diff --git a/apps/host/server/agent/README.md b/apps/host/server/agent/README.md index c0bd3fc70..b05bebcf2 100644 --- a/apps/host/server/agent/README.md +++ b/apps/host/server/agent/README.md @@ -1,5 +1,8 @@ # Agent bridge (Strata chats as DeepSeek Harness sessions) +This page is for developers who work on the app host's agent code. It lists the files, the frames and routes, and the rules the +bridge enforces. Tests are listed at the end. + The Harness is a separate, pinned component (`DSH 0.2.1-alpha.3`, commit `57190202`, MIT). The host finds it through `STRATA_HARNESS_DIR` (or `/harness`), starts one `dsh --profile web` process on a loopback port, and talks to its Gateway. Nothing here downloads or builds anything. @@ -14,12 +17,13 @@ and talks to its Gateway. Nothing here downloads or builds anything. | `children.mjs` | pure projection of one helper agent's events into its row and its approval card | | `preset.mjs` | the agent preset (which tools the chat has) | | `plugins/strata-gate.mjs` | the Manual-mode approval gate (a Harness plugin, loaded by path) | +| `plugins/strata-search.mjs`, `plugins/strata-models.mjs` | the app's web search registered with the Harness; helper agents on a named model | | `plugins/strata-context.mjs`, `context-budget.mjs`, `context-fit.mjs` | measured context pressure, summaries that fit the window, the check before a message goes out | | `agent-bridge.mjs` | the chat-to-session map, turns, approvals, Stop, status | ## Frames sent on `/api/chat/stream` for an agent turn -`queued`, `start` (`agent:true`), `reasoning`, `delta`, `agent-tool` (new: upserts a tool card by call id with +`queued`, `start` (`agent:true`), `reasoning`, `delta`, `agent-tool` (upserts a tool card by call id with `running | proposed | done | denied | error`), `done` (usage, `finishReason`, `cancelled`), `error`, and `notice` (why a chat is not using the agent). The chat UI renders them with its existing components. @@ -70,7 +74,7 @@ The chat's Mode menu is real Harness behaviour. Each turn the page sends the cha What the person chose is saved by `tools-settings.mjs` (inside the host's settings file) and shown by `src/components/SettingsTools.tsx` (`GET/PUT /api/agent/tools`, `POST/PATCH/DELETE /api/agent/tools/custom[/id]`). A change reaches the agent from the next message: the profile carries a `toolsSignature`, and the bridge restarts the Harness (after the running turns end) when it differs. - Tool groups (`tools-catalog.mjs`): each is one switch in plain words and is one or more preset plugins (`withoutDisabled`); `fetch` and `search` are the two halves of the Harness's `tool-web` (both off drops it). Goals off: `/goal` answers that goals are turned off. -- Web search needs no account. The default is DuckDuckGo's HTML result page (`../web-search-ddg.mjs`), with Bing's plain result page (`../web-search-bing.mjs`) as the second source. Manners, because DuckDuckGo refuses a program that asks fast (a real box: two searches in 1.2 s answered, the third got HTTP 202 and every DuckDuckGo page refused for many minutes): a browser User-Agent and the page's own request shape (Referer, Origin, Sec-Fetch-*), one request at a time per source, 2.5 to 4 s apart (random), at most 8 a minute, a 30 minute session cache per question, a cooldown after a refusal that doubles (1, 2, 4 ... 10 minutes, `Retry-After` honoured) during which that source gets no request, no lite-page follow-up after a refusal, one retry for a timeout or a 5xx, a 15 s limit. When every source refuses, the tool answers one plain sentence (`LIMITED_SENTENCE`: "Web search is limited right now. Try again in a minute, or add a search key (Exa, Perplexity or DeepSeek) in Settings -> Tools -> Web search.") plus an instruction to the model not to retry; the turn guard in `agent-bridge.mjs` (see Turn guard) stops a model that retries anyway. **The results come from DuckDuckGo or Bing**; they are not ranked or filtered here. The person may switch to Exa, Perplexity or DeepSeek with their own key (`../web-search-keyed.mjs`; DeepSeek uses the Harness's own provider): the key is saved in the settings file, reaches the agent only as the environment variable `STRATA_SEARCH_KEY`, never the profile file and never the screen (it only learns `keySet`). A keyed service without its key falls back to DuckDuckGo. The profile pins `web.searchProvider` to the chosen id so two usable backends cannot make the choice ambiguous. +- Web search needs no account. The default is DuckDuckGo's HTML result page (`../web-search-ddg.mjs`), with Bing's plain result page (`../web-search-bing.mjs`) as the second source. Manners, because DuckDuckGo refuses a program that asks fast: a browser User-Agent and the page's own request shape (Referer, Origin, Sec-Fetch-*), one request at a time per source, 2.5 to 4 s apart (random), at most 8 a minute, a 30 minute session cache per question, a cooldown after a refusal that doubles (1, 2, 4 ... 10 minutes, `Retry-After` honoured) during which that source gets no request, no lite-page follow-up after a refusal, one retry for a timeout or a 5xx, a 15 s limit. When every source refuses, the tool answers one plain sentence (`LIMITED_SENTENCE`: "Web search is limited right now. Try again in a minute, or add a search key (Exa, Perplexity or DeepSeek) in Settings -> Tools -> Web search.") plus an instruction to the model not to retry; the turn guard in `agent-bridge.mjs` (see Turn guard) stops a model that retries anyway. **The results come from DuckDuckGo or Bing**; they are not ranked or filtered here. The person may switch to Exa, Perplexity or DeepSeek with their own key (`../web-search-keyed.mjs`; DeepSeek uses the Harness's own provider): the key is saved in the settings file, reaches the agent only as the environment variable `STRATA_SEARCH_KEY`, never the profile file and never the screen (it only learns `keySet`). A keyed service without its key falls back to DuckDuckGo. The profile pins `web.searchProvider` to the chosen id so two usable backends cannot make the choice ambiguous. - The browser: Playwright's MCP server (`@playwright/mcp`, staged next to the Harness by `apps/desktop/scripts/stage-harness.mjs`) on the person's own Chrome, Edge or Chromium (`browser-probe.mjs`), `--isolated` so it never sees their profile. Tools appear as `mcp__browser__browser_*`. In Manual, opening a page, clicking, typing, uploading, running a script in the page and closing tabs ask (plain reasons: "Open the web page ...", "Click on the page: ..."); looking (snapshot, screenshot, console, network, waiting, listing tabs) does not. Plan refuses every connected tool. The screen says plainly when no browser, or no tool server, is found. - The person's own tools: `instructions` (the tool returns their text; look-only, never asks), `command` (runs their command with the assistant's text in the `STRATA_INPUT` environment variable, never spliced into the command line; asks in Manual, refused in Plan) and `server` (a web address or a program, added as `mcp__custom-__*`; asks in Manual). Instruction and command tools are served by `custom-tools-mcp.mjs` (stdio MCP) from a JSON file the settings module rewrites on every change, so a switch in Settings is seen without a restart. - `strata-gate.mjs` implements the Manual/Plan rules above (`planDecision`, `browserDecision`, the look-only instruction server). @@ -81,7 +85,7 @@ What the person chose is saved by `tools-settings.mjs` (inside the host's settin - A helper's request for approval is the same card (state `proposed`, id `:`), answered with the same `/api/chat/approval`; a chat can only answer its own helpers. - The run (the chat's one NDJSON response) stays open until the parent is idle, every helper has finished and reported back, and no armed goal is waiting; `done` is sent then. - Stop: pauses an active goal, interrupts every helper (`subagents/interruptByParent`; `session/cancel` on a helper session is refused by the Harness), cancels the parent, waits; a report that would wake the parent afterwards is cancelled too. -- The number of helpers at once equals the engine's request slots (`maxActiveSubagents` in the profile); the Harness refuses more with a message the model reads. +- The number of helpers at once is the engine's request slots, with a floor of 4 for a local model and 8 for an API model (`maxActiveSubagents` in the profile). The Harness refuses more with a message the model reads. ## /goal @@ -90,7 +94,7 @@ What the person chose is saved by `tools-settings.mjs` (inside the host's settin ## Thinking and the output budget - A local route is reasoning-capable (`reasoning: true`) with `compat.thinkingFormat: 'chat-template'`: the request carries `chat_template_kwargs {enable_thinking, reasoning_effort}`, which the Strata server reads (a top-level `enable_thinking` is ignored by it, which is why pi-ai's `qwen` format left the model thinking). The chat's Reasoning menu travels as `thinking` in the stream body; the bridge sends it with `session/selectModel` (`reasoningEffort`) whenever the model or the level changed; a level a route does not have is dropped (never an error). API routes keep the provider default (the menu is disabled for them in the UI). -- `localOutputBudget(context)` = context minus 4096 (it was context/4 capped at 8192, which cut big answers off at exactly 8192 tokens); pi-ai sends the window minus the prompt. App-loaded models get `fit_max_tokens: true` so the engine clamps instead of answering 400. Sampling is not set here: the Harness sends none, so the server's own defaults apply. +- `localOutputBudget(context)` is 30 percent of the window, at least 4096 and at most 32768 (9,830 tokens at 32K, 19,660 at 64K). The Harness's automatic compaction reads it as the room a reply reserves. pi-ai sends the window minus the prompt. App-loaded models get `fit_max_tokens: true` so the engine clamps instead of answering 400. Sampling is not set here: the Harness sends none, so the server's own defaults apply. ## Tasks @@ -108,7 +112,7 @@ Failures the person sees are plain sentences with a next step (`AgentUnavailable ## Helper agents on a named model (`plugins/strata-models.mjs`, `model-routes.mjs`) -The Harness's `subagent` tool takes an exact `provider` route key and model id, and the app had its model choice switched off, so every helper ran on the chat's model. The profile now enables the Harness's `subagent-model-selection-settings` with every route the app can reach (local instances and saved API models), and the `strata-models` plugin wraps the two tool definitions the Harness registers for each chat (`subagent`, `list_subagent_models`) in place: +The Harness's `subagent` tool takes an exact `provider` route key and model id. The profile enables the Harness's `subagent-model-selection-settings` with every route the app can reach (local instances and saved API models), and the `strata-models` plugin wraps the two tool definitions the Harness registers for each chat (`subagent`, `list_subagent_models`) in place: - the model is found from its id, `provider/id`, the name in the model picker, `connection name`, or "local" (any letters). A close guess is not started: the tool answers "Did you mean ...?" and nothing runs until the call is made again with that model. Several fits are listed; nothing fitting lists the models that can be used and says they can be added in Models. - the tool's own text says when to use it ("ask a subagent", "ask model X"), not to look for the model with shell commands, and not to wait with sleep. The system prompt (`preset.mjs`) says the same, so it reaches every model, local or hosted. @@ -126,7 +130,7 @@ Pictures are on by default for every model that can read them: an API model read ## Turn guard (`turn-guard.mjs`) -A turn that keeps failing the same way is stopped by the app, never left to the model. Seen on a search-heavy `/plan`: every web search was refused, the model searched again and rewrote its plan after each one until the context reached 100%, then the turn ended with no answer and no message. +A turn that keeps failing the same way is stopped by the app, never left to the model. - **Counted, from what the turn shows:** web searches that failed in a row (3), searches in one turn (10 in a plan turn, 25 otherwise), a plan turn that starts its numbered list from `1.` again (4 times), and a context of 93% or more (the Harness's own compaction did not keep up). - **Stopped the way Stop stops** (`session/cancel`, helpers first) but ended as a normal turn (`done`, not cancelled, no error card), so the plan the model had written is still proposed. The plan card carries the reason (`plan.caution`); any other turn ends with one `delta` frame that has `own: true` and `key` (the English sentence, translated by the page; `GUARD_NOTES` are in the interface catalog in all eight languages). Each sentence says what happened and gives one next step. @@ -134,9 +138,7 @@ A turn that keeps failing the same way is stopped by the app, never left to the - **Never silent:** a turn that ends (`completed` or `max-tokens`) with no answer text gets "I ran out of room in this chat..." when the last request used 85% of the window or more, otherwise "I stopped without writing an answer..." (not when tool cards were shown, not after Stop, not after an error, which says why by itself). Goal runs have their own ending lines and are not guarded. - Tests: `tests/agent-turn-guard.test.mjs` (counting, always; end to end with the real Harness, the scripted model's `searchloop` and `bloat` scripts and `searchMock: 'limited'` on the bridge, which runs the real search code on a pretend web that refuses, no network). -## Context: measured pressure, summaries that fit, the check before a message goes out (2026-10-22) - -Found by the owner's hands-on test of the installed 0.2.1 (Gyro-S, 64K): a 118 KB log paste was 66,500 tokens to the engine and about 29,500 to the Harness (4 characters a token; logs and code are about 2); the engine refused it with a 400, the refusal stayed in the history, and then neither the next message nor `/compact` could go through, because the summary request carried the same history and was refused for its size too. `/compact` itself was never a command of the app: it went to the model as a question and the model answered "nothing to compact". +## Context: measured pressure, summaries that fit, the check before a message goes out - **`plugins/strata-context.mjs`** (inserted by the profile, like the gate) wraps `tokenMeter.measure` and `llm.stream` in place, once each. Pressure = the last prompt size the model server reported for an answer, plus what changed since priced at the density that measurement showed (`context-budget.mjs`: `pressureNow`, `densityOf`); never lower than the Harness's own price. Node prices are scaled the same way, so the retained tail is in real tokens. A summary request (`purpose: 'compaction'`) that would not fit the window is asked for in pieces: a message bigger than a piece is cut for the request only, each piece builds on the summary before it, a piece the server refuses is halved and tried again. `/strata-context` (a Harness command, like `/strata-gate`) answers `{tokens, window}` for a session so the host can tell the person true figures. - **`context-fit.mjs`**: before a message is sent the model server counts it (`POST /v1/messages/count_tokens`, its own tokenizer). `decideFit`: send; summarise first and then send (the message fits an otherwise empty chat); or never fits. The last one is not sent: the error frame has `code: 'CONTEXT_MESSAGE_TOO_LARGE'` and the page offers "Attach as a file" (`oversize.mjs`, `/api/chat/oversize`: the text is saved in the chat's folder, the assistant reads it in parts) or "Send a shortened preview". An engine that cannot count lets the message through; the Harness's own compact-and-retry is the net. @@ -144,3 +146,15 @@ Found by the owner's hands-on test of the installed 0.2.1 (Gyro-S, 64K): a 118 K - **Visible:** `compaction/start` shows "Summarizing earlier messages..." in the status line; `compaction/end` adds one plain line to the reply. The projector also sends a `context` frame after every model request, so the meter follows the running turn. - **A refusal that is left** (`CONTEXT_WINDOW_EXCEEDED`) is a plain sentence with the engine's text under Details and the buttons "Summarize and continue" and "Start a new chat". "Try sending the message again" is for other errors only. - Tests: `tests/agent-context.test.mjs` (arithmetic; the real Harness against the scripted worker with `window`, `charsPerToken` and `noCount`), `tests/oversize.test.mjs`, `tests/context-ui.test.mjs`. The run against a real engine is `scripts/context-e2e.mjs`. + +## History / older measurements + +- **Search refusals (2026-10):** on a real box two DuckDuckGo searches in 1.2 s answered, the third got HTTP 202, and every DuckDuckGo page refused for many minutes. + The pacing, cooldown and cache rules above come from that test. +- **Search-heavy `/plan` (2026-10):** every web search was refused, the model searched again and rewrote its plan after each one until the context reached 100%. + The turn ended with no answer and no message. The turn guard comes from that run. +- **Large log paste (2026-10-22, installed 0.2.1, Gyro-S, 64K):** a 118 KB log was 66,500 tokens to the engine and about 29,500 to the Harness + (4 characters a token; logs and code are about 2). The engine refused it with a 400. The refusal stayed in the history, so neither the next message nor `/compact` could + go through, because the summary request carried the same history. `/compact` was also not a command of the app yet: it went to the model as a question. This + produced `strata-context.mjs`, `context-fit.mjs` and the host's own `/compact`. +- **Output budget:** an earlier local budget of a quarter of the window, capped at 8192, cut big answers off at exactly 8192 tokens. diff --git a/docs/AMD_HIP.md b/docs/AMD_HIP.md index dc401cf4d..29ef51790 100644 --- a/docs/AMD_HIP.md +++ b/docs/AMD_HIP.md @@ -1,209 +1,178 @@ -# AMD Radeon: the HIP backend (gfx1100, gfx1101, gfx1200, gfx1201, gfx1030) +# AMD Radeon: the HIP backend -Strata runs on AMD Radeon cards through its HIP backend, the same engine as on NVIDIA compiled for AMD. This page -covers the build on Linux (on Windows a ready-made engine, see [Windows](#windows)) for the RX 7900 XT / XTX (RDNA3, gfx1100) and the -RX 9070 / 9070 XT / Radeon AI PRO R9700 (RDNA4, gfx1201; see [RDNA4](#rdna4-gfx1201)). The RX 7800 XT / 7700 XT -(gfx1101) and the RX 9060 XT (gfx1200) were validated by their owners (see [Community-validated -cards](#community-validated-cards)); the RX 6800 / 6900 series (RDNA2, gfx1030) builds and runs too, reported by a community machine and not yet validated by the maintainers (see [RDNA2](#rdna2-gfx1030)). Setup chooses it by itself on a PC with no NVIDIA card Strata can use (`--backend hip` on a PC with both); the -install steps for users are in [INSTALL.md](INSTALL.md#amd-cards). gfx906 (Instinct MI50 / MI60, Radeon VII; wave64) has a separate -opt-in build, see [gfx906](#gfx906-instinct-mi50--mi60-radeon-vii-wave64-built-from-source). The Ryzen AI Max "Strix Halo" APU (gfx1151, RDNA3.5, unified memory) has its own page: [STRIX_HALO.md](STRIX_HALO.md). The ready-made 0.2.0 installers and engine packages for AMD cards are listed in [Ready-made packages](#ready-made-packages-020). Other AMD architectures and mixed -AMD/NVIDIA execution in one run are not supported. +Strata runs on AMD Radeon cards through its HIP backend, the same engine as on NVIDIA compiled for AMD. This page is for +people with an RX 6000, 7000 or 9000 card or a Radeon AI PRO R9700, and for anyone building the engine from source. +The Ryzen AI Max "Strix Halo" APU has its own page: [STRIX_HALO.md](STRIX_HALO.md). Integrated Radeon chips in mini PCs are +covered under [Integrated GPUs](#integrated-gpus-mini-pcs). -The backend maps the CUDA-shaped runtime and BLAS calls to HIP/hipBLAS, uses -RDNA2/RDNA3/RDNA4's signed integer dot instruction for quantized kernels, and supplies -wave32 shuffle/packed-byte operations. CUDA-only QSA matrix instructions have -an ordered FP32 fallback. Prefill supports both dequantization plus hipBLAS GEMM and opt-in HIP ggml MMQ. -An optional, calibrated hipBLASLt path accelerates dense projections (per-architecture tables in `tools/hip`). -This does not claim bit-identical model answers across backends. See -[performance settings and evidence](AMD_HIP_PERFORMANCE.md). +## Quick start -## Install with setup (recommended) +Use the matching [desktop app or Terminal](INSTALL.md). The engine package carries the HIP runtime, so only the AMD +graphics driver must be installed (Windows: [AMD Software: Adrenalin Edition](https://www.amd.com/en/support/download/drivers.html); +Linux: the kernel's amdgpu driver, no ROCm install). Terminal: -On Linux with an RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT or Radeon AI PRO R9700 and -the kernel's amdgpu driver (no ROCm install needed): +```sh +strata launch --plan # shows what would happen, changes nothing +strata launch # sets up and serves +``` + +Source checkout on Linux ([SOURCE_INSTALL.md](SOURCE_INSTALL.md#amd-cards)): ```sh ./setup.sh --backend hip ``` -- **Detection:** setup finds the card through the kernel's KFD topology. Integrated Radeon GPUs are listed as not - supported. On a PC without an NVIDIA card Strata can use, `--backend hip` is chosen automatically. -- **ROCm:** a system ROCm 7 in `/opt/rocm` (or `$ROCM_PATH`) with hipcc and hipBLAS is used when present. Otherwise - (or when it is older than 7.0) ROCm is installed into `.venv` from AMD's TheRock wheels (~10 GB, no sudo), from the - card family's index: `gfx110X-dgpu` for gfx1100 / gfx1101, `gfx120X-all` for gfx1200 / gfx1201, `gfx103X-all` for - gfx1030, `gfx1151` for Strix Halo. The version is chosen per family: 7.10.0a20251120 where that is on the index, - 7.13.0a20260515 for `gfx103X-all` (its index starts at 7.13, #1103) and 7.14.0a20260608 for `gfx1151` (the 7.10 wheel - segfaults on kernel 7.2.8, #1267). When the index no longer offers the version, setup warns and takes the newest of - the same 7.x line, else of the same major. `STRATA_ROCM_VERSION` / `STRATA_ROCM_INDEX` override them; the gfx1030 - wheels are unvalidated (a system ROCm 7 is the tested path there). -- **Engine:** compiled on your PC for the card's architecture (10-20 minutes, once; again after a `git pull` that - changes it, or when you pick a card of another architecture). This needs a C++ compiler and git - (`sudo apt install build-essential git`). -- **hipBLASLt tuning table:** setup uses `tools/hip/-hipblaslt-.txt` only when it matches both the - card's architecture and the installed hipBLASLt version (read from `hipblaslt-version.h`; 1.2.0 is `100200`). - Otherwise it says so and the prompt's dense matrix products use plain hipBLAS (slower prompts, same answers). - A table's solution ids are valid only for that pair, and the engine refuses any other table. On gfx103x (RDNA2, - no hipBLASLt kernels) the same role falls to a rocBLAS table, `tools/hip/-rocblas-.txt`, which the - engine picks for its own rocBLAS build when setup points it at the directory ([RDNA2](#rdna2-gfx1030)). -- **Several cards:** setup takes one card (the one with the most VRAM, or `--gpu N`) unless you name more: - `./setup.sh --backend hip --gpus 1,0` splits the model's layers across them, the faster card goes last and is the Main card (`--gpu-roles 1=main,0=helper` says it yourself; docs/MULTI_GPU.md) (numbers - as setup lists them; `--gpus all` = every supported card, the most VRAM first). Every chosen card must be one of the - architectures above; the engine is compiled for each of them (cards of two families, e.g. gfx1100 + gfx1201, need - a system ROCm 7: AMD's wheels hold one family). A split pays only when no single card holds the model's experts - (see RDNA4 below). -- **Pictures:** on by default; the reader is the CPU one under 24 GB of VRAM and the GPU one from 24 GB (the same rule on NVIDIA and AMD, `strata_control/vision_reader.py`); `--vision gpu` / `--vision cpu` switch it. The engine package carries two picture readers: `strata-vision-hip`, which reads a picture on the - card (llama.cpp's HIP backend; about 1.4 GiB of VRAM, measured on an R9700: 0.06 s for a 260-token picture and 0.4 s for 972 - tokens, against 3.1 s and 14 s on the CPU), and `strata-vision`, the CPU reader. If the GPU reader cannot start (the card is - not one it was built for, the HIP runtime does not load, the card is busy) the server starts the CPU reader by itself and - prints one line; nothing is asked. `--vision cpu` picks the CPU reader on purpose. -- **Limits for now:** Setup does not offer the tuning - (calibration) on AMD yet: its controls are being checked on HIP one at a time (#566). Since 0.1.39 a tuning run by - hand (`./setup.sh --calibrate`) is saved for the AMD card it ran on and reused when setup runs again. The Monitor - shows the card's load, VRAM, temperature and power from Linux sysfs (0.1.32). +Setup picks `--backend hip` by itself on a PC with no NVIDIA card Strata can use. Other AMD architectures and mixed AMD/NVIDIA +runs are not supported. Use the [package table](#ready-made-packages) to match your card. -The rest of setup is the same as on NVIDIA: the model download, the start script, the server. +## Supported cards -## Ready-made packages (0.2.5) +| Card | Architecture | Status | +| --- | --- | --- | +| RX 7900 XT / XTX | gfx1100 | Maintainer-tested | +| RX 9070 / 9070 XT, Radeon AI PRO R9700 | gfx1201 | Maintainer-tested ([RDNA4](#rdna4-gfx1201)) | +| RX 9060 XT | gfx1200 | Validated by its owner | +| RX 7800 XT / 7700 XT | gfx1101 | Validated by its owner | +| RX 7600 XT | gfx1102 | Reported working, not validated | +| RX 6800 / 6900 series, RX 6700 XT | gfx1030, gfx1031 | Has a package, not tested on a real card ([RDNA2](#rdna2-gfx1030)) | +| Ryzen AI Max (Strix Halo) | gfx1151 | [STRIX_HALO.md](STRIX_HALO.md) | +| Radeon 780M / 760M | gfx1103 | Linux, opt-in build: [RADEON_780M.md](RADEON_780M.md) | +| Instinct MI50 / MI60, Radeon VII | gfx906 | Source build, community-tested ([gfx906](#gfx906-instinct-mi50--mi60-radeon-vii-wave64-built-from-source)) | -The release carries one AMD engine package per card family. The Terminal one-liner and `strata update` pick the package whose -GPU code covers the card. The app installer (`Strata-0.2.5-windows-x64-setup.exe`, `Strata-0.2.5-linux-x86_64.AppImage`) has no engine -inside: on its first start the app downloads the matching package for the card, or you press **Update engine** in Settings (or run `strata update`). +Strix Point (890M / 880M, gfx1150) and Krackan (860M / 840M, gfx1152) are named and not supported by setup. RDNA1 +(gfx1012) builds by hand: [OLDER_GPUS.md](OLDER_GPUS.md#amd-building-gfx906-and-gfx1012). + +DeepSeek V4.1-Flash runs on the Radeon AI PRO R9700 on Linux, in the app and in the Terminal. The RX 9070 (gfx1200) engine package +carries it but it is untested. Details: [MODELS.md](MODELS.md#deepseek-v41-flash). + + +## Ready-made packages + +The Terminal one-liner and `strata update` pick the package whose GPU code covers the card. The app installer +(`Strata-0.2.5-windows-x64-setup.exe`, `Strata-0.2.5-linux-x86_64.AppImage`) has no engine inside: on its first start the app +downloads the matching package for the card. You can also press **Update engine** in Settings (or run `strata update`). | Package (variant) | OS | GPU code inside | ROCm | | --- | --- | --- | --- | | `rx9070` | Windows | gfx1200, gfx1201 | 10.2.0a20260930 (TheRock) | | `rdna3` | Windows | gfx1100, gfx1101, gfx1102, gfx1151 | 10.2.0a20260930 | -| `rdna2` (0.2.2) | Windows | gfx1030, gfx1031 | 10.2.0a20260930 | +| `rdna2` | Windows | gfx1030, gfx1031 | 10.2.0a20260930 | | `rx9070` | Linux | gfx1200, gfx1201 | 7.10.0a20251120 (`gfx120X-all` wheels) | | `rx7900` | Linux | gfx1100, gfx1101, gfx1102 | 7.10.0a20251120 (`gfx110X-dgpu` wheels) | | `halo` | Linux | gfx1151 | 7.14.1 (TheRock) | -| `rx6000` (0.2.2) | Linux | gfx1030, gfx1031 | 7.13.0a20260515 (`gfx103X-all` wheels) | +| `rx6000` | Linux | gfx1030, gfx1031 | 7.13.0a20260515 (`gfx103X-all` wheels) | -`rdna3` and `halo` were added to the release after the first upload (feed sequence 101). Why the split: on Windows one ROCm 10.2 library -set holds the kernels of every listed architecture, so RX 7000 and Strix Halo share a package. The Linux 7.10 wheels hold one chip family -per library set, and 7.10 crashes in the HSA runtime on a Strix Halo with a new kernel (#1267), so Linux Strix Halo has its own package on 7.14.1. +Each package carries the engine, the CPU and GPU picture readers for its architectures, and the rocBLAS and hipBLASLt kernel +data for them only. A package whose GPU code does not cover the card is refused with one line that names the card family. +The RX 6000 packages are offered only to a PC whose card reports gfx1030 or gfx1031. They run the plain kernels: RDNA2 has no +matrix cores, and hipBLASLt ships no gfx1030 kernels, so there is no tuning table. -Each package carries the engine, the CPU picture reader and the GPU picture reader (`strata-vision-hip`) for the same architectures, and -the rocBLAS and hipBLASLt kernel data for them only. A package whose GPU code does not cover the card is refused with one line that -names the card family ("This build of Strata is for AMD RX 9000 cards. This computer has an AMD RX 7000 card ..."). -**The RX 6000 series (gfx1030 / gfx1031, RX 6800 / 6900 / 6700 XT) has its own package from 0.2.2** (below). The 0.2.1 packages have no code for it, -and the app said so plainly instead of naming a build to ask for. +**Automatic card.** With no card named in the model's settings, the engine start asks the package's +`strata-device --list-devices` and skips cards the package has no code for and integrated chips, while a card of its own can +run. `HIP_VISIBLE_DEVICES`, a `"gpu"` in the settings file or `hip_ordinal` is never overridden. -**RX 6000 package (0.2.2): built, not tested on a real RX 6000 card.** `rx6000` on Linux (ROCm 7.13.0a20260515, the `gfx103X-all` wheels -setup already uses for these cards) and `rdna2` on Windows (ROCm 10.2.0a20260930), each with the code for gfx1030 and gfx1031 only. It is a niche, -opt-in build: the feed offers it to a PC whose card reports gfx1030 or gfx1031 and to no other (a PC with an RX 7000 / 9000 card beside an RX 6000 -one keeps the package it always had), and the RX 7000 / 9000 / Strix Halo packages are unchanged. It runs the plain -kernels: RDNA2 has no matrix cores, so none of the WMMA paths, the gfx12 prompt defaults or the 780M prompt set apply (they are chosen by the card's -architecture, `src/core/arch_defaults.cpp`), and hipBLASLt ships no gfx1030 kernels, so there is no tuning table and the dense matrix products use plain -hipBLAS. Linux build: `VARIANT=rx6000 packaging/build_hip_linux.sh`; Windows: `STRATA_HIP_ARCHS="gfx1030;gfx1031" tools\hip\build_windows.bat`, then -`package_engine.py --variant rdna2`. The [RDNA2 section](#rdna2-gfx1030) has what community owners measured with engines built from source. Please -report a run on an RX 6000 card with the end of `strata-.log`. +**Tuning tables.** Each package carries the hipBLASLt tuning tables of `tools/hip` that match the hipBLASLt it bundles, as +`tuning/-hipblaslt-.txt`: -**Automatic card (0.2.1):** with no card named in the model's settings, the engine start asks the package's `strata-device --list-devices` -and skips cards the package has no code for and integrated chips (a mini PC's Radeon 760M / 890M is HIP's device 0) while a card of its own -can run. Device 0 stays the choice when it is already such a card. When none of the computer's cards can run, the start stops -with one line that names the cards found and what the engine is built for. `HIP_VISIBLE_DEVICES`, a `"gpu"` in the settings file or `hip_ordinal` is never overridden. +| Package | Table | +| --- | --- | +| Windows `rx9070` | gfx1201 (hipBLASLt 1.5.0) | +| Windows `rdna3` | gfx1100, gfx1151 (1.5.0) | +| Linux `rx7900` | gfx1100 (1.2.0) | +| Linux `halo` | gfx1151 (1.4.1) | +| Linux `rx9070`, `rx6000`, Windows `rdna2` | none | -What was run on real hardware: +A table changes which GEMM algorithm runs the prompt, so rounding differs and answers are not bit-identical to plain +hipBLAS. Packaged engines therefore use the table by default only on Strix Halo (gfx1151). On every other card it is off +until you ask: -- **Checked on every package** (no RX 7000 or Strix Halo card involved): it builds and links; every library the binaries import is in the - package or belongs to the OS (`packaging/binary_deps.py`); `package_engine.py --verify` passes; the GPU code inside each binary and - library lists exactly the architectures above (`llvm-objdump --offloading`); `strata --version` runs in a clean environment (on Windows with the package's `rocm\bin` folder on PATH, which Control sets); on a - gfx1201 card the Linux `halo` and `rx7900` builds refuse to start with the plain message (the Windows `rdna3` build was not run on an RX 9000 card: there is no Windows AMD machine); the feed picks the right package for each - architecture (`tests/terminal/test_release_matrix.py`). -- **Not run by the maintainers:** any model on an RX 7000 card (gfx1100-1102) or a Strix Halo (gfx1151), on either OS. The maintainers have - no such card. -- **Run by other people:** an RX 7900 XTX on Windows 11 ran the earlier ready-made Windows engine (0.1.40; [report #1173](../bench/results/2026-10-06-community-rx-7900-xtx/README.md)); - RX 7900 XTX, XT and GRE owners ran Linux builds ([COMMUNITY.md](../bench/results/COMMUNITY.md)); Strix Halo owners ran Linux builds made from source - (ROCm 7.14 line). Nobody has reported a Strix Halo on Windows, or the packaged `halo` and `rdna3` engines. -- **Tuning tables (0.2.1):** each AMD package carries the hipBLASLt tuning tables of `tools/hip` that match the hipBLASLt it bundles, as - `tuning/-hipblaslt-.txt` (listed and hashed in the package manifest), and records that version as `hipblaslt_version` in its - BUILD.json. Windows `rx9070` has the gfx1201 table, Windows `rdna3` the gfx1100 and gfx1151 ones (all hipBLASLt 1.5.0), Linux `rx7900` the - gfx1100 one (1.2.0), Linux `halo` the gfx1151 one (1.4.1). The RX 6000 packages (`rx6000`, `rdna2`, 0.2.2) have none, because hipBLASLt has no gfx1030 / gfx1031 kernels. Linux `rx9070` has none: its wheels carry hipBLASLt 1.2.0 and no table in `tools/hip` - is for that version (a table calibrated for it on an R9700 read prompts about 1% faster and decode 2% slower, so none is shipped). - The engine start picks the table for the card and uses it only when the package's hipBLASLt version is the table's; otherwise it logs - why and the dense matrix products stay on plain hipBLAS. **A table changes which GEMM algorithm runs the prompt, so the rounding differs: the - answers are not bit-identical to plain hipBLAS** (first-token KL 0.016 on an R9700, see below). Packaged engines therefore use the table by - default only on Strix Halo (gfx1151), where a community A/B measured prompts 2.11x faster with it (251 to 529 tokens/s); every other card - uses it only when you ask: `STRATA_HIPBLASLT_TABLE=on` (or `"hipblaslt_table": true` in the run config) uses it, `off` never does (on Strix - Halo too), and the start log - says which table was taken or why none was (`hipBLASLt tuning table: ...`). A `STRATA_HIPBLASLT_TUNING` you set yourself is always kept. - A source build is unchanged (see [Tuning table](#tuning-table)). Testers on Strix Halo and RX 7000 cards: please send that log line and a - prompt-speed run with `on` and `off`. +| Setting | Effect | +| --- | --- | +| `STRATA_HIPBLASLT_TABLE=on` (or `"hipblaslt_table": true` in the run config) | Use the packaged table when it matches | +| `STRATA_HIPBLASLT_TABLE=off` | Never use it, also on Strix Halo | +| `STRATA_HIPBLASLT_TUNING=` | Use this table file; always kept | -Please report a run on an RX 7000 card or a Strix Halo (card, driver, OS, the end of `strata-.log`). +The start log says which table was taken or why none was (`hipBLASLt tuning table: ...`). Please send that line and a +prompt-speed run with `on` and `off` from an RX 7000 or Strix Halo card. + +## Install with setup (source checkout, Linux) + +```sh +./setup.sh --backend hip +``` + +| Step | What setup does | +| --- | --- | +| Detection | Finds the card through the kernel's KFD topology. Integrated Radeon GPUs are listed as not supported (Strix Halo has its own page). | +| ROCm | Uses a system ROCm 7 in `/opt/rocm` (or `$ROCM_PATH`) with hipcc and hipBLAS when present. Otherwise, or when it is older than 7.0, installs ROCm into `.venv` from AMD's TheRock wheels (about 10 GB, no sudo). | +| Engine | Compiles it for the card's architecture (10-20 minutes, once; again when you pick another architecture). Needs a C++ compiler and git (`sudo apt install build-essential git`). | +| Tuning table | Uses `tools/hip/-hipblaslt-.txt` only when it matches both the architecture and the installed hipBLASLt version. Otherwise it says so and uses plain hipBLAS. On gfx103x the same role falls to a rocBLAS table. | +| Pictures | On by default. CPU reader under 24 GB of VRAM, GPU reader from 24 GB; `--vision gpu` / `--vision cpu` switch it. If the GPU reader cannot start, the CPU reader is used. | + +TheRock wheel index per family: `gfx110X-dgpu` (gfx1100, gfx1101), `gfx120X-all` (gfx1200, gfx1201), `gfx103X-all` +(gfx1030), `gfx1151` (Strix Halo). Default versions: 7.10.0a20251120, 7.13.0a20260515 for `gfx103X-all`, 7.14.0a20260608 for +`gfx1151`. When the index no longer offers the version, setup warns and takes the newest of the same line. +`STRATA_ROCM_VERSION` and `STRATA_ROCM_INDEX` override them. The gfx1030 wheels are unvalidated; a system ROCm 7 is the +tested path there. + +**Several cards.** Setup takes one card (the one with the most VRAM, or `--gpu N`) unless you name more. + +| Option | Effect | +| --- | --- | +| `--gpus 1,0` | Split the layers across these cards (numbers as setup lists them). The faster card goes last and is the Main card. | +| `--gpus all` | Every supported card, most VRAM first | +| `--gpu-roles 1=main,0=helper` | Say the roles yourself ([MULTI_GPU.md](MULTI_GPU.md)) | + +Every chosen card must be one of the supported architectures. Cards of two families (for example gfx1100 and gfx1201) need +a system ROCm 7, because AMD's wheels hold one family. A split pays only when no single card holds the model's experts. + +Setup does not offer tuning (calibration) on AMD. A tuning run by hand (`./setup.sh --calibrate`) is saved for that AMD card and +reused. The Monitor shows the card's load, VRAM, temperature and power from Linux sysfs. ## Windows -Since 0.1.34 an AMD card on Windows is set up like an NVIDIA one: download Strata, double-click `START-HERE.bat`. -On a PC with no NVIDIA card Strata can use, the AMD card is chosen by itself; with both, setup asks -(`START-HERE.bat --backend hip` picks AMD directly). +Install the [app](INSTALL.md) (`Strata-0.2.5-windows-x64-setup.exe` from the +[release page](https://github.com/Niko1221/Strata/releases/latest); it downloads the engine for the card), or +install the [Terminal](INSTALL.md#the-terminal) and run `strata launch`. You need Windows 10 or 11 (64-bit), a supported card +and a current AMD driver. No ROCm or HIP SDK, no compiler, no admin rights. `START-HERE.bat --backend hip` is a +source-checkout option for developers, not a way to start the released app. -- **You need:** Windows 10 or 11 (64-bit), one of the cards above, and a current AMD driver ([AMD Software: - Adrenalin Edition](https://www.amd.com/en/support/download/drivers.html)). Nothing else: no ROCm or HIP SDK - install, no compiler, no admin rights. -- **Detection:** setup reads the display adapters Windows lists (their PCI ids; the VRAM size from the display - driver's registry entry). An integrated Radeon is listed as not supported, except the Ryzen AI Max "Strix Halo" (Radeon 8060S / 8050S / 8040S, gfx1151: [STRIX_HALO.md](STRIX_HALO.md)); a Strix Point (890M / 880M, gfx1150), Krackan (860M / 840M, gfx1152) or Phoenix / Hawk Point (780M / 760M, gfx1103) Radeon is named as what it is and not supported (on Linux the 780M / 760M has an opt-in build: [RADEON_780M.md](RADEON_780M.md)). -- **Engine (0.2.0 and later):** the installers and `strata update` use the engine packages in [Ready-made packages](#ready-made-packages-020); the zip described next is the START-HERE route of 0.1.x. -- **Engine (START-HERE, 0.1.x):** the ready-made engine package for your card, fetched from the signed update feed (releases from 0.2 carry no flat `strata-windows-x64-hip.zip`; that zip is what `build_windows.bat` makes for a self-build) (built by `tools\hip\build_windows.bat` - for gfx1100, gfx1101, gfx1102, gfx1200, gfx1201, gfx1030 and, from 0.1.40, gfx1151 (Strix Halo, unvalidated on Windows)) goes into `engine\`. It carries the ROCm libraries the - engine loads (`engine\rocm\bin`: the HIP runtime, hipBLAS / rocBLAS / hipBLASLt with their kernels for these cards, - amd_comgr and the Microsoft C++ runtime; ROCm 10.2.0a20260930 from AMD's TheRock builds, licenses in - `engine\rocm\licenses`). The HIP runtime works through the AMD driver's own components, so the driver is the one - thing it needs from the PC. -- **The HIP runtime next to `strata.exe` (0.1.35, #468 #461):** `amdhip64_7.dll` and `amd_comgr.dll` are also put in - `engine\` (setup copies them there on every start). Windows looks in the program's folder before System32, where - some AMD drivers install their own `amdhip64_7.dll`; with that one, the bundled libraries crashed on the first - prompt (an access violation, or `hipErrorInvalidDeviceFunction`). The engine's log names the runtime it loaded - (`strata generate: HIP runtime ...`). -- **Before the ~60 GB model download** setup runs `engine\strata-device.exe --list-devices` (with `engine\rocm\bin` on - the PATH): if the HIP runtime does not see the card, setup stops there and points to the driver. It also gives the - card's HIP number: with an integrated Radeon that is device 1, not 0 (#325). From then on setup lists the AMD cards - as HIP numbers them, so `--gpu N` and the config's `"gpu"` are HIP numbers. -- **Differences from Windows-on-NVIDIA and Linux-on-AMD:** one card per model (`--gpus` is Linux-only for now), no calibration. -- Two Windows-only engine details (#247, #325): hipBLAS can return success and still leave `hipErrorInvalidValue` - set after some BF16/FP16 GEMMs (seen on gfx1201); the engine clears that one stale error after a GEMM that - succeeded, on Windows only. `hipHostGetDevicePointer` returns the host pointer itself on Windows: kernels read - mapped memory through it correctly, but a device-to-device copy into it does not land, which is why - `tests/hip/handoff` times out there (the engine does not use that copy; `tests/hip/mapped_alias` reports it). -- Two more (#380, #377, by BlueKingMuch, measured on an RX 6800 that drives the desktop): `hipMemGetInfo` on Windows - does not subtract what the desktop and other programs hold on the card, so `--expert-cache auto` filled the card - past what Windows keeps in VRAM and decode fell from 41 to 30 tok/s. The engine now lowers that free figure by - what Windows' video memory budget for the process withholds (logged once: `strata: Windows budgets N of this - card's M MiB ...`; `STRATA_WDDM_BUDGET=0` turns it off). And the PCIe probe times its copies on the host clock - there, since HIP's events read impossible speeds (3,300-26,000 GB/s), so a slow link now gets a smaller - `pcie_frac` as on NVIDIA. +- **Detection:** setup reads the display adapters Windows lists. An integrated Radeon is listed as not supported, except Strix + Halo ([STRIX_HALO.md](STRIX_HALO.md)). Phoenix / Hawk Point (780M / 760M) is not supported on Windows; on Linux it has an + opt-in build ([RADEON_780M.md](RADEON_780M.md)). +- **HIP runtime:** the engine folder holds its own `amdhip64_7.dll` and `amd_comgr.dll`, because Windows looks in the + program's folder before System32, where some AMD drivers install an `amdhip64_7.dll` that crashes the bundled libraries. The + engine log names the runtime it loaded (`strata generate: HIP runtime ...`). +- **Before the model download,** setup runs `engine\strata-device.exe --list-devices`. If the HIP runtime does not see the + card, setup stops and points to the driver. With an integrated Radeon, a discrete card is HIP device 1, not 0; `--gpu N` and + the config's `"gpu"` are HIP numbers. +- **Several cards:** use the app's card selector or `strata launch --gpu 0,1`. [MULTI_GPU.md](MULTI_GPU.md) lists the + combinations measured. A tested Linux pair does not validate the same pair on Windows. +- **VRAM budget:** `hipMemGetInfo` on Windows does not subtract what the desktop and other programs hold. The engine lowers + the free figure by what Windows' video memory budget for the process withholds (logged once). `STRATA_WDDM_BUDGET=0` turns + that off. +- **WMMA prompt attention** is opt-in on Windows (`STRATA_HIP_WMMA=1`); it is the default on gfx12 on Linux. -**What is validated (0.1.34):** #325's author ran the engine of this port on an RX 9070 XT (Windows 11, ROCm -10.2.0a20260930 in `.venv`, compiled on the PC): Coder IQ1_M at 32K, 29.2 tok/s decode, ~181 tok/s prefill, -correct answers; ctest 42 of 46. The maintainers have no Windows AMD card: the release zip was built on an NVIDIA PC, -and checked there on the Ryzen CPU's integrated Radeon (gfx1036, a test build of the same tree): with only the zip's -libraries on the PATH, `strata-device` lists the card and a hipBLAS BF16 GEMM matches the CPU; the HIP ctest passes -52 of 56, the 4 failures the same as on the RX 9070 XT (`hip_handoff`, and three tests that need a pack fixture). -The ready-made zip itself has not run a model on a discrete card yet - please report. - -**Reporting a Windows AMD run** (an issue, or on #325): your card and driver version (AMD Software > System), then +**Reporting a Windows AMD run.** Give your card, the driver version (AMD Software > System), the output of the two commands +below (from the Strata folder, after `set PATH=%CD%\engine\rocm\bin;%PATH%`), the end of `strata-.log`, and the speed +lines the server window prints for a first answer. ```bat engine\strata-device.exe --list-devices engine\strata-device.exe --selftest ``` -(from the Strata folder, after `set PATH=%CD%\engine\rocm\bin;%PATH%`), the end of `strata-.log`, and the -speed lines the server window prints for a first answer. - -**Building it yourself:** `tools\hip\build_windows.bat` (Visual Studio 2022 Build Tools with the C++ workload, Python, -git; no admin, no AMD GPU) installs ROCm from AMD's TheRock wheels into `.rocm-win`, builds, and packages -`dist\strata-windows-x64-hip.zip`; `START-HERE.bat --backend hip --prebuilt dist\` installs that one. -`tools\hip\build_windows.bat tests` also builds the HIP tests (`ctest` in `build-hip-win`, with -`.rocm-win\Lib\site-packages\_rocm_sdk_devel\bin` on the PATH). `STRATA_HIP_ARCHS` picks other architectures. +**Building it yourself.** `tools\hip\build_windows.bat` (Visual Studio 2022 Build Tools with the C++ workload, Python, git; no +admin, no AMD GPU) installs ROCm from AMD's TheRock wheels into `.rocm-win`, builds, and packages +`dist\strata-windows-x64-hip.zip`. `START-HERE.bat --backend hip --prebuilt dist\` installs that one. +`tools\hip\build_windows.bat tests` also builds the HIP tests. `STRATA_HIP_ARCHS` picks other architectures; for the RX 6000 +package use `STRATA_HIP_ARCHS="gfx1030;gfx1031"` and `package_engine.py --variant rdna2`. ## Build -Requirements: a working ROCm driver/runtime, HIP development headers and -compiler, hipBLAS and (for tuned dense prefill) hipBLASLt development files, CMake 3.24+, a C++20 host compiler, and Git. -The model still needs sufficient system RAM and fast SSD storage for its PLE -table. VRAM occupancy alone is not a throughput measurement. +Requirements: a working ROCm driver and runtime, HIP development headers and compiler, hipBLAS and (for tuned dense prefill) +hipBLASLt development files, CMake 3.24+, a C++20 host compiler, and Git. The model still needs enough system RAM and a fast +SSD for its PLE table. ```sh cmake -S . -B build-hip \ @@ -213,103 +182,35 @@ cmake -S . -B build-hip \ cmake --build build-hip --target strata -j2 ``` -`CMAKE_HIP_ARCHITECTURES` is `gfx1100`, `gfx1101`, `gfx1200`, `gfx1201`, or a list such as `"gfx1100;gfx1201"` -(one binary for both). gfx1102 (the same wave32, 64 KiB LDS and dot4 instruction) builds with a warning: it passed -ctest (#192), and one RX 7600 XT (16 GB, i7-13700K, 64 GB) ran Swift IQ3_XXS over 146 requests up to 51K context at about 43 tok/s decode (28-50) and 250-300 tok/s prompt (#942); setup accepts it (unvalidated, #938); so does gfx1030 (RDNA2: the older `v_dot4_i32_i8`, a community run in #311). At startup the engine and `strata-device` compare each GPU they use -(`gcnArchName` up to the `:` feature suffix) with the architectures the binary was compiled for, and require -wave32. A binary carried to another card stops with the card's name, its architecture and the build's list, -instead of failing later with "invalid device function". +`CMAKE_HIP_ARCHITECTURES` is `gfx1100`, `gfx1101`, `gfx1200`, `gfx1201`, or a list such as `"gfx1100;gfx1201"` (one binary +for both). gfx1102, gfx1030 and gfx1031 build with a warning. At startup the engine and `strata-device` compare each GPU +they use with the architectures the binary was compiled for and require wave32. A binary carried to another card stops with +the card's name, its architecture and the build's list. -If CMake cannot find the HIP compiler, add -`-DCMAKE_HIP_COMPILER=/path/to/rocm/llvm/bin/clang++`. On the Fedora-family test -host this was `/usr/lib64/rocm/llvm/bin/clang++`. Use the compiler and libraries -from the same ROCm installation. Do not enable both GPU backends in one build. +- If CMake cannot find the HIP compiler, add `-DCMAKE_HIP_COMPILER=/path/to/rocm/llvm/bin/clang++`. Use the compiler and + libraries from the same ROCm installation. +- Do not enable both GPU backends in one build. +- CMake fetches the pinned llama.cpp revision. For an offline build, point `STRATA_GGML_DIR` at a checkout of that exact + revision. +- Linux engine packages: `VARIANT=rx6000 packaging/build_hip_linux.sh`. -Native IQ experts are enabled by default. CMake fetches the llama.cpp revision -pinned by this repository. For an offline build, point `STRATA_GGML_DIR` at a -checkout of that exact revision; using an arbitrary newer checkout changes the -dependency being tested. - -## Opt-in chunked prompt GDN on gfx11 - -`STRATA_GDN_CHUNKED=1` runs the prompt's gated DeltaNet recurrence in 16-token chunks on the gfx11 matrix cores, -with FP16 operands and FP32 accumulation, and fuses the output norm. It is off by default: rounding differs from -the token-by-token FP32 recurrence. The CUDA switch of the same name uses a separate FP32 implementation. - -The HIP path accepts gfx1100, gfx1101, gfx1102, gfx1150 and gfx1151, and prompt chunks of at least 64 tokens. -Other cards, shorter chunks, an output row stride not divisible by eight, and `STRATA_GDN_REC_HEADS` use the -existing recurrence. Runtime measurements and numerical checks were run on gfx1151 only; building for another -target does not validate its speed or answers. The synthetic test compares the FP16 output and carried FP32 state -against the existing recurrence, and short cases against a double-precision CPU calculation. It covers partial -chunks, padded rows, weak decay and a split request; the off/default tests require identical output and state bits. -It does not establish model answer equivalence. - -`STRATA_GDN_CHUNKED=2`, which on CUDA skips the SM-count test, is the same as 1 here. - -Measured on a Ryzen AI Max+ 395 (gfx1151, ROCm 7.14.1), UD-IQ4_XS. One layer's recurrence and norm take 1.60x less time than -gfx1151's default kernels at 16K and 32K tokens (`hip_gdn_chunked --bench` against `--bench-default`, three runs each: 20.92 -> -13.11 ms at 16K, 42.14 -> 26.30 ms at 32K). A whole prompt: the same binary with only this switch changed, a fresh engine per -side, 11 pairs in alternating order after a discarded one, the engine's `prompt_ms`, every prompt read in full (no reused -tokens). Each engine reads a discarded 4K prompt first, because the first prompt after a start varies more (on those first -prompts the same pairs gave 95% intervals of [1.004, 1.032] and [0.981, 1.015]); the prompts follow each other once -`/proc/vmstat`'s page-migration and direct-reclaim counters have stood still for 3 s, and a pair in which either prompt still -met page migration or direct reclaim was dropped and measured again (3 of the 69 pairs, all 32K with the switch on; kept in, 32K -gives 1.020 and 1.015). "Section 5" is the nine variables of `STRIX_HALO.md` section 5 (with `STRATA_PF_SWITCH_MIN_T=2048`); -"defaults" is the engine without them (section 4's gfx1151 defaults stay). Times are medians; the ratio is the geometric mean of -the 11 off / on ratios with a 95% bootstrap interval; "faster" counts the pairs in which the switch was faster: - -| tokens | arm | off -> on | off / on | faster | -|---:|---|---:|---:|---:| -| 4,096 | section 5 | 3.703 -> 3.641 s | 1.018 [1.016, 1.020] | 11 of 11 | -| 4,096 | defaults | 5.914 -> 5.829 s | 1.014 [1.010, 1.017] | 10 of 11 | -| 32,768 | section 5 | 26.157 -> 25.589 s | 1.021 [1.020, 1.023] | 11 of 11 | -| 32,768 | defaults | 44.082 -> 43.479 s | 1.015 [1.012, 1.017] | 11 of 11 | -| 128,000 | section 5 | 107.149 -> 104.866 s | 1.021 [1.020, 1.023] | 11 of 11 | -| 128,000 | defaults | 193.439 -> 191.055 s | 1.014 [1.012, 1.016] | 11 of 11 | - -So about 2% on top of section 5 and about 1.5% on the defaults. At 36 gated-DeltaNet layers the layer figures predict about -0.56 s for a 32K prompt; 0.57 s (section 5) and 0.60 s (defaults) were measured. - -First-token KL against the same run without the switch, 12 prompts of 1.5K-24K tokens: UD-IQ4_XS 0.0067 mean (0.044 max) on the -defaults and 0.0039 (0.027) on top of section 5; UD-Q4_K_XL 0.0063 (0.043) and 0.0040 (0.014); the same top token in all of -them (across the 12 prompts the first token takes only two distinct values). Section 5's switches alone are 0.0068 (IQ4_XS) and -0.0031 (Q4_K_XL) against the defaults, a different change that is listed for scale and not as a limit for this one. Only the -first generated token was compared: a long generation and perplexity were not measured. - -Build and run these checks in the HIP build directory: +Run the registered checks with: ```sh -cmake --build build-hip --target hip_gdn_chunked hip_gdn_rec_head -ctest --test-dir build-hip -R 'hip_gdn_(chunked|rec_head)' --output-on-failure -for i in 1 2 3; do ./build-hip/hip_gdn_chunked --bench-default; ./build-hip/hip_gdn_chunked --bench; done +cmake --build build-hip -j2 +ctest --test-dir build-hip --output-on-failure --timeout 60 -E '^(ple_parity|platform_memory_test)$' ``` -Each benchmark run reports the GPU-event median of seven runs after two warmups, the same nonzero initial state every run: -`--bench` this path, `--bench-default` gfx1151's default kernels (`STRATA_GDN_PP=2`, `STRATA_GDN_NOY=1`). The switch is read -once per process, so the two are separate runs. It times one layer's recurrence and norm, not a whole prompt. +`ple_parity` needs an external model fixture. `platform_memory_test` needs a memlock limit above the shell's default. ## Model and serving configuration -Prepare a supported model pack and its MTP runtime using the existing tools. -For OrcaRouter IQ3_XXS, follow [the explicit compatibility conversion](ORCA.md); -this backend does not change quantization, model licenses, or tokenizers. -Add `--experts-bin` to the `tools/iq_pack.py` command: mmap requires the pack's -`experts.bin`, which the default native packing command does not emit. This -consumes additional disk space. Original GSQ-RCO IQ3_XXS uses shard 2 for PLE; -Orca uses shard 1. Keep each model's own pack and tokenizer together. +Prepare a supported model pack and its MTP runtime with the existing tools. For OrcaRouter IQ3_XXS, follow the +[compatibility conversion](ORCA.md). Add `--experts-bin` to the `tools/iq_pack.py` command: mmap requires the pack's +`experts.bin`, which the default native packing does not emit and which uses more disk. Keep each model's pack and tokenizer +together. -Use `build-hip/strata` as the executable in the server JSON. Select the AMD -device with `ROCR_VISIBLE_DEVICES`/`HIP_VISIBLE_DEVICES` if necessary. Start with -a modest context and prefill chunk size before measuring larger workloads. -Keep the PLE file on an SSD. Do not assume a configured context length proves -successful full-window inference. - -Large pinned host allocations can fail on ROCm even when ordinary RAM is -available. The existing `--mmap-experts` path avoids allocating the full pinned -expert arena; it still depends on OS file-cache residency and may stall on -storage reads. It does not make SSD access equivalent to RAM. - -Starting args for the original GSQ-RCO IQ3_XXS model (replace the paths): +Start args for the original GSQ-RCO IQ3_XXS model (replace the paths): ```sh build-hip/strata --serve \ @@ -322,516 +223,290 @@ build-hip/strata --serve \ --adapt-every 0 --pcie-frac 0 --vram-reserve-mib 1024 ``` -`--serve` is the engine's internal token protocol. For a browser or OpenAI API, -put these args in the `args` array of the [server JSON example](ORCA.md#local-server), -set `exe` to `build-hip/strata`, and use the matching pack's `tokenizer` directory. -Launch with `.venv/bin/python -m serve.server --engine strata --config strata-hip.json --port 8080`. -The worker count above was used on a 16-core CPU; measure it for your CPU. -The 4K context is a smoke-test starting point, not a model limit. The expert cache -sizes itself automatically and leaves 1 GiB of VRAM headroom. +`--serve` is the engine's internal token protocol. For a browser or OpenAI API, put these args in the `args` array of the +[server JSON example](ORCA.md#local-server), set `exe` to `build-hip/strata`, and use the matching pack's `tokenizer` +directory: -**The card also drives a Linux desktop (#560 #516):** keep more VRAM free than the default 700 MiB, e.g. -`./setup.sh --vram-reserve-mib 3072`. With the cache filling the card, the desktop's next VRAM need makes amdgpu move -GPU memory to system RAM (GTT), the OOM killer then ends KWin/plasmashell or `systemd-oomd` ends apps, or the -compositor fails with "Failed to pin framebuffer with error -12". - -The installer supports this backend (see "Install with setup" above). Pictures are on by default. They are read on the CPU (`strata-vision`) under 24 GB of VRAM and on the card (`strata-vision-hip`) from 24 GB, the same rule as on NVIDIA; if the engine has no card reader, or it cannot start, the CPU reader is used. -Setup installs one AMD card, or several with `--gpus` (the engine's layer split; see RDNA4 below). - -**Opt-in switches for RDNA2 prompts and hangs (0.1.40, off by default):** `STRATA_DENSE_MMQ=1` (#820, HIP) runs the -prompt's dense GGUF projections through the int8 MMQ kernels instead of dequantize + hipBLAS: on the RX 6900 XT a 4K -prompt read 436 -> 757 tok/s (greedy text diverges from the default after a few dozen tokens: the q8_1 activation -rounding). `STRATA_HIP_ADAPT_KERNEL_COPY=1` (#884) copies the adaptive tier's swaps with a kernel instead of the SDMA -engine, a workaround for the gfx1030 hang seen with the MMQ prompt path and adaptive swaps (untested on the reporter's -machine). Windows HIP: the doorbell kernels fence their store (#697), and the shared-expert fork is off on HIP (#816). - -**The shared-expert stream fork (`STRATA_SH_STREAM`):** it is off by default on HIP, except for the gfx1151 table, and -that default stays. The measurements disagree by platform. 2x RX 6900 XT on Linux (#1272): turning it off costs 5-13% -of decode (one card 47.5 / 46.9 / 48.3 -> 50.5 / 48.5 / 51.3 tok/s with it on). RX 6800 on Windows (#816, -`bench/results/2026-10-04-community-rx-6800-windows`): on was 12-22% slower. 4x R9700: fork off was 13-45% faster. -Linux RX 6000 users can try `STRATA_SH_STREAM=1`; the output is identical either way. Since the same card goes -both ways on different PCs, `./setup.sh --calibrate` measures it on yours (off against on, a restart each) and writes -`STRATA_SH_STREAM=1` into the config's `"env"` only when it wins by more than 3%; a later calibration that finds it no -longer faster removes it again. - -**The draft's side stream (`STRATA_MTP_SIDE_STREAM`):** HIP does not use the MTP drafter's branch stream, and since -0.1.40.2 it does not create it (#1240: an R9700 lost prompt speed with it). gfx103x is the exception and creates it: on -an RX 6800 under Windows (`STRATA_HIP_PROMPT_F16=1`, IQ3_XXS, two runs each) a 7.9K-token prompt reads at 798-808 tok/s -with the stream and 584-596 without, and a 30.4K-token one at 855-863 against 553-600; `STRATA_PREFILL_TIMING=1` puts -the whole difference in the "wait copy" phase (3,081 -> 75 ms and 12,642 -> 288 ms). Decode and the output are the same. -`STRATA_MTP_SIDE_STREAM=0` turns it off on gfx103x, `=1` creates it on another HIP card. Measured on one card and one -OS; a Linux gfx103x number is welcome. - -**Long prompts on a big card: check the lend-coverage line (#1389, measured by the reporter on an RX 7900 XTX, gfx1100, -v0.1.40.2, 50 GB RAM).** With `--resident-experts` the prompt path borrows cache slots, and the experts of those slots -must also sit in RAM. After the first request of a start the engine prints -`FileExpertSource: N of M of the prompt path's lendable slots keep their experts in RAM too`. When N is not M, the -experts not covered are read from the pack during the prompt (now also a `WARNING` line). A 74K prompt routes through -nearly all experts, so the misses repeat: the reporter saw `wait copy` at 0.2% and 2,583 tok/s with 6201 of 6201, and -36.8% and 1,810 tok/s with 5093 of 6202 (same binary, only the start differed). Short prompts (25K) read the same either -way. The resident set is sized from `MemAvailable` at start, minus `STRATA_RESIDENT_HEADROOM_GIB` (default 4, in GiB; the -`--resident-experts` switch and `--resident-budget-gib` read it). If the line is short, lower the headroom a little and -restart; if the machine then swaps or stalls, raise it again. Re-check the line after a ROCm update: the reporter's newer -runtime kept about 1 GiB more host RAM, which moved the best value from 8 to 7. Other levers from the same report (his -numbers, not re-measured here): `STRATA_PF_FUSED=1 STRATA_PF_GEMM=1 STRATA_PF_SWITCH_MIN_T=4096` 925 -> 1,092 tok/s, -`--resident-experts` 1,092 -> 2,503 tok/s, `--prefill auto:32768` +23% at 74K. The engine uses a hipBLASLt tuning table -only when its version matches the installed library; otherwise it prefills on plain hipBLAS (about 45% slower for a 131K -prompt), so keep the table and the library paired. Not worth trying (measured flat): `--kv-resident` changes, a prefill -chunk above 32768, `--spec` above 4. - -## Exact long-prompt QSA scorer (gfx1100 opt-in, gfx12 default) - -`STRATA_SCORES_THREAD=1` at engine startup enables a HIP port of #1825's score kernel on gfx1100, for batches of -at least 64 queries with at least 16,385 live index blocks. On gfx12 (RDNA4) it is on by default for those batches and -`STRATA_SCORES_THREAD=0` turns it off (0.1.45): on an R9700 `qsa_scores_gfx1100_parity --full` compares 223,664,777 scores and -11,895,800 ids bitwise equal to the warp scorer, the scorer stage with 256 queries is 3.6x faster at 131K and 4.0x at 240K, and a -fresh prompt of 131K tokens reads 8.7% faster (4 of 4 pairs) and one of 230K tokens 16.4% faster (3 of 3), with the same answer tokens and byte-identical `exact_pp` residual dumps at 70K and 131K tokens (prompts under 65K cells never reach the bound). The default, short-prompt and decode paths keep the -existing scorer. On one RX 7900 XTX / 7950X3D / ROCm 10.0.0, three paired runs of a tuned 0.1.40.3 engine reduced -fresh one-token request time by 10.45% at 130,978 prompt tokens and 22.64% at 239,906; one same-binary current-main -pair measured 12.85% and 21.71%. Cached follow-ups did not show a repeatable large gain, and a two-task coding pilot -was 3.27% slower overall. This is opt-in, not a general coding-speed recommendation. The -[report, per-run data and standalone score/ID check](../bench/results/2026-10-11-gfx1100-qsa-scores/README.md) -give the settings, exact-parity checks and limits. `STRATA_SCORES_THREAD_TRACE=1` prints an engagement line. - -## Deep contexts: the prompt's top-k selection (0.1.45, #1749) - -Past ~135K cells (and from ~90K cells on a prompt read) the prompt's top-k selection runs the wide kernel, as on CUDA, instead of the four-pass -reference radix that every deep context on HIP paid; the same ids (`qsa_select_bench` compares them with the reference). On an R9700 the -selection of a 256-query batch is 0.29 -> 0.21 ms at 98K, 1.16 -> 0.49 ms at 262K and 22.0 -> 0.93 ms at 524K; a fresh prompt reads 1.3% faster at -230K tokens and 34% faster at 500K (716 -> 962 tok/s; 1,284 tok/s with the exact scorer above, 79% in all), with the same answer tokens. -`STRATA_TOPK_OLD=1` runs the original kernel, `STRATA_TOPK_STREAM=0` the register kernel in the 90K-135K band. - -## Linux: verify timeouts while the kernel reclaims host memory (experimental workarounds) - -A `verify: timed out at layer N` or "no progress for 60 s" message does not by itself mean a kernel or handshake bug. Two -community reports found the same mechanism: the GPU's queues are suspended while the kernel reclaims host pages the GPU -has pinned through a KFD userptr, and a restore that keeps returning `-EAGAIN` leaves them suspended for tens of seconds. -Each report is one machine, one workaround, and the cause is not confirmed on either; neither is a default or a general -speed claim, and neither is known to matter on Windows or on other ROCm versions. - -- **Paged host allocations (#750, two Radeon AI PRO R9700, ROCm 7.2):** tracing correlated a timeout with about 31 s of - USERPTR queue suspension on both cards. ROCr uses USERPTR for paged host allocations unless `HSA_USERPTR_FOR_PAGED_MEM=0`. - With it in the server JSON `env`, five paired runs gave the same outputs and the same median (7 requests: 16.65 s against - 16.63 s), without the 29.8 s and 42.2 s outliers; solo requests were a little slower (3.55 s to 3.69 s). -- **`--mmap-experts` (#920, RX 6800 gfx1030, ROCm 7.2.4, 31 GiB RAM, IQ3_XXS):** every run stalled until - `GPU_PINNED_MIN_XFER_SIZE=1048576` was in the `env`, and none has since. HIP pins the pageable source pages of large - copies, here the mapped expert file. (The engine already sets this variable for `STRATA_ARENA_MMAP=1`.) - -```json -"env": { "GPU_PINNED_MIN_XFER_SIZE": "1048576" } +```sh +.venv/bin/python -m serve.server --engine strata --config strata-hip.json --port 8080 ``` -Change one setting at a time with the rest of the configuration the same, restart the engine so the runtime reads it, and -remove it to go back to the default. Check free RAM and the driver's GTT limit before large-context tests. If the engine -runs in a systemd unit, limit it with `MemoryMax`, never `MemoryHigh`: `MemoryHigh` counts the page cache that -`--mmap-experts` reads from, and a 16K prompt stalled in `pread` for 8 minutes under it. In a report, give the runtime and -kernel versions, whether reclaim coincides with KFD queue eviction, and paired timings. +- Select the AMD device with `ROCR_VISIBLE_DEVICES` or `HIP_VISIBLE_DEVICES` if needed. +- Start with a modest context and prefill chunk. The 4K context above is a smoke-test start, not a model limit. +- The worker count (15) was used on a 16-core CPU; measure it for yours. +- Keep the PLE file on an SSD. +- Large pinned host allocations can fail on ROCm even when RAM is free. `--mmap-experts` avoids allocating the full pinned + arena; it depends on OS file-cache residency and does not make SSD access equal to RAM. +- **The card also drives a Linux desktop:** keep more VRAM free than the default, for example + `./setup.sh --vram-reserve-mib 3072`. With the cache filling the card, the desktop's next VRAM need makes amdgpu move + memory to system RAM, and the OOM killer can end the compositor or apps. + +**Long prompts on a big card: the lend-coverage line.** With `--resident-experts` the prompt path borrows cache slots, and +the experts of those slots must also sit in RAM. After the first request of a start the engine prints +`FileExpertSource: N of M of the prompt path's lendable slots keep their experts in RAM too`. When N is not M, the missing +experts are read from the pack during the prompt (also a `WARNING` line). The resident set is sized from `MemAvailable` at +start minus `STRATA_RESIDENT_HEADROOM_GIB` (default 4). If N is short of M, lower the headroom a little and restart; if the +machine then swaps or stalls, raise it. Re-check after a ROCm update. + +### Switches for shared-expert and draft streams + +| Switch | Default | Effect | +| --- | --- | --- | +| `STRATA_SH_STREAM=1` | Off on HIP, except the gfx1151 table | Shared-expert fork on its own stream. Output is identical; speed goes both ways by PC. `./setup.sh --calibrate` measures it and writes it to the config's `"env"` only when it wins by more than 3%. | +| `STRATA_MTP_SIDE_STREAM` | Off on HIP, on for gfx103x | `=0` turns it off on gfx103x, `=1` creates it on another HIP card | +| `STRATA_DENSE_MMQ=1` | Off | Dense GGUF prompt projections through int8 MMQ kernels instead of dequantize + hipBLAS. Greedy text can diverge after a few dozen tokens (q8_1 rounding). | +| `STRATA_HIP_ADAPT_KERNEL_COPY=1` | Off | Adaptive-tier swaps copied by a kernel instead of the SDMA engine; a workaround for a gfx1030 hang with MMQ prompts and adaptive swaps (untested by the reporter) | + +## Speed switches + +Switches for the prompt path. A switch you set yourself, to anything, is never overridden by an architecture default. Put +them in the server config's `env`, or in the shell. + +| Switch | Cards | Default | Effect | +| --- | --- | --- | --- | +| `STRATA_HIP_WMMA` | gfx12; gfx11 on request | gfx12 Linux: on with int8 KV. Windows and gfx11: `=1` | QSA prompt attention on the matrix cores. `=0` off; `=auto` takes it only for batches of `STRATA_HIP_WMMA_MIN_Q` (64) or more queries. Not bitwise: greedy text can differ from about token 50 on. | +| `STRATA_PF_FUSED` | gfx12 native IQ packs | gfx12 Linux: on for IQ3_S, IQ3_XXS and Q2_0 gate/up. IQ2 formats and Windows: `=1` | Prompt experts on int8 matrix-core kernels instead of MMQ. `=0` keeps MMQ; `STRATA_PF_FUSED_NATIVE=0` keeps MMQ with the flag set. | +| `STRATA_SCORES_THREAD` | gfx12, gfx1100 | gfx12: on; gfx1100: `=1` | Exact long-prompt QSA scorer (below). `=0` off. | +| `STRATA_SELECT_WMMA=1` | gfx12 | Off | Matrix-core block scorer for the QSA top-k; selects slightly differently | +| `STRATA_HC_Q8` | gfx12, gfx1151 | On | 8-bit hyper-connection weights; `=0` restores the bf16 read ([below](#strix-halo-8-bit-hyper-connection-weights-default-on-gfx1151)) | +| `STRATA_GDN_CHUNKED=1` | gfx11 | Off | Chunked prompt GDN (next section) | +| `STRATA_PF_GEMM`, `STRATA_PF_PAD` | gfx11 only | Off | RDNA4 runs neither; the engine does not pad where no kernel reads the stride | +| `STRATA_HIP_ROUTER_OLD=1` | all HIP | Off | Run the portable MoE router kernel instead of the fast one | +| `STRATA_PREFILL_WARM=0` | all | On | Turn off the warm-up BF16 and FP16 products run while the expert arena loads | +| `STRATA_HIP_QUEUE_PRIME=0` | gfx1200, gfx1201, Linux | On | Turn off the ordered opening of two GPU queues at start (it avoids slow starts) | +| `STRATA_TOPK_OLD=1`, `STRATA_TOPK_STREAM=0` | HIP | Off | Run the original top-k kernel, or the register kernel in the 90K-135K band | + +### Opt-in chunked prompt GDN on gfx11 + +`STRATA_GDN_CHUNKED=1` runs the prompt's gated DeltaNet recurrence in 16-token chunks on the gfx11 matrix cores, with FP16 +operands and FP32 accumulation, and fuses the output norm. It is off by default because rounding differs from the +token-by-token FP32 recurrence. The CUDA switch of the same name uses a separate FP32 implementation. `=2` is the same as 1 +here. + +The HIP path accepts gfx1100, gfx1101, gfx1102, gfx1150 and gfx1151 and prompt chunks of at least 64 tokens. Other cards, +shorter chunks, an output row stride not divisible by eight, and `STRATA_GDN_REC_HEADS` use the existing recurrence. Speed +and numerical checks were run on gfx1151 only. + +> **Measured on:** Ryzen AI Max+ 395 (gfx1151), ROCm 7.14.1, UD-IQ4_XS, engine version not recorded, whole-prompt `prompt_ms`, 11 alternating +> pairs per row, geometric mean of off / on. "Section 5" is the nine variables of STRIX_HALO.md section 5; "defaults" is the +> engine without them. + +| Tokens | Arm | Off -> on | Off / on | +| ---: | --- | ---: | ---: | +| 4,096 | section 5 | 3.703 -> 3.641 s | 1.018 | +| 4,096 | defaults | 5.914 -> 5.829 s | 1.014 | +| 32,768 | section 5 | 26.157 -> 25.589 s | 1.021 | +| 32,768 | defaults | 44.082 -> 43.479 s | 1.015 | +| 128,000 | section 5 | 107.149 -> 104.866 s | 1.021 | +| 128,000 | defaults | 193.439 -> 191.055 s | 1.014 | + +That is about 2% on top of section 5 and 1.5% on the defaults. First-token KL against the run without the switch (12 prompts of +1.5K-24K tokens) is 0.004 to 0.007 mean, with the same top token in all of them. Long generation and perplexity were not +measured. To check it on your build: + +```sh +cmake --build build-hip --target hip_gdn_chunked hip_gdn_rec_head +ctest --test-dir build-hip -R 'hip_gdn_(chunked|rec_head)' --output-on-failure +for i in 1 2 3; do ./build-hip/hip_gdn_chunked --bench-default; ./build-hip/hip_gdn_chunked --bench; done +``` + +### Exact long-prompt QSA scorer + +`STRATA_SCORES_THREAD=1` enables a HIP port of the exact score kernel on gfx1100 for batches of at least 64 queries with at +least 16,385 live index blocks. On gfx12 it is on by default for those batches and `STRATA_SCORES_THREAD=0` turns it off. It +is bitwise equal to the warp scorer. The default, short-prompt and decode paths keep the existing scorer. +`STRATA_SCORES_THREAD_TRACE=1` prints an engagement line. + +> **Measured on:** Radeon AI PRO R9700 (gfx1201), 256-query batches, fresh prompts, engine 0.1.45. + +| Prompt | Scorer stage | Prompt read | +| --- | --- | --- | +| 131K tokens | 3.6x faster | 8.7% faster | +| 230K tokens | 4.0x faster (at 240K) | 16.4% faster | + +On one RX 7900 XTX the gain was 10-13% at about 131K tokens and 22% at 240K. Cached follow-ups and a two-task coding pilot +showed no repeatable gain, so this is opt-in on gfx1100, not a coding-speed recommendation. Data and limits: +[bench/results/2026-10-11-gfx1100-qsa-scores](../bench/results/2026-10-11-gfx1100-qsa-scores/README.md). + +### Deep contexts: the prompt's top-k selection + +Past about 135K cells (and from about 90K cells on a prompt read) the top-k selection runs the wide kernel, as on CUDA, with +the same ids as the reference radix. On an R9700 a fresh 500K-token prompt reads 34% faster than with the reference kernel +(716 -> 962 tok/s; 1,284 tok/s with the exact scorer above). + +## Linux: verify timeouts while the kernel reclaims host memory + +A `verify: timed out at layer N` or "no progress for 60 s" message does not by itself mean a kernel bug. Two community +reports found the same mechanism: the GPU's queues are suspended while the kernel reclaims host pages the GPU has pinned +through a KFD userptr. Each is one machine and one workaround; neither is a default, and neither is known to matter on +Windows or other ROCm versions. + +| Case | Workaround in the server JSON `env` | +| --- | --- | +| Paged host allocations (two R9700, ROCm 7.2, #750) | `"HSA_USERPTR_FOR_PAGED_MEM": "0"` | +| `--mmap-experts` (RX 6800, ROCm 7.2.4, 31 GiB RAM, #920) | `"GPU_PINNED_MIN_XFER_SIZE": "1048576"` | + +Merge the field into the existing config (this is not a complete run config), change one setting at a time, restart the +engine, and remove it to go back. If the engine runs in a systemd unit, limit it with `MemoryMax`, never `MemoryHigh`: +`MemoryHigh` counts the page cache that `--mmap-experts` reads from. ## RDNA4 (gfx1201) -The RX 9070 / 9070 XT and the Radeon AI PRO R9700 run the same kernels as gfx1100: wave32, 64 KiB of LDS per -workgroup, the signed dot4 instruction (`v_dot4_i32_iu8` through `__builtin_amdgcn_sudot4`) and a 100 MHz wall -clock. The first report and patch came from doplxyz (#178). Validated on 2026-09-30 on doplxyz's test machine: -an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 3900X (16 threads, no AVX-512), -47 GB RAM, Ubuntu 24.04 in a KVM/VFIO guest. +The RX 9070 / 9070 XT and the Radeon AI PRO R9700 run the same kernels as gfx1100: wave32, 64 KiB of LDS per workgroup, the +signed dot4 instruction and a 100 MHz wall clock. Switch defaults for gfx12 are in [Speed switches](#speed-switches). -- **Build:** complete HIP build with tests (`-DCMAKE_HIP_ARCHITECTURES=gfx1201 -DSTRATA_BUILD_TESTS=ON - -DSTRATA_PREFILL_MMQ=ON`), with the system ROCm 7.14 (hipBLASLt 1.4.1) and with setup's own path: TheRock - 7.10.0a20251120 wheels from `gfx120X-all` (hipBLASLt 1.2.0) and `build_engine_hip`. -- **ctest** (all 45 registered tests, R9700): 42 pass, `hip_prefill_hipblaslt_gemm` skips (no table), and two - fail for reasons outside the GPU: `ple_parity` needs the Q2_0 model fixture, `expert_multi_test` needs an - AVX-512 CPU. `hip_handoff` needed the volatile ring store: on gfx1201 a plain store to mapped pinned memory - stays in the GPU's L2 until the stream is synchronized. -- **Arch check:** a gfx1100-only build stops on the gfx1201 card at startup with the message above (engine and - `strata-device`). -- **End to end** (Coder IQ1_M, setup's arguments: `--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5`, - MTP, `--kv int8`, `--max-context 32768`; 128 greedy tokens, system ROCm 7.14, no hipBLASLt table). The answers - are coherent and the same on both cards. The live server smoke (`serve/server.py`: web page, models, props, - chat, prefix reuse, streaming, the Anthropic endpoint, a sampled code answer) passed 9/9 on each card. +- **Build and tests:** `-DCMAKE_HIP_ARCHITECTURES=gfx1201 -DSTRATA_BUILD_TESTS=ON -DSTRATA_PREFILL_MMQ=ON`. On an R9700, + `ple_parity` (needs the Q2_0 fixture) and `expert_multi_test` (needs AVX-512) fail for reasons outside the GPU. +- **Architecture check:** a gfx1100-only build stops on a gfx1201 card at startup with a plain message. +- **hipBLASLt:** the gfx120X wheel's hipBLASLt is 1.2.0 and no table ships for it, so Linux `rx9070` uses plain hipBLAS. A + table calibrated for it measured within noise end to end. Tables for 1.2.2 (system ROCm 7.2.4), 1.4.1 and 1.5.0 exist in + `tools/hip` ([Tuning table](#tuning-table)). With ROCm 7.2.4's hipBLASLt 1.2.2 the plain path is far off, and its table + roughly doubled prompt speed. +- **Two cards in one run (layer split):** `"backend": "hip", "gpu": [1, 0]` in the config (setup writes it with `--gpus 1,0`). + With every expert on the GPUs the split gives the same tokens as the R9700 alone. On an R9700 plus a 9070 XT the split + does not pay, because the R9700 alone holds all 12,288 expert pairs. A split is worth it when no single card holds the + model's experts. On Linux a split pins the whole expert arena (the 8 GiB cap is for Windows and WSL2). + `"split_skip_if_fits": true` (opt-in) runs such a pair on the first card alone when it holds every profiled expert. + The conversation cache refuses a split at start (exit 2). See [MULTI_GPU.md](MULTI_GPU.md). +- **First prompt after a start** pays two one-time hipBLAS library loads. The engine runs a tiny BF16 and FP16 product while the + arena loads so the first request does not carry them (`STRATA_COLD_TRACE=1` times them). +- **Pictures:** the card's reader (`strata-vision-hip`) keeps about 1.4 GiB of VRAM. It has not run on an RDNA3 card + (gfx1100-gfx1102) or on Windows with an AMD card. +- **Not validated:** answer-quality benchmarks and contexts beyond 16K on the 9070 XT. - | card | expert cache | peak VRAM | 4K prompt | 4K decode | 16K prompt | 16K decode | - |---|---|---|---|---|---|---| - | Radeon AI PRO R9700 32 GB | 12,288 slots, 23.4 GiB | 29.7 GiB | 982 tok/s | 45.5 tok/s | 1,402 tok/s | 48.3 tok/s | - | RX 9070 XT 16 GB | 4,931 slots, 9.4 GiB | 15.6 GiB | 782 tok/s | 30.8 tok/s | 1,235 tok/s | 35.4 tok/s | - | R9700, TheRock 7.10 wheels | 12,288 slots | | 957 tok/s | 44.2 tok/s | 1,284 tok/s | 43.9 tok/s | +> **Measured on:** Radeon AI PRO R9700 32 GB and RX 9070 XT 16 GB (gfx1201), Ryzen 9 3900X, 47 GB RAM, Ubuntu 24.04 +> (KVM/VFIO guest), system ROCm 7.14, no hipBLASLt table, engine 0.1.30, Coder IQ1_M, `--expert-cache auto --prefill auto +> --spec 4 --spec-min-p 0.5`, MTP, `--kv int8`, 32K context, 128 greedy tokens. - The engine's resident memory was about 26 GB in every run. Since 0.1.31 `__byte_perm` is one `v_perm_b32` and the - packed byte subtracts/compare work on four lanes at once (#262, ttio2tech): decode +15% on the R9700 (46.0 -> 53.0 - tok/s on a 4K prompt, 52.0 -> 60.5 warm) and +5-7% on the 9070 XT, prompts unchanged, the same tokens. -- **hipBLASLt:** the validation above used hipBLASLt 1.4.1. A table calibrated on the R9700 at the engine's shapes - with that version (0.98-1.76x per GEMM over hipBLAS) changed the end-to-end prompt speed by 0-3%, within noise, - so none was shipped for it: there the plain hipBLAS path is already close (a table I calibrated on my own R9700 - with 1.4.1 measured the same: 1.15x per GEMM, +1% prompt - see "Tuning table" below). For hipBLASLt 1.5.0 (ROCm - 10.2.0a nightly) `tools/hip/gfx1201-hipblaslt-100500.txt` is shipped (see "Tuning table" below); on one R9700 it - measured +3.9% prompt speed on 4,210-token prompts (1,590 vs 1,531 tok/s), a modest gain. - With the hipBLASLt 1.2.2 of a system ROCm 7.2.4 the plain path is far off, and `tools/hip/gfx1201-hipblaslt-100202.txt` - is shipped for it (24 of the gfx1100 table's 26 shapes; setup uses it only with that exact version): on an R9700 - with the full IQ3_XXS, `--kv int8 --kv-resident 32768`, a fresh 32K prompt read at 638 -> 1,177 tok/s and a 7K one - at 656 -> 1,164 tok/s with it, decode unchanged. `hip_prefill_hipblaslt_gemm` passes with it. -- **Both cards in one run (layer split, engine 0.1.30):** the config's `"backend": "hip", "gpu": [1, 0]` (R9700 - first) runs through `serve/server.py` (setup writes it with `--gpus 1,0` since 0.1.31). Auto split put layers 0-27 on the R9700 and - 28-47 on the 9070 XT. With every expert on the GPUs the split gives exactly the tokens of the R9700 alone (4K and - 16K prompts); checkpoints on the split (second turn, rewind, a prompt sharing a prefix, a cancelled prompt - retried) give exactly the tokens of a fresh read. The conversation cache refuses a split at start (exit 2). - On this pair the split does not pay: the R9700 alone already holds all 12,288 expert pairs. +| Card | Expert cache | 4K prompt | 4K decode | 16K prompt | 16K decode | +| --- | --- | ---: | ---: | ---: | ---: | +| R9700 32 GB | 12,288 slots | 982 tok/s | 45.5 tok/s | 1,402 tok/s | 48.3 tok/s | +| RX 9070 XT 16 GB | 4,931 slots | 782 tok/s | 30.8 tok/s | 1,235 tok/s | 35.4 tok/s | - | run (the same session, warm) | 4K prompt | 16K prompt | decode | - |---|---|---|---| - | R9700 alone | 1,794 tok/s | 1,804 tok/s | 51-52 tok/s | - | R9700 + 9070 XT, auto split | 1,384 tok/s | 1,906 tok/s | 42-43 tok/s | - | RX 9070 XT alone | 1,016 tok/s | 1,519 tok/s | 38-40 tok/s | - - A split is worth it when no single card holds the model's experts. Since 0.1.31 a split pins the whole expert - arena on Linux (the 8 GiB cap is for Windows/WSL2 only, #253): with an expert cache of 1,500 on the R9700 (most - experts streamed) the split read a 16K prompt at 1,803 tok/s instead of 1,287, with the same tokens in 5 + 5 - starts. `"split_skip_if_fits": true` in the config (0.1.31, opt-in) runs such a pair on the first card alone when it - holds every profiled expert: with the R9700 first, 4K prompts 1,776 tok/s (split: 1,244) and decode ~60 tok/s - (split: ~51), the tokens of the R9700 alone (docs/MULTI_GPU.md). -- **Speed switches (engine 0.1.32, measured on the R9700 / 9070 XT with the Coder IQ1_M pack):** - - the MoE router (`router_top10`) runs a HIP kernel without its serial FP64 sum and block barriers by default: the - same ids and weights bit for bit (`hip_router_fast` checks 65,536 rows), 39 -> 9-12 us per call, the same greedy - tokens (5 + 5 starts). Decode: 62.4 -> 70.0 tok/s on the R9700 and +4% on the 9070 XT in one A/B here (ROCm 10.2 - nightly); a user's repeated A/B/A/B with setup's TheRock 7.10 wheels and an i5-12600K measured +1-4%, inside a - 12-20% run-to-run spread (#432) - how much it gains depends on how much of decode the router is on that setup. - `STRATA_HIP_ROUTER_OLD=1` runs the portable kernel. - - `STRATA_HIP_WMMA` (on by default on gfx12 with an int8 KV since 148-amd, `STRATA_HIP_WMMA=0` turns it off; gfx11: opt-in with `=1`): the prompt path's QSA attention on RDNA4 matrix cores (the engine says so once, at the first prompt) - (`v_wmma_f32_16x16x16_f16`, FP16 hi + lo halves like the CUDA tensor-core kernel; `hip_prompt_attn_wmma` bounds it - against the FP32 kernel and FP64). 7.2-7.5x the portable kernel; R9700 prompts 4K 1,784 -> 2,427 tok/s, 16K - 1,797 -> 2,700 (a user with TheRock 7.10: +29-34%, #432). Not bitwise: greedy text differs from token ~50 on, as with the CUDA tensor-core attention. With it - the prompt path's expert ring is 96 slots (as STRATA_PREFILL_RING=96): with the default 384 the 9070 XT's 4K - prompts fell to 718 tok/s; with 96 they gain (1,017 -> 1,211; 16K 1,518 -> 2,032). PR #329 - (bsorensen110) contributed an equivalent gfx12 WMMA kernel of the same speed (within 1%); this one also masks KV - pages that KV streaming has not made resident, as the decode kernel does. On my R9700 (engine 0.1.40, - IQ3_S at 431072 ctx, `--kv int8`, the gfx1201 hipBLASLt 1.4.1 table below, which names the machine, GPU idle): - fresh prompts of 4,210 and 8,830 tokens - the longest prompt I measured there is 8,830 tokens, the 431072 ctx is - what the model runs at, not what these prompts reach - three matched trials each, the first fresh prompt after - each start left out: 1,571 -> 2,064 and 1,568 -> 2,065 tok/s at 8,830 tokens (+31.4 and +31.7%) and 1,699 -> - 1,990 at 4,210 (+17.1%), medians 1,571 -> 2,064 and a 1.27x geometric mean over those three. decode - 89.8 -> 86.9 tok/s, unchanged. `STRATA_SELECT_WMMA=1` on top changed nothing at those lengths (2,026 tok/s); - its +1.5% is a 16K/262K-ctx number. `hip_prompt_attn_wmma` passes on that card. - **Default on gfx12 on Linux (148-amd).** Windows: opt-in until tested on Windows (`STRATA_HIP_WMMA=1` turns it on; the engine says so in its log at the first prompt). Quality on an R9700 (Q2_0, int8 KV, fixed expert cache so that the default against itself is bit-identical): - teacher-forced KL against the old kernel through the real serve decode path (STRATA_FOLLOW, 28 prompts, 6,024 positions) 0.0003 (p99 0.006, top-1 - 99.8%, perplexity of the forced text x1.0003), and 0.0014 over 1,760 positions of eight 6K-token documents (top-1 98.9%, x1.0009); HumanEval 157/164 - (default 156 and 156), reasoning 25/30 (24 and 26), agent 6/6, needles at 32K and 125K x three depths 6/6, CJK 8/8, tool calls 5/5. Prompt speed, - interleaved pairs on a quiet R9700, Q2_0 32 GB: see the 148-amd table below. The first-token KL against the FP16 prompt path cannot rank it (0.0296 against 0.0280 for the old - kernel, a paired difference of +0.0016 +- 0.0026): on a 2-bit pack any change of the prompt numerics, even `--prefill 4096`, moves the first token by about 0.025. - - `STRATA_HIP_WMMA=auto`: the same kernel, chosen per call: on a gfx12 card with int8 KV and a prompt batch of - `STRATA_HIP_WMMA_MIN_Q` (64) or more queries it runs the matrix-core kernel, everything else the FP32 one (`=1` forces it - on every call and on gfx11, `=0` keeps the FP32 kernel; unset is the gfx12 default above: the matrix-core kernel on every call). A launch error switches the matrix-core - kernel off for the rest of the process and the FP32 kernel runs (before, the engine exited). Kernel timing on an R9700 - (`hip_prompt_attn_wmma`, context 1.5K-32K, 64-2,048 queries): 6.9-8.4x the FP32 kernel at every shape, error against FP64 - 3e-6 of the output scale (FP32: 2e-6). Engine, Q2_0 pack, 6 interleaved pairs, equal slots, fixed flags - (`--pcie-frac 0 --adapt-every 0`, `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_ROUTE_TAIL_SKIP=0`): 20K prompt x1.46 (`=1`) / - x1.54 (`auto`), 4K x1.33 / x1.34 (6/6 pairs; the noise arm of that run read +6.4-6.9%, so read them as +40-50% and - +30%); the 8-token answers were the default's in every run. - - The first prompt after a start pays two one-time hipBLAS library loads (the first BF16 product 1.2 s, the first FP16 - product 1.0 s on an R9700 with the gfx120X wheel's hipBLASLt 1.2.0; `STRATA_COLD_TRACE=1` times the first 80 dense - projections to show them). The engine now runs one tiny BF16 and one tiny FP16 product on a thread of its own while the - expert arena loads, so the first request does not carry them (`STRATA_PREFILL_WARM=0` turns it off; the outputs go to - scratch buffers, nothing changes; start-up time is the same, the arena load takes longer than the thread). First - 721-token prompt after the 33-token warm request, 5 interleaved pairs: 154 -> 673 tok/s. The next 4K prompt reads - ~1,090 tok/s either way (the "cold" first-prompt numbers above are about this). - - **Fast and slow starts (the "bimodal" prompt speed, found in 0.2.1).** On an R9700 / RX 9000 under Linux about half of - the fresh starts with MTP read prompts 10-23% slower than the others (p620, Q2_0: 4K 1,980 vs 2,290 tok/s, 20K 2,480 vs - 2,770; decode was never affected). The cause is where the kernel's scheduler puts the process's GPU queues. The prompt - path has two busy queues, the compute stream and the stream that copies experts to the card (about 31,000 copies and - event hand-offs per 4K prompt). The runtime opens its queues in the order the loader threads reach them, which changes - every start; when both busy queues land on the same compute pipe of the card (the card has two for this process), every - hand-off waits for a queue switch. The engine now opens two queues at normal priority (more queues cost decode: five cost 2 to 3.6%) one - after the other on the main thread before any other thread starts (`STRATA_HIP_QUEUE_PRIME=0` turns it off; any other - value is a list of `n` / `h` letters for tests). gfx1200 and gfx1201 on Linux only; nothing else changes (the answers - are identical). Not clocks, not memory placement (read speed over the expert cache was 637-640 GB/s in both kinds of - start), not CPU placement, not the MTP stream itself. - - Ship status of `tools/hip/gfx1201-hipblaslt-*.txt`: the gfx120X wheel's hipBLASLt is 1.2.0 (version number 100200); the - tables here are 100202 and 100500, so the engine says "version mismatch ... using hipBLASEx" and runs plain hipBLAS. A - table tuned on the R9700 for 100200 (32 rows, `tune_hipblaslt`) measured within noise end to end (20K prompt x1.035 vs - a noise arm x1.031), so none is shipped: the dense projections are ~10% of a 20K prompt here. - - The prompt path's QSA top-k picks its kernel by the blocks a query actually has, not the cache's capacity (#337, - bsorensen110): the same ids, on by default with AMD (NVIDIA keeps its capacity rule: there the 64K prompts read - 1-3% slower with it). `STRATA_SELECT_WMMA=1` (opt-in, gfx12) adds #337's - matrix-core block scorer; it selects slightly differently (254 of 256 queries the same) and gained +1.5% on 16K - prompts at a 262K context on the R9700. - - `STRATA_PF_FUSED=1` (opt-in, gfx1200 / gfx1201, the native IQ packs, #1277): the prompt's experts run on int8 - matrix-core kernels (`v_wmma_i32_16x16x16_iu8`: the gfx11 kernels with gfx12's lane layout) instead of MMQ. Earlier - engines ignored the flag on RDNA4 (MMQ stayed, no banner); now the engine prints `strata: prompt experts on the fused - int8 kernels (STRATA_PF_FUSED=1, #136)` once, at the first prompt. It is not an arch default: it rounds differently - from MMQ, and those defaults hold only bit-identical switches. `STRATA_PF_FUSED_NATIVE=0` keeps MMQ with the flag set - (for an A/B). R9700 (gfx1201, ROCm 7.14, Coder IQ1_M pack), prompt time as the server reports it, alternating - starts per arm: 4K 1,612 -> 1,353 ms (+19.1%), 16K 5,579 -> 5,147 ms (+8.4%), 32K 11,890 -> 10,962 ms (+8.5%); a - second run of 3 starts per arm gave +12% to +16% at 4K (it depends on the start) and +8.3% at 16K; the - kernels alone are 1.48x / 1.20x / 1.14x MMQ's at 4K / 16K / 32K. Quality: first-token KL against the FP16 prompt path - at 4K (48 prompts) is 1.10x MMQ's, which that sample cannot tell apart from MMQ's; top-1 agreed on 48 of 48 (every - prompt's top token was the same one, so that check says little); 4 of 4 long-context retrieval checks passed. - 0.1.42 re-check on an R9700 (Linux, ROCm 7.14), 12 prompts of 1.5K-24K tokens, first-token KL against the FP16 prompt path (MMQ / fused): IQ3_S 0.00332 / 0.00284, top-1 12 of 12 for both; IQ3_XXS 0.00335 / 0.00452 on one sample and 0.00344 / 0.00323 on a second (24 prompts pooled: 0.00340 / 0.00388, top-1 24 of 24 for both); the Q2_0 pack does not take these kernels on gfx12 (output bit-identical to MMQ). Because the IQ3_XXS pool is not at or below MMQ's, the flag stays opt-in. Measured on Linux only. - gfx1200 builds the same kernels and was not run; Windows and K-quant packs (`STRATA_PF_FUSED_KQ=1`) were not measured. - **Q2_0 gate/up (148-amd).** Windows: opt-in until tested on Windows (`STRATA_PF_FUSED=1`); the default below is Linux only. The native Q2_0 pack's gate/up (64-value blocks) take the same kernels and are in the gfx12 default set with IQ3_S and - IQ3_XXS (`STRATA_PF_FUSED=0` keeps MMQ). One layer, 2,048 tokens x top 10 over 512 experts: MMQ 24.5 ms, fused 4.4 ms; 8,192 tokens 31.7 vs 13.2 ms - (`hip_prefill_fused_iq`, fused against MMQ rel. RMS 1.6e-5). Engine, R9700 Q2_0 32 GB, 5 interleaved pairs: prompt 4K +16.7%, 20K +19.0% (noise arm 1.001 / 0.999). - Quality: teacher-forced KL against MMQ 0.0004 (6,022 positions, top-1 99.5%) and 0.0016 (1,760 long-document positions, top-1 98.2%), the same tasks as - above unchanged (156/164, 24/30, 6/6, 6/6, 8/8, 5/5). IQ2_XXS / IQ2_XS / IQ2_S gate/up stay opt-in on gfx12. - **Both defaults together (148-amd), R9700, interleaved starts of one engine, prompt tok/s** (base = the build before them; the new paths change - bits, so decode texts differ from the first differing token on and decode is compared with both switched off instead: 95.23 vs 95.28, same ids): - Q2_0 32 GB, 5 pairs: 4K 1,151 -> 2,286 (a slower start mode gives 1,988), 20K 1,384 -> 2,751 (2,481); with one lever each (3 pairs): WMMA 4K x1.35 / 20K x1.46, - Q2_0 fused experts x1.28 / x1.19, both x1.73 / x1.79. IQ3_XXS 32 GB, 5 pairs (only the attention is new): 4K 1,369 -> 1,436 (the base starts in a 1,170 or a - 1,370 mode), 20K 1,576 -> 1,997 (x1.27). The identity gate with both switched off (STRATA_HIP_WMMA=0, STRATA_PF_FUSED=0): 10 of 10 prompts and the - 4K / 20K residual dumps identical on both packs. - - **`STRATA_PF_GEMM` / `STRATA_PF_PAD` are gfx11 only** (0.1.46): their product is `strata_pf_gemm_f16_ld`, - built into the gfx11 matrix-core block of `wmma_gemm.cu`, so an RDNA4 card runs neither. Until 0.1.46 the padded - half was worse than inert: `STRATA_PF_PAD=1 STRATA_PF_GEMM=1` let the two K 6144 producers write row stride 6,208 - into a GEMM that only the gfx11 kernel takes, and `Gemm::native` ended the engine (`prefill gemm: a padded X (ldx - 6208, K 6144) needs STRATA_PF_PAD's path`, exit code 1 - under `serve/` an engine restart loop per request). - Measured on an RX 9070 (gfx1201, 16 GB, system ROCm 7.2.4, kernel 7.2.11, 31 GiB RAM, Coder IQ1_M): 3 of 3 starts - exited that way at `--vram-reserve-mib` 700 and 1,536, so it was not the VRAM guard. Now the producer asks - `strata_pf_padded_gemm_available()` and does not pad where no kernel reads the stride (one line at the first prompt), - and `Gemm::native` copies a padded X's rows unpadded and runs the ordinary paths instead of exiting - checked with - the gate forced off on that card: 4K prompt answers byte-identical to the unpadded run, no exit. `STRATA_PF_GEMM=1` - alone is inert there too (4K 1,247 against 1,243 tok/s, 8K 1,330 against 1,321, medians of 3 starts, i.e. noise). -- **Known:** rarely (about 1 start in 10) a HIP run's greedy output differs from another start's at some token, on - one card or two and on engine 0.1.29 as well; not yet explained. -- **Pictures:** the card's reader (`strata-vision-hip`) is measured on an R9700 (gfx1201): 57 ms for a 260-token picture and 0.40 s for a - 972-token one, against 3.13 s and 14.1 s on the CPU reader (EPYC, 32 threads); the embeddings agree with the CPU reader's (cosine mean 0.99985). It keeps - about 1.4 GiB of VRAM (+1,470 MiB on the R9700). A test picture (red circle, blue square, green triangle, the text "STRATA 42") was answered correctly with both readers, and the - fallback to the CPU reader was tested with a reader built for another card. The same reader built for gfx1103 ran on a Radeon 780M (0.40 s for 260 tokens - against 2.42 s on its CPU). It has not run on an RDNA3 card (gfx1100-gfx1102) or on Windows with an AMD card. -- **Not validated:** long contexts beyond 16K, answer-quality benchmarks. +Later engines are faster; see [History](#history--older-measurements) for the R9700 gains from WMMA attention, fused +experts and the router kernel. ## Integrated GPUs (mini PCs) -On a Radeon that shares the computer's memory (the Ryzen mini PCs: Radeon 780M / 760M `gfx1103`, 890M / 880M `gfx1150`, -Strix Halo `gfx1151`) the matrix-core prompt set below is on by default on the Radeon 780M (`gfx1103`, Linux) and opt-in -(`STRATA_IGPU_PROMPT_SET=1`) on the others: it changes bits, and only the 780M was measured. The engine picks the set from the device's own "integrated" flag and -the architecture, not from the name: a discrete card (an RX 7900 XTX `gfx1100`, an R9700 `gfx1201`, any other) never gets it, -and neither does a computer that also holds a discrete card next to the iGPU. The engine prints one line at start-up that names -what it set. +On a Radeon that shares the computer's memory (Radeon 780M / 760M gfx1103, 890M / 880M gfx1150, Strix Halo gfx1151) the +matrix-core prompt set below is on by default on the Radeon 780M (gfx1103, Linux) and opt-in on the others, because it +changes bits and only the 780M was measured. The engine picks the set from the device's own "integrated" flag and the +architecture. A discrete card never gets it, and neither does a computer that also holds a discrete card beside the iGPU. +The engine prints one line at start-up that names what it set. | Switch | What it does | -|---|---| -| `STRATA_HIP_WMMA=1` | the prompt attention on the matrix cores (needs `--kv int8`) | -| `STRATA_PF_GEMM=1` | the prompt's FP16 projections on the 128 x 256 WMMA GEMM | -| `STRATA_PA_FAST=1` | the prompt attention with single FP16 q and p | -| `STRATA_SELECT_WMMA=1` | the block scorer on the matrix cores | -| `STRATA_HC_UPMIX=1` | the hyper-connection up projection with its epilogue | +| --- | --- | +| `STRATA_HIP_WMMA=1` | Prompt attention on the matrix cores (needs `--kv int8`) | +| `STRATA_PF_GEMM=1` | The prompt's FP16 projections on the 128 x 256 WMMA GEMM | +| `STRATA_PA_FAST=1` | Prompt attention with single FP16 q and p | +| `STRATA_SELECT_WMMA=1` | The block scorer on the matrix cores | +| `STRATA_HC_UPMIX=1` | The hyper-connection up projection with its epilogue | -**Why.** (148-amd first measurement; 021-minipc repeated it with 6 interleaved pairs: prompt x1.60 at 1K, x2.03 at 4K, decode x0.999, identical greedy ids.) The dense projections are 63% of a prompt's GPU time on a 12-CU iGPU (the experts only 16%), and they are what these -switches speed up. Radeon 780M (P2006, Ryzen 7 255, Q2_0, `--kv int8`, 4K prompt, one run each; 148-amd): 109.5 tok/s with -none of them, 214.7 with the first four (+96%), 233 with `STRATA_HC_UPMIX` as well (+111%); decode is the same (16.6 against 17-19 -tok/s, inside the run-to-run noise). The prompt switches do not touch the decode kernels. +`STRATA_IGPU_PROMPT_SET=0` turns the whole set off, `=1` turns it on where it is not the default. Each switch keeps `=0` as its +off switch. `STRATA_IGPU_DEFAULTS=0` turns the set off whatever else is set. The same code applies on Windows but was not +tested there. Strix Halo has separate speed switches ([STRIX_HALO.md](STRIX_HALO.md)). -**What it costs.** They round differently from the old path (another summation order, half-precision intermediates), so the answers -differ in the last bits. Measured on an R9700 with the same kernels (Q2_0): teacher-forced KL 0.0003 (28 prompts) / 0.0014 (eight -~6K-token documents), and the task sets inside their run-to-run noise. +> **Measured on:** Radeon 780M (Ryzen 7 255), Q2_0, `--kv int8`, 4K prompt, engine version not recorded, one run each. -**Turning it on and off.** `STRATA_IGPU_PROMPT_SET=0` turns the whole set off, `=1` turns it on where it is not the default; each switch keeps `=0` as its off switch -(`STRATA_HC_UPMIX=0`); `STRATA_IGPU_DEFAULTS=0` turns the set off whatever else is set. A switch you set yourself, to anything, is never overridden. Put them in the server config's `env`, or in the shell. +| Setting | Prompt | Decode | +| --- | ---: | ---: | +| None of the switches | 109.5 tok/s | 16.6 tok/s | +| First four | 214.7 tok/s (+96%) | 17-19 tok/s (noise) | +| All five | 233 tok/s (+111%) | same | -**Windows.** The same code applies, but it has not been tested on Windows (there was no Windows iGPU machine to run it on). +A repeat with 6 interleaved pairs gave prompt x1.60 at 1K and x2.03 at 4K, decode x0.999, identical greedy ids. The +switches round differently from the old path; on an R9700 with the same kernels the teacher-forced KL was 0.0003 to 0.0014. -**Strix Halo (`gfx1151`).** Its exact speed switches are separate and unchanged ([STRIX_HALO.md](STRIX_HALO.md)); the five above -were already its "maintainers' fast configuration". The set was measured on the 780M only. - -**The 780M resets its GPU.** On the 780M's current Ubuntu firmware (gc 11.0.1 MES) a GPU reset can follow when the kernel evicts the -process's GPU queues while a kernel with barriers is running ([RADEON_780M.md](RADEON_780M.md), `notes-minipc.md`). It is -not caused by the prompt set: the default build without it resets the same way (forced compaction every 10 s killed 2 of 2 runs). -`STRATA_ARENA_PIN_GIB=none` (experimental, exact) leaves the expert arena unregistered, so that the kernel's page migration has -no big GPU mapping to invalidate; the experts then reach the GPU by copy. Measured on the 780M (30 interleaved runs, 021-minipc): the -eviction time drops from about 6 s to about 0.13 s per run, but resets still happened (4 of 20 runs against 1 of 10), and prompts/decode -were 1-4% slower, so it is not the default. A newer `linux-firmware` (MES) is the likely cure and is a host change the engine does not make. +**The 780M resets its GPU on some firmware.** On the current Ubuntu firmware (gc 11.0.1 MES) a GPU reset can follow when the +kernel evicts the process's GPU queues while a kernel with barriers runs ([RADEON_780M.md](RADEON_780M.md)). The prompt set does +not cause it. `STRATA_ARENA_PIN_GIB=none` shortened the eviction time but did not stop the resets and cost 1-4% speed, so it +is not the default. A newer `linux-firmware` is the likely cure and is a host change. ## Community-validated cards -Run by their owners, not on the maintainers' machines; setup accepts them like gfx1100 / gfx1201. setup ships a -hipBLASLt table for gfx1200 (the numbers below); gfx1101 has none (make one with [Tuning table](#tuning-table) -and compare the prompt speed with and without it). +Run by their owners, not on the maintainers' machines. Setup accepts them like gfx1100 and gfx1201. A hipBLASLt table ships +for gfx1200; gfx1101 has none (make one with [Tuning table](#tuning-table) and compare the prompt speed). -- **gfx1101, RX 7800 XT 16 GB** (jhohertz, #254; engine 0.1.29, Ryzen 9 5950X, 121 GiB RAM, system ROCm with - hipBLASLt 1.4.1): `./setup.sh --backend hip` detected the card and compiled the engine; `strata-device --selftest` - passed; ctest 30/32 (`ple_parity` needs the Q2_0 fixture, `platform_memory_test` the memlock limit). Coder IQ1_M, - 64K context, MTP, with a table the owner calibrated: fresh prompts of 4K-9K tokens at 898-953 tok/s, decode - 38-44 tok/s (128 tokens). -- **gfx1200, RX 9060 XT 16 GB** (Efeisot, #256 after #176; engine 0.1.29, Ryzen 9 7950X, 64 GiB RAM, ROCm 7.2 with - hipBLASLt 1.2.2): ctest 32/32 (without `ple_parity` and `platform_memory_test`). Coder IQ1_M, greedy, MTP - `--spec 4 --spec-min-p 0.5`, with a table the owner calibrated: +> **Measured on:** the machine in each row, Coder IQ1_M, MTP, owner-calibrated tables. - | prompt | prompt speed | decode | - |---|---|---| - | 2,374 tokens | 540 tok/s | 27.1 tok/s (0.69 draft acceptance; 31.0 at 1.00) | - | 65K (`--kv int8 --kv-resident 65536 --prefill 16384`) | 748-753 tok/s | 26-40 tok/s by acceptance | - | 130K (the same flags) | 725 tok/s | - | +| Card | Machine | Result | +| --- | --- | --- | +| gfx1101, RX 7800 XT 16 GB (#254) | Ryzen 9 5950X, 121 GiB RAM, hipBLASLt 1.4.1, engine 0.1.29, 64K context | Prompts 4K-9K at 898-953 tok/s, decode 38-44 tok/s | +| gfx1200, RX 9060 XT 16 GB (#256) | Ryzen 9 7950X, 64 GiB RAM, ROCm 7.2, hipBLASLt 1.2.2, engine 0.1.29 | 2,374-token prompt 540 tok/s, decode 27.1 tok/s; 65K prompt 748-753 tok/s; 130K prompt 725 tok/s; llama.cpp HIP on the same card: 20 tok/s decode, 450 prompt | +| gfx1102, RX 7600 XT 16 GB (#942) | i7-13700K, 64 GB, Swift IQ3_XXS, 146 requests up to 51K context | About 43 tok/s decode, 250-300 tok/s prompt; setup accepts it, unvalidated | - Greedy output was the same across runs. For comparison, llama.cpp's HIP build measured 20 tok/s decode and - 450 tok/s prompt on that card. +The shipped gfx1200 table (`gfx1200-hipblaslt-100202.txt`) against plain hipBLAS, engine 0.1.31, two runs each: - Since 0.1.38 setup ships that table (`tools/hip/gfx1200-hipblaslt-100202.txt`). On engine 0.1.31, one binary (ctest - 37/37, without `ple_parity` and `platform_memory_test`), prompts of repeated code blocks, prefill 2560 for - the short prompt and 16384 with `--kv int8 --kv-resident 65536` otherwise, two runs each - prompt speed - with the table vs plain hipBLAS (no table): - - | prompt | with the table | plain hipBLAS | gain | - |---|---|---|---| - | 2,374 tokens | 597-599 tok/s | 396-398 tok/s | 1.50x | - | 65,045 tokens | 754-757 tok/s | 380 tok/s | 1.99x | - | 130,091 tokens (the same flags) | 719 tok/s | 363-370 tok/s | 1.98x | +| Prompt | With table | Plain hipBLAS | Gain | +| --- | --- | --- | --- | +| 2,374 tokens | 597-599 tok/s | 396-398 tok/s | 1.50x | +| 65,045 tokens | 754-757 tok/s | 380 tok/s | 1.99x | +| 130,091 tokens | 719 tok/s | 363-370 tok/s | 1.98x | ## RDNA2 (gfx1030) -The RX 6800 / 6800 XT / 6900 XT / 6950 XT run the same kernels: wave32 and 64 KiB of LDS per workgroup. The one -difference is the dot instruction: RDNA2 has no `v_dot4_i32_iu8` (gfx11 and newer), so -`dp4a` uses the plain signed `v_dot4_i32_i8` through `__builtin_amdgcn_sdot4`, which compiles to a single -`v_dot4c_i32_i8` (same signed x signed byte products, modulo 2^32). There is no WMMA; the QSA scorer takes the -same ordered FP32 fallback as gfx1100. CMake lists gfx1030 as unvalidated (the build warns) until a maintainer has -run it; the report below is from a community machine: an RX 6900 XT 16 GB (gfx1030), an i7-13700KF (8 P-cores and -8 E-cores, AVX2, no AVX-512), 63 GB RAM, NixOS, ROCm 7.2.3 from nixpkgs (clang 22, hipBLAS 3.2). +The RX 6800 / 6800 XT / 6900 XT / 6950 XT run the same kernels: wave32 and 64 KiB of LDS per workgroup. RDNA2 has no +`v_dot4_i32_iu8`, so `dp4a` uses the plain signed `v_dot4_i32_i8` (`__builtin_amdgcn_sdot4`). There is no WMMA; the QSA scorer +takes the ordered FP32 fallback. CMake lists gfx1030 as unvalidated. The `rx6000` and `rdna2` packages are built and not tested on +a real RX 6000 card; the numbers here come from community machines with engines built from source. Please report a run on an +RX 6000 card with the end of `strata-.log`. -- **Build and tests** (engine 0.1.26): a complete HIP build for gfx1030, made by hand with cmake and ROCm's own - `clang++` (the nixpkgs ROCm is not an `/opt/rocm` tree, so setup's `build_engine_hip` was not exercised; the - binary was placed in `engine/` for setup to use). All 28 registered ctest tests pass on the card, including - `hip_device_selftest`. An engine 0.1.30 build of this branch configures, builds and runs clean on the same card; - the speeds below are measured on it. -- **End to end** (Swift 1.5 IQ3_XXS, `--context 131072` with `--kv-resident 32768 --adapt-every 1 - --vram-reserve-mib 1024`, 200 greedy tokens): 38-42 tok/s decode with the default 15 CPU pool workers (one per - physical core except the host thread; 8 workers: 36; 24 = every logical core: 27), consistent even with the - 131,072-token context full. -- **Prefill** (the same 15-worker configuration): 246 tok/s on a 2,000-token prompt, 330-339 tok/s at the auto - 8,192-token chunk (7,997 and 15,967 tokens; time to first token 8.9 and 48.3 s), 133 tok/s on a 522-token - prompt - short prompts are fixed overhead (19-25 tok/s on 34 tokens). Decode after a 16K prefill holds at - 45.5 tok/s. -- **16 GB card:** the expert cache holds 6,310 slots (10.2 GiB) at 16K context and 5,247 slots at 131,072, where - setup keeps the KV cache in VRAM because IQ3_XXS needs about 60 GB of RAM plus the cache to stream it. The - `--pcie-frac 0` and `--adapt-every 0` of the 7900 XTX configuration in - [AMD_HIP_PERFORMANCE.md](AMD_HIP_PERFORMANCE.md) cost 8.6 and 13.6 tok/s here (30 with the defaults): keep the - defaults on a 16 GB card. -- **hipBLASLt:** ROCm's hipBLASLt ships no gfx1030 kernels, so there is no table and the plain hipBLAS path runs. -- **Prompt GEMMs in FP16 (#835):** rocBLAS on gfx1030 has tuned kernels for FP16-in / FP16-out GEMMs only; the prompt - path's FP16-in / FP32-out and BF16 products ran generic kernels about 6.6x slower (5.6 and ~5.3 TFLOPS against 37.7 at - N = 10240, T = 7313, K = 2560). `STRATA_HIP_PROMPT_F16=1` (opt-in, off by default; the engine prints a tip on gfx103x) runs the 16-bit prompt GEMMs FP16 in and out; - unset is the old path. `STRATA_DBG_NAN=1` also counts the non-finite FP16 outputs (a sum past 65504). The output is not bit-identical to it (an FP16 rounding of each GEMM's output replaces the BF16 rounding of - the BF16 GEMMs' activations); what the distribution check showed and did not show is in - [bench/results/2026-10-04-rdna2-fp16-prompt](../bench/results/2026-10-04-rdna2-fp16-prompt/README.md). FP16 ends at - 65504: `STRATA_F16_RANGE=1` prints, per prompt, the largest value that entered or left FP16 and how many were beyond - the range (466K tokens there: outputs peak at 410, activations at 112, none beyond; the image writers saturate, a GEMM - output beyond the range would be Inf and is what the counter is for). Measured on a - second community machine - 2x RX 6900 XT 16 GB (one card for these numbers), Ryzen 5 5600X (6 cores, AVX2), 128 GB, - Ubuntu 26.04, ROCm 10.0.0, engine 0.1.39 built by hand with `-DSTRATA_PREFILL_MMQ=ON` as setup does, GSQ-RCO IQ3_S: - a 9.4K / 34.7K / 105.8K-token prompt reads at 744 / 915 / 926 tok/s instead of 439 / 466 / 461; decode is unchanged. - On that machine the HIP ctest passes 60 of 65 with and without this change: 2 skipped (`hip_prompt_attn_wmma`, no - matrix cores; `hip_prefill_hipblaslt_gemm`, no hipBLASLt table) and 3 that fail for reasons outside the engine - (`ple_parity` needs a Q2_0 PLE file that is not on that machine, `expert_multi_test` refuses the CPU without AVX-512, - `platform_memory_test` cannot `mlock` at the shell's default `ulimit -l`). -- **The verifier's hyper-connection read as one persistent kernel:** `STRATA_HC_PERSIST=1` (opt-in, HIP, 32-lane - waves) runs the read of 1-8 tokens as one kernel of 80 blocks with two grid barriers instead of three launches; a - start-up run per token count with a barrier timeout decides where it is resident, and one `strata hc:` line per card - says so. On 2x RX 6900 XT (ROCm 10.0, layer split): the read 1.35-2.10x, ms per verify window -4.2 % / -2.8 % at - 4K / 32K (6 rounds), not bit-identical (as close to a double-accumulation reference as the default read): - [bench/results/2026-10-12-hc-persist-gfx1030](../bench/results/2026-10-12-hc-persist-gfx1030/README.md). Other - RDNA cards are untested; the `strata hc:` line and a `STRATA_DECODE_TIMING` run from one are welcome. Not offered on gfx11 and - gfx12 (RDNA3, RDNA4): the barrier timeout needs `s_memrealtime`, which gfx1100 and gfx1200 / gfx1201 do not have, and an R9700 (#2226 thread) measured the - read +0.7 % / +1.0 % per window slower than the default one; the start-up line says so. A barrier that times out at run time - (never seen in a working run) leaves the partial outputs of that read in place and every later persistent read of the - process aborts at once, without a log line; the opt-in is for measurement on gfx103x until that is closed. -- **Prompt GEMMs as SGEMMs (engine 0.1.39 + `Gemm::rdna2_sgemm`):** the rocBLAS in setup's ROCm (10.2.0a20260930) - has tuned gfx1030 kernels for FP16 -> FP16, int8 and FP32 -> FP32 only. The prompt path's dense GEMMs take FP16 or - BF16 inputs and write FP32 (rocBLAS's HS / BS types), which run its fallback kernels at about 5 TFLOPS. With - `STRATA_RDNA2_SGEMM=1` (opt-in; `=all` forces it on any AMD card) the engine widens both inputs to FP32 (exact) and runs SGEMM, - which accumulates in FP32 like the native call (`src/prefill/gemm.cu`; N < 64 keeps the native call). The author ran it - as the gfx103x default; here it is a switch because it is not bit-identical to the native call (below). Measured on an RX 6800 16 GB over OCuLink (PCIe 4.0 x4, 7.1 GB/s), Ryzen 7 8845HS, 28.8 GB RAM, Windows 11, - Coder IQ1_M with setup's arguments (64K context, MTP), fresh 12K-token prompts, 4 per arm, A/B in one binary: +hipBLASLt ships no gfx1030 kernels, so there is no hipBLASLt table and the plain hipBLAS path runs. rocBLAS on gfx1030 has tuned +kernels for FP16-in / FP16-out GEMMs only; the prompt path's FP16-in / FP32-out and BF16 products run generic kernels about +6.6x slower. The switches below address that. All are opt-in and none is bit-identical to the default. - | | native GEMMs | SGEMM route | - |---|---|---| - | prompt | 312 tok/s | 452 tok/s | - | prompt GPU timeline (`STRATA_PREFILL_TIMING`) | 37.2 s | 25.5 s | - | of which gdn / qsa proj / hc read | 9.4 / 3.6 / 3.85 s | 2.8 / 1.4 / 2.84 s | - | decode | 29-34 tok/s | 29-35 tok/s | +| Switch | Effect | +| --- | --- | +| `STRATA_HIP_PROMPT_F16=1` | Runs the 16-bit prompt GEMMs FP16 in and out (the engine prints a tip on gfx103x). `STRATA_F16_RANGE=1` prints, per prompt, the largest value that entered or left FP16 (FP16 ends at 65504); `STRATA_DBG_NAN=1` counts non-finite FP16 outputs. Checks: [2026-10-04-rdna2-fp16-prompt](../bench/results/2026-10-04-rdna2-fp16-prompt/README.md). | +| `STRATA_RDNA2_SGEMM=1` | Widens both GEMM inputs to FP32 (exact) and runs SGEMM, no 65504 limit. `=all` forces it on any AMD card. With `STRATA_HIP_PROMPT_F16=1` the FP16 route runs first and SGEMM takes the products it leaves. Slower than the FP16 route. | +| `STRATA_ROCBLAS_TUNING=` | A rocBLAS solution table for the FP16 route; setup points the engine at the directory when a table for the architecture exists. The log says `rocBLAS tuning enabled (... rows ...)` or why not. See [rocBLAS table](#rocblas-table-gfx103x). | +| `STRATA_SELECT_SIMT=1` | QSA block scores on the FP32 tiled scorer instead of the warp kernel; helps long prompts (128K: +7.4%). Not bitwise the warp kernel. [Data](../bench/results/2026-10-07-rdna2-qsa-select-simt/README.md). | +| `STRATA_HC_PERSIST=1` | The verifier's hyper-connection read as one persistent kernel (HIP, wave32). Measured on 2x RX 6900 XT: -2.8% to -4.2% ms per verify window. Not offered on gfx11 and gfx12. Opt-in for measurement on gfx103x. [Data](../bench/results/2026-10-12-hc-persist-gfx1030/README.md). | +| `STRATA_FILE_CACHE_OLD_RULE` | gfx906 keeps the old file-tier cache rule; `=1` forces it on any Linux build, `=0` the current one | - The exact calls on the card (opA = T, opB = N, FP32 out): N 10240 x T 8192 x K 2560 86.7 -> 27.7 ms including - the widening; N 320 x T 8192 x K 10240 (BF16) 10.9 -> 7.5 ms. Against a float64 reference on random inputs in - [-1, 1], the SGEMM route is as close as or closer than the native call on all 10 FP16 / BF16 shapes checked; the - two differ by at most 4.3e-6 of the largest output. A 9-task code-repair benchmark (the model patches seeded bugs - in a browser game until 38 browser checks pass) solved 9/9 on the first try with both, model time 458 -> 374 s. - With the switch on and `STRATA_HIP_PROMPT_F16` unset the SGEMM route runs; with `STRATA_HIP_PROMPT_F16=1` - the FP16 route above runs first and the SGEMM route takes the products it leaves (`beta != 0`). Same RX 6800, one - session, fresh 12K-token prompts: native 310, SGEMM 439, FP16 (#835) 494 tok/s; `STRATA_F16_RANGE=1` on the Coder - IQ1_M peaked at 105.3 (activations) and 424.8 (FP16 outputs) with nothing beyond 65504, and the repair benchmark - solved 9/9 on the first try with both (model time 374 s SGEMM, 307 s FP16). The SGEMM route is the slower of the - two; no value passes through FP16, so it has no 65504 limit to stay under. It is not bit-identical to the native - call (at most 4.3e-6 of the largest output apart), which can flip a near-tied greedy token like any such change. -- **rocBLAS solution table** (for the `STRATA_HIP_PROMPT_F16=1` route above; with it off, nothing runs through the - table): rocBLAS picks the kernel for each of those FP16 GEMMs from its own table by shape, and - on gfx1030 that pick is far from the best kernel the library holds for some shapes: the GDN input projection - (N = 10240, K = 2560) below about 1,150 tokens runs at 5 TFLOPS where the best of rocBLAS's own 237 solutions for the - same shape runs at 32-38 (7x; the kernel choice flips at T = 1152). `tools/hip/tune_rocblas` enumerates the library's - solutions per shape and token bucket, checks each against the default kernel's output and writes the winners as - `tools/hip/-rocblas-.txt`; the engine reads the table through `STRATA_ROCBLAS_TUNING` (a file, or - the directory, from which it takes the file of its own rocBLAS build) and runs those solutions in place of rocBLAS's - choice, the default kernel for every shape without a row. Solution indices are valid for one architecture and one - rocBLAS build (the full version string, `5.6.0.8d1ae90e` for ROCm 10.0.0's), and the engine refuses any other - table. Shipped: `gfx1030-rocblas-5.6.0.8d1ae90e.txt` (ROCm 10.0.0, the `/opt/rocm` setup installs on Ubuntu), measured - on an RX 6900 XT. Other gfx1030 cards (RX 6800 / 6800 XT / 6950 XT, Radeon PRO W6800 / V620) run it as well - the - indices are valid on every gfx1030 - but have fewer CUs or other clocks, so their own table may pick differently; - `tune_rocblas` makes one in about 5 minutes ([Tuning table](#tuning-table)), and a report of what it changed is welcome. Setup - points the engine at the directory when a table for the card's architecture exists, and the engine's log says - `rocBLAS tuning enabled (... rows ...)` or why not. Measured on the 2x RX 6900 XT machine above (one card, #835 + - this, `STRATA_PREFILL_TIMING=1`, temperature 0, the same answers with and without): a 799-token prompt's GPU time - 3,329 -> 2,985 ms (the GDN projections 443 -> 177 ms, `hc read` 270 -> 77), 495 tokens 2,477 -> 2,139 ms (365 -> - 146), 8,874 tokens 11,978 -> 11,459 ms (the 682-token tail chunk's projections, `hc read` 1,200 -> 898); a - 1,108-token prompt did not change (3,391 -> 3,395 ms): its GDN time fell 598 -> 201 ms and the whole saving - reappeared as `wait copy`, the experts' PCIe transfer, which bounds that prompt. Per-run lines and the tuner's - summary: [bench/results/2026-10-05-rdna2-rocblas-table](../bench/results/2026-10-05-rdna2-rocblas-table/README.md). Decode does not use these GEMMs and is unchanged. See [Tuning - table](#tuning-table) for making a table for another rocBLAS build. -- **QSA block scores on the FP32 tiled scorer (`STRATA_SELECT_SIMT=1`, opt-in, as on CUDA below sm_80):** without - matrix cores the prompt path's QSA selection scores every (query, KV block) pair on the warp kernel, the one prompt - term that grows with the context. The switch runs the FP32 tiled scorer instead (FP32 in another summation order: - not bitwise the warp kernel, so off by default), tiled 32/128/16/8 on gfx103x. `qsa_select_bench`, 256 queries: - 2.69 -> 0.587 ms at 128K (the original 32/64/32/4 tiling: 0.695), 0.67 -> 0.169 ms at 32K; FP64 gate passed, - selections identical 256/256. Every tiling gives the same bytes (each score is one fmaf chain over k in order), and - the fastest tiling was the same with 60 and 72 of the card's 80 CUs (`HSA_CU_MASK`). On the 2x RX 6900 XT split - (0.1.40.3, IQ3_S, `--kv int8 --kv-resident 32768`, two cold starts per arm, ABBA) a 128K-token prompt reads 1,735 -> - 1,861 and 1,727 -> 1,858 tok/s (+7.4%), 32K +0.8%; decode is not affected (the text changes with the scores' order, - so its speed is not compared). Measurements: - [bench/results/2026-10-07-rdna2-qsa-select-simt](../bench/results/2026-10-07-rdna2-qsa-select-simt/README.md). -- **File-tier cache rule on gfx906 (`STRATA_FILE_CACHE_OLD_RULE`):** on an MI50 ROCm queues are suspended while the kernel reclaims around the process's HMM-tracked arena (#1752, KevinX8: stalls, prompt 46 -> 360 tok/s once the file tier stops churning the page cache). The gfx906 backend therefore keeps the old rule (the file cache is used only when the room left is at least the bytes read from the files); other HIP cards keep the current rule. `STRATA_FILE_CACHE_OLD_RULE=1` forces the old rule on any Linux build, `=0` the current one. -- **Windows** (the report above): setup's ready-made Windows engine 0.1.39 runs the RX 6800 beside a 780M iGPU - (HIP numbers the RX 6800 1; setup points the engine at it), and `tools/hip/build_windows.bat` with - `STRATA_HIP_ARCHS=gfx1030` builds a working engine from the `gfx103X-all` 10.2.0a20260930 wheels. -- **gfx1031** (RX 6700 XT, #524): setup knows it (the `gfx103X-all` wheels, unvalidated); its reporter runs it daily - on one card. More reports: an RX 6700 XT 12 GB run as gfx1030 on ROCm 7.2.4 (#1027: IQ2_XS, decode 30-32 tok/s, prompt about 300 tok/s, 6 of 6 - needles), and an RX 6800M 12 GB on Windows with a self-built engine (#915, #1078: Q2_0, decode 9-27 tok/s, prompt 50-114 tok/s; the - experts stream from disk with 31 GB of RAM). The ready-made Windows zip has no gfx1031 code unless it is built with it: `STRATA_HIP_ARCHS=gfx1031` - in `tools\hip\build_windows.bat` (the default list has it from 0.1.40.2 on). -- **gfx1150** (Radeon 890M, Ryzen AI 9 HX PRO 370, Strix Point, #1217): builds and runs from 0.1.40.2 as an unvalidated target (CMake warns; the - device code already covers it: the WMMA guards and `gfx_arch_is_gfx11_wmma()` include it). One community machine: MINISFORUM N5 PRO, 96 GB DDR5-5600, GTT - raised to 64 GiB (`ttm.pages_limit=16777216 ttm.page_pool_size=16777216`), TheRock ROCm 7.14.1 for gfx1150, an unprivileged LXC container, IQ3_XXS: chat and - tool calls work, decode 15-19 tok/s, the HIP ctest passes apart from tests that need files a public checkout lacks. - Prompts read at about 125-130 tok/s on plain hipBLAS and 1.5-1.7x faster with `tools/hip/gfx1150-hipblaslt-100401.txt` (ROCm 7.14.1's - hipBLASLt; 3.6K tokens 124 -> 216 tok/s, 7K 130 -> 226 tok/s, decode unchanged): - [bench/results/2026-10-07-community-gfx1150](../bench/results/2026-10-07-community-gfx1150/README.md). - Setup does not install for it yet (an integrated Radeon other than Strix Halo is named and not supported): build by hand with `-DCMAKE_HIP_ARCHITECTURES=gfx1150` - and point `STRATA_HIPBLASLT_TUNING` at the table yourself. -- **gfx1152** (Radeon 860M / 840M, Ryzen AI 300 "Krackan", #1625): in CMake's unvalidated list like gfx1150, so - `-DCMAKE_HIP_ARCHITECTURES=gfx1152` builds with a warning. It runs the portable kernels (the matrix-core guards name - gfx1150 and gfx1151 only), no card has reported on it, and setup does not install for it (an integrated Radeon other - than Strix Halo is named and not supported). A report from a real 860M is welcome. -- **gfx1032** (RX 6600/6600 XT): unvalidated; built with `STRATA_HIP_ARCHS=gfx1030;gfx1032` and ran under Windows as - the *helper* expert GPU of an RX 6800 ([SECOND_GPU.md](SECOND_GPU.md)), 3.4-3.8k expert slots in its 8 GB and - +8-18% decode over the 6800 alone. Its 14 WGP make a helper row about 2-3x the 6800's cost, so it pays as a - cache tier for the CPU pool, not as a compute peer. No report of it as a primary card. -- **Not validated:** setup's own build path and the - `gfx103X-all` wheels on gfx1030, images, answer-quality benchmarks. RDNA1 (gfx1012, RX 5500 XT) builds by hand: - [OLDER_GPUS.md](OLDER_GPUS.md#amd-building-gfx906-and-gfx1012). +On a 16 GB card keep the defaults for `--pcie-frac` and `--adapt-every`; the 7900 XTX values in +[AMD_HIP_PERFORMANCE.md](AMD_HIP_PERFORMANCE.md) cost 8.6 and 13.6 tok/s on an RX 6900 XT. + +> **Measured on:** RX 6900 XT 16 GB (gfx1030), i7-13700KF, 63 GB RAM, NixOS, ROCm 7.2.3, engine 0.1.30, Swift 1.5 IQ3_XXS, +> `--context 131072 --kv-resident 32768 --adapt-every 1 --vram-reserve-mib 1024`, 15 CPU pool workers (one per physical +> core except the host thread). + +| Workload | Result | +| --- | --- | +| Decode, 200 greedy tokens | 38-42 tok/s (8 workers: 36; 24 workers: 27) | +| Prefill, 2,000-token prompt | 246 tok/s | +| Prefill, auto 8,192-token chunk (8K and 16K prompts) | 330-339 tok/s | +| Decode after a 16K prefill | 45.5 tok/s | + +The 16 GB card holds 6,310 expert slots at 16K context and 5,247 at 131,072. + +Other gfx103x and RDNA2-adjacent cards (community reports): + +| Card | Report | +| --- | --- | +| gfx1031, RX 6700 XT 12 GB (#524, #1027) | Setup knows it (`gfx103X-all` wheels, unvalidated). IQ2_XS on ROCm 7.2.4: decode 30-32 tok/s, prompt about 300 tok/s, 6 of 6 needles. The default list of `build_windows.bat` includes it. | +| RX 6800M 12 GB, Windows, self-built (#915, #1078) | Q2_0: decode 9-27 tok/s, prompt 50-114 tok/s; experts stream from disk with 31 GB RAM | +| gfx1032, RX 6600 / 6600 XT | Unvalidated. Ran under Windows as the helper expert GPU of an RX 6800 ([SECOND_GPU.md](SECOND_GPU.md)): +8-18% decode over the 6800 alone. No report as a primary card. | +| gfx1150, Radeon 890M (#1217) | Builds and runs as an unvalidated target. MINISFORUM N5 PRO, 96 GB, TheRock ROCm 7.14.1, IQ3_XXS: decode 15-19 tok/s, prompts 125-130 tok/s on plain hipBLAS and 1.5-1.7x faster with `gfx1150-hipblaslt-100401.txt`. Setup does not install for it: build with `-DCMAKE_HIP_ARCHITECTURES=gfx1150` and point `STRATA_HIPBLASLT_TUNING` at the table. [Report](../bench/results/2026-10-07-community-gfx1150/README.md). | +| gfx1152, Radeon 860M / 840M (#1625) | Builds with a warning; portable kernels; no report; setup does not install for it | + +On Windows, setup's ready-made engine runs an RX 6800 beside a 780M iGPU (HIP numbers the RX 6800 as 1). ## gfx906 (Instinct MI50 / MI60, Radeon VII): wave64, built from source -gfx906 is wave64 and has no WMMA and no packed byte arithmetic, so the wave32 backend above refuses it. A separate opt-in build compiles the CUDA sources as HIP through a small compat layer -(`include/strata/platform/hip_compat/`), with a CUDA warp mapped to a logical half of the 64-lane wavefront -(32-wide shuffles, a ballot of its own half). The hot kernels have wave64 layouts of their own (below). Setup does -not build it yet: build by hand, and run `serve/server.py` with a config as on any other card. +gfx906 is wave64 and has no WMMA and no packed byte arithmetic, so the wave32 backend refuses it. A separate opt-in build +compiles the CUDA sources as HIP through a small compat layer (`include/strata/platform/hip_compat/`), with a CUDA warp mapped +to a logical half of the 64-lane wavefront. Setup does not build it: build by hand and run `serve/server.py` with a config as +on any other card. -**Community-tested, opt-in (0.1.42).** The gfx906 build is maintained with its users: we have no gfx906 card, so what is here was measured by community members on their MI50s (the attention switches `STRATA_GFX906_ATTN_QUERY_SWIZZLE=1` and `STRATA_GFX906_ATTN_REDUCE12=1`, PRs #1661 and #1718 by 0FL01, are off by default and compiled only into this build) and the engine here is checked to compile for gfx906, nothing more. Nothing in it changes the builds for other cards. A wave32 card seen by this build is refused at start with a message (#1728). Please report how it runs on your card. +The gfx906 build is community-tested: the maintainers have no gfx906 card, and the engine here is checked to compile for +gfx906 and nothing more. The attention switches `STRATA_GFX906_ATTN_QUERY_SWIZZLE=1` and `STRATA_GFX906_ATTN_REDUCE12=1` are +off by default and compiled only into this build. A wave32 card seen by this build is refused at start. The Qwen3.6 family +is not part of this build; `strata --families` lists what the binary runs. Please report how it runs on your card. -**ROCm.** AMD's current ROCm releases no longer ship gfx906 libraries. The build and the measurements below used -HIP 7.14 from the community image [`mixa3607/rocm-gfx906:7.14-complete`](https://hub.docker.com/r/mixa3607/rocm-gfx906) -(rocBLAS/hipBLAS with gfx906 kernels), on the kernel's amdgpu driver (Ubuntu 24.04, kernel 6.8). hipBLASLt is not -used. +**ROCm.** AMD's current ROCm releases no longer ship gfx906 libraries. The build used HIP 7.14 from the community image +[`mixa3607/rocm-gfx906:7.14-complete`](https://hub.docker.com/r/mixa3607/rocm-gfx906) (rocBLAS/hipBLAS with gfx906 kernels) +on the kernel's amdgpu driver. hipBLASLt is not used. ```sh # inside the image, with this checkout at /src and llama.cpp at the pinned commit in third_party/llama.cpp @@ -842,65 +517,46 @@ cmake -S . -B build-906 -G Ninja -DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURE ninja -C build-906 strata ``` -`STRATA_HIP_GFX906` turns on the CUDA targets and refuses `STRATA_ENABLE_HIP` (one HIP build at a time). Run the -container with `--device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --ulimit memlock=-1`; on some boards +`STRATA_HIP_GFX906` turns on the CUDA targets and refuses `STRATA_ENABLE_HIP` (one HIP build at a time). Run the container with +`--device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --ulimit memlock=-1`; on some boards `HSA_OVERRIDE_GFX_VERSION=9.0.6` is needed for the runtime to accept the card. -0.2.1 builds `ninja -C build-906 strata` again (0.1.44 to 0.2.0 stopped at `cublasSgemm` in `src/qwen35/q36_engine.cu`, at -`cuda.h` in `strata_mmq_w4a4` and at three link errors, #3172 #3271; compiled for gfx906 on ROCm 7.10 with clang 22, all 161 targets, -not run on a card). The Qwen3.6 family (`qwen35moe`) is not part of the gfx906 build: its engine calls cuBLAS entry points the compat -layer does not map and its kernels assume 32-lane warps. `strata --families` lists what the binary runs (`qwen4exp`: Qwen3.8-Flash-Next). +**What differs from the CUDA build** (each A/B switch restores the CUDA layout, and was checked bitwise against it): -**What differs from the CUDA build** (all under `STRATA_HIP_GFX906`; each A/B switch restores the CUDA layout): +| | gfx906 | Switch | +| --- | --- | --- | +| Grouped native experts | Mode 7: signs as `dp4a(g ^ m, u) - dp4a(m, u)`, grids and activations in LDS. IQ formats only (K-quant / Q5_1 / Q8_0 experts take the CUDA layout). | `STRATA_EXP_MODE=2` | +| MMVQ | One wavefront per row, 64-lane butterfly, no LDS | `STRATA_MMVQ_WAVE=0` | +| Router top-10 | One wavefront, register argmax | none | +| Verify-window routing | The window's tokens in one multi-column BF16 projection and one top-k | `STRATA_ROUTE_PER_TOKEN=1` | +| Hyper-connection read | Norm loads in flight, 8 lanes per row for `up` | `STRATA_GR_FAST=0` | +| `gr_down_multi`, opt-in | Split along K, about 3% per window; sums in another order | `STRATA_GR_SPLIT=1` | +| IQ4 table lookup | llama.cpp's `v_perm_b32` sequence | none | +| QSA prompt attention | The FP32 kernel | none | -| | gfx906 | switch | -|---|---|---| -| grouped native experts | mode 7: signs as `dp4a(g ^ m, u) - dp4a(m, u)` (no byte SIMD), grids and activations in LDS, one weight load per group; the IQ formats only (Unsloth's K-quant / Q5_1 / Q8_0 experts take the CUDA layout) | `STRATA_EXP_MODE=2` (previous AMD layout) | -| MMVQ | one wavefront per row, 64-lane butterfly, no LDS | `STRATA_MMVQ_WAVE=0` | -| router top-10 | one wavefront, register argmax | - | -| verify-window routing (256-expert router) | the window's tokens in one multi-column BF16 projection + one top-k, ~3 ms of a ~55 ms window | `STRATA_ROUTE_PER_TOKEN=1` | -| hyper-connection read | norm loads in flight, 8 lanes per row for `up` | `STRATA_GR_FAST=0` | -| `gr_down_multi`, opt-in | split along K (8 rows x 5 slices of 2048), ~3% per window; sums in another order than the single-token read, so off by default | `STRATA_GR_SPLIT=1` | -| IQ4 table lookup | llama.cpp's `v_perm_b32` sequence (HIP's `__byte_perm` is a scratch array) | - | -| QSA prompt attention | the FP32 kernel (no tensor cores / WMMA) | - | +**Two cards: try `--pipeline-windows 2` (off by default on gfx906).** The first card starts the next verify window while the +second still verifies this one ([MULTI_GPU.md](MULTI_GPU.md)). Add `--trim-stage-weights` with the explicit `--layer-split`. +In the config: `"args": [ ..., "--pipeline-windows", "2", "--trim-stage-weights" ]`. `--pipeline-windows 0` is the plain loop. +`--batch` did not help a single interactive user. Nobody has compared the tokens against the serial loop on gfx906. -Every switch above was checked bitwise against the layout it replaces: the parity tests on synthetic data, -`native_expert_bench` on real GGUF rows (one mode against another), and the greedy text of a fixed request. +> **Measured on:** 2x Instinct MI50 16 GB, Ryzen 7 5700X, 64 GB DDR4-3600, ROCm 7.14 (community image), engine 0.1.43.1, +> original IQ3_XXS, 131,072 context, `--layer-split 27 --spec 3 --kv int8 --kv-resident 32768`, median of 7 runs, 1.6K-token +> prompt, 128 greedy tokens (#3173). One machine, one model. -**Measured** on 2x Instinct MI50 16 GB (85 W power limit each, PCIe 3.0 x16 both, peer access between them), -Xeon E5-2666 v3 (10 cores, AVX2), 32 GB DDR4, SATA SSD; Coder IQ1_M (`--native` pack), 131,072-token context, -`--kv int8 --kv-resident 32768`, `--layer-split 27`, `--prefill 4096`, MTP `--spec 4 --spec-min-p 0.5`, -`--pcie-frac 0`, `STRATA_ARENA_MMAP=1`. Measured on the port this build was cut from (builds on the 0.1.30 and 0.1.33 bases, the same -kernels, plus the layer-split weight trim and the mapped arena, which are separate pull requests): +| Setting | Decode | Prompt | +| --- | ---: | ---: | +| Without pipelining | 49.2 tok/s | 353 tok/s | +| `--pipeline-windows 2` | 66.9 tok/s (+36%) | 353 tok/s | +| plus `--trim-stage-weights` | +2% | 371 tok/s | -- all 12,288 experts held in VRAM across the two cards; -- a 17,043-token agent prompt (Claude Code's first request, tools included) read in 43.5 s (392 tok/s), decode - on it 57.7-59.3 tok/s; -- 25 GB of the 32 GB RAM available while serving; -- six repeats of that request with a 6,000-token output cap and a 900-second two-client load test run six times - (407 requests, 202,940 tokens): no stalls, no errors, no GPU faults. - -For comparison on the same machine: llama.cpp (3cf0325, ROCm) on the same Coder IQ1_M measured 26.9 tok/s `tg128`. - -**Not done:** setup.py detection and an automatic build, Windows, images (`--vision`), MI60 and Radeon VII (the -same gfx906 ISA; not run), a single-card run, the tensor-split experiment (the halves of every layer on two cards; -it works but is not part of this build). - -**Two cards: try `--pipeline-windows 2` (community measurement, #3173; off by default on gfx906).** On 2x Instinct MI50 16 GB -(Ryzen 7 5700X, 64 GB DDR4-3600, ROCm 7.14 from the community image, engine 0.1.43.1) with the original IQ3_XXS at 131,072 tokens -of context, `--layer-split 27`, `--spec 3 --kv int8 --kv-resident 32768`, median of 7 runs (1.6K-token prompt, 128 greedy tokens): -decode 49.2 tok/s without it, 66.9 with `--pipeline-windows 2` (+36%); the prompt speed stayed 353 tok/s. The first card starts the -next verify window while the second still verifies this one ([MULTI_GPU.md](MULTI_GPU.md)). On RDNA4 the tokens are the serial -loop's up to CPU rounding; nobody has compared them on gfx906 yet. Add `--trim-stage-weights` (with the explicit `--layer-split`): prompt reading 353 -> 371 tok/s, decode +2%. -In the config: `"args": [ ..., "--pipeline-windows", "2", "--trim-stage-weights" ]`. One machine, one model, one run protocol: measure it -on yours (`--pipeline-windows 0` is the plain loop) before you keep it. `--batch` did not help one user there (aggregate 20 -> 23 tok/s -at two requests, each request slower), so leave it off for a single interactive user. +Not done on gfx906: setup detection and automatic build, Windows, images (`--vision`), MI60 and Radeon VII (same ISA, not run), +a single-card run. Other measurements are under [History](#history--older-measurements). ## Tuning table -A hipBLASLt table holds solution ids that are valid only for one GPU architecture and one hipBLASLt version, so it -is calibrated on the card with the ROCm the engine runs with. The shipped gfx1100 table's rows are the engine's -dense GEMM shapes; to calibrate them for another card or version (a few minutes): +A hipBLASLt table holds solution ids that are valid only for one GPU architecture and one hipBLASLt version, so it is calibrated +on the card with the ROCm the engine runs with. To calibrate the shipped gfx1100 table's rows for another card or version (a +few minutes): ```sh cmake --build build-hip --target tune_hipblaslt @@ -908,216 +564,167 @@ CASES=$(awk 'NR>2 {printf " --case %s,%s,%s,%s,%s", $1, $5, $2, $3, $4}' tools/h ./build-hip/tune_hipblaslt $CASES --tuning-out table.txt ``` -The file's second line names the architecture and version (`STRATA_HIPBLASLT_TUNING_V1 gfx1201 100401`); save it -as `tools/hip/-hipblaslt-.txt` for setup, or point `STRATA_HIPBLASLT_TUNING` at it. Compare the -prompt speed with and without it before keeping it. For an engine package, `packaging/stage_tuning.py` (Linux builds) and -`tools/hip/package_windows.py` copy the tables that match the bundled hipBLASLt into the package's `tuning/` folder. +The file's second line names the architecture and version (`STRATA_HIPBLASLT_TUNING_V1 gfx1201 100401`). Save it as +`tools/hip/-hipblaslt-.txt` for setup, or point `STRATA_HIPBLASLT_TUNING` at it. Compare the prompt speed with and +without it before keeping it. For an engine package, `packaging/stage_tuning.py` (Linux) and `tools/hip/package_windows.py` +copy the tables that match the bundled hipBLASLt into the package's `tuning/` folder. `STRATA_HIPBLASLT_VERBOSE=1` reports +fallbacks; look for `fallbacks=0`. The test `hip_prefill_hipblaslt_gemm` is a smoke test (it checks 2 rows), and it does not +prove that every solution id is valid. -**Measured for 0.2.1 on a Radeon AI PRO R9700 (gfx1201), the Linux `rx9070` package 0.1.45.3, Q2_0, hipBLASLt 1.2.0, one card, the -box to itself.** A 1.2.0 table (32 rows, `tune_hipblaslt` over the gfx1201 100500 table's cases) against plain hipBLAS, 6 interleaved pairs -with a fresh engine per run and a second plain arm as the noise arm: prompt 4K 2,302 vs 2,119 tok/s and 20K 2,671 vs 2,507 (5 and 6 of 6 pairs -faster, but a plain run alone lands on either of two speeds, 1,970 or 2,290 at 4K, so the medians mix modes; inside one mode the table is -+0.5% at 4K and +1% at 20K), decode 91.2 vs 93.2 tok/s (-2.1%, 0 of 6 pairs faster). Not bit-identical: the final residual of a 4K and a -20K prompt differs, a 48-token greedy answer after the 20K prompt differs, and the first-token KL against plain is 0.0157 mean, 0.0716 max over -24 prompts of 1.5K-24K tokens (plain against plain: 0 and 0.00004; top-1 token the same 24 of 24; 24-token answers equal 1 of 6). That is why -no gfx1201 table is shipped for hipBLASLt 1.2.0 and why packaged engines use a table only when asked. +Tables in `tools/hip`: -Shipped tables: +| File | Card, calibrated with | Notes | +| --- | --- | --- | +| `gfx1100-hipblaslt-100100.txt`, `-100200.txt`, `-100202.txt` | RX 7900 XTX | 100202 is for system ROCm 7.2.4 | +| `gfx1100-hipblaslt-100401.txt` | RX 7900 XTX, ROCm 10.0.0 | 43 rows; setup uses it only with hipBLASLt 1.4.1 | +| `gfx1100-hipblaslt-100500.txt` | RX 7900 XTX, ROCm 10.2.0a20261003 nightly | 32 rows | +| `gfx1201-hipblaslt-100202.txt` | R9700, for hipBLASLt 1.2.2 | 24 of the gfx1100 table's 26 shapes | +| `gfx1201-hipblaslt-100500.txt` | R9700, ROCm 10.2.0a20260914 (`gfx120X-all` nightly) | 32 rows. The engine can check only the version number, so recalibrate for a different 1.5.0 build. | +| `gfx1200-hipblaslt-100202.txt` | RX 9060 XT | See [Community-validated cards](#community-validated-cards) | +| `gfx1151-hipblaslt-100400.txt`, `-100401.txt`, `-100500.txt` | Radeon 8060S (Strix Halo) | 100400 is for the 7.14.0a wheels setup installs; 90 rows. Used by default on Strix Halo. | +| `gfx1150-hipblaslt-100401.txt`, `-100500.txt` | Radeon 890M | Setup does not use it; point `STRATA_HIPBLASLT_TUNING` at it | +| `gfx1103-hipblaslt-100401.txt` | Radeon 780M | 143 rows. Setup installs for gfx1103 only with `STRATA_EXPERIMENTAL_GFX1103=1`. [Report](../bench/results/2026-10-12-community-gfx1103/README.md). | -- `gfx1100-hipblaslt-100100.txt`, `gfx1100-hipblaslt-100200.txt`: RX 7900 XTX. -- `gfx1100-hipblaslt-100401.txt`: RX 7900 XTX, calibrated with the packaged ROCm 10.0.0 - (`rocm/dev-ubuntu-24.04:10.0.0-full`, hipBLASLt 1.4.1) on a Ryzen 7 9800X3D, over the 26 dense GEMM - geometries of the shipped gfx1100 table. Without it, that stack reads a prompt through plain hipBLAS: on - Swift 1.5 IQ3_XXS at 262144 ctx, fresh-prompt prefill measured 682 -> 1,038 tok/s at 1.7K tokens and - 857 -> 1,524 tok/s at 6.5K (medians of 3, decode unchanged), with `hip_prefill_hipblaslt_gemm` reporting - `fallbacks=0`. setup uses it only when the installed hipBLASLt reports 1.4.1. - The 43-row revision adds the two geometries the 26 dense ones do not reach, calibrated with - `tune_hipblaslt` from the same tree and the engine's 32 MiB workspace: `f16 1280 2560 1280` (the - small-T expert GEMM, T=8..999) and `bf16 10240 2560 10240`. A prompt on Qwen3.8-Flash-Next GSQ-RCO - IQ2_XS at 192000 ctx otherwise falls back on the expert GEMM for nearly every launch: - `STRATA_HIPBLASLT_VERBOSE=1` reported 642 fallbacks over 642 distinct T values (1..1873) with the - 26-row table. Adding the rows measured 711 -> 938 tok/s (+31.9 %, mean of 3 interleaved pairs: - +36.0 / +32.4 / +27.6 %) at a 3151-token prompt, decode unchanged (58.6 -> 60.6 tok/s, inside this - box's 2.3 % run drift), and `fallbacks=0`. The lookup takes the nearest T bucket for a matching - geometry, so the buckets cover the whole small-T range rather than one row per T. -- `gfx1100-hipblaslt-100500.txt`: RX 7900 XTX (gfx1100, 24 GB), calibrated with a ROCm 10.2.0a20261003 nightly - SDK (`libamdhip64.so.7.17.26392`, hipBLASLt 1.5.0, version number 100500). Same 16 dense GEMM geometries at - T=4096 and T=8192 as the other gfx1100 tables (32 rows), calibrated with `tune_hipblaslt` run against that - library. On this ROCm the engine refuses the 100100/100200 tables (version mismatch) and prefills on plain - hipBLAS; with this table a 0.1.38-lineage nightly engine prefills a 131071-token prompt at 1687 tok/s vs 926 - without it (median of 3 clean runs each, greedy; decode unchanged). Per-run data and the calibration command: - the `2026-10-04-community-rx7900xtx-hipblaslt-100500` folder of PR #745. -- `gfx1201-hipblaslt-100500.txt`: Radeon AI PRO R9700 (gfx1201, 32 GB), calibrated with ROCm 10.2.0a20260914 - (AMD's `gfx120X-all` nightly, hipBLASLt 1.5.0, library build `d3164197`). 16 dense GEMM geometries at T=4096 and - T=8192, 32 rows. setup uses it only when the installed hipBLASLt reports 1.5.0 (it is found in `/opt/rocm` - when that is a system ROCm 7 or newer). The version number is the only thing the engine can check, so another - 1.5.0 build could number its solutions differently. `hip_prefill_hipblaslt_gemm` (with `STRATA_HIPBLASLT_TUNING` - set) is a smoke test: it refuses a table for another architecture or version and checks that one BF16 and one - F16 row exist and agree with hipBLASEx, which covers 2 of the 32 rows. It does not prove that a solution id is - valid: the engine falls back to hipBLASEx for an id the library rejects, and the test still passes. Run it with - `STRATA_HIPBLASLT_VERBOSE=1` and look for `fallbacks=0` in its summary line, and recalibrate with - `tune_hipblaslt` before using this table with a different 1.5.0 build. -- `gfx1151-hipblaslt-100400.txt`: Radeon 8060S (Ryzen AI Max+ 395, gfx1151, 128 GB; Fedora 44, kernel 7.2.8), - calibrated with the ROCm 7.14.0a20260608 wheels that setup installs for gfx1151 since 0.1.40.2 (#1267): their - hipBLASLt is 1.4.0 (version number 100400), so neither `gfx1151-hipblaslt-100401.txt` nor `-100500.txt` applied and - setup's engine read the prompt through plain hipBLAS. The same 90 rows as the 100401 table (its geometries, the - prompt shapes at `--prefill 16384` included), calibrated with `tune_hipblaslt --workspace-mib 32` and that table's - cases: 3.1x to 14.1x faster than plain hipBLAS per shape (median 6.2x). With `STRATA_HIPBLASLT_TUNING` set, - `hip_prefill_hipblaslt_gemm` passes (`launches=4 fallbacks=0`), `hip_prefill_hcd_exact_parity` passes (solutions - 1176 / 1177 at 9 chunk sizes, 0 outputs differ, `fallbacks=0`), and the HIP ctest passes 61 of 62 (`ple_parity` - needs the Q2_0 model fixture). setup uses it only when the installed hipBLASLt reports 1.4.0. The prompt speed - of a whole model with this table is not measured yet. +> **Measured on:** the card in each row, fresh prompts, decode unchanged unless stated; plain hipBLAS against the table. -- `gfx1150-hipblaslt-100401.txt`: Radeon 890M (gfx1150, Strix Point APU, 16 CUs), calibrated with AMD's TheRock ROCm 7.14.1 for - gfx1150 (hipBLASLt 1.4.1 `cd957402`, version number 100401) and the engine's 32 MiB workspace. The 16 dense GEMM geometries a - Qwen3.8-Flash-Next prompt logs on that card (the same 16 as the gfx1201 100500 table) at T = 64, 128, ..., 16384: 144 rows. Two - `tune_hipblaslt` passes back to back; each row keeps the solution with the lowest median of all six repetitions, because one - repetition stalled now and then (a 100 ms call read up to 1.6 s) and the tool's mean-of-three picks differed in 26 rows between - the passes. Every row beats plain hipBLAS, 1.55-22.6x per GEMM (geometric mean 4.5x), and `STRATA_HIPBLASLT_VERBOSE=1` reported - no fallbacks over prompts of 28-7,880 tokens. Setup does not use it (it does not install for gfx1150): point - `STRATA_HIPBLASLT_TUNING` at it. Measurements and the per-row timings: - [bench/results/2026-10-07-community-gfx1150](../bench/results/2026-10-07-community-gfx1150/README.md). -- `gfx1103-hipblaslt-100401.txt`: Radeon 780M (gfx1103, RDNA3, 12 CUs, unified memory), calibrated with ROCm 10.1 - (`rocm/dev-ubuntu-24.04:10.1.0-full`, hipBLASLt 1.4.1, version number 100401) and the engine's 32 MiB workspace. The - 16 dense GEMM geometries a Qwen3.8-Flash-Next prompt logs on the card at T = 64, 128, ..., 16384: 144 cases, two - `tune_hipblaslt` passes back to back, each row the solution with the lowest median of all six repetitions. One row - is left out (`bf16 N=1 K=2560 T=64`: no candidate beat plain hipBLAS at 0.04 ms), so the table has **143 rows**, - 1.05-23.30x per GEMM, geometric mean 4.95x. Without it the engine prefills on plain hipBLAS: a 4K prompt on Q2_0 - read 90 tok/s, 187 with the table and 277 with the WMMA opt-ins (one 780M; 32K context, MTP). The merged table - prefilled the 4K prompt at 187.2 tok/s. A context ladder to 131K (shipping configuration, fresh prompts) puts the - full 128K read at 198-221 tok/s, its last ~14K at ~206, and decode at ~10 tok/s at that depth; the numbers are in - the report folder. `hip_prefill_hipblaslt_gemm` reports `fallbacks=0`; - `hip_prefill_hcd_exact_parity` skips, because the table picks no 1176 / 1177 for the HC down GEMM (the skip the - gfx1150 table gets too). Setup installs for gfx1103 only with `STRATA_EXPERIMENTAL_GFX1103=1` and then uses the - table when the installed hipBLASLt reports 1.4.1. Measurements: - [bench/results/2026-10-12-community-gfx1103](../bench/results/2026-10-12-community-gfx1103/README.md). +| Table | Card, workload | Prompt speed | +| --- | --- | --- | +| `gfx1100-hipblaslt-100401.txt` | RX 7900 XTX, Swift 1.5 IQ3_XXS, 262144 ctx, hipBLASLt 1.4.1, medians of 3 | 1.7K tokens 682 -> 1,038 tok/s; 6.5K tokens 857 -> 1,524 tok/s | +| 43-row revision | RX 7900 XTX, Qwen3.8-Flash-Next GSQ-RCO IQ2_XS, 3,151-token prompt | 711 -> 938 tok/s (+31.9%) | +| `gfx1100-hipblaslt-100500.txt` | RX 7900 XTX, engine 0.1.38 lineage | 131,071-token prompt 926 -> 1,687 tok/s | +| `gfx1201-hipblaslt-100202.txt` | R9700, full IQ3_XXS, `--kv int8 --kv-resident 32768` | 32K prompt 638 -> 1,177 tok/s; 7K prompt 656 -> 1,164 tok/s | +| `gfx1201-hipblaslt-100500.txt` | One R9700, 4,210-token prompts | 1,531 -> 1,590 tok/s (+3.9%) | +| `gfx1151-hipblaslt-100400.txt` | Strix Halo | 3.1x to 14.1x faster per GEMM (median 6.2x); whole-model prompt speed not measured | +| `gfx1103-hipblaslt-100401.txt` | Radeon 780M, Q2_0, 4K prompt, 32K context, MTP | 90 -> 187 tok/s; 277 with the WMMA opt-ins | -Calibrated on my PC and kept in this checkout, not shipped: `gfx1201-hipblaslt-100401.txt`, a Radeon AI PRO -R9700 (gfx1201, 32 GB, 1002:7551) on a Ryzen 9 9950X3D with 62 GiB RAM, CachyOS (kernel 7.3.0-rc6-1-cachyos-rc, -`linux-firmware-amdgpu 20260916`, the kernel's amdgpu driver), system ROCm `rocm-gfx120x-bin 10.1.0` (hipBLASLt -1.4.1, HIP 7.16.26385, clang 24.0.0), Python 3.14.7 in setup's own venv, engine 0.1.40 compiled there for gfx1201, -the model files on a 4 TB NVMe (btrfs), IQ3_S at 131072 ctx with `--kv int8`, the GPU idle. `tune_hipblaslt` over -the 32 geometries of `gfx1201-hipblaslt-100500.txt`: 1.15x geometric mean per GEMM over hipBLAS - 1.55-1.68x at -N=512/640 K=2560, 0.99-1.13x at N=10240/12288 K=2560. End to end with `tools/hip/bench_prefill.py` over two -server starts, fresh prompts of 4,210 and 8,830 tokens, three matched trials each - the first fresh prompt after -each start is left out, it is cold (949 and 1,423 tok/s there): 1,606 -> 1,613, 1,701 -> 1,726 and 1,605 -> 1,610 -tok/s (+0.5, +1.5, +0.4%), medians 1,606 -> 1,613 and a 1.008x geometric mean over those three, decode -87.1 -> 85.1 tok/s - within noise, as the R9700 paragraph above found. `hip_prefill_hipblaslt_gemm` passes with it: `tuning enabled (30 rows, gfx1201, -version 100401)`, `launches=4 fallbacks=0`. +Tuning on the R9700 with hipBLASLt 1.2.0 or 1.4.1 gave 0 to 3% end to end (within noise), so no table ships for those and +Linux `rx9070` uses plain hipBLAS. A 1.2.0 table is not bit-identical: first-token KL 0.0157 mean against plain over 24 +prompts, top-1 the same 24 of 24. -Two rows were dropped from that table: bf16 N=10240 K=320 ldy=10240 at T=4096 and at T=8192. At T=8192 hipBLAS was -faster in five separate `tune_hipblaslt` runs - 0.609-0.619 ms against 0.648-0.677 ms for the best of the 16 -hipBLASLt candidates (0.90-0.95x), with four different winning solution ids, all slower than hipBLAS. Dropping only -the T=8192 row does not send that call back to hipBLAS: `closest()` matches dtype/N/K/ldy and takes the nearest T -bucket, so T=8192 would run the T=4096 row; both rows go or neither. - -Two calibration notes from my machine. The first case of a fresh `tune_hipblaslt` process times its hipBLAS -baseline while the clocks are still ramping: that T=4096 shape read 0.31-0.63 ms across five runs while the case -timed right after it held 0.606-0.619 ms every time, so time a heavy case first and read the later ones. And the -first fresh prompt after the model loads is not a measurement: it gave 949 and 1,423 tok/s in the two arms of that -A/B, while the three trials after it differed by 0.4-1.5%. +When you calibrate, time a heavy case first: the first case of a fresh `tune_hipblaslt` process times its baseline while the +clocks are still ramping. The first fresh prompt after a model loads is also not a measurement. ### rocBLAS table (gfx103x) -On RDNA2 the prompt GEMMs run through rocBLAS in FP16 (#835), and hipBLASLt has no gfx1030 kernels; the rocBLAS table -does for rocBLAS what the one above does for hipBLASLt (the [RDNA2 section](#rdna2-gfx1030) has the measurements). A -table is valid for one architecture and one rocBLAS build - the full version string, tweak hash included - so it is -calibrated on the card with the rocBLAS the engine loads (`lib_dirs` in the run configuration; `/opt/rocm/lib` is -rocBLAS 5.6.0.8d1ae90e on ROCm 10.0.0, while Ubuntu's own `librocblas5` package is a 5.1 build): +On RDNA2 the prompt GEMMs run through rocBLAS in FP16, and hipBLASLt has no gfx1030 kernels. The rocBLAS table does for +rocBLAS what the table above does for hipBLASLt. A table is valid for one architecture and one rocBLAS build (the full +version string, tweak hash included), so calibrate it on the card with the rocBLAS the engine loads (`/opt/rocm/lib` is +rocBLAS 5.6.0.8d1ae90e on ROCm 10.0.0): ```sh cmake --build build-hip --target tune_rocblas -LD_LIBRARY_PATH=/opt/rocm/lib ./build-hip/tune_rocblas --tuning-out table.txt # 14 shapes x 8 token buckets, ~5 min +LD_LIBRARY_PATH=/opt/rocm/lib ./build-hip/tune_rocblas --tuning-out table.txt # 14 shapes x 8 token buckets, about 5 min ``` -The tool times rocBLAS's default kernel and every solution the library lists for each shape at T = 256 ... 8192 (the -engine's dense GEMM shapes; `--case T,N,K,ldy` adds one, `--tokens` changes the buckets), checks the fastest ones -against the default kernel's output (within FP16 rounding, no write outside the N x T result) and that they also run at -smaller, odd token counts, and writes one row per shape and bucket: the winning solution index where it beats the -default by `--min-gain` (1.05), `default` otherwise - a `default` row is what keeps a neighbouring bucket's solution from -reaching token counts where rocBLAS's own choice measured best. The file's header names the architecture and version -(`STRATA_ROCBLAS_TUNING_V1 gfx1030 5.6.0.8d1ae90e`); save it as `tools/hip/-rocblas-.txt` (the version -with trailing dots dropped) for setup, or point `STRATA_ROCBLAS_TUNING` at the file. `STRATA_ROCBLAS_VERBOSE=1` prints -the solution picked for each shape and a `tuned_launches=... fallbacks=...` summary at exit; a solution rocBLAS refuses -at run time (status 11 once while it sets up its workspace is normal) falls back to the default kernel for that call, and -after three refusals for good. `hip_prefill_rocblas_gemm` (with `STRATA_ROCBLAS_TUNING` set; skips otherwise) is the -smoke test: it refuses a table for another architecture or build and checks Gemm's FP16 route against hipBLASEx on the -table's first two shapes. +The tool times rocBLAS's default kernel and every solution the library lists for each shape at T = 256 ... 8192 (`--case +T,N,K,ldy` adds a shape, `--tokens` changes the buckets), checks the fastest against the default's output, and writes one row +per shape and bucket: the winning solution where it beats the default by `--min-gain` (1.05), `default` otherwise. The header +names the architecture and version (`STRATA_ROCBLAS_TUNING_V1 gfx1030 5.6.0.8d1ae90e`). Save it as +`tools/hip/-rocblas-.txt` (trailing dots dropped) for setup, or point `STRATA_ROCBLAS_TUNING` at the file. +`STRATA_ROCBLAS_VERBOSE=1` prints the solution picked per shape and a `tuned_launches=... fallbacks=...` summary at exit. A +solution rocBLAS refuses at run time falls back to the default kernel for that call, and after three refusals for good. +`hip_prefill_rocblas_gemm` (with `STRATA_ROCBLAS_TUNING` set) is the smoke test. -## Original backend validation (PR #94) - -The following is historical validation of the original backend, not a fresh -test count for this replacement. Current build/test evidence and performance -limits are recorded in [AMD_HIP_PERFORMANCE.md](AMD_HIP_PERFORMANCE.md). - -Tested against upstream `c1e903310f211e6630780c3bd2038778c071c68d` (0.1.20), -with pinned llama.cpp `3cf03257f219afbe7334045ff7c6a06ac68c627d`. -The host has an RX 7900 XTX (24 GiB), Ryzen 9 7950X3D, 64 GiB system RAM, -and Fedora-family Linux with ROCm 7.1.52802 / Clang 20. The original model used -for the smoke check was on mechanical RAID0; cold page faults were slow. Keep -latency-sensitive data on SSD and measure warm residency separately. - -- Complete HIP Release build, including the executable and parity targets. -- All 25 tests selected by the command below passed on 2026-09-29. -- Intrinsic parity includes signed packed-byte dot products and overflow. -- CPU/GPU mapped-memory handoff covers delayed publication, changing payloads, - and repeated graph replay in both directions. -- GR coverage includes the maximum eight-token fused batch and graph replay - after inputs change. Native QSA checks the scalar HIP fallback. -- Q8_K quantization is byte-exact against the existing reference. HIP's - `__fmul_rn` can become ordinary multiplication; disabling contraction in - `quantize_act.cu` preserves the separately rounded product before the magic - rounding bias is added. CUDA's compile flags remain unchanged. -- The mmap regression covers canonical and differing native layer sizes, - truncated/oversized files, lookup bounds, and reopening after errors. -- CUDA 13 / GCC 15 / sm_89 executable build passed; GR, sampler, quantization, - and mmap tests passed on the same tree. This is a regression check, not a - claim of complete CUDA inference validation. -- CPU-only `strata-plan` build passed without either GPU backend. -- Real GSQ-RCO IQ3_XXS server smoke with mmap, the shipped expert profile, - automatic GPU cache, and MTP passed arithmetic, executable Python addition, - system-marker recall, and 1,170-token batched-prefill recall. All four requests - ended normally with correct answers; one engine process served them all. - The cache held 11,142 experts (18.08 GiB), with 874 MiB of VRAM free after - startup. These short checks do not establish a throughput or quality benchmark. - -Build all targets before running the registered focused checks: - -```sh -cmake --build build-hip -j2 -ctest --test-dir build-hip --output-on-failure --timeout 60 \ - -E '^(ple_parity|platform_memory_test)$' -``` - -`ple_parity` requires an external model fixture. `platform_memory_test` requests -256 MiB of locked memory; the test shell's 8 MiB memlock limit prevented that -test from passing. These are explicit exclusions, not skipped tests counted as -passes. Additional unpublished upstream fixture suites are not covered. - -The published performance table explicitly identifies the measured development -snapshot; it must not be read as a benchmark of every subsequent rebase. Vision, long-context stress, broad answer-quality -equivalence, other AMD cards, and mixed-vendor inference are not validated here. +Shipped: `gfx1030-rocblas-5.6.0.8d1ae90e.txt` (ROCm 10.0.0, the `/opt/rocm` setup installs on Ubuntu), measured on an RX 6900 XT. +Other gfx1030 cards run it too, but a table calibrated for them may pick differently; `tune_rocblas` makes one in about 5 +minutes. Data: [2026-10-05-rdna2-rocblas-table](../bench/results/2026-10-05-rdna2-rocblas-table/README.md). ## Strix Halo: 8-bit hyper-connection weights (default on gfx1151) -On a Strix Halo (gfx1151) the hyper-connection down / up projections are read as Q8_0 by default (quantized at load from the pack, or the GGUF's own Q8_0), -which halves the 13.3 MB each read moves. Decode is about 9% faster (5 interleaved pairs per prompt on IQ3_S, MTP 4 drafts / min-p 0.5; json / code / tool -call / chat / agent between +7.9% and +10.7%). `STRATA_HC_Q8=0` restores the bf16 read. Other cards are unchanged (gfx12 has had it on since 0.1.4x). -Not bit-identical to the bf16 read: a greedy answer can pick another token at a near tie and then go its own way. +On Strix Halo (gfx1151) and gfx12 the hyper-connection down and up projections are read as Q8_0 by default (quantized at load +from the pack, or the GGUF's own Q8_0), which halves the 13.3 MB each read moves. Decode is about 9% faster on Strix Halo +(+7.9% to +10.7% across json, code, tool call, chat and agent prompts). `STRATA_HC_Q8=0` restores the bf16 read. It is not +bit-identical to the bf16 read: a greedy answer can pick another token at a near tie and go its own way. -How far it moves the model, measured with the quant-quality yardstick (teacher forced, 5 texts of 1,000 tokens, reference = bf16 read; RTX A4000, IQ3_XXS, -2026-10-22 — the Q8_0 weights are the same bytes on every card): +> **Measured on:** RTX A4000, IQ3_XXS, quant-quality yardstick (teacher forced, 5 texts of 1,000 tokens, reference = bf16 +> read), 2026-10-22. The Q8_0 weights are the same bytes on every card. -| text | top-1 same | KL | +| Text | Top-1 same | KL | | --- | ---: | ---: | -| code | 97.9% | 0.0077 | -| documents | 93.5% | 0.0229 | +| Code | 97.9% | 0.0077 | +| Documents | 93.5% | 0.0229 | | English | 94.7% | 0.0107 | -| other languages | 98.0% | 0.0060 | -| agent transcript | 93.0% | 0.0779 | -| all 4,995 positions | 95.4% | 0.0251 | +| Other languages | 98.0% | 0.0060 | +| Agent transcript | 93.0% | 0.0779 | +| All 4,995 positions | 95.4% | 0.0251 | -For scale, the same yardstick with the bf16 read on both sides and only the experts computed on the CPU instead of the GPU (a rounding-only difference the -engine takes whenever an expert is not resident) gives code 0.0062, documents 0.0195, English 0.0088, other languages 0.0048, agent 0.0745, all 0.0228 -(top-1 95.4%). The 8-bit read is at that level. The agent text's number rests on a few near-tie positions (the top 20 of 999 carry half of it); finer blocks -(16 or 8 values) did not lower it, and 8-bit hyper-connection weights in the first six layers alone give the full figure, in the last 24 layers a third of it. +For scale, bf16 on both sides with only the experts computed on the CPU instead of the GPU (a rounding-only difference the +engine takes whenever an expert is not resident) gives KL 0.0228 over all positions with top-1 95.4%. The 8-bit read is at +that level. + +## History / older measurements + +Dated results from earlier engines. They show how the numbers moved and are not instructions. Narrative from the original +write-ups (method notes, bug hunts, per-run logs) was dropped; the numbers and their labels are kept. + +**RDNA4 (gfx1201), R9700 (Q2_0 32 GB unless stated), Linux, ROCm 7.14, interleaved starts.** + +| Change | Engine | Result | +| --- | --- | --- | +| Packed byte ops (`__byte_perm` as one `v_perm_b32`, #262) | 0.1.31 | R9700 decode +15% (46.0 -> 53.0 tok/s, 4K prompt); 9070 XT +5-7% | +| Router kernel without serial FP64 sum | 0.1.32 | R9700 decode 62.4 -> 70.0 tok/s; a user's A/B with TheRock 7.10 and an i5-12600K: +1-4% inside 12-20% spread (#432) | +| WMMA prompt attention (Coder IQ1_M) | 0.1.32 | R9700 prompts 4K 1,784 -> 2,427, 16K 1,797 -> 2,700 tok/s; a user on TheRock 7.10: +29-34% (#432). Kernel 7.2-7.5x the portable one | +| WMMA prompt attention (IQ3_S, 431072 ctx, gfx1201 hipBLASLt 1.4.1 table) | 0.1.40 | 8,830-token prompt 1,571 -> 2,064 tok/s (+31.4%); 4,210 tokens 1,699 -> 1,990 (+17.1%); decode 89.8 -> 86.9 (unchanged) | +| WMMA gfx12 default (Q2_0) | not recorded | 20K prompt x1.46 (`=1`) / x1.54 (`auto`); 4K x1.33 / x1.34; quality: KL 0.0003 (6,024 positions), HumanEval 157/164 (default 156) | +| Fused int8 prompt experts (`STRATA_PF_FUSED=1`, Coder IQ1_M, ROCm 7.14) | 0.1.42 | 4K prompt 1,612 -> 1,353 ms (+19.1%), 16K +8.4%, 32K +8.5%. First-token KL vs MMQ: IQ3_S 0.00284 vs 0.00332, IQ3_XXS 0.00388 vs 0.00340 (pooled 24 prompts) | +| Fused experts for Q2_0 gate/up | not recorded | One layer, 2,048 tokens: MMQ 24.5 ms, fused 4.4 ms. Prompt 4K +16.7%, 20K +19.0%; KL vs MMQ 0.0004 | +| Both defaults together (Q2_0 32 GB) | not recorded | 4K 1,151 -> 2,286 tok/s, 20K 1,384 -> 2,751; IQ3_XXS 20K 1,576 -> 1,997 | +| Queue priming at start (Linux) | 0.2.1 | About half of fresh Linux starts read prompts 10-23% slower (4K 1,980 vs 2,290 tok/s, p620); queue priming at start fixed it, answers identical | +| Layer split R9700 + 9070 XT (4K / 16K prompt, decode) | 0.1.30 | R9700 alone 1,794 / 1,804 tok/s, 51-52 decode; split 1,384 / 1,906, 42-43; 9070 XT alone 1,016 / 1,519, 38-40 | +| Layer split with experts streamed (R9700 cache 1,500, 16K prompt) | 0.1.31 | Split 1,803 tok/s against 1,287 | +| `split_skip_if_fits` (R9700 first) | 0.1.31 | 4K prompts 1,776 tok/s (split: 1,244), decode about 60 tok/s (split: about 51) | +| Pictures on the R9700 | not recorded | `strata-vision-hip`: 57 ms for a 260-token picture, 0.40 s for 972 tokens, against 3.13 s and 14.1 s on an EPYC CPU reader; cosine 0.99985. A 780M with a gfx1103 reader: 0.40 s against 2.42 s | +| Windows RX 9070 XT (#325), engine 0.1.34 | 0.1.34 | Coder IQ1_M at 32K: 29.2 tok/s decode, about 181 tok/s prefill, correct answers; ctest 42 of 46 | +| Top-k selection, R9700 | 0.1.45 | 256-query selection 0.29 -> 0.21 ms at 98K, 1.16 -> 0.49 ms at 262K, 22.0 -> 0.93 ms at 524K | + +**Packages and tables.** + +| Result | Where | +| --- | --- | +| R9700 and Linux `rx9070`: a 1.2.0 table read 4K prompts 2,302 vs 2,119 tok/s and 20K 2,671 vs 2,507, decode 91.2 vs 93.2 (-2.1%), mixed by start mode (0.2.1, package 0.1.45.3, Q2_0) | Not shipped | +| Strix Halo: prompts 251 -> 529 tok/s (2.11x) with the table (community A/B) | Default on gfx1151 | +| R9700, own table with hipBLASLt 1.4.1, IQ3_S 431072 ctx: 8,830-token prompts 1,606 -> 1,613 tok/s (+0.5%), decode 87.1 -> 85.1 (0.1.40) | Kept local, not shipped | +| RX 6800 (Windows, #380 #377): `--expert-cache auto` filled the card past what Windows keeps, decode fell 41 -> 30 tok/s; fixed by the video-memory budget (0.1.37) | Fixed | + +**Shared-expert stream, measured by platform.** 2x RX 6900 XT Linux (#1272): turning it off costs 5-13% decode (one card 47.5 / +46.9 / 48.3 -> 50.5 / 48.5 / 51.3 tok/s with it on). RX 6800 Windows (#816): on was 12-22% slower. 4x R9700: fork off was 13-45% +faster. + +**Draft side stream, RX 6800 Windows (`STRATA_HIP_PROMPT_F16=1`, IQ3_XXS, two runs each).** A 7.9K-token prompt read 798-808 +tok/s with the stream and 584-596 without; 30.4K tokens 855-863 against 553-600. Decode unchanged. + +**Lend coverage, RX 7900 XTX (gfx1100), v0.1.40.2, 50 GB RAM, 74K prompt (#1389).** 6201 of 6201 slots covered: `wait copy` 0.2%, +2,583 tok/s. 5093 of 6202: 36.8%, 1,810 tok/s. Same reporter's other levers: `STRATA_PF_FUSED=1 STRATA_PF_GEMM=1 +STRATA_PF_SWITCH_MIN_T=4096` 925 -> 1,092 tok/s; `--resident-experts` 1,092 -> 2,503 tok/s; `--prefill auto:32768` +23% at 74K. + +**RDNA2, other measurements.** + +| Workload | Result | +| --- | --- | +| `STRATA_HIP_PROMPT_F16=1`, 2x RX 6900 XT, GSQ-RCO IQ3_S, engine 0.1.39 | 9.4K / 34.7K / 105.8K prompts 744 / 915 / 926 tok/s instead of 439 / 466 / 461; decode unchanged | +| `STRATA_RDNA2_SGEMM=1`, RX 6800 over OCuLink, Ryzen 7 8845HS, Windows 11, Coder IQ1_M, 12K prompts, engine 0.1.39 | Prompt 312 -> 452 tok/s; GPU timeline 37.2 -> 25.5 s; decode 29-34 vs 29-35 tok/s; same session: native 310, SGEMM 439, FP16 494 tok/s | +| rocBLAS table, 2x RX 6900 XT, 799-token prompt | GPU time 3,329 -> 2,985 ms; 8,874 tokens 11,978 -> 11,459 ms; 1,108 tokens unchanged (bounded by expert PCIe transfer) | +| `STRATA_SELECT_SIMT=1`, 2x RX 6900 XT, IQ3_S, 0.1.40.3 | 128K prompt 1,735 -> 1,861 tok/s (+7.4%); 32K +0.8%. `qsa_select_bench`, 256 queries: 2.69 -> 0.587 ms at 128K | +| `STRATA_HC_PERSIST=1`, 2x RX 6900 XT, ROCm 10.0 | Read 1.35-2.10x; -4.2% / -2.8% ms per verify window at 4K / 32K | +| Draft side stream, RX 6800 Windows | See above | +| `STRATA_PF_PAD`/`STRATA_PF_GEMM` on RX 9070 (gfx1201), ROCm 7.2.4, Coder IQ1_M | 4K prompt 1,247 vs 1,243 tok/s, 8K 1,330 vs 1,321 (noise); an earlier crash with padded GEMMs on RDNA4 was fixed in 0.1.46 | + +**Package checks (0.2.1 to 0.2.3).** Initial package checks (build, link, `package_engine.py --verify`, GPU code listing, +`strata --version` in a clean environment, feed selection by architecture) passed without an RX 7000 or Strix Halo card. +Later evidence: [release 0.2.3](release-notes-0.2.3.md) records Windows Strix Halo inference with engine 0.1.48. An RX 7900 XTX +on Windows 11 ran the 0.1.40 Windows engine ([report #1173](../bench/results/2026-10-06-community-rx-7900-xtx/README.md)); +other RX 7900 owners ran Linux builds ([COMMUNITY.md](../bench/results/COMMUNITY.md)). + +**Original backend validation (PR #94, engine 0.1.20, RX 7900 XTX, Ryzen 9 7950X3D, 64 GiB, ROCm 7.1).** 25 registered tests +passed on 2026-09-29. A real GSQ-RCO IQ3_XXS server smoke with mmap, expert profile, automatic cache and MTP passed four +requests; the cache held 11,142 experts (18.08 GiB). Short checks, not a benchmark. Current evidence: +[AMD_HIP_PERFORMANCE.md](AMD_HIP_PERFORMANCE.md). + +**MI50 (gfx906), port this build was cut from (builds on 0.1.30 / 0.1.33).** 2x MI50 16 GB (85 W each), Xeon E5-2666 v3, 32 GB, +Coder IQ1_M, 131,072 context, `--kv int8 --kv-resident 32768 --layer-split 27 --prefill 4096`, MTP `--spec 4 --spec-min-p 0.5`: +all 12,288 experts in VRAM; a 17,043-token agent prompt read in 43.5 s (392 tok/s), decode 57.7-59.3 tok/s; 407 requests, 202,940 +tokens in a load test without stalls. llama.cpp (3cf0325, ROCm) on the same machine: 26.9 tok/s `tg128`. + +**Community gfx1150 (890M).** Prompts 3.6K tokens 124 -> 216 tok/s and 7K 130 -> 226 tok/s with the table, decode unchanged. + +**gfx1151 table (100400).** `hip_prefill_hipblaslt_gemm` passes (`launches=4 fallbacks=0`); HIP ctest 61 of 62. diff --git a/docs/DETAILS.md b/docs/DETAILS.md index f037c65cb..43538ddca 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -1,32 +1,51 @@ -# Strata - the details +# Strata engine and server reference -The technical side of Strata: every measured number, the API, images, all settings and how the engine works. -New here? Start with the [README](../README.md); installing step by step is in [INSTALL.md](INSTALL.md), the models in -[MODELS.md](MODELS.md), common problems in [TROUBLESHOOTING.md](TROUBLESHOOTING.md). +This page is the technical reference for the Qwen engine and its server: measured speed, the HTTP API, the settings in +`strata-.json`, memory modes and the speed switches. It is for people who run Strata from a source checkout, tune +an installation or connect their own software. It describes the current release. Dated measurements are in +[History / older measurements](#history--older-measurements) at the end. -> **On this page:** [Speed](#speed-measured) · [Other GPUs](#other-gpus-estimated) · [Which model?](#which-model) · -> [Requirements](#before-you-start) · [Windows](#windows) · [Linux](#linux) · [API](#using-it) · -> [MCP tools](#tools-from-mcp-servers) · [MCP server](#manage-strata-from-your-ai-assistant-mcp-server) · -> [Images](#images-vision) · -> [Troubleshooting](#troubleshooting) · [How it works](#how-it-works) +New here? Start with the [README](../README.md). Installing is in [INSTALL.md](INSTALL.md), the models are in +[MODELS.md](MODELS.md), common problems are in [TROUBLESHOOTING.md](TROUBLESHOOTING.md). DeepSeek V4.1-Flash has its own +engine and is covered in [MODELS.md](MODELS.md); nothing below applies to it unless stated. + +> **On this page:** [Quick start](#quick-start) · [Speed](#speed-measured) · [Which model?](#which-model) · +> [Windows](#windows) · [Linux](#linux) · [Memory modes](#memory-modes) · [Sharing the GPU](#sharing-the-gpu-with-other-programs-optional) · +> [API](#using-it) · [MCP tools](#tools-from-mcp-servers) · [MCP server](#manage-strata-from-your-ai-assistant-mcp-server) · +> [Images](#images-vision) · [Speed settings](#speed-settings) · [CPU workers](#cpu-workers-and-thread-placement) · +> [Per-card defaults](#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-decode-switches-on-by-default) · +> [Troubleshooting](#troubleshooting) · [How it works](#how-it-works) · [Credits](#credits-and-licenses) · +> [History](#history--older-measurements) + +**Which flags go where.** `strata launch` has its own flags (`strata launch --help`). The engine and server flags below go +in the `"args"` list of `strata-.json` (source setup writes it) or on the `serve/server.py` command line. +`START-HERE.bat`, `SETUP.bat`, `./setup.sh` and `run-` exist only in a [source checkout](SOURCE_INSTALL.md). An +installed app has none of them. + +## Quick start + +1. Install and start a model: [INSTALL.md](INSTALL.md) (app or Terminal) or [SOURCE_INSTALL.md](SOURCE_INSTALL.md). +2. Check that the server answers. The default address is `http://127.0.0.1:8080`: + + ```sh + curl http://127.0.0.1:8080/health + curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \ + -d '{"model": "strata", "messages": [{"role": "user", "content": "Write a haiku about GPUs."}], "max_tokens": 512}' + ``` + +3. Point your app at `http://127.0.0.1:8080/v1` with any API key, or run `strata connect` to list the apps Strata can configure. +4. Change a setting by editing `strata-.json` and restarting. Setup rewrites the file when you run it again and + keeps the keys it does not manage. --- + ## Speed (measured) -RTX 5070 **12 GB**, Ryzen 5 7600 (6 cores), 64 GB DDR5-5200, Windows, engine 0.1.26 with the settings setup writes -(`--prefill auto`, 8-bit KV above 4K, KV streaming from 64K). One code-agent prompt per length, 256 generated tokens, -MTP speculative decoding on. "262K" is the model's full context window (a 259,943-token prompt). The IQ2_XS row was -measured with Swift 1.5's IQ2_XS, which runs at the original's speed. - -**Engine 0.1.36 (#136), the same PC:** Q2_0's prompt experts run on fused int8 tensor-core kernels (RTX 30 and newer): -4K 1,294 -> 1,570, 32K 2,170 -> 2,653, 128K 2,123 -> 2,468 tokens/s (+16-22%), as close to an FP16 reference as the -previous kernels (closer at 32K: teacher-forced KL 0.009 vs 0.012). The decode path's block selection and greedy -argmax run on thread-block clusters (RTX 50, sm_90+; other cards keep the previous kernels; the same tokens): Q2_0 output at 4K 89 -> 93.5, at 128K -64.5 -> 76.4 tokens/s. `STRATA_PF_FUSED=0` keeps the previous prompt kernels (byte-identical answers to 0.1.35); -`STRATA_PF_FUSED=1` also runs the native IQ packs' fused kernels (opt-in: IQ2_XS prompts +12% at 4K, +3% at 32K, the -IQ3 packs about even; on HIP they run on gfx11 and on gfx1200 / gfx1201, see AMD_HIP.md); `STRATA_QSA_CLUSTER=0` / `STRATA_ARGMAX_MULTI=0` turn the decode kernels off. The tables -below are 0.1.26's. +**Measured on:** RTX 5070 12 GB, Ryzen 5 7600 (6 cores), 64 GB DDR5-5200, Windows, Qwen3.8-Flash-Next ISTA-DASLab packs, +engine 0.1.26 with the settings setup writes (`--prefill auto`, 8-bit KV above 4K, KV streaming from 64K). One +code-agent prompt per length, 256 generated tokens, MTP speculative decoding on. "262K" is the full context window (a +259,943-token prompt). The IQ2_XS row used Swift 1.5's IQ2_XS, which runs at the original's speed. ### Prompt processing (tokens/s) @@ -37,15 +56,9 @@ below are 0.1.26's. | **IQ3_XXS** | 482 | 1,007 | 1,745 | 1,609 | 1,602 | - | | **IQ3_S** | 427 | 913 | 1,624 | 1,640 | 1,443 | - | | **Coder** | 656 | 1,583 | 2,177 | 2,236 | 2,208 | 1,034** | -| **IQ3_S** (AMD RX 7900 XTX, gfx1100) | 760 | 1,275 | 1,641 | 1,594 | 1,494 | - | -Engine 0.1.26; `bench/results/2026-09-29-speed-0126`. The AMD RX 7900 XTX row: engine 0.1.38 on a ROCm -10.2 nightly with a tuned gfx1100 hipBLASLt-100500 table (upstream as [#755](https://github.com/Niko1221/Strata/pull/755)), -the median of 3 clean cells per tier -(Ryzen 9 7900X, 96 GB; 1K-128K one-shot runs, 256 generated tokens, -greedy). At 32K-128K that is 8-28% faster than 0.1.22. † not measured -again: 0.1.22. \* measured with images on (the image encoder's VRAM reserve leaves fewer experts cached). \*\* not -measured again: 0.1.14. +† not measured again since 0.1.22. \* measured with images on (the image encoder's VRAM reserve leaves fewer experts +cached). \*\* not measured again since 0.1.14. ### Output (tokens/s) @@ -56,829 +69,342 @@ measured again: 0.1.14. | **IQ3_XXS** | 61.9 | 61.6 | 58.5 | 57.2 | 49.0 | - | | **IQ3_S** | 52.4 | 53.3 | 48.3 | 46.3 | 45.5 | - | | **Coder** | 58.9 | 55.1 | 54.9 | 53.2 | 43.0 | 42.8† | -| **IQ3_S** (AMD RX 7900 XTX, gfx1100) | 65.1 | 60.4 | 64.4 | 61.8 | 59.2 | - | -Engine 0.1.26, the same runs. The AMD RX 7900 XTX row: engine 0.1.38 on a ROCm 10.2 nightly (tuned -gfx1100 hipBLASLt-100500 table, upstream as [#755](https://github.com/Niko1221/Strata/pull/755)), median of 3 clean cells per tier (same box; decode is flat in -context - the GDN linear attention is O(1) per token; 1K/4K prefill ran slightly under the 0.1.31 -packaged-ROCm numbers, 32K-128K ahead). † not measured again: 0.1.14. +† not measured again since 0.1.14. Raw data: `bench/results/2026-09-29-speed-0126`. -Output speed depends on the text as well: speculative decoding runs faster when more of the drafted tokens are -accepted, so a different answer to the same prompt moves it by several percent. Run back to back on the 4K prompt, -0.1.14 writes 88.5 tokens/s and 0.1.12 85.7. The numbers before 0.1.13 (prompts about half as fast): -[`bench/results/2026-09-24-final`](../bench/results/2026-09-24-final/matrix.md); these: -[`bench/results/2026-09-28-speed-0114`](../bench/results/2026-09-28-speed-0114/README.md). +- Output speed depends on the text. Speculative decoding runs faster when more drafted tokens are accepted, so a different + answer to the same prompt moves it by several percent. +- Time to first token is prompt length divided by prompt speed. With Q2_0 that is about 4 s at 4K, 25 s at 32K, under + 2 minutes at 128K and 4.5 minutes at 262K. +- IQ3_XXS and IQ3_S at 262K are not measured. With 43 and 50 GB of experts, a 260K-token context brings a 64 GB PC to its + memory limit by setup's estimate, so setup recommends up to 128K there. A longer context that you choose is kept, with a note. +- Other cards, other CPUs and newer engines differ. The 0.1.36 fused prompt kernels and later changes raised these + numbers on RTX 30 and newer cards: see [History](#history--older-measurements). For your own PC, run the benchmark in + [BENCHMARK_PROTOCOL.md](BENCHMARK_PROTOCOL.md). -IQ3_XXS and IQ3_S at 262K are not measured: with their 43 / 50 GB of experts, a 260K-token context brings a 64 GB PC -to its memory limit by setup's estimate (the experts + the context's KV cache + 24 GB), so setup recommends up to 128K -with them on 64 GB. A longer context you choose (`--context 262144`, or a pick in its list) is kept, with a note: users -ran IQ3_S at 256K on 64 GB with RAM to spare (#406). In the low-RAM mode the KV cache stays in VRAM and the context -does not count against RAM. IQ3_S (engine 0.1.4 or newer) is published for the original model and for Swift 1.5 -(the same shard layout as Swift's other sizes). - -**KV streaming (engine 0.1.5):** at 64K and more, setup keeps the context's KV cache in RAM and only the part the -attention reads in VRAM (`--kv-resident 32768`), so more experts fit on the GPU. Q2_0 at 262K: 50.9 -> 62.6 tokens/s -(1,589 -> 3,872 experts in VRAM); at 128K about +6%. The attention reads exactly the same values (only where the KV lives changes); it -costs ~13.7 KB of RAM per context token (1.7 GB at 128K). Existing installs: run `START-HERE.bat --setup` once to turn -it on. Setup turns it on when the RAM has room for it; `--kv-streaming on|off` overrides that (on past the RAM test with a -note; never under WSL, which cannot stream). - -Under WSL2 the driver caps what `cudaHostAlloc` may pin at about 1 GiB, so the KV host copy is pinned with `cudaHostRegister` of ordinary memory there (detected through `/dev/dxg`); `STRATA_KV_HOST_REGISTER=1` forces that order on any system and `=0` forces `cudaHostAlloc` first (the other method is the fallback in both cases). On Linux without WSL nothing changes by default; with the switch the engine prints `KV streaming: the host copy is pinned with cudaHostRegister` (#1498 by yjamil8; WSL not tested here, the forced path was run on a Tesla P100: `--kv int8 --kv-resident 32768`, answers and decode speed unchanged). - -**4-bit KV cache (engine 0.1.8, optional):** `START-HERE.bat --setup` asks above 8K context (or pass `--kv q4_0`). It -halves the KV cache's memory with a Hadamard rotation before 4-bit rounding (PR #21), about 4% faster at 128K, but it -is measurably less precise on long documents (perplexity +8-12%; needle tests still pass). 8-bit stays the default. -Details: [`bench/results/2026-09-27-kv-q4`](../bench/results/2026-09-27-kv-q4/README.md). - -**Hybrid K8V4 KV cache (engine 0.1.25, optional, PR #120):** `--kv k8v4` (`START-HERE.bat --setup --kv k8v4`) keeps -the keys at 8 bits and stores the values as rotated 4-bit: 23% less KV memory than 8-bit, so more experts fit in -VRAM. RTX 3090, the Coder at 198K context: 99 instead of 85 tokens/s output, the same needle results, prompts 2-5% -slower. It streams its KV cache like the other formats (`--kv-resident N`): on an RTX 2060 SUPER 8 GB at 128K with -20,480 resident cells, +780 expert slots over resident K8V4, and it scores better than 4-bit KV on long documents. - -**Reproducible greedy output (0.1.30, opt-in, `STRATA_IQ_MT_MIN=1`):** with the IQ models, the CPU computes an -expert for one token with ggml's dot product and for several tokens with Strata's multi-token kernels, which round -slightly differently. How many tokens share an expert depends on the drafts in a verify window, so the same prompt -at temperature 0 can end in a different (equally good) answer when the drafting, the cache state or a resumed -conversation differ (issue #152). `STRATA_IQ_MT_MIN=1` (in the config's `env`) uses the multi-token kernels for -every group: the answer then no longer depends on the drafting. Measured on a Ryzen 7600 (AVX-512): IQ3_S decode --1..-3%, the other models the same; the default stays the fastest rule. On an Intel CPU of Alder Lake or later -without AVX-512, where the AVX-2 kernel gathers the IQ3_S grid, `STRATA_IQ3S_MT1=1` (opt-in) gives IQ3_S the multi-token -kernel for one token, which is the faster one there; it changes a lone token's rounding, so it is off by default. Through the server, two more things carry -over from one request to the next (#410): the adaptive tier moves experts between RAM and VRAM (the GPU and the CPU -round an expert differently), and the prompt cache resumes a repeated prompt and reads only its tail through the -decode path. For byte-identical repeats add `--prompt-cache 0 --adapt-swaps 0 --pcie-frac 0` to the engine's args -as well (#410): the PCIe share of the missed experts (computed on the GPU instead of the CPU) still made the first -answer after a start differ from the next ones. Measured here (IQ3_XXS, a 3.6K-token prompt, 4 repeats): with all -three switches 1 answer of 4, without `--pcie-frac 0` 2 of 4 (the first one differs), with the defaults 2 of 4. -`--pcie-frac 0` costs decode speed (the missed experts all run on the CPU), so keep it for A/B runs. - -**Coupled drafts with Gumbel-max picks (opt-in, `STRATA_SPEC_COUPLED=1` and `STRATA_SPEC_GUMBEL=1`):** for a request -that samples (temperature above 0), `STRATA_SPEC_COUPLED=1` lets the draft layer sample its guesses with the target's own -chain and random draw instead of taking its most likely token. `STRATA_SPEC_GUMBEL=1` changes how both of them pick from -that chain: the token with the largest p / E, where E is exponential noise keyed by the seed, the position and the token -id (the Gumbel-max trick; it is still an exact sample of the same distribution). With the default pick, one random number -walked over the candidates sorted by probability, a draft and a target whose candidate lists differ in one token tend to -land on different tokens; with noise keyed by the token, a token gets the same noise on both sides, so they agree on the -tokens they share. Greedy requests are unchanged, and without the variables nothing changes. Measured on a Ryzen AI -Max+ 395 (Radeon 8060S, Linux, ROCm 7.14.1, the iGPU alone), UD-Q4_K_XL, `--spec 4 --mtp` (the base model's draft -layer), temperature 1.0 / top_p 0.95 / top_k 20, a 1.3K-token prompt and 512 output tokens, 12-13 requests per arm: -drafts accepted 52.8% -> 59.9%, tokens per verify window 2.65 -> 2.88, output 41.1 -> 44.9 tokens/s (+9%); a window -costs the same (draft 8.8 -> 9.1 ms of 64). `sampler_parity` checks the pick against a host reference on every sampled -path, and that its frequencies match the softmax. On an RTX 3060 (IQ3_XXS, `--spec 4 --mtp`, temperature 1.0 / top_p 0.95 / top_k 20, 12 interleaved pairs of 200-token story and code requests) `STRATA_SPEC_COUPLED=1` raised the accepted drafts from 62.1% to 66.9% and adding `STRATA_SPEC_GUMBEL=1` to 68.0%, but decode speed did not move beyond the run-to-run spread (median 43.3 tok/s with coupled alone, 42.8 with Gumbel as well): try it on your own card before relying on it. - -**The draft layer's tokens (0.1.27, `--draft-vocab`):** the MTP draft layer can only propose tokens from a subset -of the vocabulary (`mtp/rt/draft_vocab.bin`). Since 0.1.27 the subset includes every Chinese, Japanese and Korean -token (106,299 ids), so answers in those languages are 15-38% faster (Q2_0, RTX 5070). Its head takes ~180 MiB of -VRAM, which the expert cache leaves free for it (0.1.28). `START-HERE.bat --setup --draft-vocab en` keeps the -English/code subset from before (40,525 ids, ~110 MiB less VRAM, English answers 1-2% faster; CJK answers get -almost no drafts). `--draft-vocab cyrillic` takes the English/code subset plus the whole Cyrillic script (58,963 -ids): the shipped subsets hold 142 of the vocabulary's 18,580 Cyrillic tokens, so Ukrainian or Russian answers got -1.4 tokens a round; with it 2.1, and 83 -> 109 tokens/s (RTX 5090, the NVFP4 fork), English unchanged. -`--draft-vocab fr` (0.1.39, #597) takes the English/code subset plus the 5,686 tokens that cover 99% of a French -Wikipedia corpus (46,211 ids, `tools/draft_vocab.py --corpus`): 23.6% of French text's tokens were outside the -English/code subset, 0.7% are outside this one. Drafts accepted in French answers 0.51 -> 0.60 (IQ3_XXS, RTX 5070, -8 prompts x 2 passes; English 0.61 -> 0.63 and code 0.77 -> 0.78, no loss), and 141 -> 158 tok/s in French on an -RTX 5090 (IQ3_S, the reporter's measurement). -`tools/draft_vocab.py` builds and inspects subsets. When the start stops with "the draft head does not fit" (a -12 GB card with a long context, #474), the engine says how much the head needs, how much VRAM is free and which -smaller subset fits, and the server's start error repeats it; setup suggests `--draft-vocab en` on cards under -14 GB (only a suggestion: nothing changes unless you pass it). - -**Serving without the draft layer (`--mtp` is optional):** `serve` runs without `--mtp`. Drafts then come from the -suffix/prompt-lookup drafter only (or one token per round), every token is still verified against the model, and the -draft layer's VRAM (~0.7-1 GiB with its head) goes to the expert cache: 1,000 -> 1,678 slots in one A/B on an 8 GB -card. The conversation cache (`--conversation-cache-mib`) stays on: a parked conversation then carries no draft K/V. - -**Low-RAM mode (engine 0.1.26, chosen by setup):** normally all of a model's experts are copied into RAM (23-50 GB, -pinned) and the GPU holds a copy of the most-used ones. On a PC whose RAM cannot hold them beside the system (the -experts plus ~10 GB), setup instead maps them from one file in the model's folder (`--mmap-experts`, the pack's -`experts.bin`, +23-50 GB of disk). The OS file cache holds what the GPU does not, and it can give that memory back. -On the Coder the engine's committed memory drops from 36 to ~13 GB, with the same answers. With a big GPU (an RTX -5090 holds all of the Coder's experts, most of Q2_0's) it runs at nearly the usual speed. With a small one, most -experts come from the SSD and it is much slower (setup says so). `START-HERE.bat --setup --low-ram on|off` overrides -the choice. - -**Low-RAM mode, resident (engine 0.1.30):** when the experts the GPU does not hold fit the RAM (with the same ~10 GB -beside them), setup picks the resident variant instead (`--resident-experts`): at start the engine copies exactly those -experts from `experts.bin` into RAM (page-locked when the driver allows, else locked in RAM), so while it answers -nothing is read from the SSD, however little RAM the OS leaves for its file cache. Examples with setup's context: a -32 GB PC with a 24 GB GPU runs Q2_0, IQ2_XS and the Coder this way (~16-18 GB of experts in RAM, the GPU holds the -other ~18 GB), a 32 GB PC with a 12-16 GB GPU the Coder; IQ3_XXS on a 32 GB PC stays mapped. The details: -- The prompt path borrows room in the GPU's expert cache for its buffers and puts those experts back after the prompt; - as far as the RAM allows, their experts are kept in RAM too (so a prompt reads nothing from the SSD either). -- The cache still follows the conversation (`--adapt-every`): a swap copies the evicted expert back from VRAM into the - RAM place of the one that replaces it, so the RAM copy keeps holding exactly what the GPU does not. -- `STRATA_RAM_ADAPT=` (opt-in, `--serve`, one GPU): the RAM copy follows the conversation too. Without it the - copy is chosen once, at start, so when the RAM cannot hold every expert the GPU does not, an expert outside the copy - is read from the SSD every time the conversation routes to it. With it, before a decode window and while no adaptive - round is in flight, up to `n` experts the windows keep routing to (the adaptive tier's decayed count, 2 or more) take - the RAM places of experts of their own layer that are not routed to (count below 0.5): one pair per layer, the - largest difference first, one blob read each. The RAM stays the same size; the routing, the weights and the VRAM - tier's rule are untouched, and an expert that left the copy is read from the file again when it is needed. It needs - the adaptive tier (`--adapt-every`), and does nothing with `STRATA_EXCHANGE_ROTATE=1`, a layer split, a peer or - helper GPU, or `--batch`. Prompt routing is not used to choose replacements, and no selection runs during prefill. - Later prompts can read from a RAM copy changed by earlier decode requests. The log reports how many experts moved. - Measured on an RTX 3060 12 GB with 32 GB of RAM (Linux, IQ2_XS, 20.5 GiB of experts in RAM and about 6,000 experts on - the file tier, `--adapt-async 1`, `--spec 3`, power-saver), against v0.1.43: 47 frozen Korean requests from - my own use (tools, follow-ups and background tasks, 281-33,241 prompt tokens), replayed in order in fresh engines: - release, this option, release. Each starts from the same held learned profile; the private prompts are not attached. - With default tail skip (7), decode is 43.26 -> 50.40 tok/s (+16.5%) against the closing release, computed as total - generated tokens / total decode time; the two release arms are within 0.2%. Short/long new-input requests gain - 16.3%/16.9%; total prompt speed differs by 2.5%, within run variation. Decode file-tier demand reads per generated - token fall from 14.15 -> 2.67; 8,056 successful RAM promotions read another blob each, giving 3.16 with those added. - These are file-tier counts, not physical SSD I/O. RAM residency does not exempt an expert from tail skip. One machine, - one quantization; tail skip off, blocking adaptation, `--spec 2` and the soak were not repeated on v0.1.43. -- `--adapt-async 1` (opt-in, `--serve`): the swaps of a round advance between decode windows on a helper thread - (copy back, copy in, move into RAM) instead of one window waiting for the whole round. With `--batch` slots it - advances between their windows too (docs/BATCHING.md); not with `--batch-groups` or `--peer-device` (the blocking - tier runs there). With `--pipeline-windows 2` (two windows in flight, see - [MULTI_GPU.md](MULTI_GPU.md)) the copies into the evicted slots and the moves into RAM also wait until the windows - that were in flight when they were decided have completed (`STRATA_PIPELINE_ADAPT_ASYNC=0`: the blocking tier - there). It is not bit-exact from run to run: which window first computes a swapped-in expert on the GPU (which - rounds differently from the CPU) depends on when its copy lands. - It stays on the blocking tier (said in the log) when the exchange buffers are not page-locked; the stats line - reports ms per round. -- **The arena too** (one GPU, no layer split): with every expert in RAM `--adapt-async 1` runs the same tier copy-in only: - nothing to copy back or move into RAM. The evicted slot is marked non-resident before the round's copies are posted, and an - expert becomes resident only once its copy has landed; `STRATA_ADAPT_ASYNC_WAIT=1` (the default there) waits for the round at - the next tick, so the window an expert turns GPU-resident does not depend on timing. The pick runs on the host thread - (`STRATA_ADAPT_PICK_INLINE=1`, one round per window); `STRATA_ADAPT_ASYNC_BATCH=N` posts at most N copies per tick (default 0 - = all; 16 measured slower). Every K/V grow and the `VRAM` command land the round in flight first. RTX 3090 + Ryzen 5 5600X, - cold prompts: x1.07-1.11 decode, 12 of 12 prompts faster - ([report](../bench/results/2026-10-11-community-rtx3090-5600x-arena-async/README.md)). -- **Pin by need** (`STRATA_PIN_BY_NEED=1`, opt-in, the arena on one GPU): Windows page-locks about half the RAM at most (the - WDDM shared budget), so the arena registers its first ~30 GiB and the layers past it get no PCIe share. With the variable - every blob gets a page-aligned stride (+0.15 % RAM) and the expert profile's non-GPU-resident experts are registered by rank - instead, up to `STRATA_ARENA_PIN_GIB` (29 by default on Windows). `STRATA_PCIE_MIN1=1` sends one miss over PCIe as soon as a - layer has two (the share otherwise rounds to 0 at 1-2 misses). RTX 3090 + Ryzen 5 5600X: prompt read +5-7 % at 56K-117K - tokens; decode level alone, +2-3 % with `--adapt-async 1` and `--pcie-frac 0.6` - ([report](../bench/results/2026-10-11-community-rtx3090-5600x-arena-pin-by-need/README.md)). -- **Pinned-memory headroom** (Windows CUDA, on by default; `STRATA_PIN_RESERVE_MIB=N`, `0` = off, sets it on any OS): the arena registers - slice by slice until the driver refuses one, and on a 64 GB PC that left no page-locked memory at all for the few small buffers the - engine allocates after it (the verify window's staging: `verify: mapped staging allocation failed`, exit 1, on Swift 1.5 IQ2_XS). The - engine now holds 640 MiB (given back as soon as the arena is placed) and 160 MiB (kept until a required buffer is refused, or READY) while - the arena pins, so the arena stops about one layer earlier; a refused required buffer gives both back and tries once more. If that still - fails the error ends with `[pinned-memory-exhausted]` and names the `STRATA_ARENA_PIN_GIB` to start with. Linux has no such ceiling - (the NVIDIA driver pins past `ulimit -l`), so the reserve is off there. Test knobs: `STRATA_TEST_PIN_BUDGET_MIB=N` makes the engine's own - pinned allocations fail beyond N MiB, `STRATA_TEST_PIN_LEFT_KIB=N` lowers that ceiling after the arena to what is charged plus N KiB. -- `STRATA_EXCHANGE_ROTATE=1` (opt-in): an adaptive swap hands buffer ownership over instead of copying the evicted - blob into the RAM copy (fully page-locked copy; layers of different blob sizes keep spare buffers per size). Same - tokens, fewer host copies; it works with `--adapt-async 1` too. Swift 1.5 IQ3_S on an RTX 5090 with 64 GB RAM - (Windows, about 30 swaps per verify window): 164 -> 188 tok/s on one forced 1,500-token text, 155 -> 178 over 23 - coding-agent turns. Details and the measurements: [EXCHANGE_ROTATION.md](EXCHANGE_ROTATION.md). Measured by us on an RTX 3060 12 GB (IQ3_XXS, resident RAM, default flags, 5 interleaved pairs): decode 58.96 -> 62.03 tok/s (+5.2%, 5/5 pairs), prompt speed unchanged; tokens identical with real adaptive swaps (Tesla P100, resident, blocking adaptation). It adds about 1.4 GiB of page-locked RAM, so leave it off when the RAM is already tight. -- The answers are the plain mapped mode's for the same expert placement: the bytes are the file's. With a page-locked - copy the GPU also takes its usual share of the misses over PCIe (`--pcie-frac`), as with enough RAM; `--pcie-frac 0` - (or `STRATA_RESIDENT_PIN=0`) gives the mapped mode's exact tokens. -- The engine leaves 4 GB of the RAM it finds free (`STRATA_RESIDENT_HEADROOM_GIB`); when even the experts the GPU does - not hold do not fit, it says so and runs the plain mapped mode. The server log shows, per request, how many expert - reads went to the file (`resident RAM: ... blob reads from the file`: 0 in steady use). -- `--low-ram resident|mmap` forces one variant (also on a PC with enough RAM, e.g. to try it). -- Several GPUs (#364, #384): setup recommends one GPU in the low-RAM mode (the resident variant has no layer split - yet), and asks; `--gpus 0,1` (or answering 2) shares the model across them with the mapped variant - (`--mmap-experts`): the cards together hold more of the experts, and two users measured it 1.3-1.6x faster than - one card, but the OS file cache can fill the RAM to 0 free during long prompts. `--yes` keeps one GPU. A config - with `--resident-experts` started with `--gpus` switches to `--mmap-experts` with a note, and the engine runs that - pair as `--mmap-experts` with a warning instead of refusing it. **Since 0.1.40 (#642, #848)** setup requires an - engine that runs the resident variant on a split, so it keeps both cards, resident, as for any config: - [MULTI_GPU.md](MULTI_GPU.md#using-it). - -**A mapped arena for small RAM (Linux, opt-in, 0.1.39, PR #640):** `STRATA_ARENA_MMAP=1` maps a native pack's expert -arena read-only from the pack's `experts.bin` instead of reading it into locked RAM, for a PC whose GPUs hold most -experts but whose RAM is small (2x 16 GB GPUs with 32 GB of RAM: ~1 GB -> 25 GB available while serving). The first -start writes `experts.bin` (when the drive has room for it), later starts map it; the pages of the experts a GPU holds -are handed back to the OS. Run it with `--pcie-frac 0` (the GPUs get no mapped alias). Without the variable nothing -changes. - -**Releasing mapped expert pages on Windows (opt-in):** `STRATA_FILE_RELEASE=1` lets `FileExpertSource` trim the -full file-backed pages of experts after their GPU uploads complete, including the slots lent to prefill and then -refilled. It works with `experts.bin` and the direct GGUF views; shared boundary pages and private/pinned buffers -are left alone. Unset or `0` keeps the previous behavior. On one 32 GB Windows 11 laptop with an RTX 4080 Laptop -and a Thunderbolt RTX 3090, IQ2_XS with `--mmap-experts --layer-split 12 --trim-stage-weights` raised median available -RAM from 0.51 to 12.19 GiB across 1K/4K/16K prompt trials, with 3.7-4.4% lower prefill throughput. It did not reduce -committed memory or physical SSD reads. This is a working-set hint, not an unmap or a guarantee that the OS drops -its file cache. See the [configuration, measurements and limits](../bench/results/2026-10-04-windows-mapped-release/README.md). - -**Low-RAM mode without `experts.bin` (engine 0.1.31):** for the native packs (IQ2_XS, IQ3_XXS, IQ3_S, the Coder, Swift, -Q2_0 packed by `tools/iq_pack.py`; not the canonical Q2_0 pack setup makes for AVX-512 CPUs) the mapped mode no longer -needs the pack's `experts.bin`: when the pack has none, the engine -maps the model's GGUF files themselves and reads each expert's gate, up and down rows from where `native_experts.txt` -says they are (the files are checked against it first: every tensor's name, type, shape, offset and bounds). That -saves the 23-50 GB copy on the disk. The answers are the same: on the Coder, 64 greedy tokens from `experts.bin` and -from the GGUF gave identical tokens and logits. An expert read from the GGUF is three reads instead of one, so the -engine fetches a layer's missing experts on 8 threads (`STRATA_FETCH_THREADS`) with one batched page request -(Windows `PrefetchVirtualMemory`). With an `experts.bin` in the pack, nothing changes. Setup does not use this yet. - -**Linux file tier, 0.2.0 defaults (zero-copy page-cached experts, io prefetch, the decode gate):** when experts are read -from a GGUF read in place (the native packs, also with `--mmap-experts`), Linux now runs three things by default -(0.2.1: an `experts.bin` pack gets io prefetch and the decode gate when the page cache has room for at least 55% of the experts, see below; zero-copy -does not apply to it, its blob is one range the kernels already read from the mapping). -(1) **Zero-copy** (`STRATA_COLD_OVERLAP=4`): an expert of a GGUF read in place whose three tensors' pages are all in the page -cache (mincore) is computed straight from the mapping; before, every such expert was first assembled into a staging buffer -(a copy of ~1.5 MB per Q2_0 expert, on the decode's critical path). Same bytes, same rows, same output (identity gates -IDENTICAL). (2) **io prefetch** (`STRATA_IO_PREFETCH=1`, the path described below): whole-blob `pread`s and the router- -predicted experts of the next layers read ahead. (3) **the decode gate** (`STRATA_IO_PF_GATE=1`): every 16 decode windows the -decode's read-ahead is judged - it runs only while the page cache has room for the experts (the memory available, the -tightest cgroup limit included, over the experts' bytes: off below 0.55, on above 0.65, `STRATA_IO_PF_GATE_LO` / `_HI`) and -while it pays (at least 128 MB read ahead in 16 windows and 10% of it used, `STRATA_IO_PF_GATE_USEFUL`; a probe every 256 -windows when it closed for that). Closed, the decode reads as it did without io prefetch (the look-ahead predicts one layer -and warms with an madvise hint); the prompt path keeps its preads either way. Measured on an RTX PRO 5000 emulating a 12 GB -card (`--expert-cache 4448`, Q2_0 GGUF in place, page cache dropped before each engine, RAM limited by a cgroup), the 0.2.0 -defaults against the 0.1.4x ones, 3 interleaved pairs, story / code / doc8k decode and 4K / 20K prompts: 32 GB of RAM -1.70 / 2.56 / 2.88x decode (34.0 / 28.0 / 21.7 -> 57.6 / 71.7 / 62.6 tok/s), 1.54 / 1.91x prompts; 16 GB 1.16 / 1.25 / 1.25x -decode, 1.63 / 1.68x prompts (the gate stays closed there); `--resident-budget-gib 18` in 32 GB 1.16 / 1.10 / 1.30x decode, -1.55 / 1.91x prompts; greedy token ids equal on story and code (doc8k varies run to run with the adaptive tier in both). On a -real RTX 3060 12 GB (32 GB cgroup, 3 pairs) zero-copy alone gave 1.14 / 1.44 / 1.47x decode, 1.06 / 1.02x prompts. With them, -the container RAM check counts a cgroup's clean page cache (active and inactive) as available, as MemAvailable does on a bare -host: before, the 18 GiB RAM tier in a 32 GiB cgroup re-checked "1.2 GiB available" once io prefetch's startup reads had left -their cache in the group, and its file tier switched to unbuffered reads (which also left zero-copy off). Off switches: `STRATA_COLD_OVERLAP=0`, `STRATA_IO_PREFETCH=0`, -`STRATA_IO_PF_GATE=0`; the startup log names what is on. Windows and macOS are unchanged. - -**Linux, `experts.bin` packs (0.2.1):** io prefetch and the decode gate are on by default when the page cache has room for at least 55% of the experts -(the tightest cgroup limit or MemAvailable over the pack's expert bytes, asked once at start; `STRATA_IO_PF_GATE_LO`). Below that the pack reads as before and the log says so; `STRATA_IO_PREFETCH=1` forces it, `0` turns it off. Measured on an RTX 3060 12 GB, IQ3_XXS pack (43 GB `experts.bin`), `--mmap-experts`, cgroup RAM limit, page cache dropped before each engine, 5 interleaved pairs: 32 GB decode 1.38x / 1.21x / 1.33x (story / code / doc8k, 5/5 pairs faster each), prompts 4K 1.04x, 20K 1.06x. At 16 GB io prefetch gave decode +3..8% but the cold 4K prompt 4-6% slower, so it stays off there (room 38%). Identity gates (fixed flags, no RAM limit and a 32 GB cgroup): 10/10 texts, exact_pp 4K / 20K residual dumps IDENTICAL. - -**Linux file-tier I/O path (`STRATA_IO_PREFETCH=1`; 0.1.40.2, on by default on Linux since 0.2.0 - see above):** the mapped file tier reads experts through -page faults. With this on, the layer's uncached experts and the router-predicted ones of the next layer (`STRATA_LOOKAHEAD_K`, -`STRATA_IO_PREFETCH_DEPTH`, default 2 when on) are read by I/O threads (`STRATA_IO_PF_THREADS`, default 8) with whole-blob -`pread`s into the page cache, and the CPU kernels read the mapping as before. The bytes are the same, so the output is the -same (greedy IQ3_XXS and Q2_0 hashes identical, prefetch off vs on). `STRATA_IO_PF_STAGE=1` instead stages every uncached -expert in a buffer and makes the layer wait for them; `STRATA_IO_PF_AHEAD=0` keeps only the demand reads. The per-request -log line "file tier I/O" shows what the OS read from the drive (/proc/self/io, major faults) against the engine's expert -reads; `STRATA_IO_STATS=1` (or prefetch on) adds the page-cache hit / miss split (mincore) and the read-ahead used / unused counts. -Measured (interleaved A/B, 10 pairs, 200-token greedy medians, page cache dropped before each run, memory limit by cgroup): - -| box, lane | default (fill) | `STRATA_IO_PF_STAGE=1` | -|---|---|---| -| RTX 3060, 32 GB, IQ3_XXS | -3.4% | **+25.7%** | -| RTX 3060, 16 GB, IQ3_XXS | -1.8% | -32.6% | -| Radeon 780M iGPU, 16 GB, Q2_0 (adaptive tier off) | -0.7% (story +3.8%, code +4.1%) | -19% | - -With the experts warm in the page cache (46 GB on the 3060, 60 GB on the iGPU box) the file tier already runs at the -resident speed, so there is nothing to win; the gap is the cold start and the low-RAM lanes. So it is off by default, and -the staging variant is worth trying only on a 32 GB-class box. **iGPU caveat:** on a Radeon 780M, prefetch together with the -adaptive tier (`--adapt-every`, on by default) under memory pressure reset the GPU in about 7 of 9 runs (the engine -prints a warning and runs as asked); with `--adapt-every 100000` there were no resets in ~40 runs. The cause is not found. - -**A RAM budget (engine 0.1.31, `--resident-budget-gib N`):** the resident variant for a model whose experts do not all -fit: the N GiB of experts the GPU cache does not hold that the expert profile ranks hottest are copied into RAM at -start (locked; page-locked when the driver allows the whole budget), and the rest are read from the files. -It implies `--mmap-experts` and leaves 4 GiB of headroom. On Windows, available commit capacity also -limits the budget; a larger N is clamped to the smaller limit less 4 GiB and a 256 MiB margin, with a message. -A clamped budget no longer fails the safety check that follows (#403). A budget that cannot be kept at all is a -warning, with every expert read from the files. Setup sets N with `--resident-budget-gib N`. When the budget has room -left after the ranked experts, the slots the prompt path borrows keep their experts in RAM too, as they do without a -budget (the engine prints `the prompt path's lend region: N of its M slots keep their experts in RAM too`); before, a -budget always left them out, and every prompt read the lent slots' experts from the model files again (RX 6900 XT, -UD-Q4_K_XL, budget 71 GiB with 7.5 GiB unused: 0.5-2.6 GB of file reads per agent turn and 1.9-2.8 GB from the SSD -per fresh 8K-16K prompt went to zero; fresh prompts read 3-8% faster, the prompt path's host staging fell from -0.8-2.1 s to 0.06-0.5 s per request). With -the GGUF read in place it also warms the next layer's likely experts: while the CPU works on a layer, a thread applies -the next layer's router to this layer's input and asks the OS for the pages of the predicted experts that neither the -GPU nor the RAM budget holds (only pages - the experts computed are the same; `STRATA_LOOKAHEAD=0` turns it off). This -is what runs [Unsloth's UD-Q4_K_XL](UNSLOTH_Q4.md) (72 GiB of experts) on a 64 GB PC: 7-8.5 tokens/s at N = 40 on an -RTX 5070, against ~3 tokens/s before these changes. - -**Opt-in switches measured on a hybrid Intel CPU under Windows (Core i7-13700K, 8 P + 8 E cores, RTX 5090, 64 GB RAM, -Windows 11, Swift 1.5 IQ3_XXS at 128K):** -- `--pool-affinity measure` (hybrid CPUs, native packs): the `all` and `p-cores` layouts are timed at start on the - pack's own experts (about a second) and the faster one is taken. Engine 0.1.41 built from source, the same prompt: - `all` (15 workers) spent 6.7 ms per verify window in the CPU pool and decoded 148 tok/s, `p-cores` (7 workers) 2.2 ms - and 188-191 tok/s; in another run of the same day `all` took 2.4 ms, so the difference comes and goes. The - measurement is taken once, at start: over four server starts of one day it read `all` at 14.8, 21.9 and 25.3 ms - against 6.1 to 6.2 ms for `p-cores` and took `p-cores`, and once read `all` at 6.6 ms against 8.6 ms and took `all`. - It does not look again during a session, so where the E-cores' speed changes while the engine runs, - `--pool-affinity p-cores` is the steady choice. Not measured: other CPUs, Linux, AMD cards. - -**Exact one-token expert rows (0.1.45, opt-in, `STRATA_CPU_IQ_ALL_EXACT1=1`, #1415 #1544):** below the #152 rule -(a lone token in an expert's group) an AVX-2 CPU without AVX-512 runs ggml-cpu's `vec_dot` for the i-quant gate/up rows. -With the switch, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS and IQ3_S take Strata's AVX-2 decode instead, with ggml's float order -(the same per-lane integer sums, one fmadd per block, the format constant after `hsum_float_8`), so the output does not -change, only the time (`iq_avx2_parity`: 0 of 5,120 outputs per format differ from ggml's; the identity gate with the -switch on is 10/10 plus exact_pp 4K/20K identical on RTX 3060, 2x A4000 and R9700 boxes). It is used only where -ggml-cpu itself is the AVX2 build (not an ISA-floor build, not AVX-512) and not with `STRATA_IQ_MT_MIN=1`. Per expert, -gate/up, one thread: IQ3_S -32% on a Threadripper PRO 3975WX (Zen 2) and -11% on a Ryzen 5 5600X (Zen 3), IQ2_XXS -5..-6%, -IQ3_XXS -5% (Zen 2) / +1% (Zen 3), IQ2_S 0% (Zen 2) / +7% (Zen 3, slower); on a Core Ultra 7 265 (VNNI, gather) IQ3_S -60%, -IQ3_XXS -39%, IQ2_XXS -34%, IQ2_XS -24%, IQ2_S -3% on a P-core. End to end, with the engine's decode timing: the CPU part of a -window falls 5% on a Xeon E5-2643 v3 + RTX A4000 (22.2 -> 21.1 ms of a 46.9 ms window, decode +0.5% over 10 interleaved pairs, -+2.5% on the story and code prompts) and on a Core Ultra 7 265 + RTX 3060 (8.2 -> 7.8 ms of 39.5 ms, decode +0.4%), and does not -move on a Threadripper PRO 3975WX + R9700 (14.5 ms, +0.4%, 5 pairs); no case reached +3% end to end, so the switch stays off. - -**Switches added in 0.1.40 (all off unless noted; none changes the default output):** -- `--kv-grow` (or `STRATA_KV_GROW=1`; `--no-kv-grow` turns it off): the K/V takes VRAM only for the cells the requests - reach, and the expert cache holds the rest, giving slots back as the context grows. It needs one GPU, an expert - profile, the whole K/V in VRAM (no KV streaming) and every expert in RAM (not the resident low-RAM mode); otherwise, - and with `--batch`, `--vram-elastic` or `--peer-device`, the engine says so and stays off. - On HIP (ROCm 6+) it runs through hipMemAddressReserve / hipMemCreate / hipMemMap (2 MiB chunks; the card must report - virtual memory management, `STRATA_HIP_VMM=0` turns it off); until then every AMD card printed "--kv-grow is off". - Radeon AI PRO R9700 with `--vram-reserve-mib 17084` (a 16 GB card), Q2_0 pack, `--max-context 196608`, 5 interleaved pairs: - expert-cache slots 6,213 -> 7,968 and decode 96.3 -> 98.5 tok/s (x1.023, 5/5; the noise arm read x0.999); at - `--max-context 262144` the default engine finds no VRAM left for the cache on that budget and does not start, `--kv-grow` - starts with 7,877 slots (94.4 tok/s). Prompts of 20K and 60K tokens through the growing K/V read as fast as without it - (x1.002, x1.001) and gave identical answers. At contexts up to 128K the K/V is small and nothing changes; it pays on cards of - about 24 GB or less with contexts of 128K and more. -- `--host-core last` (or `STRATA_HOST_CORE=last`, Windows): the host thread runs on the last physical core and the - workers take the first. Windows sends a GPU's interrupts to the first core, where a host spinning on the GPU's flags - waits for them (`--host-core first` is the default; the startup log names the cores). It leaves a hybrid CPU as it - is. `--host-core sibling` (or `STRATA_HOST_CORE=sibling`) keeps the host on the first core but on its other - hardware thread (SMT), so the interrupts keep the first logical processor and the workers keep every core; it works - on hybrid CPUs too, and is `first` on a core without SMT. On a 4060 Ti + 5080 layer split with an i9-14900KF it - turned a decode that swung by ±6% from run to run into a steady one, 8% faster on average. -- `STRATA_ADAPT_LAG=2` (#764): a window takes the adaptive tier's swaps once they are two windows old (default 1, - as 0.1.39). `STRATA_PREFILL_EQUAL=1` (#693): a prompt segment is read in chunks of equal size, not full chunks and a - short last one (changes the rounding). `STRATA_OWNED_PRICE=exact` (#796): the cache sizing prices the prompt path's - own buffers by their real allocation instead of 0.1.39's rule. -- `STRATA_Q2_BITPLANE=1`: a bit-plane row kernel for Q2_0 on CPUs with AVX2 and no AVX-512 (changes the last bits). - `STRATA_NO_AVXVNNI=1` turns the AVX-VNNI forms of the i-quant, IQ4_NL and Q2_0 rows off on CPUs that have them - (Alder Lake, Sapphire Rapids and later); they give the same bits as the AVX2 forms, 4-25% faster rows. -- Draft layer: `--lookup-chain-min M` (the shortest context match `--lookup-chain` extends, default 3), - `--mtp-hnorm stream` (one norm per stream, as llama.cpp's MTP graph) and `--mtp-draft-vocab FILE` (a token subset - of your own instead of `/draft_vocab.bin`). -- Elsewhere: HIP `STRATA_DENSE_MMQ=1` and `STRATA_HIP_ADAPT_KERNEL_COPY=1` in [AMD_HIP.md](AMD_HIP.md#model-and-serving-configuration); - `STRATA_KEEP_EMPTY_TURNS=1` and `STRATA_TOPK_STREAM=0` in [TROUBLESHOOTING.md](TROUBLESHOOTING.md). - - -**Read-ahead at start (Linux):** the weights, the native dense matrices, the GPU cache's fill from the profile, the -resident RAM copy and the MTP draft files are asked for ahead of their reads (madvise / posix_fadvise WILLNEED in -128 KiB steps), so the drive sees a deep queue instead of one page fault at a time. Measured on a Gen3 NVMe (RTX 5090, -32 GB, Q2_0 resident at 262K): ready in 70 s instead of ~920 s; the fill went from 39 MB/s to 3.2 GB/s. -`STRATA_READ_AHEAD=0` turns it off; `STRATA_FILL_AHEAD=N` sets how many fill pairs are asked for ahead (default 256). - -**How much came from where:** with `--stats` the engine prints the tiers of the decode (`expert tiers`: blobs from the -RAM copy, blobs and MB from the files, the time spent reading them; `routing prefetch`: how many of the file reads had -been warmed). The server log has the same per request (`expert tiers: GPU ... hits ...; RAM ... blobs, files ... -blobs ... MB read`), and `GET /metrics` lists `ram_blobs`, `file_blobs` and `file_mb` for each recent request (with -engine 0.1.31 or newer). It also lists each request's speculative drafts, `drafts_offered` and `drafts_accepted` -(`null` when the engine did not report them), and their sums since the server started in `totals` (#457). -Its `hit_rate` is the VRAM share of the experts looked up while answering: experts the GPU reads over PCIe -(`--pcie-frac`) are not in it, so a higher `--pcie-frac` raises it even when decoding gets slower. `pcie_share` -(engine 0.1.39 or newer, #588) is their share of all routed experts, and the server log and the Monitor tab show it -beside the hit rate. - -**Where a decode window's time goes (profiling, #610):** start the server with `STRATA_DECODE_TIMING=1` (and -`STRATA_VERIFY_PROFILE=1` for the GPU's side) in the environment. After each request the engine log then has one -`strata decode timing:` line - windows, tokens per window, and per window the verify time split into the wait for the -GPU, the CPU expert pool (plan, activation quantization, jobs) and the stage, plus commit and draft - and one -`strata decode GPU stages (ms/window):` line with the GPU time of each stage (GDN and QSA layers, the VRAM expert -hits, the router, the head, ...). The GPU profile times every stage with events, so it slows the decode a little: -use it to compare, not to measure speed. This works with every pack; `--gpu-stages` (a one-token replay of -per-layer graphs) refuses a native (IQ) pack, which has no such graphs. - -**How good the next-layer expert prediction is (measurement only):** `STRATA_LOOKAHEAD_STATS=1` runs the router look-ahead on any expert tier and -scores it against the routing of the next layer: every 100 windows the engine log has a `strata lookahead stats:` line with the experts per layer -outside the GPU cache, the recall and precision of the top-k / k+4 / k+10 guesses and of the vote-ranked top 4 / 8 / 12, how much of the PCIe share they -would cover, and the predictor thread's time per window. It costs CPU (the predictor is a thread beside the expert pool), so use it to measure, not to -time. `tools/replay_cache_policy.py` replays a `--dump-routing` trace through the adaptive tier's swap policy offline. - -Time to first token is prompt length / prompt speed: with Q2_0 about 4 s at 4K, 25 s at 32K, under 2 minutes at 128K -and 4.5 minutes at 262K (engine 0.1.13 made long prompts about twice as fast, below). - -**Faster prompts (engine 0.1.13):** the prompt is read in chunks of up to 8,192 tokens instead of 2,048 (`--prefill -auto`: the largest chunk whose buffers fit in the expert-cache slots it borrows, and a request borrows only what its -prompt needs); the experts are multiplied by llama.cpp's quantized MMQ kernels instead of being expanded to FP16 -first; the next layer's experts stream over PCIe while the current layer's attention runs; the PLE block runs for the -whole chunk at once; unpinned experts are copied by helper threads. Measured on the RTX 5070 12 GB, 64 GB RAM, -32K-token prompt: Q2_0 572 -> 1,290 tokens/s, IQ3_S 383 -> 1,208. Through the server (Q2_0, 128K context): 999 tokens -353 -> 438 tokens/s, 6,927 tokens 529 -> 1,077, 28,584 tokens 584 -> 1,249. Output speed is unchanged. Needles 5/5 -(1K-262K). Details and the quality check: -[`bench/results/2026-09-28-prefill-speed`](../bench/results/2026-09-28-prefill-speed/README.md). Existing installs -switch to `--prefill auto` the next time START-HERE / setup.sh starts them. The raw numbers: -[`bench/results/`](../bench/results/). The [paper](paper/Strata-Paper.pdf) explains every number. - -## Other GPUs (estimated) - -Not measured - estimated from the runs above (same CPU and 64 GB RAM): the GPU part scaled by memory bandwidth, the CPU -part by how many more experts the card's VRAM holds. Treat as **±20%**. Numbers are *prompt / output* tokens/s. -The prompt figures predate engine 0.1.13, which about doubled prompt speed on the measured card; how much of that a -card gains depends on its PCIe link (the experts stream over it), so they are still the older estimates. - -| GPU | Model | 1K | 4K | 32K | 64K | 128K | 262K | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| RTX 5060 Ti 16GB | Q2_0 | ~341 / ~80 | ~472 / ~87 | ~501 / ~81 | ~492 / ~72 | ~476 / ~62 | ~435 / ~53 | -| | IQ2_XS | ~291 / ~80 | ~406 / ~77 | ~434 / ~63 | ~426 / ~62 | ~413 / ~51 | ~383 / ~47 | -| | IQ3_XXS | ~249 / ~66 | ~359 / ~65 | ~381 / ~56 | ~374 / ~54 | ~363 / ~45 | - | -| RTX 3090 24GB | Q2_0 | ~355 / ~128 | ~491 / ~140 | ~521 / ~130 | ~512 / ~115 | ~495 / ~100 | ~453 / ~85 | -| | IQ2_XS | ~303 / ~131 | ~422 / ~128 | ~451 / ~103 | ~444 / ~102 | ~430 / ~85 | ~398 / ~78 | -| | IQ3_XXS | ~260 / ~106 | ~374 / ~103 | ~396 / ~89 | ~390 / ~85 | ~378 / ~71 | - | - -More VRAM matters more than a faster GPU: every extra GB holds ~700 more experts, and every expert on the GPU is one the -CPU does not have to compute. A 3090's 24 GB takes most of the CPU work away. (Since 0.1.14 the expert profile ranks -all 24,576 experts; before, the cache stopped at 8,000, about 10-14 GB. `tools/make_profile.py` builds a profile from -your own prompts: run the engine once with `--dump-routing trace.bin`, see the tool's help. The shipped profile already -ranks every pair, so a trace only changes the order with `--reorder` (0.1.39, #587): your trace's pairs first, then the -base's, then the rest.) +More VRAM matters more than a faster GPU. Every extra GB holds about 700 more experts, and every expert on the GPU is one +the CPU does not have to compute. `tools/make_profile.py` builds an expert profile from your own prompts: run the engine +once with `--dump-routing trace.bin`, then see the tool's help. The shipped profile ranks every expert pair, so a trace +changes the order only with `--reorder` (your trace's pairs first, then the base's, then the rest). ## Which model? -All three are [ISTA-DASLab's GSQ-RCO quantizations](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF) -of [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next). +[MODELS.md](MODELS.md) is the page for choosing a model: sizes, memory, providers and what fits. The families are +DeepSeek V4.1-Flash, Qwen3.8-Flash-Next (with its Coder, Swift, Gyro, Unsloth and NVFP4 variants) and Qwen3.6-35B-A3B +([QWEN36.md](QWEN36.md)). The ISTA-DASLab Qwen3.8-Flash-Next packs are: | Model | Download | RAM it uses | Speed | Quality | | --- | ---: | ---: | --- | --- | | **Q2_0** | 66 GB | ~34 GB experts + ~6 GB | fastest | good | | **IQ2_XS** | 68 GB | ~36 GB experts + ~6 GB | close to Q2_0 | a bit better | -| **IQ3_XXS** | 76 GB | ~43 GB experts + ~6 GB | slower (more CPU work) | best | +| **IQ3_XXS** | 76 GB | ~43 GB experts + ~6 GB | slower (more CPU work) | best of the three | -With 64 GB of RAM all three fit (close the browser for IQ3_XXS, and keep its context at 128K or less). With 48 GB only Q2_0 / IQ2_XS may fit. With 32 GB: the Coder (below). +IQ3_S is a fourth, larger pack: see [MODELS.md](MODELS.md#pick-by-memory). With 64 GB of RAM these three fit (close the browser +for IQ3_XXS and keep its context at 128K or less). With 48 GB only Q2_0 and IQ2_XS may fit. With 32 GB use the Coder. ### Or: the Coder (half the experts, for code) -**[Qwen3.8-Flash-Next GSQ-RCO Coder](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF)** is -ISTA-DASLab's expert-pruned release: 256 of each layer's 512 experts are kept (still 10 active per token), chosen with -RCO on code, agentic and vision calibration data; its authors report 91.3% of the full model's SWE-bench Verified and -98.7% of LiveCodeBench v6. One size, named IQ1_M for its 1.89 bits per *original* parameter; the kept experts are -stored like IQ3_S (IQ2_S-IQ4_XS gate/up, IQ4_NL/Q2_0 down). Shard 1 is 29.6 GB (experts: 23 GB of RAM), so it runs -on **32 GB of RAM**, and at 262K on 64 GB. Its shard 2 and its vision encoder are the original's files: with the -original installed, setup downloads only shard 1. Strata ships its expert profile (`data/expert-profile-coder.bin`, -the shipped ranking mapped onto the kept experts through the release's `rco-allocation.txt`: 72% of the expert -reads hit the GPU on a 12 GB card). Images work; the experimental speed projection loads and runs on it (it was made -for the full model). - -``` -START-HERE.bat --setup --family coder -``` +[ISTA-DASLab's expert-pruned release](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF) keeps 256 of +each layer's 512 experts (10 still active per token), chosen on code, agent and vision data. Shard 1 is 29.6 GB (23 GB of +experts in RAM), so it runs on 32 GB of RAM. Images work. Install it with `strata launch --family coder` +(from a source checkout: `START-HERE.bat --setup --family coder`). Details: [MODELS.md](MODELS.md#coder). ### Or: Swift 1.5 (a fine-tune that thinks shorter) -The setup's first question also offers **[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)**, -UkisAI's fine-tune of Qwen3.8-Flash-Next, trained to reach the answer with much less thinking (its authors: 63% fewer -thinking tokens, 1.8x sooner answers, under 1% accuracy loss). Same architecture, the original's sizes except Q2_0, -its own -vision encoder; Strata runs it at the same speed (4K, IQ2_XS: 465 prompt / 78.7 output tokens/s, vs 467 / 78.3 for -the original). Its authors recommend **IQ2_XS** (their Q2_0 is marked experimental). Its license is the Swift Open -License 1.0 - read it on the model page. +[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF) is UkisAI's fine-tune. It has the same +architecture and speed as the original, its own vision encoder, and thinks for fewer tokens. Its authors recommend IQ2_XS. +The license is the Swift Open License 1.0. Install it with `strata launch --family swift --model iq2_xs` +(from a source checkout: `START-HERE.bat --setup --family swift --model IQ2_XS`). Details: [MODELS.md](MODELS.md#swift-15). -Our small check (8 reasoning questions, default thinking, IQ2_XS): both models got **8/8**; Swift used **1,234** -output tokens in 28 s, the original **2,682** in 46 s - most of the difference from one question the original -thought about for 1,524 tokens. Not a benchmark, but consistent with the claim. + -``` -START-HERE.bat --setup --family swift --model IQ2_XS -``` +### Unsloth's UD-Q4_K_XL -### Experimental: Unsloth's UD-Q4_K_XL - -A 4-bit quantization of the same model (111 GB, 72 GiB of experts). Setup offers it from engine 0.1.32 -(`--family unsloth --model UD-Q4_K_XL`: a RAM budget of your RAM less 24 GB, the rest read from the SSD); what it -does and the manual workflow are in **[docs/UNSLOTH_Q4.md](UNSLOTH_Q4.md)**. On a 64 GB PC with a 12 GB RTX 5070 it writes 7-8.5 -tokens/s, most experts read from the SSD; it picks the same tokens as llama.cpp on the same file at 97.5-99% of -the positions of short greedy answers, 90-91% after a 16K prompt, differing mostly at near-ties -([measured](UNSLOTH_Q4.md#quality-against-llamacpp-on-the-same-file)). - -## Before you start - -You need **only a graphics driver**: NVIDIA 580 or newer (update it with the NVIDIA App or from -[nvidia.com/drivers](https://www.nvidia.com/drivers)), or for AMD the one in [INSTALL.md](INSTALL.md#what-you-need). -Everything else is installed for you the first time. - -| | | -| --- | --- | -| GPU | NVIDIA **RTX 20, 30, 40 or 50 series**, **12 GB VRAM or more** (8 GB runs, slowly). Measured on an RTX 5070 and an RTX 3090; RTX 20 (Turing, since 0.1.27) was tested by a contributor on an RTX 2070. Or AMD **Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700, RX 6800 / 6900 series**: [AMD_HIP.md](AMD_HIP.md). | -| RAM | **64 GB** recommended (see the table above). | -| CPU | x86-64 with AVX2 (any Intel/AMD desktop CPU from the last ~8 years). AVX-512 (Ryzen 7000/9000) is a bit faster. Older CPUs without AVX2 are experimental and slow: [Older CPUs](INSTALL.md#older-cpus-experimental). | -| Disk | ~70-80 GB free for the model, ~6 GB for the MTP layer (+1 GB with images). **Q2_0 on an AVX-512 CPU** also writes a one-time ~40 GB pack for the fast CPU kernel (34 GB of experts in its layout, plus the dense weights). An NVMe SSD is strongly recommended. | -| OS | Windows 10/11, or Linux (Ubuntu 22.04/24.04 and Debian 12/13 get everything installed automatically). | - -What the first start installs: in this folder `.venv/`, `engine/` and `third_party/`; the model files (`models/`, -`packs/`, `mtp/`, 70-120 GB) in **`Strata-data` next to this folder**, so a new copy of Strata (an update unzipped -elsewhere) finds them and sets itself up the same way. The place is remembered per user (`%APPDATA%\Strata\settings.json`, -`~/.config/strata/settings.json`); `--data-dir` chooses another. Installs from before 0.1.16 are moved there by the next -start (a rename on the same drive; files on another drive are used where they are). -Python 3.12 if you have none (for your user account, no admin), a private Python environment, NVIDIA's CUDA libraries -(from pip, ~0.4 GB), the ready-made Strata engine for RTX 20/30/40/50, the model and the MTP draft layer. If no -ready-made engine fits your PC, it offers to install the build tools (Visual Studio Build Tools + CUDA Toolkit on -Windows, `build-essential` + CUDA on Ubuntu) and compiles the engine for your GPU (asks first; 20-40 minutes once). +A 4-bit quantization of the same model (111 GB, 72 GiB of experts). Install it with `--family unsloth --model UD-Q4_K_XL` +(RAM budget: your RAM less 24 GB, the rest read from the SSD). What it does and the manual workflow are in +[UNSLOTH_Q4.md](UNSLOTH_Q4.md). --- ## Windows -### Double-click `START-HERE.bat` +On Windows the source wrapper is `START-HERE.bat` ([source installation](SOURCE_INSTALL.md#from-source-windows)). For a +released app, run its installer and open Strata from the Start menu. For the Terminal, use `strata launch`. Watch the +setup plan and the startup log for what is actually chosen on your machine, and see [TROUBLESHOOTING.md](TROUBLESHOOTING.md) +if a component is missing. -**The first time** it asks four questions and does the rest: +Terminal chat from a checkout: `.venv\Scripts\python chat.py`. -1. **Which model?** Qwen3.8-Flash-Next (the original) or Swift 1.5 (the fine-tune that thinks shorter). -2. **Which size?** Q2_0, IQ2_XS or IQ3_XXS (it recommends one for your RAM). -3. **How much context?** 8K to 256K tokens (it recommends one for your VRAM). -4. **Images?** yes / no (see [Images](#images-vision)). +### Tuning for your PC (`--calibrate`) -Then it downloads and prepares everything (the model is 66-76 GB, so the first start takes a while; an interrupted -download continues where it stopped) and **starts the model**: your browser opens `http://127.0.0.1:8080`, the Strata -app. It has three tabs: -- **Chat:** streaming answers, the model's thinking (folded away once it answers), code with a copy button, pictures when - images are on, and sampling and thinking-level settings. Chats stay in your browser. -- **Monitor:** what the model is doing (reading the prompt, with progress, or writing, at how many tokens/s); GPU load, - VRAM, temperature, power and PCIe traffic; CPU, RAM and disk; the context in use; the last requests. +Some engine settings depend on the PC more than on the model: -On a Mac (Metal backend) the Monitor's GPU load and VRAM come from IOKit's IOAccelerator nodes - the driver's Device -Utilization % and its in-use system memory, against the device's Metal working set - and the card says the rest: there -is no temperature or power reading without root (`powermetrics`), and an Apple Silicon GPU sits on the die, so there is -no PCIe link. The engine's own VRAM figures are in its log. -- **About:** the model and engine settings, and the addresses to connect other apps. +| Setting | What it controls | +| --- | --- | +| `--pcie-frac` | Share of the experts missing from VRAM that are copied to the GPU instead of computed by the CPU. A fast PCIe link and a slow CPU want more; a laptop's narrow link wants less. | +| `--spec-min-p` | How sure the draft layer must be to add another guess to a check. | +| `--pool-workers` | How many CPU threads compute experts. On CPUs with efficiency cores, fewer can be faster. | +| `--adapt-every`, `--adapt-swaps`, `--adapt-decay` | How busily the VRAM expert cache follows the conversation. A CPU that reads missed experts slowly (DDR3 or DDR4 in few channels) gains from more swaps. | +| `--prefill-pipe` | On two or more cards (layer split), the prompt chunks that keep the cards reading at once. See [MULTI_GPU.md](MULTI_GPU.md). | -`http://127.0.0.1:8080/?q=your question` opens it with a new chat already asking. The API is at -`http://127.0.0.1:8080/v1` for your apps. - -**Every time after that**, `START-HERE.bat` just starts the model (30-90 s to load 34-43 GB into RAM). Nothing is -downloaded again. Closing the window stops the model. - -``` -START-HERE.bat --setup install another model, or change context / images -SETUP.bat the same (double-click it) -START-HERE.bat --model IQ2_XS --context 32768 --vision yes --yes no questions -START-HERE.bat --gguf-dir D:\models\IQ2_XS use GGUF files you already have -START-HERE.bat --data-dir E:\Strata-data keep the model files somewhere else -START-HERE.bat --port 8081 another port -START-HERE.bat --gpu 1 another GPU (numbered as nvidia-smi; setup picks the one with the most VRAM) -START-HERE.bat --calibrate tune the engine for this PC (about 15-30 minutes, longer on a slow card), then start -``` - -With more than one model installed, it asks which one to start. `run-.bat` starts a model directly. - -**Tuning for your PC (`--calibrate`, engine 0.1.19).** These engine settings depend on the PC more than on the model: -- the share of the experts missing from VRAM that are copied to the GPU instead of computed by the CPU - (`--pcie-frac`: a fast PCIe link and a slower CPU want more, a laptop's narrower link less); -- how sure the draft layer must be to add another guess to a check (`--spec-min-p`); -- how many CPU threads compute experts (`--pool-workers`: on CPUs with efficiency cores, fewer can be faster); -- how busily the VRAM expert cache follows the conversation (`--adapt-every`, `--adapt-swaps`, `--adapt-decay`): a PC - whose CPU reads the missed experts slowly (DDR3 or DDR4 in few channels) gains from swapping more. On a Xeon - E5-2673 v3 with DDR3 and an RTX 4060 Ti on PCIe 3.0 x8, 160 swaps every window measured +7.9% over the default - (Q2_0, with the swaps taking effect a window later, #764); see #906; -- on two or more cards (a layer split), the prompt chunks that keep the cards reading at once (`--prefill-pipe`, the - prompt read speed of fresh prompts rather than the output speed; see docs/MULTI_GPU.md). - -The defaults were measured on a Ryzen 5 7600 with an RTX 5070. Setup offers to measure them on your PC after an -install; `START-HERE.bat --calibrate` (Linux: `./setup.sh --calibrate`) does it any time. It measures the output -speed with each setting and keeps one only when it is more than 3% faster. The result is remembered per PC and model -(in the settings file next to the data folder's record), so updates keep it. - -Measuring the worker count needs a fresh engine, so the model is loaded more than once: the PC is busy, and can -stop responding for a minute or two, once per restart. It then starts the model, like a plain `START-HERE.bat`. - -**Manual CPU task granularity.** `--pool-tasks N` sets the target total number of row tasks in each batched -CPU expert Gate/Up and Down phase, not the number of threads or tasks per expert. The default `0` keeps -three tasks per participating thread (including the host when enabled). Values `1..4096` are capped by -each phase's row count; native batches larger than the pool's capacity apply the target to each sub-batch. -For example, `--pool-workers 13 --pool-tasks 192` uses 14 participating threads with the default host worker. -More tasks can reduce imbalance between cores, but also add scheduling overhead: compare against `0` with -the same worker count and workload. This does not change kernels, phase barriers, or PCIe placement, and -does not affect the legacy single-token/oracle fallback. Setup's `--calibrate` does not tune it yet. -For the server, add `"--pool-tasks", "192"` to the existing `args` list in its configuration, then restart it. - -**How many CPU threads the engine starts (`--pool-workers`).** Without the setting the engine picks the number of -CPU expert workers from the CPU, not from the count of logical threads: one worker per *physical* core (never an SMT -sibling) except the core of the main thread, one core less on a CPU with 17 or more cores, only the cores of the main -thread's CPU socket on a multi-socket machine, at most 32, no more than a container's CPU limit (docker `--cpus`) allows, -and on a hybrid CPU with more efficiency than performance cores the performance cores but one plus half of the efficiency -cores. When the OS does not say which logical processors share a core (some containers), it counts half of them. The start -line says what bounded the number (`CPU pool: 31 workers by default (32 physical cores, 64 logical processors; bounded by ...)`). -The workers wait by spinning, so a worker without a core of its own takes time from the others: on a 20-core CPU 24 -workers decoded 44% slower than the default, on a 32-core EPYC 63 workers 35% slower. A count in the config -(`--pool-workers N`) is always used as given; when it is above what the CPU uses well the engine prints a plain warning -with the recommended number at start (`strata cpu pool: warning: ...`). Setup writes no worker count except the hybrid -CPU's, capped the same way, and gives the CPU vision encoder at most 16 threads (`--threads` in the `vision` entry of the -config changes that). - -**Two sockets (community measurements, #3201; the default is the host's socket only).** Two dual-socket boxes gave opposite -results for workers on both sockets, so measure on yours: `--pool-workers N` with N above one socket's cores, a fresh server -for each value, the same prompt, decode tok/s from the log. On 2x Xeon Gold 6226 (2x12 cores, 4 of 6 memory channels per socket, -RTX 3090, IQ4_XS at 262K, about 25% of the experts computed on the CPU) the default (11 workers) gave 52.6 tok/s and 22 workers -gave 50.3, lower and noisier. On 2x Xeon E5-2670 v2 (AVX only, 2x CMP 50HX, IQ3_XXS at 393K, a 98% VRAM hit rate: the CPU computes -about 2% of the experts) the default (9 workers) gave 65.5 tok/s and `--pool-workers 19` gave 75.1 (+15%, six runs, 74.9 to 75.1). -So the cores of the second socket can pay off when the CPU's part of a window is short and its slowest expert sets the pace, and cost -when the sockets wait for each other's counters. Prompt speed did not change in either. Check the IOMMU too: on a distro kernel -with it translating every DMA (`cat /sys/kernel/iommu_groups/*/type | sort | uniq -c` shows `DMA-FQ`), pinned host-to-device copies ran at -7.7 to 9.2 GB/s on PCIe 3.0 x16 and at 12.3 GB/s with `iommu=pt` on the kernel command line, which moved prompt speed by about 30% there. -A NUMA-aware pool (one slice of workers per socket, each reading its own socket's copy of the CPU experts) gave +10 to +14% on -the first box in a contributor's branch; 0.2.2 has it (on by itself with 2 or more NUMA nodes, `STRATA_NUMA_POOL=0` turns it off: see "NUMA-split CPU expert pool" below). Do not wrap the server in `numactl --cpunodebind` when you test -workers across sockets: it pins them to one node. +The defaults were measured on a Ryzen 5 7600 with an RTX 5070. From a source checkout, `START-HERE.bat --calibrate` (Linux: `./setup.sh --calibrate`) +measures output speed with each setting on your PC and keeps one only when it is more than 3% faster. The result is +remembered per PC and model, and updates keep it. Measuring the worker count restarts the model more than once, so the PC +is busy and can stop responding for a minute or two at each restart. ### Running it at startup (Task Scheduler) -To have the model up at logon, people start the serve from **Task Scheduler** (or a service). Beware: Windows -throttles such contexts, and the model's ~40 GB expert load then crawls at **~0.05 GiB/s (13-14 minutes)** -instead of **~1.4-1.5 GiB/s (~35 seconds)** - a 24x slower start. Measured on an RTX 5070 Ti + Ryzen 7 9800X3D -+ NVMe, same binary, same args, same cache state: +To have the model up at logon, start the server from Task Scheduler or a service. Windows throttles such contexts by +default, and the expert load then crawls. -| How the serve starts | Expert load | +**Measured on:** RTX 5070 Ti, Ryzen 7 9800X3D, NVMe, a ~40 GB expert load, same binary, arguments and cache state. + +| How the server starts | Expert load | | --- | ---: | -| Double-click / terminal / SSH | 1.42-1.52 GiB/s (~35 s) | +| Double-click, terminal or SSH | 1.42-1.52 GiB/s (~35 s) | | Task Scheduler with its defaults | 0.05 GiB/s (821-841 s) | | Task Scheduler with the two settings below | 1.42 GiB/s (35 s) | In the task's properties set both of these (the defaults are the opposite): -- **Priority level: Normal** (Options tab; the default is Below normal), and -- **Run with highest privileges** (General tab; without it the task runs with a limited user token - which - also strips `SeLockMemoryPrivilege`, the privilege Windows large pages need). +- **Priority level: Normal** (Options tab; the default is Below normal). +- **Run with highest privileges** (General tab). Without it the task runs with a limited token, which also removes + `SeLockMemoryPrivilege`, the privilege Windows large pages need. -(Both were changed at once, so the isolated effect of each is not measured.) If the model still starts -slowly, the engine prints a hint under its `loaded ... GiB at ...` line naming this cause. +Both were changed at once, so the effect of each alone is not measured. If the model still starts slowly, the engine prints a +hint under its `loaded ... GiB at ...` line. -**Large pages need a new logon (#1412).** Granting "Lock pages in memory" (`secpol.msc`, User Rights Assignment) to the account that runs the engine takes effect at the next logon: Windows puts the privilege in the access token when the session starts, so log off and on (or reboot) after granting it. Until then the startup log still says the large pages were refused. `tools\windows\enable_large_pages.ps1` (run as Administrator) grants it to the current account with `secedit`. - -### Chat in the terminal (optional) - -``` -.venv\Scripts\python chat.py -``` +**Large pages need a new logon.** Granting "Lock pages in memory" (`secpol.msc`, User Rights Assignment) to the account +that runs the engine takes effect at the next logon. Log off and on after granting it. Until then the startup log says +large pages were refused. `tools\windows\enable_large_pages.ps1` (run as Administrator) grants it to the current account. --- ## Linux +These notes are for **source setup**. Normal installs use [the Terminal or AppImage](INSTALL.md). + ```bash ./setup.sh ``` -The same questions, the same automatic install (it uses `sudo apt` for Python and, only if it has to compile, -for the build tools), and the same start: `http://127.0.0.1:8080`. Later runs of `./setup.sh` (or `./run-.sh`) -start the model directly. Options as on Windows (`./setup.sh --setup`, `--model Q2_0 --yes`, `--gguf-dir /data/Q2_0`). -Terminal chat: `.venv/bin/python chat.py`. -Building from git, trying a pull request, the monitor from a checkout: [INSTALL.md](INSTALL.md#build-from-git-on-linux-to-try-a-pull-request-or-change-the-code). +Source setup offers choices for the detected machine and can install missing Debian or Ubuntu dependencies with `sudo apt`. +It starts the server on `http://127.0.0.1:8080` unless you configure another address or port. Later runs of `./setup.sh` +or `./run-.sh` start the model directly. The options are the same as on Windows (`./setup.sh --setup`, +`--model Q2_0 --yes`, `--gguf-dir /data/Q2_0`). Terminal chat: `.venv/bin/python chat.py`. Building from git and trying a pull +request: [SOURCE_INSTALL.md](SOURCE_INSTALL.md#build-from-git-on-linux-to-try-a-pull-request-or-change-the-code). -- **Updating:** `git pull`, then `./setup.sh`: it compiles the engine again when its source changed (a minute or - two for the changed files). If that compile fails, it says so and starts the engine you had. -- **Other distributions** (Arch, Fedora, ...): install the C++ compiler and the CUDA Toolkit 13 with your package - manager first (Arch: `sudo pacman -S base-devel cuda`); setup finds `nvcc` on PATH, in `/usr/local/cuda*` and in - `/opt/cuda*`, and does the rest. -- **WSL** works (Ubuntu 24.04 tested), with one limit: the NVIDIA driver pins only about 1 GB of RAM there, so KV - streaming (`--kv-resident`) is off and the KV cache stays in VRAM, and the experts are copied to the GPU from - unpinned RAM (slower prompts than native Linux). +- **Updating:** `git pull`, then `./setup.sh`. It compiles the engine again when its source changed. If the compile + fails, it says so and starts the engine you had. +- **Other distributions** (Arch, Fedora and others): install the C++ compiler and the CUDA Toolkit 13 with your package + manager first (Arch: `sudo pacman -S base-devel cuda`). Setup finds `nvcc` on `PATH`, in `/usr/local/cuda*` and in + `/opt/cuda*`. +- **WSL** works (Ubuntu 24.04 tested) with one limit. The NVIDIA driver pins only about 1 GB there, so KV streaming + (`--kv-resident`) is off, the KV cache stays in VRAM and experts are copied from unpinned RAM, which makes prompts + slower than on native Linux. + +--- + +## Memory modes + +### KV cache + +| Option | What it does | +| --- | --- | +| `--kv int8` (default) | 8-bit KV cache. | +| `--kv q4_0` | Half the KV memory with a Hadamard rotation before 4-bit rounding. About 4% faster at 128K, measurably less precise on long documents (perplexity +8-12%; needle tests still pass). Setup asks above 8K context. | +| `--kv k8v4` | Keys at 8 bits, values as rotated 4-bit: 23% less KV memory than 8-bit, so more experts fit in VRAM. More precise than `q4_0` on long documents. Prompts are 2-5% slower. | +| `--kv-streaming on\|off\|auto` | Keeps the context's KV cache in RAM and only the part attention reads in VRAM (`--kv-resident 32768`), so more experts fit on the GPU. Costs about 13.7 KB of RAM per context token (1.7 GB at 128K). Setup turns it on from 64K when the RAM has room; `on` overrides the RAM test with a note. Never under WSL. | +| `--kv-grow` (or `STRATA_KV_GROW=1`; `--no-kv-grow` turns it off) | The KV takes VRAM only for the cells the requests reach; the expert cache holds the rest and gives slots back as the context grows. Needs one GPU, an expert profile, the whole KV in VRAM and every expert in RAM. Otherwise the engine says so and stays off. On HIP it uses virtual memory management (`STRATA_HIP_VMM=0` turns that off). It pays on cards of about 24 GB or less with contexts of 128K and more. | + +`STRATA_KV_HOST_REGISTER=1` forces the KV host copy to be pinned with `cudaHostRegister` (the order WSL2 uses +automatically); `0` forces `cudaHostAlloc` first. + +### Low-RAM modes (chosen by setup) + +Normally all of a model's experts are copied into RAM (23-50 GB, pinned) and the GPU holds a copy of the most-used ones. +When the RAM cannot hold them beside the system (the experts plus about 10 GB), setup picks one of these: + +| Mode | Flag | What it does | +| --- | --- | --- | +| Mapped | `--mmap-experts` | Maps the experts from one file in the model's folder (the pack's `experts.bin`, +23-50 GB of disk), or from the GGUF files themselves for native packs that have no `experts.bin`. The OS file cache holds what the GPU does not and can give that memory back. With a small GPU most experts come from the SSD and it is much slower; setup says so. | +| Resident | `--resident-experts` | When the experts the GPU does not hold fit the RAM, the engine copies exactly those into RAM at start. Nothing is read from the SSD while it answers. | +| RAM budget | `--resident-budget-gib N` | The N GiB of experts the GPU does not hold that the profile ranks hottest are copied into RAM; the rest are read from the files. Implies `--mmap-experts` and leaves 4 GiB of headroom. On Windows the available commit limit also caps N, and a clamped budget is reported. This is what runs [Unsloth's UD-Q4_K_XL](UNSLOTH_Q4.md) on a 64 GB PC. | + +- In a source checkout, `START-HERE.bat --setup --low-ram on|off` overrides the choice, and `--low-ram resident|mmap` forces one variant. +- The resident variant keeps 4 GB of the RAM it finds free (`STRATA_RESIDENT_HEADROOM_GIB`). When even the missing experts do + not fit, it says so and runs the mapped mode. The server log shows per request how many expert reads went to the file + (`resident RAM: ... blob reads from the file`, 0 in steady use). +- The adaptive tier (`--adapt-every`) keeps following the conversation in the resident mode. `--adapt-async 1` (opt-in, + `--serve`) moves the swaps between decode windows on a helper thread. With `--batch` slots it works between their + windows too, but not with `--batch-groups` or `--peer-device`. It is not bit-exact from run to run. +- `STRATA_RAM_ADAPT=` (opt-in, `--serve`, one GPU, needs `--adapt-every`): the RAM copy also follows the conversation, + with up to `n` experts swapped per round. +- Several GPUs: setup recommends one GPU in the low-RAM mode and asks. `--gpus 0,1` shares the model across both cards + ([MULTI_GPU.md](MULTI_GPU.md#using-it)). `--yes` keeps one GPU. +- With a page-locked copy the GPU also takes its share of the misses over PCIe (`--pcie-frac`). `--pcie-frac 0` (or + `STRATA_RESIDENT_PIN=0`) gives the mapped mode's exact tokens. +- Linux, opt-in: `STRATA_ARENA_MMAP=1` maps a native pack's expert arena read-only from `experts.bin` instead of reading + it into locked RAM (for example 2x 16 GB GPUs with 32 GB of RAM). Run it with `--pcie-frac 0`. +- Windows, opt-in: `STRATA_FILE_RELEASE=1` trims the file-backed pages of experts after their GPU uploads. It is a + working-set hint, not an unmap, and it does not reduce committed memory or SSD reads + ([measurements and limits](../bench/results/2026-10-04-windows-mapped-release/README.md)). +- Windows, opt-in: `STRATA_PIN_BY_NEED=1` (arena on one GPU) registers the profile's non-GPU-resident experts by rank up to + `STRATA_ARENA_PIN_GIB` (29 by default on Windows) instead of the first ~30 GiB. +- Windows CUDA keeps 640 MiB then 160 MiB of page-lockable memory free while the arena pins (`STRATA_PIN_RESERVE_MIB=N`, `0` + turns it off, and it sets the reserve on any OS). If a required buffer is still refused, the error ends with + `[pinned-memory-exhausted]` and names the `STRATA_ARENA_PIN_GIB` to start with. +- `STRATA_EXCHANGE_ROTATE=1` (opt-in): an adaptive swap hands buffer ownership over instead of copying the evicted blob into + the RAM copy. Same tokens, fewer host copies, about 1.4 GiB more page-locked RAM. See + [EXCHANGE_ROTATION.md](EXCHANGE_ROTATION.md). + +### Linux file tier + +When experts are read from GGUF files in place, Linux runs three things by default. Off switches are in brackets. + +| Part | Variable | What it does | +| --- | --- | --- | +| Zero-copy | `STRATA_COLD_OVERLAP=4` (`0` = off) | An expert whose three tensors' pages are all in the page cache is computed straight from the mapping, with no staging copy. | +| io prefetch | `STRATA_IO_PREFETCH=1` (`0` = off) | I/O threads read the layer's uncached experts and the router-predicted ones of the next layers with whole-blob `pread`s into the page cache. Tunables: `STRATA_LOOKAHEAD_K`, `STRATA_IO_PREFETCH_DEPTH` (2), `STRATA_IO_PF_THREADS` (8), `STRATA_IO_PF_STAGE=1`, `STRATA_IO_PF_AHEAD=0`. | +| Decode gate | `STRATA_IO_PF_GATE=1` (`0` = off) | Every 16 decode windows the read-ahead is judged. It runs only while the page cache has room for the experts (off below 0.55, on above 0.65: `STRATA_IO_PF_GATE_LO` and `_HI`) and while it pays (`STRATA_IO_PF_GATE_USEFUL`). | + +For an `experts.bin` pack, io prefetch and the gate switch on when the page cache has room for at least 55% of the experts +(the tightest cgroup limit or `MemAvailable`, asked once at start). Below that the pack reads as before and the log says so; +`STRATA_IO_PREFETCH=1` forces it. The container RAM check counts a cgroup's clean page cache as available. The startup log +names what is on. Windows and macOS are unchanged. On a Radeon 780M iGPU, prefetch together with the adaptive tier under +memory pressure reset the GPU in most runs; with `--adapt-every 100000` there were none, and the cause is not found. + +The weights and the MTP draft files are asked for ahead of their reads at start (madvise and `posix_fadvise` WILLNEED, 128 KiB +steps), which matters on a slow drive: `STRATA_READ_AHEAD=0` turns it off, `STRATA_FILL_AHEAD=N` sets how many fill pairs +are requested ahead (default 256). + +The prompt chunk follows the host's memory (Linux, file tier, on by default). When the page cache is smaller than the file +set, a planner picks the chunk and ring with the fewest predicted bytes read per prompt from the RAM copy, the page cache that +is left, the drive class and the PCIe link. It prints `chunk 8192 -> 3840, ring 96 -> 96: ...` when it changes the pick and +`chunk 8192 kept: ...` when it does not. It does nothing when the GPU holds every expert, when the file set fits the cache, +and on Windows. `STRATA_PREFILL_HOST_PLAN=0` turns it off; `STRATA_SSD_GBPS` overrides the drive speed; +`--prefill N` and `STRATA_PREFILL_RING` always win. + +### Reading the tiers + +With `--stats` the engine prints the tiers of the decode (`expert tiers`: blobs from the RAM copy, blobs and MB from the +files, read time; `routing prefetch`: how many file reads had been warmed). The server log has the same per request, and +`GET /metrics` lists `ram_blobs`, `file_blobs`, `file_mb`, `drafts_offered` and `drafts_accepted` per recent request, with +`totals` since the start. `hit_rate` is the VRAM share of the experts looked up; experts the GPU reads over PCIe +(`--pcie-frac`) are not in it, so a higher `--pcie-frac` raises it even when decoding gets slower. `pcie_share` is their +share of all routed experts. The Monitor tab shows both. + +Profiling: start the server with `STRATA_DECODE_TIMING=1` (and `STRATA_VERIFY_PROFILE=1` for the GPU side). After each request +the engine log has a `strata decode timing:` line and a `strata decode GPU stages (ms/window):` line. The GPU profile slows +decoding a little: use it to compare, not to measure speed. `STRATA_LOOKAHEAD_STATS=1` scores the next-layer expert +prediction every 100 windows and costs CPU. `tools/replay_cache_policy.py` replays a `--dump-routing` trace through the +adaptive tier's swap policy offline. + +### Draft layer + +| Option | Effect | +| --- | --- | +| `--mtp` | Use the model's MTP draft layer (setup passes it). The server also runs without it: drafts then come from the suffix and prompt-lookup drafter only, every token is still verified, and the draft layer's ~0.7-1 GiB of VRAM goes to the expert cache. | +| `--spec N`, `--spec-min-p P` | Draft depth and the confidence needed to add a guess. Setup's default is `--spec 4 --spec-min-p 0.5`. | +| `--draft-vocab en\|cyrillic\|fr` (setup) | The draft layer can only propose tokens from a subset of the vocabulary. The default subset includes every Chinese, Japanese and Korean token. `en` keeps the English and code subset (~110 MiB less VRAM, English 1-2% faster, CJK gets almost no drafts). `cyrillic` and `fr` add those scripts. `tools/draft_vocab.py` builds and inspects subsets. | +| `--lookup-chain-min M`, `--mtp-hnorm stream`, `--mtp-draft-vocab FILE` | Finer controls: the shortest context match `--lookup-chain` extends (default 3), one norm per stream, and a token subset of your own. | + +When the start stops with "the draft head does not fit" (a 12 GB card with a long context), the engine says how much it needs +and which smaller subset fits. Setup suggests `--draft-vocab en` on cards under 14 GB; nothing changes unless you pass it. + +**Reproducible output.** With the IQ models the CPU rounds one-token expert groups and multi-token groups differently, so the +same prompt at temperature 0 can end in a different, equally good answer when drafting, cache state or a resumed conversation +differ. `STRATA_IQ_MT_MIN=1` uses the multi-token kernels for every group, so the answer no longer depends on drafting (IQ3_S +decode -1 to -3% on a Ryzen 7600). For byte-identical repeats also add `--prompt-cache 0 --adapt-swaps 0 --pcie-frac 0` to +the engine args. `--pcie-frac 0` costs decode speed, so keep it for A/B runs. --- ## Sharing the GPU with other programs (optional) -By default the model stays loaded until you close Strata. On a PC that also games, renders or runs another model -server, three server options (all off by default; also as keys in `strata-.json`) give the VRAM back: +By default the model stays loaded until you close Strata. Three server options, all off by default and also keys in +`strata-.json`, give the VRAM back: | Option | Config key | What it does | | --- | --- | --- | -| `--idle-unload 600` | `"idle_unload_s": 600` | unload the model after 600 s without requests; the next request loads it again | -| `--min-free-vram-mib 11000` | `"min_free_vram_mib": 11000` | load an unloaded model only when that much VRAM is free (it waits up to 15 s for memory being given back), else answer **503** "the GPU is in use by another program" instead of starting into what a game left (with several GPUs it checks the first one) | -| `--before-load "cmd"` | `"before_load": "cmd"` or `["cmd", "arg"]` | a command run before the model is loaded again, e.g. one that unloads another server's model | -| | `"before_load_timeout_s": 120` | how long that command may run (default 120 s). When it exits non-zero or times out (a peer that is not running makes `curl` exit 7), the free VRAM decides: with `min_free_vram_mib` set and not enough free, the model is **not** loaded and the request gets **503** with `Retry-After`; with room (or no limit set) it loads, with a warning in the log. `"before_load_strict": true` always refuses on a failure; `"before_load_ignore_failure": true` always loads | +| `--idle-unload 600` | `"idle_unload_s": 600` | Unload the model after 600 s without requests. The next request loads it again. | +| `--min-free-vram-mib 11000` | `"min_free_vram_mib": 11000` | Load an unloaded model only when that much VRAM is free (it waits up to 15 s for memory being given back). Otherwise answer **503** "the GPU is in use by another program". With several GPUs it checks the first one. | +| `--before-load "cmd"` | `"before_load": "cmd"` or `["cmd", "arg"]` | A command run before the model is loaded again, for example one that unloads another server's model. | +| | `"before_load_timeout_s": 120` | How long that command may run. When it fails or times out, the free VRAM decides: with `min_free_vram_mib` set and not enough free, the request gets **503** with `Retry-After`; otherwise the model loads with a warning. `"before_load_strict": true` always refuses on a failure; `"before_load_ignore_failure": true` always loads. | -`POST /unload` unloads it now (`409` while a request is running; `POST /unload?wait=1` waits for the running turn first, up to 300 s, or `?wait=`, so a second server can really free the card) and `POST /load` loads it ahead of a request (both -with `Content-Type: application/json`, e.g. `curl -X POST -H "Content-Type: application/json" localhost:8080/unload`); -`/health` says `"loaded"`, `/v1/models` lists it as `unloaded` (like llama.cpp's router), `/props` sets -`is_sleeping` and the Monitor shows the state. Unloading ends the engine process - and the image encoder, when images -are on; it is started again first, as at a start - so their VRAM and RAM go straight back. The model files stay in -the OS file cache, so loading again takes seconds while that RAM is not needed elsewhere. Measured on an RTX 5060 Ti -16 GB with Q2_0 in the low-RAM mode: unloading takes ~0.3 s, and a request to an unloaded model answered after -4.6 s (text) or 14.7 s (a picture, image encoder on the CPU). +`POST /unload` unloads now (`409` while a request is running; `?wait=1` waits for the running turn, up to 300 s, or +`?wait=`). `POST /load` loads ahead of a request. Both need `Content-Type: application/json`: -**Giving part of the VRAM back while it keeps serving (#533, opt-in, one NVIDIA GPU; every request and answer: -[VRAM_ELASTIC.md](VRAM_ELASTIC.md)).** With `"vram_elastic": true` -in the config (the engine flag `--vram-elastic`), the expert cache is allocated in 512 MiB segments -(`"vram_segment_mib"`), and `POST /v1/vram` with `{"reserve_mib": 8000}` shrinks it between requests until that much -VRAM is free for another program; `{"reserve_mib": null}` grows it back towards its full size, keeping the reserve -the engine started with (`--vram-reserve-mib`), and `{"reserve_mib": 0}` takes all of it back. A request that is -running finishes first. The experts of the segments given back are computed on the CPU, like any expert outside the -cache, so answers keep coming, slower; growing back puts the same experts in the same slots. Nothing resizes on its -own. Not with a layer split, the helper caches, `--peer-device` or the resident low-RAM mode. Measured on an RTX 5070 -12 GB with Q2_0 (4.8 GiB cache): `{"reserve_mib": 6000}` took 78 ms and freed 4.3 GiB (the cache keeps 0.5 GiB for -the prompt path), decode 44 -> 33 tok/s; growing back took 92 ms and the answers were token for token the ones before -the shrink. Without the flag nothing changes (the same answers as without it). +```sh +curl -X POST -H "Content-Type: application/json" localhost:8080/unload +``` -**Keep what the expert cache learned across restarts (opt-in, engine 0.1.36, #477):** a start fills the GPU's expert -cache from the shipped profile, and the adaptive tier (`--adapt-every`) then moves in the experts your requests use. -With `"expert_profile_save": "expert-profile-learned.bin"` in `strata-.json` the engine saves that as a -profile - the experts in VRAM first, then the routing it counted since the start, then the shipped order - on a -clean exit and every 10 minutes between requests (`"expert_profile_save_every": 5` for another interval, `0` for -exit only), written to a temporary file and renamed, so a crash never leaves half a file. The next start begins from -it instead of the config's `--expert-profile` when it is a profile of the same model (else from the config's, as -before). A relative path is in the Strata folder; one file per model, and a profile per project works the same way -(point the key at another file). The file is a fingerprint of what you used the model for: it stays on your PC. -Without the key nothing is counted or written. Setup rewrites the config when run again: add the key again then. -On an Intel Arc card with `STRATA_VERIFY_NO_HOST` (the SYCL default on Battlemage, no adaptive tier) the profile is ranked from the device's own routing counts and applies at the next start; a profile learned from one kind of text speeds that kind up (code, +13-19% decode on the Arc Pro B70, same tokens) and slows other kinds (prose, -18%), so keep one file per kind of work. +`/health` says `"loaded"`, `/v1/models` lists the model as `unloaded` while it is, `/props` sets `is_sleeping` and the +Monitor shows the state. Unloading ends the engine process and the image encoder, so their VRAM and RAM return at once. The +model files stay in the OS file cache, so loading again takes seconds while that RAM is not needed elsewhere. -**The expert cache size is a budget, and on Windows an over-sized one pages instead of failing.** `--expert-cache N` -(`"expert_cache": N`) is `N` times the largest expert blob, and the engine compares it only with the free VRAM it -reads **before** the slots are written. `--expert-cache auto` (the default) sizes from that same figure but **checks -again once the slots are actually written** and shrinks if they do not fit; that second check is auto-only (the loop -breaks on `!auto_cache`). Under WDDM an allocation is not resident until it is touched, and the free figure read -before it can be about a gigabyte too high, and the driver's default sysmem fallback puts an over-committed -allocation in system memory instead of failing - so the start looks normal, the banner still reports the slot count, -and only the speed collapses. The startup line is the tell: `... MiB of VRAM free with everything loaded`, and below -about 256 MiB it adds `LOW: requests may stall; add --vram-reserve-mib N` with the value that would have left room. +**Lazy start.** `serve/server.py --engine strata --config strata-.json --lazy` (or `"lazy_load": true`) starts the HTTP +API without the engine or the vision encoder. The first generation request or `POST /load` loads them, including +`before_load` and `min_free_vram_mib`. `POST /v1/load` and `/v1/unload` are JSON control aliases (they need the API key, +`application/json` and no foreign browser Origin, and return **409** while a request is active or queued). `/api/health` +aliases `/health`, and `/v1/status` exposes `loaded` and `auto_load`. -Measured on an RTX 4080 SUPER 32 GB, IQ3_S, engine 0.1.39, `--max-context 1048576` (yarn factor 4), fresh -43,969 / 45,670 / 47,956-token prompts with 200 generated each, one engine start per row (the rest of the -configuration is the one in `bench/results/2026-10-04-community-rtx-4080s-iq3s`): +**Giving part of the VRAM back while it keeps serving** (opt-in, one NVIDIA GPU, [VRAM_ELASTIC.md](VRAM_ELASTIC.md)). With +`"vram_elastic": true` (engine flag `--vram-elastic`) the expert cache is allocated in 512 MiB segments +(`"vram_segment_mib"`). `POST /v1/vram` with `{"reserve_mib": 8000}` shrinks it between requests until that much VRAM is free; +`{"reserve_mib": null}` grows it back toward its full size, and `{"reserve_mib": 0}` takes all of it back. The experts of the +segments given back are computed on the CPU, so answers keep coming, slower. Nothing resizes on its own. Not with a layer +split, the helper caches, `--peer-device` or the resident low-RAM mode. -| expert cache | slots | VRAM free at start | decode tok/s | -| --- | ---: | --- | --- | -| `--expert-cache 8900` (11,631 slots, 22.07 GiB) | 11,631 | 0 MiB (`LOW`, suggests `--vram-reserve-mib 1212`) | 13.7 / 14.7 | -| `auto` | 11,178 | 217 MiB (`LOW`) | 102.1 / 122.3 | -| `auto` with `--vram-reserve-mib 1500` | 10,766 | 1,075 MiB | 88.9 / 96.5 / 101.4 | -| `--expert-cache 7000` | 9,148 | 4,130 MiB | 94.1 / 97.3 | -| the same 11,631-slot cache at `--max-context 524288` | 11,631 | 299 MiB | 113.2 / 107.9 | +**Keep what the expert cache learned across restarts** (opt-in). `"expert_profile_save": "expert-profile-learned.bin"` saves +the experts in VRAM, then the counted routing, then the shipped order, on a clean exit and every 10 minutes between +requests (`"expert_profile_save_every": 5` for another interval, `0` for exit only). The next start begins from it when it +is a profile of the same model. A relative path is in the Strata folder. The file is a fingerprint of what you used the model +for and stays on your PC. A profile learned from one kind of text speeds that kind up and can slow others, so keep one file +per kind of work. Setup rewrites the config when run again, so add the key again then. -453 slots (0.86 GiB) separate 13.7 from 102 tok/s, and the slower run had the **higher** cache hit rate (91.4 / 94.7% -against 90.4 / 93.9%), so this is not misses: being at 0 MiB free is what costs 7x. Two ways to stay out of it: leave -the cache on `auto`, or keep an explicit size and raise `--vram-reserve-mib`, which the engine deducts **before** it -sizes the cache (700 -> 1000 -> 1500 took the free figure from 217 to 575 to 1,075 MiB and cost about 400 slots). +### The expert cache size is a budget -**Checking it on Windows.** `\GPU Process Memory()\Shared Usage` and `Dedicated Usage` (Performance Monitor, or -`Get-Counter`) look like the right instrument, but on this machine they did **not** separate a 13.7 tok/s run from a -102 tok/s one: `Shared` read about 55.5 GB at 262/524K and 62.4 GB at 1M in every run, because it counts the -deliberately pinned memory (the expert arena, 50.3 GB here, plus the pinned K/V, 6.19 -> 12.38 GiB at 1M). The free -line in the log is the reliable tell, and on NVIDIA hardware `CUDA - Sysmem Fallback Policy` set to -`Prefer No Sysmem Fallback` for the engine's executable is the other way to make an over-sized cache fail instead of -paging (not measured here). Reported with the measurements in -[issue #781](https://github.com/Niko1221/Strata/issues/781). +`--expert-cache N` (`"expert_cache": N`) is `N` times the largest expert blob. The engine compares it only with the free VRAM +it reads before the slots are written. `--expert-cache auto` (the default) checks again once the slots are written and +shrinks if they do not fit. Under Windows an over-sized fixed cache does not fail. The driver pages the excess to system +memory, the start looks normal, and only the speed collapses (see the history table for one measurement). -The same shape appears on a second, smaller card. On a 16 GB RTX 5080 at 786,432, UD-IQ4_XS, INT8 KV, with -`Prefer No Sysmem Fallback` set and an explicit `--expert-cache 4000`, taking `--vram-reserve-mib` down from 770 to 0 -grew the cache from 2,782 to 3,115 slots (768 MiB) while the card's used memory moved 129 MiB, the hit rate stayed -flat (59.9% to 63.0%) and decode fell from 40.9 to 16.6 tok/s. Every step there reached READY with 0 MiB free, so -with the fallback off it still starts and degrades instead of failing - the slack left for the rest of the card is -worth more than the extra slots, which is the same ordering measured above (enkynakamura, #781; one 256-token sample -per step, so the upper rows are noisy). - -**The free figure moves after startup.** The `... MiB of VRAM free with everything loaded` line is read before the -verify windows are captured (each captured window, and each `--batch` slot layout, takes some tens of MiB: 538 MiB at -that line was 395 MiB after the last capture in #2065, an RTX 4090 D with `--batch 2` at 262,144 context). Below about -a gigabyte on Windows the driver can page GPU memory to system RAM instead of failing: decode fell from 91 to 3 tok/s -at 70K tokens and then a window never finished, the watchdog stopping the engine after 60 s. From 0.1.43.1 each request -re-reads the free VRAM, says `WARNING: only N MiB of VRAM free at the start of this request` once it is under 1 GiB -(512 MiB elsewhere) with the `--vram-reserve-mib` that would leave room, and the stall report prints the figure. The -engine only says it; the fix is a larger `--vram-reserve-mib`, a smaller `--max-context` or `--batch`, or -`Prefer No Sysmem Fallback` for the engine. - -**A stall named "request" can be the OS paging the engine's own memory.** Before a request that switches -conversation, the engine parks the outgoing one in RAM: copies of its running checkpoints and K/V, which then sit in -ordinary memory until the conversation comes back. On a 64 GB PC with `--resident-budget-gib 40`, an 8 GiB conversation -cache and a 256K context, the working set was 8-30 GB of 56 GB committed and the engine took about 190 page faults -a second (under 1 MB/s if each is a pagefile read) in #2049: one silent step of more than 60 s, five times, on 0.1.41 -and 0.1.43 alike (thread dumps: the main thread inside the copy of the checkpoints, at the same instruction each time). The log's `verify window (last window, not the -current stage) ... host at layer step 48; the GPU rang 48` line in that report is the previous request's last window, -not the stalled one. From 0.1.43.1 the park is named in the stage (`request: parking the previous conversation`), its -copies report progress to the watchdog, and Windows counts page faults at 8 a unit instead of 64 when it decides -whether the file tier is busy. The cure for the slowness itself is memory: a smaller `--resident-budget-gib`, a smaller -`--conversation-cache-mib`, or the pagefile on an SSD. +- The startup line `... MiB of VRAM free with everything loaded` is the tell. Below about 256 MiB it adds + `LOW: requests may stall; add --vram-reserve-mib N` with the value that would have left room. +- Keep the cache on `auto`, or keep an explicit size and raise `--vram-reserve-mib`, which the engine deducts before it sizes + the cache. +- That figure is read before the verify windows are captured, so it moves down by tens of MiB per captured window or `--batch` + layout. Each request re-reads the free VRAM and prints `WARNING: only N MiB of VRAM free at the start of this request` below + 1 GiB on Windows (512 MiB elsewhere), with the `--vram-reserve-mib` that would leave room. The fix is a larger + `--vram-reserve-mib`, a smaller `--max-context` or `--batch`, or the NVIDIA setting `CUDA - Sysmem Fallback Policy` set to + `Prefer No Sysmem Fallback` for the engine's executable. +- Performance Monitor's `GPU Process Memory` counters did not separate fast from slow runs because they count deliberately + pinned memory. Use the free line in the log. +- A stall named "request" can be the OS paging the engine's own memory while it parks the outgoing conversation (the stage + reads `request: parking the previous conversation`). The cure is memory: a smaller `--resident-budget-gib`, a smaller + `--conversation-cache-mib`, or the pagefile on an SSD. --- ## Using it -The server listens on `http://127.0.0.1:8080` (change with `--port` in setup, or edit the run script). +The server listens on `http://127.0.0.1:8080` (`--port` in setup, or edit the run script). | API | Endpoint | | --- | --- | | OpenAI Chat Completions (stream and non-stream, tools) | `POST /v1/chat/completions` | | Anthropic Messages (stream and non-stream, tools) | `POST /v1/messages` | | OpenAI Responses (stream and non-stream, tools; stateless, [below](#the-responses-api-and-codex-cli)) | `POST /v1/responses` | -| Model list / health | `GET /v1/models`, `GET /models`, `GET /health` | +| Model list and health | `GET /v1/models`, `GET /models`, `GET /health` | | Model properties | `GET /props` (also accepts `?model=`) | -| What the model is doing right now | `GET /status`, `GET /slots` (busy or idle, with `n_prompt_tokens`; one entry per slot with `--batch`) | -| Save / restore the conversation to a file (session files, below) | `POST /slots/0?action=save\|restore` | -| Everything the Monitor tab shows (engine, live state, last requests, hardware) | `GET /metrics` | -| The same for Prometheus, with vLLM's metric names (asked with `Accept: text/plain` or `?format=prometheus`) | `GET /metrics` | -| Request totals and what is running in Prometheus' text format, under llama-server's metric names | `GET /metrics/prometheus` | +| What the model is doing now | `GET /status`, `GET /slots` (busy or idle, with `n_prompt_tokens`; one entry per slot with `--batch`) | +| Save or restore the conversation to a file | `POST /slots/0?action=save\|restore` ([session files](#session-files-disk)) | +| Everything the Monitor tab shows | `GET /metrics` (JSON) | +| The same for Prometheus with vLLM's metric names | `GET /metrics` with `Accept: text/plain` or `?format=prometheus` | +| Request totals in llama-server's metric names | `GET /metrics/prometheus` | | The MCP servers, their state and tools ([below](#tools-from-mcp-servers)) | `GET /mcp` | - -`GET /metrics` answers a Prometheus scrape (`Accept: text/plain` or `application/openmetrics-text`) in the text -format with vLLM's names - `vllm:num_requests_running` / `_waiting`, `vllm:kv_cache_usage_perc`, the token and -request counters, `vllm:prefix_cache_queries_total` / `_hits_total` (prompt tokens read / reused), -`vllm:spec_decode_num_draft_tokens_total` / `_accepted_tokens_total` (the MTP drafts), and the histograms -`vllm:time_to_first_token_seconds`, `vllm:inter_token_latency_seconds` and `vllm:e2e_request_latency_seconds` - so the -dashboards and alerts written for a vLLM server read this one. With an API key, the scraper sends it as a bearer -token. Any other request keeps the JSON. - -The JSON's own facts that vLLM has no name for come in the same scrape under `strata:`, named after their JSON key, -so a Monitor-tab panel and a Grafana panel read the same value: `strata:live_state{state="..."}`, `strata:live_tok_s`, -`strata:live_prefill_tok_s_mean`, `strata:live_prompt_read`, `strata:engine_max_context`, -`strata:totals_prompt_seconds_total` / `strata:totals_decode_seconds_total`, `strata:last_hit_rate` and -`strata:last_decode_tok_s` (the last request's), and the hardware - `strata:gpu_util`, `strata:gpu_mem_used_bytes`, -`strata:gpu_mem_total_bytes`, `strata:gpu_temp_celsius`, `strata:gpu_power_watts` (one sample per card, label -`gpu`), `strata:cpu`, `strata:ram_used_bytes`, `strata:ram_total_bytes`. A value the server does not have (no GPU -telemetry, an older engine) has no sample rather than a zero. - -Ready to use, in `docs/monitoring/`: `prometheus.yml` (a scrape config, the API key as a bearer token), -`servicemonitor.yaml` (the same for the Prometheus Operator) and `grafana-strata.json`, a Grafana dashboard to -import (it asks for the Prometheus data source): requests running and waiting, tokens per second, time to first -token, time between tokens and request duration (p50 / p95), the prompt cache and MTP draft rates, and the engine's -own panels (state, batch slots, rates, expert cache hit rate, GPU and RAM). Its first row uses only vLLM's names, so -it also reads a vLLM server. - -`/models` and `/v1/models` list only the loaded model, with its context limit and input modalities. `/props` exposes the original chat template, context limit, configured generation defaults (shared settings take precedence), model path and engine version when available. Context means the full engine context, not the resident KV window. `n_predict: -1` means no fixed output cap. Unconfigured sampling fields are omitted. `/slots` uses llama.cpp's names: `n_ctx` and `n_prompt_tokens` (the running request's prompt size, kept after it ends: what a front-end's context meter divides by `n_ctx`). `total_slots` in `/props` is the number of `--batch` slots (1 without). `autoload` has no effect; an unknown `model` returns 404. These metadata endpoints and `/slots` require the API key when one is configured. They do not load, unload or restart models. - -`chat_template_caps` uses llama.cpp's field names. Strata checks them when loading the active template by rendering -small requests; a rejected or omitted feature is `false`, and a template that fails a probe in any way only turns that -feature off (the server still starts). `supports_tools` checks tool definitions and XML call -instructions; `supports_tool_calls` checks that calls in the history use Strata's XML format and keep tool results. -`supports_parallel_tool_calls` checks multiple calls in one assistant turn, and `supports_system_role` checks a -leading system message. `supports_preserve_reasoning` means older assistant turns keep their `reasoning_content` -in the rendered prompt. It does not describe whether a new reply returns reasoning. These are compatibility hints -for clients, not a guarantee that the model will follow a request. Zed's llama.cpp provider reads this object to -offer tools. - -```bash -curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ - "model": "strata", "messages": [{"role": "user", "content": "Write a haiku about GPUs."}], "max_tokens": 512 }' -``` +| API request monitor ([below](#api-request-monitor)) | `GET /api-monitor`, `GET /api/requests` | ```python from openai import OpenAI @@ -887,147 +413,166 @@ r = client.chat.completions.create(model="strata", messages=[{"role": "user", "c print(r.choices[0].message.content) ``` -- **Thinking levels: none, low, medium, high.** The model thinks before it answers (streamed as - `reasoning_content`, Anthropic: `thinking` blocks). Choose how much per request - in the chat page (the "Thinking" - menu), in `chat.py` (`/think low`), or over the API: +- **Prometheus.** With `Accept: text/plain` the metrics use vLLM's names (`vllm:num_requests_running`, `vllm:kv_cache_usage_perc`, + token and request counters, `vllm:prefix_cache_queries_total` and `_hits_total`, `vllm:spec_decode_num_draft_tokens_total` + and `_accepted_tokens_total`, and the time-to-first-token, inter-token and end-to-end histograms), so dashboards written + for vLLM read this server. Strata's own facts come under `strata:` and are named after their JSON key (for example + `strata:live_tok_s`, `strata:last_hit_rate`, `strata:gpu_util`, `strata:ram_used_bytes`). A value the server does not have + has no sample. With an API key, the scraper sends it as a bearer token. Ready-made files are in `docs/monitoring/`: + `prometheus.yml`, `servicemonitor.yaml` and `grafana-strata.json`. +- **Metadata.** `/models` and `/v1/models` list only the loaded model, with its context limit and input modalities. `/props` exposes + the chat template, context limit, configured generation defaults, model path and engine version. `n_predict: -1` means no + fixed output cap. `/slots` uses llama.cpp's names (`n_ctx`, `n_prompt_tokens`). `total_slots` is the number of `--batch` slots + (1 without). These endpoints require the API key when one is configured and never load, unload or restart models. +- **`chat_template_caps`** (llama.cpp's field names) is checked by rendering small requests when the template loads. A rejected + feature is `false`. `supports_tools`, `supports_tool_calls`, `supports_parallel_tool_calls`, `supports_system_role` and + `supports_preserve_reasoning` are hints for clients, not a promise that the model follows a request. Zed's llama.cpp provider + reads this object to offer tools. - | API | how | - | --- | --- | - | OpenAI | `"reasoning_effort": "none" \| "low" \| "medium" \| "high"` (also `"reasoning": {"effort": ...}`, or `"chat_template_kwargs": {"enable_thinking": false}`) | - | Anthropic | `"output_config": {"effort": "low" \| "medium" \| "high"}`, `"thinking": {"type": "disabled"}`, or `"thinking": {"type": "enabled", "budget_tokens": N}` (under 2K = low, under 8K = medium, more = high) | +### Thinking levels: none, low, medium, high - Without a setting the model uses its own default, **high**. `none` answers at once (fastest); `low` keeps the thinking - short. The levels are instructions the model was trained with, not a hard token limit: on easy questions all three - think briefly, on hard ones `high` thinks longest and is most accurate. - To start with another level for every request that names none, put `"reasoning_effort": "low"` (or `none`, `medium`, - `high`) in `strata-.json`, outside `"args"` (#1641): the engine has no such option, and `--reasoning-effort` in - `"args"` is moved to this key with a note at startup. The web app's shared Chat settings ("Use for other apps too") and a - request's own level win over it. -- **Keep Windows awake while it works (opt-in, #1727).** `"prevent_sleep": true` in `strata-.json` keeps the PC - from going to sleep while at least one request is running or waiting (Windows' `SetThreadExecutionState` with - `ES_SYSTEM_REQUIRED`; the screen may still turn off), and gives the normal sleep rules back when the server is idle - and when it exits. Default off. On Linux and macOS it does nothing (one line at startup says so). -- **A hard thinking budget (opt-in).** `"reasoning_budget_tokens": N` (or llama.cpp's `"thinking_budget_tokens": N`, which - clients written for llama-server send) in a request (OpenAI or Anthropic) caps the - thinking at N tokens: when it gets there the server ends it with a short wrap-up line and ``, and the model - answers from there (the engine continues from what it already holds, so nothing is read again). The wrap-up is - part of the thinking the client sees and counts as output tokens. `"reasoning_budget_tokens": N` in - `strata-.json` sets it for every request; a request's own value wins, and `0` means no budget. Off by default; - Anthropic's `"thinking": {"budget_tokens": N}` chooses the level, as above, and since 0.1.45 is also the budget (a - `reasoning_budget_tokens` in the same request wins). -- **max_tokens counts the thinking, so the answer keeps its room (0.1.45).** A client that caps the whole reply (the - desktop agent's harness sends `max_completion_tokens: 8192` on every request) and a task the model thinks long about - used to be cut off in the middle of the thinking, with no answer. Now a request with thinking on, no budget of its - own and `max_tokens` of 1,024 or more gets one: the thinking is wrapped up when only a quarter of the cap (at least - 512 tokens) is left, and the model answers in that room (8,192: thinking up to 6,144, answer 2,048). Requests without - `max_tokens` (the rest of the context) get a budget of three quarters of it. `"answer_room": false` in - `strata-.json` turns it off. Qwen recommends up to 262,144 tokens of reasoning and 131,072 of answer: a client - should not cap the reply far below what the context allows. -- **A reply that ends inside its thinking (#1053, opt-in).** Some turns write a sentence of reasoning and then the - end-of-turn token with no ``: the content is empty and an agent stops. `"reasoning_close_retry": true` in - `strata-.json` closes the thinking once (as the thinking budget does) and continues, once per request, only for - a reply that ended that way with no answer and no tool call. Off by default. -- **A reply stuck on one token is ended (0.1.39, #606).** When a reply repeats the same token 256 times in a row, the - server ends it there with `finish_reason` `"length"` and says so in its window: a model in a loop, or a broken - state that answers one token forever (#606 saw 36,689 tokens of `!`). `"repeat_stop_tokens": N` in - `strata-.json` sets the run length; `0` turns it off (for a request that really wants one token many times). -- **Non-finite logits end the request and drop the cached state (#879, on by default).** Rarely, a window's logits - become NaN and the reply degenerates into token 0 (`!`) repeated (#606); the poison sits in the conversation's K/V - and recurrent state, so with the prompt cache every later turn of that conversation stays poisoned. The q8_1 clamps - of 0.1.39 and later remove the overflows found so far; this guard catches whatever cause is left. Every verify window - of `--serve` (the decode windows, the `--pipeline-windows` decode windows of a layer split, the prompt's short - tail, batch slots) checks its head logits on the GPU beside the greedy pick (one small kernel into mapped flags, read - after the sync the window already does). A flagged window emits none of its tokens, the engine prints one - `non-finite logits (#879)` line with the position, ends the request with `DONE ... nonfinite` (a slot: - `BDONE nonfinite`), and the next request first drops the live session, every checkpoint, every parked - conversation and every idle slot (a slot still running finishes its reply and is not kept after it) and reads its - prompt from token 0. The server retries a request that had sent nothing yet once, transparently; a reply that had - started ends as `"length"` with a line in the server window; a second fire on the retry ends the request with an - error (503 before an answer, an error event in a stream). When the flagged window was one that reads the prompt's - last few tokens (the part after the last turn boundary, read through the verify windows, `--short-read`), the retry - reads its whole prompt through the batched path instead (#3465: a reporter's prompts failed in that tail every time and - passed with `--short-read 0`); the engine prints a line saying so. With replicas, the server forgets what that replica held. - `repeat_stop_tokens` stays as the backstop. The kernel writes only its flags and the host only reads them, so an - answer without a NaN is the one the engine gives with `STRATA_NAN_GUARD=0`, which records no kernel: on an RTX - 3080 with the RVN IQ3_S config at fixed expert sets (a chat replay of 8 turns on a 16K-token system prompt), the - guard on and off gave byte-identical answers in 3 pairs (one of them with `STRATA_ROUTE_TAIL_SKIP=0`, one with - `STRATA_SH_STREAM=0`); one more guard-off run differed from all the others from its first answer on, while a second - guard-off run matched the guard-on runs, so that was a difference between runs, not the guard. Its cost there: 4.8 us of GPU time a window on average (7.8 us at most), 0.009% of the decode - time (Nsight Systems, 424 windows). The one-shot command line (`strata` without `--serve`) and the Intel (SYCL) port - in `sycl/` do not check. - Debug switch: `STRATA_NAN_INJECT=[,...]` reports the n-th window the guard checks in the engine process - (counted from 1 over prompt-tail, decode and batch windows) as non-finite, to exercise the whole path without a real - fire. With `=1,60` on that 3080: a new chat's first window was flagged and the server sent the request again from - token 0 (the client saw one normal answer); the 60th, mid-reply in the next turn, ended that reply as `"length"` - after 58 tokens; the turn after it reused 0 cached tokens and answered normally. -- **Repeated reasoning (opt-in, #728).** The single-token guard above does not see a model that repeats whole - passages. `"reasoning_loop_recovery"` in `strata-.json` is `false` (the default), `"stop"` or `"recover"` - (`true` means `"recover"`). Every 512 output tokens, at a complete character and parser boundary, the reasoning is - measured over its last 2,000 words and punctuation marks (counting passages over the last 30,000 words), and its - last 2,048 characters are checked for a period of at most 64 characters (#1753: a loop written as one long word, - such as a 24k-digit string cycling an 11-digit pattern, is a single word to the count and never repeats one - token); a periodic tail counts as fully repeated. If at - least 25% belong to 12-word passages seen three times, `"stop"` ends the reply there as `"length"` and says so in - the server window. `"recover"` stops and drains that generation, then goes on once from all its generated token - ids with the template's low-effort sentence in place of the xhigh one in the first system message (a splice of - token ids: the rest of the prompt is not decoded or re-encoded). The task stays the same; no answer or `` - is inserted, and both passes share the original output limit. On recovery only, temperature is raised to at least - 1.0 and presence penalty to at least 1.5; a client's top-p, top-k and seed are never changed. `/metrics` records - `reasoning_recoveries` and the coverage. This is a policy, not a numerical engine fix or a guarantee of an answer. - `"recover"` needs the exact xhigh sentence in the first system message and skips requests with images; re-reading - the changed prefix costs prompt time. Either mode can mistake repeated useful code or checks for a loop, so keep - the recorded answer quality alongside the completion rate when testing it. -- **Changing the effort without re-reading the prompt (opt-in, 0.1.39, #458).** The effort's instruction is the - first thing in the prompt, so a request that only changes the effort (an agent's "think harder" switch, `none` for - a quick tool step) reads the whole conversation again. `"effort_position": "end"` in `strata-.json` renders - every request's prompt start as the default effort's and puts a `low` / `medium` effort in a short system turn - right before the answer (no thinking: the empty thinking block, as always); the engine (0.1.39+, which the server - checks) keeps its checkpoint in front of that turn, so the next request reuses the conversation whatever its - effort. The default (`"start"`) prompt is unchanged. The model sees a level that is not the default in another - place than it was trained with; how well it follows it there is not measured yet. -- **Anthropic requests that don't ask for thinking (opt-in, 0.1.32, #278).** By default a `/v1/messages` request - with no `"thinking"`, effort or budget thinks as the model's template does. `"anthropic_thinking": "on_request"` in - `strata-.json` renders such a request without thinking - Anthropic's own rule, and what Claude Code's short - helper calls (a session title in a few dozen tokens) need; its real turns ask for thinking when it is on there. -- **Streaming.** With `"stream": true` everything arrives as it is made: the thinking, the answer, and tool calls - (the tool's name first, then its arguments piece by piece, like OpenAI and Anthropic do). While the model reads a - long prompt the stream sends keep-alives, so agents do not time out; the server window prints progress every - 15 s, and `GET /status` says what it is doing (`reading the prompt`, `answering`, tokens so far). Closing the - connection or pressing stop in your app really stops the model, so the next request starts at once. -- **Which tool calls come back as calls.** A call counts only when it names a tool the request declared. A call for - another name, or any call in a Chat Completions or Anthropic request that declared no tools, is returned as the - text it was written as, so a client never runs a tool it did not offer (the Responses route keeps its own rule - for Codex's compaction requests). A call whose envelope drifted by a token, `<=NAME>`, `` or `` - instead of ``, is read as the call when NAME is a declared tool. A `` block that still - does not read as a call is counted in the server window. -- **Tool calls in other forms (opt-in).** `"tool_call_recovery": true` in `strata-.json` reads a call of a - tool the request declared also when the model writes it as `` instead of ``, as JSON - (`{"name": ..., "arguments": ...}`) inside ``, as a `` block at the start of a line - without `` (outside code), or as a second call inside the same `` (without the switch the - second call's parameters merge into the first). Anything else in those forms stays the text it is. Measured on - 1,462 agent turns (Qwen3.8 under Claude Code, from signalnine/q27's drift corpus): 98.3% read as intended with it, - 92.9% without. Off by default, so what a client gets back is unchanged unless you turn it on. -- **opencode's environment block in the user turn (opt-in).** opencode writes an `` block (working directory, - today's date) near the start of its system message, ahead of ~40K characters of instructions and tool definitions that - are the same in every session. The prefix cache only reuses a prompt's start, so every new folder and every new day - read ~14K tokens again. `"env_to_user": true` moves that block to the front of the first user message: the system - turn is then the same for every session and the disk prefix cache restores it. Measured (RTX 5070 Ti Laptop, Swift 1.5 - IQ3_XXS, a new opencode session in a second folder): first request 14,073 tokens with 13,693 from the cache, 1.1 s, - against 14,753 read, 11.9 s, without it; the model still answered the folder and date correctly (3/3). Off by default. -- **Prefill progress in the stream (opt-in).** `"return_progress": true` puts that progress on the stream instead of - sending only the keep-alive, as one extra field on a chunk with an empty delta: `prompt_progress` with `total`, - `cache`, `processed` and `time_ms`. Those are llama.cpp's four fields and mean the same there (`time_ms` is the time - since the prompt started reading, and the work still to do is `(total-cache) - (processed-cache)`), so a client that - draws a prefill bar for llama.cpp draws one here too. It is off unless the request asks, as it is in llama.cpp. - The engine says one line per `--prefill` chunk, so that chunk is the step: measured on two RTX 3090s with - `--prefill auto` (8192 tokens), a 42,131 token prompt read in 15 s sent six of them. A prompt shorter than one chunk - sends nothing, and that is on purpose: its only line arrives once the prompt is read, because the last tokens go - through the verify windows rather than the batched path, so it stops up to `--short-read` tokens short of the end. -- **Chat apps.** Any app with an "OpenAI-compatible" provider works: base URL `http://127.0.0.1:8080/v1`, any API key. - Zed's llama.cpp provider reads `/props` and offers tools from `chat_template_caps`. Its generic OpenAI-compatible - provider does not: add `"capabilities": {"tools": true, "chat_completions": true}` on that model in Zed's settings. -- **OpenCode** (#543). A starting point for `opencode.jsonc` (in your project, or `~/.config/opencode/`); the field - names are OpenCode's, so check its config docs if your version differs: +The model thinks before it answers (streamed as `reasoning_content`; Anthropic: `thinking` blocks). Choose how much per request +in the chat page, in `chat.py` (`/think low`) or over the API: + +| API | How | +| --- | --- | +| OpenAI | `"reasoning_effort": "none" \| "low" \| "medium" \| "high"` (also `"reasoning": {"effort": ...}` or `"chat_template_kwargs": {"enable_thinking": false}`) | +| Anthropic | `"output_config": {"effort": "low" \| "medium" \| "high"}`, `"thinking": {"type": "disabled"}`, or `"thinking": {"type": "enabled", "budget_tokens": N}` (under 2K = low, under 8K = medium, more = high; also the hard budget) | + +Without a setting the model uses its own default, **high**. `none` answers at once. The levels are instructions the model was +trained with, not a hard token limit. + +### Config keys (`strata-.json`, outside `"args"`) + +All are optional. Setup rewrites the config when run again and keeps keys it does not manage. The About tab's Model settings +card edits some of them (the sampling defaults, `reasoning_budget_tokens`, `fit_max_tokens`, `anthropic_thinking`, +`effort_position`, `aliases`, `idle_unload_s`, `lazy_load`, `engine_silence_s`, `api_monitor`, `open_browser` and +`--vram-reserve-mib`). An empty field removes the key, the earlier file is kept as `strata-.json.bak`, and the change +applies at the next start. Network, key, MCP and program keys are not editable there. + +| Key | Default | Effect | +| --- | --- | --- | +| `reasoning_effort` | model default (high) | Level for requests that name none. The engine has no such option; `--reasoning-effort` in `"args"` is moved here with a note. A request's level and the web page's shared Chat settings win. | +| `reasoning_budget_tokens` | `0` (no budget) | Caps the thinking at N tokens: the server ends it with a short wrap-up line and `` and the model answers from there. The wrap-up counts as output tokens. A request's own `reasoning_budget_tokens` (or llama.cpp's `thinking_budget_tokens`) wins. | +| `answer_room` | on | A request with thinking on, no budget and `max_tokens` of 1,024 or more gets one: thinking is wrapped up when a quarter of the cap (at least 512 tokens) is left. Requests without `max_tokens` get three quarters of the context. `false` turns it off. Clients should not cap the reply far below what the context allows. | +| `reasoning_close_retry` | off | Closes the thinking once and continues when a reply ended inside its thinking with no answer and no tool call (#1053). | +| `reasoning_loop_recovery` | off | `"stop"` ends, `"recover"` (or `true`) restarts once, a reply whose reasoning repeats whole passages. Checked every 512 output tokens. It can mistake repeated useful code for a loop, so keep the answer quality alongside the completion rate when testing it (#728). | +| `repeat_stop_tokens` | `256` | A reply that repeats one token this many times in a row ends as `"length"`. `0` turns it off. | +| `effort_position` | `"start"` | `"end"` puts a `low` or `medium` effort in a short system turn right before the answer, so a request that only changes the effort reuses the conversation cache. How well the model follows a level in that place is not measured. | +| `anthropic_thinking` | as the template | `"on_request"` renders a `/v1/messages` request that asks for no thinking without thinking, as Anthropic does. Claude Code's short helper calls need it. | +| `tool_call_recovery` | off | Also reads a declared tool's call written as ``, as JSON inside ``, as a `` block without ``, or as a second call inside one ``. On 1,462 agent turns 98.3% read as intended with it and 92.9% without. | +| `env_to_user` | off | Moves opencode's `` block (working directory, date) to the first user message so the system turn is the same for every session and the disk prefix cache restores it. | +| `return_progress` | off | Per request `"return_progress": true` adds llama.cpp's `prompt_progress` (`total`, `cache`, `processed`, `time_ms`) to the stream, one update per `--prefill` chunk. A prompt shorter than one chunk sends nothing. | +| `prevent_sleep` | off | Windows only: keeps the PC awake while a request runs or waits (`SetThreadExecutionState`). On Linux and macOS one startup line says it does nothing. | +| `sampling` | the model card | Defaults for requests that leave sampling fields out, for example `{"temperature": 1.0, "top_p": 0.95, "top_k": 20}`. | +| `aliases` | none | Extra model names listed in `/v1/models`, like llama-server's `--alias`. Any other name is still served. | +| `fit_max_tokens` | `true` | A request whose `max_tokens` runs past the context gets it shortened to the room left. `false` (or `--no-fit-max-tokens`) returns 400 instead. A prompt that leaves no room is always refused. | +| `engine_silence_s` | `300` | Ends the engine after this many seconds without a line during a request (`0` = wait forever). While a prompt is read, each chunk may take three times the previous one's time. | +| `parallel` | `1` | Up to N requests decoded together in batch slots ([BATCHING.md](BATCHING.md)). | +| `api_monitor` | off | Records requests for `/api-monitor`. | +| `lazy_load` | off | Start without loading the model. | +| `slot_save_path` | none | Enables session files. | +| `host`, `api_key`, `cors_origins`, `trusted_origins`, `allowed_hosts` | local only | See [Network access](#network-access). | +| `codex_thread_titles`, `codex_compaction_cache` | off | See [Codex CLI](#the-responses-api-and-codex-cli). | +| `mcp_servers`, `mcp` | none | See [Tools from MCP servers](#tools-from-mcp-servers). | + +### Streaming and tool calls + +- With `"stream": true` everything arrives as it is made: thinking, answer and tool calls (the tool's name first, then its + arguments piece by piece). While the model reads a long prompt the stream sends keep-alives. The server window prints progress + every 15 s and `GET /status` says what it is doing. Closing the connection or pressing stop in your app stops the model. +- A tool call counts only when it names a tool the request declared. Any other call, or any call in a request that declared + no tools, comes back as the text it was written as. A call whose envelope drifted by a token (`<=NAME>`, `` + or `` instead of ``) is read as the call when NAME is a declared tool. +- JSON output: `response_format: {"type":"json_object"}` or `{"type":"json_schema", ...}` on `/v1/chat/completions`. The schema + must accept only JSON objects at its root (`"type":"object"`, or `anyOf`/`oneOf` of object schemas). Local `#` references + work and remote ones are refused. This is schema prompting followed by server validation, not grammar-constrained + decoding, with one generation per request. `jsonschema` is used when installed (`python -m pip install "jsonschema>=4.23,<5"`); + without it the answer is only checked to be one JSON object. Malformed output returns **502** with + `error.code: structured_output_failed`, an invalid schema **400**. JSON formats combined with tools or MCP are refused. + Streaming buffers the answer and emits content only after validation. +- `logit_bias` and token/bias pair lists work on CUDA and HIP engines without continuous batching ([LOGIT_BIAS.md](LOGIT_BIAS.md)). +- Stop strings (`stop`, `stop_sequences`), `temperature`, `top_p`, `top_k`, `min_p` and `seed` are honored per request. + `top_k` keeps at most 64 candidates. Penalties (`presence_penalty`, `frequency_penalty`, `repetition_penalty`, + `penalty_last_n`; llama.cpp's `repeat_penalty` and `repeat_last_n` work too) apply to every token the speculative check + verifies, as if decoded one at a time, and make a request 1-11% slower because more drafts are rejected. +- Sampling defaults, in order of precedence: the request, the web page's shared Chat settings, the config's `sampling` block, + then the model card. Without a block, thinking requests get temperature 1.0, top_p 0.95, top_k 20, and non-thinking + requests get 0.7, 0.8, 20 with presence_penalty 1.5 (Qwen3.6-35B-A3B uses presence_penalty 1.5 in both). Greedy is only for + whoever asks: `temperature: 0` in the request or `"sampling": {"temperature": 0}` in the config. With the default adaptive + tier a sampled result is not reproducible from run to run. For seed-reproducible output add `--adapt-every 100000`. +- Sampled drafting is opt-in and not faster in a way that holds up. `STRATA_SPEC_COUPLED=1` (or `--coupled-draft`) and + `STRATA_SPEC_GUMBEL=1` raise the accepted-draft rate on some machines; `STRATA_SPEC_PROB=1` is speculative rejection + sampling. `STRATA_SPEC_DEPTH=1` prints acceptance by draft depth and `STRATA_SPEC_MIN_TEMP=0.9` limits sampled drafting + to hotter requests. Try them on your own card before relying on them. +- **Non-finite logits end the request and drop the cached state (on by default).** If a verify window's logits are NaN, the + window emits no tokens, the request ends, and the next request drops the live session, checkpoints and parked + conversations and reads its prompt from token 0. The server retries a request that had sent nothing once, transparently. + A started reply ends as `"length"`, and a second hit on the retry returns an error. `STRATA_NAN_GUARD=0` turns the check off; + `STRATA_NAN_INJECT=[,...]` flags the n-th checked window to test the path. The one-shot command line and the + Intel (SYCL) port do not check. + +### Context and limits + +- Context is chosen in setup (8K-262K). Requests longer than that are refused, never cut silently. +- `max_tokens`, `max_completion_tokens` and `max_output_tokens` of `-1`, `0` or left out mean no cap: the answer may use the room + left in the context. +- The `model` field is not matched against the loaded model, because one server serves one model and OpenAI clients often + send names of their own. Only `GET /v1/models` says what is loaded. +- One request at a time unless `"parallel": N` is set. Images only when set up with them. No video. + +### Context beyond 262K (rope scaling, experimental, off unless you pick it) + +The model was trained on 262,144 positions. Rope scaling rescales the rotation angles so that longer contexts stay usable, +with llama.cpp's types and flag names: `linear` (Position Interpolation) and `yarn`. Use **yarn**. Without the flags +nothing changes. Scaled contexts need proportionally more VRAM and RAM (~13 KB for the KV cache and ~0.26 KB for the rope +tables per token). The scaling is fixed for the whole run, because the KV cache stores keys after rotating them. + +- Source setup: `START-HERE.bat --setup --context 393216` picks yarn and derives the factor (final context / 262,144, at least 1). + `--rope-scaling` and `--rope-scale` override; an explicit `--rope-scaling none` for a context past 262,144 is refused. +- Engine flags: `--rope-scaling none|linear|yarn`, `--rope-scale F`, `--rope-freq-base`, `--rope-freq-scale`, + `--yarn-orig-ctx` (262,144), `--yarn-ext-factor`, `--yarn-attn-factor`, `--yarn-beta-fast` (32), `--yarn-beta-slow` (1). + A fine-tune's `rope.scaling.*` keys are the defaults the flags override. +- Within the trained range a scaled run is a slightly different model, so setup turns scaling on only past 262,144. + Pictures read the same scaled table; that is unmeasured, and the evidence in History is all text. + +### Network access + +- **Local only by default** (`127.0.0.1`). For other devices on your network run setup with + `--host 0.0.0.0 --api-key some-long-secret`, or add `"host": "0.0.0.0"` and `"api_key": "..."` to the config. A key is + recommended, not required: without one the server starts and warns that anyone who can reach the port can use the model + and its tools. `--no-api-key` (or `"api_key": "none"`) says it is on purpose. The server window prints this PC's addresses. + On Windows allow the port in the firewall for private networks: + `New-NetFirewallRule -DisplayName "Strata 8080" -Direction Inbound -Protocol TCP -LocalPort 8080 -Action Allow -Profile Private` + (admin PowerShell). +- **From the internet:** put a tunnel in front, for example `cloudflared tunnel --url http://127.0.0.1:8080`. Set a key first. + Several keys (one per client, so one can be withdrawn): `--api-key key1,key2`, `"api_key": "key1,key2"` or + `"api_key": ["key1", "key2"]`. `STRATA_API_KEY` sets one too. Streamed answers carry `X-Accel-Buffering: no`. When you + open the web page through a proxy or tunnel whose address differs, add it: `"trusted_origins": ["https://strata.example.com"]`. +- **Browser pages (CORS):** off by default. `"cors_origins": ["https://chat.example.com"]` lets those origins call `/v1/*`; + `["*"]` lets any page and is only sensible with an API key. It never opens `/settings`, `/unload` or the MCP tools. It + belongs to the engine's own server; the desktop app answers browser pages on its own API address and already allows + pages of any site there. +- **Host names (DNS rebinding):** without an API key the server answers only requests whose `Host` is `localhost`, an IP + address, the address it listens on, or (when it listens beyond this PC) this PC's name and `host.docker.internal`. Others + get **403** naming the setting. Add a name with `"allowed_hosts": ["strata.example.com"]` or + `STRATA_ALLOWED_HOSTS=strata.example.com` (comma-separated). `".example.com"` allows every name below it and `["*"]` + turns the check off, as does setting `api_key`. +- **Web pages without a key:** a `POST` to `/v1/*` carrying an `Origin` header is answered only for Strata's own page, pages on + `localhost` or an allowed host, origins in `trusted_origins` or `cors_origins`, and browser extensions and desktop apps, and only with + a JSON body. Other pages and `Origin: null` get **403**. Clients without an `Origin` (curl, the SDKs) are not affected. With + an API key, the key decides. `POST /unload`, `/load` and `/slots/0?action=...` also take only JSON from no `Origin`, + Strata's own page or a trusted origin. + +### Connecting apps + +- **Any "OpenAI-compatible" app:** base URL `http://127.0.0.1:8080/v1`, any API key. Zed's llama.cpp provider reads `/props` + and offers tools from `chat_template_caps`. Zed's generic OpenAI-compatible provider does not; add + `"capabilities": {"tools": true, "chat_completions": true}` on that model in Zed's settings. +- **Claude Code:** set `ANTHROPIC_BASE_URL=http://127.0.0.1:8080` and `ANTHROPIC_MODEL` to a Claude model name it knows + (Strata ignores the name), plus any `ANTHROPIC_AUTH_TOKEN` (or your `api_key`). +- **OpenCode:** a starting point for `opencode.jsonc`. The field names are OpenCode's, so check its config docs if your version + differs. ```jsonc { @@ -1042,8 +587,8 @@ print(r.choices[0].message.content) "name": "Qwen3.8-Flash-Next (Strata)", // context: what you chose in setup; output: what one reply may use (prompt + output must fit) "limit": { "context": 262144, "output": 32768 }, - "options": { "reasoningEffort": "high" }, // sent as reasoning_effort - "variants": { // switch between them in OpenCode + "options": { "reasoningEffort": "high" }, + "variants": { "low": { "reasoningEffort": "low" }, "medium": { "reasoningEffort": "medium" }, "none": { "reasoningEffort": "none" } @@ -1056,471 +601,115 @@ print(r.choices[0].message.content) } ``` - Set `limit.context` to the context you chose in setup: OpenCode compacts the conversation before it gets there. - Keep `limit.output` well under it: a request whose prompt plus `max_tokens` runs past the context has its `max_tokens` - shortened to the room left (see **Context** below). For a hard cap on the thinking, add - `"reasoning_budget_tokens": N` to `strata-.json` (see above). -- **Claude Code** (Strata 0.1.17 or newer): set `ANTHROPIC_BASE_URL=http://127.0.0.1:8080` and - `ANTHROPIC_MODEL` to a Claude model name it knows (it refuses names it doesn't; Strata ignores the name), plus any - `ANTHROPIC_AUTH_TOKEN` (or your `api_key`, if you set one). -- **Codex CLI** (0.1.39): see [the Responses API](#the-responses-api-and-codex-cli) below. -- **Context.** Chosen in setup (8K-262K). Requests longer than that are refused, never silently cut. A request whose - `max_tokens` would run past the context is answered with `max_tokens` shortened to the room left (agents such as - Claude Code ask for 32,000 on every request); the thinking budget follows the shortened cap. For the strict llama.cpp - behaviour (400) set `"fit_max_tokens": false` in `strata-.json` or pass `--no-fit-max-tokens` to - `serve/server.py`. A prompt that leaves no room at all is always refused. -- **`max_tokens` and the `model` field.** `max_tokens`, `max_completion_tokens` and `max_output_tokens` of `-1` or `0`, like a - field that is left out, mean "no cap": the answer may use the room left in the context (clients send `-1` for that). The `model` field is not matched - against the loaded model on purpose: one server serves one model, and OpenAI clients often send a name of their own (`gpt-4o`, `local`), - so any name is answered by the loaded model (a name from `aliases` and the model's own name work the same). Only `GET /v1/models` says what - is loaded. -- **Model aliases** (0.1.32). `"aliases": ["qwen", "local-model"]` in `strata-.json` lists the model under - those names too in `/v1/models` (each with its own `id`, and in the model's `aliases`), like llama-server's - `--alias`; a request naming one is answered under that name. Any other name is still served, as before. -- **Model settings in the web page (0.1.39, #564).** The About tab's Model settings card shows and changes a few of - the keys above in `strata-.json`: the `sampling` defaults (temperature, top_p, top_k, min_p), - `reasoning_budget_tokens`, `fit_max_tokens`, `anthropic_thinking`, `effort_position`, `aliases`, `idle_unload_s`, - `lazy_load`, `engine_silence_s`, `api_monitor`, `open_browser` and `--vram-reserve-mib`. An empty field removes the - key (its default). Every other key of the file stays as it is, the earlier file is kept as - `strata-.json.bak`, and the model uses the change from its next start. Only Strata's own page can save - (JSON, the API key when one is set, as for the Chat settings); the network, key, MCP and program keys are not - editable there. -- **From other devices on your network.** The server listens on your PC only (`127.0.0.1`) unless you say otherwise: - run setup with `START-HERE.bat --setup --host 0.0.0.0 --api-key some-long-secret` (or add `"host": "0.0.0.0"` and - `"api_key": "..."` to `strata-.json`). The key is recommended, not required: without one the server still starts and - warns that anyone who can reach the port can use the model and its tools (to say it is on purpose, use `--no-api-key` on - the server or `"api_key": "none"` in the config, which silences the config-empty note). The server window then prints this PC's addresses - (`from other devices: http://192.168.x.x:8080/`); open that on the other device, or use `.../v1` as an API base URL. - On Windows the firewall blocks it until you allow it: accept its prompt for Python (private networks), or run - `New-NetFirewallRule -DisplayName "Strata 8080" -Direction Inbound -Protocol TCP -LocalPort 8080 -Action Allow -Profile Private` - in an admin PowerShell, and make sure the network is set to Private. -- **From the internet.** Put a tunnel in front of it, for example [cloudflared](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/do-more-with-tunnels/trycloudflare/): - `cloudflared tunnel --url http://127.0.0.1:8080`. **Set a key first**, or anyone with the link can use your PC: - add `"api_key": "some-long-secret"` to `strata-.json` (or set the `STRATA_API_KEY` environment variable); - clients then send it as their API key. Several keys (one per client, so one can be withdrawn alone): separate them - with commas, `--api-key key1,key2` or `"api_key": "key1,key2"` as llama.cpp does, or give a list, - `"api_key": ["key1", "key2"]` (the way to use a key that contains a comma); a request passes with any one of them. - Streamed answers carry `X-Accel-Buffering: no`, so nginx-style proxies pass - each token on at once. The web app's settings and MCP tools only answer Strata's own page: when you open it through - a proxy or tunnel whose address differs, add that address, e.g. `"trusted_origins": ["https://strata.example.com"]`. - With the key set, any `Host` name reaches the server (see Host names below). -- **From web apps in a browser (CORS).** Off by default. `"cors_origins": ["https://chat.example.com"]` lets pages of - those origins call `/v1/*` from the browser (Open WebUI's direct connections, browser extensions); `["*"]` lets any - page do it - only sensible with an API key. It never opens `/settings`, `/unload` or the MCP tools. - `cors_origins` belongs to the engine's own server (`strata serve`, `./setup.sh`, port 8080 by default). The desktop - app answers browser pages on its own API address (the one under Serving, port 47831 or the next free one) and already allows pages of - any site there, so its "Allow browser origins" switch in the Expert parameters does nothing. -- **Host names (DNS rebinding).** A web page of another site can point its own name at `127.0.0.1` and then reach - this server as if it were its own, so without an API key the server answers only requests whose `Host` is a name - it knows (with a key the check is off: such a page cannot send the key, and tunnels and proxies that pass their - own name on keep working): - `localhost` (and `*.localhost`), any IP address (`127.0.0.1`, `[::1]`, `192.168.x.x`, ...), the address it - listens on and, when it listens beyond this PC (`0.0.0.0` or a LAN address), this PC's name (`mypc`, `mypc.local`) - and `host.docker.internal`; any port. Others get **403** naming the setting, and the server window prints one line - for each. Reach it under another name (a reverse proxy that keeps the name, a tunnel, a DNS name on your network, - another container's name for it)? Add the name: `"allowed_hosts": ["strata.example.com"]` in - `strata-.json` or `STRATA_ALLOWED_HOSTS=strata.example.com` (comma-separated); `".example.com"` allows that - name and every name below it, and `["*"]` turns the check off (so does setting `api_key`). The hosts of - `trusted_origins` count as allowed. Requests without a `Host` header (HTTP/1.0 clients) pass. -- **Web pages without an API key.** Without `api_key`, a `POST` to `/v1/*` that carries an `Origin` header (a - browser page sent it) is answered only for Strata's own page, pages on `localhost` or an allowed host name (any - port), the origins in `trusted_origins` or `cors_origins`, and browser extensions and desktop apps - (`chrome-extension://`, `moz-extension://`, `app://`: no web site can send those), and only with a JSON body; any - other page, and `Origin: null`, gets **403**. Clients that send no `Origin` (curl, the OpenAI and Anthropic SDKs, - other servers) are not affected. With - an API key, the key decides. `POST /unload` and `POST /load` take `Content-Type: application/json` from Strata's - own page (or no `Origin`), like `/settings`. `POST /slots/0?action=save|restore` keeps the Host and API-key checks - and also takes only JSON from no `Origin`, Strata's own page or a trusted origin - also when an API key is set. + Set `limit.context` to the context you chose in setup and keep `limit.output` well under it. `strata connect` writes + configurations for these apps for you. +- **Codex CLI:** [below](#the-responses-api-and-codex-cli). -**Conversation cache.** A request that continues a chat reads only the part after what the engine already holds: the -live session, or one of the checkpoints it keeps in RAM (up to 6, ~118 MB each, taken at the start of each new -assistant turn and every 16K prompt tokens). A checkpoint is used only when the prompt starts with exactly its tokens -and pictures. The oldest checkpoint - in practice the end of the system prompt, which every chat of the same client -shares - is kept for good while the rest rotates by least recent use, so a NEW chat that shares that prefix starts -reading after it instead of from token 0. A prompt read from the start is also checkpointed at the end of its system -prompt when that is 2,048 tokens or more (engine 0.1.20; PR #62 + #65), so that root exists for agent clients with long -system prompts and tool lists. Claude Code stamps its system prompt with a billing header that changes on every -request (`cch=...`) and every session (the 4th part of `cc_version=`); the server pins both stamps (to `f`s, as -llama.cpp does), so the system prompt and the tool list in front of it are the same prompt on every turn. Engine options: `--prompt-cache N` (0 = off), `--prompt-cache-every N`, -`--prompt-cache-root N` (0 = no system-prompt checkpoint), `--turn-token ID`. -A one-shot request that no later request continues (a classification call, a probe) can send -`"strata_checkpoint": false` in its body: it saves no checkpoint at its last turn nor every 16K tokens, so what -follows the reused prefix (or the root) is read in one run, and its session is not kept or parked for a next request. -It still starts from a checkpoint it matches, and still saves the system-prompt root when that reaches -`--prompt-cache-root`. Without the field (or with `true`) nothing changes. +### Conversation cache -**Checkpoints spread back from the end (opt-in, `--prompt-cache-spread`).** Least recent use keeps the newest points: -every turn adds a checkpoint at its start, so after a few short turns the 6 sit within the last 1-2K tokens. An agent -client that rewrites its history a few thousand tokens back (it trims an old tool output, drops a step) then matches -none of them and resumes from the system prompt - the whole conversation is read again. With the flag the checkpoint -that leaves is the one whose removal leaves the smallest gap for its distance from the newest, so the kept points thin -out geometrically: after 200 turns of 1,000 tokens they sit 0, 1K, 3K, 8K, 43K and 200K tokens before the end, and an -edit d tokens back reads about d again (6K back: 8K instead of 200K; 42K back: 43K). The same number of checkpoints, no -more RAM; the system-prompt root, a pinned prefix and the newest point are never dropped, and the resume is the usual -exact-prefix match. Measured on 2x CMP 50HX (layer split, IQ3_XXS) with the pi coding agent: before, a 210,528-token -prompt that was the same as the session for 204,349 tokens resumed at 9,525 and read 201,003 again (100 s). With the -flag, an 82,593-token conversation grown turn by turn with a turn ~6 steps back rewritten resumed at 73,610 tokens -(10.2 s against 40.0 s read in full), and with the expert placement fixed (`--adapt-swaps 0`, fixed `--pcie-frac`) the -greedy answer was the same text as the full read. Without the flag nothing changes. +A request that continues a chat reads only the part after what the engine already holds: the live session, or one of the +checkpoints in RAM (up to 6, about 118 MB each, taken at the start of each new assistant turn and every 16K prompt tokens). A +checkpoint is used only when the prompt starts with exactly its tokens and pictures. The oldest checkpoint, in practice +the end of the system prompt that every chat of a client shares, is kept for good, so a new chat that shares the prefix +starts after it. A prompt of 2,048 tokens or more is also checkpointed at the end of its system prompt. Claude Code stamps its +system prompt with a billing header that changes on every request; the server pins both stamps so the prompt stays the same. -**One long document, many questions: a pinned shared prefix (opt-in, 0.1.40.2, [RESEARCH_RUNS.md](RESEARCH_RUNS.md)).** -A request can say where the document ends: `"strata_prefix": {"messages": 1}` (the first message is the document), -`{"message": 0, "chars": 210000}` (the first 210,000 characters of message 0, for a client that sends the document and -the question in one message) or `{"tokens": 12345}`. The engine reads the prompt in two parts there, keeps the checkpoint -at the boundary pinned (retention never evicts it, a parked conversation holding it stays parked, `SAVE` keeps it) and, -when a later question resumes from it, does not park the question it leaves. So the next question starts at the end of -the document, and 20 questions cost one read of it. Without the field the server's checkpoints are where they were: the -last turn's start, the system prompt's end and every `--prompt-cache-every` tokens, so a question that follows a long -document in the same chat re-reads up to 16K tokens of it (55 s at 262K on a Tesla P100). The field is checked against the -prompt's own ids (the prefix is always the longest common start), a prefix that cannot be marked is said in the server's -window and ignored, and a malformed field is a 400. One pinned prefix at a time per engine (a new one replaces it); -it needs `--prompt-cache 3` or more. `tools/research_run.py` runs a document and a question list against a server with and -without it. The engine's own key is `pin=N` on the `GEN` line. +| Option | Effect | +| --- | --- | +| `--prompt-cache N` | Number of checkpoints (`0` = off, which also disables parking). | +| `--prompt-cache-every N`, `--prompt-cache-root N`, `--turn-token ID` | Checkpoint interval, the system-prompt checkpoint (`0` = none) and the turn token. | +| `--prompt-cache-spread` (opt-in) | The checkpoint that leaves is the one whose removal leaves the smallest gap, so the kept points thin out geometrically back from the end. An agent that rewrites its history a few thousand tokens back then resumes near the edit instead of at the system prompt. Same RAM. | +| `"strata_checkpoint": false` (per request) | A one-shot request that no later request continues saves no checkpoints and keeps no session. | +| `"strata_prefix": {"messages": 1}` (per request, opt-in) | One long document, many questions. Marks where the document ends: `{"messages": 1}`, `{"message": 0, "chars": 210000}` or `{"tokens": 12345}`. The checkpoint at the boundary is pinned, so 20 questions cost one read of the document. One pinned prefix at a time per engine; needs `--prompt-cache 3` or more; a malformed field is a 400. See [RESEARCH_RUNS.md](RESEARCH_RUNS.md). | +| `--conversation-cache-mib N`, `--conversation-cache-slots N` | Park conversations in a bounded host-RAM cache so alternating requests (controller and worker histories, sub-agents) reuse their prompts. Without `--conversation-cache-mib` the budget is 40% of the RAM left above `--conversation-cache-min-free-mib` once the model is loaded, at most 8192 MiB in steps of 256, off when under 1024; the engine log says which. `0` turns it off. It preserves histories; it does not run requests at once. | +| `--conversation-cache-min-tokens N` | Park only conversations of at least N tokens, so short side requests do not evict a long conversation. Default `0` (park all). | +| `--conversation-cache-park-gain N` (opt-in) | Park a conversation again only after it advanced N tokens past the copy it would replace. | +| `--conversation-cache-min-free-mib N` | RAM floor, default 2560. Parking and session files are skipped when less than the snapshot plus the floor is available. | +| `--memory-limit-mib N` (or `STRATA_MEMORY_LIMIT_MIB`) | Total RAM this engine's container may use, for when a container cannot see its own limit. Config `args`: `"--memory-limit-mib", "102400"`. `0` = none. | -**Multiple conversations (opt-in).** Add `--conversation-cache-mib 8192 ---conversation-cache-slots 4` to the engine arguments to park up to four conversations -in a bounded 8 GiB host-RAM cache. This preserves controller/worker histories when -their requests alternate; it does not execute requests concurrently. No client session -ID is required: only exact token/image prefixes with matching steering mode are reused. -Without `--conversation-cache-mib` the budget is sized from the RAM that is free once the model is loaded (0.2.3, #3493): 40% of what is left above `--conversation-cache-min-free-mib`, at most 8192, in steps of 256, and off when that is under 1024; the engine log says which in one line. An agent Harness that alternates a long main conversation with shorter ones (sub-agents, summaries) used to read every prompt again (`Reused` 0); with the cache on, the next turn of each one reuses its prompt. `--conversation-cache-mib N` sets it and `--conversation-cache-mib 0` turns it off; `--prompt-cache 0` also disables parking. -With `--layer-split`, a parked conversation holds one image per stage (each later stage's -session and its share of the conversation checkpoints), every image is validated before -anything is overwritten, and each is restored on its own GPU. Verified on a 4-GPU split: -a 9,276-token conversation restored in 51 ms, and its follow-up decodes exactly the tokens -it decodes when the conversation never left (`tools/parking_test.py`). FP16, -INT8, Q4_0 and identity-layout K8V4 snapshots are supported; the K8V4 draft ring -remains INT8, as in upstream. Windows/HIP and multi-GPU runtime coverage must be -reported separately from Linux/CUDA evidence. +The RAM figure the floor is checked against is the smallest of `MemAvailable`, the room under each cgroup limit visible to +the engine (v2 `memory.max` and `memory.high`, v1 `memory.limit_in_bytes`, less clean inactive file cache) and the +`--memory-limit-mib` cap. The startup line `strata generate: memory guard: limit X GiB (source: flag|cgroup|meminfo) ...` names +the source. It covers parking and session files only, not the startup sizing of the arena, the resident budget or the Windows commit check. -The web page's Monitor tab has a **Conversation cache** card (0.1.39, #596): the parked conversations against the -slots and the RAM budget, how many were parked, restored and evicted, and the last switch (read from the engine's -log), and for every setup how many prompt tokens the cache gave back - in the last request and since the start. -`/metrics` has the same under `"conversation_cache"`. +Snapshots hold running state, checkpoints, used K/V pages and draft-layer K/V. They add host RAM, not VRAM. Oldest parked entries +are evicted first. A new conversation that starts at a checkpoint inside a parked one (sub-agents that share a system prompt) +restores that prefix and leaves the parked conversation in place (the log says "borrowed"). Oversized snapshots or host +allocation failures fall back to ordinary prompt processing. With `--layer-split` a parked conversation holds one image per +stage. Parked snapshots are not kept across restarts; session files are. `STRATA_SNAPSHOT_FULL_CAPTURE=1` disables K/V +retention for diagnostics. The Monitor tab's **Conversation cache** card shows parked, restored and evicted counts and the +prompt tokens the cache gave back; `/metrics` has the same under `"conversation_cache"`. -Snapshots contain running state, checkpoints, used K/V pages, and draft-layer K/V. -They add host RAM, not another model or VRAM allocation. The byte budget also counts -an incoming snapshot during a switch. After a restore, unchanged K/V pages can be -retained for the next parking operation; growth appends storage without copying -the existing pages, with room for the next turns of at most an eighth of each -buffer (and at most 16 MiB), which the budget counts like the rest of the snapshot. -Rewinds refresh the affected pages, and running state and -checkpoints are captured again. Retained active K/V counts against the same byte -budget and is discarded before evicting parked entries under memory pressure. -Oldest parked entries are evicted first. A new conversation that starts with a -checkpoint inside a parked one (subagents that share a system prompt and tool -list, a compacted history) restores that prefix and leaves the parked -conversation where it is, so its next turn still resumes in full; the engine -log says "borrowed". A match on a parked conversation's live state or its newest -checkpoint is that conversation going on and moves it into the session as -before; so does a match that reaches a pinned shared prefix (`pin=N`), which -keeps one conversation holding the pin. When the outgoing conversation fits only -in the parked one's room, it keeps its place and the parked one is moved in -whole, as before. -Oversized snapshots or host allocation failures fall back to ordinary prompt processing. -`--conversation-cache-min-tokens N` (default 0 = park every conversation) parks only conversations of at least N -tokens. A parked conversation costs one of the slots whatever its length, and parking is a switch-time decision, so a -client that interleaves short side requests with one long conversation spends the cache on the side requests and, once -it is full, evicts the long one oldest-first - exactly the reuse it wanted. Setting the minimum above the side -requests' length keeps the long conversation parked. The check runs before `make_room()`, so a skipped park evicts -nothing and stores nothing; the log line is `conversation cache: skip parking (N tokens, below the M minimum; -parked=P)`. 0 is the pre-flag behaviour and the flag leaves the answers untouched - it only decides which -conversations keep a slot. -`--conversation-cache-park-gain N` (opt-in, default 0 = park every turn) parks a conversation again only after it -advanced N tokens past the copy it would supersede; a turn that grew by a few hundred tokens keeps the older copy -instead of writing a whole snapshot that the next turn drops. The cost is that a resume from the older copy reads up -to N tokens again (#2265). -`--conversation-cache-min-free-mib N` (default 2560) additionally requires that -RAM headroom remain available: the engine checks before allocation (against the -snapshot's estimated size, less the K/V it already retains) and again after capture. -Unknown telemetry or insufficient RAM skips parking, and the same figure and floor -decide whether a session file may be saved or restored (below). An 8 GiB budget -is a cap, not a recommendation for every machine. +### Session files (disk) -**The RAM figure (`available_host_bytes()`).** The headroom is the smallest of three -numbers, sampled each time it is asked and not a reservation (other writers can take the -room afterwards): - -1. `MemAvailable` (Windows: `GlobalMemoryStatusEx`'s available physical memory). -2. The room under each memory limit of the engine's own cgroup and of every ancestor - visible under `/sys/fs/cgroup` (the group named in `/proc/self/cgroup`, or the deepest - part of that path this mount shows): cgroup v2 `memory.max` and `memory.high`, each - minus `memory.current`; cgroup v1 `memory.limit_in_bytes` minus - `memory.usage_in_bytes` (both less the clean inactive file cache, below). `max` and v1's "unlimited" count as no limit; a file that is - absent (no memory controller at that level) is skipped; a file that is there but - cannot be read or parsed makes the cgroup terms unusable: without `--memory-limit-mib` the guard then uses - `MemAvailable` alone, as 0.1.41 did, and prints one warning per process; with the flag the sample is unknown, and unknown skips parking. -3. `--memory-limit-mib N` (or `STRATA_MEMORY_LIMIT_MIB=N`; the flag wins, `0` = none): - N MiB is the total this engine's container or cgroup may use, minus what that - container uses now. Use it when the container cannot see its own limit. The usage - is `memory.current` of the top cgroup this mount shows (the container's own, in a - container with its own cgroup namespace); if that file cannot be read it is - `MemTotal - MemAvailable`, which is a weaker figure (it is only as true as the - container's `/proc/meminfo`), and if `MemTotal` is missing too the sample is - unknown. The cap is taken in addition to 1 and 2, never instead of them: whichever - number is smallest decides. It is an engine argument, so the server config carries it in - `args` (`"--memory-limit-mib", "102400"`); the variable also works from the config's - `"env"`. - -Usage is `memory.current` as the kernel reports it, **less one reclaimable kind: clean -inactive file cache**. That is the group's own `memory.stat` `inactive_file` (cgroup v1: -`total_inactive_file`), less its dirty and writeback pages when `memory.stat` lists them -(`file_dirty`, `file_writeback`; v1 `total_dirty`, `total_writeback`), and never more than -`memory.current`. The kernel gives such pages back at `memory.high` or the limit before -it kills anything, and a loaded model's file reads leave a lot of them behind. Nothing -else is credited: `active_file` (it holds the mlocked pages), `shmem` (pinned expert -complements, the KV pool) and anonymous memory stay charged in full, since a kill at a -hard limit is worse than a skipped park. For each limit the room is -`limit - (memory.current - credit)`, so it never exceeds the limit. A `memory.stat` that is missing or cannot be -parsed gives no credit; it does not make the sample unknown. The operator's cap uses the -`memory.stat` of the same top group whose `memory.current` it reads. A parked -conversation is part of the engine's own usage once captured; the check before -capture is what stops it from growing past the cap. - -The startup log has one line for it, `strata generate: memory guard: limit X GiB -(source: flag|cgroup|meminfo), current Y GiB, reclaimable cache credited Z GiB; -available ...`, where `source` names the -number that was smallest at that moment (`meminfo`: MemTotal and `MemTotal - -MemAvailable`), and a skipped park or a refused save/restore says how much it had and -from which source. The line is printed before the model is loaded, so its credit is the -cache at that moment, not at the time of a park. Without a cgroup limit and without `--memory-limit-mib` the figure is -plain `MemAvailable`, as before. - -What it does not touch: the startup sizing of the expert arena, the file tier's resident budget, the Windows commit -check and `STRATA_PIN_GUARD` (#1250: whether to page-lock the expert copy in steps follows `MemFree`, and each step -is checked against the cgroup-aware `host_available_memory`) keep their own probes (`--resident-budget-gib`, #633, -#1250), and `--memory-limit-mib` does not reach them. This figure is for what runs after the start: parking a -conversation and saving or restoring a session file, which read `MemAvailable` alone. `host_available_memory` -(expert_source.cpp) and this figure both read the cgroup limits; this one also reads `memory.high` and the operator's -cap; an unreadable cgroup file makes it fall back to `MemAvailable` (one warning), or unknown when a cap is set. - -Earlier measurement on a Proxmox LXC where the engine's container cannot see its limit (a 100 GiB cap on the parent -cgroup, outside the container's namespace; the container's own `memory.max` and `memory.high` read `max`): -`/proc/meminfo` showed about 25 GiB `MemAvailable` while about 1 GiB was really left, and right after the engine had -loaded the container's `memory.current` was 98.29 GiB with `anon` 1.29 GiB, `shmem` 65.51 GiB, `inactive_file` -3.81 GiB (clean page cache left from reading the model files) and `active_file` 0.01 GiB, of which the 3.8 GiB of -clean cache is what the credit gives back. There `--memory-limit-mib 102400` is the cap, and the floor -(`--conversation-cache-min-free-mib`) is measured against what is left under it: with the engine already near the cap, -a floor of several GiB refuses every park; a few hundred MiB keeps a margin for the allocator and the kernel while -letting a snapshot that really fits go through. - -Measured on the test machine (2x RTX 3080 20 GB, UD-Q4_K_XL, a 90 GiB container, resident RAM mode, 2 slots), one restart -per arm, `--conversation-cache-min-free-mib 2048`. With `--memory-limit-mib 101376` (the cap our deployment runs with) -the engine parked on every attempt: 135 parks and 0 skips over one run of 308 requests. With `--memory-limit-mib 78000` -the start line read `limit 76.2 GiB (source: flag) ... available 73.0 GiB (MemAvailable 88.6 GiB)`, and 8 of 8 parking -attempts were skipped (`skip parking (physical RAM admission; need 226 MiB plus 2048 MiB floor, 0 MiB available, source -flag)`), none parked. The container's `memory.current` sat at 89.7-89.9 GiB, above the 76.2 GiB cap, so the headroom -under the cap is 0, while `MemAvailable` read 15.7-15.9 GiB in the same samples; a check on `MemAvailable` alone would -have let those parks through (the old check was not run). The requests were served normally (median of 6 decodes -79.6 tok/s); only parking is refused, and the log says so. The unit test `memory_guard_test` (CPU only, no model) passes -130 checks. Not measured: a cap between 78000 and 101376, a run without the flag, any effect on throughput. - -The shared snapshot core validates all layers and checkpoints before applying any -state. Invalid entries are discarded; transfer/synchronization failure is fatal -rather than permission to continue with partial state. Indexer spare keys and the -moving spare row are preserved, including checkpoint rewinds. -The engine log reports parking, restoration, bytes, evictions, individual snapshot -sizes and K/V bytes reused during capture. `STRATA_SNAPSHOT_FULL_CAPTURE=1` disables -retention for diagnostic comparisons. Parked snapshots are not -persisted across restarts; the session files below are. - -**Session files (disk).** The conversation the engine holds can be saved to a file and restored later, also after a -restart of the same engine version, so a long prompt is not read again. The server exposes the save and restore -requests of llama-server's slot API, for its single slot 0, when started with `--slot-save-path DIR` (also -`"slot_save_path"` in the config); NAME must be a plain file name inside DIR. There is no erase action and the file -format is Strata's own, not llama.cpp's: +The conversation the engine holds can be saved to a file and restored later, also after a restart of the same engine version, +so a long prompt is not read again. Start the server with `--slot-save-path DIR` (or `"slot_save_path"` in the config). The +file format is Strata's own, not llama.cpp's, and there is no erase action. ```bash curl -X POST "http://127.0.0.1:8080/slots/0?action=save" -H "Content-Type: application/json" -d '{"filename": "chat1.bin"}' -# {"id_slot": 0, "filename": "chat1.bin", "n_saved": 63025, "n_written": 1198691396, "timings": {"save_ms": 709.5}} curl -X POST "http://127.0.0.1:8080/slots/0?action=restore" -H "Content-Type: application/json" -d '{"filename": "chat1.bin"}' -# {"id_slot": 0, "filename": "chat1.bin", "n_restored": 63025, "n_read": 1198691396, "timings": {"restore_ms": 898.2}} ``` -DIR becomes one absolute path at start (`--slot-save-path` relative to the server's working directory, the config's -`slot_save_path` relative to the config's `"cwd"`), created with mode 0700 when missing. It should be private to the -user that runs Strata: the files hold the conversation's token IDs and state, are created with mode 0600 on Linux -(on Windows they inherit the folder's permissions), and nothing deletes them - about 1.2 GB per 63K-token -conversation. Before it writes, a save checks that the disk has room for the whole new file plus -`--session-min-free-mib` (an engine argument, in the config's `args`; default 4096, 0 = no check) - also when it -replaces a file, whose space comes back only after the rename. This is a preflight, not a quota or a reservation. -NAME may -not contain a path, a drive, a stream (`:`), a Windows device name (`NUL`, `CON.bin`, `COM1`...), a control character, -a leading dot or a trailing dot or space. +- `DIR` becomes an absolute path at start and is created with mode 0700. Files are created with mode 0600 on Linux (on Windows + they inherit the folder's permissions), nothing deletes them, and they are large (about 1.2 GB per 63K-token conversation). + Keep `DIR` private to the user that runs Strata. +- `NAME` must be a plain file name inside `DIR`: no path, drive, stream (`:`), Windows device name, control character, leading + dot or trailing dot or space. A save checks that the disk has room for the whole file plus `--session-min-free-mib` (engine + argument, default 4096, `0` = no check). +- `--session-save-reclaim` (engine argument, off by default) lets a save that fails its RAM check first release reconstructible + host caches (retained K/V, oldest unpinned parked conversations, unpinned checkpoints), and the server waits up to 30 s for + RAM accounting to catch up. It trades later prefix-cache hits for room to save and cannot guarantee a save. +- Clients still send their messages and images; the file holds engine state and token identity, not a chat export. A follow-up + that continues the restored conversation reuses its state; a short tail may be read again. Only the deepest checkpoint is saved. +- A file is bound to the model files (sampled by size and first and last MiB, not by path) and to the settings that change what + the cells mean (engine version, CUDA or HIP, `--kv`, `STRATA_KV_ROT`, `--kv-resident`, `--max-context`, `--mtp-window`, the + rope configuration, a loaded control vector and the arithmetic switches). Sampling, seeds, draft tuning and the expert + tier are not in it. Do not change model files while their sessions are kept. The hash detects accidental corruption and does + not authenticate a file: restore only files this engine wrote. +- Errors: `501` without `--slot-save-path` or with parallel requests; `400` for a slot other than 0, an unknown action, a refused + file name or an invalid file; `404` for a missing file; `503` when the model is not loaded or RAM is short; `507` when the disk has + no room; `500` for other I/O failures or an engine that ended. `error.published: true` means a save failed after its new + file had replaced the old one. Requests must be `application/json` (else `415`) and come from no browser page, Strata's own + page or a trusted origin. +- Not supported with `--layer-split`, `--peer-device`, `--batch` or `--prompt-cache 0`. The engine has been run on Linux and CUDA + only for this feature. Exact token replay across restarts is not guaranteed: expert residency and CPU/GPU rounding can + change later output. -`--session-save-reclaim` (an engine argument in the config's `args`, off by default) lets a SAVE that fails its -RAM preflight first release reconstructible host caches: retained K/V buffers, oldest unpinned parked -conversations, then unpinned checkpoints other than the one selected for the file. It stops as soon as measured -available RAM meets the original allocation estimate plus `--conversation-cache-min-free-mib`; it does not lower -that floor or assume that freed allocations have reached the OS. Pinned prefixes and the selected checkpoint -stay intact, including their order and tie preference. Unknown RAM telemetry or an unknown allocation estimate -does not evict anything. The usual admission check still runs before copying state. +### API request monitor -With this option the HTTP server also allows up to 30 seconds for physical RAM accounting to catch up after -cache reclamation. Only an unpublished `memory` refusal from the SAVE RAM preflight is retried, at most three -times, when measured available RAM reaches the reported estimate plus the original floor and 2 MiB for rounding. -The same live engine process and service FIFO are held throughout; a process change, death, unknown telemetry -or missing estimate ends the wait. This does not unload a model or trim another process's working set. Persistent -pressure still returns the last refusal. The wait is outside the engine's reported `save_ms`. - -This can trade later prefix-cache hits for room to save. Released caches stay released even if admission or file -I/O later fails; the current tokens, images, steering and live device state are unchanged. It cannot guarantee a -SAVE under arbitrary memory pressure. With the option off, or enough RAM at the first probe, cache retention and -the session file format are unchanged. RESTORE is unchanged. - -The request must be `Content-Type: application/json` (else `415`) and come from no browser page, Strata's own or a -trusted origin (another site's `Origin` gets `403`, also with an API key); the Host and API-key checks apply as -everywhere. Errors: `501` without `--slot-save-path` or with parallel requests; `400` for a slot other than 0, an unknown action, a refused -file name, or a file the engine refuses as invalid (not a session file, corrupt, another model or configuration, over -this session's limits; the session is as it was); `404` for a restore of a missing file; `503` while the model is not -loaded, when the RAM to read the file is not there or an allocation failed; `507` when the disk has no room (the -free-space reserve, or the OS reports no space or quota); `500` for any other I/O failure (permissions, read, write, -flush, rename) and when the engine ended (a restore transfer failure, below, an engine that said nothing for -`engine_silence_s`, or one that answered out of protocol) - the next request starts it again. The status follows the -engine's category (`error.kind`: `invalid`, `memory`, `storage`, `io`), never the words of the message. A save that -failed after its new file had replaced the old one says so with `error.published: true` (below). A save or restore waits for the running request -(the same queue), shows in `/status` and counts as activity for the idle unload. A later request whose messages -continue the restored conversation reuses the restored state or its checkpoint; a short tail may be read again (35 -tokens in the measurement below). Only the deepest checkpoint is saved, so an edit further back reads more again. -Clients still send their messages (and images): the file holds engine state and token/image identity, not a chat -export. Underneath, `strata --serve` takes `SAVE ` and `RESTORE ` on stdin between requests and answers -`SAVED `, `RESTORED `, `SERR ` (failed, -the engine and the session as they were) or `FATAL ` (then exits). On the way, `SESSION ` follows -every block of the file that moved (at most 16 MiB, the last partial block and a small file included), and -`SWAIT ` comes before a step that blocks in one call (`fingerprint`, `capture`, `flush`, `publish`, -`validate`, `transfer`): that step is allowed those seconds - 60 plus one per 4 MiB it concerns, at most 3600 - by the -engine's watchdog and by the server, then it counts as stuck. The next line clears the allowance. The server checks -every line: a malformed or unknown one, counts that are negative or go back, an allowance outside 1..3600 or a time -that is not a finite number end the engine as out of step. - -One file holds the running state, the deepest checkpoint, every QSA layer's K/V up to the conversation's length and -the draft layer's K/V. Format v1, little-endian, fixed-width integers, IEEE-754 floats (a big-endian build does not -compile): a 64-byte header (magic `STRSESS\x01`, version u32, header size u32, model fingerprint u64, config -fingerprint u64, payload length u64, two reserved u64 that must be 0, a hash of the first 56 bytes), the payload -(geometry, layer range, cvec flag, the live state, the checkpoints, the K/V layers; every array preceded by its u64 -count), the payload hash and the end marker `STRSEND\x01`. The hash is a 64-bit function with xxHash64-style rounds, -not the standard XXH64 stream; it detects accidental corruption and does not authenticate a file: restore only files -this engine wrote. An unknown version is refused; a new format gets a new version number. - -A file is bound to the model inputs and to the settings that change what the saved bytes mean. The model fingerprint -samples (size, first and last MiB) every file the engine loads, by its role: the GGUF shards (also -`--native-dense-gguf`, the head shards and `--embd-gguf`), the PLE shard, the pack's index/dense/embedding files and -`native_experts.txt`, every file the expert source resolved - the pack's `experts.bin`, or for a pack read in place each -layer's gate/up/down GGUF as `native_experts.txt` names it, also one outside the CLI shards - and the MTP's files; not -other files in those folders, and not the path, so a moved model folder still matches. A -change in the middle of a file that keeps its size is not seen: do not change model files while their sessions are -kept. The config fingerprint covers the engine version string (another version is refused; two builds of the same version -are not told apart - there is no build hash), CUDA or HIP, -`--kv`, `STRATA_KV_ROT`, `--kv-resident`, `--max-context`, `--mtp-window`, the resolved rope configuration (type, -base, factor, freq scale, original context, the YaRN knobs - the cached K is post-RoPE), the loaded control vector (a -digest of the tables uploaded: every file's content times its exact scale, the mode, the layer range and the -direction) and the arithmetic switches (`--native-*`, `--no-ple`, the A/B arms). Sampling, seeds, draft tuning and -the expert tier are not in it: they change what comes next, not what the saved cells hold. - -A save writes a temporary file with a new hidden name beside `path` (created exclusively, never an existing file or -link), flushes it to the disk, renames it over `path` (`MoveFileExW` with write-through on Windows, which promises no transaction -on every filesystem) and, on POSIX, flushes the folder (Windows has no folder flush; the file flush and the -write-through rename are all it does). A save that fails before the rename keeps the old file at `path` and removes -only its own temporary file. Once the rename is done the old file is gone: if the folder flush then fails, the save -fails with `published` set - the new file's bytes are complete and flushed, but its name may not survive a power loss. -A filesystem that cannot flush a folder (`EINVAL`) is not a failure; the engine logs it. A restore opens `path` without following a symbolic link (or a Windows reparse point) and refuses -anything but a regular file with one name; it checks the size, the header, both fingerprints (before the payload is -parsed; the first 16 MiB block, header included, is already read), that the parse's peak (the image, the read buffer, the per-segment overhead) fits in RAM above the parking -floor (`--conversation-cache-min-free-mib`), the file's size against the largest this session can restore, the -geometry and layer range before any state array, and every count against the bytes left and this session's exact -limits (context and cells, checkpoints, layers, each running-state array, each K/V part) before allocating it, the payload hash -and then the usual snapshot validation - all before any device write, and a refusal leaves the current session as it -was. A transfer failure after the device writes began ends the engine (`FATAL`) rather than decode from a partial -state; the server reports `500` and starts it again. A restore does not park the outgoing session. Not supported with -`--layer-split`, `--peer-device`, `--batch` (the config's `"parallel"`, #465; the server answers `501`) or -`--prompt-cache 0` (the RAM conversation cache need not be on). On Linux the file -moves with `O_DIRECT` in 16 MiB blocks when the filesystem takes it (buffered I/O otherwise, or with -`STRATA_SESSION_BUFFERED=1`); on Windows with buffered I/O. The engine has been run on Linux/CUDA only. An earlier -revision's CPU file-I/O test passed as a 32-bit Windows executable under Wine; the current code has not been built -for Windows, and the Windows engine, HIP and AMD cards have not been run. - -A session file saves conversation state, not all of the process's execution history. Exact future token replay -across restarts is not guaranteed: expert residency and CPU/GPU rounding can change later output. In a 63K test the -first 32-token continuation after a restore matched the process that kept running; on the next continuation three -restored processes agreed with one another but differed from that process from the 28th token on. A refused, -corrupted restore in between did not change the restored processes' continuation. The cause of this divergence has -not been isolated. - -Measured on an RTX 4070 Ti (12 GB), Ryzen 9 5900X, 64 GB RAM, NVMe ext4, IQ3_XXS, a 63,025-token conversation -(engine 0.1.38 with this change, binary sha256 `3bbe4fc3...`): file 1.20 GB, save 0.65 s including the flushes to -disk (10 more saves of the 1.20 GB state after a further turn: 0.59-0.61 s replacing one file, 0.80-1.18 s to new -names), restore 1.05 s in a new engine process, then the next 32-token turn in 1.14 s with the same 32 output tokens as -the same turn without a restart (1.00 s); the cold first turn takes 25.5 s. The save reported 73 `SESSION` lines, never -more than 16 MiB apart. A symbolic link, a second hard link and a file with one flipped payload byte were refused -(the last also while a restored conversation was live, which then went on to answer); a fresh engine with another -`--rope-freq-base` refused the file and then answered; a save with a RAM floor above the machine's RAM was refused as -`memory` before any copy, the file it would have replaced untouched. The times are single runs. - -**Current limits (v1):** one request at a time unless `"parallel": N` is set (opt-in batch slots, up to N requests -decoded together: [BATCHING.md](BATCHING.md)), and one conversation cached at a time (switching between two chats -re-reads the other one unless the opt-in cache above is enabled, or each conversation keeps its own batch slot); images only when set up with them (below); no video. **Temperature / top_p / top_k / min_p / -seed** are honored per request (OpenAI and Anthropic fields), and so are stop strings (OpenAI `stop`, a string or up -to 4; Anthropic `stop_sequences`): the answer ends before the first one, which is not sent, and the engine stops -there (`finish_reason` "stop"; `stop_reason` "stop_sequence" with `stop_sequence` set to the one found); with the default adaptive expert tier a sampled result -is not reproducible run to run - for seed-reproducible output add `--adapt-every 100000` (static residency) to the -engine arguments. The run config's optional `sampling` block sets the defaults for requests that leave the fields out -(`"sampling": {"temperature": 1.0, "top_p": 0.95, "top_k": 20}`); a request's own fields always win. **With no -block at all (0.1.45) a request without sampling keys gets the sampling the model card recommends**, picked by the -request's thinking state: Qwen3.8-Flash-Next (and Gyro) thinking temperature 1.0, top_p 0.95, top_k 20; not thinking temperature 0.7, -top_p 0.8, top_k 20, presence_penalty 1.5. Qwen3.6-35B-A3B thinking 1.0 / 0.95 / 20 with presence_penalty 1.5, not thinking -0.7 / 0.8 / 20 with 1.5 (min_p 0 and repetition_penalty 1 are "off"). Order of precedence: the request, the web page's shared Chat settings, -the config's `sampling` block, the card. Greedy is only for who asks for it: `temperature: 0` in the request, or `"sampling": {"temperature": 0}` in the config -(before 0.1.45 a config without a block ran greedy, and greedy decoding is what makes small local models loop; #2685). The web page sends no sampling of its own until you change its Sampling settings (it showed and sent 0.6 before); an old stored 0.6 with the default top_p and top_k counts as not set. The penalties (`presence_penalty`, `frequency_penalty`, -`repetition_penalty`, with `penalty_last_n` capping how many recent tokens they count over, default 64 when any -penalty is set; llama.cpp's names `repeat_penalty` and `repeat_last_n` work too, in the config's `sampling` block and in a request, #1819) ride the same path; they count the tokens the request has consumed, so a repetition penalty -suppresses what the model itself just said, not the prompt alone. Since engine 0.1.19 they apply to every token -the speculative decoding checks at once, exactly as if it decoded one token at a time (before, only the first of -each batch got them). That makes requests with penalties 1-11% slower than in 0.1.18: the draft layer guesses -without penalties, so more of its guesses are now rejected. Requests without penalties are unchanged. `top_k` keeps at most 64 candidates: `0` ("off") or anything above 64 uses all 64. - -**Batch MTP on a layer split (`--batch-mtp`).** Our measurement on 2x R9700 with all experts in VRAM: 80.2 -> 62.7 tok/s (-21.9%, 0/5 pairs faster) against the pipelined default, so it pays only when the experts do not all fit in VRAM (docs/BATCHING.md). - -**Token biases (opt-in).** The chat API supports request-scoped `logit_bias` objects and token/bias pair lists on supporting CUDA/HIP engines without continuous batching. A value of -100 (or `false` in a pair list) excludes a token. Unsupported engines return an error. See [LOGIT_BIAS.md](LOGIT_BIAS.md) for formats, limits and validation. - -**Sampled drafting (opt-in, 0.1.40.2).** With temperature above 0 the engine keeps a draft only when it equals the token the model sampled for that position, and the draft layer proposes its best guess. Two opt-in switches change how the draft layer drafts a sampled request; greedy requests (temperature 0) are never touched and stay byte-identical. `STRATA_SPEC_COUPLED=1` (or `--coupled-draft`) drafts by sampling with the request's own settings and the random number the checking row will use; the text for a seed does not change. `STRATA_SPEC_PROB=1` is speculative rejection sampling: the draft layer samples its guess from its own distribution q, the check accepts it with probability min(1, p/q) (p is the model's distribution after penalties, top_k, top_p, min_p and temperature) and otherwise samples the replacement from the leftover distribution, so every token is distributed exactly as without drafts (the text for a seed then depends on the drafts, so it is reproducible only for the same engine, flags and prompt). Guesses without a distribution (prompt lookup, suffix drafts) keep the exact-match rule, which is already exact. Neither is faster in a way that holds up: on an RTX 3060 (IQ3_XXS, 200-token story and code answers, 10 interleaved pairs) the default's median output speed was 42.4 / 43.2 / 43.3 tok/s at temperature 0.3 / 0.7 / 1.0, coupled drafting 40.6 / 42.4 / 43.3 and rejection sampling 41.0 / 42.6 / 42.6, with run-to-run differences of up to 5% between repeats; drafts kept per round fell at 0.3 and rose by about 5 points at 1.0, which the cost of the longer sampled windows ate. `STRATA_SPEC_DEPTH=1` prints the acceptance by draft depth for each request; `STRATA_SPEC_MIN_TEMP=0.9` limits sampled drafting to hotter requests. Proof that the rejection rule keeps the distribution: `spec_prob_test` (chi-square, 2 million trials per case) and `spec_verify_parity` (GPU against the host reference). +Off by default, because it keeps prompts and answers in memory. Turn it on with `"api_monitor": true` or +`serve/server.py --api-monitor`; otherwise the two endpoints answer 404. `/api-monitor` shows the model state, load and unload +controls, active and queued requests, request bodies, output, separate reasoning and response bodies, with load, queue, +first-token and decode timings. `GET /api/requests` returns summaries and `GET /api/requests?id=` one request. The +monitor keeps the newest 100 requests in memory until restart, 262,144 characters per field, with visible truncation flags. +Headers are not recorded. Treat the history as sensitive when Strata is reachable from a network, and set an API key. --- ### The Responses API and Codex CLI -`POST /v1/responses` (0.1.39, #451) speaks OpenAI's newer Responses API, which Codex CLI uses (it no longer -speaks Chat Completions). It runs on the same path as `/v1/chat/completions`, so the thinking levels, the thinking -budget, the conversation cache and the same API key, Host and Origin checks apply. - -It is **stateless**: nothing is stored, so the client sends the whole conversation in `input` every time (Codex does, -with `store: false`). `previous_response_id`, `conversation`, `background` and the retrieve/delete/cancel endpoints -are refused with an error that says so. +`POST /v1/responses` speaks OpenAI's newer Responses API, which Codex CLI uses. It shares the chat path, so thinking levels, +budgets, the conversation cache and the API key, Host and Origin checks apply. It is **stateless**: the client sends the whole +conversation in `input` every time. `previous_response_id`, `conversation`, `background` and the retrieve, delete and cancel +endpoints are refused with an error that says so. | Request | What Strata does | | --- | --- | -| `input` as a string, or as items | `message` items (`user`, `assistant`, `system`, `developer`; text and images), `reasoning`, `function_call`, `function_call_output`, `custom_tool_call(_output)` | -| `instructions` | The system message (with leading `developer` messages; later ones become user messages, as on the chat path) | -| `tools` | `function` tools, `namespace` tools (the model sees `namespace.name`; calls come back with `namespace` and `name`), `custom` tools (one free-form `input` string). Hosted tools (`web_search`, `file_search`, ...) are left out: the model cannot run them | +| `input` as a string or items | `message` items (`user`, `assistant`, `system`, `developer`; text and images), `reasoning`, `function_call`, `function_call_output`, `custom_tool_call(_output)` | +| `instructions` | The system message (with leading `developer` messages) | +| `tools` | `function`, `namespace` and `custom` tools. Hosted tools (`web_search`, `file_search`, ...) are left out because the model cannot run them | | `tool_choice` | `"none"` hides the tools; anything else lets the model choose (it cannot be forced) | -| `reasoning.effort` | `none`/`minimal`, `low`, `medium`, `high`/`xhigh`; without it the config's `reasoning_effort` (0.1.44.1; 0.1.44 ignored it here), else the model's default (high) | -| `max_output_tokens` | The output cap (thinking included). Running out ends the response `incomplete` (`max_output_tokens`) | -| `text.format` | `json_schema` and `json_object` use the [JSON response formats](#json-response-formats) (checked, not constrained) | -| `thread_source` `thread_title` in `client_metadata["x-codex-turn-metadata"]` | Codex's thread-title request (opt-in: `"codex_thread_titles": true` in `strata-.json`, off by default), sent beside each user turn from another session with `tools: []`: answered without the engine as `{"title": ""}`, so the conversation keeps the prompt cache. The title is the user's line, not one the model wrote | -| `temperature`, `top_p`, `reasoning_budget_tokens`, ... | As on the chat path | +| `reasoning.effort` | `none`/`minimal`, `low`, `medium`, `high`/`xhigh`; without it the config's `reasoning_effort`, else the model's default | +| `max_output_tokens` | The output cap, thinking included. Running out ends the response `incomplete` | +| `text.format` | `json_schema` and `json_object` use the JSON formats above (checked, not constrained) | +| `client_metadata["x-codex-turn-metadata"]` thread-title request | Opt-in `"codex_thread_titles": true`: answered without the engine as `{"title": ""}` so the conversation keeps the prompt cache | -The model's thinking comes back as a `reasoning` output item with `reasoning_text` content (streamed as -`response.reasoning_text.delta`). This model writes no separate summaries, so `summary` is empty. With -`"include": ["reasoning.encrypted_content"]` the item also carries `encrypted_content`: an opaque string (base64, not -encrypted; the client already holds the text). Send the reasoning items back with the rest of the conversation, as -Codex does: their thinking goes back into the prompt, so it matches what the model wrote and the conversation cache -is reused. - -With `"stream": true` the events are the official ones, in order: `response.created`, `response.in_progress`, then -for each output item `response.output_item.added`, its deltas (`response.reasoning_text.delta`, -`response.output_text.delta`, `response.function_call_arguments.delta`), its `...done` events and -`response.output_item.done`, and at the end `response.completed`, `response.incomplete` or `response.failed`. -While a long prompt is read, a `response.in_progress` event goes out every 15 s, which Codex counts as activity. -Errors before the answer starts have the Responses form (`{"error": {"message", "type", "param", "code"}}`); after -it started they arrive as `response.failed`. +The thinking comes back as a `reasoning` output item with `reasoning_text` content. The model writes no summaries, so `summary` is +empty. With `"include": ["reasoning.encrypted_content"]` the item also carries an opaque string (base64, not encrypted). +Send reasoning items back with the conversation, as Codex does, so the prompt matches and the cache is reused. Streaming +events are the official ones, in order, ending in `response.completed`, `response.incomplete` or `response.failed`. A +`response.in_progress` event goes out every 15 s while a long prompt is read. **Codex CLI.** In `~/.codex/config.toml` (Windows: `%USERPROFILE%\.codex\config.toml`): @@ -1539,217 +728,108 @@ stream_idle_timeout_ms = 600000 # a first, long prompt can take minutes # env_key = "STRATA_API_KEY" # only if the server has an api_key: the variable holding it ``` -Then run `codex` (or `codex exec "..."`) as usual. Codex warns `Model metadata for ... not found` for a local model -name; that is expected. Measured with Codex CLI 0.160.0 and Q2_0 on an RTX 5070: its first prompt (instructions and -tool descriptions) was 9,443 tokens, read in 10 s; in a tool loop, each later turn reused about 96% of the prompt from -the cache and read only the new part in 1-2 s. On Windows, Codex's sandbox rejected every shell command in that test -until it was started with `-c 'windows.sandbox="unelevated"'` (a Codex setting, not Strata's). +Run `codex` (or `codex exec "..."`). Codex warns `Model metadata for ... not found` for a local model name; that is expected. +On Windows, Codex's own sandbox rejected shell commands until it was started with `-c 'windows.sandbox="unelevated"'` (a +Codex setting, not Strata's). -**Codex's compaction (opt-in: `"codex_compaction_cache": true` in `strata-.json`).** When the context fills up (or -on `/compact`), Codex sends the conversation once more with a request to summarize it, and with `tools: []`. The template -writes the tools at the top of the prompt, so that prompt would share only its first few tokens with the conversation the -engine holds and be read again from the start, at its longest. With the option on, Strata keeps which Codex conversation -sent the last prompt other than a compaction (`session_id` and `thread_id` from `client_metadata["x-codex-turn-metadata"]`) -and the tools that prompt was rendered with; a request Codex marks `"request_kind": "compaction"`, from that same -conversation and without tools of its own, is rendered with them. Only the prompt: for the output parser and in the -response the request's tools stay what Codex sent, none, so a tool call the model writes comes back as a plain function -call without its namespace or custom-tool form. It is one entry, replaced by each such prompt; a compaction of another -conversation, a request without that metadata (Codex before 0.140) or a restart renders the request as sent, and so does -one that would not fit the context with the kept tools. Off by default: the prompt of a compaction differs from the -one the client sent. The author measured, with Codex CLI 0.160.0 and an engine that only counts the shared prompt start, -that a compaction after an 85,000-token conversation reused 84,895 of its 84,997 tokens and read 102; without it, 41 of -80,683. - -## Tools from MCP servers - -The chat page can give the model tools from [MCP](https://modelcontextprotocol.io) servers, as LM Studio and Claude -Desktop do: reading your files, fetching web pages, searching, anything an MCP server offers. List the servers in -`strata-.json` under `"mcp_servers"` - the same shape as Claude Desktop's `mcpServers` block, which you can -also paste as it is (key `"mcpServers"`): - -```json -"mcp_servers": { - "files": {"command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "C:\\Users\\me\\Documents\\notes"]}, - "search": {"url": "http://127.0.0.1:3000/mcp", "headers": {"Authorization": "Bearer ..."}} -}, -"mcp": {"timeout_s": 60, "max_result_chars": 20000, "max_rounds": 8} -``` - -Or keep them in their own file and start the server with `--mcp-config path\to\claude_desktop_config.json` (a file -with an `mcpServers` block; add it to the `serve/server.py` line of your run script). Restart Strata after a change. - -- **A program** (`command`, `args`, optional `env` and `cwd`) is started by Strata and spoken to over its - stdin/stdout; `npx`, `uvx`, `python` and friends are found on `PATH` as usual (Node.js is needed for `npx` - servers). **An address** (`url`, optional `headers`) uses MCP's Streamable HTTP transport (the older SSE-only - transport is not supported). `"disabled": true` leaves an entry out. -- The servers start with Strata, in the background; the server window says what each one offers - (`MCP server 'files': 14 tools (...)`), or why it did not start - its tools are then left out and the chat works - without them. The Monitor tab lists them, and the Sampling drawer has **Use tools from MCP servers** (on by - default). A server that stops later is started again at its next call. -- In the chat each call shows as a small block (tool, arguments, result); the model reads the result and goes on, - up to `max_rounds` calls in a row per answer. A tool that fails or takes longer than `timeout_s` (default 60 s) - gives the model an `error: ...` result instead of ending the chat. Results longer than `max_result_chars` - (default 20,000 characters) are cut, with a note, before the model reads them. Stop stops a running tool too. -- Only the chat page uses them. API clients (omp, Claude Code, OpenAI and Anthropic SDKs) see the API exactly as - before and keep their own tools; a request to `/v1/chat/completions` opts in with `"strata_mcp": true` (it then - gets `strata_mcp` tool events in the stream). - -**Security.** MCP tools run on your PC with your user's rights, and **the model decides when to call them** - also -because of what it reads (a web page or a file can contain instructions). Give a filesystem server only the folders -it needs, prefer read-only tools, and don't add servers you don't trust. The tools can only be used from the chat -page itself (a request with another site's Origin or without a JSON content type is refused); if Strata is reachable -from other devices, set an API key. - -**Context extension past 262K (rope scaling, EXPERIMENTAL, off unless you pick it).** The model was trained on -262,144 positions (rotary base 1e7). Rope scaling rescales the rotation angles so that longer contexts stay usable, -with llama.cpp's types and flag names. `linear` is Position Interpolation: every angle is shrunk by the factor. -`yarn` keeps the high-frequency angles, interpolates the low-frequency ones, and adds the magnitude correction -that keeps the attention temperature where training put it. **Without the flags nothing changes:** an unscaled -run computes exactly what it did before the feature existed, bit for bit. Scaled contexts need proportionally -more VRAM/RAM for the KV cache and the rope tables (~13 KB and ~0.26 KB per token). - -What was measured (contributors' runs, RTX 5080 + IQ3_S, native path, in PR #84): per-position perplexity on the -same tokens, with only the scaling flag changed. At 293K tokens (1.12x the trained length), `yarn` with factor 2 -lowered the NLL by 0.18 nats against both `none` and `linear` 2 (2.04 vs 2.22 / 2.22). That is 3-4x the path noise -measured at the same length. `linear` 2 was indistinguishable from `none`. At 2.7K and 32K no arm separated from the -noise. Needle tests do not tell the arms apart: the unscaled model also finds a needle at 413K. Long real-document -Q&A worked with `yarn` 2 at 421K and `yarn` 4 at 714K (8/8 each), and a 1M-token `yarn` 4 run read end to end. -Taken together, use **yarn**. It is still experimental: the numbers come from one machine and one quant. - -- setup: `START-HERE.bat --setup --context 393216` asks nothing extra - it picks the method (yarn; one - question when run interactively) and derives the factor from the final context for you (final context / - 262,144, at least 1: 1.5 at 393K, 2 at 512K, 1 inside the trained range; `--rope-scaling`/`--rope-scale` - override; an explicit `--rope-scale` is kept as given even when it is too small for the context actually - served, so check it if you set one). An explicit `--rope-scaling none` for a context past - 262,144 is refused: the setup will not configure a run with the stock angles past the trained range. If - the RAM check reduces a chosen 384K/512K back inside the trained range, an omitted method adds no - scaling, and an explicitly chosen one stays at factor 1 - the trained angles, no expansion (not a - switch for rope as a whole: explicitly supplied rope settings keep their behavior). -- engine: `--rope-scaling none|linear|yarn`, `--rope-scale F`, and the raw ggml knobs `--rope-freq-base`, - `--rope-freq-scale`, `--yarn-orig-ctx` (default 262,144), `--yarn-ext-factor`, `--yarn-attn-factor`, - `--yarn-beta-fast` (32), `--yarn-beta-slow` (1). The model file's `rope.scaling.*` keys, when a - fine-tune ships them, are the defaults the flags override. - -The scaling is fixed for the whole run - the engine stores keys in its cache after rotating them, so one -cache must never mix two scalings, and there is no per-request form. Within the trained 262,144 a scaled -run is a slightly different model: the rescaled angles, and `yarn`'s magnitude correction, apply at every -position, not only past the trained end. That is why the setup turns scaling on only for a context past -262,144. Pictures read the same scaled table (their (t, h, w) positions feed it). That should work, but it is -unmeasured: all the runs above are text. +**Codex's compaction** (opt-in: `"codex_compaction_cache": true`). When the context fills up (or on `/compact`), Codex sends the +conversation once more with a summarize request and `tools: []`. The template writes tools at the top of the prompt, so that +prompt would be read again from the start. With the option on, Strata remembers which Codex conversation sent the last +non-compaction prompt (`session_id` and `thread_id`) and the tools it was rendered with, and renders a compaction request from +that same conversation with them. It is one entry, replaced by each such prompt. A compaction of another conversation, a +request without that metadata or a restart renders the request as sent. The prompt of a compaction then differs from the one +the client sent, which is why it is off by default. --- +## Tools from MCP servers + +The chat page can give the model tools from [MCP](https://modelcontextprotocol.io) servers, as LM Studio and Claude Desktop do. +List the servers in `strata-.json` under `"mcp_servers"`, the same shape as Claude Desktop's `mcpServers` block (key +`"mcpServers"` also works when pasted as is). Merge these fields into the existing config: + +```json +{ + "mcp_servers": { + "files": {"command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "C:\\Users\\me\\Documents\\notes"]}, + "search": {"url": "http://127.0.0.1:3000/mcp", "headers": {"Authorization": "Bearer ..."}} + }, + "mcp": {"timeout_s": 60, "max_result_chars": 20000, "max_rounds": 8} +} +``` + +Or keep them in their own file and start the server with `--mcp-config path\to\claude_desktop_config.json`. Restart Strata +after a change. + +- A **program** (`command`, `args`, optional `env` and `cwd`) is started by Strata and spoken to over stdin and stdout. Node.js is + needed for `npx` servers. An **address** (`url`, optional `headers`) uses MCP's Streamable HTTP transport (the older SSE-only + transport is not supported). `"disabled": true` leaves an entry out. +- Servers start with Strata. The server window says what each offers or why it did not start. The Monitor tab lists them, and + the Sampling drawer has **Use tools from MCP servers** (on by default). A server that stops later is started again at its next call. +- In the chat each call shows as a small block. The model reads the result and goes on, up to `max_rounds` calls in a row. A tool + that fails or takes longer than `timeout_s` gives the model an `error: ...` result. Results longer than `max_result_chars` are + cut with a note. Stop stops a running tool too. +- Only the chat page uses them. API clients see the API exactly as before and keep their own tools. A request to + `/v1/chat/completions` opts in with `"strata_mcp": true`. + +**Security.** MCP tools run on your PC with your user's rights, and the model decides when to call them, also because of what +it reads (a web page or file can contain instructions). Give a filesystem server only the folders it needs, prefer read-only +tools and add only servers you trust. If Strata is reachable from other devices, set an API key. + ## Manage Strata from your AI assistant (MCP server) -`tools/strata_mcp.py` is an MCP server for Claude Code, Claude Desktop, Cursor, VS Code, Codex and other assistants. -Once it is added, you can ask your assistant "install Strata for this PC", "start Strata" or "is Strata running?". -In Claude Code, add it with: +`tools/strata_mcp.py` is an MCP server for Claude Code, Claude Desktop, Cursor, VS Code, Codex and other assistants. Once it is +added you can ask your assistant to install Strata, start it or check whether it is running. In Claude Code: ```bash claude mcp add strata -- python C:\Users\you\Strata\tools\strata_mcp.py ``` -It has eight tools: status (the running model, what is installed, the hardware, a recommended size), the model -list, install, start, stop, logs, a speed test, and connection settings for other apps. - -Install runs `setup.py` with `--yes` in the background. Before it downloads anything, it shows the plan and waits -for your OK. Start and stop work like the run scripts and the server's own unload. The MCP server only ends -processes it started itself. It uses only Python's standard library, so it works before `.venv` exists. - -The config snippets for every client, the tool arguments and the safety rules are in -[docs/MCP_SERVER.md](MCP_SERVER.md). This is the opposite direction from -[Tools from MCP servers](#tools-from-mcp-servers) above, where the Strata model calls *your* MCP tools. +It has tools for status, the model list, install, start, stop, logs, a speed test and connection settings. Install runs +`setup.py` with `--yes` in the background and shows the plan first. The MCP server only ends processes it started itself and +uses only Python's standard library. Config snippets for every client, the tool arguments and the safety rules are in +[MCP_SERVER.md](MCP_SERVER.md). This is the opposite direction from [Tools from MCP servers](#tools-from-mcp-servers) above. --- ## Images (vision) -(Qwen3.6-35B-A3B reads pictures the same way, with its own `mmproj-F16.gguf`: [QWEN36.md](QWEN36.md).) +Qwen3.8-Flash-Next has a vision encoder (`mmproj-Qwen3.8-Flash-Next-BF16.gguf`, 0.9 GB). Qwen3.6-35B-A3B reads pictures the +same way with its own `mmproj-F16.gguf` ([QWEN36.md](QWEN36.md)). Pictures are **on by default** wherever a model can read +them. The app, `strata launch` and setup use one rule for the reader: -The model has a vision encoder: [`mmproj-Qwen3.8-Flash-Next-BF16.gguf`](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF) -(0.9 GB, a 27-layer ViT plus the projector into the language model). Pictures are **on by default** wherever a model can read -them. The app, `strata launch` and setup use one rule for the reader: the **CPU reader under 24 GB** of graphics memory (nothing is -taken from the card's memory, so the experts keep their room) and the **graphics card's reader from 24 GB**, on NVIDIA and AMD -cards; a Mac reads on its GPU (Metal) whatever its memory. You can switch pictures off (`--vision no`, or "No" in the app and in `strata launch`) or pick the reader per model -(`--vision gpu` or `--vision cpu`). The setup downloads the encoder and puts a small helper (`strata-vision`, from llama.cpp's -`mtmd` library) beside the engine; it adds it to your start script. Nothing else changes. Where a reader is missing, pictures -stay off and the screen says so: the Intel Arc engine has none yet. On AMD the card's reader is `strata-vision-hip`, a second program in the engine package; if it cannot start, the server -starts the CPU reader by itself and prints one line ([AMD_HIP.md](AMD_HIP.md)). - -| Encoder on | Time per picture | Cost | +| Graphics memory | Default reader | Cost | | --- | --- | --- | -| **GPU** (the default from 24 GB) | **0.1-0.5 s** (up to 1,024 image tokens) | ~1.4 GB of VRAM is kept free for it, so the expert cache is smaller: text output is a few % slower (table below) | -| CPU (the default under 24 GB) | about 3 s at 300 image tokens on 8 cores (6-13 s on 4 threads); more tokens take longer, in proportion (#767, #625) | nothing on the GPU | +| Under 24 GB (NVIDIA and AMD) | CPU reader | Nothing from the card, so the experts keep their room. About 3 s for a 300-token picture on 8 cores; more tokens take longer in proportion. | +| 24 GB or more | The card's reader | 0.1-0.5 s per picture (up to 1,024 image tokens). About 1.4 GB of VRAM is kept free, so the expert cache is smaller and text output is a few percent slower. | +| Mac | The GPU (Metal) | Whatever its memory. | -A picture becomes up to 1,024 tokens of the context (a 640x480 photo: 300). The same picture sent again, as chat apps -do on every turn, is encoded only once. +Switch pictures off with `--vision no` (or "No" in the app and `strata launch`), or pick the reader with `--vision gpu` or +`--vision cpu`. Setup downloads the encoder and puts a small helper (`strata-vision`, from llama.cpp's `mtmd` library) beside +the engine. Where no reader exists, pictures stay off and the screen says so (the Intel Arc engine has none yet). On AMD the +card's reader is `strata-vision-hip`; if it cannot start, the server starts the CPU reader and prints one line ([AMD_HIP.md](AMD_HIP.md)). -**More image tokens (0.1.39, #625):** `--vision-tokens N` at setup (`START-HERE.bat --setup --vision cpu ---vision-tokens 768`) sets the most tokens a picture becomes - `"max_tokens"` in the `"vision"` section of -`strata-.json`, which you can also edit by hand. More tokens keep more detail (small text, charts, screenshots) -and take longer to encode, on the CPU most of all; a setup run again keeps the value. +A picture becomes up to 1,024 tokens of the context (a 640x480 photo: 300). The same picture sent again, as chat apps do on +every turn, is encoded once. -**A minimum number of image tokens (0.1.40, #767):** `"min_tokens": N` in the `"vision"` section of -`strata-.json` (edit it by hand; a setup run again keeps it) is passed to the encoder as `--min-tokens N` -(mtmd's `image_min_tokens`): a small picture is scaled up to at least N tokens. llama.cpp's mtmd prints that Qwen-VL -models want at least 1,024 for grounding tasks (pointing, counting small items); the CPU default stays at 300 at most -and no minimum, because a larger minimum changes the image answers and costs encode time (about 3 s at 300 tokens on -8 cores, in proportion to the tokens). +### Vision settings (the `"vision"` section of `strata-.json`) -**A Q8_0 encoder (#625):** `"mmproj"` in the `"vision"` section can point to another mmproj file of this model, for -example a Q8_0 one (llama.cpp's `convert_hf_to_gguf.py --mmproj --outtype q8_0` makes one): the encoder's library -reads quantized weights, the file is smaller (590 MiB against 865 MiB for the BF16 one, from `llama-quantize -mmproj-Qwen3.8-Flash-Next-BF16.gguf mmproj-Qwen3.8-Flash-Next-Q8_0.gguf Q8_0`), and on the CPU it uses less RAM -(about 280 MiB less in the encoder) and can encode faster than BF16 at the default 300 tokens. Setup downloads the -BF16 file, and a setup run again keeps a file of your own that still exists. **Recommended for `--vision cpu`.** -Accuracy against BF16, per image token: the cosine of the embeddings is 0.997-0.999 on average (#625's report, with -a community Q8_0 file, at 300, 768 and 1,024 tokens), and 0.9988 and 0.9987 on two pictures with the file made by -the command above (300 tokens; the worst single token 0.94-0.96). Encode time above 768 tokens is the same as BF16's -within about 3%. Numbers on more pictures (charts, small text) are welcome in #625. +| Setting | Effect | +| --- | --- | +| `--vision-tokens N` at setup, or `"max_tokens"` | The most tokens a picture becomes. More keep more detail (small text, charts) and take longer, on the CPU most of all. A setup run again keeps the value. | +| `"min_tokens": N` | Scales a small picture up to at least N tokens (passed to the encoder as `--min-tokens`). Qwen-VL grounding tasks want at least 1,024. A larger minimum changes image answers and costs encode time. | +| `"mmproj"` | Points to another mmproj file of this model, for example a Q8_0 one (`llama-quantize mmproj-Qwen3.8-Flash-Next-BF16.gguf mmproj-Qwen3.8-Flash-Next-Q8_0.gguf Q8_0`): 590 MiB against 865 MiB, about 280 MiB less encoder RAM, and it can encode faster on the CPU. **Recommended for `--vision cpu`.** Embedding cosine against BF16 is 0.997-0.999. Setup downloads the BF16 file and keeps a file of your own that still exists. | +| `"cuda_device": N` | Runs the encoder on a spare card (numbered like `nvidia-smi`). Lower `--vram-reserve-mib` in `"args"` to 700 as well. The card needs code in the ready-made encoder (RTX 20/30/40/50). | +| `"elastic": true` (with `"vram_elastic": true`) | Starts the GPU encoder only when a picture arrives, keeping `"elastic_mib"` (default 2048) free, and stops it after `"elastic_idle_s"` (default 120) without one ([VRAM_ELASTIC.md](VRAM_ELASTIC.md#the-image-encoder-on-demand)). It turns `--kv-grow` off. | +| `--flash-attn on\|off\|auto` (`strata-vision`) | Flash attention is turned off on the CPU when ggml has AVX-512 (a build from source), because ggml's fallback kernel is several times slower and accumulates in FP16. Release builds use AVX2 and leave it on. | -**Flash attention on the CPU (#660):** ggml's fast CPU kernel for flash attention needs the encoder's head size (72) -to be a multiple of the vector width: 8 floats with AVX2, which the release builds use, but 16 with AVX-512, which a -build from source gets on a CPU that has it (Zen 4 and newer, some Intel). There ggml falls back to a kernel that is -several times slower and accumulates in FP16, so `strata-vision` turns flash attention off on the CPU when its ggml has -AVX-512 and leaves it at `auto` (on) otherwise; `--flash-attn on|off|auto` overrides that. A 1024x1024 picture -(1,024 image tokens) on a Ryzen 7 7700X, 8 threads, an F16 mmproj, one encode per fresh process: - -| Build | Flash attention | Encode | Peak RSS | Output vs FP32 attention | -| --- | --- | --- | --- | --- | -| AVX2 (as released) | on (default) | 8.3 / 9.9 s | 1,152 MiB | 1.2% | -| AVX2 | off | 15.2 / 14.0 s | 2,194 MiB | 1.1% | -| AVX-512 (from source) | on | 43.5 s | 1,152 MiB | 26% | -| AVX-512 (from source) | off (default) | 13.3 s | 2,195 MiB | - | - -The last column is the relative L2 distance of the embeddings from the AVX-512 run without flash attention; two builds -differ by about 1% anyway. - -**A spare GPU for the encoder (0.1.33, #408):** with a card the engine doesn't use, add `"cuda_device": 2` (numbered -like `nvidia-smi`) to the `"vision"` section of `strata-.json`: the encoder then runs on that card alone. Lower -`--vram-reserve-mib` in `"args"` to 700 as well, so the engine's cards keep that VRAM for the expert cache. The -encoder's card needs code in the ready-made encoder (RTX 20/30/40/50). - -**The encoder only while there are pictures (opt-in, `"elastic": true` in the `"vision"` section, with the config's -`"vram_elastic": true`; [VRAM_ELASTIC.md](VRAM_ELASTIC.md#the-image-encoder-on-demand)):** the GPU encoder does not start -with the server, so the expert cache keeps all of the VRAM. The first picture's turn makes the cache keep -`"elastic_mib"` (default 2048) free and starts the encoder there (a few seconds); later pictures use it as it is, and -after `"elastic_idle_s"` (default 120) without a picture it stops and the cache takes the VRAM back. Measured on an R9700 -32 GB (gfx1201, Linux, UD-IQ4_XS, the BF16 encoder at up to 1,024 tokens, #2189 for HIP's virtual memory): the encoder -took 1,339 MiB and started in 4.7 s (2.7 s the next time); a request with one picture took 8.6 s with the start and -5.0 s on a later cold start, and 2 / 5 / 10 pictures with the encoder running took 3.8 / 5.6 / 9.0 s (~0.45 s to encode -each, the same as on a second card). Without the key nothing changes. +On an NVIDIA card under 24 GB the CPU encoder is the default, because the GPU encoder takes ~850 expert slots of a 16 GB card's ~4,500 +(about 5% of decode). `--vision gpu` is kept with a note. ### Sending a picture -**Terminal chat:** type `/image `, press Enter, then type your question. - -``` -you> /image C:\Users\me\Pictures\receipt.jpg -(picture attached: receipt.jpg - now type your question) -you> What is the total on this receipt? -``` - -**OpenAI API** (an `image_url` part: a `data:` URL, an `http(s)://` URL or a local file path): +Terminal chat: type `/image `, press Enter, then type your question. OpenAI API: an `image_url` part (a `data:` URL, an +`http(s)://` URL or a local file path). ```python import base64 @@ -1762,523 +842,272 @@ r = client.chat.completions.create(model="strata", messages=[{"role": "user", "c print(r.choices[0].message.content) ``` -**Anthropic API:** an `image` block with a `base64` (or `url`) source, as usual. +Anthropic API: an `image` block with a `base64` (or `url`) source. JPEG, PNG, BMP, GIF, WebP, TIFF and AVIF work (the last +ones are converted to PNG first). Chat apps with image upload work the same way. -JPEG, PNG, BMP, GIF, WebP, TIFF and AVIF work (the last ones are converted to PNG first; agents such as omp send -WebP). Chat apps with image upload work the same way. - -### Speed with images on (4K context, measured) - -| Model | Prompt tok/s, images off | Prompt tok/s, images on | Output tok/s, images off | Output tok/s, images on | -| --- | ---: | ---: | ---: | ---: | -| **Q2_0** | 541 | 531 | 90.7 | 86.9 | -| **IQ2_XS** | 468 | 458 | 77.1 | 74.0 | -| **IQ3_XXS** | 411 | 401 | 64.7 | 59.5 | - -These numbers date from 2026-09-24 (an engine before 0.1.13). "Off" is the published setup; "on" runs with the encoder loaded on the GPU and its VRAM kept free, so ~1,000 fewer -experts fit in VRAM and the CPU computes a few more per token: 2-8% slower. For text, turning images on changes -nothing else (the output is bit-identical when the VRAM is the same). - -**On a 16 GB card (0.1.45, #2406).** The encoder's room is ~850 expert slots of a 16 GB card's ~4,500, and where most experts are -computed on the CPU that moves the hit rate. Measured on Linux with the config setup writes (IQ3_S on a Tesla P100, IQ3_XXS on an RTX -A4000, 65K context, KV 8-bit, 5 rounds of fresh engine starts, medians): GPU encoder vs CPU encoder, chat decode 31.1 vs 32.6 tok/s -(P100, +4.8%) and 46.0 vs 48.4 (A4000, +5.2%), 4K prompt 315.5 vs 316.0 and 1,022 vs 1,053 tok/s, 16K prompt unchanged / +0.8%; -expert slots 3,819 vs 4,522 (P100), 4,686 vs 5,435 (A4000). The "elastic" encoder (`"vision": {"elastic": true}` with -`"vram_elastic": true`; it frees the encoder after 120 s idle, but turns `--kv-grow` off) gave the CPU encoder's text speed at 65K context (32.5, 47.7 -tok/s) and 2.7 s per picture once loaded (first picture 4.5-5.8 s); at 196K context with q4_0 KV (where it cannot keep `--kv-grow`) it did not -(A4000: GPU 44.7, CPU 47.4, elastic 44.9 tok/s; +6% for the CPU encoder). A report on an RTX 4080 SUPER under Windows measured 35-44% of decode and -36-62% of prompt speed with the GPU encoder (VRAM free 61 MiB); that size did not reproduce on the Linux cards, so it may come from -Windows' VRAM management when the card is full (not measured by us). A picture on the CPU encoder takes 4-6 s (a 300-token picture) against 2.7 s on the P100's -GPU encoder (1.7 s on the A4000). Since 0.1.45 setup gives "images: yes" the CPU encoder on an NVIDIA card under 24 GB (pictures read at up to -300 tokens; `--vision-tokens 1024` for more detail); `--vision gpu` is kept with a note. - -A question about a picture (a 640x480 newspaper page = 300 image tokens, a 328-token prompt; the whole request, encoder -on the GPU, measured through the API): - -| Model | Answer ("MEN WALK ON MOON") | Picture encoded | Prompt | Output | -| --- | ---: | ---: | ---: | ---: | -| **Q2_0** | 1.9 s | ~0.1 s | 207 tok/s | 65-73 tok/s | -| **IQ2_XS** | 2.3 s | ~0.1 s | 173 tok/s | 50-60 tok/s | -| **IQ3_XXS** | 2.7 s | ~0.1 s | 144 tok/s | 41-50 tok/s | - -(Short prompts run below the 4K prompt speed: a 2,048-token chunk is where the prompt path is efficient. Short answers -run below the long-output speed: the first rounds have no draft yet.) - -**How it works inside:** the encoder turns the picture into rows of the same width as the model's word embeddings; -Strata puts them where the prompt has `<|image_pad|>` tokens and gives each one its 2-D position (row and column in the -picture; the model uses interleaved M-RoPE). Answers match llama.cpp's multimodal implementation token for token on our -test images. +**How it works inside.** The encoder turns the picture into rows of the width of the model's word embeddings. Strata puts them +where the prompt has `<|image_pad|>` tokens and gives each its 2-D position (row and column; interleaved M-RoPE). Answers +match llama.cpp's multimodal implementation token for token on our test images. --- -**Stager waits (0.1.40.2, `STRATA_STAGER_SLEEP`):** the prompt-staging threads sleep while they wait on Linux and spin on Windows, because spinning is a little faster there (a 5070 read an 8K Q2_0 prompt 1.2% faster). `STRATA_STAGER_SLEEP=1` makes Windows sleep too, which is the choice when sharing the machine matters more than speed: on a Ryzen 9 7940HS laptop with an RTX 4070 (IQ3_S, 64K context) sleeping waits read 5,914 and 22,305 token prompts 5-6% slower while the whole-machine CPU use fell from 77-90% to 21-25% (issue #1101, thanks to midhatn). `STRATA_STAGER_SLEEP=0` forces spinning on Linux. The output is the same either way. +## Speed settings -**Stager threads for a GGUF read in place (0.1.41, #1353):** a native pack read from its GGUF shards in place (UD-Q4_K_XL, a Q8_0 pack) copies each expert with 32 threads and a 128-deep ring when the bytes are page faults on an SSD. If the shards' pages are already in the page cache (a big-RAM box, a warm start) those copies are memory copies and the same profile ran 4-7x slower than the 4-thread default (a 124K-token Q8 prompt: 57 -> 350-410 tok/s with the default). The engine now checks a sample of the experts with `mincore` at the first prompt (Linux) and takes the RAM profile when 90% or more of their pages are resident, saying so in its log; a cold or partly cached GGUF keeps the SSD profile. `STRATA_STAGER_SSD=1` / `0` forces either profile; `STRATA_STAGER_THREADS` still wins. The bytes copied are the same. +Defaults first, then switches you can try. A switch you set yourself, to anything, is never overridden. "Output" means the +bits of the answer: a switch that changes rounding can change a greedy answer at a near tie. -## Short prompts: the CPU shares the experts (on by default on one NVIDIA GPU, `STRATA_PREFILL_CPU_SHARE`) +### On by default -A prompt chunk of a few thousand tokens (an agent's tool result, a test's output, a short follow-up) streams every routed -expert that is not in VRAM over PCIe, while the CPU pool that decodes sits idle and RAM already holds those experts. -`STRATA_PREFILL_CPU_SHARE=auto` hands the pool the experts few of the chunk's tokens route to, measures per layer how -long each side takes and gives the CPU the share at which both end together (`STRATA_PREFILL_CPU_SHARE=0.4` fixes a -share). `auto` also checks that sharing pays: it times layers with the share and without it (the first ones alternate until three comparisons are in, then one in 29 runs the other way) and shares only while the layers that share are the faster ones. Where sharing is slower (every share was, on an RX 7900 GRE with a Ryzen 7 5700X3D under ROCm, #1282), the experts stay on the GPU apart from those measuring layers. It is off unless you set it, and then the output is byte-identical to the build without it. +| Setting | Applies to | Off switch | What it does | +| --- | --- | --- | --- | +| `STRATA_PREFILL_CPU_SHARE` (`auto`, cap `STRATA_PREFILL_CPU_SHARE_MAX=1024`) | CUDA, one GPU, no `--batch`, prompt chunks under 1,024 tokens | `STRATA_PREFILL_CPU_SHARE=0` | The CPU pool computes the experts few of the chunk's tokens route to, while the GPU works on the rest. `auto` measures per layer how long each side takes and checks that sharing pays. Skipped with one log line when RAM or commit is short. Output changes in the last bits. Layer split, batch slots and HIP keep it off unless you set it. | +| `STRATA_ROUTE_TAIL_SKIP` (7) | CUDA, and HIP gfx1200/gfx1201; one GPU or a layer split | `STRATA_ROUTE_TAIL_SKIP=0` | A missed expert that every token of a verify window routes at rank 7 or lower is neither copied nor computed. Not applied when every expert is in VRAM, with `--batch` slots, with a peer tier or helper GPU, on other HIP and SYCL, or when 90% or more of the experts are cached (`STRATA_TAIL_SKIP_MAX_CACHED=` moves that line; it is 60% on an AMD layer split). Answers differ slightly from a run without it. | +| PCIe share from the CPU's speed | CUDA, one GPU, no `--pcie-frac` | `--pcie-frac N` always wins; `STRATA_PCIE_FRAC_DEFAULT=old` keeps the probe-only rule | The share starts at the probe's value and is refined in the first decode windows from the CPU pool's time per missed expert: `L / (L + 1.85 R)`, rounded to 0.05 and kept between 0.10 and 0.60, changed at most three times. Each change prints a `PCIe share:` line. On HIP it is the default for gfx1200/gfx1201; `STRATA_PCIE_FRAC_DEFAULT=auto` turns it on for other HIP parts. `STRATA_PCIE_AUTO_SPLIT=1` (HIP, opt-in) refines a two-stage layer split. | +| Chunk follows the host's memory | Linux file tier | `STRATA_PREFILL_HOST_PLAN=0` | See [Linux file tier](#linux-file-tier). | +| Fused int8 prompt experts | gfx12 (R9700): IQ3_S and IQ3_XXS packs. 12.0 NVIDIA cards with 80 or more SMs: IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS gate/up | `STRATA_PF_FUSED=0` | Replaces the MMQ prompt kernels. `STRATA_PF_FUSED=1` opts in every covered format on other cards (on HIP gfx11 and gfx1200/gfx1201 too; see [AMD_HIP.md](AMD_HIP.md)). | +| Bare `--prefill auto` takes 16384-token chunks | gfx12, Linux | `STRATA_GFX12_CHUNK=0` or `STRATA_GFX12_DEFAULTS=0` | An explicit `--prefill N`, `auto:N` or `STRATA_PREFILL_AUTO_MAX` is untouched. | +| Per-architecture bitwise sets | sm_86, sm_60, sm_120 | See [below](#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-decode-switches-on-by-default) | | +| `STRATA_ADAPT_CALLBACK_WARMUP` | Linux | `=0` | Starts the GPU host-callback thread before the CPU pool pins the host thread. | +| `STRATA_ADAPT_SIGNAL` | CUDA | `host` / `copy` | The overlap's per-layer "landed" signal is a stream-ordered copy into pinned memory on Linux and a host function on Windows. | +| `--ple-io direct` | all | `--ple-io ram` or `mmap` | The n-gram table is read unbuffered from the SSD. A startup probe warns when direct reads are slow (under 15K rows/s) but never switches by itself. `STRATA_PLE_AUTO_MMAP=1` brings back the automatic switch to `mmap`. On a rotational disk setup adds `--ple-io ram` when the RAM holds the table. | +| `STRATA_STAGER_SSD` | Linux | `STRATA_STAGER_SSD=1`/`0` forces a profile | A GGUF read in place copies experts with 4 threads when its pages are in the page cache (checked with `mincore` at the first prompt) and with 32 threads and a 128-deep ring when they are on an SSD. `STRATA_STAGER_THREADS` wins. | -**0.1.41: on by default** on CUDA builds with one GPU and no `--batch` slots, for prompt chunks below 1,024 tokens: unset behaves as `auto` with `STRATA_PREFILL_CPU_SHARE_MAX=1024`, and the engine prints one line at start saying so. Measured (interleaved whole-engine runs, 8 rounds, medians): 512 tokens -28..-34% and 1,000 tokens -20..-26% prompt time on an RTX 3060, a Tesla P100 and an RTX 5070, 8/8 pairs each; mean KL against the share off 0.004 (max 0.025). The answers can differ slightly from 0.1.40.3. **`STRATA_PREFILL_CPU_SHARE=0` turns it off and gives 0.1.40.3's exact bytes.** The layer split, batch slots and AMD (HIP) builds keep it off unless you set the variable; `STRATA_PREFILL_CPU_SHARE_MAX=3072` (the extension to larger chunks) also stays opt-in: it lost 2% at 2,048 tokens on the 5070. +`STRATA_STAGER_SLEEP=1` makes the prompt-staging threads sleep while they wait on Windows too (they sleep on Linux and spin on +Windows by default, which is a little faster). Use it when sharing the machine matters more than speed. `=0` forces spinning on Linux. -With it on, chunks below 3,072 tokens are staged after their routing (only those can hand the CPU a share) instead of -streaming every expert from 1,024 tokens on; `STRATA_PREFILL_CPU_SHARE_MAX=1024` keeps the old limit. The CPU takes -experts it reads from RAM as they are: the arena, the page-locked copy of the resident RAM mode, or the mapped -`experts.bin`'s pages in the file cache (`--mmap-experts`), not only page-locked ones. Serve without `--batch`; on a -layer split every stage has the pool and one stage at a time takes it for a chunk. +### Opt-in switches -**0.1.42: not armed by default when memory is short.** A 0.1.41 report (RTX 5090, RAM already tight) had chats failing and the engine restarting until `STRATA_PREFILL_CPU_SHARE=0` was set. The default is now skipped, with one line in the log saying why, when the RAM this process can get (the cgroup limit counts on Linux) is below the share's own buffers plus 3 GiB, when on Windows the commit limit (RAM + page file) has less than 4 GiB left beyond them, or when the experts it would read are file pages (the mapped `experts.bin` outside the GPU cache and the RAM copy) that the free RAM plus 3 GiB cannot keep. An explicit `STRATA_PREFILL_CPU_SHARE=auto` (or a number) is kept and only warns. The share's two page-locked buffers (about 50 MB) are now taken once, at their largest, and a failed allocation ends the share for that run instead of failing the request. +None changes the default answers unless the table says so. Each was measured against the default with interleaved off/on +pairs of whole engine runs; the numbers are in [History](#history--older-measurements), on the cards named there. Try +them on your own card. -When on, the CPU's rows are computed in the CPU's own activation format, so the output changes in the last bits (first -token KL against off: mean 0.006, max 0.026 nats over 22 prompts; about half of the 32-token greedy answers on 500 and -1,000-token prompts are identical, the rest part at a near tie after about 23 tokens). Chunks of 3,072 tokens and more -read the same either way. `auto` picks the share from wall-clock timings, so the same request can be split -differently - and answer differently - from one run to the next (a resumed conversation's re-read differed in 31 of -1,536 teacher-forced argmax tokens, #1684); a fixed share repeats, `0` gives the exact bytes of a run without it. +| Switch | Applies to | What it does | Verdict | +| --- | --- | --- | --- | +| `STRATA_ADAPT_OVERLAP=1` | One GPU, the whole arena in RAM, no `--adapt-async`, `--pipeline-windows`, `STRATA_ADAPT_LAG`, `STRATA_ADAPT_NOWAIT`, `STRATA_DMA_BATCH`; never on a layer split | The adaptive tier's copies land while the next window runs, with the same tokens. The startup log names the state and why it cannot run. | Gains on RTX A4000 and Tesla P100 (+4 to +8%), loses on an RTX 5070 under Windows and an RTX 5090 Laptop. Opt-in on every backend. With the per-card defaults set `STRATA_ADAPT_LAG=1 STRATA_ADAPT_NOWAIT=0` beside it. | +| `STRATA_ADAPT_LAG=2` | CUDA | The window takes the adaptive tier's swaps once they are two windows old. | Gains on a Tesla P100; on by default on sm_86 and sm_60. | +| `STRATA_STAGE_PIN=1` | GGUF read in place | Page-locks the stage buffers the cache fill copies from. Unset, it stays off: with it on, IQ3_S decode was corrupted after a 4096-token prompt in an earlier release gate. | Keep off until that is fixed. | +| `STRATA_PCIE_BALANCE=1` | CUDA, one GPU, no `--batch`, no `--pcie-frac` | Each verify layer reads over PCIe the number of its missed experts that balances the two sides. | Helps an RTX A4000 arena (+5.6%), hurts others slightly. | +| `STRATA_GDN_CHUNKED=1` | CUDA, 128 or more SMs by itself (`=2` forces it); HIP gfx1100-gfx1102, gfx1150, gfx1151 | The DeltaNet recurrence of the prompt in chunks of 32 tokens (CUDA) or 16-token WMMA chunks (HIP). Changes rounding. | 1.8x on the recurrence on an RTX 5090; slower on smaller cards. See [AMD_HIP.md](AMD_HIP.md#opt-in-chunked-prompt-gdn-on-gfx11). | +| `STRATA_FS_SLOTS=N` | CUDA | Foresight swap space: N VRAM slots per layer refilled with missed and predicted experts. | The smaller cache cost more than the slots returned in every box measured. Left off. | +| `STRATA_MMQ_DIRECT=1` | CUDA, native packs; needs a build with `-DSTRATA_MMQ_TAB=ON` (off by default because it costs 36-38 MiB of VRAM per GPU) | The prompt's MMQ reads resident experts in their cache slots instead of copying them. Same bits. | Faster "read in" time with every expert resident. | +| `--prefill-head-alias` | `--serve`, one GPU | The output head's VRAM holds the prompt buffers while a prompt is read ([PREFILL_HEAD_ALIAS.md](PREFILL_HEAD_ALIAS.md)). | For 8 GB cards. | +| `STRATA_EMB_REUSE_ACCOUNT=1` | CUDA | The planner counts the VRAM saved because the half-output buffer `bo` reuses the embedding buffer `emb`, so the auto chunk can grow. Changes prompt rounding. | Prompt about +3.9% on an RTX 3060, no KL measurement yet. | +| `STRATA_VERIFY_ASYNC_LAUNCH=1` | CUDA, single-request verifier | Runs the CUDA graph launch of a verify window on a separate host thread so the caller keeps serving CPU expert doorbells. A diagnostic for Compute Sanitizer runs. | | +| `STRATA_CPU_IQ_ALL_EXACT1=1` | AVX2 CPUs without AVX-512 | Strata's AVX-2 decode for one-token i-quant gate/up rows, with ggml's float order, so the output does not change. On by default on sm_86. | End to end +0.4 to +2.5%, so left off elsewhere. | +| `STRATA_Q2_BITPLANE=1` | AVX2 CPUs without AVX-512 | A bit-plane row kernel for Q2_0. Changes the last bits. | | +| `STRATA_NO_AVXVNNI=1` | CPUs with AVX-VNNI | Turns the AVX-VNNI row forms off (they give the same bits, 4-25% faster rows). | A diagnostic. | +| `STRATA_PREFILL_EQUAL=1` | CUDA | A prompt segment is read in chunks of equal size, not full chunks and a short last one. Changes rounding. | | +| `STRATA_OWNED_PRICE=exact` | CUDA | The cache sizing prices the prompt path's own buffers by their real allocation. | | +| `STRATA_IQ_MT_MIN=1`, `STRATA_IQ3S_MT1=1` | CPU | See [Draft layer](#draft-layer) (reproducible output; the second gives IQ3_S the multi-token kernel for one token on Alder Lake and later without AVX-512). | | +| `STRATA_ROUTE_PRIOR`, `STRATA_ROUTE_RESIDENT`, `STRATA_ROUTE_RESIDENT_MTP` | CUDA | Routing changes that add speed and also add KL. | Not recommended. | +| `STRATA_HOST_CORE=last\|sibling` (or `--host-core`) | Windows | `last` puts the host thread on the last physical core, away from the core that gets the GPU's interrupts. `sibling` keeps it on the first core's other hardware thread, so the workers keep every core. The default `first` leaves a hybrid CPU as it is. | One layer-split box went from a ±6% swing to steady and 8% faster. | +| `--pool-affinity measure` | Hybrid CPUs, native packs | Times the `all` and `p-cores` layouts at start (about a second) and takes the faster. The measurement is taken once; where the E-cores' speed changes while the engine runs, `--pool-affinity p-cores` is the steady choice. | | +| `STRATA_KEEP_EMPTY_TURNS=1`, `STRATA_TOPK_STREAM=0` | | See [TROUBLESHOOTING.md](TROUBLESHOOTING.md). | | +| HIP `STRATA_DENSE_MMQ=1`, `STRATA_HIP_ADAPT_KERNEL_COPY=1` | HIP | See [AMD_HIP.md](AMD_HIP.md#model-and-serving-configuration). | | -Prompt time, medians of 10 interleaved pairs (off / auto, ms, `--expert-cache 1500`): +### CUDA opt-in kernels (sm_80 and newer) -| machine | 512 tokens | 1,000 tokens | 2K / 4K / 16K | -|---|---|---|---| -| RTX 5070, Ryzen 5 7600, Q2_0 | 1,376 / 1,019 (-26%) | 1,788 / 1,392 (-22%) | unchanged | -| RTX 3060, Core Ultra 7 265, IQ3_XXS | 1,620 / 1,159 (-28%) | 2,196 / 1,770 (-19%) | unchanged | -| Tesla P100, Xeon E5-2690 v4, IQ3_XXS | 4,805 / 3,126 (-35%) | 6,821 / 5,085 (-25%) | unchanged | +Each was measured as an A/B on an RTX 3060 and an RTX A4000; the gain is small there and larger on the Turing cards the authors +measured. "Exact" means the 10-prompt and 4K/20K identity gate with the switch on equals the gate with it off. -Up to 3,072 tokens and with mapped experts, fresh prompts read after an 8K prewarm, medians of 5 interleaved rounds -(off / auto, ms; Ryzen 9 5900XT, 96 GB DDR4-3200, Windows 11, `--mmap-experts`, q4_0 KV; before this the share took no -expert here: none was page-locked): +| Switch | What | Answers | +| --- | --- | --- | +| `STRATA_ATTN_DAILY=1` | Decode attention, two cells per step | Exact; on by default on sm_86 and 12.0 | +| `STRATA_QSA_ROWS=1` ([QSAROWS.md](QSAROWS.md)) | Block scores of a verify window with queries in registers | Exact; on by default on sm_86 | +| `STRATA_EXPDEC=1` ([EXPERTDEC.md](EXPERTDEC.md)) | Gate/up of the window's resident experts with the q8_1 rows in shared memory | Exact | +| `STRATA_SIDE_MIXER=1/2` | The mixer's input-only work on graph branches | Exact; on by default on sm_86, sm_60 and 12.0 | +| `STRATA_B6_MMVQ_ROWS=1` | Exact multi-column MMVQ reusing the activation fragment across rows | Exact; on by default on sm_86 and sm_60 | +| `STRATA_QSA_SPLIT_ATTN=1` | Decode attention split across blocks on the int8 mma path (sm_80+, int8 KV) | Rounding level | +| `STRATA_HC_UPMIX=1` | Hyper-connection up projection and `gr_mix_r` in one kernel (sm_80+) | Exact | +| `STRATA_PROMPT_ATTN_IMMA=1` | Int8-KV prompt attention on INT8 tensor cores | Rounding level | +| `STRATA_HC_REQ8=1` | Hyper-connection projections as int8 plus a scale per 32 (more expert-cache slots) | Rounding level | +| `STRATA_QUANT_GATHER=1`, `STRATA_PREFILL_GROUP_GATHER=0` | Per-row gather for the MoE input quantization; the short-prompt grouped gather off | No gain measured; keep the defaults | -| machine | 512 tokens | 1,000 tokens | 2,000 tokens | 3,000 tokens | -|---|---|---|---|---| -| RTX 5070 Ti (PCIe 3.0 x8), Swift 1.5 IQ3_XXS (huihui-ai's build) | 3,322 / 1,706 (-49%) | 3,990 / 2,357 (-41%) | 5,413 / 3,354 (-38%) | 5,525 / 4,309 (-22%) | -| RTX 3060 + RTX 5070 Ti (layers 0-11 / 12-47, both PCIe 3.0 x8), IQ3_S | 3,452 / 2,037 (-41%) | 4,506 / 2,680 (-41%) | 6,376 / 3,869 (-39%) | 6,733 / 5,138 (-24%) | +`STRATA_DEQUANT_WIDE` (wide dequantization of Q4_K, Q5_K, Q6_K and IQ4_XS for the prompt's dense weights) is on by default on +sm_86 and sm_61, bit-exact. `STRATA_DEQUANT_WIDE=0` turns it off. -A coding agent's recorded conversation on the two cards (a 100K-token start, then 8 turns of 1-5K tokens of code, 128 -tokens written a turn, the same tokens read by both): 137.4 s off, 130.6 s auto (reading 121.1 -> 114.3 s). +### RTX 20 (Turing, sm_75) kernels (opt-in) -## Missed experts in a verify window: the tail is skipped (on by default on CUDA, `STRATA_ROUTE_TAIL_SKIP`) +Kernels by rafatxf (#2069-#2072, #2093, #2220-#2222), imanu86 (#2081) and xrip (#1821) for compute capability 7.5 are bitwise +equal to the ones they replace. Nobody on the team has an sm_75 card to confirm them, so they are off until asked for. +`STRATA_SM75_PERF=1` turns on every bitwise Turing kernel on a 7.5 card (no other card is affected). Each has its own variable +that wins in both directions: `STRATA_SM75_IL`, `STRATA_SM75_EXP`, `STRATA_GR_UP75` / `STRATA_GR_DOWN_TG`, `STRATA_SCORES_HALVE`, +`STRATA_TOPK_SPLIT`, `STRATA_TOPK_SAMPLE`, `STRATA_ATTN_V3`, `STRATA_Q2_WARP`, `STRATA_FUSE_WAIT=2`, `STRATA_QIL`, +`STRATA_TC_SM75`, `STRATA_PA_V1_PREFETCH`, `STRATA_SCORES_THREAD` (a thread-per-head block scorer; `=1` works on any CUDA card). +Options that change answers stay separate: `STRATA_SELECT_SIMT=1|2`, `STRATA_PA_V2T=1`, `STRATA_PF_FUSED=1 STRATA_PF_FUSED_SM75=1` +(Q2_0 pack, 7.5 card only) and `STRATA_BF16_TC=1` with `STRATA_XN_F16`. -When a card cannot hold every routed expert, most of a decode token's waiting time is spent on the experts that are not in VRAM: -their copy over PCIe or their rows on the CPU pool. In a verify window of several tokens, an expert that **every** token of the window -routes at the bottom of its top 10 (rank 7 or lower, the smallest weights) adds little to any of them. **0.1.42: on by default on CUDA builds**: -such a missed expert is neither copied nor computed, its contribution is zero, and the other experts keep the weight the router gave them -(no renormalisation). An expert that any token ranks above 7, and every expert already in VRAM, is served as before. +--- -**`STRATA_ROUTE_TAIL_SKIP=0` turns it off** and restores the answers of a build without it byte for byte (checked: `mg_norepeat` 10 of 10 and `exact_pp` 4096 and 20000 -tokens identical to 0.1.42 without the change, on the RTX 3060, the RTX A4000 alone and as a 2-GPU split, and the Tesla P100). Another rank -(`STRATA_ROUTE_TAIL_SKIP=8`, `=9`; 1-9 are accepted) is taken as given. The start-up line -`STRATA_ROUTE_TAIL_SKIP=7 is ON (default)` names the opt-out. -Since 0.1.43 the same default holds on HIP for gfx1200 / gfx1201 (RDNA4), under the same rules and with the same `STRATA_ROUTE_TAIL_SKIP=0` opt-out; other AMD parts are unchanged. +## CPU workers and thread placement -Where it applies: serve on a CUDA build with one GPU or a layer split (every stage skips by itself). Not applied, and no line printed: when every expert is -in VRAM (nothing is missed), with `--batch` / `parallel` slots (a window then mixes requests, and one request's skip would depend on another's -tokens: kept off until measured), with a peer-expert tier or a helper GPU, on HIP and SYCL builds, and (0.1.42.1, #1884) when 90% or more -of all the experts are in the GPU cache: the few misses are cheap there and the skip costs more draft acceptance than it saves (a Tesla V100 32 GB -with Coder IQ1_M, 94.8% of the experts cached, 99.9% decode hits: draft acceptance 70.1% to 68.2%, decode 1-2% slower than 0.1.41). A start-up line -says so; `STRATA_ROUTE_TAIL_SKIP=7` turns it on anyway, and `STRATA_TAIL_SKIP_MAX_CACHED=` moves the line (101 = never). On an AMD layer split the line is 60% (2x R9700 at 16 GB-equivalent: Q2_0 with 73% cached lost 6.1% story decode with the skip, IQ3_S with 45% cached gained 7-10%). The request log gets -`route tail skip: N missed experts skipped` (cumulative) while it is active. +### How many CPU threads (`--pool-workers`) -**Answers differ slightly from 0.1.41** (`STRATA_ROUTE_TAIL_SKIP=0` brings them back). Teacher-forced KL of the decode path (the default's own greedy text and the default sampled at T=0.8, 28 chat prompts of code, -reasoning, chat, six languages and long-document summaries, 220 tokens each, about 6,000 scored tokens per set, through the real verify windows with the MTP drafts): +Without the setting the engine picks the number of CPU expert workers from the CPU, not from the count of logical threads: one +worker per physical core (never an SMT sibling) except the core of the main thread, one core less on a CPU with 17 or more +cores, only the cores of the main thread's socket on a multi-socket machine, at most 32, no more than a container's CPU limit +allows, and on a hybrid CPU with more efficiency than performance cores the performance cores but one plus half of the +efficiency cores. The start line says what bounded the number (`CPU pool: 31 workers by default (32 physical cores, ...)`). +Workers wait by spinning, so a worker without a core of its own takes time from the others. A count in the config is always +used as given; when it is above what the CPU uses well, the engine prints a warning with the recommended number +(`strata cpu pool: warning: ...`). Setup writes no worker count except the hybrid CPU's, and gives the CPU vision encoder at most 16 +threads (`--threads` in the `vision` entry changes that). -| | RTX 3060 12 GB, IQ3_XXS | RTX A4000 16 GB, Q2_0 | -|---|---|---| -| KL(0.1.41 behaviour \|\| tail skip), mean, greedy / sampled text | 0.0058 / 0.0056 | 0.0032 / 0.0030 | -| p99, max (greedy) | 0.061, 0.29 | 0.035, 0.32 | -| top-1 token agreement | 97.7% / 97.5% | 98.2% / 98.2% | -| perplexity of the forced text, skip / no skip | x1.011 / x1.007 | x1.005 / x1.003 | -| KL of the first token after the prompt | 0.024 / 0.033 | 0.015 / 0.016 | -| the same engine run twice (the floor; adaptive tier and CPU/GPU split depend on timing) | 0.0006-0.0007 | 0.0007 | -| `STRATA_PREFILL_CPU_SHARE` on / off (already the default) | 0.0006-0.0007 (first token 0.002-0.0035) | 0.0005-0.0007 (first token 0.002-0.004) | -| Q2_0 against IQ3_XXS, the same text | - | 0.1006 (p99 1.08, top-1 90.6%) | +`--pool-tasks N` sets the target total number of row tasks in each batched CPU expert gate/up and down phase (the default `0` keeps +three per participating thread; values `1..4096` are capped by each phase's row count). More tasks can reduce imbalance between +cores and add scheduling overhead, so compare against `0` with the same worker count. Add `"--pool-tasks", "192"` to the `args` +list and restart. Setup's `--calibrate` does not tune it. -The effect is not even: code and reasoning prompts have a KL of 0.002-0.004, chat and non-English prompts 0.004-0.009. -Task check, greedy, thinking off (HumanEval every second problem, 82; 30 generated reasoning tasks), default / tail skip: -RTX 3060 76 and 78 of 82 / 78 of 82, reasoning 27 and 27 / 29; RTX A4000 76 and 76 / 76, reasoning 27 and 30 / 27. The runs do not separate the settings -(one run each; a change of 3 reasoning tasks is noise). +### Two sockets and NUMA -Decode speed, `STRATA_ROUTE_TAIL_SKIP` default against `=0`, 5 interleaved rounds on fresh engines, a story, a code answer and a 6K-token document, 200 tokens each, -median over the rounds (ratio of tokens/s, default over `=0`; rounds in which the default was faster): +The default uses the host's socket only. With 2 or more NUMA nodes the pool splits itself by node on Linux: one slice per +node, each with its own task counter, and each node's half of every CPU-side expert's rows is placed in that node's memory. The +default worker count under the split is the physical cores of both sockets minus the host core (minus one more from 16 workers +on, at most 32). An explicit `--pool-workers N` is spread alternately across the sockets. -| machine | story | code | 6K document | all three | -|---|---|---|---|---| -| RTX 3060 12 GB, IQ3_XXS, resident experts | 1.104 (5/5) | 1.197 (5/5) | 1.241 (5/5) | **1.156 (5/5)** | -| RTX A4000 16 GB alone, Q2_0 | 1.151 (5/5) | 1.135 (5/5) | 1.101 (5/5) | **1.129 (5/5)** | -| Tesla P100 16 GB, IQ3_XXS, `--pcie-frac 0.29` | 1.149 (5/5) | 1.182 (5/5) | 1.118 (5/5) | **1.146 (5/5)** | -| two RTX A4000, layer split (hit rate 99.7%) | 0.987 (0/5) | 1.004 (5/5) | 1.002 (3/5) | 0.998 (1/5) | +- It runs when Linux has exactly 2 nodes holding the pool's threads, balanced slices, pinned threads and no + `STRATA_EXCHANGE_ROTATE=1`; otherwise the log says `NUMA pool off: ` and the pool runs as before. One node is exactly + the pool as before and logs nothing. Windows reads the node count but does not split yet. +- `STRATA_NUMA_POOL=0` turns it off (also `off`, `false`, `no`) and wins over `STRATA_NUMA_SPLIT=1`. `STRATA_NUMA_SPLIT=1` asks for + the split on any machine; `=0` is off; `=test` builds the slices on any machine for scheduling tests, not for speed. +- Only native expert layers are split. If a node lacks free memory for its half, the log warns `NUMA placement at risk` and + after the load prints FAILED when a node holds under 95% of its slice. +- Do not wrap the server in `numactl --cpunodebind` when testing workers across sockets: it pins them to one node. +- Workers on both sockets paid off on one box and cost on another, so measure on yours: `--pool-workers N` with N above one + socket's cores, a fresh server per value, the same prompt, decode tok/s from the log. +- Check the IOMMU. On a distro kernel where it translates every DMA (`cat /sys/kernel/iommu_groups/*/type | sort | uniq -c` + shows `DMA-FQ`), pinned copies ran slower than with `iommu=pt` on the kernel command line. -The gain follows how much of a token is spent waiting for missed experts (hit rate 75-90% on the single cards in the profile); a 2-GPU -split that holds nearly every expert (hit rate 99.7%) gains nothing. It does not touch prompt reading. Switches that stay opt-in and are not part of this: -`STRATA_ROUTE_PRIOR` (adds speed, doubles the KL), `STRATA_ROUTE_RESIDENT` and `STRATA_ROUTE_RESIDENT_MTP` (the latter is not faster than the former). +### Which logical processors (`--pool-cores`, experimental) -## The PCIe share of the missed experts is set from the CPU's speed (CUDA, single GPU, no `--pcie-frac`) - -Of the experts missing from VRAM, a share goes over PCIe to the GPU and the rest is computed by the CPU pool; `--pcie-frac` sets the share. -Up to 0.1.41 the default came from the PCIe probe only (0.55 from 20 GB/s up), which ignores how fast the CPU is: with the route tail skip on, the -best share was 0.15-0.20 on an RTX 3060 (default 0.55), 0.20 on a Tesla P100 (default 0.29), and the RTX A4000 is flat from 0.15 to 0.40. - -Since 0.1.42, on CUDA with one GPU and no `--pcie-frac`, the share starts at the old value and is refined in the first decode windows from what the engine -already times: the wall time of the CPU pool per missed expert. With `R` = expert bytes / that time (GB/s) and `L` = the probe's link speed, the share is -`L / (L + 1.85 R)`, rounded to 0.05 and kept between 0.10 and 0.60; it is changed at most three times (after 48, 96 and 96 more windows, about 10 s of -decoding), and only by a full step. Each change prints one `PCIe share:` line. A request that sets its own `pcie_frac` is left alone. -**`--pcie-frac N` always wins; `STRATA_PCIE_FRAC_DEFAULT=old` keeps the 0.1.41 rule.** Not used with a layer split, `--batch`, a peer GPU, when every -expert is in VRAM, or on SYCL or on HIP other than gfx1200 / gfx1201 (RDNA4, where the measured share is the default since 0.1.43; `STRATA_PCIE_FRAC_DEFAULT=old` opts out; on the other HIP parts `STRATA_PCIE_FRAC_DEFAULT=auto` turns it on: an RX 7900 XTX showed no regression, no gain was claimed, #2224). The canonical Q2_0 pack uses it on CUDA since 0.1.43, on Linux by default and on Windows with `STRATA_PCIE_AUTO_CANON=1` (an RTX 5070 there: code +2.5%, a short answer -3.3%). The constant 1.85 is a fit to three machines, so it lands within a step of each machine's best, not on it. On HIP, `STRATA_PCIE_AUTO_SPLIT=1` (opt-in, 0.1.43) also refines a two-stage layer split: the pool speed is read once and each stage takes the share its own probed link gives (2x R9700 at 16 GB-eq, tail skip on: Q2_0 story +4.5%, code +9.9%, 4K +8.8%; IQ3_S story -2.1%, code +8.0%, 4K +23.3%; with three or more stages the pipelined loop does not run it). - -Measured (interleaved pairs of whole engine runs, new default against `STRATA_PCIE_FRAC_DEFAULT=old`, medians, route tail skip on): - -| card | the pool takes | share (old -> new) | decode, new / old | pairs faster | -|---|---|---|---|---| -| RTX 3060 12 GB, IQ3_XXS | 32-37 us per expert | 0.55 -> 0.20 | **1.300** | 5/5 | -| Tesla P100 16 GB, IQ3_XXS | 84-97 us | 0.29 -> 0.20-0.25 | 1.011 | 4/5 | -| RTX A4000 16 GB, Q2_0 | 97-115 us | 0.30 -> 0.30 (unchanged) | 0.999 (same setting) | - | - -Explicit-share runs are byte-identical to 0.1.41 (`mg_norepeat` 10 of 10, `exact_pp` 4096 and 20000). With the new default the answers differ a little -from the old default, as any change of the share does (CPU and GPU round differently): teacher-forced KL against the old default 0.0015 on the 3060 -(run-to-run floor 0.0013) and 0.0011 on the A4000 (floor 0.0012; at that time the share was 0.25). - -`STRATA_PCIE_BALANCE=1` (opt-in, CUDA, one GPU, no `--batch`, no `--pcie-frac`; #1548 by sergiywith / stuchapin909): instead of one share for every layer, each verify layer -reads over PCIe the number of its missed experts that balances the two sides. The spin kernels stamp the GPU's clock (the plan's arrival, the PCIe part's start, the wait for the CPU rows), -the host times its pool, and least squares with forgetting fit the pool's `a + c*n_cpu + d*n_pcie` and the GPU's `g0 + g*n_pcie`; the engine prints the fit (`pcie model`) at the end of `generate`. -Every 64th layer with a miss moves one expert to the other side so both fits stay measured. Unset, nothing changes (the logits equal the build before). Measured by us (5 interleaved pairs, box default flags, IQ3_XXS): RTX A4000 arena decode 48.45 -> 51.15 tok/s (+5.6%, 5/5 pairs), RTX 3060 resident 59.12 -> 58.15 (-1.6%), Tesla P100 37.95 -> 37.48 (-1.3%): opt-in, worth trying on A4000-class cards. - -## The prompt chunk follows the host's memory (Linux, file tier; on by default, `STRATA_PREFILL_HOST_PLAN=0` turns it off) - -When the experts that do not fit in VRAM are read from the model files through the page cache, the size of the prompt chunk decides how many times a long -prompt walks over those files, and how big the working set is each time (the chunk's activation buffers and the lent slots take RAM and VRAM from the -cache). Before engine 0.1.45.3 the chunk was always today's pick (8192 on most cards). The planner prices the host from what it can measure at start: the RAM copy and its -lent slots, the page cache that is left, the drive class (read from the GGUF path) and the PCIe link, and picks the chunk (and the ring) with the fewest -predicted bytes read per prompt. It prints one `chunk 8192 -> 3840, ring 96 -> 96: ...` line when it changes the pick, and `chunk 8192 kept: ...` when it does not. -A prompt that fits today's chunk keeps one chunk; a longer one is read in equal chunks of at most the plan's size (no stub at the end). No weight or expert -changes; only the rounding of the batched prompt path moves (first-window logits KL 0.002-0.012 against the fixed chunk, same top-1; the text matched in 30 of 30 -requests; a run at a fixed chunk is bit-identical to itself). - -Where it applies: a Linux file tier whose page cache is smaller than the file set at today's chunk. It does nothing, and says `kept`, when the GPU holds every -expert, when the file set fits the cache (the owner-class RTX 5070 box: 14.8 GiB set against 41.2 GiB of cache, 2024 vs 2035 tok/s at 20K), when the set can -never fit (a 16 GiB cgroup, 23.1 GiB set: 99.9 vs 98.9 tok/s at 20K, the fewest passes win), and on a fast NVMe where the bigger chunk wins even though it thrashes (P2003, 8192 kept). -Windows reads the file tier unbuffered (no page cache), so it keeps today's pick. The plan is made for a prompt of `--max-context` tokens (at most 32768, at least 4096). `STRATA_SSD_GBPS` overrides the drive speed, `STRATA_PREFILL_PLAN_TOKS` -the prompt speed (tokens/s) it assumes; `--prefill N` and `STRATA_PREFILL_RING` always win. - -Measured (RTX 3060 12 GB, IQ2_XS, 20 GiB of experts in RAM, cgroup 34 GB, SATA speed emulated at 450 MB/s, page cache dropped and a fresh engine per run, 5 interleaved -pairs, plan off against the planner's `8192 -> 3840`, medians): - -| prompt (tokens) | plan off, tok/s | planner, tok/s | ratio | pairs faster | drive read, GB (off -> planner) | decode after, tok/s | -|---|---|---|---|---|---|---| -| 4,101 | 138.1 | 138.2 | 1.001 (one chunk either way) | 3/5 (tie) | 12.8 -> 12.7 | 27.0 -> 28.2 | -| 20,400 | 334.0 | 822.3 | **2.46x** | 5/5 | 26.1 -> 2.7 | 25.4 -> 42.6 | -| 28,887 | 418.8 | 910.4 | **2.17x** | 5/5 | 29.5 -> 0.6 | 28.6 -> 45.7 | - -The same box with its real NVMe and a 32 GB cap picks 8192 again (bigger chunk 480 vs 327 tok/s at 3328, even though it thrashes), and the 16 GiB cap is unchanged. Identity gate (fixed flags, -`--expert-cache 1500`): the plan line says `kept`. The host model is `include/strata/program/prefill_plan.hpp` with `tests/program/prefill_plan_test.cpp`. - -## More opt-in switches measured for 0.1.41 (all off unless you set them) - -None of these changes the default answers; each was measured against the default with interleaved off/on pairs of whole -engine runs (a fresh engine per side, warmed, 200-token greedy answers on three fixed prompts, medians; "pairs faster" counts -the pairs in which the switched arm won). They are here so you can try them on your own card. - -- **`STRATA_GDN_CHUNKED=1`: the DeltaNet recurrence of the prompt, opt-in**. On CUDA it runs in chunks of 32 tokens - (sergiywith, #1372; WY form, FP32, - other bits than the default: first-token KL against the default was 0.002-0.009 nats on average, the same top token in - every prompt tried). The kernel needs 128 SMs in one wave, so by itself it runs only on a card with 128 or more - (RTX 5090: 1.8x on the recurrence). On smaller cards `STRATA_GDN_CHUNKED=2` forces it for a measurement, and it is - slower there: 0.73x on an RTX 3060 (28 SMs) and 0.87x on an RTX 5070 (48 SMs) at 2K-32K tokens (`gdn_rec_parity --bench`), - and the prompt time did not improve: RTX 5070 2K 1.01x, 8K 1.00x of the default's time; RTX 3060 2K, 8K, 16K 1.01x, no pair faster. - On HIP, gfx1100-gfx1102, gfx1150 and gfx1151 use 16-token WMMA chunks with FP16 operands and a fused output norm; other cards and short - chunks keep the existing path. It changes rounding too (first-token KL against the default 0.006-0.007 on average, the same top - token in 12 of 12 prompts); on a Radeon 8060S (gfx1151) the layer's recurrence is 1.6x faster and a 4K-128K prompt after a warm-up prompt 1.4-2.1%; see - [the HIP conditions and numerical checks](AMD_HIP.md#opt-in-chunked-prompt-gdn-on-gfx11). -- **`STRATA_FS_SLOTS=N`: the Foresight swap space** (q8atnight, #1348): N VRAM slots per layer, refilled on a copy stream - with the experts that just missed and with the ones the next layer's router predicts (`STRATA_FS_AHEAD=0` for the - misses only), so that an expert the cache lacks can be computed on the GPU instead of by the CPU. The model's own router - decides everything; the slots only hold copies of the same bytes (`STRATA_FS_VERIFY=1` reads every landed slot back and - compares it: 0 bad in 2,300+ reads on an RTX 5070). The slots are taken from the VRAM the expert cache would have - (raise `--vram-reserve-mib` by about N x 80 MiB), and in every box measured the smaller cache cost more than the slots - returned: decode 4 slots, median tok/s B/A (pairs faster): RTX 3060 0.95 (0/6) with misses only, 0.91 (0/6) with the - look-ahead; Tesla P100 0.97 (2/6) and 1.00 (4/6); 16 slots on the P100 0.96 (0/6); RTX 5070 0.98 (1/6). The reporter - measured the same on 2x RTX 3090 (the misses per layer are below one there, so the CPU round trip stays). Left off. -- **`STRATA_STAGE_PIN`: pinned stage buffers** (Zhong Uncle, #1237; **opt-in, `STRATA_STAGE_PIN=1`**. It was on by default in - development, but the 0.1.41 release gate found IQ3_S decode corrupted after a 4096-token prompt with it on, likely a stage - buffer recycled before its asynchronous copy finished; it stays off until that is fixed). Where the experts are served from the GGUF in - place (`--mmap-experts`, or too little RAM for the arena) the buffers the cache fill copies from are page-locked, so the - copy no longer goes through the driver's bounce buffer (about 0.4-0.6 GiB of pinned RAM, one buffer falls back to - pageable when the driver refuses). Decode, median tok/s on, off, 6 pairs of whole runs: Tesla P100 with the RAM capped at - 24 GB (cgroup) 13.9 -> 15.0 (+8%, 6/6 pairs faster); RTX 5070 `--mmap-experts` +1.2% (5/6); RTX 3060 `--mmap-experts` - -0.1% (5/6). Boxes that hold all experts in RAM never use the stage buffers. -- **`STRATA_ADAPT_LAG=2`: the adaptive tier's copies are waited for one window later** (#764). Decode, 6 pairs: Tesla P100 - (PCIe 3.0 x16) +3.5% (6/6 pairs faster), RTX 5070 +0.2% (4/6), RTX 3060 -1.3% (0/6), so it stays opt-in. - Where the tier does not wait for its copies (`STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG` above 1) the device's residency table now follows each eviction before the next window (a small patch kernel before every verifier launch; #1517 by sergiywith): before, the pool could compute an evicted expert on the CPU from a stale activation (`STRATA_ADAPT_CHECKRES=1` counts such layers; `STRATA_ADAPT_EVICT_SYNC=0` turns the fix off). The default (the tier waits) is untouched. -- **`STRATA_ADAPT_OVERLAP` (`=1` turns it on, on every backend; the startup log names the state and, when it cannot run, why): the adaptive tier's copies land while the next - window runs, with the same tokens.** The window takes the swaps at the same moment as with #463's wait (so the same experts run - on the GPU and the CPU in every window), but the copies go in layer order and the verify window holds a layer's plan back - until that layer's copies have landed, instead of the whole window waiting for the round (about 10 ms every fourth window on - an A4000, PCIe 3.0 x16). One GPU, the whole arena in RAM, no `--adapt-async`, `--pipeline-windows`, `STRATA_ADAPT_LAG`, - `STRATA_ADAPT_NOWAIT` or `STRATA_DMA_BATCH`. A layer split (two or more GPUs) never runs it, in any version: the startup log says - `off: not on a layer split` (0.1.43 printed that line only when the variable was set, and ran without the overlap on a split just the - same, so 0.1.43.1's opt-in default changes nothing there; measured on 2x A4000, #2281). On by default on CUDA on Linux since 0.1.43, opt-in on Windows and HIP; - opt-in on every backend since #2010, after an RTX 5090 Laptop lost 21% decode with it. What it is worth per box: an RTX A4000 (6 - interleaved pairs each, decode median tok/s: Q2_0 16 GB 62.5 -> 66.2 (+5.9%), Q2_0 as a 12 GB card 51.3 -> 54.2 (+5.7%), - IQ3_XXS 16 GB 48.6 -> 52.6 (+8.3%), IQ3_XXS as 12 GB 39.3 -> 41.5 (+5.5%), 6/6 pairs faster every time) and a Tesla P100 - (+4.4%) gain, but an RTX 5070 on Windows loses 2.6% on Q2_0 (IQ3_XXS no faster) and an RTX 5090 Laptop (WSL2, Core Ultra 9 - 275HX, native IQ3_XXS pack, `--expert-cache-per-layer`) loses 21%: 84.6 vs 106.6 tok/s medians of 5 generations of 256 - tokens from one build, the slowest of the fast run above the fastest of the slow one (#2010). -- **`STRATA_ADAPT_CALLBACK_WARMUP` (Linux, on by default since 0.1.43.1; =0 turns it off): start the GPU host-callback thread before the CPU - pool pins the host thread.** A callback thread created later can inherit the host's one-CPU affinity and compete - with the verifier spinning for its signal (#1975). The warm-up runs and synchronizes one no-op host callback before - constructing the pool; swap selection, per-layer callbacks and residency updates stay unchanged. On an RTX 5090 / - Ryzen 9 9950X, a paired 4K three-prompt test averaged 142.82 -> 207.75 tok/s. Other GPUs and drivers remain untested; - see the [reproducer, settings and limitations](../bench/results/2026-10-11-rtx-5090-callback-affinity/README.md). -- **`STRATA_ADAPT_SIGNAL=host`: the overlap's per-layer "landed" signal as a host function** (the 0.1.43 way). From 0.1.43.1 the default is a - 4-byte stream-ordered copy into pinned memory, which needs no driver callback thread: on a 20-core desktop (RTX 3060) the host - functions still cost ~7 ms per window with the thread started early, because the pool's workers occupy every other CPU. - **On Windows the host function stays the default** (0.1.44, #2188): under WDDM the copy and its signal waited in the - driver's command buffer while the window spun on the pinned int, and the first request stopped with "the adaptive - tier's copies for layer 45 never landed" (RTX 4070 SUPER, driver 617.42). `STRATA_ADAPT_SIGNAL=copy` asks for the copy - there; the engine now flushes the copy stream after queueing the swaps, which is meant to fix it but has not been run - on Windows yet. -- **`--prefill-head-alias` (`--serve`, one GPU): the output head's VRAM holds the prompt buffers while a prompt is read**, - and the head is read back in before the first generated token. It was not measured with the paired method above. On - an 8 GB RTX PRO 2000 Laptop (Windows, IQ3_XXS, 590 MiB reserve) it let the prompt path use 2,048-token chunks - instead of 512: prefill 292.5 -> 667.1 tok/s on a 29,854-token prompt (medians of 12 and 6 fresh-engine trials). - Reads of up to 512 fresh tokens cost about 0.6-0.75 s more per request, and the card had 0 MiB free while decoding. - Details, limits and the measurement: [PREFILL_HEAD_ALIAS.md](PREFILL_HEAD_ALIAS.md). - `STRATA_ADAPT_NOWAIT` or `STRATA_DMA_BATCH`; otherwise the log says why it is off. RTX A4000, 6 interleaved pairs each, - decode median tok/s: Q2_0 16 GB 62.5 -> 66.2 (+5.9%), Q2_0 as a 12 GB card 51.3 -> 54.2 (+5.7%), IQ3_XXS 16 GB 48.6 -> 52.6 (+8.3%), - IQ3_XXS as 12 GB 39.3 -> 41.5 (+5.5%), 6/6 pairs faster every time. -- **`STRATA_MMQ_DIRECT=1`: the prompt's MMQ reads resident experts in their cache slots** (CUDA, native packs; needs an - engine built with `-DSTRATA_MMQ_TAB=ON`, which is off by default). The prompt path multiplies its experts in groups of - 16 with llama.cpp's MMQ, which wants a group's weights in one buffer at a fixed stride, so every expert a chunk routes - to is copied there first (2.3-3.1 MB each, per chunk and layer), resident ones too. With this on, a Strata-side copy of - MMQ's outer scheduler (`src/prefill/moe_mmq_tab.cuh`) takes one pointer per expert instead, and a resident expert is - read where it is; streamed, peer-GPU and other experts are still copied. The launch (tile size, stream-k split, - fixup) is the one MMQ makes for the copied group and the tile code is llama.cpp's own, so the answers are the same - bits (`prefill_mmq_tab_test`: 874 products bitwise equal to the copied path, all nine types - Q2_0, IQ2_XXS, IQ2_XS, - IQ2_S, IQ3_XXS, IQ3_S, IQ4_NL, IQ4_XS, Q8_0 - and every tile size; the optional Q5_0 and K-quant experts get no - pointer table and keep the copy). Each sized cache slot keeps 256 bytes after its expert, because MMQ reads up to - 136 bytes past a 640-wide down matrix; they are zeroed before each group. That is about 2.3 MiB over 9,300 slots (one - slot fewer out of 6,998 on one RTX 3090). `STRATA_MMQ_DIRECT=2` keeps that layout but copies as before (an A/B - reference); 0 or unset is the default path, byte for byte. `STRATA_MMQ_DIRECT_LOG=1` adds each stage's count of experts - read in place to the log after every prompt. Under HIP, on a build without `STRATA_MMQ_TAB`, or when CMake turned - that option off (it does so, with a message, when the llama.cpp tree's `mmq.cuh` differs from the one the copy - mirrors), the variable is ignored. - **Why the build option is off:** the engine loads every kernel at start (`CUDA_MODULE_LOADING=EAGER`), and the nine - pointer-table instances (+15 MB of binary) take 36-38 MiB of VRAM on every GPU (sm_86), with the variable unset too - - expert slots every model would lose, as with `STRATA_MMQ_KQUANTS`. They also cost ~70% of the upstream MMQ instances' - compile time per CUDA arch (175 s of CPU at sm_86). Without the option the device code is byte for byte that of a - build without the feature. - Measured on 4x RTX 3090, UD-IQ4_XS, split `12,24,36` with `--trim-stage-weights` and `--batch 16` (every expert - resident), the engine driven directly, its "read in" time, medians of 8: a 170-token prompt 331 -> 251 ms (-24%) and - 369 -> 290 ms (-21%) in two series, a 1,217-token prompt 773 -> 634 ms (-18%) and 838 -> 675 ms (-19%). A hack that - skipped the copies altogether ran 225 and 577 ms, but it is no bound for an exact implementation: it changes the - routing (12,003 expert-layer entries fell to 777 on one prompt) and so does less work. On one RTX 3090 with most - experts streamed from RAM the copies of streamed experts stay, and the prompt time did not move (within 1.5%). - Exactness, with `STRATA_PREFILL_CPU_SHARE=0` and no adaptive swaps: the first decode window's logits after the prompt - (`STRATA_DUMP_FIRST_LOGITS`, fp32) and 32 greedy tokens were byte-identical between 2 and 1 (and 0) for prompts of - 180-2,619 tokens - on the 4-GPU split with `--prefill 2048` (chunks of 1,024 tokens and more: the streamed walk), - the same with 9 GB less expert cache per GPU (20-31% of the expert reads streamed, so groups mix in-place, copied and - mid-group flushed experts), and on one GPU with `--expert-cache-per-layer`; the 150 greedy tokens of each of - `tools/batch_test.py --n 16` prompts (solo and batch) were identical too, and to those of a build without the - option. The uniform cache (no profile) holds no resident expert when a native pack's prompt is read (serve needs the - profile, and a one-shot run fills the cache only while it decodes), so `expert_cache_slot_guard_test` covers it. - -## Which logical processors the CPU workers use, and `--pool-cores` (experimental, off by default) - -The CPU expert workers read the experts that are not in graphics memory, and their speed is set by how many physical cores -read memory at the same time. So the engine starts one worker per physical core, never two workers on the two hardware -threads (SMT siblings) of one core, and keeps the first core for the main thread (the one that waits for the graphics -card). The start-up log says where everything sits: +The engine starts one worker per physical core and keeps the first core for the main thread (the one that waits for the GPU). +The start-up log says where everything sits: ``` strata generate: CPU workers on logical processors 2,4,...,30 (15 workers, one per physical core; host on 0) ``` -The numbers depend on the system. Windows numbers the two threads of a core next to each other (core k is logical -processors 2k and 2k+1), so the workers sit on the even numbers. Linux usually numbers the first thread of every core first -and the siblings after them (a 6-core, 12-thread CPU: cores 0 to 5, their siblings 6 to 11), so the workers sit on the low -numbers. The engine does not guess: it reads the pairing from the system (Windows `GetLogicalProcessorInformationEx`, Linux -`/sys/devices/system/cpu/cpu*/topology`). Where the system does not say (some containers), the line says so -("the OS did not say which logical processors share a core") and the default count is half of the logical processors. The -line also says when two workers do share a core. `ps -L -o tid,psr,comm` on Linux, and Process Explorer or any tool that reads -thread names on Windows, show the workers as `strata-pool-N` and the main thread as `strata-host`; the other helper threads -carry names too (`strata-stage`, `strata-adapt`, `strata-io`, ...). +Windows numbers the two threads of a core next to each other, so the workers sit on the even numbers; Linux usually numbers the first +thread of every core first. The engine reads the pairing from the system (Windows `GetLogicalProcessorInformationEx`, Linux +`/sys/devices/system/cpu/cpu*/topology`) and says so when the OS does not tell it. Only workers and the main thread are pinned +(thread names: `strata-pool-N`, `strata-host`, and `strata-stage`, `strata-adapt`, `strata-io` for helpers), so a task manager +showing only the first logical processors busy does not mean the pool is misplaced. The CPU picture encoder is a separate process +that the OS places. -Only the workers and the main thread are pinned. Every other thread (the prompt copy threads, the adaptive tier's job thread, -the graphics driver's threads) is placed by the operating system, and the picture encoder is a separate process whose threads -are not pinned either. So a task manager that shows only the first few logical processors busy does not mean the pool is -misplaced: while the experts are in graphics memory the workers have nothing to do. On Linux the helper threads the engine -starts after the main thread is pinned inherit its single CPU; `--aux-cpus` (next section) moves them. +`--pool-cores LIST` (or `STRATA_POOL_CORES=LIST`, `"pool_cores": "2,4,6,8"` in the config, or `strata launch --pool-cores 2,4,6,8`) +pins the workers to exactly these logical processors, in this order. Use it to leave cores for other programs, use one chiplet +or test a layout. The list is numbers, ranges and `first-last:step` (`0,2,4,6`, `2-14:2`), as Task Manager and `lscpu` show them. +The worker count is the length of the list; `--pool-workers N` takes the first N entries. Nothing is refused. The engine warns +when two entries are the hardware threads of one core, when an entry is the host thread's processor, and when one is not a +processor the process may use (it is left out). The option moves threads only and never changes a result. The Qwen3.6 engine +(`strata-q36`) keeps its own `--pool-workers` and default placement. -What that looks like on a Ryzen AI MAX+ 395 (16 cores, 32 threads, Windows 11, Balanced power plan), with every expert in -graphics memory: snapshots of four runs of the engine showed the unpinned threads on logical processors 0 to 15 only (the first -eight cores, both threads of each) and 16 to 31 idle. Windows listed 16 to 31 as parked ("Parking Status" 1) while nothing -pinned ran there. The 15 workers sat on 2, 4, ..., 30 and did nothing. The CPU picture encoder (`strata-vision`, 16 threads -there) did the same: 16 threads on logical processors 0 to 15, 10.1 s for a 1024x1024 picture. Pinning those 16 threads to one -per core (2, 4, ..., 30) made it far slower (74 to 193 s), and 8 threads pinned to 8 real cores took 12.3 s, so the encoder -stays where the operating system puts it (the cause was not looked into; the parked second chiplet is the obvious suspect). +### Other threads (`--aux-cpus`, Linux, off by default) -With the experts on the CPU, the workers of the same machine were each at 100% on exactly the even numbers (the CPU pool alone -on random expert data, so only the placement and the memory traffic are real). Why the default is not "the first N logical -processors": 8 workers on logical processors 1 to 8 (four cores, both threads) read 67 GB/s, 8 workers on real cores -2, 4, ..., 16 read 94 GB/s, and the default 15 workers on 2, 4, ..., 30 read 146 GB/s (medians of three interleaved 10-second -runs; 31 workers on 1 to 31 read 123 GB/s). +A thread the engine starts after the host thread is pinned inherits the host's one CPU: the adaptive tier's job thread, prefill +helpers, the router look-ahead. `--aux-cpus` puts every thread that is neither a pool worker nor the host on spare CPUs. It +moves threads only and never changes a result. -**`--pool-cores LIST`** (or `STRATA_POOL_CORES=LIST`, or `"pool_cores": "2,4,6,8"` in the run config, or -`strata launch --pool-cores 2,4,6,8`) pins the workers to exactly these logical processors, in this order. It is for people who -want to choose the cores themselves (leave some for other programs, use one chiplet, test a layout); the default is what was -measured. The list is numbers, ranges and `first-last:step`: `0,2,4,6`, `2-14:2`. The numbers are the ones Task Manager and -`lscpu` show. The worker count is the length of the list; `--pool-workers N` takes the first N entries (more workers than -entries: the extra ones run unpinned). Nothing is refused; the engine warns: - -``` -strata cpu pool: --pool-cores: logical processors 0 and 1 are the two hardware threads of one core: two workers on it share the core's memory bandwidth and slow each other down (one per physical core is the default) -strata cpu pool: --pool-cores: logical processor 0 is also the host thread's: a worker there competes with the thread that waits for the graphics card -strata cpu pool: --pool-cores: left out: 99 is not a logical processor this process may run on -``` - -Processors the process may not use (not there, or outside a `taskset` / affinity limit) are left out; if none is left the -pool uses its default placement. A list that cannot be read (`two,four`) is said and ignored. The main thread keeps the first -core (`--host-core` still decides that), so leave that processor out of the list. The option moves threads only and never -changes a result. It applies to the main engine; the Qwen3.6 engine (`strata-q36`) keeps its own `--pool-workers` and the -default placement. - -## Thread placement: `--aux-cpus` (Linux, off by default) - -The CPU pool pins its workers (one logical CPU per physical core) and the host thread pins itself to the pool's reserved -core. A thread the engine starts after that inherits the host's one CPU: the adaptive tier's job thread, the prefill -helpers, the router look-ahead. CUDA driver threads created before pinning can run on any allowed CPU, the workers' -included; a lazily created host-callback thread can instead inherit the pinned host's one CPU (see the warm-up above). -`--aux-cpus` puts every thread that is neither a pool worker nor the host thread on spare CPUs. It moves threads only and -never changes a result. - -- **Which CPUs.** `--aux-cpus auto` (or `STRATA_AUX_CPUS=auto`): the SMT siblings of the host's core first, then the CPUs of - physical cores that have no pool worker. A CPU that shares a core with a worker is never taken; when no CPU is spare - the feature does nothing and the start-up line says so. A list (`--aux-cpus 24,26` or `24-27`) is used as given, minus - the host's CPU and the workers' CPUs, and only CPUs the process may run on. `--aux-cpus off` is the default. -- **How.** Threads the engine starts pin themselves first thing: the adaptive tier's job thread and its per-round - threads, the prefill copy threads and helpers, the router look-ahead, the PLE reader, the file readers, the stdin reader - and the watchdog. The threads someone else started (the CUDA driver's) are moved by a sweep over `/proc/self/task` - once the pool exists and again at the start of every request. The pool's workers and the host thread are never moved, - and neither is a thread somebody pinned to one CPU. An explicit `STRATA_ADAPT_JOB_CPU` still wins for the adaptive tier's - job thread. `STRATA_AUX_STAGER=0` leaves the prefill copy threads where they were started: with a one-CPU spare set up - to 32 of them (`STRATA_STAGER_THREADS`) would share that CPU, which can slow the read of a long prompt. -- **Per request.** `"strata_tune": {"aux_cpus": 1}` (`serve/server.py` forwards `1`, `0`, `true` and `false`) moves the - threads to the set; `0` puts every moved thread back to the placement it had. A request without the key goes back to the - start-up setting. With `--batch` the most recently admitted request's value holds, as for the other `strata_tune` keys. - An engine started without `--aux-cpus` plans the set anyway, so `1` takes `auto`. -- **What it prints.** At start: `strata aux cpus: threads that are neither pool workers nor the host go to CPUs 24 (on; - N threads placed so far)`, or `(off at start; a request's aux_cpus=1 turns it on)`. A request that changes the state - prints `strata serve: aux cpus ON/off for this request`. -- **Not covered.** Other processes (the Python launcher, a container runtime) and the engine's host thread. -- **Measured.** One machine: 2x RTX 3080 20 GB at 220 W, Xeon E5-2696 v4 (24 logical CPUs in the container, 16 pool - workers each on its own CPU), UD-Q4_K_XL, layer split 23, `--batch-mtp`, two slots, adaptive tier on. `auto` put the - host thread on CPU 2 and the helper threads on CPU 24. Same binary in every arm, only the flag differs. Two pairs of - whole engine restarts in the order `auto`, off, `auto`, off (12 solo decodes and 10 two-stream rounds per restart, - medians): solo decode 79.7 and 80.4 tok/s with `auto` against 72.1 and 72.4 off (off / `auto` = 0.904 and 0.900), two - streams in aggregate 90.8 and 90.7 against 80.8 and 82.8 (0.890 and 0.912). The same configuration restarted twice - moved solo by 1.009 and 1.005 and two streams by 0.999 and 1.024. Live on one engine, switching per request with - `strata_tune` and rotating the order (20 solo requests per arm, 16 two-stream rounds): off / on = 0.888 solo and - 0.917 two streams, against 0.985 and 0.990 for an A/A control of the same setting. The host thread's involuntary - context switches over a bench of about ten minutes were 43.8k and 42.8k with the flag off and 3.9k and 2.8k with - `auto`. Prompt reads (24K, 25K, 51K, 104K) do not depend on the flag: off is +0.7% to +1.5% against `auto`, inside the - restart noise. The mean and the p99 of the doorbell-to-flag time are not worse without the flag; what appears is rare - stalls above 100 us, in 7 and 8 of 12 requests off against 0 or 1 with `auto`. That preemption of the host causes the - loss is an inference; the counters do not prove it. The warm-up requests of each run were discarded. - -**NUMA-split CPU expert pool (0.2.2: on by itself when the machine has 2 or more NUMA nodes; off with `STRATA_NUMA_POOL=0`):** on a 2-socket Linux box it splits the CPU expert pool into -one slice per NUMA node, each with its own task counter and barrier, and at load it places each node's half of every -CPU-side expert's rows in that node's memory (first touch by threads pinned to that node; the model file's page cache is -released first), so a slice reads its own socket's memory. Under the split the default worker count is the physical cores -of both sockets minus the host core, minus one more kept free from 16 workers on, capped at 32 (22 on 2x12 cores); an explicit `--pool-workers N` is spread alternately across the -sockets. Measured on a dual Xeon Gold 6226 (2x12 cores, 64 GB, 4 of 6 memory channels per socket populated), RTX 3090, -PCIe 3.0 with the kernel's IOMMU in passthrough (`iommu=pt`), `orcarouter-qwen3.8-flash-next-uncensored-iq4_xs` (the -IQ4_XS pack) at 262K context, server decode median over 12 samples: 59.3 tok/s with the split against 54.05 for the 0.2.0 -default (11 workers, one socket), +9.7% (+14% and +11.5% in two earlier sessions with the IOMMU translating DMA). Without -the split, 22 workers gave 43.25 tok/s, and the split at 12 workers gave 51.45; prompt speed is unchanged. It needs Linux, exactly 2 -NUMA nodes and enough free memory on each node for its half: otherwise the log warns `NUMA placement at risk`, and after -the load it judges the placement from move_pages samples and prints FAILED when a node holds under 95% of its slice. Only -native expert layers are split (`ArenaExpertSource` and the `cudaHostAlloc` path are not, and the engine says so at start-up -for the arena path). The pool refuses the split with `STRATA_EXCHANGE_ROTATE=1` and when the two sockets' worker counts are -lopsided, logs the reason (`NUMA split off: ...`) and runs as usual. `STRATA_NUMA_SPLIT=test` builds the same slices by -alternating the threads between two slices on any machine, with no node lookup, no pinning requirement and no memory -placement; it is for testing the per-slice scheduling, not for speed. - -*0.2.2: when it runs.* The split was a contributor's opt-in (`STRATA_NUMA_SPLIT=1`, issue #3201, measured on one dual Xeon Gold 6226 box). From 0.2.2 the pool counts the NUMA nodes that have CPUs (Linux: `/sys/devices/system/node/node*/cpulist`; Windows: `GetNumaHighestNodeNumber`) and, with 2 or more and neither `STRATA_NUMA_POOL` nor `STRATA_NUMA_SPLIT` set, tries the split by itself. **One node is exactly the pool as before: nothing is read, nothing is logged, no worker count changes.** On 2 or more nodes the split runs only when it can (Linux, exactly 2 nodes holding the pool's threads, balanced slices, pinned threads, no `STRATA_EXCHANGE_ROTATE=1`); otherwise the pool keeps its old worker count and cores and logs why: `strata cpu pool: NUMA pool off: (N NUMA nodes found), the pool runs as before` (a 4-node machine, a container that sees one node's CPUs, Windows). When it runs the log says `strata cpu pool: NUMA pool on: N NUMA nodes found, one slice of the pool per node (STRATA_NUMA_POOL=0 turns it off)`, then the `NUMA split on: slice 0 = node ...` line. **Off switch: `STRATA_NUMA_POOL=0`** (also `off`, `false`, `no`); it wins over `STRATA_NUMA_SPLIT=1`, and on a 2-node machine logs `NUMA pool off (STRATA_NUMA_POOL=0)`. `STRATA_NUMA_SPLIT=1` still asks for the split on any machine (and keeps the contributor's worker counts when it cannot run), `STRATA_NUMA_SPLIT=0` is off. On Windows the node count is read but the split itself is Linux-only for now: a Windows machine with 2 nodes logs the reason and runs as before. The 0.2.2 default was built from the report and tested on one-node machines only (identity gate and speed A/B unchanged there; the node logic runs in `numa_pool_test` on made-up topologies); it has not been run on a two-socket machine by us. Log: `strata cpu pool: NUMA split on: slice 0 = node 0 (N threads), slice 1 = -node 1 (M threads)` at start, then `FileExpertSource: NUMA placement: node 0 (xx.x % of its slices), node 1 - (xx.x % of its slices); free before: ...; after the drop: ...; pages taken and page-locked in N s since the -placement started`. +- `--aux-cpus auto` (or `STRATA_AUX_CPUS=auto`): the SMT siblings of the host's core first, then the CPUs of physical cores + that have no pool worker. A list (`--aux-cpus 24,26` or `24-27`) is used as given, minus the host's and the workers' CPUs. + `--aux-cpus off` is the default. When no CPU is spare the feature does nothing and the start-up line says so. +- Threads the engine starts pin themselves first; threads started by someone else (the CUDA driver's) are moved by a sweep over + `/proc/self/task` once the pool exists and again at the start of each request. An explicit `STRATA_ADAPT_JOB_CPU` wins for the + adaptive tier's job thread. `STRATA_AUX_STAGER=0` leaves the prefill copy threads where they started. +- Per request: `"strata_tune": {"aux_cpus": 1}` moves the threads to the set; `0` puts them back. +- It prints `strata aux cpus: threads that are neither pool workers nor the host go to CPUs 24 (on; N threads placed so far)`. +- Not covered: other processes and the engine's host thread. The measurement is one machine: see [History](#history--older-measurements). --- -## Experimental speed projection (EXPERIMENTAL, off by default) + +## Experimental speed projection (experimental, off by default) -**This is an experiment, not a finished feature.** It ships with Strata but stays off unless you turn it on. +This is an experiment. It ships with Strata and stays off unless you turn it on. It is a 480 KB control vector for the original +Qwen3.8-Flash-Next (`data/experimental-speed-projection/`, not Swift 1.5). After each of layers 4-44 the engine removes one +direction from every hyper-connection stream: `h -= (h . v) v`. -A 480 KB control vector for Qwen3.8-Flash-Next (`data/experimental-speed-projection/`, see its README). After each -of layers 4-44 the engine removes one direction from every hyper-connection stream of the residual: `h -= (h . v) v`, -one unit vector `v` per layer, exactly as llama.cpp does with the package's `--cvec-mode project` patches. +**What it changes.** The vector's package describes it as a **refusal-direction projection**. With it the model declines far fewer +requests, and removing refusals removes a safety behavior, so you are responsible for what the model writes with it on. It also +shifts ordinary answers a little and costs 0.2-0.4% per token. What a chat's tokens/s does depends on the text, so measure it on +your own prompts. The Monitor marks every request ESP or stock. -**What it changes.** The vector's own package describes it as a **refusal-direction projection**: with it the model -declines far fewer requests (it reports 1 of 50 vs 50 of 50 on its test set), and removing refusals removes a safety -behaviour - you are responsible for what the model writes with it on. It also shifts ordinary answers a little -(measured below). It is not an optimization in the engine: on the same text it costs 0.2-0.4% per token. What a -chat's tokens/s does with it on depends on the text the model writes (length, repetition, how well the drafts land), -so measure it on your own prompts; the Monitor marks every request ESP or stock. - -**Turning it on (at setup).** `START-HERE.bat --setup` asks "Turn on the experimental speed projection?" (default: -no), or pass `--experimental-speed-projection on` (`off`, or a path to another vector GGUF). Only for the original -Qwen3.8-Flash-Next, not Swift 1.5. It writes these engine flags (llama.cpp's) into `strata-.json`: +**Turning it on.** In a source checkout, `START-HERE.bat --setup` asks "Turn on the experimental speed projection?" (default no), or pass +`--experimental-speed-projection on` (`off`, or a path to another vector GGUF). It writes these engine flags into `strata-.json`: ``` --control-vector-scaled \data\experimental-speed-projection\Qwen3.8-Flash-Next-experimental-speed-projection.gguf:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer ``` -The engine log then says `control vector mode = project, dir = per-layer, layers 4..44 (41 steered)`, and the web -app's About tab lists it. (`--cvec-mode add` is llama.cpp's stock additive mode, for additive vectors.) +The engine log then says `control vector mode = project, dir = per-layer, layers 4..44 (41 steered)`. A loaded vector is on for +every request unless it says otherwise: the Sampling drawer has a switch, and the API takes `"experimental_speed_projection": false` +(config default: `"sampling": {"experimental_speed_projection": false}`). Switching drops the conversation cache once. Switched +off, the output is token for token the stock model's. -**Per request.** A loaded vector is on for every request unless it says otherwise: the web app's Sampling drawer has -a switch, and the API takes `"experimental_speed_projection": false` in the request body (OpenAI and Anthropic; a -config default goes in `"sampling": {"experimental_speed_projection": false}`). Switching drops the conversation -cache once, since the model state was computed the other way. Switched off, the output is token-for-token the stock -model's. +--- -**Measured here** (Q2_0, fixed experts, 2,557 teacher-forced tokens of code, a document and a chat): the top-1 token -changes at 10% of positions, mean KL from the stock model 0.063 nats (max 4.1), perplexity +15% on code, +2.3% on -the document, +0.4% on the chat. Details: `bench/results/2026-09-27-esp/`. +## RTX 30 series (sm_86) and Tesla P100 (sm_60): bitwise decode switches on by default + +Switches that are bit-exact on every card are on by default per architecture (`include/strata/core/nv_defaults.hpp`, set before +the engine reads its switches; a switch you set yourself, to anything, is kept). The table is used when every visible card has +that one compute capability. The identity gate (10 prompts plus exact_pp 4K/20K, fixed flags) is identical with the sets on. + +| Architecture | On by default | Off switch | +| --- | --- | --- | +| sm_86 (RTX 3050-3090, RTX A2000-A6000) | `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_CPU_IQ_ALL_EXACT1=1`, `STRATA_SIDE_MIXER=2`, `STRATA_ATTN_DAILY=1`, `STRATA_GR_UP75=1`, `STRATA_GU_SWIGLU=1`, `STRATA_QSA_ROWS=1`, `STRATA_B6_MMVQ_ROWS=1` | `STRATA_SM86_PERF=0` | +| sm_60 (Tesla P100) | `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_SIDE_MIXER=2`, `STRATA_FUSE_WAIT=2`, `STRATA_B6_MMVQ_ROWS=1`, `STRATA_FUSE_DECODE=1`, `STRATA_GR_UP75=1` | `STRATA_SM60_PERF=0` | +| sm_120 (RTX 50 and RTX PRO Blackwell) | `STRATA_SIDE_MIXER=2`, `STRATA_ATTN_DAILY=1`, `STRATA_GR_UP75=1`, `STRATA_GR_DOWN_TG=1`, `STRATA_GU_SWIGLU=1` | `STRATA_SM120_PERF=0` | + +`STRATA_ADAPT_OVERLAP=1` does not run beside `STRATA_ADAPT_LAG` / `STRATA_ADAPT_NOWAIT`: set `STRATA_ADAPT_LAG=1 +STRATA_ADAPT_NOWAIT=0` with it. Switches that did not pay or were mixed are left out of every set: `STRATA_EXPDEC`, +`STRATA_GR_NORM_FAST`, `STRATA_RAM_ADAPT`, `STRATA_KV_HOST_REGISTER`, `STRATA_HOST_CORE`, `STRATA_GR_DOWN_TG` (sm_86), +`STRATA_DEQUANT_WIDE` beyond its own default. `STRATA_EXCHANGE_ROTATE` never reached READY on the RTX 3060 and +`STRATA_PCIE_BALANCE` is slower on a single card. + +On 12.0 cards (sm_120) these also apply: + +- The interleaved small-batch matvec (`native_mmvq_il`, [MMVQ_IL_TABLE.md](MMVQ_IL_TABLE.md)) runs 5 to 8 tokens, with a 12.0 table + and, on cards with 80 or more SMs, its own 2-4 token table. `STRATA_MMVQ_IL_WIDE=0` and `STRATA_MMVQ_IL_120=0` turn them off. +- With 80 or more SMs the native packs' prompt experts run on the fused int8 kernels (`STRATA_PF_FUSED`) for IQ2_S, IQ3_XXS, + IQ3_S and IQ4_XS gate/up. IQ2_XXS and IQ2_XS keep MMQ. Smaller 12.0 cards (an RTX 5070 with 48 SMs) keep MMQ because the fused + kernels were 4.7% slower on 20K prompts there. `STRATA_PF_FUSED=0` keeps MMQ. +- `--spec 4 --spec-min-p 0.5` stays the default depth. +- Setup (Linux, not WSL): when the RAM covers the model's need plus the n-gram table plus 16 GB (Q2_0 93 GB, IQ3_S 107 GB), the table + is kept in RAM (`--ple-io ram`), so a new prompt does not wait for its rows from the SSD. With RAM short, ram mode costs + prompt time and start-up time, which is why smaller PCs keep the SSD reads. + +A prompt chunk above 8192 (`--prefill auto:16384` or `auto:32768`) is not bit-exact on CUDA, so the default stays 8192 on +NVIDIA. It is exact on gfx12, where 16384 is the default. `STRATA_ROUTE_PRIOR=0.5` stays opt-in because it raises KL. --- ## Troubleshooting +[TROUBLESHOOTING.md](TROUBLESHOOTING.md) is the main page. These are the cases specific to the server and engine. + | Symptom | What to do | | --- | --- | -| `the NVIDIA driver is too old` | Update the driver (NVIDIA App or nvidia.com/drivers), restart, run `START-HERE.bat` again. | +| `the NVIDIA driver is too old` | Update the driver, restart, run setup again. | | Python or the build tools could not be installed | Install what it names (links are printed), then run it again. Everything already done is kept. | -| `port 8080 is already in use` | Strata is already running (look for its window), or another program uses the port: `START-HERE.bat --port 8081`. | -| `cudaHostRegister ... out of memory` in the log | Normal on Windows: the engine pins the experts in per-layer slices instead. Only a problem if the load then fails. | -| `ExpertCache: cudaMalloc(...) failed: out of memory` although VRAM is free | Windows' page file is off or tiny: every allocation on the graphics card is also charged to Windows' commit (RAM + page file). Set a fixed page file size, the same initial and maximum size (e.g. 65536 MB: System > About > Advanced system settings > Performance > Advanced > Virtual memory > Custom size), and restart: a page file Windows grows on demand ("System managed", or an initial size below the maximum) may not grow in time while the card's memory is charged (#60: "System managed" and 4096-32768 MB still failed, a fixed 64 GB worked). Since 0.1.19 the engine retries with a smaller cache instead of stopping, and setup warns about a page file under 4 GB (issue #60). | -| The first start takes minutes | It is reading 34-55 GB into RAM; the second start is faster while the files are in the OS cache. | -| The PC freezes for a few minutes at the start | Normal, most of all the first time (the server window says when it happens): the engine loads the experts into RAM, pins part of it for the GPU and sizes the expert cache. Wait; don't close the window. Still frozen after 10 minutes: restart the PC, close other programs, try again, or pick a smaller size. | -| `the engine stopped unexpectedly (exit code ...)` | The engine process ended mid-answer - usually out of RAM (Linux ends the biggest program: `sudo dmesg \| grep -i -E 'killed process\|out of memory'`). The next request starts it again by itself. If it repeats: close other programs or pick a smaller size. The server also warns at start when the model's experts leave less than ~6 GB of RAM for everything else. | -| Slow output, disk light busy | Not enough free RAM: close other programs, or choose Q2_0 / IQ2_XS. | +| `port 8080 is already in use` | Strata is already running, or another program uses the port: `strata launch --port 8081` (source checkout: `START-HERE.bat --port 8081`). | +| `cudaHostRegister ... out of memory` in the log | Normal on Windows: the engine pins the experts in per-layer slices instead. A problem only if the load then fails. | +| `ExpertCache: cudaMalloc(...) failed: out of memory` although VRAM is free | Windows' page file is off or tiny. Every allocation on the card is also charged to Windows' commit (RAM + page file). Set a fixed page file size, the same initial and maximum (for example 65536 MB), and restart. The engine retries with a smaller cache, and setup warns about a page file under 4 GB. | +| The first start takes minutes, or the PC freezes for a few minutes | The engine is reading 34-55 GB into RAM, pinning part of it and sizing the expert cache. Wait. The second start is faster while the files are in the OS cache. Still frozen after 10 minutes: restart, close other programs, or pick a smaller size. | +| `the engine stopped unexpectedly (exit code ...)` | The engine ended mid-answer, usually out of RAM (Linux ends the biggest program: `sudo dmesg \| grep -i -E 'killed process\|out of memory'`). The next request starts it again. If it repeats, close other programs or pick a smaller size. The server warns at start when the experts leave less than ~6 GB of RAM. | +| Slow output, disk light busy | Not enough free RAM: close other programs, or choose Q2_0 or IQ2_XS. | | `prompt ... exceeds the context` | The request is longer than the context you chose: run setup again with a bigger `--context`. | -| `the setup refuses --rope-scaling none for a past-trained context` | A context past the trained 262,144 needs the rotary angles rescaled (experimental rope scaling), and the setup will not configure one with the stock angles there. Let it pick (`START-HERE.bat --setup --context 393216` adds yarn and a covering factor), or pass `--rope-scaling linear` or `yarn` yourself. | -| Slower than the tables | The monitor plugged into the GPU and other GPU programs take VRAM from the expert cache; RAM running below its rated speed (enable EXPO/XMP in the BIOS) slows the CPU half. | -| `this server was started without the vision encoder` | The model was set up for text only: run setup again with `--vision gpu`. | -| Setup puts the model on an old drive after you moved or reinstalled Strata (#1070) | Setup remembers the data folder you used last in its settings file (`%APPDATA%\Strata\settings.json` on Windows, `~/.config/strata/settings.json` on Linux; the `data_dir` entry) and uses it again. Choose another place with `--data-dir DIR` (model files, packs, MTP layer) or `--models-dir DIR` (the GGUF files only), or delete the `data_dir` line; `--gguf-dir DIR` uses GGUF files you already have without copying. To move a model, copy its folder and run setup with the new `--data-dir`. | +| Slower than the tables | A monitor plugged into the GPU and other GPU programs take VRAM from the expert cache. RAM below its rated speed (enable EXPO or XMP in the BIOS) slows the CPU half. | +| `this server was started without the vision encoder` | The model was set up for text only: run setup again with `--vision gpu` or `--vision cpu`. | +| Setup puts the model on an old drive after you moved or reinstalled Strata | Setup remembers the data folder in its settings file (`%APPDATA%\Strata\settings.json` on Windows, `~/.config/strata/settings.json` on Linux; the `data_dir` entry). Choose another place with `--data-dir DIR` (model files, packs, MTP layer) or `--models-dir DIR` (the GGUF files only), or delete the `data_dir` line. `--gguf-dir DIR` uses GGUF files you already have without copying. | | A picture is refused or `cannot read the image` | The file is not a picture Pillow can open (JPEG, PNG, WebP, GIF, BMP, TIFF, AVIF work). | -| Pictures are slow (3-30 s) | The encoder runs on the CPU, which is the default under 24 GB of graphics memory (the card's reader is the default from 24 GB). To use the card's reader: run setup again with `--vision gpu` (needs ~1.4 GB of VRAM; it takes room from the expert cache). | -| A request never finishes: "reading the prompt", GPU "100%" at low power | The GPU ran out of VRAM (engines before 0.1.9 could end with ~30 MiB free at large contexts). Run `START-HERE.bat` once to get engine 0.1.9 or newer; the log then says `... MiB of VRAM free with everything loaded` (a few hundred) and names the `--vram-reserve-mib` to add if it is low. | -| Output much slower than the tables above, only with a large `--context`, and the start says a small `... MiB of VRAM free with everything loaded` | The expert cache does not fit the card at that context. With `--expert-cache auto` the engine shrinks it to fit; with a fixed number it does not check again after the slots are written, and on Windows the extra is put in system memory instead of failing, so only the speed shows it. Use `auto`, a smaller number, or raise `--vram-reserve-mib` (it is deducted before the cache is sized). See "The expert cache size is a budget" under Sharing the GPU. | -| Generation stops mid-answer, GPU "100%", one CPU core busy | Fixed in engine 0.1.12 (issue #29, a race in the CPU expert pool on big-VRAM cards). Since then a request that stops moving ends with an error instead of hanging (after 2 minutes; 1 minute from 0.1.13): the log says `no progress for ... s ... (issue #29)` with where it stopped, and the next request starts the engine again. If you see that line, please open an issue with it. Engine 0.1.13 adds a stall report under it (what every expert-pool thread and the GPU handshake were doing, memory and page faults) and, on Windows, a `strata-stall-.dmp` file with every thread's stack: attach both. (`STRATA_WATCHDOG_S` sets the time in seconds; 0 turns it off.) Engine 0.1.14 fixes the stall those reports found (issue #31: with the IQ packs the host could wait forever inside the NVIDIA driver while copying experts in a verify window; the experts are now copied by a GPU kernel, `--pcie-mode dma` restores the old way). | -| `no progress for 60 s ... reading the prompt` on Linux, and the stall report says `threads waiting on the disk (state D): 16 ...` | The engine waits for the drive, not a deadlock: the n-gram table is read at random (`--ple-io direct`), which a rotational disk cannot keep up with (#605). The engine warns at start when the table is on one; `--ple-io ram` (Linux, needs RAM for the table) or the model on an SSD fixes it. Setup adds `--ple-io ram` itself on a rotational disk when the RAM holds the table (0.1.39). | -| `the engine said nothing for ... s during the request` or `... did not finish the request after it was stopped (STOP)` | Issue #481: the engine and the server lost step (the engine waits for its next command, the server for the request's end; GPU at 0 %, nothing in the log). The server ends the engine after 300 s without a line from it during a request (while a prompt is read: each chunk may take three times the previous one's time, the first one up to its tokens at 50 tok/s more), the request ends with an error and the next request starts the engine again. `"engine_silence_s": 600` in `strata-.json` sets the time (0 = wait forever, as before). If you see it, please add the end of the engine log to #481. | -| `the engine said nothing for ... s and used no CPU or disk in that time (frozen ...)` | Issue #1317: a quicker version of the check above for an engine that is not slow but stopped (a deadlock inside a CUDA call, a driver stall): after 90 s without a line, if the engine process has also used no CPU time and moved no disk bytes in that time, the server ends it and the next request starts it again. An engine that is silent but still working is never ended by this check, the server prints one note (`... but is still working; it is not ended`) and leaves it to `engine_silence_s`. `STRATA_ENGINE_STALL_S=180` sets the time, `0` turns the check off (it needs `psutil`, which setup installs). | -| `out of memory: cudaFuncSetAttribute` in the log (IQ3_XXS, long prompt) | Fixed in engine 0.1.15: CUDA loaded a kernel's code when it was first needed, and mid-prompt there was no VRAM left for it. Run `START-HERE.bat` (Windows) or `./setup.sh` (Linux) once to update. | -| The desktop app or the Terminal says it could not find an engine, or that a model needs a newer engine; a Mac says an app is "damaged" | [TROUBLESHOOTING.md](TROUBLESHOOTING.md#the-desktop-app-and-the-terminal). | +| Pictures are slow (3-30 s) | The encoder runs on the CPU, the default under 24 GB of graphics memory. Run setup again with `--vision gpu` (needs ~1.4 GB of VRAM, taken from the expert cache). | +| Output much slower than the tables, only with a large `--context`, and the start shows a small `... MiB of VRAM free with everything loaded` | The expert cache does not fit the card at that context. Use `--expert-cache auto`, a smaller number, or a larger `--vram-reserve-mib`. See [The expert cache size is a budget](#the-expert-cache-size-is-a-budget). | +| A request ends with `no progress for ... s ... (issue #29)`, with a stall report under it | A request that stops moving ends with an error after 2 minutes, and the next request starts the engine again. `STRATA_WATCHDOG_S` sets the time in seconds (`0` turns it off). Open an issue with the line, the stall report and, on Windows, the `strata-stall-.dmp` file. | +| `no progress for 60 s ... reading the prompt` on Linux with `threads waiting on the disk (state D)` in the stall report | The engine waits for the drive: the n-gram table is read at random (`--ple-io direct`), which a rotational disk cannot keep up with. `--ple-io ram` (needs RAM for the table) or the model on an SSD fixes it. Setup adds `--ple-io ram` itself on a rotational disk when the RAM holds the table. | +| `the engine said nothing for ... s during the request` | The engine and the server lost step. The server ends the engine after `engine_silence_s` (300 by default), the request ends with an error and the next request starts the engine again. If it repeats, add the end of the engine log to issue #481. | +| `the engine said nothing for ... s and used no CPU or disk in that time (frozen ...)` | A quicker version of the check above for an engine that stopped, not slowed: after 90 s the server ends it. `STRATA_ENGINE_STALL_S=180` sets the time, `0` turns it off (needs `psutil`, which setup installs). | +| The desktop app or the Terminal could not find an engine, or a Mac says an app is "damaged" | [TROUBLESHOOTING.md](TROUBLESHOOTING.md#the-desktop-app-and-the-terminal). | | Anything else | The engine log is `strata-.log` in this folder. In the app: Settings, Engine, Diagnostics, View logs, Save logs. In the Terminal: `strata doctor`. | --- @@ -2287,37 +1116,21 @@ the document, +0.4% on the chat. Details: `bench/results/2026-09-27-esp/`.

memory tiers

-- **GPU (VRAM):** attention and DeltaNet mixers, the gated-residual weights, routers, shared experts, output head, the MTP - draft layer, the KV cache (from 64K: only its most-read part, the rest streams from RAM), and an **expert cache** that fills the rest of VRAM with the most-used experts (it adapts to - the conversation while you chat). -- **RAM:** all 24,576 experts, pinned. The CPU computes the experts that are not on the GPU **in place**, at the same time - as the GPU works on the cached ones (AVX-512 / AVX2 kernels, ggml's for the i-quants). -- **SSD:** the 28.8 GB n-gram table, a few rows per token, read unbuffered past the OS cache (`--ple-io direct`, the - default, made for SSDs; on a rotational disk `--ple-io ram` keeps the table in RAM, #605). A start-up probe warns when direct reads measure slow (under 15K rows/s) but never switches by itself (since 0.1.43.1; `STRATA_PLE_AUTO_MMAP=1` brings back the 0.1.42 switch to `--ple-io mmap`): the bar is calibrated for NVMe, a SATA SSD tops out at 10-14K rows/s with direct's 16 reads in flight, and on one (Linux, 77.8K-token prompt) mmap read 2.1K rows/s against direct's 14.4K (#2040), while on a Windows box with a slow mapped file mmap stalled prompts (#1984). The engine reads the table - in the format the GGUF has it: IQ4_NL (the default table), Q4_0, Q5_0, Q5_1, Q8_0, FP8 (E4M3 with a scale) or BF16. - Measured on 4,000 random rows against the checkpoint's own BF16 table, the mean per-row error is Q8_0 0.53%, FP8 2.64%, - Q5_1 3.78%, Q5_0 4.25%, IQ4_NL 7.60%, Q4_1 7.80%, Q4_0 8.55% (a Q8_0 table is 54 GB, an IQ4_NL one 28.8 GB). On the Q2_0 model, 700 teacher-forced tokens, the mean KL against the BF16 table is 0.0117 (Q5_0), 0.0121 (Q5_1), 0.0125 - (Q8_0), 0.0128 (FP8), 0.0140 (Q4_0) and 0.0146 (IQ4_NL), with the perplexity within 1% of BF16's either way: any change to the - table moves the 2-bit model by about 0.012, so the formats are hard to tell apart. Setup does not offer another table; - it only changes which GGUF the engine is given. -- **Speculation:** the model's own MTP layer drafts up to 3 tokens; one pass over all 48 layers checks them. 2.4-3.2 - tokens per pass on average. When the reply repeats the context (code edits, quoted text), **prompt lookup** (engine - 0.1.7) drafts up to 5 tokens from the earlier copy, but only where its measured acceptance and cost say it pays: - code edits 6-11% faster, other text unchanged. The drafts are checked like the MTP's, so the output is the same. -- **Prompts** are processed in chunks of up to 8,192 tokens (`--prefill auto`; 32,768 opt-in) with the experts - streamed to the GPU over PCIe. Since engine 0.1.39b (#583) the streamed ring is sized in bytes for the pack (a - native pack with bigger blobs gets fewer ring slots and keeps more cache slots) and `--prefill auto` picks the - largest chunk that keeps that ring full. It only gives up ring slots where 0.1.39's rule held the chunk at 4,096 - tokens or less; from 6,144 on it keeps 0.1.39's ring and only looks for a bigger chunk beside it (shrinking the - ring there measured slower: the Coder -12%), and a prompt that fits 0.1.39's chunk keeps 0.1.39's ring (one - chunk: a smaller ring only slowed it). RTX 5070, 32K prompts against `STRATA_RING_BYTES=0`, 2-3 interleaved - rounds: IQ3_XXS +18.5% at a 1,500-slot cache and +9% in another memory state, the Coder +3%, IQ3_S unchanged; 4K - prompts unchanged (-0.7% to +1.1%, bit-identical). Q2_0 (a fixed `--prefill 2048`) is not affected. It changes - the bits of a long prompt against 0.1.39 (the experts go through a different - mix of cached and streamed groups). Measured quality, teacher-forced over the next 2,001 tokens of a long document - against the FP16 prompt path (IQ3_XXS, `--prefill auto`): 8K prompt KL 0.042 (0.1.39: 0.054), top-1 agreement - 93.5% (92.2%); 32K prompt KL 0.020 (0.019), top-1 95.3% (95.0%) - the same band as before. `STRATA_RING_BYTES=0` - restores 0.1.39's ring, loan and chunk choice. +- **GPU (VRAM):** attention and DeltaNet mixers, the gated-residual weights, routers, shared experts, the output head, the MTP draft + layer, the KV cache (from 64K only its most-read part; the rest streams from RAM) and an **expert cache** that fills the rest of + VRAM with the most-used experts and adapts to the conversation. +- **RAM:** all 24,576 experts, pinned. The CPU computes the experts that are not on the GPU in place, at the same time as the GPU + works on the cached ones (AVX-512 and AVX2 kernels, ggml's for the i-quants). +- **SSD:** the 28.8 GB n-gram table, a few rows per token, read unbuffered past the OS cache (`--ple-io direct`). The engine reads the + table in the format the GGUF has it: IQ4_NL (the default table), Q4_0, Q5_0, Q5_1, Q8_0, FP8 (E4M3 with a scale) or BF16. Setup does not + offer another table. +- **Speculation:** the model's own MTP layer drafts up to 3 tokens and one pass over all 48 layers checks them, for 2.4-3.2 tokens + per pass on average. When the reply repeats the context (code edits, quoted text), prompt lookup drafts up to 5 tokens from the + earlier copy, but only where its measured acceptance and cost say it pays. Drafts are checked like the MTP's, so the output is the + same. +- **Prompts** are processed in chunks of up to 8,192 tokens (`--prefill auto`; 32,768 opt-in) with the experts streamed to the GPU + over PCIe. The streamed ring is sized in bytes for the pack, and `--prefill auto` picks the largest chunk that keeps that ring + full. `STRATA_RING_BYTES=0` restores the earlier ring, loan and chunk choice. The full story, with measurements, bottlenecks and what comes next: **[docs/paper/Strata-Paper.pdf](paper/Strata-Paper.pdf)**. @@ -2325,224 +1138,197 @@ The full story, with measurements, bottlenecks and what comes next: **[docs/pape ## Credits and licenses -Strata itself: [MIT](../LICENSE). The model files are not part of it; their licenses apply to them (below). +Strata itself: [MIT](../LICENSE). The model files are not part of it; their licenses apply to them. - Model: [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team; quantizations: - [ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF). - The Coder: [ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF) - (Apache-2.0 per its card); its support in Strata came from @pjgmobile's PR #54. - Swift 1.5: [ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF) - by UkisAI. Their licenses apply to the weights. -- [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT): the i-quant formats, the GPU dot products and - dequantizers transcribed in `src/kernels/cuda/iq_kernels.cu`, the CPU backend linked for the i-quant experts, the - `mtmd` library behind the image encoder (`tools/vision/`), and `gguf-py` used by the tools. See - `third_party/ggml/LICENSE`. + [ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF). The Coder: + [ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF) + (Apache-2.0 per its card); its support in Strata came from @pjgmobile's PR #54. Swift 1.5: + [ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF) by UkisAI. + Their licenses apply to the weights. +- [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT): the i-quant formats, the GPU dot products and dequantizers + transcribed in `src/kernels/cuda/iq_kernels.cu`, the CPU backend linked for the i-quant experts, the `mtmd` library behind the + image encoder (`tools/vision/`), and `gguf-py` used by the tools. See `third_party/ggml/LICENSE`. - Ideas from [Splash](https://github.com/incoai/splash), [ninfer](https://github.com/Neroued/ninfer) and [HyperQwen](https://github.com/syv-ai/HyperQwen); references in the paper. -- The web app's font: [Outfit](https://github.com/Outfitio/Outfit-Fonts) (SIL Open Font License 1.1, see - `serve/web/fonts/OFL.txt`). Its Monitor tab started from @code-martin's dashboard idea (PR #22). -- The experimental speed projection's vector (`data/experimental-speed-projection/`): Qwen Community License 1.0, - made from the model's activations (see its README). +- The web app's font: [Outfit](https://github.com/Outfitio/Outfit-Fonts) (SIL Open Font License 1.1, see `serve/web/fonts/OFL.txt`). Its + Monitor tab started from @code-martin's dashboard idea (PR #22). +- The experimental speed projection's vector (`data/experimental-speed-projection/`): Qwen Community License 1.0, made from the model's + activations (see its README). -### Start the API without loading the model +--- -`serve/server.py --engine strata --config strata-.json --lazy` (or `"lazy_load": true` in that -config) starts the lightweight HTTP API without starting the native engine or the vision encoder. The first -generation request, or `POST /load`, starts the vision encoder first when configured, then loads the engine through -the existing reload path, including `before_load` and `min_free_vram_mib`. This also works with image requests: -the load finishes before Strata encodes the image. `POST /unload` stops both processes. Eager startup remains the -default. +## History / older measurements -`POST /v1/load` and `/v1/unload` are JSON control aliases for integrations, accepting `{}` or -`{"model":""}` and returning model status. They require the configured API key, -`application/json`, and no foreign browser Origin. They return **409** while a request is active or queued, -and **404** for an unknown model. Existing `/load` and `/unload` behavior is preserved. `/api/health` aliases -`/health`; `/v1/status` exposes `loaded` and `auto_load`. The unloaded model remains discoverable. +Dated results. They were measured on the engine version and hardware named in each row. They are evidence for the defaults +above, not claims about the current release or other machines. Where a row says "default", the setting is now on or off as +described in the sections above. Raw data is in [`bench/results/`](../bench/results/). -Unloading and shutdown close the native engine's stdin after sending `QUIT`, allowing Windows' detached -stdin reader to see EOF. Cleanup waits for process exit before releasing handles; if forced shutdown still -times out, the server keeps ownership and reports an error rather than claiming the model was unloaded. +### Engine speed tables and tuning -### JSON response formats +| Engine | Measured on | Result | +| --- | --- | --- | +| 0.1.5 | RTX 5070 12 GB, Q2_0, 262K | KV streaming: 50.9 -> 62.6 tok/s output (1,589 -> 3,872 experts in VRAM); about +6% at 128K | +| 0.1.8 | Setup, `--kv q4_0` | 4-bit KV (Hadamard-rotated): about 4% faster at 128K, perplexity +8-12%. [`bench/results/2026-09-27-kv-q4`](../bench/results/2026-09-27-kv-q4/README.md) | +| 0.1.13 | RTX 5070 12 GB, 64 GB RAM, 32K prompt | Prompt chunks of 8,192, quantized MMQ kernels: Q2_0 572 -> 1,290 tok/s, IQ3_S 383 -> 1,208. Through the server (Q2_0, 128K context): 999 tokens 353 -> 438 tok/s, 6,927 tokens 529 -> 1,077, 28,584 tokens 584 -> 1,249. Output speed unchanged. [`bench/results/2026-09-28-prefill-speed`](../bench/results/2026-09-28-prefill-speed/README.md) | +| 0.1.14 | RTX 5070, 4K prompt, back to back | Output 88.5 tok/s (0.1.12: 85.7). Tables before 0.1.13: [`bench/results/2026-09-24-final`](../bench/results/2026-09-24-final/matrix.md); 0.1.14: [`bench/results/2026-09-28-speed-0114`](../bench/results/2026-09-28-speed-0114/README.md) | +| 0.1.25 | RTX 3090, Coder, 198K context, `--kv k8v4` | 99 instead of 85 tok/s output, same needle results, prompts 2-5% slower. On an RTX 2060 SUPER 8 GB at 128K: +780 expert slots over resident K8V4 | +| 0.1.36 (#136) | RTX 5070, Q2_0, fused int8 kernels (RTX 30 and newer) | Prompt 4K 1,294 -> 1,570, 32K 2,170 -> 2,653, 128K 2,123 -> 2,468 tok/s (+16-22%), teacher-forced KL 0.009 at 32K vs 0.012 before. Decode on thread-block clusters (RTX 50, sm_90+): Q2_0 output 4K 89 -> 93.5, 128K 64.5 -> 76.4 tok/s. `STRATA_PF_FUSED=0` keeps the previous prompt kernels; `STRATA_QSA_CLUSTER=0` and `STRATA_ARGMAX_MULTI=0` turn the decode kernels off. IQ2_XS prompts with `STRATA_PF_FUSED=1`: +12% at 4K, +3% at 32K | +| 0.1.38 | AMD RX 7900 XTX (gfx1100), ROCm 10.2 nightly, tuned hipBLASLt-100500 table ([#755](https://github.com/Niko1221/Strata/pull/755)), Ryzen 9 7900X 96 GB, IQ3_S, median of 3 | Prompt 1K 760, 4K 1,275, 32K 1,641, 64K 1,594, 128K 1,494 tok/s (32K-128K 8-28% faster than 0.1.22); output 65.1, 60.4, 64.4, 61.8, 59.2 tok/s | +| 0.1.30 | Ryzen 7600 (AVX-512) | `STRATA_IQ_MT_MIN=1`: IQ3_S decode -1 to -3%, others the same. IQ3_XXS, 3.6K prompt, 4 repeats: all three reproducibility switches 1 answer of 4, without `--pcie-frac 0` 2 of 4, defaults 2 of 4 | +| 0.1.27, 0.1.39 | RTX 5070 Q2_0 (CJK); RTX 5070 IQ3_XXS (fr); RTX 5090 | `--draft-vocab` CJK answers 15-38% faster; `cyrillic` 83 -> 109 tok/s (NVFP4 fork, RTX 5090); `fr` drafts accepted 0.51 -> 0.60 (English 0.61 -> 0.63, code 0.77 -> 0.78), 141 -> 158 tok/s in French on an RTX 5090 (IQ3_S, the reporter's number) | +| not stated | Serving without `--mtp`, 8 GB card (A/B) | Expert-cache slots 1,000 -> 1,678 | +| 0.1.40 | Radeon AI PRO R9700, `--vram-reserve-mib 17084`, Q2_0, `--max-context 196608`, 5 interleaved pairs | `--kv-grow`: slots 6,213 -> 7,968, decode 96.3 -> 98.5 tok/s (x1.023, 5/5). At `--max-context 262144` the default engine did not start; `--kv-grow` started with 7,877 slots (94.4 tok/s) | +| 0.1.40 | 4060 Ti + 5080 layer split, i9-14900KF | `--host-core last`: a decode that swung ±6% became steady, 8% faster on average | +| 0.1.41 | Core i7-13700K (8 P + 8 E), RTX 5090, 64 GB, Windows 11, Swift 1.5 IQ3_XXS, 128K | `--pool-affinity measure`: `all` (15 workers) 6.7 ms per verify window in the CPU pool, 148 tok/s; `p-cores` (7 workers) 2.2 ms, 188-191 tok/s. Over four starts of one day `all` read 14.8, 21.9 and 25.3 ms against 6.1 to 6.2 for `p-cores`, and once `all` won | +| 0.1.45 | Ryzen 5 5600X etc. | `STRATA_CPU_IQ_ALL_EXACT1`: per expert, one thread, IQ3_S -32% on Threadripper PRO 3975WX, -11% on Ryzen 5 5600X; Core Ultra 7 265 IQ3_S -60%. End to end: Xeon E5-2643 v3 + RTX A4000 CPU part -5% (decode +0.5%, +2.5% on story and code), Core Ultra 7 265 + RTX 3060 +0.4%, Threadripper + R9700 +0.4% | -`POST /v1/chat/completions` accepts `response_format: {"type":"json_object"}` or -`{"type":"json_schema","json_schema":{"name":"answer","strict":true,"schema":{"type":"object","properties":{"answer":{"type":"integer"}},"required":["answer"],"additionalProperties":false}}}`. -The schema must accept only JSON objects at its root: `"type":"object"`, or an `anyOf`/`oneOf` whose branches are all -object schemas (an `allOf` with an object member, or a local `$ref` to one, also counts), as apps written for -llama.cpp's `json_schema` send. A root that also allows an array, string, number, boolean or null is refused. Local -`#` references work; remote references are refused. -`json_schema` is checked with the Python package `jsonschema` when it is installed (`python -m pip install -"jsonschema>=4.23,<5"`; setup does not add it); without it the answer is only checked to be one JSON object, and the -server says so once. +### Images -This is **schema prompting followed by server validation**, not grammar-constrained decoding. One generation -is made per request, with no hidden retry. Successful responses contain a validated JSON object. Malformed JSON, -duplicate keys, non-finite numbers, schema violations and incomplete generations return **502** with -`error.code: structured_output_failed`; invalid request schemas return **400**. JSON formats combined with -tools/MCP are refused explicitly. Without `response_format`, ordinary text and tool behavior stays the same. +| Engine | Measured on | Result | +| --- | --- | --- | +| before 0.1.13 | RTX 5070, 4K context | Images off / on, prompt tok/s: Q2_0 541 / 531, IQ2_XS 468 / 458, IQ3_XXS 411 / 401; output tok/s: 90.7 / 86.9, 77.1 / 74.0, 64.7 / 59.5 (the encoder's VRAM costs ~1,000 experts, so 2-8% slower) | +| before 0.1.13 | Same card, a 640x480 newspaper page (300 image tokens, 328-token prompt), encoder on the GPU | Answer in 1.9 s (Q2_0), 2.3 s (IQ2_XS), 2.7 s (IQ3_XXS); picture encoded in ~0.1 s | +| not stated (#660) | Ryzen 7 7700X, 8 threads, F16 mmproj, 1024x1024 picture | Encode time and peak memory: AVX2 with flash attention 8.3 / 9.9 s, 1,152 MiB; AVX2 without 15.2 / 14.0 s, 2,194 MiB; AVX-512 with flash attention 43.5 s; AVX-512 without 13.3 s | +| 0.1.45 (#2406) | Linux, IQ3_S on a Tesla P100 and IQ3_XXS on an RTX A4000, 65K context, KV 8-bit, 5 rounds | GPU vs CPU encoder, decode 31.1 vs 32.6 tok/s (P100) and 46.0 vs 48.4 (A4000); expert slots 3,819 vs 4,522 and 4,686 vs 5,435. The elastic encoder matched the CPU encoder's text speed at 65K and not at 196K with q4_0 KV. A CPU picture takes 4-6 s (300 tokens), the GPU encoder 2.7 s on the P100, 1.7 s on the A4000 | +| not stated (#2189) | Radeon AI PRO R9700 32 GB, gfx1201, Linux, UD-IQ4_XS | Elastic encoder: took 1,339 MiB, started in 4.7 s (2.7 s the next time); a request with one picture 8.6 s with the start; 2 / 5 / 10 pictures with the encoder running 3.8 / 5.6 / 9.0 s | -Structured SSE buffers the answer while sending keep-alive comments. It emits content only after validation, -then usage/timings and `[DONE]`; failures emit an SSE error and `[DONE]` without invalid content deltas. -`/v1/status.structured_output` advertises the formats, validation method and buffered streaming behavior. +### Other GPUs (estimated, before engine 0.1.13) -### API request monitor +Not measured. Estimated from the runs above (same CPU, 64 GB RAM): the GPU part scaled by memory bandwidth, the CPU part by how +many more experts the card's VRAM holds. Treat as ±20%. Numbers are *prompt / output* tokens/s. The prompt figures predate 0.1.13, +which about doubled prompt speed on the measured card. -Off by default, since it keeps prompts and answers in memory: turn it on with `"api_monitor": true` in -`strata-.json` (or `serve/server.py --api-monitor`); otherwise nothing is recorded and the two endpoints below -answer 404. -Open `/api-monitor` to inspect API traffic without opening a chat. It shows the model state, safe -load/unload controls, active/queued requests, original request bodies, output, separate reasoning and -non-stream response bodies. Total wall-clock includes FIFO waits and automatic loading; load, queue, -first-token, prompt/output tokens and engine decode timing are shown separately. +| GPU | Model | 1K | 4K | 32K | 64K | 128K | 262K | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| RTX 5060 Ti 16GB | Q2_0 | ~341 / ~80 | ~472 / ~87 | ~501 / ~81 | ~492 / ~72 | ~476 / ~62 | ~435 / ~53 | +| | IQ2_XS | ~291 / ~80 | ~406 / ~77 | ~434 / ~63 | ~426 / ~62 | ~413 / ~51 | ~383 / ~47 | +| | IQ3_XXS | ~249 / ~66 | ~359 / ~65 | ~381 / ~56 | ~374 / ~54 | ~363 / ~45 | - | +| RTX 3090 24GB | Q2_0 | ~355 / ~128 | ~491 / ~140 | ~521 / ~130 | ~512 / ~115 | ~495 / ~100 | ~453 / ~85 | +| | IQ2_XS | ~303 / ~131 | ~422 / ~128 | ~451 / ~103 | ~444 / ~102 | ~430 / ~85 | ~398 / ~78 | +| | IQ3_XXS | ~260 / ~106 | ~374 / ~103 | ~396 / ~89 | ~390 / ~85 | ~378 / ~71 | - | -`GET /api/requests` returns compact summaries; `GET /api/requests?id=` returns one retained request. -Both use the existing API-key check. The monitor retains the newest **100 requests in memory** until restart, -with **262,144 characters per input/output/reasoning/response field** and visible truncation flags. The actual API -responses are unaffected. Headers are not recorded, and the monitor key is kept in this tab's session storage. -Treat request history as sensitive input/output when exposing Strata on a network: set an API key as above. -The page uses relative URLs and works through the existing host binding or a reverse proxy. +### Memory modes and the file tier -### Prompt buffers: `bo` shares `emb` (#1454) +| Engine | Measured on | Result | +| --- | --- | --- | +| 0.1.26 | Coder | Low-RAM mode: committed memory 36 -> ~13 GB, same answers | +| 0.1.31 | Coder | Mapped mode without `experts.bin`: 64 greedy tokens from `experts.bin` and from the GGUF gave identical tokens and logits | +| 0.1.31 | RX 6900 XT, UD-Q4_K_XL, budget 71 GiB with 7.5 GiB unused | The prompt path's lent slots keep their experts in RAM: 0.5-2.6 GB of file reads per agent turn and 1.9-2.8 GB from the SSD per fresh 8K-16K prompt went to zero, fresh prompts read 3-8% faster | +| 0.1.32 | RTX 5070 12 GB, 64 GB RAM | UD-Q4_K_XL with a RAM budget: 7-8.5 tok/s at N = 40 against ~3 before | +| not stated | Gen3 NVMe, RTX 5090, 32 GB, Q2_0 resident, 262K | Read-ahead at start: ready in 70 s instead of ~920 s; the fill went from 39 MB/s to 3.2 GB/s | +| 0.1.39 | 2x 16 GB GPUs, 32 GB RAM | `STRATA_ARENA_MMAP=1`: ~1 GB -> 25 GB available while serving | +| 0.1.40.2 | RTX 3060 and Radeon 780M, interleaved A/B, 10 pairs, 200-token greedy, page cache dropped | io prefetch with staging (`STRATA_IO_PF_STAGE=1`) vs fill: RTX 3060 32 GB IQ3_XXS +25.7%; 16 GB -32.6%; Radeon 780M 16 GB Q2_0 -19% | +| 0.2.0 | RTX PRO 5000 emulating a 12 GB card (`--expert-cache 4448`, Q2_0 GGUF in place, cgroup RAM limit), 3 interleaved pairs | Zero-copy, io prefetch and the decode gate vs 0.1.4x: 32 GB RAM decode 1.70 / 2.56 / 2.88x (story / code / doc8k; 34.0 / 28.0 / 21.7 -> 57.6 / 71.7 / 62.6 tok/s), prompts 1.54 / 1.91x; 16 GB decode 1.16 / 1.25 / 1.25x (the gate stays closed), prompts 1.63 / 1.68x. On a real RTX 3060 12 GB (32 GB cgroup) zero-copy alone gave 1.14 / 1.44 / 1.47x decode | +| 0.2.1 | RTX 3060 12 GB, IQ3_XXS `experts.bin` pack (43 GB), `--mmap-experts`, cgroup RAM limit, 5 pairs | io prefetch and the gate: 32 GB decode 1.38x / 1.21x / 1.33x (story / code / doc8k), prompts 4K 1.04x, 20K 1.06x. At 16 GB decode +3..8% but the cold 4K prompt 4-6% slower, so it stays off (room 38%) | +| 0.1.43 | RTX 3060 12 GB, 32 GB, Linux, IQ2_XS, 20.5 GiB experts in RAM, v0.1.43 as baseline, 47 frozen Korean requests replayed in fresh engines | `STRATA_RAM_ADAPT`: decode 43.26 -> 50.40 tok/s (+16.5%); file-tier demand reads per generated token 14.15 -> 2.67. One machine, one quantization | +| 0.1.45.3 | RTX 3060 12 GB, IQ2_XS, 20 GiB of experts in RAM, cgroup 34 GB, SATA emulated at 450 MB/s, 5 pairs | Prompt-chunk planner `8192 -> 3840`: 4,101 tokens 138.1 vs 138.2 tok/s (one chunk either way); 20,400 tokens 334.0 -> 822.3 tok/s (2.46x, drive read 26.1 -> 2.7 GB, decode after 25.4 -> 42.6); 28,887 tokens 418.8 -> 910.4 (2.17x). With a real NVMe and a 32 GB cap it picks 8192 again | +| 2026-10-11 reports | RTX 3090 + Ryzen 5 5600X | `--adapt-async 1` on the arena: x1.07-1.11 decode, 12 of 12 prompts faster ([report](../bench/results/2026-10-11-community-rtx3090-5600x-arena-async/README.md)). `STRATA_PIN_BY_NEED=1`: prompt read +5-7% at 56K-117K tokens ([report](../bench/results/2026-10-11-community-rtx3090-5600x-arena-pin-by-need/README.md)) | +| 2026-10-11 report | RTX 5090 / Ryzen 9 9950X, paired 4K three-prompt test | `STRATA_ADAPT_CALLBACK_WARMUP`: 142.82 -> 207.75 tok/s ([report](../bench/results/2026-10-11-rtx-5090-callback-affinity/README.md)) | +| 2026-10-04 report | 32 GB Windows 11 laptop, RTX 4080 Laptop + Thunderbolt RTX 3090, IQ2_XS, `--mmap-experts --layer-split 12 --trim-stage-weights` | `STRATA_FILE_RELEASE=1`: median available RAM 0.51 -> 12.19 GiB across 1K/4K/16K prompts, prefill 3.7-4.4% lower | +| not stated | Swift 1.5 IQ3_S, RTX 5090, 64 GB RAM, Windows, ~30 swaps per verify window | `STRATA_EXCHANGE_ROTATE=1`: 164 -> 188 tok/s on one 1,500-token text, 155 -> 178 over 23 coding-agent turns ([EXCHANGE_ROTATION.md](EXCHANGE_ROTATION.md)); RTX 3060 IQ3_XXS decode 58.96 -> 62.03 tok/s (+5.2%, 5/5) | -The prompt path's half-output buffer `bo` reuses the embedding buffer `emb`, which is dead after the first hyper-connection broadcast: T x 2560 floats less VRAM per chunk (320 MiB at 32768 rows). The planner still counts those bytes by default, so the auto chunk and the cache slots the prompt path borrows are exactly those of 0.1.40.3 and the output bits are unchanged. `STRATA_EMB_REUSE_ACCOUNT=1` lets the planner use the saved bytes: where VRAM limits the chunk it grows (RTX 3060, IQ3_XXS: 6400 to 6656 tokens, 1652 to 1670 borrowed slots, prompt about +3.9%). A different chunk changes the prompt path's rounding, so prompt residuals are not byte-identical to the default; greedy output matched in our runs. Opt-in until it has a KL measurement. +### Expert cache over-sizing (Windows) -**Fused int8 prompt experts on gfx12 (RDNA4, R9700): on by default for IQ3_S and IQ3_XXS packs.** `STRATA_PF_FUSED=0` turns them off (back to the MMQ prompt kernels, the previous bits). R9700, 1 card: prompt 4K +15.7%, 16K +13..+17%, 32K +19..+20%, a 96K prompt +16% (answers identical text); first-token KL against the FP16 prompt path over 96 prompts is equal to MMQ within noise (IQ3_XXS 0.00613 against 0.00683, IQ3_S 0.00416 against 0.00372). Other quants and other GPUs are unchanged; `STRATA_PF_FUSED=1` still opts in every covered format. +**Measured on:** RTX 4080 SUPER 32 GB, IQ3_S, engine 0.1.39, `--max-context 1048576` (yarn factor 4), fresh 43,969 / 45,670 / 47,956-token +prompts with 200 generated each, one engine start per row (configuration in `bench/results/2026-10-04-community-rtx-4080s-iq3s`; +issue #781). -**gfx12 (Linux): a bare `--prefill auto` may take 16384-token chunks.** Explicit `--prefill N`, `auto:N` and `STRATA_PREFILL_AUTO_MAX` are untouched; `STRATA_GFX12_CHUNK=0` or `STRATA_GFX12_DEFAULTS=0` keeps 8192. +| Expert cache | Slots | VRAM free at start | Decode tok/s | +| --- | ---: | --- | --- | +| `--expert-cache 8900` | 11,631 | 0 MiB (`LOW`, suggests `--vram-reserve-mib 1212`) | 13.7 / 14.7 | +| `auto` | 11,178 | 217 MiB (`LOW`) | 102.1 / 122.3 | +| `auto` with `--vram-reserve-mib 1500` | 10,766 | 1,075 MiB | 88.9 / 96.5 / 101.4 | +| `--expert-cache 7000` | 9,148 | 4,130 MiB | 94.1 / 97.3 | +| 11,631-slot cache at `--max-context 524288` | 11,631 | 299 MiB | 113.2 / 107.9 | -### CUDA diagnostics: a host thread for the verify-window launch +453 slots (0.86 GiB) separate 13.7 from 102 tok/s, and the slower run had the higher cache hit rate, so being at 0 MiB free is +what costs 7x. A second card, a 16 GB RTX 5080 at 786,432 context with UD-IQ4_XS, INT8 KV and `Prefer No Sysmem Fallback`: taking +`--vram-reserve-mib` from 770 to 0 grew the cache from 2,782 to 3,115 slots while decode fell from 40.9 to 16.6 tok/s. An RTX 4090 D +with `--batch 2` at 262,144 context saw 538 MiB at the free line become 395 MiB after the last capture (#2065). In #2049 a 64 GB PC with +`--resident-budget-gib 40`, an 8 GiB conversation cache and a 256K context stalled for more than 60 s five times while parking. -`STRATA_VERIFY_ASYNC_LAUNCH=1` runs `Verifier::run`'s CUDA graph launch on a separate -host thread. This lets its caller serve the CPU expert doorbells while an -instrumented `cudaGraphLaunch` blocks waiting for the graph. The caller joins the -launch and checks its real error before returning a frontier; an early service -failure releases the mapped GPU waits before joining. The switch is off by -default and applies to this single-request verifier entry point. Direct batch -and pipelined graph launches and the HIP/SYCL paths retain their existing scheduling. +### Defaults introduced by measurement -For Compute Sanitizer, set this variable before running the usual model command -under `compute-sanitizer --tool memcheck --error-exitcode 86`. Build with -`STRATA_BUILD_TESTS=ON` and run `host_graph_launch_test` for the CUDA graph/CPU -signal, early cleanup and `--errors` error-propagation checks. Its `--synchronous` -mode is the negative control under memcheck: an eight-second watchdog releases -the original launch/service deadlock and the test fails. +| Engine | Measured on | Change | Result | +| --- | --- | --- | --- | +| 0.1.41 | RTX 3060, Tesla P100, RTX 5070, 8 interleaved rounds | `STRATA_PREFILL_CPU_SHARE` on by default for chunks under 1,024 tokens | 512 tokens -28 to -34% and 1,000 tokens -20 to -26% prompt time, 8/8 pairs each; mean KL 0.004 (max 0.025). Prompt time (off / auto, ms, `--expert-cache 1500`, 10 pairs): RTX 5070 Q2_0 1,376 / 1,019 at 512 tokens, 1,788 / 1,392 at 1,000; RTX 3060 IQ3_XXS 1,620 / 1,159 and 2,196 / 1,770; Tesla P100 IQ3_XXS 4,805 / 3,126 and 6,821 / 5,085; 2K, 4K, 16K unchanged | +| not stated | Ryzen 9 5900XT, 96 GB DDR4-3200, Windows 11, `--mmap-experts`, q4_0 KV, 5 rounds | Share up to 3,072 tokens with mapped experts | RTX 5070 Ti (PCIe 3.0 x8), Swift 1.5 IQ3_XXS: 512 / 1,000 / 2,000 / 3,000 tokens 3,322 / 1,706, 3,990 / 2,357, 5,413 / 3,354, 5,525 / 4,309 ms (-49, -41, -38, -22%). RTX 3060 + RTX 5070 Ti layer split IQ3_S: -41, -41, -39, -24%. A coding agent's conversation (100K start, 8 turns): 137.4 s off, 130.6 s auto | +| 0.1.42 | RTX 3060 12 GB IQ3_XXS; RTX A4000 16 GB Q2_0; 28 chat prompts, ~6,000 scored tokens per set | `STRATA_ROUTE_TAIL_SKIP` on by default (CUDA) | KL against 0.1.41 behavior, greedy / sampled: 0.0058 / 0.0056 (3060), 0.0032 / 0.0030 (A4000); top-1 agreement 97.7% / 97.5% and 98.2% / 98.2%; perplexity x1.011 and x1.005; same-engine floor 0.0006-0.0007. HumanEval (82) 76 and 78 vs 78 (3060), 76 and 76 vs 76 (A4000): the runs do not separate the settings | +| 0.1.42 | Same cards, 5 interleaved rounds, story / code / 6K document | Decode, default over `=0` | RTX 3060 IQ3_XXS 1.104 / 1.197 / 1.241 (all three 1.156, 5/5); RTX A4000 Q2_0 1.151 / 1.135 / 1.101 (1.129, 5/5); Tesla P100 IQ3_XXS `--pcie-frac 0.29` 1.149 / 1.182 / 1.118 (1.146, 5/5); two RTX A4000 layer split (99.7% hit rate) 0.998 (1/5). Tesla V100 32 GB Coder IQ1_M, 94.8% cached: drafts accepted 70.1% -> 68.2%, decode 1-2% slower, which is why the 90% line exists | +| 0.1.42 | 3 cards, interleaved pairs, tail skip on | PCIe share from the CPU's speed | RTX 3060 IQ3_XXS pool 32-37 us per expert, share 0.55 -> 0.20, decode x1.300 (5/5); Tesla P100 84-97 us, 0.29 -> 0.20-0.25, x1.011 (4/5); RTX A4000 Q2_0 97-115 us, 0.30 unchanged. KL against the old default 0.0015 on the 3060 (floor 0.0013). Canonical Q2_0 pack on an RTX 5070 under Windows: code +2.5%, short answer -3.3%. HIP `STRATA_PCIE_AUTO_SPLIT=1` on 2x R9700 (16 GB-eq): Q2_0 story +4.5%, code +9.9%, 4K +8.8%; IQ3_S story -2.1%, code +8.0%, 4K +23.3% | +| 0.1.45 | RTX 3060 12 GB, RTX A4000 16 GB (one card), Tesla P100 16 GB, IQ3_XXS, 8 interleaved pairs, noise arm | sm_86 and sm_60 bitwise sets | The 16-switch set gave +9.6% (3060), +14.6% (A4000), +17.8% (P100) decode, 8/8 pairs each. Engine 0.1.45.3 per-architecture sets: 3060 60.36 -> 66.52 tok/s (+10.2%), A4000 51.16 -> 57.82 (+13.0%), P100 36.56 -> 41.76 (+14.2%); with the off switches 1.002, 0.994 and 0.991 of before. A prompt chunk of 16384 / 32768 changed prompt speed by -3 to +8% at 20K but is not bit-exact on CUDA. `STRATA_ROUTE_PRIOR=0.5` on an RTX 3060: KL 0.0126 against a floor of 0.0012 | +| not stated | RTX PRO 5000 Blackwell 72 GB (110 SMs, sm_120), Linux, CUDA 13.2, IQ3_S all experts in VRAM, `--spec 4 --spec-min-p 0.5`, 6 pairs, 300-token greedy | sm_120 set, wider matvec table, fused prompt experts | Decode story 168.8 -> 192.0 tok/s (+13.8%), code 224.6 -> 260.0 (+15.8%), 8K document 186.6 -> 211.9 (+13.6%); prompts 4K / 20K / 98K 3829 / 4613 / 4767 -> 4212 / 4870 / 4929 (+9.9 / +5.4 / +3.4%), 6/6 each. A fixed 4-token verify round 13.70 -> 11.80 ms. KL against MMQ 0.0002; HumanEval 156/164. Deeper speculation (`--spec 3` to `--spec 8`) was 1.03-1.18x of before, `--spec 4` stays. `--ple-io ram` on this card: 4K -18%, 20K -9%, 98K -2% | +| not stated | R9700, 1 card, IQ3_S / IQ3_XXS packs | Fused int8 prompt experts on gfx12 | Prompt 4K +15.7%, 16K +13..+17%, 32K +19..+20%, 96K +16%; first-token KL equal to MMQ within noise (IQ3_XXS 0.00613 vs 0.00683, IQ3_S 0.00416 vs 0.00372) | +| 0.1.45 | RTX 3060 / RTX A4000, IQ3_XXS, 5 pairs, decode after a 30K prompt | CUDA opt-in kernels | `STRATA_ATTN_DAILY` +3.2% (5/5) / +0.7%; `STRATA_QSA_ROWS` -0.3% / +0.3%; `STRATA_EXPDEC` +0.5% / +1 to +5% (short context); `STRATA_SIDE_MIXER` +0.5% / -0.6%; `STRATA_B6_MMVQ_ROWS` +0.2..1% / +0.5..2%; `STRATA_QSA_SPLIT_ATTN` +2.8% / +3.5% (KL 0.0016 vs floor 0.0012); `STRATA_HC_UPMIX` prompt +0.6% / +0.4%; `STRATA_PROMPT_ATTN_IMMA` prompt +2.5%; `STRATA_HC_REQ8` -0.2% / +2.2%. `STRATA_DEQUANT_WIDE`: time to first token of a 150 / 300 / 450-token prompt -11 to -15% (3060), -6 to -10% (A4000), 5/5 pairs | -## RTX 30 series (sm_86) and Tesla P100 (sm_60): bitwise decode switches on by default +### Opt-in switches measured and left off (0.1.41 and later) -The 0.1.45 switch set that is bit-exact on every card was measured switch by switch on a 12 GB RTX 3060, a 16 GB RTX A4000 (one card) -and a 16 GB Tesla P100, IQ3_XXS, default flags (adaptive tier on), fresh engine per run, 8 interleaved pairs, equal expert-cache slots, -with a noise arm: the whole 16-switch set gave +9.6% (3060), +14.6% (A4000) and +17.8% (P100) decode, 8/8 pairs each, and the identity -gate (10 prompts + exact_pp 4K/20K, fixed flags) is IDENTICAL with it on. Removing one switch at a time (4 rounds each) kept the ones that -paid on their architecture. They are on by default now, per architecture (`include/strata/core/nv_defaults.hpp`, set before the -engine reads its switches; a switch you set yourself, to anything, is kept): +| Switch | Measured on | Result | +| --- | --- | --- | +| `STRATA_GDN_CHUNKED=1` | RTX 3060 (28 SMs), RTX 5070 (48 SMs) | Recurrence 0.73x and 0.87x of the default at 2K-32K; prompt time RTX 5070 2K 1.01x, 8K 1.00x; RTX 3060 no pair faster. HIP Radeon 8060S (gfx1151): recurrence 1.6x faster, 4K-128K prompt 1.4-2.1% | +| `STRATA_FS_SLOTS=N` | 4 slots, 6 pairs | Decode B/A: RTX 3060 0.95 (0/6) misses only, 0.91 (0/6) with look-ahead; Tesla P100 0.97 (2/6) and 1.00 (4/6); 16 slots on the P100 0.96 (0/6); RTX 5070 0.98 (1/6); 2x RTX 3090 the same | +| `STRATA_STAGE_PIN=1` | 6 pairs | Tesla P100 with RAM capped at 24 GB: 13.9 -> 15.0 tok/s (+8%, 6/6); RTX 5070 `--mmap-experts` +1.2% (5/6); RTX 3060 -0.1% (5/6) | +| `STRATA_ADAPT_LAG=2` | 6 pairs | Tesla P100 (PCIe 3.0 x16) +3.5% (6/6), RTX 5070 +0.2% (4/6), RTX 3060 -1.3% (0/6). On a Xeon E5-2673 v3 with DDR3 and an RTX 4060 Ti on PCIe 3.0 x8, 160 swaps every window measured +7.9% (#906) | +| `STRATA_ADAPT_OVERLAP=1` | 6 interleaved pairs each | RTX A4000: Q2_0 16 GB 62.5 -> 66.2 (+5.9%), Q2_0 as a 12 GB card 51.3 -> 54.2 (+5.7%), IQ3_XXS 16 GB 48.6 -> 52.6 (+8.3%), IQ3_XXS as 12 GB 39.3 -> 41.5 (+5.5%), 6/6 each; Tesla P100 +4.4%; RTX 5070 Windows -2.6% on Q2_0; RTX 5090 Laptop (WSL2, Core Ultra 9 275HX, native IQ3_XXS pack, `--expert-cache-per-layer`) 84.6 vs 106.6 tok/s, -21% (#2010) | +| `STRATA_PCIE_BALANCE=1` | 5 pairs, IQ3_XXS | RTX A4000 arena 48.45 -> 51.15 tok/s (+5.6%, 5/5); RTX 3060 resident 59.12 -> 58.15 (-1.6%); Tesla P100 37.95 -> 37.48 (-1.3%) | +| `STRATA_MMQ_DIRECT=1` | 4x RTX 3090, UD-IQ4_XS, split `12,24,36`, `--trim-stage-weights`, `--batch 16`, medians of 8 | "Read in" time of a 170-token prompt 331 -> 251 ms (-24%) and 369 -> 290 ms (-21%); a 1,217-token prompt 773 -> 634 ms (-18%) and 838 -> 675 ms (-19%). One RTX 3090 with most experts streamed from RAM: within 1.5%. Greedy tokens identical to a build without it | +| `--prefill-head-alias` | 8 GB RTX PRO 2000 Laptop, Windows, IQ3_XXS, 590 MiB reserve, 29,854-token prompt | Prefill 292.5 -> 667.1 tok/s (2,048-token chunks instead of 512); reads of up to 512 fresh tokens cost 0.6-0.75 s more; 0 MiB free while decoding | +| `STRATA_EMB_REUSE_ACCOUNT=1` | RTX 3060, IQ3_XXS | Chunk 6400 -> 6656 tokens, 1652 -> 1670 borrowed slots, prompt about +3.9% | +| `STRATA_SPEC_COUPLED` / `STRATA_SPEC_GUMBEL` | Ryzen AI Max+ 395 (Radeon 8060S, Linux, ROCm 7.14.1), UD-Q4_K_XL, `--spec 4 --mtp`, temperature 1.0, 12-13 requests per arm | Drafts accepted 52.8% -> 59.9%, output 41.1 -> 44.9 tok/s (+9%). RTX 3060 IQ3_XXS, 12 interleaved pairs: 62.1% -> 66.9% -> 68.0% accepted, decode 43.3 vs 42.8 tok/s, inside the run-to-run spread | +| `STRATA_SPEC_COUPLED` / `STRATA_SPEC_PROB` (0.1.40.2) | RTX 3060 IQ3_XXS, 200-token answers, 10 pairs | Median tok/s at temperature 0.3 / 0.7 / 1.0: default 42.4 / 43.2 / 43.3, coupled 40.6 / 42.4 / 43.3, rejection sampling 41.0 / 42.6 / 42.6 | +| `STRATA_STAGER_SLEEP=1` | Ryzen 9 7940HS laptop, RTX 4070, IQ3_S, 64K (issue #1101) | Prompts of 5,914 and 22,305 tokens read 5-6% slower; whole-machine CPU use 77-90% -> 21-25%. An RTX 5070 read an 8K Q2_0 prompt 1.2% faster with spinning | +| Stager threads for a GGUF read in place | Big-RAM box, 124K-token Q8 prompt | 57 -> 350-410 tok/s with the 4-thread RAM profile instead of the 32-thread SSD profile (#1353) | -- sm_86 (RTX 3050-3090, RTX A2000-A6000): `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_CPU_IQ_ALL_EXACT1=1`, - `STRATA_SIDE_MIXER=2`, `STRATA_ATTN_DAILY=1`, `STRATA_GR_UP75=1`, `STRATA_GU_SWIGLU=1`, `STRATA_QSA_ROWS=1`, `STRATA_B6_MMVQ_ROWS=1`. - `STRATA_SM86_PERF=0` turns the set off. -- sm_60 (Tesla P100): `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_SIDE_MIXER=2`, `STRATA_FUSE_WAIT=2`, `STRATA_B6_MMVQ_ROWS=1`, - `STRATA_FUSE_DECODE=1`, `STRATA_GR_UP75=1`. `STRATA_SM60_PERF=0` turns the set off. +### Threads, sockets and sessions -The table is used when every visible card has that one compute capability. `STRATA_ADAPT_OVERLAP=1` (opt-in) does not run beside -`STRATA_ADAPT_LAG` / `STRATA_ADAPT_NOWAIT`: set `STRATA_ADAPT_LAG=1 STRATA_ADAPT_NOWAIT=0` with it. Left out because they did not pay or -were mixed: `STRATA_EXPDEC`, `STRATA_GR_NORM_FAST`, `STRATA_RAM_ADAPT`, `STRATA_KV_HOST_REGISTER`, `STRATA_HOST_CORE`, -`STRATA_GR_DOWN_TG`, `STRATA_DEQUANT_WIDE`; `STRATA_EXCHANGE_ROTATE` never reaches READY on the 3060 and `STRATA_PCIE_BALANCE` is slower on a -single card. +| Engine | Measured on | Result | +| --- | --- | --- | +| not stated | 20-core CPU; 32-core EPYC | 24 workers decoded 44% slower than the default; 63 workers 35% slower | +| #3201 | 2x Xeon Gold 6226 (2x12 cores, 4 of 6 memory channels per socket), RTX 3090, IQ4_XS at 262K, about 25% of experts on the CPU | Default (11 workers) 52.6 tok/s, 22 workers 50.3 (lower, noisier) | +| #3201 | 2x Xeon E5-2670 v2 (AVX only), 2x CMP 50HX, IQ3_XXS at 393K, 98% VRAM hit rate | Default (9 workers) 65.5 tok/s, `--pool-workers 19` 75.1 (+15%, six runs). Prompt speed unchanged in both. IOMMU: pinned copies 7.7 to 9.2 GB/s with it translating (`DMA-FQ`), 12.3 GB/s with `iommu=pt`, about 30% prompt speed | +| 0.2.2 | Dual Xeon Gold 6226, RTX 3090, PCIe 3.0, `iommu=pt`, IQ4_XS pack at 262K, 12 samples | NUMA split 59.3 tok/s against 54.05 for the 0.2.0 default (+9.7%; +14% and +11.5% in two earlier sessions without `iommu=pt`); without the split 22 workers 43.25; the split at 12 workers 51.45. Prompt speed unchanged. Not run by us on a two-socket machine | +| not stated | Ryzen AI MAX+ 395 (16 cores), Windows 11, Balanced, every expert in VRAM | Unpinned threads sat on logical processors 0-15 and 16-31 were parked; pool alone on random expert data: 8 workers on 1-8 read 67 GB/s, 8 on real cores 2,4,...,16 94 GB/s, default 15 on 2,4,...,30 146 GB/s, 31 on 1-31 123 GB/s. CPU picture encoder 16 threads 10.1 s for a 1024x1024 picture; pinned one per core 74 to 193 s; 8 threads pinned to 8 cores 12.3 s | +| not stated | 2x RTX 3080 20 GB at 220 W, Xeon E5-2696 v4 (24 logical CPUs, 16 workers), UD-Q4_K_XL, layer split 23, `--batch-mtp`, two slots | `--aux-cpus auto` vs off: solo decode 79.7 and 80.4 against 72.1 and 72.4 tok/s; two streams 90.8 and 90.7 against 80.8 and 82.8; prompt reads independent of the flag; host-thread context switches over ten minutes 43.8k and 42.8k off, 3.9k and 2.8k with `auto`. A/A control 0.985 and 0.990 | +| 0.1.38 | RTX 4070 Ti 12 GB, Ryzen 9 5900X, 64 GB, NVMe ext4, IQ3_XXS, 63,025-token conversation | Session file 1.20 GB, save 0.65 s, restore 1.05 s in a new engine, next 32-token turn 1.14 s against 1.00 s without a restart; the cold first turn takes 25.5 s | +| not stated | 2x RTX 3080 20 GB, UD-Q4_K_XL, 90 GiB container, resident RAM mode, 2 slots, 308 requests | `--memory-limit-mib 101376`: 135 parks, 0 skips. With 78000 the container sat above the cap and all 8 parking attempts were skipped while requests were served normally (median of 6 decodes 79.6 tok/s) | +| not stated | 4-GPU split | A 9,276-token conversation restored in 51 ms and decoded exactly the tokens it does when the conversation never left (`tools/parking_test.py`) | +| not stated | 2x CMP 50HX layer split, IQ3_XXS, pi coding agent | `--prompt-cache-spread`: a 210,528-token prompt that matched the session for 204,349 tokens resumed at 9,525 and read 201,003 again (100 s); with the flag, an 82,593-token conversation with a turn ~6 steps back rewritten resumed at 73,610 tokens (10.2 s against 40.0 s) | +| not stated | RTX 5070 Ti Laptop, Swift 1.5 IQ3_XXS, a new opencode session in a second folder | `env_to_user`: first request 14,073 tokens with 13,693 from the cache, 1.1 s, against 14,753 read, 11.9 s | +| 0.1.40.2 | Tesla P100 | A question after a long document re-reads up to 16K tokens (55 s at 262K) without `strata_prefix` | +| not stated | Codex CLI 0.160.0, Q2_0, RTX 5070 | First prompt 9,443 tokens in 10 s; later tool-loop turns reused about 96% from the cache and read the new part in 1-2 s. Compaction after an 85,000-token conversation reused 84,895 of 84,997 tokens with `codex_compaction_cache` and 41 of 80,683 without | +| not stated | Signalnine/q27's drift corpus, 1,462 agent turns (Qwen3.8 under Claude Code) | `tool_call_recovery`: 98.3% read as intended, 92.9% without | +| not stated | RTX 3080, RVN IQ3_S config, 8-turn chat replay on a 16K-token system prompt | NaN guard on and off: byte-identical answers in 3 pairs; cost 4.8 us of GPU time a window (7.8 us at most), 0.009% of decode time | -The per-architecture sets themselves (no environment variables), against the engine without them, IQ3_XXS, default flags, 8 -interleaved pairs of fresh engines, the same number of expert-cache slots in every arm (engine 0.1.45.3 and its test builds): +### Context beyond 262K (contributors' runs, RTX 5080, IQ3_S, native path, PR #84) -| Card | Decode before -> after | Gain | Pairs faster | Same engine against itself | With `STRATA_SM86_PERF=0` / `STRATA_SM60_PERF=0` | -| --- | ---: | ---: | ---: | ---: | ---: | -| RTX 3060 12 GB (sm_86) | 60.36 -> 66.52 tok/s | +10.2% | 8/8 | 0.999 | 1.002 (equal to before) | -| RTX A4000 16 GB (sm_86) | 51.16 -> 57.82 tok/s | +13.0% | 8/8 | 0.979 | 0.994 | -| Tesla P100 16 GB (sm_60) | 36.56 -> 41.76 tok/s | +14.2% | 8/8 | 0.975 | 0.991 | +Per-position perplexity on the same tokens, only the scaling flag changed. At 293K tokens (1.12x the trained length) `yarn` with +factor 2 lowered the NLL by 0.18 nats against `none` and `linear` 2 (2.04 vs 2.22 and 2.22), 3-4x the path noise. `linear` 2 was +indistinguishable from `none`, and at 2.7K and 32K no arm separated from the noise. Needle tests do not tell the arms apart (the unscaled +model also finds a needle at 413K). Long real-document Q&A worked with `yarn` 2 at 421K and `yarn` 4 at 714K (8/8 each), and a +1M-token `yarn` 4 run read end to end. One machine and one quant. -Two things looked at and **not changed**. A prompt chunk above 8192 (`--prefill auto:16384` or `auto:32768`) changes prompt speed by -3% to +8% -on 12-16 GB cards at 20K tokens (at 16384 / 32768: RTX 3060 +4.5% / +4.5%, Tesla P100 +3.1% / +7.7%, RTX A4000 -3.4% / +6.0%), but it is not bit-exact on -CUDA (the residual dump differs on the P100 and the A4000; 32768 also changed the output tokens on the P100), so the default -stays 8192 on NVIDIA. It is exact on gfx12, where 16384 is the default. And `STRATA_ROUTE_PRIOR=0.5` stays opt-in: on an RTX 3060 -(IQ3_XXS) its teacher-forced KL is 0.0126 against a run-to-run floor of 0.0012 (top-1 agreement 96.6% against 99.1%); on an RTX A4000 -HumanEval scored 154/164 against 157 and 159 for the default twice, and reasoning 25/30 against 24 and 26 (agent 6/6, needles 3/3, CJK -8/8 and tool calls 5/5 for all), which is inside the run-to-run spread. +### n-gram table formats and the prompt ring -## Compute capability 12.0 cards: faster decode (bitwise) and fused prompt experts by default +**Measured on:** 4,000 random rows against the checkpoint's own BF16 table, and 700 teacher-forced tokens on the Q2_0 model. Mean per-row +error: Q8_0 0.53%, FP8 2.64%, Q5_1 3.78%, Q5_0 4.25%, IQ4_NL 7.60%, Q4_1 7.80%, Q4_0 8.55% (a Q8_0 table is 54 GB, an IQ4_NL one +28.8 GB). Mean KL against the BF16 table: 0.0117 (Q5_0), 0.0121 (Q5_1), 0.0125 (Q8_0), 0.0128 (FP8), 0.0140 (Q4_0), 0.0146 (IQ4_NL), perplexity within +1% of BF16's either way. -Measured on an RTX PRO 5000 Blackwell 72 GB (110 SMs, sm_120, Linux, CUDA 13.2), IQ3_S with every expert in VRAM, setup's -flags (`--spec 4 --spec-min-p 0.5`), 6 interleaved pairs of fresh engines, 300-token greedy answers, medians: +RTX 5070, 32K prompts against `STRATA_RING_BYTES=0`, 2-3 interleaved rounds (engine 0.1.39b, #583): IQ3_XXS +18.5% at a 1,500-slot cache and ++9% in another memory state, the Coder +3%, IQ3_S unchanged; 4K prompts unchanged. Teacher-forced over 2,001 tokens of a long document +against the FP16 prompt path (IQ3_XXS): 8K prompt KL 0.042 (0.1.39: 0.054), top-1 agreement 93.5% (92.2%); 32K prompt KL 0.020 (0.019), +top-1 95.3% (95.0%). -| | story | code | 8K document | 4K / 20K / 98K prompt (tok/s) | -|---|---|---|---|---| -| before (tok/s) | 168.8 | 224.6 | 186.6 | 3829 / 4613 / 4767 | -| now | 192.0 (+13.8%, 6/6 pairs) | 260.0 (+15.8%, 6/6) | 211.9 (+13.6%, 6/6) | 4212 / 4870 / 4929 (+9.9% / +5.4% / +3.4%, 6/6 each) | +### Experimental speed projection -Decode is bitwise: with `STRATA_PF_FUSED=0` the answers are the same tokens as before and the identity gate passes (10 prompts -plus the 4K and 20K prompt dumps, with the fixed small cache and with every expert resident). The prompts change at rounding -level because of the fused prompt experts (below). What changed: +**Measured on:** Q2_0, fixed experts, 2,557 teacher-forced tokens of code, a document and a chat. The top-1 token changes at 10% of positions, +mean KL from the stock model 0.063 nats (max 4.1), perplexity +15% on code, +2.3% on the document, +0.4% on the chat. Details: +`bench/results/2026-09-27-esp/`. -- On a 12.0 card these bitwise switches are on unless set: `STRATA_SIDE_MIXER=2` (the mixer's input-only work on graph - branches), `STRATA_ATTN_DAILY=1` (the Daily's decode attention), `STRATA_GR_UP75=1` and `STRATA_GR_DOWN_TG=1` (the - hyper-connection read's up projection with its loads hoisted, and its down projection with a block row per token; token - pairs, `=2`, below 80 SMs), - `STRATA_GU_SWIGLU=1` (new: the routed experts' gate/up, SwiGLU and q8_1 in one launch). A fixed 4-token verify round went - 13.70 -> 11.80 ms with them. `STRATA_SM120_PERF=0` turns the set off; each switch's own variable still wins in both directions. -- The interleaved small-batch matvec (`native_mmvq_il`, docs/MMVQ_IL_TABLE.md) also runs 5 to 8 tokens (it stopped at 4, and a - 5-token window cost 2 ms more than a 4-token one), with a 12.0 table for 5-8 tokens and, on 12.0 cards with 80+ SMs, its own - 2-4 token table. Bitwise (`mmvq_il_parity`, 1-8 tokens). `STRATA_MMVQ_IL_WIDE=0` / `STRATA_MMVQ_IL_120=0` turn them off. -- On 12.0 cards with 80 or more SMs the native packs' prompt experts run on the fused int8 kernels (`STRATA_PF_FUSED`, the rule - gfx12 has) for IQ2_S, IQ3_XXS, IQ3_S and IQ4_XS gate/up; IQ2_XXS / IQ2_XS keep MMQ until measured. An RTX 5070 (48 SMs, IQ2_XS - pack, experts partly in RAM) was 4.7% slower on 20K prompts with them (0/6 pairs), so smaller 12.0 cards keep MMQ. Quality on IQ3_S: teacher-forced KL against MMQ 0.0002 - (5998 positions; MMQ against itself 0.0000) and 0.0010 after ~6K-token documents (1760 positions); first-token KL against - the FP16 prompt path 0.01181 vs MMQ's 0.01236 (48 prompts of 1.5K-24K tokens); HumanEval 156/164, reasoning 26/30, agent - 6/6, needles 6/6, CJK 8/8, tool calls 4/5, the same as MMQ. `STRATA_PF_FUSED=0` keeps MMQ. -- With the cheaper rows, deeper speculation was measured again (3 pairs): `--spec 3` 1.145 / 1.020 / 1.123 of before, - `--spec 4` 1.122 / 1.141 / 1.132, `--spec 5` 1.079 / 1.101 / 1.074, `--spec 6` 1.060 / 1.177 / 1.097, `--spec 8` 1.027 / - 1.105 / 1.069 (story / code / document). `--spec 4 --spec-min-p 0.5` stays the default. -- Setup (Linux, not WSL): when the RAM covers the model's need + the n-gram table + 16 GB (Q2_0 93 GB, IQ3_S 107 GB), the - table is kept in RAM (`--ple-io ram`), so a new prompt does not wait for its rows from the SSD: on this card 4K -18%, 20K - -9%, 98K -2% with a fresh engine per prompt, the same bits. On an R9700 the read was already hidden; with RAM short (a 48 GB - cap) ram mode cost +20% prompt and +14 s of start-up, which is why smaller PCs keep the SSD reads. +### Older runs of the quality checks -Not measured: other 12.0 cards (RTX 5070), Windows, partly-resident caches, the IQ3_XXS and IQ2_XS packs. - -## CUDA opt-in switches measured for 0.1.45 (RTX 3060 and RTX A4000, sm_86; all off unless you set them) - -Ported from open pull requests and A/B'd as whole engine runs against the default (a fresh engine per side, IQ3_XXS, 5 pairs, medians; -bit-exact means the 10-prompt + 4K/20K identity gate with the switch on equals the gate with it off). Decode after a 30K-token prompt -unless noted; the gain is small on these two cards and larger on the Turing cards the authors measured. - -| Switch | What | Answers | RTX 3060 / A4000 | -|---|---|---|---| -| `STRATA_ATTN_DAILY=1` (#2075) | decode attention: two cells per step, a merge that skips empty chunks | bit-exact | +3.2% (5/5) / +0.7% | -| `STRATA_QSA_ROWS=1` (#2059) | block scores of a verify window, queries in registers ([QSAROWS.md](QSAROWS.md)) | bit-exact | -0.3% / +0.3% | -| `STRATA_EXPDEC=1` (#2058) | gate/up of the window's resident experts with the q8_1 rows in shared memory ([EXPERTDEC.md](EXPERTDEC.md)) | bit-exact | +0.5% / +1 to +5% (short context) | -| `STRATA_SIDE_MIXER=1/2` (#1889) | the mixer's input-only work on graph branches | bit-exact | +0.5% / -0.6% (short context) | -| `STRATA_B6_MMVQ_ROWS=1` (#1418) | exact multi-column MMVQ reusing the activation fragment across rows (K-quants, IQ4_XS) | bit-exact | +0.2..1% / +0.5..2% | -| `STRATA_QSA_SPLIT_ATTN=1` (#1888) | decode attention split across blocks on the int8 mma path (sm_80+, int8 KV) | rounding-level (teacher-forced KL 0.0016 against a floor of 0.0012) | +2.8% / +3.5% | -| `STRATA_HC_UPMIX=1` (#1370) | the hyper-connection up projection and `gr_mix_r` in one kernel (sm_80+) | bit-exact here (gate identical) | prompt +0.6% / +0.4% | -| `STRATA_PROMPT_ATTN_IMMA=1` (#1367) | int8-KV prompt attention on INT8 tensor cores | rounding-level (KL at the floor) | prompt read +2.5% | -| `STRATA_HC_REQ8=1` (#1262) | the hyper-connection projections as int8 + a scale per 32 (more expert-cache slots) | rounding-level (KL 0.0018) | decode -0.2% / +2.2% | -| `STRATA_QUANT_GATHER=1` (#1368) | turns the once-per-token MoE input quantization back to the per-row gather | n/a | no difference measured | -| `STRATA_PREFILL_GROUP_GATHER=0` (#1107) | turns the short-prompt grouped gather off | n/a | on is 0.6-1.3% faster | - -`STRATA_DEQUANT_WIDE` (the wide dequantization of Q4_K / Q5_K / Q6_K / IQ4_XS for the prompt's dense weights) is now on by default on -sm_86 as well as sm_61: bit-exact, and the time to the first token of a 150 / 300 / 450-token prompt fell 11-15% on the RTX 3060 and 6-10% on the RTX A4000 (5 of 5 pairs each). -`STRATA_DEQUANT_WIDE=0` turns it off. - -### RTX 20 (Turing, sm_75) kernel series (opt-in, 0.1.45) - -Kernels by rafatxf (#2069-#2072, #2093, #2220-#2222), imanu86 (#2081) and xrip (#1821) for compute capability 7.5. They were measured -on one RTX 2080 Ti (decode +5..6 % at 4K-32K, prompts at depth up to +30 %, per the PRs) and are bitwise equal to the kernels they -replace, but nobody on the team has an sm_75 card to confirm them, so they are off until asked for: - -`STRATA_SM75_PERF=1` turns on every bitwise Turing kernel below on a 7.5 card (no other card is ever affected). Each has its own -variable, which wins in both directions (`=0` off, `=1` on): `STRATA_SM75_IL` (interleaved small-batch matvec table), -`STRATA_SM75_EXP` (expert layout mode 4), `STRATA_GR_UP75` / `STRATA_GR_DOWN_TG` (hyper-connection read), `STRATA_SCORES_HALVE` -and `STRATA_TOPK_SPLIT` (decode QSA selection), `STRATA_TOPK_SAMPLE` (prompt top-k, one read of the scores), `STRATA_ATTN_V3` -(decode attention), `STRATA_Q2_WARP` (Q2_0 matvec), `STRATA_FUSE_WAIT=2` and `STRATA_QIL` (fewer launches per decode layer), -`STRATA_TC_SM75` (BF16 dequant, 16-byte stores), `STRATA_PA_V1_PREFETCH` (prompt attention v1 prefetch), -`STRATA_SCORES_THREAD` (#1825: the prompt path's thread-per-head block scorer, bitwise the warp kernel's scores; `=1` takes it on any -CUDA card, where it measured +1..3 % prompt speed on a Tesla P100 at 30-60K tokens, so it is not a default there). - -Options that change answers (other summation order) are separate and explicit: `STRATA_SELECT_SIMT=1|2` (FP32 tiled block scorer, -KL 0.01 against the warp scorer per #2071), `STRATA_PA_V2T=1` (v2 prompt attention, #2092/#2146), `STRATA_PF_FUSED=1 -STRATA_PF_FUSED_SM75=1` (fused int8 prompt experts for the Q2_0 pack, #2145; only on a 7.5 card, sm_80+ is unchanged), `STRATA_BF16_TC=1` with `STRATA_XN_F16` (#2091). +- Our small Swift 1.5 check (8 reasoning questions, default thinking, IQ2_XS): both models got 8/8; Swift used 1,234 output tokens in 28 s, + the original 2,682 in 46 s. Speed: 4K IQ2_XS 465 prompt / 78.7 output tokens/s against 467 / 78.3 for the original. +- UD-Q4_K_XL on a 64 GB PC with a 12 GB RTX 5070: 7-8.5 tok/s, picks the same tokens as llama.cpp on the same file at 97.5-99% of the + positions of short greedy answers and 90-91% after a 16K prompt ([UNSLOTH_Q4.md](UNSLOTH_Q4.md#quality-against-llamacpp-on-the-same-file)). diff --git a/docs/DOCKER.md b/docs/DOCKER.md index 676326655..81672d749 100644 --- a/docs/DOCKER.md +++ b/docs/DOCKER.md @@ -1,9 +1,8 @@ # Strata in Docker -Run Strata as a container: one command, nothing to install on the computer except Docker and your graphics card's driver. -The images come with Strata's engine already built for your card, so nothing is compiled on your side. The model is -downloaded on the first start into a volume (a download that stops continues where it stopped), and every later start goes -straight to serving. +Run Strata in a container with a prebuilt engine. Install Docker, the host GPU driver and the GPU-access +requirements in the table below. NVIDIA and AMD images can download and prepare a model on first start; +Intel Arc needs prepared model files. Keep the data volume to reuse downloads and settings. > **On this page:** [Which image](#which-image-for-which-card) · [One command per card](#one-command-per-card) · > [Docker Compose](#docker-compose) · [Settings](#settings) · [The API key](#the-api-key) · [Connect your apps](#connect-your-coding-apps) · @@ -12,7 +11,7 @@ straight to serving. Docker on a **Mac** has no access to the Mac's graphics chip (no Metal): use the Mac app or the Terminal ([MAC.md](MAC.md), [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md)). Docker Desktop on **Windows** passes NVIDIA cards through -(WSL 2) but not Radeon or Arc cards; on Windows the app is the easier way for every card. +(WSL 2) but not Radeon or Arc cards; on Windows use the native app for supported NVIDIA or AMD cards. Intel Arc has no native Windows engine; follow the Linux guide. ## Which image for which card @@ -23,11 +22,11 @@ Docker on a **Mac** has no access to the Mac's graphics chip (no Metal): use the | AMD **RX 9000** series and Radeon AI PRO R9700 (RDNA 4, gfx1200 / gfx1201) | `0.2.3-rocm-gfx120x` | the `amdgpu` kernel driver (Linux 6.8+) | | AMD **RX 7000** series (RDNA 3, gfx1100 / gfx1101 / gfx1102) | `0.2.3-rocm-gfx110x` | the `amdgpu` kernel driver | | AMD **Ryzen AI Max** ("Strix Halo", Radeon 8060S, gfx1151) | `0.2.3-rocm-gfx1151` | the `amdgpu` kernel driver | -| Intel **Arc** A-series and B-series (including Arc Pro B70) | `0.2.3-sycl` (only for a release that carries the Intel engine package; otherwise [build it](#build-it-yourself)) | the `i915` or `xe` kernel driver; **prepared model files** (see [Intel](#intel-arc)) | +| Intel **Arc** A-series and B-series (including Arc Pro B70) | `0.2.3-sycl` (if no image exists for a release, [build it](#build-it-yourself)) | the `i915` or `xe` kernel driver; **prepared model files** (see [Intel](#intel-arc)) | -Size of the images on disk (built from the engine packages of the 0.2.0-beta release): `cuda13` 2.2 GB, `cuda12` 2.9 GB, `rocm-gfx120x` 2.8 GB, -`rocm-gfx110x` 2.4 GB, `rocm-gfx1151` 1.5 GB; the download is about a third of that (measured on the 0.1.45 packages: `cuda13` 640 MB, `cuda12` 845 MB, -`rocm-gfx120x` 505 MB, `rocm-gfx110x` 330 MB, `sycl` 450 MB). The model comes on top, on the first start. +**Measured on:** images built from the 0.1.45 engine packages. Size on disk: `cuda13` 2.2 GB, `cuda12` 2.9 GB, `rocm-gfx120x` 2.8 GB, +`rocm-gfx110x` 2.4 GB, `rocm-gfx1151` 1.5 GB. The download is about a third of that: `cuda13` 640 MB, `cuda12` 845 MB, +`rocm-gfx120x` 505 MB, `rocm-gfx110x` 330 MB, `sycl` 450 MB. The model comes on top, on the first start. Current sizes may differ. Not sure which NVIDIA image? Take `cuda13` when `nvidia-smi` shows driver 580 or newer and the card is an RTX; otherwise `cuda12`. A wrong pick does no harm: the container starts, says in plain words that the engine does not fit this card, and stops. @@ -40,7 +39,7 @@ tag if you want updates only when you decide. The examples below use these image ## One command per card Each command runs Strata in the background, keeps your files in the volume `strata-data`, and listens only on this computer -(`127.0.0.1:8080`). Replace `MODEL` and the key. The first start downloads the model (tens of GB): watch it with +(`127.0.0.1:8080`). Choose `FAMILY` and `MODEL`, and generate your own key. The examples select ISTA-DASLab originals explicitly. The first start downloads the model (tens of GB): watch it with `docker logs -f strata`. Already have the model's GGUF files? Add `-e MODEL_PATH=/models -v /your/folder:/models:ro` and nothing is downloaded for the model itself ([Files you already have](#files-you-already-have)). @@ -49,7 +48,7 @@ downloaded for the model itself ([Files you already have](#files-you-already-hav ```sh docker run -d --name strata --gpus all -p 127.0.0.1:8080:8080 \ --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \ - -v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \ + -v strata-data:/data -e FAMILY=qwen -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \ ghcr.io/niko1221/strata:0.2.3-cuda13 ``` @@ -59,7 +58,7 @@ docker run -d --name strata --gpus all -p 127.0.0.1:8080:8080 \ docker run -d --name strata --device /dev/kfd --device /dev/dri \ --group-add video --group-add $(getent group render | cut -d: -f3) -p 127.0.0.1:8080:8080 \ --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \ - -v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \ + -v strata-data:/data -e FAMILY=qwen -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \ ghcr.io/niko1221/strata:0.2.3-rocm-gfx120x ``` @@ -68,7 +67,7 @@ docker run -d --name strata --device /dev/kfd --device /dev/dri \ ```sh docker run -d --name strata --device /dev/dri --group-add $(getent group render | cut -d: -f3) \ -p 127.0.0.1:8080:8080 --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \ - -v strata-data:/data -e MODEL=IQ3_S -e MODEL_PATH=/models -v /srv/gguf/IQ3_S:/models:ro \ + -v strata-data:/data -e FAMILY=qwen -e MODEL=IQ3_S -e MODEL_PATH=/models -v /srv/gguf/IQ3_S:/models:ro \ -e API_KEY=$(openssl rand -hex 24) ghcr.io/niko1221/strata:0.2.3-sycl ``` @@ -96,7 +95,7 @@ Ready-made files are in [docker/compose/](../docker/compose): `docker-compose.nv ```sh cd docker/compose -cp .env.example .env # set MODEL=..., API_KEY=... +cp .env.example .env # set FAMILY=qwen, MODEL=..., API_KEY=... docker compose -f docker-compose.nvidia.yml up -d docker compose -f docker-compose.nvidia.yml logs -f ``` @@ -114,16 +113,17 @@ screens; leave out what you do not care about. | Variable | What it does | Default | | --- | --- | --- | | `MODEL` | model size: `IQ2_XS`, `IQ3_XXS`, `IQ3_S`, `Q2_0`, `UD-Q4_K_XL`, ... ([MODELS.md](MODELS.md)) | Strata picks what suits the card, and says so in the log | -| `FAMILY` | `qwen`, `swift`, `coder`, ... when a size exists in several families | the usual one | +| `FAMILY` | `qwen`, `unsloth`, `gyro`, `coder`, `swift`, ...; use `qwen` for the original ISTA-DASLab packs | installer recommendation; may differ by hardware | | `MODEL_PATH` | GGUF files you already have: a folder or one `.gguf`, **inside the container** (mount it with `-v`, `:ro` is fine) | download | | `CONTEXT` | how much text it remembers at once: `32k`, `64k`, `128k`, `256k` or a number | the largest that fits | | `PARALLEL` | requests answered at the same time, 1 to 8 | fitted to the card | | `VISION` | `off`, `on` (the model reads pictures), `gpu`, `cpu`; the picture-reader file (about 0.9 GB) is downloaded with the model | `off` | | `KV` | conversation memory format: `int8`, `q4_0` (half the memory), `k8v4`, `fp16` | `int8` | | `GPU` / `GPUS` | card numbers as the monitor lists them, `0` or `0,1`; empty or `all` = the engine's choice. Changing it later makes the model's run settings again (the files stay) | all that fit | -| `GPU_ROLES` | a role for each card, `1=main,0=helper` (`main`, `helper`, `off`; cards you do not name are off; not together with `GPU` / `GPUS`). Passed to `launch --gpu-roles`; see [MULTI_GPU.md](MULTI_GPU.md) | -| `TUNE` | `quick` (up to about 20 minutes, once), `thorough`, `skip`; done once and kept in the preset (`RETUNE=1` tunes again) | `skip` | +| `GPU_ROLES` | a role for each card, `1=main,0=helper` (`main`, `helper`, `off`; cards you do not name are off; not together with `GPU` / `GPUS`). Passed to `launch --gpu-roles`; see [MULTI_GPU.md](MULTI_GPU.md) | none | +| `TUNE` | `quick` (runtime depends on hardware and workload), `thorough`, `skip`; done once and kept in the preset (`RETUNE=1` tunes again) | `skip` | | `API_KEY` / `API_KEY_FILE` | the key clients must send ([below](#the-api-key)); the file form is for Docker secrets | made for you | +| `TUNE_ALLOW_SMALL` | `1` passes `--tune-allow-small` to `launch` (allow tuning on a small card) | `0` | | `PORT` | the port inside the container | `8080` | | `PRESET` | the preset name in `/data` | `docker` | | `STRATA_SET` | advanced engine settings, space separated, like `strata launch --set` (`"prefill=65536 STRATA_PIN_BY_NEED=1"`) | none | @@ -131,7 +131,7 @@ screens; leave out what you do not care about. Your choices are saved as a preset in the volume. Start again with no settings and the saved preset is used; start with settings and they replace the saved ones (tuning results are kept). To change one thing, recreate the container with the new variable; the model and tuning stay in `/data`. Anything else: `docker exec strata strata launch --help`, or -`docker run --rm IMAGE strata launch --plan --model q2_0` to see what a choice would do without starting anything. +`docker run --rm IMAGE strata launch --plan --family qwen --model q2_0` to see what a choice would do without starting anything. What is in the volume `/data`: `Strata-data/` (model files, the prepared pack), `state/` (presets, tuning, the key), `install/` (run settings and logs), `home/`. The engine and the Terminal are in the image, so updating the image never touches your files. @@ -142,14 +142,14 @@ container's user). Nothing large is written anywhere else in the container. ### Files you already have Mount the folder with the model's GGUF files read-only and name it: `-v /srv/gguf/IQ3_XXS:/models:ro -e MODEL_PATH=/models --e MODEL=IQ3_XXS`. Nothing is downloaded for the model itself; the prepared pack is made in the volume. +-e FAMILY=qwen -e MODEL=IQ3_XXS`. Nothing is downloaded for the model itself; the prepared pack is made in the volume. - The folder holds the shards of one model directly (`...-00001-of-00002.gguf`, `...-00002-of-00002.gguf`), not a folder of folders. If you mount a parent folder, the container says which sub-folder to name in `MODEL_PATH`. - The folder has to be mounted: a `MODEL_PATH` that is not in the container, or a folder with no `.gguf` files, stops the start at once with a message, and nothing is downloaded. - Read-only (`:ro`) is fine. The pack (about 40 GB for IQ3_XXS) and the draft layer (about 6 GB, fetched once) are written to the - volume, so it needs that room. With pictures on (the default) the picture reader (0.9 GB) is fetched too; `VISION=off` skips it. + volume, so it needs that room. With `VISION=on` the picture reader (0.9 GB) is fetched too; `VISION=off` skips it. - Links to the files work when the links' targets are mounted too, and `MODEL` is set (without `MODEL`, Strata identifies the files itself and does not follow links). - Without `MODEL_PATH`, files you put into the volume by hand (`/data/Strata-data/models//`) are found as well, if they have the @@ -284,9 +284,9 @@ This is the way to get an image for a computer without registry access or after folder (made by `packaging/package_engine.py`, or fetched by `strata update` into a components folder) and Docker on Linux: ```sh -docker/build.sh --flavor cuda --engine ~/components/engine/0.1.45.3-linux-x86_64-cuda-cuda13 --tag strata:0.2.3-cuda13 -docker/build.sh --flavor hip --engine ~/components/engine/0.1.45.3-linux-x86_64-hip-rx9070 --tag strata:0.2.3-rocm-gfx120x -docker/build.sh --flavor sycl --engine ~/components/engine/ --tag strata:0.2.3-sycl +docker/build.sh --flavor cuda --engine ~/components/engine/ --tag strata:0.2.3-cuda13 +docker/build.sh --flavor hip --engine ~/components/engine/ --tag strata:0.2.3-rocm-gfx120x +docker/build.sh --flavor sycl --engine ~/components/engine/ --tag strata:0.2.3-sycl ``` `build.sh` assembles the build context (the Terminal's files and the package) and runs `docker build`. The build makes the @@ -332,27 +332,12 @@ the job starts the image once to see that the Terminal inside it runs. | `rocm-gfx120x` | `linux-x86_64`, hip, variant `rx9070` (gfx1200 / gfx1201) | | `rocm-gfx110x` | `linux-x86_64`, hip, variant `rx7900` (gfx1100 / gfx1101 / gfx1102) | | `rocm-gfx1151` | `linux-x86_64`, hip, variant `halo` (Strix Halo, gfx1151) | -| `sycl` | `linux-x86_64`, sycl; **only when the feed lists one** (the 0.2.0-beta feed has none, so no Intel image is built from it) | +| `sycl` | `linux-x86_64`, sycl; only when the feed lists one | A flavor the feed has no package for is left out, with a line in the log. The same steps run from a Linux computer with Docker: `docker/publish.sh --release-tag v0.2.3` builds everything and pushes nothing; add `--push` (after `docker login ghcr.io`) to publish. The images run on the runner without a graphics card, so the card checks (`docker/smoke-test.sh`) stay a manual step on a computer with the card. -## Tested - -2026-10-19, images built from the engine packages of the 0.2.0-beta release (engine 0.1.45.3), fetched and checked against the signed feed by -`docker/fetch-engines.py` (the packages are the ones `strata update` installs). Run on real cards with the checks below (`docker/smoke-test.sh`): `cuda13` -and `cuda12` on one RTX A4000 (driver 580), `rocm-gfx120x` on one Radeon AI PRO R9700. All passed: serving 71-161 s after `docker run` with the prepared -files, `docker stop` took 4-5 s with exit code 0, the card's memory was free again, and the second start reused the saved preset. `rocm-gfx110x` and -`rocm-gfx1151` (Strix Halo) were built and passed the package and library checks but have not run on a card. On a computer without an AMD card the -`rocm-gfx120x` container says in plain words that the engine does not match the card and exits with code 3. - -2026-10-17, images built from the 0.1.45 engine packages, on real cards: NVIDIA RTX 3060 (`cuda13` -and `cuda12`), Radeon AI PRO R9700 (`rocm-gfx120x`), Intel Arc Pro B70 (`sycl`). On each: the container serves, `GET /v1/models` answers (and -refuses a request without the key), a one-word chat and a tool call work, `docker exec strata monitor --once` works, -`docker stop` ends it with exit code 0 and the graphics memory is free again, and a second start reuses the saved preset. -Not run on a card: `cuda12` on Pascal / Volta (it ran on the RTX 3060), `rocm-gfx110x` (RX 7000). They are built and checked the same way. - ## Known limits - Linux x86-64 hosts only. Docker on Windows reaches NVIDIA cards only; Docker on a Mac cannot use the GPU. @@ -361,3 +346,14 @@ Not run on a card: `cuda12` on Pascal / Volta (it ran on the RTX 3060), `rocm-gf - An Arc card needs prepared model files (see [Intel](#intel-arc)). - AMD **RX 7000** (gfx110x) and RDNA 3 iGPUs: the gfx110X image is built for gfx1100 / 1101 / 1102 (RX 7900, 7800, 7700, 7600). The Radeon 780M (gfx1103) is experimental and has its own engine package, not in these images. + +## History / older measurements + +**Tested on real cards.** Checks (`docker/smoke-test.sh`): the container serves, `GET /v1/models` answers (and refuses a request +without the key), a one-word chat and a tool call work, `docker exec strata monitor --once` works, `docker stop` ends it with exit +code 0 and the graphics memory is free again, and a second start reuses the saved preset. + +| Date | Images built from | Card | Result | +| --- | --- | --- | --- | +| 2026-10-19 | 0.2.0-beta engine packages (engine 0.1.45.3), fetched and checked against the signed feed | `cuda13` and `cuda12` on one RTX A4000 (driver 580); `rocm-gfx120x` on one Radeon AI PRO R9700 | All passed. Serving 71-161 s after `docker run` with the prepared files; `docker stop` took 4-5 s with exit code 0. `rocm-gfx110x` and `rocm-gfx1151` passed the package and library checks but did not run on a card. On a computer without an AMD card the `rocm-gfx120x` container says the engine does not match the card and exits with code 3. | +| 2026-10-17 | 0.1.45 engine packages | RTX 3060 (`cuda13`, `cuda12`), Radeon AI PRO R9700 (`rocm-gfx120x`), Intel Arc Pro B70 (`sycl`) | All checks passed on each. Not run on a card: `cuda12` on Pascal / Volta (it ran on the RTX 3060), `rocm-gfx110x` (RX 7000). | diff --git a/docs/GYRO.md b/docs/GYRO.md index 4a84aabf0..6c390e1b5 100644 --- a/docs/GYRO.md +++ b/docs/GYRO.md @@ -1,178 +1,208 @@ # Gyro (agentionai rotor quants) on Strata -Gyro-S and Gyro-M are Qwen3.8-Flash-Next quantizations by agentionai -([agentionai/Qwen3.8-Flash-Next-Gyro-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)): the routed -experts are stored as trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a Hadamard-rotated basis. Strata reads the -published files unchanged. **Setup lists them as a model family of their own** ("Qwen3.8-Flash-Next (Gyro)"); neither -is anybody's default. +This page is for people who want to run Gyro-S or Gyro-M, the smallest Qwen3.8-Flash-Next options, or who want to know +how they compare with the other sizes. They are quantizations by agentionai +([agentionai/Qwen3.8-Flash-Next-Gyro-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)). The +routed experts are stored as trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a Hadamard-rotated basis. Strata +reads the published files unchanged. + +Setup lists them as a model family of their own, "Qwen3.8-Flash-Next (Gyro)". They are not the first recommendation +([MODELS.md](MODELS.md)). Setup may still preselect Gyro for some memory tiers, so choose the provider explicitly if +you care. + +> **On this page:** [Quick start](#quick-start) · [Sizes](#sizes) · [Compared with other sizes](#compared-with-other-sizes) · +> [Platforms](#platforms) · [Intel Arc (SYCL)](#intel-arc-sycl) · [On a Mac](#on-a-mac) · +> [Install by hand](#install-by-hand) · [Numerics](#numerics) · [Known limits](#known-limits) · +> [Credit and licenses](#credit-and-licenses) · [History](#history--older-measurements) + +## Quick start + +```sh +strata launch --family gyro --model Gyro-S # or Gyro-M +``` + +Setup downloads the pinned revision of the file (size and SHA-256 checked) and prepares the pack +(`tools/iq_pack.py --compat-bf16`: the Hadamard table and the dense weights, 1.5 GB). It also prepares the draft head +(Strata's own MTP head, made from the original model as for every Qwen3.8-Flash-Next file) and the start script. If the +ready-made engine is too old for the pack, setup replaces it or the app offers Update engine. + +## Sizes | | Gyro-S | Gyro-M | | --- | --- | --- | | File | `Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf` | `Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf` | -| **Download** (the n-gram table is inside the file) | **58.5 GB** | **92.0 GB** | -| **Memory it needs** (VRAM + RAM together) | **27.6 GB** | **35 GB** | +| Download (the n-gram table is inside the file) | 58.5 GB | 92.0 GB | +| Memory it needs (VRAM + RAM together) | 27.6 GB | 35 GB | | Experts (trellis) | TQK6 gate/up, TQK7 down | TQ2_T | -| In plain words | the smallest option, for PCs with little memory; lower quality than the 2-3-bit sizes | about IQ3_XXS quality with about 20% less GPU memory; decodes about 16% slower than Q2_0 on a 12 GB RTX 3060 (24% slower before engine 0.2.2) | +| In plain words | The smallest option, for PCs with little memory. Lower quality than the 2 to 3-bit sizes. | About IQ3_XXS quality with about 20% less GPU memory. | -About half of each file is the n-gram (PLE) table: it stays on the SSD and is read row by row for every token, so keep +About half of each file is the n-gram (PLE) table. It stays on the SSD and is read row by row for every token, so keep the file on a fast local drive. The memory figure is what the weights take once loaded. -Not defaults, by measurement (our evaluation, 2026-10): against Q2_0 at the same GPU bytes Gyro-M halves the KL -divergence (0.159 against 0.312 on 128 chunks) and is level with IQ3_XXS (0.157), but decodes 16-21% slower on a 12 GB -card (RTX 3060, measured before engine 0.2.2: 16% since) and an R9700, and its reasoning set scored lower than Q2_0's (49/60 against 57/60). Gyro-S is 13% worse -in KL than IQ2_XS (0.268 against 0.236) with 24% fewer bytes. So: pick Gyro-S where nothing bigger fits, Gyro-M when -the 20% memory saving over IQ3_XXS matters. Setup and the MCP server mention them only where the other sizes do not -fit the PC; they warn, never refuse. Gyro-L (agentionai's 50 GiB file, 2026-10-12) is not supported: it uses another -trellis family (types 147-149) that the engine does not have. +Gyro-L (agentionai's 50 GiB file) is not supported. It uses another trellis family (types 147 to 149) that the engine +does not have. -## Install +**Pictures.** Gyro reads pictures with the original model's picture reader file, the same one every +Qwen3.8-Flash-Next model uses. Setup downloads it with the model. The reader opens only the vocabulary and the header of +the Gyro file. Pictures are on by default, with the CPU reader under 24 GB of graphics memory and the card's reader from +24 GB. -```sh -./setup.sh --setup --family gyro --model Gyro-S # or Gyro-M; Windows: START-HERE.bat --setup ... -``` +## Compared with other sizes -Setup downloads the pinned revision of the file (size and SHA-256 checked), prepares the pack (`tools/iq_pack.py ---compat-bf16`: the Hadamard table and the dense weights, 1.5 GB), the draft head (Strata's own MTP head, prepared from -the original model as for every Qwen3.8-Flash-Next file) and the start script. Needs engine 0.1.45.1 or newer: the pack is native_experts.txt v5 (Hadamard-folded), which the engine reads since the -APR port, and the released 0.1.45 stops at v4. A ready-made engine that is older is replaced by one compiled from this source -(setup), or the app says the model needs a newer engine and offers Update engine. +Gyro is not a default, by our measurements. Setup and the MCP server mention it only where the other sizes do not fit +the PC. They warn and never refuse. -**Supported:** NVIDIA (CUDA, Linux and Windows) and AMD (HIP, experimental: the authors validated gfx1151; we ran -gfx1201 with identical output to rc2). **Intel Arc** (SYCL, Linux, built from source) runs both files: see [Intel Arc](#intel-arc-sycl). **Apple Silicon Macs** (Metal) run both files from -0.2.2: see [On a Mac](#on-a-mac). +- Pick Gyro-S where nothing bigger fits. +- Pick Gyro-M when the 20% memory saving over IQ3_XXS matters. -**Pictures.** Gyro reads pictures with the original model's picture reader file (the same one every Qwen3.8-Flash-Next -model uses; setup downloads it with the model). The reader opens only the vocabulary and the header of the Gyro file, so -the trellis tensors do not matter to it. Pictures are on by default, with the CPU reader under 24 GB of graphics memory and -the card's reader from 24 GB. This needs engine 0.1.45.1 or newer, like the Gyro files themselves. +> **Measured on:** our evaluation, 2026-10, Gyro files against ISTA-DASLab files at the same GPU bytes, KL divergence on +> 128 chunks (lower is better). -**RTX 50 cards (sm_120).** Gyro gives correct output on them with engine 0.1.45.3. The fused gate/up SwiGLU kernel, which -is the default on compute capability 12.0 cards, handles only rows of 80 dot products; Gyro's trellis rows have 160, and -before the fix they went through it anyway. Gyro-S then answered with loops, garbage or nothing: on an RTX PRO 5000, 0 of -8 test questions were answered right before the fix and 8 of 8 after it (greedy and with the app's sampling). Now those -rows use the full-warp kernel, and the identity gate (10 prompts, 4K and 20K prompts) is identical to the run with the fused -kernel off. If Gyro misbehaves on an RTX 50 card, update the engine. +| Comparison | Result | +| --- | --- | +| Gyro-M against Q2_0 | KL 0.159 against 0.312 | +| Gyro-M against IQ3_XXS | KL 0.157, level | +| Gyro-S against IQ2_XS | KL 0.268 against 0.236 (13% worse), with 24% fewer bytes | +| Reasoning set, Gyro-M against Q2_0 | 49/60 against 57/60 | + +> **Measured on:** RTX 3060 12 GB, Core Ultra 7, Ubuntu, CUDA 13.4, default flags (MTP draft 4, auto cache), engine +> 0.2.2, the pinned Gyro files, medians of story, code and facts prompts (200 tokens each) over 5 interleaved pairs of +> whole engine starts. + +| Decode | tok/s | Against Q2_0 on the same card (65.5 and 64.5 tok/s) | +| --- | ---: | --- | +| Gyro-S | 62.5 | 7% slower | +| Gyro-M | 54.8 | 16% slower | + +On an R9700, Gyro-M decodes about 21% slower than Q2_0 (our evaluation, 2026-10). Gyro prompts are 7 to 9% slower than +Q2_0's. What is left of Gyro-M's gap is mostly its dense weights: its Q8_0 matrix-vector kernels take 26% of the GPU +time in a decode profile. + +## Platforms + +| Platform | Status | +| --- | --- | +| NVIDIA (CUDA, Linux and Windows) | Supported, including RTX 50 cards (sm_120). | +| AMD (HIP) | Experimental. The authors validated gfx1151. We ran gfx1201 with output identical to rc2's. | +| Intel Arc (SYCL, Linux, built from source) | Runs both files. See [Intel Arc (SYCL)](#intel-arc-sycl). | +| Apple Silicon (Metal) | Runs both files. See [On a Mac](#on-a-mac). | + +If Gyro misbehaves on an RTX 50 card, update the engine. **Prompt speed and the ggml build.** The trellis prompt kernels (MMQ) are agentionai's, from their llama.cpp fork (MIT). -Strata builds against upstream ggml, which does not have them, so it adds them itself at configure time: -`cmake/strata_ggml_tq_overlay.cmake` writes a copy of the pinned ggml-cuda headers into the build folder with the three -trellis types routed like IQ2_XS (the tile loader is the fork's, `third_party/ggml-tq/`); only Strata's own MMQ library sees -the copy, every other type is compiled from the untouched files (identity gate below). The ready-made engines read Gyro -prompts through it as fast as rc2 does, and bit for bit the same. If the pinned ggml changes and an anchor of the script -no longer matches, the overlay switches itself off with a message and prompts take the slower FP16 path. A build against the -fork (`-DSTRATA_GGML_DIR=`, as rc2 does) uses the fork's own instances instead. +Strata builds against upstream ggml, which does not have them, so it adds them at configure time. +`cmake/strata_ggml_tq_overlay.cmake` writes a copy of the pinned ggml-cuda headers into the build folder with the +three trellis types routed like IQ2_XS. The tile loader is the fork's (`third_party/ggml-tq/`). Only Strata's own MMQ +library sees the copy; every other type is compiled from the untouched files. The ready-made engines read Gyro prompts +through it. If the pinned ggml changes and an anchor of the script no longer matches, the overlay switches itself off +with a message and prompts take the slower FP16 path. A build against the fork (`-DSTRATA_GGML_DIR=`) +uses the fork's own instances instead. + +**Decode kernels.** The trellis decode kernels keep the 4 KB codebook in shared memory (`STRATA_IQ_STAGE_GRID=0` reads it +from the global table again) and take four rows per warp. The sums are added in the same order as the unstaged kernels, +so answers are bit for bit the same. A Tesla P100 (compute capability 6.0) runs its own variant, `STRATA_TQ_STAGE`. ## Intel Arc (SYCL) -Gyro-S and Gyro-M run on Intel Arc cards on Linux, with the SYCL engine that setup builds from this source ([INTEL.md](INTEL.md)). -Setup lists them for an Arc like for any card and warns, never refuses, when VRAM and RAM together are short. Tested on one -card: an Arc A750 (8 GB, i915 driver) in a PC with a Ryzen 5 5600X (6 cores, AVX2) and 62 GB of RAM, on the first-release files of Gyro-S and -Gyro-M (the same size as the pinned ones, a different SHA-256). The pinned Gyro-S passes the same parity run (16 cases, 0 failures) and -answers correctly through the server; it was not compared with CUDA. -Not tested by us: a B-series card (supported since 2026-10-22; `strata launch` says it is new and how to report a problem), two Arc cards (the second-card expert helper refuses a Gyro pack and says so), an Arc on the xe driver. The Battlemage AOT build -(`bmg-g31`) compiles with the trellis kernels; it was not run. +Gyro-S and Gyro-M run on Intel Arc cards on Linux, with the SYCL engine that setup builds from source +([INTEL.md](INTEL.md)). Setup lists them for an Arc like for any card and warns, never refuses, when VRAM and RAM +together are short. -The trellis decode kernels do the same integer and float operations in the same order as the CUDA ones (int8 codebook, q8_1 activations), -the Hadamard rotation of the expert input and of the SwiGLU output runs on the card, and the prompt takes the grouped XMX path, -which now decodes the trellis types into its tiles. Experts in the card's cache run on the card; the other misses go to the CPU pool or -over the PCIe link, as with every pack. +**Tested by us:** one Arc A750 (8 GB, i915 driver) in a PC with a Ryzen 5 5600X (6 cores, AVX2) and 62 GB of RAM. It ran +the first-release files of Gyro-S and Gyro-M (same size as the pinned ones, different SHA-256). The pinned Gyro-S passes +the same parity run (16 cases, 0 failures) and answers correctly through the server. It was not compared with CUDA. -**Checks.** `tq_sycl_parity [layer ...]` (also `--synthetic [--hadamard] GU/DOWN` and `--bench GU/DOWN`) takes real -experts of the file (layers 0-3, 10, 20, 30 and 47, two experts each) and compares the card with a float reference and with the CPU path: +**Not tested by us:** a B-series card (`strata launch` says it is new and how to report a problem), two Arc cards (the +second-card expert helper refuses a Gyro pack and says so) and an Arc on the xe driver. The Battlemage AOT build +(`bmg-g31`) compiles with the trellis kernels but was not run. -- the decode kernels equal a host replay of their own arithmetic **in every bit** (0 of 3,840 gate/up values and 0 of 7,680 down values - differ, Gyro-S and Gyro-M, all 16 cases each); the prompt dequantizers equal the fp16 codebook in every bit (0 of 4.9 million values); - the card's Hadamard rotation equals the CPU's bit for bit (the fp16 form within 3e-4); -- against the float reference the card is within 1.7-2.0e-2 relative L1 error (the int8 codebook; 3e-2 is the bound every expert type - gets), the CPU path within 0.9e-2. The card and the CPU path therefore agree within tolerance, not bit for bit: the CPU path uses an - int16 codebook and Q8_0 activations, the card the int8 codebook and q8_1, as on CUDA. +The trellis decode kernels do the same integer and float operations in the same order as the CUDA ones (int8 codebook, +q8_1 activations). The Hadamard rotation of the expert input and of the SwiGLU output runs on the card, and the prompt +takes the grouped XMX path, which decodes the trellis types into its tiles. Experts in the card's cache run on the card. +Misses go to the CPU pool or over the PCIe link, as with every pack. -**Against CUDA.** The same Gyro-S file on an RTX 3060 (the f-022-int engine) and on the A750, both with 64 cached experts, no PCIe share and -no CPU prompt share, ten prompts of 26-100 tokens, 48 greedy tokens with the MTP draft: 7 of the 10 prompts give the same 48 tokens. The other three (prompts 5, 7 and 8) agree for 9, 9 and 6 tokens and then differ at a near-tie. The first-token KL is 1.3e-4 to 1.1e-2 and the top token is always the same. The two SYCL prompt paths (oneMKL and XMX) also flip prompt 7 against each other, so the prompt-path rounding alone moves it. A 1,406-token prompt gives the same 16 tokens on both. +**Checks.** `tq_sycl_parity [layer ...]` (also `--synthetic [--hadamard] GU/DOWN` and +`--bench GU/DOWN`) takes real experts of the file (layers 0 to 3, 10, 20, 30 and 47, two experts each) and compares the +card with a float reference and with the CPU path. -**Measured** (A750, setup's config: 32K context, 300 MiB VRAM reserve, MTP draft of 4, PCIe probe 10.4 GB/s, median of 5 interleaved pairs): +| Check | Result | +| --- | --- | +| Decode kernels against a host replay of their own arithmetic | Equal in every bit (0 of 3,840 gate/up and 0 of 7,680 down values differ; Gyro-S and Gyro-M, 16 cases each) | +| Prompt dequantizers against the fp16 codebook | Equal in every bit (0 of 4.9 million values) | +| Card's Hadamard rotation against the CPU's | Bit for bit (the fp16 form within 3e-4) | +| Card against the float reference | Within 1.7 to 2.0e-2 relative L1 error (3e-2 is the bound every expert type gets) | +| CPU path against the float reference | Within 0.9e-2 | -| | before | now | -| --- | ---: | ---: | -| Gyro-S prompt, 1,406 tokens | 40.2 tok/s | **82.0 tok/s** (5/5 pairs) | -| Gyro-S prompt, 2,377 tokens | | 101.9 tok/s | -| Gyro-S prompt, 20,000 tokens | | 195 tok/s | -| Gyro-S decode | 13.7 tok/s | **15.8 tok/s** (5/5 pairs) | -| Gyro-M decode (8K context) | | 14.2 tok/s (12.6-14.4), prompt 62 tok/s at 1,406 tokens | +The card and the CPU path agree within tolerance, not bit for bit. The CPU path uses an int16 codebook and Q8_0 +activations, the card the int8 codebook and q8_1, as on CUDA. -The decode speed is the CPU's and the link's: with 8 GB the card holds 410 of the 24,576 experts (the dense weights, the draft layer, -the KV cache and the prompt buffers take the rest), so about 15 experts per layer and token come from the CPU pool -(5 workers, AVX2, 8 GB/s: the trellis decode is compute bound, 48 cycles a block) and about 5 over the link. Both move about 10 experts a -millisecond, so the link takes a bigger share than for the i-quants: for packs with trellis experts the engine scales the share with a 12 GB/s -reference instead of 20 (0.48 instead of 0.29 here; fixed 0.45 and 0.60 measured the same; 0.40 gained only 7%). The card's kernels are not the -limit: 20 microseconds an expert (53-68 GB/s of weights) against 18 for IQ3_S, 15 experts take 0.3 ms. +**Against CUDA.** Gyro-S on an RTX 3060 and on the A750, both with 64 cached experts, no PCIe share and no CPU prompt +share, ten prompts of 26 to 100 tokens, 48 greedy tokens with the MTP draft: 7 of the 10 prompts give the same 48 tokens. +The other three agree for 9, 9 and 6 tokens and then differ at a near-tie. The first-token KL is 1.3e-4 to 1.1e-2 and +the top token is always the same. A 1,406-token prompt gives the same 16 tokens on both. -Smaller prompt chunks leave room for more cached experts and cost prompt speed: `--prefill 1024` gives 1,090 slots and +13% decode on a -1,406-token prompt but -24% prompt speed, `--prefill 512` 1,429 slots, -36%. The default stays `--prefill auto`. +> **Measured on:** Arc A750 8 GB, Ryzen 5 5600X, 62 GB RAM, setup's config (32K context, 300 MiB VRAM reserve, MTP draft +> of 4, PCIe probe 10.4 GB/s), median of 5 interleaved pairs, engine 0.2.2 era. -**Gyro-M on 8 GB.** Its dense weights are bigger (2.9 GiB of native projections against 1.9): with setup's 32K context the card is 750 MiB -short of one cache slot and the engine stops with that sentence. It starts with `--max-context 8192 --vram-reserve-mib 100 --prefill 1024` -(412 slots). A card with 12 GB or more does not need that. +| | Result | +| --- | ---: | +| Gyro-S prompt, 1,406 tokens | 82.0 tok/s | +| Gyro-S prompt, 2,377 tokens | 101.9 tok/s | +| Gyro-S prompt, 20,000 tokens | 195 tok/s | +| Gyro-S decode | 15.8 tok/s | +| Gyro-M decode (8K context) | 14.2 tok/s (12.6 to 14.4); prompt 62 tok/s at 1,406 tokens | -**Memory.** Gyro-S needs about 27.6 GB of VRAM and RAM together; the n-gram table (28 GB) stays on the SSD, so on an Arc setup does not -put `--ple-io ram` in the config for Gyro (it does for the other packs on an i915 card). One run used about 24 GB of RAM for the -expert arena. Keep the file on a fast SSD. +Decode speed is the CPU's and the link's. With 8 GB the card holds 410 of the 24,576 experts, so about 15 experts per +layer and token come from the CPU pool (5 workers, AVX2, 8 GB/s; the trellis decode is compute bound) and about 5 over +the link. For packs with trellis experts the engine scales the PCIe share with a 12 GB/s reference instead of 20 (0.48 +instead of 0.29 here). The card's kernels are not the limit: 20 microseconds an expert, against 18 for IQ3_S. -## Decode speed on NVIDIA (engine 0.2.2) +Smaller prompt chunks leave room for more cached experts and cost prompt speed. On the 1,406-token prompt, +`--prefill 1024` gives 1,090 slots, +13% decode and -24% prompt speed. `--prefill 512` gives 1,429 slots and -36% +prompt speed. The default stays `--prefill auto`. -The trellis decode kernels read the codebook (2048 two-byte entries, 8 random lookups for every 16 weights) from the global -table. On an RTX 3060 that made the gate/up and down kernels wait on the L1 cache, not on memory: 18 L1 sectors for every -lookup request, the card's L1 at 64% busy while the DRAM was 27% busy. The kernels now keep the 4 KB codebook in shared memory -(`STRATA_IQ_STAGE_GRID=0` reads it in place again) and take four rows per warp, 32 rows per thread block, so the copy is shared -and no lane idles in the 160-dot gate/up rows or the 40-dot down rows. The sums are added in the same order as before, so the -answers are bit for bit the old ones: the identity gate (10 prompts, 4096 and 20000 token prompts) is identical on Gyro-S, -Gyro-M, Q2_0 and IQ3_XXS, and the Q2_0 and IQ3_XXS kernels did not change. A Tesla P100 (compute capability 6.0) runs its own variant of the same idea, STRATA_TQ_STAGE (Gyro-S +30% there, 5 pairs). +**Gyro-M on 8 GB.** Its dense weights are bigger (2.9 GiB of native projections against 1.9). With setup's 32K context +the card is 750 MiB short of one cache slot and the engine stops with that message. It starts with +`--max-context 8192 --vram-reserve-mib 100 --prefill 1024` (412 slots). A card with 12 GB or more does not need that. -RTX 3060 12 GB, Core Ultra 7, Ubuntu, CUDA 13.4, default flags (MTP draft 4, auto cache), 5 interleaved pairs of whole engine -starts, medians of the story, code and facts prompts (200 tokens each), the pinned Gyro files, engine 0.2.1 against 0.2.2. +**Memory.** Gyro-S needs about 27.6 GB of VRAM and RAM together. The n-gram table (28 GB) stays on the SSD, so on an Arc +setup does not put `--ple-io ram` in the config for Gyro (it does for the other packs on an i915 card). One run used +about 24 GB of RAM for the expert arena. -| | decode before, tok/s | after | gain | pairs faster | 4K / 20K prompt | -| --- | ---: | ---: | ---: | --- | --- | -| Gyro-S | 54.3 | 62.5 | +15.2% | 5 of 5 (10 of 10 requests per prompt) | unchanged (0.996, 1.000) | -| Gyro-M | 50.2 | 54.8 | +9.1% | 5 of 5 (10 of 10) | unchanged (1.004, 0.999) | -| Q2_0 (not changed) | 66.4 | 67.1 | 1.010 (noise) | 3 of 5 | unchanged (1.003, 0.999) | +## On a Mac -Against Q2_0 on the same card, before: Gyro-S 54.2 against 64.4 tok/s (16% slower), Gyro-M 50.5 against 65.6 (23% slower). -After: Gyro-S 61.1 against 65.5 (7% slower), Gyro-M 54.1 against 64.5 (16% slower). What is left of Gyro-M's gap is mostly its dense weights: in a -decode profile its Q8_0 matrix-vector kernels take 26% of the GPU time, and they read as many bytes as the card's memory -delivers. The trellis kernels themselves are limited by the shared-memory bank conflicts of the random codebook lookups now. Prompt speed is not changed by this: Gyro prompts stay 7-8% (Gyro-S) and 8-9% (Gyro-M) slower than Q2_0's. +Apple Silicon Macs run Gyro-S and Gyro-M with Metal kernels (`metal/kernels`), the same published files and the same +Hadamard-folded pack. What a Mac needs, the measured speeds on a 32 GB Mac mini and what was checked are in +[MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts). Setup: -## Credit and licenses +```sh +./setup.sh --family gyro --model Gyro-S +``` -The trellis types, the Hadamard rotation, the CPU and GPU kernels and the pack step were written by Laurent Zuijdwijk -(agentionai) in agentionai/Strata `rc2` (9ff02c9, MIT, Strata v0.1.40.3 plus Gyro) and ported here commit by commit with -his authorship kept; `third_party/ggml/ggml-common-tq.h` and the MMQ instances come from agentionai/llama.cpp (MIT, the -ggml authors). The weights are agentionai's quantization of Qwen3.8-Flash-Next under the Qwen Community License 1.0 -(). +The app and `strata launch` list it under Gyro models. -## How it runs (by hand) +## Install by hand -The rest of this page is the authors' own guide for a manual install and their measurements on their machines (RTX 5090, -3090, 4090 and a Strix Halo, with Strata rc2). The numbers are theirs; ours are in the next section. +Setup does these steps for you. By hand, from a [source checkout](SOURCE_INSTALL.md): -## 1. Build +### 1. Build -Setup does these steps for you; by hand, from this checkout. For the fast prompt path Strata's GGML dependency is the agentionai llama.cpp fork, which has the trellis types: +For the fast prompt path, Strata's GGML dependency can be the agentionai llama.cpp fork, which has the trellis types: ```sh git clone --depth 1 https://github.com/agentionai/llama.cpp ../agention-llama.cpp -# NVIDIA. RTX 50 series: use CUDA 13 (CUDA 12.8 crashes the prompt path on sm_120: issues #220, #224) +# NVIDIA. RTX 50 series: use CUDA 13 (CUDA 12.8 crashes the prompt path on sm_120) cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \ -DCMAKE_CUDA_ARCHITECTURES=120 -DSTRATA_GGML_DIR=$PWD/../agention-llama.cpp -# AMD (experimental: kernel parity and an end-to-end run validated on gfx1151; in a non-root container add -# --group-add $(getent group render | cut -d: -f3), otherwise HIP finds no device) +# AMD (experimental; in a non-root container add --group-add $(getent group render | cut -d: -f3), +# otherwise HIP finds no device) cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_HIP=ON -DSTRATA_PREFILL_MMQ=ON \ -DCMAKE_HIP_ARCHITECTURES=gfx1201 -DSTRATA_GGML_DIR=$PWD/../agention-llama.cpp cmake --build build -j python3 -m venv .venv && .venv/bin/pip install -r requirements.txt ``` -## 2. Download and pack +### 2. Download and pack ```sh .venv/bin/pip install huggingface_hub hf_transfer @@ -183,11 +213,10 @@ HF_HUB_ENABLE_HF_TRANSFER=1 .venv/bin/hf download agentionai/Qwen3.8-Flash-Next- build/native_expert_parity --load-check packs/gyro-s models/Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf ``` -The pack is 1.5 GB (dense weights, tokenizer, the Hadamard and expert tables); the experts are read from the -GGUF. Keep the GGUF on a fast local SSD: the n-gram table is read from it for every token. -`--load-check` must end with `0 failures`. +The pack is 1.5 GB (dense weights, tokenizer, the Hadamard and expert tables). The experts are read from the GGUF, so +keep it on a fast local SSD. `--load-check` must end with `0 failures`. -## 3. Draft head +### 3. Draft head ```sh .venv/bin/python tools/mtp_fetch.py fetch --out mtp @@ -196,7 +225,7 @@ GGUF. Keep the GGUF on a fast local SSD: the n-gram table is read from it for ev cp data/draft_vocab.bin mtp/rt/draft_vocab.bin ``` -## 4. Serve +### 4. Serve Save as `strata-gyro-s.json` at the repository root: @@ -223,147 +252,126 @@ Save as `strata-gyro-s.json` at the repository root: .venv/bin/python -m serve.server --engine strata --config strata-gyro-s.json --port 8080 ``` -The web interface is at `http://127.0.0.1:8080`; API clients use `http://127.0.0.1:8080/v1` (OpenAI) or -`/v1/messages` (Anthropic). `--expert-cache auto` sizes the expert cache to the free VRAM: on a 32 GB card at -128k context it holds 23,464 of 24,576 experts (22.9 GiB). On smaller cards it holds fewer and streams the rest -from RAM; nothing else changes. +The web interface is at `http://127.0.0.1:8080`. API clients use `http://127.0.0.1:8080/v1` (OpenAI) or `/v1/messages` +(Anthropic). `--expert-cache auto` sizes the expert cache to the free VRAM. On a 32 GB card at 128K context it holds +23,464 of 24,576 experts (22.9 GiB). On smaller cards it holds fewer and streams the rest from RAM. -## Measured here (the port, 2026-10-15/16) +## Numerics -One AMD Radeon AI PRO R9700 (gfx1201, ROCm 7), one card of four, exclusive box lock, `--prefill auto --spec 4 --spec-min-p 0.5 ---mtp --kv int8`, 22K context, interleaved pairs (the arm that goes first alternates), medians, the files of the first -Gyro release (2026-10-07). A = agentionai/Strata rc2 9ff02c9, B = this port. +- **Decode** (Strata's own expert kernels) reads the trellis weights through an int8 copy of the codebook with one + global scale (`tq_lut_i8`, relative RMS error 0.85% of the codebook's standard deviation, far below the quantization + error itself), against q8_1 activations. +- **Prompt processing** decodes them differently. The FP16 path uses the fp16 codebook (`tq_lut_f16`, what GGML's own + dequantizer reads). The MMQ path uses the llama.cpp fork's kernels, which read the int8 codebook as well. +- Decode and prompt results therefore agree within tolerance, not bit for bit. `native_expert_parity` checks one expert + on the decode kernels against GGML's float dequantizer and a float SwiGLU: relative L1 error below 3e-2, the bound + every expert type gets there. Its "bitwise equal" lines compare Strata's decode-once and per-entry decode kernels with + each other, not with a float reference. +- Strata's own CPU kernels for the trellis types (`src/kernels/cpu/tq_*.cpp`, AVX2 and AVX-512, picked at run time) + take about 45% of ggml-cpu's time per expert on AVX2 and decode a block once for all the tokens of a verify window + (`build/tq_cpu_parity --bench` measures them on your CPU). `STRATA_NO_TQ_KERNELS=1` goes back to ggml-cpu. + `STRATA_TQ_ISA=avx2` or `scalar` caps the instruction set. -| B, ggml of | file | decode tok/s A / B (B/A) | pairs B faster | prompt 4K ms A / B | prompt 20K ms A / B | +## Known limits + +- The prompt helper on a second GPU does not run for a Gyro pack. Setup of two cards still runs the layer split and the + decode peer. The prompt peer says it is not used for a Hadamard-folded pack, and the prompt stays on the first card. + Prompt processing on a peer GPU over P2P is not validated end to end. +- Token parity against llama.cpp has not been measured. +- `strata --tokens-file ... --max-new N` (benchmark mode) decodes N tokens past the end of the answer. Compare speeds + only up to the answer's end. +- AMD: the trellis kernels pass parity on gfx1151 (ROCm 7.2.1) and a short end-to-end run there generated correctly + (28 to 31 tokens/s). There was no speed tuning or RDNA4 speed run. An AMD GPU other than gfx1151 runs with a warning. +- Gyro-M: 29.9 GiB of experts do not all fit a 32 GB card next to the KV cache. `auto` streams the rest. + +## Credit and licenses + +The trellis types, the Hadamard rotation, the CPU and GPU kernels and the pack step were written by Laurent Zuijdwijk +(agentionai) in agentionai/Strata `rc2` (MIT) and ported here commit by commit with his authorship kept. +`third_party/ggml/ggml-common-tq.h` and the MMQ instances come from agentionai/llama.cpp (MIT, the ggml authors). The +weights are agentionai's quantization of Qwen3.8-Flash-Next under the Qwen Community License 1.0 +(). + +## History / older measurements + +### Engine 0.2.1 against 0.2.2 (decode kernels) + +> **Measured on:** RTX 3060 12 GB, Core Ultra 7, Ubuntu, CUDA 13.4, default flags, 5 interleaved pairs of whole engine +> starts, the pinned Gyro files. + +| | Decode before, tok/s | After | Gain | 4K / 20K prompt | +| --- | ---: | ---: | ---: | --- | +| Gyro-S | 54.3 | 62.5 | +15.2% | unchanged (0.996, 1.000) | +| Gyro-M | 50.2 | 54.8 | +9.1% | unchanged (1.004, 0.999) | +| Q2_0 (not changed) | 66.4 | 67.1 | 1.010 (noise) | unchanged (1.003, 0.999) | + +Against Q2_0 before the change: Gyro-S 54.2 against 64.4 tok/s (16% slower), Gyro-M 50.5 against 65.6 (23% slower). +Gyro-S gained +30% on a Tesla P100 from the `STRATA_TQ_STAGE` variant (5 pairs). + +### Arc A750 before the XMX prompt path + +> **Measured on:** Arc A750 8 GB, setup's config, median of 5 interleaved pairs. + +| | Before | Later | +| --- | ---: | ---: | +| Gyro-S prompt, 1,406 tokens | 40.2 tok/s | 82.0 tok/s | +| Gyro-S decode | 13.7 tok/s | 15.8 tok/s | + +### The port against agentionai's rc2 (2026-10-15/16) + +> **Measured on:** one AMD Radeon AI PRO R9700 (gfx1201, ROCm 7), one card of four, `--prefill auto --spec 4 +> --spec-min-p 0.5 --mtp --kv int8`, 22K context, interleaved pairs, medians, the first Gyro release files (2026-10-07). +> A = agentionai/Strata rc2, B = this port (Strata 0.1.45 base). + +| B's ggml | File | Decode tok/s A / B | Pairs B faster | Prompt 4K ms A / B | Prompt 20K ms A / B | | --- | --- | ---: | --- | ---: | ---: | -| agentionai fork | Gyro-S | 85.4 / 89.7 (1.051) | 5/5 | 5418 / 5313 (0.98) | 14358 / 13859 (0.97) | -| agentionai fork | Gyro-M | 72.7 / 82.7 (1.136) | 5/5 | 5183 / 5520 (1.07) | 13421 / 14164 (1.06) | -| upstream + overlay (the ready-made engine) | Gyro-S | 85.3 / 89.6 (1.050) | 5/5 | 5413 / 5298 (0.98) | 14372 / 13854 (0.96) | -| upstream + overlay | Gyro-M | 72.6 / 82.6 (1.138) | 5/5 | 5682 / 5524 (0.97) | 15005 / 14160 (0.94) | -| upstream, no overlay (FP16 prompt path) | Gyro-S | 85.3 / 89.6 (1.050) | 5/5 | 5425 / 6939 (1.28) | 14364 / 21677 (1.51) | +| agentionai fork | Gyro-S | 85.4 / 89.7 | 5/5 | 5418 / 5313 | 14358 / 13859 | +| agentionai fork | Gyro-M | 72.7 / 82.7 | 5/5 | 5183 / 5520 | 13421 / 14164 | +| upstream + overlay (the ready-made engine) | Gyro-S | 85.3 / 89.6 | 5/5 | 5413 / 5298 | 14372 / 13854 | +| upstream + overlay | Gyro-M | 72.6 / 82.6 | 5/5 | 5682 / 5524 | 15005 / 14160 | +| upstream, no overlay (FP16 prompt path) | Gyro-S | 85.3 / 89.6 | 5/5 | 5425 / 6939 | 14364 / 21677 | -Decode is faster than rc2's because the port sits on the newer Strata (0.1.45); the same engine change makes IQ3_XXS 7% faster -(rc2 86.9, port 93.1 tok/s, 2 pairs). Prompts of the ISTA files gained 11-16% from it too (IQ3_XXS 4K 6184 to 5480 ms, 20K -16587 to 13864 ms). Gyro-M's first run against rc2 showed its prompts 5-7% slower; in the second rc2's own Gyro-M prompts -took 15.0 s instead of 13.4 s (the 92 GB file's page-cache state), the port's stayed at 14.2 s, and Gyro-M's prompts are 3-6% -faster than rc2's: that difference was rc2's baseline, not a regression. Without the overlay (plain upstream ggml) the trellis -prompt kernels are missing and prompts go through the FP16 path: 28% (4K) to 51% (20K) slower than rc2, and a different -rounding. Decode is unaffected either way. +Without the overlay, prompts are 28% (4K) to 51% (20K) slower than rc2's. Decode is unaffected. The identity gate (10 +prompts, 4096 and 20000 token prompts) was identical between rc2 and the port for Gyro-S and Gyro-M, and between the +split base and the port for Q2_0 and IQ3_XXS, on AMD gfx1201 and NVIDIA sm_86. The pack step writes byte-identical files +to rc2's tools for Gyro-S. -Identity (the fixed gate flags, 10 prompts, exact_pp 4096 and 20000 tokens, same card for both arms): +### The authors' measurements (rc2) -- Q2_0 and IQ3_XXS, f-split against the port (upstream ggml): 10/10 identical, residual dumps and outputs IDENTICAL. -- Gyro-S and Gyro-M, rc2 against the port with the fork's ggml and with upstream + overlay: 10/10 identical; exact_pp 4096 / - 20000 residual dumps and outputs IDENTICAL (run alone; a run beside other engines on the box once showed a different dump). -- Q2_0 and IQ3_XXS, f-split against the port with the overlay on, on AMD gfx1201 and on NVIDIA sm_86: IDENTICAL. -- The pack step (`tools/iq_pack.py --compat-bf16`) writes byte-identical files to rc2's tools for Gyro-S. +> **Measured on:** the authors' machines, agentionai/Strata rc2, 2026-10-02 to 04; greedy or the stated sampling, MTP +> draft `--spec 4 --spec-min-p 0.5`, int8 KV. llama.cpp is the agentionai fork with the CUDA trellis fixes. The numbers +> are theirs. -## Measured (the authors', rc2) +Decode tokens/s, server protocol (128K context, `auto` cache, thinking off, temperature 0.7, top-k 20, top-p 0.95, +min-p 0.05, 512 new tokens, 3 repetitions): -All runs: greedy or the stated sampling, same prompt tokens as llama.cpp, MTP draft `--spec 4 --spec-min-p 0.5`, -int8 KV. "llama.cpp" is the agentionai fork with the CUDA trellis fixes (in its `main` since 2026-10-04). +| GPU and file | Experts cached | Prose | JSON | Code | Copy | Prefill 4K | Prefill 16K | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| RTX 5090 32 GB, Gyro-S, Strata | 23,463 | 131.7 | 200.1 | 195.2 | 201.8 | 3,456 | 4,883 | +| RTX 5090, Gyro-S, llama.cpp with MTP draft | | 116.8 | 213.0 | 179.0 | 220.7 | 2,520 | 2,471 | +| RTX 5090, Gyro-S, llama.cpp no draft | | 121.8 | 123.2 | 122.5 | 121.2 | | | +| RTX 5090, Gyro-M, Strata | 18,007 | 113.8 | 170.8 | 153.0 | 175.7 | 3,369 | 4,854 | +| RTX 3090 24 GB, Gyro-S, Strata | 15,756 | 31.8 | 45.2 | 42.2 | 45.1 | 1,402 | 1,863 | +| RTX 3090, Gyro-M, Strata | 11,811 | 27.8 | 39.4 | 34.4 | 37.7 | 1,363 | 1,910 | +| RTX 3090, Gyro-S, llama.cpp `-ncmoe 10`, 32K, no draft | | 16.0 | 16.2 | 16.2 | 16.5 | 468 | 479 | -### RTX 5090 (32 GB), Gyro-S, all experts on the GPU (2026-10-03) +Single prompts of 34 to 177 tokens, 384 new tokens, `--max-context 16384`, Gyro-S, RTX 5090, all experts cached +(2026-10-03): prose 139.9, JSON 221.3, code 209.0 tok/s. Gyro-M on the same card with llama.cpp needs `-ncmoe 8` and +decodes at 70.8 tok/s without a draft. A single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3, +JSON 76.3 tok/s. On 24 GB the CPU matters (the box had a Zen 2 EPYC, AVX2 only). These Strata rows predate Strata's +own trellis CPU kernels. -Decode tokens/s, single prompts of 34-177 tokens, 384 new tokens, `--max-context 16384` (all 24,576 experts cached): +Smaller expert caches, emulated on the RTX 5090 (2026-10-02, before the int8 decode kernels). A real 8 or 16 GB card is +a slower GPU, often on a slower link: read these rows as "it stays usable", not as a prediction for a specific card. -| | prose | JSON | code | -|---|---:|---:|---:| -| Strata | 139.9 | 221.3 | 209.0 | - -Server, rc2's `gyro_setup.sh` config (128k context, 23,463 experts cached), thinking off, temperature 0.7, -top-k 20, top-p 0.95, min-p 0.05, 512 new tokens, 3 repetitions: - -| | prose | JSON | code | copy | -|---|---:|---:|---:|---:| -| Strata | 131.7 | 200.1 | 195.2 | 201.8 | -| llama.cpp, MTP draft | 116.8 | 213.0 | 179.0 | 220.7 | -| llama.cpp, no draft | 121.8 | 123.2 | 122.5 | 121.2 | - -Long real-text prompts, prefill tokens/s: 4k tokens 3,456 (llama.cpp 2,520), 16k tokens 4,883 (llama.cpp 2,471). - -### RTX 5090, Gyro-M (2026-10-03) - -The experts (29.9 GiB) do not all fit next to the KV cache; `auto` caches 18,007 of 24,576. Same server protocol: -prose 113.8, JSON 170.8, code 153.0, copy 175.7 tokens/s; prefill 4k 3,369, 16k 4,854 tokens/s. llama.cpp needs -`-ncmoe 8` for this file on a 32 GB card and decodes at 70.8 tokens/s without a draft. - -### RTX 3090 (24 GB), 2026-10-03 - -Same server protocol, 128k context, `auto` cache: - -| | experts cached | prose | JSON | code | copy | prefill 4k | prefill 16k | -|---|---:|---:|---:|---:|---:|---:|---:| -| Gyro-S | 15,756 | 31.8 | 45.2 | 42.2 | 45.1 | 1,402 | 1,863 | -| Gyro-M | 11,811 | 27.8 | 39.4 | 34.4 | 37.7 | 1,363 | 1,910 | -| llama.cpp, Gyro-S, `-ncmoe 10`, 32k context, no draft | (10 layers on the CPU) | 16.0 | 16.2 | 16.2 | 16.5 | 468 (`-ncmoe 11`) | 479 | - -On a 24 GB card, cache misses run on the CPU, so the CPU matters: this box had a Zen 2 EPYC (AVX2, no AVX-512). -The llama.cpp row is the build before the CUDA trellis fixes; those added about 3 % at 24 GB, where the CPU layers -dominate. - -The Strata rows predate Strata's own CPU kernels for the trellis types (`src/kernels/cpu/tq_*.cpp`, AVX2 and -AVX-512, picked at run time). Before them a cache miss ran ggml-cpu's `vec_dot`, one token at a time; the new kernels -take about 45 % of its time per expert on AVX2 and decode a block once for all the tokens of a verify window -(`build/tq_cpu_parity --bench` measures them on your CPU). `STRATA_NO_TQ_KERNELS=1` goes back to ggml-cpu; -`STRATA_TQ_ISA=avx2` or `scalar` caps the instruction set (for comparisons). - -### Smaller caches (RTX 5090, 2026-10-02, before the int8 decode kernels) - -| Expert cache | prose | JSON | code | -|---|---:|---:|---:| +| Expert cache | Prose | JSON | Code | +| --- | ---: | ---: | ---: | | 24,576 slots (all experts) | 116.4 | 194.3 | 173.5 | | 22,000 slots | 111.4 | 175.7 | 166.1 | | 9,000 slots (about a 16 GB card's room) | 81.4 | 74.0 | 96.8 | | 3,000 slots (about an 8 GB card's room) | 48.0 | 52.4 | 57.9 | -The smaller caches were emulated on the 5090 (fast GPU, PCIe 5.0 x16 at ~51 GB/s). A real 8 or 16 GB card is a -slower GPU, often on a slower link: read those rows as "it stays usable", not as a prediction for a specific card. -The decoded output was read and is fluent and correct; token-level parity against llama.cpp has not been -checked. A real chat at 128k context worked through the web interface. - -### Two GPUs (2026-10-04, 2× RTX 4090, no NVLink/P2P) - -- **Layer split** (Strata splits the layers across both cards automatically) works with Gyro, including the - idle-card prompt helper (`STRATA_PREFILL_HELP=1`). Repeated runs give identical output. -- **Decode with a second card** (`--peer-device 1` or `--expert-cache-device1 auto`) works and matches single-GPU output. -- **Prompt processing on the peer card** needs GPU-to-GPU access (NVLink or data-centre GPUs); without it Strata keeps - the prompt on the first GPU and logs `no P2P`. - -Single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3, JSON 76.3 tokens/s. - -## Numerics - -- **Decode** (Strata's own expert kernels) reads the trellis weights through an int8 copy of the codebook with one - global scale (`tq_lut_i8`, relative RMS error 0.85 % of the codebook's standard deviation, far below the - quantization error itself), against q8_1 activations. -- **Prompt processing** decodes them otherwise: the FP16 path through the fp16 codebook (`tq_lut_f16`, what GGML's - own dequantizer reads); the MMQ path through the llama.cpp fork's kernels, which read the int8 codebook as well - on the fork's current `main`. -- So decode and prompt results agree within tolerance, not bit for bit. `native_expert_parity` checks one expert - on the decode kernels against GGML's float dequantizer and a float SwiGLU: relative L1 error below 3e-2, the - bound every expert type gets there. Its "bitwise equal" lines compare Strata's decode-once and per-entry decode - kernels with each other, not with a float reference. - -## Known limits - -- **Not in this port** (rc2 has them; see the list at the end of this page): the prompt helper on a second GPU for a Gyro - pack. Setup of two cards still runs the layer split and the decode peer; the prompt peer says it is not used for a - Hadamard-folded pack and the prompt stays on the first card. Gyro-L is not supported. -- **Source builds:** the SYCL tree has the trellis kernels (Intel Arc, above), and so has the Metal engine (On a Mac, below). - -- **Not measured yet:** token parity against llama.cpp. -- **Benchmark mode only:** `strata --tokens-file ... --max-new N` decodes N tokens past the end of the answer; - compare speeds only up to the answer's end. -- **AMD:** the trellis kernels pass parity on gfx1151 (ROCm 7.2.1) and a short end-to-end run there generated - correctly (~28-31 tokens/s); no speed tuning or RDNA4 run yet. -- **Gyro-M:** 29.9 GiB of experts do not all fit a 32 GB card next to the KV cache; `auto` streams the rest. -- **Not validated end to end yet** (Strata runs them and logs a warning): an AMD GPU other than gfx1151, and - prompt processing on a peer GPU over P2P. - -## On a Mac - -Apple Silicon Macs run Gyro-S and Gyro-M from 0.2.2 (Metal kernels in `metal/kernels`, the same published files, the same -Hadamard-folded pack). What a Mac needs, the measured speeds on a 32 GB Mac mini and what was checked are in -[MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts). Setup: `./setup.sh --family gyro --model Gyro-S` (the app and -`strata launch` list it under Gyro models). +Two RTX 4090 without NVLink or P2P (2026-10-04): the layer split works with Gyro, including the idle-card prompt helper +(`STRATA_PREFILL_HELP=1`), and repeated runs give identical output. Decode with a second card (`--peer-device 1` or +`--expert-cache-device1 auto`) matches single-GPU output. Prompt processing on the peer card needs GPU-to-GPU access +(NVLink or data-centre GPUs). Without it Strata keeps the prompt on the first GPU and logs `no P2P`. diff --git a/docs/IMAGES.md b/docs/IMAGES.md index 108642a1d..9d7b866ab 100644 --- a/docs/IMAGES.md +++ b/docs/IMAGES.md @@ -1,88 +1,115 @@ # Image generation (Qwen-Image-2.1-Turbo) -Strata's server can also make images: `POST /v1/images/generations`, OpenAI's images API. The model is -[Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) (8 steps, CFG 1) in +This page is for people who want Strata's server to make images through `POST /v1/images/generations`, OpenAI's +images API. It is optional, off unless you ask for it, and separate from reading pictures in chat (for that, use +`strata launch --vision on` with a supported model and reader). + +It needs a [source checkout](SOURCE_INSTALL.md) and build tools. The steps below are not part of a normal app install. + +The model is [Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) (8 steps, CFG 1) in [Unsloth's GGUF quants](https://huggingface.co/unsloth/Qwen-Image-2.1-Turbo-GGUF), run by [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) (`sd-cli`), which setup compiles for your GPU. -It is optional and off unless you ask for it. - -The model is under the **Qwen Research License**, not Apache 2.0: read +It is under the **Qwen Research License**, not Apache 2.0. Read [its license](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo/blob/main/LICENSE) before you use the images commercially. -## Set it up +## Quick start - ./setup.sh --setup --images qwen-image (Linux) - START-HERE.bat --setup --images qwen-image (Windows) +```sh +./setup.sh --setup --images qwen-image # Linux +START-HERE.bat --setup --images qwen-image # Windows +``` -or, for an install that is already set up, without running the whole setup again: +For an install that is already set up, without running the whole setup again: - .venv/bin/python tools/image/install.py --config strata-.json +```sh +.venv/bin/python tools/image/install.py --config strata-.json +``` -Setup compiles `sd-cli` (stable-diffusion.cpp at a pinned commit) into `engine-image/`, downloads three files into -the data folder's `models/qwen-image-2.1-turbo/` and checks each against its SHA-256, and writes an `"images"` section -into the model's run config. +Then request an image: -| piece | file | size | -|---|---|---| -| denoiser (by GPU memory) | `qwen-image-2.1-turbo-Q4_K_M.gguf` up to 8 GB, `Q6_K` up to 12 GB, `Q8_0` above | 4.2 / 6.3 / 7.6 GB | -| text encoder | `Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf` | 5.1 GB | +```sh +curl http://127.0.0.1:8080/v1/images/generations -H "Content-Type: application/json" \ + -d '{"prompt": "a cartoon sloth mascot waving, flat vector illustration", "size": "1024x1024", "seed": 42}' +``` + +## What setup does + +Setup compiles `sd-cli` (stable-diffusion.cpp at a pinned commit) into `engine-image/`, downloads three files into the +data folder's `models/qwen-image-2.1-turbo/`, checks each against its SHA-256 and writes an `"images"` section into the +model's run config. + +| Piece | File | Size | +| --- | --- | --- | +| Denoiser, chosen by GPU memory | `qwen-image-2.1-turbo-Q4_K_M.gguf` up to 8 GB, `Q6_K` up to 12 GB, `Q8_0` above | 4.2 / 6.3 / 7.6 GB | +| Text encoder | `Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf` | 5.1 GB | | VAE | `qwen_image_2.1_vae_bf16.safetensors` | 0.7 GB | -Options: `--image-quant Q4_K_M|Q6_K|Q8_0` (a bigger one than recommended is kept, with a warning), -`--image-backend cuda|hip|vulkan|cpu`, `--image-gpu-policy auto|keep|shrink|unload`, `--images no` (takes the section -out of the config). +### Setup options -Backends: CUDA on NVIDIA (needs the CUDA Toolkit), HIP on AMD cards with their own memory (Linux, the same ROCm as the -engine), Vulkan on AMD APUs/iGPUs (Radeon 780M, Strix Point/Halo), Intel Arc and AMD under Windows. A Linux PC without -the Vulkan development files gets the LunarG Vulkan SDK unpacked into `third_party/` (no sudo); on Windows install the -Vulkan SDK first. On Vulkan setup adds `--vae-tiling --vae-tile-overlap 0.25`: an untiled 1024x1024 VAE decode on a Radeon 780M is one -GPU job long enough to trip the kernel driver's job timeout (no image), and a quarter's overlap decodes in half the -time of sd.cpp's default half overlap. +| Option | Effect | +| --- | --- | +| `--image-quant Q4_K_M\|Q6_K\|Q8_0` | Picks the denoiser. A bigger one than recommended is kept, with a warning. | +| `--image-backend cuda\|hip\|vulkan\|cpu` | Picks the backend. | +| `--image-gpu-policy auto\|keep\|shrink\|unload` | Sets `images.gpu_policy` ([below](#sharing-the-gpu-with-the-text-model)). | +| `--images no` | Takes the section out of the config. | -## Use it +### Backends - curl http://127.0.0.1:8080/v1/images/generations -H "Content-Type: application/json" \ - -d '{"prompt": "a cartoon sloth mascot waving, flat vector illustration", "size": "1024x1024", "seed": 42}' +| Backend | Used for | +| --- | --- | +| CUDA | NVIDIA. Needs the CUDA Toolkit. | +| HIP | AMD cards with their own memory, on Linux, with the same ROCm as the engine. | +| Vulkan | AMD APUs and iGPUs (Radeon 780M, Strix Point and Halo), Intel Arc, and AMD under Windows. | +| CPU | Fallback. | -| field | | -|---|---| -| `prompt` | the description (required) | -| `size` | `WIDTHxHEIGHT`, each side a multiple of 32 from 256 to 2048 (`max_side`); default and `auto`: `1024x1024` | -| `n` | images, 1 to 4 (`max_n`); image *i* uses seed + *i* | -| `response_format` | `b64_json` (default) or `url` (`/v1/images/files/.png`; the server keeps the newest 64) | -| `seed` | Strata's: a whole number for a repeatable image (default: random; the answer says which) | -| `steps` | Strata's: 8 (Turbo's schedule). Another count uses stable-diffusion.cpp's own schedule, which Turbo was not made for | +A Linux PC without the Vulkan development files gets the LunarG Vulkan SDK unpacked into `third_party/` (no sudo). On +Windows, install the Vulkan SDK first. On Vulkan, setup adds `--vae-tiling --vae-tile-overlap 0.25`. An untiled +1024x1024 VAE decode on a Radeon 780M is one GPU job long enough to trip the kernel driver's job timeout, and the +quarter overlap decodes in half the time of sd.cpp's default half overlap. -The answer is OpenAI's shape (`created`, `data[].b64_json` or `data[].url`, `revised_prompt` = the prompt) plus +## Request fields + +| Field | Meaning | +| --- | --- | +| `prompt` | The description (required). | +| `size` | `WIDTHxHEIGHT`, each side a multiple of 32 from 256 to 2048 (`max_side`). Default and `auto`: `1024x1024`. | +| `n` | Images, 1 to 4 (`max_n`). Image *i* uses seed + *i*. | +| `response_format` | `b64_json` (default) or `url` (`/v1/images/files/.png`; the server keeps the newest 64). | +| `seed` | Strata's: a whole number for a repeatable image. Default: random; the answer says which. | +| `steps` | Strata's: 8 (Turbo's schedule). Another count uses stable-diffusion.cpp's own schedule, which Turbo was not made for. | + +The answer has OpenAI's shape (`created`, `data[].b64_json` or `data[].url`, `revised_prompt` = the prompt) plus `data[].seed` and a `strata` object with the seconds, steps and backend. The settings are the model card's for stable-diffusion.cpp: 8 steps, CFG 1, euler, Turbo's own sigmas, `--diffusion-fa`. An image from the API is the PNG -plain `sd-cli` writes with the same arguments, byte for byte. +that plain `sd-cli` writes with the same arguments, byte for byte. ## Sharing the GPU with the text model -One job at a time: an image job takes the same turn a chat request takes, so a chat request waits for the image and -the other way round. What the text model does meanwhile is the config's `images.gpu_policy`: +One job runs at a time. An image job takes the same turn a chat request takes, so a chat request waits for the image +and the other way round. The config's `images.gpu_policy` decides what the text model does meanwhile. -| policy | the text model during an image job | -|---|---| -| `auto` (default) | `keep` when the image job fits next to it (an APU always), else `shrink` when the engine was started with `--vram-elastic`, else `unload` | -| `keep` | stays loaded and idle | -| `shrink` | gives VRAM from its expert cache (as `POST /v1/vram`) and takes it back afterwards; the conversation stays | -| `unload` | its conversation is saved to a file, the engine is stopped, and the next start restores the conversation (`"reload": "eager"` starts it again right after the image; default `lazy`: with the next chat request) | +| Policy | The text model during an image job | +| --- | --- | +| `auto` (default) | `keep` when the image job fits next to it (an APU always), else `shrink` when the engine was started with `--vram-elastic`, else `unload`. | +| `keep` | Stays loaded and idle. | +| `shrink` | Gives VRAM from its expert cache (as `POST /v1/vram`) and takes it back afterwards. The conversation stays. | +| `unload` | Its conversation is saved to a file and the engine is stopped. The next start restores the conversation. `"reload": "eager"` starts it again right after the image; the default `lazy` starts it with the next chat request. | -Each image runs its own `sd-cli` process, so no image memory stays on the GPU between images. Loading the three -files takes a few seconds per image from the page cache. +Each image runs its own `sd-cli` process, so no image memory stays on the GPU between images. Loading the three files +takes a few seconds per image from the page cache. -## Speed +## History / older measurements -Seconds per image through the API, Q8_0, 8 steps (one `sd-cli` process per image, loading included): +> **Measured on:** seconds per image through the API, Q8_0, 8 steps, one `sd-cli` process per image with loading +> included. Strata version not recorded. -| GPU | backend | 1024x1024 | 512x512 | -|---|---|---|---| -| RTX A4000 16 GB | CUDA | 27 | 8.8 | -| Radeon 780M (iGPU) | HIP | 131 | 29 | -| Radeon 780M (iGPU) | Vulkan | 179 | 43 | +| GPU | Backend | 1024x1024 | 512x512 | +| --- | --- | --- | --- | +| RTX A4000 16 GB | CUDA | 27 s | 8.8 s | +| Radeon 780M (iGPU) | HIP | 131 s | 29 s | +| Radeon 780M (iGPU) | Vulkan | 179 s | 43 s | -On the 780M HIP is faster but runs through the ROCm path whose queue evictions reset this GPU on current firmware -(a driver/firmware defect, not this feature; no reset was seen in these image runs); Vulkan is setup's default there, `--image-backend hip` picks HIP. +On the 780M, HIP is faster but runs through the ROCm path whose queue evictions reset this GPU on the firmware tested +(a driver or firmware defect, not this feature; no reset was seen in these image runs). Vulkan is setup's default +there. `--image-backend hip` picks HIP. diff --git a/docs/INSTALL.md b/docs/INSTALL.md index dc3dd2a7f..1e2ae6463 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -1,426 +1,148 @@ -# Installing Strata +# Install Strata -Everything about installing, starting, updating and configuring Strata, on Windows, Linux and Mac, with an NVIDIA -or an AMD graphics card or an Apple Silicon Mac. The short version is in the [README](../README.md#install). An AI -coding assistant can do all of this for you with [AI_SETUP.md](AI_SETUP.md). +This page is for anyone installing or updating Strata (version 0.2.5). Use the **desktop app** for a graphical +interface, or the **Terminal** for `strata launch` and coding tools. Both use released engine packages. A normal +install does not need a source checkout or a startup `.bat` file. -There are three ways to install it. Take the first one unless you have a reason not to: +## Quick start -1. **[The app](#the-app):** a desktop program with the engine inside. Install it, pick a model, chat. -2. **[The Terminal](#the-terminal):** one line in a terminal, then `strata launch`. For scripts, servers and AI agents. -3. **[From source](#from-source-windows):** `START-HERE.bat` on Windows, `./setup.sh` on Linux and Mac. It builds or - fetches the engine itself. +| I want | Do this | +| --- | --- | +| A window with chat | Download the installer from the [latest release](https://github.com/Niko1221/Strata/releases/latest) ([steps](#the-app)) | +| The terminal, or my coding tool | Run the one-line install, then `strata launch` ([steps](#the-terminal)) | +| DeepSeek V4.1-Flash | Pick it in the app, or run `strata launch --family deepseek41` ([model page](MODELS.md#deepseek-v41-flash)) | -> **On this page:** [What you need](#what-you-need) · [The app](#the-app) · [The Terminal](#the-terminal) · -> [From source: Windows](#from-source-windows) · [Linux](#from-source-linux) ([build from git](#build-from-git-on-linux-to-try-a-pull-request-or-change-the-code)) · [Mac](#from-source-mac) · -> [AMD cards](#amd-cards) · [Several cards](#two-or-three-cards) · [Docker](#docker-linux) · -> [Older CPUs](#older-cpus-experimental) · [Updating](#updating) · [Where things are stored](#where-things-are-stored) · -> [Setup's questions](#setups-questions) · [Tuning](#tuning-for-your-pc) · [All options](#options-without-questions) +[Requirements](#what-you-need) · [Desktop app](#the-app) · [Terminal](#the-terminal) · +[Update](#updating) · [Files and folders](#where-things-are-stored) · [Build from source](SOURCE_INSTALL.md) ## What you need -| | | +| Computer | Guide | | --- | --- | -| **GPU** | **NVIDIA** RTX 20, 30, 40 or 50 series, **12 GB VRAM or more** (8 GB runs, slowly). Measured on an RTX 5070 and an RTX 3090; RTX 20 (Turing, since 0.1.27) was tested by a contributor on an RTX 2070. **AMD** Radeon RX 7900 XT / XTX, RX 9070 / 9070 XT and Radeon AI PRO R9700 (validated), RX 7800 XT / 7700 XT and RX 9060 XT (validated by their owners), RX 6800 / 6900 series (community-reported), with 12 GB of VRAM or more. See [AMD cards](#amd-cards). | -| **RAM** | Enough for the size you pick ([which model](MODELS.md#pick-by-memory)); **64 GB** runs every size. A big GPU makes up for less RAM - the [low-RAM mode](MODELS.md#a-big-graphics-card-and-little-ram). | -| **CPU** | x86-64 with AVX2 (any Intel/AMD desktop CPU from the last ~8 years). AVX-512 (Ryzen 7000/9000) is a bit faster. Older CPUs without AVX2 are experimental and slow: [Older CPUs](#older-cpus-experimental). | -| **Disk** | The app and its engine take about 1-3 GB. ~70-80 GB free for the model, ~6 GB for the MTP layer (+1 GB with images). Q2_0 on an AVX-512 CPU also writes a one-time ~40 GB copy of its experts for the fast CPU kernel. On Linux with an AMD card, ROCm takes ~10 GB more when setup installs it. An NVMe SSD is strongly recommended: it makes the first start much faster. | -| **OS** | Windows 10/11, Linux (Ubuntu 22.04/24.04 and Debian 12/13 get everything installed automatically; the app also runs on Fedora 44), or a Mac with Apple Silicon (M1 or newer) and macOS 15 or newer ([Mac](MAC.md)). | -| **Driver** | **NVIDIA:** a current driver, version 580 or newer ([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA App). **AMD:** on Linux the kernel's amdgpu driver (no ROCm install needed); on Windows a current AMD Software: Adrenalin Edition driver ([amd.com/support](https://www.amd.com/en/support)). | +| Windows 10/11 or Linux x86-64 with an NVIDIA RTX GPU | Use the app or the Terminal. The engine for your card is downloaded for you. [Older NVIDIA cards](OLDER_GPUS.md) use the CUDA 12 engine. DeepSeek V4.1-Flash runs on RTX 30, 40 and 50 cards in the app and the Terminal. | +| Windows or Linux x86-64 with a supported AMD Radeon | [AMD guide](AMD_HIP.md), including the card-to-package table. DeepSeek V4.1-Flash runs on the Radeon AI PRO R9700 under Linux, in the app and the Terminal. | +| Linux x86-64 with Intel Arc | [Intel Arc guide](INTEL_ARC.md); install the driver/compute runtime and check model preparation requirements | +| Apple Silicon Mac, macOS 15 or newer | [Mac guide](MAC.md). Intel Macs are not supported by the Mac package. | +| AMD Ryzen AI Max / Strix Halo | [Strix Halo guide](STRIX_HALO.md) | -The driver is the only thing you install yourself. With the app, the engine is inside the installer. With the -Terminal, `strata launch` fetches the engine for your card from the signed update feed, and nothing is compiled. -Installing from source sets up the rest as well, the first time: Python 3.12 if you have none (for your user -account, no admin), a private Python environment in `.venv/`, the Strata engine, the model and the MTP draft layer. -On NVIDIA it uses the ready-made engine for RTX -20/30/40/50 and NVIDIA's CUDA libraries from pip (~0.4 GB); if no ready-made engine fits your PC, it offers to install -the build tools (Visual Studio Build Tools + CUDA Toolkit on Windows, `build-essential` + CUDA on Ubuntu) and compiles -the engine for your GPU (asks first; 20-40 minutes once). More: [details](DETAILS.md#before-you-start). +Memory and disk needs depend on the **model, quantization, context and enabled features**. Check +[which model fits](MODELS.md#pick-by-memory); download size is not the same as RAM use. Unified memory counts once. +Use an SSD and leave room for helper files and model preparation. The Terminal's `--plan` prints what is missing. -The normal Python install uses `requirements.txt` and does not install CMake or Ninja. If setup has to compile an -engine or image encoder, it uses existing CMake/Ninja tools when available and installs missing pinned helpers from -`requirements-build.txt` into `.venv`. An older `.venv` keeps any build helpers it already has; setup does not remove -packages during an update. +Install a graphics driver that supports the package you choose. CUDA 13 packages require NVIDIA driver 580 or newer; +older NVIDIA cards use CUDA 12. AMD and Intel requirements are in their guides. Packaged GPU runtimes do not replace +hardware drivers. Models and updates need an internet connection to download. ## The app -The app is the easiest way. It has the engine inside, so there is nothing to build and no setup window. +1. Open the [latest release](https://github.com/Niko1221/Strata/releases/latest). +2. Download the one installer for your system. There is one per system, not one per graphics card: + - **Windows:** run `Strata-0.2.5-windows-x64-setup.exe`. It installs for your user and needs no administrator rights. + - **Linux:** make `Strata-0.2.5-linux-x86_64.AppImage` executable, then open it. If FUSE is unavailable, run the + downloaded file with `--appimage-extract-and-run`. [Linux help](TROUBLESHOOTING.md). + - **Mac:** open `Strata-0.2.5-macos-arm64.dmg`, drag Strata into Applications, then open it. [Mac help](MAC.md). +3. On Windows and Linux the installer has no engine inside. On first start the app shows the engine for your + graphics card with its size and one **Download** button, with progress and resume. If you already downloaded the + engine file, choose **I already have the file** and it installs without internet. The Mac file keeps its small + Metal engine inside. +4. Choose a model and its storage folder, let it download, then chat. DeepSeek V4.1-Flash is in the model list on + the supported cards ([model page](MODELS.md#deepseek-v41-flash)); it has three sizes, an 8K context and a **Speed** + choice, Faster or Full quality. -1. Download the installer for your system from the [releases page](https://github.com/Niko1221/Strata/releases/latest). - The page says which installer is for which graphics card (NVIDIA or AMD). Take the one that matches yours. - - **Windows 10 / 11:** run the `.exe`. It installs for your user and needs no administrator rights. Windows may - say "Windows protected your PC", because the installer is not code-signed yet. Choose **More info**, then - **Run anyway**. The installer asks for the folder (by default `%LOCALAPPDATA%\Programs\Strata`): on a PC - with little space on C:, pick a folder on another drive that you can write to (not Program Files, which needs - an administrator). Updates from inside the app go into the same folder. The engine inside the app - (`%LOCALAPPDATA%\Strata\components`) stays on C:; `STRATA_COMPONENTS_DIR` moves it, and the models go to the - folder you choose in the app. - - **Linux:** the `.AppImage`. Make it runnable (right-click, Properties, Permissions, "Allow executing file as - program", or `chmod +x`), then double-click it. It needs FUSE 2: `sudo apt install libfuse2` on Ubuntu 22.04, - `sudo apt install libfuse2t64` on Ubuntu 24.04 and newer, `sudo dnf install fuse fuse-libs` on Fedora. Without - FUSE you can still start it with `--appimage-extract-and-run` after the file name. The NVIDIA build needs - driver 580 or newer. - - **Mac:** the `.dmg`. Open it, drag Strata onto Applications, then open Strata from there. It needs Apple - Silicon and macOS 15 or newer. If macOS says the app is "damaged", the file is fine: see [MAC.md](MAC.md). -2. Start Strata. The first start is a little slower on Linux and Mac, because the engine inside the app is copied - into your home folder. -3. The first screen reads your PC and recommends a model that fits. Press the install button. The download is - big (40-70 GB). If it stops, it continues where it stopped. -4. Chat. **Closing the window stops Strata and the engine.** +The `strata-engine-…` and `strata-control-…` archives on the release page are components, not installers. The +installer is the same file for a first install and for an update. There is no Intel desktop installer yet; Intel Arc +runs on Linux only, through the [Terminal](INTEL_ARC.md). -**Update:** Settings, then Updates, then **Check for updates**. The buttons **Update Strata** and **Update engine** -appear only when something is new. Your models and saved settings are kept. Unload the model first: an engine that -is running cannot be replaced. **Roll back** in the same place brings the previous engine back. +The first launch can take time to check and prepare large model files. Watch its progress and logs; there is no fixed +startup time for every model or computer. Closing the desktop app stops its managed engine. -**Where things are:** the app's settings and chats are in the `.strata` folder in your user folder, the engine -in `%LOCALAPPDATA%\Strata\components` (Linux and Mac: `~/.local/share/strata/components`), and the models in the -`Strata` folder in your Documents (you can change the model folder in the app). Removing the app does not remove -them. **For a bug report:** Settings, Engine, Diagnostics, View logs, **Save logs**. - -On a Mac pictures are on by default and read on the GPU ([MAC.md](MAC.md)). Everything else about pictures is in [Setup's questions](#setups-questions). +The provider order recommended here is **ISTA-DASLab, Unsloth, Gyro, then others**, when a compatible pack fits. +Current app recommendations can differ; choose the model explicitly when you want a particular provider. ## The Terminal -The Terminal is the `strata` command. It uses the same engine, models and saved settings as the app, and it is the -way for scripts, servers and AI agents. Linux (x86-64), Mac (Apple Silicon) and Windows (x64) are supported. +Install once, then use `strata launch`. The Terminal includes Python and downloads the engine for your computer. -Linux and Mac: +**Linux / Mac** ```sh curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh ``` -Windows (PowerShell): +**Windows PowerShell** ```powershell irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex ``` -The installer finds out which computer this is, downloads the Terminal archive, checks its SHA-256, unpacks it into -your home folder (`~/.local/share/strata/terminal`, or `%LOCALAPPDATA%\Strata\terminal` on Windows; no -administrator rights), checks the release's signed update feed with the Terminal it just unpacked (if that check -fails it removes what it unpacked), and puts `strata` on your PATH. On Windows, open a new terminal window -afterwards. Then: +On Windows, open a **new terminal** afterwards. On Linux and Mac, if `strata` is not found, use the full path printed +by the installer (normally `~/.local/bin/strata`) or add that folder to your shell's PATH. ```sh -strata launch # guided screens: model, context, pictures, who may connect, then serve -strata launch --model q2_0 --yes --no-tui # the same without questions (scripts, agents) +strata launch ``` -`strata launch` downloads the engine for your graphics card (signed, and test-started before it is used) and the -model, starts serving on `http://127.0.0.1:8080` and shows the monitor. It never opens a browser unless you add -`--open`. Its flags and screens: [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md). +Follow the screens to choose a model, context, storage folder and access settings. The monitor opens after startup. +Use the address printed by Strata; the default local endpoint is `http://127.0.0.1:8080`. +The Terminal opens a browser only when you add `--open`. -**Update:** `strata update --check` says what would change, `strata update` installs the new engine (it is test-started -first and switched only if that works; the previous engine is kept), `strata components rollback` goes back. It -refuses while a model is being served. **Check your PC and the installed engine:** `strata doctor`. +For a script or AI agent, inspect a plan before downloading. This example selects ISTA-DASLab Q2_0 explicitly: -To read the install script before you run it, download it first (`curl -fsSL
-o install.sh`), read it, -then run `sh install.sh`. - -## From source: Windows - -1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or - `git clone` it). -2. Double-click **`START-HERE.bat`**. -3. Answer a few questions - or just press Enter each time for the recommended choice ([the questions](#setups-questions)). - -Then it downloads everything (the model is ~70 GB, so the first time takes a while - you can stop and it picks up -where it left off) and **starts the model**. Your browser opens the Strata app at `http://127.0.0.1:8080`. - -> **While the model starts, your PC can be slow or stop responding for 1-3 minutes** (longest the first time): Strata -> loads 35-55 GB into your RAM and locks part of it for the graphics card. That's normal - wait, and don't close the -> window. The window tells you what it is doing. - -**Next time**, just double-click `START-HERE.bat` again: it starts right away (30-90 s to load the model), nothing is -downloaded twice. Close its window to stop the model. `SETUP.bat` (the same as `START-HERE.bat --setup`) installs -another model or changes the settings. Starting Strata from Task Scheduler at logon needs two task settings, or the -start is 24x slower: [Running it at startup](DETAILS.md#running-it-at-startup-task-scheduler). - -## From source: Linux - -```bash -./setup.sh +```sh +strata launch --family qwen --model q2_0 --plan +strata launch --family qwen --model q2_0 --yes --no-tui ``` -The same questions, the same automatic install (it uses `sudo apt` for Python and, only if it has to compile, for -the build tools), and the same start: `http://127.0.0.1:8080`. Later runs of `./setup.sh` (or `./run-.sh`) -start the model directly; `./setup.sh --setup` installs another model or changes the settings. Other distributions, -WSL and compiling: [details](DETAILS.md#linux). +`--no-tui` leaves the server running in the background. Check it, connect a tool, or stop it: -On Nix/NixOS, you can use the flake: - -```bash -nix develop -c ./setup.sh +```sh +strata launch --status +strata connect +strata launch --stop ``` -### Build from git on Linux (to try a pull request or change the code) - -You do not need the app or the AppImage for this: the source install builds the engine on your PC and keeps -everything in the checkout, so you can edit, rebuild and test. - -```bash -git clone https://github.com/Niko1221/Strata.git && cd Strata -./setup.sh # first run: .venv, the engine build (build/, or build-hip/ on a Radeon), the model, then it serves -``` - -- **What it needs:** `git`, a C++ compiler and `cmake` (setup installs the build tools through `sudo apt` on - Debian and Ubuntu; elsewhere install them first), and the CUDA Toolkit for NVIDIA or ROCm with the HIP compiler for - a Radeon card ([AMD_HIP.md](AMD_HIP.md)). Python goes into `.venv` of the checkout. -- **After you change the code:** run `./setup.sh` again. It compiles only what changed and starts the model. To - update without starting: `git pull`, then `./update.sh`. By hand, the build folder is an ordinary CMake build: - `cmake --build build --target strata -j8` (`build-hip` for a Radeon card; the options are in - [AMD_HIP.md](AMD_HIP.md)). -- **To try a pull request:** `git fetch origin pull//head:pr- && git checkout pr-`, then - `./setup.sh`. `git checkout main` goes back; the next `./setup.sh` rebuilds. -- **Settings for the engine** (for example `STRATA_HIPBLASLT_TUNING=/path/to/table.txt`) go into the `"env"` object of - `strata-.json`, which `./setup.sh` writes next to the checkout. `./run-.sh` starts the model with it. -- **The Terminal monitor from a checkout:** the monitor needs one extra package, which the source install does not - put into `.venv`: - - ```bash - .venv/bin/pip install -r requirements-monitor.txt - STRATA_PYTHON="$PWD/.venv/bin/python3" bin/strata monitor --url http://127.0.0.1:8080 - ``` - - (`bin/strata` is the same `strata` command as the Terminal's: `bin/strata --help` lists the rest.) -- **Only compile, without the installer:** the engine is a normal CMake project. For NVIDIA: - `cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native && - cmake --build build --target strata -j8`. The options, and the flags setup passes, are in `setup.py` (search for - `-DSTRATA_`). - -## From source: Mac - -On an Apple Silicon Mac with macOS 15 or newer, install Apple's Command Line Tools once (`xcode-select --install`), -then run `./setup.sh` in the Strata folder. It builds the Mac (Metal) engine on your Mac and asks the same -questions. The app and the Terminal do not build anything. All of it, and what fits which memory size, is in -[MAC.md](MAC.md). - -## AMD cards - -The app and the Terminal pick the AMD engine for a Radeon card by themselves. From source the steps are the same as with NVIDIA: `START-HERE.bat` on Windows, `./setup.sh` on Linux. Setup finds the Radeon -card and chooses the AMD (HIP) engine by itself on a PC with no NVIDIA card Strata can use; `--backend hip` chooses -it on a PC that has both. Supported cards: RX 7900 XT / XTX (gfx1100), RX 7800 XT / 7700 XT (gfx1101), RX 9060 XT -(gfx1200), RX 9070 / 9070 XT and Radeon AI PRO R9700 (gfx1201), and the RX 6800 / 6900 series (gfx1030). Integrated -Radeon GPUs are listed as not supported. - -On Linux setup uses a system ROCm 7 when there is one, or installs ROCm into `.venv` from AMD's wheels (~10 GB, no -sudo), and compiles the engine on your PC for the card (10-20 minutes, once; it needs a C++ compiler and git: -`sudo apt install build-essential git`). Several AMD cards share the model with `--gpus`, as on NVIDIA. - -What differs from NVIDIA for now: `--calibrate` is NVIDIA-only, and Unsloth's 4-bit model needs an NVIDIA card. Measurements per card, the -build by hand and the tuning tables: [AMD_HIP.md](AMD_HIP.md). - -## Two or three cards - -**Two or three NVIDIA cards?** Just run `START-HERE.bat`: it lists your cards, says which ones Strata can use, and -asks whether to share the model across them (recommended when two can). An install made on one card asks once at -its next start. Or choose yourself: `START-HERE.bat --gpus 0,2` (both, remembered), `--gpus all`, or `--gpu 0` (one -card, this start only). Each card keeps the experts of its own layers, and prompts flow through the cards in a -pipeline: on an RTX 5080 + RTX 3090 prompts were read 18-20% faster than on the 5080 alone, decoding on par. -Every card must be an RTX 20 series or newer with 8 GB or more. See [MULTI_GPU.md](MULTI_GPU.md). - -**Several AMD cards:** setup takes one card (the one with the most VRAM, or `--gpu N`) unless you name more: -`--gpus 1,0` splits the model's layers across them, the faster card goes last and is the Main card (`--gpu-roles 1=main,0=helper` says it yourself; docs/MULTI_GPU.md; numbers as setup lists them). A -split pays only when no single card holds the model's experts ([AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201)). - -## Docker (Linux) - -Ready-made images for NVIDIA (CUDA 13 and CUDA 12), AMD (RX 9000 / R9700 and RX 7000) and Intel Arc cards, with the Terminal -inside: `docker run` one command, the model downloads on the first start into a volume, nothing is compiled on your computer. -Which image for which card, the commands, Docker Compose, the API key, updating and troubleshooting: [DOCKER.md](DOCKER.md). -Building from source: `docker build -t strata:src .` (CUDA 13; [DOCKER.md](DOCKER.md#build-it-yourself)). - -## Older CPUs (experimental) - -Strata's ready-made engine needs AVX2 (Intel Haswell 2013, AMD Zen 2017 or newer). Since 0.1.39 an older CPU - AVX -only (Sandy Bridge / Ivy Bridge, Xeon E5 v1/v2, AMD Bulldozer) or SSE4.2 only (Nehalem / Westmere, Xeon X5600) - runs -as an **experimental** build that setup compiles on that PC. Run setup as usual: it says the CPU has no AVX2, warns -that this is experimental and slow, and compiles the engine (10-20 minutes, once; Windows needs the Visual Studio Build -Tools and the CUDA Toolkit, which setup offers to install). It does not stop. The ready-made engine still refuses such -a CPU, and says so. - -What differs in that build (`STRATA_ISA_FLOOR=avx` or `none`, chosen from the CPU): - -- ggml-cpu is compiled for AVX or SSE4.2 instead of AVX2, and the CPU's share of the experts runs on its kernels. - Strata's own AVX2 / AVX-512 kernels stay in the program but are used only on a CPU that has them. -- Only the i-quant models' packs run (every model setup installs on such a CPU); the AVX-512 Q2_0 pack does not. -- Speed: the GPU part is unchanged and the CPU part is slower, so a GPU with more VRAM (more experts cached there) - helps most. With the older paths forced on a Ryzen 5 7600 + RTX 5070 and the expert cache held at 1500 slots, greedy - decode went from 26 to 11-17 tok/s with the AVX build and to 3.5-3.8 tok/s with the SSE4.2 build (Coder and IQ3_XXS, - 10 prompts each, one run). An old Xeon with DDR3 will be slower than these. -- Untested on a real old CPU by us. It was checked here by forcing the older paths on a Ryzen 5 7600 - (`STRATA_FORCE_ISA=avx` or `sse`): the answers were sane and passed the engine's checks, but not the same token for - token as the normal build (ggml-cpu rounds the CPU experts differently, as the AVX2 kernel does from the AVX-512 - one). Contributors ran earlier versions of the same approach on a Xeon E5-2680 (2.87 tok/s with a GTX 1080 Ti) - and on Xeon E5-2687W / X5690 machines. -- AMD cards: Linux only (setup compiles the AMD engine there anyway); NVIDIA on Windows and Linux. -- Two-socket Xeons: memory on the other socket is slow; keep the RAM on one CPU if you can. - -By hand: `cmake ... -DSTRATA_ISA_FLOOR=avx` (or `none`) builds it; `STRATA_ISA_FLOOR=avx` set for setup builds it on -any PC (to try it). Problems and results on real hardware are welcome as GitHub issues. +If a model is already running with different settings, add `--restart` to replace that run; `--yes` alone does not +restart it. A restart interrupts current answers. See [all launch options](TERMINAL_LAUNCH.md), +[advanced Terminal commands](TERMINAL.md), and [agent setup](AI_SETUP.md). ## Updating -**The app:** Settings, Updates, **Check for updates**, then **Update Strata** or **Update engine** (see [the app](#the-app)). +**Desktop app:** open Settings and find the update controls, check for updates, and use the offered app or engine update. +Unload running models before replacing the engine. Your model files and settings are kept. -**The Terminal:** `strata update --check`, then `strata update` (see [the Terminal](#the-terminal)). Add `--app` to -download the app's installer too. Models and saved tuning presets are never downloaded, replaced or deleted by an -update. How the signed update feed works: [packaging/UPDATES.md](../packaging/UPDATES.md). What Strata downloads and -how it checks it: [SAFETY.md](SAFETY.md). +**Terminal:** -**From source: `UPDATE.bat`** (Linux: `./update.sh`) updates Strata without starting the model - for when the GPU is busy with -something else, or you just want the new version ready. In a `git clone` it runs `git pull`, then does what -`START-HERE.bat` does before a start: the engine (a new ready-made one when the new version needs it; on Linux a -compiled engine is compiled again when its source changed), the Python packages, and each installed model's settings -and draft subset. The model files are not touched (at most a new engine is downloaded) and no question is asked. -Close the model's window first (a running engine cannot be replaced); start the model later with `START-HERE.bat` as -usual. In a copy that was downloaded as a zip it says to download the new zip (below): it cannot fetch new files -itself. +```sh +strata update --check +strata update +strata --version +``` -Or by hand: download the new version and unzip it anywhere (or `git pull`), then run `START-HERE.bat` (Linux: -`./setup.sh`) in it. The model files are kept in a `Strata-data` folder next to your Strata folder, so a new copy -finds them and sets itself up the same way - nothing big is downloaded again. On Linux after a `git pull`, setup -compiles the engine again when its source changed (a minute or two for the changed files); if that compile fails, it -says so and starts the engine you had. +Stop models first. The next Terminal command uses the updated Control component. The app, Control and engine have +separate version numbers, so they can differ. + +Use `strata components rollback` to return to the previous engine. See [update details](../packaging/UPDATES.md). +Source checkouts have a [separate update workflow](SOURCE_INSTALL.md#updating). ## Where things are stored -- **Your chats: only in your browser.** The Chat tab keeps the conversation, its settings and the API key you typed - in the browser's local storage (`strata.*` keys) - not on the server and not in the Strata folder. Pictures are not - kept, only their names. Another browser or a private window starts empty; clearing the site's data deletes them. -- **How the model starts:** `strata-.json` in the Strata folder (context, GPUs, host, API key, ...), written - by setup; next to it `run-.bat` / `.sh`, the log `strata-.log` and, when you use "Use for other - apps too", `strata-.shared-settings.json`. Running setup again for the same model (another context, say) - rewrites the keys setup writes (`exe`, `args`, `port`, `gpu`, `host`, `api_key`, `vision`, ...) and keeps the - ones you added (`sampling`, `mcp_servers`, `mcp`, `cors_origins`, ...); the earlier file is kept as - `strata-.json.bak` (0.1.39). Engine options you added to `"args"` by hand are not carried over: setup names - them, and you add them again. -- **The model files** (`models/`, `packs/`, `mtp/`, 70-120 GB): in **`Strata-data` next to the Strata folder**, or - wherever `--data-dir` put them. -- **Where that data folder is:** `%APPDATA%\Strata\settings.json` on Windows, `~/.config/strata/settings.json` on - Linux ([details](DETAILS.md#before-you-start)). -- **The program itself:** `.venv/`, `engine/` and `third_party/` in the Strata folder. -- **Where github.com cannot be reached:** a build from source downloads llama.cpp's source (ggml, gguf-py) from - GitHub at the commit `LLAMA_CPP_COMMIT` in `setup.py` names. Put that commit's tree in `third_party/llama.cpp` - yourself (from a mirror or another PC) and setup uses it instead (#585). +| Item | Default location | +| --- | --- | +| Desktop chats and host state | `~/.strata` (or `STRATA_HOME`) | +| Engine components, Windows | `%LOCALAPPDATA%\Strata\components` | +| Engine components, Linux / Mac | `$XDG_DATA_HOME/strata/components`, normally `~/.local/share/strata/components` | +| Terminal files | The `terminal` folder beside `components`; the installer prints the exact path | +| Models | The folder chosen in the app or launch screens; an existing model library is reused | -### Keeping the models on another disk +The desktop app normally proposes `Documents/Strata/Models` for a new library. The Terminal discovers saved +folders from earlier installs. Use the path shown by the app or `strata models list`, not an assumed folder name. +`--data-dir` changes where new downloads go; it does not move existing models unless you add `--move-data`. +See [model discovery](TERMINAL_LAUNCH.md#models-you-already-have). -The model files are the only big part (70 to 190 GB). Put them where there is room; the app and the Terminal use the same folder. +## More options -- **In the app:** the first screen shows the model folder with a button to change it, and Settings > Storage changes it - later. Pick a folder you can save files in; a whole drive (`D:\`) becomes `D:\Strata`, which Strata makes and says so. - Change folder then asks what to do with your models: move them (copied file by file and checked before the old copies are removed; you can stop at any time), - keep them where they are and also use the new folder, or start empty (the old files are never deleted). From the Terminal the same move is - `strata models move NEW_FOLDER` (add `--plan` to see what it would do first). -- **In the Terminal:** `strata launch --data-dir D:\Strata` once. The folder is remembered for the next `strata launch` - and `strata models list`. Until you give it one, the Terminal uses the folder the app chose; the app keeps its own setting - (Settings > Storage) and does not read the Terminal's. `--move-data` moves the models you already have into it; without it they stay where they - are and the new folder gets the new downloads. Use your own files in place with `strata launch --model-path FOLDER` (a - folder of shards, a folder above it, or one `.gguf` file); a folder that holds several sizes needs the one `.gguf` file. -- **Drives that are not NTFS** (exFAT, FAT32, a network share) work: Strata writes straight into `\models` and makes no - link there. -- **What stays on C:** the engine (`%LOCALAPPDATA%\Strata\components`, 1 to 2 GB per version; `STRATA_COMPONENTS_DIR` moves it), - the app itself (the installer asks), your settings, chat history, tuning results and logs (`%LOCALAPPDATA%\Strata`, - `%APPDATA%\Strata`; a few MB). Nothing else large: downloads, the prepared model data (`packs`) and the draft layer (`mtp`) are - written in the model folder, and so are the half-finished files of an interrupted download. -- **Docker:** `-v /mnt/big/strata:/data` (a folder on the big disk) instead of the volume `strata-data`; everything above lives in it. - -## Setup's questions - -These are the questions of the installer from source (`START-HERE.bat`, `./setup.sh`). The app's first screen and -`strata launch` ask the same things in their own look. - -- **Which model and size?** The original, Swift 1.5, the Coder or Unsloth's 4-bit, and Q2_0, IQ2_XS, IQ3_XXS or - IQ3_S - see [which model](MODELS.md). -- **How much context?** How much text it can keep in mind at once (it suggests one for your card). 384K and - 512K (experimental) extend the model past its trained 262K by rope scaling - the setup turns it on itself (yarn and a - covering factor; `--rope-scaling`/`--rope-scale` override) ([details](DETAILS.md): "Context extension past - 262K"). -- **Images?** Whether it should also read pictures. Pictures are on by default for models that can read them. The - image encoder (0.9 GB) runs on the CPU when your card has less than 24 GB (about 3-13 s per picture, nothing taken - from the GPU) and on the GPU from 24 GB (0.1-0.5 s per picture, ~1.4 GB of VRAM kept free for it). The rule is the - same for NVIDIA and AMD cards; `--vision gpu` and `--vision cpu` choose the reader yourself, `--vision no` turns - pictures off. On a Radeon AI PRO R9700 the GPU reader took 57 ms for a 260-token picture and the CPU reader 3.1 s. - A Mac reads on its GPU (0.45 s for a 260-token picture on an M4). [Details](DETAILS.md#images-vision). -- **Experimental speed projection?** Off unless you say yes - [read what it does](DETAILS.md#experimental-speed-projection-experimental-off-by-default) - first. It changes how the model answers, and only the original model offers it. - -With `--yes` setup takes the recommended answer to every question. - -## Tuning for your PC - -Every PC is different: `START-HERE.bat --calibrate` (Linux: `./setup.sh --calibrate`) measures a few engine settings -on yours and keeps the fastest (about 15-30 minutes, longer on a slow card; on an RTX 5070 with a Ryzen 5 7600 it made the Coder 7% faster). -It keeps a setting only when it is more than 3% faster, and the result is remembered per PC and model, so updates -keep it. Measuring the CPU worker count needs a fresh engine, so the model is loaded more than once: the PC is -busy, and can stop responding for a minute or two, once per restart. When it finishes it **starts the model**, -so the server is already running when it returns - do not start it a second time. -NVIDIA cards for now. [What it measures](DETAILS.md#double-click-start-herebat). - -## Options without questions - -For the Terminal, `strata launch --help` lists every flag; the most used are in [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md). These are the options of the installer from source: - -``` -START-HERE.bat --setup install another model, or change context / images -SETUP.bat the same (double-click it) -START-HERE.bat --model IQ2_XS --context 32768 --vision yes --yes no questions -START-HERE.bat --gguf-dir D:\models\IQ2_XS use GGUF files you already have -START-HERE.bat --inspect D:\models\some.gguf what a GGUF really holds and whether Strata runs it (no download) -START-HERE.bat --data-dir E:\Strata-data keep the model files somewhere else -START-HERE.bat --port 8081 another port -START-HERE.bat --gpu 1 another GPU (setup picks the one with the most VRAM) -START-HERE.bat --gpus 0,2 several GPUs sharing the model -START-HERE.bat --vram-reserve-mib 2048 leave 2 GB of VRAM free for other programs (remembered) -START-HERE.bat --no-browser do not open the chat page when the model is ready (remembered; - --browser undoes it) -START-HERE.bat --setup --backend hip the AMD engine on a PC that also has an NVIDIA card -START-HERE.bat --setup --host 0.0.0.0 --api-key reachable from other devices, with a key -START-HERE.bat --calibrate tune the engine for this PC (about 15-30 minutes, longer on a slow card), then start -START-HERE.bat --check only check this PC -``` - -**Leaving VRAM for other programs (#493):** Strata fills the graphics card's free VRAM with experts (the expert cache) -and leaves `--vram-reserve-mib` MiB free: 700 by default. For a game, a 3D program or another model beside it, leave -more: `START-HERE.bat --vram-reserve-mib 2048` (Linux: `./setup.sh --vram-reserve-mib 2048`) writes it into the -model's `strata-.json` and starts it; at setup (`--setup --vram-reserve-mib 2048`) it goes into the new config. -By hand: add `"--vram-reserve-mib", "2048"` to the config's `"args"` list and restart. The expert cache is then that -much smaller, so answers can be a little slower. The engine sizes its cache from the VRAM free when it starts: what -another program already holds then is left alone anyway; the reserve is room for what it takes later. - -**An AMD card that also drives a Linux desktop (#560 #516):** with the default reserve the expert cache fills the -card, and when the desktop (the compositor, a browser, a new app) needs more VRAM, the amdgpu driver moves GPU memory -to system RAM, which the model's experts already fill: the OOM killer then ends the desktop session (KWin, plasmashell) -or the compositor fails ("Failed to pin framebuffer"). `./setup.sh --vram-reserve-mib 3072` fixed it in both reports -(about 2.3 GB fewer experts in VRAM, a few % of speed). Setup and the server window say so on such a PC. - -**A card under 8 GB (#496):** when the default reserve leaves the expert cache too little room, the engine lowers the -reserve (down to 300 MiB) until it fits, and warns if the card then ends nearly full. If the start stops with "no VRAM -is left for the expert cache", that log line says how much is short; an 8K context and `--draft-vocab en` at setup -free the most. - -**Model files downloaded by hand, or from a mirror (#495):** setup's step 5 prints the folder it expects them in -(`Strata-data\models\\`, e.g. `Strata-data\models\IQ3_XXS\`): put them there with their original names, or -point setup at them with `--gguf-dir`. To see what a file really holds before using or downloading it, `--inspect` -reads only its headers (a few MB, also over the network: a file, a folder, a URL, `ms:owner/repo` for ModelScope or -`hf:owner/repo`): every weight group's real bits per weight and storage types, and whether it is one of the files setup -installs (under any name), stored like one of them, or not one Strata runs (another architecture, or experts in a -format no Strata kernel reads). A name says little: Unsloth's `UD-IQ3_XXS` stores its routed experts at 3.22 bits per -weight, `UD-Q4_K_XL` at 5.10, the original `Q2_0` at 2.25 and `IQ3_S` at 3.33. To let setup download from a Hugging Face mirror itself, set `HF_ENDPOINT` -first (Windows: `set HF_ENDPOINT=https://hf-mirror.com`, Linux: `export HF_ENDPOINT=https://hf-mirror.com`): the same -pinned revisions and checks apply, and the MTP draft layer comes from there too. - -**From ModelScope (mainland China):** every repository setup downloads from (the GSQ-RCO models, Swift 1.5, the -Coder, Unsloth's files, and the original checkpoint the MTP draft layer is taken from) is on -[ModelScope](https://www.modelscope.cn) under the same name, with the same files. Setup never switches to it by -itself (a download that fails says how): use `--source modelscope` (`STRATA_SOURCE=modelscope`). ModelScope serves a -repository's current files (no pinned revision), so each file is checked against the SHA-256 ModelScope publishes for -it (self-attested: a changed upstream repository would carry its own hash), and the MTP tensors against the -pinned checkpoint's own hashes, as from Hugging Face; setup says so when you choose it. Measured on a PC in mainland -China on 2026-10-05: 11-14 MB/s; the Q2_0 and IQ3_S files matched their published SHA-256, and Q2_0's shard 1 matched -the copy from hf-mirror.com. - -On Linux the same options go to `./setup.sh`. `START-HERE.bat --help` lists them all. The server's own settings -(sharing the GPU with games, MCP tools, CORS, API keys, the API itself) are in the [details](DETAILS.md#using-it). +- [Docker](DOCKER.md) +- [Build or run from a source checkout](SOURCE_INSTALL.md) +- [Multiple GPUs](MULTI_GPU.md) +- [Troubleshooting and logs](TROUBLESHOOTING.md) +- [Model compatibility](MODELS.md) diff --git a/docs/INTEL.md b/docs/INTEL.md index beaa6ad79..156150390 100644 --- a/docs/INTEL.md +++ b/docs/INTEL.md @@ -1,389 +1,122 @@ -# Strata on an Intel Arc +# Strata on an Intel Arc: the SYCL engine -Strata's engine is CUDA. On an Intel Arc it runs as **Strata's own engine, ported to SYCL** (`sycl/`, the -section "The engine itself on Intel" below), behind the same Strata server: OpenAI and Anthropic APIs, streaming, -tool calls, MCP and the web app all unchanged. llama.cpp's SYCL backend is the comparison point in the tables (it -runs the same GGUF at about a third of the speed). +This page is for people who build the Intel Arc engine from source or run it in a container, and for those who maintain the +port. For the released Linux engine package, start with [INTEL_ARC.md](INTEL_ARC.md). -Everything Intel-specific lives in `sycl/`; no shared file of upstream's is changed, so upstream merges stay clean. -`sycl/setup_intel.py` and `sycl/serve/server_intel.py` wrap upstream's `setup.py` and `serve/server.py` from -outside. +Strata's engine is CUDA. On an Intel Arc it runs as Strata's own engine ported to SYCL (`sycl/`), behind the same server: +OpenAI and Anthropic APIs, streaming, tool calls, MCP and the web app are unchanged. Everything Intel-specific lives in `sycl/`; +no shared upstream file is changed. `sycl/setup_intel.py` and `sycl/serve/server_intel.py` wrap upstream's `setup.py` and +`serve/server.py` from outside. llama.cpp's SYCL backend runs the same GGUF at about a third of the speed. -Written for and measured on an **Arc Pro B70 (32 GB)** running the Coder (IQ1_M) on Ubuntu 24.04, in a -PCIe 3.0 x8 slot (the card trains at Gen3 x8 there; it can do Gen5 x16). +DeepSeek V4.1-Flash does not run on Intel. Pictures are on by default ([below](#pictures-on-an-arc)). -## What you get (2026-10-01) +## Quick start -| | NVIDIA (Strata engine) | Intel Arc, Strata SYCL port | Intel Arc, llama.cpp | -|---|---|---|---| -| models | all four | the Coder IQ1_M; the original IQ2_XS too (experts beyond VRAM in the host mirror: 58-64 tok/s) | the Coder (fits VRAM) | -| model files | the same GGUFs | the same GGUFs + a native pack | the same GGUFs | -| where the model lives | experts in RAM, hot ones on the card | every expert in VRAM (`--stream-experts`: no host copy), shard 2's lookup table read from the SSD by row | all of shard 1 on the card, shard 2 paged from disk | -| RAM needed | 32-64 GB | little (23 GB is fine) | little (23 GB is fine) | -| decode, Coder IQ1_M | 44-51 tok/s on an RTX 5070 | **78.2 tok/s** (19-token prompt, 256 greedy tokens), **75.7** after a 2,184-token prompt, 69 after 40K, 66.6 after 128K, 55.9 after 256K; 48-72 through the API | 23-25 tok/s | -| prompt reading | ~1,870 tok/s | **790 tok/s** at 2,184 tokens, 1,117 at 40K, 888 at 128K, 757 at 256K | ~150 tok/s (424 at 105K) | -| speculative decoding (MTP) | yes | yes (the base checkpoint's draft layer; 70-85% of drafts accepted on code) | no | -| images | yes | not yet | not yet | -| context | up to 262K | 256K measured, 56 tok/s decode there (`--kv-resident`: the KV in pinned host memory, the attended window in VRAM) | 131K measured ceiling | +On Ubuntu 24.04 or newer, with Intel's GPU driver installed (`ls /dev/dri` shows a `renderD128`; see +[INTEL_ARC.md](INTEL_ARC.md#what-you-need)): -Decode speed with speculative decoding depends on the text: code drafts well, prose less so (see "Speed depends -on the text"). The SYCL numbers are greedy runs of the engine (as in "How to run it by hand" below, 256 new -tokens) on engine 0.1.31-sycl (2026-10-01; 0.1.32-0.1.38-sycl reproduce them exactly); -"Decode round 2" below lists what each change bought. +```sh +sudo apt install docker.io git +sudo usermod -aG docker,render $USER # log out and in; `docker ps` must work without sudo +cd Strata +docker build --pull -t strata-sycl-dev sycl/tools # once, about 6 minutes, 13 GB +# build the engine: see "How to build it", then +./setup.sh --backend sycl [setup's options, e.g. --model IQ3_S --context 32768 --port 8085] +``` + +`setup.sh` makes its own Python environment in `.venv`. Do not run `python3 sycl/setup_intel.py` yourself: Ubuntu's system +Python refuses it (`externally-managed-environment`). `./setup.sh --backend sycl` is the one command for the first install and +every later start. Then `run-.sh` starts the model; the first start of a 30 GB model takes about two minutes. ## Setup -On Ubuntu 24.04 or newer, from nothing. The Intel GPU driver (the `xe` or `i915` kernel driver and the compute runtime; -[INTEL_ARC.md](INTEL_ARC.md), "What you need") is assumed: `ls /dev/dri` shows a `renderD128`. +`sycl/setup_intel.py` imports upstream's `setup.py` and replaces the Intel steps. The model choice, download, pack, tokenizer, +MTP draft layer and the context and KV questions are setup's own. -1. **Docker, once.** The engine runs in a container, so the PC needs Docker and no oneAPI: +| Part | What changes on an Arc | +| --- | --- | +| GPU check | The Arc is found in sysfs (vendor 8086 under `xe` or `i915`) and offered through setup's AMD path (it builds locally and has no images). | +| Engine | Uses the SYCL build (`build-sycl-aot/strata` or `build-sycl/strata`, run in the `strata-sycl-dev` image by `sycl/serve/strata-sycl.sh`). | +| RAM rule | Does not apply on an `xe` card: the port streams the experts from the GGUF into VRAM (`--stream-experts`), so RAM only decides KV streaming. `--check` lists what fits by VRAM. | +| Config | Container paths and `"backend": "sycl"`. VRAM reserve 1,024 MiB up to 32K context and 2,048 MiB above, 300 MiB on a card under 12 GB. A `--vram-reserve-mib N` you give is kept. KV streaming (`--kv-resident 32768`) is on from 64K when the RAM holds the KV. | +| Fit rule | A model that does not fit the card at the default context gets a context down the ladder 32K, 16K, 8K, a smaller prompt chunk and a lower reserve until it does. Setup prints one plain line per change. | +| Prompt chunk | 4,096 tokens; 8,192 on a Battlemage card with 24 GB or more above 32K context. A `--prefill` in the config is kept. | +| Alchemist (A-series) | A different config: [Arc A750 and the other Alchemist cards](#arc-a750-and-the-other-alchemist-cards). | +| Engine settings | The config's `"env"` block reaches the engine. `strata-sycl.sh` forwards every `STRATA_`, `ONEAPI_`, `UR_`, `IGC_`, `SYCL_` and `ZES_` variable and sets `STRATA_VERIFY_DEVICE_PLAN=1`, `STRATA_VERIFY_NO_HOST=1` and `STRATA_STAGER_THREADS=12` unless the environment says otherwise (for example `"env": {"STRATA_VERIFY_NO_HOST": "0"}`). A value of `0` is not passed on, because the engine tests these switches by presence. | - sudo apt install docker.io git - sudo usermod -aG docker,render $USER - - Log out and in again (or reboot) so the groups apply. `docker ps` must work without `sudo`; the `render` group lets - the container open the GPU. -2. **Get Strata and build the engine** ("How to build it" below, about 10 minutes the first time). -3. **Run setup:** - - ./setup.sh --backend sycl [setup.py's options, e.g. --model IQ3_S --context 32768 --port 8085] - - `setup.sh` makes its own Python environment in `.venv`, so Ubuntu's "externally managed" system Python is not - touched. **Do not run `python3 sycl/setup_intel.py` yourself:** it installs setup's packages into whichever Python runs - it, and Ubuntu's refuses (`externally-managed-environment`). `./setup.sh --backend sycl` is the one command, for the - first install and every later start. - -Setup is upstream's `setup.py` with the Intel steps swapped in (`sycl/setup_intel.py` imports it and replaces those steps; -setup.py itself is unchanged). The model choice, download, pack, tokenizer, MTP draft layer and the context and KV -questions are setup's own. What changes: - -- **GPU check:** the Arc is found in sysfs (vendor 8086 under `xe` or `i915`) and offered through setup's AMD - path. That is the path that builds locally and has no images. -- **Engine step:** it uses the SYCL build (`build-sycl-aot/strata` or `build-sycl/strata`, run in the `strata-sycl-dev` - image by `sycl/serve/strata-sycl.sh`) instead of compiling CUDA or HIP. -- **RAM rule:** this does not apply on an `xe` card. The CUDA engine keeps every expert in RAM; the port streams them from - the GGUF into VRAM (`--stream-experts`), so RAM only decides the KV streaming. `--check` lists what fits by VRAM. -- **The config:** it uses the container's paths and `"backend": "sycl"`. The VRAM reserve is 1,024 MiB up to - 32K and 2,048 MiB with 4,096-token prompt chunks above that, 300 MiB on a card under 12 GB; a - `--vram-reserve-mib N` you give is kept as given. KV streaming (`--kv-resident 32768`) is on from - 64K up when the RAM holds the KV. A `model_switcher` or `sampling` block from an earlier config is kept. - `run-.sh` starts `sycl/serve/server_intel.py`. -- **Arc A-series (`i915`, e.g. the A750 with 8 GB):** a different config, see "Arc A750 and the other Alchemist cards" below. -- **Engine settings:** the config's `"env"` block reaches the engine. `strata-sycl.sh` forwards every variable starting with - `STRATA_`, `ONEAPI_`, `UR_`, `IGC_`, `SYCL_` or `ZES_` into the container and sets `STRATA_VERIFY_DEVICE_PLAN=1`, - `STRATA_VERIFY_NO_HOST=1` and `STRATA_STAGER_THREADS=12` unless the environment says otherwise (for example - `"env": {"STRATA_VERIFY_NO_HOST": "0"}`). - -If a future `setup.py` drops a step this relies on, it stops with a message instead of writing a wrong config. -The models, packs and checkout must sit under the folder `strata-sycl.sh` mounts at `/work` (the one above the -checkout, or `STRATA_SYCL_ROOT`). An earlier Strata install on the same PC keeps its data folder in -`~/.config/strata/settings.json`; pass `--data-dir /Strata-data` to put this one under the -mount. A symlink to a folder outside the mount does not work in the container; a hard link does. - -Then `run-.sh` (or `./setup.sh --backend sycl` again) starts the model. The first start of a 30 GB model -takes about two minutes. `--port N` and `--host 0.0.0.0` work as in upstream's setup. +The models, packs and checkout must sit under the folder `strata-sycl.sh` mounts at `/work` (the one above the checkout, or +`STRATA_SYCL_ROOT`). An earlier Strata install on the same PC keeps its data folder in `~/.config/strata/settings.json`; pass +`--data-dir /Strata-data` to put this one under the mount. A symlink to a folder outside the mount does +not work in the container; a hard link does. If a future `setup.py` drops a step this relies on, setup stops with a message +instead of writing a wrong config. ## How to build it -The engine is built in Docker, in the `strata-sycl-dev` image (oneAPI 2026.1 compiler, oneMKL and Level Zero, from a -community llama.cpp SYCL image), so the PC needs no oneAPI. From the checkout (its folder is called `Strata` here; the -folder above it is what the container mounts at `/work`): - - cd Strata - docker build --pull -t strata-sycl-dev sycl/tools # once: about 6 minutes, a 13 GB image - H=$(basename "$PWD") - -**Arc B-series** (Pro B70: `bmg-g31`; B580, B570 and Pro B60: `bmg-g21`; `ocloc ids bmg-g21` in the image lists the -device ids). Ahead-of-time code, no compile at the first start: - - docker run --rm -u $(id -u):$(id -g) -v "$PWD/..:/work" -e AOT=bmg-g31 -e REPO=/work/$H \ - -e BUILD_DIR=/work/$H/build-sycl-aot strata-sycl-dev "cd /work/$H && bash sycl/tools/build.sh strata" - -**Arc A-series** (Alchemist: A750, A770, A580, A380): no `AOT=`. The code is compiled from SPIR-V at the first start -(JIT, about a minute). Nobody has run an AOT build for these cards. - - docker run --rm -u $(id -u):$(id -g) -v "$PWD/..:/work" -e REPO=/work/$H \ - -e BUILD_DIR=/work/$H/build-sycl strata-sycl-dev "cd /work/$H && bash sycl/tools/build.sh strata" - -`--pull` matters: the image starts from a community image whose tag moves, and `docker build` reuses a copy of it from an earlier -build unless it is told to fetch the current one. An old copy can hold a device compiler (ocloc from Compute Runtime -26.22.38646, IGC 2.36.3) that crashes on the Battlemage kernels ("IGC: Internal Compiler Error: Floating point exception", 77 device -images, then `gen compiler command failed with exit code 245` at the last link, #3219). `sycl/tools/build.sh` checks the -version before the compile and stops with the fix. If you cannot update the image, `-e I8_ESIMD=OFF` builds without the fused int8 -kernels (stubs: the prompt's experts take the slower XMX path, the answers do not change); `-e ALLOW_BAD_OCLOC=1` builds anyway. - -The build takes about 3 minutes and ends with `BUILD EXIT 0` and `errors: 0`. The binary is `build-sycl-aot/strata` -(B-series) or `build-sycl/strata` (A-series); setup finds either one. Run the `docker build` again after pulling a -Strata update that changes `sycl/tools/Dockerfile`, and the `docker run` after every update of the engine (setup -says when the engine is older than it needs). - -## Things that matter on this GPU - -- **`SYCL_CACHE_PERSISTENT` must be 0.** The persistent JIT cache segfaults on Xe2 during the first - compile. The start script sets it; if you run llama-server by hand, do too. -- **The whole model goes on the card** (`--n-gpu-layers 999`), except `per_layer_token_embd.weight`, - the single 28.8 GB tensor of shard 2. `--override-tensor per_layer_token_embd=CPU` keeps it in host - memory, mmapped and paged from the SSD by row, which is how the model's authors serve it. Host RSS - stays around 2 GB. -- **Thinking.** The model reasons before it answers. Strata's web app has the setting; for an API - client, put `{"reasoning_effort": "none"}` in `strata-.shared-settings.json` next to the - config, or send `chat_template_kwargs: {"enable_thinking": false}` per request. Without one of - those a short `max_tokens` is spent entirely inside the think block and the answer looks empty. -- **`/health` says 503 while loading**; the server polls `/props` instead. - -## How much context fits - -The architecture keeps this small: only every fourth layer is full attention (12 of 48); the other -36 are gated-delta-net layers with a fixed-size recurrent state that does not grow with context. -Per token, the 12 attention layers keep 2 KV heads x 256 x (K + V) at q8_0, plus the -sparse-attention indexer's keys. **Measured: about 20 KiB per token** (VRAM grows 0.6 GB per 32k -of context with everything else unchanged). - -| context | VRAM in use, model loaded | headroom on 32 GB | status | -|---|---|---|---| -| 32,768 | 28.4 GB | 3.5 GB | measured, the default | -| 65,536 | 29.0 GB | 2.9 GB | measured, loads | -| 98,304 | 29.6 GB | 2.3 GB | measured, loads | -| 131,072 | 30.3 GB | 1.5 GB | **measured, the practical ceiling**: a 104,798-token prompt (79% of the window) read in 247 s at 424 tok/s, then answered | -| 163,840 | ~31.0 GB | ~0.9 GB | not attempted: under the 1.2 GB safety margin | -| 262,144 | ~32.6 GB | none | **does not fit - asking for it took the host down** | - -**Do not ask for more than fits.** On this driver a GPU allocation past VRAM does not fail: the xe -driver evicts buffers into host RAM, the kernel runs out of memory, and the machine livelocks until -its hardware watchdog resets it. A 262K request did exactly that on 2026-09-29, and llama.cpp's own -"failed to fit params" check fired too late to prevent it. Compute the KV size first and leave -1.5 GB free. - -Prompt reading gets faster with size, because the work batches better: ~150 tok/s on a 2.7k-token -prompt, 424 tok/s on a 105k-token one. So a full 128k window costs about five minutes to read, -not fifteen. - -## The engine itself on Intel: the SYCL port (`sycl/`) - -Everything above runs the model through llama.cpp. This is the other half: Strata's own engine - the 50 CUDA -kernels and the host code that drives them (streams, events, graph capture, pinned memory) - built for the -Arc with oneAPI. It is a migration of the tree, not a new backend: the engine has no backend seam to slot -into. - -**Layout.** `sycl/src` and `sycl/include` mirror the tree and hold only the files the port changes; the -SYCL build (`sycl/CMakeLists.txt`) takes every other source from the original location. No upstream file -is edited. `sycl/include/dpct/` is the vendored SYCLomatic helper library, so the port builds without the -migration tool. - -**How it was made, so it can be redone.** - -1. `sycl/tools/Dockerfile` - the dev image: the llama.cpp SYCL image plus SYCLomatic (`dpct` 2025.3), - ninja, and the CUDA 12.8 headers that `sycl/tools/get-cuda-headers.sh` pulls out of NVIDIA's pip wheels - (dpct parses CUDA; it needs the headers, not the toolkit). -2. `sycl/tools/migrate.sh` - writes a compilation database for the 86 CUDA-touching translation units and - runs dpct over them. 85 migrate; dpct reports no line it could not migrate, and about 1,400 advisory - notes. -3. `sycl/tools/fixups.py` - what dpct got wrong or could not do, as an idempotent script with a reason per - item. The ones that mattered: - - CUDA's null stream means the default stream; dpct turned it into a null `sycl::queue*`. Every - stream cast now goes through `strata::q_of()` (`sycl/include/strata/sycl_queue.hpp`). - - `__ldg((const float*) p)` came out as `*p`, reading one byte of a float scale (two sites, s_gemv). - - `__fadd_rn(a, b ? c : d)` lost its parentheses (two sites). - - the ggml lookup tables were threaded through kernel parameters with the wrong table per template; - they are plain `static const` arrays read from device code, as ggml-sycl does. - - helper headers renamed since dpct 2025.3 (`entangle`, `chunked_partition`), graph introspection and - `cudaGraphUpload` (no SYCL equivalents), `%globaltimer` (the stage profiler reads zeros). -4. `sycl/tools/build.sh` - configure + build with icpx inside the image. Two compiler flags are load-bearing: - `-fp-model=precise` (icpx defaults to a fast FP model) and `-cl-fp32-correctly-rounded-divide-sqrt` - for the device compiler - the Arc's fp32 divide is not correctly rounded by default (OpenCL allows - 2.5 ulp), and Strata's quantizers are byte-exact against ggml through `amax / 127`. Measured: without - it `quantize_act_parity` has 303k mismatches, with it none. - -**Where it stands (2026-09-29).** The whole tree builds and links: the `strata` binary and 22 kernel -parity tests. On the B70, 19 of the 22 pass byte-for-byte or within their tolerances; the other three -need model fixtures (`iq_parity`, `ple_parity`) or the tensor-core kernel below (`qsa_prompt_attn`). - -| parity test | result | -|---|---| -| bf16_gemv, cvec, dequant_s2, elementwise, gdn, gr, kv_q4, kv_q8, kv_stream, qsa, quantize_act, rope, router_top10, s2_gemv, s2_gemv_q8, s_gemv, s_gemv_q8k, sampler, shared_expert | pass | -| iq_parity, ple_parity | `ctest` generates the `iq_parity` fixtures into the build tree (`iq_parity_fixtures`, needs python3 with numpy; skipped without). `ple_parity` needs the Q2_0 shard 2 GGUF and the dense pack: configure with `-DSTRATA_PLE_GGUF=` (and `-DSTRATA_PLE_FIXTURE_DIR=`); a missing input is a hard error, as in the CUDA build (#1795) | -| qsa_prompt_attn_parity | needs the tensor-core kernel (below) | -| kv_hybrid_parity (0.1.27, hybrid K8V4 KV) | appends, gathers and attention pass; its last step is that same tensor-core kernel | - -**The engine end to end (2026-09-29, later the same day).** It generates. Prompt "Write a Python function -that returns the n-th Fibonacci number", greedy, 64 tokens: - -> `` The user wants a Python function that returns the n-th Fibonacci number. Let me consider the -> options: 1. **Recursive approach** - Simple but O(2^n) time complexity, very slow for large n. 2. -> **Iterative approach** - O(n) time, O(1) ... - -Measured on the same card, same 2,185-token prompt, 64 greedy tokens (AOT build, `--spec 2`, -`--no-prefill-borrow`), against llama.cpp's SYCL build serving the same GGUF: - -| | llama.cpp SYCL | Strata SYCL port | -|---|---|---| -| prompt reading | 138-319 tok/s | **560 tok/s** (693 at 2,000 tokens, 180 at 300) | -| decode, suffix drafter only (`--spec 2`) | 24.2-26.0 tok/s | 20.2-20.8 tok/s | -| decode with the MTP draft layer (`--spec 4 --mtp`) | - | **42.7 tok/s** at the 2,185-token context (81% accepted, 2.9 tokens/round); 45.3 on a short prompt | - -Prompt reading is where Strata's design pays (oneMKL GEMM over dequantised experts, the whole model on -the card). Decode is at ~80%: a speculative round costs ~53 ms whatever its size and the suffix drafter -is accepted 9% of the time, so nearly every round yields one token; `--spec 4` (16.9 tok/s) was worse -than `--spec 2` (20.8) for that reason. The kernels themselves run at 130-160 GB/s of weights on a card -that streams 600 GB/s (`sycl/probe/bw.cpp`, incompressible data) - the same class llama.cpp reaches - -and five variants of the hot Q6_K matvec (lanes per row, rows per warp, unroll, 16-byte loads, software -pipelining; `mmvq_bench`) all landed in that band. The draft source was the lever: the base -Qwen3.8-Flash-Next checkpoint's MTP layer (`tools/mtp_fetch.py fetch`, `mtp_pack.py --experts q2_0`, -`mtp_rt.py`; 4.9 GB downloaded, 809 MiB of VRAM) drafts for the Coder fine-tune at 87% acceptance: -3.4 tokens per 53 ms round, 45 tok/s, the same greedy tokens. The `mtp.cpp` path ran on the first try -under the same flags. - -Without AOT the runtime JIT-compiles every kernel on first use, ~47 s the first time a process runs; -`SYCL_CACHE_PERSISTENT=1 SYCL_CACHE_DIR=` keeps that across runs (15 MB). - -How to run it by hand (a greedy test run, the way every number in this section was measured; inside the -`strata-sycl-dev` image, AOT build in `build-sycl-aot/`): +The engine is built in Docker, in the `strata-sycl-dev` image (oneAPI 2026.1 compiler, oneMKL and Level Zero, from a community +llama.cpp SYCL image), so the PC needs no oneAPI. From the checkout (the folder above it is what the container mounts at +`/work`): +```sh +cd Strata +docker build --pull -t strata-sycl-dev sycl/tools # once: about 6 minutes, a 13 GB image +H=$(basename "$PWD") ``` + +**Arc B-series** (Pro B70: `bmg-g31`; B580, B570 and Pro B60: `bmg-g21`; `ocloc ids bmg-g21` in the image lists device ids). +Ahead-of-time code, no compile at the first start: + +```sh +docker run --rm -u $(id -u):$(id -g) -v "$PWD/..:/work" -e AOT=bmg-g31 -e REPO=/work/$H \ + -e BUILD_DIR=/work/$H/build-sycl-aot strata-sycl-dev "cd /work/$H && bash sycl/tools/build.sh strata" +``` + +**Arc A-series** (Alchemist: A750, A770, A580, A380): no `AOT=`. The code is compiled from SPIR-V at the first start (JIT, about +a minute). Nobody has run an AOT build for these cards. + +```sh +docker run --rm -u $(id -u):$(id -g) -v "$PWD/..:/work" -e REPO=/work/$H \ + -e BUILD_DIR=/work/$H/build-sycl strata-sycl-dev "cd /work/$H && bash sycl/tools/build.sh strata" +``` + +The build takes about 3 minutes and ends with `BUILD EXIT 0` and `errors: 0`. Setup finds `build-sycl-aot/strata` or +`build-sycl/strata`. Run the `docker build` again after pulling an update that changes `sycl/tools/Dockerfile`, and the +`docker run` after every engine update (setup says when the engine is older than it needs). + +| `build.sh` variable | Effect | +| --- | --- | +| `AOT=bmg-g31` or `bmg-g21` | Ahead-of-time device code. Empty: SPIR-V, JIT at first start. | +| `I8_ESIMD=OFF` | Build without the fused int8 ESIMD expert kernels (stubs; the prompt's experts take the slower XMX path, answers do not change). | +| `ALLOW_BAD_OCLOC=1` | Build although this `ocloc` is known to crash on the Battlemage kernels | +| `SPIN_MAX=N` | The bound on a device spin (`STRATA_SYCL_SPIN_MAX`); see [Things that matter](#things-that-matter-on-this-gpu) | + +`--pull` matters because the image starts from a community image whose tag moves, and `docker build` reuses an old copy unless +told to fetch the current one. An old copy can hold a device compiler (ocloc from Compute Runtime 26.22.38646, IGC 2.36.3) that +crashes on the Battlemage kernels (`IGC: Internal Compiler Error: Floating point exception`, then `gen compiler command failed +with exit code 245`, #3219). `sycl/tools/build.sh` checks the version and stops with the fix. + +## How to run it by hand + +A greedy test run, inside the `strata-sycl-dev` image, AOT build in `build-sycl-aot/`: + +```sh STRATA_VERIFY_DEVICE_PLAN=1 STRATA_VERIFY_NO_HOST=1 \ build-sycl/strata --pack --native --ple-gguf \ --expert-profile data/expert-profile-coder.bin --expert-cache auto --stream-experts \ --prefill auto --spec 4 --spec-min-p 0.5 --max-context 8192 --tokens --max-new 64 --greedy ``` -- `--stream-experts` (this port): no resident host copy of the experts; every one of the 12,288 goes from - the GGUF into the VRAM cache through a small staging ring (`GgufExpertSource`). Upstream needs 32 GB of - RAM for this model; with the flag the engine runs on 23 GiB. -- `STRATA_VERIFY_DEVICE_PLAN=1`: the GPU plans each layer itself (upstream's E-6; off by default there). -- `STRATA_VERIFY_NO_HOST=1` (this port): the host waits for the whole window graph instead of per-layer - rings. Only valid with every expert resident, which is the case on a 32 GB card. -- `--no-prefill-borrow`: the prompt path must not lend expert slots (a lent expert is served from the host - behind a flag the GPU does not see reliably here; long prompts hung without it). +| Flag or variable | Meaning | +| --- | --- | +| `--stream-experts` | No resident host copy of the experts: every one goes from the GGUF into the VRAM cache through a small staging ring. The engine runs on 23 GiB of RAM; upstream needs 32 GB for this model. | +| `STRATA_VERIFY_DEVICE_PLAN=1` | The GPU plans each layer itself | +| `STRATA_VERIFY_NO_HOST=1` | The host waits for the whole window graph instead of per-layer rings. Required on `xe` whenever part of the experts is not in VRAM. | +| `--no-prefill-borrow` | The prompt path does not lend expert slots. The port borrows by default only above a 32K context (`--prefill-borrow` / `--no-prefill-borrow` decide explicitly). | -What the port had to get right beyond compiling (each is an entry in `sycl/tools/fixups.py` or a flag): +With `--tokens-file` instead of `--tokens` for long prompts (80K ids exceed Linux's single-argument limit). With a native pack +`--prefill-until N` skips the token loop, so compare output tokens between paths, never only timings. -- **Host-mapped flags.** Strata's decode is a GPU/CPU handshake through host-mapped memory. `volatile` - device loads do not bypass the caches on Intel; system-scope atomics do, and they are the only thing - measured to work (`sycl/probe/doorbell.cpp`, six variants). Host-to-device visibility *during* a kernel - stays unreliable on this platform, which is why the all-resident path waits for the window instead. -- **Bounded spins.** A device spin that never sees its flag is not a hang of one process: the xe driver - times the queue out and resets the GT node by node - a window graph has 2,400 - and the card stays - wedged until a reboot (twice). Every spin is capped (`kSpinMax`). -- **32-lane sub-groups.** The kernels are written for warps; dpct pinned 134 of 289 launches, the rest - would run at Xe2's default 16 (`-fsycl-default-sub-group-size=32`). -- **Synchronous copies.** `cudaMemcpy` blocks; dpct's default-queue `memcpy` did not wait. With the - streaming ring that filled the expert cache from overwritten buffers: non-deterministic residuals, NaN by - layer 5. Found with the per-layer residual ladder (`STRATA_VERIFY_DEBUG=1` prints R after every layer). -- `native_expert_parity` hand-ported: the GPU native expert kernel matches ggml's float reference on real - IQ1_M rows (rel 1.1e-2, the same class as the CPU path). - -Profiling: `sycl/tools/Dockerfile.unitrace` builds the dev image with Intel's unitrace; `unitrace -d` around -the engine plus `sycl/rank_kernels.py` gives device time per kernel. `mmvq_bench` and `native_expert_parity -NATIVE_BENCH=1` time the two hot kernel families in isolation (warm the clocks first: a 5 ms run measures -the ramp, not the kernel). - -**After the 0.1.25-0.1.27 merge (2026-09-30, same card, same flags, AOT).** Fibonacci prompt (19 tokens), -64 greedy tokens: decode 46.5 tok/s (85% of drafts accepted). The 2,184-token prompt: 566 tok/s prompt, -38.3 tok/s decode (77% accepted on that text); the draft layer's prompt pass is now upstream's batched one -(38.6 ms for 2,184 tokens, from 105 ms for 19 before it). Parity: 19 of 22, byte-exact `quantize_act` back. - -**Speed work, 2026-09-30 (after the 0.1.27 merge).** What moved the numbers, measured on the B70 with the -same prompts as above (outputs identical before and after each change): - -| change | effect | -|---|---| -| every window graph and the drafter's graphs captured at load (`STRATA_WARM_GRAPHS=0`: as before) | the first request no longer pays 250-290 ms of captures | -| the commit graph left running while the drafter's round runs on its own queue (`Verifier::commit(wait=false)` + `commit_finish`) | decode 38.3 -> 43.0 tok/s at the 2,184-token context, 46.5 -> 49.8 short | -| upstream 0.1.29 merged (GDN recurrence pipelining, the block-scores read, sampler) | 2,184-token prompt 765 -> 792 tok/s, decode at that context 43.1 -> 45.2 | -| the draft layer's batched prompt pass takes prompts under 64 rows (it fell back to per-6-token graphs) | its prompt cost on 19 tokens 106 -> 16 ms | -| one device module per kernel (`-fsycl-device-code-split=per_kernel`) | the first launch in the prompt path (the embedding gather) 245 -> 1 ms | -| the expert dequant writes each thread's run of FP16 values as one vector store (it wrote them one 2-byte store at a time) | dequant per expert 0.085 -> 0.030 ms; prompt 496 -> 575 tok/s at 2,184 tokens, 720 -> 841 at 8,000 | -| SWAR sign compare/subtract in the expert dots, a local-memory resident-plan kernel, a split-K fused down kernel (bit-identical to the single-token one) | parity-clean, no measurable decode change; kept | -| the PLE row reader's default queue depth 16 -> 64 blocking O_DIRECT threads (`STRATA_IO_THREADS`; 128 and 256 change nothing: the drive tops out near 85k IOPS on the 27k random 4 KB reads a 2,184-token prompt needs) | the rows of that prompt 466 -> 330 ms | -| a short first prompt chunk (`STRATA_PREFILL_FIRST`, default 256 tokens) so the GPU starts while the remaining rows are still being read | 2,184 tokens: 711 -> 765 tok/s, 8,000: 857 -> 986; time to first token about 150 ms less. The chunk boundary moves rounding in the expert GEMMs, so a long greedy continuation can diverge late (token 45 of 64 on the test prompt) | - -Draft policy sweep (2,184-token prompt, 128 tokens): windows of 6 (`--spec 4`) at 41.6-43.5 tok/s; `--spec 2` -33.6, `--spec 6` 35-37; `--spec-min-p` 0.3-0.7 within noise. Profiles (unitrace, `strata-sycl-dev:metrics` with -Intel's metrics libraries and `dev.xe.observation_paranoid=0`): the expert dot kernels are ALU-bound (77% XVE -active), the dense projections memory-latency bound (92-128 GB/s at 84-93% occupancy), a decode round is 80-85% -kernel time and ~5 us of launch gap per node over 2,400-2,600 nodes. - -**XMX.** oneMKL's FP16 GEMMs already run on the XMX units (30-60 TFLOP/s in `xmx_gemm_bench`); the prompt path is -bound by the dequant that feeds them, not by the products. Two joint_matrix kernels were written and are correct -but lose to the existing paths on this card, so both stay opt-in: `xmx_gemm_iq` (a fused dequant + GEMM straight -from the quantized rows, 4-5x slower than dequant + oneMKL) and `qsa_prompt_attn_xmx` (the port of the mma.sync -prompt attention; `qsa_prompt_attn_parity` passes at 1e-6 of scale, 3x slower than the FP32 fallback per chunk; -`STRATA_PROMPT_ATTN_XMX=1`). Where the prompt time goes now (2,184 / 8,000 tokens): dequant 28% / 27%, GEMMs -22% / 20%, the per-layer host grouping 9.5% / 9%, attention 7% / 14%, embeddings + PLE rows 13% / 5%. - -A trap worth knowing: with a native pack `--prefill-until N` does not feed the rest of the prompt through the -token loop (it is skipped for native packs), so the tokens after N are dropped and the model free-runs. Compare -output tokens between paths, never only timings. - -**An 80,000-token prompt (2026-09-30).** Where the time goes changes completely at this scale: - -- **The VRAM plan.** `--expert-cache auto` fills the card down to `--vram-reserve-mib` *before* the KV state - (about 1 GB in INT8 at 81,920 cells) and the prompt chunk buffers (about 2 GB at `--prefill 8192`) exist. With - the 1,536 MiB default that put 30.95 of the B70's 32.6 GB in use, the driver started migrating buffers and the - run never finished. `--max-context 81920 --kv int8 --vram-reserve-mib 3072 --prefill 4096` peaks at 28.8 GB. - (80k ids also exceed Linux's 128 KB single-argument limit: `--tokens-file`.) -- **Streamed experts.** The reserve costs cache slots (10,348 of 12,288 at 3 GB), and the prompt path streams the - missing experts from the GGUF for every chunk. Two port bugs sat on this path and are fixed: the stream plan - held `GgufExpertSource::blob()` pointers across ~1,900 reads of a 512-slot ring (so blobs were overwritten - before they were copied: the 75 s / 1,065 tok/s measured first was computed partly with the wrong experts), and - the stream-all walk hangs on this card in its first large chunk (the copy engine stops on a barrier). The stager - threads now read the blobs themselves, and the port uses the per-layer routed-only walk (`STRATA_PREFILL_RING=8` - upstream, the port's default; `STRATA_PREFILL_STREAM_ALL=1` restores the other for debugging). -- **Correct result: 80,000 tokens in 101 s = 790 tok/s.** GPU time: expert down GEMM 26.0 s, attention 15.4 s, - dequant 12.7 s, QSA block selection 9.3 s (0.2 s at 8k: it scans every block of the context per query), gate/up - GEMM 8.9 s, gather 6.1 s, the per-layer grouping sync 5.2 s, GDN recurrence 4.8 s. The PLE rows are free at this - scale (97.6% row-cache hits). -- **Decode after such a prompt, and the fix.** Without lending cache slots to the prompt path (`--no-prefill-borrow`, - which the port used from the start) the reserve evicts ~1,900 experts for good and decode after the prompt runs at - 2 tok/s. Borrowing lends ~940 slots to the prompt path and refills them in about a second afterwards: 80,000 - tokens in 75 s = **1,062 tok/s, decode 35-40 tok/s** after it, at `--vram-reserve-mib 2048 --prefill 4096`. - It costs ~1 s on short prompts (2,184 tokens: 610 vs 792 tok/s), so the port borrows by default only above a - 32K context (`--prefill-borrow` / `--no-prefill-borrow` decide explicitly). - -**Long contexts (2026-10-01, engine 0.1.32-sycl, setup's flags: `--kv-resident 32768 --vram-reserve-mib 2048 ---prefill 4096`, the prompt path borrowing cache slots - the default above 32K).** Same text repeated to length, -then 256 greedy tokens: - -| context | KV | prompt | decode after | peak VRAM | experts in the host mirror | -|---|---|---|---|---|---| -| 128K | int8 | 888 tok/s (144 s) | **66.6 tok/s** | 29.9 GB | none | -| 128K | q4_0 | 848 tok/s (151 s) | 62.8 tok/s | 29.7 GB | none | -| 128K | k8v4, KV in VRAM | **1,006 tok/s** (127 s) | **66.8 tok/s** | 30.7 GB | 103 (0.2 GB) | -| 256K | int8 | 757 tok/s (338 s) | **55.9 tok/s** | 30.2 GB | none | -| 256K | q4_0 | 737 tok/s (347 s) | 51.7 tok/s | 30.0 GB | none | -| 256K | k8v4, KV in VRAM | 823 tok/s (311 s) | 52.2 tok/s | 30.9 GB | 865 (1.7 GB) | - -Borrowing keeps every expert in VRAM (the prompt buffers live in lent cache slots, refilled after the prompt). -Without it (`--no-prefill-borrow`) the buffers keep their own VRAM and ~1,000-2,300 experts move to the pinned host -mirror: the prompt is 2-5% faster, decode after it 10-23% slower (0.1.31 without borrowing: 128K int8 934 / 59.5, -q4_0 887 / 53.4, k8v4 1,026 / 54.4; 256K int8 784 / 48.7, q4_0 763 / 43.2, k8v4 838 / 42.3 tok/s). - -KV streaming (`--kv-resident`) keeps the whole KV in pinned host memory and only the attended window in VRAM, so -the KV pushes no experts out; setup.py turns it on from 64K up and keeps INT8 (the faster of the -two at every size here). Without it the KV pushes experts out and decode after the prompt fell to 4-9 tok/s at -128K-256K (2026-09-30). `--kv k8v4` does not support streaming yet: setup keeps its KV in VRAM, which pushes up to -more experts out; those come from the pinned host mirror over PCIe (planned item 2), so decode holds at 42 tok/s at -256K (it was 3.8 before the mirror). k8v4 reads prompts fastest (no streaming copies) and works through the FP32 -attention fallback (its dedicated prompt kernel is the XMX one below). - -**XMX prompt attention v2.** 64-cell chunks, vector-packed K^T and V, hi+lo Q in one accumulator per scale group, -one accumulator update per chunk: 1.4-1.5x faster than v1 and correct in all modes (`qsa_prompt_attn_parity`, -and `kv_hybrid_parity` passes completely with it), but still ~2x slower than the FP32 fallback (13.4 vs 5.5 ms per -chunk, INT8, 32K context). This attention is gather-bound: each query position selects its own ~2,000 cells, so -the K/V fetch dominates and only 12 of the 16 matrix rows are real heads. Opt-in: `STRATA_PROMPT_ATTN_XMX=1` -(64-cell chunks, 120 KB of local memory) or `=32`. - -**XMX v2 in the full matrix, and why a grouped kernel is not next (2026-09-30).** With `STRATA_PROMPT_ATTN_XMX=1` -(stage timing on, ~8-10% overhead) the long prompts run 23-33% slower than with the FP32 attention: 128K int8 666 vs -960 tok/s, k8v4 596 vs 983; 256K k8v4 506 vs 752, int8 536 vs 718 (q4_0 does not use the XMX kernel). The obvious -fix, gathering the union of neighbouring positions' cells once, depends on how much their selections overlap. -Measured on the last chunk of an 80K prompt (first QSA layer, `STRATA_DUMP_SEL=`): the union of 8 consecutive -positions is 3.3x one position's 2,051 cells (16: 5.1x), and only 12% of a selection is shared by all 8. Grouping -would cut the K/V gather to ~40% but multiply the arithmetic by 3-5x: at best 5-8 s of an 80K prompt. Not built. - -**Serving the port (2026-09-30).** `serve/server.py --engine strata` runs the SYCL engine unchanged through -`sycl/serve/strata-sycl.sh`, an `exe` that starts the binary inside the oneAPI runtime image with the serve pipes -attached (paths in the config's `args` are the container's, the data root mounted at `/work`). A config: +Serve the port with `serve/server.py --engine strata` and `sycl/serve/strata-sycl.sh` as the `exe`, which starts the binary inside +the oneAPI runtime image with the serve pipes attached. Paths in the config's `args` are the container's. A config: ```json {"engine": "strata", "exe": "/sycl/serve/strata-sycl.sh", @@ -391,635 +124,389 @@ attached (paths in the config's `args` are the container's, the data root mounte "--expert-profile", "data/expert-profile-coder.bin", "--expert-cache", "auto", "--stream-experts", "--prefill", "auto", "--spec", "4", "--spec-min-p", "0.5", "--mtp", "/work/mtp/rt", "--max-context", "32768", "--kv", "int8", "--vram-reserve-mib", "1024"], - "sampling": {"temperature": 0.6, "top_p": 0.95, "top_k": 20, "repetition_penalty": 1.05}, ...} + "sampling": {"temperature": 0.6, "top_p": 0.95, "top_k": 20, "repetition_penalty": 1.05}} ``` -Measured through the OpenAI API with those sampling defaults: 36 tok/s decode on a first turn, 33 on a follow-up -(which reuses the conversation's cached prompt), against 24-26 for llama.cpp on the same card. After the decode work -(2026-10-01 build): 48-72 tok/s on a ~190-token answer, prompt included (the first request after a start is the slower -one); 55-59 tok/s over 5,000-18,000-token answers on the 2026-09-30 build. The reserve matters: -with `--stream-experts` there is no host copy of the experts, so any expert left out of VRAM is read from the SSD -and computed on the CPU whenever it is routed. At 1,536 MiB the cache came up 128 experts short and decode fell to -5-10 tok/s; 1,024 MiB fits all 12,288 with 2 GB of VRAM still free. +With `--stream-experts` any expert left out of VRAM is read from the SSD and computed on the CPU when routed, so the reserve +matters: on the B70 with Coder IQ1_M a 1,536 MiB reserve left the cache 128 experts short and decode fell to 5-10 tok/s, while +1,024 MiB fits all 12,288 with 2 GB of VRAM free. -**Logprobs (2026-10-02).** `/v1/chat/completions` takes OpenAI's `"logprobs": true` and `"top_logprobs": 0..20`, -streamed or not, through `sycl/serve/server_intel.py`; the reply carries `choices[0].logprobs.content[]` (`token`, -`logprob`, `bytes`, `top_logprobs`), and with thinking on the thinking tokens under `reasoning_content`. The engine -(the port only) gets `logprobs=K` on its GEN line and writes an `LP logprob id:logprob ...` line after each `T` -line, from the verify window's head logits - the distribution the token was taken from, before temperature, -penalties and sampling. Upstream's server ignores the lines, and an engine never asked writes none. Each token -costs one 1 MB logits row copy when asked. Measured (IQ2_XS, "capital of France"): "Paris" at -0.0019 (p 0.998), the -same answer and no measurable time without it. Not on `/v1/completions` or the Anthropic endpoint yet. +## Things that matter on this GPU -**The Monitor tab on Intel.** `sycl/serve/server_intel.py` is `serve/server.py` with two additions made at run -time: +- **`SYCL_CACHE_PERSISTENT` must be 0.** The persistent JIT cache segfaults on Xe2 during the first compile. The start script + sets it. +- **Do not ask for more VRAM than the card has.** On the `xe` driver an allocation past VRAM does not fail: the driver evicts + buffers into host RAM, the kernel runs out of memory, and the machine can livelock until its hardware watchdog resets it. + Compute the KV size first and leave about 1.5 GB free. +- **Thinking.** The model reasons before it answers. Put `{"reasoning_effort": "none"}` in + `strata-.shared-settings.json` next to the config, or send `chat_template_kwargs: {"enable_thinking": false}` per + request. Otherwise a short `max_tokens` is spent inside the think block and the answer looks empty. +- **`/health` says 503 while loading;** the server polls `/props` instead. +- **Bounded spins.** A device spin that never sees its flag makes the `xe` driver time out the queue and reset the GT, and the + card can stay wedged until a reboot. Every spin is capped. The cap is the build option `STRATA_SYCL_SPIN_MAX` (2,000,000 + reads, a few seconds, unless the build is an AOT build for a `bmg` card, which keeps 20,000). +- **JIT builds** add `-ze-opt-greater-than-4GB-buffer-required` so kernels keep 64-bit addresses past 4 GiB into an allocation. + oneMKL's own precompiled kernels do not get it; the GEMM stages an operand past 4 GiB (`STRATA_SYCL_GEMM_STAGE=1` forces + that on Battlemage). +- **Aliased pages in big allocations (xe).** The driver can return a large arena with two 2 MiB pages mapped onto the same + memory. Every allocation of 32 MiB or more goes through `strata::malloc_device_guarded`, which checks the range and allocates + again behind a spacer until it is clean (a warning names each retry; under 0.5 s at startup; `STRATA_ARENA_ALIAS_CHECK=0` + skips it). `STRATA_VERIFY_ALL_SLOTS=1` reads every filled cache slot back and compares it with the GGUF. +- **Monitor tab.** `server_intel.py` plugs `sycl/serve/xe_telemetry.py` into the Monitor (temperature, power, PCIe link from + sysfs). Load and VRAM come from `/run/gpustat.json`, written by a small root sampler (`sycl/tools/gpustat.py`, installed with + `gpustat.service`), because `xe` reports them per client in `/proc/*/fdinfo`, which only root can read. A run config may name + a `model_switcher`; the web app's header then gets a Model menu. +- **Logprobs.** `/v1/chat/completions` takes `"logprobs": true` and `"top_logprobs": 0..20` through `server_intel.py`, streamed + or not. Not on `/v1/completions` or the Anthropic endpoint. -- **GPU readings.** When NVML has no card, it plugs `sycl/serve/xe_telemetry.py` into `serve/telemetry.py`'s - `gpu_reader`. That reads the Arc through sysfs (temperature, power and its cap, the PCIe link the card - trained at). Load and VRAM come from `/run/gpustat.json`: xe reports them per client in `/proc/*/fdinfo`, - which only root can read, so a small root sampler writes them (`sycl/tools/gpustat.py`; install it with its - `gpustat.service` as its header says). -- **A Model menu.** A run config may name a `model_switcher` (an RPC taking `{"mode": m}`) for a host that swaps - models on one card. The web app's header then gets a Model menu (`sycl/serve/web/switcher.js`, injected into - the page). +## How much context fits -**Speed depends on the text.** Decode with speculative decoding tracks how often the draft layer guesses right. -The served engine at 32K context: 45-51 tok/s on the test prompt (continuing a Fibonacci function, 77-85% -of drafts accepted), 30-39 tok/s on a chat answer with prose (55-72%); sampling and the repetition penalty cost -nothing measurable. The engine used to abort at exit in serve mode (a queue wait in a destructor after the -runtime's teardown began); it now exits directly once its requests are done. +Only 12 of the 48 layers are full attention, so the KV grows slowly. KV streaming (`--kv-resident`) keeps the whole KV in pinned +host memory and only the attended window in VRAM; setup turns it on from 64K and keeps INT8 KV. `--kv k8v4` does not support +streaming: setup keeps its KV in VRAM, which pushes more experts into the host mirror. -**Planned (2026-09-30), in order.** Ranked by payoff on this card; the first three were taken first. +> **Measured on:** Arc Pro B70 32 GB, Coder IQ1_M, engine 0.1.32-sycl, setup's flags (`--kv-resident 32768 +> --vram-reserve-mib 2048 --prefill 4096`, the prompt borrowing cache slots), the same text repeated to length, 256 greedy tokens. -1. Expert dot products on XMX in integer mode: a decode window (up to 6 tokens) fits one INT8 DPAS (1-8 rows), the - i-quant grids decode to small integers, the activations are already INT8. Today: scalar dp4a, ALU-bound (77%). - **Parked**: three `joint_matrix` versions (opt-in `STRATA_EXPERT_XMX=1`, do not enable) were 1.4x and 2-3x slower - than dp4a, and the third (B filled in place with `joint_matrix_apply`) hung the GPU. At 1-6 rows the grid decode - and the packed-B layout cost more than the DPAS saves. -2. Experts missing from VRAM read from pinned host memory over PCIe instead of the SSD (the 256K decode collapse). - **Done**: at start-up every expert without a VRAM slot is read into a pinned host mirror (`STRATA_MIRROR_MIB`, by - default free RAM less 4 GiB), and the device-built verify plan points the expert kernels straight at it. Before, - that plan dropped non-resident experts, so output with misses was wrong. Forced test (`--expert-cache 8000`, - 1,879 experts / 3.6 GiB out of VRAM): decode 2.6 -> 40.9 tok/s (full cache 43.3), prompt 451 -> 640 tok/s - (776), output tokens identical to the full-cache run. About 3.6 GiB of VRAM can go to context for ~6% of decode. -3. KV streaming (`--kv-resident`) from 64K up: only the attended window of the KV in VRAM. **Done**: 256K decodes at - 31 tok/s (from 4-5). -4. QSA block selection on XMX: every query against every pooled block, a dense product that grows with the context. -5. The hot decode kernels re-tuned for Xe2's native 16-wide sub-groups (twice the registers per thread). - **Tried, no gain**: SIMD16 builds of the multi-column mmvq and the wide Q6_K kernel (`STRATA_MMVQ_SG`: 16 all, - 1 IQ4_XS only, 2 short outputs only; default 32). Alone (`mmvq_sg_bench`) SIMD16 is up to 1.45x faster on IQ4_XS - and 1.3x on the 640-row shared-expert projections, 10-25% slower on the large K-quant ones; in the engine no - setting beats SIMD32 beyond run-to-run noise (SIMD32 itself lands at 39 or 45 tok/s). Output tokens identical. - The same bench shows the real headroom: the dense decode kernels stream 100-280 GB/s of a 608 GB/s card - (Q6_K, the most common type, ~146 GB/s at 2-4 columns). - **Found and fixed: misaligned loads.** A Q6_K block is 210 bytes, so every block's `ql`/`qh` runs start only - 2-byte aligned, and the B70 splits a misaligned 16-byte load into pieces. The same weights repacked at a - 224-byte stride ran 2.3-4.7x faster (`q6k_align_bench`). Without changing the layout, the wide Q6_K kernel now - does two aligned 16-byte loads and a shift (`load16_a2`; both stay inside the block) and takes every Q6_K - shape and window width (it used to be gated to n_out >= 4096, 1-4 columns): 2560->10240 at 1/2/4/6 columns - 160/145/141/101 -> 493/432/331/273 GB/s, the 248K-row head 150 -> 407 GB/s at 1 column. Output tokens - identical. **Coder decode 44.9 -> 54 tok/s** (2k prompt, greedy). `STRATA_MMVQ_A2=0` restores the old path. - Still misaligned: IQ4_XS (136 B, 8-aligned, ~100 GB/s), IQ4_NL (18 B), Q8_0 (34 B); Q4_K/Q5_K are aligned - but use 4-byte loads in the multi kernel (230-280 GB/s). +| Context | KV | Prompt | Decode after | Peak VRAM | +| --- | --- | ---: | ---: | ---: | +| 128K | int8 | 888 tok/s (144 s) | 66.6 tok/s | 29.9 GB | +| 128K | q4_0 | 848 tok/s | 62.8 tok/s | 29.7 GB | +| 128K | k8v4, KV in VRAM | 1,006 tok/s | 66.8 tok/s | 30.7 GB | +| 256K | int8 | 757 tok/s (338 s) | 55.9 tok/s | 30.2 GB | +| 256K | q4_0 | 737 tok/s | 51.7 tok/s | 30.0 GB | +| 256K | k8v4, KV in VRAM | 823 tok/s | 52.2 tok/s | 30.9 GB | -**Decode round 2 (2026-09-30), from a unitrace of 256 decoded tokens.** Each step measured on its own, 19-token prompt, -256 tokens, two runs each, output tokens identical throughout: +Borrowing keeps every expert in VRAM (the prompt buffers live in lent cache slots, refilled after the prompt). Without it +(`--no-prefill-borrow`) about 1,000-2,300 experts move to the pinned host mirror: the prompt is 2-5% faster and decode after it +10-23% slower. -| | decode | -|---|---| -| after the Q6_K alignment fix | 53.9 tok/s | -| + `resident_plan` grouping in parallel (thread 0 alone took 73 us per layer, ~3.5 ms a round) | 57.1 | -| + the GR down kernel reads the activations directly (the 60 KB SLM tile per group capped occupancy) | 58.7 | -| + wide 16-byte-load kernels for Q4_K, Q5_K, IQ4_XS (codebook in registers) | 62.6 | -| + wide kernels for Q8_0 (MTP dense, shexp down of layer 0) and IQ4_NL (shexp down); Q6_K's activation loads aligned | 65.6 | -| + the GR down projection sliced by columns (each group stages one 128-column slice of xn, 4 lanes per row) + a fixed-order reduce | 70.6 | -| + IQ4_NL expert dots (the down projection in 39 of 48 layers): codebook in registers, weights as aligned 4-byte loads, bitwise the same ints (2026-10-01) | 76.1 | -| + GR norm per (token, stream), bitwise the same; aligned weight loads in the IQ3_XXS/IQ3_S/IQ2_S gate/up dots | **77.9** | +## Arc Pro B70 with a model that does not fit: Flash-Next IQ3_S -The last step changes the output: identical for 146 tokens, then a near-tie after a comma goes the other way (the new -kernels sum in a different order). Q8_0 kernels 4-6x (189 -> 31 us at 2560 x 10240, 2 columns), IQ4_NL 640 -> 2560 -28 -> 6.6 us. The aligned-load helper only loads its second chunk when the address is unaligned, so it never reads a -16-byte chunk without a needed byte and cannot cross a page at the end of an allocation. `STRATA_MMVQ_WIDE_32=0` -restores the old Q8_0/IQ4_NL kernels. The sliced GR down kernel (GR read 108.6 -> 76.5 us at 6 tokens, sums equal to -~1e-7) flips that near-tie back: its output is the original one. `STRATA_GR_DOWN_SLICED=0` restores the direct kernel. +IQ3_S has 24,576 experts of 1.97 MB (46 GB). The 32 GB card holds about 11.5k of them (22 GiB of cache) and the rest sit in a +pinned host mirror that the GPU reads over PCIe. Every prompt chunk streams the experts the cache does not hold, so a 4K prompt +moves about 26 GB; bigger chunks read faster. -Kernel level: IQ4_XS 107 -> 323 GB/s at 2 columns (61 -> 212 at 6), Q4_K/Q5_K 1.3-1.6x; the GR read 135 -> 110 us at -6 tokens (bitwise equal). Switches to the old paths: `STRATA_PLAN_PARALLEL=0`, `STRATA_GR_DOWN_DIRECT=0`, -`STRATA_MMVQ_WIDE_K=0`. (`iq_parity` reports 10 "missing fixture" failures with and without the change: the oracle -fixtures are not in the port's tree.) +- `setup_intel.py` writes `--prefill 4096` for a card with 24 GB or more (the chunk's buffers take about 700 cache slots). + `--prefill 8192 --prefill-borrow` lends cache slots instead and keeps all of them for decode, at about 1 s of refill per + prompt. Setup recommends 8,192 for a Battlemage card with 24 GB or more above 32K context. +- `STRATA_PREFILL_FIRST` (a short first chunk that starts the GPU while PLE rows are read) defaults to 256 tokens only when the + cache holds every expert, else 0. +- `STRATA_VERIFY_NO_HOST=1` is required whenever part of the experts is not in VRAM (`strata-sycl.sh` sets it). Without it the + host/GPU handshake is not visible across the bus on `xe` and the logits turn NaN a few tokens in. +- On `xe`, setup writes `data/expert-profile-sycl.bin`, a ranking built from the routing of 45 varied prompts, so the cache + holds the experts a chat, code or non-English prompt uses (`tools/make_profile_mix.py`, `tools/collect_profile_traces.py`). + An `i915` card keeps the shipped profile. +- The prompt's expert copies run on the card's copy engine on Battlemage (`STRATA_COPY_ENGINE=1` forces it elsewhere, `=0` + turns it off). The experts in the last 10% of the cache slots are also mirrored so that lent slots refill by DMA + (`STRATA_LEND_MIRROR=0` off). -`STRATA_IQ3S_WIDE=1` (opt-in, 2026-10-10) reads the dense IQ3_S matrices (attn_q, attn_gate, ssm_out, attn_k, the -shared experts' gate/up) with a wide kernel shaped like the Q4_K..IQ4_XS ones. unitrace, B70, IQ3_XXS, one card, T=1: -worst attn_k 77 -> 123 GB/s, best attn_q 114 -> 307 GB/s; these matrices take 0.60 ms of kernel time per token -instead of 1.36 ms. The greedy output parts from the default at token 198 of 256 (the sub-blocks are summed in -another order). Measured again for 0.1.45 (B70, IQ3_S, stock-like config, 5 interleaved pairs): decode 33.39 -> 33.29 tok/s -(-0.3%), 4K prompt 40.71 -> 40.79, so it stays opt-in; the gain the author reports is on two cards. +> **Measured on:** Arc Pro B70 32 GB (xe, kernel 7.0, oneAPI 2026.1, PCIe 13.3 GB/s host to device), 60 GB RAM, Flash-Next IQ3_S, +> 11.5k experts resident, greedy, `--spec 4 --spec-min-p 0.5 --mtp`, INT8 KV, `--stream-experts --vram-reserve-mib 1024 +> --ple-io ram`, engine 0.1.40-sycl to 0.1.43. The 10-prompt `mg_norepeat` gate passes 10 of 10. -**Q6_K decode kernels (0.1.45, #2340 leadgtr7, #2207 Andy Slater).** The Q6_K wide kernel took its six-bit offset (v - 32) -through a per-byte saturating subtract; it is now four integer ops (bitwise equal), and the Q4_K / Q5_K / IQ4_XS / Q6_K -kernels load the Q8_1 activations as one dword-vector. No switch. mmvq_bench Q6_K, B70: 2560x10240 1 column 44.7 -> 26.5 us, -4 columns 64.5 -> 46.2; A750: 2560x10240 4 columns 251 -> 102 us. End to end on the B70 (IQ3_S, 5 interleaved pairs, answers -identical): decode 32.24 -> 33.37 tok/s on a story prompt (+3.5%), 39.11 -> 40.77 after a 4K prompt (+4.2%), prompt rate unchanged. -`STRATA_MMVQ_LARGE_GRF` (default on for Arc A-series only, `=1` forces it on, `=0` off) launches the 2-8 column wide kernels with -256 registers per thread, where they spilled at 128 (the same bytes are produced either way). `STRATA_Q6_PLANAR` (default on, -`=0` off) converts the Q6_K output head and the MTP draft head to ql / qh / scales / d planes at load (same bytes, no VRAM change, -bitwise equal): 2560x248320 on a B70 800 -> 662 us (1 column), 1230 -> 1020 us (4 columns); end to end it adds +0.3% on the B70 -(33.16 -> 33.25 tok/s, within noise) on top of the kernel changes above. +| Workload | Result | +| --- | --- | +| 4,095-token prompt, `--prefill 4096` | 980-1,002 tok/s | +| 8,169-token prompt, `--prefill 8192` lending cache slots | 1,117 tok/s (9.4k slots) | +| Decode, 200-token story (66% of drafts accepted) | 30.5 tok/s | +| Decode, 200-token code answer (90% accepted) | 40.8 tok/s | +| 128K config, 4K prompt incl. refill (0.1.43) | 3.3 s (was 7.2 s); 16K 13.5 s (was 25.5); 49K 44 s (was 78) | +| `--prefill 8192` against 4096, 16K / 32K prompt (IQ3_S, setup's 128K flags) | 1,347-1,362 against 1,222 tok/s (+10-11%); 1,361-1,383 against 1,187 (+15-16%); decode after unchanged | +| Fused int8 expert prompt path (`auto`) against FP16 chain, 1,024 / 4,095 tokens | 783.5 against 645.8 (+21%); 1,407.7 against 1,257.0 (+12%) | -**Two-speed runs, explained (2026-09-30).** Identical greedy runs decode at either ~45 or ~39 tok/s. A per-gather -trace of the PLE reader (`STRATA_PLE_TRACE=1`) shows the slow runs pay one 226 ms PLE read stall in the first -decode round, after the window graphs are captured; every other read and round matches the fast runs. Prompt time -plus decode time is the same in both modes (4.95-5.17 s): the stall lands either in the prompt's PLE wait or in the -first decode round, so it is a once-per-process cost, not lost throughput. Ruled out: NVMe APST, the I/O scheduler -(`none`), CPU starvation (93% idle during decode), I/O thread count. Compare runs on time to first token + decode. -6. INT8 prompt GEMMs: experts dequantized to INT8, oneMKL/oneDNN INT8 on XMX (half the dequant bytes, 2x rate). -7. Fewer graph nodes per decode round (~2,500 at ~5 us): norm+rope, scores+top-k, gate+quantize fused. -8. Wider speculation (two draft branches per verify window): the kernels are latency-bound, so it is nearly free. -9. A model bigger than VRAM: the original Qwen3.8-Flash-Next IQ2_XS (35.5 GB of experts; the same path suits Q2_0 - and Swift 1.5) on the B70 with 23 GB of RAM. ~24 GB of experts in VRAM, the rest (~10-12 GB) in the pinned host - mirror the device plan reads over PCIe (item 2). The port has the IQ2_XS/IQ3_XXS/Q2_0 expert kernels. Open: whether - a 10+ GB pinned mirror fits beside everything else in 23 GB, and decode with that share of experts on a Gen3 x8 - link (3.6 GB mirrored measured 40.9 tok/s; expect less). Needs the 68 GB download. - **Done (2026-10-01)**: shard 1 downloaded (39.2 GB; shard 2 is byte-identical to the Coder's, so a hard link), - a native pack (`tools/iq_pack.py`, 6 s), the original model's expert profile, the same MTP draft layer. 18,329 of - 24,576 experts in VRAM (24.6 GiB), 6,247 in the pinned host mirror (8.4 GiB, 9 s to fill), host RAM never below - 12 GB free. **Decode 50.8 tok/s** on the 19-token prompt and **60.6** after 2,184 tokens, prompt 549 tok/s; - coherent, correct answers on both. Q2_0 and Swift 1.5 (similar size) should behave the same; IQ3_XXS/IQ3_S - (43-50 GB of experts) would need 19-26 GB mirrored, more than 23 GB of RAM allows. +Draft length (200 tokens, `--spec-min-p 0.5`): `--spec 6` against `--spec 4` is +6.5% on code and -1.5% on prose, so the setup +default (4, 0.5) stays. A `--prefill 8192` chunk is not bit-identical to 4096 (first-token KL 1.1e-2 at 16K, top-1 the same). -**Keeping up with upstream.** A merge of upstream `main` into `b70` leaves the copies in `sycl/` behind -wherever upstream touched a file they mirror. They are refreshed by re-migration, not by hand (done for -0.1.25-0.1.27, 2026-09-30): +**Fused int8 expert path (Battlemage default).** On a Battlemage GPU (B70, B580, B570, detected by device architecture) the +engine computes the prompt's routed experts with ESIMD kernels that decode the quantized weights in registers into the s8 DPAS B +operand and multiply them by q8_1 activations. Covered: gate/up IQ3_S, IQ3_XXS, IQ2_S and down IQ4_NL, Q2_0. Other formats keep +the FP16 path, as does every other GPU. `STRATA_PREFILL_MMQ` unset is `auto`, which times both chains at the first layer of each +format and takes int8 below the measured crossover (130-250 rows per expert); `=0` turns it off, `=1` forces it. Activations are +quantized, so bits differ from the FP16 path (KL 2e-4 nats mean over 6,002 positions, top-1 99.7%). Details: +[INTEL_INT8_EXPERTS.md](INTEL_INT8_EXPERTS.md). -1. Merge upstream into `b70`. No shared file should conflict: the Intel code is all in `sycl/`. Then check that - `sycl/setup_intel.py --check` and `sycl/serve/server_intel.py --engine mock` still run against the new - setup.py and server.py. -2. Migrate the *old* upstream tree (a `git archive` of the pre-merge commit) with `migrate.sh` into - `sycl-base`, and the merged tree into `sycl-new`; about 3 minutes each in the dev image. -3. `sycl/tools/normalize.sh ` on both: dpct's file names, the unchanged files, the - message serials, then `fixups.py`. Two runs of dpct on the same source now differ only in the kernel - name hashes it generates. -4. For every file present in both: `git merge-file sycl/F sycl-base/F sycl-new/F`. Files upstream did not - change are left alone. Copies dpct never produced (`verify.cpp`, `mtp.cpp`, `remote_experts.cpp` - include no CUDA header directly) take upstream's diff by hand; new parity tests are copied in and - listed in `sycl/CMakeLists.txt`. -5. A fixup whose pattern upstream changed shows up as a compile error (the PTX gate became an `#elif` - under upstream's `__HIPCC__` guard); extend the fixup, re-run it, rebuild. +## Arc A750 and the other Alchemist cards -**The 0.1.31 merge (2026-10-01).** Upstream 0.1.29 -> 0.1.31 (132 commits) by the steps above: 45 migrated files -changed for real (27 more only in dpct's kernel-name hashes), 38 conflicts resolved, verify.cpp/mtp.cpp/ -conversation_state.cpp ported by hand. A symbol audit (the port's feature identifiers counted before and after) is -part of the procedure now: it caught a dropped mirror hook and four doorbell waits that had lost their spin bound -(upstream renamed the loops; `fixups.py` covers both spellings). What the merge needed beyond conflicts: - -- **The expert kernels.** Upstream sends the i-quant experts to new multi kernels (one warp per row, 8 rows a - group); the port's grid is sized for its own kernels (8 lanes a row, 32 rows a group), so only a quarter of each - expert's rows were written and the output was end-of-text tokens. The port's kernels stay the default (they also - serve upstream's new Q4_K/Q5_K/Q5_1/Q8_0 experts); `STRATA_EXPERT_SPLIT=1` runs upstream's, with their grid. -- **Prompt-slot borrowing hung** in the first chunk of any prompt with it on (40K, 128K; GPU busy) - fixed after - the 0.1.32 merge, see "Prompt-slot borrowing: the hang" below. -- **Upstream's new tests found three dpct mistranslations from the first migration.** dpct writes `__fadd_rn(a, b)` - as `a + b` without parentheses, so `__fadd_rn(sum, c ? x : y)` became `sum + c ? x : y` (replaced, not added): in - the native QSA indexer's per-token pooled key (the decode path: blocks completed while generating got wrong - pooled keys; prompt blocks come from the batched kernel and were right), in the native QSA scores (an opt-in - path) and in the s2 activation rounding (every value 0; the Q2_0 s2 pack, unused here). Fixed, and `fixups.py` - now parenthesises. `qsa_parity`'s batch-vs-sequential test passes; decode output after long prompts changed. -- **Upstream's new kernels against the port's on the B70** (decode tok/s, Coder 19 / 2,184-token prompts, IQ2_XS - 19): port defaults 78.1 / 75.7 / 58.6; `STRATA_EXPERT_SPLIT=1` 65.6-69.5 / 67.2 / 49.1 (-11 to -16%); - `STRATA_GR_V3=1` 73.2 / 72.4 (-4 to -6%). Both stay opt-in. -- **Lanes per row of the port's expert kernels** (`STRATA_GU_LANES` / `STRATA_DOWN_LANES`, 4/8/16/32 at run time): - gate/up 8 or 16 x down 4 or 8 all decode in 37.7-38.6 ms per verify round (2 runs each, Coder, both prompts); tok/s - differences between them are draft acceptance (each split rounds differently). The default (8 / 8) stays. -- **Speed after the merge:** Coder 78.2 / 75.7 tok/s (as before), IQ2_XS 58.6 / 64.2 (from 50.8 / 60.6). -- **Still failing:** (`iq_multi_parity` on IQ2_XS: fixed later, see "Two model-specific bugs"), `s2_expert_grouped_parity` (the s2 path), `kv_hybrid_parity`'s last step - (needs the unported tensor-core prompt kernel). `ple_parity` and `native_expert_parity` need model files. - -**The 0.1.32 merge (2026-10-01).** Upstream 0.1.31 -> 0.1.32 (89 commits) by the same steps; the hash-only -differences are now canonicalized before `git merge-file` (the port's kernel-name hashes are kept), which left 10 -files with real conflicts. What it needed: - -- **Upstream's async commit** (`set_commit_async` / `wait_commit`) maps onto the port's own deferred commit - (`commit(n, err, false)` + `commit_finish`); `wait_commit` is `commit_finish`. -- **Upstream's new hyper-connection read variants** (split / staged, chosen per card by a bit-for-bit self-test at - start) are not used by the port, which keeps its sliced down / split norm read; the self-test segfaulted on the - B70, so on SYCL it runs only with `STRATA_HC_CHECK=1` (open). The port's split-norm kernel was renamed - (`gr_norm_split_port_kernel`): upstream now has one of the same name. -- Two kernel names collided after hash canonicalization (renamed), and fused_gr's per-block shared-memory query is - a fixup now. -- **Result:** every output identical to 0.1.31 (Coder 19 / 2,184-token prompts and IQ2_XS, 256 greedy tokens), - same speeds (Coder 77.7-78.1 / 75.7 tok/s, IQ2_XS 58.5), 40K prompt 1,201 tok/s then 65.1 tok/s decode. - `kv_hybrid_parity` and `qsa_prompt_attn_parity` pass (the tests now turn on the XMX prompt attention they - check); `iq_multi_parity` (IQ2_XS) and `s2_expert_grouped_parity` as before. - -**The 0.1.33 merge (2026-10-01).** Upstream 0.1.32 -> 0.1.33 (17 commits): no shared file conflicted (the Intel -code is all in `sycl/`); six engine files changed upstream and were merged by the same re-migration (only those six: -the other files dpct produced differently were left alone). Two conflicts: `--resident-cpu-experts` (upstream's new -`resident_cpu_explicit` beside the port's `--stream-experts`) and the prompt attention's compute-capability check -(the port keeps its XMX dispatch). `sycl/setup_intel.py` and `sycl/serve/server_intel.py` ran unchanged against the -new setup.py and server.py. Every output identical (Coder, IQ2_XS, the 40K prompt with borrowing); `gr_parity` and -`qsa_prompt_attn_parity` pass. - -**Prompt-slot borrowing: the hang (fixed 2026-10-01).** From 0.1.31 on, a prompt that borrowed cache slots stopped -in its first full chunk with the GPU at 100% and the host waiting on the compute queue. It was never a borrowing -bug: borrowing is just the only way the Coder streams experts during a prompt (every expert is resident otherwise), -and the streamed path's stager deadlocked. Its threads read experts into a ring of 16 page-locked buffers and, to -reuse one, waited on the copy queue's event for the DMA that last read it (the CUDA form, `cudaEventSynchronize`). -Under the Level Zero v2 adapter that event, once a host thread had waited on it, no longer released the other -queue's barrier that also listed it, and the GPU waited forever - only once a layer streamed more than 16 experts -(the ring wrapped). The v1 adapter (`SYCL_UR_USE_LEVEL_ZERO_V2=0`) and a 256-buffer ring both ran; so did a sync -after every phase (`STRATA_PREFILL_SYNC=1`, kept as a debug switch). The fix: the copy queue writes a sequence -number into page-locked host memory after each DMA and the stager polls it - no host thread waits on a queue's -event. Output is bit-identical to the run without borrowing; borrowing is the default above 32K again. - -**Two model-specific bugs found with Swift 1.5 (fixed 2026-10-01).** UkisAI's Swift 1.5 IQ2_XS (setup's `swift` -family, the same GSQ-RCO layout) decoded token 0 forever: NaN logits from layer 13 on. - -- **`--stream-experts` read the wrong shard.** Since upstream 0.1.31, `ExpertLayout::gguf_file` is per layer AND role - (`3 * layer + role`; a shard boundary can fall inside a layer). The port's `GgufExpertSource` still indexed it by - layer, so a layer in shard 2 was read from shard 1 at shard 2's offsets: garbage IQ1_M scales, infinities after the - fp16 dequantization. The original model keeps every expert in shard 1 and never noticed; Swift's layers 13-47 are - in shard 2. `STRATA_DBG_NAN=1` now also reports the experts' fp16 GEMM inputs (activations, dequantized gate/up). -- **IQ2_XS products were garbage** (the long-failing `iq_multi_parity` IQ2_XS case): ggml's `vec_dot_iq2_xs` reads - its four 16-bit codes through a `uint16_t*` to an `int2`, type punning the SYCL device compiler does not honour; - the dequantizer, which reads the codes directly, was exact. The codes are extracted by shifts now (the dot and the - expert kernels' `Split<17>`); the test checks against a CPU decode from ggml's tables too and passes. No model here - uses IQ2_XS tensors, Swift included - this was not Swift's bug, only found on the way. - -Swift 1.5 IQ2_XS after both: correct answers, 51-118 words of thinking on short questions, 48 tok/s raw decode. - -**Load time (2026-10-02).** Starting a model is mostly the expert cache's fill from the GGUF (`--stream-experts`): -the Coder's 23.4 GiB of experts took 64-76 s, the IQ2_XS's 24.9 GiB 91 s, because the fill read each expert's -three slices, copied the blob and waited for the copy before the next read. It is a pipeline now: the slots are -admitted in profile order first (the same placement), the reads go in file order by up to 8 threads into -page-locked batches of 64, and a batch's copies run while the next one is read. Cold page cache: 76.2 s -> 18.8 s -(0.33 -> 1.34 GB/s); a whole Coder start from the engine's launch to its first token 82 s -> 26 s, the IQ2_XS -120 s -> 41 s (its 8.2 GB host mirror is ~9 s of the rest). Output identical. `STRATA_FILL_SERIAL=1` is the old fill. - -**The stream-all prompt walk is back (2026-10-03).** It copies every expert the VRAM cache does not hold, layer by -layer ahead of the compute, instead of only the routed ones. It hung on the B70 until the stager and PLE fixes of -2026-10-02 and runs since. It now runs by default when the VRAM holds more than 90% of the (layer, expert) pairs; -past that it would copy several times the routed experts (the IQ2_XS keeps a quarter of them in the RAM mirror: a -2,184-token prompt fell from ~560 to 254 tok/s with it). `STRATA_PREFILL_STREAM_ALL=1` / `=0` force it on or off. -Coder, outputs identical either way: - -| prompt | routed-only | stream-all | -|---|---|---| -| 40K int8 | 1,106 tok/s | 1,160 | -| 128K int8, KV streaming | 897 | 920 | -| 128K k8v4 | 1,006 | 1,041 | -| 256K int8, KV streaming | 760 | 780 | - -Measured alongside: the 2,184-token prompt reads 784-799 tok/s on both the 0.1.35+PR and the 0.1.38 build (three -runs each), so the 825 seen once on 0.1.35 was session variance, not a regression. The prompt path at 40K (Coder, -GPU timeline): expert dequant 15.7%, expert GEMMs 23.4%, host grouping 6.5%, gather 5.7%, combine 2.4%, prompt -attention 19.2%, QSA select 6.8%, DeltaNet recurrence 5.8%, hyper-connection reads 4.6%. Upstream's sm_90 cluster -kernels (greedy sampler, QSA top-k) would save next to nothing here: the sampler reads ~6 MB a round against a 13 ms -token. Q4_0 tensor-core prompt attention (#452) is not worth porting: the port's XMX prompt attention is 2-3x slower -than the plain kernel it uses for every KV format. - -**The 0.1.38 merge (2026-10-03).** Upstream 0.1.35 -> 0.1.38 (83 commits) merged the seven PRs the port had taken -early, so their header forks in `sycl/include` are gone; the merge base for the 3-way merge was the port's own "main + -PRs" migration. New in the port with it: the DeltaNet output norm without its dead FP32 store, two prompt-path -fixes (the fused layout's buffers, the streamed walk's resident lookup), `STRATA_GR_DOWN_MAX4=1` (opt-in; neutral on -the B70: 2,184-token prompt 793 vs 788 tok/s, decode 76.2 vs 76.3). Stubbed in the port: `--peer-device` (a second -GPU as an expert-cache tier: upstream's code is CUDA calls; `open` refuses with a message) and the fused int8 prompt -kernels (#136, part of the MMQ library the port does not build). Off on SYCL: the sm_90 thread-block-cluster greedy -sampler and QSA top-k (no SYCL counterpart; the callers take the plain kernels). `STRATA_GDN_KEYHEAD=1` stays opt-in -(still 8% slower here: 727 vs 788 tok/s). `qsa_prompt_attn_parity` now also tests upstream's Q4_0 tensor-core mode -(#452), which the port does not have: those cases report "refused" (the int8 and fp16 XMX cases pass); a `--kv -q4_0` prompt takes the older kernel, as before. Outputs identical (Coder, IQ2_XS, 40K); 128K int8 KV streaming -892 / 66.8 tok/s. - -**0.1.35 and seven open upstream PRs (2026-10-02).** Upstream main 0.1.33 -> 0.1.35 merged as before, then seven open -PRs ported into `sycl/` ahead of upstream (one dpct run of main + all of them, 3-way merged into only the files they -touch; `sycl/tools/merge_upstream.py`). Their header changes live in `sycl/include` until upstream merges them. - -| PR | what | on the B70 | -|---|---|---| -| #385 (sergqwer) | stager: a buffer's first job of a generation waits for the previous DMA from it | race fix; same output | -| #463 (constantindjonkam) | decode waits for an adaptive swap before reading the residency table | determinism fix | -| #453 (architectds) | the drafter's batched K/V on the KV-streaming ring | 128K int8: 888 -> 895 tok/s prompt, 66.6 -> 67.0 decode | -| #374 (sergqwer) | the first chunk's PLE rows read beside layer 0 | part of the 2,184-token prompt's 784 -> 825 tok/s | -| #363 (BlueKingMuch) | the PCIe expert call: group stride, fused SwiGLU + q8_1 | IQ2_XS decode +1.7%; re-done on the port's lane kernels | -| #407 (sergqwer) | `--adapt-tuned` (opt-in) | neutral here; stays opt-in | -| #413 (BlueKingMuch) | DeltaNet recurrence per key head | bit-identical but 8% slower prompt here: off (`STRATA_GDN_KEYHEAD=1`) | - -#374 needed two port fixes: its next-chunk read assumed every chunk is the full chunk length (the port's first chunk -is 256 tokens: the second chunk's rows were read from the wrong place), and its host wait on the PLE upload's event, -now inside the layer loop, deadlocked the prompt past ~4K tokens under the Level Zero v2 adapter (the stager's bug -again): the upload is marked by a polled sequence number now. #413's gate is an NVIDIA SM-count rule and its parity -test an SM-holding NVIDIA bench (not built). Outputs identical to 0.1.33 (Coder 19 / 2,184 tokens, IQ2_XS, 40K). - -**Not ported yet (2026-10-01).** - -- Three kernels carry inline PTX (`mma.sync` tensor-core matrix ops, `ldmatrix`, `cp.async`): - `qsa_prompt_attn`, `qsa_select`'s block scores, `native_qsa_score`. The SYCL build takes the "older card" - fallback the CUDA build uses below sm_80. `qsa_prompt_attn` also has an XMX version (`joint_matrix`, opt-in - `STRATA_PROMPT_ATTN_XMX=1`): correct, but slower than the fallback (see "XMX prompt attention v2"). -- The ggml MMQ prefill path (`moe_mmq.cu`) needs llama.cpp's ggml-cuda sources; not built. -- AOT device code is what runs: `AOT=bmg-g31 BUILD_DIR=.../build-sycl-aot` (the JIT build costs ~47 s of - compiling on the first window). - -## Arc A750 and the other Alchemist cards (`i915`, 2026-10-07) - -Measured on an Arc A750 (8 GB, `i915`, PCIe 4.0) with the Flash-Next IQ3_XXS in a PC with 64 GB of RAM, a Ryzen 5 5600X -(AVX2, no AVX-512). `./setup.sh --backend sycl` writes a different config for an `i915` card (and a 300 MiB VRAM reserve and -`--draft-vocab en` for any card under 12 GB): +An Alchemist card (A750, A770 and so on) has no FP64 hardware, and the A750 runs on the `i915` driver. `./setup.sh --backend sycl` +writes a different config for an `i915` card, and a 300 MiB VRAM reserve and `--draft-vocab en` for any card under 12 GB. | | Arc Pro B70 (`xe`, 32 GB) | Arc A750 (`i915`, 8 GB) | -|---|---|---| -| experts | `--stream-experts`: from the GGUF into VRAM, the rest in a pinned RAM mirror | no `--stream-experts`: all of them in a RAM arena (39.97 GiB for IQ3_XXS), 406 of 24,576 in VRAM, the CPU computes the others | +| --- | --- | --- | +| Experts | `--stream-experts`: from the GGUF into VRAM, the rest in a pinned RAM mirror | No `--stream-experts`: all of them in a RAM arena (about 40 GiB for IQ3_XXS), a few hundred in VRAM, the CPU computes the others | | `STRATA_VERIFY_NO_HOST` | 1 (the wrapper's default) | 0, written to the config's `"env"` | | `--vram-reserve-mib` | 1024 / 2048 | 300 | -| `--ple-io` | `ram` (rotational-disk rule of setup) | `ram` | -| FP64 | native | emulated: `IGC_EnableDPEmulation=1`, `OverrideDefaultFP64Settings=1`, `NEOReadDebugKeys=1` in `"env"` | -| build | AOT `bmg-g31` | JIT (no `AOT=`) | +| `--ple-io` | `ram` | `ram` | +| FP64 | Native | Emulated: `IGC_EnableDPEmulation=1`, `OverrideDefaultFP64Settings=1`, `NEOReadDebugKeys=1` in `"env"` | +| Build | AOT `bmg-g31` | JIT (no `AOT=`) | -- **Why no `--stream-experts`.** The mirror is one pinned host allocation for everything the card does not hold (40 GB here). A - single `sycl::malloc_host` above a few GB fails on the A750 (`STRATA_MIRROR_MIB` 3,000 worked, 8,192 and 24,000 did not, also - with `memlock` unlimited and the relaxed-allocation-limit variables), and the engine then refuses to start with - `24170 experts are neither in VRAM nor mirrored`. Without the flag the engine loads the GGUF's experts into an ordinary RAM - arena and the CPU computes the experts that are not in the 0.67 GiB cache. That needs the RAM: about 43 GB for IQ3_XXS plus - the 29 GB n-gram table with `--ple-io ram` (the table is not locked in the container, so it is reclaimable; do not give the - container `--ulimit memlock=-1`, the locked table and the arena then do not fit in 64 GB and the PC swaps). Setup warns when the RAM is less than the model's. -- **Why `STRATA_VERIFY_NO_HOST=0`.** With it set the GPU plans each layer itself and the CPU's experts are never asked for: - the window hangs (the i915 log says `Fence expiration time out`) or, if the pool is refused, the engine says - `REFUSED: ... neither in VRAM nor mirrored`. It is the switch of the B70's case, where every expert the card lacks is in the mirror. - The engine takes a value of `0` (or empty) as "off", and `strata-sycl.sh` does not pass such a value on. -- **FP64.** An Alchemist has no FP64 hardware. One kernel on the sampled path declares `double`, and the SYCL runtime refuses it at - its first launch (`'double' is not supported in 'Intel(R) Arc(TM) A750 Graphics' device`, the engine dies on the first request - that has a temperature). With the three variables above the driver emulates it; the cost is the sampler's tail - (see "FP64." in the B70 section). -- **The first request after a start used to return `!!!!!` (token 0) or crash the engine.** The GPU waits for the CPU's experts at - every layer in a spin that is bounded (`strata::kSpinMax`, `sycl/include/strata/sycl_doorbell.hpp`): a spin that never ends - hangs the driver. The bound was 20,000 reads, a few tens of milliseconds, which is enough when nothing is waited for (the B70) - and is not for a CPU layer in the first request (cold pages, 13 s for a 26-token prompt). The GPU gave up, went on with the - experts' outputs missing, the next layers' routing was garbage (one expert chosen ten times for a token: `nt=10`) and the - CPU pool wrote past its token arrays and died with a segmentation fault (exit code 139); when it did not die the logits were NaN - and the sampler's answer was token 0. Later requests ran with warm pages and were right. The bound is now a build option, - `STRATA_SYCL_SPIN_MAX` (CMake; `SPIN_MAX=` for `sycl/tools/build.sh`): 2,000,000 reads (a few seconds) unless the build is an - AOT build for a `bmg` card, which keeps 20,000. Before the change 11 of 16 runs of the engine (three requests each, a 26-token prompt) died or gave - token 0 in the first request; with 2,000,000 reads 4 of 4 were right in all three requests. -- **Kernels misread memory more than 4 GiB into an allocation (fixed in the build).** On an Arc A770 (16 GB, `xe`, Coder - IQ1_M, `--stream-experts`, a 4.46 GiB expert cache) every prompt answered `!!!!` (token 0): the experts in the cache's - last 0.46 GiB dequantized to NaN in the prompt path. A copy of those slots read back equal to the GGUF - (`STRATA_VERIFY_ALL_SLOTS` passed) and a fresh device copy of the same bytes dequantized clean; only a kernel reading - the slot in place got wrong bytes. In a JIT build IGC addresses a kernel's pointer arguments as a 32-bit offset from - the allocation's base. The JIT build now passes `-ze-opt-greater-than-4GB-buffer-required` (`sycl/CMakeLists.txt`), - which keeps the kernels on 64-bit addresses. After the change, prompts of 73 to 542 tokens had 0 non-finite values - (`STRATA_DBG_NAN=1`) and the 542-token prompt answered right. The speed cost is not measured. The AOT (`bmg-*`) build is - unchanged. oneMKL's own precompiled kernels do not get the option; the GEMM stages an operand past 4 GiB for that. -- **Speed.** About 10 to 15 tok/s decode (76 to 92% of the drafts accepted), a 26-token first prompt in 13 s and later short prompts in 0.2 to - 1.3 s; the A750's PCIe link probes at 10.6 GB/s. -- **A750 and the xe error counters.** The engine segfaults in a worker thread of the CPU pool when it exits (dmesg only, the server - has already printed "stopped"); it is not a GPU event. -- **The prompt's GDN recurrence runs a sub-group kernel on Alchemist** (`sycl/include/strata/prefill/gdn_xe.hpp`, - variant 6 of `gdn_recurrence_variant`). It keeps the column kernel's four row groups and their FMA chains in the same - order, but a 16-lane sub-group holds four columns x four row groups, so the two cross-row-group sums are sub-group - shuffles instead of SLM round trips and a token needs one work-group barrier instead of five (256 GRFs a thread). - Measured on an Arc A770 (16 GB, `xe`, oneAPI 2026.1 JIT), 2026-10-13: `gdn_parity --prefill-tokens 8192` 51.0 -> - 24.5 ms, 32,768 tokens 204-236 -> 110-113 ms (48 heads), 0 mismatches; end to end, Coder IQ1_M, `--max-context - 131072 --kv int8 --kv-resident 20480 --vram-reserve-mib 1200 --resident-experts`, prompt 386 -> 396-407 tok/s at - 8K and 478-480 -> 511-518 tok/s at 32K, the same output tokens. On by default on Alchemist only, the column kernels everywhere else (not measured there). `STRATA_GDN_XE=0` - turns it off, `STRATA_GDN_XE=1` turns it on for any Intel GPU with 16-lane sub-groups. To check another Intel card: - `STRATA_GDN_XE=1 ./gdn_parity --selftest --prefill-only` (or `ctest -R prefill_gdn_parity`), and `--prefill-tokens - 8192 --prefill-repeat 5` for the time of both kernels. Not taken: the same kernel with its state in SLM (no register - spills, the same bits, ~4x slower: 13.7 vs 3.1-3.3 ms at T=1000). +- **Why no `--stream-experts` on the A750.** The mirror is one pinned host allocation, and a single `sycl::malloc_host` above a few + GB fails on the A750. Without the flag the engine loads the experts into an ordinary RAM arena and the CPU computes the experts + that are not in the 0.67 GiB cache. That needs about 43 GB of RAM for IQ3_XXS plus the 29 GB n-gram table with `--ple-io ram`. + Setup warns when the RAM is less than the model's. +- **Why `STRATA_VERIFY_NO_HOST=0`.** With it set the GPU plans each layer itself and the CPU's experts are never asked for; the + window hangs (`Fence expiration time out` in the i915 log). +- **FP64.** One kernel on the sampled path declared `double`; the sampler's tail now accumulates in float + (`-DSTRATA_SYCL_SAMPLER_FP64=1` keeps double). Setup still writes the three emulation keys for an A-series card, because other + kernels that use double were not exercised without them on long contexts or with vision. On the A750 a sampled decode ran at + 12.0 tok/s without the emulation keys against 9.8 with them (median of 4 interleaved pairs). +- **Prompt GDN recurrence** runs a sub-group kernel on Alchemist (variant 6, `STRATA_GDN_XE`): on by default there, `=0` off, `=1` + on any Intel GPU with 16-lane sub-groups. On Battlemage it stays off (-3.5%). +- **Grouped XMX expert launch** (`STRATA_PF_XMX=2`) is the default on Alchemist: the prompt path decodes and multiplies a group of + experts in one fused launch on the XMX units. `=0` turns it off; `=1` dequantizes first and runs one grouped GEMM per product. + Battlemage keeps the older path (the same kernels are 7-11% slower there). +- **Streaming copies.** Only experts in USM host memory (the `--stream-experts` mirror) are copied directly; every other expert + goes through the stager's pinned buffers, because SYCL has no `cudaHostRegister` and a direct copy out of pageable memory could + hang a prompt. `STRATA_PF_DMA_PAGEABLE=1` restores the direct copy. +- The engine segfaults in a worker thread of the CPU pool when it exits (dmesg only, after the server printed "stopped"). It is + not a GPU event. -### Prompt speed on Alchemist: the grouped XMX expert launch is the default (engine 0.1.45.3, `STRATA_PF_XMX`) +> **Measured on:** Arc A750 8 GB (i915, PCIe 4.0), Ryzen 5 5600X (AVX2, no AVX-512), 64 GB RAM, Flash-Next IQ3_XXS, default +> flags, engine 0.1.45.3. -The A750's prompt path decodes and multiplies a group of experts in one fused launch on the XMX units (`STRATA_PF_XMX=2`, `sycl/src/prefill/xmx_moe.dp.cpp`) -instead of dequantizing each expert to FP16 and calling oneMKL. It is the default on Alchemist cards; `STRATA_PF_XMX=0` turns it off and `=1` is the -variant that dequantizes first and runs one grouped XMX GEMM per product. Battlemage keeps the old path (the same kernels are 7-11% slower there). -Measured on the A750 (IQ3_XXS, default flags, interleaved rounds against the run before): a 1K-token prompt 35.1 -> 46.4 tok/s, 2K 61.7 -> 83.1, 4K 79.3 -> 110.4 -(+39%, 5 of 5 rounds), 6K 66.3 -> 114.8; decode unchanged (18.7 tok/s), the same number of cache slots. It rounds differently from the FP16 path: -teacher-forced KL against the default over 6,006 forced tokens 0.0002 (the default against itself: 0.0001; with the default PCIe share the A750's own run-to-run noise -is 0.012), first-token KL 0.0017 (floor 0.0005), top-1 agreement 99.7%. HumanEval (every 2nd, 82) 73, 69 and 74 for the mode and two default runs, reasoning 21 / 23 / 22 of -30, a 35-question Chinese/Japanese/Korean set 35/35 in all, needles at 2K-6K 9/9, an 8K-context tool-call agent 6/6. +| Workload | Result | +| --- | --- | +| Prompt, 1K / 2K / 4K / 6K tokens, grouped XMX launch | 46.4 / 83.1 / 110.4 / 114.8 tok/s (before: 35.1 / 61.7 / 79.3 / 66.3) | +| Decode | 18.7 tok/s (unchanged by the grouped launch); 10-15 tok/s in the first report | +| PCIe probe | 10.6 GB/s | +| Quality of the grouped launch against the default | teacher-forced KL 0.0002 over 6,006 tokens (default against itself 0.0001), first-token KL 0.0017, top-1 99.7%; HumanEval (every 2nd, 82) 73, 69, 74; needles at 2K-6K 9/9; 8K-context tool-call agent 6/6 | -### A750: a prompt that never finished is fixed (engine 0.1.45.3, commit `fca3c5958`) +**Gyro on an Arc.** Gyro-S and Gyro-M (trellis-coded experts, [GYRO.md](GYRO.md#intel-arc-sycl)) run on the SYCL engine. They were +tested on the A750 only. Setup writes the usual A750 config (32K context, 300 MiB reserve) without `--ple-io ram`, because the +28 GB n-gram table stays on the SSD while the expert arena takes 24 GB of RAM (Gyro-S). Gyro-M needs `--max-context 8192 +--vram-reserve-mib 100 --prefill 1024` on 8 GB. A second Arc card does not take Gyro's experts (the engine refuses and says so). +The Battlemage AOT build compiles with the trellis kernels and was not run. -On an Arc A750 some prompts never finished: with default flags a 2,744-token documentation excerpt hung in 6 of 6 runs (the -thread spun in `sched_yield` inside `libze_intel_gpu` with every engine idle), and the same hang was seen on a 5.3K-token needle prompt. The -cause: SYCL has no `cudaHostRegister`, so the expert arena is ordinary pageable memory even where the engine calls it -pinned. Copying a 2.3 MB expert straight out of it went through the Level Zero driver's staging path on the calling thread, -and on some prompts that call never returned. Now only experts in USM host memory (the `--stream-experts` pinned mirror) -are copied directly (the answer is remembered per 2 MiB); every other expert goes through the stager's pinned buffers, like -an unpinned one. **`STRATA_PF_DMA_PAGEABLE=1` restores the old direct copy.** +> **Measured on:** Arc A750 (i915, 8 GB), Ryzen 5 5600X, 62 GB RAM, Gyro-S, engine version not recorded. -Checked: the 2,744-token repro finished 3 of 3 times after the fix (0 of 6 before); the first-token logits and the tokens are -identical old against new on the A750 (1.4K, 2.5K and 4K prompts) and on the Arc Pro B70 (stock config, and without -`--stream-experts`); speed is unchanged where it did not hang (A750 4K prompt 79.4 against 79.3 tok/s, B70 stock 1,246 -against 1,240), and on the B70 without `--stream-experts` the prompt is faster (680-740 against 77-456 tok/s). The -regression test is `sycl/tools/test_prompt_deadlock.sh` (the 2,744-token prompt; it times out on the old build and passes on -the fix). - -## Not done - -- Images on the llama.cpp config: not wired. On the SYCL port they are wired, see "Pictures on an Arc" below. Open there: a - nobody has run the SYCL picture reader on a card of 24 GB or more. Pictures are on for a Battlemage card too since 2026-10-22 (see "Default" below), - with the CPU reader. -- Speculative decoding on the llama.cpp path (the GGUF carries no draft layer llama.cpp can use). The SYCL - port has it (MTP draft layer, `--mtp`). - -## Pictures on an Arc (2026-10-19, branch f-021-arcvis) - -Pictures follow the rule of every other card ([MODELS.md](MODELS.md#the-versions)): a picture reader on the CPU under 24 GB of VRAM, on -the card from 24 GB. `strata_control/vision_reader.py` is the one place for it. On an Arc: - -- **The CPU reader** is `strata-vision` (llama.cpp's mtmd). Setup compiles it into `engine-sycl/`, the server runs it on the host, and the - engine in the container reads the picture's embeddings from the host's `/dev/shm`, which `strata-sycl.sh` shares - (`STRATA_SYCL_SHM=0` keeps a private one). On the A750 a 448x448 picture takes 1.73 s. -- **The SYCL reader** is `strata-vision-sycl`, the same program on llama.cpp's SYCL backend (`tools/vision`, `STRATA_VISION_SYCL=ON`). - `setup --backend sycl --vision gpu` compiles it on the host when the oneAPI compiler (`icx`, `icpx`, `intel-oneapi-mkl-devel`) is - installed; without it setup says so and keeps the CPU reader. It needs the PC's oneAPI runtime, not the container. If it does not - start, the server falls back to the CPU reader with one log line. On the A750: 0.16 s per picture instead of 1.73 s (after a 30 s - first start that compiles the kernels), and the embeddings match the CPU reader's (cosine 0.9959 to 0.9999 over 8 pictures). It takes - VRAM the experts would use, which is why the rule gives it to 24 GB cards only. -- **Default.** On for every Arc card with the CPU reader, a B-series (Battlemage) card included. An Arc of 24 GB or more also keeps the CPU reader by - default: the SYCL reader has not run on such a card, so `--vision gpu` asks for it and `vision_reader.INTEL_GPU_READER_BY_DEFAULT` is the switch to - flip once one has. **B-series cards are supported and untested by us** (the owner decision of 2026-10-22: users will report failures). One known - risk: the first picture on the Arc Pro B70 (engine 0.1.45, IQ3_S, 2026-10-19, CPU reader) logged an xe engine reset (ccs + bcs) and - `Fault response: Unsuccessful -EINVAL`, and the request ended 503. The CPU reader was in use, so the fault is in the engine on xe, not in a GPU - reader; it has not been looked at since. It is not a refusal: setup and `strata launch` print one plain line that the card is new to Strata and - how to report a problem, and `--vision no` (setup) or `--vision off` (`strata launch`) turns pictures off. - -Tested on the A750 (IQ3_XXS, 8K context, engine from f-021-int, JIT build, 2026-10-19): a picture with a red circle, a blue square and the -words "STRATA 42", 8 variants, answered correctly by the CPU reader 10 of 10 (a first set with smaller text read "42" as "412"), and -by the SYCL reader 10 of 10; two pictures in one request, a picture kept in the history of a second turn, and a long text prompt after the -pictures all answered; a text request after every picture answered; 47 picture requests, outside and inside the container (inside, one -answer was cut off by the 200-token limit of the test), no GPU fault in the kernel log, no 503. - -## Gyro on an Arc (2026-10-20, branch f-022-arcgyro) - -Gyro-S and Gyro-M (trellis-coded experts, [GYRO.md](GYRO.md#intel-arc-sycl)) run on the SYCL engine. Tested on the Arc A750 only (i915, 8 GB, -Ryzen 5 5600X, 62 GB of RAM). Setup writes the usual A750 config for them (32K context, 300 MiB VRAM reserve), without `--ple-io ram`: the -28 GB n-gram table stays on the SSD, because the expert arena already takes 24 GB (Gyro-S) of the RAM. - -| A750, Gyro-S | | +| Workload | Result | | --- | ---: | -| prompt, 1,406 tokens | 82 tok/s (40 with dequantize + oneMKL; the grouped XMX launch now takes the trellis types, 5/5 pairs) | -| prompt, 2,377 / 20,000 tokens | 102 / 195 tok/s | -| decode, MTP draft of 4 | 15-16 tok/s (13.7 with the old link share, 5/5 pairs) | -| experts in the card's cache | 410 of 24,576 | +| Prompt, 1,406 tokens | 82 tok/s (40 with dequantize + oneMKL) | +| Prompt, 2,377 / 20,000 tokens | 102 / 195 tok/s | +| Decode, MTP draft of 4 | 15-16 tok/s | +| Experts in the card's cache | 410 of 24,576 | +| Gyro-M, 8K context | decode 14 tok/s, prompt 62 tok/s | -Decode is bound by the CPU pool (5 workers, 8 GB/s) and the PCIe link (10.4 GB/s measured), not by the card's kernels (20 microseconds an -expert). For trellis packs the engine takes the link share against a 12 GB/s reference instead of 20, which gives 0.48 on this card instead -of 0.29. A smaller prompt chunk makes room for more cached experts and slows the prompt: `--prefill 1024` +13% decode, -24% prompt speed; -the default stays `--prefill auto`. Gyro-M needs `--max-context 8192 --vram-reserve-mib 100 --prefill 1024` on 8 GB (14 tok/s decode, -62 tok/s prompt). A second Arc card does not take Gyro's experts (the engine refuses and says so). The Battlemage AOT build (`bmg-g31`) -compiles with the trellis kernels; it was not run, there is no B-series card here. +Decode is bound by the CPU pool and the PCIe link (10.4 GB/s), not by the card's kernels. `--prefill 1024` gave +13% decode and +-24% prompt speed; the default stays `--prefill auto`. -## Arc Pro B70 with a model that does not fit: Flash-Next IQ3_S (2026-10-07, branch f-141-intel) +## Pictures on an Arc -The earlier B70 rows are for models whose experts all fit in VRAM. IQ3_S has 24,576 experts of 1.97 MB (46 GB): the 32 GB -card holds 11.5k of them (22 GiB of cache) and the rest sit in a pinned host mirror that the GPU reads over PCIe. Tested on -the B70 VM (xe driver, kernel 7.0, oneAPI 2026.1, PCIe measured at 13.3 GB/s host to device), engine 0.1.40-sycl. + +Pictures follow the rule of every other card ([MODELS.md](MODELS.md#the-versions)): the picture reader runs on the CPU under 24 GB +of VRAM and on the card from 24 GB. `strata_control/vision_reader.py` is the one place for that rule. On an Arc, pictures are on by +default with the CPU reader, a B-series card included. An Arc of 24 GB or more also keeps the CPU reader; `--vision gpu` asks for +the GPU reader. -**Two things made it produce garbage, both fixed or documented here.** +- **CPU reader:** `strata-vision` (llama.cpp's mtmd). Setup compiles it into `engine-sycl/`, the server runs it on the host, and + the engine in the container reads the embeddings from the host's `/dev/shm`, which `strata-sycl.sh` shares + (`STRATA_SYCL_SHM=0` keeps a private one). +- **SYCL reader:** `strata-vision-sycl`, the same program on llama.cpp's SYCL backend. `setup --backend sycl --vision gpu` + compiles it on the host when the oneAPI compiler (`icx`, `icpx`, `intel-oneapi-mkl-devel`) is installed; otherwise setup says so + and keeps the CPU reader. It needs the PC's oneAPI runtime, not the container. If it does not start, the server falls back to + the CPU reader with one log line. It takes VRAM the experts would use, which is why the rule gives it to 24 GB cards only. Nobody + has run it on a card of 24 GB or more. +- **Known risk on Battlemage.** The first picture on the Arc Pro B70 (engine 0.1.45, IQ3_S, CPU reader) logged an xe engine + reset (ccs + bcs) and `Fault response: Unsuccessful -EINVAL`, and the request ended 503. It has not been looked at since. + Setup and `strata launch` print one plain line that the card is new to Strata and how to report a problem. `--vision no` + (setup) or `--vision off` (`strata launch`) turns pictures off. -- **Aliased pages in big allocations (fixed in the engine).** The xe driver returned the 22 GiB expert-cache arena with two - of its 2 MiB pages mapped onto the same memory: a write at +2 MiB showed at +1022 MiB. Cache slots there held other - experts' bytes, and the prompt came out NaN at the first layer that routed to one (layer 4, expert 249); the same run - with another `--max-context`, or with an MTP window of 4 instead of 6, was clean. `STRATA_VERIFY_ALL_SLOTS=1` - (`STRATA_VERIFY_FIND=1`, `STRATA_VERIFY_DUMP=dir`) reads every filled slot back and compares it with the GGUF; that is how - the aliasing was found (the wrong bytes were slot 0's blob). Every allocation of 32 MiB or more now goes through - `strata::malloc_device_guarded`: it tags every 64 KiB, reads the tags back in a second kernel, and allocates again behind a - growing spacer until the range is clean (a warning names each retry; the check costs under 0.5 s at startup; - `STRATA_ARENA_ALIAS_CHECK=0` skips it). `sycl/probe/arena_rw.cpp` and `alias_scan.cpp` do not reproduce it on their own: it - needs the engine's allocation history. -- **`STRATA_VERIFY_NO_HOST=1` is required whenever part of the experts is not in VRAM.** `sycl/serve/strata-sycl.sh` sets - it; a by-hand run must too. Without it the per-layer host/GPU handshake is not visible across the bus on xe and the logits - turn NaN a few tokens into the answer, differently on every run. With it (and the mirror covering every miss) the run is - deterministic: two runs gave identical tokens. +> **Measured on:** Arc A750 (8 GB), IQ3_XXS, 8K context, JIT build, 448x448 picture. -**Numbers** (greedy, `--spec 4 --spec-min-p 0.5 --mtp`, INT8 KV, 32K context, `--stream-experts --vram-reserve-mib 1024 ---ple-io ram`, 11.5k experts resident; the 10-prompt `mg_norepeat` gate passes 10 of 10, the xe error counters did not move): +| Reader | Time per picture | +| --- | ---: | +| CPU (`strata-vision`) | 1.73 s | +| SYCL (`strata-vision-sycl`), after a 30 s first start that compiles the kernels | 0.16 s | -| | before | after | -|---|---|---| -| 4,095-token prompt, `--prefill auto` (2,048-token chunks) | 618 tok/s | 730 | -| 4,095-token prompt, `--prefill 4096` | 784 | **980-1,002** | -| 8,169-token prompt: auto / 4096 / 8192 lending cache slots / 8192 own buffers | 676 | 953 / 1,037 / 1,117 (9.4k slots) | -| decode, 200-token story (66% of drafts accepted, 1.98 tokens per round) | 31.4 tok/s | 30.5 with 4096-token chunks (700 fewer cache slots) | -| decode, 200-token code answer (90% accepted, 3.65 tokens per round) | not measured | 40.8 with 4096-token chunks | +The embeddings match the CPU reader's (cosine 0.9959 to 0.9999 over 8 pictures). A test picture (red circle, blue square, the words +"STRATA 42") was answered correctly by both readers 10 of 10; two pictures in one request, a picture kept in a second turn's history, +and text requests after the pictures all answered, with no GPU fault in the kernel log. -(Interleaved A/B pairs, medians of 3-5. "Before" is the same build with the old short first chunk; the decode figures are -the old default and `--prefill 4096`.) +## The SYCL port (`sycl/`) -**Why the prompt is slow here, and the two changes.** Every prompt chunk streams the experts the cache does not hold over -PCIe, and a 4K prompt touches nearly all 512 experts of every layer: 13.5k missing experts x 1.97 MB = 26 GB, 2 s of the -bus per chunk. So the prompt reads faster in fewer, bigger chunks, and the short first chunk (`STRATA_PREFILL_FIRST`, 256 -tokens, there to start the GPU while the PLE rows are read) cost a whole extra pass of that stream. It is now 256 only when -the cache holds every expert, else 0. `setup_intel.py` writes `--prefill 4096` for a card with 24 GB or more (the chunk's -buffers take ~700 cache slots; `--prefill 8192 --prefill-borrow` lends cache slots instead and keeps all of them for decode, -at ~1 s of refill per prompt). Where the 4K prompt's GPU time goes now (GPU timeline, timing on): expert dequant 20%, the two -expert GEMMs 34%, QSA attention 14%, host grouping 6.5%, GDN 9%, hyper-connection reads 4.6%. +**Layout.** `sycl/src` and `sycl/include` mirror the tree and hold only the files the port changes; `sycl/CMakeLists.txt` takes every +other source from the original location. `sycl/include/dpct/` is the vendored SYCLomatic helper library. -**Draft length** (200 tokens, medians of 5 interleaved pairs, `--spec-min-p 0.5`): `--spec 6` against `--spec 4` is +6.5% on -code (43.4 against 40.8 tok/s, 84% accepted against 90%) and -1.5% on prose (29.9 against 30.4). A 2-run sweep of the rest -(`--spec 2/3/5`, min-p 0.3/0.7) was within the run-to-run noise of ~1 tok/s; the setup default (4, 0.5) stays. +**How it is made, so it can be redone.** -**FP64.** Arc Alchemist emulates it (`IGC_EnableDPEmulation=1`); on the A750 one row of the sampler's top_p/temperature tail -costs 451 us in double and 20 us in float (`sycl/probe/fp64_cost.cpp`), and a sampled decode ran at 7.6 tok/s against 12 for -a greedy one. On the B70 the same kernels cost 3.4 against 2.1 us, so FP64 is not a B70 problem. Of the kernels that use -double, the native-pack path (the packs setup builds) runs only the sampler (a 16-token greedy run created 173 kernels, the -double ones among them: `sample_tokens` only; the native router, combine, gate and norm kernels are float). The sampler's tail -now accumulates in `strata::samp_acc_t`, float by default (`-DSTRATA_SYCL_SAMPLER_FP64=1` keeps double): A750 sampled decode -7.62 -> 8.10 tok/s (medians of 5 interleaved pairs, runs 7.1-9.2), `sampler_parity` 0 failures on both cards, and the B70 gate's -sampled cases picked the same tokens. +1. `sycl/tools/Dockerfile` is the dev image: the llama.cpp SYCL image plus SYCLomatic (`dpct` 2025.3), ninja, and the CUDA 12.8 + headers that `sycl/tools/get-cuda-headers.sh` pulls from NVIDIA's pip wheels (dpct parses CUDA; it needs headers, not the toolkit). +2. `sycl/tools/migrate.sh` writes a compilation database for the CUDA-touching translation units and runs dpct over them. +3. `sycl/tools/fixups.py` fixes what dpct got wrong, as an idempotent script with a reason per item. Examples: CUDA's null stream + becomes the default stream (every stream cast goes through `strata::q_of()`), `__ldg` and `__fadd_rn` translations that lost + their meaning, ggml lookup tables as plain `static const` arrays. +4. `sycl/tools/build.sh` configures and builds with icpx inside the image. Two compiler flags are load-bearing: + `-fp-model=precise` (icpx defaults to a fast model) and `-cl-fp32-correctly-rounded-divide-sqrt` for the device compiler. The + Arc's fp32 divide is not correctly rounded by default, and Strata's quantizers are byte-exact against ggml through `amax / 127`. -The sampler's launchers also kept dpct's `aspect::fp64` check, which threw "'double' is not supported" at the first sampled token on an A750 without the emulation keys even though the kernels had been float since the accumulator change; 0.1.41 drops the check (it stays in a `-DSTRATA_SYCL_SAMPLER_FP64=1` build). Measured on the A750 (IQ3_XXS, `--spec 4 --mtp`, sampled, 100 tokens, 4 interleaved pairs, one binary): without `IGC_EnableDPEmulation` / `OverrideDefaultFP64Settings` 11.0 / 11.6 / 12.4 / 12.4 tok/s, with them 9.3 / 9.9 / 9.8 / 9.8 (median 12.0 against 9.8, faster in 4 of 4 pairs). Setup still writes the keys for an A-series card: the kernels other than the sampler's that use double (the oracle router, norms, PLE, QSA indexer) were not exercised without them on long contexts or with vision, so the keys stay the safe default. +**What the port had to get right.** -**XMX.** oneMKL's FP16 GEMMs already run on the matrix engines; per expert the dequant (42 us for a gate/up matrix) costs -twice the GEMM (20 us), and the fused dequant+XMX kernel (`xmx_gemm_bench`) is 0.22x of the pair on this card. The speed left -on this card for this model is the dequant kernels (at ~200 GB/s against a 600 GB/s card), the host grouping, and the QSA -prompt attention (14%); the multi-column int8 DPAS decode kernels of llama.cpp PR 29864 target K-quant and Q8_0 weights, not -the IQ-quant experts, and were not ported. +- Host-mapped flags: `volatile` device loads do not bypass the caches on Intel; system-scope atomics do. Host-to-device visibility + during a kernel stays unreliable, which is why the all-resident path waits for the window instead. +- 32-lane sub-groups (`-fsycl-default-sub-group-size=32`): the kernels are written for warps. +- Synchronous copies: `cudaMemcpy` blocks, dpct's default-queue `memcpy` did not wait. +- Stager and PLE waits: no host thread waits on a queue's event under the Level Zero v2 adapter; the copy queue writes a sequence + number into page-locked host memory and the stager polls it. +- Profiling: `sycl/tools/Dockerfile.unitrace` builds the dev image with Intel's unitrace; `unitrace -d` around the engine plus + `sycl/rank_kernels.py` gives device time per kernel. -## Arc Pro B70, Flash-Next IQ3_S: prompt and decode speed in 0.1.43 (branch f-143-intel) +**Keeping up with upstream.** A merge of upstream `main` leaves the copies in `sycl/` behind wherever upstream touched a file they +mirror. They are refreshed by re-migration: -Measured on the B70 (xe, 32 GB, 60 GB RAM, IQ3_S, 11.5k of 24.6k experts in VRAM, the rest in the pinned mirror; unitrace from pti-gpu -for the kernel timelines). All the changes below leave the tokens as they were (first-token logits compared byte for byte, the 10-prompt -gate identical in the 32K and the 128K configuration); every one has an off switch. +1. Merge upstream. No shared file should conflict. Check that `sycl/setup_intel.py --check` and + `sycl/serve/server_intel.py --engine mock` still run against the new `setup.py` and `server.py`. +2. Migrate the old upstream tree with `migrate.sh` into `sycl-base` and the merged tree into `sycl-new`. +3. Run `sycl/tools/normalize.sh ` on both (dpct's file names, unchanged files, message serials, then `fixups.py`). +4. For every file present in both: `git merge-file sycl/F sycl-base/F sycl-new/F`. Copies dpct never produced (`verify.cpp`, + `mtp.cpp`, `remote_experts.cpp`) take upstream's diff by hand; new parity tests are copied in and listed in `sycl/CMakeLists.txt`. +5. A fixup whose pattern upstream changed shows up as a compile error; extend the fixup, re-run it, rebuild. Run a symbol audit + (count the port's feature identifiers before and after). -- **Prompt: the expert copies run on the card's copy engine** (`sycl/include/strata/copy_queue.hpp`). The Level Zero v2 adapter runs every - SYCL queue on the compute engine, a copy-only queue included: the ~10,000 expert DMAs of a 4K prompt (2 MB, ~80 us each) sat between the - expert kernels and never overlapped one (unitrace: 0 of 2,000 copies overlapped a kernel). An immediate command list on the copy-only queue - group (ordinal 1) wrapped with `sycl::make_queue` puts them on `Copy Engine<1,0>`; events work across the two queues as before. 4K prompt - 3.82 -> 3.25 s, 16K 15.5 -> 13.5 s. On by default for a Battlemage card; `STRATA_COPY_ENGINE=1` forces it elsewhere, `=0` turns it off (an A750 - gained 1.4% and the prompt hung there once the stager's completion flag was a copy-engine memcpy, so Alchemist stays off). -- **The 128K configuration setup writes** (`--kv-resident 32768`, the prompt borrows cache slots) paid three more costs: `kv_append` wrote the - host copy of the KV with 1-byte stores (2 s of a 4K prompt, now 4-byte lane stores), every expert GEMM staged two 6.5/3.3 MB operands because they - sit past 4 GiB in the borrowed region (an A770 oneMKL bug that did not show on this card: off on Battlemage, `STRATA_SYCL_GEMM_STAGE=1` - brings it back), and the lent slots were refilled by a single-threaded GGUF read (0.9 s for 930 slots, now a DMA from the pinned mirror: - the experts in the last 10% of the cache slots are mirrored too, `STRATA_LEND_MIRROR=0` off). 4K prompt incl. the refill 7.2 -> 3.3 s, - 1K 3.1 -> 1.6 s, 16K 25.5 -> 13.5 s, 49K 78 -> 44 s; a 684-token request through the server 2.64 -> 1.59 s, a 3,083-token one 6.1 -> 2.95 s (the same answer). -- **Decode: `data/expert-profile-sycl.bin`.** The shipped profile ranks the experts for a document summary; a chat, code or non-English prompt finds - only 60% of its routed experts in the cache (85% for a summary). `tools/make_profile_mix.py` builds a ranking from several routing traces with - every trace counting the same, `tools/collect_profile_traces.py` records them (`bench/profile_corpus.json`: 45 prompts) through the engine's - device-side routing counts (`STRATA_USAGE_DUMP=PATH`). The profile built from the 45 prompts and eight summaries holds 71% / 84% - (leave-one-out) and gave +5% to +64% decode (mean +21%) on 12 prompts that were in neither set, identical tokens; setup_intel writes it for a - card on xe (the i915 A750 keeps the shipped one: its CPU computes the misses, so the residency moves roundings). -- Decode graph without the nodes nobody answers under `STRATA_VERIFY_NO_HOST` (`STRATA_VERIFY_LEAN=0` off): -2%. -- Tried and not worth it on this card: the grouped XMX expert GEMM for Battlemage (built, `STRATA_PF_XMX=1/2`, correct to 6e-7, 11-15% slower), a - miss-first or hit-first group order in the device plan (-3% / -7%), spreading the expert chains over several queues (one compute engine), - `STRATA_LFUSE`, `STRATA_VERIFY_EAGER`. +`sycl/tools/merge_upstream.py` ports open upstream PRs ahead of upstream the same way. -## Measured, 2026-10-01, Arc Pro B70, Coder IQ1_M, 32K context, INT8 KV: the SYCL port (engine 0.1.31-sycl) +**Not ported.** Three kernels carry inline PTX (`mma.sync`, `ldmatrix`, `cp.async`): `qsa_prompt_attn`, `qsa_select`'s block +scores and `native_qsa_score`. The SYCL build takes the "older card" fallback the CUDA build uses below sm_80. +`qsa_prompt_attn` also has an XMX version (`STRATA_PROMPT_ATTN_XMX=1`) that is correct but slower than the fallback. The ggml MMQ +prefill path (`moe_mmq.cu`) is not built. `--peer-device` (a second GPU as an expert-cache tier) is stubbed and refuses with a +message. The sm_90 thread-block-cluster sampler and QSA top-k have no SYCL counterpart. Images on the llama.cpp config are not wired; +on the SYCL port they are ([Pictures](#pictures-on-an-arc)). -| | | -|---|---| -| decode | 78.2 tok/s on a 19-token prompt, 75.7 after a 2,184-token prompt (256 greedy tokens, MTP + suffix drafts) | -| prompt | 780 tok/s on 2,184 tokens; 934 tok/s at 128K, 784 at 256K | -| through the API | 48-72 tok/s on a ~190-token chat answer, prompt included (the first request after a start is the slower one) | -| VRAM | all 12,288 experts resident, ~1.9 GB free with everything loaded (`--vram-reserve-mib 1024`) | -| RAM | no host copy of the experts (`--stream-experts`) | +## Switch reference -## Measured, 2026-09-29, Arc Pro B70, Coder IQ1_M, 32K context, q8_0 KV: llama.cpp +All are environment variables for the engine. Put them in the config's `"env"` block. Defaults are for the current engine. -| | | -|---|---| -| load | ~100 s | -| VRAM | 28.4 of 32 GB | -| decode | 23-25 tok/s; GPU 92% busy at 165 W, CPU idle | -| prefill | 149 tok/s on a 2,701-token prompt; 424 tok/s on a 104,798-token prompt at 131k context | -| quality | correct code on every test; matches the NVIDIA path token for token in spirit, not measured | +| Switch | Default | Effect | +| --- | --- | --- | +| `STRATA_VERIFY_NO_HOST` | 1 (`strata-sycl.sh`); 0 on `i915` | See above | +| `STRATA_MIRROR_MIB` | Free RAM less 4 GiB | Size of the pinned host mirror for experts without a VRAM slot. Set `0` on a layer split where the mirror would not be used. | +| `STRATA_PF_XMX` | 2 on Alchemist | Grouped XMX expert launch; `0` off, `1` dequantize first | +| `STRATA_GDN_XE` | On for Alchemist | Sub-group GDN recurrence | +| `STRATA_COPY_ENGINE` | On for Battlemage | Expert copies on the copy engine | +| `STRATA_PREFILL_MMQ` | `auto` on Battlemage | Fused int8 prompt experts | +| `STRATA_PREFILL_STREAM_ALL` | On when VRAM holds more than 90% of the (layer, expert) pairs | Copy every expert the cache does not hold, layer by layer; `=0` / `=1` force it | +| `STRATA_PREFILL_FIRST` | 256 if the cache holds every expert, else 0 | Tokens in the short first prompt chunk | +| `STRATA_LEND_MIRROR` | On | Mirror the experts in the last 10% of cache slots; `=0` off | +| `STRATA_VERIFY_LEAN` | On | Decode graph without unanswered nodes; `=0` off | +| `STRATA_FILL_SERIAL=1` | Off | The old serial expert-cache fill (the pipelined fill is the default) | +| `STRATA_MMVQ_A2=0`, `STRATA_MMVQ_WIDE_32=0` | On | Restore the older matvec kernels | +| `STRATA_MMVQ_LARGE_GRF` | On for Arc A-series | 256 registers per thread for the 2-8 column wide kernels | +| `STRATA_Q6_PLANAR` | On | Q6_K output head and MTP head as planes (same bytes) | +| `STRATA_ATTN_RS` | 2 | QSA prompt attention probability layout (bit-exact variants) | +| `STRATA_IQ3S_WIDE=1` | Off | Wide kernel for dense IQ3_S matrices: neutral on one B70 (decode -0.3%), the author reports gains on two cards | +| `STRATA_EXPERT_SPLIT=1` | Off | Upstream's i-quant multi kernels (-11 to -16% decode here) | +| `STRATA_PROMPT_ATTN_XMX=1` | Off | joint_matrix prompt attention (-24% to -27% prompt on the B70) | +| `STRATA_SYCL_GEMM_STAGE=1` | Off on Battlemage | Stage GEMM operands past 4 GiB | +| `STRATA_I8_NO_ESIMD` | n/a | Compile-time: stubs for the int8 ESIMD kernels (set for a device without them) | +| `STRATA_DBG_NAN=1` | Off | Report non-finite values, including the experts' FP16 inputs | -## Arc Pro B70: `--prefill 8192` in the long-context config (2026-10-18, branch f-151-b70) +## History / older measurements -`setup_intel.py` now recommends 8,192-token prompt chunks for a Battlemage card with 24 GB or more when the context is over 32K (the -prompt borrows cache slots; `recommended_prefill()`), 4,096 everywhere else, and it keeps a chunk size you put in the config. -Arc Pro B70, IQ3_S, setup's 128K flags (`--kv-resident 32768 --vram-reserve-mib 2048`), `--pcie-frac 0 --adapt-every 0`, 2 rounds, -32 greedy tokens after the prompt (identical in every run): +Dated results from earlier engines. They show how the numbers moved and are not instructions. Narrative from the original work +log (debugging stories, per-merge notes, planned-item lists) was dropped; the numbers and their labels are kept. -| prompt | `--prefill 4096` | `--prefill 8192` | cache slots | decode after | -|---|---|---|---|---| -| 16K | 1,222 tok/s | 1,347-1,362 (+10-11%) | 11,795 both | 33-38 / 38 | -| 32K | 1,187 | 1,361-1,383 (+15-16%) | 11,794-11,795 both | 35 / 38.6 | +> **Measured on:** Arc Pro B70 32 GB (xe), PCIe 3.0 x8, Ubuntu 24.04, Coder IQ1_M, 32K context, INT8 KV, the SYCL port, +> engine 0.1.31-sycl (2026-10-01), against llama.cpp SYCL (2026-09-29, q8_0 KV). -The lent slots are refilled after the prompt, so the bigger chunk costs the decode nothing. First-token logits against 4096 -(`STRATA_DUMP_FIRST_LOGITS`, whole vocabulary): KL 1.1e-2 nats at 16K and 3.0e-2 at 32K, top-1 the same, largest logit -difference 1.0 (the chunk boundary moves where the QSA block selection is made, so a different chunk size is not bit-identical; -two runs of the same size are, KL 0). With the 32K context (own prompt buffers, no borrowing) a bigger chunk is +20% on a 16K -prompt (1,236 -> 1,478-1,482) but takes 1,397 cache slots (10,926 -> 9,529) and decode after it falls 9-16%, so 4,096 stays there. +| | NVIDIA (RTX 5070, Strata) | Intel Arc, SYCL port | Intel Arc, llama.cpp | +| --- | --- | --- | --- | +| Decode, Coder IQ1_M | 44-51 tok/s | 78.2 tok/s (19-token prompt), 75.7 (2,184-token prompt); 48-72 through the API | 23-25 tok/s | +| Prompt | about 1,870 tok/s | 790 tok/s at 2,184 tokens, 1,117 at 40K, 888 at 128K, 757 at 256K | about 150 tok/s (424 at 105K) | +| Experts | in RAM, hot ones on the card | all 12,288 in VRAM (`--stream-experts`), about 1.9 GB VRAM free | all of shard 1 on the card | +| Load time | | about 26 s to first token (was 82 s before the pipelined fill) | about 100 s | -## Arc Pro B70: the fused int8 expert path for the prompt (2026-10-18, branch f-151-b70, default on for Battlemage) +**Decode, Coder IQ1_M on the B70 (19-token prompt, 256 greedy tokens), by change.** Each step measured alone, output tokens +identical unless noted. -On a Battlemage GPU (B70, B580, B570: detected by the device architecture id, `bmg-g21` and `bmg-g31`, not by name) the engine computes the routed experts of the prompt with ESIMD kernels that decode the quantized weights in -registers straight into the s8 DPAS B operand (no FP16 copy of the expert, 2.1 MB of weights read instead of 9.8 MB written and read -back) and multiply them by q8_1-quantized activations; `sycl/src/prefill/esimd_moe.dp.cpp`, behaviour note and switches in -[INTEL_INT8_EXPERTS.md](INTEL_INT8_EXPERTS.md). Gate/up IQ3_S, IQ3_XXS, IQ2_S and down IQ4_NL, Q2_0 are covered (47 of the 48 layers of -the mixed IQ3_S pack; the other layer, IQ4_XS gate/up, and any other format keep the FP16 path). Default (STRATA_PREFILL_MMQ unset) is `auto` on Battlemage and the FP16 chain on every other GPU (Alchemist A750 / A770, Lunar Lake, other vendors); `=0` turns it off, `=1` forces it, `=auto` is the default. The activations are -quantized, so the bits differ from the FP16 path. `auto` times both chains on this GPU at the first layer of each format and takes int8 -below the measured crossover (130-250 rows per expert here), per group of experts. +| Change | Decode | +| --- | ---: | +| First working engine (suffix drafter only, `--spec 2`) | 20.2-20.8 tok/s | +| With the base checkpoint's MTP draft layer (`--spec 4 --mtp`, 81-87% accepted) | 42.7-45.3 tok/s | +| After the 0.1.25-0.1.27 merge | 46.5 | +| Commit graph left running while the drafter runs on its own queue | 49.8 | +| Q6_K alignment fix (a Q6_K block is 210 bytes; two aligned 16-byte loads) | 53.9-54 | +| `resident_plan` grouping in parallel | 57.1 | +| GR down kernel reads activations directly | 58.7 | +| Wide 16-byte-load kernels for Q4_K, Q5_K, IQ4_XS | 62.6 | +| Wide kernels for Q8_0 and IQ4_NL; Q6_K activation loads aligned | 65.6 | +| GR down projection sliced by columns | 70.6 | +| IQ4_NL expert dots with codebook in registers | 76.1 | +| GR norm per (token, stream); aligned weight loads in IQ3 gate/up dots | 77.9 (output differs after token 146) | -Arc Pro B70, IQ3_S pack, 32K config, `--prefill 4096`, fixed flags (`STRATA_PREFILL_CPU_SHARE=0 STRATA_ROUTE_TAIL_SKIP=0 --pcie-frac 0 ---adapt-every 0`), 5 interleaved rounds per size, prefill tok/s (medians; the noise arm is the default run again): +Prompt, same card, 2,184-token prompt: 560 tok/s (first engine) to 792 tok/s after dequant vector stores (0.085 to 0.030 ms per +expert), a short first prompt chunk (711 to 765 tok/s), one device module per kernel and the PLE reader at 64 threads +(466 to 330 ms for that prompt's rows). Draft sweep at 2,184 tokens: `--spec 4` 41.6-43.5 tok/s, `--spec 2` 33.6, `--spec 6` +35-37. Decode depends on the text: 45-51 tok/s on code (77-85% of drafts accepted), 30-39 on prose (55-72%). -| prompt | default | noise arm | `=1` | `=auto` | -|---|---|---|---|---| -| 1,024 tokens | 645.8 | 648.3 | 775.4 (+20%) | 783.5 (+21%) | -| 4,095 tokens | 1,257.0 | 1,256.8 | 1,413.8 (+12.5%) | 1,407.7 (+12.0%) | +**Flash-Next IQ2_XS on the B70 (original model, 18,329 of 24,576 experts in VRAM, 6,247 in the pinned mirror).** Decode 50.8 tok/s +on a 19-token prompt and 60.6 after 2,184 tokens, prompt 549 tok/s (0.1.31); after the 0.1.31 merge 58.6 / 64.2. A forced test +(`--expert-cache 8000`, 1,879 experts out of VRAM) took decode from 2.6 to 40.9 tok/s (full cache 43.3) once the host mirror existed. -The output tokens are identical in all runs. Quality (teacher-forced KL against the FP16 path over 6,002 positions of 28 prompts): mean -2e-4 nats, p99 6e-3, top-1 99.7%, perplexity ratio 1.0004; a `--prefill 8192` chunk moves the first-token distribution by 1e-2 to 3e-2 -for comparison. At more than ~130 rows per expert (a chunk of 8,192 tokens or more) the int8 chain stops winning because every row block -decodes the weights again (1.03x at 194 rows against 1.4-1.9x at 60): use `auto`. +**Long prompts.** 80,000 tokens at `--max-context 81920 --kv int8 --vram-reserve-mib 2048 --prefill 4096` with slot borrowing: +75 s (1,062 tok/s), decode 35-40 tok/s after it; without borrowing, decode after the prompt ran at 2 tok/s. 128K int8 with +KV streaming: 897 tok/s routed-only against 920 with the stream-all walk; 40K 1,106 against 1,160; 128K k8v4 1,006 against 1,041; +256K 760 against 780 (0.1.38). Prompt time at 40K (GPU timeline): expert dequant 15.7%, expert GEMMs 23.4%, prompt attention +19.2%, host grouping 6.5%, QSA select 6.8%, DeltaNet 5.8%. -### Arc Pro B70, what else was measured on the prompt path (2026-10-18, branch f-151-b70) +**Context sizes on llama.cpp (2026-09-29, Coder IQ1_M, q8_0 KV, 32 GB card).** About 20 KiB of KV per token. -All on the 4,095-token IQ3_S prompt, 32K config, `--prefill 4096`, fixed flags, 3-4 interleaved rounds with a noise arm (prefill tok/s, -medians, output tokens identical in every arm): +| Context | VRAM in use | Status | +| --- | ---: | --- | +| 32,768 | 28.4 GB | Measured, the default | +| 65,536 | 29.0 GB | Loads | +| 131,072 | 30.3 GB | Practical ceiling: a 104,798-token prompt read in 247 s at 424 tok/s | +| 262,144 | about 32.6 GB | Does not fit; asking for it took the host down | -| change | result | -|---|---| -| `STRATA_GDN_XE=1` (variant 6, the sub-group recurrence that is the default on Alchemist; bit-identical) | 1,251 -> 1,207 (-3.5%): stays off on Battlemage | -| `STRATA_PROMPT_ATTN_XMX=1` / `=32` (the joint_matrix prompt attention, 64- / 32-cell chunks) | 1,258 -> 952 / 913 (-24% / -27%): stays opt-in | -| QSA prompt attention, probabilities stored `[cell][head]` and read as three vectors per cell in the value phase (`STRATA_ATTN_RS=2`, now the default; first-token logits md5 equal at 4K and 16K) | 1,254 -> 1,265 (+0.9%, 3 of 3 rounds) | -| the same plus a 16-shuffle reduce-scatter of the 12 heads' score sums (`STRATA_ATTN_RS=1`; also bit-exact) | 1,254 -> 1,236 (-1.5%): the score phase is not shuffle bound | -| int8 expert groups of 24 or 32 experts instead of 16 | 1,416 -> 1,420 / 1,416: no effect | -| K-quant / Q5_1 / Q8_0 multi-entry decode kernels (`STRATA_KQ_SPLIT`, on by default; bit-identical to `=0`), `native_grouped_parity --kqbench 16`, synthetic UD-Q4_K_XL-shaped experts | 1 token: 76.4 vs 75.0 us (-1.8%); 2 tokens: 122.5 vs 135.9 us (+10%); 4 tokens (an MTP window): 202.6 vs 233 us (+13%); Q6_K 4 tokens 387 vs 485 (+20%), Q8_0 290 vs 341 (+15%) | +**Load time.** The expert-cache fill became a pipeline (reads in file order by up to 8 threads into page-locked batches of 64, +copies overlapped): a cold fill of the Coder's 23.4 GiB went from 76.2 s to 18.8 s (0.33 to 1.34 GB/s), a whole start from +82 s to 26 s, and the IQ2_XS from 120 s to 41 s. -The B70's int8 expert kernels also build and pass the parity test as a SPIR-V JIT build and as an AOT build for `bmg-g21` (compile only: no -G21 card here); an AOT build for another device (Alchemist, Xe-HPC) gets stubs (`STRATA_I8_NO_ESIMD`). +**Upstream PRs ported early (0.1.35).** + +| PR | What | On the B70 | +| --- | --- | --- | +| #385 | Stager: a buffer's first job of a generation waits for the previous DMA | Race fix, same output | +| #463 | Decode waits for an adaptive swap before reading the residency table | Determinism fix | +| #453 | Drafter's batched K/V on the KV-streaming ring | 128K int8: 888 to 895 tok/s prompt, 66.6 to 67.0 decode | +| #374 | First chunk's PLE rows read beside layer 0 | Part of 2,184-token prompt 784 to 825 tok/s | +| #363 | PCIe expert call: group stride, fused SwiGLU + q8_1 | IQ2_XS decode +1.7% | +| #407 | `--adapt-tuned` (opt-in) | Neutral | +| #413 | DeltaNet recurrence per key head | 8% slower prompt: off (`STRATA_GDN_KEYHEAD=1`) | + +**Alchemist prompt GDN kernel (Arc A770 16 GB, xe, oneAPI 2026.1 JIT, 2026-10-13).** `gdn_parity --prefill-tokens 8192` 51.0 to +24.5 ms; 32,768 tokens 204-236 to 110-113 ms. End to end, Coder IQ1_M, `--max-context 131072 --kv int8 --kv-resident 20480 +--vram-reserve-mib 1200 --resident-experts`: prompt 386 to 396-407 tok/s at 8K and 478-480 to 511-518 at 32K. To check another +Intel card: `STRATA_GDN_XE=1 ./gdn_parity --selftest --prefill-only`. + +**Q6_K decode kernels (0.1.45, #2340, #2207).** B70, `mmvq_bench` Q6_K 2560x10240, 1 column 44.7 to 26.5 us, 4 columns 64.5 to +46.2; A750 4 columns 251 to 102 us. End to end on the B70 (IQ3_S, 5 interleaved pairs, answers identical): decode 32.24 to 33.37 +tok/s on a story prompt (+3.5%), 39.11 to 40.77 after a 4K prompt (+4.2%). + +**Fused int8 expert prompt path (B70, IQ3_S, 32K config, fixed flags, 5 interleaved rounds).** 1,024 tokens: default 645.8, `=1` +775.4 (+20%), `auto` 783.5 (+21%). 4,095 tokens: 1,257.0, 1,413.8 (+12.5%), 1,407.7 (+12.0%). At more than about 130 rows per +expert (a chunk of 8,192 tokens or more) the int8 chain stops winning (1.03x at 194 rows against 1.4-1.9x at 60). + +**Other prompt-path trials on the B70 (4,095-token IQ3_S prompt, tok/s).** + +| Change | Result | +| --- | --- | +| `STRATA_GDN_XE=1` on Battlemage | 1,251 to 1,207 (-3.5%) | +| `STRATA_PROMPT_ATTN_XMX=1` / `=32` | 1,258 to 952 / 913 | +| `STRATA_ATTN_RS=2` (now default) | 1,254 to 1,265 (+0.9%) | +| `STRATA_ATTN_RS=1` | 1,254 to 1,236 (-1.5%) | +| Int8 expert groups of 24 or 32 | No effect | +| Grouped XMX expert GEMM on Battlemage (`STRATA_PF_XMX=1/2`) | 11-15% slower | +| Miss-first / hit-first group order in the device plan | -3% / -7% | + +**Via the OpenAI API, sampled (0.1.31-sycl era).** 36 tok/s decode on a first turn, 33 on a follow-up, against 24-26 for llama.cpp +on the same card. Logprobs for "Paris" (IQ2_XS) -0.0019 (p 0.998). + +**Tests (0.1.x ports).** On the B70, 19 of the 22 kernel parity tests passed byte for byte or within tolerance; the others needed +model fixtures or the tensor-core kernel. Parity gaps that remain: `s2_expert_grouped_parity` (the s2 path), `kv_hybrid_parity`'s +last step on the XMX kernel, `ple_parity` and `native_expert_parity` need model files. + +**Fixed bugs worth knowing if you see the symptom** (fixed long ago; the symptom is listed so a search finds this line). `--stream-experts` read +the wrong shard for layers past a shard boundary (NaN logits from layer 13 on, Swift 1.5). IQ2_XS products were garbage because of +a type pun the SYCL compiler does not honour. A prompt that borrowed cache slots hung in its first chunk (stager deadlock under +Level Zero v2). An A750 prompt of 2,744 tokens hung (direct copy out of pageable memory; regression test +`sycl/tools/test_prompt_deadlock.sh`). The first request after a start returned token 0 on `i915` when the spin bound was too short. +An A770 JIT build misread memory past 4 GiB into an allocation. The xe driver aliased pages in a 22 GiB arena (NaN at layer 4). diff --git a/docs/INTEL_ARC.md b/docs/INTEL_ARC.md index 34ffd1844..4adf07a3c 100644 --- a/docs/INTEL_ARC.md +++ b/docs/INTEL_ARC.md @@ -1,97 +1,74 @@ -# Intel Arc (experimental) +# Intel Arc on Linux -Strata 0.1.39 includes an **experimental Intel Arc engine**: Strata's own engine ported to SYCL (Intel oneAPI). -maxfridbe wrote it in [#423](https://github.com/Niko1221/Strata/pull/423), with fixes from the people testing it. -The code is in `sycl/`, and the port's own notes, measurements and maintenance procedure are in -[docs/INTEL.md](INTEL.md). This page covers what you need, how to build it, and what has been tested. +Strata runs on Intel Arc cards on Linux through its own engine ported to SYCL (Intel oneAPI). This page is for people with +an Arc A-series (Alchemist) or B-series (Battlemage) card. It covers the released Terminal route, what you need, how to +build the engine from source, and which cards were tested. The port's workflow and measurements are in [INTEL.md](INTEL.md). -**Experimental means:** the Intel engine is still young and Linux only. When the port arrived (0.1.39) the maintainers had no -Intel GPU and every result on Arc hardware came from the community. Since 0.1.40.2 the maintainers also test on an Arc Pro B70 -(32 GB, `xe`) and an Arc A750 (8 GB, `i915`); the results from those two cards are marked in the table below. Other cards -are still community reports. The NVIDIA and AMD engines are unchanged: the Intel build is a separate CMake target, off by default. +maxfridbe wrote the port in [#423](https://github.com/Niko1221/Strata/pull/423). The code is in `sycl/`. Intel Arc is an +official platform since 0.1.40.2. The maintainers test on an Arc Pro B70 (32 GB, `xe` driver) and an Arc A750 (8 GB, `i915` +driver); results from other cards are community reports. The Mac and Windows builds have no Intel engine. -**Where the engine comes from.** On Linux, `strata update` (Terminal) and Settings > Update (app) download the Intel engine package, `linux-x86_64-sycl`, for a PC whose graphics card is made by Intel, when the release feed lists one (`strata update --check` shows what it would install). One package serves Alchemist (A-series) and Battlemage (B-series) cards: the GPU code is compiled from SPIR-V at the first start, because the A-series cards have no FP64 units and the engine's FP64 kernels cannot be built ahead of time for them. It carries oneAPI's runtime libraries; the Level Zero loader and the compute runtime stay the PC's own (see "What you need" below), and the user must be in the `render` group (`strata doctor` says so). The package is built by `packaging/build_sycl_linux.sh`. It was tested on an Arc A750 only. The release zips (the ones `START-HERE.bat` and `setup.sh` use) have no Intel engine; to build one from source, follow the steps below. The Docker route is a ready-made image, `ghcr.io/niko1221/strata:0.2.3-sycl`, which is published only for a release that carries a Linux Intel engine package; with none, [build the image yourself](DOCKER.md#build-it-yourself) from the engine you built (the image needs prepared model files; see [Docker: Intel Arc](DOCKER.md#intel-arc)). The Mac and Windows builds have no Intel engine. +## Quick start -## What has been run, and by whom +Install the [Terminal](INSTALL.md#the-terminal), then: -| | Hardware | Result | Reported in | -|---|---|---|---| -| Port author | Arc Pro B70 32 GB, Ubuntu 24.04 | Coder IQ1_M: 70-78 tok/s decode, ~790 tok/s prompt; IQ2_XS: 51-64 tok/s; 256K context measured | #423, [INTEL.md](INTEL.md) | -| Community | 2x Arc Pro B70, `--layer-split` | Flash-Next IQ3_XXS 66 tok/s decode, 394 tok/s prompt (with the `stage_room` fix that is now in 0.1.39) | #423 | -| Community | Arc Pro B50 16 GB | Coder IQ1_M ~23 tok/s, IQ2_XS ~25-27 tok/s, up to 128K | #423 | -| Community | Arc B580 12 GB, WSL2 | IQ2_XS ~21 tok/s, Coder ~15 tok/s; **device loss also seen** | #423 | -| Community | 2x Arc Pro B60 24 GB, `--layer-split` | Coder IQ1_M: 56.6 tok/s decode, 428 tok/s prompt (2,129 tokens); Flash-Next IQ2_XS: 58.6-61.3 tok/s decode, 436 tok/s prompt; all experts in VRAM; 8K context | [bench/results/2026-10-04-community-2x-arc-pro-b60](../bench/results/2026-10-04-community-2x-arc-pro-b60/README.md) | -| Community | one Arc Pro B60 24 GB | Coder IQ1_M with 4,042 of 12,288 experts mirrored in RAM: 11.9 tok/s decode (needs the ring-wait fix from the same report) | same | -| Strata maintainers (0.1.40.2 and later) | Arc Pro B70 32 GB (xe) and Arc A750 8 GB (i915), test machines | Flash-Next IQ3_S on the B70 with 11.5k of 24.6k experts resident: 10-prompt gate passes, prompt 980-1,000 tok/s on 4K tokens, decode 30 (prose) to 41 (code) tok/s; the xe aliased-pages fix and the `NO_HOST` requirement are in [INTEL.md](INTEL.md) | INTEL.md, "Arc Pro B70 with a model that does not fit" | +```sh +strata update --check # shows the Intel engine package the feed would install +strata doctor --live # checks the driver, the render group and the card +strata launch --plan # shows what would happen, changes nothing +strata launch +``` -The 0.1.39 port re-migrates 0.1.38's port onto the 0.1.39 engine sources (the #606 NaN fix, the #649 verify -trace, the new prompt paths). 0.1.39's `sycl/` did not compile against 0.1.39's own engine sources (#784: the -`ThreadAffinity` type of #626 and the layer range of `NativeDense::load` from #559), and its ring waits never saw a -slow layer as still running (#866, #867). Both are fixed in 0.1.40, and the two B60 reports below were measured with -exactly those two fixes on top of 0.1.39. **No one has run an unpatched 0.1.39 port on an Arc**, and the other rows were -measured on earlier versions. +On Linux, `strata update` (Terminal) and Settings > Update (app) download the Intel engine package `linux-x86_64-sycl` for a PC +whose graphics card is made by Intel. One package serves Alchemist and Battlemage: the GPU code is compiled from SPIR-V at the +first start, because the A-series cards have no FP64 units. The package carries oneAPI's runtime libraries. The Level Zero +loader and the compute runtime stay the PC's own (see [What you need](#what-you-need)), and your user must be in the `render` +group (`strata doctor` says so). It was tested on an Arc A750 only. The package is built by `packaging/build_sycl_linux.sh`. -The two B60 rows ran `6f32ec0` plus two small `sycl/` fixes (the compile fix and the ring-wait fix), AOT `bmg-g21`, on Ubuntu 24.04 with -`xe`, Level Zero V2, NEO 26.09.37435.12 and oneAPI 2026.1.1, without Docker. Host: Ryzen 5 5600, 64 GB RAM, PCIe 3.0 x8 per card. -Full flags, per-request timings and engine logs are in the report linked in the table. +**Model preparation is separate.** `strata launch` reuses compatible prepared local files, but it does not run the Intel +source or container setup for you. When a model needs preparation, follow the [source workflow](INTEL.md#setup). Installing the +engine package alone does not prove the selected model is ready; check a loaded server with a real request. -## What was tested here (0.1.39) - -- **Compiles:** Ubuntu 24.04 (WSL2), Intel oneAPI DPC++ 2026.1.1 + oneMKL 2026.1. The whole `sycl/` project - builds with 0 errors: the `strata` engine plus all 157 targets (the kernel parity tests and benches). The build is - SPIR-V (JIT). The AOT build (`STRATA_SYCL_AOT`) was not built here, because it needs `ocloc`. -- **Kernel parity tests on a CPU** (`ONEAPI_DEVICE_SELECTOR=opencl:cpu`, Intel's OpenCL CPU runtime, AMD Ryzen 5 7600): - 14 of 25 pass. That is the same set that PR #423's own 0.1.38 port passes on that device: the failures are tight - float tolerances on the CPU's math (rel 3e-6 against a 1e-6 limit), model fixtures that are not present, and two - tests that time out on a CPU. This is a check that the kernels compile and compute, not a test of an Arc. -- **Setup:** `--backend sycl` warns, then hands over to `sycl/setup_intel.py` on Linux (unit tests: - `tools/test_setup_sycl.py`). -- **Unchanged:** the CUDA engine's greedy output, checked byte-identical against the gated 0.1.39 build (Q2_0 and Coder). - The HIP build is not affected (the option is off by default). - -**Not tested by anyone yet:** Windows (no native build path; see below), Arc on WSL2 for the 0.1.39 port, integrated Arc -GPUs (the B390 / Panther Lake in #515; the shared-memory planning does not exist yet), and pictures on a Battlemage card -(the first picture on the Arc Pro B70 reset the GPU; they are off by default there, [INTEL.md](INTEL.md#pictures-on-an-arc-2026-10-19-branch-f-021-arcvis)). -Pictures work on the A750 (on by default, read on the CPU). The Alchemist A-series was tested later, on an A750 (rows above and -[INTEL.md](INTEL.md), "Arc A750 and the other Alchemist cards"). +Docker: the ready-made image is `ghcr.io/niko1221/strata:0.2.3-sycl`, published only for a release that carries a Linux Intel +engine package. Otherwise [build the image yourself](DOCKER.md#build-it-yourself). The image needs prepared model files +([Docker: Intel Arc](DOCKER.md#intel-arc)). ## What you need -- **Linux**, e.g. Ubuntu 24.04, with Intel's GPU driver (the `xe` or `i915` kernel driver plus the compute runtime / - Level Zero; on Ubuntu, `intel-opencl-icd libze1 libze-intel-gpu1`, or Intel's - [client GPU guide](https://dgpu-docs.intel.com/driver/client/overview.html)). -- **Intel oneAPI**: the DPC++ compiler (`icpx`, 2025.3 or newer; 2026.1 is what was built here) and **oneMKL**. About 5 GB. -- `cmake` 3.24+, `ninja`, `git` (the build fetches ggml unless you point `STRATA_GGML_DIR` at a llama.cpp checkout), - Python 3. -- For `./setup.sh --backend sycl` today: **Docker**, and nothing else from the list above except the GPU driver. - `sycl/setup_intel.py` runs the engine in the `strata-sycl-dev` image built from `sycl/tools/Dockerfile`, and the - engine is built in that image too ("Build with Docker" below). On Ubuntu: `sudo apt install docker.io git`, then - `sudo usermod -aG docker,render $USER` and log in again. Note that the Dockerfile starts from a community - llama.cpp SYCL image (`ghcr.io/snailium/...`), not an Intel or Strata image. -- VRAM: the port keeps the experts on the card (`--stream-experts`). A 32 GB card holds the Coder IQ1_M or IQ2_XS. - Smaller cards mirror part of the experts in RAM and are slower. +- **Linux**, for example Ubuntu 24.04, with Intel's GPU driver: the `xe` or `i915` kernel driver plus the compute runtime and + Level Zero. On Ubuntu: `intel-opencl-icd libze1 libze-intel-gpu1`, or follow Intel's + [client GPU guide](https://dgpu-docs.intel.com/driver/client/overview.html). +- For a source build without Docker: **Intel oneAPI** with the DPC++ compiler (`icpx`, 2025.3 or newer) and **oneMKL** + (about 5 GB), `cmake` 3.24+, `ninja`, `git`, Python 3. The build fetches ggml unless you point `STRATA_GGML_DIR` at a + llama.cpp checkout. +- For `./setup.sh --backend sycl`: **Docker** and the GPU driver, nothing else from the list above. `sycl/setup_intel.py` + builds and runs the engine in the `strata-sycl-dev` image from `sycl/tools/Dockerfile`, which starts from a community + llama.cpp SYCL image (`ghcr.io/snailium/...`), not an Intel or Strata image. On Ubuntu: + `sudo apt install docker.io git`, then `sudo usermod -aG docker,render $USER` and log in again. +- VRAM: the port keeps the experts on the card (`--stream-experts`). A 32 GB card holds the Coder IQ1_M or IQ2_XS. Smaller + cards mirror part of the experts in RAM and are slower. ## Build with Docker (what `setup --backend sycl` uses) -Two commands, the same as in [INTEL.md](INTEL.md), "How to build it" (about 10 minutes the first time). From the checkout, -called `Strata` here: +About 10 minutes the first time. From the checkout, called `Strata` here: ```sh cd Strata -docker build --pull -t strata-sycl-dev sycl/tools # once: ~6 minutes, a 13 GB image (--pull: INTEL.md says why) +docker build --pull -t strata-sycl-dev sycl/tools # once: about 6 minutes, a 13 GB image H=$(basename "$PWD") # Arc B-series: bmg-g31 (Pro B70), bmg-g21 (B580, B570, Pro B60). Arc A-series (Alchemist): leave out AOT (JIT) and use build-sycl docker run --rm -u $(id -u):$(id -g) -v "$PWD/..:/work" -e AOT=bmg-g31 -e REPO=/work/$H \ -e BUILD_DIR=/work/$H/build-sycl-aot strata-sycl-dev "cd /work/$H && bash sycl/tools/build.sh strata" ``` -Then `./setup.sh --backend sycl` (below). Do not run `python3 sycl/setup_intel.py` by hand: on Ubuntu it fails with -`externally-managed-environment`; `setup.sh` runs it in its own `.venv`. +`--pull` matters: [INTEL.md](INTEL.md#how-to-build-it) says why. Then run `./setup.sh --backend sycl`. Do not run +`python3 sycl/setup_intel.py` by hand: on Ubuntu it fails with `externally-managed-environment`; `setup.sh` runs it in its own +`.venv`. -## Build without Docker (Linux) +## Build without Docker -This builds the engine for running it by hand ("How to run it by hand" in [INTEL.md](INTEL.md)); `setup` itself uses the -Docker build above. Install oneAPI from Intel's apt repository (this is what was used here): +This builds the engine for running it by hand ([INTEL.md](INTEL.md#how-to-run-it-by-hand)). `setup` itself uses the Docker build. +Install oneAPI from Intel's apt repository: ```sh wget -qO- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB \ @@ -102,8 +79,7 @@ sudo apt update sudo apt install intel-oneapi-compiler-dpcpp-cpp intel-oneapi-mkl-devel ninja-build cmake git ``` -Then build the engine. Either command works: the first goes through the top-level CMake option, the second -configures the `sycl/` project directly. +Build the engine through the top-level CMake option or by configuring the `sycl/` project directly: ```sh source /opt/intel/oneapi/setvars.sh @@ -112,16 +88,14 @@ cmake -S . -B build-sycl -G Ninja -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=ic cmake --build build-sycl --target strata ``` -Options: +| Option | Effect | +| --- | --- | +| `-DSTRATA_SYCL_AOT=bmg-g31` | Compile ahead of time for Arc Pro B70. Needs `ocloc` (Intel's `intel-ocloc` package). Without it the first start JIT-compiles every kernel (about 47 s). | +| `-DSTRATA_SYCL_AOT=bmg-g21` | The same for B580, B570 and Pro B60 | +| `-DSTRATA_SYCL_PARITY=OFF` | Skip the kernel tests (on by default). Run them with `ctest --test-dir build-sycl` on the card. | -- `-DSTRATA_SYCL_AOT=bmg-g31` (Arc Pro B70) or `bmg-g21` (B580 / B570 / Pro B60) compiles the GPU code ahead of - time. This needs `ocloc` (Intel's `intel-ocloc` package). Without it, the first start JIT-compiles every kernel, - which takes about 47 s. -- `-DSTRATA_SYCL_PARITY=OFF` skips the kernel tests (on by default). Run them with - `ctest --test-dir build-sycl` on the card. - -`setup_intel.py` looks for `build-sycl-aot/strata` or `build-sycl/strata` in the checkout. With the -top-level option the engine is at `build-sycl/sycl/strata`, so either use `-S sycl` or set `STRATA_SYCL_BIN`. +`setup_intel.py` looks for `build-sycl-aot/strata` or `build-sycl/strata` in the checkout. With the top-level option the +engine is at `build-sycl/sycl/strata`, so use `-S sycl` or set `STRATA_SYCL_BIN`. ## Setup and running @@ -129,50 +103,76 @@ top-level option the engine is at `build-sycl/sycl/strata`, so either use `-S sy ./setup.sh --backend sycl [setup's usual options, e.g. --model IQ2_XS --context 32768] ``` -This prints the experimental warning and continues with `sycl/setup_intel.py` (in setup's own Python environment). -That script finds the Arc in sysfs, uses the SYCL engine you built, and writes the config and `run-.sh`. -On a card under 12 GB or an `i915` card (the A-series) it writes the small-card config ([INTEL.md](INTEL.md), "Arc A750 -and the other Alchemist cards"). It still downloads and packs the model -the usual way. To run the engine by hand (no Docker), see "How to run it by hand" in [INTEL.md](INTEL.md). +Setup continues with `sycl/setup_intel.py`, which finds the Arc in sysfs, uses the SYCL engine you built, and writes the +config and `run-.sh`. On a card under 12 GB or an `i915` card (the A-series) it writes the small-card config +([INTEL.md](INTEL.md#arc-a750-and-the-other-alchemist-cards)). It downloads and packs the model the usual way. Windows has no +path: `setup --backend sycl` on Windows stops and points here. -Things that matter on an Arc (details in INTEL.md): +Things that matter on an Arc (details in [INTEL.md](INTEL.md)): -- **Do not ask for more VRAM than the card has.** On the `xe` driver, an allocation past VRAM can push buffers into - RAM until the machine runs out of memory and stalls. Leave about 1.5 GB free. +- **Do not ask for more VRAM than the card has.** On the `xe` driver an allocation past VRAM can push buffers into RAM until + the machine runs out of memory and stalls. Leave about 1.5 GB free. - `SYCL_CACHE_PERSISTENT=0`: the persistent JIT cache crashed on Xe2 during the first compile. -- Two cards: `ONEAPI_DEVICE_SELECTOR=level_zero:*` (the image pins `level_zero:0`; `strata-sycl.sh` now passes the - variable through) and `--layer-split`. -- **`--layer-split auto` on two equal cards (0.1.44, #1440):** when several splits hold all the experts and predict the - same speed, the search now takes the one that leaves its fullest card the most VRAM to spare (within 0.2% of the best - predicted time, `STRATA_SPLIT_HEADROOM_TOL`). It used to take the first one tried, which filled the second card to its - last MiB. A full card under `xe` moves buffers to system memory instead of failing: 2x B70 with IQ3_S, auto picked - K=24 (~11 tok/s) where `--layer-split 30` ran ~54. The startup line names both splits when the choice moves. - `STRATA_SPLIT_HEADROOM=0` keeps the old pick. -- **Arc Pro B60:** the PCI id is `8086:e211` (`lspci -nn`, the kernel's `xe` id list files it with the BMG-G21 cards), `sycl-ls` prints - `Intel(R) Arc(TM) Pro B60 Graphics 20.1.0`, and `ocloc ids bmg-g21` prints 20.1.0, so `-DSTRATA_SYCL_AOT=bmg-g21` is the right target. - `setup_intel.py` knows both B60 ids, `e211` and `e221` (both seen on B60 cards). On a `--layer-split` the startup line "N experts are neither in VRAM nor mirrored" - counts the other card's layers; the lines `100% of the experts resident` that follow are the ones to read. Set `STRATA_MIRROR_MIB=0` there, - or the pinned mirror is allocated and not used. -- **`setvars.sh` and `set -u`:** `source /opt/intel/oneapi/setvars.sh` in a shell with `set -u` stops at `OCL_ICD_FILENAMES: unbound variable` - (oneAPI 2026.1.1). Source it before `set -u`, or run `set +u` around it. +- Two cards: `ONEAPI_DEVICE_SELECTOR=level_zero:*` (the image pins `level_zero:0`; `strata-sycl.sh` passes the variable + through) and `--layer-split`. +- `--layer-split auto` on two equal cards takes the split that leaves the fullest card the most VRAM to spare, within 0.2% of + the best predicted time (`STRATA_SPLIT_HEADROOM_TOL`). `STRATA_SPLIT_HEADROOM=0` turns that off. A full card under `xe` moves + buffers to system memory instead of failing. The startup line names both splits when the choice moves. +- **Arc Pro B60:** PCI id `8086:e211` (`lspci -nn`), `sycl-ls` prints `Intel(R) Arc(TM) Pro B60 Graphics 20.1.0`, so + `-DSTRATA_SYCL_AOT=bmg-g21` is the right target. `setup_intel.py` knows both B60 ids, `e211` and `e221`. On a `--layer-split` + the startup line "N experts are neither in VRAM nor mirrored" counts the other card's layers; read the following + `100% of the experts resident` lines. Set `STRATA_MIRROR_MIB=0` there, or the pinned mirror is allocated and not used. +- **`setvars.sh` and `set -u`:** `source /opt/intel/oneapi/setvars.sh` in a shell with `set -u` stops at + `OCL_ICD_FILENAMES: unbound variable` (oneAPI 2026.1.1). Source it before `set -u`. - **Without Docker:** `sycl/serve/strata-sycl.sh` needs Docker. Natively, `source setvars.sh`, export `SYCL_CACHE_PERSISTENT=0`, - `ZES_ENABLE_SYSMAN=1` and `STRATA_VERIFY_DEVICE_PLAN=1`, then run `build-sycl-aot/strata` or `sycl/serve/server_intel.py` with an `exe` that does this. - A normal user is not in the `render` group on Ubuntu, and then `sycl-ls` shows only the CPU device. -- `STRATA_VERIFY_NO_HOST=1` (set by `strata-sycl.sh` unless the config's `"env"` says otherwise) is needed on `xe` - cards when part of the experts is not in VRAM (the host-to-GPU flag stores are not visible there, NaN logits - otherwise) and is left off on the `i915` Arc A-series: see [INTEL.md](INTEL.md). The config's `"env"` block and any - `STRATA_*`, `ONEAPI_*`, `UR_*`, `IGC_*`, `SYCL_*` or `ZES_*` variable of the server's environment reach the engine - through `strata-sycl.sh` (for example `STRATA_MIRROR_MIB`, `IGC_EnableDPEmulation`). + `ZES_ENABLE_SYSMAN=1` and `STRATA_VERIFY_DEVICE_PLAN=1`, then run `build-sycl-aot/strata` or `sycl/serve/server_intel.py` + with an `exe` that does this. A normal user is not in the `render` group on Ubuntu, and then `sycl-ls` shows only the CPU. +- `STRATA_VERIFY_NO_HOST=1` (set by `strata-sycl.sh` unless the config's `"env"` says otherwise) is needed on `xe` cards when part + of the experts is not in VRAM, and is left off on the `i915` A-series ([INTEL.md](INTEL.md)). The config's `"env"` block and + any `STRATA_*`, `ONEAPI_*`, `UR_*`, `IGC_*`, `SYCL_*` or `ZES_*` variable of the server's environment reach the engine + through `strata-sycl.sh`. + +## Tested cards + +> **Measured on:** the hardware in each row, Linux. Versions are the engine version of that report, not the current release. + +| Who | Hardware | Result | Reported in | +| --- | --- | --- | --- | +| Port author | Arc Pro B70 32 GB, Ubuntu 24.04, 0.1.31-sycl | Coder IQ1_M: 70-78 tok/s decode, about 790 tok/s prompt; IQ2_XS: 51-64 tok/s; 256K context | #423, [INTEL.md](INTEL.md) | +| Community | 2x Arc Pro B70, `--layer-split` | Flash-Next IQ3_XXS 66 tok/s decode, 394 tok/s prompt | #423 | +| Community | Arc Pro B50 16 GB | Coder IQ1_M about 23 tok/s, IQ2_XS about 25-27 tok/s, up to 128K | #423 | +| Community | Arc B580 12 GB, WSL2 | IQ2_XS about 21 tok/s, Coder about 15 tok/s; device loss also seen | #423 | +| Community | 2x Arc Pro B60 24 GB, `--layer-split`, Ryzen 5 5600, 64 GB, PCIe 3.0 x8 per card | Coder IQ1_M: 56.6 tok/s decode, 428 tok/s prompt (2,129 tokens); Flash-Next IQ2_XS: 58.6-61.3 tok/s decode, 436 tok/s prompt; all experts in VRAM; 8K context | [report](../bench/results/2026-10-04-community-2x-arc-pro-b60/README.md) | +| Community | One Arc Pro B60 24 GB | Coder IQ1_M with 4,042 of 12,288 experts mirrored in RAM: 11.9 tok/s decode | same | +| Maintainers | Arc Pro B70 32 GB (xe), 0.1.40.2 and later | Flash-Next IQ3_S with 11.5k of 24.6k experts resident: 10-prompt gate passes, prompt 980-1,000 tok/s on 4K tokens, decode 30 (prose) to 41 (code) tok/s | [INTEL.md](INTEL.md#arc-pro-b70-with-a-model-that-does-not-fit-flash-next-iq3_s) | +| Maintainers | Arc A750 8 GB (i915) | Flash-Next IQ3_XXS about 10-19 tok/s decode | [INTEL.md](INTEL.md#arc-a750-and-the-other-alchemist-cards) | + +Two cards on `xe` (2x Arc Pro B70, IQ3_S, 0.1.44): `--layer-split auto` picked a split at about 11 tok/s where +`--layer-split 30` ran about 54. The auto choice now prefers the split with the most VRAM headroom (above). + +**Not tested by anyone:** Windows (no native build path), Arc on WSL2 with the current engine, integrated Arc GPUs (the B390 / +Panther Lake in #515; shared-memory planning does not exist), and pictures on a Battlemage card with a GPU reader. Pictures work +on the A750 (on by default, read on the CPU). See [INTEL.md](INTEL.md#pictures-on-an-arc). ## Windows -There is no Windows path yet. `setup --backend sycl` on Windows stops and points here. oneAPI exists for Windows, -but `sycl/CMakeLists.txt` uses GCC-style flags (`-mavx512f`, `-fp-model=precise`, `-qmkl`) and the runner is a -bash/Docker script, so a native Windows build would need work. Nobody has tried it. WSL2 with an Arc has been -used by one tester (the B580 row above), but setup cannot detect the card there, because it reads `/sys/class/drm`, -which WSL2 does not have. +There is no Windows path. `setup --backend sycl` on Windows stops and points here. oneAPI exists for Windows, but +`sycl/CMakeLists.txt` uses GCC-style flags (`-mavx512f`, `-fp-model=precise`, `-qmkl`) and the runner is a bash/Docker script. +Nobody has tried a native build. WSL2 with an Arc was used by one tester (the B580 row above), but setup cannot detect the card +there, because it reads `/sys/class/drm`, which WSL2 does not have. ## Reporting a problem -Open an issue with: the card, the driver version, `sycl-ls` output, the oneAPI version, the model and flags, and the -engine's stderr. Results from real cards are what move this from experimental to supported. +Open an issue with the card, the driver version, the `sycl-ls` output, the oneAPI version, the model and flags, and the engine's +stderr. Results from real cards are what move more cards from tested to validated. + +## History / older measurements + +The first port (0.1.39) compiled with 0 errors on Ubuntu 24.04 (WSL2) with oneAPI DPC++ 2026.1.1 and oneMKL 2026.1: the +`strata` engine plus 157 targets, as SPIR-V (JIT). Kernel parity tests on a CPU (`ONEAPI_DEVICE_SELECTOR=opencl:cpu`, Ryzen 5 +7600) passed 14 of 25, the same set as the 0.1.38 port; the failures were tight float tolerances, missing model fixtures and two +CPU timeouts. That was a compile-and-compute check, not a test of an Arc. The CUDA engine's greedy output stayed byte-identical. + +The two B60 rows ran commit `6f32ec0` plus two small `sycl/` fixes, AOT `bmg-g21`, Ubuntu 24.04, `xe`, Level Zero V2, NEO +26.09.37435.12 and oneAPI 2026.1.1, without Docker. No one ran an unpatched 0.1.39 port on an Arc; the other rows were measured on +earlier versions. diff --git a/docs/LLAMA_SWAP.md b/docs/LLAMA_SWAP.md index c5b4148c4..042f5e290 100644 --- a/docs/LLAMA_SWAP.md +++ b/docs/LLAMA_SWAP.md @@ -1,55 +1,60 @@ # Running Strata behind llama-swap -[llama-swap](https://github.com/mostlygeek/llama-swap) puts several model servers behind one address and starts the one a -request names, stopping the others to free the GPU. Strata works as one of its models. This page is what we needed to -get it running; it was tested with llama-swap's default stop behaviour, Strata 0.1.39 (Coder IQ1_M on a Tesla V100) and -clients reaching the machine by its host name. +[llama-swap](https://github.com/mostlygeek/llama-swap) puts several model servers behind one address, starts the one a +request names and stops the others to free the GPU. This page is for people who already run llama-swap and want +Strata as one of its models. It needs a [source checkout](SOURCE_INSTALL.md), not the desktop installer. -## The entry +## Quick start -Set Strata up as usual first (`./setup.sh --no-start ...`), then point llama-swap at the server directly. The start script -setup writes (`run-.sh`) fixes the port at 8080, so call `serve/server.py` with llama-swap's port instead: +1. Set up Strata without starting it: `./setup.sh --no-start`. This writes `strata-.json` in the checkout. +2. Add an entry to llama-swap's config. The start script that setup writes (`run-.sh`) fixes the port at 8080, + so call `serve/server.py` with llama-swap's port instead: ```yaml -healthCheckTimeout: 300 # Strata takes ~1.5-2 min to load; give it room +healthCheckTimeout: 300 # model loading takes minutes; give it room models: strata: cmd: >- sh -c 'cd /path/to/Strata && exec .venv/bin/python serve/server.py - --engine strata --config strata-coder-iq1_m.json --port ${PORT}' - env: ["STRATA_ALLOWED_HOSTS=my-server"] # the host name your clients use -- see below + --engine strata --config strata-.json --port ${PORT}' + env: ["STRATA_ALLOWED_HOSTS=my-server"] # the host name your clients use, see below checkEndpoint: /health ttl: 1800 ``` -- `exec` makes the Python server the process llama-swap stops. On llama-swap's stop (SIGTERM) the server stops its engine: - in our test the engine was gone and the V100's memory freed within 2 s. -- `checkEndpoint: /health` -- Strata's `/health` answers only once the model is loaded. Measured load time on our machine - (model files on a SATA SSD): 80 s, 105 s while another download was using the disk. -- If other models share the GPU, put them in a llama-swap group or let swapping unload them; Strata needs the card's - memory to itself. +3. If clients reach llama-swap by a host name, keep the `STRATA_ALLOWED_HOSTS` line and set it to that name + ([Host not allowed](#host-is-not-allowed)). -## The trap: "Host ... is not allowed" +## The entry -The first request through llama-swap failed at once with: +- `exec` makes the Python server the process llama-swap stops. On SIGTERM the server stops its engine and the + GPU memory is freed. +- `checkEndpoint: /health` works because Strata's `/health` answers only once the model is loaded. +- Strata needs the card's memory to itself. If other models share the GPU, put them in a llama-swap group or let + swapping unload them. + +## Host is not allowed + +A client that reaches the proxy by name (`http://my-server:8040`) gets: ```text 403 Host 'my-server:8040' is not allowed (DNS rebinding protection) ``` Strata only answers to the host names it knows, so a web page cannot reach it by pointing a DNS name at 127.0.0.1. -llama-swap passes the client's Host header through, so a client that reaches the proxy by name (`http://my-server:8040`) -is refused. Requests by IP address and to `localhost` always pass, which is why it can work in a quick test and fail -from another machine. Any one of these fixes it: +llama-swap passes the client's Host header through. Requests by IP address and to `localhost` always pass, so a quick +local test can work while the same setup fails from another machine. Any one of these fixes it: -- `STRATA_ALLOWED_HOSTS=my-server` in the entry's `env` (comma-separated names; ports are ignored), or -- `"allowed_hosts": ["my-server"]` in `strata-.json`, or -- an `"api_key"`, which turns the check off -- then every client must send the key. +| Fix | Notes | +| --- | --- | +| `STRATA_ALLOWED_HOSTS=my-server` in the entry's `env` | Comma-separated names; ports are ignored. | +| `"allowed_hosts": ["my-server"]` in `strata-.json` | Same effect, stored in the config. | +| An `"api_key"` in the config | Turns the host check off. Every client must then send the key. | ## Strata's own pages through llama-swap -llama-swap forwards `/upstream//...` unchanged, so Strata's status and health pages stay reachable, and calling one +llama-swap forwards `/upstream//...` unchanged. Strata's status and health pages stay reachable, and calling one also loads the model without sending a chat request: ```bash @@ -57,10 +62,25 @@ curl http://my-server:8040/upstream/strata/health curl http://my-server:8040/upstream/strata/v1/status ``` -## Claude Code and other Anthropic-API clients: the thinking level +## Claude Code and other Anthropic-API clients -Claude Code (and the Claude Agent SDK) send `"output_config": {"effort": "high"}` with every request unless told -otherwise, so Strata thinks at its highest level (seen by logging the requests). `CLAUDE_CODE_EFFORT_LEVEL=medium` (or -`low`) in the client's environment changes it. One measurement, for scale only: on one real coding task in our setup, -`medium` finished in 6.5 min against 9.7 min at `high`, both correct; `low` took 10.1 min (more trial-and-fix steps). -One run each -- not a general result. +Claude Code and the Claude Agent SDK send `"output_config": {"effort": "high"}` with every request unless told +otherwise, so Strata thinks at its highest level. Set `CLAUDE_CODE_EFFORT_LEVEL=medium` (or `low`) in the client's +environment to change it. + +## History / older measurements + +> **Measured on:** Tesla V100, Coder IQ1_M, Strata 0.1.39, llama-swap default stop behaviour, model files on a SATA SSD. + +| Item | Result | +| --- | --- | +| Model load time | 80 s; 105 s while another download used the disk | +| Engine gone and V100 memory freed after llama-swap's stop | within 2 s | + +> **Measured on:** one real coding task, Strata 0.1.39 setup, one run per level. This is not a general result. + +| `CLAUDE_CODE_EFFORT_LEVEL` | Time to a correct result | +| --- | --- | +| `medium` | 6.5 min | +| `high` | 9.7 min | +| `low` | 10.1 min (more trial-and-fix steps) | diff --git a/docs/MAC.md b/docs/MAC.md index 136bbc160..56d16318c 100644 --- a/docs/MAC.md +++ b/docs/MAC.md @@ -1,301 +1,320 @@ # Strata on a Mac (Apple Silicon, Metal) -Strata runs on Apple Silicon Macs through its own engine on Metal. This is the same engine as on a PC. The NVIDIA -kernels are replaced by Metal ones (`metal/`), and the host code and the server are the PC's. There are two ways in: +This page is for people with an Apple Silicon Mac (M1 or newer). Strata runs there on its own Metal engine: the NVIDIA kernels +are replaced by Metal ones (`metal/`), and the host code and the server are the PC's. Intel Macs cannot run it. -- **The Strata app (`.dmg`) and the Terminal (`strata launch`).** The Metal engine comes ready-made: the `.dmg` has it - inside, and the Terminal fetches it as an engine component. Nothing is built on your Mac and you need no Command Line - Tools ([below](#the-desktop-app-and-strata-launch-on-a-mac)). macOS 15 or newer, as for the source build. If macOS says the - app is "damaged", see [Troubleshooting](TROUBLESHOOTING.md#the-desktop-app-and-the-terminal); how the builds are signed: +DeepSeek V4.1-Flash does not run on a Mac yet. It runs in the Terminal on NVIDIA RTX 30 and 50 cards and on the Radeon AI PRO +R9700 ([MODELS.md](MODELS.md#deepseek-v41-flash)). + +> **On this page:** [What you need](#what-you-need) · [Install](#install) · [Defaults](#defaults-and-options) · +> [Gyro](#gyro-s-and-gyro-m-agentionais-trellis-experts) · [Measured](#measured-m5-max-128-gb) · [Limits](#limits-this-version) · +> [Switches](#switches) · [How it works](#how-it-works) · [History](#history--older-measurements) + +## Quick start + +There are two ways in: + +- **The Strata app (`.dmg`) and the Terminal (`strata launch`).** The Metal engine comes ready-made: the `.dmg` has it inside and + the Terminal fetches it as an engine component. Nothing is built on your Mac and you need no Command Line Tools. If macOS says + the app is "damaged", see [Troubleshooting](TROUBLESHOOTING.md#the-desktop-app-and-the-terminal). How the builds are signed: [packaging/mac/README.md](../packaging/mac/README.md). - **From source: `./setup.sh`.** It sets things up the way `START-HERE.bat` does on Windows and builds the engine on your Mac - (about 1-15 minutes). + (about 1 to 15 minutes). -**Pictures:** on by default in the app, the Terminal and the source build. The reader (`strata-vision`) comes in the engine package -and runs on the Mac's GPU through Metal ([Limits](#limits-this-version)). The NVIDIA, AMD and Intel engines are unchanged. +Pictures are on by default in the app, the Terminal and the source build ([Limits](#limits-this-version)). The NVIDIA, AMD and +Intel engines are unchanged. -**Status:** new. It is measured on three Macs: an M5 Max with 128 GB, where the model stays in memory; and an M4 Mac mini -with 32 GB and an M1 Max with 34 GB, where the experts stream from the SSD (the M1 Max: 7.9 tok/s decode, 186 tok/s on a -9,940-token prompt, Q2_0, the expert cache at 64 slots, as reported with the kernel fix #2348). Other chips are untested, and reports help +**Status:** new. It was measured on three Macs: an M5 Max with 128 GB, where the model stays in memory, and an M4 Mac mini with +32 GB and an M1 Max with 34 GB, where the experts stream from the SSD. Other chips are untested, and reports help ([COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md)). ## What you need -- **An Apple Silicon Mac (M1 or newer).** Intel Macs cannot run it. +- **An Apple Silicon Mac (M1 or newer).** - **macOS 15 or newer.** The engine keeps its buffers resident through a residency set, which macOS 15 introduced. -- **The Command Line Tools:** `xcode-select --install`. That is the compiler. Xcode itself is not needed. -- **Enough unified memory for the pack.** The GPU maps the model's main file and keeps it in memory while the model runs. Metal lets one - program keep about 3/4 of the memory resident (107.5 GB of 128 GB, measured on the M5 Max). +- **For a source build only:** Command Line Tools (`xcode-select --install`). +- **Enough unified memory for the pack.** The GPU maps the model's main file and keeps it in memory while the model runs. Metal + lets one program keep about 3/4 of the memory resident (107.5 GB of 128 GB on the M5 Max). -| Pack | The file the GPU maps | Needs about (with the prompt buffers, KV cache, draft layer) | Recommended from | -|---|---|---|---| +| Pack | The file the GPU maps | Needs about (with prompt buffers, KV, draft layer) | Recommended from | +| --- | --- | --- | --- | | Q2_0 | 37.6 GB | 46 GB | 64 GB | | IQ3_XXS | 47.0 GB | 55 GB | 74 GB (where IQ3_S does not fit) | | IQ3_S (recommended; the default from 96 GB) | 54.8 GB | 63 GB | 96 GB | -| UD-Q4_K_XL (Unsloth, recommended: higher quality) | 82.6 GB | 91 GB | 128 GB | -| UD-Q5_K_XL, UD-Q6_K_XL (Unsloth) | 103.9 / 114.8 GB | stream from the SSD on 128 GB | 128 GB, streamed | -| Gyro-S (agentionai, one 58.5 GB file) | 29.7 GB (the file's other 28.8 GB is the n-gram table) | 38 GB | 48 GB; streams below that (measured on 32 GB) | -| Gyro-M (agentionai, one 92.0 GB file) | 63.2 GB (+ the 28.8 GB table) | 71 GB | 96 GB; streams below that (not measured on a Mac) | +| UD-Q4_K_XL (Unsloth; higher quality) | 82.6 GB | 91 GB | 128 GB | +| UD-Q5_K_XL, UD-Q6_K_XL (Unsloth) | 103.9 / 114.8 GB | streams from the SSD on 128 GB | 128 GB, streamed | +| Gyro-S (agentionai, one 58.5 GB file) | 29.7 GB (the other 28.8 GB is the n-gram table) | 38 GB | 48 GB; streams below that | +| Gyro-M (agentionai, one 92.0 GB file) | 63.2 GB (+ the 28.8 GB table) | 71 GB | 96 GB; streams below that | -- **The n-gram table:** each pack also has a 28.8 GB n-gram table (the model's second file). It is read from the SSD a few rows at a time and does not count against the memory above. -- **Disk:** the download is 66-84 GB, plus about 7 GB for the pack and the draft layer. A Mac that streams also needs room for the experts' copy (below): 34-40 GB. -- **A Mac whose memory cannot hold the pack (32-48 GB):** the engine streams the experts from the SSD instead. It decides this by itself when the experts are more than 60% of Metal's share. - - It keeps the most-used experts in an arena: Metal's share, or the memory macOS reports as available less what it keeps for macOS, whichever is smaller. It keeps 7 GB, or 15% of the RAM when that is more, and it also keeps the n-gram table's share. It counts as free only what macOS can give without swapping (free and purgeable memory and the clean file cache, not other programs' idle memory). It checks again just before it takes the arena, and fills the arena in 1.5 GB steps, checking again before each one; it stops early rather than push macOS below what it keeps. On a Mac with 64 GB or more the arena, the Metal buffers and the host memory together also stay under 60% of the RAM (see "The engine's whole anonymous footprint" below). - - It reads the rest as each token needs them, and reads ahead the experts the next layer will probably use. - - Setup writes a second copy of the experts for such a Mac (`.experts`, 40 GB for IQ3_XXS). In it each expert's three matrices sit together, so the SSD reads an expert with one request instead of three. On the 32 GB M4 that is about 17% faster decoding and 30% faster prompts, with the same tokens. Setup writes it only when the disk has room for it plus 10 GB; otherwise it warns, and the engine reads the model file as before. - - Setup configures such a Mac without the MTP draft layer: one token per step is faster there. - - Measured at 32 GB (below). Smaller Macs are untested. -- **Setup recommends, and never forces.** It recommends the pack by memory; a bigger pack chosen with `--model` is kept, with a warning about what it risks. +- **The n-gram table.** Each pack also has a 28.8 GB n-gram table (the model's second file). It is read from the SSD a few rows at + a time and does not count against the memory above. +- **Disk.** The original Q2_0 and IQ3 packs download roughly 66 to 84 GB, plus preparation and draft data. Unsloth packs are much + larger and Gyro uses different files. A streamed model can also need a separate experts file (34 to 40 GB for the examples + below). Check the selected pack rather than treating these sizes as a universal limit. +- **Setup recommends and never forces.** It recommends the pack by memory. A bigger pack chosen with `--model` is kept, with a + warning about what it risks. + +### A Mac whose memory cannot hold the pack (32 to 48 GB) + +The engine streams the experts from the SSD. It decides this itself when the experts are more than 60% of Metal's share. + +- It keeps the most-used experts in an arena and reads the rest as each token needs them, with read-ahead for the next layer's + likely experts. The arena is Metal's share, or the memory macOS reports as available less what it keeps for macOS (7 GB, or 15% + of the RAM when that is more), whichever is smaller. It counts as free only what macOS can give without swapping, fills the + arena in 1.5 GB steps and stops early rather than push macOS below its reserve. +- On a Mac with 64 GB or more, the arena, the Metal buffers and the host memory together stay under 60% of the RAM, and the + filled arena is wired (`mlock`) so macOS does not compress it while the GPU is idle. `STRATA_METAL_WIRE_ARENA=0` turns the + wiring off. `STRATA_METAL_ARENA_GB` sets the arena yourself, with a warning when it is over the cap. The engine prints the + planned footprint (`planned resident footprint ... cap ...`) and the footprint after the fill. +- Setup writes a second copy of the experts (`.experts`, 40 GB for IQ3_XXS) in which each expert's three matrices sit + together, so the SSD reads an expert with one request. On the 32 GB M4 that is about 17% faster decoding and 30% faster prompts + with the same tokens. Setup writes it only when the disk has room for it plus 10 GB, otherwise it warns, and it does not write + it on a Mac with 96 GB or more. +- Setup configures such a Mac without the draft layer (one token per step is faster there). A streaming Mac with 96 GB or more + keeps the draft layer and writes `--spec 3`. +- Prompts stream in chunks of 4096 on a streaming Mac with 96 GB or more, and 1024 on a smaller one. `STRATA_METAL_PROMPT_CHUNK=N` + sets it. +- The model's file must be on the internal SSD. An external disk is untested and likely much slower. ## Install -``` +For everyday use follow [the app or Terminal installation](INSTALL.md). The commands below are the source installation, which +builds the Metal engine locally. + +```sh git clone https://github.com/Niko1221/Strata && cd Strata ./setup.sh # asks the size; --yes takes the recommended answers ``` What the first run does (each step is skipped when it is already done): -1. **Python.** It needs Python 3.10 or newer, but macOS ships 3.9. If that is all your Mac has, `setup.sh` downloads a standalone Python 3.12 (25 MB, pinned by SHA-256) into `.python`. No sudo, no Homebrew. +1. **Python.** It needs Python 3.10 or newer, but macOS ships 3.9. If that is all your Mac has, `setup.sh` downloads a standalone + Python 3.12 (25 MB, pinned by SHA-256) into `.python`. No sudo, no Homebrew. 2. **Checks your Mac:** the chip, the memory, the macOS version, the Command Line Tools and the free disk. -3. **Python packages** into `.venv`. CMake and Ninja come from pip too, installed just before the build in step 4. -4. **Builds the engine** (`build-metal/strata`). It uses Apple's metal-cpp headers (pinned by SHA-256) and llama.cpp's ggml at the commit the PC build pins. +3. **Python packages** into `.venv`. CMake and Ninja come from pip too, installed just before the build. +4. **Builds the engine** (`build-metal/strata`) with Apple's metal-cpp headers (pinned by SHA-256) and llama.cpp's ggml at the + commit the PC build pins. 5. **Downloads the model.** The download resumes if interrupted. `--gguf-dir` uses files you already have. -6. **Prepares the model:** the pack (seconds), and the MTP draft layer (about 5 GB of the original checkpoint). On a Mac that streams: the experts' copy instead of the draft layer (about 3 minutes on the M4, checked byte for byte against the model file). +6. **Prepares the model:** the pack (seconds) and the MTP draft layer (about 5 GB of the original checkpoint). On a Mac that + streams, the experts' copy takes the draft layer's place (about 3 minutes on the M4). 7. **Writes `strata-.json` and `run-.sh`, then starts the server.** -The config uses the Mac defaults: `--spec 2` and the Q8_0 hyper-connection read (below). After that, `./setup.sh` or `./run-q2_0.sh` starts the model directly. The server is the same as on a PC: -- the OpenAI-compatible API on `http://127.0.0.1:8080/v1`; -- the Anthropic API on `/v1/messages`; -- the chat page on `http://127.0.0.1:8080/`. +After that, `./setup.sh` or `./run-q2_0.sh` starts the model directly. The server is the same as on a PC: the OpenAI-compatible +API on `http://127.0.0.1:8080/v1`, the Anthropic API on `/v1/messages`, and the chat page on `http://127.0.0.1:8080/`. -Options: `--model Q2_0|IQ3_XXS|IQ3_S|UD-Q4_K_XL|UD-Q5_K_XL|UD-Q6_K_XL` (the UD files are Unsloth's), `--vision yes|no`, `--context N`, `--port`, `--host` / `--api-key`, `--draft-vocab`, `--no-browser`, -`--data-dir`, `--models-dir`, `--gguf-dir`, `--setup` (another model or new settings), `--no-start`, `--check` (only -check the Mac), `--build` (build the engine again). +| Option | Meaning | +| --- | --- | +| `--model Q2_0\|IQ3_XXS\|IQ3_S\|UD-Q4_K_XL\|UD-Q5_K_XL\|UD-Q6_K_XL\|Gyro-S\|Gyro-M` | The pack. The `UD-` files are Unsloth's. | +| `--family qwen\|unsloth\|gyro` | The model family. | +| `--vision yes\|no\|gpu\|cpu` | Picture reader. | +| `--context N`, `--port`, `--host`, `--api-key` | Server settings. | +| `--draft-vocab`, `--no-browser` | Draft vocabulary; do not open the browser. | +| `--data-dir`, `--models-dir`, `--gguf-dir` | Where files go, or where yours are. | +| `--setup`, `--no-start`, `--check`, `--build` | Choose another model or settings; install only; only check the Mac; build the engine again. | ### The desktop app and `strata launch` on a Mac -The Strata app (and the Terminal's `strata launch`) do not build anything on your Mac. The Metal engine comes ready-made, as an engine component: from the update feed, or inside the app's installer. The model install is `setup.py` in its Control mode (`STRATA_CONTROL_INSTALL=1`): it downloads (or uses with `--gguf-dir`) and prepares the model files and writes a run config whose engine is `component:strata-engine`. It does not run `metal/setup_mac.py`, needs no Command Line Tools, and writes the same Metal flags (`--mmap-experts`, `--spec 2`, no draft layer when the experts stream from the SSD). +The app and `strata launch` build nothing. The Metal engine comes ready-made as an engine component, from the update feed or +inside the app's installer. The model install is `setup.py` in its Control mode (`STRATA_CONTROL_INSTALL=1`). It downloads, or +uses with `--gguf-dir`, and prepares the model files, and writes a run config whose engine is `component:strata-engine`. It does +not run `metal/setup_mac.py`, needs no Command Line Tools, and writes the same Metal flags. -The model list and the recommendation judge a Mac by its unified memory with the rules of `strata_control/mac_fit.py`, the module `setup_mac.py` uses too: Q2_0, IQ3_XXS and IQ3_S are offered (the Unsloth many-shard files need the repack of `./setup.sh` for now); a pack that does not stay in memory is listed as "read from the SSD", and a Mac under 16 GB of unified memory is told the model does not fit. +The model list and the recommendation judge a Mac by its unified memory with the rules of `strata_control/mac_fit.py`, the module +`setup_mac.py` uses too. Q2_0, IQ3_XXS, IQ3_S and the Gyro packs are offered. Unsloth's many-shard files need the source +installer's repack for now. A pack that does not stay in memory is listed as "read from the SSD", and a Mac under 16 GB of unified +memory is told the model does not fit. -## Measured (M5 Max, 128 GB, macOS 27.2) +The ready-made Mac engine is in the app and in the Terminal's engine component, not in the release zips. The zips (for +`START-HERE.bat` and `setup.sh`) have no Mac engine: `./setup.sh` builds it. -**Decode**, in tokens per second over 200 generated tokens, with greedy answers. The table gives two prompts: a short question -("prompt1") and a story. +## Defaults and options + +The config uses the Mac defaults: `--spec 2` with MTP (best on mixed text; an explicit `--spec N` in the config's `args` is kept) +and the Q8_0 hyper-connection read. `STRATA_HC_Q8=0` in the config's `env` gives the exact BF16 read. The Mac engine is +deterministic: two runs give identical numbers. + +### Gyro-S and Gyro-M (agentionai's trellis experts) + +The Mac engine runs [Gyro](GYRO.md). The routed experts are trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a +Hadamard-rotated basis, read from the published file unchanged. The kernels are in `metal/kernels/quant.metal`, `moe.metal` and +`moe_mm.metal`. Setup offers Gyro-S and Gyro-M next to the other sizes (`./setup.sh --family gyro --model Gyro-S`), never as the +default, and says what the choice risks on your Mac's memory. The 28.8 GB n-gram table is inside the file and is read from the SSD +like every other pack's, so Gyro-S needs 58.5 GB of disk and about 38 GB of memory to stay resident. + +Gyro-M runs on the same kernels (checked tensor by tensor) but no Mac tested here could hold it. `metal/tests/gyro_parity.sh` +checks the decoder against the format definition and the expert kernels against a double-precision reference. +`STRATA_METAL_GYRO_TILES=0` turns off the dense-weight tiles. + +## Measured (M5 Max, 128 GB) + +**Measured on:** Apple M5 Max, 128 GB, macOS 27.2, Qwen3.8-Flash-Next packs, greedy, 200 generated tokens, Strata engine version +not recorded. Decode tok/s for a short question and a story ("prompt1 / story"). | | Q2_0 | IQ3_XXS | IQ3_S | -|---|---|---|---| -| Setup's defaults (`--spec 2` with MTP, the suffix drafter on) | **119.5 / 108.2** | **108.9 / 97.0** | - | +| --- | --- | --- | --- | +| Setup's defaults (`--spec 2` with MTP, suffix drafter on) | **119.5 / 108.2** | **108.9 / 97.0** | - | | `--spec 2` with MTP, suffix drafter off | 114.0 / 107.9 | 103.5 / 96.9 | 98.2 / 94.1 | | No MTP (one token per step) | 87.4 / 87.3 | 78.7 / 78.6 | 74.5 / 74.3 | -The output with MTP is the same as without it, token for token, in every run. MTP only proposes tokens, and the -full model checks every one of them. +The output with MTP is the same as without it, token for token. MTP only proposes tokens, and the full model checks each one. -**Prompt reading** (time to the first token), on the needle-test prompts below: +Prompt reading (time to first token), on the needle-test prompts: | | 4K tokens | 32K tokens | -|---|---|---| -| Q2_0 | ~1,430 tok/s | ~1,460 tok/s | -| IQ3_XXS | ~1,260 tok/s | ~1,240 tok/s | -| IQ3_S | ~1,320 tok/s | ~1,140-1,190 tok/s | +| --- | --- | --- | +| Q2_0 | about 1,430 tok/s | about 1,460 tok/s | +| IQ3_XXS | about 1,260 tok/s | about 1,240 tok/s | +| IQ3_S | about 1,320 tok/s | about 1,140-1,190 tok/s | -The first prompt after the Mac starts is slower, while macOS reads the model file (IQ3_S: 413 tok/s on the first 4K -prompt, cold). +The first prompt after the Mac starts is slower while macOS reads the model file (IQ3_S: 413 tok/s on the first 4K prompt). +llama.cpp on the same Mac with Q2_0: 56.6 tok/s decode on the story, 887 / 809 tok/s on 4K / 32K prompts. -**For comparison**, llama.cpp on the same Mac with Q2_0: 56.6 tok/s decode on the story, and 887 / 809 tok/s on 4K / 32K prompts. - -### Unsloth's UD-Q4/Q5/Q6_K_XL on the M5 Max - -IQ3_S and UD-Q4_K_XL are the recommended sizes. Setup offers IQ3_S as the default and names UD-Q4_K_XL as the -higher-quality choice where it fits. Decode is tokens per second over 200 tokens (greedy, MTP); prompts are the warm rate -of the second request on (the first request after the start pays a few seconds of first-use cost). +**Unsloth packs on the same Mac.** IQ3_S and UD-Q4_K_XL are the recommended sizes. Decode is tok/s over 200 tokens (greedy, MTP); +prompts are the warm rate from the second request on. | | IQ3_S | UD-Q4_K_XL | UD-Q5_K_XL (streamed) | UD-Q6_K_XL (streamed) | -|---|---|---|---|---| +| --- | --- | --- | --- | --- | | Quality against Q8_0 (top-1 / KL) | 87.9% / 0.142 | 92.8% / 0.055 | not measured | not measured | | Decode, short question, no MTP / MTP | 75.4 / 101.9 | 60.7 / 94.0 | - | - | | Decode, story, no MTP / MTP | 74.8 / 94.7 | 59.7 / 73.7 | 28 / 46-51 (`--spec 3`) | not measured / 47-51 (`--spec 3`) | -| Prompt 4K / 32K | ~1,180 / ~1,290 | ~1,150 / ~1,290 | ~1,070 / ~1,110 | ~1,020 / ~1,020 | +| Prompt 4K / 32K | about 1,180 / 1,290 | about 1,150 / 1,290 | about 1,070 / 1,110 | about 1,020 / 1,020 | | llama.cpp Metal, decode (story / code) | 51.1 / 44.1 | 48.4 / 47.3 | 45.1 / 43.2 | does not run (Insufficient Memory) | -| llama.cpp Metal, prompt 4K / 32K | 791 / 730 | 821 / 739 | 729 / 693 | - | -- **UD-Q5 and UD-Q6 cannot stay resident:** their repacked file (103.9 and 114.8 GB) is bigger than the largest buffer Metal gives (86.6 GB), so the experts stream from the SSD into an arena of about 65-67 GB on the 128 GB M5 Max (about 68% of Q5's experts, 62% of Q6's; it was 80-86 GB before the footprint cap below, and the decode and prompt speeds did not change: Q6 story 51 tok/s, 15K prompt 1,020 tok/s at both sizes). -- **The cost is a fixed hand-shake per layer** (the engine reads the router's choice and makes sure the experts are in the arena), not the SSD. A window of 2 tokens costs 38 ms against ~17 resident, so a longer window pays: Q5 story 28 tok/s without MTP, 41.8 with `--spec 2`, 50.6 with `--spec 3`, 46 with `--spec 4`. Setup writes `--spec 3` and keeps the MTP draft layer for a streaming Mac with 96 GB or more. -- **The engine's whole anonymous footprint is limited to 60% of the RAM on a Mac with 64 GB or more (82.4 GB of the M5 Max's 137.4 GB): the arena, every Metal buffer and the host memory the process holds, not only the Metal buffers.** macOS keeps a floor of file cache (`vm.vm_page_filecache_min`: 37.2 GB here) and compresses anonymous memory instead of dropping cache below it. The arena is anonymous, and the GPU wires it only while a command buffer runs, so while it is filled it is ordinary pageable memory. Two kernel panics on the M5 Max (the watchdog daemon starved for 92 s) had the engine at 95 and 90 GiB resident and the compressor at 2-4 million pages. Measured again with a memory watchdog every second: an old-build arena of 100 GB compressed 19 GB of itself in 7 seconds and swapped 24,000 pages; the old automatic arena (69 GB) with a 40 GB file write and a 16 GB program beside it took the compressor from 21,000 to 4.07 million pages in 10 seconds (pressure "warn"). With 30% left (footprint 94.5 GB) and the file write alone, 12,000 pages were swapped out; with 35% left, 316; with 40% left and the write and the 16 GB program, none and the compressor at 0.3 million pages. The old rule (arena + Metal buffers under 67%) counted 7.5 GB of buffers where the process held 11-15 GB, so its real total was about 100 GB. A resident pack is different (clean file pages that macOS drops, never compresses): UD-Q4_K_XL resident (wired 91.7 GB) took the same 40 GB write and 16 GB program with no swap-out and nothing compressed, so resident packs are limited to 70% of the RAM as before. The arena is also wired (`mlock`) once it is filled: unwired it is ordinary pageable memory whenever the GPU is idle (the GPU's own wiring ends a few seconds after its last command buffer; a Metal residency set did not hold it), and a 20 GB file written beside the idle engine took the arena into the compressor and froze the Mac (a third watchdog panic, in a control run of this work). Wired, the same write at full speed raised the compressor by 0.7 million pages (the engine's non-arena memory), swapped out 408 pages and changed nothing else. `STRATA_METAL_WIRE_ARENA=0` turns it off; it is on for Macs with 64 GB or more. The engine prints the planned total (`planned resident footprint ... cap ...`) and the footprint after the fill. `STRATA_METAL_ARENA_GB` still sets the arena as you say, with a warning when it is over the cap. -- **The experts' copy** (`.experts`, 98 GB for Q5) did not pay on the M5 Max: decode 50.3 against 50.2 tok/s without it, 4K prompt 741 against 685 (noisy). Setup does not write it on a Mac with 96 GB or more. -- **Prompts stream in chunks of 4096** on a streaming Mac with 96 GB or more (the engine's own default there; 1024 on a smaller one): every chunk makes one pass over the experts that are not in the arena, so a longer chunk reads them fewer times. UD-Q5, warm, 2 interleaved rounds: chunks of 1024 gave 700 tok/s on the 4K prompt, 2048 gave 920 and 4096 gave 1,100 (32K: 710, 970, 1,100); the decode did not change. `STRATA_METAL_PROMPT_CHUNK=N` sets it. On a resident model 4096 instead of 2048 gained only 2-4% (Q2_0), so that default stays. +UD-Q5 and UD-Q6 cannot stay resident: their repacked file (103.9 and 114.8 GB) is bigger than the largest buffer Metal gives +(86.6 GB). The experts stream from the SSD into an arena of about 65 to 67 GB. The cost is a fixed hand-shake per layer, not the +SSD, so a longer window pays (Q5 story: 28 tok/s without MTP, 41.8 with `--spec 2`, 50.6 with `--spec 3`). -### M4 Mac mini, 32 GB, experts streamed from the SSD - -The arena was 11.5-14.7 GB. Tokens per second, one token per step (no MTP), medians of 5 interleaved runs: +**Measured on:** Apple M4 Mac mini, 32 GB, experts streamed from the SSD (arena 11.5 to 14.7 GB), one token per step (no MTP), +medians of 5 interleaved runs, tok/s. | | Q2_0 | IQ3_XXS | llama.cpp, CPU only (Q2_0 / IQ3_XXS) | -|---|---|---|---| +| --- | --- | --- | --- | | Decode, short question (200 tokens) | **14.1** | **10.0** | 6.3 / 4.1 | | Decode, story (200 tokens) | **13.5** | **9.2** | 6.3 / 4.1 | | Decode after a 6K-token document | **13.1** | **7.7** | 5.1 / 2.8 | | Prompt, 4K tokens | **93.6** | **65.8** | 21.2 / 8.7 | -- The greedy answers equal the M5 Max's (model in memory) on 10 of 10 test prompts, for both packs. -- No memory was swapped out during the runs; the least memory left to macOS was 7.1 GB. -- MTP was slower here (Q2_0 story 12.1-12.3 with it against 12.3-12.8 without). The draft layer's 1.2 GB do more as expert cache. +The greedy answers equal the M5 Max's (model in memory) on 10 of 10 test prompts. MTP was slower here, because the draft layer's +1.2 GB do more as expert cache. The arena's size matters most: with 5.7 to 9.3 GB (a Mac with less memory free) the same M4 read +about 8.3 tok/s on a story. On an M1 Max with 34 GB (Q2_0, expert cache at 64 slots, kernel fix #2348) decode was 7.9 tok/s and a 9,940-token prompt read at 186 tok/s. -**The experts' copy** (`.experts`, one SSD request per expert): IQ3_XXS after the 6K-token document, the arena held at 12 GB, medians of 5 interleaved runs each: +**Measured on:** Apple M4 Mac mini, 32 GB, Gyro-S, experts streamed (arena 11 to 12 GB of the 25.8 GB of experts), no experts' copy, +one token per step, no draft layer. Decode story, 120 tokens, 5 interleaved pairs, medians: **9.14 tok/s** (8.70 with the +dense-weight tiles off). Prompt of 4,065 tokens: **109 tok/s**. The engine held 18.6 GB of the 32 GB. Against the CUDA engine on +the same Gyro-S file (5 text segments, 4,995 positions, RTX 3060) the Mac is as far from either CUDA run as the two CUDA runs are +from each other: top-1 agreement 95.4% against 95.8%, mean KL 0.0207 against 0.0186. -| | Decode | The 6K prompt | SSD wait per token | -|---|---|---|---| -| The model file (three reads per expert) | 7.37 tok/s | 64.7 tok/s | 46.6 ms | -| The experts' copy (one read per expert) | **8.62 tok/s** | **84.1 tok/s** | 31.5 ms | +## Limits (this version) -- The tokens are identical in all 10 runs. -- The 4K-token prompt (3 interleaved runs each, default arena): 72.9 tok/s with the copy against 62.5 with the model file (+17%), the same tokens. -- 32K-token prompts through the server, with the copy: 29,768 tokens at 89.2 tok/s (5.6 minutes to the first token), 33,803 tokens at 84.4 tok/s. The code word was found both times. -- A 34-minute server run with the copy (47 requests: 32K and 6K needles, short questions, a story, code; context 34,816) got every answer right and did not slow down over time. Decode was 7.2-7.7 tok/s on short answers and 9.9-10.4 on the story; the 6K prompts ran at 80-90 tok/s. macOS's memory pressure was never critical: one 2-second "warn" sample while the arena filled at the start. About 90 MB were swapped out in those same 2 seconds and nothing after; three restarts afterwards (smaller arenas, less memory free) swapped nothing. -- The SSD alone measures the same: 2.97 GB/s for one 1.75 MB read per expert against 2.30 GB/s for three 0.58 MB reads (6 threads, uncached). -- The arena's size matters most. With the model file, a 7.5 GB arena gave 5.5 tok/s, 10.5 GB gave 6.7 and 12 GB gave 7.5. The read-ahead depth, the I/O threads and the eviction order each moved decode by less than 3%. -- The Q2_0 copy is not measured yet. The numbers in the table above are with the model file. -- **Measured again on the release build, with less memory free:** the M4 had 13-17 GB available when the engine started (another user's programs held the rest), so the arena was 5.7 GB (Q2_0) and 9.3 GB (IQ3_XXS, with the experts' copy). One token per step, medians of 3: Q2_0 story 8.3 tok/s, 4K prompt 80; IQ3_XXS story 8.3, 4K prompt 89. That is the arena's size effect above, not a regression: with a 6.6 GB arena the engine used to stop at the first prompt ("no free slot"); a small arena now gives the prompt's 1.6 GB of buffers their own allocation, also when no chunk of the arena is big enough for them (on a 24 GB M2 whose arena shrank below the prompt's needs, every prompt stopped with "no arena region for the prompt buffers"), and a 26-minute server run with a 5.4 GB arena (29 requests, 6K and 31K needles found) was clean. -- With the default arena (12.1-12.2 GB here), the experts' copy gives 8.35-8.48 tok/s decode and 82.5-82.8 tok/s on the prompt. Nothing was swapped out. macOS's memory pressure stayed "normal", except for one 2-second "warn" sample while the arena fills; keeping 7 GB for macOS shows the same sample. -- Those runs and the server run below kept 5 GB for macOS. The default is now 7 GB with the stepped fill: after the 6K document, 7.3 and 6.7 tok/s decode (arenas 9.3 and 7.6 GB, as much as was free), the same tokens, the memory pressure "normal" throughout and nothing swapped out. +- **Images (vision):** on by default. The app and the Terminal use the `strata-vision` that sits beside `strata` in the engine + package; `./setup.sh` builds it and asks "Should the model also read images?" (default yes). With `--gpu` it reads on the Mac's + GPU (ggml's Metal backend), without it on the CPU. A Mac always gets the GPU reader, whatever its memory. The footprint is about + 1.18 GB on the GPU and 1.05 GB on the CPU. `--vision cpu` is still allowed and prints a note. The CPU reader is slow because + ggml has no vector kernel for BF16 on ARM. If the GPU reader cannot start, the server prints one line and starts it on the CPU. + Checked on one Mac mini M4 only; the M5 Max is untested for pictures. -**Why `--spec 2`:** measured with MTP on the same two prompts. + **Measured on:** Mac mini M4, 32 GB, Qwen3.8-Flash-Next BF16 projector. -| Window | Q2_0 (prompt1 / story) | IQ3_XXS (prompt1 / story) | -|---|---|---| + | Picture | GPU reader | CPU reader | + | --- | --- | --- | + | 260 tokens (640x400) | 0.45 s | 50.7 s | + | 972 tokens (1600x1200) | 2.4 s | 194 s | + +- **Context:** measured up to 32K (needles, KL). Setup recommends 32K below 96 GB and 64K from 96 GB. Up to the model's 262K can + be set, but it is not measured. +- **One request at a time.** `--parallel` (batch slots) is not ported. Several clients queue. +- **Not ported:** the control vector (`--control-vector-scaled`) and the PC tuning (`--calibrate`). +- **Streaming** is measured at 32 GB only. +- **Families:** the original Qwen3.8-Flash-Next, Unsloth's UD-Q4/Q5/Q6_K_XL and agentionai's Gyro-S are measured. Setup repacks + Unsloth's shards into one file for the GPU; a Gyro file is used as it is. The Swift and Coder releases are not set up on Mac + yet, and Qwen3.6 does not run on the Mac engine. + +### Quality + +- **The Q8_0 hyper-connection read** is the default. Teacher-forced over 28 prompts (code, reasoning, chat, six languages, + documents), about 6,000 positions per set, BF16 reference: mean KL 0.0007 and top-1 agreement 98.9% on the model's own text, + 99.1% on another model's text, perplexity unchanged. For scale, IQ3_XXS against Q2_0 is a KL of 0.094. +- **The fast prompt path** against reading the prompt through decode windows: mean KL 0.0004, top-1 99.4 to 99.6%. +- **Needles:** a code word hidden in 8K, 16K and 32K tokens at 10/50/90% depth was found 9 of 9 with Q2_0 and with IQ3_XXS. +- **Same answers as on a PC:** against the CUDA engine on a Tesla P100 (IQ3_XXS and IQ3_S) the first 200 tokens of a reply on the + short question are identical; against the HIP engine on a Radeon AI PRO R9700 (Q2_0), 36 of 39 first tokens are identical. The + differences are near-ties between the two best tokens, from a different floating-point order. + +## Switches + +For testing. All are off unless set; the defaults are described above. + +| Variable | Effect | +| --- | --- | +| `STRATA_HC_Q8=0` | The BF16 hyper-connection read (exact) instead of Q8_0 | +| `STRATA_METAL_PROMPT=window` | Read the prompt through decode windows (slow, exact fallback; about 190 tok/s) | +| `STRATA_METAL_PROMPT_CHUNK=N` | Prompt chunk size (default 2048 resident; see streaming above) | +| `STRATA_METAL_MTP_Q8=0` | The draft layer's matrices in BF16 instead of Q8_0 | +| `STRATA_METAL_MTP_CHAIN=0` | One GPU submission per draft step instead of one per round | +| `STRATA_METAL_FUSE=0`, `STRATA_METAL_LAZY_FOLD=0` | Turn off two decode fusions (same tokens, slower) | +| `STRATA_METAL_PARTIAL=1`, `STRATA_METAL_ROUTE=1` | Two decode variants that measured no faster. `ROUTE` is also wrong on an M1 Max (stale routing logits) | +| `STRATA_METAL_PROMPT_EXPERTS=v3` | The M5 layout for the prompt's expert GEMMs. Fails on an M1 Max, so it is off there (default v2) | +| `STRATA_METAL_STREAM=1` / `=0` | Stream the experts from the SSD even when the pack fits / never (the default decides by size) | +| `STRATA_METAL_CONTIG=0` | Streaming: read the model file even when the experts' copy is there | +| `STRATA_METAL_ARENA_GB=N`, `STRATA_METAL_KEEP_FREE_GB=N` | Streaming: the arena's size, or the memory left to macOS | +| `STRATA_METAL_LEAVE_SHARE=0.4` | Streaming, 64 GB or more: the share of the RAM left to macOS when the footprint cap is set | +| `STRATA_METAL_WIRE_ARENA=0` | Streaming, 64 GB or more: leave the filled arena pageable (default: wired) | +| `STRATA_METAL_IGNORE_REPACK=1` | Start although a model repack or setup's preparation is running (the engine refuses by default) | +| `STRATA_METAL_KEEP_HOST_WEIGHTS=1` | Streaming: keep the host code's own copies of the weights (2 to 3 GB) | +| `STRATA_METAL_TRACE=1`, `STRATA_METAL_PROFILE=1`, `STRATA_METAL_KPROF=T` | Per-window traces, a time breakdown, per-kernel GPU times | + +## How it works + +The code is in `metal/` (the engine's side in `metal/src/engine`, the kernels in `metal/kernels`). + +- **Host code:** the PC host code compiles unchanged for macOS. `metal/include/metal_compat` maps the CUDA runtime calls it makes + onto a small Metal runtime (`metal/src/gpu`). +- **Decode and prompt work** run in Metal kernels: one command buffer per verify window, experts read in place from the mapped + model file, hyper-connections, GDN state and QSA attention on the GPU, and a whole MTP draft round in one command buffer. +- **The PC builds are byte-identical** with this code present. Every shared file's change is behind `STRATA_GPU_METAL` or `__APPLE__`. + +## History / older measurements + +**Why `--spec 2`.** Measured on: Apple M5 Max, 128 GB, decode tok/s with MTP on the two prompts (prompt1 / story). + +| Window | Q2_0 | IQ3_XXS | +| --- | --- | --- | | `--spec 2` | 111 / 106 | 102 / 95 | | `--spec 3` | 116 / 92 | 108 / 95 | | `--spec 4` | 111 / 81 | 108 / 83 | | `--spec 5-6` | slower | slower | -`--spec 2` is the best default on mixed text. An explicit `--spec N` in the config's `args` is kept. +**The experts' copy.** Measured on: Apple M4 Mac mini, 32 GB, IQ3_XXS after a 6K-token document, arena held at 12 GB, medians of 5 +interleaved runs, identical tokens in all 10 runs. -### Gyro-S and Gyro-M (agentionai's trellis experts) +| | Decode | The 6K prompt | SSD wait per token | +| --- | --- | --- | --- | +| The model file (three reads per expert) | 7.37 tok/s | 64.7 tok/s | 46.6 ms | +| The experts' copy (one read per expert) | **8.62 tok/s** | **84.1 tok/s** | 31.5 ms | -From 0.2.2 the Mac engine runs [Gyro](GYRO.md): the routed experts are trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a -Hadamard-rotated basis, read from the published file unchanged. The kernels are ours (`metal/kernels/quant.metal`, `moe.metal`, -`moe_mm.metal`); the codebook and the decoder are the CPU engine's, the rotation is applied to the gate/up and down inputs as on CUDA. -Setup offers Gyro-S and Gyro-M next to the other sizes (`./setup.sh --family gyro --model Gyro-S`), never as the default, and says -what the choice risks on your Mac's memory. The 28.8 GB n-gram table is inside the file and is read from the SSD like every other -pack's, so Gyro-S needs 58.5 GB of disk and about 38 GB of memory to stay resident (29.7 GB of weights). +The 4K prompt read 72.9 tok/s with the copy against 62.5 without. A 34-minute server run with the copy (47 requests, context +34,816) got every answer right and did not slow down. With the model file, an arena of 7.5 GB gave 5.5 tok/s, 10.5 GB gave 6.7 and +12 GB gave 7.5. On the 128 GB M5 Max the copy for Q5 did not pay (decode 50.3 against 50.2 tok/s). The SSD alone measures 2.97 GB/s +for one 1.75 MB read per expert against 2.30 GB/s for three 0.58 MB reads. The Q2_0 copy was not measured. -**Measured on the M4 Mac mini, 32 GB** (the experts stream from the SSD; no experts' copy, because the copy would need 26 GB more -disk than this Mac had; one token per step, no draft layer; the arena was 11-12 GB of the 25.8 GB of experts): +**Prompt chunk on a streaming Mac.** Measured on: M5 Max, UD-Q5, warm, 2 interleaved rounds. Chunks of 1024, 2048 and 4096 gave 700, +920 and 1,100 tok/s on the 4K prompt (32K: 710, 970, 1,100). Decode did not change. On a resident model 4096 instead of 2048 gained +only 2 to 4% (Q2_0), so that default stays. -| | Gyro-S | -|---|---| -| Decode, story, 120 tokens (5 interleaved pairs, medians) | **9.14 tok/s** (9.07-9.15) | -| Decode, with the dense-weight tiles turned off (`STRATA_METAL_GYRO_TILES=0`) | 8.70 tok/s (8.69-8.73); same tokens in all 10 runs | -| Prompt, 4,065 tokens (3 runs) | **109 tok/s** (108-110) | -| Memory the engine held | 18.6 GB of the 32 GB | +**Memory footprint cap.** Measured on: M5 Max, 128 GB. Before the cap, a streaming arena of 100 GB compressed 19 GB of itself in 7 +seconds, and the kernel watchdog panicked three times. With 40% of the RAM left to macOS and the arena wired, a 40 GB file write +beside a 16 GB program caused no swap-out. A resident pack (UD-Q4_K_XL, wired 91.7 GB) took the same load with nothing compressed, +so resident packs are limited to 70% of the RAM. -Where the time goes: a decode window of two tokens takes 111 ms. The routed experts are about 40 ms of it, the dense weights -(Gyro keeps its attention, state-space and shared-expert matrices in Q5_K and Q6_K: 3.6 GB read a token) -about 66 ms before the tile change (a run with those kernels skipped). The kernel for one expert layer (ten experts, one token) takes -278 us on the M4, 37.5 GB/s; IQ3_XXS takes 333 us and Q2_0 166 us. The trellis decode is limited by latency and ALU, not by memory: -32-bit funnel shifts for the stream bits, fetching the next unit while decoding the current one, one lane a row with the unit index -folded away and the codebook in constant memory were all tried and none beat the plain kernel, so it is the plain kernel with the -tile that measured fastest. The prompt kernels (fp16 matrix units) are on par with IQ3_XXS: 68 ms against 62 ms a layer for -1,024 tokens. +**Gyro-S kernel profile.** Measured on: M4 Mac mini, 32 GB. A decode window of two tokens takes 111 ms: the routed experts about +40 ms, the dense weights (Q5_K and Q6_K, 3.6 GB read a token) about 66 ms before the tile change. The kernel for one expert layer +(ten experts, one token) takes 278 us at 37.5 GB/s; IQ3_XXS takes 333 us and Q2_0 166 us. The trellis decode is limited by latency +and ALU, not by memory. The prompt kernels run 68 ms a layer against 62 ms for IQ3_XXS at 1,024 tokens. -**Checked against the CPU engine's decoder, on the real file:** the decoder equals the format's definition written out bit by bit -(614,400 sampled blocks a layer, none differ); the Hadamard rotation equals the CPU's bit for bit (f32 and f16); the expert -kernels reproduce a double-precision reference to 1e-7 relative on real experts of layers 0 and 47 of Gyro-S and of layers 0, 23 -and 47 of Gyro-M (the 92 GB file was read by range requests, 16 experts a layer: it does not fit this Mac's disk). Run it with -`metal/tests/gyro_parity.sh`. +**Quality against the CUDA engine, Gyro-S file.** `tools/quant_quality.py`, teacher-forced over 4,995 positions, CUDA = RTX 3060. -**Against the CUDA engine, same file** (the pinned Gyro-S; `tools/quant_quality.py`, teacher-forced over the 5 text segments of -`data/quant_quality/v1`, 4,995 positions; CUDA = an RTX 3060 with the engine of this release, run twice: experts on the GPU, experts on the CPU): - -| Reference -> run | top-1 agreement | mean KL | KL p99 | perplexity of the text | -|---|---|---|---|---| -| CUDA GPU experts -> CUDA CPU experts (the noise floor: two numerics of one engine) | 95.80 % | 0.0186 | 0.195 | 4.462 / 4.477 | -| CUDA GPU experts -> Mac | 95.46 % | 0.0207 | 0.241 | 4.486 / 4.477 | -| CUDA CPU experts -> Mac | 95.38 % | 0.0187 | 0.272 | 4.486 / 4.462 | - -The Mac is as far from either CUDA run as the two CUDA runs are from each other (the CUDA GPU path quantizes the activations to q8_1 and -reads an int8 copy of the codebook, the CPU path an int16 one; the Mac multiplies fp16 codebook values with the float activations). - -### Quality - -- **The Q8_0 hyper-connection read is the default on Mac.** It keeps a Q8_0 copy of the hyper-connection matrices instead of BF16. `STRATA_HC_Q8=0` in the config's `env` gives the exact BF16 read. -- **How the KL was measured:** teacher-forced over 28 prompts (code, reasoning, chat, six languages, documents), about 6,000 positions per set. The reference is BF16. - - On the model's own text: mean KL 0.0007, top-1 agreement 98.9%. - - On another model's text: mean KL 0.0007, top-1 agreement 99.1%. - - Perplexity of the text is unchanged (×0.9995 and ×1.0006). - - For scale, IQ3_XXS against Q2_0 is a KL of 0.094. - - The Mac engine is deterministic: two runs give identical numbers. -- **The fast prompt path** (the prompt kernels) against reading the prompt through decode windows: mean KL 0.0004, top-1 99.4-99.6%. -- **Needles:** a code word hidden in 8K, 16K and 32K tokens of text, at 10/50/90% depth. Found 9 of 9 with Q2_0 and with IQ3_XXS, both with Q8_0 and with BF16. With the fast prompt path (4K-32K) the answers are the same. -- **Same answers as on a PC:** - - against the CUDA engine on a Tesla P100 (IQ3_XXS and IQ3_S), the first 200 tokens of a reply are identical on the short question; - - on the story, the first IQ3_S tokens differ at token 1, a near-tie (margin 0.12). Following the CUDA engine's tokens, 4 of 200 picks differ, all at margins 0.06-0.21; - - against the HIP engine on a Radeon AI PRO R9700 (Q2_0), 36 of 39 first tokens are identical. - - The differences are near-ties between the two best tokens, from a different floating-point order. - -## Limits (this version) - -- **Images (vision):** on by default. The app and the Terminal use the `strata-vision` that sits beside `strata` in the engine package; - `./setup.sh` builds it itself and asks "Should the model also read images?" (default yes). One program does both jobs: with `--gpu` it - reads on the Mac's GPU (ggml's Metal backend, the shaders are built into the program), without it on the CPU. A Mac always gets the GPU - reader, whatever its memory. The 24 GB rule of the PC cards does not apply: the Mac has no separate card memory to keep free, and the - reader's footprint is about the same on either side (1.18 GB on the GPU, 1.05 GB on the CPU, mostly the 0.9 GB projector file). - Measured on a Mac mini M4 (32 GB), Qwen3.8-Flash-Next's BF16 projector: - - | Picture | GPU reader | CPU reader | - |---|---|---| - | 260 tokens (640x400) | 0.45 s | 50.7 s | - | 972 tokens (1600x1200) | 2.4 s | 194 s | - - `--vision cpu` is still allowed and prints that note. The CPU reader is slow because ggml has no vector kernel for BF16 on ARM (a build - for newer chips, with the BF16 and int8 instructions, took the same 50.7 s). If the GPU reader cannot start, the server prints one line - and starts the same program on the CPU. With Q2_0 streaming from the SSD on that M4, the STRATA 42 test picture (red circle, blue square, - green triangle, the text "STRATA 42") was described correctly with both readers. Checked on one Mac only; the M5 Max is untested for pictures. -- **Context:** measured up to 32K (needles, KL). Setup recommends 32K below 96 GB and 64K from 96 GB. Up to the model's 262K can be set, but it is not measured. -- **One request at a time.** `--parallel` (batch slots) is not ported. Several clients queue. -- **Not ported:** the control vector (`--control-vector-scaled`) and the PC tuning (`--calibrate`). -- **Streaming:** measured at 32 GB only. The model's file must be on the internal SSD; an external disk is untested and likely much slower. -- **Families:** the original Qwen3.8-Flash-Next, Unsloth's UD-Q4/Q5/Q6_K_XL and agentionai's Gyro-S are measured (above). Setup repacks Unsloth's shards into one file for the GPU; a Gyro file is used as it is. Gyro-M runs on the same kernels (checked tensor by tensor) but no Mac here could hold it. The Swift and Coder releases are not set up on Mac yet, and Qwen3.6 does not run on the Mac engine. -- **The ready-made Mac engine is in the app and in the Terminal's engine component, not in the release zips.** The zips - (for `START-HERE.bat` and `setup.sh`) have no Mac engine: `./setup.sh` builds it. - -## Switches (for testing; all off unless set, the defaults are above) - -| Variable | Effect | -|---|---| -| `STRATA_HC_Q8=0` | The BF16 hyper-connection read (exact) instead of Q8_0 | -| `STRATA_METAL_PROMPT=window` | Read the prompt through decode windows (the slow, exact fallback; ~190 tok/s) | -| `STRATA_METAL_PROMPT_CHUNK=N` | Prompt chunk size (default 2048) | -| `STRATA_METAL_MTP_Q8=0` | The draft layer's matrices in BF16 instead of Q8_0 | -| `STRATA_METAL_MTP_CHAIN=0` | One GPU submission per draft step instead of one per round | -| `STRATA_METAL_FUSE=0`, `STRATA_METAL_LAZY_FOLD=0` | Turn off two decode fusions (same tokens, slower) | -| `STRATA_METAL_PARTIAL=1`, `STRATA_METAL_ROUTE=1` | Two decode variants that measured no faster here (same tokens) | -| `STRATA_METAL_STREAM=1` / `=0` | Stream the experts from the SSD even when the pack fits / never (the default decides by size) | -| `STRATA_METAL_CONTIG=0` | Streaming: read the model file even when the experts' copy is there | -| `STRATA_METAL_ARENA_GB=N`, `STRATA_METAL_KEEP_FREE_GB=N` | Streaming: the arena's size, or the memory left to macOS when the arena is sized | -| `STRATA_METAL_LEAVE_SHARE=0.4` | Streaming, Macs with 64 GB or more: the share of the RAM left to macOS and its file cache when the footprint cap is set (for measuring) | -| `STRATA_METAL_WIRE_ARENA=0` | Streaming, Macs with 64 GB or more: leave the filled arena pageable (default: wired with mlock) | -| `STRATA_METAL_IGNORE_REPACK=1` | Start the engine although a model repack or setup's preparation is running beside it (the engine refuses by default: it froze this Mac) | -| `STRATA_METAL_KEEP_HOST_WEIGHTS=1` | Streaming: keep the host code's own copies of the weights (2-3 GB) instead of giving them back to macOS | -| `STRATA_METAL_TRACE=1`, `STRATA_METAL_PROFILE=1`, `STRATA_METAL_KPROF=T` | Per-window traces, a time breakdown, per-kernel GPU times | - -**Two opt-ins an M1 Max cannot run (measured on an M1 Max, macOS 26.6.2, Q2_0):** `STRATA_METAL_PROMPT_EXPERTS=v3` - the -prompt's expert GEMMs with the M5's layout, `mm_gu3` - loses the threadgroup once a prompt spreads over more than 40 -threadgroups of it (35 pass, 40 fail; 190 tiles and the engine's tile both fail, one tile passes), so it is off there (the -default is v2). `STRATA_METAL_ROUTE=1` - the router GEMV and `moe_route` fused into one dispatch - routes with stale -logits and gives a different top-10 per run (two identical runs differ in ids, weights and plan; the logits themselves -stay bitwise): the fusion hands the routing to the last threadgroup of the GEMV through a device counter, and this -Metal's device atomics are relaxed-only, with no acquire to pair with the writers' fence. Both are off by default. The -`no faster` measurement in the table is from the Macs above; on an M1 Max `ROUTE` is also wrong. - -## How it works - -The code is in `metal/` (the engine's side in `metal/src/engine`, the kernels in `metal/kernels`). In short: -- **Host code:** the PC host code compiles unchanged for macOS. `metal/include/metal_compat` maps the CUDA runtime calls it makes onto a small Metal runtime (`metal/src/gpu`). -- **The decode and prompt work** run in Metal kernels (`metal/kernels`, `metal/src/engine`): - - one command buffer per verify window; - - the experts are read in place from the mapped model file; - - the hyper-connections, the GDN state and the QSA attention stay on the GPU; - - a whole MTP draft round runs in one command buffer. -- **The PC builds are byte-identical** with this code present. Every shared file's change is behind `STRATA_GPU_METAL` or `__APPLE__`, and the CUDA and HIP objects are compared after each change. +| Reference -> run | top-1 agreement | mean KL | KL p99 | perplexity | +| --- | --- | --- | --- | --- | +| CUDA GPU experts -> CUDA CPU experts (noise floor) | 95.80% | 0.0186 | 0.195 | 4.462 / 4.477 | +| CUDA GPU experts -> Mac | 95.46% | 0.0207 | 0.241 | 4.486 / 4.477 | +| CUDA CPU experts -> Mac | 95.38% | 0.0187 | 0.272 | 4.486 / 4.462 | diff --git a/docs/MCP_SERVER.md b/docs/MCP_SERVER.md index 772919d39..086a96b3d 100644 --- a/docs/MCP_SERVER.md +++ b/docs/MCP_SERVER.md @@ -1,13 +1,14 @@ # Strata's MCP server: let your AI assistant install and run Strata -`tools/strata_mcp.py` is an [MCP](https://modelcontextprotocol.io) server for the AI assistant you already use -(Claude Code, Claude Desktop, Cursor, VS Code, Codex ...). Add it once, then ask in plain words: +This page is for people who use an AI assistant and want it to install, start, stop and check Strata. `tools/strata_mcp.py` +is an [MCP](https://modelcontextprotocol.io) server for the assistant you already use (Claude Code, Claude Desktop, +Cursor, VS Code, Codex ...). Add it once ([Quick start](#add-it)), then ask in plain words: - "Install Strata for this PC." - "Start Strata." / "Stop Strata." - "Is Strata running? How much VRAM does it use?" - "Which Strata model fits my PC?" -- "Strata didn't start - what does the log say?" +- "Strata didn't start. What does the log say?" - "How fast is Strata on this PC?" (a short speed test) - "How do I connect Open WebUI / the OpenAI SDK / Claude Code to Strata?" @@ -21,6 +22,9 @@ direction: *your assistant* manages Strata. ## Add it +This is an optional **source-checkout tool**. It is not required to install the released app or Terminal. +For normal agent-assisted installation, use [AI_SETUP.md](AI_SETUP.md). + You need a copy of Strata: the zip from GitHub, or `git clone https://github.com/Niko1221/Strata`. Below, `C:\Users\you\Strata` (Windows) and `/home/you/Strata` (Linux) stand for that folder. Use `python3` on Linux. On Windows, use `py` if `python` is not on your PATH. @@ -110,9 +114,10 @@ The command is `python /tools/strata_mcp.py`. The transport is st | Tool | What it does | Arguments (all optional) | | --- | --- | --- | | `strata_status` | Whether a model is running: the model, engine version, context, busy or idle, VRAM and RAM in use. What is installed: models, start scripts, engine version, data folder, downloads. An install in progress. This PC's hardware: OS, CPU, RAM, GPUs, page file, free disk. The model and context setup recommends for this PC. | `port`, `hardware` (true/false) | -| `strata_models` | Every family and size setup offers, with download size, RAM needed, whether it fits this PC, and whether it is installed. Also the context lengths and the recommendation. | - | -| `strata_install` | Runs setup without questions, in the background (see below). Without `confirm: true` it only returns the plan. A second call shows the progress. `cancel: true` stops the install. | `family` (qwen, swift, coder, unsloth), `model` (Q2_0, IQ2_XS, IQ3_XXS, IQ3_S, IQ1_M, UD-IQ4_XS, UD-Q4_K_XL), `context` (8192 ... 524288), `vision` (no, yes, gpu, cpu), `kv` (int8, q4_0, k8v4), `gpu` (a number) or `gpus` ("0,2", "all"), `backend` (auto, cuda, hip), `low_ram`, `port`, `data_dir`, `confirm`, `cancel` | -| `strata_start` | Starts an installed model in the background and waits until it answers. If it is not ready after `wait_seconds` (default 50), it returns "loading". | `model` (an id from `strata_status`, e.g. `q2_0`, `coder-iq1_m`; default: the most recently used), `port`, `gpu`, `wait_seconds` (0-600) | +| `strata_models` | Every family and size setup offers (qwen, swift, coder, unsloth, gyro, nvfp4, and Qwen3.6-35B-A3B), with download size, RAM needed, whether it fits this PC, and whether it is installed. Also the context lengths and the recommendation. | - | +| `strata_remove` | Deletes an installed model's own files (GGUFs, pack folder, run config) and nothing another installed model still uses. Without `confirm: true` it only returns the plan. Stop the model first. | `model` (required), `confirm` | +| `strata_install` | Runs setup without questions, in the background (see below). Without `confirm: true` it only returns the plan. A second call shows the progress. `cancel: true` stops the install. | `family` and `model` (the ones `strata_models` lists), `context` (1024 ... 2097152; usual sizes 8192 ... 524288), `vision` (no, yes, gpu, cpu), `kv` (int8, q4_0, k8v4), `gpu` (a number) or `gpus` ("0,2", "all"), `backend` (auto, cuda, hip), `low_ram` (auto, on, off, resident, mmap), `port`, `data_dir`, `gguf_dir` (a folder with the model's GGUF files, checked and not downloaded again), `confirm`, `cancel` | +| `strata_start` | Starts an installed model in the background and waits until it answers. If it is not ready after `wait_seconds` (default 50), it returns "loading". | `model` (an id from `strata_status`, e.g. `q2_0`, `coder-iq1_m`, `swift-iq3_xxs`; default: the most recently used), `port`, `gpu`, `wait_seconds` (0-600) | | `strata_stop` | Unloads the model through the server's own API, then ends the server it started. | `port`, `force` | | `strata_logs` | The last lines of the server log, the engine log or the setup log. | `source` (server, engine, setup), `model`, `lines` (1-500) | | `strata_benchmark` | Sends a short fixed request (greedy, thinking off) and reports output and prompt tokens/s. | `port`, `max_tokens` (16-512) | @@ -130,7 +135,7 @@ Every result is JSON with a one-line `summary`. It is sent both as text and as M - the context setup recommends for the GPU's VRAM; - pictures on where the model can read them (the reader runs on the CPU under 24 GB of VRAM and on the GPU from 24 GB, on NVIDIA and AMD cards; a Mac reads on its GPU). -The first call returns the plan: the download size (58-111 GB), the free disk space, the RAM check, and the exact +The first call returns the plan: the download size, the free disk space, the RAM check, and the exact setup command. The assistant should show you the plan and call again with `confirm: true` once you agree. It then runs the same steps `START-HERE.bat` / `setup.sh` run: @@ -164,7 +169,7 @@ VRAM and pinned RAM itself, as when you close the window. Then the server ends: While a request is running, the tool refuses unless you pass `force: true`. -**Status.** The tool reads `GET /health`, `/v1/status`, `/status` and, on older servers, `/metrics` on 127.0.0.1. +**Status.** The tool reads `GET /health`, `/v1/status`, `/status` and `/metrics` on 127.0.0.1. If a config of the folder sets an `api_key`, the tool sends that key with its own requests. It never shows the key. ## Safety diff --git a/docs/MODELS.md b/docs/MODELS.md index f60f94c89..17206ac8e 100644 --- a/docs/MODELS.md +++ b/docs/MODELS.md @@ -1,251 +1,265 @@ # Which model? Sizes, versions and what fits -Strata's main model is [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), in several **sizes** -(the same model, compressed more or less) and several **versions** (the original, a coding version, a fine-tune, other -makers' compressions). It also runs a smaller model, [Qwen3.6-35B-A3B](#qwen36-35b-a3b). The desktop app, the Terminal -(`strata launch`) and the installer recommend one for your PC; this page explains the choice. Back to the -[README](../README.md). +This page helps you choose a model and check that your computer can run it. It covers DeepSeek V4.1-Flash, +Qwen3.8-Flash-Next (and its variants) and Qwen3.6-35B-A3B. For installing, see [INSTALL.md](INSTALL.md). -> **On this page:** [Pick by memory](#pick-by-memory) · [Speed](#how-fast-is-each-size) · [The sizes](#the-sizes) · -> [Will it fit?](#will-it-fit) · [The versions](#the-versions) · [Qwen3.6](#qwen36-35b-a3b) · -> [Adding another model](#adding-or-switching-models) +> **On this page:** [Quick start](#quick-start) · [Pick by memory](#pick-by-memory) · +> [DeepSeek V4.1-Flash](#deepseek-v41-flash) · [Qwen3.8-Flash-Next](#qwen38-flash-next) · +> [Qwen3.6-35B-A3B](#qwen36-35b-a3b) · [Adding or switching models](#adding-or-switching-models) · +> [History](#history--older-measurements) + +## Quick start + +1. Find your memory in [Pick by memory](#pick-by-memory) and choose the first provider that fits. +2. Print the plan. Nothing is downloaded or changed: + + ```sh + strata launch --plan --family qwen --model iq2_xs + ``` + +3. Start it with `strata launch` (guided), or run the same command without `--plan` and add `--yes --no-tui`. + +**Provider order:** ISTA-DASLab, then Unsloth, then Gyro, then others, using the first compatible pack that fits the +model and your hardware. ISTA-DASLab is the default recommendation. Gyro saves memory and is slower. Swift 1.5, NVFP4 +and OrcaRouter are other options. An explicit choice by the user always wins. The order applies within a model family: +Qwen3.6 has Unsloth packs only, and DeepSeek has the IST-DAS sets. The order is a recommendation. It says nothing about which +provider wins a given quality or speed comparison. + +Setup may preselect something else: a Swift or Gyro size at a low memory tier, or Unsloth on Strix Halo. Read the plan +and pass `--family` and `--model` when you want a particular provider. ## Pick by memory -The tiers count RAM and VRAM (the graphics card's memory) together, the same as the [README](../README.md#the-other-sizes). -`strata launch`, setup and the install tool of the MCP server pre-select by them; the desktop app does from 0.2.1. Every -other size stays a choice. Strata leaves 5 GB for Windows and the system, so the sum is RAM + VRAM minus 5 GB: a PC with a card of 12 GB and 32 GB of RAM is in the 40 GB tier (32 + 12 - 5 = 39), and a PC with a 16 GB card and 32 GB of RAM is too, not in the 48 GB tier (#3425). +Figures are the model's weights in RAM plus VRAM. Leave memory for the OS and other apps. Unified memory counts once. +They are estimates; a given context may still not fit. -| Your memory (RAM + VRAM) | Pre-selected | Notes | -| --- | --- | --- | -| **32 GB** | **Gyro-S**, or the **Coder** for code | The Coder needs 32 GB of RAM and keeps half of the experts, so it is weaker outside code ([why](#coder)). Gyro-S has lower quality than the 2-3-bit sizes, and the 2-bit Q2_0 decodes faster | -| **40 GB** | **Swift 1.5** IQ2_XS (39.2 GB) | Gyro-M (35 GB) is the other choice: close to IQ3_XXS in quality, but it writes answers 16-21% slower than Q2_0. Gyro runs on NVIDIA, AMD and Intel Arc cards and, from 0.2.2, on Macs (Gyro-S measured on a 32 GB Mac mini) | -| **48 GB** | **IQ3_XXS** (47.0 GB) | Slower, better | -| **64 GB** | **IQ3_S** (54.8 GB), the original. Swift 1.5's IQ3_S is an equal choice | The slowest, the best: it matches the full model on the published tests | -| **96 GB or more** | [Unsloth's UD-IQ4_XS to Q8_0](#unsloths-files-ud-iq4_xs-to-q8_0), 4 to 8 bit | UD-IQ4_XS keeps every expert in RAM from about 80 GB of RAM, Q8_0 from about 136 GB. Up to 192 GB, the 5-8-bit files fit in RAM. Setup still pre-selects IQ3_S here, except on a Strix Halo | +**Qwen3.8-Flash-Next, ISTA-DASLab original packs** -With less memory than a row says, Strata's [low-RAM mode](#a-big-graphics-card-and-little-ram) starts by itself and the -model runs slower, much slower with a small card. The figures above are RAM + VRAM; [Will it fit?](#will-it-fit) counts -RAM alone, for the sizes that keep their experts in RAM. - -Not sure? Take what the installer pre-selects. The Coder fits a 32 GB PC, but it keeps only 256 of the 512 experts, -chosen on code data, so it is weaker outside code and in languages other than English. For general use, take a full -model (Q2_0, IQ2_XS or IQ3_S) whenever your memory allows. - -## How fast is each size - -These are older numbers, kept because they compare the sizes with each other. Newer engines are faster on some cards -(each change is listed with its own measurement in the [details](DETAILS.md)); the 0.2.0 engine (0.1.45.3) has not been -re-measured on the whole table. Measured on an RTX 5070 (12 GB), a Ryzen 5 7600 and 64 GB of RAM, engine 0.1.26 (the -Q2_0 row: engine 0.1.36, which made Q2_0 prompts faster): - -| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt | -| --- | ---: | ---: | ---: | -| **Q2_0** | 94 tokens/s | 76 tokens/s | 2,650 tokens/s | -| **IQ2_XS** | 79 tokens/s | 63 tokens/s | 2,090 tokens/s | -| **IQ3_XXS** | 62 tokens/s | 49 tokens/s | 1,750 tokens/s | -| **IQ3_S** | 53 tokens/s | 46 tokens/s | 1,620 tokens/s | -| **Coder** (IQ1_M) | 55 tokens/s | 43 tokens/s | 2,180 tokens/s | - -Measured on an AMD RX 9070 XT (16 GB), a Ryzen 9 3900X and 47 GB of RAM (Linux, setup's own install, 2026-10-02, an engine -from before 0.1.40; Q2_0 is above setup's RAM estimate for 47 GB and was installed with `--model Q2_0 --yes`): - -| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt | -| --- | ---: | ---: | ---: | -| **Q2_0** | 60 tokens/s | 48 tokens/s | 1,160 tokens/s | -| **IQ2_XS** | 52 tokens/s | 36 tokens/s | 1,110 tokens/s | -| **Coder** (IQ1_M) | 44 tokens/s | 33 tokens/s | 1,420 tokens/s | - -- **Writes answers** = how fast the reply appears (tokens per second; a token is about ¾ of a word). -- **Reads your prompt** = how fast it takes in what you send (long documents, code, chat history), measured on a - 32K-token prompt; a 4K prompt reads at 910-1,580 tokens/s. A 32K prompt takes about 15 seconds with Q2_0. - -A card with more VRAM is faster, because more of the model fits on the GPU. For an RTX 3090 (24 GB) the -[details](DETAILS.md#other-gpus-estimated) give an estimate, not a measurement. All measurements, long-context numbers -and estimates for other cards are in the [details](DETAILS.md#speed-measured). The AMD measurements per card (RX 9070 XT, -Radeon AI PRO R9700, RX 7800 XT, RX 9060 XT, RX 6900 XT) are in [AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201). - -Every PC is different: `START-HERE.bat --calibrate` measures a few engine settings on yours and keeps the fastest -(about 15-30 minutes, longer on a slow card; on the PC above it made the Coder 7% faster; NVIDIA cards for now). -`strata launch` and the app have their own tuning step (quick: about 20 minutes). Measured Strata on your own -PC? See [Community benchmark results](COMMUNITY_BENCHMARKS.md) for a report template and how to share your results -in a pull request. - -## The sizes - -| Model | RAM+VRAM Requirements | Speed | Quality | +| Size | RAM + VRAM for the model | Speed | Quality | | --- | ---: | --- | --- | -| **Q2_0** | 37.6 GB | fastest | good | -| **IQ2_XS** | 39.2 GB | fast | better (**recommended** at 40 GB, as Swift 1.5) | -| **IQ3_XXS** | 47.0 GB | slower | great | -| **IQ3_S** | 54.8 GB | slowest | best: matches the full model on the published tests | +| Q2_0 | 37.6 GB | fastest | good | +| IQ2_XS | 39.2 GB | fast | better | +| IQ3_XXS | 47.0 GB | slower | great | +| IQ3_S | 54.8 GB | slowest | best; matches the full model on the published tests | -The download is 66-76 GB for the three smaller sizes ([details](DETAILS.md#which-model)); the first start also -fetches the MTP draft layer (~6 GB, +1 GB with images). +- **Less memory, coding only:** [Coder](#coder) runs around 32 GB of RAM with a suitable GPU. +- **Less memory, general use:** compare Unsloth, then [Gyro](#gyro-s-and-gyro-m-agentionai), which uses the least memory and is slower. +- **More memory:** [Unsloth's 4 to 8 bit files](#unsloths-files-ud-iq4_xs-to-q8_0) need 72 to 136 GB for all experts in RAM. +- **A smaller model:** [Qwen3.6-35B-A3B](#qwen36-35b-a3b), 32 to 52 GB of RAM. +- **A very large model:** [DeepSeek V4.1-Flash](#deepseek-v41-flash), 12 GB graphics card, 46 to 64 GB of RAM and 257 GB of disk. -## Will it fit? +If RAM is short, setup can read experts from the SSD instead ([below](#will-it-fit)). That works and is much slower. -Shard 1 is the part of the model that gets loaded when it starts: its experts go into your **RAM**, the rest onto -your graphics card (the second shard, a 29 GB lookup table, stays on the SSD). So it fits when your **RAM is at least -the experts + about 10 GB** for Windows and your other programs - the experts are most of shard 1: 34 GB for Q2_0, -35.5 for IQ2_XS, 43 for IQ3_XXS, 50 for IQ3_S, 23 for the Coder (the dense weights in shard 1 go to the graphics -card). With 64 GB of RAM every size fits (IQ3_S with little else open); with 48 GB, Q2_0 and IQ2_XS. A bigger graphics card makes it faster, but it doesn't lower the RAM needed -- except in the low-RAM mode below. +## DeepSeek V4.1-Flash -### A big graphics card and little RAM +DeepSeek V4.1-Flash-1017 runs in the **desktop app** (pick it in the model list) and in the **Terminal** (`strata launch`). +It uses its own engine, a separate download that the app and `strata launch` fetch for you. -On a PC whose RAM cannot hold the experts beside the system, setup maps them from the model's files instead and keeps -in RAM only what the graphics card does not hold (the [low-RAM mode](DETAILS.md#speed-measured), chosen by setup). For -example, a 32 GB PC with a 24 GB GPU runs Q2_0, IQ2_XS and the Coder this way, and a 32 GB PC with a 12-16 GB GPU the -Coder. With a small card most experts then come from the SSD and it is much slower (setup says so). -`START-HERE.bat --setup --low-ram on|off` overrides the choice. +| Set (IST-DAS) | Download | Quality against the original model | +| --- | ---: | --- | +| 1.75 bpw | 239 GB | perplexity +21%; code and agent text +8.9% and +10.2%; weaker on plain prose | +| 2.00 bpw | 257 GB | perplexity +16%; code and agent text +8.5% and +5.9% | +| 2.25 bpw | 274 GB | perplexity +7.7% to +8.3%; code and agent text +3.5% and +2.5% | -## The versions +The download includes the shared n-gram (Engram) table and the DSpark drafter. Pick 2.25 for coding and agent work if +you have the disk. Perplexity is measured on a fixed set of 4,995 test texts; lower is closer to the original. -**Pictures.** The versions in setup's menu read pictures, and pictures are on by default wherever a model can -read them. The picture reader runs on the CPU when the graphics card has less than 24 GB (a few seconds per -picture, nothing taken from the card) and on the graphics card from 24 GB (about 1-1.4 GB of VRAM; 0.1-0.5 s per -picture). The rule is the same on NVIDIA and AMD cards. `--vision off` (or "No" in the app and in `strata launch`) turns -pictures off; `--vision cpu` or `--vision gpu` picks the reader. On AMD, `strata-vision-hip` is the card's reader; if it -cannot start, Strata uses the CPU reader by itself ([AMD_HIP.md](AMD_HIP.md)). The Mac reads pictures on its GPU -([MAC.md](MAC.md)). On an Intel Arc the rule is the same, with two differences: an Arc of 24 GB or more keeps the -CPU reader unless you ask for `--vision gpu` (the Intel GPU reader has only run on an 8 GB A750), and a Battlemage card (B580, B70) -has pictures off until you pass `--vision yes` ([INTEL.md](INTEL.md#pictures-on-an-arc-2026-10-19-branch-f-021-arcvis)). More: [Images](DETAILS.md#images-vision). +**Supported hardware** -### Qwen3.8-Flash-Next +| Hardware | Systems | +| --- | --- | +| NVIDIA RTX 30, 40 and 50 series, 12 GB or more | Windows, Linux | +| AMD Radeon AI PRO R9700 | Linux | -The original, in all four sizes. With images, and with the -[experimental speed projection](DETAILS.md#experimental-speed-projection-experimental-off-by-default) as an option. +Tested on an RTX 3060 (Linux), an RTX 5070 (Windows) and four R9700 cards (Linux). The RTX 30 kernels were also checked +on an RTX A4000. Windows on RTX 30 and the RX 9070 are built but were not run. Mac and Strix Halo do not run it yet. + +**What you need:** a graphics card with 12 GB or more, 46 GB of RAM at least (64 GB is the tested comfortable size), +an NVMe SSD, and the disk space above. Setup warns when disk or RAM is short and does not stop you. + +```sh +strata launch --family deepseek41 --model ist-das-2.00bpw --plan +strata launch --family deepseek41 --model ist-das-2.00bpw +``` + +`--model-path DIR` starts from files you already have. The context is 8192 tokens (8K), the size it was checked at. + +**Speed: Faster or Full quality.** The app asks, and the Terminal takes `--speed faster|full` (or `--full-quality`). Faster uses 5 of the +model's 6 experts per word, a small quality change that is usually not noticeable. It is the default on a PC with 16 GB of graphics +memory or less. Full quality uses all 6. It needs DeepSeek engine 0.1.50 or newer, which the app and `strata launch` fetch. + +**Measured on:** IST-DAS 2.00 bpw, Strata 0.2.4 (before the Speed choice). + +| Hardware | Output | Prompt (8K) | +| --- | ---: | ---: | +| RTX 5070 12 GB, Ryzen 5 7600, 64 GB RAM | about 5 tokens/s (code) | 231 tokens/s | +| RTX 3060 12 GB, 46 GB RAM | 6 to 9 tokens/s | about 103 tokens/s | +| 4x Radeon AI PRO R9700 | about 31 tokens/s; 55 to 69 with DSpark | about 360 tokens/s (557 at 32K) | + +- The 2.25 bpw set reads 13% more data per token and is a little slower. +- A slow SSD lowers every number, because 12 GB cards spend most of each token waiting on the SSD and the CPU. +- DSpark (a small draft model) raised the RTX 3060 PC from 6.1-6.7 to 8.2-8.7 tokens/s and the R9700 box from about 31 to 55-69. It gave no gain on the RTX 5070 PC with a 6-core CPU, so `--dspark auto` leaves the drafter unloaded there. +- Answers can differ between runs with the same prompt, because the experts on the card change from moment to moment. + + + +## Qwen3.8-Flash-Next + +Four ISTA-DASLab sizes are the original model; the versions below are variants or other providers' packs of it. +Every version in setup's menu reads pictures. [Picture rules](#pictures). + +### Will it fit? + +Shard 1 is loaded at start: its experts go into **RAM**, the rest onto the graphics card. The second shard (a 29 GB +lookup table) stays on the SSD. The model fits when **RAM is at least the experts plus about 10 GB** for the OS and +other programs. + +| Size | Experts in shard 1 | +| --- | ---: | +| Q2_0 | 34 GB | +| IQ2_XS | 35.5 GB | +| IQ3_XXS | 43 GB | +| IQ3_S | 50 GB | +| Coder | 23 GB | + +With 64 GB of RAM every size fits (IQ3_S with little else open). With 48 GB, Q2_0 and IQ2_XS fit. A bigger graphics +card makes it faster and does not lower the RAM needed, except in low-RAM mode. + + + +**Big graphics card, little RAM.** When RAM cannot hold the experts beside the system, setup maps them from the model +files and keeps in RAM only what the card does not hold. A 32 GB PC with a 24 GB GPU runs Q2_0, IQ2_XS and the Coder +this way, and a 32 GB PC with a 12 to 16 GB GPU runs the Coder. With a small card most experts come from the SSD and it +is much slower; setup says so. A source checkout can force the mode with `--low-ram on|off` +([SOURCE_INSTALL.md](SOURCE_INSTALL.md)). `strata launch` has no `--low-ram` flag. + +The download is 66 to 76 GB for the three smaller sizes ([details](DETAILS.md#which-model)). The first start also +fetches the MTP draft layer (about 6 GB, 1 GB more with pictures). + +### Pictures + +Pictures are on by default wherever a model can read them. The reader runs on the CPU when the card has less than +24 GB (a few seconds per picture, no VRAM used) and on the card from 24 GB (about 1 to 1.4 GB of VRAM, 0.1 to 0.5 s per +picture). The rule is the same on NVIDIA and AMD. + +| Setting | Effect | +| --- | --- | +| `--vision off` | no pictures (also "No" in the app and in `strata launch`) | +| `--vision cpu` / `--vision gpu` | choose the reader | + +- **AMD:** `strata-vision-hip` is the card's reader. If it cannot start, Strata uses the CPU reader ([AMD_HIP.md](AMD_HIP.md)). +- **Mac:** pictures run on the GPU ([MAC.md](MAC.md)). +- **Intel Arc:** an Arc with 24 GB or more keeps the CPU reader unless you pass `--vision gpu`. A Battlemage card (B580, B70) has pictures off until you pass `--vision on` ([INTEL.md](INTEL.md)). + +More: [Images](DETAILS.md#images-vision). ### Coder -**[Coder](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF)** - ISTA-DASLab's coding version: -it keeps 256 of each layer's 512 experts, chosen on code (with agentic and image) data, and drops the rest (91% of -the full model's SWE-bench Verified score, 99% of LiveCodeBench, by its authors). One size (IQ1_M: its experts stored -like IQ3_S): shard 1 is **29.6 GB**, so it fits a PC with **32 GB of RAM**, runs 262K context on 64 GB, and reads long -prompts the fastest of all. With half the experts gone it is weaker outside code and in languages other than -English, Chinese and other CJK text included (#438: Chinese answers came out wrong or looping where English was -fine). For general use, or whenever your RAM allows, take a size that keeps every expert: Q2_0, IQ2_XS or IQ3_S. -More: [details](DETAILS.md#or-the-coder-half-the-experts-for-code). +**[Coder](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF)** is ISTA-DASLab's coding version. +It keeps 256 of each layer's 512 experts, chosen on code (with agentic and image data), and drops the rest. Its authors +report 91% of the full model's SWE-bench Verified score and 99% of LiveCodeBench. There is one size, IQ1_M (experts +stored like IQ3_S); shard 1 is **29.6 GB**. A 32 GB RAM computer also needs enough GPU memory and room for buffers; +read the launch plan. -``` -START-HERE.bat --setup --family coder +It is weaker outside code and in languages other than English, Chinese and other CJK text included. For general use, +or whenever RAM allows, take a size that keeps every expert. More: [details](DETAILS.md#or-the-coder-half-the-experts-for-code). + +```sh +strata launch --family coder --model IQ1_M ``` ### Swift 1.5 -**[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** - a fine-tune by UkisAI that -thinks much shorter before answering, so you get the answer sooner, with about the same quality. Same speed per -token, and about the same RAM as the same size of the original. Its own license applies (see its page). -More: [details](DETAILS.md#or-swift-15-a-fine-tune-that-thinks-shorter). +**[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** is a fine-tune by UkisAI that +thinks much shorter before answering, with about the same quality. Speed per token and RAM match the same size of the +original. Its own license applies. More: [details](DETAILS.md#or-swift-15-a-fine-tune-that-thinks-shorter). -``` -START-HERE.bat --setup --family swift --model IQ2_XS +```sh +strata launch --family swift --model IQ2_XS ``` ### Unsloth's files: UD-IQ4_XS to Q8_0 -The fourth version in setup's menu (`--family unsloth`) is **Unsloth's quantizations** of the same model, from ~4 to 8 -bits. They keep the original's experts at more bits than the 2-3-bit sizes above, so they need more RAM. Setup keeps -every expert in RAM where they and 16 GB more fit (the fastest way); with less RAM it keeps as many as fit and reads -the rest from the SSD while it answers (much slower), and for the bigger files it asks first. Pictures are on by default -(the CPU reader under 24 GB of graphics memory, the card's reader from 24 GB). +`--family unsloth` is Unsloth's 4 to 8 bit quantizations of the same model. They keep the experts at more bits than +the sizes above, so they need more RAM. Setup keeps every expert in RAM when they fit with 16 GB to spare. With less RAM +it keeps as many as fit and reads the rest from the SSD (much slower), and for the bigger files it asks first. -| Size | Download | Experts | RAM for all of them in RAM | Same top token as Q8_0 | AMD R9700 (32 GB): story / code / 32K prompt | -| --- | ---: | ---: | ---: | ---: | --- | -| **UD-IQ4_XS** (setup's first) | 94 GB | 55 GiB | ~72 GB | 88.9% | - | -| **UD-Q4_K_XL** | 111 GB | 72 GiB | ~88 GB | 93.1% | 63 / 87 tok/s / 1,110-1,320 tok/s | -| **UD-Q5_K_XL** | 158 GB | 92 GiB | ~108 GB | 94.6% | 57-60 / 82 / 810-1,260 | -| **UD-Q6_K_XL** | 169 GB | 102 GiB | ~118 GB | 94.6% | 54 / 76 / 1,020 | -| **Q8_0** | 188 GB | 120 GiB | ~136 GB | 100% (the reference) | 53-57 / 66-74 / 1,230-1,300 | +| Size | Download | Experts | RAM for all experts | Same top token as Q8_0 | +| --- | ---: | ---: | ---: | ---: | +| **UD-IQ4_XS** (setup's first) | 94 GB | 55 GiB | ~72 GB | 88.9% | +| **UD-Q4_K_XL** | 111 GB | 72 GiB | ~88 GB | 93.1% | +| **UD-Q5_K_XL** | 158 GB | 92 GiB | ~108 GB | 94.6% | +| **UD-Q6_K_XL** | 169 GB | 102 GiB | ~118 GB | 94.6% | +| **Q8_0** | 188 GB | 120 GiB | ~136 GB | 100% (the reference) | -For comparison, IQ3_S on the same card: 75 / 95 tok/s, 87.7% the same top token as Q8_0. The speed figures come from -engines older than 0.1.45.3 (the conditions are in [QUANTS.md](QUANTS.md)). The quality numbers, the -speed bound and NVIDIA/AMD/Intel details: [QUANTS.md](QUANTS.md); the files: [UD-Q4_K_XL](UNSLOTH_Q4.md), -[UD-Q6_K_XL](UNSLOTH_Q6.md). +For comparison, IQ3_S matches the top token of Q8_0 87.7% of the time. Quality numbers, the speed bound and +NVIDIA, AMD and Intel details are in [QUANTS.md](QUANTS.md); the files are described in [UD-Q4_K_XL](UNSLOTH_Q4.md) and +[UD-Q6_K_XL](UNSLOTH_Q6.md). Older AMD speeds are in [History](#history--older-measurements). -``` -START-HERE.bat --setup --family unsloth --model UD-Q5_K_XL +```sh +strata launch --family unsloth --model UD-Q5_K_XL ``` ### Gyro-S and Gyro-M (agentionai) -The fifth version in setup's menu (`--family gyro`) is **agentionai's Gyro quantizations** of the same model: trellis-coded -experts in a rotated basis. They are the smallest options and never setup's default. The download includes the -28 GB n-gram table; the memory figure is VRAM and RAM together. +`--family gyro` is agentionai's Gyro quantization: trellis-coded experts in a rotated basis. They are the smallest +options and never setup's default. The download includes the 28 GB n-gram table. The memory figure is VRAM and RAM together. -| Size | Download | Memory it needs (VRAM + RAM) | In plain words | +| Size | Download | Memory (VRAM + RAM) | In plain words | | --- | ---: | ---: | --- | | **Gyro-S** | 58.5 GB | 27.6 GB | the smallest option, for PCs with little memory; lower quality | -| **Gyro-M** | 92.0 GB | 35 GB | about IQ3_XXS quality with about 20% less GPU memory; decodes 16-21% slower than Q2_0 (RTX 3060 12 GB, R9700) | +| **Gyro-M** | 92.0 GB | 35 GB | about IQ3_XXS quality with about 20% less GPU memory | -Gyro runs on NVIDIA and AMD (experimental) cards and on Intel Arc cards on Linux (tested on an Arc A750; the Arc engine is built from this release's source, see [INTEL.md](INTEL.md)) and, from 0.2.2, on Apple Silicon Macs ([MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts): Gyro-S is measured on a 32 GB Mac mini, Gyro-M only tensor by tensor). Gyro reads pictures (the same rule as above). On NVIDIA and AMD it needs -engine 0.1.45.1 or newer; if the engine is older, the app says "This model needs a newer engine" and offers Update -engine. Gyro gives correct output on RTX 50 cards with engine 0.1.45.3. Details, measurements and credit: -[GYRO.md](GYRO.md). +Gyro runs on NVIDIA and AMD (experimental), on Intel Arc under Linux, and on Apple Silicon Macs. It reads pictures. +On an RTX 3060 12 GB and an R9700, Gyro-M decodes 16 to 21% slower than Q2_0. Details, measurements and credit: +[GYRO.md](GYRO.md). Intel Arc: [INTEL_ARC.md](INTEL_ARC.md). Mac: [MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts). -``` -START-HERE.bat --setup --family gyro --model Gyro-S +```sh +strata launch --family gyro --model Gyro-S ``` - + -### NVFP4: three versions (NVIDIA cards) +### NVFP4 (NVIDIA cards) -The sixth version in setup's menu (`--family nvfp4`) keeps the experts in NVIDIA's 4-bit NVFP4 format. There are three -versions of it. The app shows them by who made the weights: +`--family nvfp4` keeps the experts in NVIDIA's 4-bit NVFP4 format. There are three versions: -| App name | Setup name (`--model`) | What it is | Download | +| App name | `--model` | What it is | Download | | --- | --- | --- | ---: | -| **NVFP4 (Strata's version)** | `NVFP4` | Strata's own conversion (GPTQ; the ready-made pack is sergqwer's strata-nvfp4). The best quality of the three | 128 GB | -| **NVFP4 (NVIDIA's version)** | `NVFP4-NVIDIA` | NVIDIA's own NVFP4 checkpoint (plain rounding), converted for Strata. The same size and speed as the first | 128 GB | -| **NVFP4 (NVIDIA, trained for FP4)** | `NVFP4-QAD` | NVIDIA's retrained 4-bit model, slightly faster on short prompts ([below](#nvfp4-qad)) | 102 GB (+110 GB of disk to prepare) | +| **NVFP4 (Strata's version)** | `NVFP4` | Strata's own conversion (GPTQ; the ready pack is sergqwer's strata-nvfp4). Best quality of the three | 128 GB | +| **NVFP4 (NVIDIA's version)** | `NVFP4-NVIDIA` | NVIDIA's own checkpoint (plain rounding), converted for Strata. Same size and speed as the first | 128 GB | +| **NVFP4 (NVIDIA, trained for FP4)** | `NVFP4-QAD` | NVIDIA's retrained model, slightly faster on short prompts | 102 GB (+110 GB of disk to prepare) | -All three need about 80 GB of RAM to keep every expert in RAM, run on NVIDIA cards only, and read pictures (the -same rule as above). RTX 50 cards have FP4 tensor cores; other NVIDIA cards run NVFP4 at 8-bit precision. The first two -need engine 0.1.44 or newer, the third 0.1.45 or newer. Details and measurements: [NVFP4.md](NVFP4.md). +All three need about 80 GB of RAM to keep every expert in RAM, run on NVIDIA cards only and read pictures. RTX 50 cards +have FP4 tensor cores; other NVIDIA cards run NVFP4 at 8-bit precision. Details and measurements: [NVFP4.md](NVFP4.md). -``` -START-HERE.bat --setup --family nvfp4 --model NVFP4 -``` +**NVFP4-QAD** is NVIDIA's [Qwen3.8-Flash-Next-FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local), +retrained for 4-bit experts (quantization-aware distillation). Setup downloads NVIDIA's files as they are (102 GB) and +turns them into a Strata pack on your PC once (20 to 40 minutes, about 110 GB more disk while it runs; the 102 GB can be +deleted afterwards). Strata does not redistribute NVIDIA's weights. Pictures use the original model's reader file, which +setup downloads with it. Licenses: NVIDIA Open Model Agreement and Qwen's Community License. +More: [NVFP4.md](NVFP4.md#nvidias-fp4-local-checkpoint). - - -#### NVIDIA's retrained 4-bit model (NVFP4-QAD) - -NVIDIA retrained Qwen3.8-Flash-Next for 4-bit experts (quantization-aware distillation) and published it as -[Qwen3.8-Flash-Next-FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local). In setup's NVFP4 family it is -the size **NVFP4-QAD**. It is not a ready-made pack: setup downloads NVIDIA's files as they are (102 GB) and **turns them -into a Strata pack on your PC** (once, 20-40 minutes; about 110 GB more disk while it runs, and the 102 GB can be deleted -afterwards). Nothing of NVIDIA's weights is redistributed by Strata. - -| Size | Download | Memory it needs | In plain words | -| --- | ---: | ---: | --- | -| **NVFP4-QAD** (app: NVFP4 (NVIDIA, trained for FP4)) | 102 GB (+110 GB of disk to prepare) | like NVFP4: ~80 GB of RAM keeps every expert in RAM | NVIDIA's FP4, trained for FP4 - slightly faster on short prompts than the other two NVFP4 versions (Strata's version has the best quality) | - -NVIDIA cards only. It reads pictures (through the original model's picture reader file, which setup downloads with it). Its licenses are the NVIDIA Open Model Agreement and Qwen's Community License. -Technical details and measurements: [NVFP4.md](NVFP4.md#nvidias-fp4-local-checkpoint). - -``` -START-HERE.bat --setup --family nvfp4 --model NVFP4-QAD +```sh +strata launch --family nvfp4 --model NVFP4 ``` ### OrcaRouter Uncensored IQ3_XXS -For **OrcaRouter's Flash-Next Uncensored IQ3_XXS**, see the [manual compatibility setup](ORCA.md). It needs an -explicit packing conversion and is not an installer menu option. +For **OrcaRouter's Flash-Next Uncensored IQ3_XXS**, see the [manual compatibility setup](ORCA.md). It needs an explicit +packing conversion and is not an installer menu option. ## Qwen3.6-35B-A3B A smaller model from the same maker: [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (35 billion -parameters, 3 billion active per token, 256 experts, Apache 2.0 license). Strata runs [Unsloth's GGUFs with the MTP -draft layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) on its own engine path, chosen by the file. It is -in the app's model list and in setup (`--family qwen36`). It runs on NVIDIA and AMD cards; the Intel Arc engine and the -Mac engine do not run it. It reads pictures and serves several requests at once. +parameters, 3 billion active per token, 256 experts, Apache 2.0). Strata runs +[Unsloth's GGUFs with the MTP draft layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) on its own engine +path, chosen by the file. It is in the app's model list and in setup (`--family qwen36`). It runs on NVIDIA and AMD +cards; the Intel Arc engine and the Mac engine do not run it. It reads pictures and serves several requests at once. | Size | File | RAM setup plans for | In plain words | | --- | ---: | ---: | --- | @@ -254,24 +268,75 @@ Mac engine do not run it. It reads pictures and serves several requests at once. | **Q8_0** | 37.8 GB | 52 GB | 8-bit, the closest to the full model | The picture reader file adds 0.9 GB. The engine keeps every expert it can on the graphics card and computes the rest on -the CPU. Measured with engine 0.1.45 and UD-Q4_K_XL: decode 117.8 tokens/s on a Radeon AI PRO R9700 (32 GB) and 59.7 -tokens/s on an RTX 3060 (12 GB, most experts on the CPU). Pictures, requests at once, the full measurements and the -limits: [QWEN36.md](QWEN36.md). +the CPU. -``` -START-HERE.bat --setup --family qwen36 --model UD-Q4_K_XL +**Measured on:** UD-Q4_K_XL, decode only. + +| Hardware | Decode | +| --- | ---: | +| Radeon AI PRO R9700 (32 GB) | 117.8 tokens/s | +| RTX 3060 (12 GB), most experts on the CPU | 59.7 tokens/s | + +Pictures, parallel requests, full measurements and limits: [QWEN36.md](QWEN36.md). + +```sh +strata launch --family qwen36 --model UD-Q4_K_XL ``` ## Adding or switching models -- **In the app:** the Models page lists every model with its size and whether it fits your computer; pick one and press - the install button. -- **In the Terminal:** `strata launch` has a Model step with the same list (and "Use model files I already have"). - `strata models list` shows what is installed; `strata models import --plan` shows what a set of files - you already have would be, without changing anything. -- **With the installer:** `SETUP.bat` (the same as `START-HERE.bat --setup`; on Linux `./setup.sh --setup`) adds another - model any time. +- **In the app:** the Models page lists every model with its size and whether it fits your computer. Pick one and press the install button. +- **In the Terminal:** `strata launch` has a Model step with the same list and "Use model files I already have". + `strata models list` shows what is installed. `strata models import --plan` shows what a set of + files you already have would become, without changing anything. +- **From a source checkout only:** `SETUP.bat` or `./setup.sh --setup` adds another model ([SOURCE_INSTALL.md](SOURCE_INSTALL.md)). -Files another model shares are not downloaded again (the Coder uses the original's shard 2 and vision encoder). With more -than one model installed, `START-HERE.bat` asks which one to start; `run-.bat` (Linux: `run-.sh`) starts one -directly. +Shared files are reused. In the Terminal, choose a model in `strata launch`, pass an explicit family and model, or use a +saved `--preset`. Add `--restart` when replacing a running launch. Generated `run-` scripts belong to a source +checkout, not to the installed app. + +## Speed on your own PC + +`--tune quick` and `--tune thorough` measure settings for the selected model, and the app offers tuning too. Results +depend on the backend and machine. All speed measurements, long-context numbers and estimates for other cards are in +[DETAILS.md](DETAILS.md#speed-measured). AMD measurements per card are in [AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201). +Measured Strata on your own PC? See [Community benchmark results](COMMUNITY_BENCHMARKS.md). + + + +## History / older measurements + +These numbers compare the sizes with each other. Newer engines are faster on some cards, and the tables were not +re-measured in full for the current release. + +**Measured on:** RTX 5070 (12 GB), Ryzen 5 7600, 64 GB RAM; ISTA-DASLab sizes; engine 0.1.26 (Q2_0 row: engine 0.1.36). + +| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt (32K) | +| --- | ---: | ---: | ---: | +| **Q2_0** | 94 tokens/s | 76 tokens/s | 2,650 tokens/s | +| **IQ2_XS** | 79 tokens/s | 63 tokens/s | 2,090 tokens/s | +| **IQ3_XXS** | 62 tokens/s | 49 tokens/s | 1,750 tokens/s | +| **IQ3_S** | 53 tokens/s | 46 tokens/s | 1,620 tokens/s | +| **Coder** (IQ1_M) | 55 tokens/s | 43 tokens/s | 2,180 tokens/s | + +**Measured on:** AMD RX 9070 XT (16 GB), Ryzen 9 3900X, 47 GB RAM, Linux; engine before 0.1.40, 2026-10-02. Q2_0 is +above setup's RAM estimate for 47 GB and was installed with `--model Q2_0 --yes`. + +| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt (32K) | +| --- | ---: | ---: | ---: | +| **Q2_0** | 60 tokens/s | 48 tokens/s | 1,160 tokens/s | +| **IQ2_XS** | 52 tokens/s | 36 tokens/s | 1,110 tokens/s | +| **Coder** (IQ1_M) | 44 tokens/s | 33 tokens/s | 1,420 tokens/s | + +- A token is about three quarters of a word. A 4K prompt reads at 910 to 1,580 tokens/s on the RTX 5070; a 32K prompt takes about 15 seconds with Q2_0. +- The older NVIDIA `--calibrate` run on the RTX 5070 PC raised Coder speed by 7%. + +**Measured on:** AMD Radeon AI PRO R9700 (32 GB), Unsloth files, engines before 0.1.45.3 (conditions in [QUANTS.md](QUANTS.md)). + +| Size | Story | Code | 32K prompt | +| --- | ---: | ---: | ---: | +| **UD-Q4_K_XL** | 63 tokens/s | 87 tokens/s | 1,110 to 1,320 tokens/s | +| **UD-Q5_K_XL** | 57 to 60 | 82 | 810 to 1,260 | +| **UD-Q6_K_XL** | 54 | 76 | 1,020 | +| **Q8_0** | 53 to 57 | 66 to 74 | 1,230 to 1,300 | +| *IQ3_S, same card* | 75 | 95 | - | diff --git a/docs/MULTI_GPU.md b/docs/MULTI_GPU.md index 123654286..36b35a81b 100644 --- a/docs/MULTI_GPU.md +++ b/docs/MULTI_GPU.md @@ -1,532 +1,301 @@ # Strata on two or more GPUs (layer split) -One model can run across several NVIDIA cards in one PC. The layers are split into contiguous ranges, one per GPU: -the first card runs layers 0 to K-1, the next card runs K onward, and so on; the last card also runs the output head -and the draft (MTP) layer. Each card keeps an expert cache for **its own layers only**, so two cards hold about twice -the experts one card holds - for the Coder model on a 16 GB + 24 GB pair, nearly all of them, which is where the -speed comes from (decode then barely touches the CPU pool). +This page is for people with two or more graphics cards of the same make in one PC. A layer split puts one model on +several cards. NVIDIA is described here; AMD limits and measurements are in [AMD_HIP.md](AMD_HIP.md). -This is pipeline (layer) parallelism, not tensor parallelism: a token crosses from one card to the next once per -verify window (a few hundred KB through pinned RAM), not twice per layer. No NVLink or peer-to-peer access is -needed; cards on x4 or x1 slots work, and the PCIe share of each card is probed on its own link. A `--pcie-frac` you give -is every card's share and skips those probes; there is no per-card setting yet. - -**Two models on two cards (the app).** A layer split puts one model on several cards. The other way to use two cards is two -models, one per card, each answering on its own. In the app, turn on **Run multiple models** on the Models page: the card -list then says which card a loaded model uses ("In use by ..."), starts that card as Off for the next model, and -explains a failed start on a card that is taken. Measured on a PC with Radeon AI PRO R9700 cards (two of four, Linux): Q2_0 on -one card and IQ3_XXS on another both loaded (33 s and 59 s with the files cached, 114 s the first time), both answered chats -(about 60 tokens/s each), the Monitor listed both, and unloading one left the other answering. This was run with the AMD build only; -the same app code drives NVIDIA cards, but we have not run two models on two NVIDIA cards. - -**One make of card per engine.** An engine drives one make of card (NVIDIA, AMD or Intel). The app, `strata launch` and setup -offer only the cards the engine can drive, never split a model across two makes, say in one plain line when the other make has -the bigger card, and say that two models at once on two makes is not supported. +> **On this page:** [Using it](#using-it) · [Main, Helper, Off](#main-helper-off) · [How the split works](#how-the-split-works) · +> [Tuning options](#tuning-options) · [Pipeline windows](#pipeline-windows) · [Two models on two cards](#two-models-on-two-cards) · +> [DeepSeek](#deepseek-v41-flash) · [Limits](#limits) · [History](#history--older-measurements) ## Using it -**Nothing to type.** `START-HERE.bat` (Linux: `./setup.sh`) lists your NVIDIA cards and says for each one whether -Strata can use it: +**Desktop app:** choose Main, Helper or Off for each card. **Terminal:** run `strata launch` and use the Cards step, or name +the cards: +```sh +strata launch --gpu 0,1 +strata launch --gpu-roles 1=main,0=helper # cards you do not name are Off ``` - Your NVIDIA GPUs: - GPU 0: NVIDIA GeForce RTX 5080, 16 GB VRAM - can be used - GPU 1: NVIDIA GeForce GTX 1080 Ti, 11 GB VRAM - not supported - older than the RTX 20 series (compute capability 6.1; Strata needs 7.5 or newer) - GPU 2: NVIDIA GeForce RTX 3090, 24 GB VRAM - can be used - ... - 1) RTX 3090 + RAM: fits: needs 44 GB of the 70 GB available (recommended) - 2) RTX 3090 + RTX 5080 + RAM (layer split): fits: needs 44 GB of the 84 GB available - prompts are read about 20% faster; the cards share the layers (docs/MULTI_GPU.md) - 3) RTX 5080 + RAM: fits: needs 44 GB of the 66 GB available -Which GPUs? [1]: + +Choose a model that fits the combined memory. A split can add capacity without adding speed. + +**Source checkout only:** `START-HERE.bat` and `./setup.sh` have their own card picker. Their flags are not `strata launch` flags: + +| Source setup flag | Meaning | +| --- | --- | +| `--gpus 0,2` | These cards together, numbered as `nvidia-smi` numbers them. Strata puts the faster card last. Remembered. | +| `--gpus all` | Every card that can share the model. | +| `--gpu 0` | One card (at a start: for that start only). | +| `--layer-split auto` | The default. Or the first layer of each later card, for example `18` or `16,32`. | +| `--gpu-roles 1=main,0=helper` | Roles, saved. | + +Or edit an existing run config (`strata-*.json`) and restart. Merge these fields into the config; this is not a complete config: + +```json +{ + "gpu": [0, 2], + "layer_split": "auto" +} ``` +Docker: `GPU_ROLES=1=main,0=helper` is passed as `--gpu-roles` ([DOCKER.md](DOCKER.md)). + ## Main, Helper, Off -Every card has one of three roles: +Every card has one of three roles. | Role | What it does | -|---|---| -| **Main** | Runs the **last** stage of the split: the output head, the draft layer and the check of its guesses. A prompt chunk waits on it. Exactly one card is Main. | -| **Helper** | Holds a share of the layers and of the expert cache, so more of the model fits on graphics cards instead of being read from the drive. Prompts are read about 20% faster. | +| --- | --- | +| **Main** | Runs the last stage of the split: the output head, the draft layer and the check of its guesses. A prompt chunk waits on it. Exactly one card is Main. | +| **Helper** | Holds a share of the layers and of the expert cache, so more of the model sits on graphics cards instead of being read from the drive. The effect on speed depends on the cards and the workload. | | **Off** | Strata does not touch the card (a game, the screen, a card that is too small). | -The Main card is the **last** card in the engine's order, not the first. The first card runs the first layers and, by default, -the picture reader. (Older versions of this page said "the first is the main one"; that has not been true since 0.1.41.1.) +The Main card is the last card in the engine's order. The first card runs the first layers and, by default, the picture reader. -**Recommended roles** (the same rule in the app, `strata launch` and setup; `strata_control/gpu_roles.py`): the faster card -is Main (speed is multiprocessors x max clock, the engine's own measure; cards within 5% are the same card, and the one with -more memory wins, then the lower number; two identical cards: the highest number is Main, so the engine order stays 0, 1 as it always was; a card with clearly more memory beats a faster one that is at least half as fast). -Every other card starts as Helper, except a card with **8 GB or less** when a bigger card is there (Off; with only 8 GB cards, say two GTX 1080, the second one is a Helper), built-in graphics beside a card with memory of -its own (Off) and a card the engine cannot run (Off). Every role is one click (the app), one key (`strata launch`: M, H, O; -R goes back to the recommendation) or one line (below). +**Recommended roles** (the same rule in the app, `strata launch` and setup; `strata_control/gpu_roles.py`): -Only a few things are warned about, in one short line each: built-in graphics as Main, a card the engine cannot run, too -little memory for the chosen setup (part of the model would be read from the drive). **Every card Off** is allowed. The CUDA, -HIP and SYCL engines cannot start without a card, so the server then says so in one line and stops; an engine that can run on -the processor (the Mac engine) says "Performance is degraded on CPU only." and goes on. +- The faster card is Main. Speed is multiprocessors x max clock. Cards within 5% count as the same card, and the one with more memory wins, then the lower number. Of two identical cards the highest number is Main. A card with clearly more memory beats a faster one that is at least half as fast. +- Every other card starts as Helper, except a card with 8 GB or less when a bigger card is present (Off; with only 8 GB cards the second one is a Helper), built-in graphics beside a card with memory of its own (Off), and a card the engine cannot run (Off). +- A role is one click (app), one key (`strata launch`: M, H, O; R restores the recommendation) or one line (above). -**Setting roles** +Only a few things are warned about, one short line each: built-in graphics as Main, a card the engine cannot run, and too +little memory for the chosen setup. All cards Off is allowed. The CUDA, HIP and SYCL engines cannot start without a card and +say so; the Mac engine says "Performance is degraded on CPU only." and goes on. -``` -strata launch --gpu-roles 1=main,0=helper cards you do not name are Off -./setup.sh --gpu-roles 1=main,0=helper (START-HERE.bat --gpu-roles ... on Windows); saved -strata launch the Cards step: M Main, H Helper, O Off, R recommended, D details -``` - -or in the run settings (`strata-*.json`), which is what the app writes: +**In the run config** (what the app writes). Merge these fields into the existing config: ```json -"gpu_roles": {"1": "main", "0": "helper", "2": "off"}, -"gpu": [0, 1], -"gpu_order": "as_given" +{ + "gpu_roles": {"1": "main", "0": "helper", "2": "off"}, + "gpu": [0, 1], + "gpu_order": "as_given" +} ``` -`gpu` is the same choice for older engines and the Monitor: the Helpers (slowest first), then the Main card, with the Off -cards left out. The server reads `gpu_roles`; when `gpu` was edited by hand and no longer matches, `gpu` wins and the log says -so (a settings file that names its cards keeps them). Nothing is rewritten on upgrade: a file with only `"gpu": [1, 0]` shows -as Main = the last card, Helper = the first, the rest Off. The recommendation itself is not saved, so a card you add later is -recommended again; a card you add after choosing roles starts Off. Docker: `GPU_ROLES=1=main,0=helper` is passed as `--gpu-roles`. +`gpu` is the same choice for older engines and the Monitor: the Helpers (slowest first), then the Main card, Off cards left out. +The server reads `gpu_roles`. If `gpu` was edited by hand and no longer matches, `gpu` wins and the log says so. A file with only +`"gpu": [1, 0]` shows as Main = the last card, Helper = the first, the rest Off. The recommendation is not saved, so a card you +add later is recommended again; a card you add after choosing roles starts Off. -**One rule for which cards count** (`strata_control/gpu_fit.py` keeps the memory arithmetic): not an integrated chip beside -a real card, at least 4 GB of graphics memory to be listed, 8 GB or less starts Off beside a bigger card. The arithmetic is the app's `fitOn`; -each extra card of a split costs an estimated 2 GB of its own buffers. +**Which cards count** (`strata_control/gpu_fit.py` holds the memory arithmetic): not an integrated chip beside a real card; at +least 4 GB of graphics memory to be listed; 8 GB or less starts Off beside a bigger card. Each extra card of a split costs an +estimated 2 GB of its own buffers. -**Choosing the cards without roles** (at setup or at any start): +**Not supported** (setup says so and names the cards that can be used): -``` ---gpus 0,2 these cards together, as nvidia-smi numbers them; Strata puts the faster card last. Remembered. ---gpus all every card that can share the model ---gpu 0 one card (at a start: for that start only) ---layer-split auto (default) or the first layer of each later card, e.g. 18 or 16,32 -``` +- A card older than the RTX 20 series (compute capability below 7.5). Pascal and Volta cards are admitted when you name them + with `--gpus` (or `--gpu`, `--cuda 12`) and then run the experimental CUDA 12 engine ([OLDER_GPUS.md](OLDER_GPUS.md)). +- A card with less than 8 GB of VRAM together with others, because each card holds a copy of the dense weights and its own + prompt buffers. If you name it with `--gpus`, setup states the risk and asks (`--yes` goes ahead). +- Intel GPUs in a split, and a mix of NVIDIA and AMD cards. An engine drives one make of card. The app, `strata launch` and setup + offer only the cards the engine can drive, and say so when the other make has the bigger card. AMD cards share a model among + themselves: `./setup.sh --backend hip --gpus 1,0` ([AMD_HIP.md](AMD_HIP.md)). -**Not supported** (setup says so and names the cards that can be used instead): -- a card older than the RTX 20 series (compute capability below 7.5: GTX 10 and older), unless you name it: Pascal - and Volta cards (Tesla P100 / P40, GTX 10, V100) are admitted when you choose them with `--gpus` (or `--gpu`, - `--cuda 12`) and then run the experimental CUDA 12 engine, see [OLDER_GPUS.md](OLDER_GPUS.md). Two Tesla P40s - (`"gpu": [0, 1]` in the config) ran the layer split in a community benchmark: IQ2_XS decode 19.6 tok/s on one card, - 34.8 on both (#1028, experimental, one report); -- a card with less than 8 GB of VRAM, together with others (each card holds a copy of the dense weights and its - own prompt buffers) - unless you name it with `--gpus`: then setup says the risk and asks (`--yes` with the named - cards goes ahead); -- Intel GPUs, and a mix of NVIDIA and AMD cards. (AMD cards share a model among themselves: `./setup.sh --backend - hip --gpus 1,0`, see [AMD_HIP.md](AMD_HIP.md).) +## How the split works -Or edit an existing config (`strata-*.json`), then restart: +The layers are cut into contiguous ranges, one per card. The first card runs layers 0 to K-1, the next runs K onward, and the +last card also runs the output head and the draft (MTP) layer. Each card keeps an expert cache for its own layers only, so two +cards hold about twice the experts one card holds. -```json -"gpu": [0, 2], -"layer_split": "auto" -``` +This is pipeline parallelism, not tensor parallelism. A token crosses from one card to the next once per verify window (a few +hundred KB through pinned RAM). No NVLink or peer-to-peer access is needed, cards on x4 or x1 slots work, and each card's PCIe +share is probed on its own link. A `--pcie-frac` you give applies to every card and skips those probes. -`"layer_split"` is `"auto"` (placed by each card's free VRAM) or the **first layer of each later card**: one rising -number per card after the first, not a count of layers per card. With 4 cards and a 48-layer model, `"24,36,42"` -(or `[24, 36, 42]`) puts layers 0-23 on the first card, 24-35 on the second, 36-41 on the third and 42-47 on the last. -The server checks it before the start and says what is wrong (0.1.39, #644). +**What each card holds** -**Card order with `"auto"`** (NVIDIA, #1352): the faster card (multiprocessors x max clock) goes **last** - the last stage runs the head, the draft layer and the verify, and a prompt chunk waits on it (a 4070 Ti SUPER + 5060 Ti read a 6K prompt in 50 s one way round and 15 s the other). Cards whose scores are within 5% of each other (two of the same model) keep your order; a manual split keeps it too. `"gpu_order": "as_given"` keeps the order you wrote under `"auto"` as well. The engine log line `layer split: card order ...` says when it changed. +- Every card: a copy of the dense weights, its own session state (the KV cache of the full context), its verify window, its + prompt-path buffers, and an expert cache for its layers filled from the expert profile. +- The last card: also the output head and the draft layer (about 0.8 GB). +- Host RAM: the expert arena once, shared by all cards. The CPU pool computes whatever no card holds. -**An unequal pair under `"auto"` (#2644).** The search prices a decode window, and it can give almost every layer to the card it -rates faster. A card with one or two layers holds a handful of experts and clamps the prompt chunk of the whole pipeline -(#1616), so the search then asks for at least 3 layers per card - which on a 4060 Ti 16 GB + 3060 12 GB (48 layers, IQ3_S) -was K=3: the 3060 ran 3 layers, 12 GB idle, 452 tok/s of prefill against 802 at 28 layers on the 3060 (the reporter's sweep of -`"layer_split"` 12, 18, 24, 28, 32: 582, 693, 762, 802, 803 tok/s; the 3060's 360 GB/s of memory bandwidth against the 4060 Ti's -288 counts for more than its SMs). On two cards the engine now first asks for a placement in which each card owns at least its -share of the layers by its room for experts, and the log line says which rule chose the split. Measured on 2x RTX A4000 made -unequal (stage 0 priced at 0.57 of the speed with 3.7 GiB less room, `STRATA_SPLIT_SPEED_SCALE=0.57` and `--vram-reserve-mib 4400`; 14.5K-token -prompt, IQ3_XXS, 5 interleaved pairs): the old rule chose K=3 (3 layers on stage 0), the new one K=20; prompt reading 945 -> 1,441 tok/s (+52%, -5/5 pairs), decode 50.2 -> 65.9 tok/s (+31%, 5/5). Equal cards and three or more GPUs are unchanged. -On x16 + x4 rigs the link-miss scaling (#2007, from a 4x gap) can also move `auto` off the balanced placement (2x RTX 3090, x16 + x4: -K=31 with miss cost x4.20, 114-119 tok/s decode against 130-136 at K=27, #2418): `STRATA_SPLIT_UNIFORM=1` (opt-in, with `"gpu_order": "as_given"`) -prices every link the same, as before 0.1.41. -Prefill that is flat in the prompt length (no gain from 4K to 16K) is the sign of an idle card; **on an unequal pair sweep -`"layer_split"` once** - a 4K-only benchmark cannot see it. +Prompts are read in chunks that flow through the cards in turn: while a later card reads chunk c, the first card reads chunk c+1. +Conversation checkpoints save and restore every card's state. -**Skip the split when the first card holds everything** (opt-in, 0.1.31): `"split_skip_if_fits": true` in the config -(engine flag `--split-skip-if-fits`, with `--layer-split auto`) runs on the first card alone when it holds every -profiled expert plus the context's KV, the draft layer and the reserve, and says so in the log; otherwise the split -stays. On an R9700 32 GB + RX 9070 XT the R9700 holds all of the Coder's experts: with the flag 4K prompts read at -1,776 tok/s instead of 1,244 (split) and decode runs at ~60 tok/s instead of ~51 (16K prompts ~5% slower than split). +**`"layer_split"`** is `"auto"` (placed by each card's free VRAM) or the first layer of each later card: one rising number per +card after the first, not a count per card. With four cards and a 48-layer model, `"24,36,42"` puts layers 0-23 on the first +card, 24-35 on the second, 36-41 on the third and 42-47 on the last. The server checks the value before the start and says what +is wrong. -**Short prompts on a split (0.1.32, #340).** 0.1.30 gave each card's prompt path a loan from its own expert cache, -refilled after every request; on cards that hold nearly all their experts that cost short prompts up to a third of -their speed. 0.1.32 refills all cards at once, uses a smaller streaming ring on a split, and lets a card with free -VRAM keep its own prompt buffers - the same output as 0.1.31, measured on an R9700 + RX 9070 XT: 2K prompts 993 -> -1,265 tok/s, 16K 1,852 -> 1,950, decode unchanged. `STRATA_SPLIT_OWN=1` (opt-in) gives every card its own buffers: -2K 1,450 and 16K 2,227 tok/s there, but a full card then keeps a different set of experts resident, so the output -differs from the default's (stable and coherent); `STRATA_SPLIT_OWN=auto` does that only where the buffers are at -most 12% of each card's VRAM. - -**The idle card can help one-chunk prompts (opt-in, `STRATA_PREFILL_HELP=1`).** A prompt that fits one chunk runs the stages one after the other, so while -one card reads its layers the other idles. With it on, each stage hands a share of its streamed experts to the idle card: it -streams them over its own PCIe link into its own (lent) prompt buffers, computes their rows on the MMQ path and sends -them back - `--peer-device`'s peer streaming, without P2P (the activations and the rows go through mapped host memory, -read by copy kernels, so they do not queue behind the expert blobs on either card's copy engine). The share falls with -the prompt (0.41 of the streamed experts at 1.5K tokens, 0.32 at 3K) and is off from ~3.3K tokens, where the stages -overlap anyway and no share paid. Measured on 2x RTX 3090 (UD-Q4_K_XL, no P2P), prompt tok/s without / with: 1.1K 496 / -723, 1.5K 674 / 833, 2K 878 / 1,128, 2.5K 1,017 / 1,296, 3K 1,260 / 1,440; 4K and 8K unchanged, decode unchanged. It -costs no VRAM (the idle card's own prompt buffers) and ~110 KB of mapped host memory per token of the largest chunk it -helped (~360 MB at 3.3K tokens). The rows it computes round like a different MMQ grouping, so the output is not -bit-identical to the default's (it is repeatable: same prompt, same output), which is why it is **opt-in**: -`STRATA_PREFILL_HELP=1` turns it on. Native packs on the MMQ prompt path only (not with the fused prompt kernels, -`STRATA_PF_FUSED=1`, nor with `--peer-device`). With it on, `STRATA_PREFILL_HELP_FRAC=f` fixes the share. -On two Radeon AI PRO R9700 (each limited to 16 GB with `--vram-reserve-mib 17084`, Q2_0 pack, layer split auto, 5 interleaved pairs, -noise arm within 0.2%), `STRATA_PREFILL_HELP=1` gave prompt tok/s without -> with: 700 tokens 638 -> 672 (x1.053, 5/5), 1.5K 839 -> 809 -(x0.964), 3K 1,105 -> 929 (x0.840), 6K 1,284 -> 1,211 (x0.943): the PCIe 4.0 x16 links of this box (28.4 GB/s each) make the share lose -from 1.5K tokens up, so it stays off (`STRATA_PREFILL_HELP_FRAC` was not tuned for it). - -The engine flags behind it: `--layer-split K1[,K2..]|auto` and `--split-device D1[,D2..]` (the later stages' -devices; default the next visible ones). `--layer-split K --split-device 0` runs both stages on one card sharing -everything - the bit-exact check of the hand-off, not a speed mode. - -**The draft layer's prompt K/V in batches on a split (opt-in, `STRATA_SPLIT_MTP_BATCH=1`).** With `--mtp` the draft -layer sits on the last card, and each prompt chunk ends by filling that layer's K/V for the chunk's rows. On one card the -prompt path does it in batches (`Prefill::draft_kv`, a few large matrix products per chunk); on a split the drafter's own -pass always did it, one graph launch and four device copies per group of 8 rows (1,024 groups per 8,192-token chunk). -With the variable set, the last card's prompt path batches it as it does on one card. Restart-only; default off, so the -default start is unchanged. It needs a `--native` pack (the GGUF-form token table) and a paged or ring (`--kv-resident`) -draft K/V (`STRATA_MTP_BATCH_RING=0` sends a ring back to the old pass); otherwise it says why once and keeps the old pass: -`strata serve: draft layer prompt K/V batched on CUDA ...` or `strata serve: STRATA_SPLIT_MTP_BATCH=1 declined, the -drafter's own pass runs: `. The draft layer's K/V come out of Q8_1 x Q8_0 MMQ instead of its own mmvq, so the -drafts, and how many are accepted, can move; the target's tokens are decided by the verify window. -`tools/split_mtp_batch_parity.py` compares the greedy texts of the two arms. Measured on one machine (2x RTX 3080 20 GB -at 220 W, Xeon E5-2696 v4, UD-Q4_K_XL, `--layer-split 23`, `--batch-mtp`, two slots, `--prefill auto:16384`), two full -restarts per arm in the order off, on, off, on, the same binary in every arm: pooled medians of the prompt read were 1,996 -tok/s without and 2,129 with at 25K tokens (+6.7%), 2,448 and 2,553 at 51K (+4.3%), 2,846 and 2,940 at 104K (+3.3%); all 8 -on/off ratios of medians are above 1 (1.03 to 1.08), and the two off restarts differ by at most 2% at 25K and 0.3% at the -other sizes. The host's "after each chunk" time per request fell from 0.9-1.3 s to 0.1-0.25 s (one restart per arm). Greedy -output (`--adapt-every 0 --suffix-draft 0`, five prompts of 256 tokens and one 25.5K-token prompt) was identical in 6 of 6 -prompts in all five comparisons, the two runs inside one process (A/A) included, and the drafts accepted were 826 of 1,210 -in every run, so no difference in acceptance was visible. Decode is not shown to be unchanged: the medians with the variable -set were about 3% lower (pooled solo 82.6 -> 80.2 tok/s, two streams 93.2 -> 90.8), and the same configuration restarted -twice moved up to 4.7%, so two restarts per arm can neither confirm nor exclude it; the change touches only the prompt path. - -**Each card loads only its own layers' dense weights** (0.1.39, PR #639) with explicit split points (`--layer-split -27`, not `auto`): every card used to keep a full copy (~3.4 GB for the Coder) though its stage reads only its own -layers, and the VRAM it frees goes to that card's expert cache (2x MI50 16 GB, Coder: 8,819 -> 10,626 experts in -VRAM, decode 39.2 -> 41.7 tok/s). Opt-in for now, on AMD and NVIDIA alike: `STRATA_STAGE_TRIM=1` (please report how -it goes). -A card holding more experts can change which experts run on the GPU, so the output can differ slightly from a run -without it. - -**The resident RAM mode works on a split** (`--resident-experts`, the low-RAM mode). The RAM copy of the experts -leaves out the ones every card's cache holds, not only the first card's, and when the rest does not fit whole it keeps -the hottest by the expert profile over all the layers. An adaptive swap copies the evicted expert back into RAM from -the card that owns its layer. Before, `--resident-experts` with a split ran as `--mmap-experts`, and setup recommended -one card in the low-RAM mode; with an engine that has it (`RESIDENT_SPLIT_ENGINE` in setup.py), setup keeps the cards -together and the experts no card holds in RAM. The same copy, sized by a RAM budget (`--resident-budget-gib`, the -Unsloth UD-Q4_K_XL and UD-IQ4_XS configs), is kept on a split as well ([UNSLOTH_Q4.md](UNSLOTH_Q4.md)). Swift 1.5 IQ3_XXS at -160K (q4_0 KV, `--prefill 4096`, `--spec 4` with the stock draft layer), RTX 4060 Ti (layers 0-19) + RTX 5080 (20-47), -i9-14900KF, 32 GB of RAM, Windows 11, four greedy prompts at a time, decode tok/s: - -| | first four prompts | after three more rounds | -|---|---|---| -| 5080 alone, `--resident-experts` | 25 | 29 | -| split, `--mmap-experts` (what `--resident-experts` became on a split) | 32 | 64 | -| split, `--resident-experts` (22 GiB of experts locked in RAM) | 71 | 69 | - -The split with `--mmap-experts` catches up once the OS file cache holds the experts, on a PC with nothing else -running; the resident copy is there from the first request and stays locked when other programs need the RAM. With -`--pcie-frac 0 --adapt-every 0` the split's greedy output is the same with either mode. - -**The prompt path's loan on a split and its streamed ring (`STRATA_SPLIT_RING`).** Under a split the prompt path borrows -the tail slots of each card's expert cache for its buffers (they are given back after the prompt), and the resident RAM -copy keeps those experts too, so a borrowed expert is not read back from the model file during a prompt. How many slots -are borrowed depends on the prompt path's streamed ring: a split uses 96 ring slots when at least 75% of the (layer, -expert) pairs sit in some card's cache, and `STRATA_SPLIT_RING=N` sets N slots (`0` goes back to the pinned-share rule). -The ring used to be chosen after the RAM copy's regions were sized, so the regions were sized for a different ring than -the prompt path then used: with `STRATA_SPLIT_RING=384` it borrowed 5,396 slots and the copy kept 4,662, and the -other 734 were read from the model file in every chunk. The ring is now chosen first (`Prefill::set_ring_override` -says it must be set before the buffers are counted). The start-up line `strata serve: lend sizing: the prompt path -borrows N slots (ring R slots at chunk C), K of them keep their experts in RAM too` shows both numbers, and a -`WARNING` follows when K is below N. Without a split, or without `--resident-experts`, nothing changes. With a split -the order of the two steps changed for every ring choice, the 96-slot rule included: its regions used to be sized for -the larger default ring, so there the copy probably kept more slots than were borrowed (not measured). The 5,396 / -4,662 figures were measured on the test machine (2x RTX 3080 20 GB at 220 W, Xeon E5-2696 v4, UD-Q4_K_XL, -`--layer-split 23`, resident RAM mode, `--prefill auto:16384`, 2 slots), one restart per arm, with `STRATA_SPLIT_RING=384` -and `STRATA_PREFILL_RING` unset. Before the change a 104K-token prompt made 413 blob reads and read 10.3 GB of experts -from the model file per request; after it, 0, with all 5,396 borrowed slots kept in RAM, for 2.15 GiB more pinned RAM -(13.61 -> 15.76 GiB). 25K and 51K prompts read nothing from the file in either case. With the override unset and -`STRATA_PREFILL_RING=384` the lend lines are identical before and after (5,396 / 5,396, 15.76 GiB). No speed gain is -shown: the medians of three 104K reads were 2,803 tok/s before and 2,907 after, but the spread before is 2,540-2,897 -(its first read faulted 4.2 GB in from NVMe) and there is no A/A restart. Here the page cache absorbed the 10 GB per -request; with less spare page cache it would be disk traffic. Not measured: both ring variables unset. - -**A separate VRAM reserve for the later cards:** `--vram-reserve-later-mib N` (default: `--vram-reserve-mib`'s value). -The card that drives the monitors needs more headroom than one that drives none; with the display on the last card, -`--vram-reserve-mib 300 --vram-reserve-later-mib 1800` gives the first card's cache that VRAM. - -**auto** tries every placement (all of them for two, three or, since 0.1.40, four cards; proportional to the free VRAM beyond that) and -keeps the one whose caches would hold the most of the expert profile, hottest pairs weighted most; ties go to the -placement that leaves the fullest card the most room. The startup log prints the choice: +**`auto`** tries every placement for two, three or four cards (proportional to free VRAM beyond that) and keeps the one whose +caches hold the most of the expert profile, hottest pairs weighted most. Ties go to the placement that leaves the fullest card +the most room. The startup log prints the choice: ``` strata generate: layer split auto: K=19 - the caches hold 11767 of 12288 profiled pairs (fullest device 100%) strata serve: layer split: layers 0-18 (CUDA0), 19-47 (CUDA1), one hand-off per window ``` -## What each card holds +**Card order with `auto`:** the faster card goes last, because the last stage runs the head, the draft layer and the verify, and +a prompt chunk waits on it. Cards within 5% keep your order, and so does a manual split. `"gpu_order": "as_given"` keeps your +order under `auto` as well. The log line `layer split: card order ...` says when the order changed. -- **every card**: a copy of the dense weights (~3.4 GB for the Coder), its own session state (the KV cache of the full - context), its verify window and its prompt-path buffers, and an expert cache for its layers filled from the profile; -- **the last card**: also the output head and the draft layer (~0.8 GB); -- **host RAM**: the expert arena once, shared by all cards (the CPU pool computes whatever no card holds). +**Unequal cards under `auto`.** The search asks for at least 3 layers per card and, on two cards, first for a placement in which +each card owns at least its share of the layers by its room for experts. The log line says which rule chose the split. Prefill +that stays flat from 4K to 16K tokens is the sign of an idle card. On an unequal pair, sweep `"layer_split"` once, because a +4K-only benchmark cannot see it. On x16 plus x4 rigs the link-miss scaling can move `auto` off the balanced placement; +`STRATA_SPLIT_UNIFORM=1` (with `"gpu_order": "as_given"`) prices every link the same. -Prompts are read in chunks that flow through the cards in turn; while a later card reads chunk c, the first card -already reads chunk c+1. (0.1.44, AMD gfx12: the auto chunk on a split stays at 8192, not the single-card 16384: a 16K prompt is then two chunks -that overlap instead of one, a 32K one four instead of two. Same tokens; 2x / 3x / 4x R9700 prompt tok/s at 16K +36% / +45% / +53%, -32K +18% / +21% / +29%, 64K +4% / +12% / +17%, 8K unchanged. `STRATA_SPLIT_CHUNK16K=1` keeps 16384.) Conversation checkpoints save and restore every card's state; the adaptive expert swaps copy -into the card that owns the layer. +## Tuning options -## Overlapped hand-off (`STRATA_SPLIT_OVERLAP=1`, opt-in) +These are engine flags and environment variables. Put flags in the config's `"args"` and variables in its `"env"` object, then +restart. Defaults are off unless the table says otherwise. -By default each verify window runs the stages one after another: the host syncs stage 1, then stages and launches -stage 2's graph, and so on. `STRATA_SPLIT_OVERLAP=1` (stages on different cards only) gives the hand-off a ready -flag: the earlier stage's GPU writes the hand-off with volatile stores and raises the flag (`handoff_publish`), the -next stage's graph is staged and launched on a helper thread while the earlier stage is still running and waits for -the flag on its GPU (`wait_flag_ge`), and the host goes straight from serving one stage's layers to the next, syncing -the earlier stage only once the chain is done. It takes one stream sync and one graph launch per window off the -critical path. Batch windows (several conversations) keep the serial order. It is off beside `--pipeline-windows`, whose loop overlaps the stages its own way. +| Setting | What it does | Notes | +| --- | --- | --- | +| `"split_skip_if_fits": true` (flag `--split-skip-if-fits`) | Runs on the first card alone when it holds every profiled expert plus the context's KV, the draft layer and the reserve. Otherwise the split stays. | Opt-in. Needs `--layer-split auto`. | +| `STRATA_SPLIT_OWN=1` or `auto` | Gives every card its own prompt buffers. `auto` does it only where the buffers are at most 12% of each card's VRAM. | Output differs from the default's (stable and coherent). | +| `STRATA_PREFILL_HELP=1` | The idle card computes a share of the streamed experts of a one-chunk prompt. `STRATA_PREFILL_HELP_FRAC=f` fixes the share. | Helps prompts up to about 3.3K tokens on slow links only; not bit-identical; native packs on the MMQ prompt path only. | +| `STRATA_SPLIT_MTP_BATCH=1` | The last card fills the draft layer's prompt K/V in batches, as on one card. | Needs a `--native` pack and a paged or ring draft K/V; says why once if it declines. | +| `STRATA_STAGE_TRIM=1` | Each card loads only its own layers' dense weights (needs explicit `--layer-split K`, not `auto`). The freed VRAM goes to that card's expert cache. | Output can differ slightly. | +| `--vram-reserve-later-mib N` | A separate VRAM reserve for the later cards (default: `--vram-reserve-mib`). Use it when the display sits on the last card. | `--vram-reserve-mib 300 --vram-reserve-later-mib 1800` gives the first card that VRAM. | +| `--prefill-pipe R` (or `STRATA_PREFILL_PIPE=R`) | Reads each prompt in the chunk count with the least pipeline time for b/a = R tokens. `1` evens out today's chunk count. | Opt-in. `./setup.sh --calibrate` measures it per PC and writes it to the config. | +| `STRATA_SPLIT_RING=N` | Sets the prompt path's streamed ring to N slots (`0`: the pinned-share rule). A split uses 96 when at least 75% of the (layer, expert) pairs sit in some card's cache. | The startup line `lend sizing: ...` shows the result. | +| `STRATA_SPLIT_CHUNK16K=1` | Keeps the 16384-token auto chunk on AMD gfx12 splits (default 8192). | | +| `STRATA_SPLIT_OVERLAP=1` | Overlapped hand-off between stages on different cards. | Helps AMD; no gain on NVIDIA. See [below](#overlapped-hand-off). | +| `--split-device D1[,D2..]` | The later stages' devices (default: the next visible ones). `--layer-split K --split-device 0` runs both stages on one card; a bit-exact hand-off check, not a speed mode. | | -Measured on 2x RX 7900 XTX (gfx1100, ROCm, Linux), Ryzen 9 9900X, 52 GB RAM, Swift 1.5 IQ3_XXS, 128K context, 8-bit -KV, layers 0-25 / 26-47; 512-token greedy answers, median of 8 runs: +**The resident RAM mode works on a split.** `--resident-experts` (the low-RAM mode) leaves out of the RAM copy the experts any +card's cache holds. An adaptive swap copies the evicted expert back into RAM from the card that owns its layer. A RAM budget +(`--resident-budget-gib`) works on a split as well ([UNSLOTH_Q4.md](UNSLOTH_Q4.md)). With `--pcie-frac 0 --adapt-every 0` the +split's greedy output is the same in either mode. -| | serial (default) | `STRATA_SPLIT_OVERLAP=1` | one card | -|---|---|---|---| -| decode, code prompt | 53 tok/s | 70.1 tok/s | ~70 tok/s | -| decode, prose prompt | 45 tok/s | 55.9 tok/s | ~68 tok/s | +**Under WDDM (Windows and WSL2)** only 8 GiB of the expert arena is pinned on a multi-GPU run; the rest streams through the +pinned staging ring. Linux pins the whole arena. `STRATA_ARENA_PIN_GIB=N` pins at most N GiB, `0` the whole arena, on any OS. -Greedy output is byte-identical with the flag at 0 and 1; the quiz set scored the same (47/51). On NVIDIA it -gives nothing: 2x RTX A4000 (IQ3_XXS, 5 interleaved pairs, 0.2.1 engine), decode story +0.3 %, code +2.1 %, facts -0.3 % against -+2.8 % of noise, prompts +0.1 %, with the same tokens. A sync and a graph launch cost less there. The second card then reads about 94 % -busy in `nvidia-smi` (a kernel waiting for the flag), against 58 % without the flag, with no more tokens per second. +### Overlapped hand-off -The reading stage's wait for the flag is bounded on the GPU: `STRATA_SPLIT_WAIT_MS` (default 30000, 0 = no bound). -If the flag does not arrive in time, the wait marks an error word and returns, the window's graph completes, and the -host fails that window with an error naming the setting (the server then restarts the engine) instead of using a -stale hand-off. Normally the host's own per-layer wait (20 s, #267) ends a stalled window first and releases every -GPU wait, the hand-off's included. Test hook: `STRATA_TEST_HANDOFF_DROP=N` withholds the flag in the first stage's -N-th window, as if the publish were lost. -## Prompt chunks sized for the pipeline (`--prefill-pipe`, opt-in; setup's `--calibrate` sets it) +By default each verify window runs the stages one after another. `STRATA_SPLIT_OVERLAP=1` (stages on different cards only) gives +the hand-off a ready flag, so the next stage's graph is launched while the earlier stage still runs and waits on the flag on +its GPU. It takes one stream sync and one graph launch per window off the critical path. Batch windows keep the serial order, and +it is off beside `--pipeline-windows`. The wait is bounded by `STRATA_SPLIT_WAIT_MS` (default 30000, `0` = no bound); on timeout +the window fails with an error naming the setting and the server restarts the engine. -`--prefill auto` takes the largest chunk the buffers allow (up to 8192), which is what one card wants: a chunk costs a -fixed pass over its layers' experts (b: dequantized, or streamed over PCIe where they are not in VRAM) plus a part per -token (a), so few large chunks are cheapest. A layer split reads a prompt as a pipeline - stage s reads chunk c while -stage s+1 reads chunk c-1 - and then takes about sum_i (b + a C_i) + (S - 1) max_i (b + a C_i) for chunks C_i over S -stages: large or uneven chunks keep the later cards waiting (a 10.7K-token prompt in a 7680 and a 3011 chunk over -four stages: the first stage carried 24 of the 36 s), small ones pay b again and again. +## Pipeline windows -`--prefill-pipe R` reads each prompt in the chunk count with the least pipeline time for b/a = R tokens, when that -beats today's split by 3% or more: every count from the fewest the buffers allow is tried, each chunk evened out on the -256-token grid (at least 512, at most the buffers' chunk; keep `--prefill auto` so long prompts may use large chunks). -b/a differs by an order of magnitude between rigs - about 700 tokens on four GP100 dies (every expert in VRAM, slow -arithmetic), about 5000 on two RTX 3090 Ti (part of the experts streamed per chunk, fast arithmetic) - and a wrong one -costs: a fixed 1024 read 19-24% slower on the two RTX 3090 Ti and 22-36% slower on two RTX 4090 (reported in the PR by -MistyMoonR). So it is neither guessed nor learned from the prompts the engine serves (that made a prompt's chunks, and -so its output bits, depend on how busy the PC happened to be): like the PCIe share and the draft floor, it is a -setting of the PC that setup's `--calibrate` measures once (`tools/calibrate.py`: on a layer split, the prompt read -speed of a 2K- and an 8K-token prompt - the same text every time, after a header of its own so nothing is reused - -with the chunks off, evened out and b/a 768, 1536, 3072 and 6144; the best is kept only when it reads faster than off -in each of three paired rounds and by more than 3% in the median) and writes into the config. Measured so: four GP100 -dies keep b/a 768 (355-356 against 219-220 tok/s in every round, +62%); two RTX 3090 Ti keep it off (nothing read -faster than off, b/a 768 20% slower). A first version that read a different stretch of text per value and confirmed in -two rounds picked 1536 on the RTX 3090 Ti pair from one fast read - where the experts are streamed, the read speed -depends on the text. `--prefill-pipe 1` needs no -b/a: it is `STRATA_PREFILL_EQUAL=1`'s rule (#693) - today's chunk count, evened out on the 256-token grid, and only -when today's last chunk has 1024 tokens or more (3072 with `STRATA_PREFILL_CPU_SHARE` set). A chunk that size streams -every expert its card does not hold, a shorter one only the experts it routes to, so a prompt just past the chunk size -reads as one full chunk and a nearly free tail, and evening that out would add a second full pass over the experts: -the first version evened every split out, and on two RTX 4090 MistyMoonR's calibration read 1,601 tok/s with it -against 2,129 off. 0 (the default) is today's split; -`STRATA_PREFILL_PIPE=` is the same as the flag where the config cannot carry it. The chunk geometry changes the -rounding, which is why it is opt-in. +`--pipeline-windows 2` lets the first card start the next verify window while the last card still verifies this one. The next +window is a guess: that this window is accepted whole and that its bonus token is the one the draft layer predicts. If the guess +holds, half of the next window is already done. If not, the first card restores its state and builds the window from the real +tokens. The tokens are the serial loop's, up to CPU rounding. -A prompt segment (the prompt between two of its cuts: the system-prompt root, the last turn, a `pin=`) is planned once, -on its whole length and R, and every run of it reads in that chunk. So its chunks, and the loan its buffers are laid -out for, are the same whether it is read in one run, in pieces between the windows of decoding `--batch` slots, or goes -on after a `BYIELD` or from a periodic checkpoint taken inside it (the slot and the checkpoint keep the plan; the read -goes on in it when it continues the same segment, else it plans again and says so in the log). The output bits then -match as far as the experts the cards hold do (`--adapt-swaps 0` for byte-identical repeats, as without the pipe). -Before this, a read beside decoding slots re-planned every piece, and one that went on after a `BYIELD` re-planned what -was left. A chunk the pipe planned takes no `STRATA_PREFILL_CPU_SHARE`: the stages read -such chunks at the same time, and which stage got the CPU pool first would decide which experts the CPU computed. The -DONE line ends with `chunk=` (the first chunk of the request's longest batched segment) and `experts_streamed=` (the -expert blobs its prompt copied to the cards), and the INFO line carries `prefill_cap`, `prefill_stages`, -`stream_all_min` and `prefill_pipe`. One line in the log says what a plan chose: +- **Default:** on for AMD RDNA4 (gfx1200 / gfx1201) with `--serve`, the draft layer, and a split of two or more stages when the + drafter can sit on the first card, otherwise three or more. The log says "is on by default for a layer split of N stages". + `--pipeline-windows 0` or `STRATA_PIPELINE_AUTO=0` turns it off. NVIDIA builds keep it opt-in (`--pipeline-windows 2`). +- **Draft layer placement:** with the pipeline on, the draft layer loads on the first card, whose window has finished when the + chain starts. `STRATA_MTP_STAGE0=0` keeps the old place. +- **Off, with one log line saying why:** `--batch` slots, `--peer-device`, the helper caches (`--expert-cache-device1..3`, which + `--remote-expert-opt` builds on), a split onto one GPU, no draft layer, and on three or more stages `--pipeline-windows 1` and + `--adapt-async 1`. A request with repetition penalties or coupled draft sampling decodes serially. +- **Cost:** a second verify window on each card, 160 MiB less expert cache per card, plus two copies of the first card's + recurrent state (about 3 MiB per GDN layer it runs). +- **Config:** `"args": [ ..., "--pipeline-windows", "2" ]` with `"layer_split": "20"`. - strata prefill: 10058 tokens in 1280-token chunks over 4 stages (--prefill-pipe 706: b/a) +`--remote-expert-opt` does something only with a helper cache; on a plain layer split it is inert. +The `STRATA_PIPELINE_*` tuning and test variables are read only with `STRATA_PIPELINE_DEBUG=1`. -Four GP100 dies (PH402 SKU 200, `--layer-split 12,24,36`, Flash-Next IQ3_XXS, 64K context), prompt tok/s, fresh -prompts, two interleaved rounds (each within 0.2%), with b/a 1024 (the same chunks as 700-890, measured on this rig): +## Two models on two cards -| Prompt tokens | `--prefill auto` | `--prefill 2048` | `--prefill auto` + `--prefill-pipe 1024` (chunk) | +The other way to use two cards is two models, one per card, each answering on its own. In the app, turn on **Run multiple +models** on the Models page. The card list then says which card a loaded model uses ("In use by ..."), starts that card as Off +for the next model, and explains a failed start on a card that is taken. Two models at once on two makes is not supported. + +**Measured on:** a PC with two of four Radeon AI PRO R9700 cards, Linux, AMD build, Q2_0 on one card and IQ3_XXS on the other. +Both loaded (33 s and 59 s with the files cached, 114 s the first time) and answered chats at about 60 tokens/s each. Unloading +one left the other answering. The same app code drives NVIDIA cards, but two models on two NVIDIA cards were not run. + +## DeepSeek V4.1-Flash + +DeepSeek V4.1-Flash runs in the app and the Terminal and spreads its experts and its dense weights over several cards. It does not use +the Qwen layer split described above. On four Radeon AI PRO R9700 cards it reaches about 31 tokens/s. Hardware, sets and speeds: +[MODELS.md](MODELS.md#deepseek-v41-flash). + +## Limits + +- Every card needs compute capability 7.5 (RTX 20 or newer), unless you opt into the CUDA 12 engine for older cards. +- Works across cards: images (`--vision`), control vectors, KV streaming (`--kv-resident`) and mid-prompt checkpoints + (`--prompt-cache-every`). Each card keeps its own part. +- The older helper-GPU caches (`--expert-cache-device1..3`, [SECOND_GPU.md](SECOND_GPU.md)) take the visible GPUs no stage runs on. + A weak helper can make decoding slower, because its per-layer round trip costs more than the CPU pool needs for those experts. +- `--mmap-experts` needs a canonical pack (`experts.bin`); a native (IQ) pack says so at start. +- The prompt path has its own buffers on every card (1.5 GB each at the default 2048-token chunk; `--prefill 1024` halves that). + An explicit `--expert-cache` on the first card is capped to leave room for them. +- More cards pay off when the model's routed experts do not fit the faster ones. An extra slow card costs its own round per + window and can make a good pair slower. +- For several conversations at once, `--batch N --batch-groups G --trim-stage-weights` pipelines them through the cards + ([BATCHING.md](BATCHING.md)). + +## History / older measurements + +Dated numbers behind the settings above. Each row is one machine; none is a claim about yours. + + + +**Layer split, Coder model.** Measured on: RTX 5080 + RTX 3090, Ryzen 9 9950X3D, Coder, 32K context, engine 0.1.20-era +(`bench/results/2026-09-29-layer-split/`). + +| Setup | Prompt 16K / 28K tok/s | Decode story / code tok/s | +| --- | --- | --- | +| 5080 alone | 1,726-2,017 / 1,970 | 83-87 / 88-105 | +| 5080 + 3090, best split (K=26) | 2,039 / 2,357 | 84 / 110 | +| 5080 + 3090, auto (K=22) | 2,037 / 2,073 | 80 / 109 | + +A third card (RTX 2080 Ti) made that pair slower: 68 / 90 tok/s decode. + +**Option measurements** + +| Setting | Measured on | Result | +| --- | --- | --- | +| Unequal pair, `auto` rule | 2x RTX A4000 made unequal, IQ3_XXS, 14.5K prompt, 5 interleaved pairs | prompt 945 -> 1,441 tok/s (+52%), decode 50.2 -> 65.9 (+31%) | +| Unequal pair, hand sweep of `"layer_split"` | 4060 Ti 16 GB + 3060 12 GB, IQ3_S | K=12/18/24/28/32: 582 / 693 / 762 / 802 / 803 prompt tok/s; the old rule chose K=3: 452 | +| x16 + x4 link scaling | 2x RTX 3090, x16 + x4 | K=31: 114-119 tok/s decode, K=27: 130-136 | +| `split_skip_if_fits` | R9700 32 GB + RX 9070 XT, Coder | 4K prompts 1,244 -> 1,776 tok/s, decode ~51 -> ~60; 16K prompts ~5% slower | +| `STRATA_SPLIT_OWN=1` | R9700 + RX 9070 XT | 2K prompts 1,450 and 16K 2,227 tok/s (default 1,265 and 1,950) | +| `STRATA_PREFILL_HELP=1` | 2x RTX 3090, UD-Q4_K_XL, no P2P | prompt tok/s without / with: 1.1K 496 / 723, 2K 878 / 1,128, 3K 1,260 / 1,440; 4K and 8K unchanged | +| `STRATA_PREFILL_HELP=1` | 2x R9700 (16 GB each), Q2_0, PCIe 4.0 x16 | 700 tokens 638 -> 672; 1.5K 839 -> 809; 3K 1,105 -> 929; slower from 1.5K, so it stays off | +| `STRATA_SPLIT_MTP_BATCH=1` | 2x RTX 3080 20 GB, Xeon E5-2696 v4, UD-Q4_K_XL, `--layer-split 23` | prompt read +6.7% at 25K, +4.3% at 51K, +3.3% at 104K; decode not shown unchanged (about 3% lower, within restart noise) | +| `STRATA_STAGE_TRIM=1` | 2x MI50 16 GB, Coder | experts in VRAM 8,819 -> 10,626; decode 39.2 -> 41.7 tok/s | +| Prompt chunk on a split, AMD gfx12 | 2x / 3x / 4x R9700 | 16K prompt +36% / +45% / +53%; 32K +18% / +21% / +29%; 64K +4% / +12% / +17% | +| `STRATA_SPLIT_OVERLAP=1` | 2x RX 7900 XTX, Swift 1.5 IQ3_XXS, 128K, median of 8 | code decode 53 -> 70.1 tok/s (one card ~70); prose 45 -> 55.9 (one card ~68) | +| `STRATA_SPLIT_OVERLAP=1` | 2x RTX A4000, IQ3_XXS | +0.3% story, +2.1% code, -0.3% facts, inside the noise | +| Two Tesla P40, layer split | community report, IQ2_XS (#1028), experimental | decode 19.6 tok/s on one card, 34.8 on both | + +**Resident RAM mode on a split.** Measured on: RTX 4060 Ti (layers 0-19) + RTX 5080 (20-47), i9-14900KF, 32 GB RAM, Windows 11, +Swift 1.5 IQ3_XXS, 160K context, four greedy prompts at a time, decode tok/s. + +| Setup | First four prompts | After three more rounds | +| --- | ---: | ---: | +| 5080 alone, `--resident-experts` | 25 | 29 | +| split, `--mmap-experts` | 32 | 64 | +| split, `--resident-experts` (22 GiB locked in RAM) | 71 | 69 | + +**Prompt chunks for the pipeline (`--prefill-pipe`).** Prompt tok/s, fresh prompts, two interleaved rounds per row. + +| Prompt tokens | `--prefill auto` | `--prefill 2048` | `--prefill-pipe 1024` (chunk) | | ---: | ---: | ---: | ---: | | 2,013 | 180.9 | 181.4 | 228.0 (1024) | | 3,865 | 207.6 | 276.5 | 312.4 (1024) | | 10,041 | 261.1 | 433.0 | 432.6 (2048) | | 28,998 | 475.7 | 554.1 | 559.7 (3328) | -Two RTX 3090 Ti (CUDA 12.9 build for sm_86, two stages, Swift 1.5 IQ3_XXS, 1M context, 13,371 of 24,576 experts in -VRAM), prompt tok/s, fresh prompts, two starts per arm, each reading the four prompts twice (the cold first pass after -a card was freed is left out); b/a 5836 and 6432 (timed on the rig), against 1024: +Measured on: four GP100 dies (PH402 SKU 200), `--layer-split 12,24,36`, Flash-Next IQ3_XXS, 64K context. On 2x RTX 3090 Ti +(Swift 1.5 IQ3_XXS, 1M context) calibration kept it off, and a fixed 1024 chunk read 19-24% slower there and 22-36% slower on +2x RTX 4090 (community report). -| Prompt tokens | `--prefill auto` | + b/a 5836 / 6432 (chunk) | for comparison: b/a 1024 | -| ---: | ---: | ---: | ---: | -| 2,035 | 694-815 | 698-816 (today's) | 560-562 | -| 3,887 | 1,127-1,207 | 1,115-1,227 (today's) | 880-885 | -| 10,063 | 1,437-1,553 | 1,654-1,681 (2 x 5120) | 1,109-1,113 | -| 29,019 | 2,532-2,603 | 2,566-2,620 (today's) | 2,064-2,089 | +**Pipeline windows.** Decode tok/s. -## Limits (for now) +| Measured on | Result | +| --- | --- | +| 2 cards, R9700, code, serial loop -> draft layer on the first card | 103 -> 125 (+21%); 3 cards +19%, 4 cards +22% | +| RTX 4060 Ti + RTX 5080, i9-14900KF, 32 GB, Swift 1.5 IQ3_XXS, 160K, greedy 500 tokens | mean 72.4 -> 84.0 (+16%): Python 90.4 -> 103.1, English prose 63.0 -> 71.1, copy-heavy edit 90.5 -> 117.9 | +| 4x RX 7900 XT, Flash-Next GSQ-RCO IQ3_S, 262K, `--layer-split 12,24,36` | 4K 59.4 -> 67.6; 32K 51.8 -> 65.6 | -- **Works across cards** (bench/results/2026-09-29-layer-split-limits): - - images (`--vision`): each card keeps its own image-position table; - - control vectors and the experimental speed projection: each card holds the vector's tables, switched on and - off per request on all of them; - - KV streaming (`--kv-resident`): each card streams the KV of its own session; - - mid-prompt checkpoints (`--prompt-cache-every`): each card saves its part of a checkpoint when it has read that - chunk; - - the older helper-GPU caches (`--expert-cache-remote`, docs/SECOND_GPU.md): they take the visible GPUs no stage - runs on, and hold only experts no stage's cache holds. On the test rig, a 2080 Ti helper made decoding slower, - as it did without a split: its per-layer round trip costs more than the CPU pool needs for those experts. -- `--mmap-experts` needs a canonical pack (`experts.bin`), with or without a split; a native (IQ) pack says so at - start. -- The prompt path has its own buffers on every card (1.5 GB each at the default 2048-token chunk; `--prefill 1024` - halves that) instead of borrowing cache slots as one card does. An explicit `--expert-cache` on the first card is - capped to leave room for them. -- Under WDDM (Windows, and WSL2) only 8 GiB of the expert arena is pinned (more, mapped into two GPU contexts, - leaves WDDM refusing allocations); the rest streams through the pinned staging ring. A Linux driver has no such - limit, so there the whole arena is pinned (since 0.1.31; the cap cost a 4090 + 3060 split two thirds of its - prompt speed, #253). `STRATA_ARENA_PIN_GIB=N` pins at most N GiB, `0` the whole arena, on any OS. -- Every card needs compute capability 7.5 (RTX 20 or newer). The pre-sm_80 QSA scorer path is fp32 FMAs, so a - Turing card runs the same kernels instead of the tensor-core prompt attention. - -## Measured - -The Coder on an RTX 5080 + RTX 3090 (Ryzen 9 9950X3D), 32K context; details in -`bench/results/2026-09-29-layer-split/`: - -| | Prompt 16K / 28K tok/s | Decode story / code tok/s | -|---|---|---| -| 5080 alone | 1,726-2,017 / 1,970 | 83-87 / 88-105 | -| 5080 + 3090, best split (K=26) | 2,039 / 2,357 | 84 / 110 | -| 5080 + 3090, auto (K=22) | 2,037 / 2,073 | 80 / 109 | - -- **Prompts gain the most** (+18-20%): each card reads its own layers of the chunk while the other reads the next. -- **Decode is on par with the faster card alone**, and ahead on code. Once both caches hold nearly every routed - expert, the per-layer GPU time decides. -- **Correctness:** one GPU is byte-identical to 0.1.20, and the hand-off itself is bit-exact. - -**Which cards and in what order:** -- Put the fastest card first; auto gives it as many layers as its cache allows. -- Leave out a much slower card when two already hold the model. An RTX 2080 Ti as a third card made the 5080 + - 3090 pair slower (68 / 90 tok/s decode): every extra card costs its own round per window. -- More cards pay off when the model's routed experts do not fit the faster ones. - -## One conversation with both cards busy (`--pipeline-windows`, default on two or more AMD gfx12 stages) - -**Default on AMD RDNA4 (gfx1200 / gfx1201).** With `--serve`, the draft layer and a layer split of **two or more stages**, -`--pipeline-windows 2` is on unless you say otherwise (the log says "is on by default for a layer split of N stages"; two -stages since 0.1.44, three or more since 0.1.43). It is no quality trade: the tokens are the serial loop's up to CPU rounding -(below). `--pipeline-windows 0` or `STRATA_PIPELINE_AUTO=0` turns it off; `--pipeline-windows 2` asks for it on any split. -The same options that switch it off when you ask for it (`--batch`, `--peer-device`, the helper caches, no draft layer, -`--adapt-async 1`) leave it off without a line when it is only the default. NVIDIA builds keep it opt-in. - -**Where the draft layer sits (0.1.44).** The draft layer's chain used to run on the last card, beside the last stage's window. -On 2x / 4x R9700 the two streams then slowed each other, in most engine starts and for the whole life of that engine: the -last stage's window took 27 ms instead of 13 ms while a chain ran (14.9 ms in the lucky starts, which also decoded 25% faster; -the 0.1.43 notes' "128.9 does not reproduce" was this). With the pipeline on, the engine now loads the draft layer on the -**first** card, whose window has finished when the chain starts: the chain's source rows are copied across once per chain and the -draft head's rows once at load. Same tokens; `STRATA_MTP_STAGE0=0` keeps the old place. Exact-flags A/B, 5 pairs, decode tok/s -(serial loop -> now): 2 cards code 103 -> 125 (+21%), 4K prompt +8%, prose equal; 3 cards code +19%, 4K +12%, prose +2%; 4 cards -code +22%, prose +4%, 4K +15%. - -With a split the cards take turns on a verify window: the first card runs its layers and hands off, then waits while -the last card runs the rest, the head and the draft. `--pipeline-windows 2` lets the first card start the **next** -window while the last card still verifies this one. The next window is a guess: that this window is accepted whole -and that its bonus token is the one the draft layer predicts (the draft layer is run on through the drafts of the -window in flight). When the guess holds, half of the next window is already done; when it does not, the first card -puts its state back (a copy of its recurrent state taken while the window ran) and the next window is built from the -real tokens. A guessed window is only started when the draft layer's estimate says it is likely to be kept. Windows -copied from earlier context (`--suffix-draft`) are guessed past as well, which is where edits that copy text gain -most. `--pipeline-windows 1` overlaps only the short prompt reads that go through the verify windows -(`--short-read`). - -Two or more cards, `--serve`. With three or more stages every stage but the last is a front stage: the guessed -window follows the verified one through them one card behind (each front stage keeps its own pair of recurrent-state -copies and puts its state back on a wrong guess), and only the last stage waits for the verdict. `--pipeline-windows 1` -and the asynchronous tier stay two-stage. In the config: - -``` -"args": [ ..., "--pipeline-windows", "2" ], -"layer_split": "20" -``` - -- **Cost**: a second verify window on each card, 160 MiB more kept out of each card's expert cache, plus two copies - of the first card's recurrent state (about 3 MiB per GDN layer it runs) on the first card with `2` (on every front - card with three or more stages; only the first card's room is kept out of its cache, the others allocate theirs after). -- **Same text**: the last card only ever runs windows that are verified, and every window row computes what it - would in any other window, so the tokens are the serial loop's. With `STRATA_IQ_MT_MIN=1 --pcie-frac 0 - --adapt-every 0` the greedy output is identical bit for bit to the serial loop's with the same expert caches. The pipeline keeps - its VRAM out of the caches, so against a run without the flag a few experts move from a card to the CPU, which - rounds them differently, and a near-tie can flip (a serial run given the same caches through `--vram-reserve-mib` - matches it exactly). -- **Off, with one line in the log saying why**, with `--batch` slots, `--peer-device`, the helper caches - (`--expert-cache-device1..3`, which `--remote-expert-opt` builds on), a split onto one GPU (`--split-device 0`), or no - draft layer; on three or more stages also `--pipeline-windows 1` and `--adapt-async 1`. A request with repetition penalties (`penalty_last_n`) or coupled - draft sampling decodes serially. -- **With the resident RAM mode's asynchronous swaps** (`--adapt-async 1`, [DETAILS.md](DETAILS.md)) a round's steps - advance between the verified windows. Each card's copies are queued by the decode loop itself while that card has - no window in flight, after every window that may still read what they overwrite has finished; the moves into RAM - wait the same way. -- **Measured** (Swift 1.5 IQ3_XXS, 160K context, q4_0 KV, the stock draft layer, RTX 4060 Ti (layers 0-19) + - RTX 5080 (20-47), i9-14900KF, 32 GB of RAM with the resident RAM mode on the split (#848); greedy, 500 tokens, two - interleaved pairs of three rounds, decode tok/s): Python code 90.4 -> 103.1, C code 76.6 -> 81.6, English prose - 63.0 -> 71.1, Italian prose 41.6 -> 46.3, a copy-heavy edit (a 5 KB file back with a rename) 90.5 -> 117.9; mean - 72.4 -> 84.0 (+16%). It pays when the windows are GPU-bound: with the experts read through the OS file cache - (`--mmap-experts` on that 32 GB PC) the file reads dominate and it measured no faster. -- **Measured on four stages** (Flash-Next GSQ-RCO IQ3_S, 262K context, int8 KV, layer split 12,24,36 on 4x RX 7900 XT, - every expert in VRAM; greedy, reasoning off, 256 tokens, repository-text prompts): decode 59.4 -> 67.6 tok/s at 4K - (median of 3) and 51.8 -> 65.6 at 32K. A guessed window that holds costs ~15 ms against ~43 ms for a fresh one; about - a third of the guesses held. The greedy text was identical to the serial loop's on all four prompts (`STRATA_IQ_MT_MIN=1 - --pcie-frac 0 --adapt-every 0`), and with every guess forced wrong (`STRATA_PIPELINE_FORCE_MISS=1`). -- **HIP**: the draft layer's per-row `gr_read` used the last stage's session scratch (`ss.block.gr`), which that stage's - verifier also uses; a chain launched at a verdict and the next window's last stage then ran on the card at once and - the window's head read a clobbered mix (wrong tokens after a guess that held). The draft layer now has its own scratch. - -The `STRATA_PIPELINE_*` tuning and test variables (THETA, FORCE_MISS, SWITCH, LOG, TRACE, SPEC_DEPTH and the like) are read only with -`STRATA_PIPELINE_DEBUG=1`. `--pipeline-windows` and `--adapt-async 1` combine: the engine turns the asynchronous tier -off beside `--pipeline-windows 2` only when `STRATA_PIPELINE_ADAPT_ASYNC=0` is set. What switches either one off is -printed once at start ("is off: ..."). `--remote-expert-opt` does something only with a helper cache -(`--expert-cache-device1..3`); on a plain layer split it is inert, and setup no longer writes it there (#1447). - -Measured on 2x RTX 3090 (sm_86, 250 W limits; GPU0 PCIe 4.0 x16, GPU1 x4, no NVLink; Ryzen 7 9800X3D, 32 GB RAM), -Qwen3.8-Flash-Next GSQ-RCO IQ3_S, `--resident-experts`, KV int8, `--spec 4 --mtp`, `"layer_split": "29"` for the -`--pipeline-windows` rows (without `--remote-expert-opt`); the others are setup's config (`auto`). Decode is the mean of -runs 2-6 of 1,500-token coding replies at temperature 0.6, prefill one cold 19.9K-token prompt; one run per arm unless -noted (reported by adambenhassen, #1447; not repeated on our boxes): +**Several settings on one rig.** Measured on: 2x RTX 3090 (250 W, x16 + x4, no NVLink), Ryzen 7 9800X3D, 32 GB RAM, Flash-Next +GSQ-RCO IQ3_S, `--resident-experts`, int8 KV, `--spec 4 --mtp`; decode is the mean of 1,500-token coding replies, prefill one +cold 19.9K prompt; one run per arm (community report #1447, not repeated here). | Arm | Decode tok/s | Prefill tok/s | -|---|---:|---:| -| setup defaults (two runs) | 137.0 / 136.9 | 1798 | -| `--adapt-async 1` (two runs) | 145.2 / 143.7 | 1800 / 1797 | -| `--adapt-async 1`, no `--remote-expert-opt` | 145.0 | 1807 | -| `STRATA_ADAPT_LAG=2` | 142.8 | 1800 | -| `STRATA_EXCHANGE_ROTATE=1` | 139.6 | 1800 | -| `STRATA_PF_FUSED=1` | 137.2 | 1913 | -| `--pipeline-windows 2` | 136.7 | 2097 | -| `STRATA_SPEC_COUPLED=1` | 137.6 | 1804 | -| `STRATA_SPEC_PROB=1` | 137.2 | 1788 | -| `STRATA_SPEC_COUPLED=1` + `STRATA_SPEC_GUMBEL=1` | 136.5 | 1805 | -| async + lag 2 + rotate + pf_fused | 145.0 | 1829 | -| pw2 + lag 2 + rotate + pf_fused | 135.9 | 2296 | -| pw2 + async + lag 2 + rotate + pf_fused | 142.0 | 2302 | - -The last row keeps most of the asynchronous tier's decode gain and the pipeline's +28% prefill; no stalls in any arm. -These are opt-in settings on one rig, not defaults. - -## Several conversations at once - -With a layer split, `--batch N --batch-groups G --trim-stage-weights` decodes several conversations together and -pipelines them through the cards: see [BATCHING.md](BATCHING.md). +| --- | ---: | ---: | +| setup defaults | 137.0 | 1,798 | +| `--adapt-async 1` | 144.5 | 1,799 | +| `--pipeline-windows 2` | 136.7 | 2,097 | +| `STRATA_PF_FUSED=1` | 137.2 | 1,913 | +| pipeline-windows 2 + async + lag 2 + rotate + pf_fused | 142.0 | 2,302 | diff --git a/docs/NVFP4.md b/docs/NVFP4.md index f64ffd1a7..fdfdb42e3 100644 --- a/docs/NVFP4.md +++ b/docs/NVFP4.md @@ -1,201 +1,155 @@ # NVFP4 routed experts +This page is for people who run, convert or test Qwen3.8-Flash-Next packs whose routed experts are NVFP4. They need an +NVIDIA card. For the choice between the three versions, see [MODELS.md](MODELS.md#nvfp4). + The engine can serve a Qwen3.8-Flash-Next whose routed experts are NVFP4, as NVIDIA's ModelOpt stores them: 64-value blocks of E2M1 codes with four UE4M3 sub-block scales, plus one FP32 `weight_scale_2` per expert and projection. -Everything else (attention, GDN, shared experts, routers, the head) stays in the formats the native path already -reads. Measured on Windows 11, RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB DDR5-5600. +Everything else (attention, GDN, shared experts, routers, the head) stays in the formats the native path already reads. -**The three NVFP4 versions in setup and the app.** `NVFP4` is "NVFP4 (Strata's version)", `NVFP4-NVIDIA` is "NVFP4 -(NVIDIA's version)", and `NVFP4-QAD` is "NVFP4 (NVIDIA, trained for FP4)" ([the short guide](MODELS.md#nvfp4)). All three -read pictures, through the original model's picture reader file that setup downloads with them (see "Images" below and -the FP4-Local section). The other sections of this page apply to all three unless they say otherwise. +## Quick start -## Getting a model +Choose a ready-made pack with the Terminal. Manual conversion below needs a source checkout. +```sh +strata launch --family nvfp4 --model NVFP4 ``` + +| Size in setup and the app | Name | Source | +| --- | --- | --- | +| `NVFP4` | NVFP4 (Strata's version) | GPTQ-quantized pack, `Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata` | +| `NVFP4-NVIDIA` | NVFP4 (NVIDIA's version) | `nvidia/Qwen3.8-Flash-Next-NVFP4` quantizes the routed experts the same way (NVFP4, group 16; other layers BF16, PLE table FP8, MTP experts FP8). Setup offers it as the ready-made pack `Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata`. | +| `NVFP4-QAD` | NVFP4 (NVIDIA, trained for FP4) | `nvidia/Qwen3.8-Flash-Next-FP4-Local`, converted on your PC ([below](#nvidias-fp4-local-checkpoint)) | + +All three read pictures through the original model's picture reader file that setup downloads with them +([Images](#images)). The other sections of this page apply to all three unless they say otherwise. Setup runs a pack in +the RAM-budget mode (`--resident-budget-gib`, the RAM less 24 GB, as for Unsloth's files) when the RAM is below the 68 GB +of experts plus room. + +## Getting a model by hand + +```sh python tools/nvfp4_convert.py --model --outfile .gguf python tools/iq_pack.py --gguf .gguf --out ``` - `nvfp4_convert.py` runs llama.cpp's own converter (`third_party/llama.cpp`, the commit setup pins) with the type - policy the engine needs. The experts are repacked without loss: `tools/nvfp4_verify.py` compares them with the + policy the engine needs. The experts are repacked without loss. `tools/nvfp4_verify.py` compares them with the checkpoint bit for bit, and `tools/nvfp4_verify_gguf.py` checks the GGUF. - - The small projections the BF16 kernels read (routers, SSM gates, indexer, PLE and hyper-connection projections) - are written as BF16. - - Everything served natively goes to Q8_0: attention, `ssm_out`, the shared experts, the output head, and the - token embedding. + - The small projections the BF16 kernels read (routers, SSM gates, indexer, PLE and hyper-connection projections) are + written as BF16. + - Everything served natively goes to Q8_0: attention, `ssm_out`, the shared experts, the output head and the token + embedding. - The 51.2e9-value PLE table is left out. -- `iq_pack.py` writes `experts.bin` for an NVFP4 GGUF by itself. Each blob is `[gate | up | down]` plus a 16-byte - tail `{s_gate, s_up, s_down, 0}`. The GGUF has no room for these scales, so the engine reads NVFP4 experts from +- `iq_pack.py` writes `experts.bin` for an NVFP4 GGUF by itself. Each blob is `[gate | up | down]` plus a 16-byte tail + `{s_gate, s_up, s_down, 0}`. The GGUF has no room for these scales, so the engine reads NVFP4 experts from `experts.bin` only. Run it like an IQ pack, with the GGUF as its own dense-weight source and the PLE table from an IQ GGUF's PLE shard: -``` +```sh strata --pack --native .gguf --native-dense-gguf .gguf \ --ple-gguf --mtp \ --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 ... ``` -The NVFP4 checkpoints change only the routed experts. The PLE table, the MTP layer and the expert profile of the -original model therefore work as they are. +The NVFP4 checkpoints change only the routed experts. The PLE table, the MTP layer and the expert profile of the original +model therefore work as they are. The conversion was tested end to end with +`jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4`. The NVIDIA checkpoint takes the same path but was +not run here as downloaded. -Tested end to end with `jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4`. By its -`hf_quant_config.json`, NVIDIA's `nvidia/Qwen3.8-Flash-Next-NVFP4` quantizes the routed experts the same way -(NVFP4, group 16; the other layers BF16, the PLE table FP8, the MTP experts FP8), so it takes the same path. It has -not been run here as downloaded. Setup offers it as a ready-made pack, the size `NVFP4-NVIDIA`. +## Expert size and the PCIe share -## Measured +An NVFP4 expert blob is 2.76 MB, against 1.51 MB for IQ2_XS. The cache holds half as many experts, and a miss costs twice +the bytes over PCIe. The PCIe share of the missed experts is therefore 0.25 for an NVFP4 pack, not the usual 0.55. At +0.55 the copy kernel was 40% of all GPU kernel time (nsys, 300-token decode). Measurements are in +[History](#history--older-measurements). -RTX 5090, 64K context, int8 K/V, `--vram-reserve-mib 1500`: - -| | NVFP4 pack | IQ2_XS, same engine | -| --- | ---: | ---: | -| experts in RAM | 63.3 GiB | 33.0 GiB | -| expert cache slots | ~8,350 | ~17,300 | -| decode, 400-token answer | 114-116 tokens/s | 132-139 tokens/s | -| a 32K prompt | 3,900 tokens/s | 5,600 tokens/s | - -An NVFP4 expert blob is 2.76 MB, against 1.51 MB for IQ2_XS: the cache holds half as many, and a miss costs twice -the bytes over PCIe. The PCIe share of the missed experts is therefore 0.25 for an NVFP4 pack, not the usual 0.55. -At 0.55 the copy kernel was 40% of all GPU kernel time (nsys, 300-token decode). A sweep at 262K context gave the -following decode rates: - -| PCIe share | short prompt | 32K prompt | -| --- | ---: | ---: | -| 0.55 | 103 tokens/s | 105 | -| 0.40 | 110 | 117 | -| 0.25 | 117 | 124 | -| 0.10 | 112 | 118 | - -## Measured in Strata (2026-10-13) - -The numbers above are the fork author's, on his RTX 5090 and 128 GB. These were measured here, on the ready-made GPTQ -pack (`--family nvfp4 --model NVFP4`). - -Quality, the quant yardstick (`tools/quant_quality.py`, 4,995 teacher-forced positions of code, docs, English, other -languages and an agent transcript, against Unsloth's Q8_0; RTX 3060 box): - -| | same top token as Q8_0 | mean KL | KL p99 | -| --- | ---: | ---: | ---: | -| NVFP4 GPTQ, experts on the CPU | 92.09% | 0.070 | 1.12 | -| NVFP4 GPTQ, 1,500 experts on the GPU | 91.79% | 0.069 | 1.07 | -| the same with `STRATA_ROUTE_TAIL_SKIP=7` | 88.59% | 0.118 | 1.76 | -| NVFP4 GPTQ with the GSQ-RCO IQ4_NL n-gram table instead of the FP8 one | 92.15% | 0.069 | 1.07 | -| for comparison, the same tool: Unsloth UD-Q5_K_XL | 94.55% | 0.038 | 0.64 | -| Unsloth UD-Q4_K_XL | 93.09% | 0.056 | - | -| Unsloth UD-IQ4_XS | 88.91% | 0.101 | 1.60 | -| GSQ-RCO IQ3_S | 87.69% | 0.144 | 1.98 | - -The route tail skip moves the answers by more than the quantization itself, so it is off by default for NVFP4 -experts. `STRATA_ROUTE_TAIL_SKIP=7` turns it on: +7-25% decode on the RTX 3060, +47-55% on the RTX 5070 (64 GB). - -Decode experts (`nvfp4_decode_bench`, 64 distinct experts per call, the share of the card's DRAM bandwidth): - -| tokens per expert | RTX 5070 | RTX 3060 | Tesla P100 | -| --- | ---: | ---: | ---: | -| 1 | 0.294 ms (89%) | 0.554 ms (89%) | 0.910 ms | -| 2 | 0.298 ms (88%) | 0.596 ms | 1.247 ms | -| 4 | 0.335 ms (79%) | 0.766 ms | 1.954 ms | - -Each weight is decoded once for all the tokens of a verify window (before the port's kernel work: 2.26 ms on the 3060 -at 4 tokens), bit for bit the same results. - -Prompt experts on the RTX 5070 (`mmq_expert_bench`, one launch of 16 experts with 625 rows each, gate/up + down): w4a8 -1.52 ms, w4a4x2 (`STRATA_PREFILL_NVFP4=w4a4x2`) 1.13 ms, w4a4x2 with down too (`STRATA_PREFILL_X2_MIN_K=640`) -0.995 ms, UD-IQ4_XS's experts 1.20 ms. First-token KL against the exact FP16 path (24 prompts, 1.5-8K tokens): w4a8 -0.0068, w4a4x2 0.0056, w4a4x2 with down 0.0044. On a 64 GB PC the prompt waits for the experts read from the SSD, -so the modes give the same prompt speed there; w4a8 stays the default until a 96 GB+ RTX 50 PC measures the others. -On RTX 20-40 the NVFP4 prompt products are 1.35-1.6x slower than IQ3_S / IQ4_NL ones at 32K: NVFP4's 16-value scales -are applied in FP32 after every int8 step, which only Blackwell's FP4 tensor cores avoid. - -Below its RAM need (68 GB of experts plus room) setup runs the pack in the RAM-budget mode (`--resident-budget-gib`, -the RAM less 24 GB), as for Unsloth's files. Mapped experts (`--mmap-experts`) on a 64 GB Windows PC filled the RAM. - -Images: the packs keep Qwen's vision tower unquantized, so the original's image encoder reads pictures for them -(`--vision`; strata-vision opens the dense GGUF for the vocabulary). Checked on an RTX 5070 and an RTX 3060 against -IQ3_XXS's answers on the same picture (a number in the picture read, two shapes with their colours and sides). The -encoder on the GPU takes ~1.1 GiB of VRAM: on the 3060 at 8K context the cache held 1,293 experts with images against -1,734 without. Pictures are on by default; setup and the app pick the CPU reader under 24 GB of graphics memory and the -GPU reader from 24 GB (`--vision cpu` / `--vision gpu` change it). +`STRATA_ROUTE_TAIL_SKIP` is off by default for NVFP4 experts, because it moves the answers by more than the quantization +itself. `STRATA_ROUTE_TAIL_SKIP=7` turns it on. ## Where the global scales go Each scale multiplies its own projection's FP32 output, as llama.cpp applies `.scale`: `h = silu(s_gate * (G x)) * (s_up * (U x))`, `y = s_down * (D h)`. -The scales are ~1e-4 each. Folding `s_down` into up is algebraically the same, but it leaves the hidden near 1e-5. -The hidden is then rounded to Q8_0 (CPU) or q8_1 (GPU) blocks with an FP16 scale, and `amax / 127` lands deep in -FP16's subnormals: one expert's output was off by 2-12% instead of 1.1% (`nvfp4_avx512_parity --expert`). The FP16 -prompt path had the same problem in its dequantized up weights (~1e-8). Any future fold must be checked against -FP16's range. +The scales are about 1e-4 each. Folding `s_down` into up is algebraically the same, but it leaves the hidden near 1e-5. +The hidden is then rounded to Q8_0 (CPU) or q8_1 (GPU) blocks with an FP16 scale, and `amax / 127` lands deep in FP16's +subnormals. One expert's output was off by 2 to 12% instead of 1.1% (`nvfp4_avx512_parity --expert`). The FP16 prompt +path had the same problem in its dequantized up weights (about 1e-8). Any future fold must be checked against FP16's +range. ## Prompt path precision `STRATA_PREFILL_NVFP4` picks how the batched prompt path multiplies the experts: -| mode | kernel | activations | one product vs FP64 (`mmq_nvfp4_parity`) | -|---|---|---|---| +| Mode | Kernel | Activations | One product against FP64 (`mmq_nvfp4_parity`) | +| --- | --- | --- | --- | | `w4a8` (default) | llama.cpp MMQ, int8 tensor cores | q8_1, float block scales | 0.53% | | `w4a4` | llama.cpp MMQ, Blackwell FP4 MMA (sm_120a only) | NVFP4 | 8.6% | +| `w4a4x2` | Blackwell FP4 MMA | NVFP4 | not in this table; first-token KL in [History](#history--older-measurements) | | `fp16` | dequantize + cuBLAS FP16 GEMM | FP16 | reference | -`src/prefill/mmq_nvfp4_w4a8.cu` compiles mmq.cuh's int8 NVFP4 path with Blackwell's FP4 MMA hidden from both the -device code and the host's tile config, so it is the same on every architecture; decode already multiplies at this -precision. The FP4 MMA exists only in the arch-specific target, so `src/prefill/mmq_nvfp4_w4a4.cu` (the FP4 x FP4 -kernel and its FP4 activation quantizer) is the one unit CMake builds with `12x` -> `12xa` (`strata_mmq_w4a4`); -the rest of the engine keeps the architectures as given and runs on sm_121 too. A card without that unit's image -(another architecture, or sm_121 against a 120a build) falls back to `w4a8` with a note. +`src/prefill/mmq_nvfp4_w4a8.cu` compiles mmq.cuh's int8 NVFP4 path with Blackwell's FP4 MMA hidden from both the device +code and the host's tile config, so it is the same on every architecture. Decode already multiplies at this precision. +The FP4 MMA exists only in the arch-specific target, so `src/prefill/mmq_nvfp4_w4a4.cu` (the FP4 x FP4 kernel and its FP4 +activation quantizer) is the one unit CMake builds with `12x` to `12xa` (`strata_mmq_w4a4`). The rest of the engine keeps +the architectures as given and runs on sm_121 too. A card without that unit's image (another architecture, or sm_121 +against a 120a build) falls back to `w4a8` with a note. -First-token KL divergence against `fp16`, 8 prompts (code and prose, 1K-8K tokens). The noise floor is the same -`fp16` path cut into 4096-token chunks instead of 8192: another summation order, same arithmetic. +> **Measured on:** first-token KL divergence against `fp16`, 8 prompts (code and prose, 1K to 8K tokens). Hardware and +> Strata version not recorded. The noise floor is the same `fp16` path cut into 4096-token chunks instead of 8192. -| mode | KL mean | KL median | KL max | top-1 agreement | prefill, 4-8K prompts | -|---|---|---|---|---|---| +| Mode | KL mean | KL median | KL max | Top-1 agreement | Prefill, 4-8K prompts | +| --- | --- | --- | --- | --- | --- | | noise floor | 0.00023 | 0.00003 | 0.0016 | 8/8 | - | | `w4a8` | 0.0018 | 0.0010 | 0.0096 | 8/8 | 2,503 tok/s | | `w4a4` | 0.0080 | 0.0027 | 0.040 | 7/8 | 2,904 tok/s | | `fp16` | reference | | | | 1,905 tok/s | -`w4a8` is the default. It is 4.5x closer to the reference than `w4a4`, at 86% of its speed, and it is the precision -every generated token already runs at. `fp16` is the choice when the prompt must be read exactly. +`w4a8` is the default. It is 4.5x closer to the reference than `w4a4`, at 86% of its speed, and it is the precision every +generated token already runs at. `fp16` is the choice when the prompt must be read exactly. -`STRATA_DUMP_MOE_INPUT` + `mmq_nvfp4_parity --real` test one product on a real prompt's rows: layer 20, its 12 -busiest experts, max|x|/rms 4 median and 9 at most. The errors: +`STRATA_DUMP_MOE_INPUT` with `mmq_nvfp4_parity --real` tests one product on a real prompt's rows: layer 20, its 12 +busiest experts, max|x|/rms 4 median and 9 at most. -| mode | gate/up | down (heavier SwiGLU tails) | +| Mode | Gate/up error | Down error (heavier SwiGLU tails) | | --- | ---: | ---: | | `w4a8` | 0.46% | 0.90% | | `w4a4` | 7.2% | 8.4% | | `fp16` | 0.017% | 0.021% | +On RTX 20 to 40 cards the NVFP4 prompt products are 1.35 to 1.6x slower than IQ3_S / IQ4_NL ones at 32K, because +NVFP4's 16-value scales are applied in FP32 after every int8 step, which only Blackwell's FP4 tensor cores avoid. + ## CPU experts -`src/kernels/cpu/nvfp4_avx512.cpp` computes the CPU pool's NVFP4 rows in 512-bit lanes. Each 64-value block is -decoded once for every token of a verify window: `vpdpbusd` on |E2M1| codes, with the sign moved onto the -activation. The arithmetic is ggml-cpu's `ggml_vec_dot_nvfp4_q8_0`, with the float additions ordered differently -(worst 4.5e-7 relative, `nvfp4_avx512_parity`). `STRATA_NO_NVFP4_512=1` falls back to ggml-cpu. +`src/kernels/cpu/nvfp4_avx512.cpp` computes the CPU pool's NVFP4 rows in 512-bit lanes. Each 64-value block is decoded +once for every token of a verify window: `vpdpbusd` on |E2M1| codes, with the sign moved onto the activation. The +arithmetic is ggml-cpu's `ggml_vec_dot_nvfp4_q8_0`, with the float additions ordered differently (worst 4.5e-7 relative, +`nvfp4_avx512_parity`). `STRATA_NO_NVFP4_512=1` falls back to ggml-cpu. -A CPU without AVX-512 (Zen 1-3, Intel Core 4th-10th and 12th-14th gen, Core Ultra, Xeon E5 v3/v4) runs the same rows in 256-bit lanes -(`src/kernels/cpu/nvfp4_avx2.cpp`): ggml's AVX2 arithmetic step for step, so every row is bit-equal to ggml-cpu's -(`nvfp4_cpu_parity`, 0 of 1280x9 / 2560x9 rows differ on a Ryzen 5 5600X, a Ryzen 7 255 and a Core Ultra 7 265). -`STRATA_NO_NVFP4_256=1` falls back to ggml-cpu. An NVFP4 pack is a native pack (its folder holds `native_experts.txt`), -so the engine starts on such a CPU; the AVX-512 requirement belongs to the canonical Q2_0 pack only (#2347: a `--pack` -folder without `native_experts.txt` is read as that pack, and the engine now says so before it refuses). +A CPU without AVX-512 (Zen 1 to 3, Intel Core 4th to 10th and 12th to 14th gen, Core Ultra, Xeon E5 v3/v4) runs the same +rows in 256-bit lanes (`src/kernels/cpu/nvfp4_avx2.cpp`): ggml's AVX2 arithmetic step for step, so every row is bit-equal +to ggml-cpu's (`nvfp4_cpu_parity`, 0 of 1280x9 and 2560x9 rows differ on a Ryzen 5 5600X, a Ryzen 7 255 and a Core Ultra +7 265). `STRATA_NO_NVFP4_256=1` falls back to ggml-cpu. An NVFP4 pack is a native pack (its folder holds +`native_experts.txt`), so the engine starts on such a CPU. The AVX-512 requirement belongs to the canonical Q2_0 pack +only; a `--pack` folder without `native_experts.txt` is read as that pack, and the engine says so before it refuses. -| tokens | ggml-cpu | AVX-512 | speedup | -|---|---|---|---| +> **Measured on:** one expert's gate+up (1280 rows of 2560, cache-resident), AVX-512 CPU, Strata version not recorded. +> The pool itself is bound by DRAM. + +| Tokens | ggml-cpu | AVX-512 | Speedup | +| --- | --- | --- | --- | | 1 | 0.102 ms | 0.058 ms | 1.76x | | 4 | 0.410 ms | 0.125 ms | 3.29x | | 7 | 0.719 ms | 0.196 ms | 3.66x | -These are one expert's gate+up (1280 rows of 2560, cache-resident). The pool itself is bound by DRAM. +> **Measured on:** the pool in DRAM terms (`nvfp4_avx512_parity --bw THREADS TOKENS`: 1536 experts, 4.2 GB), GB/s of +> weights read. Tokens is the tokens an expert serves in one verify window (1 to 2 in decode). -The pool in DRAM terms (`nvfp4_avx512_parity --bw THREADS TOKENS`: 1536 experts, 4.2 GB, GB/s of weights read; tokens = tokens an -expert serves in one verify window, 1-2 in decode): - -| CPU (threads) | kernel | 1 thread, 1 tok | all cores, 1 tok | 2 tok | 4 tok | -|---|---|---|---|---|---| +| CPU (threads) | Kernel | 1 thread, 1 tok | All cores, 1 tok | 2 tok | 4 tok | +| --- | --- | --- | --- | --- | --- | | Ryzen 7 255, Zen 4 (8) | AVX-512 | 15.4 | 58.6 | 57.6 | 47.7 | | | AVX2 rows | 13.9 | 58.9 | 58.1 | 42.1 | | | ggml-cpu | 12.6 | 58.7 | 44.8 | 22.4 | @@ -204,18 +158,25 @@ expert serves in one verify window, 1-2 in decode): | Core Ultra 7 265, DDR5-5600 (20) | AVX2 rows | 16.0 | 74.2 | 73.0 | 60.6 | | | ggml-cpu | 10.3 | 74.1 | 62.7 | 37.1 | -Without AVX-512 a core reads ~10% slower (13.9 vs 15.4 GB/s), but a pool that reads DRAM runs at the same rate up to two tokens a -window and ~12% below at four. The CPU's memory decides, not its vector width. +Without AVX-512 a core reads about 10% slower (13.9 against 15.4 GB/s). A pool that reads DRAM runs at the same rate up +to two tokens a window and about 12% below at four. The CPU's memory decides, not its vector width. + +## Images + +The packs keep Qwen's vision tower unquantized, so the original's image encoder reads pictures for them (`--vision`; +strata-vision opens the dense GGUF for the vocabulary). Pictures are on by default. Setup and the app pick the CPU reader +under 24 GB of graphics memory and the GPU reader from 24 GB (`--vision cpu` and `--vision gpu` change it). The encoder on +the GPU takes about 1.1 GiB of VRAM. ## Other GPUs Only the optional `w4a4` needs Blackwell. The default `w4a8` is ggml's int8 MMQ (sm_75 and up), and decode uses the -engine's own q8_1 kernels, which decode UE4M3/E2M1 in software. The engine was built for sm_75/86/89 as PTX and run -on the RTX 5090 with the host answering as that generation. On those builds `mmq_nvfp4_parity` and -`nvfp4_expert_gpu_parity` gave results identical to sm_120. +engine's own q8_1 kernels, which decode UE4M3/E2M1 in software. The engine was built for sm_75/86/89 as PTX and run on an +RTX 5090 with the host answering as that generation. On those builds `mmq_nvfp4_parity` and `nvfp4_expert_gpu_parity` gave +results identical to sm_120. -HIP: the decode kernels use no CUDA-only intrinsics. The NVFP4 MMQ instance and the W4A8 unit are CUDA-only and -guarded (`GGML_USE_HIP`). Neither was built for HIP here. +HIP: the decode kernels use no CUDA-only intrinsics. The NVFP4 MMQ instance and the W4A8 unit are CUDA-only and guarded +(`GGML_USE_HIP`). Neither was built for HIP. Setup does not offer the NVFP4 packs on AMD and Intel. ## Tests @@ -227,26 +188,109 @@ guarded (`GGML_USE_HIP`). Neither was built for HIP here. ## NVIDIA's FP4-Local checkpoint `nvidia/Qwen3.8-Flash-Next-FP4-Local` (setup size **NVFP4-QAD**) is NVIDIA's own retrained 4-bit model: ModelOpt 0.47 -safetensors, experts retrained by quantization-aware distillation. Strata reads it by converting it on the user's PC; -nothing of NVIDIA's weights is redistributed (licenses: NVIDIA Open Model Agreement and Qwen Community License 1.0). +safetensors, experts retrained by quantization-aware distillation. Strata reads it by converting it on the user's PC. +Nothing of NVIDIA's weights is redistributed (licenses: NVIDIA Open Model Agreement and Qwen Community License 1.0). What the checkpoint holds, and what `tools/nvfp4_local_pack.py` does with it: | In the checkpoint | Becomes | | --- | --- | -| routed experts: NVFP4 (E2M1, E4M3 scale per 16, F32 scale per tensor); the scales stored in NVIDIA's "CSF" form (`.nvfp4_csf_fixed` U8 + `.nvfp4_csf_exceptions` U32) | the scales are decoded to the plain E4M3 matrix (byte-exact; the format is the one vLLM's loader documents), then repacked exactly like every other NVFP4 pack: `experts.bin` with the same 36-byte super-blocks and the 16-byte scale tail. Checked: 304 complete experts and the whole `experts.bin` (SHA-256) equal the route through a full GGUF | -| attention, GDN projections, shared experts, indexer: MXFP8 (E4M3 values, E8M0 scale per 32) | dequantized exactly, then Q8_0 (same 32-value blocks; +0.6% RMS on top of MXFP8's own ~3%), or BF16 for the small projections the BF16 kernels read | -| n-gram (PLE) table, 26.8 GiB: NVFP4 rows of 160 values | a one-tensor GGUF of 90-byte rows (80 B codes + 10 E4M3 scales, one F32 for the table), read by the new PLE format `NVFP4` (`strata.ple.format = nvfp4x16`) in `ngram.cpp`; same row size as IQ4_NL | -| MTP head: NVFP4 experts, BF16 rest | a Q2_0 draft layer built with `mtp_pack.py` / `mtp_rt.py` (88.7% of drafts accepted over 225 test requests) | -| vision tower | not converted. Pictures are read by the original model's picture reader (the shared `mmproj` file that setup downloads, as for the other NVFP4 versions); the dense GGUF supplies the vocabulary | +| Routed experts: NVFP4 (E2M1, E4M3 scale per 16, F32 scale per tensor); the scales stored in NVIDIA's "CSF" form (`.nvfp4_csf_fixed` U8 + `.nvfp4_csf_exceptions` U32) | The scales are decoded to the plain E4M3 matrix (byte-exact; the format is the one vLLM's loader documents), then repacked exactly like every other NVFP4 pack: `experts.bin` with the same 36-byte super-blocks and the 16-byte scale tail. Checked: 304 complete experts and the whole `experts.bin` (SHA-256) equal the route through a full GGUF. | +| Attention, GDN projections, shared experts, indexer: MXFP8 (E4M3 values, E8M0 scale per 32) | Dequantized exactly, then Q8_0 (same 32-value blocks; +0.6% RMS on top of MXFP8's own ~3%), or BF16 for the small projections the BF16 kernels read. | +| N-gram (PLE) table, 26.8 GiB: NVFP4 rows of 160 values | A one-tensor GGUF of 90-byte rows (80 B codes + 10 E4M3 scales, one F32 for the table), read by the PLE format `NVFP4` (`strata.ple.format = nvfp4x16`) in `ngram.cpp`; same row size as IQ4_NL. | +| MTP head: NVFP4 experts, BF16 rest | A Q2_0 draft layer built with `mtp_pack.py` / `mtp_rt.py` (88.7% of drafts accepted over 225 test requests). | +| Vision tower | Not converted. Pictures are read by the original model's picture reader (the shared `mmproj` file that setup downloads); the dense GGUF supplies the vocabulary. | `python tools/nvfp4_local_pack.py prepare --model --out ` runs every step (setup does this): the n-gram table, the dense GGUF (no experts), the pack (experts written straight from the checkpoint) and the draft layer. It needs -about 110 GB of free disk beside the 102 GB download, which can be deleted afterwards. On a Threadripper PRO with the files -on NVMe it took about 12 minutes. +about 110 GB of free disk beside the 102 GB download, which can be deleted afterwards. On a Threadripper PRO with the +files on NVMe it took about 12 minutes. -Measured on an RTX PRO 5000 72 GB (Blackwell), 128K context, int8 KV, every expert in VRAM (`tools/strata_bench.py`): -decode 159 tokens/s (median of chat answers, 137-216), 4K prompt 3,981 tokens/s, 16K 4,813, 32K 4,916 (first token 6.6 s), -agent session 151 tokens/s; the engine holds 69 GB of RAM beside 72 GB of VRAM. Quality is in the table below -(`tools/quant_quality.py`, set v1, against Unsloth's Q8_0): top-1 90.5%, mean KL 0.085, perplexity 3.95 (Q8_0: 4.05). -This is a retrained model, so its distance to the original's Q8_0 includes real weight changes, not only rounding. +> **Measured on:** RTX PRO 5000 72 GB (Blackwell), 128K context, int8 KV, every expert in VRAM +> (`tools/strata_bench.py`). Strata version not recorded. + +| Measure | Result | +| --- | --- | +| Decode (median of chat answers) | 159 tok/s (137 to 216) | +| Prompt, 4K / 16K / 32K | 3,981 / 4,813 / 4,916 tok/s (first token at 32K: 6.6 s) | +| Agent session | 151 tok/s | +| Memory | 69 GB of RAM beside 72 GB of VRAM | + +Quality (`tools/quant_quality.py`, set v1, against Unsloth's Q8_0): top-1 90.5%, mean KL 0.085, perplexity 3.95 (Q8_0: +4.05). This is a retrained model, so its distance to the original's Q8_0 includes real weight changes, not only rounding. + +## History / older measurements + +### The fork author's measurements + +> **Measured on:** Windows 11, RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB DDR5-5600, 64K context, int8 K/V, +> `--vram-reserve-mib 1500`, the fork author's engine. + +| | NVFP4 pack | IQ2_XS, same engine | +| --- | ---: | ---: | +| Experts in RAM | 63.3 GiB | 33.0 GiB | +| Expert cache slots | about 8,350 | about 17,300 | +| Decode, 400-token answer | 114 to 116 tok/s | 132 to 139 tok/s | +| A 32K prompt | 3,900 tok/s | 5,600 tok/s | + +PCIe share sweep at 262K context (decode tok/s): + +| PCIe share | Short prompt | 32K prompt | +| --- | ---: | ---: | +| 0.55 | 103 | 105 | +| 0.40 | 110 | 117 | +| 0.25 | 117 | 124 | +| 0.10 | 112 | 118 | + +### Measured in Strata (2026-10-13) + +> **Measured on:** the ready-made GPTQ pack (`--family nvfp4 --model NVFP4`), Strata 0.1.44 era. Quality on an RTX 3060 +> box with `tools/quant_quality.py`: 4,995 teacher-forced positions of code, docs, English, other languages and an agent +> transcript, against Unsloth's Q8_0. + +| | Same top token as Q8_0 | Mean KL | KL p99 | +| --- | ---: | ---: | ---: | +| NVFP4 GPTQ, experts on the CPU | 92.09% | 0.070 | 1.12 | +| NVFP4 GPTQ, 1,500 experts on the GPU | 91.79% | 0.069 | 1.07 | +| The same with `STRATA_ROUTE_TAIL_SKIP=7` | 88.59% | 0.118 | 1.76 | +| NVFP4 GPTQ with the GSQ-RCO IQ4_NL n-gram table instead of the FP8 one | 92.15% | 0.069 | 1.07 | +| For comparison, Unsloth UD-Q5_K_XL | 94.55% | 0.038 | 0.64 | +| Unsloth UD-Q4_K_XL | 93.09% | 0.056 | - | +| Unsloth UD-IQ4_XS | 88.91% | 0.101 | 1.60 | +| GSQ-RCO IQ3_S | 87.69% | 0.144 | 1.98 | + +`STRATA_ROUTE_TAIL_SKIP=7` gave +7 to 25% decode on the RTX 3060 and +47 to 55% on the RTX 5070 (64 GB). + +> **Measured on:** decode experts (`nvfp4_decode_bench`, 64 distinct experts per call), share of the card's DRAM bandwidth +> in brackets. + +| Tokens per expert | RTX 5070 | RTX 3060 | Tesla P100 | +| --- | ---: | ---: | ---: | +| 1 | 0.294 ms (89%) | 0.554 ms (89%) | 0.910 ms | +| 2 | 0.298 ms (88%) | 0.596 ms | 1.247 ms | +| 4 | 0.335 ms (79%) | 0.766 ms | 1.954 ms | + +Each weight is decoded once for all the tokens of a verify window, bit for bit the same results. Before the kernel work, +the RTX 3060 took 2.26 ms at 4 tokens. + +> **Measured on:** prompt experts on an RTX 5070 (`mmq_expert_bench`, one launch of 16 experts with 625 rows each, +> gate/up + down); first-token KL against the exact FP16 path over 24 prompts of 1.5K to 8K tokens. + +| Mode | Time | First-token KL | +| --- | ---: | ---: | +| `w4a8` (default) | 1.52 ms | 0.0068 | +| `w4a4x2` (`STRATA_PREFILL_NVFP4=w4a4x2`) | 1.13 ms | 0.0056 | +| `w4a4x2` with down too (`STRATA_PREFILL_X2_MIN_K=640`) | 0.995 ms | 0.0044 | +| UD-IQ4_XS's experts | 1.20 ms | - | + +On a 64 GB PC the prompt waits for the experts read from the SSD, so the modes give the same prompt speed there. `w4a8` +stays the default until a 96 GB or larger RTX 50 PC measures the others. + +> **Measured on:** RTX 5070 and RTX 3060, images with an NVFP4 pack against IQ3_XXS's answers on the same picture (a +> number in the picture read, two shapes with their colours and sides). + +The answers matched. On the 3060 at 8K context the cache held 1,293 experts with images (GPU encoder) against 1,734 +without. + +Mapped experts (`--mmap-experts`) on a 64 GB Windows PC filled the RAM, which is why setup uses the RAM-budget mode below +the RAM need. diff --git a/docs/NVIDIA_V100.md b/docs/NVIDIA_V100.md index 1536d0ea0..41f5e0bcc 100644 --- a/docs/NVIDIA_V100.md +++ b/docs/NVIDIA_V100.md @@ -1,12 +1,22 @@ -# NVIDIA Tesla V100 / Titan V (Volta, sm_70): the community build +# NVIDIA Tesla V100 / Titan V (Volta, sm_70) -Strata's ready-made engine and the support matrix start at compute capability 7.5 (RTX 20). A V100 (7.0) runs the same -engine compiled for it, through the experimental build that issue #236 added. It is **community-tested, not part of the -supported cards**: no ready-made engine is published for it and the numbers below come from one machine. +This page is for people running Strata on a V100 or Titan V. Volta uses the experimental CUDA 12 engine, which current releases +provide as an engine component through the app installer or the Terminal ([INSTALL.md](INSTALL.md)). CUDA 13 cannot run Volta. +How setup selects the CUDA 12 engine and the driver requirements are in [OLDER_GPUS.md](OLDER_GPUS.md). Speeds below were +reported on the named machines. They are not a guarantee for another V100 setup. -## Build +> **On this page:** [Quick start](#quick-start) · [What runs on Volta](#what-runs-on-volta) · [Opt-in switches](#opt-in-switches) · +> [Checking the kernels](#checking-and-benchmarking-the-volta-kernels) · [Limits](#limits) · [History](#history--older-measurements) -CUDA 13 dropped Volta, so the build needs a **CUDA 12.x** toolkit (12.8 was used) and a host GCC that toolkit accepts. +## Quick start + +1. Install the app or Terminal for your system ([INSTALL.md](INSTALL.md)). Setup picks the CUDA 12 engine when the model runs on + a Volta card. +2. To choose it by hand from a source checkout: `START-HERE.bat --setup --gpu 0 --cuda 12` (Linux: `./setup.sh`). + +### Build from source + +CUDA 13 dropped Volta, so the build needs a CUDA 12.x toolkit (12.8 was used) and a host GCC that toolkit accepts. ```sh cmake -S . -B build -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF \ @@ -14,53 +24,49 @@ cmake -S . -B build -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF \ cmake --build build --target strata -j ``` -Setup does the same by itself for a model on a V100 (the experimental CUDA 12 engine, in `engine-cuda12/`: ready-made -on Windows, compiled with a CUDA 12.x toolkit on Linux; [OLDER_GPUS.md](OLDER_GPUS.md)). For a PC with a V100 and a newer card, a build for both architectures (`-DCMAKE_CUDA_ARCHITECTURES="70;86"`) is the -natural choice (a community report in issue #509 ran an RTX 3080 beside a V100 that way; it was not repeated here). On the -measuring PC a Quadro RTX 4000 (sm_75) sits beside the V100s and was hidden with `CUDA_VISIBLE_DEVICES` because the sm_70-only -build has no code for it. +A source checkout can fetch or build its CUDA 12 engine; `--build` requests compilation with the toolkit (see +[OLDER_GPUS.md](OLDER_GPUS.md)). For a PC with a V100 and a newer card, build for both architectures +(`-DCMAKE_CUDA_ARCHITECTURES="70;86"`). A sm_70-only build has no code for a card such as a Quadro RTX 4000 (sm_75); hide it with +`CUDA_VISIBLE_DEVICES`. ## What runs on Volta -The flag only lowers the floor (CMake and the runtime device check) and replaces two intrinsics Volta lacks. It does not -select "old" kernels everywhere: which kernel runs is decided per architecture when the engine is built. +The flag lowers the floor (CMake and the runtime device check) and replaces two intrinsics Volta lacks. Which kernel runs is +decided per architecture when the engine is built. | Part of the prompt path | On sm_70 | | --- | --- | -| MoE experts | opt-in (`STRATA_PF_WMMA=1`, only when every visible card is compute capability 7+; MMQ otherwise): `gemm_iq_f16_grouped`: a group of experts per launch on the FP16 tensor cores, the weights dequantized in shared memory (+10% prompt speed against MMQ's `dp4a` kernels, [bench](../bench/results/2026-10-07-v100-prompt-experts/README.md); FP16 activations instead of MMQ's q8_1, so the answers differ) | +| MoE experts | MMQ by default. Opt-in `STRATA_PF_WMMA=1` (only when every visible card is compute capability 7 or newer): `gemm_iq_f16_grouped`, a group of experts per launch on the FP16 tensor cores. FP16 activations instead of MMQ's q8_1, so answers differ. | | Dense projections (dequantized weights) | FP16 tensor-core GEMMs (cuBLAS / CUTLASS `s884`) | -| BF16 projections (hyper-connection, router, indexer, ...) | converted to FP16 and run on the FP16 tensor cores (#655, #540; `STRATA_BF16_TC=0`: cuBLAS BF16, an FP32 SIMT kernel on Volta) | -| QSA attention for decode and verify windows (and prompts with `STRATA_PROMPT_ATTN_OLD=1`) | #540's kernel (fewer shuffles, bit-exact; `STRATA_ATTN_PRE75=0`: the one other cards run) | +| BF16 projections (hyper-connection, router, indexer) | converted to FP16 and run on the FP16 tensor cores (`STRATA_BF16_TC=0`: cuBLAS BF16, an FP32 SIMT kernel on Volta) | +| QSA attention for decode and verify windows | the #540 kernel, fewer shuffles, bit-exact (`STRATA_ATTN_PRE75=0`: the one other cards run) | | QSA prompt attention, int8 / FP16 / K8V4 KV | `prompt_attn_v70_kernel` on `mma.m8n8k4` (`STRATA_PROMPT_ATTN_OLD=1`: the decode kernel, one query at a time) | | QSA prompt attention, Q4_0 KV | the decode kernel | | QSA block scores | the warp kernel (the tensor-core scorer needs sm_80) | -| Part of a decode / verify window | On sm_70 | +| Part of a decode or verify window | On sm_70 | | --- | --- | -| Routed experts in VRAM | gfx906's expert mode 8: the codebook grid and the group's activations in shared memory, SwiGLU fused (`STRATA_EXP_MODE=0`: the CUDA layout other cards run) | -| Dense 2-4 column GEMVs and the head | the interleaved `native_mmvq_il` with a Volta rows table (`STRATA_MMVQ_IL=0`: `native_mmvq`) | -| Hyper-connection read, up projection | gfx906's latency-hidden up kernel (`STRATA_GR_FAST=0`: the plain one) | +| Routed experts in VRAM | expert mode 8 (codebook grid and the group's activations in shared memory, SwiGLU fused). Opt-in, see below. | +| Dense 2-4 column GEMVs and the head | the interleaved `native_mmvq_il` with a Volta rows table. Opt-in, see below. | +| Hyper-connection read, up projection | the latency-hidden up kernel. Opt-in, see below. | -All three are bitwise the kernels they replace; on a V100-SXM2 they take a verify window's GPU work from 22.8 to 21.2 ms -([bench/results/2026-10-07-v100-decode-kernels](../bench/results/2026-10-07-v100-decode-kernels/README.md)). +## Opt-in switches -## Measured +None of these is on by default. Each was measured by its contributor on a V100, not by the maintainers. -One V100-PCIE-32GB, i9-7960X, Unsloth `UD-IQ4_XS` (not a setup model), int8 KV, MTP `--spec 2`; prompt read speed of random-word prompts, tokens/s, -with and without the Volta attention kernel (`STRATA_PROMPT_ATTN_OLD=1`): +| Switch | What it does | +| --- | --- | +| `STRATA_SM70_TABLE=1` | Turns on the three decode kernels in the second table above together. Bitwise the default kernels' output. Individual switches `STRATA_EXP_MODE`, `STRATA_MMVQ_IL` and `STRATA_GR_FAST` still apply on top. | +| `STRATA_FUSE_DECODE=1` | The decode GEMV / MoE fusions of #1097. Bitwise the unfused kernels. | +| `STRATA_PF_WMMA=1` | The FP16 tensor-core prompt experts; also takes IQ4_NL down projections. | +| `STRATA_PF_WMMA2=1` | A second prompt-expert kernel for IQ2_S gate/up and IQ4_NL down (#2356). | +| `STRATA_ARGMAX_ROWS_SM70=1` | The multi-block greedy pick on sm_7x (#2357), bitwise the one-block pick. | +| `STRATA_GR_NORM_FAST=1` | The register-split hyper-connection norm (#2358). | +| `STRATA_Q8_GATHER_WIDE=0` | Turns off the 8-wide int8 KV gather (#1095) that the experimental build uses below compute capability 7.5. | +| `STRATA_GDN_REC_WARP=2` | The chain-split DeltaNet recurrence of the prompt (#1098). Changes the FP32 summation order, so the default stays the pipelined kernel. `1` is the bit-exact sync-free variant; `STRATA_GDN_CHUNK=1` the chunked path. | -| Prompt tokens | old attention path | `prompt_attn_v70_kernel` | -| --- | ---: | ---: | -| 7,194 | 1,010 | 1,164 | -| 28,650 | 1,063 | 1,251 | -| 114,338 | 973 | 1,123 | - -Decode was 38-51 tok/s in every run and is not affected. The profile, the parity numbers, the needle test and the limits are in -[bench/results/2026-10-03-v100-prompt-attn](../bench/results/2026-10-03-v100-prompt-attn/README.md). - -What the kernel does and does not guarantee: against an FP64 reference its error is about 2-3e-6 (output scale 3.6; the FP32 kernel it replaces: 2e-6; 5e-6 before the hi and lo halves got separate accumulator chains); 15 of 15 -needles were found; greedy output was identical on the 28,650- and 114,338-token prompts and differed late in the answer on the 7,194-token one (another -summation order). +Volta's tensor cores take FP16 only. A model's BF16 weights are converted, and a weight outside FP16's range would saturate (none +did in the check in the history section). ## Checking and benchmarking the Volta kernels @@ -69,37 +75,41 @@ cmake -S . -B build -DSTRATA_PARITY_PROMPT_ATTN=ON ... && cmake --build build -- CUDA_VISIBLE_DEVICES=0 ./build/qsa_prompt_attn_parity 32768 2048 5 # synthetic, no model; exit 0 = PASS ``` -It compares the tensor-core attention with the FP32 kernel and an FP64 host reference and times both. The Q4_0 cases are skipped below -sm_80. +It compares the tensor-core attention with the FP32 kernel and an FP64 host reference and times both. The Q4_0 cases are skipped +below sm_80. ## Limits -- A layer split over two V100 was not measured with this kernel. The cards work in turn on a single request, so a split adds expert-cache room, not prompt speed. -- sm_60 (Pascal) is covered by the same flag, and community runs exist: two Tesla P40 with a layer split (#1028, bench - [2026-10-05-community-2x-p40](../bench/results/2026-10-05-community-2x-p40/README.md)), a Tesla P100 alone and with an RTX 2070 SUPER as - the expert-helper card (#1069: 30.5 tok/s alone, 35.8-38.4 with the helper), and a V100 + P100 pair (#1079: +6-11% decode from 0.1.39 to - 0.1.40 with the P100 as helper cache). One reporter's numbers each; this page's own kernel measurements are V100 only. -- Volta's tensor cores take FP16 only: a model's BF16 weights are converted, and a weight outside FP16's range would saturate (none did in - the check above). +- A layer split over two V100 was not measured with this kernel. The cards work in turn on a single request, so a split adds + expert-cache room, not prompt speed. +- Pascal (sm_60) is covered by the same flag. Community runs exist for it; this page's own kernel measurements are V100 only. -## The decode kernels measured on a V100 (opt-in: `STRATA_SM70_TABLE=1`) +## History / older measurements -The interleaved 2-4 column matvec with a Volta rows table, expert mode 8 and the latency-hidden norm/up (bitwise the default -kernels' output, GPU work per verify window 22.8 -> 21.2 ms on a V100-SXM2 in the contributor's measurement) are used on -sm_70 only when `STRATA_SM70_TABLE=1` is set. They become the default once the contributor has confirmed them on a V100 with -the 0.1.41 code. Without the variable sm_70 runs what 0.1.40.3 ran. The single switches (`STRATA_EXP_MODE`, `STRATA_MMVQ_IL`, -`STRATA_GR_FAST`) still apply on top. +**Prompt attention kernel.** Measured on: one V100-PCIE-32GB, i9-7960X, Unsloth UD-IQ4_XS, int8 KV, MTP `--spec 2`, random-word +prompts, prompt read speed in tok/s with and without the Volta kernel (`STRATA_PROMPT_ATTN_OLD=1`). Decode was 38-51 tok/s in +every run and is not affected. Report: [bench/results/2026-10-03-v100-prompt-attn](../bench/results/2026-10-03-v100-prompt-attn/README.md). -## More opt-in switches (CUDA 12 experimental build) +| Prompt tokens | old attention path | `prompt_attn_v70_kernel` | +| --- | ---: | ---: | +| 7,194 | 1,010 | 1,164 | +| 28,650 | 1,063 | 1,251 | +| 114,338 | 973 | 1,123 | -None of these is on by default. Each was measured by its contributor on a V100, not by the maintainers. +Against an FP64 reference the kernel's error is about 2-3e-6 (the FP32 kernel it replaces: 2e-6). 15 of 15 needles were found. +Greedy output was identical on the 28,650- and 114,338-token prompts and differed late in the answer on the 7,194-token one. -| Switch | What it does | Contributor's number | -| --- | --- | --- | -| `STRATA_FUSE_DECODE=1` | the decode GEMV / MoE operator fusions of #1097 (gate/up + SwiGLU in one kernel, SwiGLU + quantizer in one kernel, all activation images of a vector in one pass, two BF16 projections in one kernel); bitwise the unfused kernels | 4-16 us per token saved, not visible end to end | -| `STRATA_PF_WMMA=1` | the FP16 tensor-core prompt experts (above); also takes IQ4_NL down projections (K=640, #1771) | +10% prompt speed (#1401) | -| `STRATA_PF_WMMA2=1` | a second prompt-expert kernel for IQ2_S gate/up and IQ4_NL down (#2356) | +2.5-4.3% prompt speed | -| `STRATA_ARGMAX_ROWS_SM70=1` | the multi-block greedy pick on sm_7x (#2357), bitwise the one-block pick | +0.7-0.8% decode | -| `STRATA_GR_NORM_FAST=1` | the register-split hyper-connection norm on CUDA (#2358) | +0.5-0.6% decode | -| `STRATA_Q8_GATHER_WIDE=0` | turns off the 8-wide int8 KV gather (#1095) that the experimental build uses below compute capability 7.5 | +11% on the gather kernel | -| `STRATA_GDN_REC_WARP=2` | the chain-split DeltaNet recurrence of the prompt on a V100 (#1098); changes the summation order (FP32 level), so the default stays the pipelined kernel; `1` is the bit-exact sync-free variant, `STRATA_GDN_CHUNK=1` the chunked path | +6-9% on the recurrence | +**Other measurements** + +| Measured on | Result | +| --- | --- | +| V100-SXM2, contributor's measurement | decode kernels (`STRATA_SM70_TABLE=1`): GPU work per verify window 22.8 -> 21.2 ms ([report](../bench/results/2026-10-07-v100-decode-kernels/README.md)) | +| V100, `STRATA_PF_WMMA=1` | prompt +10% against MMQ ([report](../bench/results/2026-10-07-v100-prompt-experts/README.md)) | +| V100, `STRATA_PF_WMMA2=1` | prompt +2.5-4.3% | +| V100, `STRATA_ARGMAX_ROWS_SM70=1` / `STRATA_GR_NORM_FAST=1` | decode +0.7-0.8% / +0.5-0.6% | +| V100, `STRATA_Q8_GATHER_WIDE=0` | +11% on the gather kernel | +| V100, `STRATA_GDN_REC_WARP=2` | +6-9% on the recurrence | +| V100, `STRATA_FUSE_DECODE=1` | 4-16 us per token saved, not visible end to end | +| Tesla P100 alone and with an RTX 2070 SUPER as expert helper (#1069) | 30.5 tok/s alone, 35.8-38.4 with the helper | +| Two Tesla P40, layer split (#1028) | [report](../bench/results/2026-10-05-community-2x-p40/README.md) | +| V100 + P100 pair (#1079) | +6-11% decode from engine 0.1.39 to 0.1.40 with the P100 as helper cache | diff --git a/docs/OLDER_GPUS.md b/docs/OLDER_GPUS.md index 0bb73ccca..92c26f9c8 100644 --- a/docs/OLDER_GPUS.md +++ b/docs/OLDER_GPUS.md @@ -1,51 +1,61 @@ # Older GPUs (experimental) -Strata's supported cards are NVIDIA RTX 20 / 30 / 40 / 50 and the AMD cards in [AMD_HIP.md](AMD_HIP.md). The cards -below run through **opt-in, experimental** paths that community members wrote and measured on their own machines. The -maintainers have one of these cards, a Tesla P100, and test on it; they have none of the others: each path is -compile-checked and unit-tested here, and the ready-made engines and their output stay exactly as they were. Numbers are -the reporters' own, on one machine each, unless the text says the maintainers measured them. +This page is for people with an NVIDIA card older than the RTX 20 series, an older AMD card, or a Turing card they want to tune. +For a normal install use the app installer for your hardware or the [Terminal](INSTALL.md#the-terminal). Pascal and Volta use the +CUDA 12 engine package. The `--cuda`, `--gpus`, `START-HERE.bat` and `UPDATE.bat` examples below are +[source-setup](SOURCE_INSTALL.md) options, not Terminal launch flags. + +Strata's supported cards are NVIDIA RTX 20 / 30 / 40 / 50 and the AMD cards in [AMD_HIP.md](AMD_HIP.md). The cards below run +through opt-in, experimental paths that community members wrote and measured. The maintainers own one of these cards, a Tesla +P100, and test on it. Every other path is compile-checked and unit-tested, and the ready-made engines and their output are +unchanged. Numbers are the reporters' own, on one machine each, unless the text says the maintainers measured them. + +> **On this page:** [Support matrix](#support-matrix) · [CUDA 12 engine](#nvidia-the-cuda-12-engine) · +> [Tesla P100](#tesla-p100-sm_60) · [GP100 Q8_0 kernel](#pascal-gp100-a-q8_0-decode-kernel-opt-in) · +> [Tesla P40](#tesla-p40-sm_61-what-was-measured-and-changed) · [Turing switches](#quadro-rtx-8000-and-other-turing-cards) · +> [AMD](#amd-building-gfx906-and-gfx1012) · [History](#history--older-measurements) ## Support matrix ### NVIDIA -| Cards | Compute capability | How it runs | What is different on it | Reported | -| --- | --- | --- | --- | --- | -| Tesla P100 | 6.0 | the CUDA 12 engine | `__dp4a` emulated (bit-exact); BF16 projections through fp32; opt-in: `STRATA_Q8_SM60=1` runs the Q8_0 dense projections and head on a GP100 kernel ([below](#pascal-gp100-a-q8_0-decode-kernel-opt-in)) | 2x P100, IQ3_S, engine 0.1.39: prompt 526-532 tok/s, decode 28-32 tok/s at 128K prompt tokens, 3 of 3 needle checks at 122K ([report](../bench/results/2026-10-06-community-2x-p100/README.md), #1157); PH402 (4 GP100 dies), IQ3_XXS: decode 28.3 -> 38.9 tok/s with `STRATA_Q8_SM60=1` and a Q8_0 dense shard | -| Tesla P40 / P4, GTX 10 series | 6.1 | the CUDA 12 engine | BF16 projections through fp32 (cuBLAS has no BF16 GEMM there, #395); the dense weights dequantized by a wide kernel, expert mode 8 and its own interleaved-matvec table; opt-in `STRATA_DENSE_F32=1`: the large dense GEMMs as `cublasSgemm` ([below](#tesla-p40-sm_61-what-was-measured-and-changed)) | P40, IQ3_S, engine 0.1.30: prompt 217-374 tok/s, decode 30-33 tok/s (#395); 0.1.43.1 with the sm_61 changes: prompt 255 tok/s at 565 tokens (was 205), decode 37-38 tok/s (was 34-35); with `STRATA_DENSE_F32=1` 464-474 tok/s at 7K tokens (was 360-432) | -| Tesla V100, Titan V | 7.0 | the CUDA 12 engine | BF16 projections on the FP16 tensor cores (#655, #540); the prompt attention on `mma.m8n8k4` (#600); a leaner attention kernel (#540) | V100-PCIE-32GB, UD-IQ4_XS: prompt 1,123-1,251 tok/s (#600); V100 32GB, IQ2_XS: prompt +22% from #540 | -| RTX 20 (Turing) | 7.5 | **supported**, the ready-made engine | opt-in: `STRATA_BF16_TC=1` runs the BF16 projections on the FP16 tensor cores | RTX 2080 Ti, Q2_0: prompt +15-18% (#655) | +| Cards | Compute capability | How it runs | What is different on it | +| --- | --- | --- | --- | +| Tesla P100 | 6.0 | the CUDA 12 engine | `__dp4a` emulated (bit-exact); BF16 projections through fp32; seven scheduling switches on by default ([below](#tesla-p100-sm_60)); opt-in `STRATA_Q8_SM60=1` ([below](#pascal-gp100-a-q8_0-decode-kernel-opt-in)) | +| Tesla P40 / P4, GTX 10 series | 6.1 | the CUDA 12 engine | BF16 projections through fp32 (cuBLAS has no BF16 GEMM there); dense weights dequantized by a wide kernel; expert mode 8 and its own interleaved-matvec table; opt-in `STRATA_DENSE_F32=1` ([below](#tesla-p40-sm_61-what-was-measured-and-changed)) | +| Tesla V100, Titan V | 7.0 | the CUDA 12 engine | BF16 projections on the FP16 tensor cores; prompt attention on `mma.m8n8k4`; a leaner attention kernel. See [NVIDIA_V100.md](NVIDIA_V100.md) | +| RTX 20 (Turing) | 7.5 | **supported**, the ready-made engine | opt-in `STRATA_BF16_TC=1` runs the BF16 projections on the FP16 tensor cores | -A card needs enough VRAM to be useful: 12 GB or more is recommended, as for every card (a 2 GB GT 1030 is compute -capability 6.1 too, and cannot hold any of the model). +A card needs enough VRAM to be useful. 12 GB or more is recommended, as for every card. A 2 GB GT 1030 is compute capability 6.1 +too and cannot hold any of the model. ### AMD -| Cards | Architecture | How it runs | Reported | -| --- | --- | --- | --- | -| RX 6800 / 6900 series | gfx1030 | setup (`--backend hip`), unvalidated; #540's attention kernel is the default there, with 8 cells per step and DPP lane exchanges (bit-exact; `STRATA_ATTN_PRE75=0` runs the standard kernel) | [AMD_HIP.md](AMD_HIP.md#rdna2-gfx1030); RX 6900 XT, IQ3_S: prompts +4-6% alone, +7-12% with #835 ([bench/results/2026-10-04-rdna2-pre75-attention](../bench/results/2026-10-04-rdna2-pre75-attention/README.md)) | -| RX 6700 XT | gfx1031 | setup (`--backend hip`), unvalidated (#524) | used daily by its reporter, one card | -| RX 5500 XT (RDNA1) | gfx1012 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1012` (HIP 5.7 or 7) | 8 GB card, IQ3_S, 8K prompt: 15.3 tok/s decode (#442) | -| RX 5700 XT (RDNA1) | gfx1010 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1010`, on ROCm 7.14's `gfx101X-dgpu` wheels; needs `ROCR_VISIBLE_DEVICES=0` in a PC that also has an AMD iGPU | 8 GB card, Coder IQ1_M, 32K context: prompt 115 / 144 / 149 tok/s and decode 18.0 / 20.3 / 23.3 tok/s at 4K / 16K / 30K prompt tokens, 6 of 6 needle checks at 8K and 30K, 61 of 61 ctest on the card, 607 of 12,288 experts in VRAM | -| RX 5700 and the 6 GB RX 5600 (RDNA1) | gfx1010 | as above | the same Navi 10 silicon as the RX 5700 XT, so the same build; a 6 GB card leaves ~50 expert slots, expect the RX 5500 XT's range, not this one's | -| Radeon PRO V520 / Pro 5600M (RDNA1) | gfx1011 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1011` | untested on hardware: it is the same RDNA1 ISA as gfx1010 (Navi 12), and the engine builds for it with 0 errors on the same wheels | -| Instinct MI50 / MI60, Radeon VII | gfx906 (wave64) | built by hand: `-DSTRATA_HIP_GFX906=ON` | 2x MI50, Coder IQ1_M, 128K context: decode 50.1 / 47.8 / 45.7 tok/s at 4K / 32K / 128K prompt tokens, prompt ~520 tok/s (#677) | +| Cards | Architecture | How it runs | +| --- | --- | --- | +| RX 6800 / 6900 series | gfx1030 | setup (`--backend hip`), unvalidated. The #540 attention kernel is the default there (`STRATA_ATTN_PRE75=0` runs the standard kernel). See [AMD_HIP.md](AMD_HIP.md#rdna2-gfx1030) | +| RX 6700 XT | gfx1031 | setup (`--backend hip`), unvalidated; used daily by its reporter on one card | +| RX 5500 XT (RDNA1) | gfx1012 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1012` (HIP 5.7 or 7) | +| RX 5700 XT (RDNA1) | gfx1010 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1010` on ROCm 7.14's `gfx101X-dgpu` wheels; needs `ROCR_VISIBLE_DEVICES=0` on a PC that also has an AMD iGPU | +| RX 5700, 6 GB RX 5600 (RDNA1) | gfx1010 | as above; a 6 GB card leaves about 50 expert slots | +| Radeon PRO V520 / Pro 5600M (RDNA1) | gfx1011 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1011`; builds with 0 errors, untested on hardware | +| Instinct MI50 / MI60, Radeon VII | gfx906 (wave64) | built by hand: `-DSTRATA_HIP_GFX906=ON` | ## NVIDIA: the CUDA 12 engine -CUDA 13 dropped Pascal and Volta: it cannot compile for them. Setup therefore keeps a **second engine**, built with -CUDA 12.9 and `-DSTRATA_EXPERIMENTAL_SM60=ON`, in its own folder (`engine-cuda12\`, beside `engine\`). One engine runs -per model, so the choice is made per model, by the oldest card that model runs on: +CUDA 13 dropped Pascal and Volta and cannot compile for them. Setup therefore keeps a second engine, built with CUDA 12.9 and +`-DSTRATA_EXPERIMENTAL_SM60=ON`, in its own folder (`engine-cuda12\`, beside `engine\`). One engine runs per model, so the +choice is made per model, by the oldest card that model runs on: -- **Every card the model uses is RTX 20 or newer:** the ready-made CUDA 13 engine, as always. -- **A card is Pascal or Volta:** the CUDA 12 engine. Setup says so (`CUDA 12: sm_70 is older than CUDA 13 supports - ...`). On Windows it downloads `strata-windows-x64-cuda12.zip` with NVIDIA's CUDA 12 libraries (from pip, like the - CUDA 13 ones); on Linux, or with `--build`, it compiles the engine with a CUDA 12.x toolkit. +- Every card the model uses is RTX 20 or newer: the ready-made CUDA 13 engine. +- A card is Pascal or Volta: the CUDA 12 engine. Setup says so (`CUDA 12: sm_70 is older than CUDA 13 supports ...`). + +Release builds deliver engine components through the signed feed (`strata update`). For a source checkout, `--build` compiles with +a CUDA 12.x toolkit ([source installation](SOURCE_INSTALL.md)). ### Opting in -An older card is used only when you choose it; a PC with a newer card keeps recommending the newer one. +An older card is used only when you choose it. A PC with a newer card keeps recommending the newer one. | You | Setup | | --- | --- | @@ -53,22 +63,21 @@ An older card is used only when you choose it; a PC with a newer card keeps reco | name the card: `START-HERE.bat --setup --gpu 1`, or `--gpus 0,1` with a newer card | uses it; the model gets the CUDA 12 engine | | `--cuda 12` (or `STRATA_CUDA=12`) | the CUDA 12 engine for this model, on any card | | `--cuda 13` | the CUDA 13 engine even with an older card (a warning: it has no code for that card) | -| `STRATA_EXPERIMENTAL_SM60=1` | the older cards are listed as usable (the setting from #295 still works) | +| `STRATA_EXPERIMENTAL_SM60=1` | the older cards are listed as usable | -The choice is kept in the model's config (`"cuda": 12`): its starts and `UPDATE.bat` keep it, and other models keep -their own engine. Setting the model up again chooses again (by its cards; add `--cuda 12` to keep a forced choice). A Pascal / Volta card added to a model at a start (`--gpus`) moves that model to the -CUDA 12 engine. +The choice is kept in the model's config (`"cuda": 12`). Starts and `UPDATE.bat` keep it, and other models keep their own engine. +Setting the model up again chooses again (add `--cuda 12` to keep a forced choice). A Pascal or Volta card added to a model at a +start (`--gpus`) moves that model to the CUDA 12 engine. -`--cuda 12` is also the way to run Strata with an NVIDIA driver older than 580: CUDA 12 needs 528 or newer on Windows -(527.41, NVIDIA's minor-version compatibility) and 525 on Linux; setup's "driver too old" stop says so. Such old -drivers were not tested here. +`--cuda 12` is also the way to run Strata with an NVIDIA driver older than 580. The CUDA 12 engine needs driver 528 or newer on +Windows and 525 on Linux (`CUDA12_MIN_DRIVER` in `setup.py`); setup's "driver too old" stop says so. The Docker `cuda12` image +asks for 575 ([DOCKER.md](DOCKER.md)). Drivers that old were not tested here. ### Mixed cards -A model that shares a V100 with an RTX 30 / 40 card runs both on the CUDA 12 engine (it has code for sm_60 to sm_89, -plus PTX). An RTX 50 card (sm_120) in a CUDA 12 engine is a warning, not a stop: CUDA 12.8 and newer compile for it, -but engines built with 12.8 crashed on long prompts there (#220, #224). Keep the RTX 50 card on its own model (`--gpu -N`), where it runs the CUDA 13 engine. +A model that shares a V100 with an RTX 30 / 40 card runs both on the CUDA 12 engine (code for sm_60 to sm_89, plus PTX). An RTX +50 card (sm_120) in a CUDA 12 engine is a warning, not a stop: engines built with 12.8 crashed on long prompts there (#220, +#224). Keep the RTX 50 card on its own model (`--gpu N`), where it runs the CUDA 13 engine. ### Building it yourself @@ -79,71 +88,167 @@ cmake -S . -B build-cuda12 -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF -DST cmake --build build-cuda12 --target strata -j ``` -Setup does the same when it compiles: it looks for the newest CUDA 12.x toolkit (`STRATA_NVCC=` picks -one, #601; on glibc 2.43 use 12.8, see [TROUBLESHOOTING.md](TROUBLESHOOTING.md)). +Setup does the same when it compiles: it looks for the newest CUDA 12.x toolkit. `STRATA_NVCC=` picks one. On +glibc 2.43 use 12.8 ([TROUBLESHOOTING.md](TROUBLESHOOTING.md)). On GCC 12.3 with nvcc (openEuler 24.03, CUDA 12.8) one report +needed `-D_BITS_OPT_RANDOM_H` added to the host flags (#1074, not reproduced here). -On GCC 12.3 with nvcc (openEuler 24.03, CUDA 12.8, 2x V100) the build needed `-D_BITS_OPT_RANDOM_H` added to the host flags (#1074; one -report, not reproduced here). The `size_t` error in `vmm.hpp` that the same report hit is fixed in 0.1.40.2. +### What the flag changes -### What the flag changes, and what it does not +`-DSTRATA_EXPERIMENTAL_SM60=ON` lowers the runtime floor to compute capability 6.0 and compiles the older cards' code paths into +that build only: the Volta prompt attention, the #540 attention kernel (used below sm_75), and the Pascal BF16 path. The +ready-made CUDA 13 engine has none of them, so its kernels and output are unchanged. -`-DSTRATA_EXPERIMENTAL_SM60=ON` lowers the runtime floor to compute capability 6.0 and compiles the older cards' code -paths into that build only: the Volta prompt attention (#600), #540's attention kernel (used below sm_75), and the -Pascal BF16 path. The ready-made CUDA 13 engine has none of them, so its kernels and its output are unchanged. The -FP16 path for BF16 projections is in every build but runs by default only below sm_75; RTX 20 owners can try it with -`STRATA_BF16_TC=1` (`=0` turns it off on a V100). It is not bitwise the same as cuBLAS's BF16 kernel (FP16 tensor-core -sums round differently): #540 measured a mean KL of 8.4e-3 on the next-token distribution of 24 code prompts on a V100 -(the same top-1 in 23), the size of other summation-order changes; #655 a worst relative difference of 3.5e-5 per -product on an RTX 2080 Ti. +The FP16 path for BF16 projections is in every build but runs by default only below sm_75. RTX 20 owners can try it with +`STRATA_BF16_TC=1` (`=0` turns it off on a V100). It is not bitwise the same as cuBLAS's BF16 kernel, because FP16 tensor-core +sums round differently. -### Tesla P100 (sm_60): seven decode switches on by default (engine 0.1.45.3) +### Switches by card -On a Tesla P100 the engine now turns on a set of seven switches that change only how the work is scheduled, so the answers -stay bit for bit the same: `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_SIDE_MIXER=2`, `STRATA_FUSE_WAIT=2`, -`STRATA_B6_MMVQ_ROWS=1`, `STRATA_FUSE_DECODE=1` and `STRATA_GR_UP75=1`. `STRATA_SM60_PERF=0` turns the whole set off, and a -switch you set yourself, to any value, is kept. The set is used when every visible card has compute capability 6.0. -Measured by the maintainers on a Tesla P100 16 GB, IQ3_XXS, default flags, 8 interleaved pairs of fresh engines: decode 36.56 --> 41.76 tokens/s (+14.2%, faster in 8 of 8 pairs; a run against itself gave 0.975), with the same number of expert-cache -slots; the identity gate (10 prompts and 4K/20K prompt dumps) is identical with the set on. The full list, the RTX 30 set -and the per-switch results: [DETAILS.md](DETAILS.md#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-decode-switches-on-by-default). +| Variable | Card | Default | What it does | +| --- | --- | --- | --- | +| `STRATA_SM60_PERF=0` | P100 | set of 7 on | Turns off the seven scheduling switches below. | +| `STRATA_Q8_SM60=1` | GP100 | off | Q8_0 dense projections and head on a GP100 kernel. | +| `STRATA_IQ_STAGE_TABLES=1` | Pascal | off | IQ3_XXS / IQ3_S tables staged in shared memory (bitwise the same output). | +| `STRATA_DEQUANT_WIDE=0` | sm_61 | on | Back to the old dequantization. | +| `STRATA_SM61_TABLE=0` | sm_61 | on | Back to the CUDA expert layout and `native_mmvq`. | +| `STRATA_DENSE_F32=1` | Pascal | off | Large dense GEMMs as `cublasSgemm`. | +| `STRATA_BF16_TC=0\|1` | sm_7x | on below sm_75 | BF16 projections on the FP16 tensor cores. | +| `STRATA_PROMPT_ATTN_OLD=1` | Volta | off | The decode kernel for prompts instead of the Volta kernel. | +| `STRATA_ATTN_PRE75=0` | sm_70, gfx103x | on | Turns the #540 attention kernel off. `=1` turns it on for another wave32 AMD card. | -### Pascal GP100: a Q8_0 decode kernel (opt-in) +[NVIDIA_V100.md](NVIDIA_V100.md) has the V100 build, its switches and the parity test. -GP100 (P100, PH402) is the one Pascal chip without `__dp4a`, and the decode kernels are written for the cards that have -it: on a GP100 die the dense projections of a verify window read their weights at 40-80 GB/s of its ~730, decoding K- -and i-quant blocks (Q6_K, IQ4_XS, Q4_K, IQ3_S, ...) one int at a time. Two changes, both measured on a PH402 only: +## Tesla P100 (sm_60) -- **`STRATA_Q8_SM60=1`** packs every Q8_0 dense matrix and the head into the planes of `STRATA_Q8_PACKED` (any shape) - and runs them on `src/kernels/cuda/q8_sm60.cuh`: a warp (half or quarter warp for narrow rows) per row, 16-byte - weight loads, the window's q8_1 activations copied once per workgroup into shared memory as an int8 plane, the dot - on `STRATA_DP4A` (vmad), a shuffle sum - no barrier per row. Shapes it declines (a 6144-wide row with 7-8 columns) - keep the usual kernel. Not bitwise the exact kernels (the per-32 products are added in another order). The packed - copies cost their size in VRAM beside the GGUF layout the prompt path reads (Flash-Next: 2.9 GiB of dense - projections over the stages and 0.6 GiB for the head). -- The GSQ-RCO packs have no Q8_0 dense matrices, so the kernel needs a copy of the model whose dense projections and head are - Q8_0: `python tools/q8_dense_gguf.py -00001-of-00002.gguf ` writes one: every shard that holds - such matrices is converted (Swift 1.5 keeps them in both of its shards) and the rest are hard-linked beside them - (+3.5 GiB; the experts keep their offsets, so the native pack is reused). Point `--native` and - `--ple-gguf` at the new folder. Requantizing costs at most 0.4% of a tensor's |w|max (Q8_0 is finer than the - sources). -- **Opt-in `STRATA_IQ_STAGE_TABLES=1`**: IQ3_XXS's and IQ3_S's codebooks and sign-mask tables are staged in shared memory by - the grouped expert kernels (IQ3_XXS's ported from shinbunbun/llama-cpp-p100-patches 29 and 30, IQ3_S's the same for its - sign bytes; bitwise the same output). IQ3_S was the one gate/up format whose codebook those kernels still read from - global memory: the contributor measured a verify window's layer of IQ3_S experts (20 experts, 24 entries) 24% faster on a - GP100 die (517 -> 393 us with a Q2_0 down projection, 492 -> 369 us with IQ4_NL), and Swift 1.5 IQ3_XXS (13 of its 48 - layers are IQ3_S) 3.2% faster at 1M context (39.3 -> 40.5 tok/s); IQ3_XXS's tables alone were within noise (+0-4%). The - maintainers' interleaved A/Bs on a Tesla P100 (PCIe, IQ3_XXS pack and IQ3_S pack, 5-6 pairs each) showed no end-to-end - change (decode 0.97-1.03x, inside the noise of that box), so it is off by default (the contributor's version had it on - for compute capability 6.0). On an RTX 3090 Ti, forced on, the same IQ3_S layers took 3-13% longer (those kernels already - read at 490-665 GB/s there). -- Neither change is measured on compute capability 6.1 (P40, P4, GTX 10 series), which has `__dp4a`. Both build for - sm_61, and on a card with `__dp4a` the Q8_0 kernel's dot is the hardware instruction (`tools/q8_sm60_check.cu` - passes on an RTX 3090 Ti); numbers from a 6.1 card with `STRATA_Q8_SM60=1` or `STRATA_IQ_STAGE_TABLES=1` would - show whether either helps there. +On a Tesla P100 the engine turns on seven switches that change only how the work is scheduled, so answers stay bit for bit the +same: `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_SIDE_MIXER=2`, `STRATA_FUSE_WAIT=2`, `STRATA_B6_MMVQ_ROWS=1`, +`STRATA_FUSE_DECODE=1` and `STRATA_GR_UP75=1`. `STRATA_SM60_PERF=0` turns the whole set off, and a switch you set yourself, to +any value, is kept. The set is used when every visible card has compute capability 6.0. The full list and per-switch results are +in [DETAILS.md](DETAILS.md#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-decode-switches-on-by-default). -Measured on a PH402 SKU 200 (two boards, four GP100 dies of 48 SMs and 32 GB HBM2), application clocks locked at -1050 MHz, Windows 11, driver 581.80 (TCC), CUDA 12.9, engine 0.1.40.2; Flash-Next IQ3_XXS, 32K, `--layer-split 12,24,36 ---prefill 2048`; greedy, 256 tokens, medians of 2-3 interleaved starts, decode tok/s: +## Pascal GP100: a Q8_0 decode kernel (opt-in) + +GP100 (P100, PH402) is the one Pascal chip without `__dp4a`. Its dense projections read their weights at 40-80 GB/s of about +730, decoding K- and i-quant blocks one int at a time. Two opt-in changes, both measured on a PH402 only: + +- **`STRATA_Q8_SM60=1`** packs every Q8_0 dense matrix and the head into the planes of `STRATA_Q8_PACKED` and runs them on + `src/kernels/cuda/q8_sm60.cuh`: a warp per row, 16-byte weight loads, activations copied once per workgroup into shared memory, + the dot on `STRATA_DP4A`, no barrier per row. Shapes it declines keep the usual kernel. It is not bitwise the exact kernels. + The packed copies cost their size in VRAM (Flash-Next: 2.9 GiB of dense projections over the stages and 0.6 GiB for the head). +- The GSQ-RCO packs have no Q8_0 dense matrices, so the kernel needs a copy whose dense projections and head are Q8_0: + `python tools/q8_dense_gguf.py -00001-of-00002.gguf ` writes one (+3.5 GiB; experts keep their offsets, + so the native pack is reused). Point `--native` and `--ple-gguf` at the new folder. +- **`STRATA_IQ_STAGE_TABLES=1`** stages IQ3_XXS and IQ3_S codebooks in shared memory in the grouped expert kernels. Bitwise the + same output. Interleaved A/Bs on a Tesla P100 showed no end-to-end change, so it is off by default. On an RTX 3090 Ti, forced on, + the same layers took 3-13% longer. + +Neither change is measured on compute capability 6.1, which has `__dp4a`. Two practical notes for GP100 cards: the default +application clock is 759 MHz and a layer split leaves each die idle most of a window, so lock the clocks +(`nvidia-smi -i -ac 715,1050` as administrator). And a second engine pinned to the same logical CPUs halves this one's +decode; keep two engines on different logical CPUs. + +`tools/q8_sm60_check.cu` checks the kernel against a CPU reference on these shapes (1-8 columns). + +## Tesla P40 (sm_61): what was measured and changed + +Three things run differently on compute capability 6.1 by default and a fourth is opt-in. Each gives the same bits as what it +replaces. + +| Change | What it replaces | Switch | +| --- | --- | --- | +| Wide dequantization (a thread per 8 values, 16-bit loads, 16-byte stores) for Q4_K / Q5_K / Q6_K / IQ4_XS | a thread per 32 values with byte loads | on; `STRATA_DEQUANT_WIDE=0` | +| Expert mode 8 for the experts in VRAM (decode and verify windows) | the CUDA layout | on; `STRATA_SM61_TABLE=0` (or `STRATA_EXP_MODE=0`) | +| The interleaved small-batch matvec with a P40 table (`kIlRows_61`) | `native_mmvq` | on; `STRATA_SM61_TABLE=0` (or `STRATA_MMVQ_IL=0`) | +| The large dense GEMMs as `cublasSgemm` on FP32 copies of the FP16 operands | cuBLAS's FP16-operand kernels (4.3 TFLOPS instead of 7.6 for most chunk sizes) | **opt-in**, `STRATA_DENSE_F32=1` | + +The GEMM route is opt-in because its equality is a property of the cuBLAS build, not of the code. It was measured with cuBLAS +12.4.5.8 only. Run `gemm_f32_parity` (a build with `-DSTRATA_BUILD_TESTS=ON`; exit 0 = every size bitwise) with your toolkit's +library before you set it. + +`STRATA_GR_FAST=1` is slower on the P40 below 5 tokens, and `STRATA_PREFILL_CPU_SHARE=auto` and `STRATA_ADAPT_OVERLAP=1` gave +nothing there. + +Two things to know when measuring on such a card: + +- **Temperature.** A P40 has no fan of its own. At 89 C the driver reports `SW Thermal Slowdown: Active` and the clock falls from + 1,531 to about 1,300 MHz (`nvidia-smi -q -d PERFORMANCE`). Start every arm of an A/B at the same temperature, or the later arm loses. +- **Greedy answers are not identical run to run on a CPU without AVX-512**, even with `--prompt-cache 0 --adapt-swaps 0 + --pcie-frac 0`. Add `STRATA_IQ_MT_MIN=1` (and `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_ROUTE_TAIL_SKIP=0`) to compare two builds' + answers byte for byte. + +Profiling switches: `STRATA_PREFILL_TIMING=1`, `STRATA_DECODE_TIMING=1`, `STRATA_VERIFY_PROFILE=1`. + +## Quadro RTX 8000 and other Turing cards + +Turing (sm_75) has FP16 tensor cores but no native BF16 or TF32 tensor cores. Two existing, opt-in switches are worth comparing +on your own workload: + +- `STRATA_BF16_TC=1` converts BF16 projection operands to FP16 and uses FP16 tensor cores with FP32 accumulation. Default on sm_75 + is off; `=0` restores the cuBLAS BF16 path. Values outside FP16 range clamp to +-65504, and differently ordered sums can round + differently. +- `STRATA_SELECT_SIMT=1` uses the tiled FP32 QSA block scorer on the pre-sm_80 CUDA path instead of the warp scorer. `=0` keeps + the warp scorer. It does not enable TF32 and its summation order differs. + +Set the variables before starting the engine, or in the model config's `env` object, then restart. Merge these fields into the +existing config; this is not a complete run config: + +```json +{ + "env": { + "STRATA_BF16_TC": "1", + "STRATA_SELECT_SIMT": "1" + } +} +``` + +`STRATA_BF16_TC=1` improved prompt throughput by 20-34% on one Quadro RTX 8000 and changed replies in 3 of 9 paired requests, so +use it only if different rounding is acceptable. It did not improve decode. Numbers are in the history section. These switches are +existing code ([gemm.cu](../src/prefill/gemm.cu), [qsa_select.cu](../src/kernels/cuda/qsa_select.cu)), not new kernels. + +## AMD: building gfx906 and gfx1012 + +Setup does not build these. Build by hand and run `serve/server.py` with a config, as on any other card. + +- **gfx906** (MI50 / MI60 / Radeon VII, wave64): a separate opt-in build, `-DSTRATA_HIP_GFX906=ON + -DCMAKE_HIP_ARCHITECTURES=gfx906` (not `STRATA_ENABLE_HIP`). Current ROCm no longer ships gfx906 libraries; the reporter used a + community ROCm 7.14 image. Recipe, kernels and measurements: + [AMD_HIP.md](AMD_HIP.md#gfx906-instinct-mi50--mi60-radeon-vii-wave64-built-from-source). +- **gfx1012** (RX 5500 XT): the wave32 backend, `-DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1012`. HIP 5.7 (Ubuntu's + packages) works: older hipBLAS is used through rocBLAS, legacy HIP names and the missing `__syncwarp` are version-gated, and + RDNA1's missing signed dot4 uses llama.cpp's SDWA sequence (`-DSTRATA_GFX1012_PORTABLE_DOT=ON`: the portable one). + +## Reports welcome + +These paths stay experimental until more people run them. A report with the card, the driver or ROCm version, the model and the +engine log (`strata-.log`) in an issue helps. [COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md) has the format. + +## History / older measurements + +Dated numbers. Each row is one machine and one engine version, as labelled. They are not a promise for your card. + +**Community reports by card** + +| Card | Measured on | Result | +| --- | --- | --- | +| 2x Tesla P100 | IQ3_S, engine 0.1.39 ([report](../bench/results/2026-10-06-community-2x-p100/README.md), #1157) | prompt 526-532 tok/s; decode 28-32 tok/s at 128K prompt tokens; 3 of 3 needle checks at 122K | +| PH402 (4 GP100 dies) | IQ3_XXS, engine 0.1.40.2 | decode 28.3 -> 38.9 tok/s with `STRATA_Q8_SM60=1` and a Q8_0 dense shard | +| Tesla P40 | IQ3_S, engine 0.1.30 (#395) | prompt 217-374 tok/s; decode 30-33 tok/s | +| Tesla P40 | IQ3_S, engine 0.1.43.1 with the sm_61 changes | prompt 255 tok/s at 565 tokens (was 205); decode 37-38 tok/s (was 34-35); with `STRATA_DENSE_F32=1` 464-474 tok/s at 7K tokens (was 360-432) | +| V100-PCIE-32GB | UD-IQ4_XS (#600) | prompt 1,123-1,251 tok/s | +| V100 32GB | IQ2_XS (#540) | prompt +22% | +| RTX 2080 Ti | Q2_0 (#655) | prompt +15-18% with `STRATA_BF16_TC=1` | +| RX 6900 XT | IQ3_S ([report](../bench/results/2026-10-04-rdna2-pre75-attention/README.md)) | prompts +4-6% alone, +7-12% with #835 | +| RX 5500 XT, 8 GB | IQ3_S, 8K prompt (#442) | decode 15.3 tok/s | +| RX 5700 XT, 8 GB | Coder IQ1_M, 32K context | prompt 115 / 144 / 149 tok/s and decode 18.0 / 20.3 / 23.3 tok/s at 4K / 16K / 30K; 6 of 6 needle checks; 607 of 12,288 experts in VRAM | +| 2x MI50 | Coder IQ1_M, 128K context (#677) | decode 50.1 / 47.8 / 45.7 tok/s at 4K / 32K / 128K; prompt about 520 tok/s | + +**Tesla P100 default switch set.** Measured on: Tesla P100 16 GB, IQ3_XXS, default flags, engine 0.1.45.3, 8 interleaved pairs of +fresh engines: decode 36.56 -> 41.76 tok/s (+14.2%, faster in 8 of 8 pairs), same number of expert-cache slots. The identity gate +(10 prompts and 4K / 20K prompt dumps) is identical with the set on. + +**GP100 Q8_0 kernel.** Measured on: PH402 SKU 200 (two boards, four GP100 dies, 48 SMs and 32 GB HBM2 each), application clocks +locked at 1050 MHz, Windows 11, driver 581.80 (TCC), CUDA 12.9, engine 0.1.40.2, Flash-Next IQ3_XXS, 32K, `--layer-split +12,24,36 --prefill 2048`, greedy, 256 tokens, medians of 2-3 interleaved starts, decode tok/s. | | ~10.7K-token document | the same, cached | short reasoning | code | mean | | --- | ---: | ---: | ---: | ---: | ---: | @@ -151,140 +256,35 @@ Measured on a PH402 SKU 200 (two boards, four GP100 dies of 48 SMs and 32 GB HBM | Q8_0 dense shard, `STRATA_Q8_PACKED=1` | 27.5 | 31.0 | 34.4 | 31.8 | 31.1 | | Q8_0 dense shard, `STRATA_Q8_SM60=1` | 37.5 | 38.4 | 41.2 | 38.5 | 38.9 | -GPU time per verify window (`STRATA_VERIFY_PROFILE`): dense projections 19.2 -> 11.2 ms, head 4.25 -> 2.1 ms; the -draft policy then picks longer windows (2.10 -> 2.31 tokens per window). Prompt reads are unchanged (437-438 tok/s at -10.7K). `tools/q8_sm60_check.cu` checks the kernel against a CPU reference on these shapes (1-8 columns) and times it. +GPU time per verify window: dense projections 19.2 -> 11.2 ms, head 4.25 -> 2.1 ms. Prompt reads are unchanged (437-438 tok/s at +10.7K). Locking clocks at 1050 MHz took one case from 25.0 to 29.1 tok/s at 10.7K context. -Two things that mattered as much on this card: its default application clock is 759 MHz, and a layer split leaves -each die idle most of a window, so the dies decode below 1050 MHz unless the clocks are locked -(`nvidia-smi -i -ac 715,1050` as administrator; 25.0 -> 29.1 tok/s at 10.7K context). And a second engine on -the same PC that pins its threads to the same logical CPUs halves this one's decode while it runs (the host thread -is time-sliced); keeping the two on different logical CPUs (SMT siblings) avoided it. +**Tesla P40 sm_61 kernels.** Measured on: one Tesla P40 24 GB in a 20-vCPU VM (Xeon E5-2695 v4, 98 GB RAM), CUDA 12.4 (cuBLAS +12.4.5.8), IQ3_S with MTP (`--spec 4`), int8 KV, `--max-context 262144`, engine 0.1.43.1 ([report](../bench/results/2026-10-12-p40-sm61-kernels/README.md)). -A/B switches: `STRATA_DEQUANT_WIDE=0`, `STRATA_SM61_TABLE=0` (sm_61: the kernels 0.1.43.1 ran), `STRATA_DENSE_F32=1` (Pascal, opt-in), -`STRATA_Q8_SM60=0|1` (Pascal GP100, opt-in), `STRATA_BF16_TC=0|1`, `STRATA_PROMPT_ATTN_OLD=1` (the decode kernel for prompts), `STRATA_ATTN_PRE75=0` -(#540's kernel off; on gfx103x with HIP that kernel is the default, see the AMD table above, and `=1` turns it on for -another wave32 AMD card). [NVIDIA_V100.md](NVIDIA_V100.md) has the V100 build, its measurements and the parity test. +| Change | Result on the P40 | +| --- | --- | +| Wide dequantization | Q6_K 10240 x 2560: 7.2 -> 0.33 ms (10 -> 227 GB/s); Q4_K 5.2x, Q5_K 3.8x, IQ4_XS 3.6x | +| Expert mode 8 | an 18-expert call at 2 tokens: IQ3_S 390 -> 255 us, IQ3_XXS 345 -> 252, IQ2_S 290 -> 238, IQ4_XS 328 -> 261 | +| Interleaved matvec table | IQ4_XS 2560 x 10240 at 2 columns 112 -> 65 us; Q6_K 12288 x 2560 at 4 columns 189 -> 127 us | +| `STRATA_DENSE_F32=1` | 8.4-9.8 TFLOPS at every size from 512 tokens; qkv at 7,041 tokens 85 -> 45 ms | -### Tesla P40 (sm_61): what was measured and changed +Through the server, medians of 3 requests, two runs per build alternated, card at 65 C. By default: a 565-token prompt 203 / 205 +-> 256 / 255 tok/s, decode 35.0 / 34.2 -> 38.2 / 37.2 tok/s. With `STRATA_DENSE_F32=1`: a 7,047-token prompt 430 / 364 -> 474 / 473 +tok/s. Of a 7K-token prompt, 36% is the dense projections' GEMMs, 28% the experts, 13% the hyper-connection read and 12% +attention. -One Tesla P40 24 GB in a 20-vCPU VM (Xeon E5-2695 v4, 98 GB RAM), CUDA 12.4 (cuBLAS 12.4.5.8), IQ3_S with MTP -(`--spec 4`), int8 KV, `--max-context 262144`, engine 0.1.43.1; the numbers and every run are in -[bench/results/2026-10-12-p40-sm61-kernels](../bench/results/2026-10-12-p40-sm61-kernels/README.md). -`STRATA_PREFILL_TIMING=1`, `STRATA_DECODE_TIMING=1` and `STRATA_VERIFY_PROFILE=1` show a card that is the bottleneck -everywhere: the prompt's GPU timeline equals its wall time (waiting for expert copies 0.2%, host grouping 0.2% of a -7K-token prompt) and a decode window is 49 ms of GPU stages with 1-3 ms of CPU experts. Of a 7K-token prompt, 36% is -the dense projections' GEMMs, 28% the experts (ggml's MMQ, which has its own Pascal dp4a configuration), 13% the -hyper-connection read and 12% the attention. +**Quadro RTX 8000 (sm_75).** Measured on: one 48 GiB Quadro RTX 8000, 260 W limit, Xeon W-2295, CUDA 12.4, GCC 13, 2026-10-07, +source build based on v0.1.40.3 with other local changes; IQ3_S with MTP (`--spec 4`), `--max-context 262144`, INT8 KV with +`--kv-resident 32768`, `--pcie-frac 0.20`, `--expert-cache 19000`, `--prefill auto`, prompt cache and adaptive swaps off. Three +seeded synthetic prompts per size, 128 output tokens, median engine-reported rates. -Three things run differently on compute capability 6.1 by default and a fourth is opt-in. Each gives the same bits as -what it replaces: - -| Change | What it replaces | Measured on the P40 | Switch | -| --- | --- | --- | --- | -| Wide dequantization (a thread per 8 values, 16-bit loads, 16-byte stores) for Q4_K / Q5_K / Q6_K / IQ4_XS | a thread per 32 values with byte loads: 0.5 s of every prompt chunk, whatever its length | Q6_K 10240 x 2560: 7.2 -> 0.33 ms (10 -> 227 GB/s); Q4_K 5.2x, Q5_K 3.8x, IQ4_XS 3.6x; `dequant_wide_parity` 360 cases bitwise | on; `STRATA_DEQUANT_WIDE=0` | -| Expert mode 8 for the experts in VRAM (decode and verify windows) | the CUDA layout | an 18-expert call at 2 tokens: IQ3_S 390 -> 255 us, IQ3_XXS 345 -> 252, IQ2_S 290 -> 238, IQ4_XS 328 -> 261; `native_expert_bench` bitwise | on; `STRATA_SM61_TABLE=0` (or `STRATA_EXP_MODE=0`) | -| The interleaved small-batch matvec with a table measured on the P40 (`kIlRows_61`) | `native_mmvq` | IQ4_XS 2560 x 10240 at 2 columns 112 -> 65 us, Q6_K 12288 x 2560 at 4 columns 189 -> 127 us, the Q5_K head at 3 columns 2,545 -> 1,766 us; `mmvq_il_parity` bitwise | on; `STRATA_SM61_TABLE=0` (or `STRATA_MMVQ_IL=0`) | -| The large dense GEMMs (qkv, query, gate, output projections) as `cublasSgemm` on FP32 copies of the FP16 operands | cuBLAS's FP16-operand kernels, which run at 4.3 TFLOPS instead of 7.6 for most chunk sizes (a 7K-token prompt reads in 16.3 or 19.5 s by its exact token count) | 8.4-9.8 TFLOPS at every size from 512 tokens; qkv at 7,041 tokens 85 -> 45 ms; `gemm_f32_parity` 5,201 chunk sizes bitwise | **opt-in**, `STRATA_DENSE_F32=1` | - -The GEMM route is opt-in because its equality is a property of the cuBLAS build, not of the code: it was measured with -cuBLAS 12.4.5.8 only. Run `gemm_f32_parity` (a build with `-DSTRATA_BUILD_TESTS=ON`; exit 0 = every size bitwise) with -your toolkit's library before you set it. - -Through the server, medians of 3 requests per run, two runs per build alternated, each run started with the card at -65 C (see below). By default: a 565-token prompt 203 / 205 -> 256 / 255 tok/s, decode after it 35.0 / 34.2 -> 38.2 / -37.2 tok/s, a 7,047-token prompt 1-3% faster at the same token count. With `STRATA_DENSE_F32=1` as well: the -7,047-token prompt 430 / 364 -> 474 / 473 tok/s (the old build reads it at ~361 or ~431 tok/s depending on whether the -chunk has more than 7,040 tokens; with the route every request read at 464-474), decode after it 32.1 / 33.4 -> 35.9 / -34.0. With `STRATA_DEQUANT_WIDE=0 STRATA_SM61_TABLE=0` and the route off the new build measures as the old one (210 / -365 tok/s, 33.3 tok/s). - -Measured and **not** changed: the latency-hidden hyper-connection read (`STRATA_GR_FAST=1`) is slower on the P40 below -5 tokens (82 -> 88 us at one); `STRATA_PREFILL_CPU_SHARE=auto` and `STRATA_ADAPT_OVERLAP=1` gave nothing. - -Two things to know when measuring on such a card: - -- **Temperature.** A P40 has no fan of its own. This one reaches 87-89 C in a few minutes of work; at 89 C the driver - reports `SW Thermal Slowdown: Active` and the clock falls from 1,531 to ~1,300 MHz (`nvidia-smi -q -d PERFORMANCE`, - `nvidia-smi --query-gpu=temperature.gpu,clocks.gr --format=csv -l 2`). Start every arm of an A/B at the same - temperature, or the later arm loses. -- **Greedy answers are not identical run to run on a CPU without AVX-512**, even with `--prompt-cache 0 - --adapt-swaps 0 --pcie-frac 0`: an expert computed on the CPU rounds differently for a lone token than in a window - of several (`native_gu_mt_min`), and how many drafts a window holds depends on timing. Add `STRATA_IQ_MT_MIN=1` - (and `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_ROUTE_TAIL_SKIP=0`) to compare two builds' answers byte for byte. - -### Quadro RTX 8000: measurements of existing switches - -The Quadro RTX 8000 is Turing (sm_75): it has FP16 tensor cores, but no native BF16 or TF32 tensor cores. Two -**existing, opt-in** CUDA switches are worth comparing on your own workload: - -- `STRATA_BF16_TC=1` converts BF16 projection operands to FP16 and uses FP16 tensor cores with FP32 accumulation. - The default on sm_75 is off; `=0` restores the cuBLAS BF16 path. The switch applies to 7.x cards, not the native - BF16 path on sm_80 and newer. Conversion clamps finite values outside FP16's range to ±65504; small values and - differently ordered sums can round differently. -- `STRATA_SELECT_SIMT=1` uses the tiled FP32 QSA block scorer on the pre-sm_80 CUDA path instead of the default - warp scorer. `=0` keeps the warp scorer. It does not enable TF32 on Turing, and its summation order differs. - -The existing paths are in [gemm.cu](../src/prefill/gemm.cu) and [qsa_select.cu](../src/kernels/cuda/qsa_select.cu). - -Set the environment before starting the engine, or set the switches in the model config's `env` object, then -restart. For example, this config fragment opts into both (keep the config's other settings): - -```json -"env": { - "STRATA_BF16_TC": "1", - "STRATA_SELECT_SIMT": "1" -} -``` - -This section records an investigation, **not a new kernel or a change to defaults**. Measurements on 2026-10-07 -used one 48 GiB Quadro RTX 8000 (physical GPU 3), a 260 W power limit, a Xeon W-2295, CUDA 12.4 and GCC 13. -The source-built engine was based on v0.1.40.3 with other local changes integrated; unrelated opt-in changes -were disabled in these arms. These are not measurements of an otherwise clean documentation-only checkout, -not results at the launcher's 200 W setting, and not a speed promise for every sm_75 card. - -The model was IQ3_S with MTP (`--spec 4`), `--max-context 262144`, INT8 KV with `--kv-resident 32768`, -`--pcie-frac 0.20`, a fixed `--expert-cache 19000` request and `--prefill auto` (8,192-token ceiling). -The engine reported 21,189 cache slots including the prompt loan. Prompt caching and adaptive swaps were off -(`--prompt-cache 0 --adapt-swaps 0`). Each size used three fresh, seeded synthetic prompts and 128 output tokens -per request; the table gives median engine-reported rates, not end-to-end server latency. - -| Existing switch settings | 537 tokens: prompt / decode tok/s | 4,057 tokens: prompt / decode tok/s | 32,057 tokens: prompt / decode tok/s | +| Switches | 537 tokens: prompt / decode tok/s | 4,057 tokens | 32,057 tokens | | --- | --- | --- | --- | | BF16_TC=0, SELECT_SIMT=0 | 493 / 86.8 | 1,055 / 79.9 | 1,135 / 70.6 | | BF16_TC=1, SELECT_SIMT=0 | 661 / 83.7 | 1,262 / 75.4 | 1,416 / 68.4 | | BF16_TC=1, SELECT_SIMT=1 | 679 / 87.1 | 1,290 / 78.2 | 1,501 / 71.6 | -BF16_TC improved prompt throughput by about 20–34%, but did not improve decode in these comparisons. Adding -SELECT_SIMT improved the 32K prompt rate by about 6% over BF16_TC alone; this is not a universal decode win. -BF16_TC changed replies in 3 of the 9 paired requests, so use it only if different rounding is acceptable. -All five basic arithmetic/JSON checks passed in each arm, which is a smoke check, **not a model-quality study**. -A separate synthetic QSA scorer check at 131,072 context tokens matched selected IDs for all 256 queries and -passed its FP64 accuracy gate. That check does not prove identical selections on arbitrary prompts or model quality. - -Two other existing tuning options did not justify a recommendation on this machine: - -- `--prefill auto:16384`, with BF16_TC on and SELECT_SIMT off, increased the prompt's cache loan from 4.38 to - 7.38 GiB for only about 0.6% more 32K prompt throughput (1,425 versus 1,416 tok/s). Just 24 MiB of VRAM remained - at the last graph capture. Keep the 8,192 ceiling rather than spend that memory margin for this small gain. -- `STRATA_PREFILL_CPU_SHARE=auto` gave no gain at the two tested sizes (537 and 4,057 tokens): prompt rates were - 647 and 1,253 tok/s versus 661 and 1,262 with BF16_TC alone; decode was also lower. Larger prompts were not - tested for this setting. - -## AMD: building gfx906 and gfx1012 - -Setup does not build these; build by hand and run `serve/server.py` with a config, as on any other card. - -- **gfx906** (MI50 / MI60 / Radeon VII, wave64): a separate opt-in build, `-DSTRATA_HIP_GFX906=ON - -DCMAKE_HIP_ARCHITECTURES=gfx906` (not `STRATA_ENABLE_HIP`). Current ROCm no longer ships gfx906 libraries; the - reporter used a community ROCm 7.14 image. Recipe, kernels and measurements: [AMD_HIP.md](AMD_HIP.md#gfx906-instinct-mi50--mi60-radeon-vii-wave64-built-from-source). -- **gfx1012** (RX 5500 XT): the wave32 backend, `-DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1012`. HIP 5.7 - (Ubuntu's packages) works: older hipBLAS (0.x) is used through rocBLAS, the legacy HIP names and the missing - `__syncwarp` are version-gated, and RDNA1's missing signed dot4 uses llama.cpp's SDWA sequence - (`-DSTRATA_GFX1012_PORTABLE_DOT=ON`: the portable one). - -## Reports welcome - -These paths stay experimental until more people run them. A report with the card, the driver / ROCm version, the -model and the engine log (`strata-.log`) in an issue helps; [COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md) -has the format for measurements. +`SELECT_SIMT` added about 6% to the 32K prompt rate over BF16_TC alone. `--prefill auto:16384` gained about 0.6% at 32K for 3 GiB +more prompt-cache loan, and `STRATA_PREFILL_CPU_SHARE=auto` gave no gain, so neither is recommended. All five basic arithmetic and +JSON checks passed in each arm; that is a smoke check, not a quality study. diff --git a/docs/ORCA.md b/docs/ORCA.md index c8ea68a54..70e30abe2 100644 --- a/docs/ORCA.md +++ b/docs/ORCA.md @@ -1,25 +1,30 @@ # OrcaRouter IQ3_XXS compatibility -Validated on Linux with an RTX 5090 (32 GB), Ryzen 9 9950X3D and 128 GB RAM on 2026-09-27. +This page is for people who want to run `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF` (**IQ3_XXS**, two shards, +85.20 GB) with Strata. It is a manual conversion workflow from a [source checkout](SOURCE_INSTALL.md). OrcaRouter is +not an installer menu entry, and there is no `--family orca` command. -The target is `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, **IQ3_XXS** (two shards, 85.20 GB total). -It has Strata's Qwen4Exp geometry, but ordinary quantization also compresses the hyperconnection, -small attention and PLE projections which Strata's kernels read as BF16. +## Quick start -`tools/iq_pack.py --compat-bf16` expands only those projections, rounding to BF16 using nearest-even. -This adds rounding relative to the GGUF's dequantized weights; it does not recover the original BF16 -checkpoint or its quality. Expert weights, native attention weights, embeddings and the 28.8 GB PLE -lookup table remain unchanged. The latter stays on disk. The pack records its conversions in -`compat-bf16.json`. +1. Build Strata normally, then pack the model ([Preparation](#preparation)). +2. Prepare the optional draft head (MTP) with the same section's commands. +3. Save the config from [Local server](#local-server) and start the server. -The native loader retains this pack's converted PLE key. Its existing native Q2_0 key path remains -available to the original model. Hugging Face snapshot symlinks are supported; all split shards must -finish downloading before packing. Pass the snapshot filename, not its hash-named blob target. +## How it works + +The model has Strata's Qwen4Exp geometry. Ordinary quantization also compresses the hyperconnection, small attention +and PLE projections, which Strata's kernels read as BF16. `tools/iq_pack.py --compat-bf16` expands only those +projections, rounding to BF16 with nearest-even. This adds rounding relative to the GGUF's dequantized weights. It does +not recover the original BF16 checkpoint or its quality. + +Expert weights, native attention weights, embeddings and the 28.8 GB PLE lookup table stay unchanged, and the lookup +table stays on disk. The pack records its conversions in `compat-bf16.json`. Hugging Face snapshot symlinks work. All +split shards must finish downloading before packing. Pass the snapshot filename, not its hash-named blob target. ## Preparation -Build Strata normally using the pinned llama.cpp dependency. Python needs numpy, regex and gguf-py -from that dependency (`STRATA_GGUF_PY` can point to its `gguf-py` directory). +Python needs numpy, regex and gguf-py from the pinned llama.cpp dependency. `STRATA_GGUF_PY` can point to its +`gguf-py` directory. ```sh .venv/bin/python tools/iq_pack.py \ @@ -27,11 +32,10 @@ from that dependency (`STRATA_GGUF_PY` can point to its `gguf-py` directory). --out packs/orca-iq3_xxs --compat-bf16 ``` -Use this model's own tokenizer exported into the pack. Do not share another model's `dense.bin`, -and do not rename the Orca files to impersonate one of setup's GSQ-RCO models. +Use this model's own tokenizer, exported into the pack. Do not share another model's `dense.bin`, and do not rename +the Orca files to pass as one of setup's GSQ-RCO models. -Strata's persistent server also uses the MTP runtime (optional: without `--mtp` it drafts by lookup only), prepared by -the existing tools: +The persistent server can use the MTP runtime. It is optional: without `--mtp` the server drafts by lookup only. ```sh .venv/bin/python tools/mtp_fetch.py fetch --out mtp @@ -40,15 +44,15 @@ the existing tools: cp data/draft_vocab.bin mtp/rt/draft_vocab.bin ``` -This uses the original model's draft head, as setup does for Swift. The target model verifies its -proposals; draft acceptance and performance must be measured for this fine-tune. +This uses the original model's draft head, as setup does for Swift. The target model verifies every proposal. Measure +draft acceptance and speed for this fine-tune; the original model's numbers do not carry over. ## Local server -Save the following as `strata-orca-iq3_xxs.json` at the repository root, replacing both `/path/to/` -entries with the **same first shard**. Orca's PLE table is in shard 1. Start with a 32K context and -512-token prefill chunks; the expert cache sizes itself to available VRAM. The expert arena alone -needs about 49.8 GiB of available system RAM, plus draft/runtime buffers and other applications. +Save the following as `strata-orca-iq3_xxs.json` at the repository root. Replace both `/path/to/` entries with the +**same first shard**, because Orca's PLE table is in shard 1. The config uses a 32K context and 512-token prefill +chunks. The expert cache sizes itself to the available VRAM. The expert arena alone needs about 49.8 GiB of free system +RAM, plus draft and runtime buffers and your other applications. ```json { @@ -70,27 +74,32 @@ needs about 49.8 GiB of available system RAM, plus draft/runtime buffers and oth } ``` -Run from the repository root: +Run it from the repository root: ```sh .venv/bin/python -m serve.server --engine strata --config strata-orca-iq3_xxs.json --port 8080 ``` -The web interface is at `http://127.0.0.1:8080`; API clients use `http://127.0.0.1:8080/v1`. -Vision is not configured in this text-only example. Performance of the original GSQ-RCO model -does not establish this fine-tune's speed or accuracy. +The web interface is at `http://127.0.0.1:8080`. API clients use `http://127.0.0.1:8080/v1`. This example is text-only; +vision is not configured. -## Scope and validation +## Scope -- IQ3_XXS is the target of this change; other Orca quantizations are not validated. -- IQ3_M additionally uses Q5_0 expert down matrices, which the native GPU expert path does not support. -- The installer model menu is unchanged. This is an explicit local packing workflow. -- Focused conversion and split-file tests: `.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`. -- All eight packing tests, 17 server tests and the GPU gated-residual parity check passed. -- The complete model packed successfully: 460 tensors converted, 1.39 GiB of converted BF16 weights; - 1.43 GiB total packed dense weights. The 48 expert layers remain IQ3_XXS gate/up and IQ4_NL down. -- Real server checks passed: arithmetic, Python code, translation, a longer explanation, multi-turn context - reuse (25 prompt tokens reused), streaming cancellation and a subsequent correct response. The engine - PID stayed unchanged. These are smoke tests, not a quality benchmark or a full-context stress test. -- With the configuration above, a 111-token explanation generated at 77.7 tokens/s (engine decode timing); - total request time was 1.77 seconds. This single short measurement is not a general throughput claim. +- IQ3_XXS is the only validated Orca quantization. +- IQ3_M uses Q5_0 expert down matrices, which the native GPU expert path does not support. +- The installer model menu does not list this model. +- Packing tests: `.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`. + +## History / older measurements + +> **Measured on:** Linux, RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB RAM, 2026-09-27, the configuration above. + +| Check | Result | +| --- | --- | +| Packing | 460 tensors converted, 1.39 GiB of converted BF16 weights, 1.43 GiB total packed dense weights; the 48 expert layers stay IQ3_XXS gate/up and IQ4_NL down | +| Tests | all eight packing tests, 17 server tests and the GPU gated-residual parity check passed | +| Server smoke tests | arithmetic, Python code, translation, a longer explanation, multi-turn reuse (25 prompt tokens reused), streaming cancellation and a correct answer afterwards; the engine PID stayed unchanged | +| Decode speed | 111-token explanation at 77.7 tokens/s (engine decode timing), 1.77 s total request time | + +These are smoke tests and one short measurement, not a quality benchmark, a full-context stress test or a general +throughput claim. diff --git a/docs/QWEN36.md b/docs/QWEN36.md index a2455b827..96bf4486e 100644 --- a/docs/QWEN36.md +++ b/docs/QWEN36.md @@ -1,23 +1,51 @@ # Qwen3.6-35B-A3B: pictures, several requests at once and speed -Strata runs [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (35B mixture of experts, 3B active, 256 experts, top-8) from -[Unsloth's GGUFs with the MTP layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) (UD-Q4_K_XL, UD-Q6_K_XL, Q8_0) on its own -engine path (`src/qwen35`; `strata` switches to it from the file's architecture). Pick it in the app's model list, in `strata launch` or in setup (`--family qwen36`). Besides text it -reads **pictures** and serves **several requests at once**, the same two options the Flash-Next models have. +This page is for people who run [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (35B mixture of +experts, 3B active, 256 experts, top-8) with Strata and want to know how pictures, parallel requests and the prompt +chunk behave. For sizes and memory, see [MODELS.md](MODELS.md). -Measured on an AMD Radeon AI PRO R9700 32 GB (HIP, Linux, UD-Q4_K_XL, every expert in VRAM) and on an RTX 3060 12 GB (CUDA, 20 threads, -UD-Q4_K_XL in the hybrid mode: 22-37 of 256 experts per layer in VRAM, the rest on the CPU), llama.cpp 972b5fc on the same files. +Strata runs [Unsloth's GGUFs with the MTP layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) +(UD-Q4_K_XL, UD-Q6_K_XL, Q8_0) on its own engine path (`src/qwen35`; `strata` switches to it from the file's +architecture). It runs on NVIDIA and AMD cards. The Intel Arc and Mac engines do not run it. + +## Quick start + +Pick it in the app's model list, in `strata launch` or in setup: + +```sh +strata launch --family qwen36 --model ud-q4_k_xl +strata launch --family qwen36 --model ud-q4_k_xl --vision on --parallel 2 +``` + +| Option | Effect | +| --- | --- | +| `--vision on\|off\|gpu\|cpu` | Terminal: pictures and which reader. Source setup uses `--vision yes\|no\|gpu\|cpu` and `--vision-tokens N`. | +| `--parallel N` | How many requests are answered at once, 1 to 8. Source setup has the same option. | +| `--prefill 512` | Engine flag in the config's `args`, not a `strata launch` option: the prompt chunk size ([Prompt chunk](#prompt-chunk)). | ## Pictures -The vision encoder is Unsloth's `mmproj-F16.gguf` (0.9 GB, pinned revision, SHA-256 checked by setup): a 27-layer ViT with a 2x2 merger, -the same projector family as the Flash-Next models' (`qwen3vl_merger`; only its output width differs: 2048 here, 2560 there). Strata runs -it with the same helper (`strata-vision`, llama.cpp's `mtmd`), so the encoder choices are the Flash-Next ones -([Images](DETAILS.md#images-vision)). Pictures are on by default, the same rule as for every model on NVIDIA and AMD cards: -the CPU reader under 24 GB of graphics memory (a few seconds per picture, no VRAM), the card's reader from 24 GB (GPU encoder, -~1.2 GB of VRAM at up to 1,024 image tokens; on AMD it is `strata-vision-hip`, with the CPU reader as its fallback). -`--vision yes|no|gpu|cpu` and `--vision-tokens N` work as there; what you ask for is kept, with a note. The Mac engine does -not run this model. +The vision encoder is Unsloth's `mmproj-F16.gguf` (0.9 GB, pinned revision, SHA-256 checked by setup): a 27-layer ViT +with a 2x2 merger, the same projector family as the Flash-Next models' (`qwen3vl_merger`; only its output width +differs: 2048 here, 2560 there). Strata runs it with the same helper (`strata-vision`, llama.cpp's `mtmd`), so the +encoder choices are the Flash-Next ones ([Images](DETAILS.md#images-vision)). + +Pictures are on by default, by the same rule as for every model on NVIDIA and AMD cards: + +- The CPU reader runs under 24 GB of graphics memory. It takes a few seconds per picture and no VRAM. +- The card's reader runs from 24 GB (GPU encoder, about 1.2 GB of VRAM at up to 1,024 image tokens). On AMD it is + `strata-vision-hip`, with the CPU reader as its fallback. + +Your explicit choice is kept, with a note. A picture becomes up to 1,024 tokens of the context (the CPU default is 300). +The engine puts the encoder's rows where the prompt has `<|image_pad|>` tokens and gives every row its 2-D position, +interleaved M-RoPE as llama.cpp does for this model (rotary sections 11 / 11 / 10). + +Pictures work with several requests at once, with the MTP drafts and with the conversation cache. The same conversation +sent again continues from the held state when its pictures are the same (matched by the server's key for each +picture). A different picture starts that conversation again. + +> **Measured on:** RTX 3060 12 GB (CUDA) and an R9700 box (32 threads) and a 3060 box (10 threads), UD-Q4_K_XL, Strata +> 0.1.45. | Encoder | Time per picture (640x400 = 260 tokens / 800x450 = 350 tokens) | VRAM | | --- | --- | --- | @@ -25,179 +53,200 @@ not run this model. | CPU, 32 threads (R9700 box) | 0.9 s / 1.1 s | none | | CPU, 10 threads (3060 box) | 1.6 s / 2.1 s | none | -A picture becomes up to 1,024 tokens of the context (the CPU default is 300). The engine puts the encoder's rows where the prompt has -`<|image_pad|>` tokens and gives every row its 2-D position, interleaved M-RoPE exactly as llama.cpp does for this model (rotary sections -11 / 11 / 10, a picture's rows at (t, h, w) = (p, p + row, p + column), the text after it continuing at p + max(columns, rows)). +## Several requests at once -**Checked against llama.cpp** (its `mtmd` and its own graph, the encoder on the CPU on both sides, the same prompt ids, the answer -llama.cpp gives fed to both so the logits are compared step by step, 24 steps; the "floor" is llama.cpp's CPU backend against its own -HIP run, the noise two correct engines have on this model): +`"parallel": N` in the model's config (`strata-qwen36-.json`) or `--parallel N` makes the engine keep N +conversations open (`--batch N`, up to 8) and decode them together. Every window carries one token of each +conversation, so the dense weights, the head and every routed expert that conversations share are read once for all of +them. The server speaks the same engine protocol as for Flash-Next ([BATCHING.md](BATCHING.md)). -| Picture and question | Prompt tokens | First token | Mean KL, llama.cpp HIP vs Strata | Floor (llama.cpp CPU vs HIP) | +A new request waits only for the read of its own prompt, which runs between the decode windows of the others. A +conversation continues in the slot that holds its start, or the engine copies the longer held start from another slot. +A request that runs alone (the solo path, with its MTP drafts) moves into a batch slot when another arrives, and back +when it is alone again, and gives the same text either way. +`STRATA_Q36_READ_WINDOWS=4` lets the decoding slots produce more tokens during a long prompt read, at the cost of a +slower read. The default is one decode window per chunk. + +**MTP drafts.** A request that has the engine to itself keeps its MTP drafts. With several requests the batch window has +no drafts unless `--batch-mtp` is in the config's `args` (opt-in). Each slot's window then also carries the proposal its +MTP block made, one per slot, up to the rows to spare in the 8-row window. + +**What a slot costs.** Each slot keeps its own cache in VRAM: 1.5 GB at a context of 32K (fp32 keys and values of the 10 +attention layers and the MTP block, 45 KB per token) and 0.07 GB of recurrent state. Setup writes the context for one +conversation, so N slots take N times that from the room the experts would have. On a card that does not hold every +expert (12 and 16 GB, the hybrid mode) the expert slots lose that room, and a batch reads more distinct experts from RAM. + +Setup offers `--parallel` where the sessions do not eat the experts' room, says the price in the same breath and asks +"How many requests at once?" Enter keeps one at a time, and any number you give is kept. + +| Card type | Slots setup offers | +| --- | --- | +| Every expert still fits | Up to 3 to 4 (3 on a 32 GB card at 32K) | +| Hybrid mode | The count whose sessions take at most 35% of the room the experts would have. 12 GB: 4 slots at 8K, none at 32K. 16 GB: 2 slots at 32K. | + +The engine says what the sessions take before it allocates them and refuses, with the numbers, a count that cannot fit. + +In the hybrid mode the text of a request can differ a little between running alone and running beside others. The +experts that sit on the GPU or on the CPU change with the routing of all the requests, and the two compute with +different rounding. With the slot tables fixed (`STRATA_Q36_FIXED_SLOTS=1`, a test setting) every slot equals its solo +run there too. + +## Prompt chunk + +The MoE prompt kernels read every expert once per chunk, so a bigger chunk feeds them more rows per expert. With every +expert in VRAM the engine takes the largest of 4096 / 2048 / 1024 that fits beside the experts, the 0.77 GiB reserve +and the sessions' caches. + +| Situation | Chunk | +| --- | --- | +| Every expert in VRAM, 8K to 64K context | 4096 | +| Every expert in VRAM, 100K context | 2048 | +| Every expert in VRAM, 131K context | 512 | +| `--batch 2..4` | 1024 at most, so the other slots are not held for a whole chunk | +| `--batch 8` | 512 | +| Hybrid mode | 2048; 1024 or 512 on a card with 7.5 or 5.5 GiB free | + +In the hybrid mode a bigger chunk takes expert slots (an RTX 3060 at chunk 4096 read prompts 5% faster and decoded 13% +slower). `--prefill 512` (or `STRATA_Q36_CHUNK=512`) gives the 512 size, and then the text is the base engine's, bit +for bit. The chunk size changes the GEMM tiles the BLAS picks and with them the last bits of the prompt state. This is +the size of the llama.cpp CPU-against-HIP difference on the same file, not a quality loss. A prompt shorter than the +chunk (every chat turn after the first) is one chunk either way. + +## Speed and exactness + +Every change on the decode path gives the old kernels' bits. `STRATA_Q36_VERIFY=1 strata ... --no-graph` runs the old +kernel beside each new one on the real activations and compares them bit for bit. `STRATA_Q36_OLD_KERNELS=1` switches +the new kernels off. Gate/up for Q4_K experts uses the V2K kernels (`STRATA_EXPERT_V2K`, the default on gfx12). + +> **Measured on:** AMD Radeon AI PRO R9700 32 GB (HIP gfx1201), UD-Q4_K_XL, Strata 0.1.45 against the engine before it, +> 5 interleaved pairs, medians; llama.cpp 972b5fc HIP on the same file (`-fa 1`). + +| | Before | 0.1.45 | llama.cpp | +| --- | ---: | ---: | ---: | +| Decode tg128 | 83.2 tok/s | 117.8 | 81.9 to 82.1 | +| Prompt pp4096 | 3,453 | 6,220 (chunk 4096) | 3,794 (`-ub 2048`) / 3,042 (`-ub 512`) | + +More numbers are in [History](#history--older-measurements). + +## Limits + +- Eight slots at most (the window holds 8 rows). `--batch-mtp` shares those rows between slots and proposals. +- A sampling request in a batch decodes one token per window without proposals. +- `logit_bias` is not available on this engine path. The server refuses a request that has it (as it does with + `--batch` for Flash-Next). +- A slot that stopped being alone goes back to the solo path only through the server's own hand-over. The MTP block's + cache has holes for the tokens a slot decoded in batch windows, so the drafts of a later solo stretch are a little + worse. The text is not affected. +- Intel Arc (SYCL) and Mac do not run this family. + +## Testing + +The gates are in `tools/qwen35/` and talk to the engine through its serve protocol (no server needed). Run them under +the box's GPU lock. + +| Script | Checks | +| --- | --- | +| `q36_solo_gate.py` + `q36_cmp_gate.py` | The 10 parity prompts x 128 greedy tokens through `GEN`, compared with another build's. | +| `q36_batch_gate.py --slots N --waves --long 2500` | Every slot's tokens equal its solo tokens, slots reused, a long prompt read beside decoding slots. | +| `q36_proto_edge.py` | `max_new` 1, `BSTOP`, a busy slot, `STOP` during a prompt read, held-state reuse, sampled requests, the context's end. | +| `q36_interleave_probe.py` | What the decoding slots see while a long prompt is read. | +| `q36_vision_cmp.py` (with `q36_mtmd_dump.cpp` built against llama.cpp) | The picture comparison against llama.cpp. | +| `q36_vision_proto.py` | Pictures through `GENI` / `BGENI` (solo, in a batch, the conversation cache). | +| `q36_vision_serve.py`, `q36_serve_par.py`, `q36_serve_handoff.py` | The same through the server. | +| `q36_par_bench.py` | The aggregate-throughput A/B against `llama-server -np N`. | + +## History / older measurements + +### Pictures checked against llama.cpp + +> **Measured on:** R9700 32 GB (HIP), UD-Q4_K_XL, Strata 0.1.45. llama.cpp's `mtmd` and its own graph, the encoder on the +> CPU on both sides, the same prompt ids, llama.cpp's answer fed to both, logits compared step by step over 24 steps. +> The floor is llama.cpp's CPU backend against its own HIP run. + +| Picture and question | Prompt tokens | First token | Mean KL, llama.cpp HIP vs Strata | Floor | | --- | ---: | --- | ---: | ---: | | 640x400, text in the image | 288 | same | 0.029 | 0.056 | | 640x400, list the shapes | 294 | same | 0.035 | 0.039 | | 800x450, text in the image | 378 | same | 0.021 | 0.031 | | 800x450, describe the shapes | 381 | same | 0.0005 | 0.0019 | -| 1024x1024, the title (1,024 image tokens, read in 3 chunks) | 1,042 | same | 0.0022 | 0.0026 | -| two pictures in one request, the text in each | 655 | differs (a near tie: the floor's KL at this step is 0.16 too) | 0.014 | 0.022 | +| 1024x1024, the title (1,024 image tokens, 3 chunks) | 1,042 | same | 0.0022 | 0.0026 | +| two pictures in one request | 655 | differs (a near tie; the floor's KL at this step is 0.16) | 0.014 | 0.022 | -The answers are the same text as llama.cpp's: `STRATA 42` (with llama.cpp's own spelling ` STRATA42`), `CODE 4217`, `Red circle / Blue square -/ Green triangle`, `A red square is positioned on the left ... A blue circle is positioned on the right ...`. With the picture options on, -a request without pictures gives the same tokens as without them (10 prompts x 128 greedy tokens, identical). +The answers are the same text as llama.cpp's. With the picture options on, a request without pictures gives the same +tokens as without them (10 prompts x 128 greedy tokens). -Pictures work with several requests at once, with the MTP drafts and with the conversation cache: the same conversation sent again (a chat -app resends its pictures) continues from the held state when its pictures are the same (they are matched by the server's key for each -picture); a different picture starts that conversation again. +### Aggregate decode speed, several requests at once -## Several requests at once +> **Measured on:** same prompts, greedy, 128 new tokens each, wall time from first request to last answer, 5 +> interleaved pairs, medians, Strata 0.1.45 with `--batch 8 --mtp 0` against `llama-server -np N`. -`"parallel": N` in the model's config (`strata-qwen36-.json`, or `--parallel N` at setup) makes the engine keep N conversations open -(`--batch N`, up to 8) and decode them together: every window carries one token of each conversation, so the dense weights, the head and -every routed expert two conversations share are read once for all of them. The server needs no change: it speaks the same engine protocol -as for Flash-Next ([BATCHING.md](BATCHING.md)). A request that arrives while another is read waits only for that read: the prompt of a -new request is read **between** the decode windows of the others, and a conversation continues in the slot that holds its start (or the -engine copies the longer held start from another slot). A 3,008-token prompt read in 0.95 s (3,200 tok/s, 6 chunks) beside three decoding -slots: each of them paused at most 160 ms (one chunk) instead of the whole read, and produced 20 tokens meanwhile (one window per chunk, -the default); `STRATA_Q36_READ_WINDOWS=4` gives them 65 tokens during a 1.14 s read (more of the time, a slower read). -A request that runs alone (the solo path, with its MTP drafts) and is moved into a batch slot when another arrives, and back when it is -alone again, gives the text it gives running alone (the server's hand-over reuses the held state). +| Card | Requests | Strata tok/s | llama.cpp | Ratio | +| --- | ---: | ---: | ---: | ---: | +| R9700 32 GB, UD-Q4_K_XL | 1 | 83.9 | 73.8 | 1.14 | +| | 2 | 173.5 | 107.4 | 1.61 | +| | 4 | 252.0 | 165.0 | 1.53 | +| | 8 | 325.4 | 208.1 | 1.56 | +| RTX 3060 12 GB, hybrid (llama.cpp `-ncmoe 28 -t 8`, the best split of a sweep) | 1 | 64.0 | 41.1 | 1.56 | +| | 2 | 85.1 | 52.5 | 1.62 | +| | 4 | 99.3 | 63.0 | 1.58 | -Aggregate decode speed, N requests at once, same prompts, greedy, 128 new tokens each, wall time from the first request to the last -answer, **R9700 32 GB, UD-Q4_K_XL, llama.cpp `llama-server -np N`** (5 interleaved pairs, median; Strata `--batch 8 --mtp 0`): +All pairs were faster for Strata. Every slot's tokens equal the tokens of the same request alone. -| Requests at once | Strata | llama.cpp | Ratio | Pairs faster | -| ---: | ---: | ---: | ---: | ---: | -| 1 | 83.9 tok/s | 73.8 | 1.14 | 5/5 | -| 2 | 173.5 | 107.4 | 1.61 | 5/5 | -| 4 | 252.0 | 165.0 | 1.53 | 5/5 | -| 8 | 325.4 | 208.1 | 1.56 | 5/5 | +### Several requests with the base engine, and with `--batch-mtp` -**RTX 3060 12 GB, UD-Q4_K_XL in the hybrid mode** (each engine with the slots it needs: `--batch N` takes room from the expert slots; llama.cpp -`llama-server -np N -ncmoe 28 -t 8`, the best split of a sweep on this card; 5 interleaved pairs, medians; the box was shared with other -jobs between runs, never during one): +> **Measured on:** R9700 32 GB, 5 interleaved pairs, Strata 0.1.45 (before the speed work below). -| Requests at once | Strata | llama.cpp | Ratio | Pairs faster | -| ---: | ---: | ---: | ---: | ---: | -| 1 | 64.0 tok/s | 41.1 | 1.56 | 5/5 | -| 2 | 85.1 | 52.5 | 1.62 | 5/5 | -| 4 | 99.3 | 63.0 | 1.58 | 5/5 | +| Requests | `--mtp 0` | `--mtp 2 --batch-mtp` | Ratio | +| ---: | ---: | ---: | ---: | +| 1 | 83.7 tok/s | 134.0 | 1.60 | +| 2 | 173.1 | 207.0 | 1.20 | +| 4 | 251.0 | 275.0 | 1.10 | +| 8 | 324.0 | 320.5 | 0.99 (no row to spare: no proposals) | -Every slot's tokens equal the tokens of the same request alone (the base engine's), at 2, 4 and 8 slots, with a 2.5K-token prompt read beside -decoding slots, with slots reused for later requests, with pictures and with sampling (same seed). Through the server: four concurrent chat -requests give the texts of the same four one after the other. +> **Measured on:** hybrid cards, UD-Q4_K_XL, medians of 5 interleaved pairs, `--mtp 2`; "together" is the total tokens +> per second of N simultaneous requests against one request at a time with the draft layer. -**MTP drafts.** A request that has the engine to itself keeps its MTP drafts (the server sends it alone while nothing else runs). With -several requests the batch window has no drafts unless `--batch-mtp` is in the config's `args` (opt-in): every slot's window then also -carries the proposal its MTP block made (one per slot, up to the rows to spare in the 8-row window) and a confirmed proposal moves two -tokens in the window. Same prompts, 5 interleaved pairs, R9700: - -| Requests at once | `--mtp 0` | `--mtp 2 --batch-mtp` | Ratio | Pairs faster | -| ---: | ---: | ---: | ---: | ---: | -| 1 | 83.7 tok/s | 134.0 | 1.60 | 5/5 | -| 2 | 173.1 | 207.0 | 1.20 | 5/5 | -| 4 | 251.0 | 275.0 | 1.10 | 5/5 | -| 8 | 324.0 | 320.5 | 0.99 | 0/5 (no row to spare: no proposals) | - -The text is the same with and without it (every slot equal to its solo tokens). - -**What a slot costs.** Each slot keeps its own cache in VRAM: 1.5 GB at a context of 32K (fp32 keys and values of the 10 attention layers -and the MTP block, 45 KB per token) and 0.07 GB of recurrent state. Setup writes the context for one conversation, so N slots take N times -that from the room the experts would have: on a card that does not hold every expert (12 and 16 GB, the hybrid mode) that is room the -expert slots lose, and a batch reads more distinct experts from RAM. Setup **offers `--parallel`** where the sessions do not eat the experts' room: up to 3-4 where every expert still fits (3 on a 32 GB card at -32K), and on a card that does not (the hybrid mode) the count whose sessions take at most 35% of the room the experts would have, so it -depends on the context you chose (12 GB: 4 slots at 8K, none at 32K; 16 GB: 2 slots at 32K). It says the price in the same breath, asks -"How many requests at once?" (Enter keeps one at a time), and any number you give is kept. Measured on the hybrid cards (UD-Q4_K_XL, medians of 5 -interleaved pairs, `--mtp 2` as setup writes it; "together" = total tokens per second of N simultaneous requests, against ONE request at a time -with the draft layer): - -| card, context | slots | one request alone, vs 1 slot | N together, vs one at a time | +| Card, context | Slots | One request alone, against 1 slot | N together, against one at a time | | --- | ---: | ---: | ---: | | RTX 3060 12 GB, 8K | 2 / 4 / 8 | -3.5% / -4.3% / -12% | 0.99x / 1.15x / 1.10x | | RTX A4000 16 GB, 32K | 2 / 4 | -10.6% / -29.9% | 1.03x / 0.76x | | Tesla P100 16 GB, 32K | 2 / 4 | -7% / -23% | 0.98x / 0.81x | -Without the draft layer (`--mtp 0`) the same slots give 1.26-1.64x together (3060 at 8K: 4 slots 1.64x for -5.5% alone), but a single request -with drafts is already about 1.6x as fast as one without, so on a small card the main gain is that a second user starts at once; the -speed gain is small. On a card that holds every expert (R9700 32 GB, 8K prompts) 4 slots run 1.45x one-at-a-time-with-drafts (1.59x with -`--batch-mtp`). The engine says what the sessions take before it allocates them and refuses (with the numbers) a count that cannot fit. +Without the draft layer (`--mtp 0`) the same slots give 1.26x to 1.64x together (3060 at 8K: 4 slots 1.64x for -5.5% +alone). A single request with drafts is already about 1.6x as fast as one without, so on a small card the main gain is +that a second user starts at once. On a card that holds every expert (R9700 32 GB, 8K prompts), 4 slots run 1.45x +one-at-a-time-with-drafts (1.59x with `--batch-mtp`). -In the hybrid mode the text of a request can differ a little between running alone and running beside others: the experts that sit on the -GPU or on the CPU change with the routing of all the requests, and the two compute with different rounding. With the slot tables fixed (`STRATA_Q36_FIXED_SLOTS=1`, a test setting) every slot equals its -solo run there too, with and without `--batch-mtp`. +### Prompt read beside decoding slots -## Speed (0.1.45) +> **Measured on:** R9700 32 GB, Strata 0.1.45, three decoding slots. -Decode and prompt kernels of this engine path were rewritten for speed; **every change on the decode path gives the old kernels' bits** -(`STRATA_Q36_VERIFY=1 strata ... --no-graph` runs the old kernel beside each new one on the real activations and compares them bit for bit: -0 mismatches in 18,360 / 8,187 / 6,399 comparisons on the R9700 and the RTX 3060; `STRATA_Q36_OLD_KERNELS=1` switches them all off). The -10 parity prompts (greedy 128) give the base engine's tokens, plain and with `--mtp 2`. Where the prompt's chunk size changes, the numbers -change a little: see "Prompt chunk". +A 3,008-token prompt was read in 0.95 s (3,200 tok/s, 6 chunks). Each decoding slot paused at most 160 ms (one chunk) +and produced 20 tokens meanwhile. With `STRATA_Q36_READ_WINDOWS=4` they produced 65 tokens during a 1.14 s read. -**What was slow** (rocprofv3, R9700, UD-Q4_K_XL, plain decode, 11.25 ms of kernel time per token): the big Q8_0 products (attention, GDN -projections, head: 1.9 GB per token) run at 530 GB/s, the memory bound. The rest did not: the shared expert's and the attention k/v products -(140 per token) at 18 us, the Q4_K gate/up kernel at 250 GB/s, `k_moe_out` (one block) 17 us, `k_group` (one thread walking 256 experts) 9 us. -Now: those products one warp per row (a launch of 2048 warps took 21 us on gfx1201 where 1024 warps took 5 us, so the grid is capped), shared -gate+up+SwiGLU+q8_1 in one launch, the MoE end with 1024 threads and the next layer's q8_1 image, add+norm+q8_1 in one launch, router top-k and -grouping in one launch, the router and the shared gate in one product, Q4_K gate/up on the V2K kernels (`STRATA_EXPERT_V2K`, gfx12 default). -The prompt's GDN conv runs in tiles of 16 tokens side by side and the delta rule loads one token ahead with two barriers per token (both exact). +### Speed work in 0.1.45 -R9700 32 GB (p620, HIP gfx1201), UD-Q4_K_XL, base = the engine before this work, 5 interleaved pairs, 3 repetitions each, medians; llama.cpp -972b5fc HIP on the same file (`-fa 1`, best of `-ub` 512 / 2048): +> **Measured on:** R9700 32 GB (HIP gfx1201), UD-Q4_K_XL, 5 interleaved pairs, 3 repetitions each, medians; base = the +> engine before 0.1.45. -| | base | 0.1.45 | ratio | pairs faster | llama.cpp | -| --- | ---: | ---: | ---: | ---: | ---: | -| decode tg128 | 83.2 tok/s | 117.8 | 1.42 | 5/5 | 81.9-82.1 | -| prompt pp4096 | 3,453 | 6,220 (chunk 4096) | 1.80 | 5/5 | 3,794 (-ub 2048) / 3,042 (-ub 512) | -| `--mtp 2`, 10 prompts greedy 128 | 167.7 | 183.7 | 1.10 | 3/3 | 129 (draft-mtp, n-max 2, earlier run) | -| 1 / 2 / 4 / 8 requests at once (`--batch 8 --mtp 0`) | 85.2 / 182.4 / 271.9 / 361.1 | 122.4 / 205.1 / 303.3 / 386.9 | 1.44 / 1.12 / 1.12 / 1.07 | 5/5 | | +| | Base | 0.1.45 | Ratio | +| --- | ---: | ---: | ---: | +| `--mtp 2`, 10 prompts greedy 128 | 167.7 tok/s | 183.7 | 1.10 | +| 1 / 2 / 4 / 8 requests (`--batch 8 --mtp 0`) | 85.2 / 182.4 / 271.9 / 361.1 | 122.4 / 205.1 / 303.3 / 386.9 | 1.44 / 1.12 / 1.12 / 1.07 | +| Prompt pp4096 at chunk 512 / 1024 / 2048 / 4096 | | 3,453 / 4,786 / 5,766 / 6,223 | | -UD-Q6_K_XL on the same card (every main layer's experts in VRAM, so the prompt chunk stays 512): decode 80.4 -> 106.9 tok/s (1.33), pp4096 -2,160 -> 2,200 (llama.cpp 3,486 at -ub 2048, 2,738 at -ub 512; decode 81.1), `--mtp 2` 154.7 -> 167.3. `--chunk 2048` there moves the experts -to the hybrid placement (205 of 256 per layer in VRAM): prompt 3,500 tok/s, decode 94 - an option, not the default. +UD-Q6_K_XL on the same card (every main layer's experts in VRAM, chunk stays 512): decode 80.4 to 106.9 tok/s, pp4096 +2,160 to 2,200 (llama.cpp 3,486 at `-ub 2048`, 2,738 at `-ub 512`), `--mtp 2` 154.7 to 167.3. `--chunk 2048` there moves +the experts to the hybrid placement (205 of 256 per layer in VRAM): prompt 3,500 tok/s, decode 94. That is an option, +not the default. -RTX 3060 12 GB (P2003, CUDA sm_86, 20 threads), UD-Q4_K_XL in the hybrid mode (37 of 256 experts per layer in VRAM): the GPU part of a -token is the dense Q8_0 at the card's bandwidth (6 ms of the 17) and the rest waits for the CPU pool (9 ms of experts), so only the small -kernels' share is gained: decode tg128 58.5 -> 59.7 tok/s (+2%), pp2048 1,958 -> 2,022 (+3.3%), 10/10 identical, plain and `--mtp 2` -(84.3 -> 84.7). llama.cpp 972b5fc CUDA (`-ncmoe 26 -t 8 -ub 2048`): pp2048 1,328, tg128 48-52. +> **Measured on:** RTX 3060 12 GB (CUDA sm_86, 20 threads), UD-Q4_K_XL in the hybrid mode (37 of 256 experts per layer +> in VRAM), Strata 0.1.45. -**Prompt chunk (the default changed).** The MoE prompt kernels read every expert once per chunk, so a bigger chunk feeds them more rows per -expert: R9700 pp4096 at chunk 512 / 1024 / 2048 / 4096 = 3,453 / 4,786 / 5,766 / 6,223 tok/s. With every expert in VRAM the engine now takes the -largest of 4096 / 2048 / 1024 that fits beside the experts, the 0.77 GiB reserve and the sessions' caches without giving the packed decode -copies back (8K-64K context: 4096, 100K: 2048, 131K: 512; `--batch 2..4`: 1024 at most so the other slots are not held for a whole chunk, -`--batch 8`: 512). The hybrid mode keeps 2048 (a bigger chunk takes expert slots: RTX 3060 chunk 4096 reads prompts 5% faster and decodes 13% -slower) and goes down to 1024 / 512 on a card with 7.5 / 5.5 GiB free. **`--prefill 512` (or `STRATA_Q36_CHUNK=512`) gives the old size, and -then the text is the base engine's, bit for bit.** The chunk size changes the GEMM tiles the BLAS picks and with them the last bits of the -prompt state: KL against the 512 chunk on six real 1.5-3.2K-token documents (48 teacher-forced rows each): chunk 1024 4.0e-3, 2048 4.2e-3, -4096 3.6e-3 (per prompt 2.3e-3 .. 5.9e-3), top-1 agreement 44-48 of 48 rows - the size of the llama.cpp CPU-vs-HIP difference on the same -file (4.2e-3), not a quality loss. Greedy 64 tokens after those six documents: the default chunk gives the base engine's text for 2 of 6, -`--prefill 512` for 6 of 6. A prompt shorter than the chunk (every chat turn after the first) is one chunk either way: unchanged. -The maintainers keep the new default's reference outputs (six documents, greedy 64) and check later changes against them. +Decode tg128 58.5 to 59.7 tok/s (+2%), pp2048 1,958 to 2,022 (+3.3%), 10/10 prompts identical, plain and with `--mtp 2`. +llama.cpp 972b5fc CUDA (`-ncmoe 26 -t 8 -ub 2048`): pp2048 1,328, tg128 48 to 52. The GPU part of a token is the dense +Q8_0 at the card's bandwidth and the rest waits for the CPU pool, so only the small kernels' share is gained. -**Tried and dropped.** The MTP drafts' head over a subset of the vocabulary (`--draft-vocab N`, kept as an opt-in; the answer never -changes): the first 16K-131K rows gain 5-8% on English and code but lose 10-15% on Chinese / Russian / Arabic / Hindi (70% of a Chinese -prompt's token ids are above 65,536), and a candidate list built from the tokens seen so far loses on both. The shared expert on a second -stream (captured into the decode graph) decodes at 36 tok/s instead of 117 with this ROCm. Chunked (WY) GDN and the PCIe-assisted hybrid decode -were not attempted. - -## Not done / limits - -- Eight slots at most (the window holds 8 rows); `--batch-mtp` shares those rows between slots and proposals. -- A sampling request in a batch decodes one token per window without proposals. `logit_bias` is not available on this engine path: the - server refuses a request that has it (as it does with `--batch` for Flash-Next). -- A slot that stopped being alone goes back to the solo path only through the server's own hand-over; the MTP block's cache has holes for - the tokens a slot decoded in batch windows (the drafts of a later solo stretch are a little worse; the text is not affected). -- Intel Arc (SYCL) does not run this family. - -## Testing - -The gates are in `tools/qwen35/` and talk to the engine through its serve protocol (no server needed); run them under the box's GPU lock: - -- `q36_solo_gate.py` + `q36_cmp_gate.py`: the 10 parity prompts x 128 greedy tokens through `GEN`, compared with another build's. -- `q36_batch_gate.py --slots N --waves --long 2500`: every slot's tokens equal its solo tokens, slots reused, a long prompt read beside - decoding slots. `q36_proto_edge.py`: `max_new` 1, `BSTOP`, a busy slot, `STOP` during a prompt read, held-state reuse, sampled requests, - the context's end. `q36_interleave_probe.py`: what the decoding slots see while a long prompt is read. -- `q36_vision_cmp.py` (with `q36_mtmd_dump.cpp` built against llama.cpp): the picture comparison above; `q36_vision_proto.py`: pictures through - `GENI` / `BGENI` (solo, in a batch, the conversation cache); `q36_vision_serve.py` and `q36_serve_par.py` / `q36_serve_handoff.py`: - the same through the server. -- `q36_par_bench.py`: the aggregate-throughput A/B against `llama-server -np N`. +KL against the 512 chunk on six real 1.5K to 3.2K-token documents (48 teacher-forced rows each): chunk 1024 4.0e-3, +2048 4.2e-3, 4096 3.6e-3, top-1 agreement 44 to 48 of 48 rows. Greedy 64 tokens after those documents: the default chunk +gives the base engine's text for 2 of 6, `--prefill 512` for 6 of 6. diff --git a/docs/SECOND_GPU.md b/docs/SECOND_GPU.md index d17091000..1432de7a4 100644 --- a/docs/SECOND_GPU.md +++ b/docs/SECOND_GPU.md @@ -1,227 +1,130 @@ -# CUDA1–3 expert experiment +# CUDA1-3 expert caches and the peer tier (experimental) -CUDA0 keeps the dense layers, KV/state, MTP and its existing expert cache. Up to -three independent expert caches fill CUDA1, CUDA2 and CUDA3 in that order. Each -expert has one owner: CUDA0, one secondary GPU, or the CPU pool. Each secondary -GPU computes its rows while the CPU pool works; the results return through -private pinned host buffers before CUDA0 continues the layer. The GPU launches -can overlap, but each layer waits for all three. +This page is for people who tune the engine by hand and want a second, third or fourth card to hold extra experts. These are +experimental engine settings. For ordinary multi-GPU use in the app or `strata launch`, see [MULTI_GPU.md](MULTI_GPU.md). The +settings go into the run config (`strata-*.json`) as `"args"` entries; they are engine flags, not `strata launch` flags. -The shipped profile ranks 8,000 `(layer, expert)` pairs. CUDA0 takes its usual -`--expert-cache auto` allocation. With only CUDA1, it takes the first remaining -pairs. When CUDA2 or CUDA3 is enabled, the remaining profiled pairs are assigned -round robin to the secondary GPUs. The ranked list is then extended with all -other pairs in expert-then-layer order and distributed in the same way. This -fills the additional VRAM and lets all secondary GPUs see frequent experts, but -unranked experts may hardly ever be routed. The startup -log reports slots and GiB per card; each request reports the number of expert -entries actually computed on each secondary GPU. Remote outputs are packed -before returning over PCIe/USB4: the request log compares MiB transferred -with the previous full-row transfer. The input and routing metadata use pinned -host staging. This reduces transfer traffic and launch overhead, but does not -release the expert arena in system RAM or significantly change VRAM use. +> **On this page:** [How it works](#how-it-works) · [Helper caches](#helper-caches) · [Faster helper decode](#faster-helper-decode---remote-expert-opt) · +> [Peer tier](#peer-tier---peer-device) · [Resident RAM mode](#the-resident-ram-mode-with-the-helper-caches) · +> [Choosing the primary device](#choosing-the-primary-device-on-windows-strata_primary_device) · [History](#history--older-measurements) -The optional `--gpu-placement layer` gives each layer to exactly one secondary -GPU (layer number modulo the number of enabled cards). It fills that card with -the most frequently routed experts of its layers first, then the unranked ones. -This can reduce USB4 device switches and synchronization when several secondary -cards serve the same layer in the default `stripe` mode. All three cards still -keep their requested expert slots; the primary CUDA0 cache and CPU fallback -continue to work. The log counts active layer launches on each card, so the -tradeoff can be measured alongside tokens per second. This placement is -experimental and may be slower if a given layer needs more than one card's -compute capacity. +## How it works -The experiment requires the corresponding visible CUDA devices and a CUDA -enabled build. Each card must keep at least 512 MiB free. Tiers must be enabled -in order. Under WDDM (Windows, WSL2), CUDA registration of the host expert arena is capped at -8 GiB to leave room for the contexts and MTP on CUDA0 (`STRATA_ARENA_PIN_GIB` overrides it; on Linux the whole -arena is registered). The rest remains -available to the CPU pool; the PCIe expert path is available only for the -registered layers. Without secondary GPUs, the original uncapped registration -behavior applies. +CUDA0 keeps the dense layers, the KV and state, the draft layer (MTP) and its own expert cache. Up to three independent expert +caches fill CUDA1, CUDA2 and CUDA3 in that order. Each expert has one owner: CUDA0, one helper card, or the CPU pool. Each +helper computes its rows while the CPU pool works, and the results return through private pinned host buffers before CUDA0 +continues the layer. Each layer waits for all helpers. -On an existing Linux installation, apply the patch to its source tree, then run: +- No peer-to-peer access is needed. Prompt prefill still uses CUDA0; helpers serve decode, including draft verification. +- Each helper card must keep at least 512 MiB free. Enable the tiers in order (CUDA1 before CUDA2). +- The startup log reports slots and GiB per card, and each request reports how many expert entries each helper computed. + Compare identical requests at one, two and four GPUs. VRAM use alone does not show useful offload. +- Longer contexts reserve more KV on CUDA0 and shrink its automatic expert cache. +- Under WDDM (Windows, WSL2) only 8 GiB of the host expert arena is registered with CUDA when helpers are on + (`STRATA_ARENA_PIN_GIB` overrides it). On Linux the whole arena is registered. +- The CUDA0 expert-cache GPU hit path gives outputs that diverge from cache-off runs. Treat performance as experimental until the + generated tokens are validated. -```sh -./setup.sh --setup --gpu1-experts 5000 --gpu2-experts 5000 --gpu3-experts 5000 -``` +## Helper caches -On Windows, from PowerShell in the installation folder: +| Engine flag | Meaning | +| --- | --- | +| `--expert-cache-device1 N\|auto` | Fill CUDA1 with N experts, or as many as fit. | +| `--expert-cache-device2 N\|auto` | Fill CUDA2 with more experts. | +| `--expert-cache-device3 N\|auto` | Fill CUDA3 with more experts. | +| `--expert-cache-remote-placement stripe\|layer` | `stripe` (default) spreads the ranked expert list over the helpers. `layer` gives each layer to one helper (layer number modulo the number of cards), which can reduce device switches on USB4 but may be slower. | -```powershell -.\START-HERE.bat --setup --gpu1-experts 5000 --gpu2-experts 5000 --gpu3-experts 5000 -``` +The shipped profile ranks 8,000 `(layer, expert)` pairs. CUDA0 takes its usual `--expert-cache auto` share. With only CUDA1 it +takes the first remaining pairs. With CUDA2 or CUDA3 the remaining ranked pairs go round robin to the helpers, then all other +pairs in expert-then-layer order, so the extra VRAM fills even though unranked experts may rarely be routed. -To test the layer placement instead, add `--gpu-placement layer` to the setup -command. To go back, run the same command with `--gpu-placement stripe`. After -the new engine is built, the mode can also be changed without a rebuild: edit -`--expert-cache-remote-placement` in `strata-iq3_xxs.json` from `layer` to -`stripe` or vice versa and restart the server. If the argument is absent, the -default is `stripe`. +`auto` fills each helper from its assigned ranking using actual aligned expert bytes and free VRAM, keeping the 512 MiB allowance. +It sizes capacity at startup, not throughput, and a card's truncated tail is not moved to another card. Explicit numbers still work. -Choose the same model, size, context and image settings as your current -installation when prompted. Setup recompiles the engine, updates its server -configuration, and starts it. Later starts keep those settings. To tune the -slots without recompiling, edit the values after `--expert-cache-device1`, -`--expert-cache-device2` and `--expert-cache-device3` in the stored -`strata-iq3_xxs.json`, then restart with `.\run-iq3_xxs.bat`. - -Compare identical requests at one, two and four GPUs, preferably with several -repeats. VRAM use alone does not show useful offload: compare the per-request -CUDA1–3 counts and tokens per second. More GPU contexts and synchronization -may lower the speed. Prompt prefill still uses CUDA0; secondary GPUs serve -decode, including MTP verification. No peer-to-peer access is required. - -Longer contexts reserve more KV/state memory on CUDA0, which reduces its -automatic expert cache. On a 64 GB PC `setup.py` limits IQ3_XXS to 128K even -if 262K was selected; the secondary caches still require the host expert arena. -To try 262K on 64 GB, choose Q2_0 or IQ2_XS instead and check that CUDA0 still -has enough free VRAM after loading. Benchmark long prompts separately from -short decode requests. - -The existing warning about the CUDA0 expert-cache GPU hit path still applies: -its outputs diverge from cache-off runs. Treat performance as experimental -until the generated tokens have been validated. - -## Optional helper decode optimization - -`--remote-expert-opt` (`--serve` only) optimizes the CUDA1-3 helper caches -above. The engine's default is off; since 0.1.39b setup adds it to a config on -two or more GPUs (`--gpus`, or "use both" at start). It acts only when a helper -cache is configured; a layer split runs exactly as before. To leave it out: -setup's `--no-remote-expert-opt`, or `"remote_expert_opt": false` in the -model's `strata-*.json` (kept when setup runs again). Measured by the PR's -author: dual RTX 4090 +63% mixed / +132% code decode over the plain helper -path; RTX 5090 + 4090 +28% / +63%. The primary cache avoids admitting experts already held -by a helper, and helpers replace cold experts with frequently routed CPU -misses using their existing same-layer slots. Each helper reduces its expert -outputs to a weighted partial sum on its GPU before returning one vector per -token. Tokens with no CPU expert work skip CPU activation quantization. - -For an existing server configuration with CUDA0 as the primary and CUDA1 as a -helper, use these engine arguments alongside the model and profile arguments: +Example engine arguments beside the model and profile arguments: ```text --expert-cache auto --expert-cache-device1 auto --remote-expert-opt ``` -`--expert-cache-device1`, `--expert-cache-device2` and -`--expert-cache-device3` now also accept `auto`: fill each helper from its -assigned ranking using actual aligned expert bytes and free VRAM, retaining -the existing 512 MiB allowance. Explicit numeric budgets still work. This is -startup capacity sizing, not throughput balancing; with several helpers a -card's truncated candidate tail is not redistributed to another card. +To change slot counts without recompiling, edit the values in the stored config and restart. -The optimization uses the existing host scheduling and pinned-host transport, -not P2P or tensor parallelism. Each layer still waits for its participating -helpers. It changes floating-point summation order, so enabled output is not -claimed to be bitwise identical. Only two-card CUDA operation has been measured -by its author; a community machine measured two cards on HIP (2x RX 6900 XT, -PCIe 4.0 x8 each, IQ3_S): there the expert plan's PCIe share had been taking -experts the helper already held, which made the primary card wait 43 ms per -verify window instead of 19 (decode 40 instead of 66 tok/s). #854 keeps a -helper's experts out of that share; before it, `--pcie-frac 0` avoided the -cost ([bench/results/2026-10-04-rdna2-helper-pcie-share](../bench/results/2026-10-04-rdna2-helper-pcie-share/README.md)). -Three/four cards have not been validated. Without the switch, the -existing decode path remains in use. This does not optimize the separate -`--peer-device` path below or change its existing incompatibility with helper -caches. +**Size a helper against the card's real free VRAM.** On Windows `auto` reads the process's WDDM budget. It cannot see the +desktop, the vision encoder (a separate process), a streaming host or other apps. A helper auto-sized on a card that also drives +the display over-commits, and WDDM backs the excess with system RAM. Use explicit counts on a card that shares its VRAM. + +## Faster helper decode (`--remote-expert-opt`) + +`--remote-expert-opt` (serve mode only) optimizes the helper caches. The primary cache avoids admitting experts a helper already +holds, helpers replace cold experts with frequently routed CPU misses, each helper reduces its expert outputs to a weighted +partial sum on its GPU before returning one vector per token, and tokens with no CPU expert work skip CPU activation +quantization. It does nothing without a helper cache, and a plain layer split runs as before. + +- Setup adds it to a config with two or more GPUs and a helper cache. To leave it out, use setup's `--no-remote-expert-opt` or + `"remote_expert_opt": false` in the config (kept when setup runs again). +- It changes the floating-point summation order, so output is not claimed to be bitwise identical. +- It switches `--pipeline-windows` off ([MULTI_GPU.md](MULTI_GPU.md#pipeline-windows)). Keep one of the two. +- Only two-card CUDA operation has been measured by its author. Three and four cards are not validated. +- It does not apply to the `--peer-device` path below. ## Peer tier (`--peer-device`) -`--peer-device N` puts a second adaptive expert cache on CUDA device N. It -takes the ranked pairs CUDA0's cache does not hold, as many as fit. The peer -computes the rows of its own experts, for decode windows and for prompt -chunks; the activations and the results cross NVLink or another P2P path. The -tier adapts while the server runs, like the primary cache. It is an -alternative to the CUDA1-3 caches above, not a third tier beside them. +`--peer-device N` puts a second adaptive expert cache on CUDA device N. It takes the ranked pairs CUDA0's cache does not hold, as +many as fit. The peer computes the rows of its own experts for decode windows and for prompt chunks. Activations and results cross +NVLink or another peer-to-peer path. It is an alternative to the helper caches above, not a third tier beside them. -- `--peer-device N` (default off): enable the tier on CUDA device N (N >= 1). -- `--peer-reserve-mib M` (default 600): leave M MiB free on the peer card; the - cache takes what remains. The prompt-path buffers need this headroom. - Keep a few hundred MiB free on both cards beyond this: the prompt path loads its kernels and cuBLAS on first use - (a start with 5 MiB left on the peer and 485 MiB on the primary stopped at the first prompt with "cuBLAS status 14"). -- Whether the two cards can really exchange data is tested once at start (#2666): two RTX A4000 behind a host - bridge reported peer access, enabled it, and `cudaMemcpyPeerAsync` then copied nothing, so the peer computed on - zeros and the first prompt came back NaN. A test copy each way now decides; if it arrives wrong the log says so - and the prompt path uses the host route (the one without P2P: activations and rows through mapped host memory). - `STRATA_PEER_NO_P2P=1` asks for that route by hand. -- `--peer-slots N` (default 0): cap the tier at N experts; 0 = as many as fit. -- `--peer-adapt-swaps N` (default -1): swaps per adaptive round on the peer; - -1 uses the primary's `--adapt-swaps`. -- `--peer-prefill-rows N` (default -1): the share of each prompt chunk's rows - the peer computes; -1 is half of chunk x top-k, 0 keeps prompt rows on the - primary. +| Flag | Default | Meaning | +| --- | --- | --- | +| `--peer-device N` | off | Enable the tier on CUDA device N (N >= 1). | +| `--peer-reserve-mib M` | 600 | VRAM left free on the peer card. Keep a few hundred MiB free on both cards beyond this, because the prompt path loads its kernels and cuBLAS on first use. | +| `--peer-slots N` | 0 | Cap the tier at N experts; 0 means as many as fit. | +| `--peer-adapt-swaps N` | -1 | Swaps per adaptive round on the peer; -1 uses the primary's `--adapt-swaps`. | +| `--peer-prefill-rows N` | -1 | Share of each prompt chunk's rows the peer computes; -1 is half of chunk x top-k, 0 keeps prompt rows on the primary. | -Through the server, list both cards in the config's `"gpu"` (e.g. `[0, 1]`, numbered as nvidia-smi numbers them) -and add `--peer-device 1` to its `"args"`; the number is the card's position in that list (with `"gpu": [2, 0]`, -`--peer-device 1` is nvidia-smi's card 0). Several GPUs in `"gpu"` are otherwise a layer split: the server adds -`--layer-split` only when `--peer-device` is not among the args. +Whether the two cards can really exchange data is tested once at start with a copy each way. If the copy arrives wrong, the log +says so and the prompt path uses the host route (activations and rows through mapped host memory). `STRATA_PEER_NO_P2P=1` asks +for that route by hand. -`--peer-device` requires `--expert-profile` and an enabled expert cache, and -the device must be visible; it refuses otherwise. It also refuses -`--layer-split` (a different second-GPU mode: use one or the other) and -`--expert-cache-device1..3` (the peer tier already caches experts on that card): +Through the server, list both cards in the config's `"gpu"` (for example `[0, 1]`, numbered as `nvidia-smi` numbers them) and add +`--peer-device 1` to `"args"`. The number is the card's position in that list (with `"gpu": [2, 0]`, `--peer-device 1` is +`nvidia-smi`'s card 0). Several GPUs in `"gpu"` are otherwise a layer split; the server adds `--layer-split` only when +`--peer-device` is not among the args. - strata generate: --peer-device cannot be combined with --layer-split (use one or the other) - strata generate: --peer-device cannot be combined with --expert-cache-device1..3 (the peer tier already caches experts there) +`--peer-device` requires `--expert-profile` and an enabled expert cache, and the device must be visible. It refuses +`--layer-split` and `--expert-cache-device1..3`: -Without `--peer-device` the binary is unchanged; its output is byte-identical -to the release. With `--peer-device` and the same expert set split across the -two cards, the generated tokens are byte-identical to the single-GPU run under -the exactness gate. +```text +strata generate: --peer-device cannot be combined with --layer-split (use one or the other) +strata generate: --peer-device cannot be combined with --expert-cache-device1..3 (the peer tier already caches experts there) +``` -With a peer the prompt path keeps the MMQ path; the fused int8 prompt path is -not yet combined with the peer's rows. Mapped host buffers gain -`cudaHostAllocPortable` only with a peer, since only then does a second -context write them. The tier size is manual for now (`--peer-reserve-mib`, -`--peer-slots`); automatic sizing on small cards wants the buffer lending of -#216 and is a follow-up. +With a peer, the prompt path keeps the MMQ path (the fused int8 prompt path is not combined with the peer's rows). The tier size +is manual (`--peer-reserve-mib`, `--peer-slots`). With the same expert set split across the two cards, generated tokens are +byte-identical to the single-GPU run under the exactness gate. ## The resident RAM mode with the helper caches -`--resident-experts` (the low-RAM mode) works with the helper caches: the -experts a helper holds are left out of the RAM copy as a split stage's are, and -the helpers' refills read through the source - the RAM copy when it holds the -bytes, the files otherwise. Measured here on an RX 6800 (gfx1030, 16 GB) + RX -6600 (gfx1032, 8 GB), Windows 11, an i5-11400F and 32 GB of RAM (the 6800 alone: -~37.5 decode tok/s, the same worker configuration): the 6600's 3.4-3.8k-expert -cache lifted decode to ~40.6 / 44.2 tok/s on 8K-token requests at 3800 slots, -with the CPU pool's share of the experts down from ~2.9 to ~0.4 entries per -layer-window. The helper's rows cost about 0.3-0.6 ms per layer on the pool's -critical path - usually less than the CPU rows and page faults they replace, -which is why the 6600 (about 2-3x the 6800's cost per row) still pays off as a -cache tier. - -**Size a helper against the card's *real* free VRAM.** `--expert-cache-device1 -auto` reads `cudaMemGetInfo`, which on Windows reports the *process's* WDDM -budget: it cannot see the desktop, the vision encoder (a separate process), a -streaming host or other apps on that card. A helper auto-sized on a card that -also drives the display or the encoder over-commits, WDDM backs the excess with -system RAM ("shared GPU memory"), and that memory leaves the machine's RAM - on -a 32 GB PC it pushed the resident complement into the pagefile and made the -helper read its own cache over PCIe. An explicit count -(`--expert-cache-device1 3800` here, of ~4100 that fit an empty card) kept the -cache in real VRAM; freeing another app's sticky allocation on the same card -was worth about the same again. Prefer explicit counts on a card that shares -its VRAM with anything else. +`--resident-experts` (the low-RAM mode) works with the helper caches. The experts a helper holds are left out of the RAM copy, +and the helpers' refills read through the source: the RAM copy when it holds the bytes, the model files otherwise. ## Choosing the primary device on Windows (`STRATA_PRIMARY_DEVICE`) -The engine runs the model on the device the runtime numbers 0. Windows' HIP -runtime enumerates the cards in a fixed order that `HIP_VISIBLE_DEVICES` only -filters - it cannot reorder - so on a PC whose faster card comes second (an RX -6800 after an RX 6600, say) a run with both visible could only put the model on -the slow one. `STRATA_PRIMARY_DEVICE=N` picks another visible ordinal as the -primary; unset or 0 is upstream's behavior. +The engine runs the model on the device the runtime numbers 0. Windows' HIP runtime enumerates cards in a fixed order that +`HIP_VISIBLE_DEVICES` only filters, so when the faster card comes second, the model can only land on the slower one. +`STRATA_PRIMARY_DEVICE=N` picks another visible ordinal as primary. Unset or 0 keeps the default. -- the helper caches (`--expert-cache-device1..3`) take the visible devices - other than the primary, in order; -- the peer tier's cross-device checks use the same ordinal; -- a layer split is refused with this set (its later stages must be > 0, and - stage 0 is the primary); -- NVIDIA cards do not need it: `CUDA_VISIBLE_DEVICES` with - `CUDA_DEVICE_ORDER=PCI_BUS_ID` already reorders. +- The helper caches (`--expert-cache-device1..3`) take the visible devices other than the primary, in order. +- The peer tier uses the same ordinal. +- A layer split is refused when this is set. +- NVIDIA cards do not need it: `CUDA_VISIBLE_DEVICES` with `CUDA_DEVICE_ORDER=PCI_BUS_ID` already reorders. -Measured on RX 6800 + RX 6600, Windows 11; the report accompanies this change. +## History / older measurements + +Each row is one machine, reported by the author of the change or a community member. + +| Setting | Measured on | Result | +| --- | --- | --- | +| `--remote-expert-opt` | dual RTX 4090 (PR author) | +63% mixed, +132% code decode over the plain helper path | +| `--remote-expert-opt` | RTX 5090 + RTX 4090 (PR author) | +28% mixed, +63% code | +| Helper PCIe share on HIP | 2x RX 6900 XT, PCIe 4.0 x8 each, IQ3_S (community) | before the fix, decode 40 tok/s with the primary waiting 43 ms per verify window; after, 66 tok/s and 19 ms. `--pcie-frac 0` avoided the cost ([report](../bench/results/2026-10-04-rdna2-helper-pcie-share/README.md)) | +| Resident RAM mode with a helper | RX 6800 (16 GB) + RX 6600 (8 GB), Windows 11, i5-11400F, 32 GB RAM | RX 6800 alone ~37.5 tok/s; with a 3,800-slot helper ~40.6 / 44.2 tok/s on 8K-token requests; CPU pool share of experts down from ~2.9 to ~0.4 entries per layer-window | diff --git a/docs/STRIX_HALO.md b/docs/STRIX_HALO.md index 2b89cdf53..41304ac9b 100644 --- a/docs/STRIX_HALO.md +++ b/docs/STRIX_HALO.md @@ -1,26 +1,47 @@ # Strix Halo (Ryzen AI Max, gfx1151) -Strata runs on the Ryzen AI Max "Strix Halo" APUs (Radeon 8060S / 8050S, RDNA3.5, `gfx1151`, one pool of LPDDR5X shared by the -CPU and the GPU) through the same HIP backend as the Radeon cards in [AMD_HIP.md](AMD_HIP.md). **Status: experimental.** -Everything here was built and measured on one machine, the maintainers' Strix Halo box (Ryzen AI Max+ 395, 128 GB, -Ubuntu 26.04, kernel 7.0, ROCm 7.14.1). Setup (`setup.sh`, `START-HERE.bat`) recognizes it from 0.1.40, by its PCI id (`1002:1586`, one id for the 8060S, 8050S and 8040S) and by -`gfx1151` exactly: a Strix Point (890M / 880M, gfx1150), Krackan (860M / 840M, gfx1152) or Phoenix / Hawk Point (780M / 760M, gfx1103) Radeon is -named as what it is and not supported. On Linux setup compiles the engine for gfx1151 itself (the steps below are the same, by hand). The 0.2.0-beta release also has ready-made packages: `Strata-0.2.0-beta-linux-amd-strixhalo.AppImage` (engine `halo`, gfx1151, ROCm 7.14.1) and, on Windows, `Strata-0.2.0-beta-windows-hip-rdna3.exe` (engine `rdna3`: gfx1100, gfx1101, gfx1102 and gfx1151, ROCm 10.2.0a20260930); any app installer plus **Update engine** (or `strata update`) fetches the same package. What was run: both packages were built, linked and scanned, and the Linux package refuses to start on a card of another family (tried on an RX 9000 card); the maintainers have no Strix Halo and ran neither on one. The measurements on this page come from source builds on Linux (the maintainers' earlier box and community machines). Nobody has run a Strix Halo on Windows. The packaged engines do not carry the hipBLASLt table of section 3 yet (planned for 0.2.1), so their prompt reading is slower than the numbers below until then. A Windows zip of the 0.1.x START-HERE route without gfx1151 code is refused with a message, never used. Setup counts the GPU's memory as the BIOS carve-out plus the shared memory -(the GTT pool; on Windows half the RAM) and recommends UD-IQ4_XS from about 80 GB of memory; it changes no BIOS or kernel setting. +This page is for people with a Ryzen AI Max "Strix Halo" PC (Radeon 8060S / 8050S, RDNA 3.5, `gfx1151`, one pool of LPDDR5X +shared by the CPU and the GPU). Strata runs there through the same HIP backend as the Radeon cards in [AMD_HIP.md](AMD_HIP.md). +**Status: experimental.** -**Windows start-up (0.2.1.2).** On a Strix Halo under Windows the engine no longer registers the expert arena with the GPU driver: registering it took all free VRAM (91 GiB before, 0 after, on a 64 GB carve-out + 32 GB shared PC), the expert cache got no slots and the engine stopped with "the embedding buffer failed" (0.2.1 and earlier). The arena stays in RAM, held by the working-set lock, and the cache gets its normal slots. `STRATA_ARENA_PIN_GIB=N` (or `0`) registers it as before, `STRATA_UMA_REGISTER=1` too. Linux is unchanged. +DeepSeek V4.1-Flash does not run on Strix Halo yet. It runs in the app and the Terminal on NVIDIA RTX 30, 40 and 50 cards and on the Radeon AI +PRO R9700 ([MODELS.md](MODELS.md#deepseek-v41-flash)). -What the engine does differently on this chip: +> **On this page:** [Quick start](#quick-start) · [What the engine does differently](#what-the-engine-does-differently) · +> [1. Toolchain](#1-the-toolchain-no-root-needed) · [2. Engine](#2-the-engine) · [3. Tuning table](#3-the-prompt-table-and-running) · +> [4. Defaults](#4-what-is-on-by-default-on-gfx1151) · [5. Opt-in](#5-what-is-not-on-by-default-it-changes-bits) · +> [5b. Desktop and RAM](#5b-the-desktop-the-ram-and-a-discrete-card-beside-the-strix-halo-linux) · [6. Measured](#6-measured) · +> [History](#history--older-measurements) -- **Unified memory.** The GPU's "VRAM" is the GTT pool of the same RAM. `device_free_bytes()` (the figure `--expert-cache auto` and - the memory plan use) counts what Linux can give back (`MemAvailable` less 6 GiB for the system, `STRATA_UMA_HEADROOM_GIB` - changes the 6) instead of only `MemFree`, per device, for any integrated GPU on Linux (this applies to a DGX Spark too). +## Quick start + +Use the [current release](https://github.com/Niko1221/Strata/releases/latest). You do not need to build ROCm for a packaged install. + +| System | What to install | Engine variant | +| --- | --- | --- | +| Linux | `Strata-0.2.5-linux-x86_64.AppImage` (it downloads the engine on first start), or `strata launch` | `linux-x86_64-hip-halo` | +| Windows | `Strata-0.2.5-windows-x64-setup.exe` (it downloads the engine on first start), or `strata launch` | `windows-x86_64-hip-rdna3` | + +Packaged engines carry the matching hipBLASLt tuning table for gfx1151 and select it on their own. Setup recognizes `gfx1151` and +PCI ID `1002:1586`; other integrated Radeon chips have separate support limits. Unified memory is counted once. Setup does not +change BIOS or kernel settings. The manual Linux toolchain below is for reproduction and builds, not a prerequisite for the app. + +## What the engine does differently + +- **Unified memory.** The GPU's "VRAM" is the GTT pool of the same RAM. `device_free_bytes()` (the figure `--expert-cache auto` + and the memory plan use) counts what Linux can give back (`MemAvailable` less 6 GiB for the system; `STRATA_UMA_HEADROOM_GIB` + changes the 6) for any integrated GPU on Linux, a DGX Spark included. - **gfx11 matrix cores.** The prompt attention, the block scorer, the prompt GEMMs and the prompt experts have WMMA kernels for - gfx1100 / 1101 / 1102 / 1103 / 1150 / 1151. A gfx11 part outside that list (gfx1152) is not given them: it takes the portable path. -- **The hipBLASLt table** `tools/hip/gfx1151-hipblaslt-100401.txt` (ROCm 7.14.1's hipBLASLt 1.x `100401`), including the prompt shapes - at `--prefill 16384` (T 16383 / 16384). Setup's own ROCm for gfx1151 (the 7.14.0a20260608 wheels, hipBLASLt 1.4.0) takes - `tools/hip/gfx1151-hipblaslt-100400.txt`: the same shapes, calibrated against that library (docs/AMD_HIP.md, Tuning table). -- **The expert cache and the RAM budget share the RAM.** For a model with a RAM budget (UD-IQ4_XS, UD-Q4_K_XL) setup writes a number for `--expert-cache` (the RAM less 24 GB less the budget), not `auto`: `auto` counts all the memory the OS can give back at start, and with a 55 GiB budget and a desktop open it took the room the OS needed (#1715, a global OOM on a 128 GB box). The number is in the config's `args`; `--expert-cache N` there sets it. An engine run by hand should pass one too. + gfx1100 / 1101 / 1102 / 1103 / 1150 / 1151. A gfx11 part outside that list (gfx1152) takes the portable path. +- **The hipBLASLt table.** `tools/hip/gfx1151-hipblaslt-100401.txt` is for ROCm 7.14.1's hipBLASLt (`100401`), including the prompt + shapes at `--prefill 16384`. Setup's own ROCm for gfx1151 (the 7.14.0a20260608 wheels, hipBLASLt 1.4.0) takes + `tools/hip/gfx1151-hipblaslt-100400.txt`, calibrated against that library. +- **The expert cache and the RAM budget share the RAM.** For a model with a RAM budget (UD-IQ4_XS, UD-Q4_K_XL) setup writes a number + for `--expert-cache` (the RAM less 24 GB less the budget), not `auto`, because `auto` counts all memory the OS can give back at + start and can take the room the OS needs. The number is in the config's `args`. An engine run by hand should pass one too. +- **Windows.** The engine does not register the expert arena with the GPU driver. The arena stays in RAM, held by the + working-set lock, and the expert cache gets its normal slots. `STRATA_ARENA_PIN_GIB=N` (or `0`) or `STRATA_UMA_REGISTER=1` + registers it. Linux registers the whole arena by default. ## 1. The toolchain (no root needed) @@ -37,18 +58,17 @@ export PATH=$SDK/bin:$SDK/lib/llvm/bin:$PATH export LD_LIBRARY_PATH=$SDK/lib:$SDK/lib/rocm_sysdeps/lib:$SDK/lib/llvm/lib:${LD_LIBRARY_PATH:-} ``` -You also need CMake 3.24+, Ninja, git and a C++ host toolchain with libstdc++ headers (`sudo apt install build-essential cmake ninja-build git`). -The maintainers' box has no system compiler: they point ROCm's clang at a user-space copy of gcc 15 with -`--gcc-toolchain=/usr` (added to `CMAKE_HIP_FLAGS` below; leave it out when the system has one). +You also need CMake 3.24+, Ninja, git and a C++ host toolchain with libstdc++ headers +(`sudo apt install build-essential cmake ninja-build git`). If the system has no compiler, point ROCm's clang at a user-space gcc +with `--gcc-toolchain=/usr` in `CMAKE_HIP_FLAGS`. -The kernel's amdgpu driver and `/dev/kfd` access (your user in the `render` and `video` groups) are all the GPU side needs. -To let the GPU use most of a 128 GB machine the maintainers' box boots with `ttm.pages_limit=29360128 ttm.page_pool_size=29360128` -(a 112 GiB GTT; the default is about half of the RAM) and `amd_iommu=off`; both are host settings the engine does not need to run, -only to have the room. +The GPU side needs the kernel's amdgpu driver and `/dev/kfd` access (your user in the `render` and `video` groups). To let the GPU +use most of a 128 GB machine, the maintainers' box boots with `ttm.pages_limit=29360128 ttm.page_pool_size=29360128` (a 112 GiB GTT; +the default is about half of the RAM) and `amd_iommu=off`. Both are host settings the engine does not need to run, only to have the room. ## 2. The engine -llama.cpp's `ggml` sources are pinned by this repository: CMake fetches the revision it needs. For an offline build, point +llama.cpp's `ggml` sources are pinned by this repository, and CMake fetches the revision it needs. For an offline build, point `STRATA_GGML_DIR` at a checkout of exactly that revision. ```sh @@ -62,50 +82,52 @@ cmake -S . -B build-halo -G Ninja -DCMAKE_BUILD_TYPE=Release \ cmake --build build-halo --target strata -j 8 # about 25 minutes the first time ``` -`gfx1151` is "unvalidated" in the build's own wording (it warns). `STRATA_PREFILL_MMQ=ON` builds ggml's MMQ kernels the prompt -experts need. The tests: `cmake --build build-halo -j 8` builds them all, `ctest --test-dir build-halo` runs them; -`hip_prefill_hcd_exact_parity` and `hip_prefill_hipblaslt_gemm` need the table (next section) and skip without it. +The build warns that `gfx1151` is "unvalidated". `STRATA_PREFILL_MMQ=ON` builds ggml's MMQ kernels the prompt experts need. To test: +`cmake --build build-halo -j 8` builds the tests and `ctest --test-dir build-halo` runs them. `hip_prefill_hcd_exact_parity` and +`hip_prefill_hipblaslt_gemm` need the table (next section) and skip without it. ## 3. The prompt table, and running -Hand the engine the table (`setup` does it for the supported cards; for gfx1151 you do it): without it the prompt's dense matrix -products use plain hipBLAS and are slower. +A packaged engine finds its table itself. For a hand-built engine, hand it the table, because without it the prompt's dense matrix +products use plain hipBLAS and are slower: ```sh export STRATA_HIPBLASLT_TUNING=$PWD/tools/hip/gfx1151-hipblaslt-100401.txt ``` -The engine accepts it only for the architecture and the hipBLASLt version named on the table's second line +The engine accepts a table only for the architecture and hipBLASLt version named on its second line (`STRATA_HIPBLASLT_TUNING_V1 gfx1151 100401`: ROCm 7.14.1) and says so at start-up (`prefill gemm: hipBLASLt tuning enabled (90 rows, -gfx1151, version 100401)`). Then run `strata` with the same arguments as on any other card (the model packs, `--serve`, the -settings of [DETAILS.md](DETAILS.md)); a pack of the Unsloth UD-IQ4_XS or UD-Q4_K_XL GGUFs is what the numbers below used. +gfx1151, version 100401)`). Then run `strata` with the same arguments as on any other card (the model packs, `--serve`, the settings +of [DETAILS.md](DETAILS.md)). The numbers below used packs of the Unsloth UD-IQ4_XS or UD-Q4_K_XL GGUFs. ## 4. What is on by default on gfx1151 -At start the engine looks at GPU 0; on a gfx1151 it sets the switches below unless you set them yourself -(`STRATA_GFX1151_DEFAULTS=0` turns the whole table off, and the start-up line names what was set). **Every one of them gave -byte-identical output on the maintainers' box** (the same token ids, 4K / 32K / 64K and four models, and a bitwise harness per kernel): -they change speed, not answers. Nothing here applies to CUDA or to another AMD architecture. +At start the engine looks at GPU 0. On a gfx1151 it sets the switches below unless you set them yourself. +`STRATA_GFX1151_DEFAULTS=0` turns the whole table off, and the start-up line names what was set. Every one of them gave +byte-identical output on the maintainers' box (same token ids at 4K / 32K / 64K and on four models, plus a bitwise harness per +kernel): they change speed, not answers. Nothing here applies to CUDA or to another AMD architecture. | Switch | What it does | -|---|---| -| `STRATA_GDN_HEAD`, `STRATA_GDN_PP=2`, `STRATA_GDN_CONVL2`, `STRATA_GDN_NOY` | the prompt's gated-delta-net recurrence: four lanes per column, two columns per lane, the convolution and the norms fused | -| `STRATA_HCD_EXACT` | the hyper-connection down projection (N 320, K 10240) in a WMMA kernel that reproduces hipBLASLt solution 1176 / 1177's summation order (`hip_prefill_hcd_exact_parity` checks it bit for bit); it runs only where the table makes hipBLASLt pick one of those two, and says once if it cannot | -| `STRATA_CVEC_FUSE`, `STRATA_PF_PAD` | a steered layer's write, control vector and next norm in one pass; padded GEMM row strides | -| `STRATA_TOPK_REG_MIN_BLOCKS=4096` | the prompt's QSA top-k uses the register kernel from 16K cells of context instead of 28K (the original kernel is slower there; the selected cells are identical, `qsa_topk_parity` checks it) | -| `STRATA_Q8_PACKED`, `STRATA_Q6_PACKED`, `STRATA_MMVF_ROWS`, `STRATA_ATTN_LANECELL` | decode layouts and kernels (packed Q8_0 / Q6_K weights, 4-row BF16 GEMV, one-cell-per-thread attention scores) | -| `STRATA_EXPERT_V2`, `STRATA_TSUM`, `STRATA_LFUSE` | the grouped decode experts (IQ3_S / IQ4_NL), single-butterfly warp sums, fewer launches around the shared expert and the KV append | -| `STRATA_GDN_SPLIT`, `STRATA_QFUSE`, `STRATA_PLE_BATCH` | the GDN step over four blocks per head, activation images written by their producers, the PLE key / value projections of a verify window at once | -| `STRATA_SH_STREAM=1` | the shared expert on a second stream beside the routed experts. It is off by default on other AMD cards (#816), but it pays on gfx1151: decode +1.8% (UD-IQ4_XS) and +6.7% (UD-Q4_K_XL) at 8K | +| --- | --- | +| `STRATA_GDN_HEAD`, `STRATA_GDN_PP=2`, `STRATA_GDN_CONVL2`, `STRATA_GDN_NOY` | The prompt's gated-delta-net recurrence: four lanes per column, two columns per lane, convolution and norms fused. | +| `STRATA_HCD_EXACT` | The hyper-connection down projection (N 320, K 10240) in a WMMA kernel that reproduces hipBLASLt solution 1176 / 1177's summation order (`hip_prefill_hcd_exact_parity` checks it bit for bit). It runs only where the table makes hipBLASLt pick one of those two, and says once if it cannot. | +| `STRATA_CVEC_FUSE`, `STRATA_PF_PAD` | A steered layer's write, control vector and next norm in one pass; padded GEMM row strides. | +| `STRATA_TOPK_REG_MIN_BLOCKS=4096` | The prompt's QSA top-k uses the register kernel from 16K cells of context instead of 28K. The selected cells are identical (`qsa_topk_parity`). | +| `STRATA_Q8_PACKED`, `STRATA_Q6_PACKED`, `STRATA_MMVF_ROWS`, `STRATA_ATTN_LANECELL` | Decode layouts and kernels: packed Q8_0 / Q6_K weights, 4-row BF16 GEMV, one-cell-per-thread attention scores. | +| `STRATA_EXPERT_V2`, `STRATA_TSUM`, `STRATA_LFUSE` | The grouped decode experts (IQ3_S / IQ4_NL), single-butterfly warp sums, fewer launches around the shared expert and the KV append. | +| `STRATA_GDN_SPLIT`, `STRATA_QFUSE`, `STRATA_PLE_BATCH` | The GDN step over four blocks per head, activation images written by their producers, the PLE key / value projections of a verify window at once. | +| `STRATA_SH_STREAM=1` | The shared expert on a second stream beside the routed experts. Off by default on other AMD cards, but it pays on gfx1151 (decode +1.8% UD-IQ4_XS, +6.7% UD-Q4_K_XL at 8K). | + +Two further defaults change bits. `STRATA_HC_Q8` reads the hyper-connection weights as 8-bit (`STRATA_HC_Q8=0` restores the BF16 +read), and on Windows the prompt is read in chunks of 8192 tokens (`STRATA_GFX1151_CHUNK=0` keeps 6144). About `STRATA_HC_Q8`: it +halves the hyper-connection weight bytes (13.29 -> 7.15 MB per read) and gave +8.4% decode on `json_extract`, at the cost of answers +that are not bit-identical to the BF16 read. How far they move: [AMD_HIP.md](AMD_HIP.md) ("Strix Halo: 8-bit hyper-connection weights"). ## 5. What is not on by default (it changes bits) -**Five of the switches below (`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA`, `STRATA_SELECT_WMMA`) are one opt-in on an integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"). It was measured on a Radeon 780M only, where it is the default.** `STRATA_PF_FUSED`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well. `STRATA_HC_Q8` is on by default on gfx1151 (below); `STRATA_HC_Q8=0` turns it off. - -These are faster on gfx1151 and **round differently** from the default path (a different summation order or a half-precision -intermediate), so they are opt-in and the maintainers gate them with KL against the default (`STRATA_PF_SWITCH_MIN_T=4096` keeps -the short prompts on the default numerics, since these pay on long ones). The maintainers' fast configuration sets all of -them: +These are faster on gfx1151 and round differently from the default path (a different summation order or a half-precision +intermediate), so they are opt-in. The maintainers gate them with KL against the default. `STRATA_PF_SWITCH_MIN_T=4096` keeps +short prompts on the default numerics, since these pay on long ones. The maintainers' fast configuration sets all of them: ```sh export STRATA_PF_FUSED=1 # the prompt experts on the matrix cores (IQ2 / IQ3 / IQ4_XS gate/up, Q2_0 / IQ4_NL / Q8_0 down) @@ -115,64 +137,64 @@ export STRATA_PA_FAST=1 # the prompt attention with single FP16 q and p export STRATA_HIP_WMMA=1 # the prompt attention on matrix cores export STRATA_SELECT_WMMA=1 # the block scorer on matrix cores export STRATA_PF_SWITCH_MIN_T=4096 -export STRATA_PREFILL_STREAM_MIN=128 # prompt reads of 128-1,023 tokens on the fused experts too (below) +export STRATA_PREFILL_STREAM_MIN=128 # prompt reads of 128-1,023 tokens on the fused experts too ``` -`STRATA_HC_Q8` (on by default on gfx1151 since 0.2.2; `STRATA_HC_Q8=0` restores the bf16 read) measured on the Strix Halo (IQ3_S, gfx1151, 2026-10-21, MTP 4 drafts / min-p 0.5): **+8.4%** -on `json_extract` at fixed windows (53.55 -> 58.03 tok/s), at the cost of **94 of 250 output tokens differing** from -the BF16 read (first at 60). It halves the hyper-connection weight bytes (13.29 -> 7.15 MB per read; cold read -93.8 -> 20.3 us, 142 -> 352 GB/s). The answers are not bit-identical to the bf16 read; how far they move is in docs/AMD_HIP.md ("Strix Halo: 8-bit hyper-connection weights"). +`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA` and `STRATA_SELECT_WMMA` together are one opt-in on an +integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"), measured on a Radeon 780M only, where +it is the default. `STRATA_PF_FUSED`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well. -`STRATA_PF_FUSED=1` reaches only chunks of `STRATA_PREFILL_STREAM_MIN` tokens or more (1,024 by default, the floor measured on -a discrete card, where a smaller chunk does not pay for streaming every expert); a smaller read runs its experts through MMQ. -With every expert in the unified memory nothing streams, and MMQ is the slow part: an agent's turn - a tool result of ~1,000 -tokens on a cached conversation - read in 3.1-6.0 s at 1,024 and 0.9-2.5 s at 128 (C2T8, `--no-prefill-borrow`; 12-turn -replay 3.4 -> 2.1 s per turn). Its output is the fused path's, which reads of 1,024 tokens or more already take; it is not -under `STRATA_PF_SWITCH_MIN_T`, which gates only the hyper-connection and padding switches. A UD-Q4_K_XL or other K-quant -pack takes the fused experts only with `STRATA_PF_FUSED_KQ=1` as well. - -`STRATA_PF_F16=1` (gfx11 only; add it to the list above) runs the covered prompt experts on the F16 matrix cores instead of -the int8 ones: K-quant gate/up and Q5_1 down (with `STRATA_PF_FUSED_KQ=1`), IQ2 / IQ3 / IQ4_XS gate/up, Q2_0 / IQ4_NL / Q8_0 -down. One layer, 16,384 tokens: Q4_K / Q5_1 85.3 -> 53.5 ms, IQ3_S / IQ4_NL gate/up 11.65 -> 9.87 and down 6.11 -> 4.71 ms. -It rounds differently from the int8 kernels. First-token KL against a run with no switches, 12 natural-text prompts of -1.5K-24K tokens (Strata's own docs and sources), top-1 the same in 12 of 12: UD-Q4_K_XL mean 0.0068 (median 0.0021) against -0.0031 (0.0024) for `STRATA_PF_FUSED=1 STRATA_PF_FUSED_KQ=1` without it; UD-IQ4_XS 0.0085 (0.0036) against 0.0068 (0.0032). One -1.5K prompt carries most of the Q4_K_XL mean (0.049). - -`STRATA_REFILL_PARTS=1` (opt-in, exact; HIP and CUDA, a GGUF read in place) refills the expert-cache slots a prompt borrowed straight from the GGUF mapping (an expert's gate, up and down tensors) instead of through an assembled 4 MB copy per expert: the same bytes in the slot (checked against the assembled blob on 162 slots), 0.1 to 0.3 s less at the end of a prompt; gfx1151, UD-Q4_K_XL and UD-IQ4_XS: a 4K-8K prompt +2.3% to +3.5%, 32K +1.1%. It is not in the table above because only gfx1151 has measured it. - -Two engine flags help on long contexts and are not Strix-specific: `--mtp-window 8192` (the draft layer attends to the last 8,192 -cells: 64K output +3.5%, 32K -0.4%) and `--spec`, `--lookup-chain`, `--mtp-q4` (see [DETAILS.md](DETAILS.md)). +| Switch | What it does | +| --- | --- | +| `STRATA_PREFILL_STREAM_MIN=N` | `STRATA_PF_FUSED=1` reaches only chunks of N tokens or more (1,024 by default, the floor measured on a discrete card). With every expert in unified memory nothing streams and MMQ is the slow part, so a smaller N (128) speeds up an agent's turn (a tool result of about 1,000 tokens on a cached conversation: 3.1 to 6.0 s at 1,024 against 0.9 to 2.5 s at 128). | +| `STRATA_PF_FUSED_KQ=1` | A UD-Q4_K_XL or other K-quant pack takes the fused experts only with this as well. | +| `STRATA_PF_F16=1` (gfx11 only) | Runs the covered prompt experts on the F16 matrix cores instead of the int8 ones. It rounds differently from the int8 kernels. | +| `STRATA_REFILL_PARTS=1` (exact; HIP and CUDA, a GGUF read in place) | Refills the expert-cache slots a prompt borrowed straight from the GGUF mapping instead of through an assembled 4 MB copy per expert. Only gfx1151 has measured it. | +| `--mtp-window 8192` | Not Strix-specific: the draft layer attends to the last 8,192 cells. | +| `--spec`, `--lookup-chain`, `--mtp-q4` | See [DETAILS.md](DETAILS.md). | ## 5b. The desktop, the RAM, and a discrete card beside the Strix Halo (Linux) -The Strix Halo chip usually draws your desktop, and its graphics memory is your RAM. Two things follow, and the engine now takes both into account. +The Strix Halo chip usually draws your desktop, and its graphics memory is your RAM. The engine takes both into account. -- **One memory, counted once.** The expert cache of an integrated GPU lives in the same RAM as the expert arena. The automatic cache size is the smaller of - what the GPU reports free and what the OS reports available, less a reserve for the desktop (the larger of 8 GiB and 8% of the RAM; the start-up line says so - when it changes the size). `STRATA_UMA_RESERVE_GIB=N` sets the reserve (`0` = the old sizing, nothing kept back); `--expert-cache N` is never changed. -- **A card of its own beside the chip** (for example a Radeon AI PRO R9700). A card of its own is the recommended Main card and the chip stays Off in the card roles. - If you name the chip in a layer split anyway (`--gpus 0,1`, or the config's `"gpu": [0, 1]`), the split runs as asked, the chip is placed on the first layers and - never on the last stage (the output head, the draft layer and the check of its guesses), the picture reader is started on the card of its own, and the - start prints one plain line about it. `"gpu_order": "as_given"` and a manual `layer_split` keep your order. `setup --gpus all` leaves the chip out - when two cards of their own are present; name it by number to use it. -- **The CPU.** On Linux with built-in Radeon graphics the engine starts at nice +5, so the desktop keeps its share of the CPU while the engine's workers spin. +- **One memory, counted once.** The automatic expert cache size is the smaller of what the GPU reports free and what the OS reports + available, less a reserve for the desktop (the larger of 8 GiB and 8% of the RAM; the start-up line says so when it changes the + size). `STRATA_UMA_RESERVE_GIB=N` sets the reserve (`0` = nothing kept back). `--expert-cache N` is never changed. +- **A card of its own beside the chip** (for example a Radeon AI PRO R9700). That card is the recommended Main card and the chip + stays Off in the card roles. If you name the chip in a layer split anyway (`--gpus 0,1`, or the config's `"gpu": [0, 1]`), the + split runs as asked, the chip takes the first layers and never the last stage, the picture reader starts on the card of its own, + and the start prints one plain line about it. `"gpu_order": "as_given"` and a manual `layer_split` keep your order. + `setup --gpus all` leaves the chip out when two cards of their own are present; name it by number to use it. +- **The CPU.** On Linux with built-in Radeon graphics the engine starts at nice +5, so the desktop keeps its share of the CPU. `STRATA_ENGINE_NICE=0` turns it off, `STRATA_ENGINE_NICE=N` sets another value. -What is not known yet: how much of the screen freeze a long compute job on the same chip causes, and whether registering the whole expert arena with the driver -(the Linux default on gfx1151; Windows uses the unregistered arena since 0.2.1.2) hurts when the GTT is small. `STRATA_ARENA_PIN_GIB=none` runs the unregistered -arena on any OS; reports with `amdgpu_top` and `dmesg` are welcome. +Not known yet: how much of the screen freeze a long compute job on the same chip causes, and whether registering the whole expert +arena with the driver (the Linux default on gfx1151) hurts when the GTT is small. `STRATA_ARENA_PIN_GIB=none` runs the unregistered +arena on any OS. Reports with `amdgpu_top` and `dmesg` are welcome. ## 6. Measured -Ryzen AI Max+ 395 (128 GB, gfx1151), the maintainers' fast configuration of section 5 on top of the defaults of section 4, `--spec 4`, -`--mtp`, `--lookup-chain 3`, `--mtp-q4 all`, `--prefill 16384`, int8 KV, all experts in the unified memory. Prompt tok/s is -(prompt tokens - 1) / prompt time, output tok/s is the tokens / decode time; medians of 3 runs, one fresh process per run, the two -engines interleaved. "Before" is the maintainers' earlier engine branch (the same switches set by hand), "now" is this release. -Every pair printed the same token ids. +**Measured on:** a Strix Halo PC (gfx1151, Windows), IQ3_S model, Strata 0.2.3 (engine 0.1.48), 3 pairs against 0.2.2, from +[release-notes-0.2.3.md](release-notes-0.2.3.md). Answer speed +6% to +19% (tool calls +19%, code +18%, 64K prompt +13%, 8K prompt ++12%, 4K prompt +8%, JSON +6%, agent chat +9%). Prompt speed +5% to +14% (8K +14%, 64K +5%, 4K unchanged because it is one chunk). +Prompt speed at 4K, 8K and 64K is 685, 783 and 749 tokens per second. This is specific evidence, not validation of every APU or +quantization. + +The Linux numbers from the maintainers' Ryzen AI Max+ 395 (128 GB, Ubuntu 26.04, kernel 7.0, ROCm 7.14.1) are in the history +section below. + +## History / older measurements + + + +**Measured on:** Ryzen AI Max+ 395, 128 GB, gfx1151, Linux, Strata 0.1.40 code, the maintainers' fast configuration of section 5 on +top of the defaults of section 4, `--spec 4`, `--mtp`, `--lookup-chain 3`, `--mtp-q4 all`, `--prefill 16384`, int8 KV, all experts +in unified memory. Prompt tok/s is (prompt tokens - 1) / prompt time. Medians of 3 runs, one fresh process per run, two engines +interleaved, every pair printed the same token ids. "Before" is the maintainers' earlier engine branch with the same switches set +by hand. | Model | Context | Prompt before -> now | Output before -> now | -|---|---|---|---| +| --- | --- | --- | --- | | UD-IQ4_XS | 8K | 1,212 -> 1,293 (+6.7%) | 51.7 -> 53.8 (+4.0%) | | UD-IQ4_XS | 128K | 1,241 -> 1,320 (+6.4%) | 49.3 -> 51.4 (+4.2%) | | UD-Q4_K_XL | 8K | 1,136 -> 1,169 (+2.9%) | 50.6 -> 52.6 (+3.9%) | @@ -183,29 +205,33 @@ Every pair printed the same token ids. | IQ3_XXS | 128K | 1,259 -> 1,252 (-0.6%) | 48.4 -> 50.9 (+5.1%) | UD-IQ4_XS at 4K / 32K / 64K (5-6 interleaved pairs): prompt +9.2% / +6.0% / +5.3%, output +5.0% / +3.9% / +4.6%. -IQ3_XXS at 8K was measured again with 6 interleaved pairs (an earlier 3-run set had read 5% low while other jobs shared the GPU): prompt equal, output +5.6%. With the defaults off the same build's prompt is 8% lower (1,137), so the table helps there too. -### Re-measured on the merged 0.1.40 code - -The same box and method, no environment switches at all (the arch defaults only), on the final merged tree. 4 interleaved pairs per cell, -medians; "j" is the build of the table above (the Aurora port before the other 0.1.40 work was merged), "merged" is the release -code. Every run printed the same token ids, and the ids also match `strata-int29` (the v4.2 config) on UD-IQ4_XS and UD-Q4_K_XL at -a8 / 4K / 64K. +**Same box, merged 0.1.40 code, no environment switches (arch defaults only).** 4 interleaved pairs per cell, medians. "j" is the +build of the table above, "merged" is the 0.1.40 release code. Every run printed the same token ids. Prompt numbers at 8K swing by +more than the differences because other jobs shared the box, so read them as unchanged. | Model | Context | Prompt j -> merged | Output j -> merged | -|---|---|---|---| -| UD-IQ4_XS | 8K | 1,260 -> 1,159 (-8.0%, spread 1,070-1,320) | 53.94 -> 53.14 (-1.5%) | +| --- | --- | --- | --- | +| UD-IQ4_XS | 8K | 1,260 -> 1,159 | 53.94 -> 53.14 (-1.5%) | | UD-IQ4_XS | 128K | 1,317 -> 1,299 (-1.4%) | 51.38 -> 50.72 (-1.3%) | -| UD-Q4_K_XL | 8K | 1,163 -> 1,104 (-5.0%, spread 890-1,240) | 52.66 -> 50.47 (-4.2%) | +| UD-Q4_K_XL | 8K | 1,163 -> 1,104 | 52.66 -> 50.47 (-4.2%) | | UD-Q4_K_XL | 128K | 1,252 -> 1,211 (-3.3%) | 43.28 -> 42.64 (-1.5%) | | IQ3_S | 8K | 1,283 -> 1,265 (-1.4%) | 59.81 -> 59.10 (-1.2%) | | IQ3_S | 128K | 1,311 -> 1,310 (-0.1%) | 42.74 -> 42.09 (-1.5%) | | IQ3_XXS | 8K | 1,127 -> 1,212 (+7.6%) | 51.64 -> 50.95 (-1.3%) | | IQ3_XXS | 128K | 1,252 -> 1,250 (-0.1%) | 50.88 -> 50.22 (-1.3%) | -The prompt numbers at 8K swing by more than the differences (other jobs share this box), so read them as unchanged. The output -column moved by a steady -1.2% to -1.5% (-4% on UD-Q4_K_XL at 8K, where the default arm also had slow runs): since 0.1.39 the -shared-expert stream fork is off by default on HIP (#816). It overlaps the shared expert with the routed experts on a second -stream and does help on gfx1151, so 0.1.40 turns it on there by default (section 4). It changes no bits (same ids). 4 interleaved pairs at 8K, merged -code with the fork off -> on: UD-IQ4_XS output 53.1 -> 54.05 (+1.8%), UD-Q4_K_XL 49.5 -> 52.8 (+6.7%, the default arm ranged 46.0-52.1), -prompt unchanged. With the fork on, the merged code's output is at or above the j build's (IQ4_XS 54.05 vs 53.94). +With the shared-expert stream (`STRATA_SH_STREAM=1`, now a default) at 8K, 4 pairs: UD-IQ4_XS output 53.1 -> 54.05 (+1.8%), +UD-Q4_K_XL 49.5 -> 52.8 (+6.7%), prompt unchanged. + +**Other switch measurements** (Measured on: the same Ryzen AI Max+ 395) + +| Switch | Result | +| --- | --- | +| `STRATA_HC_Q8` (IQ3_S, 2026-10-21, MTP 4 drafts, min-p 0.5) | `json_extract` 53.55 -> 58.03 tok/s (+8.4%); 94 of 250 output tokens differ from the BF16 read (first at 60); cold read 93.8 -> 20.3 us (142 -> 352 GB/s) | +| `STRATA_PF_F16=1`, one layer, 16,384 tokens | Q4_K / Q5_1 85.3 -> 53.5 ms; IQ3_S / IQ4_NL gate/up 11.65 -> 9.87 ms and down 6.11 -> 4.71 ms | +| `STRATA_PF_F16=1`, first-token KL against no switches, 12 prompts of 1.5K-24K tokens, top-1 same in 12 of 12 | UD-Q4_K_XL mean 0.0068 (median 0.0021) against 0.0031 (0.0024) without it; UD-IQ4_XS 0.0085 (0.0036) against 0.0068 (0.0032) | +| `STRATA_REFILL_PARTS=1`, UD-Q4_K_XL and UD-IQ4_XS | 4K-8K prompt +2.3% to +3.5%, 32K +1.1% | +| `--mtp-window 8192` | 64K output +3.5%, 32K -0.4% | +| 12-turn agent replay, `STRATA_PREFILL_STREAM_MIN=128`, C2T8, `--no-prefill-borrow` | 3.4 -> 2.1 s per turn | +| Defaults off, IQ3_XXS 8K | prompt 8% lower (1,137) | diff --git a/docs/TERMINAL_LAUNCH.md b/docs/TERMINAL_LAUNCH.md index dfb790ea9..3d5834d9d 100644 --- a/docs/TERMINAL_LAUNCH.md +++ b/docs/TERMINAL_LAUNCH.md @@ -1,116 +1,231 @@ -# `strata launch`: set up, serve and watch, in the terminal +# `strata launch`: set up, serve and watch -One command for the whole path: choose a model, tune it, start serving, and watch it. It is the same questions as the one-click setup (`START-HERE.bat` / `setup.sh`), -in the Terminal's look, with presets, an Advanced table, an access choice and a Connect view for your coding apps. Text previews of every screen are in +This page is for anyone who uses the installed Terminal. One command chooses a model, tunes it, starts serving and +opens a monitor. It also connects your coding apps. Text previews of the screens are in [docs/terminal-previews/](terminal-previews/) (`launch-*.txt`). +## Quick start + +1. Install the Terminal with the one-line command in [INSTALL.md](INSTALL.md#the-terminal). +2. Run `strata launch` and follow the screens, or run a command without screens: + +```sh +strata launch --plan --family qwen --model q2_0 # what would happen; nothing changes +strata launch --family qwen --model q2_0 --yes --no-tui # download, serve, print the address, return +strata launch --status # what is serving now +strata launch --stop +``` + +`strata launch --help` lists every flag with examples. An unknown model or flag gets a plain message with the closest match. + ## Install (one line) +Linux and Mac: + ```sh -curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh # Linux, macOS -irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex # Windows PowerShell +curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh ``` -It finds out which computer this is, downloads the Terminal archive named in the release folder's `install-manifest.txt`, checks its SHA-256, unpacks it into your -home (`~/.local/share/strata/terminal`, `%LOCALAPPDATA%\Strata\terminal`; no administrator rights), checks the release's signed feed with the Terminal it just unpacked -(`strata update --check`; if that fails it removes what it unpacked), puts `strata` on your PATH and prints `Installed. Run: strata launch`. The engine (the part that talks -to your graphics card) comes with the first model: `strata launch` fetches the one for your card from the same release folder, signed, and test-starts it before using it. -The release folder is the latest GitHub release of Niko1221/Strata (`https://github.com/Niko1221/Strata/releases/latest/download`). The script only fetches and checks; read it first if you like (it is the same file as `install.sh` / `install.ps1` in the repository's `packaging` folder and in the release). Set `STRATA_FEED` to another folder to try a local one; `packaging/make_local_feed.py` builds a test-signed one. How far to trust a piped installer is in [SECURITY.md](../SECURITY.md). +Windows PowerShell (then open a new terminal): -## The screens (`strata launch`) +```powershell +irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex +``` -1. **Start**: saved presets (name, model, context, parallel requests, tuned or not, last used) or "Set up from the beginning" or "Use model files I already have". -2. **Model**: the catalog of the app and `setup.py` (one source), with download size, what it needs (graphics memory + RAM) and "fits / tight / does not fit this computer". - A warning, never a block. The n-gram table stays on your SSD and is not counted. `o` = your own files (a folder or a .gguf; checked with the fingerprint table). - *(Cards: only with two or more usable graphics cards. Each card gets a role: M Main, H Helper, O Off, R back to the recommendation, D details. `--gpu-roles 1=main,0=helper` does the same from the command line; see [MULTI_GPU.md](MULTI_GPU.md).)* -3. **Folder**: where model files are stored, free space shown. For a model that is already installed (also one the desktop app installed) it starts where the model is, says - "Already downloaded" and lists only what is really missing with its size (the draft layer, about 5.2 GB; the picture reader, 0.9 GB). It never says "download the model" when only - a helper file is missing. +More in [INSTALL.md](INSTALL.md#the-terminal). The installer needs no administrator rights and puts `strata` on your PATH. +How far to trust a piped installer is in [SECURITY.md](../SECURITY.md). The engine (the part that talks to your graphics +card) comes with the first model: `strata launch` fetches the one for your card from the same release, checks its signature +and test-starts it before using it. + +## The screens + +1. **Start:** saved presets (name, model, context, parallel requests, tuned or not, last used), "Set up from the beginning" + or "Use model files I already have". +2. **Model:** the catalog the app and `setup.py` share, with download size, what it needs (graphics memory and RAM) and + "fits / tight / does not fit this computer". This is a warning, never a block. The n-gram table stays on your SSD and is + not counted. `o` picks your own files (a folder or a `.gguf`). +3. **Folder:** where model files are stored, with free space. For a model that is already installed, including one the + desktop app installed, it starts where the model is and lists only what is missing (the draft layer, about 5.2 GB; the + picture reader, 0.9 GB). 4. **Context** (8K to 512K, with the extra memory each costs) and the memory format (`m`: 8-bit or 4-bit). -5. **Pictures**: only for models that can read them. They are **on by default** (Yes is preselected; `--vision off` or "No" turns them off). It says what it adds (about 0.9 GB: - the picture-reader file, downloaded with the model through setup's Control mode, the same path the app's "Read pictures" uses). The reader runs on the CPU - for a card under 24 GB (a few seconds per picture, nothing taken from the card) and on the graphics card from 24 GB (about 1-1.4 GB kept free), on NVIDIA and AMD cards; the - rule lives in one place, `strata_control/vision_reader.py`, for setup, `strata launch` and the app's catalog. `--vision gpu` / `--vision cpu` (or the Pictures step) switch it per model. It needs a picture reader beside the engine in the engine package; with an - engine that has none, pictures default to off and the screen says so in plain words. A model that cannot read pictures stays off with no question. -6. **Parallel requests**: "fit to your card" (12 GB: 1, 20: 2, 24: 3, 32+: 4) or 1 to 4; a warning if it will not fit. -7. **Tuning**: quick (about 20 minutes) / thorough (up to 2 hours) / skip (`strata tune auto`; the "answers may change very slightly" tick is off). The screens suggest quick the first - time (it is done once and kept); the command line skips tuning unless you add `--tune`. -8. **Advanced** (optional): every engine setting with default, tuned value and your value; edit, reset, add a setting by name (checked against the engine's own list; a warning, not a refusal); - the exact command (it includes the context, memory format and picture choice you made above; it has its own scroll area: Tab, then the arrows) and the matching `strata launch` flags. -9. **Access**: this computer only (127.0.0.1), your network (LAN address, key), Cloudflare quick tunnel (needs `cloudflared`), Tailscale (needs it installed and connected); port; API key - (optional, recommended beyond this computer: Strata makes one for you, and one keypress (`t`) turns it off. Without a key the server still starts and warns that anyone who can reach the port can use the model and its tools; `--no-api-key` does the same without the screens, and on a Cloudflare tunnel the warning is stronger. This computer only never needs a key). -10. **Save as preset** and launch. Then downloads (with progress), engine start, and the monitor. In a small window the summary shrinks so the three choices stay on screen, - and the key line says what Enter will do. No web page opens by itself; only `--open` opens one. +5. **Pictures:** only for models that can read them. They are on by default (`--vision off` or "No" turns them off). The + reader adds about 0.9 GB. It runs on the CPU for a card under 24 GB and on the graphics card from 24 GB (about 1 to 1.4 GB kept + free). The rule is in `strata_control/vision_reader.py`. `--vision gpu` or `--vision cpu` overrides it per model. If the + engine package has no picture reader, pictures default to off and the screen says so. +6. **Parallel requests:** "fit to your card" (12 GB: 1, 20: 2, 24: 3, 32 and more: 4) or a number from 1 to 8, with a warning + if it will not fit. Check the effective slots after startup. +7. **Tuning:** quick (about 20 minutes), thorough (up to 2 hours) or skip. The screens suggest quick the first time; the + command line skips tuning unless you add `--tune`. +8. **Advanced** (optional): every engine setting with its default, tuned value and your value. You can edit, reset, or add a + setting by name (checked against the engine's own list, with a warning instead of a refusal). It also shows the exact command and + the matching `strata launch` flags. +9. **Access:** this computer only (127.0.0.1), your network (LAN address and key), a Cloudflare quick tunnel (needs `cloudflared`) + or Tailscale (needs it installed and connected). Also the port and an API key. Beyond this computer a key is recommended: + Strata makes one, and `t` turns it off. Without a key the server still starts and warns that anyone who can reach the port + can use the model and its tools; `--no-api-key` does the same without screens, and the warning is stronger on a Cloudflare tunnel. +10. **Save as preset** and launch. Downloads show progress, then the engine starts and the monitor opens. No web page opens + unless you pass `--open`. -Keys: arrows choose, Enter goes on, Esc goes back, `?` explains the screen, `q` leaves. Every screen says in plain words what it means and what to do next. +Keys: arrows choose, Enter goes on, Esc goes back, `?` explains the screen, `q` leaves. + +With two or more usable graphics cards the Model step also asks for a role per card (M main, H helper, O off). On the command +line, `--gpu-roles 1=main,0=helper` does the same ([MULTI_GPU.md](MULTI_GPU.md)). + +## Flags + +```sh +strata launch --preset "IQ3_XXS 128K" # a saved preset (opens the monitor) +strata launch --family qwen --model iq3_xxs --context 128k --parallel 4 --vision off --tune quick \ + --access lan --port 8080 --save-preset "Office" --no-tui +strata launch --model-path /data/models/IQ3_XXS --no-tui # files you already have +strata launch --preset Office --set prefill=65536 --set STRATA_PIN_BY_NEED=1 --no-tui +strata launch --family qwen --model q2_0 --context 64k --restart --no-tui +strata launch --list-presets +strata launch --show-key +``` + +| Flag | What it does | +| --- | --- | +| `--preset NAME` | Start a saved preset. Other flags change that start only; add `--save-preset` to keep them. | +| `--family`, `--model` | The model family and size, for example `qwen`, `q2_0`. Any case. | +| `--model-path PATH` | Use model files you already have: a folder of shards or one `.gguf`. | +| `--data-dir DIR`, `--move-data` | Where new downloads go. `--move-data` also moves the models you already have. | +| `--context SIZE` | `32k`, `64k`, `128k`, `256k` or a number of tokens. | +| `--kv FORMAT` | Conversation memory format: `int8` (default), `q4_0` (half the memory), `k8v4`, `fp16`. | +| `--vision off\|on\|gpu\|cpu` | Picture reading. Default: on when the model and the engine can. | +| `--parallel N` | Requests answered at the same time, 1 to 8. Default: fit to your card. | +| `--mtp on\|off` | The draft layer. `off` saves about 1 GB of graphics memory and is slower. Default: on. | +| `--drafting auto\|mtp\|dflash2\|off` | Which drafter guesses ahead. `dflash2` is for Qwen3.8-Flash-Next only and downloads a 1.2 GB drafter once. If it cannot run, MTP is used and the start says why. | +| `--gpu LIST`, `--gpu-roles LIST` | Which cards to use, and a role per card. | +| `--tune quick\|thorough\|skip`, `--tune-allow-small`, `--retune` | Tuning. `--retune` measures again; a model tuned before for the same choices on this computer reuses the stored result. | +| `--set KEY=VALUE` | An advanced engine setting or `STRATA_*` variable (repeatable). Unknown names get a warning. | +| `--pool-cores LIST` | Experimental: pin CPU workers to these logical processors, for example `2-14:2` ([DETAILS.md](DETAILS.md)). | +| `--access local\|lan\|cloudflare\|tailscale`, `--port N` | Who can reach the server. | +| `--api-key KEY`, `--api-key-env NAME`, `--new-key`, `--no-api-key` | The key clients must send. | +| `--save-preset NAME`, `--list-presets`, `--delete-preset NAME` | Presets. | +| `--no-tui` | No screens: serve, print the address and return. | +| `--yes` | No questions. It never restarts a running model. | +| `--restart` | Stop a model that is serving other choices (or holds the port), then start these. | +| `--open` | Open the server's web page once it serves. | +| `--plan` | Show what would happen and exit. It lists the model, context, picture setting and what would download (0 when everything is here). | +| `--feed URL\|DIR` | The signed update feed. Default: `STRATA_UPDATE_FEED` or the project's. | +| `--output human\|json` | Output format. | +| `--status`, `--stop`, `--show-key` | Show, stop, or print the key of the running launch (add `--preset NAME` to pick one). | + +Presets are small files in `launch-presets/` beside the profiles folder. A run uses the shared profile `launch-` and the shared +tuning presets (`strata presets`), so the app sees them too. + +**Launching while a model is serving.** The same choices restart nothing and print the running addresses. Different choices +(another context, pictures, access, or another model on the same port) make the screens ask once, "Restart the model with these +settings? Answers in progress stop." With `--no-tui` the command refuses with one line naming the running model, its port and settings. Add +`--restart` or run `strata launch --stop` first. A different `--port` runs beside the first server; two big models may not fit in graphics memory together, and the launch says so. + +## DeepSeek V4.1-Flash + +DeepSeek V4.1-Flash runs in the Terminal and in the app, on NVIDIA RTX 30, 40 and 50 (Windows and Linux) and on the Radeon AI PRO R9700 (Linux). Hardware, sizes and speeds are in [MODELS.md](MODELS.md#deepseek-v41-flash). + +```sh +strata launch --plan --family deepseek41 --model ist-das-2.00bpw +strata launch --family deepseek41 --model ist-das-2.00bpw +``` + +| Model name | Download | +| --- | ---: | +| `ist-das-1.75bpw` | 239 GB | +| `ist-das-2.00bpw` | 257 GB | +| `ist-das-2.25bpw` | 274 GB | +| `ud-q8_k_xl` | 508 GB | + +- The context is 8192 tokens (8K), the size the engine was checked at. +- `--speed faster|full` picks how many of the model's 6 experts it uses per word. Faster uses 5 (a small quality change, usually not noticeable) and is the default on a PC with 16 GB of graphics memory or less and on PCs whose graphics share the computer's memory. Full uses all 6. `--full-quality` is the same as `--speed full`. Both need DeepSeek engine 0.1.50 or newer, which `strata launch` fetches. The engine's own flag is `--experts N`. +- It answers one request at a time. `--parallel` above 1 is ignored with a note. +- It cannot read pictures, so pictures are off and `--vision on` is ignored with a note. +- `--tune` runs DeepSeek's own short tuner (drafter on or off, CPU threads), not the Qwen tuner. +- `--model-path DIR` starts from the shards you already have. +- `--plan` shows where the model goes on your cards and how much is held in RAM. ## After the start: the monitor -The Strata Monitor with the new **Overview** (key 1: model state bar, speed, GPU load, VRAM, temperature, power, PCIe, CPU, disk, context fill, experts in VRAM, system RAM, recent requests; read from the -server's `/metrics`, the same data as the browser Monitor), the other five views as before, and **7 Connect** (the seventh view: the arrow keys reach it like the others, and every view's hint says 1-7): the address, the key (hidden until `k`), and the setup for each app -(`c` copy, `w` write it for you after a backup, `h` check it: one prompt and one tool call sent the way that app sends them). The list keeps two things apart: **Vendor tested** (the app -version its maker tested the setup with: history) and **This server** (Not checked / Passed / Failed for your server, after `h`). `q` asks: keep serving in the background, or stop -(stopping unloads the model and frees the graphics memory). +The monitor has seven views. Press `1` to `7` or use the arrow keys. -In the classic Windows console (the Windows 10 default window) the monitor first switches that window's font to Cascadia Mono when it is installed (the console font API: no admin, nothing saved, gone with the window) and draws the full look; otherwise it draws a solid-block look: █ bars with a thin ▄ track, ▄ / █ sparklines, square boxes and the 16 palette colours (`•` instead of `●` when the font is not Consolas). It always shows one line: "For the full look, use Windows Terminal (built into Windows 11)". Windows Terminal, Linux and macOS are never touched. `STRATA_GLYPHS=unicode|ascii|solid` forces a look. With two or more graphics cards the Overview tiles show the busiest card (load), all cards (memory), the hottest card and the total power, a "Cards" strip lists each card (up to 4, then "press 5" for the Hardware view), and the CPU tile says "2 CPUs · 64 cores" from the server's new `sockets` figure. +| Key | View | +| --- | --- | +| 1 | Overview: model state, speed, GPU load, VRAM, temperature, power, PCIe, CPU, disk, context fill, experts in VRAM, system RAM, recent requests. It reads the server's `/metrics`, like the browser Monitor. | +| 2 to 6 | Serving, Requests, Engine, Hardware, Logs | +| 7 | Connect: the address, the key (hidden until `k`) and the setup for each app | -## Without the screens (flags) +In Connect, `c` copies a setup, `w` writes it after a backup and `h` checks it by sending one prompt and one tool call the way that +app sends them. The list keeps two columns apart. **Vendor tested** is the app version its maker tested the setup with. +**This server** is Not checked, Passed or Failed for your server after `h`. -```sh -strata launch --preset "IQ3_XXS 128K" # a saved preset (opens the monitor) -strata launch --model iq3_xxs --context 128k --parallel 4 --vision off --tune quick \ - --access lan --port 8080 --save-preset "Office" --no-tui # serve and print the address -strata launch --model-path /data/models/IQ3_XXS --no-tui # files you already have -strata launch --preset Office --set prefill=65536 --set STRATA_PIN_BY_NEED=1 --no-tui -strata launch --model q2_0 --pool-cores 2-14:2 --no-tui # experimental: pin the CPU workers to these logical processors (see DETAILS.md) -strata launch --plan --model q2_0 # what would happen; nothing changes -strata launch --model q2_0 --yes --no-tui # no questions (for one: download on a computer with no engine yet) -strata launch --model q2_0 --open # also open the server's page in your browser once it serves -strata launch --model q2_0 --context 64k --restart --no-tui # a model is already serving other choices (or holds the port): stop it and start these -strata launch --list-presets | --delete-preset NAME | --status | --stop [--preset NAME] | --show-key -``` +`q` asks whether to keep serving in the background or stop. Stopping unloads the model and frees graphics memory. -`strata launch --help` lists everything with examples. An unknown model or flag gets a plain message with the closest match. Presets are small files in `launch-presets/` beside the profiles -folder; the run uses the shared profile `launch-` and the shared tuning presets (`strata presets`), so the app sees them too. Flags next to `--preset` change that start only -(add `--save-preset` to keep them). `--plan` shows the model (also for your own files, which it recognises), the context, the picture setting and what would be downloaded and how big (0 when everything is already here). A model that was tuned before for the same choices on the same computer is not measured again by `--tune quick`; the stored result is used and the run says so. Add `--retune` to measure again (the model of the same launch is stopped for the measuring and starts again with the result). +With two or more cards the Overview shows the busiest card's load, all cards' memory, the hottest card and the total power, and a +"Cards" strip lists up to 4 cards (`5` opens Hardware). The CPU tile shows sockets and cores. -**A launch while a model is already serving.** The same choices: nothing restarts, the running addresses are printed. Different choices (another context, pictures, access, or another model on the same port): the screens ask once, "Restart the model with these settings? Answers in progress stop.", with Restart and Cancel; `--no-tui` refuses with one line naming the running model, its port and settings, and says to add `--restart` or run `strata launch --stop` first. `--yes` alone never restarts. With `--restart` the old server is stopped first and the new one starts on the same port. A different port given with `--port` runs beside the first one; two big models may not fit in the graphics memory together, and the launch says so. +**Windows console.** In the classic Windows 10 console the monitor switches that window's font to Cascadia Mono when it is installed +and draws the full look. Otherwise it draws a solid-block look and shows one line suggesting Windows Terminal. Nothing is saved and +Windows Terminal, Linux and macOS are never touched. `STRATA_GLYPHS=unicode|ascii|solid` forces a look. ## Connect your apps: `strata connect` -`strata connect` lists OpenCode, Codex CLI, Claude Code, oh-my-pi (omp), pi, Hermes Agent, OpenClaw and the DeepSeek Harness (built into the app). `strata connect opencode` prints the file and -environment for the running server (address, key, model name and context filled in); `--write` writes it after asking and keeps a backup of anything it replaces; `--check` sends one prompt and one tool call -the way that app does and says whether each worked. With `--output json` (`strata connect opencode --check --output json`) it prints one JSON object with the same file and environment fields plus `checks` (one entry per check: `name`, `ok`, `detail`, `seconds`) and `checked_server`; `ok` is true only when both checks passed. The exit code is 0 when both worked and 1 when one failed or the server could not be reached; in that case the object also carries an `error` that says which of the two it was. Each configuration was tested for real against a Strata server at 32K context (the app versions are on screen); the known limits are printed in plain -words (for example: Claude Code needs `CLAUDE_CODE_MAX_CONTEXT_TOKENS`; OpenCode needs `limit.input`; Hermes refuses windows under 64K unless set up as the config does; OpenClaw needs Node 24.16+). +`strata connect` lists OpenCode, Codex CLI, Claude Code, oh-my-pi (`omp`), pi, Hermes Agent, OpenClaw and the DeepSeek Harness +(built into the app). -## Boxes the installer cannot set up +```sh +strata connect opencode # print the file and environment, with address, key, model and context filled in +strata connect opencode --write # write it after asking; anything replaced is backed up +strata connect opencode --check # send one prompt and one tool call the way that app does +strata connect opencode --check --output json +``` -`setup.py` sets up NVIDIA (CUDA) and AMD (HIP) cards. For an Intel Arc card its Linux path is a container build that the Terminal does not run: if the model's files and their prepared pack are -already on the disk, `strata launch` writes the run settings itself and says so; otherwise it says which file is missing. Nothing is downloaded or compiled in that case. +| Flag | What it does | +| --- | --- | +| `--url URL`, `--key KEY`, `--key-env NAME` | The server address and key. Default: the running launch, else `http://127.0.0.1:8080`. | +| `--model NAME`, `--context SIZE` | The model name the server reports and the context it started with. Default: the running launch's, else 32k. | +| `--preset NAME` | Take address, key and model from a saved preset. | +| `--yes`, `--show-key` | Do not ask before `--write`; print the real key instead of a masked one. | + +With `--output json` the object has the file and environment fields, `checks` (one entry per check with `name`, `ok`, `detail`, +`seconds`) and `checked_server`. `ok` is true only when both checks passed. The exit code is 0 when both worked and 1 otherwise; +then `error` says which check failed or that the server could not be reached. + +Each configuration was tested against a Strata server at 32K context. Known limits are printed in plain words, for example: +Claude Code needs `CLAUDE_CODE_MAX_CONTEXT_TOKENS`; OpenCode needs `limit.input`; Hermes refuses windows under 64K unless set up +as the config does; OpenClaw needs Node 24.16 or newer. ## Models you already have -`strata models list` looks in the same places the screens and `strata launch` look (one function, `strata_control/discovery.py`): the folder you name with `--install-dir` or -STRATA_INSTALL_DIR (then only that one), the Terminal's own folder, the folders earlier installs remembered in settings.json, and the desktop app's install folder, which the Terminal -shares (Windows `%LOCALAPPDATA%\Strata\install`; Linux and macOS `$XDG_DATA_HOME/strata/install`, by default `~/.local/share/strata/install`). The data folder is the one you set, else the -app's (`app-settings.json` beside its install folder, else a `Documents/Strata` that holds models), else Strata's default. `strata models import PATH --plan` knows the Gyro-S and Gyro-M files -(TQK6/TQK7 and TQ2_T experts) and every file set setup installs, also under another name when the header matches, and says which run config, prepared pack, draft layer and picture reader -already exist for them. Nothing is copied or moved. +`strata models list` looks in these places (one function, `strata_control/discovery.py`): + +- the folder you name with `--install-dir` or `STRATA_INSTALL_DIR` (then only that one) +- the Terminal's own folder +- folders earlier installs remembered in `settings.json` +- the desktop app's install folder, which the Terminal shares: `%LOCALAPPDATA%\Strata\install` on Windows, + `$XDG_DATA_HOME/strata/install` (normally `~/.local/share/strata/install`) on Linux and macOS + +The data folder is the one you set, else the app's (`app-settings.json` beside its install folder, else a `Documents/Strata` that holds +models), else Strata's default. + +`strata models import PATH --plan` recognises the Gyro-S and Gyro-M files (TQK6/TQK7 and TQ2_T experts) and every file set setup +installs, also under another name when the header matches. It says which run config, prepared pack, draft layer and picture reader +already exist. Nothing is copied or moved. ## When there is no engine for this computer -`strata launch` does not refuse. It says, in plain words, "Strata couldn't find an engine for this computer (the reason). The model will download, but it may not run until you install the right -engine (next step: ...)", asks whether to go on (Enter means yes; `--yes` skips the question; nothing to download means no question), downloads the model, and ends with a message that names the -engine problem and the next step. The exit code is 0 and nothing is started. +`strata launch` does not refuse. It says "Strata couldn't find an engine for this computer" with the reason, asks whether to go on +(Enter means yes; `--yes` skips the question), downloads the model, and ends with a message that names the problem and the next step. +The exit code is 0 and nothing is started. -## Cards of different makes +## Intel Arc and cards of different makes -One engine drives the cards of one make (CUDA: NVIDIA, HIP: AMD). Only those cards are offered for running and splitting; a model is never split across makes, and two models at the same -time, one on each make, is not supported yet. When the other make has the bigger card you get one line: "This computer also has an AMD R9700 (32 GB). The AMD build of Strata can use it - it fits bigger models." +- **Intel Arc:** the Terminal downloads the released Linux SYCL engine, but its model preparation for Arc is a container build that it does not + run. If the model's files and prepared pack are already on disk, `strata launch` writes the run settings itself. Otherwise it reports the preparation problem. Use + [Intel setup](INTEL_ARC.md) for the preparation route. +- **Mixed makes:** one engine drives the cards of one make (CUDA: NVIDIA, HIP: AMD). Only those cards are offered, and a model is never + split across makes. Two models at once, one per make, is not supported. If the other make has the bigger card, you get one line saying so. ## Limits -* The model download uses `setup.py` in its control mode (no pip, no engine fetch, no compiler), as the desktop app does; it needs internet and room for the files. -* `strata update`'s test start of an engine runs ` --version`; the SYCL (Intel) package's engine does not answer that flag (exit 2), so on an Intel box the engine package has to be copied into the - components folder until that is fixed. -* The Windows console has no way for the test scripts to drive the screens; they were checked by pilot tests and by hand. +- The model download uses `setup.py` in its control mode (no pip, no engine fetch, no compiler), as the desktop app does. It needs internet and room for the files. +- The Linux SYCL package updates through `strata update`. Do not copy it into the components folder by hand. diff --git a/docs/TERMINAL_QUICKSTART.md b/docs/TERMINAL_QUICKSTART.md index a11f67f43..555f02dc4 100644 --- a/docs/TERMINAL_QUICKSTART.md +++ b/docs/TERMINAL_QUICKSTART.md @@ -1,8 +1,11 @@ # Terminal quick start -New here? `strata launch` sets up a model, serves it and opens the monitor in one guided flow: [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md). +This page is for people who already have a model and want to read a running server, rerun it from a saved profile, or measure it. +To set up and serve a new model, use [`strata launch`](TERMINAL_LAUNCH.md). -These commands use the `strata` command that the one-line Terminal install puts on your PATH ([how to install](TERMINAL_LAUNCH.md#install-one-line)). The Terminal carries its own Python, so there is nothing else to install. In a source checkout, type `python strata.py` (or `python3 strata.py`) in place of `strata`, from the repository root. The commands below read, prepare and rerun a model you already have; they do not download one. The complete options, permissions and limits are in [TERMINAL.md](TERMINAL.md). +The commands use the `strata` command from the [Terminal install](INSTALL.md#the-terminal), which carries its own Python. +In a source checkout, type `python strata.py` in place of `strata`, from the repository root. These commands read, prepare +and rerun a model you already have; they do not download one. All options, permissions and limits are in [TERMINAL.md](TERMINAL.md). ## Read a running server @@ -18,7 +21,7 @@ The interactive monitor comes with the installed Terminal. Open the one-screen O strata monitor --url http://127.0.0.1:8080 ``` -The Overview puts serving state, prompt and answer speed, batch/queue pressure, four selected GPU rows, requests and activity on one screen. Press `1`–`6` for detail (`1`–`7` in the monitor that `strata launch` opens, where `7` is the Connect view), `Space` to freeze the display, `r` to refresh, and `q` to detach. Detach leaves the server running. On a machine without a terminal, use `status --json` or `monitor --once --json`. +The Overview puts serving state, prompt and answer speed, batch/queue pressure, four selected GPU rows, requests and activity on one screen. Press `1`–`7` for the views (`7` is Connect), `Space` to freeze the display, `r` to refresh, and `q` to detach. Detach leaves the server running. On a machine without a terminal, use `status --json` or `monitor --once --json`. For a remote server, use HTTPS or a local SSH tunnel. A saved connection stores only the credential environment variable's **name**: diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index df94d8bb0..aa7c4d42f 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -1,279 +1,142 @@ -# Something went wrong? +# Troubleshooting -The common problems and what to do, for the desktop app, the Terminal (`strata launch`) and the installer. Back to the [README](../README.md#something-went-wrong). The full table of -error messages, with the older engine fixes, is in the [details](DETAILS.md#troubleshooting). +This page is for anyone whose install or startup fails. Find your symptom below. Each entry gives one fix. +Start from the route you installed: [desktop app or Terminal](INSTALL.md), or [source checkout](SOURCE_INSTALL.md). +The normal release uses the app or `strata launch` and needs no `.bat` file. Detailed engine errors are in the +[engine reference](DETAILS.md#troubleshooting). + +**Quick check.** Run these and keep the output for a bug report: + +```sh +strata --version +strata doctor --live +strata launch --status +``` ## The desktop app and the Terminal -**A warning says Strata "couldn't find an engine for this computer".** -The engine is the part that talks to your graphics card. Strata only warns and still downloads the -model, and the model files stay on your computer, but the model may not run until the right engine is installed. The usual -reasons are that no engine is installed yet and the update source has none for your card, that the engine found is for -another make of card, or that the graphics driver is too old. In the app, open Settings, then Engine. In the Terminal, the -message names the next step: run `strata update` and start again. If your computer has no NVIDIA or AMD graphics card -(only an Intel UHD or Iris chip, an AMD processor graphics chip, a virtual machine, or Windows on ARM), Strata says so and -there is nothing to install: it needs a computer that has one. The model files stay where they are. +**I cannot find `START-HERE.bat` or `SETUP.bat`.** +Open the installed Strata app, or run `strata launch`. Those wrappers are in the source repository and are not the +startup path for the app. For a fresh desktop install, download the one installer for your system +(`Strata-0.2.5-windows-x64-setup.exe`, `Strata-0.2.5-linux-x86_64.AppImage` or `Strata-0.2.5-macos-arm64.dmg`), not a +`strata-engine-…` archive. -**"This model needs a newer engine." (the app), or a button called Update engine.** -Some models need an engine newer than the one installed: Gyro needs 0.1.45.1 or newer, NVFP4 needs 0.1.44, and NVFP4 (NVIDIA, -trained for FP4) needs 0.1.45. In the app, open Settings, then Engine, and press Update engine. It asks first, keeps your -model files and settings, and wants running models unloaded. In the Terminal: `strata update --check` says what would -change, `strata update` installs it (it switches to the new engine only after a test start). If the engine you -installed is the newest and the message stays, the update source has no newer engine for your card yet. +**`strata` is not found.** +On Windows, open a new PowerShell window after installing. On Linux or Mac, use the full path the installer printed, +normally `~/.local/bin/strata`, and add `~/.local/bin` to PATH if needed. Check `strata --version`. -**On a Mac: "Strata is damaged and can't be opened", "cannot be opened", or "unidentified developer".** -Nothing is wrong with the file. macOS marks everything a browser downloads, and only lets a marked program run if Apple -has checked it. Strata's release builds are checked by Apple ("notarised"): drag Strata onto Applications and open it; the -first time, macOS asks "Are you sure you want to open it?" and you press Open. A test build that is not notarised says -"damaged". Do not move it to the Trash. Open the Terminal app and remove the mark, with the line that matches what you have -(drag the file from Finder into the Terminal window to type its path): +**Strata could not find an engine for this computer.** +Run `strata doctor --live` and `strata update --check`. Check the selected GPU, driver and +[backend guide](INSTALL.md#what-you-need). Supported paths include Apple Silicon and Intel Arc on Linux; do not +assume every machine without an NVIDIA/AMD discrete card is unsupported. A model download can finish even when +no engine starts. Verify `strata launch --status` and `/health` before calling the installation successful. - xattr -dr com.apple.quarantine ~/Downloads/ # the .dmg, before opening it - xattr -dr com.apple.quarantine /Applications/Strata.app # the app, after dragging it over - xattr -dr com.apple.quarantine # an unzipped Terminal folder +**This model needs a newer engine.** +Unload running models, then use the app's engine update or `strata update`. Minimum engine versions vary by model and +backend. Check the model guide and the installed engine, not only the app version. If the feed has no compatible engine, +report the missing combination. -Then open it again. If macOS says only "the developer cannot be verified", you can instead try to open it once, then open -System Settings > Privacy & Security and press "Open Anyway" next to the message about Strata. Installs made with -`curl ... | sh` carry no mark and need none of this. To see which kind you have: `spctl -a -vv /Applications/Strata.app` -answers "accepted" and "Notarized Developer ID" for a notarised build. (How the builds are signed: `packaging/mac/README.md`.) +**An update says it succeeded but `strata --version` is still old.** +The Terminal switches to the updated Control on the next command, so run `strata --version` again. If it is still old, +run the one-line installer once more; it keeps models and settings. The app, Control and engine have separate version +numbers; see [updating](INSTALL.md#updating) and the [release notes](https://github.com/Niko1221/Strata/releases). -**Checking what is installed: `strata doctor`.** -It is a read-only check of the installed engine parts (it changes nothing): which engine and helper files a saved model -needs, and whether they are there. `strata doctor --live` also asks the computer which graphics cards it sees and compares -them with the cards the model is set to use; `--output json` prints the same as data. Put its output into a bug report. +**A new engine misbehaves after an update.** +Stop the model, then use the app's rollback control or `strata components rollback`. Source-checkout engines use a +separate rollback workflow, `python setup.py --rollback-engine`, described in [source setup](SOURCE_INSTALL.md). -**Sending logs for a bug report.** -In the app: Settings, Engine, Diagnostics, View logs, then Save logs. It writes one text file, -`strata-diagnostics-.txt`, into your Downloads folder and shows where. It holds the engine's messages, the failed -steps and the end of each model's server log. In the Terminal, send the output of `strata doctor`. If the Linux app window -crashes or will not open, run the AppImage with `--save-logs` in a terminal instead; it writes the crash log and the other -logs to one file in Downloads. If the Windows app closes right after you start it, it shows a message with the reason and writes -`desktop-crash-.log` in `%LOCALAPPDATA%\Strata\logs`; Save logs includes it, and so can you, by hand. Look through a log before you send it, and never send an API key. +**The Mac app cannot be opened.** +Check that the download is for Apple Silicon and that macOS is 15 or newer. Follow +[Mac installation](MAC.md#the-desktop-app-and-strata-launch-on-a-mac). A warning alone does not establish that a file +is safe or intact: verify the download source and release before changing quarantine settings. Release signing and +test-build handling are described in [Mac packaging](../packaging/mac/README.md). + +**The Linux AppImage does not open.** +Make the downloaded file executable. It may need FUSE 2 (`libfuse2` on Ubuntu 22.04, `libfuse2t64` on newer Ubuntu; +package names vary by distribution). Without FUSE, run the file with `--appimage-extract-and-run`. For a crash log, +run it with `--save-logs`; see the logs section below. ## While installing or starting -**My PC froze, or got very slow, the first time Strata started.** -That's normal while it starts, most of all the first time. Strata loads 35-55 GB into your RAM, locks part of it for -the graphics card, and works out how much of the model fits on your GPU. The mouse can freeze for a few minutes. -**Wait, and don't close the window.** The next starts are much faster. Still frozen after 10 minutes? Restart the -PC, close other programs (browsers use a lot of RAM) and try again. If it keeps happening, pick a smaller size (Q2_0 -or IQ2_XS). +**A download stopped.** +Retry it in the app, or rerun the same `strata launch` command. Completed model-download steps are reused. Check +free space and the error message. For an interrupted Terminal installer, rerun the install command. Only a source +checkout uses `START-HERE.bat` / `./setup.sh` to resume source setup. -**The engine stops while it starts, on a small graphics card or a PC with little RAM. Can I turn the draft layer (MTP) off?** -Yes. The draft layer is what makes answers faster, and it takes about 1 GB of graphics memory (plus some RAM and a memory -copy that grows with the context). Without it the expert cache gets that memory back and answers are slower. Turn it off -for one model: in the app, Models, Expert settings, "Draft layer (MTP)": Off; in the Terminal, -`strata launch --model q2_0 --mtp off` (older Terminals: `--set mtp=off --set spec-min-p=off`); by hand, add `"mtp": false` to the -model's `strata-.json`. The choice is kept per model and applied on every start. +**Startup takes a long time or the computer becomes slow.** +Large models need time to check files, prepare packs and fill memory. A first launch can take many minutes; there is +no fixed startup time. Watch progress and the engine log. If memory pressure is high, close memory-heavy programs and try a +smaller model or context. DeepSeek V4.1-Flash needs 46 GB of RAM at least ([model page](MODELS.md#deepseek-v41-flash)); +`strata launch --plan --family deepseek41 --model ist-das-2.00bpw` shows what your computer is missing. -**I changed `strata-.json` by hand and the change came back.** -Engine options you add to `args` by hand (control vectors, `--spec`, `--vram-reserve-mib 0`, anything the app does not offer) -are kept. The file has two helper keys for this. `user_args` lists your options; every start and every setup run puts them -over the ones the app generates, and for the same option your value wins. `args_written` is a copy of `args` as the app last -wrote it, so an option you added or changed in `args` is noticed and moved into `user_args` the next time the app or setup -touches the file. To undo one, delete it from both `args` and `user_args`. A value you save in the app's Expert settings -("GPU memory for other apps") is put in `user_args` for you. +**The NVIDIA driver is too old.** +Use the driver required by the engine package. CUDA 13 needs driver 580 or newer; older cards use the +[CUDA 12 route](OLDER_GPUS.md). After updating the driver and restarting if needed, open the app or rerun +`strata launch`. -Two things the app still chooses itself: the context size, KV type, picture reading and draft layer. Set them in the Models -screen, or with `"mtp": false` (draft layer). The app keeps the earlier file as `strata-.json.bak`. +**Port 8080 is already in use, or I changed a running model's settings.** +Check `strata launch --status`. To replace your existing launch, rerun the intended command with `--restart`; +it stops current answers. To run beside it, choose another port, for example `--port 8081`, if memory permits. +Do not stop an unrelated program just because it holds the port. `--yes` alone does not restart a running model. -**The app keeps more graphics memory free than I want (decode is slower than in 0.1.45).** -Without a value of yours the app keeps 1536 MiB free (2048 with picture reading) on a card that drives a display, and 512 MiB -(700 with pictures) on a card that has no display attached, such as a second GPU with the screen on the integrated one. It -reads that from the driver (nvidia-smi `display_active`, or the Windows adapter that has the desktop); when it cannot tell, it -assumes a display. Your own value is never raised or lowered: set "GPU memory for other apps" in Expert settings, or -`--vram-reserve-mib 0` in `args`. The engine's own warning about low free memory stays; a card that also draws the desktop -can freeze or slow to a few tokens a second with a very small reserve. +**The engine stops on a small GPU or low-memory computer.** +Try a smaller pack or context. The draft layer also uses memory. In the app, set the model's Draft layer to Off; +in the Terminal, an explicit example is: -**It stopped while downloading or installing.** -Run `START-HERE.bat` (Linux: `./setup.sh`) again. It continues where it stopped. +```sh +strata launch --family qwen --model q2_0 --context 32k --mtp off +``` -**It says the NVIDIA driver is too old.** -Update it (NVIDIA App or [nvidia.com/drivers](https://www.nvidia.com/drivers)), restart the PC, and run -`START-HERE.bat` again. +Use `--restart` if replacing an active run. Turning MTP off can slow decoding. Keep some VRAM free for the display; +setting the reserve to zero is not a general speed fix. -**It says port 8080 is already in use.** -Strata is already running. Look for its window. Or another program uses the port: `START-HERE.bat --port 8081`. -Only an address that is really taken says "already in use". Any other reason (Windows keeps the port reserved, or -the config's `"host"` is not an address of this PC) is printed as what the OS said, with what to change (#769). +**Pictures do not work.** +The model, backend and installed engine package must include a compatible picture reader. Inspect the startup +error and the model guide. `--vision off` disables reading pictures; it is different from optional +[image generation](IMAGES.md). In the Terminal, use `on|off|gpu|cpu`, not source setup's `yes|no` spelling. -**"Windows blocked the Strata engine (or the image encoder) ... Smart App Control".** -On a clean Windows 11, Smart App Control refuses programs that are not signed, and Strata's `strata.exe` and -`strata-vision.exe` are not (#735). Turn it off (Windows Security > App & browser control > Smart App Control -settings; it cannot be turned back on without a reset of Windows), or, if only the image encoder is blocked, run -`START-HERE.bat --setup --vision no`. - -**A new engine misbehaves after an update.** -An update keeps the engine it replaced in `engine\.previous` (one generation, about 210 MiB). -`python setup.py --rollback-engine` puts it back (and keeps the newer one there: run it again to go forward) (#670). - -**UPDATE.bat / update.sh say "git pull did not succeed", or "has no commit in common" (#1276).** -The repository's history was cleaned up on 2026-10-06, so a clone made before that cannot be updated with `git pull`. -The 0.1.40.2 scripts handle it: when no tracked file is edited they keep your old commits in the branch -`pre-cleanup-backup` and move the clone to the new history; otherwise they print the two commands for you. Your models, -settings and engine are untracked files and are never touched. An older UPDATE.bat / update.sh (before 0.1.40.2) cannot -do this itself: run these once in the Strata folder, then use UPDATE.bat as usual (add `git stash` first if `git status` -lists edited files): - - git fetch origin - git branch pre-cleanup-backup - git checkout -B main origin/main - -**Python or the build tools could not be installed.** -Install what it names (links are printed), then run it again. Everything already done is kept. - -**Linux: the engine does not compile (`unsupported GNU version`, or `exception specification is incompatible` for -`cospi`/`sinpi`/`rsqrt`).** Two known mismatches between the CUDA toolkit and a new Linux (#601): -- gcc newer than 14 (Ubuntu 26.04's default 15): CUDA 12.x and 13.0 refuse it. Install g++-14 and run - `CXX=g++-14 ./setup.sh` (CC, CXX and CUDAHOSTCXX name the build's compilers; setup passes them to CMake itself, - so an already-configured build folder picks them up too, #1645). -- glibc 2.43 with CUDA 12.9: the toolkit's math headers clash with glibc's, already in CMake's first test compile. - Use CUDA 12.8 or 13.x instead. Setup takes the newest toolkit it finds; `STRATA_NVCC=/usr/local/cuda-12.8/bin/nvcc - ./setup.sh` makes it use that one (only that one). - -**The first start takes minutes.** -It is reading 34-55 GB into RAM; the second start is faster while the files are in the OS cache. Started from Task -Scheduler, it can be 24x slower: see [Running it at startup](DETAILS.md#running-it-at-startup-task-scheduler). -On Linux the engine now asks the kernel to read the model files ahead (a cold start went from ~920 s to 70 s on one -PC); `STRATA_READ_AHEAD=0` turns that off. On Linux with transparent huge pages on `always`, the arena no longer asks -for `MADV_HUGEPAGE` on top (a fragmented machine spent minutes compacting memory, #771); `STRATA_NO_ARENA_THP=1` skips -that request on any setting. - -**Setup says the engine does not match GitHub's checksum.** -Setup checks the downloaded ready-made engine against the SHA-256 GitHub publishes. On a mismatch the file is corrupt or was changed on the way, so setup deletes it and stops; run setup again to download it afresh. If GitHub gives no checksum (offline, rate limited, your own mirror), setup warns and installs anyway. `STRATA_SKIP_SHA256=1` skips the check, only if you insist. +**Windows blocks an executable.** +Verify the official release and the component named in the message. Feed signatures and Windows code-signing are +different checks. If only the picture reader is blocked, `--vision off` may let text requests run. Do not disable +system-wide protection as the routine installation fix; include the blocked file and exact message in a bug report. ## While it answers -**It's very slow and the disk light keeps blinking.** -Your PC is out of free RAM. Close other programs, or pick a smaller size (Q2_0 or IQ2_XS). +**It is slow and the disk stays busy.** +The chosen model may exceed available RAM/VRAM and be using the file tier. Close other programs, check memory in +the monitor, and choose a smaller model or context. Compare performance only with the same hardware and workload. -**An answer stopped with "the engine stopped unexpectedly".** -Usually not enough RAM (on Linux the system then stops the engine). Just send your message again: Strata starts the -engine by itself. If it keeps happening, close other programs or pick a smaller size. +**The prompt exceeds the context.** +Start a new chat, shorten the input, or use the app's Summarize and continue action. A larger context needs more +memory. In the Terminal, restart your chosen model with an explicit `--context`, such as `64k`; preserve its family, +model and other choices. Do not look for `SETUP.bat` in an app install. -**On Linux the engine stops with "exit code -9", but nothing ran out of RAM.** -Check `ulimit -R`. If it prints 0 (or any number), the desktop session sets a real-time CPU limit: the audio system (PipeWire) does that when rtkit is not installed, and every program started from the desktop inherits it. The kernel can then end a program with signal 9 and no message. Install `rtkit`, or start Strata outside the session's limits, for example `systemd-run --user --collect -p LimitRTTIME=infinity -- strata serve ...`. Strata prints a warning at start when it sees this limit. +**On Linux the engine exits with code -9.** +Check system logs for an out-of-memory kill. Also check `ulimit -R`: a desktop-session real-time limit can cause a +kill even when memory is available. The engine prints a warning for that condition; see the +[detailed troubleshooting table](DETAILS.md#troubleshooting) before changing system limits. -**It says the prompt exceeds the context.** -The conversation is longer than the context you chose. Start a new chat, or run `SETUP.bat` and pick more -context. +**A hand-edited engine option disappears.** +The run config tracks explicit options in `user_args` and the generated baseline in `args_written`. Prefer the +app's Expert settings or Terminal `--set` for supported overrides. Context, KV, picture reading and drafting are +launch choices; set them through the app or the corresponding `strata launch` flags. Inspect the effective config +rather than assuming a saved preference was applied. -**It is slower than the tables.** -The monitor plugged into the graphics card and other GPU programs take VRAM from the expert cache; RAM running below -its rated speed (enable EXPO/XMP in the BIOS) slows the CPU half. +## Source-checkout problems -**An earlier reply that came back empty is gone from the chat history (0.1.40, #843).** -The server leaves out finished assistant turns that have no text, so the next reply does not copy the empty ones. -`STRATA_KEEP_EMPTY_TURNS=1` in the server's environment renders them as before. +Python/venv, CUDA compiler versions, ROCm builds, `UPDATE.bat`, `update.sh` and generated `run-` scripts +belong to [SOURCE_INSTALL.md](SOURCE_INSTALL.md). Use the backend guide for build prerequisites. The packaged +Terminal carries Python; a missing system Python is not a reason to install a compiler for the normal release. -**A Turing card (RTX 20) reads prompts above ~90K tokens differently (0.1.40, #743).** -The prompt's top-k selection takes a wider kernel there, with the same ids. `STRATA_TOPK_STREAM=0` restores the old one. +If `git pull` fails in an old clone, back up your local work and follow the recovery steps the source update script prints. +Do not overwrite local branches or edits just to make `git pull` succeed. -**Decode is several times slower with `--expert-cache N` than with `auto` (#781, #831).** -An explicit N is a byte budget that is not checked against the free VRAM once the slots are written. On a card it -fills (the log says `0 MiB of VRAM free with everything loaded - LOW`), Windows pages the GPU's memory and decode drops -from 100 to 14 tok/s with only a few hundred slots too many. Use `--expert-cache auto`, or a smaller N. +## Logs and bug reports -**A low-RAM PC (about 32 GB) stalls with `--resident-budget-gib N` (#649).** -The budget is pinned (locked) in RAM, which leaves the OS and the CPU workers little room. `STRATA_RESIDENT_PIN=0` keeps -it pageable; a smaller budget works too. +- **App:** Settings → Engine → Diagnostics → View logs → Save logs. The diagnostic file goes to Downloads. +- **Terminal:** include `strata --version`, `strata doctor --live`, `strata launch --status`, and the relevant log excerpt. +- **Windows app crash:** inspect `%LOCALAPPDATA%\Strata\logs` for `desktop-crash-.log`. +- **Linux app crash:** start the AppImage with `--save-logs` to collect diagnostics. -**Decode got slower in 0.1.40 on a 32 GB PC with `--resident-budget-gib N`, and the drive is read hard (#1194, #1116, #1085).** -The start line "the file tier reads unbuffered" means the experts outside the RAM copy are read straight from the drive. -0.1.40 chose that whenever the free RAM could not hold all of them; but the same few come back token after token, and -a cache a twentieth of their size serves most of the repeats. 0.1.40.2 asks for that much (at least 1.5 GiB beside the -4 GiB headroom) and reads through the cache otherwise, which measured 7% to 118% faster decode with a third to a sixth of -the drive traffic on a 20 to 32 GB box. On 0.1.40 / 0.1.40.1: `STRATA_UNBUFFERED_LOAD=0` does the same, `=1` forces the -unbuffered reads. - -**Linux: the start is OOM-killed while the engine allocates the page-locked expert copy (#1250).** -Page-locked pages come from the driver in one go and cannot wait for the kernel to give back file cache, so with the -model files' cache in the way (little really free RAM, much "available") the host could run out. 0.1.40.2 gives back the -cached pages of the model files it mapped before it page-locks the copy; when the really free RAM still does not cover it, -it takes the pages in 1 GiB steps while the RAM available (cgroup limits included) stays above the headroom, page-locks -what it has, and keeps the rest resident but pageable with a warning ("only N of M GiB could be page-locked"): the answers -are the same, only the copies of the pageable part to the GPU go through the CPU. `STRATA_PIN_GUARD=0` turns this off, -`STRATA_PIN_RESERVE_GIB=N` sets the RAM that must stay free (default: `STRATA_RESIDENT_HEADROOM_GIB`, 4 GiB). - -**The Windows display driver resets, then the PC blue-screens (0x141, then 0x116 in `nvlddmkm`) during an answer (#961).** -This is the NVIDIA driver, not Strata: the report shows the same wedge with other CUDA programs on that driver (610.88, -RTX 4080 SUPER). Try another driver (a Studio one, or an older one) and a lower power limit; if it only happens with -`--spec 4`, `--spec 0` avoids the verify window while you wait for a driver fix. - -**An RTX 50 card (a source build with CUDA 13.2) answers with nonsense, or reads prompts wrongly (#892, #968).** -CUDA 13.2.0 and 13.2.1's compiler (nvcc 13.2.51) miscompile some of the engine's kernels for sm_120: on our RTX 5070 the IQ2_S and IQ3_S -products are wrong (relative error 0.5 to 1.0 in the tests) with 13.2 and right with 13.0. The ready-made engine is built -with 13.0. If you compile it yourself, use CUDA 13.0, 13.1 or 13.2.2 (13.2.2, nvcc build 13.2.86, fixes it; an older one can sit next to a newer: `STRATA_NVCC=`); -setup warns when it finds 13.2.0 or 13.2.1 for such a card, and takes an older 13.x when one is installed. - -**Pictures are refused, or slow.** -"this server was started without the vision encoder": the model was set up for text only (someone chose "No", or the -engine has no picture reader) - run setup again with `--vision yes` (or `--vision gpu` / `--vision cpu`), or switch pictures -on in the app or on the Pictures step of `strata launch`. Pictures are on by default wherever a model can read them. The -reader runs on the CPU under 24 GB of graphics memory and on the card from 24 GB, on NVIDIA and AMD cards. Pictures that take -several seconds (about 3 s at 300 image tokens on 8 cores, more with more tokens) are read by the encoder on the CPU; -`--vision gpu` (NVIDIA and AMD, ~1.4 GB of VRAM) makes it 0.1-0.5 s. On AMD the card's reader is `strata-vision-hip`; if it -cannot start, the server prints "the GPU picture reader did not start (...); pictures are read on the CPU" and goes on with -the CPU reader. On a Mac the reader runs on the GPU; `--vision cpu` takes about 50 s per picture there. On Intel Arc, pictures are off by default for now (`--vision yes` turns them on). - -## AMD cards - -**"No AMD GPU found (the amdgpu driver's KFD topology is empty)" on Linux.** -The kernel's amdgpu driver is not loaded for the card. Integrated Radeon GPUs are listed as not supported; setup -lists every card it found and whether Strata can use it. - -**The engine stops at start with the card's name, its architecture and the build's list.** -The engine was compiled for another card (for example after moving the Strata folder to another PC). Run -`./setup.sh --setup --backend hip`: it compiles the engine for this card's architecture. - -**On Windows the engine exits with `0xC0000005` in `amdhip64_7.dll` before it prints anything (#654).** -Two things in your environment can cause it, and the server now repairs both before it starts the engine (it says so in -the log): `HIP_PATH`, `HIP_DEVICE_LIB_PATH` or `LLVM_PATH` naming a ROCm folder that no longer exists (a deleted build), -and a `TEMP`/`TMP` folder the engine cannot create files in (AMD's runtime compiles its first kernels through temporary -files). If you start `strata.exe` by hand, fix them yourself: `set HIP_PATH=`, and point `TEMP` and `TMP` at a normal -folder such as `C:\Temp`. - -**Windows AMD: the driver resets (VIDEO_ENGINE_TIMEOUT_DETECTED / screen flicker / the engine dies mid-answer).** -Windows gives a GPU about 2 seconds to answer; when a graphics card does not, it resets the driver (a TDR): the screen -flickers, the engine dies, and the Event Viewer shows `VIDEO_ENGINE_TIMEOUT_DETECTED` (#613, with #579 and #541 on -Linux). Strata 0.1.39 and newer say "the GPU stopped responding" in the log. It was seen with KV streaming on -(`--kv-resident 32768`, which setup turns on from 64K context) while reading a long prompt on a gfx1201 card. Try these -in order, one at a time, and tell us what changed: -1. **Send us the evidence.** The last ~80 lines of `strata-.log` in the Strata folder (the server window shows - the same lines), your AMD driver version (AMD Software > System), your card and Windows version, the Strata - version, and the exact steps (context size, how long the prompt was, which app sent it). If Windows wrote a - dump, the newest `.dmp` in `C:\Windows\LiveKernelReports\WATCHDOG` (or `C:\Windows\Minidump`) helps; it holds no - chats, only the driver state. -2. **Find the step that hangs:** add `"STRATA_PF_STEP_SYNC": "1"` to the `"env"` block of `strata-.json` - and restart. The prompt path then waits for the GPU after each step and logs any step over 250 ms, so the log names - the step. It is slower; use it to diagnose, then take it out. -3. **`"STRATA_KV_HOST_DMA": "1"`** in the same `"env"` block: no GPU kernel writes the RAM copy of the K/V during a - prompt, it is copied by DMA instead. Same answers. -4. **No KV streaming:** run setup again and pick 32K context (or remove `--kv-resident` and its number from - `"args"` in the JSON, with a context that fits the VRAM). -5. **A smaller prompt chunk:** change `--prefill` in `"args"` (for example `--prefill 512`): each GPU launch then does - less work between two checks by Windows. Prompts are read more slowly. -6. **Optional, your decision: give Windows more time (`TdrDelay`).** The default is 2 seconds. In the registry, - `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers`, a DWORD `TdrDelay` of `10` (seconds) - lets a long GPU job finish before Windows resets the driver; restart the PC. It hides the symptom and does not fix - a GPU that has really hung, and a hung GPU then freezes the screen for those 10 seconds. Strata changes no Windows - setting itself; to undo it, delete the value. -7. **Driver:** use the newest AMD Software release for your card (Adrenalin, not a Windows Update driver) and restart - after installing. If it started after a driver update, tell us both versions; going back one version is a fair - test. - -**Large pinned host allocations fail on ROCm although RAM is free.** -See [AMD_HIP.md](AMD_HIP.md#model-and-serving-configuration): the mapped expert mode avoids the full pinned arena. - -**"verify: timed out at layer N" or "no progress for 60 s" on AMD (Linux), mostly with `--mmap-experts`.** -Two reports fixed it with `"GPU_PINNED_MIN_XFER_SIZE": "1048576"` (RX 6800) or `"HSA_USERPTR_FOR_PAGED_MEM": "0"` (two -R9700) in the `env` block of the server JSON, then a restart. Experimental, one machine each; on a systemd unit use -`MemoryMax`, not `MemoryHigh`. Details: [AMD_HIP.md](AMD_HIP.md#linux-verify-timeouts-while-the-kernel-reclaims-host-memory-experimental-workarounds). - -## Still stuck? - -Look in the [full troubleshooting table](DETAILS.md#troubleshooting), or open an -[issue](https://github.com/Niko1221/Strata/issues) and attach the log: in the app, the file Save logs writes (see -[above](#the-desktop-app-and-the-terminal)); in the Terminal, the output of `strata doctor`; with the installer, -`strata-.log` from the Strata folder. Say which one you use (app, Terminal or the installer), your Strata and -engine versions and your card. +Include model and quantization, OS, GPU/VRAM, RAM, driver, install route, exact command, expected result and actual +error. Remove API keys and private data before sharing. Follow [CONTRIBUTING.md](../CONTRIBUTING.md). diff --git a/docs/UNSLOTH_Q4.md b/docs/UNSLOTH_Q4.md index eb833aae1..922ce264a 100644 --- a/docs/UNSLOTH_Q4.md +++ b/docs/UNSLOTH_Q4.md @@ -1,60 +1,52 @@ # Unsloth UD-Q4_K_XL -A regular size since MODEL-UPDATE Phase 2 (it was experimental): setup offers it from engine 0.1.32 -([below](#setup)); the manual workflow after that section works with engine 0.1.31 or newer. First validated on -Windows 10, an RTX 5070 (12 GB), 64 GB of RAM and an AVX-512 CPU (2026-10-01); since then on an R9700 (AMD, every -expert in RAM) and an RTX A4000 (62 GB, the RAM budget) - quality, speed and images in [QUANTS.md](QUANTS.md). +This page is for people who want to run Unsloth's 4-bit Qwen3.8-Flash-Next file (UD-Q4_K_XL) with Strata, and for +people who pack it by hand. It also covers UD-IQ4_XS and plain Q4_0 files. For UD-Q5_K_XL, UD-Q6_K_XL and Q8_0, see +[UNSLOTH_Q6.md](UNSLOTH_Q6.md). Quality, speed and images across the Unsloth sizes are in [QUANTS.md](QUANTS.md). The target is Unsloth's 4-bit quantization of the same model the other packs use: [unsloth/Qwen3.8-Flash-Next-GGUF, `UD-Q4_K_XL`, revision `38bb39e`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q4_K_XL). -"UD-Q4_K_XL" is a mix of formats, not Q4_K everywhere: the routed experts are Q4_K (Q5_K in layer 2) for gate/up and -Q5_1 (Q8_0 in layers 2, 4, 30, 46, 47) for down; the token embedding, the output head, the attention and shared-expert -projections and the PLE key are Q8_0; the 28.8 GB PLE table is IQ4_NL. The routed experts are 71.7 GiB, about twice -the 3-bit models'. +"UD-Q4_K_XL" is a mix of formats, not Q4_K everywhere: -## Setup +- Routed experts: Q4_K (Q5_K in layer 2) for gate/up, Q5_1 (Q8_0 in layers 2, 4, 30, 46, 47) for down. They are + 71.7 GiB, about twice the 3-bit models'. +- Token embedding, output head, attention and shared-expert projections, and the PLE key: Q8_0. +- The 28.8 GB PLE table: IQ4_NL. -`START-HERE.bat --setup` (Linux: `./setup.sh --setup`) and choose **Qwen3.8-Flash-Next (Unsloth)**; or directly: +## Quick start + +Run `strata launch` and choose **Qwen3.8-Flash-Next (Unsloth)**, or: ```sh -START-HERE.bat --setup --family unsloth --model UD-Q4_K_XL +strata launch --family unsloth --model UD-Q4_K_XL ``` -What setup does differently for this model: +## What setup does -- Every expert in RAM where its 72 GiB of experts and 16 GiB more fit the RAM (~88 GB: no `--resident-budget-gib`; - the rule is sergqwer's fork's, MIT), else the RAM budget below. It needs 48 GB of RAM or more (with less it asks, - default no; `--model UD-Q4_K_XL --yes` installs it anyway) and engine 0.1.32 or newer (checked before anything is - downloaded). NVIDIA or AMD: it ran on an R9700 too (#429's question is gone). One GPU by default in the budget mode - (the tested setup); on several, see the next point. No experimental speed projection (not tested with it). Images - (#967) are on by default on an NVIDIA card of 16 GB or more and on AMD (CPU encoder) where every expert fits in RAM; - run end to end on CUDA and HIP (QUANTS.md). -- Several GPUs, with an engine that keeps the resident RAM copy on a layer split (0.1.40 and newer, #642): the RAM - budget is kept on the split. Each card caches the experts of its own layers, and the budget holds the hottest of - the rest by the whole expert profile (the experts no card holds), so any RAM works as on one card. Setup asks (one - GPU stays the default; `--gpus 0,1`, "2" at setup or "y" when a start offers both cards takes the split), and a - start on several GPUs keeps `--resident-budget-gib`. Measured on an RTX 3090 (x16) + RTX 3080 Ti (x4), 91 GB of RAM, - budget 64 GiB, engine 0.1.41: decode 45 tok/s against 38 on the 3090 alone (the auto split gives the x4 card two - layers); 57.6 GiB of experts in RAM, no file reads per request. -- Several GPUs with an older engine (#498): when the RAM holds the GGUF files and 24 GB more (~135 GB of RAM) and two or more cards can - share it, setup asks (one GPU stays the default; `--gpus 0,1` takes the split). The split runs **without** the RAM budget: all 77 GB of experts are loaded into RAM from the GGUFs at - start, the files pass through the OS file cache while they load, and the config gets `"gpu": [0, 1]` and - `"layer_split": "auto"`. Measured on 2x RTX 3090 with 165 GiB (#498): decode 31 tok/s on one card with the budget, - 64-78 tok/s split (55 tok/s at a 128K prompt), with `MemAvailable` never under 68 GiB. With less RAM, `--gpus` - keeps one GPU and says so, and `START-HERE.bat --gpus 0,1` on an installed UD-Q4_K_XL stops with the reason - (it used to keep the budget, and the engine exited with code 2). -- It downloads the four shards below from the pinned revision `38bb39e` (resumable, like the other models), then - checks each one's size and SHA-256 against the table below; the check takes a few minutes once and is remembered - in the file's finish mark. A file with the wrong hash is deleted, so the next run downloads it again. -- It packs with `--compat-bf16` and never writes `experts.bin` (`--low-ram` does not apply). -- The RAM budget (below ~88 GB of RAM) is the PC's RAM less 24 GB: `--resident-budget-gib 40` on 64 GB, at most all 71 GiB of experts on - a PC that asks for a budget. From a 64K context up, where the KV cache moves to RAM, its size comes out of the budget. Setup's - `--resident-budget-gib N` sets another one (a bigger one is kept, with a note on what it risks). -- It recommends an 8K context on a GPU under 14 GB (every GB of KV cache is a GB less of cached experts). +- **Where the experts live.** Every expert goes into RAM when its 72 GiB plus 16 GiB more fit (about 88 GB of RAM). On a + PC with less RAM, or on a unified-memory APU, setup keeps a RAM budget: the hottest experts in RAM, the rest read + from the GGUF on the SSD. It needs 48 GB of RAM or more. With less it asks (default no), and + `--model UD-Q4_K_XL --yes` installs it anyway. +- **The RAM budget** is the PC's RAM less 24 GB (`--resident-budget-gib 40` on 64 GB), at most all 71 GiB of experts. From + a 64K context up, where the KV cache moves to RAM, its size comes out of the budget. `--resident-budget-gib N` at + setup sets another one; a bigger one is kept, with a note on what it risks. +- **Download.** The four shards below from the pinned revision `38bb39e`, resumable. Setup checks each one's size and + SHA-256 against the table below once and remembers it in the file's finish mark. A file with the wrong hash is + deleted, so the next run downloads it again. +- **Packing.** `--compat-bf16`, and never an `experts.bin` (`--low-ram` does not apply). +- **Defaults it writes** into the run config's `env`: `STRATA_HC_Q8=1` on CUDA (the GGUF's own Q8_0 hyper-connection + projections, closer to the file and faster), and `STRATA_ROUTE_TAIL_SKIP=0` where every expert is in RAM. UD-Q4_K_XL + keeps `STRATA_MMQ_KQ` opt-in. +- **Context.** It recommends 8K on a GPU under 14 GB, because every GB of KV cache is a GB less of cached experts. +- **GPUs.** NVIDIA or AMD. One GPU is the default in the budget mode. With several GPUs, each card caches the experts of + its own layers and the budget holds the hottest of the rest by the whole expert profile. Setup asks; `--gpus 0,1` + takes the split, and a start on several GPUs keeps `--resident-budget-gib`. +- **Pictures.** The reader is the CPU one below 24 GB of VRAM and the GPU one from 24 GB, when the package has a reader. + No experimental speed projection is used with this file. ## The files -Four shards, 111,334,654,784 bytes (111.3 GB) together. Shard 1 holds only the metadata; layer 11's down projection is +Four shards, 111,334,654,784 bytes (111.3 GB) together. Shard 1 holds only the metadata. Layer 11's down projection is in shard 2 and its gate/up in shard 3, which this engine handles (`native_experts.txt` v4). | File | Bytes | SHA-256 | @@ -64,23 +56,22 @@ in shard 2 and its gate/up in shard 3, which this engine handles (`native_expert | `Qwen3.8-Flash-Next-UD-Q4_K_XL-00003-of-00004.gguf` | 49,376,141,504 | `56758f40269cad5cd9b0d3d6fbae0f40f6d5be6de49e4ab392dbe83157d9cbd3` | | `Qwen3.8-Flash-Next-UD-Q4_K_XL-00004-of-00004.gguf` | 12,087,983,520 | `753bda48b98ba4f1636134a90a967de1b2d3908a236c026e464777342e53510a` | -Keep the four files together in one folder under these names (the engine finds shards 2-4 from shard 1's name; a -missing shard is an error that names it). Setup checks them itself; by hand, check them before packing -(`sha256sum -c`, or `Get-FileHash` on Windows). -Hugging Face snapshot symlinks work if you pass the snapshot's file name, not the hash-named blob it points to. +Keep the four files together in one folder under these names. The engine finds shards 2 to 4 from shard 1's name, and a +missing shard is an error that names it. By hand, check the files before packing (`sha256sum -c`, or `Get-FileHash` on +Windows). Hugging Face snapshot symlinks work if you pass the snapshot's file name, not the hash-named blob it points to. ## What it needs | | | | --- | --- | -| Disk | 111.3 GB for the four files, 1.4 GB for the pack, ~6 GB for the MTP draft layer if you have none yet. **No `experts.bin`**: the engine reads the experts from the GGUF files in place. An NVMe SSD matters: the experts that fit neither VRAM nor the RAM budget are read from it for every token. | -| RAM | 64 GB measured. The RAM budget (below) holds the most-used experts; the rest come from the SSD through the OS file cache. | -| GPU | 12 GB measured (RTX 5070): after the weights, the draft layer and the buffers, the expert cache held 1,280 of the 24,576 experts (3.7 GiB) at 4K context. | +| Disk | 111.3 GB for the four files, 1.4 GB for the pack, about 6 GB for the MTP draft layer if you have none yet. There is **no `experts.bin`**: the engine reads the experts from the GGUF files in place. An NVMe SSD matters, because experts that fit neither VRAM nor the RAM budget are read from it for every token. | +| RAM | 48 GB minimum; about 88 GB for every expert in RAM. The RAM budget holds the most-used experts and the rest come from the SSD through the OS file cache. | +| GPU | 12 GB works: after the weights, the draft layer and the buffers, the expert cache held 1,280 of the 24,576 experts (3.7 GiB) at 4K context on an RTX 5070. | ## The pack (by hand) -Build Strata (or install 0.1.31+), then, from the repository root, with gguf-py from the pinned llama.cpp -(setup installs it; `STRATA_GGUF_PY` can point to its `gguf-py` folder): +Build Strata, then from the repository root, with gguf-py from the pinned llama.cpp (setup installs it; +`STRATA_GGUF_PY` can point to its `gguf-py` folder): ```sh .venv/bin/python tools/iq_pack.py \ @@ -88,36 +79,40 @@ Build Strata (or install 0.1.31+), then, from the repository root, with gguf-py --out packs/ud-q4_k_xl --compat-bf16 ``` -About 30 seconds. It writes `index.txt`, `dense.bin` (1.4 GB), `native_experts.txt`, the tokenizer, `conversions.json` -and `compat-bf16.json`. **`--compat-bf16` is required**: this file stores 195 small projections (the hyper-connection -up/down matrices, `output_hc_*` and the PLE value) as Q8_0, while the engine reads them as BF16. Without the flag the -packer refuses and writes nothing; with it, they are dequantized and rounded to BF16 (nearest-even). Do not add -`--experts-bin`: it would write a 77 GB copy of the experts that this mode does not need. +It takes about 30 seconds and writes `index.txt`, `dense.bin` (1.4 GB), `native_experts.txt`, the tokenizer, +`conversions.json` and `compat-bf16.json`. **`--compat-bf16` is required.** This file stores 195 small projections (the +hyper-connection up/down matrices, `output_hc_*` and the PLE value) as Q8_0, while the engine reads them as BF16. +Without the flag the packer refuses and writes nothing. With it, they are dequantized and rounded to BF16 +(nearest-even). Do not add `--experts-bin`: it would write a 77 GB copy of experts that this mode does not need. ## The numbers: what is native and what is converted The routed experts, the token embedding, the output head, the attention and shared-expert projections, the PLE key and -the PLE table are used as the GGUF stores them (the engine has GPU kernels and CPU (ggml) paths for Q4_K, Q5_K, Q5_1 and -Q8_0). The small tensors the engine reads as floats are converted, and `conversions.json` lists every one with its -source/destination type, method, whether it is exact, the largest absolute error and the source bytes' SHA-256: -264 F32 tensors (routers, injections, SSM gates) whose values are already BF16's are stored as BF16 exactly; the 195 -Q8_0 projections above are rounded to BF16 (largest absolute error 0.0144, in `hc_ffn_up`); the PLE convolution (F32) -is narrowed to F16 when the engine loads it (largest error 3e-8). That is the compatibility Unsloth's file needs; it is -not the original BF16 checkpoint. The output matches llama.cpp's on the same file: -[Quality](#quality-against-llamacpp-on-the-same-file). +the PLE table are used as the GGUF stores them. The engine has GPU kernels and CPU (ggml) paths for Q4_K, Q5_K, Q5_1 and +Q8_0. + +The small tensors the engine reads as floats are converted. `conversions.json` lists every one with its source and +destination type, method, whether it is exact, the largest absolute error and the source bytes' SHA-256: + +- 264 F32 tensors (routers, injections, SSM gates) whose values are already BF16's are stored as BF16 exactly. +- The 195 Q8_0 projections above are rounded to BF16 (largest absolute error 0.0144, in `hc_ffn_up`). +- The PLE convolution (F32) is narrowed to F16 when the engine loads it (largest error 3e-8). + +That is the compatibility Unsloth's file needs. It is not the original BF16 checkpoint. The output matches llama.cpp's on +the same file ([Quality](#quality-against-llamacpp-on-the-same-file)). ## The MTP draft layer -The draft layer is the base model's, the same one the other packs use. If setup installed Strata, you have it -(`mtp/rt` in `Strata-data`). Otherwise build it as [docs/ORCA.md](ORCA.md#preparation) shows (`tools/mtp_fetch.py`, +The draft layer is the base model's, the same one the other packs use. If setup installed Strata, you have it (`mtp/rt` +in `Strata-data`). Otherwise build it as [ORCA.md](ORCA.md#preparation) shows (`tools/mtp_fetch.py`, `tools/mtp_pack.py`, `tools/mtp_rt.py`, then copy `data/draft_vocab.bin` to `mtp/rt/`). ## The server Save as `strata-ud-q4_k_xl.json` at the repository root (or next to setup's other `strata-*.json`), with `/path/to/` pointing at **shard 1**. `--ple-gguf` is not needed: the engine finds the shard that holds the PLE table -(`per_layer_token_embd.weight`, shard 2) by name. `--resident-budget-gib` turns on the mapped mode with a RAM budget -(it implies `--mmap-experts`). +(`per_layer_token_embd.weight`, shard 2) by name. `--resident-budget-gib` turns on the mapped mode with a RAM budget (it +implies `--mmap-experts`). ```json { @@ -143,61 +138,35 @@ pointing at **shard 1**. `--ple-gguf` is not needed: the engine finds the shard .venv/bin/python -m serve.server --engine strata --config strata-ud-q4_k_xl.json --port 8080 ``` -(`engine/strata.exe` is setup's engine on Windows; a self-built one is `build/strata`.) The expert profile is the base -model's (`data/expert-profile.bin`); it ranks the same 48 x 512 experts and decides which ones the GPU cache and the RAM +`engine/strata.exe` is setup's engine on Windows; a self-built one is `build/strata`. The expert profile is the base +model's (`data/expert-profile.bin`). It ranks the same 48 x 512 experts and decides which ones the GPU cache and the RAM budget take first. -**Choosing the RAM budget N:** your RAM minus 20-24 GB (the OS, the engine itself, the 126 MB router copy, and room for -the OS file cache that serves the rest). On 64 GB, 40. The engine checks available RAM and, on Windows, available -commit capacity. If N exceeds the smaller value minus 4 GiB headroom, it reduces N to that amount minus a 256 MiB -margin and reports which limit applies. Before #403's fix a clamped budget could then fail the safety check. -Everything above N comes from the SSD for every token, so N is the setting that matters -most; a bigger budget was faster in every measurement (24 / 32 / 40 GiB). When the driver page-locks the whole budget -(24 GiB did on this PC, 32 and 40 did not), the GPU also computes a share of the misses over PCIe, as in the resident -low-RAM mode; `STRATA_RESIDENT_PIN=0` uses ordinary pageable memory (the CPU then computes every miss). +**Choosing the RAM budget N.** Use your RAM minus 20 to 24 GB (the OS, the engine itself, the 126 MB router copy, and +room for the OS file cache that serves the rest). On 64 GB that is 40. The engine checks available RAM and, on Windows, +available commit capacity. If N exceeds the smaller value minus 4 GiB headroom, it reduces N to that amount minus a +256 MiB margin and reports which limit applies. Everything above N comes from the SSD for every token, so N is the +setting that matters most; a bigger budget was faster in every measurement (24 / 32 / 40 GiB). -On Windows, the allocation log reports available physical RAM and commit capacity before allocation and after a -failure. Free RAM alone does not guarantee that Windows can commit the requested memory. If the pageable fallback -also fails, the warning preserves both the GPU allocation or mapping error and the `malloc` error. +When the driver page-locks the whole budget, the GPU also computes a share of the misses over PCIe, as in the resident +low-RAM mode. `STRATA_RESIDENT_PIN=0` uses ordinary pageable memory instead, and the CPU then computes every miss. On +Windows the allocation log reports available physical RAM and commit capacity before allocation and after a failure, +because free RAM alone does not guarantee that Windows can commit the requested memory. -**Context:** measured at 4K. The KV cache takes VRAM from the expert cache, so a longer context makes decoding slower; -8K is a reasonable start on 12 GB. `--kv int8` halves the KV cache's VRAM. - -## What to expect (RTX 5070 12 GB, 64 GB RAM, Windows, one short greedy prompt) - -| RAM budget | Greedy output | Notes | -| --- | --- | --- | -| 24 GiB | 5.1-5.2 tok/s (2 runs, before the routing prefetch below) | 2.2 GB read from the SSD per verify round (~3.5 tokens) | -| 40 GiB | **7-8.5 tok/s** (6 runs: 6.7-8.6, mean 7.9) | 1.1 GB read from the SSD per verify round, ~0.33 GB per token | - -- The first answer comes about 60 s after the engine starts (mostly loading: the RAM budget is copied from the GGUF - files at start). A 30-token prompt then took 5-6 s. -- Long prompts (engine 0.1.32): a 16K prompt is read at **160 tokens/s** (15,873 tokens in 99 s; 0.1.31: 57 tokens/s, - 273-282 s). Reading a prompt streams every expert once per 4K chunk, and the third of them outside the RAM budget - come from the SSD (~35 GB per chunk), so the prompt path now reads them with 32 threads and up to 128 blobs in - flight when the experts are read from the GGUF in place (`STRATA_STAGER_THREADS` / `STRATA_STAGER_RING` override - it; 4 and 16 before, still the default for every other model). Between the chunks, the 28.8 GB PLE table's rows - take ~5 s per chunk. -- With `STRATA_MMQ_KQ=1` the engine multiplies the Q4_K / Q5_K / Q5_1 experts of a prompt with llama.cpp's MMQ - kernels (as the other models' formats always are) instead of dequantizing them to FP16: 4x less GPU time for those - products, but on this PC the prompt waits for the SSD either way (29 s per 4K chunk with or without). Opt-in: it - rounds differently from the FP16 path. On CUDA these kernels are a library next to the engine (`libstrata_kq.so` / - `strata_kq.dll`), opened only then, so they take no VRAM from other models ([QUANTS.md](QUANTS.md)); before, they were - the build option `-DSTRATA_MMQ_KQUANTS=ON` and took 1-3 expert slots from every model. -- The speed varies from run to run (6.7-8.6 tok/s at 40 GiB for the same prompt), with what the OS file cache holds. -- For comparison, IQ3_S (all its experts in RAM) writes ~53 tok/s on the same PC ([the speed tables](DETAILS.md#speed-measured)). -- Where the time goes (`--stats` on the command line): at 40 GiB about 550-750 ms of each verify round (3.5 tokens) is - reading experts from the SSD, ~75 ms the CPU's expert kernels, ~13 ms the GPU. +**Context.** The KV cache takes VRAM from the expert cache, so a longer context makes decoding slower. 8K is a reasonable +start on 12 GB. `--kv int8` halves the KV cache's VRAM. ## Quality: against llama.cpp on the same file -Strata and llama.cpp (the pinned commit `3cf0325`, a CPU build reading the GGUF memory-mapped) were given the same -token sequences, and at every position each wrote its 20 most likely next tokens with their log-probabilities: Strata -through its verify windows (`STRATA_LOGPOS` with `STRATA_LOGPOS_TOPK=20`), llama.cpp from `llama_get_logits_ith` over -one batch. Strata ran with the recommended settings (RAM budget 40 GiB, the MTP draft layer on, greedy, -`--adapt-every 100000` so the expert cache does not move during the run). Per position: whether the most likely token is the same (argmax -agreement), how many of the 5 / 10 most likely tokens both have, the KL divergence (llama.cpp's distribution against -Strata's, over llama.cpp's top 20 plus one bucket for the rest), and the perplexity of the sequence under each. +> **Measured on:** Strata 0.1.32 era engine against llama.cpp pinned commit `3cf0325` (CPU build, GGUF memory-mapped). +> Strata ran with RAM budget 40 GiB, the MTP draft layer on, greedy and `--adapt-every 100000`. Hardware is not recorded +> on this page. + +Both were given the same token sequences, and at every position each wrote its 20 most likely next tokens with their +log-probabilities. Strata used its verify windows (`STRATA_LOGPOS` with `STRATA_LOGPOS_TOPK=20`), llama.cpp used +`llama_get_logits_ith` over one batch. Per position: whether the most likely token is the same (argmax agreement), how +many of the 5 and 10 most likely tokens both have, the KL divergence (llama.cpp's distribution against Strata's, over +llama.cpp's top 20 plus one bucket for the rest), and the perplexity of the sequence under each. | Token set | Positions | Argmax agreement | Top-5 / top-10 overlap | KL | Perplexity Strata / llama.cpp | | --- | ---: | ---: | ---: | ---: | ---: | @@ -207,42 +176,44 @@ Strata's, over llama.cpp's top 20 plus one bucket for the rest), and the perplex | Real text after a 16K prompt: the last 510 tokens of 16,384 tokens of this repository's docs and code | 510 | 91.2% | 88.3% / 89.9% | 0.035 | 4.27 / 4.29 | | The same 511 positions with only 512 tokens before them | 511 | 89.2% | 86.0% / 86.3% | 0.061 | 8.99 / 8.87 | -For comparison, [eddoursul/Strata](https://github.com/eddoursul/Strata) reported 98.2-99.3% (code), 95.6-95.8% +For comparison, [eddoursul/Strata](https://github.com/eddoursul/Strata) reported 98.2 to 99.3% (code), 95.6 to 95.8% (thinking) and 97.7% (after a 16K prompt) for the same file against llama.cpp on an RTX 3090. -**Where they differ:** at near-ties. Grouped by llama.cpp's own gap between its two most likely tokens, Strata picks -the same token at every position of the code and thinking answers where that gap is 0.2 or more (695 of 695) and at -every position after the 16K prompt where it is 0.5 or more (144 of 144 in the continuation, 254 of 254 in the real -text; 50 of 54 and 117 of 121 between 0.2 and 0.5). Text after a long document has many more near-ties than a short -answer (58 of the continuation's 256 positions and 135 of the real text's 510 have a gap under 0.2, against 13 of the -code answer's 400), which is why its agreement is lower. It is not the long context itself: the same real-text -positions after only 512 tokens agree a little less (89.2%), and the perplexities match within 3% in every set. For -scale, Strata's own greedy run and its teacher-forced rerun of the continuation pick different tokens at 6% of the -positions (93.8% the same: other window sizes, other experts in VRAM), the same kind of near-tie flips. The 16K -prompts were read by Strata's batched prompt path (15,872 tokens), the rest through the verify windows. These numbers -are the FP16 prompt path's (the released engine's); with the MMQ prompt path (`-DSTRATA_MMQ_KQUANTS=ON`, above) the +The two differ at near-ties. Where llama.cpp's own gap between its two most likely tokens is 0.2 or more, Strata picks +the same token at every position of the code and thinking answers (695 of 695). After the 16K prompt it does so at every +position where the gap is 0.5 or more (144 of 144 in the continuation, 254 of 254 in the real text). Text after a long +document has many more near-ties than a short answer (58 of the continuation's 256 positions and 135 of the real text's +510 have a gap under 0.2, against 13 of the code answer's 400), which is why its agreement is lower. The perplexities +match within 3% in every set. The 16K prompts were read by Strata's batched prompt path (15,872 tokens), the rest +through the verify windows. These numbers are the FP16 prompt path's. With the MMQ prompt path (`STRATA_MMQ_KQ=1`) the continuation after the 16K prompt agrees at 92.2% (KL 0.029, three runs, identical). -To repeat it: Strata's side is `STRATA_LOGPOS=` with `STRATA_LOGPOS_TOPK=20` (engine 0.1.32) on a serve -engine with `--short-read` covering the positions to compare; llama.cpp's side was a short program over -`llama_decode` / `llama_get_logits_ith` writing the same top 20 per position. +To repeat it, set `STRATA_LOGPOS=` with `STRATA_LOGPOS_TOPK=20` on a serve engine with `--short-read` covering the +positions to compare. llama.cpp's side was a short program over `llama_decode` and `llama_get_logits_ith` writing the +same top 20 per position. ## Opt-ins and switches (environment variables) -| | | +| Variable | Effect | | --- | --- | -| `STRATA_LOOKAHEAD=0` | Turns off the routing-aware prefetch (on by default in this mode): while the CPU works on a layer, a thread applies the next layer's router to this layer's input and asks the OS to read the predicted experts' pages. Pages only, the answers are the same. About half of the SSD reads were predicted; mean +14% (6.9 -> 7.9 tok/s). `STRATA_LOOKAHEAD_K` sets the experts per token (default 10). | +| `STRATA_LOOKAHEAD=0` | Turns off the routing-aware prefetch (on by default in this mode). While the CPU works on a layer, a thread applies the next layer's router to this layer's input and asks the OS to read the predicted experts' pages. Pages only; the answers are the same. `STRATA_LOOKAHEAD_K` sets the experts per token (default 10). | +| `STRATA_MMQ_KQ=1` | Multiplies the Q4_K / Q5_K / Q5_1 experts of a prompt with llama.cpp's MMQ kernels instead of dequantizing them to FP16. It rounds differently from the FP16 path, so it is opt-in for this file. On CUDA the kernels are a library next to the engine (`libstrata_kq.so` / `strata_kq.dll`), opened only then, so they take no VRAM from other models ([QUANTS.md](QUANTS.md)). | | `STRATA_KQ256=1` | Multi-token AVX2 kernels for the Q4_K / Q5_1 / Q8_0 experts. Bit-exact with ggml's, but measured no faster, so off. | -| `STRATA_PARTIAL_PIN=1` | Registers the hottest part of the RAM budget (up to `STRATA_PARTIAL_PIN_GIB`, default 24) with the GPU driver, so the GPU computes a share of the misses over PCIe (`--pcie-frac`). Measured no faster on this PC, and it changes the numerics of those experts (GPU kernels instead of the CPU's), so off. | -| `STRATA_FETCH_THREADS=N` | Threads that read the experts from the GGUF while it answers (default 8; 16 was no faster). The prompt path has its own: `STRATA_STAGER_THREADS` (default 32 here) and `STRATA_STAGER_RING` (128). | +| `STRATA_PARTIAL_PIN=1` | Registers the hottest part of the RAM budget (up to `STRATA_PARTIAL_PIN_GIB`, default 24) with the GPU driver, so the GPU computes a share of the misses over PCIe (`--pcie-frac`). Measured no faster, and it changes the numerics of those experts, so off. | +| `STRATA_FETCH_THREADS=N` | Threads that read the experts from the GGUF while it answers (default 8). The prompt path has its own: `STRATA_STAGER_THREADS` (default 32 here) and `STRATA_STAGER_RING` (128). | +| `STRATA_RESIDENT_PIN=0` | Ordinary pageable memory for the RAM budget (see above). | -## UD-IQ4_XS (setup from 0.1.39, #621) +## UD-IQ4_XS -Unsloth's `UD-IQ4_XS` at the same revision is in setup too, as a regular choice (not experimental; the Unsloth -family's first size and its default): `START-HERE.bat --setup --family unsloth --model UD-IQ4_XS` (engine 0.1.38 or -newer). It sits between IQ3_S and UD-Q4_K_XL: its routed experts are IQ3_S gate/up (IQ4_XS -in one layer) with IQ4_NL downs (Q8_0 in five layers), 59.5 GB of them; the dense side (Q8_0 projections, the IQ4_NL -PLE table, a Q6_K head) is UD-Q4_K_XL's. Three shards, 93.7 GB: +Unsloth's `UD-IQ4_XS` at the same revision is in setup as a regular choice, the Unsloth family's first size: + +```sh +strata launch --family unsloth --model UD-IQ4_XS +``` + +It sits between IQ3_S and UD-Q4_K_XL. Its routed experts are IQ3_S gate/up (IQ4_XS in one layer) with IQ4_NL downs (Q8_0 +in five layers), 59.5 GB of them. The dense side (Q8_0 projections, the IQ4_NL PLE table, a Q6_K head) is UD-Q4_K_XL's. +Three shards, 93.7 GB: | File | Bytes | SHA-256 | | --- | ---: | --- | @@ -250,37 +221,38 @@ PLE table, a Q6_K head) is UD-Q4_K_XL's. Three shards, 93.7 GB: | `Qwen3.8-Flash-Next-UD-IQ4_XS-00002-of-00003.gguf` | 49,835,229,856 | `577a38a2392b40ca2193cea502e1d92f60b8cd370675d308e0ec21885d9daaa7` | | `Qwen3.8-Flash-Next-UD-IQ4_XS-00003-of-00003.gguf` | 43,836,407,744 | `d4634e6d84f0ebb0940be15c90d3790bf6464e3dea3a1cddc567dc0e83ad8833` | -Setup treats it like UD-Q4_K_XL (the list above): the same RAM budget (your RAM less 24 GB; at most all 55 GiB of its -experts, so a PC with ~80 GB of RAM or more holds all of them), the pack with `--compat-bf16`, no `experts.bin`, one -GPU by default (several keep the budget, as above). It does not ask on AMD: its experts' formats have prompt kernels there too. Images are an option (asked, -off by default; NVIDIA): the image path has no restriction for this pack, which uses the original model's image -encoder; not yet run with images. +Setup treats it like UD-Q4_K_XL: the same RAM budget rule (your RAM less 24 GB, at most all 55 GiB of its experts, so a +PC with about 80 GB of RAM or more holds all of them), the pack with `--compat-bf16`, no `experts.bin`, and one GPU by +default. It uses the original model's image encoder. -What is known so far: it packs and runs on a Strix Halo (AMD gfx1151, 128 GB unified memory), where prompts read at -~820 tokens/s at 8K and 64K and the answers passed the retrieval checks at 8K and 64K. It has **not** been measured on -NVIDIA yet; please report what you see. One fidelity note that applies to every `--compat-bf16` pack (UD-Q4_K_XL too): -the pack rounds the file's Q8_0 hyper-connection projections to BF16, and on the Strix Halo that rounding put -perplexity 6-9% above llama.cpp's on the same file (teacher-forced, short context). Reading the Q8_0 values instead -closes most of that gap; it is measured on AMD only and not in this release. +It packs and runs on a Strix Halo (AMD gfx1151, 128 GB unified memory), where prompts read at about 820 tokens/s at 8K +and 64K and the answers passed the retrieval checks at 8K and 64K. It has not been measured on NVIDIA; please report +what you see. + +One fidelity note applies to every `--compat-bf16` pack, UD-Q4_K_XL included. The pack rounds the file's Q8_0 +hyper-connection projections to BF16, and on the Strix Halo that put perplexity 6 to 9% above llama.cpp's on the same +file (teacher-forced, short context). `STRATA_HC_Q8=1`, which setup writes on CUDA for the UD files, reads the file's Q8_0 +values instead of the rounded BF16. The 6 to 9% gap was measured on AMD, where setup does not write it. ## Q4_0 files -A plain `llama-quantize` Q4_0 file of this model runs the same way: pack shard 1 with `--compat-bf16` as above and -point `--native` at it. Tested: +A plain `llama-quantize` Q4_0 file of this model runs the same way: pack shard 1 with `--compat-bf16` as above and point +`--native` at it. Tested: [bartowski/Qwen3.8-Flash-Next-GGUF, `Qwen3.8-Flash-Next-Q4_0`, revision `928589f`](https://huggingface.co/bartowski/Qwen3.8-Flash-Next-GGUF/tree/928589fdb66c6ff07f22ac561e3fbce76553548f/Qwen3.8-Flash-Next-Q4_0) -(3 shards, 100.6 GB). Its routed experts are Q4_0 for gate/up and for down, except down in layers 0-5, which is Q4_1; -the PLE table and the token embedding are Q4_0. The pack serves 302 tensors from the GGUF as stored and converts 362 +(3 shards, 100.6 GB). Its routed experts are Q4_0 for gate/up and for down, except down in layers 0 to 5, which is Q4_1. +The PLE table and the token embedding are Q4_0. The pack serves 302 tensors from the GGUF as stored and converts 362 (168 exact, 194 rounded to BF16, largest absolute error 0.0156; 1.23 GiB). The Q4_0 expert kernels centre the codes (q - 8) before the integer dot product, as ggml-cpu's Q4_0 x Q8_0 dot does, -instead of llama.cpp's CUDA form (uncentred codes, minus 8 x the Q8_1 block sum in float); Q4_1 follows the Q5_1 -kernel without the fifth bit. `native_expert_parity` on this file (layers 0 and 5: Q4_0/Q4_1, layers 20 and 47: -Q4_0/Q4_0): relative error against the float product 1.1-1.3e-2 on the CPU and the GPU alike, CPU vs GPU 1.1e-7 for -Q4_0/Q4_0 and 5e-4 to 1.1e-3 with Q4_1 down. +instead of llama.cpp's CUDA form (uncentred codes, minus 8 x the Q8_1 block sum in float). Q4_1 follows the Q5_1 kernel +without the fifth bit. `native_expert_parity` on this file (layers 0 and 5: Q4_0/Q4_1, layers 20 and 47: Q4_0/Q4_0) +gives a relative error against the float product of 1.1 to 1.3e-2 on the CPU and the GPU alike, and CPU against GPU of +1.1e-7 for Q4_0/Q4_0 and 5e-4 to 1.1e-3 with Q4_1 down. -Measured on one RTX 3060 12 GB (PCIe 3.0 x16), Xeon E5-2696 v4, 121 GiB DDR4-2400, Linux, CUDA 12.4; every expert in -RAM, `--max-context 131072 --kv int8 --kv-resident 20480`, expert cache `auto` (1,960 slots, 5.06 GiB), prompt chunk -`auto` (6,144 tokens); greedy, `reasoning_effort` medium, one request at a time: +> **Measured on:** one RTX 3060 12 GB (PCIe 3.0 x16), Xeon E5-2696 v4, 121 GiB DDR4-2400, Linux, CUDA 12.4. Every +> expert in RAM, `--max-context 131072 --kv int8 --kv-resident 20480`, expert cache `auto` (1,960 slots, 5.06 GiB), +> prompt chunk `auto` (6,144 tokens), greedy, `reasoning_effort` medium, one request at a time. Strata version not +> recorded. | Prompt | Prompt read | Output | | --- | --- | --- | @@ -289,28 +261,62 @@ RAM, `--max-context 131072 --kv int8 --kv-resident 20480`, expert cache `auto` ( | 24,461 tokens | 787 tok/s | 24.0 tok/s | | 67,934 tokens (a fact hidden in the middle: found) | 737 tok/s | 26.4 tok/s | -A coding prompt was answered correctly. Not measured: agreement with llama.cpp on this file (as -[above](#quality-against-llamacpp-on-the-same-file) for UD-Q4_K_XL), AMD, Windows, and bartowski's Swift-1.5 Q4_0, -whose header shows the same formats. +A coding prompt was answered correctly. Not measured: agreement with llama.cpp on this file, AMD, Windows, and +bartowski's Swift-1.5 Q4_0, whose header shows the same formats. ## Scope and validation -- UD-Q4_K_XL and UD-IQ4_XS at revision `38bb39e` are targeted, and plain Q4_0 files ([above](#q4_0-files)). Other - quantizations use formats this engine may not have kernels for; the engine checks every layer's formats at start - and refuses an unsupported one by name. -- Tests: the packer's synthetic 4-shard and conversion tests (`.venv/bin/python -m unittest discover -s tools -p - test_iq_pack.py`); CTests `gguf_split_test`, `expert_layout_test`, `native_expert_parity_*` (the three real expert - format pairs against ggml-cpu, the Q5_1 min term, Q8_0 rows, Q4_0/Q4_0 and Q4_0/Q4_1), `prefill_mmq_kquant_test` (with `-DSTRATA_MMQ_KQUANTS=ON`: the - prompt path's MMQ products for Q4_K / Q5_K / Q5_1 / Q8_0 against ggml's dequantized weights); the in-place mode against `experts.bin` on the Coder - (identical tokens and logits). -- Real runs: greedy answers to a coding prompt (correct) at every budget and setting above, identical across them; - the same tokens as llama.cpp at 97.5-99% of the positions of short greedy answers and 90-91% after a 16K prompt, - differing mostly at near-ties (above). Not yet: sampled decoding, long conversations. Please report what you see. +- UD-Q4_K_XL and UD-IQ4_XS at revision `38bb39e` are targeted, and plain Q4_0 files. Other quantizations use formats this + engine may not have kernels for. The engine checks every layer's formats at start and refuses an unsupported one by + name. +- Tests: the packer's synthetic 4-shard and conversion tests + (`.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`) and the CTests `gguf_split_test`, + `expert_layout_test`, `native_expert_parity_*` (the three real expert format pairs against ggml-cpu, the Q5_1 min term, + Q8_0 rows, Q4_0/Q4_0 and Q4_0/Q4_1). With `-DSTRATA_MMQ_KQUANTS=ON`, `prefill_mmq_kquant_test` checks the prompt path's + MMQ products for Q4_K / Q5_K / Q5_1 / Q8_0 against ggml's dequantized weights. +- Not validated: sampled decoding and long conversations. Please report what you see. ## Credits The split-artifact loader, the expert formats' GPU kernels and the packing of Unsloth's files follow [eddoursul/Strata](https://github.com/eddoursul/Strata) (PR #245), which ran these files first. The Q8_0 kernels, the -per-role shard column and the PLE convolution's F32 -> F16 fix come from @gopinath87607's PR #255; per-role shard names +per-role shard column and the PLE convolution's F32 to F16 fix come from @gopinath87607's PR #255. Per-role shard names were also proposed by @jagsan-cyber in PR #247. The quantization is [Unsloth](https://huggingface.co/unsloth)'s; the model is Qwen's ([license](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)). + +## History / older measurements + +### RAM budget, speed on a 12 GB card + +> **Measured on:** RTX 5070 12 GB, 64 GB RAM, Windows 10, AVX-512 CPU, UD-Q4_K_XL, one short greedy prompt, first +> validation (2026-10-01, engine 0.1.31/0.1.32). + +| RAM budget | Greedy output | Notes | +| --- | --- | --- | +| 24 GiB | 5.1 to 5.2 tok/s (2 runs, before the routing prefetch) | 2.2 GB read from the SSD per verify round (about 3.5 tokens) | +| 40 GiB | 7 to 8.5 tok/s (6 runs: 6.7 to 8.6, mean 7.9) | 1.1 GB read from the SSD per verify round, about 0.33 GB per token | + +- The first answer comes about 60 s after the engine starts (the RAM budget is copied from the GGUF files at start). A + 30-token prompt then took 5 to 6 s. +- A 16K prompt was read at 160 tokens/s (15,873 tokens in 99 s) on engine 0.1.32, up from 57 tokens/s (273 to 282 s) on + 0.1.31, after the prompt path began reading experts with 32 threads and up to 128 blobs in flight. +- The routing-aware prefetch added a mean of +14% (6.9 to 7.9 tok/s); about half of the SSD reads were predicted. +- With `STRATA_MMQ_KQ=1` the products took 4x less GPU time, but on this PC the prompt waited for the SSD either way (29 s + per 4K chunk). +- Where the time goes at 40 GiB (`--stats`): about 550 to 750 ms of each verify round (3.5 tokens) reading experts from + the SSD, about 75 ms the CPU's expert kernels, about 13 ms the GPU. +- For comparison, IQ3_S (all its experts in RAM) wrote about 53 tok/s on the same PC + ([the speed tables](DETAILS.md#speed-measured)). + +### Two GPUs with the RAM budget + +> **Measured on:** RTX 3090 (x16) + RTX 3080 Ti (x4), 91 GB RAM, budget 64 GiB, engine 0.1.41. + +Decode 45 tok/s against 38 on the 3090 alone (the auto split gives the x4 card two layers); 57.6 GiB of experts in RAM, +no file reads per request. + +> **Measured on:** 2x RTX 3090, 165 GiB RAM, an older engine that ran the split without the RAM budget (all 77 GB of +> experts loaded into RAM at start). + +Decode 31 tok/s on one card with the budget, 64 to 78 tok/s split (55 tok/s at a 128K prompt), with `MemAvailable` never +under 68 GiB. diff --git a/docs/UNSLOTH_Q6.md b/docs/UNSLOTH_Q6.md index 03094911f..64f652564 100644 --- a/docs/UNSLOTH_Q6.md +++ b/docs/UNSLOTH_Q6.md @@ -1,24 +1,27 @@ # Unsloth UD-Q6_K_XL -**In setup since MODEL-UPDATE Phase 2** (`START-HERE.bat --setup --family unsloth --model UD-Q6_K_XL`, engine 0.1.44 or -newer: every expert in RAM from ~118 GB, else it asks and keeps a RAM budget; quality, speed and images in -[QUANTS.md](QUANTS.md); UD-Q5_K_XL and Q8_0 likewise). The manual workflow below still works. This is Unsloth's 6-bit -file of the same model the other packs use: -[unsloth/Qwen3.8-Flash-Next-GGUF, `UD-Q6_K_XL`, revision `38bb39e`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL). -Build it for engine 0.1.38 or newer (earlier engines read `per_layer_token_embd.weight` in IQ4_NL, Q5_0 or FP8 only, -below). UD-Q4_K_XL's manual workflow asks nothing of setup, and neither does this one. +This page is for people who want to run Unsloth's 6-bit Qwen3.8-Flash-Next file with Strata, and for people who pack it +by hand. UD-Q5_K_XL and Q8_0 are set up the same way. Quality, speed and images across the Unsloth sizes are in +[QUANTS.md](QUANTS.md). UD-Q4_K_XL, with the full setup details, is in [UNSLOTH_Q4.md](UNSLOTH_Q4.md). -> **Status (MODEL-UPDATE Phase 1).** The Q6_K gate/up kernels are part of every build now (they were the opt-in -> `-DSTRATA_Q6K_EXPERTS=ON`): measured to leave other models' expert-cache slots unchanged ([QUANTS.md](QUANTS.md)). The -> MMQ prompt kernels for Q6_K come with the K-quant library and `STRATA_MMQ_KQ=1`; without it the prompt takes the FP16 -> path. The file runs on the standard engine (AMD R9700: the quality yardstick in [QUANTS.md](QUANTS.md)). +The file is [unsloth/Qwen3.8-Flash-Next-GGUF, `UD-Q6_K_XL`, revision `38bb39e`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL), +the same model the other packs use. + +## Quick start + +```sh +strata launch --family unsloth --model UD-Q6_K_XL +``` + +Setup keeps every expert in RAM from about 118 GB of RAM. Below that it asks and keeps a RAM budget, as for UD-Q4_K_XL +([What setup does](UNSLOTH_Q4.md#what-setup-does)). For UD-Q5_K_XL and UD-Q6_K_XL it writes `STRATA_MMQ_KQ=1`, the +K-quant MMQ prompt kernels, into the run config. ## What is in the file -Six shards, 169,165,382,688 bytes (169.2 GB) together ([sizes and SHA-256 from the HF tree API](https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL); -shards 1, 2 and 6 were also checked bit for bit against their Hugging Face LFS hashes after download). Shard 1 holds -only metadata; shard 2 only the output head (`output.weight`, `output_hc_*`, all Q8_0 but `output_hc_norm`, F32); -shard 3 is exactly the PLE table, nothing else. +Six shards, 169,165,382,688 bytes (169.2 GB) together ([sizes and SHA-256 from the HF tree API](https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL)). +Shard 1 holds only metadata. Shard 2 holds only the output head (`output.weight`, `output_hc_*`, all Q8_0 but +`output_hc_norm`, F32). Shard 3 is exactly the PLE table. | File | Bytes | SHA-256 | | --- | --- | --- | @@ -29,30 +32,24 @@ shard 3 is exactly the PLE table, nothing else. | `Qwen3.8-Flash-Next-UD-Q6_K_XL-00005-of-00006.gguf` | 49,419,317,344 | `9948e81ae8368144b135a7403d8c0f2cf9a6c2f9f692313865855b2832e696c9` | | `Qwen3.8-Flash-Next-UD-Q6_K_XL-00006-of-00006.gguf` | 14,838,313,984 | `e4ae255234f42b18012f6d6eec5bf615b57479b7e0a0233c47436a321ffeec3f` | -Formats, read from the shard headers with `tools/gguf_reader.py` (all six shards verified: shard 4 = blocks 0-20, -520 tensors, shard 5 = blocks 20-41, 530 tensors, and per layer the types above): +Formats, read from the shard headers with `tools/gguf_reader.py`: -- **Routed experts:** `ffn_gate_exps` and `ffn_up_exps` are Q6_K (seen in every layer of shard 6: blocks 41-47); - `ffn_down_exps` is Q8_0 (per layer 891.3 MB at 640×2560×512, against Q6_K's 688.1 MB at 2560×640×512 for gate or - up). Gate and up are Q6_K in 47 of 48 layers and Q8_0 in layer 2; down is Q8_0 in every layer (the pack's - `native_experts.txt`: 47 rows of ggml types 14/8, and layer 2 is 8/8). Q6_K gate/up is native: the - engine multiplies them directly on the GPU in every path check below, and the CPU has the ggml block layout (210 - bytes per 256 elements); the per-role layout in `native_experts.txt` v4 handles shards whose boundaries cut - through a layer like UD-Q4_K_XL's layer 11 case does. -- **Everything else:** attention and shared-expert projections, SSM, hyper-connections, the PLE key/value, the token - embedding and the output head are Q8_0, or F32/BF16 where the engine reads floats — the same pattern +- **Routed experts.** `ffn_gate_exps` and `ffn_up_exps` are Q6_K in 47 of 48 layers and Q8_0 in layer 2. `ffn_down_exps` + is Q8_0 in every layer (891.3 MB per layer at 640x2560x512, against Q6_K's 688.1 MB for gate or up). The pack's + `native_experts.txt` has 47 rows of ggml types 14/8, and layer 2 is 8/8. The engine multiplies Q6_K gate/up natively on + the GPU, and the CPU has the ggml block layout (210 bytes per 256 elements). The per-role layout in + `native_experts.txt` v4 handles shards whose boundaries cut through a layer. +- **Everything else.** Attention and shared-expert projections, SSM, hyper-connections, the PLE key/value, the token + embedding and the output head are Q8_0, or F32/BF16 where the engine reads floats. This is the same pattern [UD-Q4_K_XL](UNSLOTH_Q4.md) handles with `tools/iq_pack.py --compat-bf16`. -- **The PLE table** (`per_layer_token_embd.weight`, [160, 320001536]): **Q8_0**, its own shard 3. A row is - 5 blocks of 32 elements at 34 bytes = 170 bytes, and the whole table is 54.4 GB, against the 28.8 GB IQ4_NL of - UD-Q4_K_XL. Engine 0.1.38 reads Q8_0 rows natively: `PleTable` accepts the type, the row reader is already - parameterized by bytes-per-row (its selftest runs 90, 110 and 170 byte rows), and no GPU kernel changes — - `gather` outputs floats. (UD-Q4_K_XL's table is IQ4_NL; the FP8 table of [tools/ple_fp8_pack.py](../tools/ple_fp8_pack.py) - needs the original checkpoint, which this file is not a substitute for. Converting this Q8_0 table to FP8 would - trade table precision away for 6% smaller rows; Q8_0 read directly has none that conversion's error.) +- **The PLE table** (`per_layer_token_embd.weight`, [160, 320001536]) is **Q8_0**, in its own shard 3. A row is 5 blocks + of 32 elements at 34 bytes, 170 bytes in all, and the whole table is 54.4 GB (UD-Q4_K_XL's IQ4_NL table is 28.8 GB). + The engine reads Q8_0 rows natively. The FP8 table of [tools/ple_fp8_pack.py](../tools/ple_fp8_pack.py) needs the + original checkpoint, which this file is not a substitute for. -Q6_K gate/up costs 210 bytes per 256 elements, against Q4_K's 144, so the routed experts' share of the file grows -about 45% per element read; expected decode speed accordingly lower than UD-Q4_K_XL's on the same hardware. The -table row gather grows 170/90 = 1.89x over IQ4_NL. +Q6_K gate/up costs 210 bytes per 256 elements, against Q4_K's 144, so the routed experts' share of the file grows about +45% per element read. Decode is accordingly slower than UD-Q4_K_XL's on the same hardware. The table row gather grows +170/90 = 1.89x over IQ4_NL. ## The pack (by hand) @@ -64,22 +61,18 @@ Same workflow as [UD-Q4_K_XL](UNSLOTH_Q4.md#the-pack-by-hand), from the reposito --out packs/ud-q6_k_xl --compat-bf16 ``` -**`--compat-bf16` is required** for the same reason as there (the 195 small Q8_0 projections the engine reads as -BF16). Do not add `--experts-bin`; the experts stay in the GGUF files. Built here end to end on this file -(`packs/ud-q6_k_xl`, the packer's own tests cover the shard and conversion logic via -`.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`): 1079 tensors indexed, 303 served natively, 460 -tensors converted (264 exact, 196 rounded, max abs error 0.0144 - the same profile as UD-Q4_K_XL's), 195 -compat-BF16 tensors (1.20 GiB), arena 1.38 GiB, and `native_experts.txt` v4 with the per-role shard column on -layers 20 and 41. +**`--compat-bf16` is required**, for the same reason as there (the 195 small Q8_0 projections the engine reads as BF16). +Do not add `--experts-bin`; the experts stay in the GGUF files. The pack on the finished file has 1079 tensors indexed, +303 served natively, 460 converted (264 exact, 196 rounded, max abs error 0.0144), 195 compat-BF16 tensors (1.20 GiB), +an arena of 1.38 GiB, and `native_experts.txt` v4 with the per-role shard column on layers 20 and 41. The packer's tests: +`.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`. ## The server -Like UD-Q4_K_XL's [config](UNSLOTH_Q4.md#the-server), with the Q6 paths and names. `--ple-gguf` is not needed: the -engine finds shard 3 by the table's name. On a machine whose RAM holds the ~110 GB of routed experts plus the rest -(499 GB total, ~325 GB free, measured on the machine this was written on), there is no RAM budget and no -`--expert-cache` gymnastics: drop `--resident-budget-gib` and `--expert-cache-device1` from the Q4 config. The -54.4 GB table is mmap'd (the default) — do not switch it to `--ple-io ram` unless the RAM really is there to lock -it. `mtp/rt` is the base model's draft layer as always. +Like UD-Q4_K_XL's [config](UNSLOTH_Q4.md#the-server), with the Q6 paths and names. `--ple-gguf` is not needed: the engine +finds shard 3 by the table's name. On a machine whose RAM holds the roughly 110 GB of routed experts plus the rest, drop +`--resident-budget-gib` and `--expert-cache-device1` from the Q4 config. The 54.4 GB table is mapped (the default). Do +not switch it to `--ple-io ram` unless the RAM really is there to lock it. `mtp/rt` is the base model's draft layer. ```json { @@ -99,46 +92,28 @@ it. `mtp/rt` is the base model's draft layer as always. } ``` -On 64 GB machines treat the Q4 numbers below as the starting point: the RAM budget matters the same way, and -everything it leaves out comes from the SSD, faster now per expert (Q6_K) and per table row (170 bytes). +On a 64 GB machine, start from UD-Q4_K_XL's RAM-budget advice. The budget matters the same way, and everything it leaves +out comes from the SSD. -## What the engine did not have, and has now +## Kernels and tests -UD-Q4_K_XL's expert formats (gate/up Q4_K, down Q5_1) were in every path; Q6_K gate/up was not. This branch adds -it (the PLE Q8_0 reader is the other half): +- Decode windows: the grouped expert kernels (`native_expert_grouped`) take Q6_K gate/up (a pinned copy of llama.cpp's + `vec_dot_q6_K_q8_1` behind `Fmt<14>`). Per-token MMVQ (`native_mmvq`) has Q6_K too. +- Prompt path: llama.cpp's MMQ prompt kernels instantiate Q6_K in the K-quant library (`STRATA_MMQ_KQ=1`). Without it + the prompt takes the FP16 path. +- Tests: `native_grouped_parity` loops Q6_K/Q8_0 pairs, `prefill_mmq_kquant_test` runs Q6_K gate/up products at + 1280x2560 against a double-precision reference, and `native_expert_parity --synthetic q6_K/q8_0` checks GPU and CPU + against ggml's dequantized weights. `ple_reader_test --selftest` runs 90, 110 and 170-byte PLE rows, and + `ple_q8_parity` matches ggml's reference dequantizer bit for bit through both Mmap and Direct. +- Not measured on this page: quality against llama.cpp on this file ([QUANTS.md](QUANTS.md) has the Unsloth quality + comparison). -- **Decode windows:** the grouped expert kernels (`native_expert_grouped`) take gate/up Q6_K — a pinned copy of - llama.cpp's `vec_dot_q6_K_q8_1` behind `Fmt<14>`, and the dequantizer (`dequantize_q6_K` folded onto the kernel's - 32 threads). Per-token MMVQ (`native_mmvq`) already had Q6_K. -- **The prompt path:** llama.cpp's MMQ prompt kernels instantiate Q6_K (the K-quant library, `STRATA_MMQ_KQ=1`; it builds - `template-instances/mmq-instance-q6_k.cu` alongside q4_k/q5_k/q5_1; `moe_mmq` accepts and dispatches it). -- **The gates:** `native_expert_supported` took Q6_K/Q8_0 pairs (gate/up Q8_0 as in layer 2 was already in the - lists) — the engine previously refused the whole pack with "no GPU kernels for". +## History / older measurements -Tests: `native_grouped_parity` loops Q6_K/Q8_0 pairs; `prefill_mmq_kquant_test` runs Q6_K gate/up products at -1280×2560 against a double-precision reference; `native_expert_parity --synthetic q6_K/q8_0` checks GPU and CPU -against ggml's dequantized weights. All three are built (CUDA, `86;120`). They have not been run on a GPU: the -card was serving the file below. +> **Measured on:** Linux, RTX PRO 4500 Blackwell (compute capability 12.0) plus a second GPU, Threadripper PRO 5975WX +> (AVX2, no AVX-512), 30 expert-pool workers, 499 GB RAM, 2026-10-04. One code-review session. -## Status: what is verified and what is not - -Verified here (test clone, Linux, NVIDIA GPU, source build): - -- The six files' sizes and SHA-256 (HF tree API; shards 1, 2, 6 additionally against the downloaded blobs). -- The tensor inventory of shards 1, 2 and 6, read from the headers (`tools/gguf_reader.py`); shards 4 and 5's - tensor counts and per-layer expert types, from the pack's `conversions.json` and `native_experts.txt` v4. -- Q8_0 PLE rows: `ple_reader_test --selftest` runs 90, 110 and 170-byte rows (all OK); `ple_q8_parity` on a - synthetic Q8_0 table (`tools/ple_q8_fixture.py`) matches ggml's reference dequantizer bit for bit through both - Mmap and Direct (`max_abs 0.000e+00`). Both still pass on this branch with the Q6_K expert work in. -- The pack on the finished file (numbers above), and the engine binary builds (CUDA `86;120`, MMQ K-quants on). -- Q6_K expert source support: decode-side MMVQ was already there; the grouped verify-window kernels, the MMQ - prompt kernels and the `native_expert_supported` gate were not, and are in this branch (section above). - -The server has run this file. On 2026-10-04, Linux, an RTX PRO 4500 Blackwell (compute capability 12.0) plus a -second GPU, a Threadripper PRO 5975WX (AVX2, no AVX-512), 30 expert-pool workers and 499 GB of RAM, engine 0.1.39 -loaded 101.75 GiB of experts (8.43 GiB/s) and served one review. The cold read was 14,915 tokens at 3,176 tok/s. -Fourteen turns generated 5,799 tokens in 74.1 s (78.3 tok/s) and finished at 44,160 tokens of context; drafts were -accepted 3,485 of 4,406 (79.1%). The same machine on engine 0.1.38, before this branch was rebased onto 0.1.39, -read 249,193 tokens cold at 2,802 tok/s and then generated 1,079 tokens at 69.0 tok/s. Quality against llama.cpp -on this file has not been measured, so the page stays experimental. The three Q6_K parity programs above are the -remaining GPU check. \ No newline at end of file +| Engine | Experts loaded | Prompt | Decode | Drafts accepted | +| --- | --- | --- | --- | --- | +| 0.1.39 | 101.75 GiB at 8.43 GiB/s | 14,915 tokens cold at 3,176 tok/s | 5,799 tokens in 74.1 s (78.3 tok/s) over 14 turns, ending at 44,160 tokens of context | 3,485 of 4,406 (79.1%) | +| 0.1.38 | | 249,193 tokens cold at 2,802 tok/s | 1,079 tokens at 69.0 tok/s | | diff --git a/packaging/ENGINE-COMPONENT.md b/packaging/ENGINE-COMPONENT.md index 0d3157e85..24f19e438 100644 --- a/packaging/ENGINE-COMPONENT.md +++ b/packaging/ENGINE-COMPONENT.md @@ -1,7 +1,7 @@ # Engine component manifest (schema 2) -One format for every engine package: Windows CUDA 13 and 12, Windows HIP, Linux CUDA, Linux HIP, Linux SYCL and macOS -Metal. Control (`strata_control/engine_component.py`) reads it, checks it, and only then starts the engine. +This page is for developers who build or check engine packages. It defines the manifest that every engine package carries: +Windows CUDA 13 and 12, Windows HIP, Linux CUDA, Linux HIP, Linux SYCL and macOS Metal. Control (`strata_control/engine_component.py`) reads it, checks it, and only then starts the engine. `packaging/package_engine.py` writes it. A package you make locally is unsigned and loads. A package in the release feed is signed with the feed key (an Ed25519 key the owner keeps offline), and `strata update` / "Update engine" install it only when that signature verifies ([UPDATES.md](UPDATES.md)). The installers carry an engine of their own, so a new install needs no download to start a model. @@ -24,7 +24,7 @@ A package folder is immutable: a new build is a new folder. |---|---|---| | `schema_version` | 2 | schema 1 packages (no `archs`, `min_driver`, `build_id`, `signature`; `cuda_archs` instead of `archs`) still load | | `id`, `name` | `"strata-engine"` | the component name; anything else is refused | -| `version` | string | the engine release, e.g. `0.1.45`; the newest compatible version wins | +| `version` | string | the engine release, e.g. `0.1.50`; the newest compatible version wins | | `platform` | string | `windows-x86_64`, `linux-x86_64`, `macos-arm64` (must match this PC) | | `backend` | string | `cuda`, `hip`, `sycl`, `metal`, `cpu`; must be a combination `BACKEND_MATRIX` commits to, and the one this PC's GPU needs | | `archs` | list of strings | what the build contains. CUDA: `sm_86` (machine code for that card), `compute_89` (PTX, runs on that generation or newer). HIP: `gfx1201`. SYCL and Metal: a descriptive tag (`intel-arc`, `apple-gpu`). Required for every non-`cpu` backend. Enforced for CUDA against `nvidia-smi` compute capability; recorded for the others | @@ -40,7 +40,7 @@ A package folder is immutable: a new build is a new folder. Example (shortened): ```json -{"schema_version": 2, "id": "strata-engine", "name": "strata-engine", "version": "0.1.45", +{"schema_version": 2, "id": "strata-engine", "name": "strata-engine", "version": "0.1.50", "platform": "linux-x86_64", "backend": "cuda", "archs": ["sm_86"], "min_driver": {"nvidia": "580.65"}, "build_id": "be885026f96a-cuda13.0", "entrypoint": "strata", "files": [{"path": "strata", "sha256": "<64 hex>", "bytes": 191000000}], "signature": null} @@ -59,7 +59,7 @@ In this order, each refusal names the cause; none of them starts anything and no 6. on POSIX the entrypoint is executable. If a package fails (5) or (4) and no other package fits this PC, the start ends with `EngineIntegrityError` -(exit code 5): "The engine package failed its integrity check, so it was not started: 0.1.45 linux-x86_64-cuda: +(exit code 5): "The engine package failed its integrity check, so it was not started: 0.1.50 linux-x86_64-cuda: libcudart.so.13 does not match its recorded SHA-256. Repair or reinstall the engine component with Update engine in the app's Settings or `strata update` in the Terminal". A package for another GPU or an older driver is `EngineMissing` with its reason. A tampered newer package never falls back silently @@ -83,29 +83,35 @@ UTF-8). It contains every file hash, so one signature covers the whole package. ## Building a package - python packaging/package_engine.py --src --backend cuda --version 0.1.45 \ + python packaging/package_engine.py --src --backend cuda --version 0.1.50 \ --out /engine --arch sm_86 --arch compute_89 --min-driver nvidia=580.65 \ - [--build-id ] [--variant cuda13] [--platform linux-x86_64] [--include 'rocm/**' --include strata-vision.exe | --tree] + [--build-id ] [--variant cuda13] [--platform linux-x86_64] [--include 'rocm/**' --include strata-vision.exe | --tree] [--require-dsv41] python packaging/package_engine.py --verify # full hash check -Without `--include`/`--tree` it takes the entrypoint, the libraries beside it, `BUILD.json` and `LICENSE*`. The build id +Without `--include`/`--tree` it takes the entrypoint, the libraries beside it, `BUILD.json`, `LICENSE*`, the picture reader +`strata-vision` when the build has one, and the DeepSeek engine `strata-dsv41[.exe]` when it sits beside `strata`. +`--require-dsv41` refuses a package without `strata-dsv41`. The build id defaults to the checkout's commit (`-dirty` when the tree has changes). It prints the size, file count and the SHA-256 of the manifest file (which holds every file's hash), used as the package's digest in notes. ## What each target's package holds +Variants in the release: Windows `cuda13`, `cuda12`, `rx9070`, `rdna3`, `rdna2`; Linux `cuda13`, `cuda12`, `rx9070`, `rx7900`, `halo`, `rx6000`, plus +the SYCL and Metal packages. The packages that also carry `strata-dsv41` (DeepSeek V4.1-Flash, engine 0.1.50) are Windows `cuda13`, Linux `cuda13` +and Linux `rx9070`. Build `strata-dsv41` with `packaging/build_dsv41_cuda.sh` or `packaging/build_dsv41_windows.bat` and copy it beside `strata` before packaging. + | Package | Built on | Contents | Notes | |---|---|---|---| -| `windows-x86_64` cuda, variant `cuda13` | the owner PC, the same cmake line as the release zips (CUDA 13.0, `75;86;89;120` real + `120` PTX, `STRATA_PORTABLE=ON`) | `strata.exe`, `strata_kq.dll`, `strata-vision.exe` (built for this variant's archs by `packaging/build_vision_windows.bat cuda13 `, which writes `vision_archs` into BUILD.json), `strata-dsv41.exe` (DeepSeek-V4.1, `packaging/build_dsv41_windows.bat`, sm_86 + sm_120; `--require-dsv41`), `BUILD.json` | `min_driver` nvidia 580.65 | -| `windows-x86_64` cuda, variant `cuda12` | the owner PC (CUDA 12.9, `60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same four files, with its own `strata-vision.exe` (`packaging/build_vision_windows.bat cuda12 `: sm_60 .. sm_89; the CUDA 13 reader has no Pascal code, #3669) | experimental, `min_driver` nvidia 575.51; the CUDA 13 build wins when both fit | -| `windows-x86_64` hip | the owner PC, `tools/hip/build_windows.bat` (ROCm 10.2 nightly wheels) then `tools/hip/package_windows.py` | `strata.exe`, `strata-device.exe`, the ROCm DLLs and the rocBLAS/hipBLASLt kernels of the zip (`--tree`) | no Windows AMD test machine yet; with `STRATA_NO_VISION` unset it also builds the two picture readers (`strata-vision.exe` on the CPU, `strata-vision-hip.exe` on the GPU) | +| `windows-x86_64` cuda, variant `cuda13` | the owner PC, the same cmake line as the release zips (CUDA 13.0, `75;86;89;120` real + `120` PTX, `STRATA_PORTABLE=ON`) | `strata.exe`, `strata_kq.dll`, `strata-vision.exe` (built for this variant's archs by `packaging/build_vision_windows.bat cuda13 `, which writes `vision_archs` into BUILD.json), `BUILD.json` | `min_driver` nvidia 580.65 | +| `windows-x86_64` cuda, variant `cuda12` | the owner PC (CUDA 12.9, `60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same four files, with its own `strata-vision.exe` (`packaging/build_vision_windows.bat cuda12 `: sm_60 .. sm_89; the CUDA 13 reader has no Pascal code) | experimental, `min_driver` nvidia 575.51; the CUDA 13 build wins when both fit | +| `windows-x86_64` hip | the owner PC, `tools/hip/build_windows.bat` (ROCm 10.2 nightly wheels) then `tools/hip/package_windows.py` | `strata.exe`, `strata-device.exe`, the ROCm DLLs and the rocBLAS/hipBLASLt kernels of the zip (`--tree`) | with `STRATA_NO_VISION` unset it also builds the two picture readers (`strata-vision.exe` on the CPU, `strata-vision-hip.exe` on the GPU) | | `linux-x86_64` cuda, variant `cuda13` | a Linux box with the CUDA 13 toolkit, `VARIANT=cuda13 packaging/build_cuda_linux.sh` | `strata`, `libstrata_kq.so`, `strata-vision` (built by the same script for the same archs, `BUILD_VISION=0` for a text-only package), `libcudart/libcublas/libcublasLt` (13.x) | the GPU driver's `libcuda` stays the PC's | -| `linux-x86_64` cuda, variant `cuda12` | a Linux box with CUDA 12.9, `VARIANT=cuda12 packaging/build_cuda_linux.sh` (`60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same files with the 12.x libraries (the reader has sm_60 / 61 / 70 too) | experimental, `min_driver` nvidia 575.51; the Pascal and Volta package (the 0.2.1 one lacked sm_60 / 61 / 70, #3503); `package_engine.py` refuses a cuda12 package without them | +| `linux-x86_64` cuda, variant `cuda12` | a Linux box with CUDA 12.9, `VARIANT=cuda12 packaging/build_cuda_linux.sh` (`60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same files with the 12.x libraries (the reader has sm_60 / 61 / 70 too) | experimental, `min_driver` nvidia 575.51; the Pascal and Volta package; `package_engine.py` refuses a cuda12 package without them | | `linux-x86_64` hip | a box with ROCm (the Python wheels are enough) | `strata`, the HIP runtime, hipBLAS/hipBLASLt/rocBLAS, comgr + libLLVM, `rocblas/library` and `hipblaslt/library` for the listed gfx archs; `packaging/build_hip_linux.sh` builds it with both picture readers | | `linux-x86_64` hip, variant `rx6000` (RX 6000, gfx1030 / gfx1031) | `VARIANT=rx6000 packaging/build_hip_linux.sh` in the jammy chroot, ROCm 7.13.0a20260515 `gfx103X-all` wheels | the same set as the other HIP packages, with `rocblas/library` for gfx1030 / gfx1031; no `hipblaslt/library` kernels exist for them and no `tuning/` table | built, not tested on a real RX 6000 card (the manifest notes say so). `windows-x86_64` hip, variant `rdna2`: the same archs from `tools/hip/build_windows.bat` with `STRATA_HIP_ARCHS=gfx1030;gfx1031`, packaged with `--variant rdna2`. A PC with an RX 7000 / 9000 card beside an RX 6000 one never takes it (`engine_component.hip_preference`) | | `linux-x86_64` hip, gfx1103 (Radeon 780M / Phoenix iGPU) | `packaging/build_hip_gfx1103.sh` on the mini PC (TheRock gfx110X-all SDK) | the same set as the gfx1200/1201 package, with `rocblas/library` and `hipblaslt/library` for gfx1103 (600 MiB) | experimental, unvalidated: the iGPU resets now and then (MES REMOVE_QUEUE); never kill the engine mid-request; run configs set `STRATA_NO_ARENA_THP=1`. `engine_component.resolve` picks a HIP package only when its `archs` list a gfx target the Linux kfd topology reports | | `linux-x86_64` sycl, `archs` `intel-arc` | a Linux box with oneAPI DPC++ 2026.1 and `patchelf`, `packaging/build_sycl_linux.sh` (JIT from SPIR-V; ocloc cannot build the Alchemist FP64 kernels ahead of time, so one JIT package serves A-series and B-series) | `strata`, libsycl, the Unified Runtime adapters, the oneMKL SYCL libraries, libimf / libsvml / libirng / libintlc (about 650 MiB); every ELF file has RUNPATH `$ORIGIN` (Intel's libraries ship without it), so `env -i ./strata --version` runs | the Level Zero loader and compute runtime are the PC's driver; the user must be in the `render` group (`strata doctor` warns). No Windows package: `BACKEND_MATRIX` allows sycl on `linux-x86_64` only | -| `macos-arm64` metal | an Apple Silicon Mac | `strata`, `strata-vision` (`packaging/build_vision_mac.sh`; `package_engine.py` refuses a macOS package without it, #3697), `metal/kernels/*.metal`, `metal/engine-msl/*.metal`, `metal/ggml-common.h`, `metal/data/*.bin` | the Metal engine compiles its kernel sources when it starts; since S1 it finds them beside the executable (`metal/src/resource_path.h`: environment variable, then `/metal/`, then the compiled-in source path) | +| `macos-arm64` metal | an Apple Silicon Mac | `strata`, `strata-vision` (`packaging/build_vision_mac.sh`; `package_engine.py` refuses a macOS package without it), `metal/kernels/*.metal`, `metal/engine-msl/*.metal`, `metal/ggml-common.h`, `metal/data/*.bin` | the Metal engine compiles its kernel sources when it starts; it finds them beside the executable (`metal/src/resource_path.h`: environment variable, then `/metal/`, then the compiled-in source path) | `packaging/collect_libs.py` gathers the Linux libraries (an `ldd` closure with the dlopen'd extras named, kernel data filtered by GPU architecture) into the folder `package_engine.py` packages; libraries the distribution provides are listed in @@ -119,16 +125,14 @@ with the package). `packaging/stage_tuning.py` does it for the Linux build scrip `package_engine.py` refuses an AMD stage without `hipblaslt_version` or with a table for another version or architecture. The server's start (`serve/server.py with_component_engine`, rule in `strata_control/hipblaslt_tuning.py`) picks the table of the card and sets `STRATA_HIPBLASLT_TUNING` for the engine; a version mismatch, an unreadable card or a missing version are one log line and plain hipBLAS. -`STRATA_HIPBLASLT_TABLE=on|off` (or `"hipblaslt_table"` in the run config) overrides; unset it follows `AUTO_USES_TABLE`, which is off until the -owner decides, because a table changes the rounding of the prompt's GEMMs (docs/AMD_HIP.md, Tuning table). Tests: `tests/terminal/test_hipblaslt_tuning.py`. +`STRATA_HIPBLASLT_TABLE=on|off` (or `"hipblaslt_table"` in the run config) overrides; unset it follows `AUTO_USES_TABLE`, which is off by default, because a table changes the rounding of the prompt's GEMMs (docs/AMD_HIP.md, Tuning table). Tests: `tests/terminal/test_hipblaslt_tuning.py`. ## The picture reader must match the package `package_engine.py` (the command line, not the library call) checks the reader before it packages: * a CUDA package with `strata-vision` needs `vision_archs` in BUILD.json, and every `sm_NN` of the package must be in it. A reader reused from an - older build was the cause of #3669 (the cuda12 package on a Pascal card: the engine ran, the reader stopped with "device kernel image is - invalid"). `--allow-stale-vision` packages it anyway and prints which archs are missing; + older build fails on cards it was not built for. `--allow-stale-vision` packages it anyway and prints which archs are missing; * a macOS package without `strata-vision` is refused (`--no-picture-reader` ships a text-only Mac package on purpose); * a CUDA package with no reader is allowed: it answers text only, the app says "the engine installed here does not include the picture reader yet" and keeps Read pictures off. @@ -146,3 +150,10 @@ and the CPU reader as `vision.cpu_exe`, and the GPU reader does not start (the c busy) the server starts the CPU reader by itself and prints one line. Both programs have RUNPATH `$ORIGIN` on Linux; on Windows the ROCm DLLs they import sit beside the programs or in `rocm/bin` (the engine's PATH entry, `lib_dirs`). `strata_control/vision_reader.py` holds the choice; `tools/test_vision_reader.py` the tests. + +## History / older measurements + +- **Stale picture reader (#3669):** a cuda12 package reused a reader built for newer cards. On a Pascal card the engine ran and the reader stopped with + "device kernel image is invalid". `package_engine.py` now checks `vision_archs`. +- **Missing sm_60 / 61 / 70 (#3503):** the first cuda12 package lacked Pascal and Volta code. `package_engine.py` now refuses a cuda12 package without them. +- **Mac without a reader (#3697):** a macOS package once shipped without `strata-vision`. It is refused unless `--no-picture-reader` is given. diff --git a/packaging/README.md b/packaging/README.md index 9020c6129..651022101 100644 --- a/packaging/README.md +++ b/packaging/README.md @@ -1,42 +1,51 @@ # Strata packaging: targets, recipes and locks -Revision: Strata 0.2.0. This directory holds the per-target build recipes and the machine-readable -manifests. **It describes what can be built where, and it is explicit about what has *not* been -built or tested.** Nothing here is a support claim. +This page is for developers who build the desktop app, the Terminal archive and the engine packages. To install Strata, use +[docs/INSTALL.md](../docs/INSTALL.md). It describes the current release (app and Terminal 0.2.5, Control and engines 0.1.50). +How packages are signed and published is in [UPDATES.md](UPDATES.md), and the package format is in [ENGINE-COMPONENT.md](ENGINE-COMPONENT.md). -The installers carry the engine: the desktop installers hold an engine package for their platform and graphics card, and the first -start copies it into the per-user component folder after checking every file against its manifest. A new install therefore starts a -model without a download; later engines come from the signed feed ([UPDATES.md](UPDATES.md)). The Terminal install is the other -route: one line, and the first model brings the engine for that PC from the same release. +Release assets show what was published; the platform guides and retained reports show what was tested. A published package does not +qualify every GPU, model or feature on its platform. + +## Quick start + +| I want to | See | +| --- | --- | +| Build the desktop app | [Desktop app](#desktop-app) | +| Build an engine package | [ENGINE-COMPONENT.md](ENGINE-COMPONENT.md#building-a-package) | +| Build the Terminal archive | [The Terminal archive](#the-terminal-archive) | +| Sign and publish a release | [UPDATES.md](UPDATES.md#releasing) | +| Test an update against a local feed | [UPDATES.md](UPDATES.md#testing-locally) | ## The target matrix -A backend name alone is not a target. Each row is an OS x CPU ABI x backend tuple, and a row is -`qualified` only after the acceptance checklist has actually run on it. +Each row is an OS, CPU ABI and backend. A row is `qualified` only after the acceptance checklist +([below](#what-a-platform-must-produce-before-it-is-qualified)) has run on it. -| Target tuple | App artifact | Engine package | Status | +| Target | App artifact | Engine package recipe | Status | | --- | --- | --- | --- | -| `macos-arm64-metal` | `.app` + `.dmg` (Tauri) | Metal `strata`, bundled | **built.** The app's self-test passed from inside the mounted DMG and the engine's 17 kernel-parity suites pass on an Apple M4 GPU. A large model does not fit the 16 GiB test Mac. Carries `strata-vision` (Metal), built by `packaging/build_vision_mac.sh`. Signing and notarising: `packaging/mac/sign-and-notarize.sh` (see [mac/README.md](mac/README.md)); it has been run with stand-ins for Apple's service, not yet against the real one. | -| Terminal `darwin-arm64` | self-contained `.tar.gz` | any | **built and verified.** Runs from a moved directory in a clean environment with no usable system Python. | -| `windows-x86_64-cuda` | NSIS `.exe` (Tauri) | `cuda13` and `cuda12` packages | **built and tested** on a Windows PC with an NVIDIA card: silent install with a PATH of System32 only, a real model, an agent chat. | -| `windows-x86_64-hip` | NSIS `.exe` (Tauri) | `tools/hip/build_windows.bat` | **built; never run on a real AMD card.** | -| `linux-x86_64-cuda` | AppImage (Tauri) | `setup.py`; built in a glibc 2.35 chroot | **cuda13 built and tested** (RTX 3060 and RTX A4000: app in the browser, a real model, a picture). cuda12 built. | -| `linux-x86_64-hip` | AppImage (Tauri) | `packaging/build_hip_linux.sh` | **RX 9000 (gfx1200 / gfx1201) built and tested** on a Radeon AI PRO R9700. **RX 7000 (gfx1100 / 1101 / 1102) built; not run on an RX 7000 card.** **RX 6000 (gfx1030 / 1031, 0.2.2, engine package `rx6000` only) built; not tested on a real RX 6000 card.** | -| `linux-x86_64-sycl` | | `packaging/build_sycl_linux.sh` (one JIT package for Alchemist and Battlemage) | **engine package built; tested on an Arc A750** (Intel Arc is an official platform since 0.1.40.2, docs/INTEL_ARC.md). Arc B-series (Battlemage) cards are supported and untested by us: no B-series card was available for the package (the B70 tests of 2026-10 were with the source build), and the Terminal says so once and tells the person how to report a problem. `sycl/setup_intel.py` (Docker) remains the source-build route. | -| `windows-x86_64-sycl` | none | none | **gated.** No native Windows SYCL path exists upstream; it needs a separate feasibility gate, not a recipe. | -| `macos-x86_64-*` | none | none | **unsupported.** The macOS engine is Apple-Silicon only (`metal/CMakeLists.txt` requires Apple Silicon for the Metal runtime). | +| `macos-arm64-metal` | `.app` and `.dmg` (Tauri) | Metal `strata`, bundled with `strata-vision` (`packaging/build_vision_mac.sh`) | Built. Self-test passes from the mounted DMG. Inference measurements are in [MAC.md](../docs/MAC.md). Signing: `packaging/mac/sign-and-notarize.sh` ([mac/README.md](mac/README.md)). | +| `darwin-arm64` Terminal | self-contained `.tar.gz` | any | Built and verified. Runs from a moved directory with no system Python. | +| `windows-x86_64-cuda` | NSIS `.exe` (Tauri) | `cuda13` and `cuda12` | Built and tested on a Windows PC with an NVIDIA card: silent install with a System32-only PATH, a real model, an agent chat. | +| `windows-x86_64-hip` | NSIS `.exe` (Tauri) | `tools/hip/build_windows.bat` | Published. Strix Halo inference is recorded in [STRIX_HALO.md](../docs/STRIX_HALO.md). This does not qualify every discrete Radeon card ([AMD_HIP.md](../docs/AMD_HIP.md)). | +| `linux-x86_64-cuda` | AppImage (Tauri) | `packaging/build_cuda_linux.sh`, built in a glibc 2.35 chroot | `cuda13` built and tested on RTX 3060 and RTX A4000 (app in the browser, a real model, a picture). `cuda12` built. | +| `linux-x86_64-hip` | AppImage (Tauri) | `packaging/build_hip_linux.sh` | `rx9070` (gfx1200 / gfx1201) built and tested on a Radeon AI PRO R9700. `rx7900` (gfx1100 / 1101 / 1102) built, not run on an RX 7000 card. `rx6000` (gfx1030 / 1031) built, not tested on a real RX 6000 card. | +| `linux-x86_64-sycl` | none | `packaging/build_sycl_linux.sh` (one JIT package for Alchemist and Battlemage) | Engine package built and tested on an Arc A750. Battlemage is supported and untested by us. | +| `windows-x86_64-sycl` | none | none | Gated. No native Windows SYCL path exists upstream. | +| `macos-x86_64-*` | none | none | Unsupported. The macOS engine is Apple Silicon only. | -**Cross-compilation is not attempted for the desktop app.** Tauri bundles are per-OS and the -Windows installer must be produced on Windows. What *is* portable is -everything below the carrier: the Python contracts, the owner transport, the checks, and the engine -build recipes. +**DeepSeek V4.1-Flash.** Three packages carry engine 0.1.50 with the DeepSeek engine `strata-dsv41`: Windows NVIDIA `cuda13`, +Linux NVIDIA `cuda13` and Linux AMD `rx9070`. All other packages carry engine 0.1.50. Build `strata-dsv41` with +`packaging/build_dsv41_cuda.sh` (Linux) or `packaging/build_dsv41_windows.bat` (Windows), then package it with +`package_engine.py --require-dsv41`. The DeepSeek engine can also be a separate download (feed member `dsv41`, see [UPDATES.md](UPDATES.md#the-feed)). + +Cross-compilation is not attempted for the desktop app. Tauri bundles are per OS, and the Windows installer is built on Windows. +The Python contracts, the owner transport, the checks and the engine build recipes are portable. ## The private Node runtime -The desktop bundle ships a pinned standalone Node beside the App Host, because the packaging -requirement is that an end user supplies no runtime. It is **not committed**: it is a ~117 MB -platform binary, it cannot be reviewed in a diff, and it is an input rather than source. Fetch and -verify it instead: +The desktop bundle ships a pinned standalone Node beside the App Host, so an end user supplies no runtime. It is not committed +(about 117 MB, an input rather than source). Fetch and verify it: ```sh apps/desktop/scripts/fetch-node.sh # host platform @@ -44,11 +53,9 @@ apps/desktop/scripts/fetch-node.sh --target linux-x64 apps/desktop/scripts/fetch-node.sh --check # verify what is installed ``` -The script refuses an archive whose SHA-256 does not match the digest recorded for that target. The -digests are taken from nodejs.org's own `SHASUMS256.txt` for the pinned version, so they are not -derived from the same response as the payload. **To bump the Node version: change -`STRATA_NODE_VERSION`, add the new digests, and re-run `--check`** — the script will refuse an -unknown target's archive rather than installing it unverified. +The script refuses an archive whose SHA-256 differs from the digest recorded for that target. The digests come from nodejs.org's +`SHASUMS256.txt` for the pinned version. To bump the version, change `STRATA_NODE_VERSION`, add the new digests and run `--check`. +`fetch-node.ps1` is the Windows equivalent. ## Desktop app @@ -56,104 +63,95 @@ unknown target's archive rather than installing it unverified. cd apps/desktop ./scripts/fetch-node.sh # the sidecar (once) npm install --prefix ../host && npm --prefix ../host run build # the UI -cargo tauri build --bundles app # -> Strata.app - -# macOS DMG (headless: Tauri's default step needs a Finder session) -./scripts/make-dmg.sh - -# Windows (on Windows): produces an NSIS per-user installer -cargo tauri build --bundles nsis - -# Linux (on Linux): AppImage plus a deb -cargo tauri build --bundles appimage deb +cargo tauri build --bundles app # macOS: Strata.app +./scripts/make-dmg.sh # macOS DMG (headless; Tauri's default step needs a Finder session) +cargo tauri build --bundles nsis # Windows, on Windows: per-user NSIS installer +cargo tauri build --bundles appimage deb # Linux, on Linux ``` -Windows installer policy is set in `tauri.conf.json`: `installMode: currentUser`, so the installer -is **per-user and needs no elevation**. Uninstall retention (checklist P10) is a property of *where -the app keeps its data*, not of an installer flag: models, profiles and sessions live outside the -install directory (`~/.strata` and the platform state directories), so removing the application -cannot remove them. The packaging config was validated against this Tauri version's schema — an -earlier draft used `perMachine`/`oneClick`/`deleteAppDataOnUninstall`, which `tauri-utils` 2.10 does -not declare and which are rejected outright rather than ignored. +The Windows installer is per-user and needs no elevation (`installMode: currentUser` in `tauri.conf.json`). Models, profiles and +sessions live outside the install directory (`~/.strata` and the platform state directories), so uninstalling the app does not remove them. -**The `dmg`/`app` targets were rebuilt and the packaged self-test re-run (8/8) after the Windows and -Linux targets were added**, so the added targets do not affect the macOS artifact. +Build boxes build without a signing key: the default `tauri.conf.json` has `createUpdaterArtifacts: false`. Signing happens on the +release PC ([UPDATES.md](UPDATES.md#releasing)). ## Strata Control and the engine in the installed app -The installer carries two things that start a model, and neither fetches anything when it runs. +The installer carries Control, and on a Mac the Metal engine too. Neither fetches anything when it runs. -**Control** (`/control`, built by `apps/desktop/scripts/stage-control.py`): the same bundle as the Terminal -archive, the pinned CPython 3.12.15, the hash-locked wheels, `strata_control`, `strata_cli` and `serve`, plus a -`BUNDLE.json` with the interpreter and lock digests. The host finds it beside itself (or through `STRATA_CONTROL_DIR`), -runs it on its own interpreter (`PYTHONNOUSERSITE=1`, no `PYTHONHOME`/`PYTHONPATH`) and starts a model with -`strata profile create|use-config` + `strata runtime start`, so the engine runs under the owner runtime (loopback TCP and a -256-bit token on Windows, a private socket elsewhere; a Job Object / process group; leases) and a second client (the -Terminal) attaches to the same instance. +**Control** (`/control`, built by `apps/desktop/scripts/stage-control.py`) is the same bundle as the Terminal archive: +the pinned CPython 3.12.15, the hash-locked wheels, `strata_control`, `strata_cli` and `serve`, plus a `BUNDLE.json` with the +interpreter and lock digests. The host finds it beside itself or through `STRATA_CONTROL_DIR`, runs it on its own interpreter +(`PYTHONNOUSERSITE=1`, no `PYTHONHOME` or `PYTHONPATH`) and starts a model with `strata profile create|use-config` and +`strata runtime start`. The engine then runs under the owner runtime (loopback TCP and a 256-bit token on Windows, a private +socket elsewhere; a Job Object or process group; leases), and a second client such as the Terminal can attach to the same instance. -The app starts each engine with `runtime start --die-with-pid `, so the engine ends with the host, and the host ends -with the shell (a kill-on-close Job Object around the app on Windows; the host watching its parent everywhere). The app is a -single instance: a second launch focuses the open window. One app can run several engines (profiles `app-1`, `app-2`, ...). -docs/CLOSING.md has every close path and how each is tested. +The app starts each engine with `runtime start --die-with-pid `, so the engine ends with the host. The host ends with the shell +(a kill-on-close Job Object on Windows; the host watches its parent everywhere). The app is a single instance: a second launch focuses +the open window. One app can run several engines (profiles `app-1`, `app-2`, ...). Every close path is in [docs/CLOSING.md](../docs/CLOSING.md). -**The engine** comes with the installer (`/engine-bundle/`, the same layout as below); the first start checks it against its own manifest and copies it into the per-user component folder, where Control finds it like any engine that `strata update` installed. It is a component in the per-user component folder -(`%LOCALAPPDATA%\Strata\components`, `$XDG_DATA_HOME/strata/components`, or `STRATA_COMPONENTS_DIR`): +**The engine.** The Windows and Linux installers carry no engine. On the first start the app shows the engine for the card with its size +and one Download button (with progress and resume), or installs an engine file the person already has ("I already have the file"). +The Mac dmg keeps its small Metal engine in `/engine-bundle/`. Either way the engine ends up in the per-user +component folder, after the package is checked against its manifest (`%LOCALAPPDATA%\Strata\components`, `$XDG_DATA_HOME/strata/components`, or `STRATA_COMPONENTS_DIR`), +where Control finds it like any engine that `strata update` installed: - components/engine/--/ - engine-component.json schema 2: id/name "strata-engine", version, platform, backend, archs, min_driver, - build_id, min_runtime (optional), entrypoint, files [{path, sha256, bytes}], - signature (null for a local package, set by the release feed); full table and the checks in ENGINE-COMPONENT.md + components/engine/--[-]/ + engine-component.json the manifest (see ENGINE-COMPONENT.md) strata[.exe] *.dll | *.so the binary and the GPU runtime libraries beside it -A run config names it as `"exe": "component:strata-engine"`; at start `strata_control/engine_component.py` picks the package -for this PC (platform, then the backend its GPU needs: nvidia-smi / rocminfo / sycl-ls, or `STRATA_ENGINE_BACKEND`; CUDA -architectures, `min_driver` and `min_runtime` are checked; the newest version wins), checks every file's size and -SHA-256 (a start hashes the whole package; a mismatch is `EngineIntegrityError` and nothing starts) and puts the package -folder first on the library path. -Nothing matching is `EngineMissing`: `strata serve` ends with exit 5 and "No engine for this PC was found in ... The app never downloads an engine -while it starts or serves", and no fetch. That is the start itself. Everything before it is a warning, not a refusal: `strata launch` says in plain -words that it could not find an engine for this computer, asks whether to go on (Enter means yes), downloads the model and ends with the next step -(exit code 0, nothing started), and the app shows the same warning with an "Update engine" button in Settings. `packaging/package_engine.py` makes a local package from a build folder. +A run config names it as `"exe": "component:strata-engine"`. At start, `strata_control/engine_component.py` picks the package for this PC +(platform, then the backend its GPU needs, found with nvidia-smi, rocminfo or sycl-ls, or `STRATA_ENGINE_BACKEND`), checks every +file's size and SHA-256 and puts the package folder first on the library path. A mismatch is `EngineIntegrityError` and nothing starts. -**No acquisition at run time.** `STRATA_NO_ACQUIRE=1` (set by the Tauri shell for the host, and for Control by the host) -makes `strata_control/acquisition.py` refuse pip, downloads and compiles; on the host side `server/acquisition.mjs` -refuses the llama.cpp and cloudflared fetches (a connector placed in `components/cloudflared/` is used as it is). Models the -user chooses to download are a separate, explicit action. `apps/host/tests/no-acquire.test.mjs` runs the host under -`tests/helpers/acquire-spy.mjs` and fails on any outbound request or installer process; -`apps/desktop/scripts/installed-flow.mjs` drives an installed app end to end (stub engine anywhere, real engine with `--components`). +When no package matches, `strata serve` ends with exit 5 and "No engine for this PC was found". It never fetches. Before that point +everything is a warning: `strata launch` says it could not find an engine, asks whether to go on, downloads the model and ends with +the next step (exit 0, nothing started). The app shows the same warning with an "Update engine" button in Settings. +`packaging/package_engine.py` makes a local package from a build folder. -## What every platform must produce before it can be called qualified +**No acquisition at run time.** `STRATA_NO_ACQUIRE=1` (set by the Tauri shell for the host, and by the host for Control) makes +`strata_control/acquisition.py` refuse pip, downloads and compiles. On the host side `server/acquisition.mjs` refuses the llama.cpp +and cloudflared fetches (a connector placed in `components/cloudflared/` is used as it is). Models the user chooses to download are a +separate, explicit action. `apps/host/tests/no-acquire.test.mjs` fails on any outbound request or installer process. +`apps/desktop/scripts/installed-flow.mjs` drives an installed app end to end. + +## The Terminal archive + +The Terminal bundles a private Python and its dependencies for `windows-x86_64`, `linux-x86_64` and `darwin-arm64` +(`package_terminal.py --target ...`, built on the target OS). + +- The interpreter is python-build-standalone CPython 3.12.15, digest-checked at build time. +- Packages come from `packaging/locks/runtime--cp312.txt` and are installed with + `pip install --require-hashes --no-deps --only-binary=:all:` at build time only. `lock_wheels.py` makes the lock; review its diff when a pin changes. +- Wrappers: `bin/strata.cmd` and `bin/strata.ps1` on Windows, `bin/strata` on POSIX. `START-HERE.bat`, `SETUP.bat`, `UPDATE.bat`, `setup.sh` + and `update.sh` use a `python/` folder beside them when there is one. +- `smoke_terminal.py ` extracts an archive into a clean environment (a path with a space, a PATH with no Python) and runs + `--version`, `doctor`, `profile list`, `install` and `serve --engine mock`. +- Check each archive's `BUNDLE.json` and lock digests. An old archive is not upgraded when the source lock changes. + +## What a platform must produce before it is qualified 1. The app launches and the window serves the supplied UI from the host's origin. -2. The owner transport works: one owner per installation, a second client attaches, a stale owner - can be recovered, and no client can stop a process it does not own (R03/R04). +2. The owner transport works: one owner per installation, a second client attaches, a stale owner can be recovered, and no client can stop a process it does not own. 3. Real inference on a supported pack, with the engine build hash and the model hashes recorded. -4. `docs/work/QUALIFICATION-KIT.md` run end to end, with raw evidence retained. +4. `docs/work/QUALIFICATION-KIT.md` run end to end, with raw evidence kept. -## Honest gaps in this directory +## Release checks and gaps -* There is no CI. Every recipe here is written to be run by hand; making them automated is separate - work. -* Signing needs the owner's credentials, which are never in this tree. The update feed and every engine package are signed with the feed key - (`feed_tool.py`, [UPDATES.md](UPDATES.md)); the app installers with the Tauri updater key. The Mac app and Terminal archive are signed and - notarised on a Mac that holds the Developer ID certificate, with `packaging/mac/sign-and-notarize.sh` ([mac/README.md](mac/README.md)); for a release, `--updater-archive` also packs the notarised app as `Strata.app.tar.gz` for the in-app Mac update (the one `tauri build` makes holds the unsigned app), signed with the Tauri key on the PC and added with `feed_tool.py set-app-platform`. No - Windows code-signing certificate is configured in this tree; Windows may warn about an installer that is not signed. -* No font or asset redistribution evidence has been gathered (D05). The app currently ships the - brand assets from the supplied UI ZIP, which is fine for an internal candidate and not sufficient - for redistribution. -* The terminal distribution bundles a private Python runtime and its dependencies for `windows-x86_64`, - `linux-x86_64` and `darwin-arm64` (`package_terminal.py --target ...`, built on the target OS). The interpreter is - the pinned python-build-standalone CPython 3.12.15 (digest-checked at build time); the packages come from - `packaging/locks/runtime--cp312.txt`, installed with `pip install --require-hashes --no-deps - --only-binary=:all:` at BUILD time only (`lock_wheels.py` makes the lock; review its diff when a pin changes). - Wrappers: `bin/strata.cmd` / `bin/strata.ps1` on Windows, `bin/strata` on POSIX; START-HERE.bat, SETUP.bat, - UPDATE.bat, setup.sh and update.sh use a `python/` folder beside them when there is one. - `smoke_terminal.py ` extracts an archive into a clean environment (a path with a space, a PATH with no - Python) and runs `--version`, `doctor`, `profile list`, `install` and `serve --engine mock`. The macOS archive - predates the lock: build it again on a Mac to get the hash-pinned install (the lock is already here). -* `.gitignore` coverage for build output was added late. `apps/desktop/src-tauri/target` and the - Node sidecar were briefly committed, which left ~324 MB in the git object store. The files are - untracked now and the ignore rules are in place, but the history is not rewritten: that is a - deliberate choice, since rewriting published history is worse than a large local repository. A - developer cloning this branch gets the current tree, and the sidecar arrives via - `fetch-node.sh`. +- CI covers scripts, Terminal checks, Metal builds and Docker images ([.github/workflows](../.github/workflows)). It does not replace native desktop + packaging, signing or real-GPU acceptance on every target. Some recipes still run by hand. +- Signing needs the owner's credentials, which are never in this tree. The feed and every engine package are signed with the feed key + (`feed_tool.py`); the app installers with the Tauri updater key ([UPDATES.md](UPDATES.md)). The Mac app and Terminal archive are signed and + notarised on a Mac that holds the Developer ID certificate. No Windows code-signing certificate is configured, so Windows may warn about the installer. +- Check the licenses and provenance of bundled fonts and assets when packaging. The early D05 checklist lacked redistribution evidence; + treat that as an open checklist finding, not an audit of every release asset. + +## History / older measurements + +- **Tauri config:** an early draft used `perMachine`, `oneClick` and `deleteAppDataOnUninstall`. The Tauri schema rejects them; the + current config uses `installMode: currentUser`. +- **Repository size:** `apps/desktop/src-tauri/target` and the Node sidecar were briefly committed, which left about 324 MB in the git object + store. They are untracked and ignored now. History was not rewritten. +- **Mac packaging tests:** the first DMG self-test (8/8) and the engine's 17 kernel-parity suites ran on an Apple M4 GPU in a 16 GiB Mac, with service stand-ins for signing. + Later inference measurements on larger Macs are in [MAC.md](../docs/MAC.md). +- **Intel Arc:** official since 0.1.40.2. The 2026-10 B70 tests used the source build, not the engine package. `sycl/setup_intel.py` (Docker) remains the source-build route. diff --git a/packaging/UPDATES.md b/packaging/UPDATES.md index 197af8c9a..74258e17b 100644 --- a/packaging/UPDATES.md +++ b/packaging/UPDATES.md @@ -1,10 +1,14 @@ # Updates: the signed feed, the component updater, the app updater -Everything that downloads software on a user's PC lives behind one explicit step: `strata update` in the Terminal, or -"Update engine" / "Update Strata" in the app's Settings. Starting a model and serving never download anything +This page is for developers who build, sign and publish releases. Users update with [INSTALL.md](../docs/INSTALL.md#updating). +It covers the three updated parts, the feed format, what a PC checks, and the release steps. + +Everything that downloads software on a user's PC sits behind one explicit step: `strata update` in the Terminal, or +"Update engine" or "Update Strata" in the app's Settings. Starting a model and serving never download anything (`STRATA_NO_ACQUIRE` stays on for them; the update step is a separate Control process that is allowed to). Engines are -separate from the app and ship more often: an app-only update downloads 0 engine bytes (the update installers are built without the engine), an engine update downloads only the -package for that PC's GPU. **Models and saved tuning presets are never downloaded, replaced or deleted by any update.** +separate from the app and ship more often. An app-only update downloads no engine bytes, because the Windows and Linux installers are built +without the engine. An engine update downloads only the package for that PC's GPU. +**Models and saved tuning presets are never downloaded, replaced or deleted by any update.** ## The three things that are updated @@ -19,33 +23,41 @@ Two separate private keys, both kept by the owner, neither in git, neither the s * **feed key**: Ed25519, signs `strata-feed.json` and every engine package manifest. * **Tauri updater key**: minisign (what `tauri signer generate` makes), signs the app installers. + ## The feed (`strata-feed.json` + `strata-feed.json.sig`) `strata-feed.json.sig` is `{"alg": "ed25519", "key_id": "<16 hex>", "value": ""}` over the **exact bytes** of -`strata-feed.json` (no canonicalisation). The feed: +`strata-feed.json` (no canonicalisation). The annotated example below illustrates the format; it is not a current signed feed. +Comments are for this document only; actual feed files must be strict JSON. -```json +```jsonc { "schema_version": 1, "feed": "strata-updates", "channel": "stable", "sequence": 12, // goes up with every feed; an older one than the last accepted is refused "generated_at": "2026-10-20T10:00:00Z", "expires_at": "2027-10-20T10:00:00Z", // expires_at may be null "app": { // exactly Tauri's static updater JSON, plus sha256/size per platform - "version": "0.2.0", "notes": "...", "pub_date": "2026-10-20T10:00:00Z", - "platforms": {"windows-x86_64": {"url": "app/Strata_0.2.0_x64-setup.exe", "signature": "", "sha256": "...", "size": 71000000}} + "version": "0.2.5", "notes": "...", "pub_date": "2026-10-20T10:00:00Z", + "platforms": {"windows-x86_64": {"url": "app/Strata-0.2.5-windows-x64-setup.exe", "signature": "", "sha256": "...", "size": 71000000}} }, - "control": {"version": "0.1.48", "packages": [{"platform": "linux-x86_64", "format": "tar.gz", "url": "control/...tar.gz", + "control": {"version": "0.1.50", "packages": [{"platform": "linux-x86_64", "format": "tar.gz", "url": "control/...tar.gz", "size": 0, "sha256": "...", "installed_bytes": 0, "entry": "strata.py"}]}, "engine": {"packages": [{ - "version": "0.1.48", "platform": "linux-x86_64", "backend": "cuda", "variant": "cuda13", + "version": "0.1.50", "platform": "linux-x86_64", "backend": "cuda", "variant": "cuda13", "archs": ["sm_75", "sm_86", "sm_89", "sm_120", "compute_120"], "min_driver": {"nvidia": "580.65"}, "min_runtime": null, - "build_id": "be885026f96a-cuda13.0", "format": "tar.gz", "url": "engine/strata-engine-0.1.48-linux-x86_64-cuda-cuda13.tar.gz", + "build_id": "be885026f96a-cuda13.0", "format": "tar.gz", "url": "engine/strata-engine-0.1.50-linux-x86_64-cuda-cuda13.tar.gz", "size": 505568970, // bytes of the archive "sha256": "...", // of the archive "manifest_sha256": "...", // of the package's engine-component.json (it holds every file's hash) - "installed_bytes": 840893419, "notes": ""}]} + "installed_bytes": 840893419, "notes": ""}]}, + "dsv41": {"packages": []} // optional: the separate DeepSeek engine packages, same entry shape as "engine" } ``` +The DeepSeek V4.1-Flash engine `strata-dsv41` reaches a PC in one of two ways. The engine package carries it beside `strata` +(`package_engine.py --require-dsv41`), or it is a separate package in the feed's `dsv41` member, packed with +`pack-engine --prefix strata-dsv41`. The separate package installs in `/engine-dsv41/` beside the engine, never inside it +(`strata_control/dsv41_engine.py`), and is fetched when DeepSeek is chosen. `STRATA_DSV41_EXE` points setup at a build of your own. + `url` is relative to the feed's own location (or an absolute `https://` address; a loopback `http://` is allowed so a local test feed works; a local feed folder may only name files inside itself). For a GitHub release the assets are flat: pack with `--url-prefix ""` and upload every file beside the feed. @@ -98,7 +110,7 @@ On an offline machine, outside any git checkout: npx tauri signer generate -w D:\strata-keys\tauri-updater.key # asks for a password; prints the public key Back both up (a lost feed key means every installed PC needs a build with a new key; a lost Tauri key means the same for the app). -Then, in git, once, in the commit that ships the first signed release (done for 0.2.0, key id 5d1bc23abbeffd0d): +Then, in git, once, in the commit that ships the first signed release (the release key id is `5d1bc23abbeffd0d`): * the `keys` entry from `feed-key.pub.json` into `strata_control/update_keys.json`; * the Tauri public key into `apps/desktop/src-tauri/tauri.conf.json` (`plugins.updater.pubkey`) and into `update_keys.json` as `app_minisign_public` @@ -110,7 +122,7 @@ without their key being part of a release build. A build whose `update_keys.json feed with "this build trusts no update key". A key rotation is a release that carries both public keys; the old private key signs that release, the new one signs the next. -## Releasing (when the owner says go) +## Releasing Keys live only on the release PC (`feed-key.json`, `tauri-updater.key`; the Tauri key has no password). Every build box builds WITHOUT a key: the default `tauri.conf.json` has `createUpdaterArtifacts: false`, so `tauri build` needs no signing key and makes the same installer bytes. @@ -129,9 +141,9 @@ All signing happens on the release PC, on the finished files. python packaging/feed_tool.py sign-package --key # every engine package (the Mac app bundles its UNSIGNED copy) python packaging/feed_tool.py pack-engine --out RELEASE/engine --url-prefix= - python packaging/feed_tool.py pack-control --version 0.1.48 --platform P --out RELEASE/control --url-prefix= # --format zip for windows-x86_64 + python packaging/feed_tool.py pack-control --version 0.1.50 --platform P --out RELEASE/control --url-prefix= # --format zip for windows-x86_64 # --url-prefix= (not --url-prefix "") because PowerShell 5.1 drops an empty argument. The Control version is the VERSION file - # (0.1.48), not the app version: a Control package newer than the installed one is installed by `strata update`. + # (0.1.50), not the app version: a Control package newer than the installed one is installed by `strata update`. # Pack Linux and Mac packages on Linux / the Mac or on Windows with this tool: it sets the execute bit from the file header either way. cd apps/desktop && npx tauri build --bundles nsis --config src-tauri/tauri.update.conf.json # Windows installer (no engine inside) # Linux: tauri build --bundles appimage (no engine inside) is the AppImage @@ -140,11 +152,11 @@ All signing happens on the release PC, on the finished files. python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target windows-x86_64 --version 0.2.5 --archive Strata-0.2.5-windows-x64-setup.exe --sig Strata-0.2.5-windows-x64-setup.exe.sig --notes "..." python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target linux-x86_64 --version 0.2.5 --archive Strata-0.2.5-linux-x86_64.AppImage --sig Strata-0.2.5-linux-x86_64.AppImage.sig python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target darwin-aarch64 --version 0.2.5 --archive Strata.app.tar.gz --sig Strata.app.tar.gz.sig - python packaging/feed_tool.py build-feed --out RELEASE --sequence N --key --engine RELEASE/engine --control RELEASE/control --app-json RELEASE/tauri-latest.json + python packaging/feed_tool.py build-feed --out RELEASE --sequence N --key --engine RELEASE/engine --control RELEASE/control --app-json RELEASE/tauri-latest.json [--dsv41 RELEASE/dsv41] python packaging/feed_tool.py verify-feed RELEASE --pub `set-app-platform` checks each `.sig` against the public key in `tauri.conf.json` over the exact bytes, so a file signed with another key, or changed after -signing, stops the release. The full installers are not in `latest.json` and need no Tauri signature (they are downloaded by hand). +signing, stops the release. The Mac dmg is not in `latest.json` (the updater uses `Strata.app.tar.gz`) and needs no Tauri signature; it is downloaded by hand. ### The Mac app @@ -163,7 +175,7 @@ Mach-O in the app with the Developer ID and notarises and staples it, which chan The Windows installer and the Linux AppImage need no extra step: their `.sig` files from `tauri build` are the ones to pass to `set-app-platform`. Re-signing the Mac app after the archive was made (a second `sign-and-notarize.sh` run) makes a new archive: sign and add it again. -`build-feed` also writes `RELEASE/install-manifest.txt`, the file the one-line installers read (`install.sh` L76, `install.ps1` L45): one line per Control package, +`build-feed` also writes `RELEASE/install-manifest.txt`, the file the one-line installers read: one line per Control package, `platform version url sha256 size`, taken from the `*.entry.json` files in `RELEASE/control`. The Control package is the Terminal that `install.sh` unpacks (it must hold `bin/strata`). `install.sh` unpacks with `tar -xzf` and `install.ps1` with `Expand-Archive`, so `windows-x86_64` must be packed with `--format zip` and the other platforms as `tar.gz`; `build-feed` leaves a pack in the wrong format out of the manifest and says so. Pack Control for every platform you want the one-line install to cover (`install.sh` knows `linux-x86_64` and `macos-arm64`, `install.ps1` `windows-x86_64`); @@ -185,3 +197,10 @@ Everything above runs against a folder or a loopback server: `tests/terminal/tes `tests/terminal/test_updater.py` (end to end against a loopback HTTP feed with stub engines: update, test-start, switch; a failing test start keeps the old engine; an interrupted download resumes; rollback; Control; the app installer's minisign check), `apps/host/tests/component-updates.test.mjs` (the Settings buttons). For a real engine on a box, build a package with `packaging/package_engine.py`, serve it from a local feed made by `packaging/make_local_feed.py`, and run `strata update --feed` against it. + +## History / older measurements + +- **Keys:** the feed key and the Tauri updater key were created once and the feed key's public half was committed in `strata_control/update_keys.json` + with the first signed release (0.2.0-beta). Terminals from before that build trust no key and refuse every feed. +- **Engine-less installers:** an app update once carried an engine. The Windows and Linux installers now drop `engine-bundle` + (`tauri.update.conf.json`), so one `latest.json` entry per OS serves every card variant. diff --git a/tools/dflash2/README.md b/tools/dflash2/README.md index ff5c71558..828252f32 100644 --- a/tools/dflash2/README.md +++ b/tools/dflash2/README.md @@ -1,20 +1,30 @@ -# DFlash2 acceptance experiment (research tooling, not shipped) +# DFlash2 acceptance experiment -Measures how many tokens per round the incoai DFlash2 drafter for Qwen3.8-Flash-Next would get on OUR quantized target, offline, from the hidden states -the engine dumps. Results and method: phaseA-tests/notes-dflash2.md. All scripts keep the paths of the run (C:\dflash2-work on the owner PC, C:\strata-test\dflash2 -on the test box): edit them before reuse. +Research tooling, not shipped with the app. It measures how many tokens per round the incoai DFlash2 drafter for +Qwen3.8-Flash-Next would accept on Strata's quantized target. The measurement is offline, from hidden states the +engine dumps. It is for developers who want to repeat or extend the experiment. -Engine side (both off unless the variable is set): +The scripts keep the paths of the original run (`C:\dflash2-work` on the development PC, `C:\strata-test\dflash2` on +the test box). Edit the paths before you reuse them. The method and results are in the project's internal notes +and are not part of this tree. -- `STRATA_TAP_FILE=` (+ `STRATA_TAP_LAYERS`, default `5,15,25,35,45`): `Verifier::record_window` copies each verified row's completed layer output (the mean of - the 4 hyper-connection streams, the quantity the drafter was trained on) and `Verifier::run` appends `pos, token, taps` records. -- `STRATA_SPEC_T_CYCLE=1,2,...,8` (CLI bench only): the verify window size cycles; the report line `window cost (ms/round)` gives the cost of each width. +## Engine variables -Scripts: +Both are off unless set. -- `mk_prompts.py` builds chat-template prompt ids with the server's own tokenizer and template. -- `orch.py [gen,genf,genfm,tap]` drives `run_one.ps1` on the test box: free generation (ids, MTP statistics), the same text with MTP only, and the - teacher-forced tap dump with oracle drafts (`--spec 8 --spec-oracle --spec-follow`, a 2-token prompt: a native pack never starts the speculative loop from a 1-token prompt). -- `dflash2_ref.py` is a plain-torch port of z-lab's `DFlash2DraftModel` (checked against it by `test_port.py`), with the target's embedding and head dequantized from the GGUF. -- `run_eval.py` (walk, a few block sizes), `run_all2.py` (every position, resumable: per-anchor correctness and the selector's confidences), `analyze2.py` and - `final_tables.py` (policies, speed estimates from the measured window costs). +| Variable | Effect | +| --- | --- | +| `STRATA_TAP_FILE=` | Writes `pos, token, taps` records. `Verifier::record_window` copies each verified row's completed layer output (the mean of the 4 hyper-connection streams, the quantity the drafter was trained on) and `Verifier::run` appends it. | +| `STRATA_TAP_LAYERS` | Layers to tap, default `5,15,25,35,45`. Used with `STRATA_TAP_FILE`. | +| `STRATA_SPEC_T_CYCLE=1,2,...,8` | CLI bench only. The verify window size cycles; the report line `window cost (ms/round)` gives the cost of each width. | + +## Scripts + +| Script | Purpose | +| --- | --- | +| `mk_prompts.py` | Builds chat-template prompt ids with the server's own tokenizer and template. | +| `orch.py [gen,genf,genfm,tap]` | Drives `run_one.ps1` on the test box: free generation (ids and MTP statistics), the same text with MTP only, and the teacher-forced tap dump with oracle drafts (`--spec 8 --spec-oracle --spec-follow`, a 2-token prompt, because a native pack never starts the speculative loop from a 1-token prompt). | +| `dflash2_ref.py` | Plain-torch port of z-lab's `DFlash2DraftModel`, with the target's embedding and head dequantized from the GGUF. `test_port.py` checks it against the original. | +| `run_eval.py` | Walks the tap file with a few block sizes. | +| `run_all2.py` | Evaluates every position, resumable: per-anchor correctness and the selector's confidences. | +| `analyze2.py`, `final_tables.py` | Policies, and speed estimates from the measured window costs. |