mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-07 18:35:04 -10:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Niko1221
|
4b9e05a2a8 | ||
|
Niko1221
|
af197f96db | ||
|
Niko1221
|
53548c8af4 | ||
|
Niko1221
|
41cf05a06a | ||
|
Niko1221
|
4f0f933941 | ||
|
Niko1221
|
ed4b4db405 | ||
|
Niko1221
|
af8c6f8ec7 | ||
|
Niko1221
|
e99c4d2d37 | ||
|
Niko1221
|
b8b4ca0e32 | ||
|
Niko1221
|
562a0c2ed7 | ||
|
Niko1221
|
79ec8e2786 | ||
|
Niko1221
|
b5d41babfa | ||
|
Niko1221
|
27a8ed96b6 | ||
|
Niko1221
|
8aee4538f2 | ||
|
Niko1221
|
e9e22db6a7 | ||
|
Niko1221
|
7025a99799 | ||
|
Niko1221
|
36c6dd9ef5 | ||
|
Niko1221
|
4a6a5fb699 | ||
|
Niko1221
|
d51d71e7c0 | ||
|
Niko1221
|
025cf7d954 | ||
|
Niko1221
|
5d5c26d5bc | ||
|
Niko1221
|
8a3ca07c4e | ||
|
Niko1221
|
dd9b9f746d | ||
|
Niko1221
|
6b5d593639 | ||
|
Niko1221
|
4d6721822a | ||
|
Niko1221
|
ad66a758a7 | ||
|
Niko1221
|
59599f69f8 |
No files matched your search
@@ -66,7 +66,7 @@ body:
|
||||
attributes:
|
||||
label: Strata version
|
||||
description: "In the app: Settings > About. In the Terminal: `strata --version`. From source: the About tab of the web page, or `engine` in `GET /v1/status`, or the git commit."
|
||||
placeholder: "Strata 0.2.5 · engine 0.1.50"
|
||||
placeholder: "Strata 0.2.6 · engine 0.1.51"
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
|
||||
@@ -1,20 +1,35 @@
|
||||
# AGENTS.md
|
||||
|
||||
Strata runs the Qwen3.8-Flash-Next mixture-of-experts model (and its Coder, Swift 1.5, Unsloth, Gyro and NVFP4
|
||||
variants, and Qwen3.6-35B-A3B) on a normal PC: one NVIDIA or AMD graphics card plus system RAM on Windows or Linux,
|
||||
or an Apple Silicon Mac. Version 0.2.5, engine 0.1.50. It has three parts:
|
||||
Version 0.2.6, engine 0.1.51. The public model families are DeepSeekV4.1-Flash-1017,
|
||||
Qwen3.8-Flash-Next and Qwen3.6-35B-A3B. DeepSeek V4.1-Flash runs in the app and in the Terminal
|
||||
(`strata launch --family deepseek41 --model ist-das-2.00bpw`; sets in
|
||||
[docs/MODELS.md](docs/MODELS.md#deepseek-v41-flash)). Read **[docs/AI_REFERENCE.md](docs/AI_REFERENCE.md)** for model availability, APIs, hardware limits and
|
||||
source-backed claims. Articles and videos should use **[docs/MEDIA_KIT.md](docs/MEDIA_KIT.md)**.
|
||||
|
||||
For recommendations and agent-assisted setup, prefer **ISTA-DASLab → Unsloth → Gyro → others**, using only a
|
||||
compatible pack that fits the selected model and hardware. ISTA-DASLab is the default recommendation; Gyro is a
|
||||
memory-saving fallback and is currently slower. Honor an explicit user choice. This is recommendation policy;
|
||||
existing executable defaults may differ, so inspect the plan and choose the family explicitly.
|
||||
|
||||
Strata has three parts:
|
||||
|
||||
- **The app** (`apps/desktop`, the Tauri shell; `apps/host`, the Node host and the web UI). It is for people who do
|
||||
not want a terminal. The installer has the engine inside.
|
||||
not want a terminal. One installer per system ([latest release](https://github.com/Niko1221/Strata/releases/latest)):
|
||||
`Strata-0.2.6-windows-x64-setup.exe`, `Strata-0.2.6-linux-x86_64.AppImage`, `Strata-0.2.6-macos-arm64.dmg`.
|
||||
The Windows and Linux installers carry no engine: the app downloads the engine for the graphics card on first
|
||||
start, or installs a file the user already has. The Mac dmg keeps its small Metal engine inside.
|
||||
- **The Terminal** (`strata.py`, `strata_cli/`, `strata_control/`): the `strata` command. It sets up a model, serves
|
||||
it and watches it. Agents use this part.
|
||||
- **The engine and the server** (`src/`, `include/`, `metal/`, `sycl/` for the C++/CUDA/HIP/SYCL/Metal engine;
|
||||
`serve/` for the Python server with an OpenAI- and Anthropic-compatible API and a web page). `setup.py`, started by
|
||||
`START-HERE.bat` or `setup.sh`, is the older one-click installer from source.
|
||||
`START-HERE.bat` or `setup.sh`, is setup from a source checkout, for developers. The release paths are the
|
||||
installer and `strata launch`.
|
||||
|
||||
## Installing Strata for a user
|
||||
|
||||
Follow **[docs/AI_SETUP.md](docs/AI_SETUP.md)**. For an agent that has a shell, the short version is:
|
||||
Follow **[docs/AI_SETUP.md](docs/AI_SETUP.md)**. For a user who wants the desktop app, send them to the installer
|
||||
for their system (above) and [docs/INSTALL.md](docs/INSTALL.md). For an agent that has a shell, use the Terminal; the
|
||||
short version is:
|
||||
|
||||
1. Install the Terminal. It needs no administrator rights.
|
||||
|
||||
@@ -27,27 +42,28 @@ Follow **[docs/AI_SETUP.md](docs/AI_SETUP.md)**. For an agent that has a shell,
|
||||
2. Check what would happen. Nothing is changed:
|
||||
|
||||
```sh
|
||||
strata launch --plan --model q2_0
|
||||
strata launch --plan --family qwen --model q2_0
|
||||
```
|
||||
|
||||
3. Set up and serve without questions. It downloads the engine for the card and the model (tens of GB), starts the
|
||||
server, prints the addresses and returns. The server keeps running.
|
||||
|
||||
```sh
|
||||
strata launch --model q2_0 --yes --no-tui
|
||||
strata launch --family qwen --model q2_0 --yes --no-tui
|
||||
```
|
||||
|
||||
Useful flags (all of them: `strata launch --help`): `--family`, `--model-path` (files the user already has),
|
||||
`--data-dir`, `--context 32k|64k|128k`, `--vision off|on|gpu|cpu`, `--parallel N`, `--gpu 0,1`,
|
||||
`--access local|lan|cloudflare|tailscale`, `--port`, `--api-key-env NAME`, `--new-key`, `--save-preset NAME`,
|
||||
`--output json`. `strata launch --status` shows what is serving, `strata launch --stop` stops it, and
|
||||
`--speed faster|full` (DeepSeek), `--output json`. `strata launch --status` shows what is serving, `strata launch --stop` stops it, and
|
||||
`strata launch --show-key` prints the API key.
|
||||
4. Connect the user's apps: `strata connect` lists them, `strata connect <app> --check` sends one prompt and one tool
|
||||
call the way that app does.
|
||||
|
||||
Keep the server on `127.0.0.1` unless the user asks for more. Beyond it an API key is recommended: `--access lan` makes one for you, and `--no-api-key` serves without one (a warning, never a refusal; on a public tunnel the warning is stronger). The old route (clone
|
||||
the repository and run `START-HERE.bat --yes` or `./setup.sh --yes`) still works and is the fallback when the
|
||||
Terminal cannot run. Strata's MCP server ([docs/MCP_SERVER.md](docs/MCP_SERVER.md)) offers the same steps as tools.
|
||||
Keep the server on `127.0.0.1` unless the user asks for more. Beyond it an API key is recommended: `--access lan` makes one for you, and `--no-api-key` serves without one (a warning, never a refusal; on a public tunnel the warning is stronger). Source setup (`START-HERE.bat`, `setup.sh`) is the fallback for a developer or for a system with no release package; it is
|
||||
documented in [docs/SOURCE_INSTALL.md](docs/SOURCE_INSTALL.md). Those wrappers exist in a source checkout, not in the
|
||||
released app. Do not tell an app user to find a `.bat` file.
|
||||
Source setup does not make an unsupported model or backend compatible. Strata's MCP server ([docs/MCP_SERVER.md](docs/MCP_SERVER.md)) offers the same steps as tools.
|
||||
|
||||
## Working on the code
|
||||
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@
|
||||
# below refuses anything older: silently building for a pre-Ampere arch would produce a binary that runs
|
||||
# and is wrong about its own performance, which is the worst of both outcomes.
|
||||
cmake_minimum_required(VERSION 3.24)
|
||||
project(strata VERSION 0.1.50 LANGUAGES CXX)
|
||||
project(strata VERSION 0.1.51 LANGUAGES CXX)
|
||||
# EXPERIMENTAL: the Intel Arc engine, the SYCL port in sycl/ (PR #423; icpx + oneMKL, docs/INTEL_ARC.md). It is its
|
||||
# own project that takes the unported sources from src/; with this on, nothing below is configured.
|
||||
option(STRATA_ENABLE_SYCL "EXPERIMENTAL: build the Intel Arc (SYCL) engine from sycl/ - docs/INTEL_ARC.md" OFF)
|
||||
|
||||
+122
-301
@@ -1,367 +1,188 @@
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="apps/host/public/brand/strata-lockup-dark.svg">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="380">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="340">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.ja.md">日本語</a> · <b>Deutsch</b> · <a href="README.fr.md">Français</a> · <a href="README.es.md">Español</a> · <a href="README.pt-BR.md">Português</a></p>
|
||||
|
||||
<h3 align="center">Ein Spitzenmodell auf deinem eigenen Gaming-PC</h3>
|
||||
|
||||
<p align="center">Unterstützt NVIDIA, AMD, Intel · Windows, Linux, Mac · kostenlos und Open Source</p>
|
||||
<h3 align="center">Leistungsstarke KI auf deinem eigenen Computer.</h3>
|
||||
<p align="center">Chatten, programmieren und mit Bildern arbeiten, lokal.</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?include_prereleases&label=release" alt="Neueste Version"></a>
|
||||
<a href="#modelle">Modelle</a> · <a href="#installation"><b>Installation</b></a> ·
|
||||
<a href="#hardware">Hardware</a> · <a href="#tools">Tools</a> ·
|
||||
<a href="#hilfe">Hilfe</a> · <a href="#strata-unterstützen">Unterstützen</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Strata herunterladen"></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="MIT-Lizenz"></a>
|
||||
<a href="https://github.com/Niko1221/Strata/stargazers"><img src="https://img.shields.io/github/stars/Niko1221/Strata?style=flat" alt="GitHub-Sterne"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black" alt="Spendier mir einen Kaffee"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Strata auf Buy Me a Coffee unterstützen"></a>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Ein Voxel-Pagodengarten, den das Modell von Strata geschrieben hat, im Browser"></a><br>
|
||||
<sub>Ein Voxel-Pagodengarten aus einem einzigen Prompt, auf einer RTX 5070 mit Strata (IQ3_S, 128K Kontext) ·
|
||||
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">ganzes Video (49 s)</a></sub></p>
|
||||
<p align="center"><sub><a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.ja.md">日本語</a> · <b>Deutsch</b> · <a href="README.fr.md">Français</a> · <a href="README.es.md">Español</a> · <a href="README.pt-BR.md">Português</a></sub></p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#installation"><b>Installation</b></a> ·
|
||||
<a href="#modelle">Modelle</a> ·
|
||||
<a href="#wie-schnell-ist-es">Tempo</a> ·
|
||||
<a href="#benutzung">Benutzung</a> ·
|
||||
<a href="#wie-funktioniert-das">Wie es funktioniert</a> ·
|
||||
<a href="#etwas-ist-schiefgegangen">Hilfe</a> ·
|
||||
<a href="#strata-unterstützen">Unterstützen</a>
|
||||
</p>
|
||||
**Kostenlos, Open Source und ganz in deiner Hand.** Nutze Strata auf dem Desktop oder mit deinen liebsten Coding-Tools. Kein Konto nötig.
|
||||
|
||||
Strata lässt **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** laufen, ein Modell mit 125
|
||||
Milliarden Parametern, für das man sonst einen Server braucht, auf einem normalen PC. Ab 0.2.4 läuft im Terminal auch
|
||||
**DeepSeek V4.1-Flash**. Es chattet, schreibt Code, liest Bilder und arbeitet mit deinen Apps und Coding-Agents zusammen.
|
||||
Deine Chats und Dateien bleiben auf deinem PC.
|
||||
## Modelle
|
||||
|
||||
- **Läuft auf einem Gaming-PC.** 32 GB Arbeitsspeicher (RAM) und Grafikspeicher zusammen reichen für den Anfang.
|
||||
- **Passt zu deinen Werkzeugen.** Claude Code, Codex CLI, OpenCode und andere Agents verbinden sich über die OpenAI- und die Anthropic-API.
|
||||
- **Liest Bilder.** Das ist standardmäßig eingeschaltet.
|
||||
- **Kostenlos und privat.** MIT-Lizenz, kein Konto, keine Cloud, keine Telemetrie ([was es herunterlädt und wann](docs/SAFETY.md)).
|
||||
Wir empfehlen **zuerst ISTA-DASLab**, danach **Unsloth** und andere Anbieter, soweit ein passendes Modell in deinen Speicher passt.
|
||||
|
||||
## Was du brauchst
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.8-Flash-Next</b></summary>
|
||||
|
||||
**Du brauchst mindestens 32 GB Speicher: Arbeitsspeicher (RAM) und Grafikspeicher (VRAM) zusammen.** Bei einem Mac oder einem
|
||||
Ryzen AI Max ist das der gemeinsame Speicher. Ein PC mit 32 GB und einer Karte mit 12-16 GB führt die kleineren Größen aus,
|
||||
mit 64 GB laufen alle Größen.
|
||||
|
||||
| Läuft auf | Anleitung |
|
||||
| Anbieter | Auswahl |
|
||||
| --- | --- |
|
||||
| **NVIDIA** GeForce RTX 20, 30, 40 und 50 Serie, 12 GB oder mehr | [Installation](#installation) |
|
||||
| **AMD** Radeon RX 6800 und neuer, Radeon AI PRO R9700 | [AMD-Karten](docs/AMD_HIP.md) |
|
||||
| **Intel** Arc | [Intel Arc](docs/INTEL_ARC.md) |
|
||||
| **Apple-Silicon-Mac**, M1 oder neuer, macOS 15 oder neuer | [Mac](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max (Strix Halo)** und andere PCs mit gemeinsamem Speicher | [Strix Halo](docs/STRIX_HALO.md) |
|
||||
| Ältere Karten und Prozessoren | [Ältere GPUs](docs/OLDER_GPUS.md) |
|
||||
| **ISTA-DASLab · empfohlen** | Q2\_0, IQ2\_XS, IQ3\_XXS, IQ3\_S. [Coder: IQ1\_M](docs/MODELS.md#coder) zum Programmieren. |
|
||||
| **Unsloth** | UD-IQ4\_XS, UD-Q4\_K\_XL, UD-Q5\_K\_XL, UD-Q6\_K\_XL, Q8\_0 |
|
||||
| **Gyro · agentionai** | [Gyro-S, Gyro-M](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF). Geringerer Speicherbedarf, aber langsamer. |
|
||||
| **Weitere Optionen** | [Swift 1.5](docs/MODELS.md#swift-15), [NVFP4](docs/NVFP4.md), [OrcaRouter](docs/ORCA.md) |
|
||||
|
||||
Außerdem brauchst du etwa 80 GB freien Speicherplatz (eine SSD), Windows 10 / 11, Linux oder macOS und einen aktuellen
|
||||
Grafiktreiber. Zwei oder drei Karten desselben Herstellers können sich ein Modell teilen ([Multi-GPU](docs/MULTI_GPU.md)).
|
||||
[Größen, Speicherbedarf und Kompatibilität vergleichen](docs/MODELS.md).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.6-35B-A3B</b></summary>
|
||||
|
||||
**Unsloth:** UD-Q4_K_XL, UD-Q6_K_XL, Q8_0. Das sind die derzeit dokumentierten Pakete für dieses Modell.
|
||||
|
||||
[Modellanleitung und Kompatibilität](docs/QWEN36.md).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/deepseek.svg" width="22" alt="" align="middle"> <b>DeepSeekV4.1-Flash-1017</b></summary>
|
||||
|
||||
In der **Desktop-App** und im **Terminal**. Was nicht in den Arbeitsspeicher passt, wird von deiner SSD gelesen. Darum brauchst du viel freien Speicherplatz.
|
||||
|
||||
| Set (IST-DAS) | Download | Freier Speicherplatz nötig |
|
||||
| --- | ---: | ---: |
|
||||
| 1.75 bpw | 239 GB | etwa 245 GB |
|
||||
| 2.00 bpw | 257 GB | etwa 265 GB |
|
||||
| 2.25 bpw | 274 GB | etwa 280 GB |
|
||||
|
||||
Läuft auf NVIDIA RTX 30, 40 und 50 (Windows und Linux) und auf AMD Radeon AI PRO R9700 (Linux). Das Setup lädt die Dateien und die DeepSeek-Engine herunter und startet dann das Modell.
|
||||
|
||||
```sh
|
||||
strata launch --family deepseek41 --model ist-das-2.00bpw --plan
|
||||
```
|
||||
|
||||
Gemessen mit dem 2.00-bpw-Set auf Strata 0.2.5. „Ausgabe“ ist das Tempo der Antwort, „Prompt“ zeigt, wie schnell Strata deine Eingabe liest.
|
||||
|
||||
| Hardware | Ausgabe | Prompt |
|
||||
| --- | ---: | ---: |
|
||||
| RTX 5070 12 GB, 64 GB RAM | 4 bis 6 Token/s | etwa 110 Token/s (4K) |
|
||||
| RTX 3060 12 GB, 46 GB RAM | 9 bis 11 Token/s | etwa 155 Token/s (4K) |
|
||||
| 1x Radeon AI PRO R9700, 251 GB RAM | 21 bis 25 Token/s | 320 bis 360 Token/s (8K) |
|
||||
| 4x Radeon AI PRO R9700 | 74 bis 77 Token/s | 400 bis 450 Token/s (8K) |
|
||||
|
||||
[Speicher, Festplatte und Details](docs/DEEPSEEK41.md) · [Versionshinweise](https://github.com/Niko1221/Strata/releases)
|
||||
|
||||
</details>
|
||||
|
||||
<a id="the-other-sizes"></a>
|
||||
|
||||
Unsicher, was passt? [Wähle ein Modell für deinen Computer](docs/MODELS.md#pick-by-memory).
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Die Strata-App (am einfachsten)
|
||||
<a id="option-a-the-strata-app"></a>
|
||||
|
||||
Lade die Datei für deinen Computer herunter, öffne sie und drück auf Installieren beim empfohlenen Modell.
|
||||
### Desktop-App
|
||||
|
||||
**Windows 10 / 11**
|
||||
**[Strata für Windows, Linux oder Mac herunterladen →](https://github.com/Niko1221/Strata/releases/latest)**
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-windows-x64-setup.exe)
|
||||
Installiere die App, wähle ein Modell und leg mit dem Chat los. Es gibt einen Installer pro System. Beim ersten Start lädt die App die Engine für deine Grafikkarte herunter. [Hilfe zum Installer](docs/INSTALL.md#the-app).
|
||||
|
||||
**Linux**
|
||||
<sub>Dateien: `Strata-0.2.6-windows-x64-setup.exe` · `Strata-0.2.6-linux-x86_64.AppImage` · `Strata-0.2.6-macos-arm64.dmg`</sub>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-linux-x86_64.AppImage)
|
||||
<p align="center">
|
||||
<img src="docs/media/app-chat.png" width="900" alt="Strata-Desktop-App mit Chat und Live-Codevorschau nebeneinander">
|
||||
<br><sub>Chat, Dateien und Live-Vorschau in einem Fenster.</sub>
|
||||
</p>
|
||||
|
||||
**Mac** (Apple Silicon, macOS 15 oder neuer)
|
||||
<a id="option-b-the-terminal-one-line"></a>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-macos-arm64.dmg)
|
||||
### Terminal-App
|
||||
|
||||
Die App wählt die passende Engine für deine Grafikkarte selbst. Alle Dateien stehen auf der
|
||||
[Releases-Seite](https://github.com/Niko1221/Strata/releases/latest).
|
||||
Lieber im Terminal? Installiere Strata und starte dann **`strata launch`**.
|
||||
|
||||
| Erster Start | Was du tust |
|
||||
| --- | --- |
|
||||
| **Windows** | Starte die `.exe`. Du brauchst keine Administratorrechte. Wenn Windows "Der Computer wurde durch Windows geschützt" meldet, wähle **Weitere Informationen** und dann **Trotzdem ausführen**. |
|
||||
| **Linux** | Mach die Datei ausführbar (`chmod +x`) und starte sie. Sie braucht FUSE 2: `sudo apt install libfuse2t64` (Ubuntu 24.04 und neuer) oder `libfuse2` (22.04). |
|
||||
| **Mac** | Öffne die `.dmg` und zieh Strata in den Ordner "Programme". |
|
||||
<details open>
|
||||
<summary><b>Installationsbefehle</b></summary>
|
||||
|
||||
Das Modell ist ein großer Download (40-70 GB). Wenn er abbricht, geht er dort weiter, wo er aufgehört hat.
|
||||
|
||||
### 2. Das Terminal (eine Zeile)
|
||||
|
||||
Der Befehl `strata`, für Skripte, Server und KI-Agents. Damit läuft auch DeepSeek.
|
||||
|
||||
Linux und Mac:
|
||||
**Linux / Mac**
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
```
|
||||
|
||||
Windows (PowerShell):
|
||||
**Windows PowerShell**
|
||||
|
||||
```powershell
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
```
|
||||
|
||||
Dann startest du es. Es stellt dir eine Frage nach der anderen (Modell, Kontext, Bilder, wer sich verbinden darf), lädt,
|
||||
was es braucht, und beginnt zu laufen:
|
||||
Öffne unter Windows danach ein neues Terminal. Dann:
|
||||
|
||||
```sh
|
||||
strata launch
|
||||
```
|
||||
|
||||
Du brauchst keine Administratorrechte. Die Installation prüft die SHA-256-Prüfsumme von allem, was sie herunterlädt. Öffne
|
||||
unter Windows nach der Installation ein neues Terminalfenster. Alle Bildschirme und Flags: [docs/TERMINAL_LAUNCH.md](docs/TERMINAL_LAUNCH.md).
|
||||
|
||||
### 3. Deine KI richtet es ein
|
||||
|
||||
Füge das in Claude Code, Cursor, Codex, GitHub Copilot oder einen anderen Coding-Assistenten ein:
|
||||
|
||||
```text
|
||||
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
|
||||
```
|
||||
|
||||
Der Assistent prüft deine Grafikkarte, deinen Arbeitsspeicher und den Speicherplatz, wählt das passende Modell, installiert
|
||||
und startet es und sagt dir, wie du deine Apps verbindest. KI-Werkzeuge können Strata auch über den
|
||||
[MCP-Server](docs/MCP_SERVER.md) installieren, starten und stoppen.
|
||||
|
||||
Du willst es lieber selbst bauen? Siehe [Aus dem Quellcode bauen](#aus-dem-quellcode-bauen).
|
||||
|
||||
## Modelle
|
||||
|
||||
Die App und `strata launch` empfehlen eine Größe passend zu deinem Speicher. Kleinere Größen sind schneller, größere
|
||||
Größen sind etwas klüger.
|
||||
|
||||
### Wenig Speicher? Probier Gyro
|
||||
|
||||
[Gyro](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF) von agentionai ist die kleinste Möglichkeit, Qwen3.8-Flash-Next
|
||||
auszuführen. Es läuft auf NVIDIA, AMD, Intel Arc und Apple Silicon.
|
||||
|
||||
| Größe | Benötigter Speicher (RAM + VRAM) | Download | Geeignet für |
|
||||
| --- | ---: | ---: | --- |
|
||||
| **Gyro-S** | 27,6 GB | 58,5 GB | PCs, auf denen nichts anderes läuft. 9 Tokens/s auf einem Mac mini mit 32 GB. |
|
||||
| **Gyro-M** | 35 GB | 92,0 GB | Ungefähr die Qualität von IQ3_XXS bei 20 % weniger GPU-Speicher. |
|
||||
|
||||
Starte es mit `strata launch --model gyro-s`. Details: [docs/GYRO.md](docs/GYRO.md).
|
||||
|
||||
### Die Größen von Qwen3.8-Flash-Next
|
||||
|
||||
| Dein Speicher (RAM + VRAM) | Nimm | Warum |
|
||||
| --- | --- | --- |
|
||||
| **32 GB** | **Gyro-S**, oder den **[Coder](docs/MODELS.md#coder)** für Code | Der Coder behält die Experten, die Code braucht |
|
||||
| **40 GB** | **[Swift 1.5](docs/MODELS.md#swift-15)** IQ2_XS (39,2 GB) oder **Gyro-M** | Swift denkt kürzer nach, bevor es antwortet |
|
||||
| **48 GB** | **IQ3_XXS** (47,0 GB) | Langsamer, besser |
|
||||
| **64 GB** | **IQ3_S** (54,8 GB) | Erreicht in den veröffentlichten Tests das volle Modell |
|
||||
| **96 GB oder mehr** | **UD-IQ4_XS** bis **Q8_0** von Unsloth | Am nächsten am vollen Modell |
|
||||
|
||||
Auch mit weniger RAM, als eine Zeile nennt, kann es funktionieren, nur langsamer: Strata wechselt von selbst in seinen
|
||||
[Modus für wenig RAM](docs/MODELS.md#a-big-graphics-card-and-little-ram). Außerdem gibt es:
|
||||
[NVFP4](docs/NVFP4.md) für NVIDIA-Karten, [Qwen3.6-35B-A3B](docs/QWEN36.md) (kleiner, mehrere Anfragen gleichzeitig) und
|
||||
[OrcaRouters Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs). Alle Größen und was wo läuft:
|
||||
[docs/MODELS.md](docs/MODELS.md).
|
||||
|
||||
### DeepSeek V4.1-Flash (Terminal, ab 0.2.4)
|
||||
|
||||
Ein deutlich größeres Modell, das auf einer Grafikkarte mit 12 GB und 46 bis 64 GB RAM läuft. Strata hält die meistgenutzten
|
||||
Teile im Speicher und liest den Rest von deiner SSD. [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF)
|
||||
hat die Sets gebaut. Es läuft auf NVIDIA-RTX-Karten der Serien 30, 40 und 50 und auf der Radeon AI PRO R9700. Wähle es in
|
||||
`strata launch`. Die Desktop-App bietet es noch nicht an.
|
||||
|
||||
| Set | Download | Qualität | Schreibt Code mit |
|
||||
| --- | ---: | --- | --- |
|
||||
| **IQ1_M** (1,75 Bit) | 239 GB | Das kleinste, schwächer bei einfacher Prosa | auf einem PC noch nicht gemessen |
|
||||
| **IQ2_XXS** (2.00 Bit) | 257 GB | Gute Balance | 10 Tokens/s (RTX 3060), 5 (RTX 5070), 48 (4x R9700) |
|
||||
| **IQ2_XS** (2.25 Bit) | 274 GB | Am nächsten am Original | 15 Tokens/s (eine R9700), 80 (4x R9700) |
|
||||
| **IQ4_NL** (Engram-Tabelle) | in jedem Download oben enthalten | Die Nachschlagetabelle, die alle drei Sets teilen | |
|
||||
|
||||
Starte es mit `strata launch --model ist-das-2.00bpw` (oder `ist-das-1.75bpw`, `ist-das-2.25bpw`). Qualitätswerte,
|
||||
Geschwindigkeiten je PC und die manuelle Installation: [docs/DEEPSEEK41.md](docs/DEEPSEEK41.md).
|
||||
|
||||
## Wie schnell ist es?
|
||||
|
||||
Gemessen mit einer Grafikkarte mit 12 GB (RTX 5070, 64 GB RAM). Ergebnisse anderer Karten:
|
||||
[Tempo jeder Größe](docs/MODELS.md#how-fast-is-each-size) und [Ergebnisse der Community](docs/COMMUNITY_BENCHMARKS.md).
|
||||
|
||||
| Modell | Antworten erscheinen mit | Liest deinen Prompt mit |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████████` 94 Tokens/s | `████████████` 2.650 Tokens/s |
|
||||
| **IQ2_XS** | `██████████░░` 79 Tokens/s | `█████████░░░` 2.090 Tokens/s |
|
||||
| **IQ3_XXS** | `████████░░░░` 62 Tokens/s | `████████░░░░` 1.750 Tokens/s |
|
||||
| **IQ3_S** | `███████░░░░░` 53 Tokens/s | `███████░░░░░` 1.620 Tokens/s |
|
||||
| **Coder** | `███████░░░░░` 55 Tokens/s | `██████████░░` 2.180 Tokens/s |
|
||||
|
||||
Auf einer AMD-Karte mit 16 GB (RX 9070 XT, 47 GB RAM):
|
||||
|
||||
| Modell | Antworten erscheinen mit | Liest deinen Prompt mit |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████░░░░` 60 Tokens/s | `█████░░░░░░░` 1.160 Tokens/s |
|
||||
| **IQ2_XS** | `███████░░░░░` 52 Tokens/s | `█████░░░░░░░` 1.110 Tokens/s |
|
||||
| **Coder** | `██████░░░░░░` 44 Tokens/s | `██████░░░░░░` 1.420 Tokens/s |
|
||||
|
||||
Ein Token ist etwa ¾ eines Wortes, 60 Tokens pro Sekunde sind also schneller, als du lesen kannst. "Liest deinen Prompt"
|
||||
beschreibt, wie schnell Strata ein langes Dokument, Code oder einen Chatverlauf aufnimmt.
|
||||
|
||||
## Benutzung
|
||||
|
||||
### In der App
|
||||
|
||||
Ein Chat, ein **Monitor** für das Modell sowie für GPU, CPU und RAM, eine Seite **Modelle** und die **Einstellungen**. Der
|
||||
Chat kann in einem Ordner auf deinem PC arbeiten. Im Modus **Manuell** fragt er nach, bevor er eine Datei ändert oder einen
|
||||
Befehl ausführt. Er kann das Web durchsuchen (DuckDuckGo oder Bing, ohne Konto). Die Suchbegriffe sind das Einzige, was
|
||||
deinen PC verlässt.
|
||||
|
||||
<p align="center"><img src="docs/media/app-chat.png" width="900" alt="Die Strata-App: ein Chat, in dem der Agent einen kleinen Voxel-Pagodengarten gebaut hat, die Seite ist rechts in der Vorschau geöffnet"><br>
|
||||
<sub>Der Agent hat einen Voxel-Pagodengarten als eine HTML-Datei geschrieben, und die Seite öffnet sich in der Vorschau neben dem Chat</sub></p>
|
||||
|
||||
### Im Terminal
|
||||
|
||||
<p align="center"><img src="docs/media/terminal-monitor.png" width="900" alt="Die Übersicht des Strata-Terminal-Monitors, während das Modell eine Antwort schreibt"><br>
|
||||
<sub><code>strata monitor</code>, während das Modell antwortet: Geschwindigkeit, GPU, Speicher und die letzten Anfragen</sub></p>
|
||||
|
||||
```sh
|
||||
strata launch # guided: model, context, pictures, access, then serve
|
||||
strata launch --model q2_0 --yes --no-tui # no questions: serve and print the address
|
||||
strata launch --status # what is being served now
|
||||
strata launch --stop # stop it
|
||||
strata monitor # watch speed, GPU, memory and requests
|
||||
strata update # get the newest engine
|
||||
strata doctor # check the installed files
|
||||
```
|
||||
|
||||
Um Strata von deinem Handy oder einem anderen PC aus zu erreichen, nutze `strata launch --access lan`. Der Befehl erzeugt
|
||||
einen API-Schlüssel für dich. `strata --help` listet alles auf, und [docs/TERMINAL.md](docs/TERMINAL.md) erklärt es.
|
||||
|
||||
### Mit KI-Agents und anderen Apps
|
||||
|
||||
Ein Agent installiert Strata mit dem Einzeiler oben und führt `strata launch --model NAME --yes --no-tui` aus. Die Schritte,
|
||||
Flags und Exit-Codes stehen in [docs/AI_SETUP.md](docs/AI_SETUP.md). Die Regeln für das Repository stehen in [AGENTS.md](AGENTS.md).
|
||||
|
||||
| Deine App spricht | Basis-URL |
|
||||
| --- | --- |
|
||||
| OpenAI-API (die meisten Apps und Agents) | `http://127.0.0.1:8080/v1` (beliebiger API-Schlüssel, beliebiger Modellname) |
|
||||
| Anthropic-API (Claude Code) | `http://127.0.0.1:8080`, ohne `/v1`. Claude Code: `ANTHROPIC_BASE_URL=http://127.0.0.1:8080` |
|
||||
| OpenAI Responses API (Codex CLI) | `http://127.0.0.1:8080/v1/responses` ([Einrichtung](docs/DETAILS.md#the-responses-api-and-codex-cli)) |
|
||||
|
||||
`strata connect` gibt die genauen Einstellungen für OpenCode, Codex CLI, Claude Code, oh-my-pi, pi, Hermes Agent und OpenClaw
|
||||
aus, und `strata connect opencode --write` schreibt sie für dich (mit einer Sicherung).
|
||||
|
||||
### Gut zu wissen
|
||||
|
||||
- **Bilder** sind für jedes Modell eingeschaltet, das sie lesen kann. Hänge ein Bild im Chat an oder füge es ein, oder sende es aus deiner App. Details: [docs/IMAGES.md](docs/IMAGES.md).
|
||||
- **Das Denkniveau** kann aus, niedrig, mittel oder hoch sein, im Chat-Menü oder unter "reasoning effort" in deiner App. Aus ist am schnellsten.
|
||||
- **Die erste Nachricht** eines Chats wird komplett gelesen, etwa 1 Minute pro 30.000 Tokens. Folgenachrichten starten in Sekunden.
|
||||
- **Eine Anfrage nach der anderen** ist die Standardeinstellung. Für mehrere nutze `--parallel N` ([BATCHING.md](docs/BATCHING.md)). Auf einer Karte mit 12 GB wird dadurch jede Antwort langsamer.
|
||||
|
||||
## Aus dem Quellcode bauen
|
||||
|
||||
<details>
|
||||
<summary>Windows, Linux und Mac aus dem Repository</summary>
|
||||
|
||||
[Lade Strata herunter](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) und entpacke es, oder nutze `git clone`.
|
||||
**Windows:** Doppelklick auf `START-HERE.bat`. **Linux:** Führe `./setup.sh` im Strata-Ordner aus.
|
||||
|
||||
**Mac (Apple Silicon, macOS 15 oder neuer):** Installiere einmal die Command Line Tools von Apple mit `xcode-select --install`
|
||||
(Xcode selbst brauchst du nicht). Öffne das Terminal im Strata-Ordner und führe `./setup.sh` aus. Das Skript baut die
|
||||
Mac-Engine auf deinem Mac (etwa 7 Minuten), stellt dieselben Fragen, lädt das Modell herunter und startet es. Beim nächsten
|
||||
Mal startet `./setup.sh` sofort. Welche Größe zu deinem Mac passt: [docs/MAC.md](docs/MAC.md).
|
||||
|
||||
Der Installer erkennt deine Karte und richtet die passende Engine ein. Er fragt, welches Modell und welche Größe du willst,
|
||||
wie viel Kontext (wie viel Text sich das Modell merkt) und ob es Bilder lesen soll. Drück Enter für die empfohlene Antwort.
|
||||
Dann lädt er das Modell herunter (etwa 70 GB) und startet es. Wenn der Download abbricht, führe das Skript erneut aus, dann
|
||||
geht es weiter. Dein Browser öffnet die Strata-Seite unter `http://127.0.0.1:8080`.
|
||||
|
||||
> Während das Modell startet, kann dein PC 1-3 Minuten langsam sein oder nicht reagieren (beim ersten Mal am längsten). Strata
|
||||
> lädt 35-55 GB in deinen RAM und sperrt einen Teil davon für die Grafikkarte. Warte ab und schließe das Fenster nicht.
|
||||
|
||||
Starte beim nächsten Mal wieder `START-HERE.bat` (oder `./setup.sh`). Es startet sofort und lädt nichts doppelt herunter.
|
||||
Schließ sein Fenster, um das Modell zu beenden. `UPDATE.bat` (`./update.sh`) aktualisiert Strata, ohne es zu starten.
|
||||
Aktualisieren, Docker, mehrere Karten, wohin die Dateien kommen und alle Optionen: [docs/INSTALL.md](docs/INSTALL.md).
|
||||
|
||||
Um deinen PC zu messen, führe `BENCH.bat` (`./bench.sh`) aus. Das dauert 5-10 Minuten und schreibt einen Bericht, den du teilen
|
||||
kannst ([so geht's](docs/COMMUNITY_BENCHMARKS.md#run-the-benchmark)).
|
||||
|
||||
**Optional: mehr Tempo auf einer Karte, wenn das ganze Modell in den RAM passt.** Zwei Einstellungen zum Einschalten von
|
||||
RubenTorney (#1904, #1905). Sie helfen, wenn du eine Grafikkarte nutzt und dein RAM alle Experten aufnimmt, zum Beispiel bei
|
||||
Q2_0 oder IQ3_XXS auf einem PC mit 64 GB. Öffne `strata-<model>.json` im Strata-Datenordner ([wo](docs/INSTALL.md#where-things-are-stored)) und ergänze:
|
||||
|
||||
- `"--adapt-async", "1"` in der Liste `"args"`. Der Expertencache wird dann in einem Hilfsthread aktualisiert, statt die
|
||||
Antwort anzuhalten. Gemessen auf einer RTX 5070 (Windows, Q2_0): +4-5 % beim Decodieren, +7-10 % bei Code und langen
|
||||
Dokumenten. Die Antworten können sich leicht von der Standardeinstellung unterscheiden und bleiben von Lauf zu Lauf gleich.
|
||||
- `"STRATA_PIN_BY_NEED": "1"` im Block `"env"` (Windows). Windows kann nur etwa die Hälfte des RAM für die Grafikkarte sperren,
|
||||
und diese Einstellung sperrt zuerst die Experten, die die Karte am dringendsten braucht. Gemessen auf der RTX 5070: +4,4 % beim Lesen des Prompts.
|
||||
|
||||
```json
|
||||
"args": [ ..., "--adapt-async", "1" ],
|
||||
"env": { "STRATA_PIN_BY_NEED": "1" }
|
||||
```
|
||||
|
||||
Starte Strata nach der Änderung neu. Zum Rückgängigmachen entfernst du die beiden Einträge.
|
||||
Folge den Anweisungen, um ein Modell herunterzuladen und zu starten. [Terminal-Anleitung](docs/TERMINAL_LAUNCH.md).
|
||||
|
||||
</details>
|
||||
|
||||
## Wie funktioniert das?
|
||||
<p align="center">
|
||||
<img src="docs/media/terminal-monitor.png" width="900" alt="Strata-Terminal-App mit Modellaktivität, Tempo, Speicher und letzten Anfragen">
|
||||
<br><sub>Behalte Modell und Hardware im Blick, während es läuft.</sub>
|
||||
</p>
|
||||
|
||||
Modelle wie dieses laufen normalerweise auf Servern mit Hunderten Gigabyte Grafikspeicher. Deine Grafikkarte hat 12-24 GB.
|
||||
Strata bringt das Modell auf deinen PC, indem es die Arbeit auf den ganzen Rechner verteilt. Stell dir eine Bibliothek vor.
|
||||
<a id="build-from-source"></a>
|
||||
|
||||
- Das Modell ist ein Team aus 24.576 kleinen Spezialisten ("Experten"), wie 24.576 Nachschlagewerke. Für jedes Wort braucht es nur 10 davon.
|
||||
- **Deine Grafikkarte ist der Schreibtisch.** Er ist klein und schnell. Die etwa 4.000 am häufigsten genutzten Bücher liegen
|
||||
dort aufgeschlagen.
|
||||
- **Dein RAM sind die Regale im Lesesaal.** Alle 24.576 Bücher stehen dort. Braucht ein Wort ein Buch, das nicht auf dem
|
||||
Schreibtisch liegt, holt dein Prozessor es sofort aus dem Regal, während die Karte weiterarbeitet.
|
||||
- **Deine SSD ist das Archiv.** Sie enthält eine Nachschlagetabelle von 29 GB. Das Modell liest pro Wort nur wenige Zeilen
|
||||
daraus, deshalb muss sie nicht im RAM liegen.
|
||||
[Aus dem Quellcode bauen](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [Vollständige Installationsanleitung](docs/INSTALL.md)
|
||||
|
||||
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Die 24.576 Experten des Modells: die meistgenutzten auf der Grafikkarte, alle im RAM, eine Nachschlagetabelle auf der SSD"></p>
|
||||
<a id="what-you-need"></a>
|
||||
|
||||
Ein kleiner Helfer rät außerdem die nächsten Wörter. Das große Modell prüft sie alle auf einmal und behält die richtigen. Du
|
||||
bekommst dieselbe Antwort, 1,6-1,8-mal früher. Lange Texte werden in großen Stücken gelesen, bis zu 8.192 Tokens auf einmal,
|
||||
mit über 1.000 Tokens pro Sekunde.
|
||||
## Hardware
|
||||
|
||||
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Ein kleiner Helfer rät die nächsten Wörter, das große Modell prüft sie alle auf einmal und behält die richtigen"></p>
|
||||
| Unterstützte Hardware | Anleitung |
|
||||
| --- | --- |
|
||||
| **NVIDIA GeForce RTX 20 / 30 / 40 / 50** | [NVIDIA-Setup](docs/INSTALL.md#what-you-need) |
|
||||
| **AMD Radeon RX 7700 / 7800 / 7900, RX 9060 / 9070, AI PRO R9700** | [AMD-Setup](docs/AMD_HIP.md) |
|
||||
| **AMD Radeon RX 6800 / 6900 · experimentell** | [RX-6000-Setup](docs/AMD_HIP.md#rdna2-gfx1030) |
|
||||
| **Intel Arc · Linux** | [Intel-Setup](docs/INTEL_ARC.md) |
|
||||
| **Apple Silicon · M1 oder neuer** | [Mac-Setup](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max · Strix Halo** | [Strix-Halo-Setup](docs/STRIX_HALO.md) |
|
||||
|
||||
Die ausführliche Erklärung: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Jeder Teil mit seinen Zahlen:
|
||||
[die Details](docs/DETAILS.md#how-it-works) und das [Paper](docs/paper/Strata-Paper.pdf).
|
||||
<a id="using-it"></a>
|
||||
|
||||
## Etwas ist schiefgegangen?
|
||||
## Tools
|
||||
|
||||
- **Mein PC ist beim ersten Start von Strata eingefroren.** Strata lädt das Modell in den RAM. Warte ab und schließe das
|
||||
Fenster nicht. Nach 10 Minuten immer noch eingefroren? Starte den PC neu, schließe andere Programme und versuch es noch
|
||||
einmal, oder wähle eine kleinere Größe.
|
||||
- **Es ist beim Herunterladen oder Installieren stehen geblieben.** Starte die App oder `strata launch` noch einmal. Es
|
||||
macht dort weiter, wo es aufgehört hat.
|
||||
- **Es ist sehr langsam, oder es meldet "the engine stopped unexpectedly".** Dein PC hat nicht genug freien RAM. Schließe
|
||||
andere Programme (Browser brauchen viel) oder wähle eine kleinere Größe wie Q2_0 oder IQ2_XS.
|
||||
- **Es meldet, dass es keine Engine für diesen PC gibt.** Öffne in der App **Einstellungen, Engine** und drücke **Engine
|
||||
aktualisieren**. Führe im Terminal `strata update` aus.
|
||||
- **Es meldet, dass Port 8080 schon belegt ist.** Strata läuft bereits. Such das Fenster.
|
||||
**Claude Code · Codex CLI · OpenCode · oh-my-pi · pi.dev · Hermes Agent · OpenClaw**, und weitere über die OpenAI- und Anthropic-kompatiblen APIs.
|
||||
|
||||
Weitere Lösungen: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Du kommst nicht weiter? Öffne ein
|
||||
[Issue](https://github.com/Niko1221/Strata/issues) und hänge die Protokolle an: in der App unter **Einstellungen, Engine,
|
||||
Diagnose, Protokolle speichern**, im Terminal die Ausgabe von `strata doctor`. Ein Sicherheitsproblem meldest du bitte
|
||||
privat: [SECURITY.md](SECURITY.md).
|
||||
Führe **`strata connect`** aus, um die Einrichtung für dein Tool zu sehen. [Verbindungsanleitung](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
|
||||
|
||||
## Danksagung und Lizenz
|
||||
<a id="guides"></a>
|
||||
<a id="how-fast-is-it"></a>
|
||||
<a id="how-does-it-work"></a>
|
||||
<a id="something-went-wrong"></a>
|
||||
|
||||
Das Modell ist [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) vom Qwen-Team. Komprimiert wurde es von
|
||||
[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5), agentionai (Gyro)
|
||||
und Unsloth. ISTA-DASLab hat auch die Sets von DeepSeek V4.1-Flash gebaut. Strata nutzt Teile von
|
||||
[llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Alle Danksagungen: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits).
|
||||
Strata ist Open Source unter der [MIT-Lizenz](LICENSE). Einige Teile und alle Modelle haben eigene Lizenzen
|
||||
([welche](docs/HOW_IT_WORKS.md#license)).
|
||||
## Hilfe
|
||||
|
||||
## Mitmachen
|
||||
|
||||
Du hast einen Fehler gefunden oder willst eine Änderung oder einen Benchmark einsenden? Lies zuerst [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
Dort steht, was dazugehört, damit andere deine Arbeit nachprüfen können.
|
||||
[Alle Anleitungen](docs) · [Fehlerbehebung](docs/TROUBLESHOOTING.md) · [Fehler melden](https://github.com/Niko1221/Strata/issues/new/choose) ·
|
||||
[Benchmarks](docs/COMMUNITY_BENCHMARKS.md) · [So funktioniert es](docs/HOW_IT_WORKS.md) · [Mitmachen](CONTRIBUTING.md)
|
||||
|
||||
## Strata unterstützen
|
||||
|
||||
Strata ist kostenlos und steht unter der MIT-Lizenz. Ich baue es und teste es auf jeder Hardware, die ich bekommen kann. Wenn es auf deinem PC gut läuft, hilft mir ein Kaffee,
|
||||
weiter neue Karten zu testen und das zu beheben, was ihr meldet. Es besteht kein Druck. Strata zu nutzen und zu melden, was dir auffällt, hilft genauso viel.
|
||||
**Hilf mit, lokale KI für alle offen zu halten.** Ein Kaffee hilft, Entwicklung und Tests auf mehr Hardware zu finanzieren.
|
||||
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: Strata unterstützen" height="48"></a>
|
||||
|
||||
Du möchtest Strata sponsern? Schreib mir über [GitHub](https://github.com/Niko1221).
|
||||
[MIT-Lizenz](LICENSE) · [Danksagung und Modelllizenzen](docs/HOW_IT_WORKS.md#credits) · [Datenschutz](docs/SAFETY.md)
|
||||
|
||||
<a id="option-c-let-your-ai-set-it-up"></a>
|
||||
|
||||
<sub>Für KI-Tools und automatische Leser: <a href="AGENTS.md">technische Referenz</a> · <a href="docs/AI_SETUP.md">Setup-Anleitung</a>.</sub>
|
||||
|
||||
**Sponsoring, Hardware-Partnerschaften oder Zusammenarbeit:** [[email protected]](mailto:[email protected]).
|
||||
+122
-298
@@ -1,364 +1,188 @@
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="apps/host/public/brand/strata-lockup-dark.svg">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="380">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="340">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.fr.md">Français</a> · <b>Español</b> · <a href="README.pt-BR.md">Português</a></p>
|
||||
|
||||
<h3 align="center">Ejecuta un modelo de última generación en tu propio PC gaming</h3>
|
||||
|
||||
<p align="center">Compatible con NVIDIA, AMD, Intel · Windows, Linux, Mac · libre y de código abierto</p>
|
||||
<h3 align="center">IA potente en tu propio ordenador.</h3>
|
||||
<p align="center">Chatea, programa y trabaja con imágenes, en local.</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?include_prereleases&label=release" alt="Última versión"></a>
|
||||
<a href="#modelos">Modelos</a> · <a href="#instalación"><b>Instalación</b></a> ·
|
||||
<a href="#hardware">Hardware</a> · <a href="#herramientas">Herramientas</a> ·
|
||||
<a href="#ayuda">Ayuda</a> · <a href="#apoya-strata">Apoyar</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Descargar Strata"></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="Licencia MIT"></a>
|
||||
<a href="https://github.com/Niko1221/Strata/stargazers"><img src="https://img.shields.io/github/stars/Niko1221/Strata?style=flat" alt="Estrellas en GitHub"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black" alt="Invítame a un café"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Apoya Strata en Buy Me a Coffee"></a>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Un jardín con una pagoda de vóxeles que escribió el modelo de Strata, funcionando en el navegador"></a><br>
|
||||
<sub>Un jardín con una pagoda de vóxeles, hecho con un solo prompt en una RTX 5070 con Strata (IQ3_S, contexto de 128K) ·
|
||||
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">vídeo completo (49 s)</a></sub></p>
|
||||
<p align="center"><sub><a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.fr.md">Français</a> · <b>Español</b> · <a href="README.pt-BR.md">Português</a></sub></p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#instalación"><b>Instalación</b></a> ·
|
||||
<a href="#modelos">Modelos</a> ·
|
||||
<a href="#qué-velocidad-tiene">Velocidad</a> ·
|
||||
<a href="#uso">Uso</a> ·
|
||||
<a href="#cómo-funciona">Cómo funciona</a> ·
|
||||
<a href="#algo-ha-ido-mal">Ayuda</a> ·
|
||||
<a href="#apoya-strata">Apoyar</a>
|
||||
</p>
|
||||
**Gratis, de código abierto y tuyo para ejecutar.** Usa Strata en tu escritorio o con tus herramientas de programación favoritas. No necesitas cuenta.
|
||||
|
||||
Strata ejecuta **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)**, un modelo de 125 mil millones
|
||||
de parámetros que normalmente necesita un servidor, en un PC normal. Desde 0.2.4 también ejecuta **DeepSeek V4.1-Flash**
|
||||
en la Terminal. Conversa, escribe código, lee imágenes y funciona con tus aplicaciones y tus agentes de programación. Tus
|
||||
chats y tus archivos se quedan en tu PC.
|
||||
## Modelos
|
||||
|
||||
- **Funciona en un PC gaming.** Con 32 GB entre RAM y memoria gráfica ya puedes empezar.
|
||||
- **Funciona con tus herramientas.** Claude Code, Codex CLI, OpenCode y otros agentes se conectan por las API de OpenAI y de Anthropic.
|
||||
- **Lee imágenes**, activado por defecto.
|
||||
- **Gratis y privado.** Licencia MIT, sin cuenta, sin nube, sin telemetría ([qué descarga y cuándo](docs/SAFETY.md)).
|
||||
Recomendamos **primero ISTA-DASLab**, después **Unsloth** y otros proveedores, siempre que haya un modelo compatible que quepa en tu memoria.
|
||||
|
||||
## Lo que necesitas
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.8-Flash-Next</b></summary>
|
||||
|
||||
**Necesitas al menos 32 GB de memoria: RAM y memoria gráfica (VRAM) juntas.** En un Mac o un Ryzen AI Max, es la memoria
|
||||
unificada. Un PC de 32 GB con una tarjeta de 12 a 16 GB ejecuta los tamaños más pequeños, y con 64 GB funcionan todos los
|
||||
tamaños.
|
||||
|
||||
| Funciona en | Guía |
|
||||
| Proveedor | Opciones |
|
||||
| --- | --- |
|
||||
| **NVIDIA** GeForce RTX serie 20, 30, 40 y 50, 12 GB o más | [Instalación](#instalación) |
|
||||
| **AMD** Radeon RX 6800 y posteriores, Radeon AI PRO R9700 | [Tarjetas AMD](docs/AMD_HIP.md) |
|
||||
| **Intel** Arc | [Intel Arc](docs/INTEL_ARC.md) |
|
||||
| **Mac con Apple Silicon**, M1 o posterior, macOS 15 o posterior | [Mac](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max (Strix Halo)** y otros PC con memoria unificada | [Strix Halo](docs/STRIX_HALO.md) |
|
||||
| Tarjetas y procesadores antiguos | [GPU antiguas](docs/OLDER_GPUS.md) |
|
||||
| **ISTA-DASLab · recomendado** | Q2\_0, IQ2\_XS, IQ3\_XXS, IQ3\_S. [Coder: IQ1\_M](docs/MODELS.md#coder) para programar. |
|
||||
| **Unsloth** | UD-IQ4\_XS, UD-Q4\_K\_XL, UD-Q5\_K\_XL, UD-Q6\_K\_XL, Q8\_0 |
|
||||
| **Gyro · agentionai** | [Gyro-S, Gyro-M](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF). Usa menos memoria, pero es más lento. |
|
||||
| **Otras opciones** | [Swift 1.5](docs/MODELS.md#swift-15), [NVFP4](docs/NVFP4.md), [OrcaRouter](docs/ORCA.md) |
|
||||
|
||||
También necesitas unos 80 GB de disco libre (un SSD), Windows 10 / 11, Linux o macOS, y un controlador gráfico actual. Dos
|
||||
o tres tarjetas de la misma marca pueden compartir un modelo ([multi-GPU](docs/MULTI_GPU.md)).
|
||||
[Compara tamaños, necesidades de memoria y compatibilidad](docs/MODELS.md).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.6-35B-A3B</b></summary>
|
||||
|
||||
**Unsloth:** UD-Q4_K_XL, UD-Q6_K_XL, Q8_0. Son los paquetes documentados actualmente para este modelo.
|
||||
|
||||
[Guía del modelo y compatibilidad](docs/QWEN36.md).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/deepseek.svg" width="22" alt="" align="middle"> <b>DeepSeekV4.1-Flash-1017</b></summary>
|
||||
|
||||
En la **aplicación de escritorio** y en el **Terminal**. Lo que no cabe en la memoria se lee desde tu SSD, así que hace falta mucho espacio libre en disco.
|
||||
|
||||
| Conjunto (IST-DAS) | Descarga | Espacio libre necesario |
|
||||
| --- | ---: | ---: |
|
||||
| 1.75 bpw | 239 GB | unos 245 GB |
|
||||
| 2.00 bpw | 257 GB | unos 265 GB |
|
||||
| 2.25 bpw | 274 GB | unos 280 GB |
|
||||
|
||||
Funciona en NVIDIA RTX 30, 40 y 50 (Windows y Linux) y en AMD Radeon AI PRO R9700 (Linux). La instalación descarga los archivos y el motor de DeepSeek, y después inicia el modelo.
|
||||
|
||||
```sh
|
||||
strata launch --family deepseek41 --model ist-das-2.00bpw --plan
|
||||
```
|
||||
|
||||
Medido con el conjunto de 2.00 bpw en Strata 0.2.5. «Salida» es la velocidad de la respuesta; «Prompt» es la rapidez con la que Strata lee tu texto.
|
||||
|
||||
| Hardware | Salida | Prompt |
|
||||
| --- | ---: | ---: |
|
||||
| RTX 5070 12 GB, 64 GB de RAM | 4 a 6 tokens/s | unos 110 tokens/s (4K) |
|
||||
| RTX 3060 12 GB, 46 GB de RAM | 9 a 11 tokens/s | unos 155 tokens/s (4K) |
|
||||
| 1x Radeon AI PRO R9700, 251 GB de RAM | 21 a 25 tokens/s | 320 a 360 tokens/s (8K) |
|
||||
| 4x Radeon AI PRO R9700 | 74 a 77 tokens/s | 400 a 450 tokens/s (8K) |
|
||||
|
||||
[Memoria, disco y detalles](docs/DEEPSEEK41.md) · [Notas de la versión](https://github.com/Niko1221/Strata/releases)
|
||||
|
||||
</details>
|
||||
|
||||
<a id="the-other-sizes"></a>
|
||||
|
||||
¿No sabes qué cabe? [Elige un modelo para tu ordenador](docs/MODELS.md#pick-by-memory).
|
||||
|
||||
## Instalación
|
||||
|
||||
### 1. La aplicación Strata (la más fácil)
|
||||
<a id="option-a-the-strata-app"></a>
|
||||
|
||||
Descarga el archivo para tu ordenador, ábrelo y pulsa Instalar en el modelo que recomienda.
|
||||
### Aplicación de escritorio
|
||||
|
||||
**Windows 10 / 11**
|
||||
**[Descargar Strata para Windows, Linux o Mac →](https://github.com/Niko1221/Strata/releases/latest)**
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-windows-x64-setup.exe)
|
||||
Instala la aplicación, elige un modelo y empieza a chatear. Hay un instalador por sistema. En el primer arranque, la aplicación descarga el motor para tu tarjeta gráfica. [Ayuda con el instalador](docs/INSTALL.md#the-app).
|
||||
|
||||
**Linux**
|
||||
<sub>Archivos: `Strata-0.2.6-windows-x64-setup.exe` · `Strata-0.2.6-linux-x86_64.AppImage` · `Strata-0.2.6-macos-arm64.dmg`</sub>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-linux-x86_64.AppImage)
|
||||
<p align="center">
|
||||
<img src="docs/media/app-chat.png" width="900" alt="Aplicación de escritorio de Strata con un chat y una vista previa del código en vivo, una junto a otra">
|
||||
<br><sub>Chat, archivos y vista previa en vivo en una sola ventana.</sub>
|
||||
</p>
|
||||
|
||||
**Mac** (Apple Silicon, macOS 15 o posterior)
|
||||
<a id="option-b-the-terminal-one-line"></a>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-macos-arm64.dmg)
|
||||
### Aplicación de Terminal
|
||||
|
||||
La aplicación elige sola el motor adecuado para tu tarjeta gráfica. Todos los archivos están en la
|
||||
[página de versiones](https://github.com/Niko1221/Strata/releases/latest).
|
||||
¿Prefieres la terminal? Instala Strata y ejecuta **`strata launch`**.
|
||||
|
||||
| Primer arranque | Qué hacer |
|
||||
| --- | --- |
|
||||
| **Windows** | Ejecuta el `.exe`. No necesita permisos de administrador. Si Windows muestra "Windows protegió tu PC", elige **Más información** y luego **Ejecutar de todas formas**. |
|
||||
| **Linux** | Haz el archivo ejecutable (`chmod +x`) y lánzalo. Necesita FUSE 2: `sudo apt install libfuse2t64` (Ubuntu 24.04 y posteriores) o `libfuse2` (22.04). |
|
||||
| **Mac** | Abre el `.dmg` y arrastra Strata a Aplicaciones. |
|
||||
<details open>
|
||||
<summary><b>Comandos de instalación</b></summary>
|
||||
|
||||
La descarga del modelo es grande (40-70 GB). Si se detiene, continúa donde se quedó.
|
||||
|
||||
### 2. La Terminal (una línea)
|
||||
|
||||
El comando `strata`, para scripts, servidores y agentes de IA. Es también la forma de ejecutar DeepSeek.
|
||||
|
||||
Linux y Mac:
|
||||
**Linux / Mac**
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
```
|
||||
|
||||
Windows (PowerShell):
|
||||
**Windows PowerShell**
|
||||
|
||||
```powershell
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
```
|
||||
|
||||
Después arráncalo. Te hace una pregunta cada vez (modelo, contexto, imágenes, quién puede conectarse), descarga lo que
|
||||
necesita y empieza a servir:
|
||||
En Windows, abre una terminal nueva después. Luego:
|
||||
|
||||
```sh
|
||||
strata launch
|
||||
```
|
||||
|
||||
No necesita permisos de administrador. La instalación comprueba el SHA-256 de todo lo que descarga. En Windows, abre una
|
||||
ventana de terminal nueva después de la instalación. Todas las pantallas y opciones: [docs/TERMINAL_LAUNCH.md](docs/TERMINAL_LAUNCH.md).
|
||||
|
||||
### 3. Que lo instale tu IA
|
||||
|
||||
Pega esto en Claude Code, Cursor, Codex, GitHub Copilot u otro asistente de programación:
|
||||
|
||||
```text
|
||||
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
|
||||
```
|
||||
|
||||
El asistente revisa tu tarjeta gráfica, tu RAM y tu disco, elige el modelo adecuado, lo instala y lo arranca, y te explica
|
||||
cómo conectar tus aplicaciones. Las herramientas de IA también pueden instalar, arrancar y detener Strata mediante su
|
||||
[servidor MCP](docs/MCP_SERVER.md).
|
||||
|
||||
¿Prefieres compilarlo tú? Mira [Compilar desde el código fuente](#compilar-desde-el-código-fuente).
|
||||
|
||||
## Modelos
|
||||
|
||||
La aplicación y `strata launch` recomiendan un tamaño según tu memoria. Los tamaños pequeños son más rápidos. Los grandes
|
||||
son algo más inteligentes.
|
||||
|
||||
### ¿Poca memoria? Prueba Gyro
|
||||
|
||||
[Gyro](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF) de agentionai es la forma más pequeña de ejecutar
|
||||
Qwen3.8-Flash-Next. Funciona en NVIDIA, AMD, Intel Arc y Apple Silicon.
|
||||
|
||||
| Tamaño | Memoria que necesita (RAM + VRAM) | Descarga | Para qué sirve |
|
||||
| --- | ---: | ---: | --- |
|
||||
| **Gyro-S** | 27,6 GB | 58,5 GB | PC donde no cabe nada más. 9 tokens/s en un Mac mini de 32 GB. |
|
||||
| **Gyro-M** | 35 GB | 92,0 GB | Calidad parecida a IQ3_XXS con un 20 % menos de memoria de GPU. |
|
||||
|
||||
Arráncalo con `strata launch --model gyro-s`. Detalles: [docs/GYRO.md](docs/GYRO.md).
|
||||
|
||||
### Los tamaños de Qwen3.8-Flash-Next
|
||||
|
||||
| Tu memoria (RAM + VRAM) | Elige | Por qué |
|
||||
| --- | --- | --- |
|
||||
| **32 GB** | **Gyro-S**, o el **[Coder](docs/MODELS.md#coder)** para código | El Coder conserva los expertos que usa el código |
|
||||
| **40 GB** | **[Swift 1.5](docs/MODELS.md#swift-15)** IQ2_XS (39,2 GB) o **Gyro-M** | Swift piensa menos tiempo antes de responder |
|
||||
| **48 GB** | **IQ3_XXS** (47,0 GB) | Más lento, mejor |
|
||||
| **64 GB** | **IQ3_S** (54,8 GB) | Iguala al modelo completo en las pruebas publicadas |
|
||||
| **96 GB o más** | **UD-IQ4_XS** hasta **Q8_0** de Unsloth | Lo más cerca del modelo completo |
|
||||
|
||||
Con menos RAM de la que indica una fila también puede funcionar, solo que más lento: Strata pasa solo a su
|
||||
[modo de poca RAM](docs/MODELS.md#a-big-graphics-card-and-little-ram). También hay:
|
||||
[NVFP4](docs/NVFP4.md) para tarjetas NVIDIA, [Qwen3.6-35B-A3B](docs/QWEN36.md) (más pequeño, varias peticiones a la vez) y
|
||||
[el Uncensored IQ3_XXS de OrcaRouter](docs/MODELS.md#orcarouter-uncensored-iq3_xxs). Todos los tamaños y lo que cabe en cada equipo:
|
||||
[docs/MODELS.md](docs/MODELS.md).
|
||||
|
||||
### DeepSeek V4.1-Flash (Terminal, desde 0.2.4)
|
||||
|
||||
Un modelo mucho más grande que funciona con una sola tarjeta gráfica de 12 GB y de 46 a 64 GB de RAM. Strata mantiene en
|
||||
memoria las partes más usadas y lee el resto de tu SSD. [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF)
|
||||
hizo los conjuntos. Funciona en tarjetas NVIDIA RTX de las series 30, 40 y 50 y en la Radeon AI PRO R9700. Elígelo en
|
||||
`strata launch`. La aplicación de escritorio todavía no lo ofrece.
|
||||
|
||||
| Conjunto | Descarga | Calidad | Escribe código a |
|
||||
| --- | ---: | --- | --- |
|
||||
| **IQ1_M** (1,75 bits) | 239 GB | El más pequeño, más débil con prosa sencilla | aún sin medir en un PC |
|
||||
| **IQ2_XXS** (2.00 bits) | 257 GB | Buen equilibrio | 10 tokens/s (RTX 3060), 5 (RTX 5070), 48 (4x R9700) |
|
||||
| **IQ2_XS** (2.25 bits) | 274 GB | Lo más cerca del original | 15 tokens/s (una R9700), 80 (4x R9700) |
|
||||
| **IQ4_NL** (tabla Engram) | incluida en cada descarga de arriba | La tabla de consulta que comparten los tres conjuntos | |
|
||||
|
||||
Arráncalo con `strata launch --model ist-das-2.00bpw` (o `ist-das-1.75bpw`, `ist-das-2.25bpw`). Cifras de calidad,
|
||||
velocidades por PC e instalación manual: [docs/DEEPSEEK41.md](docs/DEEPSEEK41.md).
|
||||
|
||||
## ¿Qué velocidad tiene?
|
||||
|
||||
Medido con una tarjeta gráfica de 12 GB (RTX 5070, 64 GB de RAM). Resultados con otras tarjetas:
|
||||
[velocidad de cada tamaño](docs/MODELS.md#how-fast-is-each-size) y [resultados de la comunidad](docs/COMMUNITY_BENCHMARKS.md).
|
||||
|
||||
| Modelo | Las respuestas aparecen a | Lee tu prompt a |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████████` 94 tokens/s | `████████████` 2.650 tokens/s |
|
||||
| **IQ2_XS** | `██████████░░` 79 tokens/s | `█████████░░░` 2.090 tokens/s |
|
||||
| **IQ3_XXS** | `████████░░░░` 62 tokens/s | `████████░░░░` 1.750 tokens/s |
|
||||
| **IQ3_S** | `███████░░░░░` 53 tokens/s | `███████░░░░░` 1.620 tokens/s |
|
||||
| **Coder** | `███████░░░░░` 55 tokens/s | `██████████░░` 2.180 tokens/s |
|
||||
|
||||
Con una tarjeta AMD de 16 GB (RX 9070 XT, 47 GB de RAM):
|
||||
|
||||
| Modelo | Las respuestas aparecen a | Lee tu prompt a |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████░░░░` 60 tokens/s | `█████░░░░░░░` 1.160 tokens/s |
|
||||
| **IQ2_XS** | `███████░░░░░` 52 tokens/s | `█████░░░░░░░` 1.110 tokens/s |
|
||||
| **Coder** | `██████░░░░░░` 44 tokens/s | `██████░░░░░░` 1.420 tokens/s |
|
||||
|
||||
Un token equivale a unos ¾ de palabra, así que 60 tokens por segundo es más rápido de lo que puedes leer. "Lee tu prompt"
|
||||
es lo rápido que Strata procesa un documento largo, código o un historial de chat.
|
||||
|
||||
## Uso
|
||||
|
||||
### En la aplicación
|
||||
|
||||
Un chat, un **Monitor** del modelo y de tu GPU, CPU y RAM, una página de **Modelos** y los **Ajustes**. El chat puede
|
||||
trabajar en una carpeta de tu PC. En modo **Manual** pregunta antes de cambiar un archivo o ejecutar un comando. Puede buscar
|
||||
en la web (DuckDuckGo o Bing, sin cuenta). Las palabras que buscas son lo único que sale de tu PC.
|
||||
|
||||
<p align="center"><img src="docs/media/app-chat.png" width="900" alt="La aplicación Strata: un chat donde el agente construyó un pequeño jardín con una pagoda de vóxeles, con la página abierta en la vista previa a la derecha"><br>
|
||||
<sub>El agente escribió un jardín con una pagoda de vóxeles en un solo archivo HTML, y la página se abre en la vista previa junto al chat</sub></p>
|
||||
|
||||
### En la Terminal
|
||||
|
||||
<p align="center"><img src="docs/media/terminal-monitor.png" width="900" alt="La vista general del monitor de la Terminal de Strata mientras el modelo escribe una respuesta"><br>
|
||||
<sub><code>strata monitor</code> mientras el modelo responde: velocidad, GPU, memoria y las últimas solicitudes</sub></p>
|
||||
|
||||
```sh
|
||||
strata launch # guided: model, context, pictures, access, then serve
|
||||
strata launch --model q2_0 --yes --no-tui # no questions: serve and print the address
|
||||
strata launch --status # what is being served now
|
||||
strata launch --stop # stop it
|
||||
strata monitor # watch speed, GPU, memory and requests
|
||||
strata update # get the newest engine
|
||||
strata doctor # check the installed files
|
||||
```
|
||||
|
||||
Para llegar a Strata desde tu móvil u otro PC, usa `strata launch --access lan`. Crea una clave de API para ti.
|
||||
`strata --help` lo lista todo, y [docs/TERMINAL.md](docs/TERMINAL.md) lo explica.
|
||||
|
||||
### Con agentes de IA y otras aplicaciones
|
||||
|
||||
Un agente instala Strata con el comando de una línea de arriba y ejecuta `strata launch --model NAME --yes --no-tui`. Los
|
||||
pasos, las opciones y los códigos de salida están en [docs/AI_SETUP.md](docs/AI_SETUP.md). Las reglas del repositorio para agentes: [AGENTS.md](AGENTS.md).
|
||||
|
||||
| Tu aplicación habla | URL base |
|
||||
| --- | --- |
|
||||
| API de OpenAI (la mayoría de aplicaciones y agentes) | `http://127.0.0.1:8080/v1` (cualquier clave de API, cualquier nombre de modelo) |
|
||||
| API de Anthropic (Claude Code) | `http://127.0.0.1:8080`, sin `/v1`. Claude Code: `ANTHROPIC_BASE_URL=http://127.0.0.1:8080` |
|
||||
| API OpenAI Responses (Codex CLI) | `http://127.0.0.1:8080/v1/responses` ([configuración](docs/DETAILS.md#the-responses-api-and-codex-cli)) |
|
||||
|
||||
`strata connect` muestra los ajustes exactos para OpenCode, Codex CLI, Claude Code, oh-my-pi, pi, Hermes Agent y OpenClaw,
|
||||
y `strata connect opencode --write` los escribe por ti (con una copia de seguridad).
|
||||
|
||||
### Conviene saber
|
||||
|
||||
- **Las imágenes** vienen activadas en todos los modelos que pueden leerlas. Adjunta o pega una imagen en el chat, o envíala desde tu aplicación. Detalles: [docs/IMAGES.md](docs/IMAGES.md).
|
||||
- **El razonamiento** puede estar desactivado, bajo, medio o alto, en el menú del chat o en "reasoning effort" de tu aplicación. Desactivado es lo más rápido.
|
||||
- **El primer mensaje** de un chat se lee entero, unos 1 minuto por cada 30.000 tokens. Los mensajes siguientes empiezan en segundos.
|
||||
- **Una petición a la vez** es lo normal. Para varias, usa `--parallel N` ([BATCHING.md](docs/BATCHING.md)). En una tarjeta de 12 GB, esto hace más lenta cada respuesta.
|
||||
|
||||
## Compilar desde el código fuente
|
||||
|
||||
<details>
|
||||
<summary>Windows, Linux y Mac desde el repositorio</summary>
|
||||
|
||||
[Descarga Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) y descomprímelo, o haz `git clone`.
|
||||
**Windows:** haz doble clic en `START-HERE.bat`. **Linux:** ejecuta `./setup.sh` en la carpeta de Strata.
|
||||
|
||||
**Mac (Apple Silicon, macOS 15 o posterior):** instala una vez las Command Line Tools de Apple con `xcode-select --install`
|
||||
(no hace falta Xcode). Abre la Terminal en la carpeta de Strata y ejecuta `./setup.sh`. El script compila el motor para Mac
|
||||
en tu Mac (unos 7 minutos), hace las mismas preguntas, descarga el modelo y lo arranca. La próxima vez, `./setup.sh` lo
|
||||
arranca enseguida. Qué tamaño cabe en tu Mac: [docs/MAC.md](docs/MAC.md).
|
||||
|
||||
El instalador detecta tu tarjeta y prepara el motor adecuado. Pregunta qué modelo y qué tamaño quieres, cuánto contexto
|
||||
(cuánto texto mantiene el modelo en mente) y si debe leer imágenes. Pulsa Enter para aceptar la respuesta recomendada.
|
||||
Luego descarga el modelo (unos 70 GB) y lo arranca. Si la descarga se detiene, vuelve a ejecutar el script y continúa. Tu
|
||||
navegador abre la página de Strata en `http://127.0.0.1:8080`.
|
||||
|
||||
> Mientras el modelo arranca, tu PC puede ir lento o dejar de responder durante 1-3 minutos (más tiempo la primera vez).
|
||||
> Strata carga 35-55 GB en tu RAM y bloquea una parte para la tarjeta gráfica. Espera y no cierres la ventana.
|
||||
|
||||
La próxima vez, ejecuta de nuevo `START-HERE.bat` (o `./setup.sh`). Arranca enseguida y no descarga nada dos veces. Cierra
|
||||
su ventana para detener el modelo. `UPDATE.bat` (`./update.sh`) actualiza Strata sin arrancarlo. Actualización, Docker,
|
||||
varias tarjetas, dónde van los archivos y todas las opciones: [docs/INSTALL.md](docs/INSTALL.md).
|
||||
|
||||
Para medir tu PC, ejecuta `BENCH.bat` (`./bench.sh`). Tarda 5-10 minutos y escribe un informe que puedes compartir
|
||||
([cómo](docs/COMMUNITY_BENCHMARKS.md#run-the-benchmark)).
|
||||
|
||||
**Opcional: más velocidad en una tarjeta cuando todo el modelo cabe en la RAM.** Dos ajustes que se activan a mano, de
|
||||
RubenTorney (#1904, #1905). Ayudan cuando usas una sola tarjeta gráfica y tu RAM guarda todos los expertos, por ejemplo con
|
||||
Q2_0 o IQ3_XXS en un PC de 64 GB. Abre `strata-<model>.json` en la carpeta de datos de Strata ([dónde](docs/INSTALL.md#where-things-are-stored)) y añade:
|
||||
|
||||
- `"--adapt-async", "1"` a la lista `"args"`. Así la caché de expertos se actualiza en un hilo auxiliar en vez de detener la
|
||||
respuesta. Medido en una RTX 5070 (Windows, Q2_0): +4-5 % en decodificación, +7-10 % en código y documentos largos. Las
|
||||
respuestas pueden diferir un poco de las del ajuste por defecto y son iguales de una ejecución a otra.
|
||||
- `"STRATA_PIN_BY_NEED": "1"` en el bloque `"env"` (Windows). Windows solo puede bloquear aproximadamente la mitad de la RAM
|
||||
para la tarjeta gráfica, y este ajuste bloquea primero los expertos que la tarjeta necesita antes. Medido en la RTX 5070: +4,4 % en lectura del prompt.
|
||||
|
||||
```json
|
||||
"args": [ ..., "--adapt-async", "1" ],
|
||||
"env": { "STRATA_PIN_BY_NEED": "1" }
|
||||
```
|
||||
|
||||
Reinicia Strata después del cambio. Para deshacerlo, quita las dos entradas.
|
||||
Sigue las indicaciones para descargar e iniciar un modelo. [Guía del Terminal](docs/TERMINAL_LAUNCH.md).
|
||||
|
||||
</details>
|
||||
|
||||
## ¿Cómo funciona?
|
||||
<p align="center">
|
||||
<img src="docs/media/terminal-monitor.png" width="900" alt="Aplicación de Terminal de Strata con la actividad del modelo, la velocidad, la memoria y las peticiones recientes">
|
||||
<br><sub>Observa tu modelo y tu hardware mientras se ejecuta.</sub>
|
||||
</p>
|
||||
|
||||
Los modelos como este suelen ejecutarse en servidores con cientos de gigabytes de memoria gráfica. Tu tarjeta gráfica tiene
|
||||
12-24 GB. Strata hace que el modelo quepa repartiendo el trabajo por todo tu PC. Imagina una biblioteca.
|
||||
<a id="build-from-source"></a>
|
||||
|
||||
- El modelo es un equipo de 24.576 pequeños especialistas ("expertos"), como 24.576 libros de consulta. Cada palabra necesita solo 10 de ellos.
|
||||
- **Tu tarjeta gráfica es el escritorio.** Es pequeño y rápido. Los unos 4.000 libros que más se usan están abiertos encima.
|
||||
- **Tu RAM son las estanterías de la sala de lectura.** Los 24.576 libros están ahí. Cuando una palabra necesita un libro que
|
||||
no está en el escritorio, tu procesador lo coge enseguida de la estantería mientras la tarjeta sigue trabajando.
|
||||
- **Tu SSD es el archivo.** Guarda una tabla de consulta de 29 GB. El modelo lee unas pocas filas por palabra, así que no
|
||||
hace falta que esté en la RAM.
|
||||
[Compilar desde el código fuente](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [Guía de instalación completa](docs/INSTALL.md)
|
||||
|
||||
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Los 24.576 expertos del modelo: los más usados en la tarjeta gráfica, todos en la RAM, una tabla de consulta en el SSD"></p>
|
||||
<a id="what-you-need"></a>
|
||||
|
||||
Un pequeño ayudante también adivina las siguientes palabras. El modelo grande las comprueba todas a la vez y se queda con
|
||||
las correctas. Obtienes la misma respuesta, entre 1,6 y 1,8 veces antes. Los textos largos se leen en trozos grandes, de
|
||||
hasta 8.192 tokens a la vez, a más de 1.000 tokens por segundo.
|
||||
## Hardware
|
||||
|
||||
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Un pequeño ayudante adivina las siguientes palabras; el modelo grande las comprueba todas a la vez y se queda con las correctas"></p>
|
||||
| Hardware compatible | Guía |
|
||||
| --- | --- |
|
||||
| **NVIDIA GeForce RTX 20 / 30 / 40 / 50** | [Instalación en NVIDIA](docs/INSTALL.md#what-you-need) |
|
||||
| **AMD Radeon RX 7700 / 7800 / 7900, RX 9060 / 9070, AI PRO R9700** | [Instalación en AMD](docs/AMD_HIP.md) |
|
||||
| **AMD Radeon RX 6800 / 6900 · experimental** | [Instalación en RX 6000](docs/AMD_HIP.md#rdna2-gfx1030) |
|
||||
| **Intel Arc · Linux** | [Instalación en Intel](docs/INTEL_ARC.md) |
|
||||
| **Apple Silicon · M1 o posterior** | [Instalación en Mac](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max · Strix Halo** | [Instalación en Strix Halo](docs/STRIX_HALO.md) |
|
||||
|
||||
La explicación más larga: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Cada parte con sus cifras:
|
||||
[los detalles](docs/DETAILS.md#how-it-works) y el [artículo](docs/paper/Strata-Paper.pdf).
|
||||
<a id="using-it"></a>
|
||||
|
||||
## ¿Algo ha ido mal?
|
||||
## Herramientas
|
||||
|
||||
- **Mi PC se congeló la primera vez que arrancó Strata.** Carga el modelo en la RAM. Espera y no cierres la ventana. ¿Sigue
|
||||
congelado después de 10 minutos? Reinicia el PC, cierra otros programas e inténtalo de nuevo, o elige un tamaño más pequeño.
|
||||
- **Se detuvo durante la descarga o la instalación.** Abre de nuevo la aplicación o `strata launch`. Continúa donde se
|
||||
quedó.
|
||||
- **Va muy lento, o dice "the engine stopped unexpectedly".** Tu PC no tiene bastante RAM libre. Cierra otros programas
|
||||
(los navegadores gastan mucha) o elige un tamaño más pequeño, como Q2_0 o IQ2_XS.
|
||||
- **Dice que no hay motor para este PC.** En la aplicación, abre **Ajustes, Motor** y pulsa **Actualizar motor**. En la
|
||||
Terminal, ejecuta `strata update`.
|
||||
- **Dice que el puerto 8080 ya está en uso.** Strata ya se está ejecutando. Busca su ventana.
|
||||
**Claude Code · Codex CLI · OpenCode · oh-my-pi · pi.dev · Hermes Agent · OpenClaw**, y más, mediante las API compatibles con OpenAI y Anthropic.
|
||||
|
||||
Más soluciones: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). ¿Sigues atascado? Abre una
|
||||
[incidencia](https://github.com/Niko1221/Strata/issues) y adjunta los registros: en la aplicación desde **Ajustes, Motor,
|
||||
Diagnóstico, Guardar registros**, en la Terminal la salida de `strata doctor`. Para un problema de seguridad, avísanos en
|
||||
privado: [SECURITY.md](SECURITY.md).
|
||||
Ejecuta **`strata connect`** para ver las instrucciones de tu herramienta. [Guía de conexión](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
|
||||
|
||||
## Créditos y licencia
|
||||
<a id="guides"></a>
|
||||
<a id="how-fast-is-it"></a>
|
||||
<a id="how-does-it-work"></a>
|
||||
<a id="something-went-wrong"></a>
|
||||
|
||||
El modelo es [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), del equipo de Qwen. Lo comprimieron
|
||||
[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5), agentionai (Gyro)
|
||||
y Unsloth. ISTA-DASLab hizo también los conjuntos de DeepSeek V4.1-Flash. Strata usa partes de
|
||||
[llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Todos los créditos: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits).
|
||||
Strata es de código abierto con la [licencia MIT](LICENSE). Algunas partes y todos los modelos tienen sus propias licencias
|
||||
([cuáles](docs/HOW_IT_WORKS.md#license)).
|
||||
## Ayuda
|
||||
|
||||
## Contribuir
|
||||
|
||||
¿Has encontrado un error, o quieres enviar un cambio o un benchmark? Lee primero [CONTRIBUTING.md](CONTRIBUTING.md). Explica
|
||||
qué incluir para que otros puedan comprobar tu trabajo.
|
||||
[Todas las guías](docs) · [Solución de problemas](docs/TROUBLESHOOTING.md) · [Informar de un error](https://github.com/Niko1221/Strata/issues/new/choose) ·
|
||||
[Benchmarks](docs/COMMUNITY_BENCHMARKS.md) · [Cómo funciona](docs/HOW_IT_WORKS.md) · [Contribuir](CONTRIBUTING.md)
|
||||
|
||||
## Apoya Strata
|
||||
|
||||
Strata es gratis y tiene licencia MIT. Lo construyo y lo pruebo en el hardware que consigo. Si te funciona bien en tu PC, un café me ayuda
|
||||
a seguir probando tarjetas nuevas y a arreglar lo que nos cuentas. Sin presión. Usarlo y contar lo que encuentras ayuda igual de mucho.
|
||||
**Ayuda a mantener la IA local abierta a todos.** Un café ayuda a financiar el desarrollo y las pruebas en más hardware.
|
||||
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: apoya Strata" height="48"></a>
|
||||
|
||||
¿Te interesa patrocinar Strata? Escríbeme a través de [GitHub](https://github.com/Niko1221).
|
||||
[Licencia MIT](LICENSE) · [Créditos y licencias de los modelos](docs/HOW_IT_WORKS.md#credits) · [Privacidad](docs/SAFETY.md)
|
||||
|
||||
<a id="option-c-let-your-ai-set-it-up"></a>
|
||||
|
||||
<sub>Para herramientas de IA y lectores automáticos: <a href="AGENTS.md">referencia técnica</a> · <a href="docs/AI_SETUP.md">instrucciones de instalación</a>.</sub>
|
||||
|
||||
**Patrocinios, alianzas de hardware o colaboraciones:** [[email protected]](mailto:[email protected]).
|
||||
+122
-301
@@ -1,367 +1,188 @@
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="apps/host/public/brand/strata-lockup-dark.svg">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="380">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="340">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <b>Français</b> · <a href="README.es.md">Español</a> · <a href="README.pt-BR.md">Português</a></p>
|
||||
|
||||
<h3 align="center">Faites tourner un modèle de pointe sur votre propre PC de jeu</h3>
|
||||
|
||||
<p align="center">Compatible avec NVIDIA, AMD, Intel · Windows, Linux, Mac · gratuit et open source</p>
|
||||
<h3 align="center">Une IA puissante sur votre propre ordinateur.</h3>
|
||||
<p align="center">Discutez, programmez et travaillez avec des images, en local.</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?include_prereleases&label=release" alt="Dernière version"></a>
|
||||
<a href="#modèles">Modèles</a> · <a href="#installation"><b>Installation</b></a> ·
|
||||
<a href="#matériel">Matériel</a> · <a href="#outils">Outils</a> ·
|
||||
<a href="#aide">Aide</a> · <a href="#soutenir-strata">Soutenir</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Télécharger Strata"></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="Licence MIT"></a>
|
||||
<a href="https://github.com/Niko1221/Strata/stargazers"><img src="https://img.shields.io/github/stars/Niko1221/Strata?style=flat" alt="Étoiles GitHub"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black" alt="Offrez-moi un café"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Soutenir Strata sur Buy Me a Coffee"></a>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Un jardin de pagode en voxels écrit par le modèle de Strata, qui tourne dans le navigateur"></a><br>
|
||||
<sub>Un jardin de pagode en voxels, généré en une seule requête sur une RTX 5070 avec Strata (IQ3_S, contexte de 128K) ·
|
||||
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">vidéo complète (49 s)</a></sub></p>
|
||||
<p align="center"><sub><a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <b>Français</b> · <a href="README.es.md">Español</a> · <a href="README.pt-BR.md">Português</a></sub></p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#installation"><b>Installation</b></a> ·
|
||||
<a href="#modèles">Modèles</a> ·
|
||||
<a href="#quelle-est-sa-vitesse-">Vitesse</a> ·
|
||||
<a href="#utilisation">Utilisation</a> ·
|
||||
<a href="#comment-ça-marche-">Comment ça marche</a> ·
|
||||
<a href="#un-problème-">Aide</a> ·
|
||||
<a href="#soutenir-strata">Soutenir</a>
|
||||
</p>
|
||||
**Gratuit, open source, et à vous de l'exécuter.** Utilisez Strata sur votre bureau ou avec vos outils de programmation préférés. Aucun compte nécessaire.
|
||||
|
||||
Strata fait tourner **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)**, un modèle de 125 milliards
|
||||
de paramètres qui demande normalement un serveur, sur un PC ordinaire. Depuis la 0.2.4, il fait aussi tourner **DeepSeek
|
||||
V4.1-Flash** dans le Terminal. Il discute, écrit du code, lit des images et fonctionne avec vos applications et vos agents
|
||||
de code. Vos conversations et vos fichiers restent sur votre PC.
|
||||
## Modèles
|
||||
|
||||
- **Tourne sur un PC de jeu.** 32 Go de RAM et de mémoire graphique réunis suffisent pour commencer.
|
||||
- **Fonctionne avec vos outils.** Claude Code, Codex CLI, OpenCode et d'autres agents se connectent par les API d'OpenAI et d'Anthropic.
|
||||
- **Lit les images**, activé par défaut.
|
||||
- **Gratuit et privé.** Licence MIT, pas de compte, pas de cloud, aucune télémétrie ([ce qu'il télécharge et quand](docs/SAFETY.md)).
|
||||
Nous recommandons **d'abord ISTA-DASLab**, puis **Unsloth** et d'autres fournisseurs, lorsqu'un modèle compatible tient en mémoire.
|
||||
|
||||
## Ce qu'il vous faut
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.8-Flash-Next</b></summary>
|
||||
|
||||
**Il vous faut au moins 32 Go de mémoire : RAM et mémoire graphique (VRAM) réunis.** Sur un Mac ou un Ryzen AI Max, c'est
|
||||
la mémoire unifiée. Un PC de 32 Go avec une carte de 12 à 16 Go fait tourner les petites tailles, et 64 Go font tourner
|
||||
toutes les tailles.
|
||||
|
||||
| Fonctionne sur | Guide |
|
||||
| Fournisseur | Choix |
|
||||
| --- | --- |
|
||||
| **NVIDIA** GeForce RTX série 20, 30, 40 et 50, 12 Go ou plus | [Installation](#installation) |
|
||||
| **AMD** Radeon RX 6800 et plus récent, Radeon AI PRO R9700 | [Cartes AMD](docs/AMD_HIP.md) |
|
||||
| **Intel** Arc | [Intel Arc](docs/INTEL_ARC.md) |
|
||||
| **Mac Apple Silicon**, M1 ou plus récent, macOS 15 ou plus récent | [Mac](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max (Strix Halo)** et autres PC à mémoire unifiée | [Strix Halo](docs/STRIX_HALO.md) |
|
||||
| Cartes et processeurs plus anciens | [GPU anciens](docs/OLDER_GPUS.md) |
|
||||
| **ISTA-DASLab · recommandé** | Q2\_0, IQ2\_XS, IQ3\_XXS, IQ3\_S. [Coder : IQ1\_M](docs/MODELS.md#coder) pour programmer. |
|
||||
| **Unsloth** | UD-IQ4\_XS, UD-Q4\_K\_XL, UD-Q5\_K\_XL, UD-Q6\_K\_XL, Q8\_0 |
|
||||
| **Gyro · agentionai** | [Gyro-S, Gyro-M](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF). Utilise moins de mémoire, mais plus lent. |
|
||||
| **Autres options** | [Swift 1.5](docs/MODELS.md#swift-15), [NVFP4](docs/NVFP4.md), [OrcaRouter](docs/ORCA.md) |
|
||||
|
||||
Il vous faut aussi environ 80 Go d'espace disque libre (un SSD), Windows 10 / 11, Linux ou macOS, et un pilote graphique
|
||||
récent. Deux ou trois cartes de la même marque peuvent se partager un modèle ([multi-GPU](docs/MULTI_GPU.md)).
|
||||
[Comparer les tailles, la mémoire nécessaire et la compatibilité](docs/MODELS.md).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.6-35B-A3B</b></summary>
|
||||
|
||||
**Unsloth:** UD-Q4_K_XL, UD-Q6_K_XL, Q8_0. Ce sont les paquets actuellement documentés pour ce modèle.
|
||||
|
||||
[Guide du modèle et compatibilité](docs/QWEN36.md).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/deepseek.svg" width="22" alt="" align="middle"> <b>DeepSeekV4.1-Flash-1017</b></summary>
|
||||
|
||||
Dans l'**application de bureau** et dans le **Terminal**. Ce qui ne tient pas en mémoire est lu depuis votre SSD : il faut donc beaucoup d'espace disque libre.
|
||||
|
||||
| Jeu (IST-DAS) | Téléchargement | Espace disque libre nécessaire |
|
||||
| --- | ---: | ---: |
|
||||
| 1.75 bpw | 239 GB | environ 245 GB |
|
||||
| 2.00 bpw | 257 GB | environ 265 GB |
|
||||
| 2.25 bpw | 274 GB | environ 280 GB |
|
||||
|
||||
Fonctionne sur les NVIDIA RTX 30, 40 et 50 (Windows et Linux) et sur l'AMD Radeon AI PRO R9700 (Linux). L'installation télécharge les fichiers et le moteur DeepSeek, puis démarre le modèle.
|
||||
|
||||
```sh
|
||||
strata launch --family deepseek41 --model ist-das-2.00bpw --plan
|
||||
```
|
||||
|
||||
Mesuré avec le jeu 2.00 bpw sur Strata 0.2.5. « Sortie » est la vitesse de la réponse, « Prompt » la vitesse à laquelle Strata lit votre texte.
|
||||
|
||||
| Matériel | Sortie | Prompt |
|
||||
| --- | ---: | ---: |
|
||||
| RTX 5070 12 GB, 64 GB de RAM | 4 à 6 tokens/s | environ 110 tokens/s (4K) |
|
||||
| RTX 3060 12 GB, 46 GB de RAM | 9 à 11 tokens/s | environ 155 tokens/s (4K) |
|
||||
| 1x Radeon AI PRO R9700, 251 GB de RAM | 21 à 25 tokens/s | 320 à 360 tokens/s (8K) |
|
||||
| 4x Radeon AI PRO R9700 | 74 à 77 tokens/s | 400 à 450 tokens/s (8K) |
|
||||
|
||||
[Mémoire, disque et détails](docs/DEEPSEEK41.md) · [Notes de version](https://github.com/Niko1221/Strata/releases)
|
||||
|
||||
</details>
|
||||
|
||||
<a id="the-other-sizes"></a>
|
||||
|
||||
Vous ne savez pas ce qui tient ? [Choisir un modèle pour votre ordinateur](docs/MODELS.md#pick-by-memory).
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. L'application Strata (le plus simple)
|
||||
<a id="option-a-the-strata-app"></a>
|
||||
|
||||
Téléchargez le fichier pour votre ordinateur, ouvrez-le et appuyez sur Installer pour le modèle qu'elle recommande.
|
||||
### Application de bureau
|
||||
|
||||
**Windows 10 / 11**
|
||||
**[Télécharger Strata pour Windows, Linux ou Mac →](https://github.com/Niko1221/Strata/releases/latest)**
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-windows-x64-setup.exe)
|
||||
Installez l'application, choisissez un modèle et commencez à discuter. Il y a un seul installateur par système. Au premier lancement, l'application télécharge le moteur de votre carte graphique. [Aide pour l'installateur](docs/INSTALL.md#the-app).
|
||||
|
||||
**Linux**
|
||||
<sub>Fichiers: `Strata-0.2.6-windows-x64-setup.exe` · `Strata-0.2.6-linux-x86_64.AppImage` · `Strata-0.2.6-macos-arm64.dmg`</sub>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-linux-x86_64.AppImage)
|
||||
<p align="center">
|
||||
<img src="docs/media/app-chat.png" width="900" alt="Application de bureau Strata avec une discussion et un aperçu du code en direct côte à côte">
|
||||
<br><sub>Discussion, fichiers et aperçu en direct dans une seule fenêtre.</sub>
|
||||
</p>
|
||||
|
||||
**Mac** (Apple Silicon, macOS 15 ou plus récent)
|
||||
<a id="option-b-the-terminal-one-line"></a>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-macos-arm64.dmg)
|
||||
### Application Terminal
|
||||
|
||||
L'application choisit toute seule le bon moteur pour votre carte graphique. Tous les fichiers sont sur la
|
||||
[page des versions](https://github.com/Niko1221/Strata/releases/latest).
|
||||
Vous préférez le terminal ? Installez Strata, puis lancez **`strata launch`**.
|
||||
|
||||
| Premier démarrage | Ce qu'il faut faire |
|
||||
| --- | --- |
|
||||
| **Windows** | Lancez le `.exe`. Pas besoin de droits d'administrateur. Si Windows affiche « Windows a protégé votre ordinateur », choisissez **Informations complémentaires**, puis **Exécuter quand même**. |
|
||||
| **Linux** | Rendez le fichier exécutable (`chmod +x`) et lancez-le. Il a besoin de FUSE 2 : `sudo apt install libfuse2t64` (Ubuntu 24.04 et plus récent) ou `libfuse2` (22.04). |
|
||||
| **Mac** | Ouvrez le `.dmg` et faites glisser Strata dans Applications. |
|
||||
<details open>
|
||||
<summary><b>Commandes d'installation</b></summary>
|
||||
|
||||
Le téléchargement du modèle est gros (40 à 70 Go). S'il s'arrête, il reprend là où il s'était arrêté.
|
||||
|
||||
### 2. Le Terminal (une ligne)
|
||||
|
||||
La commande `strata`, pour les scripts, les serveurs et les agents d'IA. C'est aussi le moyen de faire tourner DeepSeek.
|
||||
|
||||
Linux et Mac :
|
||||
**Linux / Mac**
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
```
|
||||
|
||||
Windows (PowerShell) :
|
||||
**Windows PowerShell**
|
||||
|
||||
```powershell
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
```
|
||||
|
||||
Puis lancez-le. Il pose une question à la fois (modèle, contexte, images, qui peut se connecter), télécharge ce dont il a
|
||||
besoin et commence à servir :
|
||||
Sous Windows, ouvrez ensuite un nouveau terminal. Puis :
|
||||
|
||||
```sh
|
||||
strata launch
|
||||
```
|
||||
|
||||
Pas besoin de droits d'administrateur. L'installation vérifie l'empreinte SHA-256 de tout ce qu'elle télécharge. Sous
|
||||
Windows, ouvrez une nouvelle fenêtre de terminal après l'installation. Tous les écrans et toutes les options : [docs/TERMINAL_LAUNCH.md](docs/TERMINAL_LAUNCH.md).
|
||||
|
||||
### 3. Laissez votre IA l'installer
|
||||
|
||||
Collez ceci dans Claude Code, Cursor, Codex, GitHub Copilot ou un autre assistant de programmation :
|
||||
|
||||
```text
|
||||
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
|
||||
```
|
||||
|
||||
L'assistant vérifie votre carte graphique, votre RAM et votre disque, choisit le modèle adapté, l'installe, le démarre et
|
||||
vous explique comment connecter vos applications. Les outils d'IA peuvent aussi installer, démarrer et arrêter Strata via son
|
||||
[serveur MCP](docs/MCP_SERVER.md).
|
||||
|
||||
Vous préférez le compiler vous-même ? Voir [Compiler depuis les sources](#compiler-depuis-les-sources).
|
||||
|
||||
## Modèles
|
||||
|
||||
L'application et `strata launch` recommandent une taille selon votre mémoire. Les petites tailles sont plus rapides. Les
|
||||
grandes sont un peu plus intelligentes.
|
||||
|
||||
### Peu de mémoire ? Essayez Gyro
|
||||
|
||||
[Gyro](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF) d'agentionai est la façon la plus légère de faire tourner
|
||||
Qwen3.8-Flash-Next. Il fonctionne sur NVIDIA, AMD, Intel Arc et Apple Silicon.
|
||||
|
||||
| Taille | Mémoire nécessaire (RAM + VRAM) | Téléchargement | Pour quoi |
|
||||
| --- | ---: | ---: | --- |
|
||||
| **Gyro-S** | 27,6 Go | 58,5 Go | Les PC où rien d'autre ne tient. 9 tokens/s sur un Mac mini de 32 Go. |
|
||||
| **Gyro-M** | 35 Go | 92,0 Go | À peu près la qualité d'IQ3_XXS avec 20 % de mémoire GPU en moins. |
|
||||
|
||||
Lancez-le avec `strata launch --model gyro-s`. Détails : [docs/GYRO.md](docs/GYRO.md).
|
||||
|
||||
### Les tailles de Qwen3.8-Flash-Next
|
||||
|
||||
| Votre mémoire (RAM + VRAM) | Prenez | Pourquoi |
|
||||
| --- | --- | --- |
|
||||
| **32 Go** | **Gyro-S**, ou le **[Coder](docs/MODELS.md#coder)** pour le code | Le Coder garde les experts dont le code a besoin |
|
||||
| **40 Go** | **[Swift 1.5](docs/MODELS.md#swift-15)** IQ2_XS (39,2 Go) ou **Gyro-M** | Swift réfléchit moins longtemps avant de répondre |
|
||||
| **48 Go** | **IQ3_XXS** (47,0 Go) | Plus lent, meilleur |
|
||||
| **64 Go** | **IQ3_S** (54,8 Go) | Égale le modèle complet dans les tests publiés |
|
||||
| **96 Go ou plus** | **UD-IQ4_XS** à **Q8_0** d'Unsloth | Au plus près du modèle complet |
|
||||
|
||||
Avoir moins de RAM que ce qu'indique une ligne peut encore fonctionner, en plus lent : Strata passe tout seul en
|
||||
[mode peu de RAM](docs/MODELS.md#a-big-graphics-card-and-little-ram). Il existe aussi :
|
||||
[NVFP4](docs/NVFP4.md) pour les cartes NVIDIA, [Qwen3.6-35B-A3B](docs/QWEN36.md) (plus petit, plusieurs requêtes à la fois) et
|
||||
[l'Uncensored IQ3_XXS d'OrcaRouter](docs/MODELS.md#orcarouter-uncensored-iq3_xxs). Toutes les tailles et ce qui tient où :
|
||||
[docs/MODELS.md](docs/MODELS.md).
|
||||
|
||||
### DeepSeek V4.1-Flash (Terminal, depuis 0.2.4)
|
||||
|
||||
Un modèle bien plus gros, qui tourne sur une seule carte graphique de 12 Go avec 46 à 64 Go de RAM. Strata garde en mémoire
|
||||
les parties les plus utilisées et lit le reste sur votre SSD. [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF)
|
||||
a produit les versions. Il fonctionne sur les cartes NVIDIA RTX des séries 30, 40 et 50 et sur la Radeon AI PRO R9700. Choisissez-le
|
||||
dans `strata launch`. L'application de bureau ne le propose pas encore.
|
||||
|
||||
| Version | Téléchargement | Qualité | Écrit du code à |
|
||||
| --- | ---: | --- | --- |
|
||||
| **IQ1_M** (1,75 bit) | 239 Go | La plus petite, plus faible sur la prose simple | pas encore mesuré sur un PC |
|
||||
| **IQ2_XXS** (2.00 bits) | 257 Go | Bon équilibre | 10 tokens/s (RTX 3060), 5 (RTX 5070), 48 (4x R9700) |
|
||||
| **IQ2_XS** (2.25 bits) | 274 Go | Au plus près de l'original | 15 tokens/s (une R9700), 80 (4x R9700) |
|
||||
| **IQ4_NL** (table Engram) | compris dans chaque téléchargement ci-dessus | La table de consultation que les trois versions partagent | |
|
||||
|
||||
Lancez-le avec `strata launch --model ist-das-2.00bpw` (ou `ist-das-1.75bpw`, `ist-das-2.25bpw`). Chiffres de qualité,
|
||||
vitesses par PC et installation manuelle : [docs/DEEPSEEK41.md](docs/DEEPSEEK41.md).
|
||||
|
||||
## Quelle est sa vitesse ?
|
||||
|
||||
Mesuré avec une carte graphique de 12 Go (RTX 5070, 64 Go de RAM). Voir les résultats d'autres cartes :
|
||||
[vitesse de chaque taille](docs/MODELS.md#how-fast-is-each-size) et [résultats de la communauté](docs/COMMUNITY_BENCHMARKS.md).
|
||||
|
||||
| Modèle | Les réponses s'affichent à | Lit votre prompt à |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████████` 94 tokens/s | `████████████` 2 650 tokens/s |
|
||||
| **IQ2_XS** | `██████████░░` 79 tokens/s | `█████████░░░` 2 090 tokens/s |
|
||||
| **IQ3_XXS** | `████████░░░░` 62 tokens/s | `████████░░░░` 1 750 tokens/s |
|
||||
| **IQ3_S** | `███████░░░░░` 53 tokens/s | `███████░░░░░` 1 620 tokens/s |
|
||||
| **Coder** | `███████░░░░░` 55 tokens/s | `██████████░░` 2 180 tokens/s |
|
||||
|
||||
Avec une carte AMD de 16 Go (RX 9070 XT, 47 Go de RAM) :
|
||||
|
||||
| Modèle | Les réponses s'affichent à | Lit votre prompt à |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████░░░░` 60 tokens/s | `█████░░░░░░░` 1 160 tokens/s |
|
||||
| **IQ2_XS** | `███████░░░░░` 52 tokens/s | `█████░░░░░░░` 1 110 tokens/s |
|
||||
| **Coder** | `██████░░░░░░` 44 tokens/s | `██████░░░░░░` 1 420 tokens/s |
|
||||
|
||||
Un token vaut environ ¾ de mot, donc 60 tokens par seconde, c'est plus rapide que votre lecture. « Lit votre prompt »
|
||||
indique la vitesse à laquelle Strata absorbe un long document, du code ou un historique de discussion.
|
||||
|
||||
## Utilisation
|
||||
|
||||
### Dans l'application
|
||||
|
||||
Un chat, un **Monitor** du modèle et de votre GPU, CPU et RAM, une page **Models** et les **Settings**. Le chat peut
|
||||
travailler dans un dossier de votre PC. En mode **Manual**, il demande avant de modifier un fichier ou d'exécuter une
|
||||
commande. Il peut chercher sur le web (DuckDuckGo ou Bing, sans compte). Les mots recherchés sont la seule chose qui quitte
|
||||
votre PC.
|
||||
|
||||
<p align="center"><img src="docs/media/app-chat.png" width="900" alt="L'application Strata : un chat où l'agent a construit un petit jardin de pagode en voxels, avec la page ouverte dans l'aperçu à droite"><br>
|
||||
<sub>L'agent a écrit un jardin de pagode en voxels dans un seul fichier HTML, et la page s'ouvre dans l'aperçu à côté du chat</sub></p>
|
||||
|
||||
### Dans le Terminal
|
||||
|
||||
<p align="center"><img src="docs/media/terminal-monitor.png" width="900" alt="La vue d'ensemble du moniteur du Terminal Strata pendant que le modèle écrit une réponse"><br>
|
||||
<sub><code>strata monitor</code> pendant que le modèle répond : vitesse, GPU, mémoire et les dernières requêtes</sub></p>
|
||||
|
||||
```sh
|
||||
strata launch # guided: model, context, pictures, access, then serve
|
||||
strata launch --model q2_0 --yes --no-tui # no questions: serve and print the address
|
||||
strata launch --status # what is being served now
|
||||
strata launch --stop # stop it
|
||||
strata monitor # watch speed, GPU, memory and requests
|
||||
strata update # get the newest engine
|
||||
strata doctor # check the installed files
|
||||
```
|
||||
|
||||
Pour joindre Strata depuis votre téléphone ou un autre PC, utilisez `strata launch --access lan`. La commande crée une clé
|
||||
d'API pour vous. `strata --help` liste tout, et [docs/TERMINAL.md](docs/TERMINAL.md) l'explique.
|
||||
|
||||
### Avec des agents d'IA et d'autres applications
|
||||
|
||||
Un agent installe Strata avec la commande en une ligne ci-dessus et lance `strata launch --model NAME --yes --no-tui`. Les
|
||||
étapes, les options et les codes de sortie sont dans [docs/AI_SETUP.md](docs/AI_SETUP.md). Les règles du dépôt pour les agents : [AGENTS.md](AGENTS.md).
|
||||
|
||||
| Votre application parle | URL de base |
|
||||
| --- | --- |
|
||||
| API OpenAI (la plupart des applications et des agents) | `http://127.0.0.1:8080/v1` (n'importe quelle clé d'API, n'importe quel nom de modèle) |
|
||||
| API Anthropic (Claude Code) | `http://127.0.0.1:8080`, sans `/v1`. Claude Code : `ANTHROPIC_BASE_URL=http://127.0.0.1:8080` |
|
||||
| API OpenAI Responses (Codex CLI) | `http://127.0.0.1:8080/v1/responses` ([configuration](docs/DETAILS.md#the-responses-api-and-codex-cli)) |
|
||||
|
||||
`strata connect` affiche les réglages exacts pour OpenCode, Codex CLI, Claude Code, oh-my-pi, pi, Hermes Agent et OpenClaw,
|
||||
et `strata connect opencode --write` les écrit pour vous (avec une sauvegarde).
|
||||
|
||||
### Bon à savoir
|
||||
|
||||
- **Les images** sont activées pour tous les modèles qui savent les lire. Joignez ou collez une image dans le chat, ou envoyez-la depuis votre application. Détails : [docs/IMAGES.md](docs/IMAGES.md).
|
||||
- **La réflexion** peut être désactivée, faible, moyenne ou élevée, dans le menu du chat ou dans le réglage « reasoning effort » de votre application. Désactivée est le plus rapide.
|
||||
- **Le premier message** d'une discussion est lu en entier, environ 1 minute pour 30 000 tokens. Les messages suivants démarrent en quelques secondes.
|
||||
- **Une requête à la fois** par défaut. Pour en traiter plusieurs, utilisez `--parallel N` ([BATCHING.md](docs/BATCHING.md)). Sur une carte de 12 Go, chaque réponse devient alors plus lente.
|
||||
|
||||
## Compiler depuis les sources
|
||||
|
||||
<details>
|
||||
<summary>Windows, Linux et Mac depuis le dépôt</summary>
|
||||
|
||||
[Téléchargez Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) et décompressez-le, ou faites un `git clone`.
|
||||
**Windows :** double-cliquez sur `START-HERE.bat`. **Linux :** lancez `./setup.sh` dans le dossier Strata.
|
||||
|
||||
**Mac (Apple Silicon, macOS 15 ou plus récent) :** installez une fois les Command Line Tools d'Apple avec `xcode-select --install`
|
||||
(Xcode lui-même n'est pas nécessaire). Ouvrez le Terminal dans le dossier Strata et lancez `./setup.sh`. Le script compile
|
||||
le moteur Mac sur votre Mac (environ 7 minutes), pose les mêmes questions, télécharge le modèle et le démarre. La fois
|
||||
suivante, `./setup.sh` le démarre tout de suite. Quelle taille convient à votre Mac : [docs/MAC.md](docs/MAC.md).
|
||||
|
||||
L'installateur détecte votre carte et prépare le bon moteur. Il demande quel modèle et quelle taille, combien de contexte
|
||||
(la quantité de texte que le modèle garde en tête) et s'il doit lire les images. Appuyez sur Entrée pour la réponse
|
||||
recommandée. Puis il télécharge le modèle (environ 70 Go) et le démarre. Si le téléchargement s'arrête, relancez le script
|
||||
et il reprend. Votre navigateur ouvre la page de Strata à l'adresse `http://127.0.0.1:8080`.
|
||||
|
||||
> Pendant le démarrage du modèle, votre PC peut être lent ou ne plus répondre pendant 1 à 3 minutes (le plus longtemps la
|
||||
> première fois). Strata charge 35 à 55 Go dans votre RAM et en verrouille une partie pour la carte graphique. Patientez et
|
||||
> ne fermez pas la fenêtre.
|
||||
|
||||
La fois suivante, relancez `START-HERE.bat` (ou `./setup.sh`). Il démarre tout de suite et ne télécharge rien deux fois.
|
||||
Fermez sa fenêtre pour arrêter le modèle. `UPDATE.bat` (`./update.sh`) met Strata à jour sans le démarrer. Mise à jour,
|
||||
Docker, plusieurs cartes, où vont les fichiers et toutes les options : [docs/INSTALL.md](docs/INSTALL.md).
|
||||
|
||||
Pour mesurer votre PC, lancez `BENCH.bat` (`./bench.sh`). Cela prend 5 à 10 minutes et produit un rapport que vous pouvez
|
||||
partager ([comment](docs/COMMUNITY_BENCHMARKS.md#run-the-benchmark)).
|
||||
|
||||
**Facultatif : plus de vitesse sur une carte quand tout le modèle tient en RAM.** Deux réglages à activer soi-même, par
|
||||
RubenTorney (#1904, #1905). Ils aident quand vous utilisez une seule carte graphique et que votre RAM contient tous les
|
||||
experts, par exemple Q2_0 ou IQ3_XXS sur un PC de 64 Go. Ouvrez `strata-<model>.json` dans le dossier de données de Strata ([où](docs/INSTALL.md#where-things-are-stored)) et ajoutez :
|
||||
|
||||
- `"--adapt-async", "1"` à la liste `"args"`. Le cache des experts se met alors à jour sur un thread auxiliaire au lieu de
|
||||
mettre la réponse en pause. Mesuré sur une RTX 5070 (Windows, Q2_0) : +4 à 5 % en décodage, +7 à 10 % sur le code et les
|
||||
longs documents. Les réponses peuvent différer légèrement de celles du réglage par défaut, et restent identiques d'une exécution à l'autre.
|
||||
- `"STRATA_PIN_BY_NEED": "1"` dans le bloc `"env"` (Windows). Windows ne peut verrouiller qu'environ la moitié de la RAM
|
||||
pour la carte graphique, et ce réglage verrouille d'abord les experts dont la carte a le plus besoin. Mesuré sur la RTX 5070 : +4,4 % en lecture du prompt.
|
||||
|
||||
```json
|
||||
"args": [ ..., "--adapt-async", "1" ],
|
||||
"env": { "STRATA_PIN_BY_NEED": "1" }
|
||||
```
|
||||
|
||||
Redémarrez Strata après la modification. Pour annuler, supprimez les deux entrées.
|
||||
Suivez les invites pour télécharger et démarrer un modèle. [Guide du Terminal](docs/TERMINAL_LAUNCH.md).
|
||||
|
||||
</details>
|
||||
|
||||
## Comment ça marche ?
|
||||
<p align="center">
|
||||
<img src="docs/media/terminal-monitor.png" width="900" alt="Application Terminal de Strata affichant l'activité du modèle, la vitesse, la mémoire et les requêtes récentes">
|
||||
<br><sub>Suivez votre modèle et votre matériel pendant qu'il tourne.</sub>
|
||||
</p>
|
||||
|
||||
Les modèles de ce genre tournent d'habitude sur des serveurs qui disposent de centaines de gigaoctets de mémoire graphique.
|
||||
Votre carte graphique en a 12 à 24. Strata fait tenir le modèle en répartissant le travail sur tout votre PC. Imaginez une
|
||||
bibliothèque.
|
||||
<a id="build-from-source"></a>
|
||||
|
||||
- Le modèle est une équipe de 24 576 petits spécialistes (les « experts »), comme 24 576 ouvrages de référence. Chaque mot n'en demande que 10.
|
||||
- **Votre carte graphique est le bureau.** Il est petit et rapide. Les quelque 4 000 livres les plus utilisés y sont ouverts.
|
||||
- **Votre RAM, ce sont les étagères de la salle de lecture.** Les 24 576 livres s'y trouvent. Quand un mot demande un livre
|
||||
qui n'est pas sur le bureau, votre processeur va le chercher aussitôt sur l'étagère, pendant que la carte continue de travailler.
|
||||
- **Votre SSD, ce sont les archives.** Il contient une table de consultation de 29 Go. Le modèle n'en lit que quelques lignes par
|
||||
mot, elle n'a donc pas besoin de tenir en RAM.
|
||||
[Compiler depuis les sources](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [Guide d'installation complet](docs/INSTALL.md)
|
||||
|
||||
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Les 24 576 experts du modèle : les plus sollicités sur la carte graphique, tous dans la RAM, une table de consultation sur le SSD"></p>
|
||||
<a id="what-you-need"></a>
|
||||
|
||||
Un petit assistant devine aussi les mots suivants. Le grand modèle les vérifie tous d'un coup et garde les bons. Vous
|
||||
obtenez la même réponse, 1,6 à 1,8 fois plus tôt. Les textes longs sont lus par gros morceaux, jusqu'à 8 192 tokens à la
|
||||
fois, à plus de 1 000 tokens par seconde.
|
||||
## Matériel
|
||||
|
||||
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Un petit assistant devine les mots suivants ; le grand modèle les vérifie tous d'un coup et garde les bons"></p>
|
||||
| Matériel pris en charge | Guide |
|
||||
| --- | --- |
|
||||
| **NVIDIA GeForce RTX 20 / 30 / 40 / 50** | [Installation NVIDIA](docs/INSTALL.md#what-you-need) |
|
||||
| **AMD Radeon RX 7700 / 7800 / 7900, RX 9060 / 9070, AI PRO R9700** | [Installation AMD](docs/AMD_HIP.md) |
|
||||
| **AMD Radeon RX 6800 / 6900 · expérimental** | [Installation RX 6000](docs/AMD_HIP.md#rdna2-gfx1030) |
|
||||
| **Intel Arc · Linux** | [Installation Intel](docs/INTEL_ARC.md) |
|
||||
| **Apple Silicon · M1 ou plus récent** | [Installation sur Mac](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max · Strix Halo** | [Installation Strix Halo](docs/STRIX_HALO.md) |
|
||||
|
||||
L'explication plus longue : [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Chaque élément avec ses chiffres :
|
||||
[les détails](docs/DETAILS.md#how-it-works) et l'[article](docs/paper/Strata-Paper.pdf).
|
||||
<a id="using-it"></a>
|
||||
|
||||
## Un problème ?
|
||||
## Outils
|
||||
|
||||
- **Mon PC s'est figé au premier démarrage de Strata.** Il charge le modèle en RAM. Patientez et ne fermez pas la fenêtre.
|
||||
Toujours figé après 10 minutes ? Redémarrez le PC, fermez les autres programmes et réessayez, ou choisissez une taille plus petite.
|
||||
- **Ça s'est arrêté pendant le téléchargement ou l'installation.** Relancez l'application ou `strata launch`. Cela reprend là
|
||||
où ça s'était arrêté.
|
||||
- **C'est très lent, ou il affiche « the engine stopped unexpectedly ».** Votre PC n'a pas assez de RAM libre. Fermez
|
||||
d'autres programmes (les navigateurs en consomment beaucoup) ou choisissez une taille plus petite, comme Q2_0 ou IQ2_XS.
|
||||
- **Il indique qu'il n'y a pas de moteur pour ce PC.** Dans l'application, ouvrez **Settings, Engine** et appuyez sur **Update engine**
|
||||
(mettre à jour le moteur). Dans le Terminal, lancez `strata update`.
|
||||
- **Il indique que le port 8080 est déjà utilisé.** Strata tourne déjà. Cherchez sa fenêtre.
|
||||
**Claude Code · Codex CLI · OpenCode · oh-my-pi · pi.dev · Hermes Agent · OpenClaw**, et d'autres via les API compatibles OpenAI et Anthropic.
|
||||
|
||||
Autres solutions : [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Toujours bloqué ? Ouvrez une
|
||||
[issue](https://github.com/Niko1221/Strata/issues) et joignez les journaux : dans l'application via **Settings, Engine,
|
||||
Diagnostics, Save logs**, dans le Terminal la sortie de `strata doctor`. Pour un problème de sécurité, signalez-le en
|
||||
privé : [SECURITY.md](SECURITY.md).
|
||||
Lancez **`strata connect`** pour voir la configuration de votre outil. [Guide de connexion](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
|
||||
|
||||
## Remerciements et licence
|
||||
<a id="guides"></a>
|
||||
<a id="how-fast-is-it"></a>
|
||||
<a id="how-does-it-work"></a>
|
||||
<a id="something-went-wrong"></a>
|
||||
|
||||
Le modèle est [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), de l'équipe Qwen. Il a été compressé par
|
||||
[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5), agentionai (Gyro)
|
||||
et Unsloth. ISTA-DASLab a aussi produit les versions de DeepSeek V4.1-Flash. Strata utilise des parties de
|
||||
[llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Tous les remerciements : [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits).
|
||||
Strata est open source sous [licence MIT](LICENSE). Quelques parties et tous les modèles ont leurs propres licences
|
||||
([lesquelles](docs/HOW_IT_WORKS.md#license)).
|
||||
## Aide
|
||||
|
||||
## Contribuer
|
||||
|
||||
Vous avez trouvé un bug, ou vous voulez proposer une modification ou un benchmark ? Lisez d'abord [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
Il indique ce qu'il faut inclure pour que d'autres puissent vérifier votre travail.
|
||||
[Tous les guides](docs) · [Dépannage](docs/TROUBLESHOOTING.md) · [Signaler un bogue](https://github.com/Niko1221/Strata/issues/new/choose) ·
|
||||
[Benchmarks](docs/COMMUNITY_BENCHMARKS.md) · [Comment ça marche](docs/HOW_IT_WORKS.md) · [Contribuer](CONTRIBUTING.md)
|
||||
|
||||
## Soutenir Strata
|
||||
|
||||
Strata est gratuit et sous licence MIT. Je le construis et je le teste sur le matériel que je peux obtenir. S'il tourne bien sur votre PC, un café m'aide
|
||||
à continuer de tester de nouvelles cartes et à corriger ce que vous signalez. Aucune obligation. L'utiliser et signaler ce que vous constatez aide tout autant.
|
||||
**Aidez à garder l'IA locale ouverte à tous.** Un café aide à financer le développement et les tests sur davantage de matériel.
|
||||
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee : soutenir Strata" height="48"></a>
|
||||
|
||||
Vous souhaitez sponsoriser Strata ? Contactez-moi via [GitHub](https://github.com/Niko1221).
|
||||
[Licence MIT](LICENSE) · [Crédits et licences des modèles](docs/HOW_IT_WORKS.md#credits) · [Confidentialité](docs/SAFETY.md)
|
||||
|
||||
<a id="option-c-let-your-ai-set-it-up"></a>
|
||||
|
||||
<sub>Pour les outils d'IA et les lecteurs automatiques : <a href="AGENTS.md">référence technique</a> · <a href="docs/AI_SETUP.md">instructions d'installation</a>.</sub>
|
||||
|
||||
**Parrainages, partenariats matériel ou collaborations :** [[email protected]](mailto:[email protected]).
|
||||
+122
-284
@@ -1,350 +1,188 @@
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="apps/host/public/brand/strata-lockup-dark.svg">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="380">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="340">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <b>日本語</b> · <a href="README.de.md">Deutsch</a> · <a href="README.fr.md">Français</a> · <a href="README.es.md">Español</a> · <a href="README.pt-BR.md">Português</a></p>
|
||||
|
||||
<h3 align="center">最先端のモデルを、手元のゲーミングPCで動かす</h3>
|
||||
|
||||
<p align="center">NVIDIA、AMD、Intel に対応 · Windows、Linux、Mac · 無料のオープンソース</p>
|
||||
<h3 align="center">あなた自身のパソコンで、高性能な AI を。</h3>
|
||||
<p align="center">チャット、コーディング、画像の処理を、すべてローカルで。</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?include_prereleases&label=release" alt="最新リリース"></a>
|
||||
<a href="#モデル">モデル</a> · <a href="#インストール"><b>インストール</b></a> ·
|
||||
<a href="#ハードウェア">ハードウェア</a> · <a href="#ツール">ツール</a> ·
|
||||
<a href="#ヘルプ">ヘルプ</a> · <a href="#strata-を支援する">支援</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Strata をダウンロード"></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="MIT ライセンス"></a>
|
||||
<a href="https://github.com/Niko1221/Strata/stargazers"><img src="https://img.shields.io/github/stars/Niko1221/Strata?style=flat" alt="GitHub のスター"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black" alt="コーヒーをおごる"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Buy Me a Coffee で Strata を支援する"></a>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Strata のモデルが書いたボクセルの塔の庭園。ブラウザで動いています"></a><br>
|
||||
<sub>ボクセルの塔の庭園。1回のプロンプトで生成し、RTX 5070 上の Strata で動かしています（IQ3_S、128K コンテキスト） ·
|
||||
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">動画全体（49秒）</a></sub></p>
|
||||
<p align="center"><sub><a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <b>日本語</b> · <a href="README.de.md">Deutsch</a> · <a href="README.fr.md">Français</a> · <a href="README.es.md">Español</a> · <a href="README.pt-BR.md">Português</a></sub></p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#インストール"><b>インストール</b></a> ·
|
||||
<a href="#モデル">モデル</a> ·
|
||||
<a href="#どのくらい速い">速度</a> ·
|
||||
<a href="#使い方">使い方</a> ·
|
||||
<a href="#仕組み">仕組み</a> ·
|
||||
<a href="#うまくいかないとき">ヘルプ</a> ·
|
||||
<a href="#strata-を応援する">応援する</a>
|
||||
</p>
|
||||
**無料でオープンソース。自分の環境で動かせます。** デスクトップで、またはお好みのコーディングツールと一緒に Strata を使えます。アカウントは不要です。
|
||||
|
||||
Strata は **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** を動かします。1250億パラメータのモデルで、
|
||||
ふつうはサーバーが必要ですが、Strata なら一般的なPCで使えます。0.2.4 からは、ターミナル版で **DeepSeek V4.1-Flash** も動かせます。
|
||||
チャット、コード作成、画像の読み取りができ、お使いのアプリやコーディングエージェントとも連携します。チャットとファイルはPCの外に出ません。
|
||||
## モデル
|
||||
|
||||
- **ゲーミングPCで動きます。** RAMとグラフィックスメモリの合計が32 GBあれば始められます。
|
||||
- **お使いのツールと連携します。** Claude Code、Codex CLI、OpenCode などのエージェントが、OpenAI と Anthropic の API で接続できます。
|
||||
- **画像を読めます。** 初期設定でオンです。
|
||||
- **無料でプライバシーも守ります。** MIT ライセンス。アカウントもクラウドもテレメトリも不要です（[何をいつダウンロードするか](docs/SAFETY.md)）。
|
||||
まず **ISTA-DASLab** をおすすめします。次に **Unsloth** などの提供元です。メモリに収まる互換モデルがある場合に選べます。
|
||||
|
||||
## 必要な環境
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.8-Flash-Next</b></summary>
|
||||
|
||||
**RAMとグラフィックスメモリ（VRAM）の合計が、少なくとも32 GB必要です。** MacやRyzen AI Maxでは、ユニファイドメモリがその量にあたります。
|
||||
RAM 32 GBのPCに12〜16 GBのカードがあれば小さいサイズが動き、64 GBならどのサイズも動きます。
|
||||
|
||||
| 動く環境 | ガイド |
|
||||
| 提供元 | 選択肢 |
|
||||
| --- | --- |
|
||||
| **NVIDIA** GeForce RTX 20、30、40、50 シリーズ、12 GB以上 | [インストール](#インストール) |
|
||||
| **AMD** Radeon RX 6800 以降、Radeon AI PRO R9700 | [AMD のカード](docs/AMD_HIP.md) |
|
||||
| **Intel** Arc | [Intel Arc](docs/INTEL_ARC.md) |
|
||||
| **Apple Silicon の Mac**、M1 以降、macOS 15 以降 | [Mac](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max（Strix Halo）** などのユニファイドメモリ搭載機 | [Strix Halo](docs/STRIX_HALO.md) |
|
||||
| 古いカードやプロセッサ | [古いGPU](docs/OLDER_GPUS.md) |
|
||||
| **ISTA-DASLab · おすすめ** | Q2\_0, IQ2\_XS, IQ3\_XXS, IQ3\_S。[Coder: IQ1\_M](docs/MODELS.md#coder)（コーディング向け）。 |
|
||||
| **Unsloth** | UD-IQ4\_XS, UD-Q4\_K\_XL, UD-Q5\_K\_XL, UD-Q6\_K\_XL, Q8\_0 |
|
||||
| **Gyro · agentionai** | [Gyro-S, Gyro-M](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)。メモリ使用量は少ないですが、動作は遅くなります。 |
|
||||
| **その他の選択肢** | [Swift 1.5](docs/MODELS.md#swift-15), [NVFP4](docs/NVFP4.md), [OrcaRouter](docs/ORCA.md) |
|
||||
|
||||
そのほかに、約80 GBの空きディスク容量（SSD）、Windows 10 / 11、Linux、または macOS、最新のグラフィックスドライバーが必要です。
|
||||
同じメーカーのカードを2〜3枚使って、1つのモデルを動かせます（[マルチGPU](docs/MULTI_GPU.md)）。
|
||||
[サイズ、必要なメモリ、互換性を比べる](docs/MODELS.md)。
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.6-35B-A3B</b></summary>
|
||||
|
||||
**Unsloth:** UD-Q4_K_XL, UD-Q6_K_XL, Q8_0。現在、このモデルで文書化されているパックです。
|
||||
|
||||
[モデルガイドと互換性](docs/QWEN36.md)。
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/deepseek.svg" width="22" alt="" align="middle"> <b>DeepSeekV4.1-Flash-1017</b></summary>
|
||||
|
||||
**デスクトップアプリ**と**ターミナル**で使えます。メモリに収まらない分は SSD から読み込むため、空きディスク容量がたくさん必要です。
|
||||
|
||||
| セット（IST-DAS） | ダウンロード | 必要な空き容量 |
|
||||
| --- | ---: | ---: |
|
||||
| 1.75 bpw | 239 GB | 約245 GB |
|
||||
| 2.00 bpw | 257 GB | 約265 GB |
|
||||
| 2.25 bpw | 274 GB | 約280 GB |
|
||||
|
||||
NVIDIA RTX 30・40・50 シリーズ（Windows と Linux）と、AMD Radeon AI PRO R9700（Linux）で動きます。セットアップがファイルと DeepSeek エンジンをダウンロードし、モデルを起動します。
|
||||
|
||||
```sh
|
||||
strata launch --family deepseek41 --model ist-das-2.00bpw --plan
|
||||
```
|
||||
|
||||
Strata 0.2.5 で 2.00 bpw のセットを使って測定しました。「出力」は回答の速さ、「プロンプト」は Strata が入力を読み込む速さです。
|
||||
|
||||
| ハードウェア | 出力 | プロンプト |
|
||||
| --- | ---: | ---: |
|
||||
| RTX 5070 12 GB, 64 GB RAM | 4〜6 トークン/秒 | 約 110 トークン/秒 (4K) |
|
||||
| RTX 3060 12 GB, 46 GB RAM | 9〜11 トークン/秒 | 約 155 トークン/秒 (4K) |
|
||||
| 1x Radeon AI PRO R9700, 251 GB RAM | 21〜25 トークン/秒 | 320〜360 トークン/秒 (8K) |
|
||||
| 4x Radeon AI PRO R9700 | 74〜77 トークン/秒 | 400〜450 トークン/秒 (8K) |
|
||||
|
||||
[メモリ、ディスク、詳細](docs/DEEPSEEK41.md) · [リリースノート](https://github.com/Niko1221/Strata/releases)
|
||||
|
||||
</details>
|
||||
|
||||
<a id="the-other-sizes"></a>
|
||||
|
||||
どれが動くかわからない場合は、[お使いのパソコンに合うモデルを選ぶ](docs/MODELS.md#pick-by-memory)。
|
||||
|
||||
## インストール
|
||||
|
||||
### 1. Strata アプリ（いちばん簡単）
|
||||
<a id="option-a-the-strata-app"></a>
|
||||
|
||||
お使いのPCに合うファイルをダウンロードして開き、すすめられたモデルのインストールを押します。
|
||||
### デスクトップアプリ
|
||||
|
||||
**Windows 10 / 11**
|
||||
**[Windows、Linux、Mac 用の Strata をダウンロード →](https://github.com/Niko1221/Strata/releases/latest)**
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-windows-x64-setup.exe)
|
||||
アプリをインストールし、モデルを選んで、チャットを始めましょう。インストーラーはシステムごとに1つです。初回起動時に、アプリがお使いのグラフィックスカード用のエンジンをダウンロードします。[インストーラーのヘルプ](docs/INSTALL.md#the-app)。
|
||||
|
||||
**Linux**
|
||||
<sub>ファイル: `Strata-0.2.6-windows-x64-setup.exe` · `Strata-0.2.6-linux-x86_64.AppImage` · `Strata-0.2.6-macos-arm64.dmg`</sub>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-linux-x86_64.AppImage)
|
||||
<p align="center">
|
||||
<img src="docs/media/app-chat.png" width="900" alt="チャットとライブのコードプレビューを並べて表示した Strata デスクトップアプリ">
|
||||
<br><sub>チャット、ファイル、ライブプレビューを1つのウィンドウで。</sub>
|
||||
</p>
|
||||
|
||||
**Mac**（Apple Silicon、macOS 15 以降）
|
||||
<a id="option-b-the-terminal-one-line"></a>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-macos-arm64.dmg)
|
||||
### ターミナルアプリ
|
||||
|
||||
お使いのグラフィックスカードに合うエンジンは、アプリが自動で選びます。すべてのファイルは
|
||||
[リリースページ](https://github.com/Niko1221/Strata/releases/latest)にあります。
|
||||
ターミナルのほうが好みですか？ Strata をインストールして、**`strata launch`** を実行します。
|
||||
|
||||
| 初回の起動 | すること |
|
||||
| --- | --- |
|
||||
| **Windows** | `.exe` を実行します。管理者権限は不要です。「Windows によってPCが保護されました」と表示されたら、**詳細情報**を選び、**実行**を選んでください。 |
|
||||
| **Linux** | ファイルを実行可能にして（`chmod +x`）起動します。FUSE 2 が必要です。`sudo apt install libfuse2t64`（Ubuntu 24.04 以降）または `libfuse2`（22.04）を実行してください。 |
|
||||
| **Mac** | `.dmg` を開き、Strata を「アプリケーション」にドラッグします。 |
|
||||
<details open>
|
||||
<summary><b>インストールコマンド</b></summary>
|
||||
|
||||
モデルのダウンロードは大きく（40〜70 GB）、止まっても続きから再開します。
|
||||
|
||||
### 2. ターミナル（1行）
|
||||
|
||||
`strata` コマンドです。スクリプトやサーバー、AI エージェント向けです。DeepSeek を動かすにもこれを使います。
|
||||
|
||||
Linux と Mac:
|
||||
**Linux / Mac**
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
```
|
||||
|
||||
Windows（PowerShell）:
|
||||
**Windows PowerShell**
|
||||
|
||||
```powershell
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
```
|
||||
|
||||
そのあと起動します。質問は1つずつ表示されます（モデル、コンテキスト、画像、接続を許可する相手）。必要なものをダウンロードして、
|
||||
サーバーを開始します。
|
||||
Windows では、インストール後に新しいターミナルを開きます。そのあと:
|
||||
|
||||
```sh
|
||||
strata launch
|
||||
```
|
||||
|
||||
管理者権限は不要です。インストールでは、ダウンロードしたものすべての SHA-256 を検証します。Windows では、インストール後に新しいターミナルを開いてください。
|
||||
すべての画面とフラグは [docs/TERMINAL_LAUNCH.md](docs/TERMINAL_LAUNCH.md) にあります。
|
||||
|
||||
### 3. AI にセットアップを任せる
|
||||
|
||||
Claude Code、Cursor、Codex、GitHub Copilot などのコーディングアシスタントに、次の文を貼り付けます。
|
||||
|
||||
```text
|
||||
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
|
||||
```
|
||||
|
||||
AI はグラフィックスカード、RAM、ディスクを調べ、合うモデルを選んで、インストールと起動をし、アプリの接続方法を教えてくれます。
|
||||
AI ツールは [MCP サーバー](docs/MCP_SERVER.md)を通じて、Strata のインストール、起動、停止もできます。
|
||||
|
||||
自分でビルドしたい方は[ソースからビルド](#ソースからビルド)を見てください。
|
||||
|
||||
## モデル
|
||||
|
||||
アプリと `strata launch` は、メモリ量に合うサイズをすすめます。小さいサイズは速く、大きいサイズは少し賢くなります。
|
||||
|
||||
### メモリが少ないなら Gyro
|
||||
|
||||
[Gyro](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)（agentionai 製）は、Qwen3.8-Flash-Next をいちばん小さく動かせる方法です。
|
||||
NVIDIA、AMD、Intel Arc、Apple Silicon で動きます。
|
||||
|
||||
| サイズ | 必要なメモリ（RAM + VRAM） | ダウンロード | 向いているPC |
|
||||
| --- | ---: | ---: | --- |
|
||||
| **Gyro-S** | 27.6 GB | 58.5 GB | ほかのサイズが入らないPC向け。32 GB の Mac mini で毎秒 9 トークン。 |
|
||||
| **Gyro-M** | 35 GB | 92.0 GB | 品質は IQ3_XXS 程度で、GPUメモリの使用量は20%少なくなります。 |
|
||||
|
||||
`strata launch --model gyro-s` で起動します。詳細は [docs/GYRO.md](docs/GYRO.md) にあります。
|
||||
|
||||
### Qwen3.8-Flash-Next のサイズ
|
||||
|
||||
| メモリ（RAM + VRAM） | 選ぶモデル | 理由 |
|
||||
| --- | --- | --- |
|
||||
| **32 GB** | **Gyro-S**、またはコード向けの **[Coder](docs/MODELS.md#coder)** | Coder は、コードで使うエキスパートを残します |
|
||||
| **40 GB** | **[Swift 1.5](docs/MODELS.md#swift-15)** IQ2_XS（39.2 GB）、または **Gyro-M** | Swift は答える前に考える時間が短くなります |
|
||||
| **48 GB** | **IQ3_XXS**（47.0 GB） | 遅くなりますが、品質は上がります |
|
||||
| **64 GB** | **IQ3_S**（54.8 GB） | 公開されているテストでは完全版と同等です |
|
||||
| **96 GB 以上** | Unsloth の **UD-IQ4_XS** から **Q8_0** | 完全版にいちばん近いです |
|
||||
|
||||
表の行より RAM が少なくても、遅くなるだけで動くことがあります。Strata は自動で[低RAMモード](docs/MODELS.md#a-big-graphics-card-and-little-ram)に切り替わります。
|
||||
そのほかに、NVIDIA カード向けの [NVFP4](docs/NVFP4.md)、[Qwen3.6-35B-A3B](docs/QWEN36.md)（より小さく、複数のリクエストを同時に処理）、
|
||||
[OrcaRouter の Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs) があります。すべてのサイズと、どの構成で何が動くか:
|
||||
[docs/MODELS.md](docs/MODELS.md)。
|
||||
|
||||
### DeepSeek V4.1-Flash（ターミナル版、0.2.4 から）
|
||||
|
||||
ずっと大きなモデルで、12 GB のグラフィックスカード 1 枚と 46 から 64 GB の RAM で動きます。Strata はよく使う部分をメモリに置き、
|
||||
残りは SSD から読みます。セットは [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF) が作りました。
|
||||
NVIDIA RTX 30、40、50 シリーズのカードと Radeon AI PRO R9700 で動きます。`strata launch` で選んでください。デスクトップアプリはまだ対応していません。
|
||||
|
||||
| セット | ダウンロード | 品質 | コードの出力速度 |
|
||||
| --- | ---: | --- | --- |
|
||||
| **IQ1_M**（1.75 ビット） | 239 GB | いちばん小さく、ふつうの文章は弱めです | PC ではまだ測っていません |
|
||||
| **IQ2_XXS**（2.00 ビット） | 257 GB | バランスが良いです | 毎秒 10 トークン（RTX 3060）、5（RTX 5070）、48（R9700 4 枚） |
|
||||
| **IQ2_XS**（2.25 ビット） | 274 GB | 元のモデルにいちばん近いです | 毎秒 15 トークン（R9700 1 枚）、80（R9700 4 枚） |
|
||||
| **IQ4_NL**（Engram テーブル） | 上のどのダウンロードにも含まれます | 3 つのセットが共有する参照テーブルです | |
|
||||
|
||||
`strata launch --model ist-das-2.00bpw`（または `ist-das-1.75bpw`、`ist-das-2.25bpw`）で起動します。品質の数値、PC ごとの速度、手動インストールは
|
||||
[docs/DEEPSEEK41.md](docs/DEEPSEEK41.md) にあります。
|
||||
|
||||
## どのくらい速い？
|
||||
|
||||
12 GBのグラフィックスカード（RTX 5070、RAM 64 GB）での数値です。ほかのカードの結果は
|
||||
[各サイズの速度](docs/MODELS.md#how-fast-is-each-size)と[コミュニティの結果](docs/COMMUNITY_BENCHMARKS.md)を見てください。
|
||||
|
||||
| モデル | 回答が出る速さ | プロンプトを読む速さ |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████████` 94 tokens/s | `████████████` 2,650 tokens/s |
|
||||
| **IQ2_XS** | `██████████░░` 79 tokens/s | `█████████░░░` 2,090 tokens/s |
|
||||
| **IQ3_XXS** | `████████░░░░` 62 tokens/s | `████████░░░░` 1,750 tokens/s |
|
||||
| **IQ3_S** | `███████░░░░░` 53 tokens/s | `███████░░░░░` 1,620 tokens/s |
|
||||
| **Coder** | `███████░░░░░` 55 tokens/s | `██████████░░` 2,180 tokens/s |
|
||||
|
||||
16 GBの AMD カード（RX 9070 XT、RAM 47 GB）の場合:
|
||||
|
||||
| モデル | 回答が出る速さ | プロンプトを読む速さ |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████░░░░` 60 tokens/s | `█████░░░░░░░` 1,160 tokens/s |
|
||||
| **IQ2_XS** | `███████░░░░░` 52 tokens/s | `█████░░░░░░░` 1,110 tokens/s |
|
||||
| **Coder** | `██████░░░░░░` 44 tokens/s | `██████░░░░░░` 1,420 tokens/s |
|
||||
|
||||
1トークンは英単語の約¾にあたるので、毎秒60トークンは読む速さより速くなります。「プロンプトを読む速さ」は、長い文書やコード、チャット履歴を Strata が取り込む速さです。
|
||||
|
||||
## 使い方
|
||||
|
||||
### アプリで
|
||||
|
||||
チャット、モデルとGPU・CPU・RAMの状態を見る**モニター**、**モデル**ページ、**設定**があります。チャットはPC上のフォルダで作業できます。
|
||||
**手動**モードでは、ファイルを変更したりコマンドを実行したりする前に確認します。ウェブ検索もできます（DuckDuckGo または Bing、アカウント不要）。
|
||||
PCの外に出るのは検索語だけです。
|
||||
|
||||
<p align="center"><img src="docs/media/app-chat.png" width="900" alt="Strata アプリ。エージェントが小さなボクセルの塔の庭を作ったチャットで、右側のプレビューにそのページが開いています"><br>
|
||||
<sub>エージェントがボクセルの塔の庭を 1 つの HTML ファイルとして書き、そのページがチャットの横のプレビューで開きます</sub></p>
|
||||
|
||||
### ターミナルで
|
||||
|
||||
<p align="center"><img src="docs/media/terminal-monitor.png" width="900" alt="モデルが返信を書いている間の Strata ターミナルのモニター概要"><br>
|
||||
<sub>モデルが返信を書いている間の <code>strata monitor</code>。速度、GPU、メモリ、最近のリクエストが表示されます</sub></p>
|
||||
|
||||
```sh
|
||||
strata launch # guided: model, context, pictures, access, then serve
|
||||
strata launch --model q2_0 --yes --no-tui # no questions: serve and print the address
|
||||
strata launch --status # what is being served now
|
||||
strata launch --stop # stop it
|
||||
strata monitor # watch speed, GPU, memory and requests
|
||||
strata update # get the newest engine
|
||||
strata doctor # check the installed files
|
||||
```
|
||||
|
||||
スマートフォンや別のPCから Strata に接続するには、`strata launch --access lan` を使います。API キーが自動で作られます。
|
||||
`strata --help` ですべてのコマンドを確認できます。説明は [docs/TERMINAL.md](docs/TERMINAL.md) にあります。
|
||||
|
||||
### AI エージェントやほかのアプリと使う
|
||||
|
||||
エージェントは、上の1行コマンドで Strata をインストールし、`strata launch --model NAME --yes --no-tui` を実行します。手順、フラグ、終了コードは
|
||||
[docs/AI_SETUP.md](docs/AI_SETUP.md) にあります。エージェント向けのリポジトリのルールは [AGENTS.md](AGENTS.md) にあります。
|
||||
|
||||
| アプリが使うもの | ベース URL |
|
||||
| --- | --- |
|
||||
| OpenAI API（ほとんどのアプリとエージェント） | `http://127.0.0.1:8080/v1`（API キーもモデル名も何でも可） |
|
||||
| Anthropic API（Claude Code） | `http://127.0.0.1:8080`、`/v1` は付けません。Claude Code: `ANTHROPIC_BASE_URL=http://127.0.0.1:8080` |
|
||||
| OpenAI Responses API（Codex CLI） | `http://127.0.0.1:8080/v1/responses`（[設定方法](docs/DETAILS.md#the-responses-api-and-codex-cli)） |
|
||||
|
||||
`strata connect` は、OpenCode、Codex CLI、Claude Code、oh-my-pi、pi、Hermes Agent、OpenClaw の正確な設定を表示します。
|
||||
`strata connect opencode --write` なら、設定を書き込みます（バックアップ付き）。
|
||||
|
||||
### 知っておくと便利なこと
|
||||
|
||||
- **画像**は、画像を読めるモデルすべてでオンです。チャットに画像を添付または貼り付けるか、アプリから送ってください。詳細: [docs/IMAGES.md](docs/IMAGES.md)。
|
||||
- **思考のレベル**は off、low、medium、high から、チャットのメニューか、お使いのアプリの「reasoning effort」で選びます。off がいちばん速くなります。
|
||||
- **チャットの最初のメッセージ**は全文読みます。30,000トークンあたり約1分かかります。続きのメッセージは数秒で始まります。
|
||||
- **一度に1つのリクエスト**が初期設定です。複数にするには `--parallel N` を使います（[BATCHING.md](docs/BATCHING.md)）。12 GBのカードでは、1つずつの回答が遅くなります。
|
||||
|
||||
## ソースからビルド
|
||||
|
||||
<details>
|
||||
<summary>リポジトリから Windows、Linux、Mac 用にビルドする</summary>
|
||||
|
||||
[Strata をダウンロード](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip)して展開するか、`git clone` します。
|
||||
**Windows:** `START-HERE.bat` をダブルクリックします。**Linux:** Strata フォルダで `./setup.sh` を実行します。
|
||||
|
||||
**Mac（Apple Silicon、macOS 15 以降）:** 最初に一度、`xcode-select --install` で Apple のコマンドラインツールをインストールします
|
||||
（Xcode 本体は不要です）。Strata フォルダでターミナルを開き、`./setup.sh` を実行します。Mac 用エンジンをお使いの Mac 上でビルドし
|
||||
（約7分）、同じ質問をして、モデルをダウンロードし、起動します。2回目以降は `./setup.sh` ですぐに起動します。
|
||||
お使いの Mac に合うサイズは [docs/MAC.md](docs/MAC.md) を見てください。
|
||||
|
||||
インストーラーはカードを見つけて、合うエンジンを設定します。どのモデルとサイズにするか、コンテキストをどれだけにするか（モデルが一度に覚えておけるテキストの量）、
|
||||
画像を読ませるかを聞かれます。おすすめの答えでよければ Enter を押します。そのあとモデル（約70 GB）をダウンロードして起動します。
|
||||
ダウンロードが止まったら、もう一度実行すると続きから再開します。ブラウザで `http://127.0.0.1:8080` の Strata ページが開きます。
|
||||
|
||||
> モデルの起動中は、PCが1〜3分ほど（初回がいちばん長い）遅くなったり、応答しなくなったりすることがあります。Strata は35〜55 GBをRAMに読み込み、
|
||||
> その一部をグラフィックスカード用に固定するためです。ウィンドウを閉じずに待ってください。
|
||||
|
||||
次回からは `START-HERE.bat`（または `./setup.sh`）をもう一度実行するだけです。すぐに起動し、同じものを2回ダウンロードすることはありません。ウィンドウを閉じるとモデルが止まります。
|
||||
`UPDATE.bat`（`./update.sh`）は、起動せずに Strata を更新します。更新、Docker、複数のカード、ファイルの保存先、すべてのオプションは
|
||||
[docs/INSTALL.md](docs/INSTALL.md) にあります。
|
||||
|
||||
PCの性能を測るには、`BENCH.bat`（`./bench.sh`）を実行します。5〜10分かかり、共有できるレポートが作られます
|
||||
（[方法](docs/COMMUNITY_BENCHMARKS.md#run-the-benchmark)）。
|
||||
|
||||
**オプション: モデル全体がRAMに入るとき、1枚のカードをさらに速くする。** RubenTorney による、自分でオンにする2つの設定です（#1904、#1905）。
|
||||
グラフィックスカードが1枚で、RAM がすべてのエキスパートを保持できるときに効果があります。たとえば64 GBのPCでの Q2_0 や IQ3_XXS です。
|
||||
Strata のデータフォルダにある `strata-<model>.json`（[場所](docs/INSTALL.md#where-things-are-stored)）を開いて、次を追加します。
|
||||
|
||||
- `"args"` のリストに `"--adapt-async", "1"` を追加します。エキスパートキャッシュが、回答を止めずに補助スレッドで更新されます。
|
||||
RTX 5070（Windows、Q2_0）での測定: デコードが4〜5%、コードや長い文書では7〜10%速くなりました。回答は初期設定と少し違うことがありますが、実行ごとの結果は変わりません。
|
||||
- `"env"` ブロックに `"STRATA_PIN_BY_NEED": "1"` を追加します（Windows）。Windows ではグラフィックスカード用に固定できる RAM が半分ほどしかありません。
|
||||
この設定は、カードが最初に必要とするエキスパートを優先して固定します。RTX 5070 での測定: プロンプトの読み取りが4.4%速くなりました。
|
||||
|
||||
```json
|
||||
"args": [ ..., "--adapt-async", "1" ],
|
||||
"env": { "STRATA_PIN_BY_NEED": "1" }
|
||||
```
|
||||
|
||||
変更したら Strata を再起動してください。元に戻すには、2つの項目を削除します。
|
||||
画面の案内に従って、モデルをダウンロードして起動します。[ターミナルガイド](docs/TERMINAL_LAUNCH.md)。
|
||||
|
||||
</details>
|
||||
|
||||
## 仕組み
|
||||
<p align="center">
|
||||
<img src="docs/media/terminal-monitor.png" width="900" alt="モデルの動作、速度、メモリ、最近のリクエストを表示する Strata ターミナルアプリ">
|
||||
<br><sub>動作中のモデルとハードウェアの状態を確認できます。</sub>
|
||||
</p>
|
||||
|
||||
このようなモデルは、ふつう数百GBのグラフィックスメモリを持つサーバーで動きます。お使いのグラフィックスカードは12〜24 GBです。Strata は、PC全体で作業を分担して、モデルを収めます。
|
||||
図書館にたとえてみます。
|
||||
<a id="build-from-source"></a>
|
||||
|
||||
- モデルは、24,576人の小さな専門家（「エキスパート」）からなるチームです。24,576冊の参考書のようなものです。1つの単語に必要なのは、そのうち10冊だけです。
|
||||
- **グラフィックスカードは机です。** 小さくて速く、最もよく使う約4,000冊が開いて置いてあります。
|
||||
- **RAM は閲覧室の書棚です。** 24,576冊すべてがここにあります。単語に必要な本が机になければ、プロセッサがすぐに書棚から読み、その間もカードは作業を続けます。
|
||||
- **SSD は書庫です。** 29 GBの参照テーブルがしまってあります。モデルは単語ごとにその数行だけを読むので、RAM に置いておく必要はありません。
|
||||
[ソースからビルド](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [インストールガイド（全文）](docs/INSTALL.md)
|
||||
|
||||
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="モデルの24,576人のエキスパート。よく使うものはグラフィックスカードに、全員が RAM に、参照テーブルは SSD にあります"></p>
|
||||
<a id="what-you-need"></a>
|
||||
|
||||
小さな助手が、次の数単語も予想します。大きなモデルがそれをまとめて確認し、正しいものだけを残します。答えは同じで、1.6〜1.8倍早く返ります。
|
||||
長い文章は、一度に最大8,192トークンの大きな塊で、毎秒1,000トークンを超える速さで読みます。
|
||||
## ハードウェア
|
||||
|
||||
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="小さな助手が次の単語を予想し、大きなモデルがそれをまとめて確認して、正しいものだけを残します"></p>
|
||||
| 対応ハードウェア | ガイド |
|
||||
| --- | --- |
|
||||
| **NVIDIA GeForce RTX 20 / 30 / 40 / 50 シリーズ** | [NVIDIA のセットアップ](docs/INSTALL.md#what-you-need) |
|
||||
| **AMD Radeon RX 7700 / 7800 / 7900、RX 9060 / 9070、AI PRO R9700** | [AMD のセットアップ](docs/AMD_HIP.md) |
|
||||
| **AMD Radeon RX 6800 / 6900 · 試験的** | [RX 6000 のセットアップ](docs/AMD_HIP.md#rdna2-gfx1030) |
|
||||
| **Intel Arc · Linux** | [Intel のセットアップ](docs/INTEL_ARC.md) |
|
||||
| **Apple Silicon · M1 以降** | [Mac のセットアップ](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max · Strix Halo** | [Strix Halo のセットアップ](docs/STRIX_HALO.md) |
|
||||
|
||||
詳しい説明は [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md) にあります。各部分と数値は
|
||||
[詳細](docs/DETAILS.md#how-it-works)と[論文](docs/paper/Strata-Paper.pdf)を見てください。
|
||||
<a id="using-it"></a>
|
||||
|
||||
## うまくいかないとき
|
||||
## ツール
|
||||
|
||||
- **Strata を初めて起動したときに、PCが固まりました。** モデルを RAM に読み込んでいます。ウィンドウを閉じずに待ってください。10分たっても固まったままなら、
|
||||
PCを再起動し、ほかのプログラムを閉じてからもう一度試すか、小さいサイズを選んでください。
|
||||
- **ダウンロードやインストールの途中で止まりました。** アプリか `strata launch` をもう一度起動してください。止まったところから再開します。
|
||||
- **とても遅い、または「the engine stopped unexpectedly」と表示される。** PCの空き RAM が足りません。
|
||||
ほかのプログラムを閉じるか（ブラウザはメモリを多く使います）、Q2_0 や IQ2_XS のような小さいサイズを選んでください。
|
||||
- **このPCに合うエンジンがないと表示される。** アプリでは、**設定、エンジン**を開いて**エンジンを更新**を押します。ターミナルでは
|
||||
`strata update` を実行します。
|
||||
- **ポート 8080 はすでに使われていると表示される。** Strata がすでに動いています。ウィンドウを探してください。
|
||||
**Claude Code · Codex CLI · OpenCode · oh-my-pi · pi.dev · Hermes Agent · OpenClaw**。OpenAI 互換と Anthropic 互換の API を通じて、ほかのツールも使えます。
|
||||
|
||||
そのほかの対処法は [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) にあります。それでも解決しない場合は、
|
||||
[issue](https://github.com/Niko1221/Strata/issues) を作成して、ログを添付してください。アプリでは**設定、エンジン、診断、ログを保存**、
|
||||
ターミナルでは `strata doctor` の出力です。セキュリティ上の問題は、非公開で報告してください: [SECURITY.md](SECURITY.md)。
|
||||
**`strata connect`** を実行すると、お使いのツールの設定方法が表示されます。[接続ガイド](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect)。
|
||||
|
||||
## クレジットとライセンス
|
||||
<a id="guides"></a>
|
||||
<a id="how-fast-is-it"></a>
|
||||
<a id="how-does-it-work"></a>
|
||||
<a id="something-went-wrong"></a>
|
||||
|
||||
モデルは Qwen チームの [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) です。圧縮は
|
||||
[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)、UkisAI（Swift 1.5）、agentionai（Gyro）、
|
||||
Unsloth が行いました。DeepSeek V4.1-Flash のセットも ISTA-DASLab が作りました。Strata は
|
||||
[llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) の一部を使っています。すべてのクレジットは
|
||||
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits) にあります。Strata は [MIT ライセンス](LICENSE)のオープンソースです。一部のコンポーネントと
|
||||
すべてのモデルには、それぞれのライセンスがあります（[該当するもの](docs/HOW_IT_WORKS.md#license)）。
|
||||
## ヘルプ
|
||||
|
||||
## コントリビューション
|
||||
[すべてのガイド](docs) · [トラブルシューティング](docs/TROUBLESHOOTING.md) · [不具合を報告](https://github.com/Niko1221/Strata/issues/new/choose) ·
|
||||
[ベンチマーク](docs/COMMUNITY_BENCHMARKS.md) · [仕組み](docs/HOW_IT_WORKS.md) · [貢献する](CONTRIBUTING.md)
|
||||
|
||||
バグを見つけた、変更やベンチマークを送りたい、という方は、先に [CONTRIBUTING.md](CONTRIBUTING.md) を読んでください。
|
||||
ほかの人があなたの成果を確認できるように、何を書くべきかが説明してあります。
|
||||
## Strata を支援する
|
||||
|
||||
## Strata を応援する
|
||||
**ローカル AI をみんなに開かれたものに。** コーヒー1杯が、開発と、より多くのハードウェアでのテストの支えになります。
|
||||
|
||||
Strata は無料で、MIT ライセンスです。私が入手できるハードウェアで開発とテストをしています。お使いのPCで快適に動いたなら、コーヒー1杯分の支援は、
|
||||
新しいカードのテストと、報告いただいた不具合の修正を続ける助けになります。無理のない範囲で大丈夫です。使って、気づいたことを報告してもらえるだけでも同じように助かります。
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: Strata を支援する" height="48"></a>
|
||||
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a>
|
||||
[MIT ライセンス](LICENSE) · [クレジットとモデルのライセンス](docs/HOW_IT_WORKS.md#credits) · [プライバシー](docs/SAFETY.md)
|
||||
|
||||
Strata のスポンサーに関心がある方は、[GitHub](https://github.com/Niko1221) から連絡してください。
|
||||
<a id="option-c-let-your-ai-set-it-up"></a>
|
||||
|
||||
<sub>AI ツールと自動リーダー向け: <a href="AGENTS.md">技術リファレンス</a> · <a href="docs/AI_SETUP.md">セットアップ手順</a></sub>
|
||||
|
||||
**スポンサー、ハードウェア提携、共同開発のご相談:** [[email protected]](mailto:[email protected]).
|
||||
@@ -1,357 +1,191 @@
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="apps/host/public/brand/strata-lockup-dark.svg">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="380">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="340">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
<p align="center"><b>English</b> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.fr.md">Français</a> · <a href="README.es.md">Español</a> · <a href="README.pt-BR.md">Português</a></p>
|
||||
|
||||
<h3 align="center">Run a state-of-the-art model on your own gaming PC</h3>
|
||||
|
||||
<p align="center">Support for NVIDIA, AMD, Intel · Windows, Linux, Mac · free and open source</p>
|
||||
<h3 align="center">Run powerful AI on your own computer.</h3>
|
||||
<p align="center">Chat, code and work with images, locally.</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?include_prereleases&label=release" alt="Latest release"></a>
|
||||
<a href="#models">Models</a> · <a href="#install"><b>Install</b></a> ·
|
||||
<a href="#hardware">Hardware</a> · <a href="#tools">Tools</a> ·
|
||||
<a href="#help">Help</a> · <a href="#support-strata">Support</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Download Strata"></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="MIT license"></a>
|
||||
<a href="https://github.com/Niko1221/Strata/stargazers"><img src="https://img.shields.io/github/stars/Niko1221/Strata?style=flat" alt="GitHub stars"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black" alt="Buy me a coffee"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Support Strata on Buy Me a Coffee"></a>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="A voxel pagoda garden that Strata's model wrote, running in the browser"></a><br>
|
||||
<sub>A voxel pagoda garden, 1 shot prompt running on an RTX 5070 with Strata (IQ3_S, 128K context) ·
|
||||
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">full video (49 s)</a></sub></p>
|
||||
<p align="center"><sub><b>English</b> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.fr.md">Français</a> · <a href="README.es.md">Español</a> · <a href="README.pt-BR.md">Português</a></sub></p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#install"><b>Install</b></a> ·
|
||||
<a href="#models">Models</a> ·
|
||||
<a href="#how-fast-is-it">Speed</a> ·
|
||||
<a href="#using-it">Using it</a> ·
|
||||
<a href="#how-does-it-work">How it works</a> ·
|
||||
<a href="#something-went-wrong">Help</a> ·
|
||||
<a href="#support-strata">Support</a>
|
||||
</p>
|
||||
**Free, open source, and yours to run.** Use Strata on your desktop or with your favorite coding tools. No account needed.
|
||||
|
||||
Strata runs **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)**, a 125-billion-parameter model that
|
||||
normally needs a server, on a normal PC. From 0.2.4 it also runs **DeepSeek V4.1-Flash** in the Terminal. It chats, writes
|
||||
code, reads pictures and works with your apps and coding agents. Your chats and files stay on your PC.
|
||||
## Models
|
||||
|
||||
- **Runs on a gaming PC.** 32 GB of RAM and graphics memory together is enough to start.
|
||||
- **Works with your tools.** Claude Code, Codex CLI, OpenCode and other agents connect through the OpenAI and Anthropic APIs.
|
||||
- **Reads pictures**, on by default.
|
||||
- **Free and private.** MIT license, no account, no cloud, no telemetry ([what it downloads and when](docs/SAFETY.md)).
|
||||
We recommend **ISTA-DASLab first**, then **Unsloth** and other providers, where a compatible model fits.
|
||||
|
||||
## What you need
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.8-Flash-Next</b></summary>
|
||||
|
||||
**At least 32 GB of memory: RAM and graphics memory (VRAM) together.** On a Mac or a Ryzen AI Max that is the unified
|
||||
memory. A 32 GB PC with a 12-16 GB card runs the smaller sizes, and 64 GB runs every size.
|
||||
|
||||
| Works on | Guide |
|
||||
| Provider | Choices |
|
||||
| --- | --- |
|
||||
| **NVIDIA** GeForce RTX 20, 30, 40 and 50 series, 12 GB or more | [Install](#install) |
|
||||
| **AMD** Radeon RX 6800 and newer, Radeon AI PRO R9700 | [AMD cards](docs/AMD_HIP.md) |
|
||||
| **Intel** Arc | [Intel Arc](docs/INTEL_ARC.md) |
|
||||
| **Apple Silicon Mac**, M1 or newer, macOS 15 or newer | [Mac](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max (Strix Halo)** and other unified-memory PCs | [Strix Halo](docs/STRIX_HALO.md) |
|
||||
| Older cards and processors | [Older GPUs](docs/OLDER_GPUS.md) |
|
||||
| **ISTA-DASLab · recommended** | Q2\_0, IQ2\_XS, IQ3\_XXS, IQ3\_S. [Coder: IQ1\_M](docs/MODELS.md#coder) for coding. |
|
||||
| **Unsloth** | UD-IQ4\_XS, UD-Q4\_K\_XL, UD-Q5\_K\_XL, UD-Q6\_K\_XL, Q8\_0 |
|
||||
| **Gyro · agentionai** | [Gyro-S, Gyro-M](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF). Lower memory use but slower. |
|
||||
| **Other options** | [Swift 1.5](docs/MODELS.md#swift-15), [NVFP4](docs/NVFP4.md), [OrcaRouter](docs/ORCA.md) |
|
||||
|
||||
You also need about 80 GB of free disk (an SSD), Windows 10 / 11, Linux or macOS, and a current graphics driver. Two or
|
||||
three cards of the same make can share a model ([multi-GPU](docs/MULTI_GPU.md)).
|
||||
[Compare sizes, memory needs and compatibility](docs/MODELS.md).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.6-35B-A3B</b></summary>
|
||||
|
||||
**Unsloth:** UD-Q4_K_XL, UD-Q6_K_XL, Q8_0. These are the currently documented packs for this model.
|
||||
|
||||
[Model guide and compatibility](docs/QWEN36.md).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/deepseek.svg" width="22" alt="" align="middle"> <b>DeepSeekV4.1-Flash-1017</b></summary>
|
||||
|
||||
In the **desktop app** and the **Terminal**. What does not fit in memory is read from your SSD, so it needs a lot of free
|
||||
disk space.
|
||||
|
||||
| Set (IST-DAS) | Download | Free disk needed |
|
||||
| --- | ---: | ---: |
|
||||
| 1.75 bpw | 239 GB | about 245 GB |
|
||||
| 2.00 bpw | 257 GB | about 265 GB |
|
||||
| 2.25 bpw | 274 GB | about 280 GB |
|
||||
|
||||
Runs on NVIDIA RTX 30, 40 and 50 series (Windows and Linux) and AMD Radeon AI PRO R9700 (Linux). Setup downloads the
|
||||
files and the DeepSeek engine, then starts the model.
|
||||
|
||||
```sh
|
||||
strata launch --family deepseek41 --model ist-das-2.00bpw --plan
|
||||
```
|
||||
|
||||
Measured with the 2.00 bpw set on Strata 0.2.5. Output is the speed of the answer, prompt is how fast Strata reads your input.
|
||||
|
||||
| Hardware | Output | Prompt |
|
||||
| --- | ---: | ---: |
|
||||
| RTX 5070 12 GB, 64 GB RAM | 4 to 6 tokens/s | about 110 tokens/s (4K) |
|
||||
| RTX 3060 12 GB, 46 GB RAM | 9 to 11 tokens/s | about 155 tokens/s (4K) |
|
||||
| 1x Radeon AI PRO R9700, 251 GB RAM | 21 to 25 tokens/s | 320 to 360 tokens/s (8K) |
|
||||
| 4x Radeon AI PRO R9700 | 74 to 77 tokens/s | 400 to 450 tokens/s (8K) |
|
||||
|
||||
[Memory, disk and details](docs/DEEPSEEK41.md) · [Release notes](https://github.com/Niko1221/Strata/releases)
|
||||
|
||||
</details>
|
||||
|
||||
<a id="the-other-sizes"></a>
|
||||
|
||||
Not sure what fits? [Choose a model for your computer](docs/MODELS.md#pick-by-memory).
|
||||
|
||||
## Install
|
||||
|
||||
### 1. The Strata app (easiest)
|
||||
<a id="option-a-the-strata-app"></a>
|
||||
|
||||
Download the file for your computer, open it, and press install on the model it recommends.
|
||||
### Desktop app
|
||||
|
||||
**Windows 10 / 11**
|
||||
**[Download Strata for Windows, Linux or Mac →](https://github.com/Niko1221/Strata/releases/latest)**
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-windows-x64-setup.exe)
|
||||
Install the app, choose a model, and start chatting. There is one installer for each system. On first start the app
|
||||
downloads the engine for your graphics card. [Help with the installer](docs/INSTALL.md#the-app).
|
||||
|
||||
**Linux**
|
||||
<sub>Files: `Strata-0.2.6-windows-x64-setup.exe` · `Strata-0.2.6-linux-x86_64.AppImage` · `Strata-0.2.6-macos-arm64.dmg`</sub>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-linux-x86_64.AppImage)
|
||||
<p align="center">
|
||||
<img src="docs/media/app-chat.png" width="900" alt="Strata desktop app with a chat and live code preview side by side">
|
||||
<br><sub>Chat, files and a live preview in one window.</sub>
|
||||
</p>
|
||||
|
||||
**Mac** (Apple Silicon, macOS 15 or newer)
|
||||
<a id="option-b-the-terminal-one-line"></a>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-macos-arm64.dmg)
|
||||
### Terminal app
|
||||
|
||||
The app picks the right engine for your graphics card by itself. All files are on the [releases page](https://github.com/Niko1221/Strata/releases/latest).
|
||||
Prefer the terminal? Install Strata, then run **`strata launch`**.
|
||||
|
||||
| First start | What to do |
|
||||
| --- | --- |
|
||||
| **Windows** | Run the `.exe`. No administrator rights needed. If Windows says "Windows protected your PC", choose **More info**, then **Run anyway**. |
|
||||
| **Linux** | `chmod +x` the file and start it. It needs FUSE 2: `sudo apt install libfuse2t64` (Ubuntu 24.04 and newer) or `libfuse2` (22.04). |
|
||||
| **Mac** | Open the `.dmg` and drag Strata onto Applications. |
|
||||
<details open>
|
||||
<summary><b>Install commands</b></summary>
|
||||
|
||||
The model download is big (40-70 GB). If it stops, it continues where it left off.
|
||||
|
||||
### 2. The Terminal (one line)
|
||||
|
||||
The `strata` command, for scripts, servers and AI agents. It is also the way to run DeepSeek.
|
||||
|
||||
Linux and Mac:
|
||||
**Linux / Mac**
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
```
|
||||
|
||||
Windows (PowerShell):
|
||||
**Windows PowerShell**
|
||||
|
||||
```powershell
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
```
|
||||
|
||||
Then start it. It asks one question at a time (model, context, pictures, who may connect), downloads what it needs and
|
||||
starts serving:
|
||||
On Windows, open a new terminal afterwards. Then:
|
||||
|
||||
```sh
|
||||
strata launch
|
||||
```
|
||||
|
||||
No admin rights needed. The install checks the SHA-256 of everything it downloads. On Windows, open a new terminal window
|
||||
after the install. All screens and flags: [docs/TERMINAL_LAUNCH.md](docs/TERMINAL_LAUNCH.md).
|
||||
|
||||
### 3. Let your AI set it up
|
||||
|
||||
Paste this into Claude Code, Cursor, Codex, GitHub Copilot or another coding assistant:
|
||||
|
||||
```text
|
||||
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
|
||||
```
|
||||
|
||||
It checks your graphics card, RAM and disk, picks the model that fits, installs it, starts it and tells you how to connect
|
||||
your apps. AI tools can also install, start and stop Strata through its [MCP server](docs/MCP_SERVER.md).
|
||||
|
||||
Prefer to build it yourself? See [Build from source](#build-from-source).
|
||||
|
||||
## Models
|
||||
|
||||
The app and `strata launch` recommend a size for your memory. Smaller sizes are faster, larger sizes are a bit smarter.
|
||||
|
||||
### Low on memory? Start with Gyro
|
||||
|
||||
[Gyro](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF) by agentionai is the smallest way to run
|
||||
Qwen3.8-Flash-Next. It runs on NVIDIA, AMD, Intel Arc and Apple Silicon.
|
||||
|
||||
| Size | Memory it needs (RAM + VRAM) | Download | Good for |
|
||||
| --- | ---: | ---: | --- |
|
||||
| **Gyro-S** | 27.6 GB | 58.5 GB | PCs that fit nothing else. 9 tokens/s on a 32 GB Mac mini. |
|
||||
| **Gyro-M** | 35 GB | 92.0 GB | About IQ3_XXS quality with 20% less GPU memory. |
|
||||
|
||||
Start it with `strata launch --model gyro-s`. Details: [docs/GYRO.md](docs/GYRO.md).
|
||||
|
||||
### Qwen3.8-Flash-Next sizes
|
||||
|
||||
| Your memory (RAM + VRAM) | Take | Why |
|
||||
| --- | --- | --- |
|
||||
| **32 GB** | **Gyro-S**, or the **[Coder](docs/MODELS.md#coder)** for code | The Coder keeps the experts that code uses |
|
||||
| **40 GB** | **[Swift 1.5](docs/MODELS.md#swift-15)** IQ2_XS (39.2 GB) or **Gyro-M** | Swift thinks for a shorter time before it answers |
|
||||
| **48 GB** | **IQ3_XXS** (47.0 GB) | Slower, better |
|
||||
| **64 GB** | **IQ3_S** (54.8 GB) | Matches the full model on the published tests |
|
||||
| **96 GB or more** | Unsloth's **UD-IQ4_XS** to **Q8_0** | Closest to the full model |
|
||||
|
||||
Less RAM than a row says can still work, only slower: Strata switches to its
|
||||
[low-RAM mode](docs/MODELS.md#a-big-graphics-card-and-little-ram) by itself. Also available:
|
||||
[NVFP4](docs/NVFP4.md) for NVIDIA cards, [Qwen3.6-35B-A3B](docs/QWEN36.md) (smaller, several requests at once) and
|
||||
[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs). Every size and what fits where:
|
||||
[docs/MODELS.md](docs/MODELS.md).
|
||||
|
||||
### DeepSeek V4.1-Flash (Terminal, from 0.2.4)
|
||||
|
||||
A much bigger model that runs on one 12 GB graphics card with 46 to 64 GB of RAM. Strata keeps the busiest parts in memory and
|
||||
reads the rest from your SSD. [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF)
|
||||
made the sets. It runs on NVIDIA RTX 30, 40 and 50 cards and the Radeon AI PRO R9700. Pick it in `strata launch`; the desktop
|
||||
app does not offer it yet.
|
||||
|
||||
| Set | Download | Quality | Writes code at |
|
||||
| --- | ---: | --- | --- |
|
||||
| **IQ1_M** (1.75 bits) | 239 GB | The smallest; weaker on plain prose | not measured on a PC yet |
|
||||
| **IQ2_XXS** (2.00 bits) | 257 GB | Good balance | 10 tokens/s (RTX 3060), 5 (RTX 5070), 48 (4x R9700) |
|
||||
| **IQ2_XS** (2.25 bits) | 274 GB | Closest to the original | 15 tokens/s (one R9700), 80 (4x R9700) |
|
||||
| **IQ4_NL** (Engram table) | in every download above | The lookup table all three sets share | |
|
||||
|
||||
Start it with `strata launch --model ist-das-2.00bpw` (or `ist-das-1.75bpw`, `ist-das-2.25bpw`). Quality numbers, speeds
|
||||
per PC and the manual install: [docs/DEEPSEEK41.md](docs/DEEPSEEK41.md).
|
||||
|
||||
## How fast is it?
|
||||
|
||||
Based on a 12 GB graphics card (RTX 5070, 64 GB RAM). Results from other cards:
|
||||
[speed of each model](docs/MODELS.md#how-fast-is-each-size) and [community results](docs/COMMUNITY_BENCHMARKS.md).
|
||||
|
||||
| Model | Answers appear at | Reads your prompt at |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████████` 94 tokens/s | `████████████` 2,650 tokens/s |
|
||||
| **IQ2_XS** | `██████████░░` 79 tokens/s | `█████████░░░` 2,090 tokens/s |
|
||||
| **IQ3_XXS** | `████████░░░░` 62 tokens/s | `████████░░░░` 1,750 tokens/s |
|
||||
| **IQ3_S** | `███████░░░░░` 53 tokens/s | `███████░░░░░` 1,620 tokens/s |
|
||||
| **Coder** | `███████░░░░░` 55 tokens/s | `██████████░░` 2,180 tokens/s |
|
||||
|
||||
On a 16 GB AMD card (RX 9070 XT, 47 GB RAM):
|
||||
|
||||
| Model | Answers appear at | Reads your prompt at |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████░░░░` 60 tokens/s | `█████░░░░░░░` 1,160 tokens/s |
|
||||
| **IQ2_XS** | `███████░░░░░` 52 tokens/s | `█████░░░░░░░` 1,110 tokens/s |
|
||||
| **Coder** | `██████░░░░░░` 44 tokens/s | `██████░░░░░░` 1,420 tokens/s |
|
||||
|
||||
A token is about ¾ of a word, so 60 tokens per second is faster than you can read. "Reads your prompt" is how fast Strata
|
||||
takes in a long document, code or chat history.
|
||||
|
||||
## Using it
|
||||
|
||||
### In the app
|
||||
|
||||
Chat, a **Monitor** of the model and your GPU, CPU and RAM, a **Models** page and **Settings**. The chat can work in a
|
||||
folder on your PC. In **Manual** mode it asks before it changes a file or runs a command. It can search the web
|
||||
(DuckDuckGo or Bing, no account); the search words are the one thing that leaves your PC.
|
||||
|
||||
<p align="center"><img src="docs/media/app-chat.png" width="900" alt="The Strata app: a chat where the agent built a small voxel pagoda garden, with the page open in the preview on the right"><br>
|
||||
<sub>The agent wrote a voxel pagoda garden as one HTML file, and the page opens in the preview next to the chat</sub></p>
|
||||
|
||||
### In the Terminal
|
||||
|
||||
<p align="center"><img src="docs/media/terminal-monitor.png" width="900" alt="The Strata Terminal monitor overview while the model writes a reply"><br>
|
||||
<sub><code>strata monitor</code> while the model writes a reply: speed, GPU, memory and the recent requests</sub></p>
|
||||
|
||||
```sh
|
||||
strata launch # guided: model, context, pictures, access, then serve
|
||||
strata launch --model q2_0 --yes --no-tui # no questions: serve and print the address
|
||||
strata launch --status # what is being served now
|
||||
strata launch --stop # stop it
|
||||
strata monitor # watch speed, GPU, memory and requests
|
||||
strata update # get the newest engine
|
||||
strata doctor # check the installed files
|
||||
```
|
||||
|
||||
To reach Strata from your phone or another PC, use `strata launch --access lan`. It makes an API key for you.
|
||||
`strata --help` lists everything, and [docs/TERMINAL.md](docs/TERMINAL.md) explains it.
|
||||
|
||||
### With AI agents and other apps
|
||||
|
||||
An agent installs Strata with the one-line command above and runs `strata launch --model NAME --yes --no-tui`. Steps,
|
||||
flags and exit codes: [docs/AI_SETUP.md](docs/AI_SETUP.md). Repository rules for agents: [AGENTS.md](AGENTS.md).
|
||||
|
||||
| Your app speaks | Base URL |
|
||||
| --- | --- |
|
||||
| OpenAI API (most apps and agents) | `http://127.0.0.1:8080/v1` (any API key, any model name) |
|
||||
| Anthropic API (Claude Code) | `http://127.0.0.1:8080`, without `/v1`. Claude Code: `ANTHROPIC_BASE_URL=http://127.0.0.1:8080` |
|
||||
| OpenAI Responses API (Codex CLI) | `http://127.0.0.1:8080/v1/responses` ([setup](docs/DETAILS.md#the-responses-api-and-codex-cli)) |
|
||||
|
||||
`strata connect` prints the exact settings for OpenCode, Codex CLI, Claude Code, oh-my-pi, pi, Hermes Agent and OpenClaw,
|
||||
and `strata connect opencode --write` writes them for you (with a backup).
|
||||
|
||||
### Good to know
|
||||
|
||||
- **Pictures** are on for every model that can read them. Attach or paste one in the chat, or send it from your app. Details: [docs/IMAGES.md](docs/IMAGES.md).
|
||||
- **Thinking** can be off, low, medium or high, in the chat menu or your app's "reasoning effort". Off is fastest.
|
||||
- **The first message** of a chat is read in full, about 1 minute per 30,000 tokens. Follow-ups start in seconds.
|
||||
- **One request at a time** by default. For more, use `--parallel N` ([BATCHING.md](docs/BATCHING.md)). On a 12 GB card each answer gets slower.
|
||||
|
||||
## Build from source
|
||||
|
||||
<details>
|
||||
<summary>Windows, Linux and Mac from the repository</summary>
|
||||
|
||||
[Download Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it, or `git clone` it.
|
||||
**Windows:** double-click `START-HERE.bat`. **Linux:** run `./setup.sh` in the Strata folder.
|
||||
|
||||
**Mac (Apple Silicon, macOS 15 or newer):** install Apple's Command Line Tools once with `xcode-select --install`
|
||||
(Xcode itself is not needed). Open Terminal in the Strata folder and run `./setup.sh`. It builds the Mac engine on your
|
||||
Mac (about 7 minutes), asks the same questions, downloads the model and starts it. Next time, `./setup.sh` starts it
|
||||
right away. Which size fits your Mac: [docs/MAC.md](docs/MAC.md).
|
||||
|
||||
The installer finds your card and sets up the right engine. It asks which model and size, how much context (how much text
|
||||
the model keeps in mind) and whether it should read pictures. Press Enter for the recommended answer. Then it downloads the
|
||||
model (about 70 GB) and starts it. If the download stops, run it again and it continues. Your browser opens the Strata
|
||||
page at `http://127.0.0.1:8080`.
|
||||
|
||||
> While the model starts, your PC can be slow or stop responding for 1-3 minutes (longest the first time). Strata loads
|
||||
> 35-55 GB into your RAM and locks part of it for the graphics card. Wait, and don't close the window.
|
||||
|
||||
Next time, run `START-HERE.bat` (or `./setup.sh`) again. It starts right away and downloads nothing twice. Close its
|
||||
window to stop the model. `UPDATE.bat` (`./update.sh`) updates Strata without starting it. Updating, Docker, several cards,
|
||||
where the files go and every option: [docs/INSTALL.md](docs/INSTALL.md).
|
||||
|
||||
To measure your PC, run `BENCH.bat` (`./bench.sh`). It takes 5-10 minutes and writes a report you can share
|
||||
([how](docs/COMMUNITY_BENCHMARKS.md#run-the-benchmark)).
|
||||
|
||||
**Optional: more speed on one card when the whole model fits in RAM.** Two opt-in settings by RubenTorney (#1904, #1905).
|
||||
They help when you run one graphics card and your RAM holds every expert, for example Q2_0 or IQ3_XXS on a 64 GB PC. Open
|
||||
`strata-<model>.json` in the Strata data folder ([where](docs/INSTALL.md#where-things-are-stored)) and add:
|
||||
|
||||
- `"--adapt-async", "1"` to the `"args"` list. The expert cache then updates on a helper thread instead of pausing the
|
||||
answer. Measured on an RTX 5070 (Windows, Q2_0): +4-5% decode, +7-10% on code and long documents. Answers can differ
|
||||
slightly from the default and stay the same from run to run.
|
||||
- `"STRATA_PIN_BY_NEED": "1"` in the `"env"` block (Windows). Windows can lock only about half the RAM for the graphics
|
||||
card, and this locks the experts the card needs first. Measured on the RTX 5070: +4.4% prompt reading.
|
||||
|
||||
```json
|
||||
"args": [ ..., "--adapt-async", "1" ],
|
||||
"env": { "STRATA_PIN_BY_NEED": "1" }
|
||||
```
|
||||
|
||||
Restart Strata after the change. To undo, remove the two entries.
|
||||
Follow the prompts to download and start a model. [Terminal guide](docs/TERMINAL_LAUNCH.md).
|
||||
|
||||
</details>
|
||||
|
||||
## How does it work?
|
||||
<p align="center">
|
||||
<img src="docs/media/terminal-monitor.png" width="900" alt="Strata terminal app showing model activity, speed, memory and recent requests">
|
||||
<br><sub>Watch your model and hardware while it runs.</sub>
|
||||
</p>
|
||||
|
||||
Models like this one usually run on servers with hundreds of gigabytes of graphics memory. Your graphics card has
|
||||
12-24 GB. Strata makes the model fit by sharing the work across your whole PC. Picture a library.
|
||||
<a id="build-from-source"></a>
|
||||
|
||||
- The model is a team of 24,576 small specialists ("experts"), like 24,576 reference books. Each word needs only 10 of them.
|
||||
- **Your graphics card is the desk.** It is small and fast. The roughly 4,000 books used most often lie open on it.
|
||||
- **Your RAM is the shelves in the reading room.** All 24,576 books are there. When a word needs a book that is not on the
|
||||
desk, your processor reads it from the shelf at once, while the card keeps working.
|
||||
- **Your SSD is the archive.** It holds a 29 GB lookup table. The model reads a few rows of it per word, so it does not
|
||||
need to sit in RAM.
|
||||
[Build from source](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [Full install guide](docs/INSTALL.md)
|
||||
|
||||
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="The model's 24,576 experts: the busiest on the graphics card, all of them in RAM, a lookup table on the SSD"></p>
|
||||
<a id="what-you-need"></a>
|
||||
|
||||
A small helper also guesses the next few words. The big model checks all of them at once and keeps the right ones. You get
|
||||
the same answer, 1.6-1.8x sooner. Long texts are read in big pieces, up to 8,192 tokens at a time, at over 1,000 tokens
|
||||
per second.
|
||||
## Hardware
|
||||
|
||||
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="A small helper guesses the next words; the big model checks them all at once and keeps the right ones"></p>
|
||||
| Supported hardware | Guide |
|
||||
| --- | --- |
|
||||
| **NVIDIA GeForce RTX 20 / 30 / 40 / 50 series** | [NVIDIA setup](docs/INSTALL.md#what-you-need) |
|
||||
| **AMD Radeon RX 7700 / 7800 / 7900, RX 9060 / 9070, AI PRO R9700** | [AMD setup](docs/AMD_HIP.md) |
|
||||
| **AMD Radeon RX 6800 / 6900 · experimental** | [RX 6000 setup](docs/AMD_HIP.md#rdna2-gfx1030) |
|
||||
| **Intel Arc · Linux** | [Intel setup](docs/INTEL_ARC.md) |
|
||||
| **Apple Silicon · M1 or newer** | [Mac setup](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max · Strix Halo** | [Strix Halo setup](docs/STRIX_HALO.md) |
|
||||
|
||||
The longer explanation: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Every part and its numbers:
|
||||
[the details](docs/DETAILS.md#how-it-works) and the [paper](docs/paper/Strata-Paper.pdf).
|
||||
<a id="using-it"></a>
|
||||
|
||||
## Something went wrong?
|
||||
## Tools
|
||||
|
||||
- **My PC froze the first time Strata started.** It loads the model into RAM. Wait, and don't close the window. Still
|
||||
frozen after 10 minutes? Restart the PC, close other programs and try again, or pick a smaller size.
|
||||
- **It stopped while downloading or installing.** Start the app or `strata launch` again. It continues where it stopped.
|
||||
- **It is very slow, or it says "the engine stopped unexpectedly".** Your PC is short on free RAM. Close other programs
|
||||
(browsers use a lot), or pick a smaller size such as Q2_0 or IQ2_XS.
|
||||
- **It says there is no engine for this PC.** In the app, open **Settings, Engine** and press **Update engine**. In the
|
||||
Terminal, run `strata update`.
|
||||
- **It says port 8080 is already in use.** Strata is already running. Look for its window.
|
||||
**Claude Code · Codex CLI · OpenCode · oh-my-pi · pi.dev · Hermes Agent · OpenClaw**, and more through the OpenAI- and Anthropic-compatible APIs.
|
||||
|
||||
More fixes: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Still stuck? Open an
|
||||
[issue](https://github.com/Niko1221/Strata/issues) and attach the logs: in the app from **Settings, Engine, Diagnostics,
|
||||
Save logs**, in the Terminal the output of `strata doctor`. For a security problem, report it privately: [SECURITY.md](SECURITY.md).
|
||||
Run **`strata connect`** to see setup instructions for your tool. [Connection guide](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
|
||||
|
||||
## Credits and license
|
||||
<a id="guides"></a>
|
||||
<a id="how-fast-is-it"></a>
|
||||
<a id="how-does-it-work"></a>
|
||||
<a id="something-went-wrong"></a>
|
||||
|
||||
The model is [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team. It was compressed by
|
||||
[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5), agentionai (Gyro)
|
||||
and Unsloth. ISTA-DASLab also made the DeepSeek V4.1-Flash sets. Strata uses parts of
|
||||
[llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). All credits: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits).
|
||||
Strata is open source under the [MIT License](LICENSE). A few parts and every model have their own licenses
|
||||
([which ones](docs/HOW_IT_WORKS.md#license)).
|
||||
## Help
|
||||
|
||||
## Contributing
|
||||
|
||||
Found a bug, or want to send a change or a benchmark? Read [CONTRIBUTING.md](CONTRIBUTING.md) first. It says what to
|
||||
include so that others can check your work.
|
||||
[All guides](docs) · [Troubleshooting](docs/TROUBLESHOOTING.md) · [Report a bug](https://github.com/Niko1221/Strata/issues/new/choose) ·
|
||||
[Benchmarks](docs/COMMUNITY_BENCHMARKS.md) · [How it works](docs/HOW_IT_WORKS.md) · [Contribute](CONTRIBUTING.md)
|
||||
|
||||
## Support Strata
|
||||
|
||||
Strata is free and MIT-licensed. I build it and test it on whatever hardware I can get. If it runs well on your PC, a coffee helps
|
||||
me keep testing new cards and fixing what you report. No pressure. Using it and reporting what you find helps just as much.
|
||||
**Help keep local AI open to everyone.** A coffee helps fund development and testing on more hardware.
|
||||
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: support Strata" height="48"></a>
|
||||
|
||||
Interested in sponsoring Strata? Contact me via [GitHub](https://github.com/Niko1221).
|
||||
[MIT license](LICENSE) · [Credits and model licenses](docs/HOW_IT_WORKS.md#credits) · [Privacy](docs/SAFETY.md)
|
||||
|
||||
<a id="option-c-let-your-ai-set-it-up"></a>
|
||||
|
||||
<sub>For AI tools and automated readers: <a href="AGENTS.md">technical reference</a> · <a href="docs/AI_SETUP.md">setup instructions</a>.</sub>
|
||||
|
||||
**Sponsorships, hardware partnerships or collaborations:** [[email protected]](mailto:[email protected]).
|
||||
+122
-294
@@ -1,360 +1,188 @@
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="apps/host/public/brand/strata-lockup-dark.svg">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="380">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="340">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.fr.md">Français</a> · <a href="README.es.md">Español</a> · <b>Português</b></p>
|
||||
|
||||
<h3 align="center">Rode um modelo de ponta no seu próprio PC gamer</h3>
|
||||
|
||||
<p align="center">Suporte a NVIDIA, AMD, Intel · Windows, Linux, Mac · gratuito e de código aberto</p>
|
||||
<h3 align="center">IA poderosa no seu próprio computador.</h3>
|
||||
<p align="center">Converse, programe e trabalhe com imagens, localmente.</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?include_prereleases&label=release" alt="Última versão"></a>
|
||||
<a href="#modelos">Modelos</a> · <a href="#instalação"><b>Instalação</b></a> ·
|
||||
<a href="#hardware">Hardware</a> · <a href="#ferramentas">Ferramentas</a> ·
|
||||
<a href="#ajuda">Ajuda</a> · <a href="#apoie-o-strata">Apoiar</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Baixar o Strata"></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="Licença MIT"></a>
|
||||
<a href="https://github.com/Niko1221/Strata/stargazers"><img src="https://img.shields.io/github/stars/Niko1221/Strata?style=flat" alt="Estrelas no GitHub"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black" alt="Me pague um café"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Apoie o Strata no Buy Me a Coffee"></a>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Um jardim de pagodes em voxels que o modelo do Strata escreveu, rodando no navegador"></a><br>
|
||||
<sub>Um jardim de pagodes em voxels, gerado com um único prompt, rodando numa RTX 5070 com o Strata (IQ3_S, contexto de 128K) ·
|
||||
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">vídeo completo (49 s)</a></sub></p>
|
||||
<p align="center"><sub><a href="README.md">English</a> · <a href="README.zh-CN.md">简体中文</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.fr.md">Français</a> · <a href="README.es.md">Español</a> · <b>Português</b></sub></p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#instalação"><b>Instalação</b></a> ·
|
||||
<a href="#modelos">Modelos</a> ·
|
||||
<a href="#qual-é-a-velocidade">Velocidade</a> ·
|
||||
<a href="#como-usar">Como usar</a> ·
|
||||
<a href="#como-funciona">Como funciona</a> ·
|
||||
<a href="#algo-deu-errado">Ajuda</a> ·
|
||||
<a href="#apoie-o-strata">Apoiar</a>
|
||||
</p>
|
||||
**Gratuito, de código aberto e seu para executar.** Use o Strata no desktop ou com suas ferramentas de programação favoritas. Não precisa de conta.
|
||||
|
||||
O Strata roda o **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)**, um modelo de 125 bilhões de parâmetros que
|
||||
normalmente precisa de um servidor, num PC comum. A partir da 0.2.4 ele também roda o **DeepSeek V4.1-Flash** no Terminal. Ele
|
||||
conversa, escreve código, lê imagens e funciona com os seus aplicativos e agentes de programação. Suas conversas e arquivos
|
||||
ficam no seu PC.
|
||||
## Modelos
|
||||
|
||||
- **Roda num PC gamer.** 32 GB somando memória RAM e memória da placa de vídeo já bastam para começar.
|
||||
- **Funciona com as suas ferramentas.** Claude Code, Codex CLI, OpenCode e outros agentes se conectam pelas APIs da OpenAI e da Anthropic.
|
||||
- **Lê imagens**, ativado por padrão.
|
||||
- **Gratuito e privado.** Licença MIT, sem conta, sem nuvem, sem telemetria ([o que ele baixa e quando](docs/SAFETY.md)).
|
||||
Recomendamos **primeiro a ISTA-DASLab**, depois a **Unsloth** e outros provedores, quando um modelo compatível cabe na memória.
|
||||
|
||||
## O que você precisa
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.8-Flash-Next</b></summary>
|
||||
|
||||
**Você precisa de pelo menos 32 GB de memória: RAM e memória da placa de vídeo (VRAM) somadas.** Num Mac ou num Ryzen AI Max, isso é a
|
||||
memória unificada. Um PC de 32 GB com uma placa de 12 a 16 GB roda os tamanhos menores, e 64 GB rodam todos os tamanhos.
|
||||
|
||||
| Funciona em | Guia |
|
||||
| Provedor | Opções |
|
||||
| --- | --- |
|
||||
| **NVIDIA** GeForce RTX série 20, 30, 40 e 50, 12 GB ou mais | [Instalação](#instalação) |
|
||||
| **AMD** Radeon RX 6800 e mais novas, Radeon AI PRO R9700 | [Placas AMD](docs/AMD_HIP.md) |
|
||||
| **Intel** Arc | [Intel Arc](docs/INTEL_ARC.md) |
|
||||
| **Mac com Apple Silicon**, M1 ou mais novo, macOS 15 ou mais novo | [Mac](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max (Strix Halo)** e outros PCs de memória unificada | [Strix Halo](docs/STRIX_HALO.md) |
|
||||
| Placas e processadores mais antigos | [GPUs antigas](docs/OLDER_GPUS.md) |
|
||||
| **ISTA-DASLab · recomendado** | Q2\_0, IQ2\_XS, IQ3\_XXS, IQ3\_S. [Coder: IQ1\_M](docs/MODELS.md#coder) para programar. |
|
||||
| **Unsloth** | UD-IQ4\_XS, UD-Q4\_K\_XL, UD-Q5\_K\_XL, UD-Q6\_K\_XL, Q8\_0 |
|
||||
| **Gyro · agentionai** | [Gyro-S, Gyro-M](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF). Usa menos memória, mas é mais lento. |
|
||||
| **Outras opções** | [Swift 1.5](docs/MODELS.md#swift-15), [NVFP4](docs/NVFP4.md), [OrcaRouter](docs/ORCA.md) |
|
||||
|
||||
Você também precisa de cerca de 80 GB de espaço livre no disco (um SSD), Windows 10 / 11, Linux ou macOS, e um driver de vídeo
|
||||
atualizado. Duas ou três placas da mesma marca podem dividir um modelo ([multi-GPU](docs/MULTI_GPU.md)).
|
||||
[Compare tamanhos, necessidades de memória e compatibilidade](docs/MODELS.md).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.6-35B-A3B</b></summary>
|
||||
|
||||
**Unsloth:** UD-Q4_K_XL, UD-Q6_K_XL, Q8_0. Esses são os pacotes documentados atualmente para este modelo.
|
||||
|
||||
[Guia do modelo e compatibilidade](docs/QWEN36.md).
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/deepseek.svg" width="22" alt="" align="middle"> <b>DeepSeekV4.1-Flash-1017</b></summary>
|
||||
|
||||
No **aplicativo de desktop** e no **Terminal**. O que não cabe na memória é lido do seu SSD, por isso é preciso muito espaço livre em disco.
|
||||
|
||||
| Conjunto (IST-DAS) | Download | Espaço livre necessário |
|
||||
| --- | ---: | ---: |
|
||||
| 1.75 bpw | 239 GB | cerca de 245 GB |
|
||||
| 2.00 bpw | 257 GB | cerca de 265 GB |
|
||||
| 2.25 bpw | 274 GB | cerca de 280 GB |
|
||||
|
||||
Roda em NVIDIA RTX 30, 40 e 50 (Windows e Linux) e em AMD Radeon AI PRO R9700 (Linux). A instalação baixa os arquivos e o motor do DeepSeek e depois inicia o modelo.
|
||||
|
||||
```sh
|
||||
strata launch --family deepseek41 --model ist-das-2.00bpw --plan
|
||||
```
|
||||
|
||||
Medido com o conjunto de 2.00 bpw no Strata 0.2.5. “Saída” é a velocidade da resposta; “Prompt” é a rapidez com que o Strata lê o seu texto.
|
||||
|
||||
| Hardware | Saída | Prompt |
|
||||
| --- | ---: | ---: |
|
||||
| RTX 5070 12 GB, 64 GB de RAM | 4 a 6 tokens/s | cerca de 110 tokens/s (4K) |
|
||||
| RTX 3060 12 GB, 46 GB de RAM | 9 a 11 tokens/s | cerca de 155 tokens/s (4K) |
|
||||
| 1x Radeon AI PRO R9700, 251 GB de RAM | 21 a 25 tokens/s | 320 a 360 tokens/s (8K) |
|
||||
| 4x Radeon AI PRO R9700 | 74 a 77 tokens/s | 400 a 450 tokens/s (8K) |
|
||||
|
||||
[Memória, disco e detalhes](docs/DEEPSEEK41.md) · [Notas da versão](https://github.com/Niko1221/Strata/releases)
|
||||
|
||||
</details>
|
||||
|
||||
<a id="the-other-sizes"></a>
|
||||
|
||||
Não sabe o que cabe? [Escolha um modelo para o seu computador](docs/MODELS.md#pick-by-memory).
|
||||
|
||||
## Instalação
|
||||
|
||||
### 1. O app Strata (o mais fácil)
|
||||
<a id="option-a-the-strata-app"></a>
|
||||
|
||||
Baixe o arquivo para o seu computador, abra e clique em instalar no modelo que ele recomenda.
|
||||
### Aplicativo de desktop
|
||||
|
||||
**Windows 10 / 11**
|
||||
**[Baixar o Strata para Windows, Linux ou Mac →](https://github.com/Niko1221/Strata/releases/latest)**
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-windows-x64-setup.exe)
|
||||
Instale o aplicativo, escolha um modelo e comece a conversar. Há um instalador por sistema. Na primeira vez que abre, o aplicativo baixa o motor da sua placa de vídeo. [Ajuda com o instalador](docs/INSTALL.md#the-app).
|
||||
|
||||
**Linux**
|
||||
<sub>Arquivos: `Strata-0.2.6-windows-x64-setup.exe` · `Strata-0.2.6-linux-x86_64.AppImage` · `Strata-0.2.6-macos-arm64.dmg`</sub>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-linux-x86_64.AppImage)
|
||||
<p align="center">
|
||||
<img src="docs/media/app-chat.png" width="900" alt="Aplicativo de desktop do Strata com um chat e uma prévia do código ao vivo lado a lado">
|
||||
<br><sub>Chat, arquivos e prévia ao vivo em uma só janela.</sub>
|
||||
</p>
|
||||
|
||||
**Mac** (Apple Silicon, macOS 15 ou mais novo)
|
||||
<a id="option-b-the-terminal-one-line"></a>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-macos-arm64.dmg)
|
||||
### Aplicativo de Terminal
|
||||
|
||||
O aplicativo escolhe sozinho o mecanismo certo para a sua placa de vídeo. Todos os arquivos estão na
|
||||
[página de releases](https://github.com/Niko1221/Strata/releases/latest).
|
||||
Prefere o terminal? Instale o Strata e execute **`strata launch`**.
|
||||
|
||||
| Primeira inicialização | O que fazer |
|
||||
| --- | --- |
|
||||
| **Windows** | Execute o `.exe`. Não precisa de direitos de administrador. Se o Windows disser "O Windows protegeu o computador", escolha **Mais informações** e depois **Executar assim mesmo**. |
|
||||
| **Linux** | Torne o arquivo executável (`chmod +x`) e inicie. É preciso ter o FUSE 2: `sudo apt install libfuse2t64` (Ubuntu 24.04 e mais novos) ou `libfuse2` (22.04). |
|
||||
| **Mac** | Abra o `.dmg` e arraste o Strata para Aplicativos. |
|
||||
<details open>
|
||||
<summary><b>Comandos de instalação</b></summary>
|
||||
|
||||
O download do modelo é grande (40 a 70 GB). Se parar, continua de onde parou.
|
||||
|
||||
### 2. O Terminal (uma linha)
|
||||
|
||||
O comando `strata`, para scripts, servidores e agentes de IA. É também o jeito de rodar o DeepSeek.
|
||||
|
||||
Linux e Mac:
|
||||
**Linux / Mac**
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
```
|
||||
|
||||
Windows (PowerShell):
|
||||
**Windows PowerShell**
|
||||
|
||||
```powershell
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
```
|
||||
|
||||
Depois inicie. Ele faz uma pergunta por vez (modelo, contexto, imagens, quem pode se conectar), baixa o que precisa e começa
|
||||
a servir:
|
||||
No Windows, abra um novo terminal depois. Em seguida:
|
||||
|
||||
```sh
|
||||
strata launch
|
||||
```
|
||||
|
||||
Não precisa de direitos de administrador. A instalação confere o SHA-256 de tudo que baixa. No Windows, abra uma nova janela do
|
||||
terminal depois da instalação. Todas as telas e flags: [docs/TERMINAL_LAUNCH.md](docs/TERMINAL_LAUNCH.md).
|
||||
|
||||
### 3. Deixe a sua IA configurar
|
||||
|
||||
Cole isto no Claude Code, Cursor, Codex, GitHub Copilot ou outro assistente de programação:
|
||||
|
||||
```text
|
||||
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
|
||||
```
|
||||
|
||||
Ele confere a sua placa de vídeo, a RAM e o disco, escolhe o modelo que cabe, instala e inicia, e diz como conectar os seus
|
||||
aplicativos. Ferramentas de IA também podem instalar, iniciar e parar o Strata pelo [servidor MCP](docs/MCP_SERVER.md) dele.
|
||||
|
||||
Prefere compilar por conta própria? Veja [Compilar a partir do código-fonte](#compilar-a-partir-do-código-fonte).
|
||||
|
||||
## Modelos
|
||||
|
||||
O app e o `strata launch` recomendam um tamanho para a sua memória. Os tamanhos menores são mais rápidos. Os maiores são um
|
||||
pouco mais inteligentes.
|
||||
|
||||
### Pouca memória? Experimente o Gyro
|
||||
|
||||
O [Gyro](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF), da agentionai, é a menor forma de rodar o
|
||||
Qwen3.8-Flash-Next. Ele roda em NVIDIA, AMD, Intel Arc e Apple Silicon.
|
||||
|
||||
| Tamanho | Memória necessária (RAM + VRAM) | Download | Serve para |
|
||||
| --- | ---: | ---: | --- |
|
||||
| **Gyro-S** | 27,6 GB | 58,5 GB | PCs onde nenhum outro tamanho cabe. 9 tokens/s num Mac mini de 32 GB. |
|
||||
| **Gyro-M** | 35 GB | 92,0 GB | Qualidade próxima à do IQ3_XXS, com 20% menos memória da GPU. |
|
||||
|
||||
Inicie com `strata launch --model gyro-s`. Detalhes: [docs/GYRO.md](docs/GYRO.md).
|
||||
|
||||
### Os tamanhos do Qwen3.8-Flash-Next
|
||||
|
||||
| Sua memória (RAM + VRAM) | Escolha | Por quê |
|
||||
| --- | --- | --- |
|
||||
| **32 GB** | **Gyro-S**, ou o **[Coder](docs/MODELS.md#coder)** para código | O Coder mantém os experts que o código usa |
|
||||
| **40 GB** | **[Swift 1.5](docs/MODELS.md#swift-15)** IQ2_XS (39,2 GB) ou **Gyro-M** | O Swift pensa por menos tempo antes de responder |
|
||||
| **48 GB** | **IQ3_XXS** (47,0 GB) | Mais lento, melhor |
|
||||
| **64 GB** | **IQ3_S** (54,8 GB) | Empata com o modelo completo nos testes publicados |
|
||||
| **96 GB ou mais** | **UD-IQ4_XS** a **Q8_0** da Unsloth | O mais perto do modelo completo |
|
||||
|
||||
Com menos RAM do que a linha indica, ainda pode funcionar, só que mais devagar: o Strata muda sozinho para o
|
||||
[modo de pouca RAM](docs/MODELS.md#a-big-graphics-card-and-little-ram). Há também:
|
||||
[NVFP4](docs/NVFP4.md) para placas NVIDIA, [Qwen3.6-35B-A3B](docs/QWEN36.md) (menor, vários pedidos ao mesmo tempo) e
|
||||
[o Uncensored IQ3_XXS da OrcaRouter](docs/MODELS.md#orcarouter-uncensored-iq3_xxs). Todos os tamanhos e o que cabe em cada configuração:
|
||||
[docs/MODELS.md](docs/MODELS.md).
|
||||
|
||||
### DeepSeek V4.1-Flash (Terminal, a partir da 0.2.4)
|
||||
|
||||
Um modelo bem maior, que roda numa única placa de vídeo de 12 GB com 46 a 64 GB de RAM. O Strata mantém na memória as partes mais
|
||||
usadas e lê o resto do seu SSD. A [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF)
|
||||
fez os conjuntos. Ele roda em placas NVIDIA RTX das séries 30, 40 e 50 e na Radeon AI PRO R9700. Escolha no `strata launch`. O
|
||||
app de desktop ainda não o oferece.
|
||||
|
||||
| Conjunto | Download | Qualidade | Escreve código a |
|
||||
| --- | ---: | --- | --- |
|
||||
| **IQ1_M** (1,75 bits) | 239 GB | O menor, mais fraco em prosa simples | ainda não medido num PC |
|
||||
| **IQ2_XXS** (2.00 bits) | 257 GB | Bom equilíbrio | 10 tokens/s (RTX 3060), 5 (RTX 5070), 48 (4x R9700) |
|
||||
| **IQ2_XS** (2.25 bits) | 274 GB | O mais perto do original | 15 tokens/s (uma R9700), 80 (4x R9700) |
|
||||
| **IQ4_NL** (tabela Engram) | incluída em cada download acima | A tabela de consulta que os três conjuntos compartilham | |
|
||||
|
||||
Inicie com `strata launch --model ist-das-2.00bpw` (ou `ist-das-1.75bpw`, `ist-das-2.25bpw`). Números de qualidade,
|
||||
velocidades por PC e a instalação manual: [docs/DEEPSEEK41.md](docs/DEEPSEEK41.md).
|
||||
|
||||
## Qual é a velocidade?
|
||||
|
||||
Com base numa placa de vídeo de 12 GB (RTX 5070, 64 GB de RAM). Veja resultados de outras placas:
|
||||
[velocidade de cada tamanho](docs/MODELS.md#how-fast-is-each-size) e [resultados da comunidade](docs/COMMUNITY_BENCHMARKS.md).
|
||||
|
||||
| Modelo | As respostas aparecem a | Lê o seu prompt a |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████████` 94 tokens/s | `████████████` 2.650 tokens/s |
|
||||
| **IQ2_XS** | `██████████░░` 79 tokens/s | `█████████░░░` 2.090 tokens/s |
|
||||
| **IQ3_XXS** | `████████░░░░` 62 tokens/s | `████████░░░░` 1.750 tokens/s |
|
||||
| **IQ3_S** | `███████░░░░░` 53 tokens/s | `███████░░░░░` 1.620 tokens/s |
|
||||
| **Coder** | `███████░░░░░` 55 tokens/s | `██████████░░` 2.180 tokens/s |
|
||||
|
||||
Numa placa AMD de 16 GB (RX 9070 XT, 47 GB de RAM):
|
||||
|
||||
| Modelo | As respostas aparecem a | Lê o seu prompt a |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████░░░░` 60 tokens/s | `█████░░░░░░░` 1.160 tokens/s |
|
||||
| **IQ2_XS** | `███████░░░░░` 52 tokens/s | `█████░░░░░░░` 1.110 tokens/s |
|
||||
| **Coder** | `██████░░░░░░` 44 tokens/s | `██████░░░░░░` 1.420 tokens/s |
|
||||
|
||||
Um token equivale a cerca de ¾ de palavra, então 60 tokens por segundo é mais rápido do que você lê. "Lê o seu prompt" é a velocidade com que o Strata
|
||||
absorve um documento longo, um código ou um histórico de conversa.
|
||||
|
||||
## Como usar
|
||||
|
||||
### No app
|
||||
|
||||
Um chat, um **Monitor** do modelo e da sua GPU, CPU e RAM, uma página de **Modelos** e as **Configurações**. O chat pode
|
||||
trabalhar numa pasta do seu PC. No modo **Manual**, ele pergunta antes de alterar um arquivo ou executar um comando. Ele pode
|
||||
pesquisar na web (DuckDuckGo ou Bing, sem conta). Os termos da pesquisa são a única coisa que sai do seu PC.
|
||||
|
||||
<p align="center"><img src="docs/media/app-chat.png" width="900" alt="O app Strata: um chat em que o agente montou um pequeno jardim de pagodes em voxels, com a página aberta na pré-visualização à direita"><br>
|
||||
<sub>O agente escreveu um jardim de pagodes em voxels em um único arquivo HTML, e a página abre na pré-visualização ao lado do chat</sub></p>
|
||||
|
||||
### No Terminal
|
||||
|
||||
<p align="center"><img src="docs/media/terminal-monitor.png" width="900" alt="A visão geral do monitor do Terminal do Strata enquanto o modelo escreve uma resposta"><br>
|
||||
<sub><code>strata monitor</code> enquanto o modelo responde: velocidade, GPU, memória e as solicitações recentes</sub></p>
|
||||
|
||||
```sh
|
||||
strata launch # guided: model, context, pictures, access, then serve
|
||||
strata launch --model q2_0 --yes --no-tui # no questions: serve and print the address
|
||||
strata launch --status # what is being served now
|
||||
strata launch --stop # stop it
|
||||
strata monitor # watch speed, GPU, memory and requests
|
||||
strata update # get the newest engine
|
||||
strata doctor # check the installed files
|
||||
```
|
||||
|
||||
Para acessar o Strata do celular ou de outro PC, use `strata launch --access lan`. Ele cria uma chave de API para você.
|
||||
`strata --help` lista tudo, e o [docs/TERMINAL.md](docs/TERMINAL.md) explica.
|
||||
|
||||
### Com agentes de IA e outros aplicativos
|
||||
|
||||
Um agente instala o Strata com o comando de uma linha acima e executa `strata launch --model NAME --yes --no-tui`. Os passos,
|
||||
as flags e os códigos de saída estão em [docs/AI_SETUP.md](docs/AI_SETUP.md). As regras do repositório para agentes: [AGENTS.md](AGENTS.md).
|
||||
|
||||
| Seu aplicativo fala | URL base |
|
||||
| --- | --- |
|
||||
| API da OpenAI (a maioria dos apps e agentes) | `http://127.0.0.1:8080/v1` (qualquer chave de API, qualquer nome de modelo) |
|
||||
| API da Anthropic (Claude Code) | `http://127.0.0.1:8080`, sem `/v1`. Claude Code: `ANTHROPIC_BASE_URL=http://127.0.0.1:8080` |
|
||||
| OpenAI Responses API (Codex CLI) | `http://127.0.0.1:8080/v1/responses` ([configuração](docs/DETAILS.md#the-responses-api-and-codex-cli)) |
|
||||
|
||||
`strata connect` mostra as configurações exatas para OpenCode, Codex CLI, Claude Code, oh-my-pi, pi, Hermes Agent e OpenClaw,
|
||||
e `strata connect opencode --write` grava para você (com um backup).
|
||||
|
||||
### Bom saber
|
||||
|
||||
- **As imagens** vêm ativadas em todo modelo que sabe lê-las. Anexe ou cole uma imagem no chat, ou envie pelo seu aplicativo. Detalhes: [docs/IMAGES.md](docs/IMAGES.md).
|
||||
- **O raciocínio** pode ser off, low, medium ou high, no menu do chat ou no "reasoning effort" do seu aplicativo. Off é o mais rápido.
|
||||
- **A primeira mensagem** de uma conversa é lida por inteiro, cerca de 1 minuto a cada 30.000 tokens. As mensagens seguintes começam em segundos.
|
||||
- **Um pedido por vez** é o padrão. Para mais de um, use `--parallel N` ([BATCHING.md](docs/BATCHING.md)). Numa placa de 12 GB, isso deixa cada resposta mais lenta.
|
||||
|
||||
## Compilar a partir do código-fonte
|
||||
|
||||
<details>
|
||||
<summary>Windows, Linux e Mac a partir do repositório</summary>
|
||||
|
||||
[Baixe o Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) e descompacte, ou use `git clone`.
|
||||
**Windows:** dê dois cliques em `START-HERE.bat`. **Linux:** execute `./setup.sh` na pasta do Strata.
|
||||
|
||||
**Mac (Apple Silicon, macOS 15 ou mais novo):** instale uma vez as Command Line Tools da Apple com `xcode-select --install`
|
||||
(o Xcode em si não é necessário). Abra o Terminal na pasta do Strata e execute `./setup.sh`. Ele compila o mecanismo do Mac no seu
|
||||
Mac (cerca de 7 minutos), faz as mesmas perguntas, baixa o modelo e inicia. Da próxima vez, `./setup.sh` inicia
|
||||
na hora. Qual tamanho cabe no seu Mac: [docs/MAC.md](docs/MAC.md).
|
||||
|
||||
O instalador encontra a sua placa e configura o mecanismo certo. Ele pergunta qual modelo e tamanho, quanto contexto (quanto texto
|
||||
o modelo mantém em mente) e se deve ler imagens. Pressione Enter para aceitar a resposta recomendada. Depois ele baixa o
|
||||
modelo (cerca de 70 GB) e inicia. Se o download parar, execute de novo e ele continua. O navegador abre a página do Strata em
|
||||
`http://127.0.0.1:8080`.
|
||||
|
||||
> Enquanto o modelo inicia, o seu PC pode ficar lento ou parar de responder por 1 a 3 minutos (mais tempo na primeira vez). O Strata carrega
|
||||
> de 35 a 55 GB na sua RAM e trava parte dela para a placa de vídeo. Espere e não feche a janela.
|
||||
|
||||
Da próxima vez, execute `START-HERE.bat` (ou `./setup.sh`) de novo. Ele inicia na hora e não baixa nada duas vezes. Feche a janela
|
||||
dele para parar o modelo. `UPDATE.bat` (`./update.sh`) atualiza o Strata sem iniciá-lo. Atualização, Docker, várias placas,
|
||||
onde ficam os arquivos e todas as opções: [docs/INSTALL.md](docs/INSTALL.md).
|
||||
|
||||
Para medir o seu PC, execute `BENCH.bat` (`./bench.sh`). Leva de 5 a 10 minutos e gera um relatório que você pode compartilhar
|
||||
([como](docs/COMMUNITY_BENCHMARKS.md#run-the-benchmark)).
|
||||
|
||||
**Opcional: mais velocidade numa placa quando o modelo inteiro cabe na RAM.** Duas configurações opcionais do RubenTorney (#1904, #1905).
|
||||
Elas ajudam quando você usa uma placa de vídeo e a sua RAM comporta todos os experts, por exemplo Q2_0 ou IQ3_XXS num PC de 64 GB. Abra
|
||||
`strata-<model>.json` na pasta de dados do Strata ([onde fica](docs/INSTALL.md#where-things-are-stored)) e adicione:
|
||||
|
||||
- `"--adapt-async", "1"` à lista `"args"`. O cache de experts passa a ser atualizado numa thread auxiliar, em vez de pausar a
|
||||
resposta. Medido numa RTX 5070 (Windows, Q2_0): +4 a 5% na geração, +7 a 10% em código e documentos longos. As respostas podem diferir
|
||||
um pouco do padrão e continuam iguais de uma execução para outra.
|
||||
- `"STRATA_PIN_BY_NEED": "1"` no bloco `"env"` (Windows). O Windows só consegue travar cerca de metade da RAM para a placa
|
||||
de vídeo, e isso trava primeiro os experts de que a placa precisa. Medido na RTX 5070: +4,4% na leitura do prompt.
|
||||
|
||||
```json
|
||||
"args": [ ..., "--adapt-async", "1" ],
|
||||
"env": { "STRATA_PIN_BY_NEED": "1" }
|
||||
```
|
||||
|
||||
Reinicie o Strata depois da alteração. Para desfazer, remova as duas entradas.
|
||||
Siga as instruções para baixar e iniciar um modelo. [Guia do Terminal](docs/TERMINAL_LAUNCH.md).
|
||||
|
||||
</details>
|
||||
|
||||
## Como funciona?
|
||||
<p align="center">
|
||||
<img src="docs/media/terminal-monitor.png" width="900" alt="Aplicativo de Terminal do Strata mostrando a atividade do modelo, a velocidade, a memória e as solicitações recentes">
|
||||
<br><sub>Acompanhe seu modelo e seu hardware enquanto ele roda.</sub>
|
||||
</p>
|
||||
|
||||
Modelos como este costumam rodar em servidores com centenas de gigabytes de memória de vídeo. A sua placa de vídeo tem
|
||||
12 a 24 GB. O Strata faz o modelo caber dividindo o trabalho entre o PC inteiro. Imagine uma biblioteca.
|
||||
<a id="build-from-source"></a>
|
||||
|
||||
- O modelo é uma equipe de 24.576 pequenos especialistas ("experts"), como 24.576 livros de consulta. Cada palavra precisa de apenas 10 deles.
|
||||
- **A sua placa de vídeo é a mesa.** Ela é pequena e rápida. Os cerca de 4.000 livros mais usados ficam abertos nela.
|
||||
- **A sua RAM são as estantes da sala de leitura.** Os 24.576 livros estão lá. Quando uma palavra precisa de um livro que não está na
|
||||
mesa, o seu processador lê da estante na hora, enquanto a placa continua trabalhando.
|
||||
- **O seu SSD é o arquivo.** Ele guarda uma tabela de consulta de 29 GB. O modelo lê algumas linhas dela por palavra, então ela não
|
||||
precisa ficar na RAM.
|
||||
[Compilar a partir do código-fonte](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [Guia de instalação completo](docs/INSTALL.md)
|
||||
|
||||
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Os 24.576 experts do modelo: os mais usados na placa de vídeo, todos na RAM, uma tabela de consulta no SSD"></p>
|
||||
<a id="what-you-need"></a>
|
||||
|
||||
Um pequeno ajudante também adivinha as próximas palavras. O modelo grande confere todas de uma vez e fica com as certas. Você recebe
|
||||
a mesma resposta, de 1,6 a 1,8 vez mais rápido. Textos longos são lidos em blocos grandes, de até 8.192 tokens por vez, a mais de 1.000 tokens
|
||||
por segundo.
|
||||
## Hardware
|
||||
|
||||
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Um pequeno ajudante adivinha as próximas palavras; o modelo grande confere todas de uma vez e fica com as certas"></p>
|
||||
| Hardware compatível | Guia |
|
||||
| --- | --- |
|
||||
| **NVIDIA GeForce RTX 20 / 30 / 40 / 50** | [Instalação para NVIDIA](docs/INSTALL.md#what-you-need) |
|
||||
| **AMD Radeon RX 7700 / 7800 / 7900, RX 9060 / 9070, AI PRO R9700** | [Instalação para AMD](docs/AMD_HIP.md) |
|
||||
| **AMD Radeon RX 6800 / 6900 · experimental** | [Instalação para RX 6000](docs/AMD_HIP.md#rdna2-gfx1030) |
|
||||
| **Intel Arc · Linux** | [Instalação para Intel](docs/INTEL_ARC.md) |
|
||||
| **Apple Silicon · M1 ou mais novo** | [Instalação para Mac](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max · Strix Halo** | [Instalação para Strix Halo](docs/STRIX_HALO.md) |
|
||||
|
||||
A explicação mais longa: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Cada parte e seus números:
|
||||
[os detalhes](docs/DETAILS.md#how-it-works) e o [artigo](docs/paper/Strata-Paper.pdf).
|
||||
<a id="using-it"></a>
|
||||
|
||||
## Algo deu errado?
|
||||
## Ferramentas
|
||||
|
||||
- **Meu PC travou na primeira vez que o Strata iniciou.** Ele carrega o modelo na RAM. Espere e não feche a janela. Ainda
|
||||
travado depois de 10 minutos? Reinicie o PC, feche outros programas e tente de novo, ou escolha um tamanho menor.
|
||||
- **Parou durante o download ou a instalação.** Abra o app ou o `strata launch` de novo. Ele continua de onde parou.
|
||||
- **Está muito lento, ou aparece "the engine stopped unexpectedly".** O seu PC não tem RAM livre suficiente. Feche outros
|
||||
programas (os navegadores usam muita), ou escolha um tamanho menor, como Q2_0 ou IQ2_XS.
|
||||
- **Diz que não há mecanismo para este PC.** No app, abra **Configurações, Mecanismo** e clique em **Atualizar mecanismo**. No
|
||||
Terminal, execute `strata update`.
|
||||
- **Diz que a porta 8080 já está em uso.** O Strata já está rodando. Procure a janela dele.
|
||||
**Claude Code · Codex CLI · OpenCode · oh-my-pi · pi.dev · Hermes Agent · OpenClaw**, e outras, pelas APIs compatíveis com OpenAI e Anthropic.
|
||||
|
||||
Mais soluções: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Ainda sem resolver? Abra uma
|
||||
[issue](https://github.com/Niko1221/Strata/issues) e anexe os logs: no app, em **Configurações, Mecanismo, Diagnóstico,
|
||||
Salvar logs**; no Terminal, a saída de `strata doctor`. Para um problema de segurança, informe em particular: [SECURITY.md](SECURITY.md).
|
||||
Execute **`strata connect`** para ver as instruções da sua ferramenta. [Guia de conexão](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
|
||||
|
||||
## Créditos e licença
|
||||
<a id="guides"></a>
|
||||
<a id="how-fast-is-it"></a>
|
||||
<a id="how-does-it-work"></a>
|
||||
<a id="something-went-wrong"></a>
|
||||
|
||||
O modelo é o [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), da equipe Qwen. Ele foi comprimido por
|
||||
[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5), agentionai (Gyro)
|
||||
e Unsloth. A ISTA-DASLab também fez os conjuntos do DeepSeek V4.1-Flash. O Strata usa partes do
|
||||
[llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Todos os créditos: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits).
|
||||
O Strata é de código aberto sob a [Licença MIT](LICENSE). Algumas partes e todos os modelos têm licenças próprias
|
||||
([quais](docs/HOW_IT_WORKS.md#license)).
|
||||
## Ajuda
|
||||
|
||||
## Contribuindo
|
||||
|
||||
Encontrou um bug, ou quer enviar uma alteração ou um benchmark? Leia primeiro o [CONTRIBUTING.md](CONTRIBUTING.md). Ele diz o que
|
||||
incluir para que outras pessoas possam conferir o seu trabalho.
|
||||
[Todos os guias](docs) · [Solução de problemas](docs/TROUBLESHOOTING.md) · [Relatar um bug](https://github.com/Niko1221/Strata/issues/new/choose) ·
|
||||
[Benchmarks](docs/COMMUNITY_BENCHMARKS.md) · [Como funciona](docs/HOW_IT_WORKS.md) · [Contribuir](CONTRIBUTING.md)
|
||||
|
||||
## Apoie o Strata
|
||||
|
||||
O Strata é gratuito e tem licença MIT. Eu o desenvolvo e testo em todo hardware que consigo arranjar. Se ele roda bem no seu PC, um café me ajuda
|
||||
a continuar testando placas novas e corrigindo o que você relatar. Sem pressão. Usar e relatar o que encontrar ajuda tanto quanto.
|
||||
**Ajude a manter a IA local aberta para todos.** Um café ajuda a financiar o desenvolvimento e os testes em mais hardware.
|
||||
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: apoie o Strata" height="48"></a>
|
||||
|
||||
Quer patrocinar o Strata? Fale comigo pelo [GitHub](https://github.com/Niko1221).
|
||||
[Licença MIT](LICENSE) · [Créditos e licenças dos modelos](docs/HOW_IT_WORKS.md#credits) · [Privacidade](docs/SAFETY.md)
|
||||
|
||||
<a id="option-c-let-your-ai-set-it-up"></a>
|
||||
|
||||
<sub>Para ferramentas de IA e leitores automáticos: <a href="AGENTS.md">referência técnica</a> · <a href="docs/AI_SETUP.md">instruções de instalação</a>.</sub>
|
||||
|
||||
**Patrocínios, parcerias de hardware ou colaborações:** [[email protected]](mailto:[email protected]).
|
||||
+122
-278
@@ -1,344 +1,188 @@
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="apps/host/public/brand/strata-lockup-dark.svg">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="380">
|
||||
<img src="apps/host/public/brand/strata-lockup-light.svg" alt="Strata" width="340">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="README.md">English</a> · <b>简体中文</b> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.fr.md">Français</a> · <a href="README.es.md">Español</a> · <a href="README.pt-BR.md">Português</a></p>
|
||||
|
||||
<h3 align="center">在你自己的游戏电脑上运行最先进的模型</h3>
|
||||
|
||||
<p align="center">支持 NVIDIA、AMD、Intel · Windows、Linux、Mac · 免费开源</p>
|
||||
<h3 align="center">在你自己的电脑上运行强大的 AI。</h3>
|
||||
<p align="center">本地聊天、写代码、处理图片。</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?include_prereleases&label=release" alt="最新版本"></a>
|
||||
<a href="#模型">模型</a> · <a href="#安装"><b>安装</b></a> ·
|
||||
<a href="#硬件">硬件</a> · <a href="#工具">工具</a> ·
|
||||
<a href="#帮助">帮助</a> · <a href="#支持-strata">支持</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="下载 Strata"></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="MIT 许可证"></a>
|
||||
<a href="https://github.com/Niko1221/Strata/stargazers"><img src="https://img.shields.io/github/stars/Niko1221/Strata?style=flat" alt="GitHub 星标"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black" alt="请我喝杯咖啡"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="在 Buy Me a Coffee 上支持 Strata"></a>
|
||||
</p>
|
||||
|
||||
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Strata 的模型写出的体素宝塔花园，在浏览器里运行"></a><br>
|
||||
<sub>一座体素宝塔花园，由一条提示词一次生成，在装有 RTX 5070 的电脑上用 Strata 运行（IQ3_S，128K 上下文） ·
|
||||
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">完整视频（49 秒）</a></sub></p>
|
||||
<p align="center"><sub><a href="README.md">English</a> · <b>简体中文</b> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.fr.md">Français</a> · <a href="README.es.md">Español</a> · <a href="README.pt-BR.md">Português</a></sub></p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#安装"><b>安装</b></a> ·
|
||||
<a href="#模型">模型</a> ·
|
||||
<a href="#速度如何">速度</a> ·
|
||||
<a href="#使用方法">使用方法</a> ·
|
||||
<a href="#工作原理">工作原理</a> ·
|
||||
<a href="#出问题了">帮助</a> ·
|
||||
<a href="#支持-strata">支持</a>
|
||||
</p>
|
||||
**免费、开源，由你自己运行。** 在桌面上使用 Strata，或搭配你喜欢的编程工具。无需账号。
|
||||
|
||||
Strata 运行 **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)**。这是一个 1250 亿参数的模型，
|
||||
通常需要服务器才能跑，现在用普通电脑就行。从 0.2.4 起，它在终端版里也能运行 **DeepSeek V4.1-Flash**。它能聊天、写代码、
|
||||
看图片，也能配合你的应用和编程智能体使用。你的聊天记录和文件都留在你自己的电脑上。
|
||||
## 模型
|
||||
|
||||
- **游戏电脑就能跑。** 内存和显存加起来有 32 GB 就可以开始。
|
||||
- **能接入你的工具。** Claude Code、Codex CLI、OpenCode 等智能体通过 OpenAI 和 Anthropic 的 API 连接。
|
||||
- **能看图片**，默认开启。
|
||||
- **免费且隐私。** MIT 许可证，不用注册账号，不走云端，没有遥测（[它下载什么、什么时候下载](docs/SAFETY.md)）。
|
||||
我们**首先推荐 ISTA-DASLab**，其次是 **Unsloth** 和其他提供方，前提是有内存放得下的兼容模型。
|
||||
|
||||
## 硬件要求
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.8-Flash-Next</b></summary>
|
||||
|
||||
**内存和显存（VRAM）加起来至少要有 32 GB。** Mac 和 Ryzen AI Max 上指的是统一内存。32 GB 内存加一张 12-16 GB 显存的显卡，
|
||||
可以跑较小的尺寸；64 GB 可以跑所有尺寸。
|
||||
|
||||
| 适用于 | 指南 |
|
||||
| 提供方 | 可选项 |
|
||||
| --- | --- |
|
||||
| **NVIDIA** GeForce RTX 20、30、40 和 50 系列，12 GB 及以上 | [安装](#安装) |
|
||||
| **AMD** Radeon RX 6800 及更新型号、Radeon AI PRO R9700 | [AMD 显卡](docs/AMD_HIP.md) |
|
||||
| **Intel** Arc | [Intel Arc](docs/INTEL_ARC.md) |
|
||||
| **Apple 芯片 Mac**，M1 及更新，macOS 15 及以上 | [Mac](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max（Strix Halo）** 及其他统一内存的电脑 | [Strix Halo](docs/STRIX_HALO.md) |
|
||||
| 较旧的显卡和处理器 | [旧显卡](docs/OLDER_GPUS.md) |
|
||||
| **ISTA-DASLab · 推荐** | Q2\_0, IQ2\_XS, IQ3\_XXS, IQ3\_S。[Coder：IQ1\_M](docs/MODELS.md#coder)，适合写代码。 |
|
||||
| **Unsloth** | UD-IQ4\_XS, UD-Q4\_K\_XL, UD-Q5\_K\_XL, UD-Q6\_K\_XL, Q8\_0 |
|
||||
| **Gyro · agentionai** | [Gyro-S, Gyro-M](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)。占用内存更少，但速度更慢。 |
|
||||
| **其他选项** | [Swift 1.5](docs/MODELS.md#swift-15), [NVFP4](docs/NVFP4.md), [OrcaRouter](docs/ORCA.md) |
|
||||
|
||||
你还需要约 80 GB 的可用磁盘空间（固态硬盘）、Windows 10 / 11、Linux 或 macOS，以及较新的显卡驱动。同一品牌的两三张显卡可以共同运行一个模型（[多显卡](docs/MULTI_GPU.md)）。
|
||||
[比较大小、内存需求和兼容性](docs/MODELS.md)。
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/qwen.svg" width="22" alt="" align="middle"> <b>Qwen3.6-35B-A3B</b></summary>
|
||||
|
||||
**Unsloth:** UD-Q4_K_XL, UD-Q6_K_XL, Q8_0。这些是目前文档中收录的该模型的版本。
|
||||
|
||||
[模型指南和兼容性](docs/QWEN36.md)。
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><img src="docs/media/models/deepseek.svg" width="22" alt="" align="middle"> <b>DeepSeekV4.1-Flash-1017</b></summary>
|
||||
|
||||
可在**桌面应用**和**终端**中使用。内存放不下的部分会从 SSD 读取，所以需要大量空闲磁盘空间。
|
||||
|
||||
| 版本（IST-DAS） | 下载大小 | 所需空闲磁盘 |
|
||||
| --- | ---: | ---: |
|
||||
| 1.75 bpw | 239 GB | 约245 GB |
|
||||
| 2.00 bpw | 257 GB | 约265 GB |
|
||||
| 2.25 bpw | 274 GB | 约280 GB |
|
||||
|
||||
支持 NVIDIA RTX 30、40 和 50 系列（Windows 和 Linux），以及 AMD Radeon AI PRO R9700（Linux）。安装程序会下载模型文件和 DeepSeek 引擎，然后启动模型。
|
||||
|
||||
```sh
|
||||
strata launch --family deepseek41 --model ist-das-2.00bpw --plan
|
||||
```
|
||||
|
||||
以下为 Strata 0.2.5 上 2.00 bpw 版本的实测结果。“输出”是回答的速度，“提示词”是 Strata 读取你输入内容的速度。
|
||||
|
||||
| 硬件 | 输出 | 提示词 |
|
||||
| --- | ---: | ---: |
|
||||
| RTX 5070 12 GB, 64 GB 内存 | 4 到 6 tokens/s | 约 110 tokens/s（4K） |
|
||||
| RTX 3060 12 GB, 46 GB 内存 | 9 到 11 tokens/s | 约 155 tokens/s（4K） |
|
||||
| 1x Radeon AI PRO R9700, 251 GB 内存 | 21 到 25 tokens/s | 320 到 360 tokens/s (8K) |
|
||||
| 4x Radeon AI PRO R9700 | 74 到 77 tokens/s | 400 到 450 tokens/s (8K) |
|
||||
|
||||
[内存、磁盘和详细信息](docs/DEEPSEEK41.md) · [发行说明](https://github.com/Niko1221/Strata/releases)
|
||||
|
||||
</details>
|
||||
|
||||
<a id="the-other-sizes"></a>
|
||||
|
||||
不确定哪个放得下？[为你的电脑选择模型](docs/MODELS.md#pick-by-memory)。
|
||||
|
||||
## 安装
|
||||
|
||||
### 1. Strata 应用（最简单）
|
||||
<a id="option-a-the-strata-app"></a>
|
||||
|
||||
下载适合你电脑的文件，打开它，然后对推荐的模型点击安装。
|
||||
### 桌面应用
|
||||
|
||||
**Windows 10 / 11**
|
||||
**[下载适用于 Windows、Linux 或 Mac 的 Strata →](https://github.com/Niko1221/Strata/releases/latest)**
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-windows-x64-setup.exe)
|
||||
安装应用，选择模型，就可以开始聊天。每个系统只有一个安装程序。首次启动时，应用会下载适合你显卡的引擎。[安装程序帮助](docs/INSTALL.md#the-app)。
|
||||
|
||||
**Linux**
|
||||
<sub>文件: `Strata-0.2.6-windows-x64-setup.exe` · `Strata-0.2.6-linux-x86_64.AppImage` · `Strata-0.2.6-macos-arm64.dmg`</sub>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-linux-x86_64.AppImage)
|
||||
<p align="center">
|
||||
<img src="docs/media/app-chat.png" width="900" alt="Strata 桌面应用，聊天和实时代码预览并排显示">
|
||||
<br><sub>聊天、文件和实时预览都在一个窗口里。</sub>
|
||||
</p>
|
||||
|
||||
**Mac**（Apple 芯片，macOS 15 及以上）
|
||||
<a id="option-b-the-terminal-one-line"></a>
|
||||
|
||||
[](https://github.com/Niko1221/Strata/releases/latest/download/Strata-0.2.5-macos-arm64.dmg)
|
||||
### 终端应用
|
||||
|
||||
应用会自己为你的显卡选择合适的引擎。所有文件都在[发布页](https://github.com/Niko1221/Strata/releases/latest)。
|
||||
更喜欢终端？安装 Strata，然后运行 **`strata launch`**。
|
||||
|
||||
| 首次启动 | 怎么做 |
|
||||
| --- | --- |
|
||||
| **Windows** | 运行 `.exe`。不需要管理员权限。如果 Windows 提示“Windows 已保护你的电脑”，点击**更多信息**，再点**仍要运行**。 |
|
||||
| **Linux** | 给文件加上可执行权限（`chmod +x`），然后启动。需要 FUSE 2：`sudo apt install libfuse2t64`（Ubuntu 24.04 及更新）或 `libfuse2`（22.04）。 |
|
||||
| **Mac** | 打开 `.dmg`，把 Strata 拖到“应用程序”。 |
|
||||
<details open>
|
||||
<summary><b>安装命令</b></summary>
|
||||
|
||||
模型下载很大（40-70 GB）。如果中断，会从断点继续。
|
||||
|
||||
### 2. 终端版（一行命令）
|
||||
|
||||
`strata` 命令，适合脚本、服务器和 AI 智能体。运行 DeepSeek 也要用它。
|
||||
|
||||
Linux 和 Mac：
|
||||
**Linux / Mac**
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
```
|
||||
|
||||
Windows（PowerShell）：
|
||||
**Windows PowerShell**
|
||||
|
||||
```powershell
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
```
|
||||
|
||||
然后启动它。它会一次问一个问题（模型、上下文、图片、允许谁连接），下载需要的东西，并开始提供服务：
|
||||
在 Windows 上，安装后请打开一个新的终端。然后运行：
|
||||
|
||||
```sh
|
||||
strata launch
|
||||
```
|
||||
|
||||
不需要管理员权限。安装程序会校验它下载的所有内容的 SHA-256。在 Windows 上，装完后请打开一个新的终端窗口。
|
||||
所有界面和参数见 [docs/TERMINAL_LAUNCH.md](docs/TERMINAL_LAUNCH.md)。
|
||||
|
||||
### 3. 让你的 AI 来安装
|
||||
|
||||
把下面这段话粘贴到 Claude Code、Cursor、Codex、GitHub Copilot 或其他编程助手里：
|
||||
|
||||
```text
|
||||
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
|
||||
```
|
||||
|
||||
它会检查你的显卡、内存和磁盘，选出合适的模型，完成安装和启动，并告诉你怎么连接你的应用。AI 工具也可以通过 Strata 的
|
||||
[MCP 服务器](docs/MCP_SERVER.md)安装、启动和停止 Strata。
|
||||
|
||||
想自己动手编译？请看[从源码构建](#从源码构建)。
|
||||
|
||||
## 模型
|
||||
|
||||
应用和 `strata launch` 会按你的内存推荐尺寸。尺寸小的更快，尺寸大的稍微更聪明一点。
|
||||
|
||||
### 内存不够？试试 Gyro
|
||||
|
||||
[Gyro](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)（作者 agentionai）是运行 Qwen3.8-Flash-Next 占用最小的方式。
|
||||
它可在 NVIDIA、AMD、Intel Arc 和 Apple 芯片上运行。
|
||||
|
||||
| 尺寸 | 所需内存（RAM + VRAM） | 下载大小 | 适合 |
|
||||
| --- | ---: | ---: | --- |
|
||||
| **Gyro-S** | 27.6 GB | 58.5 GB | 其他尺寸都放不下的电脑。32 GB 的 Mac mini 上每秒 9 个 token。 |
|
||||
| **Gyro-M** | 35 GB | 92.0 GB | 质量约相当于 IQ3_XXS，显存占用少 20%。 |
|
||||
|
||||
用 `strata launch --model gyro-s` 启动。详情：[docs/GYRO.md](docs/GYRO.md)。
|
||||
|
||||
### Qwen3.8-Flash-Next 的尺寸
|
||||
|
||||
| 你的内存（RAM + VRAM） | 选择 | 原因 |
|
||||
| --- | --- | --- |
|
||||
| **32 GB** | **Gyro-S**，写代码可选 **[Coder](docs/MODELS.md#coder)** | Coder 保留代码用到的专家 |
|
||||
| **40 GB** | **[Swift 1.5](docs/MODELS.md#swift-15)** IQ2_XS（39.2 GB）或 **Gyro-M** | Swift 回答前思考的时间更短 |
|
||||
| **48 GB** | **IQ3_XXS**（47.0 GB） | 更慢，更好 |
|
||||
| **64 GB** | **IQ3_S**（54.8 GB） | 在公开测试中与完整模型持平 |
|
||||
| **96 GB 及以上** | Unsloth 的 **UD-IQ4_XS** 到 **Q8_0** | 最接近完整模型 |
|
||||
|
||||
内存比这一行写的少，也可能能用，只是更慢：Strata 会自动切换到[低内存模式](docs/MODELS.md#a-big-graphics-card-and-little-ram)。
|
||||
另外还有：给 NVIDIA 显卡用的 [NVFP4](docs/NVFP4.md)、[Qwen3.6-35B-A3B](docs/QWEN36.md)（更小，能同时处理多个请求），
|
||||
以及 [OrcaRouter 的 Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs)。所有尺寸，以及各种配置能跑什么：
|
||||
[docs/MODELS.md](docs/MODELS.md)。
|
||||
|
||||
### DeepSeek V4.1-Flash（终端版，0.2.4 起）
|
||||
|
||||
一个大得多的模型，用一块 12 GB 显卡加 46 到 64 GB 内存就能运行。Strata 把最常用的部分留在内存里，其余的从你的固态硬盘读取。
|
||||
这些版本由 [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF) 制作。它可在
|
||||
NVIDIA RTX 30、40、50 系列显卡和 Radeon AI PRO R9700 上运行。请在 `strata launch` 里选择它。桌面应用暂时不提供。
|
||||
|
||||
| 版本 | 下载大小 | 质量 | 写代码速度 |
|
||||
| --- | ---: | --- | --- |
|
||||
| **IQ1_M**（1.75 位） | 239 GB | 最小，普通文字表现较弱 | 还没在电脑上测量 |
|
||||
| **IQ2_XXS**（2.00 位） | 257 GB | 平衡得不错 | 每秒 10 个 token（RTX 3060），5（RTX 5070），48（4 块 R9700） |
|
||||
| **IQ2_XS**（2.25 位） | 274 GB | 最接近原模型 | 每秒 15 个 token（一块 R9700），80（4 块 R9700） |
|
||||
| **IQ4_NL**（Engram 表） | 已包含在上面每个下载中 | 三个版本共用的查找表 | |
|
||||
|
||||
用 `strata launch --model ist-das-2.00bpw`（或 `ist-das-1.75bpw`、`ist-das-2.25bpw`）启动。质量数据、各台电脑的速度和手动安装：
|
||||
[docs/DEEPSEEK41.md](docs/DEEPSEEK41.md)。
|
||||
|
||||
## 速度如何？
|
||||
|
||||
以下数据基于 12 GB 显卡（RTX 5070、64 GB 内存）。其他显卡的结果见：
|
||||
[各尺寸的速度](docs/MODELS.md#how-fast-is-each-size)和[社区测试结果](docs/COMMUNITY_BENCHMARKS.md)。
|
||||
|
||||
| 模型 | 回答输出速度 | 读取提示词速度 |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████████` 94 tokens/s | `████████████` 2,650 tokens/s |
|
||||
| **IQ2_XS** | `██████████░░` 79 tokens/s | `█████████░░░` 2,090 tokens/s |
|
||||
| **IQ3_XXS** | `████████░░░░` 62 tokens/s | `████████░░░░` 1,750 tokens/s |
|
||||
| **IQ3_S** | `███████░░░░░` 53 tokens/s | `███████░░░░░` 1,620 tokens/s |
|
||||
| **Coder** | `███████░░░░░` 55 tokens/s | `██████████░░` 2,180 tokens/s |
|
||||
|
||||
在 16 GB 的 AMD 显卡上（RX 9070 XT、47 GB 内存）：
|
||||
|
||||
| 模型 | 回答输出速度 | 读取提示词速度 |
|
||||
| --- | --- | --- |
|
||||
| **Q2_0** | `████████░░░░` 60 tokens/s | `█████░░░░░░░` 1,160 tokens/s |
|
||||
| **IQ2_XS** | `███████░░░░░` 52 tokens/s | `█████░░░░░░░` 1,110 tokens/s |
|
||||
| **Coder** | `██████░░░░░░` 44 tokens/s | `██████░░░░░░` 1,420 tokens/s |
|
||||
|
||||
一个 token 约等于 ¾ 个英文单词，所以每秒 60 个 token 比你阅读的速度还快。“读取提示词”指 Strata 吸收长文档、代码或聊天历史的速度。
|
||||
|
||||
## 使用方法
|
||||
|
||||
### 在应用里
|
||||
|
||||
聊天、显示模型以及 GPU、CPU 和内存状态的**监控**、**模型**页面和**设置**。聊天可以在你电脑上的某个文件夹里工作。
|
||||
在**手动**模式下，修改文件或运行命令之前它会先询问你。它可以搜索网页（DuckDuckGo 或 Bing，不需要账号）。
|
||||
发出电脑的只有搜索词。
|
||||
|
||||
<p align="center"><img src="docs/media/app-chat.png" width="900" alt="Strata 应用：一次对话，智能体做出了一个小型体素宝塔花园，页面在右侧的预览中打开"><br>
|
||||
<sub>智能体用一个 HTML 文件写出了体素宝塔花园，页面在聊天旁的预览中打开</sub></p>
|
||||
|
||||
### 在终端里
|
||||
|
||||
<p align="center"><img src="docs/media/terminal-monitor.png" width="900" alt="模型写回复时，Strata 终端监控的总览界面"><br>
|
||||
<sub>模型回复时的 <code>strata monitor</code>：速度、GPU、内存和最近的请求</sub></p>
|
||||
|
||||
```sh
|
||||
strata launch # guided: model, context, pictures, access, then serve
|
||||
strata launch --model q2_0 --yes --no-tui # no questions: serve and print the address
|
||||
strata launch --status # what is being served now
|
||||
strata launch --stop # stop it
|
||||
strata monitor # watch speed, GPU, memory and requests
|
||||
strata update # get the newest engine
|
||||
strata doctor # check the installed files
|
||||
```
|
||||
|
||||
要从手机或另一台电脑访问 Strata，请使用 `strata launch --access lan`。它会为你生成一个 API 密钥。
|
||||
`strata --help` 会列出所有命令，[docs/TERMINAL.md](docs/TERMINAL.md) 有详细说明。
|
||||
|
||||
### 配合 AI 智能体和其他应用
|
||||
|
||||
智能体用上面的一行命令安装 Strata，然后运行 `strata launch --model NAME --yes --no-tui`。步骤、参数和退出码见
|
||||
[docs/AI_SETUP.md](docs/AI_SETUP.md)。给智能体看的仓库规则：[AGENTS.md](AGENTS.md)。
|
||||
|
||||
| 你的应用使用 | 基础 URL |
|
||||
| --- | --- |
|
||||
| OpenAI API（大多数应用和智能体） | `http://127.0.0.1:8080/v1`（API 密钥和模型名称随便填） |
|
||||
| Anthropic API（Claude Code） | `http://127.0.0.1:8080`，不带 `/v1`。Claude Code：`ANTHROPIC_BASE_URL=http://127.0.0.1:8080` |
|
||||
| OpenAI Responses API（Codex CLI） | `http://127.0.0.1:8080/v1/responses`（[设置方法](docs/DETAILS.md#the-responses-api-and-codex-cli)） |
|
||||
|
||||
`strata connect` 会打印 OpenCode、Codex CLI、Claude Code、oh-my-pi、pi、Hermes Agent 和 OpenClaw 的具体设置，
|
||||
`strata connect opencode --write` 会帮你写入（并保留备份）。
|
||||
|
||||
### 使用须知
|
||||
|
||||
- **图片**对每个能看图片的模型都是默认开启的。在聊天里附加或粘贴图片，或者从你的应用发送图片。详情：[docs/IMAGES.md](docs/IMAGES.md)。
|
||||
- **思考级别**可以是 off、low、medium 或 high，在聊天菜单或你的应用的“reasoning effort”里设置。off 最快。
|
||||
- **聊天的第一条消息**会被完整读取，每 30,000 个 token 约需 1 分钟。之后的消息几秒钟就开始。
|
||||
- **默认一次处理一个请求。** 要处理更多，请使用 `--parallel N`（[BATCHING.md](docs/BATCHING.md)）。在 12 GB 显卡上，这会让每个回答变慢。
|
||||
|
||||
## 从源码构建
|
||||
|
||||
<details>
|
||||
<summary>从仓库构建 Windows、Linux 和 Mac 版本</summary>
|
||||
|
||||
[下载 Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip)并解压，或者用 `git clone` 获取。
|
||||
**Windows：** 双击 `START-HERE.bat`。**Linux：** 在 Strata 文件夹里运行 `./setup.sh`。
|
||||
|
||||
**Mac（Apple 芯片，macOS 15 及以上）：** 先用 `xcode-select --install` 安装一次 Apple 的命令行工具（不需要完整的 Xcode）。
|
||||
在 Strata 文件夹里打开终端，运行 `./setup.sh`。它会在你的 Mac 上构建 Mac 引擎（约 7 分钟），问同样的问题，下载模型并启动。
|
||||
下次运行 `./setup.sh` 会直接启动。你的 Mac 适合哪个尺寸：[docs/MAC.md](docs/MAC.md)。
|
||||
|
||||
安装程序会找到你的显卡，并设置合适的引擎。它会问你用哪个模型和尺寸、上下文多长（模型能记住多少文字），以及是否要读取图片。
|
||||
直接按回车就是推荐的选择。然后它会下载模型（约 70 GB）并启动。如果下载中断，再运行一次就会继续。浏览器会打开
|
||||
`http://127.0.0.1:8080` 上的 Strata 页面。
|
||||
|
||||
> 模型启动期间，你的电脑可能会变慢，或者在 1-3 分钟内没有响应（第一次最久）。Strata 会把 35-55 GB 加载到内存里，并锁定其中一部分
|
||||
> 给显卡使用。请等待，不要关闭窗口。
|
||||
|
||||
下次再运行 `START-HERE.bat`（或 `./setup.sh`）会立即启动，不会重复下载。关闭它的窗口就能停止模型。`UPDATE.bat`
|
||||
（`./update.sh`）只更新 Strata，不启动。更新、Docker、多显卡、文件存放位置和所有选项：[docs/INSTALL.md](docs/INSTALL.md)。
|
||||
|
||||
要测试你的电脑，运行 `BENCH.bat`（`./bench.sh`）。需要 5-10 分钟，并生成一份可以分享的报告
|
||||
（[方法](docs/COMMUNITY_BENCHMARKS.md#run-the-benchmark)）。
|
||||
|
||||
**可选：整个模型都放得进内存时，让单卡更快。** 这是 RubenTorney 提供的两个需要手动开启的设置（#1904、#1905）。
|
||||
当你只用一张显卡，并且内存装得下所有专家时，它们会有帮助，例如在 64 GB 的电脑上跑 Q2_0 或 IQ3_XXS。打开 Strata 数据文件夹里的
|
||||
`strata-<model>.json`（[位置](docs/INSTALL.md#where-things-are-stored)），然后添加：
|
||||
|
||||
- 在 `"args"` 列表中加入 `"--adapt-async", "1"`。这样专家缓存会在辅助线程上更新，不会让回答暂停。在 RTX 5070
|
||||
（Windows，Q2_0）上测得：解码快 4-5%，代码和长文档快 7-10%。回答可能与默认设置略有不同，但每次运行的结果保持一致。
|
||||
- 在 `"env"` 块中加入 `"STRATA_PIN_BY_NEED": "1"`（Windows）。Windows 只能为显卡锁定大约一半的内存，
|
||||
这个设置会优先锁定显卡最先需要的专家。在 RTX 5070 上测得：读取提示词快 4.4%。
|
||||
|
||||
```json
|
||||
"args": [ ..., "--adapt-async", "1" ],
|
||||
"env": { "STRATA_PIN_BY_NEED": "1" }
|
||||
```
|
||||
|
||||
修改后请重启 Strata。要撤销，删除这两项即可。
|
||||
按提示下载并启动模型。[终端指南](docs/TERMINAL_LAUNCH.md)。
|
||||
|
||||
</details>
|
||||
|
||||
## 工作原理
|
||||
<p align="center">
|
||||
<img src="docs/media/terminal-monitor.png" width="900" alt="Strata 终端应用，显示模型活动、速度、内存和最近的请求">
|
||||
<br><sub>模型运行时，随时查看模型和硬件状态。</sub>
|
||||
</p>
|
||||
|
||||
这类模型通常运行在有几百 GB 显存的服务器上。你的显卡只有 12-24 GB。Strata 把工作分给整台电脑，让模型装得下。
|
||||
可以把它想成一座图书馆。
|
||||
<a id="build-from-source"></a>
|
||||
|
||||
- 模型是一个由 24,576 位小专家（“experts”）组成的团队，好比 24,576 本参考书。每个词只需要其中 10 本。
|
||||
- **你的显卡是书桌。** 它小而快。最常用的大约 4,000 本书摊开放在上面。
|
||||
- **你的内存是阅览室的书架。** 24,576 本书都在这里。某个词需要一本不在书桌上的书时，处理器会立刻从书架上取来读，显卡则继续工作。
|
||||
- **你的固态硬盘是档案室。** 里面存着一张 29 GB 的查找表。模型每个词只读其中几行，所以不用常驻内存。
|
||||
[从源码构建](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [完整安装指南](docs/INSTALL.md)
|
||||
|
||||
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="模型的 24,576 位专家：最忙的在显卡上，全部在内存里，查找表在固态硬盘上"></p>
|
||||
<a id="what-you-need"></a>
|
||||
|
||||
另外有一个小助手会猜接下来的几个词。大模型一次检查所有猜测，保留正确的部分。得到的答案不变，但快了 1.6-1.8 倍。
|
||||
长文本按大块读取，一次最多 8,192 个 token，速度超过每秒 1,000 个 token。
|
||||
## 硬件
|
||||
|
||||
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="小助手猜接下来的几个词；大模型一次检查全部猜测，保留正确的部分"></p>
|
||||
| 支持的硬件 | 指南 |
|
||||
| --- | --- |
|
||||
| **NVIDIA GeForce RTX 20 / 30 / 40 / 50 系列** | [NVIDIA 安装](docs/INSTALL.md#what-you-need) |
|
||||
| **AMD Radeon RX 7700 / 7800 / 7900、RX 9060 / 9070、AI PRO R9700** | [AMD 安装](docs/AMD_HIP.md) |
|
||||
| **AMD Radeon RX 6800 / 6900 · 实验性** | [RX 6000 安装](docs/AMD_HIP.md#rdna2-gfx1030) |
|
||||
| **Intel Arc · Linux** | [Intel 安装](docs/INTEL_ARC.md) |
|
||||
| **Apple Silicon · M1 或更新** | [Mac 安装](docs/MAC.md) |
|
||||
| **AMD Ryzen AI Max · Strix Halo** | [Strix Halo 安装](docs/STRIX_HALO.md) |
|
||||
|
||||
更详细的解释：[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md)。每个部分和它的数据：
|
||||
[详细说明](docs/DETAILS.md#how-it-works)和[论文](docs/paper/Strata-Paper.pdf)。
|
||||
<a id="using-it"></a>
|
||||
|
||||
## 出问题了？
|
||||
## 工具
|
||||
|
||||
- **第一次启动 Strata 时我的电脑卡住了。** 它在把模型加载到内存。请等待，不要关闭窗口。10 分钟后还是卡住？重启电脑，
|
||||
关闭其他程序后再试，或者选一个更小的尺寸。
|
||||
- **下载或安装时中断了。** 重新启动应用或 `strata launch`。它会从中断的地方继续。
|
||||
- **非常慢，或者提示“the engine stopped unexpectedly”。** 你的电脑可用内存不够。关闭其他程序（浏览器很占内存），
|
||||
或者选一个更小的尺寸，比如 Q2_0 或 IQ2_XS。
|
||||
- **提示这台电脑没有可用的引擎。** 在应用里打开**设置，引擎**，点击**更新引擎**。在终端里运行 `strata update`。
|
||||
- **提示 8080 端口已被占用。** Strata 已经在运行了。找一下它的窗口。
|
||||
**Claude Code · Codex CLI · OpenCode · oh-my-pi · pi.dev · Hermes Agent · OpenClaw**，也可以通过兼容 OpenAI 和 Anthropic 的 API 使用更多工具。
|
||||
|
||||
更多解决办法：[docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md)。还是解决不了？请提交
|
||||
[issue](https://github.com/Niko1221/Strata/issues) 并附上日志：在应用里是**设置，引擎，诊断，保存日志**，在终端里是
|
||||
`strata doctor` 的输出。如果是安全问题，请私下报告：[SECURITY.md](SECURITY.md)。
|
||||
运行 **`strata connect`**，查看你所用工具的设置方法。[连接指南](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect)。
|
||||
|
||||
## 致谢与许可证
|
||||
<a id="guides"></a>
|
||||
<a id="how-fast-is-it"></a>
|
||||
<a id="how-does-it-work"></a>
|
||||
<a id="something-went-wrong"></a>
|
||||
|
||||
模型是 Qwen 团队的 [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)。压缩工作由
|
||||
[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)、UkisAI（Swift 1.5）、agentionai（Gyro）
|
||||
和 Unsloth 完成。DeepSeek V4.1-Flash 的各个版本也由 ISTA-DASLab 制作。Strata 使用了
|
||||
[llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) 的部分代码。全部致谢：
|
||||
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits)。Strata 以 [MIT 许可证](LICENSE)开源。少数部分和所有模型有各自的许可证
|
||||
（[具体是哪些](docs/HOW_IT_WORKS.md#license)）。
|
||||
## 帮助
|
||||
|
||||
## 参与贡献
|
||||
|
||||
发现了 bug，或者想提交修改或基准测试结果？请先阅读 [CONTRIBUTING.md](CONTRIBUTING.md)。其中说明了需要附上哪些内容，
|
||||
方便其他人核实你的工作。
|
||||
[全部指南](docs) · [故障排查](docs/TROUBLESHOOTING.md) · [报告问题](https://github.com/Niko1221/Strata/issues/new/choose) ·
|
||||
[基准测试](docs/COMMUNITY_BENCHMARKS.md) · [工作原理](docs/HOW_IT_WORKS.md) · [参与贡献](CONTRIBUTING.md)
|
||||
|
||||
## 支持 Strata
|
||||
|
||||
Strata 免费，采用 MIT 许可证。我在能拿到的各种硬件上开发和测试它。如果它在你的电脑上跑得不错，请我喝杯咖啡，能帮我继续测试新显卡、
|
||||
修复你们反馈的问题。不用有压力。使用它并反馈你的发现，同样很有帮助。
|
||||
**帮助本地 AI 保持开放。** 一杯咖啡能帮助我们在更多硬件上开发和测试。
|
||||
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a>
|
||||
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee：支持 Strata" height="48"></a>
|
||||
|
||||
有兴趣赞助 Strata？请通过 [GitHub](https://github.com/Niko1221) 联系我。
|
||||
[MIT 许可证](LICENSE) · [致谢和模型许可证](docs/HOW_IT_WORKS.md#credits) · [隐私](docs/SAFETY.md)
|
||||
|
||||
<a id="option-c-let-your-ai-set-it-up"></a>
|
||||
|
||||
<sub>面向 AI 工具和自动阅读器： <a href="AGENTS.md">技术参考</a> · <a href="docs/AI_SETUP.md">安装说明</a></sub>
|
||||
|
||||
**赞助、硬件合作或其他合作：** [[email protected]](mailto:[email protected]).
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "strata-desktop",
|
||||
"productName": "Strata",
|
||||
"version": "0.2.5",
|
||||
"version": "0.2.6",
|
||||
"private": true,
|
||||
"description": "Strata Desktop - local AI, tuned to one model.",
|
||||
"scripts": {
|
||||
|
||||
Generated
+1
-1
@@ -3524,7 +3524,7 @@ checksum = "ff7589a1859b09232522bb9edf076ca7fcf14ded96685d3fa629db73023ffbb7"
|
||||
|
||||
[[package]]
|
||||
name = "strata-desktop"
|
||||
version = "0.2.5"
|
||||
version = "0.2.6"
|
||||
dependencies = [
|
||||
"gtk",
|
||||
"serde",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "strata-desktop"
|
||||
version = "0.2.5"
|
||||
version = "0.2.6"
|
||||
description = "Strata Desktop — local AI, tuned to one model."
|
||||
edition = "2021"
|
||||
rust-version = "1.77"
|
||||
|
||||
@@ -701,6 +701,35 @@ SectionEnd
|
||||
!macroend
|
||||
; STRATA-MIGRATE-END
|
||||
|
||||
; STRATA-STOP-BEGIN
|
||||
; Stop what Strata itself started from the install folder (the app, its host, the node runtime, the engine) before files are removed:
|
||||
; a running copy keeps its files open, so the folder stayed behind after use. Only processes whose image path is inside INSTDIR are
|
||||
; stopped; this uninstaller's own process and any program elsewhere are left alone.
|
||||
!macro StrataStopOwnProcesses INSTALLDIR
|
||||
System::Call 'kernel32::SetEnvironmentVariableW(w "STRATA_INSTDIR", w "${INSTALLDIR}")'
|
||||
System::Call 'kernel32::GetCurrentProcessId()i.r9'
|
||||
System::Call 'kernel32::SetEnvironmentVariableW(w "STRATA_SELFPID", w "$9")'
|
||||
; the installer is a 32-bit program, so plain powershell.exe is the 32-bit one, which cannot read the image path of a 64-bit process: use the 64-bit one
|
||||
StrCpy $8 "$WINDIR\Sysnative\WindowsPowerShell\v1.0\powershell.exe"
|
||||
IfFileExists $8 +2 0
|
||||
StrCpy $8 "powershell.exe"
|
||||
nsExec::Exec '"$8" -NoProfile -NonInteractive -ExecutionPolicy Bypass -Command "$$d = [IO.Path]::GetFullPath($$env:STRATA_INSTDIR).TrimEnd([char]92) + [char]92; $$ids = @(); $$ids += Get-Process -ErrorAction SilentlyContinue | Where-Object { $$_.Path -and [IO.Path]::GetFullPath($$_.Path).StartsWith($$d, [StringComparison]::OrdinalIgnoreCase) } | ForEach-Object { $$_.Id }; try { $$ids += Get-CimInstance Win32_Process -ErrorAction Stop | Where-Object { $$_.ExecutablePath -and [IO.Path]::GetFullPath($$_.ExecutablePath).StartsWith($$d, [StringComparison]::OrdinalIgnoreCase) } | ForEach-Object { $$_.ProcessId } } catch {}; $$ids | Sort-Object -Unique | Where-Object { $$_ -ne [int]$$env:STRATA_SELFPID } | ForEach-Object { Stop-Process -Id $$_ -Force -ErrorAction SilentlyContinue }; Start-Sleep -Milliseconds 1500"'
|
||||
Pop $9
|
||||
!macroend
|
||||
; STRATA-STOP-END
|
||||
|
||||
; STRATA-CLEAN-BEGIN
|
||||
; The app writes into its own folders while it runs (Python bytecode caches under control\, for one), and the file-by-file delete below does not know them,
|
||||
; so those folders stayed behind with a few dozen files. These four folders hold only what the installer put there and what the app made there;
|
||||
; user data is never in them (it lives in the data folder), and the install folder itself is still removed with a plain RMDir.
|
||||
!macro StrataRemoveAppFolders INSTALLDIR
|
||||
RMDir /r "${INSTALLDIR}\control"
|
||||
RMDir /r "${INSTALLDIR}\host"
|
||||
RMDir /r "${INSTALLDIR}\node"
|
||||
RMDir /r "${INSTALLDIR}\harness"
|
||||
!macroend
|
||||
; STRATA-CLEAN-END
|
||||
|
||||
Section Install
|
||||
SetOutPath $INSTDIR
|
||||
|
||||
@@ -847,6 +876,7 @@ Section Uninstall
|
||||
!insertmacro NSIS_HOOK_PREUNINSTALL
|
||||
!endif
|
||||
|
||||
!insertmacro StrataStopOwnProcesses "$INSTDIR"
|
||||
!insertmacro CheckIfAppIsRunning "$INSTDIR\${MAINBINARYNAME}.exe" "${PRODUCTNAME}"
|
||||
|
||||
; Delete the app directory and its content from disk
|
||||
@@ -879,6 +909,8 @@ Section Uninstall
|
||||
{{/each}}
|
||||
|
||||
|
||||
!insertmacro StrataRemoveAppFolders "$INSTDIR"
|
||||
|
||||
; Delete uninstaller
|
||||
Delete "$INSTDIR\uninstall.exe"
|
||||
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "strata",
|
||||
"version": "0.2.5",
|
||||
"version": "0.2.6",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "strata",
|
||||
"version": "0.2.5",
|
||||
"version": "0.2.6",
|
||||
"dependencies": {
|
||||
"html-to-image": "^1.11.13",
|
||||
"katex": "^0.19.0",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "strata",
|
||||
"private": true,
|
||||
"version": "0.2.5",
|
||||
"version": "0.2.6",
|
||||
"type": "module",
|
||||
"description": "Strata \u2014 local AI, tuned to one model. Web shell + engine host.",
|
||||
"scripts": {
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Agent bridge (Strata chats as DeepSeek Harness sessions)
|
||||
|
||||
This page is for developers who work on the app host's agent code. It lists the files, the frames and routes, and the rules the
|
||||
bridge enforces. Tests are listed at the end.
|
||||
|
||||
The Harness is a separate, pinned component (`DSH 0.2.1-alpha.3`, commit `57190202`, MIT). The host finds it through
|
||||
`STRATA_HARNESS_DIR` (or `<STRATA_COMPONENTS_DIR>/harness`), starts one `dsh --profile web` process on a loopback port,
|
||||
and talks to its Gateway. Nothing here downloads or builds anything.
|
||||
@@ -14,12 +17,13 @@ and talks to its Gateway. Nothing here downloads or builds anything.
|
||||
| `children.mjs` | pure projection of one helper agent's events into its row and its approval card |
|
||||
| `preset.mjs` | the agent preset (which tools the chat has) |
|
||||
| `plugins/strata-gate.mjs` | the Manual-mode approval gate (a Harness plugin, loaded by path) |
|
||||
| `plugins/strata-search.mjs`, `plugins/strata-models.mjs` | the app's web search registered with the Harness; helper agents on a named model |
|
||||
| `plugins/strata-context.mjs`, `context-budget.mjs`, `context-fit.mjs` | measured context pressure, summaries that fit the window, the check before a message goes out |
|
||||
| `agent-bridge.mjs` | the chat-to-session map, turns, approvals, Stop, status |
|
||||
|
||||
## Frames sent on `/api/chat/stream` for an agent turn
|
||||
|
||||
`queued`, `start` (`agent:true`), `reasoning`, `delta`, `agent-tool` (new: upserts a tool card by call id with
|
||||
`queued`, `start` (`agent:true`), `reasoning`, `delta`, `agent-tool` (upserts a tool card by call id with
|
||||
`running | proposed | done | denied | error`), `done` (usage, `finishReason`, `cancelled`), `error`, and `notice`
|
||||
(why a chat is not using the agent). The chat UI renders them with its existing components.
|
||||
|
||||
@@ -70,7 +74,7 @@ The chat's Mode menu is real Harness behaviour. Each turn the page sends the cha
|
||||
What the person chose is saved by `tools-settings.mjs` (inside the host's settings file) and shown by `src/components/SettingsTools.tsx` (`GET/PUT /api/agent/tools`, `POST/PATCH/DELETE /api/agent/tools/custom[/id]`). A change reaches the agent from the next message: the profile carries a `toolsSignature`, and the bridge restarts the Harness (after the running turns end) when it differs.
|
||||
|
||||
- Tool groups (`tools-catalog.mjs`): each is one switch in plain words and is one or more preset plugins (`withoutDisabled`); `fetch` and `search` are the two halves of the Harness's `tool-web` (both off drops it). Goals off: `/goal` answers that goals are turned off.
|
||||
- Web search needs no account. The default is DuckDuckGo's HTML result page (`../web-search-ddg.mjs`), with Bing's plain result page (`../web-search-bing.mjs`) as the second source. Manners, because DuckDuckGo refuses a program that asks fast (a real box: two searches in 1.2 s answered, the third got HTTP 202 and every DuckDuckGo page refused for many minutes): a browser User-Agent and the page's own request shape (Referer, Origin, Sec-Fetch-*), one request at a time per source, 2.5 to 4 s apart (random), at most 8 a minute, a 30 minute session cache per question, a cooldown after a refusal that doubles (1, 2, 4 ... 10 minutes, `Retry-After` honoured) during which that source gets no request, no lite-page follow-up after a refusal, one retry for a timeout or a 5xx, a 15 s limit. When every source refuses, the tool answers one plain sentence (`LIMITED_SENTENCE`: "Web search is limited right now. Try again in a minute, or add a search key (Exa, Perplexity or DeepSeek) in Settings -> Tools -> Web search.") plus an instruction to the model not to retry; the turn guard in `agent-bridge.mjs` (see Turn guard) stops a model that retries anyway. **The results come from DuckDuckGo or Bing**; they are not ranked or filtered here. The person may switch to Exa, Perplexity or DeepSeek with their own key (`../web-search-keyed.mjs`; DeepSeek uses the Harness's own provider): the key is saved in the settings file, reaches the agent only as the environment variable `STRATA_SEARCH_KEY`, never the profile file and never the screen (it only learns `keySet`). A keyed service without its key falls back to DuckDuckGo. The profile pins `web.searchProvider` to the chosen id so two usable backends cannot make the choice ambiguous.
|
||||
- Web search needs no account. The default is DuckDuckGo's HTML result page (`../web-search-ddg.mjs`), with Bing's plain result page (`../web-search-bing.mjs`) as the second source. Manners, because DuckDuckGo refuses a program that asks fast: a browser User-Agent and the page's own request shape (Referer, Origin, Sec-Fetch-*), one request at a time per source, 2.5 to 4 s apart (random), at most 8 a minute, a 30 minute session cache per question, a cooldown after a refusal that doubles (1, 2, 4 ... 10 minutes, `Retry-After` honoured) during which that source gets no request, no lite-page follow-up after a refusal, one retry for a timeout or a 5xx, a 15 s limit. When every source refuses, the tool answers one plain sentence (`LIMITED_SENTENCE`: "Web search is limited right now. Try again in a minute, or add a search key (Exa, Perplexity or DeepSeek) in Settings -> Tools -> Web search.") plus an instruction to the model not to retry; the turn guard in `agent-bridge.mjs` (see Turn guard) stops a model that retries anyway. **The results come from DuckDuckGo or Bing**; they are not ranked or filtered here. The person may switch to Exa, Perplexity or DeepSeek with their own key (`../web-search-keyed.mjs`; DeepSeek uses the Harness's own provider): the key is saved in the settings file, reaches the agent only as the environment variable `STRATA_SEARCH_KEY`, never the profile file and never the screen (it only learns `keySet`). A keyed service without its key falls back to DuckDuckGo. The profile pins `web.searchProvider` to the chosen id so two usable backends cannot make the choice ambiguous.
|
||||
- The browser: Playwright's MCP server (`@playwright/mcp`, staged next to the Harness by `apps/desktop/scripts/stage-harness.mjs`) on the person's own Chrome, Edge or Chromium (`browser-probe.mjs`), `--isolated` so it never sees their profile. Tools appear as `mcp__browser__browser_*`. In Manual, opening a page, clicking, typing, uploading, running a script in the page and closing tabs ask (plain reasons: "Open the web page ...", "Click on the page: ..."); looking (snapshot, screenshot, console, network, waiting, listing tabs) does not. Plan refuses every connected tool. The screen says plainly when no browser, or no tool server, is found.
|
||||
- The person's own tools: `instructions` (the tool returns their text; look-only, never asks), `command` (runs their command with the assistant's text in the `STRATA_INPUT` environment variable, never spliced into the command line; asks in Manual, refused in Plan) and `server` (a web address or a program, added as `mcp__custom-<name>__*`; asks in Manual). Instruction and command tools are served by `custom-tools-mcp.mjs` (stdio MCP) from a JSON file the settings module rewrites on every change, so a switch in Settings is seen without a restart.
|
||||
- `strata-gate.mjs` implements the Manual/Plan rules above (`planDecision`, `browserDecision`, the look-only instruction server).
|
||||
@@ -81,7 +85,7 @@ What the person chose is saved by `tools-settings.mjs` (inside the host's settin
|
||||
- A helper's request for approval is the same card (state `proposed`, id `<helper session>:<call>`), answered with the same `/api/chat/approval`; a chat can only answer its own helpers.
|
||||
- The run (the chat's one NDJSON response) stays open until the parent is idle, every helper has finished and reported back, and no armed goal is waiting; `done` is sent then.
|
||||
- Stop: pauses an active goal, interrupts every helper (`subagents/interruptByParent`; `session/cancel` on a helper session is refused by the Harness), cancels the parent, waits; a report that would wake the parent afterwards is cancelled too.
|
||||
- The number of helpers at once equals the engine's request slots (`maxActiveSubagents` in the profile); the Harness refuses more with a message the model reads.
|
||||
- The number of helpers at once is the engine's request slots, with a floor of 4 for a local model and 8 for an API model (`maxActiveSubagents` in the profile). The Harness refuses more with a message the model reads.
|
||||
|
||||
## /goal
|
||||
|
||||
@@ -90,7 +94,7 @@ What the person chose is saved by `tools-settings.mjs` (inside the host's settin
|
||||
## Thinking and the output budget
|
||||
|
||||
- A local route is reasoning-capable (`reasoning: true`) with `compat.thinkingFormat: 'chat-template'`: the request carries `chat_template_kwargs {enable_thinking, reasoning_effort}`, which the Strata server reads (a top-level `enable_thinking` is ignored by it, which is why pi-ai's `qwen` format left the model thinking). The chat's Reasoning menu travels as `thinking` in the stream body; the bridge sends it with `session/selectModel` (`reasoningEffort`) whenever the model or the level changed; a level a route does not have is dropped (never an error). API routes keep the provider default (the menu is disabled for them in the UI).
|
||||
- `localOutputBudget(context)` = context minus 4096 (it was context/4 capped at 8192, which cut big answers off at exactly 8192 tokens); pi-ai sends the window minus the prompt. App-loaded models get `fit_max_tokens: true` so the engine clamps instead of answering 400. Sampling is not set here: the Harness sends none, so the server's own defaults apply.
|
||||
- `localOutputBudget(context)` is 30 percent of the window, at least 4096 and at most 32768 (9,830 tokens at 32K, 19,660 at 64K). The Harness's automatic compaction reads it as the room a reply reserves. pi-ai sends the window minus the prompt. App-loaded models get `fit_max_tokens: true` so the engine clamps instead of answering 400. Sampling is not set here: the Harness sends none, so the server's own defaults apply.
|
||||
|
||||
## Tasks
|
||||
|
||||
@@ -108,7 +112,7 @@ Failures the person sees are plain sentences with a next step (`AgentUnavailable
|
||||
|
||||
## Helper agents on a named model (`plugins/strata-models.mjs`, `model-routes.mjs`)
|
||||
|
||||
The Harness's `subagent` tool takes an exact `provider` route key and model id, and the app had its model choice switched off, so every helper ran on the chat's model. The profile now enables the Harness's `subagent-model-selection-settings` with every route the app can reach (local instances and saved API models), and the `strata-models` plugin wraps the two tool definitions the Harness registers for each chat (`subagent`, `list_subagent_models`) in place:
|
||||
The Harness's `subagent` tool takes an exact `provider` route key and model id. The profile enables the Harness's `subagent-model-selection-settings` with every route the app can reach (local instances and saved API models), and the `strata-models` plugin wraps the two tool definitions the Harness registers for each chat (`subagent`, `list_subagent_models`) in place:
|
||||
|
||||
- the model is found from its id, `provider/id`, the name in the model picker, `connection name`, or "local" (any letters). A close guess is not started: the tool answers "Did you mean ...?" and nothing runs until the call is made again with that model. Several fits are listed; nothing fitting lists the models that can be used and says they can be added in Models.
|
||||
- the tool's own text says when to use it ("ask a subagent", "ask model X"), not to look for the model with shell commands, and not to wait with sleep. The system prompt (`preset.mjs`) says the same, so it reaches every model, local or hosted.
|
||||
@@ -126,7 +130,7 @@ Pictures are on by default for every model that can read them: an API model read
|
||||
|
||||
## Turn guard (`turn-guard.mjs`)
|
||||
|
||||
A turn that keeps failing the same way is stopped by the app, never left to the model. Seen on a search-heavy `/plan`: every web search was refused, the model searched again and rewrote its plan after each one until the context reached 100%, then the turn ended with no answer and no message.
|
||||
A turn that keeps failing the same way is stopped by the app, never left to the model.
|
||||
|
||||
- **Counted, from what the turn shows:** web searches that failed in a row (3), searches in one turn (10 in a plan turn, 25 otherwise), a plan turn that starts its numbered list from `1.` again (4 times), and a context of 93% or more (the Harness's own compaction did not keep up).
|
||||
- **Stopped the way Stop stops** (`session/cancel`, helpers first) but ended as a normal turn (`done`, not cancelled, no error card), so the plan the model had written is still proposed. The plan card carries the reason (`plan.caution`); any other turn ends with one `delta` frame that has `own: true` and `key` (the English sentence, translated by the page; `GUARD_NOTES` are in the interface catalog in all eight languages). Each sentence says what happened and gives one next step.
|
||||
@@ -134,9 +138,7 @@ A turn that keeps failing the same way is stopped by the app, never left to the
|
||||
- **Never silent:** a turn that ends (`completed` or `max-tokens`) with no answer text gets "I ran out of room in this chat..." when the last request used 85% of the window or more, otherwise "I stopped without writing an answer..." (not when tool cards were shown, not after Stop, not after an error, which says why by itself). Goal runs have their own ending lines and are not guarded.
|
||||
- Tests: `tests/agent-turn-guard.test.mjs` (counting, always; end to end with the real Harness, the scripted model's `searchloop` and `bloat` scripts and `searchMock: 'limited'` on the bridge, which runs the real search code on a pretend web that refuses, no network).
|
||||
|
||||
## Context: measured pressure, summaries that fit, the check before a message goes out (2026-10-22)
|
||||
|
||||
Found by the owner's hands-on test of the installed 0.2.1 (Gyro-S, 64K): a 118 KB log paste was 66,500 tokens to the engine and about 29,500 to the Harness (4 characters a token; logs and code are about 2); the engine refused it with a 400, the refusal stayed in the history, and then neither the next message nor `/compact` could go through, because the summary request carried the same history and was refused for its size too. `/compact` itself was never a command of the app: it went to the model as a question and the model answered "nothing to compact".
|
||||
## Context: measured pressure, summaries that fit, the check before a message goes out
|
||||
|
||||
- **`plugins/strata-context.mjs`** (inserted by the profile, like the gate) wraps `tokenMeter.measure` and `llm.stream` in place, once each. Pressure = the last prompt size the model server reported for an answer, plus what changed since priced at the density that measurement showed (`context-budget.mjs`: `pressureNow`, `densityOf`); never lower than the Harness's own price. Node prices are scaled the same way, so the retained tail is in real tokens. A summary request (`purpose: 'compaction'`) that would not fit the window is asked for in pieces: a message bigger than a piece is cut for the request only, each piece builds on the summary before it, a piece the server refuses is halved and tried again. `/strata-context` (a Harness command, like `/strata-gate`) answers `{tokens, window}` for a session so the host can tell the person true figures.
|
||||
- **`context-fit.mjs`**: before a message is sent the model server counts it (`POST /v1/messages/count_tokens`, its own tokenizer). `decideFit`: send; summarise first and then send (the message fits an otherwise empty chat); or never fits. The last one is not sent: the error frame has `code: 'CONTEXT_MESSAGE_TOO_LARGE'` and the page offers "Attach as a file" (`oversize.mjs`, `/api/chat/oversize`: the text is saved in the chat's folder, the assistant reads it in parts) or "Send a shortened preview". An engine that cannot count lets the message through; the Harness's own compact-and-retry is the net.
|
||||
@@ -144,3 +146,15 @@ Found by the owner's hands-on test of the installed 0.2.1 (Gyro-S, 64K): a 118 K
|
||||
- **Visible:** `compaction/start` shows "Summarizing earlier messages..." in the status line; `compaction/end` adds one plain line to the reply. The projector also sends a `context` frame after every model request, so the meter follows the running turn.
|
||||
- **A refusal that is left** (`CONTEXT_WINDOW_EXCEEDED`) is a plain sentence with the engine's text under Details and the buttons "Summarize and continue" and "Start a new chat". "Try sending the message again" is for other errors only.
|
||||
- Tests: `tests/agent-context.test.mjs` (arithmetic; the real Harness against the scripted worker with `window`, `charsPerToken` and `noCount`), `tests/oversize.test.mjs`, `tests/context-ui.test.mjs`. The run against a real engine is `scripts/context-e2e.mjs`.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
- **Search refusals (2026-10):** on a real box two DuckDuckGo searches in 1.2 s answered, the third got HTTP 202, and every DuckDuckGo page refused for many minutes.
|
||||
The pacing, cooldown and cache rules above come from that test.
|
||||
- **Search-heavy `/plan` (2026-10):** every web search was refused, the model searched again and rewrote its plan after each one until the context reached 100%.
|
||||
The turn ended with no answer and no message. The turn guard comes from that run.
|
||||
- **Large log paste (2026-10-22, installed 0.2.1, Gyro-S, 64K):** a 118 KB log was 66,500 tokens to the engine and about 29,500 to the Harness
|
||||
(4 characters a token; logs and code are about 2). The engine refused it with a 400. The refusal stayed in the history, so neither the next message nor `/compact` could
|
||||
go through, because the summary request carried the same history. `/compact` was also not a command of the app yet: it went to the model as a question. This
|
||||
produced `strata-context.mjs`, `context-fit.mjs` and the host's own `/compact`.
|
||||
- **Output budget:** an earlier local budget of a quarter of the window, capped at 8192, cut big answers off at exactly 8192 tokens.
|
||||
@@ -137,7 +137,7 @@ export function createNativeControl({directory,requests=()=>[],changed=()=>{},sp
|
||||
if(closing)return state();
|
||||
/* A read that is not forced never makes the window wait for the (slow) status of Control: it gets the last state at once, and a stale one
|
||||
is refreshed in the background. Only the first read, and a forced one after an operation, waits. */
|
||||
if(refreshing)return !force&&snapshot.available?state():refreshing;if(!force&&Date.now()-lastRead<45000)return state();
|
||||
if(refreshing)return !force&&snapshot.available?state():force?refreshing.then(()=>refresh(true)):refreshing; /* a forced read that meets one already running (started before an engine update finished) waits for it and reads again, so it never settles on the old answer */if(!force&&Date.now()-lastRead<45000)return state();
|
||||
probeDrafting();
|
||||
refreshing=(async()=>{try{const [catalog,status]=packaged?await (async()=>{const s=await tool('strata_status',{hardware:true});const c=await tool('strata_models').catch(error=>({families:[],contexts:[],recommendation:{family:null,model:null},context_note:'',error:error.message}));return [c,s];})():await Promise.all([tool('strata_models'),tool('strata_status',{hardware:true})]);let visionReader=false;if(packaged){try{const e=await control.engine();const root=e?.engine?.root??(e?.engine?.exe?dirname(e.engine.exe):'');visionReader=!!root&&['strata-vision','strata-vision.exe'].some(n=>existsSync(join(root,n)));}catch{visionReader=false;}}const markImages=c=>c&&Array.isArray(c.families)?{...c,families:c.families.map(f=>({...f,imagesCapable:!!f.images,images:!!f.images&&visionReader,sizes:(f.sizes??[]).map(s=>({...s,imagesCapable:!!s.images,images:!!s.images&&visionReader}))}))}:c;snapshot={available:true,repo:packaged?status.installDir:repo(),version:packaged?(control.version??null):await execute('git',['describe','--tags','--always'],repo()).then(s=>s.trim()).catch(()=>null),summary:status.summary,catalog:markEngine(packaged?markImages(catalog):catalog,status.engine),engineInfo:{version:status.engine?.version??null,source:status.engine?.source??null},visionReader:packaged?visionReader:true,installed:installedInfo(status.installed),hardware:status.hardware,error:null,install:status.install,external:status.running??null,...(packaged?{packaged:true,installDir:status.installDir,engine:status.engine,installerError:status.installerError}:{})};lastRead=Date.now();if(status.install?.running&&![...jobs.values()].some(j=>j.state==='running'))begin('install',j=>watchInstall(j,status.install));}catch(error){snapshot={...snapshot,available:false,error:error.message};lastRead=Date.now();}emit();return state();})().finally(()=>{refreshing=null;});const run=refreshing;if(!force&&snapshot.available){run.catch(()=>{});return state();}return run;
|
||||
}
|
||||
|
||||
@@ -104,7 +104,7 @@ export function NativeModels({ onOtherModels, onReady, onImport, onConnectEngine
|
||||
// Drafting (which drafter guesses ahead): DFlash2 is for Qwen3.8-Flash-Next only (the qwen and gyro families), so no other model is offered it.
|
||||
const draftingChoices = selectedSize ? draftingOptions(selectedSize.sourceFamily ?? selectedFamily?.family, native?.capabilities?.dflash2) : null;
|
||||
const draftingWords: Record<string, string> = { auto: 'Auto (recommended)', mtp: 'MTP', dflash2: 'DFlash2', off: 'Off' };
|
||||
const values = useMemo(() => ({ family: selectedSize?.sourceFamily, model: selectedSize?.model, context, vision: selectedSize?.images ? vision : 'no', kv, low_ram: lowRam, resourceProfile, ...(cardsGpus.length > 1 ? { gpus: cardsGpus.join(',') } : cardsGpus.length === 1 ? { gpu: cardsGpus[0] } : {}), loadAfter: true, multiple: expert && multiple }), [selectedSize?.sourceFamily, selectedSize?.model, context, vision, kv, lowRam, resourceProfile, cardsGpus.join(','), expert, multiple]);
|
||||
const values = useMemo(() => ({ family: selectedSize?.sourceFamily, model: selectedSize?.model, context: selectedSize?.multi_file ? 8192 : context, vision: selectedSize?.images ? vision : 'no', kv, low_ram: lowRam, resourceProfile, ...(cardsGpus.length > 1 ? { gpus: cardsGpus.join(',') } : cardsGpus.length === 1 ? { gpu: cardsGpus[0] } : {}), loadAfter: true, multiple: expert && multiple }), [selectedSize?.sourceFamily, selectedSize?.model, context, vision, kv, lowRam, resourceProfile, cardsGpus.join(','), expert, multiple]);
|
||||
const downloadValues = { ...values, ...(draftingChoices ? { drafting } : {}), vision: selectedSize?.images && vision !== 'no' ? 'yes' : 'no', loadVision: selectedSize?.images ? vision : 'no' };
|
||||
useEffect(() => { if (!native?.catalog || family)
|
||||
return; const current = native.instances.find(i => i.id === native.defaultId); const rec = native.catalog.recommendation ?? { family: null, model: null }; const loadedFamily = families.find(f => f.sizes.some(s => s.id === current?.model)); const have = new Set((native.installed?.models ?? []).filter(m => m.ready).map(m => m.model)); const installedFamily = families.find(f => f.sizes.some(s => have.has(s.id))); const f = loadedFamily ?? installedFamily ?? families.find(f => f.family === rec.family || f.sizes.some(s => s.sourceFamily === rec.family)) ?? families[0]; if (f) {
|
||||
@@ -138,8 +138,11 @@ export function NativeModels({ onOtherModels, onReady, onImport, onConnectEngine
|
||||
if (installed.resourceProfile && !loadedSelected?.resourceProfile)
|
||||
setResourceProfile(installed.resourceProfile);
|
||||
}
|
||||
else
|
||||
else {
|
||||
setVision('yes');
|
||||
if (selectedSize.multi_file)
|
||||
setContext(8192); // DeepSeek runs at the one context its engine was checked at (as in the guided setup)
|
||||
}
|
||||
}, [selectedSize?.id, installed?.model, installed?.ready, installed?.context, installed?.kv, installed?.images, installed?.visionEnabled, installed?.visionDevice, installed?.lowRamMode, installed?.resourceProfile]);
|
||||
useEffect(() => { if (jobId && job?.state === 'done' && job.id !== toastedJob && (job.kind === 'load' || job.kind === 'install')) {
|
||||
setToastedJob(job.id);
|
||||
@@ -210,7 +213,7 @@ export function NativeModels({ onOtherModels, onReady, onImport, onConnectEngine
|
||||
<Dialog open={!!plan} title={tr(installed?.ready ? "Apply settings and load" : "Download and prepare")} compact onClose={() => !pending && setPlan(null)} actions={<><Btn disabled={pending} onClick={() => setPlan(null)}>{tr("Cancel")}</Btn><Btn variant="primary" disabled={pending || !!plan?.plan?.disk_short} onClick={() => void download()}>{tr(installed?.ready ? "Apply & load" : "Download & load")}</Btn></>}><p><strong>{tr(plan?.plan?.title ?? selectedFamily?.title)}</strong>{tr(installed?.ready ? null : <>{tr(" \u00B7 approximately ")}{tr(plan?.plan?.download_gb ?? selectedSize?.download_gb)}{tr(" GB")}</>)}</p>{tr(plan?.drafting?.needed ? <p className="small muted">{tr("Choosing DFlash2 also downloads its drafter: about {0} GB, once. If it does not fit in graphics memory, the model uses the MTP instead.", [plan.drafting.gb])}</p> : null)}{tr(installed?.ready ? <p className="small muted">{tr("The model files are already on this computer. Nothing will be downloaded; only the settings you chose are applied.")}</p> : null)}{tr(plan?.plan?.disk_short ? <ErrorNotice message={plan.plan.disk_short}/> : null)}{tr(installed?.ready ? <p className="small muted">{tr("Any currently loaded native models will be unloaded first.")}</p> : <><p className="small muted">{tr("Existing files are checked and reused. Any currently loaded native models will be unloaded before preparing files.")}</p><p className="small muted">{tr("Your computer may feel slower while weights load. You can stop the download and resume later.")}</p></>)}{tr(selectedSize?.images ? <p className="small muted">{tr("Vision encoder files are included. Picture reading loads only when enabled.")}</p> : null)}</Dialog>
|
||||
<NativeDeleteDialog id={deleting} onClose={() => setDeleting(null)}/>
|
||||
<Dialog open={manage} title={tr("Manage models")} onClose={() => setManage(false)} actions={<Btn onClick={() => setManage(false)}>{tr("Done")}</Btn>}><div className="native-manage">
|
||||
{tr(downloaded.length ? <div className="native-manage-list" role="list">{tr(downloaded.map(m => { const v = allSizes.find(x => x.id === m.model), loaded = native?.instances.some(i => i.model === m.model && i.status === 'ready'); return <div className="native-manage-row" role="listitem" key={m.model} data-selected={selectedSize?.id === m.model || undefined}>
|
||||
{tr(downloaded.some(m => m.ready || m.diskGb) ? <div className="native-manage-list" role="list">{tr(downloaded.filter(m => m.ready || m.diskGb).map(m => { const v = allSizes.find(x => x.id === m.model), loaded = native?.instances.some(i => i.model === m.model && i.status === 'ready'); return <div className="native-manage-row" role="listitem" key={m.model} data-selected={selectedSize?.id === m.model || undefined}>
|
||||
<span className="native-manage-name"><strong>{tr(v ? `${catalogVersionLabel(v)}` : m.model_name ?? m.model)}</strong><small>{tr(v ? v.model : m.model)}{tr(m.diskGb ? ` · ${sizeGb(m.diskGb)}` : '')}</small></span>
|
||||
<span className={loaded ? 'st-pill st-pill--accent' : 'st-pill'}>{tr(loaded ? 'Loaded' : 'Downloaded')}</span>
|
||||
<span className="native-manage-actions"><Btn size="sm" disabled={!!active || !v} onClick={() => { if (v) setSize(v.id); setManage(false); }}>{tr(selectedSize?.id === m.model ? 'Selected' : 'Use version')}</Btn><Btn size="sm" variant="danger" disabled={!!active} onClick={() => setDeleting(m.model)}>{tr("Delete")}</Btn></span>
|
||||
|
||||
@@ -19,7 +19,7 @@ export function SettingsStorage() {
|
||||
const library = state.library;
|
||||
const versions = library ? pickerGroups(library).flatMap(group => group.versions) : [];
|
||||
// ONE list and ONE total: the optimized models (sizes from their real files, shared files counted once) and the other local models together.
|
||||
const nativeModels = native?.installed?.models ?? [];
|
||||
const nativeModels = (native?.installed?.models ?? []).filter(m => m.ready || (m.diskGb ?? 0) > 0); // a model whose files are all gone (an old config) is not a 0.0 GB download
|
||||
const sharedGb = (native?.installed?.usage?.shared_bytes ?? 0) / 1e9;
|
||||
const nativeGb = native?.installed?.usage ? native.installed.usage.total_gb : nativeModels.reduce((n, m) => n + (m.diskGb ?? 0), 0);
|
||||
const totalGb = nativeGb + versions.reduce((n, v) => n + (v.diskGb ?? 0), 0);
|
||||
|
||||
@@ -9,6 +9,7 @@ import { NativeProgress } from './NativeModels';
|
||||
import { jobErrorText } from '../lib/native-job-diagnostics';
|
||||
import { appVersionLabel } from '../lib/app-version';
|
||||
import { fetchUpdateCheck, publishUpdateCheck, useInstalledVersions, useLastUpdateCheck } from '../lib/update-check';
|
||||
import { shownEngineVersion } from '../lib/engine-version';
|
||||
/** A check from an earlier day shows its date too, because the result is kept between visits. */
|
||||
function checkedLabel(at: string) {
|
||||
const when = new Date(at), sameDay = when.toDateString() === new Date().toDateString();
|
||||
@@ -30,7 +31,7 @@ export function SettingsUpdates() {
|
||||
const [error, setError] = useState('');
|
||||
const request = useRef<AbortController | null>(null);
|
||||
useEffect(() => () => request.current?.abort(), []);
|
||||
const installed = component === 'strata' ? appVersion : native?.available ? (native.installed?.engine?.version ?? 'Not installed') : engineVersion;
|
||||
const installed = component === 'strata' ? appVersion : native?.available ? (native.installed?.engine?.version ?? 'Not installed') : shownEngineVersion(native, engineVersion);
|
||||
const active = native?.jobs.find(j => j.state === 'running');
|
||||
const result = checking || error ? null : (fresh ?? last);
|
||||
async function operate(kind: string) {
|
||||
@@ -78,7 +79,7 @@ export function SettingsUpdates() {
|
||||
return <>
|
||||
<div className="settings-versions" aria-busy={checking}>
|
||||
{tr(versionRow('Strata', appVersion, result?.strata))}
|
||||
{tr(versionRow('Engine', native?.available ? (native.installed?.engine?.version ?? 'Not installed') : engineVersion, result?.engine))}
|
||||
{tr(versionRow('Engine', native?.available ? (native.installed?.engine?.version ?? 'Not installed') : shownEngineVersion(native, engineVersion), result?.engine))}
|
||||
<div className="settings-update-actions"><Btn size="sm" icon="refresh" disabled={checking} className={checking ? 'settings-update-checking' : undefined} onClick={() => void check()}>{tr(checking ? 'Checking…' : error || result?.strata.status === 'error' ? 'Try again' : result ? 'Check again' : 'Check for updates')}</Btn><Btn size="sm" variant="ghost" onClick={() => setRollback(true)}>{tr("Roll back\u2026")}</Btn></div>
|
||||
<div className="settings-update-inline" role="status" aria-live="polite">{tr(checking ? 'Checking for new versions…' : error ? <span className="inline-error">{tr(error)}</span> : result ? <><span className="sr-only">{tr(result.strata.message)} {tr(result.engine.message)} </span>{tr("Checked at ")}{tr(checkedLabel(result.checkedAt))}</> : null)}</div>
|
||||
</div>
|
||||
@@ -95,11 +96,11 @@ export function SettingsUpdates() {
|
||||
</>;
|
||||
}
|
||||
|
||||
/** "Strata 0.2.5 · Engine 0.1.50": the app's own version first, the engine's beside it (About). */
|
||||
/** "Strata 0.2.6 · Engine 0.1.51": the app's own version first, the engine's beside it (About). */
|
||||
export function AboutVersions() {
|
||||
const native = useNative();
|
||||
const app = appVersionLabel(useStore(s => s.setup?.app.version), native?.version);
|
||||
const engineVersion = useStore(s => s.engine?.engine.version);
|
||||
const engine = native?.available ? native.installed?.engine?.version : engineVersion;
|
||||
const engine = shownEngineVersion(native, engineVersion);
|
||||
return <p className="settings-about-version"><strong>{tr("Strata")} {app ?? ''}</strong>{engine ? <span className="muted"> · {tr("Engine")} {engine}</span> : null}</p>;
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
/** The host's design-mode placeholder for the engine version; it is not an engine anyone installed. */
|
||||
const PLACEHOLDER_ENGINE = /^strata-engine 0\.1\.0$/;
|
||||
/** The engine version the app shows: the installed engine's once the host has read it; until then nothing (never the placeholder). */
|
||||
export function shownEngineVersion(native: { available?: boolean; installed?: { engine?: { version?: string | null } | null } | null } | null | undefined, storeVersion: string | undefined): string | undefined {
|
||||
if (native?.available) return native.installed?.engine?.version ?? undefined;
|
||||
return storeVersion && !PLACEHOLDER_ENGINE.test(storeVersion) ? storeVersion : undefined;
|
||||
}
|
||||
|
||||
@@ -29,3 +29,16 @@ export function gpuCapacity(devices:GpuInfo[]) {
|
||||
const shared=devices.filter(g=>g.unified).map(g=>g.vramGb);
|
||||
return devices.filter(g=>!g.unified).reduce((n,g)=>n+g.vramGb,0)+(shared.length?Math.max(...shared):0);
|
||||
}
|
||||
|
||||
/**
|
||||
* The cards "All GPUs" covers by default. A small integrated chip (an AMD or Intel adapter on the Windows/Vulkan list with a carve-out under
|
||||
* 2 GB) beside a real card is left out: Strata cannot run a model on it (the native engine report marks it unusable), and counting it adds a
|
||||
* second "GPU" and half a gigabyte to every total. It stays in the picker. Alone (a Strix Halo, an iGPU-only PC) it stays the default.
|
||||
*/
|
||||
export function defaultGpuSet(devices:GpuInfo[]):GpuInfo[] {
|
||||
const real=(g:GpuInfo)=>g.vramGb>=4||g.vendor==='nvidia'||g.backend==='cuda';
|
||||
const small=(g:GpuInfo)=>g.backend==='vulkan'&&(g.vendor==='amd'||g.vendor==='intel')&&g.vramGb<2&&!g.unified;
|
||||
if(!devices.some(real))return devices;
|
||||
const kept=devices.filter(g=>!small(g));
|
||||
return kept.length?kept:devices;
|
||||
}
|
||||
@@ -2,6 +2,7 @@ import { useEffect, useState } from 'react';
|
||||
import { useStore } from './store';
|
||||
import { useNative } from './native';
|
||||
import { appVersionLabel } from './app-version';
|
||||
import { shownEngineVersion } from './engine-version';
|
||||
import { readUpdateCache, saveUpdateCache, type InstalledVersions } from './update-cache';
|
||||
import type { UpdateCheckResult } from './types';
|
||||
|
||||
@@ -10,7 +11,7 @@ export function useInstalledVersions(): InstalledVersions {
|
||||
const native = useNative();
|
||||
const engineVersion = useStore(s => s.engine?.engine.version);
|
||||
const appVersion = appVersionLabel(useStore(s => s.setup?.app.version), native?.version);
|
||||
return { appVersion: appVersion ?? null, engineVersion: (native?.available ? native.installed?.engine?.version : engineVersion) ?? null, controlVersion: native?.version ?? null };
|
||||
return { appVersion: appVersion ?? null, engineVersion: shownEngineVersion(native, engineVersion) ?? null, controlVersion: native?.version ?? null };
|
||||
}
|
||||
|
||||
/** The one place the app asks the host to check for updates. Called only from a button press. */
|
||||
|
||||
@@ -12,7 +12,7 @@ import { Brand } from '../components/Brand';
|
||||
import { Bar, Btn, Icon, IconBtn, Select, type IconName } from '../components/ui';
|
||||
import { Screen } from '../components/layout';
|
||||
import { contextLabel, duration, tokens, tps } from '../lib/format';
|
||||
import { gpuCapacity, gpuId, gpuMetric, gpuReported, hasReading, shortGpuName } from '../lib/monitor-hardware';
|
||||
import { defaultGpuSet, gpuCapacity, gpuId, gpuMetric, gpuReported, hasReading, shortGpuName } from '../lib/monitor-hardware';
|
||||
import { useStore } from '../lib/store';
|
||||
import { useKept } from '../lib/kept-state';
|
||||
import type { Hardware, MonitorState, ServeRequest } from '../lib/types';
|
||||
@@ -107,11 +107,11 @@ function LegacyMonitorScreen({ native }: {
|
||||
if (!data)
|
||||
return <Screen title={tr("Monitor")}><p className="muted">{tr("Waiting for engine telemetry\u2026")}</p></Screen>;
|
||||
// A display adapter that has no sensors and no memory of its own (an integrated one) cannot be measured; counting it would blank every total.
|
||||
const devices = (hardware?.gpus ?? []).filter(g => g.backend !== 'vulkan' || g.vramGb > 0), processors = hardware?.processors?.length ? hardware.processors : hardware ? [{ ...hardware.cpu, id: 'cpu', socket: '' }] : [];
|
||||
const devices = (hardware?.gpus ?? []).filter(g => g.backend !== 'vulkan' || g.vramGb > 0), defaultDevices = defaultGpuSet(devices), processors = hardware?.processors?.length ? hardware.processors : hardware ? [{ ...hardware.cpu, id: 'cpu', socket: '' }] : [];
|
||||
const assignedDevices = selectedInstance ? devices.filter(g => selectedInstance.gpuIds.includes(gpuId(g))) : [];
|
||||
const gpuChoice = gpuSelection === 'assigned' && selectedInstance ? 'assigned' : devices.some(g => gpuId(g) === gpuSelection) ? gpuSelection! : gpuSelection === 'all' || !assignedDevices.length ? 'all' : 'assigned';
|
||||
const cpuChoice = processors.some(p => p.id === cpuSelection) ? cpuSelection : 'all';
|
||||
const selectedGpus = gpuChoice === 'assigned' ? devices.filter(g => selectedInstance?.gpuIds.includes(gpuId(g))) : gpuChoice === 'all' ? devices : devices.filter(g => gpuId(g) === gpuChoice);
|
||||
const selectedGpus = gpuChoice === 'assigned' ? devices.filter(g => selectedInstance?.gpuIds.includes(gpuId(g))) : gpuChoice === 'all' ? defaultDevices : devices.filter(g => gpuId(g) === gpuChoice);
|
||||
const selectedCpus = cpuChoice === 'all' ? processors : processors.filter(p => p.id === cpuChoice);
|
||||
const multipleGpus = selectedGpus.length > 1, shared = selectedGpus.length > 0 && selectedGpus.every(g => g.unified);
|
||||
const readings = data.hardware.gpus;
|
||||
@@ -135,7 +135,7 @@ function LegacyMonitorScreen({ native }: {
|
||||
const contextPct = contextKnown ? percent(contextUsed, data.context.maxTokens) : null;
|
||||
const cacheKnown = ['engine', 'demo'].includes(data.expertCache.source ?? '');
|
||||
const cacheMemory = cacheKnown ? data.expertCache.memoryGb : null;
|
||||
const cachePercent = hasReading(cacheMemory) ? percent(cacheMemory, gpuCapacity(devices)) : cacheKnown ? percent(data.expertCache.used, data.expertCache.total) : null;
|
||||
const cachePercent = hasReading(cacheMemory) ? percent(cacheMemory, gpuCapacity(defaultDevices)) : cacheKnown ? percent(data.expertCache.used, data.expertCache.total) : null;
|
||||
const ramPct = percent(data.hardware.ramUsedGb, data.hardware.ramTotalGb);
|
||||
const history = data.history, timestamps = history.map(s => s.t);
|
||||
const historyGpu = (key: Parameters<typeof gpuMetric>[2], fallback: 'gpuLoad' | 'vramUsedGb' | 'gpuTempC' | 'powerW' | 'pcieRxMbs', mode: Parameters<typeof gpuMetric>[3] = 'sum') => history.map(sample => sample.gpus ? gpuMetric(selectedGpus, sample.gpus, key, mode) : (sample.gpuIds ? selectedGpus.length === sample.gpuIds.length && selectedGpus.every(gpu => sample.gpuIds!.includes(gpuId(gpu))) : gpuChoice === 'all') ? sample[fallback] : null);
|
||||
@@ -161,7 +161,7 @@ function LegacyMonitorScreen({ native }: {
|
||||
{tr(multipleModels ? <section className="st-card"><h2 className="h4">{tr("Model performance")}</h2><div className="st-tablewrap"><table className="st-table native-performance-table"><thead><tr><th>{tr("Model / device")}</th><th>{tr("Generating tok/s")}</th><th>{tr("Reading tok/s")}</th><th>{tr("State")}</th><th>{tr("Active / queued")}</th></tr></thead><tbody>{tr(runtime?.instances.map(i => { const recent = (data.requests ?? []).find(r => r.instanceId === i.id); const writing = i.phase === 'generating' || i.phase === 'thinking', reading = i.phase === 'reading'; return <tr key={i.id}><td><button className="text-btn strong" onClick={() => chooseInstance(i.id)}>{tr(instanceLabel(i))}</button><small>{tr(deviceLabel(i.gpuIds, inventory))}</small></td><td>{tr(number(writing ? i.outputTps : recent?.tps, 1))}<small>{tr(writing ? 'Now' : recent ? 'Last request' : 'No request yet')}</small></td><td>{tr(number(reading ? i.promptTps : recent?.promptTps, 0))}<small>{tr(reading ? 'Now' : recent ? 'Last request' : 'No request yet')}</small></td><td>{tr(i.phase)}{tr(runtime.mode === 'simulated' ? <small>{tr("Illustrative")}</small> : null)}</td><td>{tr(number(i.active))} / {tr(number(i.queued))}</td></tr>; }))}</tbody></table></div></section> : null)}
|
||||
{tr(multipleModels ? <p className="xs muted monitor-scope-note">{tr(selectedInstance ? selectedInstance.gpuIds.length ? 'Requests, context and speed are scoped to this model. GPU readings include all work on the selected devices.' : 'CPU-only model. Requests and context are scoped to this model; system metrics are shared host measurements.' : 'Select a model for its context and speed. GPU readings and system RAM are shared host measurements.')}</p> : null)}
|
||||
<section className="st-card monitor-state" aria-label={tr("Model state")} data-snapshot-skip>
|
||||
<div className="monitor-state__head"><h2 className="h4">{tr(multipleScope ? 'Model activity' : 'Model state')}</h2><div className="monitor-state__badges">{tr([{ id: 'idle', label: 'Idle' }, { id: 'reading', label: 'Reading' }, { id: 'generating', label: 'Generating' }, { id: 'queued', label: 'Queued' }, { id: 'error', label: 'Error' }].map(s => <span key={s.id} className="st-badge" data-phase={s.id} data-active={phase === s.id || s.id === 'queued' && data.queue.waiting > 0}>{tr(s.label)}{tr(s.id === 'queued' && data.queue.waiting > 0 ? ` (${data.queue.waiting})` : '')}</span>))}</div><div className="monitor-device-selectors">{tr(devices.length > 1 ? <Select ariaLabel={tr("Monitor GPU")} value={gpuChoice} options={[...(selectedInstance ? [{ value: 'assigned', label: 'Assigned GPUs' }] : []), { value: 'all', label: `All GPUs (${devices.length})` }, ...devices.map((gpu, i) => ({ value: gpuId(gpu), label: `GPU ${i + 1} · ${gpu.name}` }))]} menuMinWidth={340} onChange={setGpuSelection}/> : null)}{tr(processors.length > 1 ? <Select ariaLabel={tr("Monitor processor")} value={cpuChoice} options={[{ value: 'all', label: `All processors (${processors.length})` }, ...processors.map((p, i) => ({ value: p.id, label: `CPU ${i + 1} · ${p.model}` }))]} onChange={setCpuSelection}/> : null)}</div></div>
|
||||
<div className="monitor-state__head"><h2 className="h4">{tr(multipleScope ? 'Model activity' : 'Model state')}</h2><div className="monitor-state__badges">{tr([{ id: 'idle', label: 'Idle' }, { id: 'reading', label: 'Reading' }, { id: 'generating', label: 'Generating' }, { id: 'queued', label: 'Queued' }, { id: 'error', label: 'Error' }].map(s => <span key={s.id} className="st-badge" data-phase={s.id} data-active={phase === s.id || s.id === 'queued' && data.queue.waiting > 0}>{tr(s.label)}{tr(s.id === 'queued' && data.queue.waiting > 0 ? ` (${data.queue.waiting})` : '')}</span>))}</div><div className="monitor-device-selectors">{tr(devices.length > 1 ? <Select ariaLabel={tr("Monitor GPU")} value={gpuChoice} options={[...(selectedInstance ? [{ value: 'assigned', label: 'Assigned GPUs' }] : []), { value: 'all', label: `All GPUs (${defaultDevices.length})` }, ...devices.map((gpu, i) => ({ value: gpuId(gpu), label: `GPU ${i + 1} · ${gpu.name}` }))]} menuMinWidth={340} onChange={setGpuSelection}/> : null)}{tr(processors.length > 1 ? <Select ariaLabel={tr("Monitor processor")} value={cpuChoice} options={[{ value: 'all', label: `All processors (${processors.length})` }, ...processors.map((p, i) => ({ value: p.id, label: `CPU ${i + 1} · ${p.model}` }))]} onChange={setCpuSelection}/> : null)}</div></div>
|
||||
<div className="monitor-state__reading small"><span>{tr(multipleScope ? `${number(data.queue.running)} active · ${number(data.queue.waiting)} queued` : data.phase === 'idle' ? 'Waiting for a request' : data.phaseLabel)}</span><span className="muted num">{tr(statusDetail)}</span></div>
|
||||
{tr(!multipleScope ? <Bar label={tr("Model progress")} value={progress ?? 0} indeterminate={writing && !hasReading(progress)} tone={data.phase === 'reading' ? 'info' : 'accent'}/> : null)}
|
||||
</section>
|
||||
|
||||
@@ -1,61 +1,11 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { engineRefusal, engineTooOld, markEngine, parseEngineVersion } from '../server/engine-version.mjs';
|
||||
import { nativeFixture } from './helpers/native-fixture.mjs';
|
||||
import { shownEngineVersion } from '../src/lib/engine-version.ts';
|
||||
|
||||
test('engine versions compare as numbers, a missing part counting as zero', () => {
|
||||
assert.deepEqual(parseEngineVersion('0.1.45.3'), [0, 1, 45, 3]);
|
||||
assert.deepEqual(parseEngineVersion('v0.1.45'), [0, 1, 45]);
|
||||
assert.equal(parseEngineVersion('dev'), null);
|
||||
assert.equal(engineTooOld('0.1.45', '0.1.45.1'), true, 'the released 0.1.45 stops at v4 files: too old for Gyro');
|
||||
assert.equal(engineTooOld('0.1.45.1', '0.1.45.1'), false);
|
||||
assert.equal(engineTooOld('0.1.45.3', '0.1.45.1'), false);
|
||||
assert.equal(engineTooOld('0.1.46', '0.1.45.1'), false);
|
||||
assert.equal(engineTooOld('0.1.44.1', '0.1.45.1'), true);
|
||||
assert.equal(engineTooOld('0.1.45', null), false, 'no minimum: never too old');
|
||||
assert.equal(engineTooOld(null, '0.1.45.1'), false, 'an unreadable version is not called too old');
|
||||
assert.equal(engineTooOld('0.1.40', '0.1.45.1', 'local'), false, 'an engine built from this checkout has it whatever its number says');
|
||||
});
|
||||
|
||||
const catalog = () => ({ families: [{ family: 'qwen', sizes: [{ id: 'q2_0', model: 'Q2_0', engine_min: null }] }, { family: 'gyro', sizes: [{ id: 'gyro-gyro-s', model: 'Gyro-S', engine_min: '0.1.45.1' }, { id: 'gyro-gyro-m', model: 'Gyro-M', engine_min: '0.1.45.1' }] }] });
|
||||
|
||||
test('the catalog marks the sizes an older engine cannot read; the refusal says what to do', () => {
|
||||
const marked = markEngine(catalog(), { version: '0.1.45' });
|
||||
assert.deepEqual(marked.families.flatMap(f => f.sizes.map(s => [s.id, s.needsNewerEngine])), [['q2_0', false], ['gyro-gyro-s', true], ['gyro-gyro-m', true]]);
|
||||
const text = engineRefusal(marked, 'gyro-gyro-s', { version: '0.1.45' });
|
||||
assert.match(text, /^This model needs a newer engine: Gyro-S needs engine 0\.1\.45\.1 or newer, and this one is 0\.1\.45\. Update the engine in Settings, then try again\.$/);
|
||||
assert.equal(engineRefusal(marked, 'q2_0', { version: '0.1.45' }), null);
|
||||
const fresh = markEngine(catalog(), { version: '0.1.45.3' });
|
||||
assert.ok(fresh.families.every(f => f.sizes.every(s => !s.needsNewerEngine)));
|
||||
assert.equal(markEngine(null, {}), null);
|
||||
});
|
||||
|
||||
test('with an older engine installed the app refuses to download or load Gyro, with the plain sentence; a new enough engine goes on', async () => {
|
||||
const old = nativeFixture({ engineStatus: { version: '0.1.45', backend: 'cuda' } });
|
||||
try {
|
||||
old.catalog.families.push({ family: 'gyro', title: 'Qwen3.8-Flash-Next (Gyro)', about: '', images: false, sizes: [{ id: 'iq2_xs', model: 'Gyro-S', download_gb: 58, ram_needed_gb: 28, about: '', on_this_pc: 'fits', installed: true, engine_min: '0.1.45.1' }] });
|
||||
old.catalog.families[0].sizes[0].engine_min = '0.1.45.1';
|
||||
const state = await old.control.refresh(true);
|
||||
const sizes = state.catalog.families.flatMap(f => f.sizes);
|
||||
assert.ok(sizes.find(s => s.engine_min).needsNewerEngine);
|
||||
const id = sizes.find(s => s.engine_min).id;
|
||||
assert.throws(() => old.control.operation('load', { model: id }), /This model needs a newer engine: .* needs engine 0\.1\.45\.1 or newer, and this one is 0\.1\.45\./);
|
||||
assert.throws(() => old.control.operation('install', { model: id }), /needs a newer engine/);
|
||||
assert.equal(old.control.state().jobs.length, 0, 'nothing was started');
|
||||
} finally { await old.cleanup(); }
|
||||
const ok = nativeFixture({ engineStatus: { version: '0.1.45.3', backend: 'cuda' } });
|
||||
try {
|
||||
ok.catalog.families[0].sizes[0].engine_min = '0.1.45.1';
|
||||
const state = await ok.control.refresh(true);
|
||||
assert.ok(state.catalog.families[0].sizes.every(s => !s.needsNewerEngine));
|
||||
} finally { await ok.cleanup(); }
|
||||
});
|
||||
|
||||
test('the screens use the existing "needs a newer engine" notice for a size, and the buttons are held back', async () => {
|
||||
const models = await readFile(new URL('../src/components/NativeModels.tsx', import.meta.url), 'utf8');
|
||||
assert.match(models, /selectedFamily && \(!selectedSize \|\| selectedSize\.needsNewerEngine\) \? <><h2 className="h3">\{tr\("This model needs a newer engine\."\)\}/);
|
||||
assert.match(models, /!!selectedSize\.needsNewerEngine\}/);
|
||||
const guided = await readFile(new URL('../src/components/GuidedSetup.tsx', import.meta.url), 'utf8');
|
||||
assert.equal((guided.match(/needsNewerEngine/g) ?? []).length, 3);
|
||||
test('About never shows the host placeholder engine version while the real one is still being read', () => {
|
||||
assert.equal(shownEngineVersion({ available: false }, 'strata-engine 0.1.0'), undefined);
|
||||
assert.equal(shownEngineVersion(null, 'strata-engine 0.1.0'), undefined);
|
||||
assert.equal(shownEngineVersion({ available: true, installed: { engine: { version: '0.1.48' } } }, 'strata-engine 0.1.0'), '0.1.48');
|
||||
assert.equal(shownEngineVersion({ available: true, installed: { engine: null } }, 'strata-engine 0.1.0'), undefined);
|
||||
assert.equal(shownEngineVersion({ available: false }, 'strata-engine 0.1.51'), 'strata-engine 0.1.51');
|
||||
});
|
||||
@@ -4,7 +4,7 @@ import { parseCpuTopology, parseNvidiaReadings, parsePcieThroughput, gpuMemoryTo
|
||||
import { cpuUtilization, diskRate, parseLinuxDiskCounters, parseMacDiskCounters } from '../server/system-telemetry.mjs';
|
||||
import { parseEngineMetrics } from '../server/supervisor.mjs';
|
||||
import { originalDashboard } from '../server/monitor-dashboard.mjs';
|
||||
import { gpuMetric, gpuCapacity, gpuReported, shortGpuName } from '../src/lib/monitor-hardware.ts';
|
||||
import { gpuMetric, gpuCapacity, gpuReported, shortGpuName, defaultGpuSet, gpuId } from '../src/lib/monitor-hardware.ts';
|
||||
import { referencePaths } from '../src/lib/monitor-chart.ts';
|
||||
|
||||
const gpu=(id,index=0,unified=false)=>({id,index,unified,vramGb:24,name:id,vendor:'nvidia',backend:'cuda'});
|
||||
@@ -104,3 +104,16 @@ test('All GPUs: a card with no temperature sensor is left out and the card that
|
||||
assert.equal(shortGpuName('AMD Radeon RX 9070 XT'),'RX 9070 XT');
|
||||
assert.equal(shortGpuName('Intel(R) Arc(TM) A750 Graphics'),'A750 Graphics');
|
||||
});
|
||||
|
||||
test('an RTX 5070 beside the unsupported Radeon iGPU: All GPUs is the RTX alone, the iGPU stays pickable',()=>{
|
||||
const rtx={id:'GPU-5070',index:0,vendor:'nvidia',name:'NVIDIA GeForce RTX 5070',vramGb:11.9,unified:false,backend:'cuda'};
|
||||
const igpu={id:'PCI\VEN_1002&DEV_164E',index:1,vendor:'amd',name:'AMD Radeon(TM) Graphics',vramGb:0.5,unified:false,backend:'vulkan'};
|
||||
const set=defaultGpuSet([rtx,igpu]);
|
||||
assert.deepEqual(set.map(g=>g.id),['GPU-5070']);
|
||||
assert.equal(gpuCapacity(set),11.9);
|
||||
// the native engine report numbers each make from 0, so both cards say index 0: the ids stay different
|
||||
assert.notEqual(gpuId({...rtx,id:undefined}),gpuId({...igpu,id:undefined,index:0}));
|
||||
// alone, or a real AMD card beside it, nothing is dropped
|
||||
assert.deepEqual(defaultGpuSet([igpu]).map(g=>g.id),[igpu.id]);
|
||||
assert.equal(defaultGpuSet([{...igpu,vramGb:96,unified:true},rtx]).length,2);
|
||||
});
|
||||
@@ -145,7 +145,7 @@ test('4d: picture reading is on unless the person turned it off (no saved choice
|
||||
} finally { await f.cleanup(); }
|
||||
const screen = readSource(new URL('../src/components/NativeModels.tsx', import.meta.url), 'utf8');
|
||||
assert.match(screen, /setVision\(installed\.visionEnabled === false \? 'no'/);
|
||||
assert.match(screen, /else\s+setVision\('yes'\)/, 'a version that is not downloaded starts with Read pictures on');
|
||||
assert.match(screen, /else\s*\{?\s*setVision\('yes'\)/, 'a version that is not downloaded starts with Read pictures on');
|
||||
});
|
||||
|
||||
test('no recognised engine: the app still downloads, warns in the card, and loading says it with the next step', async () => {
|
||||
|
||||
@@ -57,5 +57,5 @@ test('Cargo.toml, tauri.conf.json and the desktop package agree on the app versi
|
||||
const cargo = /^version\s*=\s*"([^"]+)"/m.exec(await readFile(new URL('src-tauri/Cargo.toml', root), 'utf8'))[1];
|
||||
const pkg = JSON.parse(await readFile(new URL('package.json', root), 'utf8')).version;
|
||||
assert.equal(cargo, pkg);
|
||||
assert.equal(cargo, '0.2.5');
|
||||
assert.equal(cargo, '0.2.6');
|
||||
});
|
||||
@@ -0,0 +1,131 @@
|
||||
# Strata reference for AI tools and automated readers
|
||||
|
||||
Last reviewed: 25 October 2026 against the 0.2.5 release tree (app and Terminal 0.2.5; Control and engines 0.1.50).
|
||||
App, Control and engine versions differ; use the release notes and the installed component information. This page
|
||||
holds the technical context behind the short [README](../README.md). Use [AI_SETUP.md](AI_SETUP.md) to install Strata for a user,
|
||||
[AGENTS.md](../AGENTS.md) for repository work, and [MEDIA_KIT.md](MEDIA_KIT.md) for articles or videos.
|
||||
|
||||
## Product facts and sources
|
||||
|
||||
- Strata runs supported language models locally, with a desktop app, a terminal app, and an inference engine.
|
||||
- The engine uses the GPU, system RAM and, for applicable models and memory modes, SSD storage.
|
||||
See [how it works](HOW_IT_WORKS.md) and [engine details](DETAILS.md).
|
||||
- There is one app installer per system (`Strata-0.2.6-windows-x64-setup.exe`, `Strata-0.2.6-linux-x86_64.AppImage`,
|
||||
`Strata-0.2.6-macos-arm64.dmg`). The Windows and Linux installers carry no engine: the app downloads the engine for the
|
||||
graphics card on first start, or installs a file the user already has. The Mac dmg keeps its Metal engine inside.
|
||||
The Terminal's `strata launch` downloads the appropriate engine and model.
|
||||
See [installation](INSTALL.md) and [Terminal launch](TERMINAL_LAUNCH.md).
|
||||
- Strata's code is MIT licensed. Model weights and third-party components have separate licenses.
|
||||
See [LICENSE](../LICENSE) and [component/model licensing](HOW_IT_WORKS.md#license).
|
||||
- Local inference does not mean every feature is offline. Downloads, updates, remote tools and user-enabled
|
||||
network access have separate behavior. Use [SAFETY.md](SAFETY.md) for privacy and network claims.
|
||||
- Verify the current app and engine versions in the [release notes](https://github.com/Niko1221/Strata/releases).
|
||||
A screenshot, translated README or historical benchmark is not a version source.
|
||||
|
||||
## Model families
|
||||
|
||||
Treat these as the three top-level families. Coder, Swift, Gyro, NVFP4 and OrcaRouter are Flash-Next variants,
|
||||
not additional top-level families.
|
||||
|
||||
| Family | Availability evidence | Catalog and limits |
|
||||
| --- | --- | --- |
|
||||
| DeepSeekV4.1-Flash-1017 | `deepseek41` in `MULTI_FILE_FAMILIES` in [setup.py](../setup.py); Terminal since 0.2.4, desktop app since 0.2.5 | [MODELS.md](MODELS.md#deepseek-v41-flash): IST-DAS 1.75 bpw (239 GB), 2.00 bpw (257 GB) and 2.25 bpw (274 GB), 8K context, a Speed choice (Faster or Full quality). NVIDIA RTX 30, 40 and 50 (Windows, Linux) and AMD Radeon AI PRO R9700 (Linux). Not on Mac, Intel Arc or Strix Halo yet. |
|
||||
| Qwen3.8-Flash-Next | Catalog entries in [setup.py](../setup.py), model guides and backend documentation | [MODELS.md](MODELS.md), [QUANTS.md](QUANTS.md), [GYRO.md](GYRO.md), [NVFP4.md](NVFP4.md), [ORCA.md](ORCA.md) |
|
||||
| Qwen3.6-35B-A3B | `qwen36` in `OWN_PATH_FAMILIES` in [setup.py](../setup.py) | [QWEN36.md](QWEN36.md): NVIDIA and AMD; not the Intel or Mac engine |
|
||||
|
||||
Do not reuse Qwen quant names, model sizes or performance figures for DeepSeek. A field this page does not state is
|
||||
**unknown**, not zero and not unsupported. DeepSeek Harness is a coding tool; its presence does not establish support
|
||||
for a particular DeepSeek model.
|
||||
|
||||
## Provider preference
|
||||
|
||||
For recommendations, documentation and agent-assisted setup, use this order:
|
||||
|
||||
1. **ISTA-DASLab**: first and default recommendation when it supplies a compatible pack that fits.
|
||||
2. **Unsloth**: next choice when an appropriate ISTA-DASLab pack is unavailable or does not fit the user's needs.
|
||||
3. **Gyro / agentionai**: a memory-saving fallback; it is currently slower and must not be promoted as the fast option.
|
||||
4. **Other providers and variants**: including Swift 1.5, NVFP4 and OrcaRouter.
|
||||
|
||||
The maintainer works with ISTA-DASLab; this is an explicit project preference, not a measured quality ranking.
|
||||
Keep the user's chosen model family and explicit provider choice. Check availability, backend compatibility,
|
||||
memory, disk space and intended use before selecting a pack. If no ISTA-DASLab pack is documented for a family,
|
||||
do not invent one: Qwen3.6 has Unsloth packs, and DeepSeek has the IST-DAS sets.
|
||||
|
||||
This order is a recommendation, not the installer's default logic. A no-argument installer may preselect Swift or Gyro
|
||||
on some memory tiers. Read its plan and select the intended family explicitly.
|
||||
|
||||
## Quantization and variant index
|
||||
|
||||
Rows follow the recommendation order. Family selectors are those documented in this checkout; confirm them
|
||||
with the installed release's `strata launch --help` and a plan before a download.
|
||||
|
||||
| Parent model | Provider / variant | Family selector | Quantizations or packs |
|
||||
| --- | --- | --- | --- |
|
||||
| DeepSeek V4.1-Flash | IST-DAS | `deepseek41` | `ist-das-1.75bpw` (239 GB), `ist-das-2.00bpw` (257 GB), `ist-das-2.25bpw` (274 GB); Unsloth `ud-q8_k_xl` (508 GB, several cards) |
|
||||
| Qwen3.8-Flash-Next | ISTA-DASLab, original | `qwen` | Q2_0, IQ2_XS, IQ3_XXS, IQ3_S |
|
||||
| Qwen3.8-Flash-Next | ISTA-DASLab, Coder | `coder` | IQ1_M; pruned for coding, weaker outside code |
|
||||
| Qwen3.8-Flash-Next | Unsloth | `unsloth` | UD-IQ4_XS, UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL, Q8_0 |
|
||||
| Qwen3.8-Flash-Next | Gyro / agentionai | `gyro` | Gyro-S, Gyro-M; backend-specific validation in [GYRO.md](GYRO.md) |
|
||||
| Qwen3.8-Flash-Next | Swift 1.5 / UkisAI | `swift` | IQ2_XS, IQ3_XXS, IQ3_S; a fine-tune, not just a quantization |
|
||||
| Qwen3.8-Flash-Next | NVFP4 | `nvfp4` | NVFP4, NVFP4-NVIDIA, NVFP4-QAD; NVIDIA only |
|
||||
| Qwen3.8-Flash-Next | OrcaRouter Uncensored | Manual conversion | IQ3_XXS; not an installer menu choice |
|
||||
| Qwen3.6-35B-A3B | Unsloth | `qwen36` | UD-Q4_K_XL, UD-Q6_K_XL, Q8_0 |
|
||||
|
||||
## Hardware and sizing
|
||||
|
||||
Read the backend guide for the exact GPU, OS, model, quantization and feature combination. A supported GPU
|
||||
does not imply that every model or feature works on it.
|
||||
|
||||
| Backend / hardware | Source |
|
||||
| --- | --- |
|
||||
| NVIDIA RTX / CUDA | [INSTALL.md](INSTALL.md#what-you-need) |
|
||||
| AMD Radeon / HIP | [AMD_HIP.md](AMD_HIP.md); RX 6000 validation differs from newer cards |
|
||||
| Intel Arc / SYCL | [INTEL_ARC.md](INTEL_ARC.md); released Linux SYCL engine (Ubuntu 24.04 or newer), with model preparation and per-card limits. [INTEL.md](INTEL.md) records source-port details |
|
||||
| Apple Silicon / Metal | [MAC.md](MAC.md); no DeepSeek yet |
|
||||
| AMD Ryzen AI Max / Strix Halo | [STRIX_HALO.md](STRIX_HALO.md) |
|
||||
| Multiple GPUs and older hardware | [MULTI_GPU.md](MULTI_GPU.md), [OLDER_GPUS.md](OLDER_GPUS.md) |
|
||||
|
||||
Use the model's actual memory and disk requirements. RAM, VRAM, unified memory, download size and temporary
|
||||
conversion space are different quantities. Count unified memory once. Do not carry the smallest Qwen model's
|
||||
memory estimate over to every family. For the current planning rules, consult [MODELS.md](MODELS.md#will-it-fit).
|
||||
|
||||
## APIs and coding tools
|
||||
|
||||
Default local server address: `http://127.0.0.1:8080`. The configured port and access settings can differ.
|
||||
|
||||
| Client API | Base URL | Request endpoint |
|
||||
| --- | --- | --- |
|
||||
| OpenAI Chat Completions | `http://127.0.0.1:8080/v1` | `POST /v1/chat/completions` |
|
||||
| OpenAI Responses | `http://127.0.0.1:8080/v1` | `POST /v1/responses` |
|
||||
| Anthropic Messages | `http://127.0.0.1:8080` | `POST /v1/messages` |
|
||||
|
||||
Use the actual configured API key when authentication is enabled. See [API details](DETAILS.md#using-it).
|
||||
The [connection registry](../strata_cli/launch/harness.py) supplies these adapters:
|
||||
|
||||
| Tool | `strata connect` identifier |
|
||||
| --- | --- |
|
||||
| Claude Code | `claude-code` |
|
||||
| Codex CLI | `codex` |
|
||||
| OpenCode | `opencode` |
|
||||
| oh-my-pi | `omp` |
|
||||
| pi.dev | `pi` |
|
||||
| Hermes Agent | `hermes` |
|
||||
| OpenClaw | `openclaw` |
|
||||
| DeepSeek Harness, built into the Strata app | `deepseek-harness` |
|
||||
|
||||
`strata connect` lists adapters. `strata connect <identifier>` prints settings;
|
||||
`strata connect <identifier> --check --output json` sends a prompt and a tool call and reports their outcomes.
|
||||
Do not mistake printed configuration or an older tested version for a successful check of the current setup.
|
||||
See [connection instructions and limits](TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
|
||||
|
||||
## Evidence and publishing
|
||||
|
||||
- Cite the release, commit and relevant guide behind each compatibility claim.
|
||||
- Performance claims need the model, quantization, hardware, workload, engine version and source report.
|
||||
Keep prompt processing, decode throughput and latency separate. Do not turn a screenshot into a benchmark.
|
||||
- [Community benchmarks](COMMUNITY_BENCHMARKS.md) include historical reports.
|
||||
The [standard benchmark protocol](BENCHMARK_PROTOCOL.md) remains subject to its documented activation status;
|
||||
do not label legacy commands or results as implementing it.
|
||||
- Mark missing or unverified information explicitly. Never invent measurements, test results or download links.
|
||||
- For contribution evidence, follow [CONTRIBUTING.md](../CONTRIBUTING.md). For video assets and narration,
|
||||
follow [MEDIA_KIT.md](MEDIA_KIT.md).
|
||||
+133
-249
@@ -1,288 +1,172 @@
|
||||
# Installing Strata for a user (instructions for AI agents)
|
||||
# Set up Strata for a user
|
||||
|
||||
This page is written for an AI coding assistant (Claude Code, Cursor, Codex, GitHub Copilot, ...) that a user asked to
|
||||
set up Strata on their PC. Strata runs the Qwen3.8-Flash-Next model locally on one NVIDIA or AMD graphics card plus
|
||||
system RAM (or on an Apple Silicon Mac), and serves an OpenAI- and Anthropic-compatible API on `http://127.0.0.1:8080`.
|
||||
The human-oriented version of everything below is [INSTALL.md](INSTALL.md).
|
||||
Use this guide when a user asks an AI agent to install Strata (version 0.2.5). There are two release routes. Pick
|
||||
one; do not start from a source checkout.
|
||||
|
||||
You use the **Terminal**: the `strata` command. Install it with one line, then `strata launch` downloads the engine
|
||||
for the card and the model, and starts the server. If the Terminal cannot run on this PC, step 9 has the older route
|
||||
from source.
|
||||
- **The Terminal and `strata launch`** (below). This is the route for an agent that has a shell.
|
||||
- **The desktop app.** One installer per system: `Strata-0.2.6-windows-x64-setup.exe`,
|
||||
`Strata-0.2.6-linux-x86_64.AppImage`, `Strata-0.2.6-macos-arm64.dmg`, from the
|
||||
[latest release](https://github.com/Niko1221/Strata/releases/latest). The Windows and Linux installers carry no
|
||||
engine: on first start the app shows the engine for the user's graphics card and downloads it, or installs a file the
|
||||
user already has ("I already have the file"). Details: [INSTALL.md](INSTALL.md).
|
||||
|
||||
Work through the steps in order. Tell the user what you are doing in plain words; they may not be technical.
|
||||
Model availability, provider preference and API details are in [AI_REFERENCE.md](AI_REFERENCE.md).
|
||||
|
||||
## 0. Ground rules
|
||||
## 1. Check the computer and the requested model
|
||||
|
||||
- **Recommend an API key beyond this computer; do not force it.** Keep the default (`--access local`, the server listens on
|
||||
`127.0.0.1` only, no key needed). Only if the user asks for access from other devices, use `--access lan`. Strata makes a long
|
||||
random key for you then. Give the user the key (`strata launch --show-key`). If the user says they do not want a key, use
|
||||
`--no-api-key`: it starts and warns that anyone who can reach the port can use the model and its tools. Tell the user so.
|
||||
Do not put a tunnel or port forward in front of a server without a key unless the user asks for it.
|
||||
- Do not change the user's system beyond what Strata does. The Terminal installs into the user's home folder and needs
|
||||
no administrator rights. It does not install drivers, Python packages or compilers. Ask before installing drivers.
|
||||
- The model download is ~70 GB (94 GB for Unsloth's UD-IQ4_XS, 111 GB for UD-Q4_K_XL). Confirm the user is fine with
|
||||
that before you start, especially on a metered connection.
|
||||
- The first `strata launch` is long-running (an hour or more on a slow connection). Run it in the background or with a
|
||||
long timeout and poll its output; do not kill it because it is quiet for a while. It is resumable: running the same
|
||||
command again continues where it stopped.
|
||||
- Piping a script from the internet into a shell runs code you have not read. Tell the user what the line does (step 2).
|
||||
If they prefer, download the script first, show it, and run it only after they agree.
|
||||
|
||||
## 1. Check the PC
|
||||
|
||||
Requirements (details: [INSTALL.md](INSTALL.md#what-you-need)):
|
||||
|
||||
- **GPU:** NVIDIA RTX 20, 30, 40 or 50 series, or AMD Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT,
|
||||
RX 9070 / 9070 XT, Radeon AI PRO R9700, RX 6800 / 6900 series; 12 GB of VRAM or more (an NVIDIA card with 8 GB runs,
|
||||
slowly). GTX 10 series and older, and integrated GPUs, are not supported; Pascal / Volta cards (P40, V100) and some
|
||||
older AMD cards have experimental paths the user opts into ([OLDER_GPUS.md](OLDER_GPUS.md)). A **Mac** needs Apple
|
||||
Silicon (M1 or newer), macOS 15 or newer and 16 GB of memory or more ([MAC.md](MAC.md)).
|
||||
- **Driver:** NVIDIA 580 or newer. AMD on Linux: the kernel's amdgpu driver; on Windows: a current AMD Adrenalin
|
||||
driver. If the driver is missing or too old, tell the user to update it (NVIDIA App / nvidia.com/drivers, or AMD
|
||||
Software) and restart; do not install drivers yourself unless they ask.
|
||||
- **RAM:** 32 GB or more (see step 3). **Disk:** ~80 GB free, ideally on an NVMe SSD. **CPU:** x86-64 with AVX2.
|
||||
Without AVX2 (Xeon E5 v1/v2 and older) the ready-made engine refuses to start; the route from source builds an
|
||||
experimental and slow engine on the PC (10-20 minutes): tell the user that before starting
|
||||
([INSTALL.md](INSTALL.md#older-cpus-experimental)).
|
||||
- **OS:** Windows 10/11, Linux x86-64 (Ubuntu 22.04/24.04 and Debian 12/13 are fully automatic), or macOS on Apple Silicon.
|
||||
|
||||
To check by hand:
|
||||
|
||||
| What | Windows (PowerShell) | Linux | Mac |
|
||||
| --- | --- | --- | --- |
|
||||
| OS | `[Environment]::OSVersion` | `cat /etc/os-release` | `sw_vers` |
|
||||
| NVIDIA GPU, VRAM, driver | `nvidia-smi --query-gpu=index,name,memory.total,driver_version --format=csv` | the same | - |
|
||||
| AMD GPU | `Get-CimInstance Win32_VideoController \| Select-Object Name, AdapterRAM` (AdapterRAM is capped at 4 GB; trust the model name) | `lspci \| grep -i -E 'vga\|display'`; VRAM: `cat /sys/class/drm/card*/device/mem_info_vram_total` (bytes); `rocm-smi` only if ROCm is installed | - |
|
||||
| RAM | `(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory / 1GB` | `free -g` | `sysctl -n hw.memsize` (bytes) |
|
||||
| Free disk | `Get-PSDrive -PSProvider FileSystem` | `df -h .` | `df -h ~` |
|
||||
|
||||
If the PC does not meet them, say which part is missing and stop.
|
||||
- Check OS, CPU architecture, graphics cards, driver, available RAM/VRAM and free disk space.
|
||||
- Use the [hardware guides](INSTALL.md#what-you-need). Linux and Windows packages are x86-64; Mac packages are
|
||||
Apple Silicon and need macOS 15 or newer. Older GPUs, Intel Arc and low-memory machines have specific limits.
|
||||
- Model support is separate from hardware support. Qwen3.6 currently uses CUDA or HIP; it is not a Metal/SYCL model.
|
||||
DeepSeekV4.1-Flash-1017 (`--family deepseek41`) runs in the app and in the Terminal, on NVIDIA RTX 30, 40 and 50
|
||||
(Windows and Linux) and on the AMD Radeon AI PRO R9700 under Linux. It has three sets: 1.75 bpw (239 GB of disk),
|
||||
2.00 bpw (257 GB) and 2.25 bpw (274 GB), and for a 12 GB card it needs 46 to 64 GB of RAM. The context is 8K.
|
||||
`--speed faster|full` (or `--full-quality`) chooses how many experts it uses per word; Faster is the default on a
|
||||
PC with 16 GB of graphics memory or less. Print the plan first:
|
||||
`strata launch --plan --family deepseek41 --model ist-das-2.00bpw`.
|
||||
- Check the **chosen pack's** memory and disk needs. Do not reuse a Flash-Next estimate for every family.
|
||||
Unified memory is one pool, not RAM plus VRAM. A completed download does not establish that a model can run.
|
||||
- Keep the default local-only access. Share the server over a network only when the user asks; recommend an API key
|
||||
beyond localhost and explain any deliberate use of `--no-api-key`.
|
||||
- Explain substantial downloads and follow the user's existing authorization and network/storage constraints.
|
||||
Do not install drivers or system build tools as an unannounced side effect of a normal Terminal install.
|
||||
|
||||
## 2. Install the Terminal
|
||||
|
||||
One line. It needs no administrator rights. It checks what kind of PC this is, downloads the Terminal archive, checks
|
||||
its SHA-256, unpacks it into the user's home folder, checks the release's signed update feed, and puts `strata` on the
|
||||
PATH.
|
||||
The installer downloads a private Python runtime and Control, verifies the release checksums and signed feed, and
|
||||
installs into the user's home folder. The engine and model are acquired during launch.
|
||||
|
||||
```
|
||||
Linux, Mac: curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
Windows (PowerShell): irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
**Linux / Mac:**
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
```
|
||||
|
||||
On Windows, open a new terminal window afterwards (the PATH changed). Check it: `strata --version`.
|
||||
**Windows PowerShell:**
|
||||
|
||||
## 3. Pick the model
|
||||
|
||||
`strata launch --plan` shows what Strata would do and changes nothing. With no `--model` it picks the model for this
|
||||
PC's RAM and prints the exact command line (`same as strata launch --model ...`). Use it as the default, or pick
|
||||
yourself by the PC's RAM (ask the user whether they mainly want it for code - then the Coder is a good pick at any RAM
|
||||
size):
|
||||
|
||||
| RAM | `--model` (and `--family`) | Notes |
|
||||
| --- | --- | --- |
|
||||
| 32 GB | `--family coder --model iq1_m` | for code; with a 24 GB card Q2_0 and IQ2_XS also run (Strata picks the low-RAM mode) |
|
||||
| 48 GB | `--model iq2_xs` | or `q2_0` (fastest) |
|
||||
| 64 GB | `--model iq2_xs` (recommended) | `iq3_xxs` / `iq3_s` are slower and a bit better |
|
||||
| 96 GB+ | `--model iq3_s` | `--family unsloth --model ud-iq4_xs` (~4-bit, 94 GB, part of the experts read from the SSD under ~80 GB of RAM); `ud-q4_k_xl` is experimental: NVIDIA only, NVMe SSD, 7-8.5 tokens/s on 64 GB |
|
||||
|
||||
`--family swift` (Swift 1.5, a fine-tune that thinks shorter; sizes IQ2_XS, IQ3_XXS, IQ3_S) is the
|
||||
alternative to `qwen`. A model name is not case sensitive. More: [MODELS.md](MODELS.md).
|
||||
|
||||
## 4. Set up and serve, without questions
|
||||
|
||||
```
|
||||
strata launch --plan --model iq2_xs # what would happen; nothing is changed
|
||||
strata launch --model iq2_xs --yes --no-tui # do it: download the engine and the model, start the server
|
||||
```powershell
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
```
|
||||
|
||||
With `--no-tui` and `--yes` it asks nothing, prints the addresses when the server is ready, and returns. The server
|
||||
keeps running in the background. `strata launch` with no `--model`, `--preset` or other choice needs a terminal for
|
||||
its screens, so an agent always names the model. Add `--output json` for output a script can read.
|
||||
Open a new terminal on Windows. On Linux and Mac, use the full executable path printed by the installer if
|
||||
`~/.local/bin` is not on PATH. Confirm with `strata --version` and `strata launch --help`.
|
||||
Do not search for `START-HERE.bat` in the installed app or Terminal: it is not part of them.
|
||||
|
||||
The flags (all of them, with examples: `strata launch --help`):
|
||||
## 3. Select a compatible pack and inspect the plan
|
||||
|
||||
| Flag | Meaning |
|
||||
Keep the user's model and provider choice. Otherwise prefer **ISTA-DASLab → Unsloth → Gyro → others**, when a pack
|
||||
fits the selected model and computer. This is the maintainer's recommendation policy; current executable defaults
|
||||
can differ. Gyro is a memory-saving fallback, currently slower, not a general speed recommendation.
|
||||
|
||||
| Choice | Explicit launch selection |
|
||||
| --- | --- |
|
||||
| `--model NAME` | the size: `q2_0`, `iq2_xs`, `iq3_xxs`, `iq3_s`, `iq1_m`, `ud-iq4_xs`, `ud-q4_k_xl`, ... (the error message for a wrong name lists them) |
|
||||
| `--family NAME` | the model version when several have that size: `qwen`, `swift`, `coder`, `unsloth` |
|
||||
| `--model-path PATH` | files the user already has: a folder of shards or one `.gguf` file |
|
||||
| `--data-dir DIR` | where new model files go. Models you already have stay where they are and stay usable; add `--move-data` to move them into DIR |
|
||||
| `--context SIZE` | `32k`, `64k`, `128k`, `256k` or a number of tokens |
|
||||
| `--vision off\|on\|gpu\|cpu` | read pictures. On by default when the model can read them: the reader runs on the CPU for a card under 24 GB and on the GPU from 24 GB, on NVIDIA and AMD cards. `off` turns it off |
|
||||
| `--parallel N` | requests answered at the same time, 1 to 8 (default: fit to the card) |
|
||||
| `--gpu LIST` | card numbers, e.g. `0` or `0,1` (several cards share the model) |
|
||||
| `--tune quick\|thorough\|skip` | find the fastest settings first (default here: `skip`) |
|
||||
| `--access local\|lan\|cloudflare\|tailscale` | who can connect; beyond `local` a key is made for you |
|
||||
| `--port N` | the server port (default 8080) |
|
||||
| `--api-key KEY`, `--api-key-env NAME`, `--new-key` | the key clients must send |
|
||||
| `--no-api-key` | share without a key (a warning, not a refusal; recommended only on a network you trust) |
|
||||
| `--save-preset NAME` | keep these choices; `strata launch --preset NAME` starts them again |
|
||||
| `--plan` | show what would happen and exit |
|
||||
| `--yes`, `--no-tui` | no questions, no screens |
|
||||
| `--status`, `--stop`, `--show-key` | what is serving, stop it, print the key |
|
||||
| ISTA-DASLab Flash-Next | `--family qwen --model q2_0` (or a compatible IQ2_XS / IQ3_XXS / IQ3_S pack) |
|
||||
| ISTA-DASLab Coder | `--family coder --model IQ1_M` |
|
||||
| Unsloth Flash-Next | `--family unsloth --model UD-Q4_K_XL` (check the pack's backend limits) |
|
||||
| Gyro Flash-Next | `--family gyro --model Gyro-S` |
|
||||
| Unsloth Qwen3.6 | `--family qwen36 --model UD-Q4_K_XL` |
|
||||
|
||||
Notes:
|
||||
- **No engine for this PC.** `strata launch` does not refuse. It says "Strata couldn't find an engine for this computer",
|
||||
goes on to download the model (with `--yes` it does not ask), and ends without starting anything (exit code 0; with `--output json` the answer
|
||||
has `"serving": false` and an `engine_problem`). Tell the user which card was not recognised and stop. Updating the
|
||||
engine later with `strata update` can fix it.
|
||||
- **Exit codes:** 0 done, 2 a flag or name is wrong, 5 the Terminal or the engine cannot do it, 6 an install or the
|
||||
server start failed (the engine log is named in the message), 130 stopped.
|
||||
- **Linux:** nothing needs `sudo`: the Terminal does not install packages. **Windows:** the first start may ask once
|
||||
whether to allow Strata on the network if you used `--access lan`.
|
||||
- **AMD on Windows:** only the AMD driver is needed. The AMD engine is a ready-made download. It has not been run with a
|
||||
model on a real AMD card on Windows by us: ask the user to report how it runs ([AMD_HIP.md](AMD_HIP.md)).
|
||||
- **Mac:** the Metal engine is fetched like any other. Pictures are on by default and read on the Mac's GPU ([MAC.md](MAC.md)).
|
||||
These are choices, not an instruction to install every row. The full list and memory estimates are in
|
||||
[MODELS.md](MODELS.md). For example, inspect the ISTA-DASLab Q2_0 plan:
|
||||
|
||||
## 5. While it downloads, tell the user
|
||||
|
||||
- It downloads about 70 GB from Hugging Face; how long depends on their connection (at 100 Mbit/s roughly 1.5-2
|
||||
hours). It can be stopped and continues where it left off.
|
||||
- Then it prepares the model for their PC (a few minutes).
|
||||
- **When the model starts, the PC can be slow or stop responding for 1-3 minutes** (longest the first time): Strata
|
||||
loads 35-55 GB into RAM and locks part of it for the graphics card. That is normal; they should not close the window.
|
||||
- Nothing leaves their PC: the model runs locally.
|
||||
|
||||
## 6. Check that it is serving
|
||||
|
||||
`strata launch` prints `Strata is serving.` with the addresses when it is done. Later:
|
||||
|
||||
```
|
||||
strata launch --status # lists what is serving, with its address
|
||||
strata launch --stop # stops it and frees the graphics memory
|
||||
strata launch --preset NAME # starts a saved preset again (after --save-preset NAME)
|
||||
```sh
|
||||
strata launch --family qwen --model q2_0 --context 32k --plan --output json
|
||||
```
|
||||
|
||||
No browser opens by itself. Add `--open` to `strata launch` if the user wants the web page.
|
||||
Check the resolved family, model, storage folder, context, picture reader and missing downloads. If the plan names
|
||||
an unavailable backend, do not continue as though compatibility were confirmed. Intel Arc has a released Linux
|
||||
engine, but model preparation still has special requirements: [INTEL_ARC.md](INTEL_ARC.md).
|
||||
|
||||
## 7. Verify
|
||||
## 4. Start it
|
||||
|
||||
The HTTP server answers once the model is loaded (30-90 s on later starts, a few minutes the first time). Poll:
|
||||
After the plan matches the user's computer and requested setup:
|
||||
|
||||
```
|
||||
curl http://127.0.0.1:8080/health
|
||||
```sh
|
||||
strata launch --family qwen --model q2_0 --context 32k --yes --no-tui
|
||||
```
|
||||
|
||||
It returns `{"status": "ok", "model": ..., "max_context": ..., "images": ..., "api_key": ..., "loaded": true, ...}`.
|
||||
Then:
|
||||
`--no-tui` prints the address and returns after startup while the server continues in the background. Keep the
|
||||
launch process alive during model acquisition; large downloads and preparation can take a long time. Poll progress
|
||||
and logs rather than promising a universal startup time. Repeating an interrupted launch resumes completed work.
|
||||
|
||||
```
|
||||
curl http://127.0.0.1:8080/v1/models
|
||||
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"strata\", \"messages\": [{\"role\": \"user\", \"content\": \"Say hello in five words.\"}], \"max_tokens\": 64, \"reasoning_effort\": \"none\"}"
|
||||
```
|
||||
Useful options:
|
||||
|
||||
With an API key, add `-H "Authorization: Bearer <key>"`.
|
||||
|
||||
## 8. Connect the user's apps
|
||||
|
||||
- **Browser:** `http://127.0.0.1:8080` - Chat, a live Monitor, and About (settings and addresses).
|
||||
- **Any OpenAI-compatible app or agent:** base URL `http://127.0.0.1:8080/v1`, any API key (or the configured one),
|
||||
any model name.
|
||||
- **Anthropic-compatible apps:** base URL `http://127.0.0.1:8080`, without `/v1`. Those apps add `/v1/messages`
|
||||
themselves.
|
||||
- **Codex CLI and other Responses API apps:** `http://127.0.0.1:8080/v1/responses` (stateless; Codex's
|
||||
`config.toml`: [DETAILS.md](DETAILS.md#the-responses-api-and-codex-cli)).
|
||||
- **Claude Code:** `ANTHROPIC_BASE_URL=http://127.0.0.1:8080`, `ANTHROPIC_MODEL` set to a Claude model name it knows
|
||||
(Strata ignores the name), and any `ANTHROPIC_AUTH_TOKEN` (or the configured key).
|
||||
- **The easy way for a coding app:** `strata connect` lists them (OpenCode, Codex CLI, Claude Code, oh-my-pi, pi,
|
||||
Hermes Agent, OpenClaw). `strata connect <app>` prints the file and environment for the running server,
|
||||
`strata connect <app> --write` writes it after asking (and keeps a backup), `strata connect <app> --check` sends
|
||||
one prompt and one tool call the way that app does and says whether each worked.
|
||||
- **Thinking level:** `"reasoning_effort": "none" | "low" | "medium" | "high"` (default high).
|
||||
- Strata answers one request at a time; `--parallel N` in `strata launch` decodes up to N together
|
||||
([BATCHING.md](BATCHING.md)). API details: [DETAILS.md](DETAILS.md#using-it).
|
||||
|
||||
## 9. If the Terminal cannot run: install from source
|
||||
|
||||
Use this only when step 2 does not work on the PC (for example a CPU without AVX2, or no internet access to GitHub
|
||||
Releases). It needs git or the zip, and on Linux `sudo` for Python and build tools.
|
||||
|
||||
```
|
||||
git clone https://github.com/Niko1221/Strata.git
|
||||
```
|
||||
|
||||
Without git: download https://github.com/Niko1221/Strata/archive/refs/heads/main.zip and unzip it. Pick a drive with
|
||||
enough free space: the model files go to a `Strata-data` folder next to the Strata folder (or `--data-dir <path>`
|
||||
on another drive). Setup's own check prints the GPU(s), driver, RAM, CPU, free disk and whether this PC can run
|
||||
Strata, and installs nothing except Python and the `.venv` it needs to run:
|
||||
|
||||
```
|
||||
Windows: START-HERE.bat --check
|
||||
Linux: ./setup.sh --check
|
||||
```
|
||||
|
||||
Run setup without questions, from the Strata folder (Windows: in `cmd`, or `cmd /c START-HERE.bat ...` from PowerShell).
|
||||
The model is picked as in step 3, with setup's flag names (`--family qwen|swift|coder|unsloth --model Q2_0|IQ2_XS|...`):
|
||||
|
||||
```
|
||||
Windows: START-HERE.bat --yes --family qwen --model IQ2_XS --vision yes --no-start
|
||||
Linux: ./setup.sh --yes --family qwen --model IQ2_XS --vision yes --no-start
|
||||
```
|
||||
|
||||
The flags (all of them: `START-HERE.bat --help`):
|
||||
|
||||
| Flag | Meaning |
|
||||
| Option | Meaning |
|
||||
| --- | --- |
|
||||
| `--yes` | take the recommended answer to every question (no prompts) |
|
||||
| `--family qwen\|swift\|coder\|unsloth` | the model version |
|
||||
| `--model Q2_0\|IQ2_XS\|IQ3_XXS\|IQ3_S\|IQ1_M\|UD-IQ4_XS\|UD-Q4_K_XL` | the size (the Coder is IQ1_M, Unsloth UD-IQ4_XS or UD-Q4_K_XL) |
|
||||
| `--context N` | context in tokens; default by VRAM: 32768 under 14 GB, 65536 under 20 GB, else 131072 |
|
||||
| `--vision yes\|no\|gpu\|cpu` | read pictures. Pass `yes` to get the best reader for the card (the CPU under 24 GB, the GPU from 24 GB, any make); `no` turns them off |
|
||||
| `--gpu N` / `--gpus 0,1` / `--gpus all` | one card, or several sharing the model (default: the card with the most VRAM) |
|
||||
| `--backend cuda\|hip` | NVIDIA or AMD engine; chosen by itself on a PC with only one kind of card |
|
||||
| `--data-dir PATH` | where new model files go. Nothing is moved unless you add `--move-data`; the saved default folder changes only with `--move-data` or `--set-default-data-dir` |
|
||||
| `--port N` | the server port (default 8080) |
|
||||
| `--host H --api-key K` | listen beyond this PC; only together with a key |
|
||||
| `--no-start` | install only, do not start the server |
|
||||
| `--setup` | install another model or change settings of an installed one |
|
||||
| `--check` | only check the PC |
|
||||
| `--model-path PATH` | Use existing supported model files; a matching filename alone does not prove compatibility |
|
||||
| `--data-dir PATH` | Folder for new downloads; existing files stay put unless `--move-data` is also given |
|
||||
| `--vision on\|off\|gpu\|cpu` | Picture reading, subject to model and packaged-reader support |
|
||||
| `--gpu 0,1` | Selected GPUs; do not use source setup's `--gpus` spelling here |
|
||||
| `--parallel N` | Requested concurrent slots, 1–8; check what actually fits and starts |
|
||||
| `--save-preset NAME` | Keep these launch choices for later |
|
||||
| `--open` | Open the server's browser page after it starts |
|
||||
| `--restart` | Replace an already running launch with different settings; interrupts active answers |
|
||||
|
||||
`--no-start` is recommended for agents: the server runs in the foreground until its window is closed, which would
|
||||
block your shell. Start it separately:
|
||||
`--yes` does not authorize a restart by itself. When a model is already running, inspect its state before using
|
||||
`--restart` or `--stop`. Use `strata launch --help` for the installed release's full list.
|
||||
|
||||
```
|
||||
Windows (PowerShell): Start-Process -FilePath ".\run-iq2_xs.bat" (opens its own window)
|
||||
Linux: nohup ./run-iq2_xs.sh > strata-server.out 2>&1 &
|
||||
## 5. Verify the running service
|
||||
|
||||
```sh
|
||||
strata launch --status
|
||||
strata doctor --live
|
||||
```
|
||||
|
||||
Setup prints the start script it wrote (`start script: run-<model>.bat`). The name is the family tag plus the size,
|
||||
lower case: `run-iq2_xs`, `run-swift-iq2_xs`, `run-coder-iq1_m`, `run-unsloth-ud-iq4_xs`. The engine log is
|
||||
`strata-<model>.log` in the Strata folder. Then continue with step 7.
|
||||
Check the printed address and model, then query `/health` and `/v1/models` at that address. With the default port:
|
||||
|
||||
Notes:
|
||||
- **Linux:** setup uses `sudo apt` (or dnf/pacman) to install Python with venv if it is missing, and on AMD may need
|
||||
`build-essential` and `git`. You cannot type the user's password: if a `sudo` step is needed, ask the user to run it
|
||||
(e.g. `sudo apt install python3-venv build-essential git`) and then rerun setup.
|
||||
- **AMD on Linux:** setup installs ROCm into `.venv` (~10 GB, no sudo) unless a system ROCm 7 exists, and compiles the
|
||||
engine for the card (10-20 minutes, once).
|
||||
- **AMD on Windows:** setup downloads the ready-made AMD engine (~550 MB, ROCm included) - nothing is compiled and only
|
||||
the AMD driver is needed. Before the model download it runs `engine\strata-device.exe --list-devices`; if that does
|
||||
not list the card, the driver is the problem (tell the user to update AMD Software). One card per model. Pictures use
|
||||
the 24 GB rule like everywhere else.
|
||||
- **NVIDIA with no ready-made engine for the card:** setup offers to install build tools and compile (20-40 minutes);
|
||||
`--yes` accepts.
|
||||
- **Mac:** `./setup.sh` builds the Metal engine on the Mac (needs Apple's Command Line Tools, `xcode-select --install`).
|
||||
**Linux / Mac:**
|
||||
|
||||
## 10. When something fails
|
||||
```sh
|
||||
curl -fsS http://127.0.0.1:8080/health
|
||||
curl -fsS http://127.0.0.1:8080/v1/models
|
||||
```
|
||||
|
||||
| Symptom | What to do |
|
||||
| --- | --- |
|
||||
| `the NVIDIA driver is too old` | The user updates the driver (NVIDIA App or nvidia.com/drivers) and restarts; run the command again. |
|
||||
| Download or install stopped | Run the same command again; it continues. |
|
||||
| `Strata couldn't find an engine for this computer` | The card is not one the engine packages cover, or the update feed was not reachable. Check the card against the list in step 1, check the internet, run `strata doctor`, then `strata update`. |
|
||||
| `port 8080 is already in use` | Strata is already running (check `/health` and `strata launch --status`), or use `--port 8081`. |
|
||||
| Very slow, disk busy, or "the engine stopped unexpectedly" | Not enough free RAM: close programs, or choose a smaller size (`q2_0` or `iq2_xs`). |
|
||||
| `ExpertCache: cudaMalloc(...) failed: out of memory` with free VRAM (Windows) | The page file is off or tiny: set a fixed size (Custom size, the same initial and maximum size, e.g. 65536 MB) and restart; a page file Windows grows on demand ("System managed") may not grow in time (#60). |
|
||||
| `prompt ... exceeds the context` | Start again with a bigger `--context`. |
|
||||
| No AMD GPU found (Linux) | The amdgpu driver lists no GPU; check the card and driver. Integrated GPUs are not supported. |
|
||||
| A Mac says the app is "damaged" (the app, not the Terminal) | The file is fine: [MAC.md](MAC.md) has the one command. |
|
||||
**Windows PowerShell:**
|
||||
|
||||
More: [TROUBLESHOOTING.md](TROUBLESHOOTING.md) and the [full table](DETAILS.md#troubleshooting). If it still fails,
|
||||
run `strata doctor`, collect its output and the engine log named in the error message, and suggest an issue at
|
||||
https://github.com/Niko1221/Strata/issues.
|
||||
```powershell
|
||||
Invoke-RestMethod http://127.0.0.1:8080/health
|
||||
Invoke-RestMethod http://127.0.0.1:8080/v1/models
|
||||
```
|
||||
|
||||
## Alternative: the MCP server
|
||||
Use the configured credential for protected endpoints. Confirm the model is loaded, then send a real request or
|
||||
run the relevant tool check below. **Exit code 0 alone is insufficient:** launch can finish after downloading a model
|
||||
without starting it when no usable engine is available. Report that as downloaded, not installed and verified.
|
||||
|
||||
Strata also has an MCP server, so an AI tool can check, install, start and stop Strata through tool calls
|
||||
(`strata_status`, `strata_models`, `strata_install`, `strata_start`, `strata_stop`, `strata_logs`) instead of the
|
||||
shell commands above. See [MCP_SERVER.md](MCP_SERVER.md).
|
||||
## 6. Connect a tool
|
||||
|
||||
```sh
|
||||
strata connect
|
||||
strata connect opencode
|
||||
strata connect opencode --check --output json
|
||||
```
|
||||
|
||||
The first command lists adapters; the second prints settings; the third sends a prompt and tool-call check. Use the
|
||||
adapter the user actually needs. `--write` changes that tool's configuration and keeps a backup; use it within the
|
||||
user's authorization. Printed configuration and a previous vendor test are not a successful current-server check.
|
||||
|
||||
- OpenAI-compatible clients: use the printed server address plus `/v1`.
|
||||
- Anthropic-compatible clients: use the server root; the client adds `/v1/messages`.
|
||||
- Use the actual API key when enabled. Tool identifiers and limits: [connection guide](TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
|
||||
- Flash-Next's template defaults to thinking on with `xhigh` effort; `high` maps to `xhigh`. Requests, shared Chat
|
||||
settings and run-config settings can override defaults. Do not assert a sampling configuration without inspecting it.
|
||||
|
||||
## 7. Handoff, update or stop
|
||||
|
||||
Tell the user which model is running, where its files are, its address, how the tool check went, and any limitations.
|
||||
|
||||
```sh
|
||||
strata launch --preset NAME
|
||||
strata launch --status
|
||||
strata launch --stop
|
||||
strata update --check
|
||||
```
|
||||
|
||||
Stop active models before `strata update`. Very old Terminals require rerunning the installer; see
|
||||
[updating](INSTALL.md#updating). Show credentials only to the intended user; redact them from shared logs and reports.
|
||||
|
||||
## 8. Source-only and unsupported setups
|
||||
|
||||
A source build is a separate workflow: [SOURCE_INSTALL.md](SOURCE_INSTALL.md). `START-HERE.bat`, `SETUP.bat`,
|
||||
`UPDATE.bat`, `setup.sh` and generated `run-<model>` scripts belong to that workflow. Do not pass their flags to
|
||||
`strata launch`, promise a source build will support an unknown model, or describe a source-only command as an
|
||||
installed-app command. If a required backend or artifact is missing, say exactly what is missing.
|
||||
+570
-963
File diff suppressed because it is too large.
Load diff
+996
-2210
File diff suppressed because it is too large.
Load diff
+35
-39
@@ -1,9 +1,8 @@
|
||||
# Strata in Docker
|
||||
|
||||
Run Strata as a container: one command, nothing to install on the computer except Docker and your graphics card's driver.
|
||||
The images come with Strata's engine already built for your card, so nothing is compiled on your side. The model is
|
||||
downloaded on the first start into a volume (a download that stops continues where it stopped), and every later start goes
|
||||
straight to serving.
|
||||
Run Strata in a container with a prebuilt engine. Install Docker, the host GPU driver and the GPU-access
|
||||
requirements in the table below. NVIDIA and AMD images can download and prepare a model on first start;
|
||||
Intel Arc needs prepared model files. Keep the data volume to reuse downloads and settings.
|
||||
|
||||
> **On this page:** [Which image](#which-image-for-which-card) · [One command per card](#one-command-per-card) ·
|
||||
> [Docker Compose](#docker-compose) · [Settings](#settings) · [The API key](#the-api-key) · [Connect your apps](#connect-your-coding-apps) ·
|
||||
@@ -12,7 +11,7 @@ straight to serving.
|
||||
|
||||
Docker on a **Mac** has no access to the Mac's graphics chip (no Metal): use the Mac app or the Terminal
|
||||
([MAC.md](MAC.md), [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md)). Docker Desktop on **Windows** passes NVIDIA cards through
|
||||
(WSL 2) but not Radeon or Arc cards; on Windows the app is the easier way for every card.
|
||||
(WSL 2) but not Radeon or Arc cards; on Windows use the native app for supported NVIDIA or AMD cards. Intel Arc has no native Windows engine; follow the Linux guide.
|
||||
|
||||
## Which image for which card
|
||||
|
||||
@@ -23,11 +22,11 @@ Docker on a **Mac** has no access to the Mac's graphics chip (no Metal): use the
|
||||
| AMD **RX 9000** series and Radeon AI PRO R9700 (RDNA 4, gfx1200 / gfx1201) | `0.2.3-rocm-gfx120x` | the `amdgpu` kernel driver (Linux 6.8+) |
|
||||
| AMD **RX 7000** series (RDNA 3, gfx1100 / gfx1101 / gfx1102) | `0.2.3-rocm-gfx110x` | the `amdgpu` kernel driver |
|
||||
| AMD **Ryzen AI Max** ("Strix Halo", Radeon 8060S, gfx1151) | `0.2.3-rocm-gfx1151` | the `amdgpu` kernel driver |
|
||||
| Intel **Arc** A-series and B-series (including Arc Pro B70) | `0.2.3-sycl` (only for a release that carries the Intel engine package; otherwise [build it](#build-it-yourself)) | the `i915` or `xe` kernel driver; **prepared model files** (see [Intel](#intel-arc)) |
|
||||
| Intel **Arc** A-series and B-series (including Arc Pro B70) | `0.2.3-sycl` (if no image exists for a release, [build it](#build-it-yourself)) | the `i915` or `xe` kernel driver; **prepared model files** (see [Intel](#intel-arc)) |
|
||||
|
||||
Size of the images on disk (built from the engine packages of the 0.2.0-beta release): `cuda13` 2.2 GB, `cuda12` 2.9 GB, `rocm-gfx120x` 2.8 GB,
|
||||
`rocm-gfx110x` 2.4 GB, `rocm-gfx1151` 1.5 GB; the download is about a third of that (measured on the 0.1.45 packages: `cuda13` 640 MB, `cuda12` 845 MB,
|
||||
`rocm-gfx120x` 505 MB, `rocm-gfx110x` 330 MB, `sycl` 450 MB). The model comes on top, on the first start.
|
||||
**Measured on:** images built from the 0.1.45 engine packages. Size on disk: `cuda13` 2.2 GB, `cuda12` 2.9 GB, `rocm-gfx120x` 2.8 GB,
|
||||
`rocm-gfx110x` 2.4 GB, `rocm-gfx1151` 1.5 GB. The download is about a third of that: `cuda13` 640 MB, `cuda12` 845 MB,
|
||||
`rocm-gfx120x` 505 MB, `rocm-gfx110x` 330 MB, `sycl` 450 MB. The model comes on top, on the first start. Current sizes may differ.
|
||||
|
||||
Not sure which NVIDIA image? Take `cuda13` when `nvidia-smi` shows driver 580 or newer and the card is an RTX; otherwise `cuda12`.
|
||||
A wrong pick does no harm: the container starts, says in plain words that the engine does not fit this card, and stops.
|
||||
@@ -40,7 +39,7 @@ tag if you want updates only when you decide. The examples below use these image
|
||||
## One command per card
|
||||
|
||||
Each command runs Strata in the background, keeps your files in the volume `strata-data`, and listens only on this computer
|
||||
(`127.0.0.1:8080`). Replace `MODEL` and the key. The first start downloads the model (tens of GB): watch it with
|
||||
(`127.0.0.1:8080`). Choose `FAMILY` and `MODEL`, and generate your own key. The examples select ISTA-DASLab originals explicitly. The first start downloads the model (tens of GB): watch it with
|
||||
`docker logs -f strata`. Already have the model's GGUF files? Add `-e MODEL_PATH=/models -v /your/folder:/models:ro` and nothing is
|
||||
downloaded for the model itself ([Files you already have](#files-you-already-have)).
|
||||
|
||||
@@ -49,7 +48,7 @@ downloaded for the model itself ([Files you already have](#files-you-already-hav
|
||||
```sh
|
||||
docker run -d --name strata --gpus all -p 127.0.0.1:8080:8080 \
|
||||
--ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \
|
||||
-v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \
|
||||
-v strata-data:/data -e FAMILY=qwen -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \
|
||||
ghcr.io/niko1221/strata:0.2.3-cuda13
|
||||
```
|
||||
|
||||
@@ -59,7 +58,7 @@ docker run -d --name strata --gpus all -p 127.0.0.1:8080:8080 \
|
||||
docker run -d --name strata --device /dev/kfd --device /dev/dri \
|
||||
--group-add video --group-add $(getent group render | cut -d: -f3) -p 127.0.0.1:8080:8080 \
|
||||
--ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \
|
||||
-v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \
|
||||
-v strata-data:/data -e FAMILY=qwen -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \
|
||||
ghcr.io/niko1221/strata:0.2.3-rocm-gfx120x
|
||||
```
|
||||
|
||||
@@ -68,7 +67,7 @@ docker run -d --name strata --device /dev/kfd --device /dev/dri \
|
||||
```sh
|
||||
docker run -d --name strata --device /dev/dri --group-add $(getent group render | cut -d: -f3) \
|
||||
-p 127.0.0.1:8080:8080 --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \
|
||||
-v strata-data:/data -e MODEL=IQ3_S -e MODEL_PATH=/models -v /srv/gguf/IQ3_S:/models:ro \
|
||||
-v strata-data:/data -e FAMILY=qwen -e MODEL=IQ3_S -e MODEL_PATH=/models -v /srv/gguf/IQ3_S:/models:ro \
|
||||
-e API_KEY=$(openssl rand -hex 24) ghcr.io/niko1221/strata:0.2.3-sycl
|
||||
```
|
||||
|
||||
@@ -96,7 +95,7 @@ Ready-made files are in [docker/compose/](../docker/compose): `docker-compose.nv
|
||||
|
||||
```sh
|
||||
cd docker/compose
|
||||
cp .env.example .env # set MODEL=..., API_KEY=...
|
||||
cp .env.example .env # set FAMILY=qwen, MODEL=..., API_KEY=...
|
||||
docker compose -f docker-compose.nvidia.yml up -d
|
||||
docker compose -f docker-compose.nvidia.yml logs -f
|
||||
```
|
||||
@@ -114,16 +113,17 @@ screens; leave out what you do not care about.
|
||||
| Variable | What it does | Default |
|
||||
| --- | --- | --- |
|
||||
| `MODEL` | model size: `IQ2_XS`, `IQ3_XXS`, `IQ3_S`, `Q2_0`, `UD-Q4_K_XL`, ... ([MODELS.md](MODELS.md)) | Strata picks what suits the card, and says so in the log |
|
||||
| `FAMILY` | `qwen`, `swift`, `coder`, ... when a size exists in several families | the usual one |
|
||||
| `FAMILY` | `qwen`, `unsloth`, `gyro`, `coder`, `swift`, ...; use `qwen` for the original ISTA-DASLab packs | installer recommendation; may differ by hardware |
|
||||
| `MODEL_PATH` | GGUF files you already have: a folder or one `.gguf`, **inside the container** (mount it with `-v`, `:ro` is fine) | download |
|
||||
| `CONTEXT` | how much text it remembers at once: `32k`, `64k`, `128k`, `256k` or a number | the largest that fits |
|
||||
| `PARALLEL` | requests answered at the same time, 1 to 8 | fitted to the card |
|
||||
| `VISION` | `off`, `on` (the model reads pictures), `gpu`, `cpu`; the picture-reader file (about 0.9 GB) is downloaded with the model | `off` |
|
||||
| `KV` | conversation memory format: `int8`, `q4_0` (half the memory), `k8v4`, `fp16` | `int8` |
|
||||
| `GPU` / `GPUS` | card numbers as the monitor lists them, `0` or `0,1`; empty or `all` = the engine's choice. Changing it later makes the model's run settings again (the files stay) | all that fit |
|
||||
| `GPU_ROLES` | a role for each card, `1=main,0=helper` (`main`, `helper`, `off`; cards you do not name are off; not together with `GPU` / `GPUS`). Passed to `launch --gpu-roles`; see [MULTI_GPU.md](MULTI_GPU.md) |
|
||||
| `TUNE` | `quick` (up to about 20 minutes, once), `thorough`, `skip`; done once and kept in the preset (`RETUNE=1` tunes again) | `skip` |
|
||||
| `GPU_ROLES` | a role for each card, `1=main,0=helper` (`main`, `helper`, `off`; cards you do not name are off; not together with `GPU` / `GPUS`). Passed to `launch --gpu-roles`; see [MULTI_GPU.md](MULTI_GPU.md) | none |
|
||||
| `TUNE` | `quick` (runtime depends on hardware and workload), `thorough`, `skip`; done once and kept in the preset (`RETUNE=1` tunes again) | `skip` |
|
||||
| `API_KEY` / `API_KEY_FILE` | the key clients must send ([below](#the-api-key)); the file form is for Docker secrets | made for you |
|
||||
| `TUNE_ALLOW_SMALL` | `1` passes `--tune-allow-small` to `launch` (allow tuning on a small card) | `0` |
|
||||
| `PORT` | the port inside the container | `8080` |
|
||||
| `PRESET` | the preset name in `/data` | `docker` |
|
||||
| `STRATA_SET` | advanced engine settings, space separated, like `strata launch --set` (`"prefill=65536 STRATA_PIN_BY_NEED=1"`) | none |
|
||||
@@ -131,7 +131,7 @@ screens; leave out what you do not care about.
|
||||
Your choices are saved as a preset in the volume. Start again with no settings and the saved preset is used; start with
|
||||
settings and they replace the saved ones (tuning results are kept). To change one thing, recreate the container with the new
|
||||
variable; the model and tuning stay in `/data`. Anything else: `docker exec strata strata launch --help`, or
|
||||
`docker run --rm IMAGE strata launch --plan --model q2_0` to see what a choice would do without starting anything.
|
||||
`docker run --rm IMAGE strata launch --plan --family qwen --model q2_0` to see what a choice would do without starting anything.
|
||||
|
||||
What is in the volume `/data`: `Strata-data/` (model files, the prepared pack), `state/` (presets, tuning, the key), `install/`
|
||||
(run settings and logs), `home/`. The engine and the Terminal are in the image, so updating the image never touches your files.
|
||||
@@ -142,14 +142,14 @@ container's user). Nothing large is written anywhere else in the container.
|
||||
### Files you already have
|
||||
|
||||
Mount the folder with the model's GGUF files read-only and name it: `-v /srv/gguf/IQ3_XXS:/models:ro -e MODEL_PATH=/models
|
||||
-e MODEL=IQ3_XXS`. Nothing is downloaded for the model itself; the prepared pack is made in the volume.
|
||||
-e FAMILY=qwen -e MODEL=IQ3_XXS`. Nothing is downloaded for the model itself; the prepared pack is made in the volume.
|
||||
|
||||
- The folder holds the shards of one model directly (`...-00001-of-00002.gguf`, `...-00002-of-00002.gguf`), not a folder of folders.
|
||||
If you mount a parent folder, the container says which sub-folder to name in `MODEL_PATH`.
|
||||
- The folder has to be mounted: a `MODEL_PATH` that is not in the container, or a folder with no `.gguf` files, stops the start at
|
||||
once with a message, and nothing is downloaded.
|
||||
- Read-only (`:ro`) is fine. The pack (about 40 GB for IQ3_XXS) and the draft layer (about 6 GB, fetched once) are written to the
|
||||
volume, so it needs that room. With pictures on (the default) the picture reader (0.9 GB) is fetched too; `VISION=off` skips it.
|
||||
volume, so it needs that room. With `VISION=on` the picture reader (0.9 GB) is fetched too; `VISION=off` skips it.
|
||||
- Links to the files work when the links' targets are mounted too, and `MODEL` is set (without `MODEL`, Strata identifies the files itself
|
||||
and does not follow links).
|
||||
- Without `MODEL_PATH`, files you put into the volume by hand (`/data/Strata-data/models/<model folder>/`) are found as well, if they have the
|
||||
@@ -284,9 +284,9 @@ This is the way to get an image for a computer without registry access or after
|
||||
folder (made by `packaging/package_engine.py`, or fetched by `strata update` into a components folder) and Docker on Linux:
|
||||
|
||||
```sh
|
||||
docker/build.sh --flavor cuda --engine ~/components/engine/0.1.45.3-linux-x86_64-cuda-cuda13 --tag strata:0.2.3-cuda13
|
||||
docker/build.sh --flavor hip --engine ~/components/engine/0.1.45.3-linux-x86_64-hip-rx9070 --tag strata:0.2.3-rocm-gfx120x
|
||||
docker/build.sh --flavor sycl --engine ~/components/engine/<a-sycl-package-folder> --tag strata:0.2.3-sycl
|
||||
docker/build.sh --flavor cuda --engine ~/components/engine/<cuda13-package-folder> --tag strata:0.2.3-cuda13
|
||||
docker/build.sh --flavor hip --engine ~/components/engine/<hip-rx9070-package-folder> --tag strata:0.2.3-rocm-gfx120x
|
||||
docker/build.sh --flavor sycl --engine ~/components/engine/<sycl-package-folder> --tag strata:0.2.3-sycl
|
||||
```
|
||||
|
||||
`build.sh` assembles the build context (the Terminal's files and the package) and runs `docker build`. The build makes the
|
||||
@@ -332,27 +332,12 @@ the job starts the image once to see that the Terminal inside it runs.
|
||||
| `rocm-gfx120x` | `linux-x86_64`, hip, variant `rx9070` (gfx1200 / gfx1201) |
|
||||
| `rocm-gfx110x` | `linux-x86_64`, hip, variant `rx7900` (gfx1100 / gfx1101 / gfx1102) |
|
||||
| `rocm-gfx1151` | `linux-x86_64`, hip, variant `halo` (Strix Halo, gfx1151) |
|
||||
| `sycl` | `linux-x86_64`, sycl; **only when the feed lists one** (the 0.2.0-beta feed has none, so no Intel image is built from it) |
|
||||
| `sycl` | `linux-x86_64`, sycl; only when the feed lists one |
|
||||
|
||||
A flavor the feed has no package for is left out, with a line in the log. The same steps run from a Linux computer with Docker:
|
||||
`docker/publish.sh --release-tag v0.2.3` builds everything and pushes nothing; add `--push` (after `docker login ghcr.io`) to publish.
|
||||
The images run on the runner without a graphics card, so the card checks (`docker/smoke-test.sh`) stay a manual step on a computer with the card.
|
||||
|
||||
## Tested
|
||||
|
||||
2026-10-19, images built from the engine packages of the 0.2.0-beta release (engine 0.1.45.3), fetched and checked against the signed feed by
|
||||
`docker/fetch-engines.py` (the packages are the ones `strata update` installs). Run on real cards with the checks below (`docker/smoke-test.sh`): `cuda13`
|
||||
and `cuda12` on one RTX A4000 (driver 580), `rocm-gfx120x` on one Radeon AI PRO R9700. All passed: serving 71-161 s after `docker run` with the prepared
|
||||
files, `docker stop` took 4-5 s with exit code 0, the card's memory was free again, and the second start reused the saved preset. `rocm-gfx110x` and
|
||||
`rocm-gfx1151` (Strix Halo) were built and passed the package and library checks but have not run on a card. On a computer without an AMD card the
|
||||
`rocm-gfx120x` container says in plain words that the engine does not match the card and exits with code 3.
|
||||
|
||||
2026-10-17, images built from the 0.1.45 engine packages, on real cards: NVIDIA RTX 3060 (`cuda13`
|
||||
and `cuda12`), Radeon AI PRO R9700 (`rocm-gfx120x`), Intel Arc Pro B70 (`sycl`). On each: the container serves, `GET /v1/models` answers (and
|
||||
refuses a request without the key), a one-word chat and a tool call work, `docker exec strata monitor --once` works,
|
||||
`docker stop` ends it with exit code 0 and the graphics memory is free again, and a second start reuses the saved preset.
|
||||
Not run on a card: `cuda12` on Pascal / Volta (it ran on the RTX 3060), `rocm-gfx110x` (RX 7000). They are built and checked the same way.
|
||||
|
||||
## Known limits
|
||||
|
||||
- Linux x86-64 hosts only. Docker on Windows reaches NVIDIA cards only; Docker on a Mac cannot use the GPU.
|
||||
@@ -361,3 +346,14 @@ Not run on a card: `cuda12` on Pascal / Volta (it ran on the RTX 3060), `rocm-gf
|
||||
- An Arc card needs prepared model files (see [Intel](#intel-arc)).
|
||||
- AMD **RX 7000** (gfx110x) and RDNA 3 iGPUs: the gfx110X image is built for gfx1100 / 1101 / 1102 (RX 7900, 7800, 7700, 7600). The
|
||||
Radeon 780M (gfx1103) is experimental and has its own engine package, not in these images.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
**Tested on real cards.** Checks (`docker/smoke-test.sh`): the container serves, `GET /v1/models` answers (and refuses a request
|
||||
without the key), a one-word chat and a tool call work, `docker exec strata monitor --once` works, `docker stop` ends it with exit
|
||||
code 0 and the graphics memory is free again, and a second start reuses the saved preset.
|
||||
|
||||
| Date | Images built from | Card | Result |
|
||||
| --- | --- | --- | --- |
|
||||
| 2026-10-19 | 0.2.0-beta engine packages (engine 0.1.45.3), fetched and checked against the signed feed | `cuda13` and `cuda12` on one RTX A4000 (driver 580); `rocm-gfx120x` on one Radeon AI PRO R9700 | All passed. Serving 71-161 s after `docker run` with the prepared files; `docker stop` took 4-5 s with exit code 0. `rocm-gfx110x` and `rocm-gfx1151` passed the package and library checks but did not run on a card. On a computer without an AMD card the `rocm-gfx120x` container says the engine does not match the card and exits with code 3. |
|
||||
| 2026-10-17 | 0.1.45 engine packages | RTX 3060 (`cuda13`, `cuda12`), Radeon AI PRO R9700 (`rocm-gfx120x`), Intel Arc Pro B70 (`sycl`) | All checks passed on each. Not run on a card: `cuda12` on Pascal / Volta (it ran on the RTX 3060), `rocm-gfx110x` (RX 7000). |
|
||||
+262
-254
@@ -1,178 +1,208 @@
|
||||
# Gyro (agentionai rotor quants) on Strata
|
||||
|
||||
Gyro-S and Gyro-M are Qwen3.8-Flash-Next quantizations by agentionai
|
||||
([agentionai/Qwen3.8-Flash-Next-Gyro-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)): the routed
|
||||
experts are stored as trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a Hadamard-rotated basis. Strata reads the
|
||||
published files unchanged. **Setup lists them as a model family of their own** ("Qwen3.8-Flash-Next (Gyro)"); neither
|
||||
is anybody's default.
|
||||
This page is for people who want to run Gyro-S or Gyro-M, the smallest Qwen3.8-Flash-Next options, or who want to know
|
||||
how they compare with the other sizes. They are quantizations by agentionai
|
||||
([agentionai/Qwen3.8-Flash-Next-Gyro-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)). The
|
||||
routed experts are stored as trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a Hadamard-rotated basis. Strata
|
||||
reads the published files unchanged.
|
||||
|
||||
Setup lists them as a model family of their own, "Qwen3.8-Flash-Next (Gyro)". They are not the first recommendation
|
||||
([MODELS.md](MODELS.md)). Setup may still preselect Gyro for some memory tiers, so choose the provider explicitly if
|
||||
you care.
|
||||
|
||||
> **On this page:** [Quick start](#quick-start) · [Sizes](#sizes) · [Compared with other sizes](#compared-with-other-sizes) ·
|
||||
> [Platforms](#platforms) · [Intel Arc (SYCL)](#intel-arc-sycl) · [On a Mac](#on-a-mac) ·
|
||||
> [Install by hand](#install-by-hand) · [Numerics](#numerics) · [Known limits](#known-limits) ·
|
||||
> [Credit and licenses](#credit-and-licenses) · [History](#history--older-measurements)
|
||||
|
||||
## Quick start
|
||||
|
||||
```sh
|
||||
strata launch --family gyro --model Gyro-S # or Gyro-M
|
||||
```
|
||||
|
||||
Setup downloads the pinned revision of the file (size and SHA-256 checked) and prepares the pack
|
||||
(`tools/iq_pack.py --compat-bf16`: the Hadamard table and the dense weights, 1.5 GB). It also prepares the draft head
|
||||
(Strata's own MTP head, made from the original model as for every Qwen3.8-Flash-Next file) and the start script. If the
|
||||
ready-made engine is too old for the pack, setup replaces it or the app offers Update engine.
|
||||
|
||||
## Sizes
|
||||
|
||||
| | Gyro-S | Gyro-M |
|
||||
| --- | --- | --- |
|
||||
| File | `Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf` | `Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf` |
|
||||
| **Download** (the n-gram table is inside the file) | **58.5 GB** | **92.0 GB** |
|
||||
| **Memory it needs** (VRAM + RAM together) | **27.6 GB** | **35 GB** |
|
||||
| Download (the n-gram table is inside the file) | 58.5 GB | 92.0 GB |
|
||||
| Memory it needs (VRAM + RAM together) | 27.6 GB | 35 GB |
|
||||
| Experts (trellis) | TQK6 gate/up, TQK7 down | TQ2_T |
|
||||
| In plain words | the smallest option, for PCs with little memory; lower quality than the 2-3-bit sizes | about IQ3_XXS quality with about 20% less GPU memory; decodes about 16% slower than Q2_0 on a 12 GB RTX 3060 (24% slower before engine 0.2.2) |
|
||||
| In plain words | The smallest option, for PCs with little memory. Lower quality than the 2 to 3-bit sizes. | About IQ3_XXS quality with about 20% less GPU memory. |
|
||||
|
||||
About half of each file is the n-gram (PLE) table: it stays on the SSD and is read row by row for every token, so keep
|
||||
About half of each file is the n-gram (PLE) table. It stays on the SSD and is read row by row for every token, so keep
|
||||
the file on a fast local drive. The memory figure is what the weights take once loaded.
|
||||
|
||||
Not defaults, by measurement (our evaluation, 2026-10): against Q2_0 at the same GPU bytes Gyro-M halves the KL
|
||||
divergence (0.159 against 0.312 on 128 chunks) and is level with IQ3_XXS (0.157), but decodes 16-21% slower on a 12 GB
|
||||
card (RTX 3060, measured before engine 0.2.2: 16% since) and an R9700, and its reasoning set scored lower than Q2_0's (49/60 against 57/60). Gyro-S is 13% worse
|
||||
in KL than IQ2_XS (0.268 against 0.236) with 24% fewer bytes. So: pick Gyro-S where nothing bigger fits, Gyro-M when
|
||||
the 20% memory saving over IQ3_XXS matters. Setup and the MCP server mention them only where the other sizes do not
|
||||
fit the PC; they warn, never refuse. Gyro-L (agentionai's 50 GiB file, 2026-10-12) is not supported: it uses another
|
||||
trellis family (types 147-149) that the engine does not have.
|
||||
Gyro-L (agentionai's 50 GiB file) is not supported. It uses another trellis family (types 147 to 149) that the engine
|
||||
does not have.
|
||||
|
||||
## Install
|
||||
**Pictures.** Gyro reads pictures with the original model's picture reader file, the same one every
|
||||
Qwen3.8-Flash-Next model uses. Setup downloads it with the model. The reader opens only the vocabulary and the header of
|
||||
the Gyro file. Pictures are on by default, with the CPU reader under 24 GB of graphics memory and the card's reader from
|
||||
24 GB.
|
||||
|
||||
```sh
|
||||
./setup.sh --setup --family gyro --model Gyro-S # or Gyro-M; Windows: START-HERE.bat --setup ...
|
||||
```
|
||||
## Compared with other sizes
|
||||
|
||||
Setup downloads the pinned revision of the file (size and SHA-256 checked), prepares the pack (`tools/iq_pack.py
|
||||
--compat-bf16`: the Hadamard table and the dense weights, 1.5 GB), the draft head (Strata's own MTP head, prepared from
|
||||
the original model as for every Qwen3.8-Flash-Next file) and the start script. Needs engine 0.1.45.1 or newer: the pack is native_experts.txt v5 (Hadamard-folded), which the engine reads since the
|
||||
APR port, and the released 0.1.45 stops at v4. A ready-made engine that is older is replaced by one compiled from this source
|
||||
(setup), or the app says the model needs a newer engine and offers Update engine.
|
||||
Gyro is not a default, by our measurements. Setup and the MCP server mention it only where the other sizes do not fit
|
||||
the PC. They warn and never refuse.
|
||||
|
||||
**Supported:** NVIDIA (CUDA, Linux and Windows) and AMD (HIP, experimental: the authors validated gfx1151; we ran
|
||||
gfx1201 with identical output to rc2). **Intel Arc** (SYCL, Linux, built from source) runs both files: see [Intel Arc](#intel-arc-sycl). **Apple Silicon Macs** (Metal) run both files from
|
||||
0.2.2: see [On a Mac](#on-a-mac).
|
||||
- Pick Gyro-S where nothing bigger fits.
|
||||
- Pick Gyro-M when the 20% memory saving over IQ3_XXS matters.
|
||||
|
||||
**Pictures.** Gyro reads pictures with the original model's picture reader file (the same one every Qwen3.8-Flash-Next
|
||||
model uses; setup downloads it with the model). The reader opens only the vocabulary and the header of the Gyro file, so
|
||||
the trellis tensors do not matter to it. Pictures are on by default, with the CPU reader under 24 GB of graphics memory and
|
||||
the card's reader from 24 GB. This needs engine 0.1.45.1 or newer, like the Gyro files themselves.
|
||||
> **Measured on:** our evaluation, 2026-10, Gyro files against ISTA-DASLab files at the same GPU bytes, KL divergence on
|
||||
> 128 chunks (lower is better).
|
||||
|
||||
**RTX 50 cards (sm_120).** Gyro gives correct output on them with engine 0.1.45.3. The fused gate/up SwiGLU kernel, which
|
||||
is the default on compute capability 12.0 cards, handles only rows of 80 dot products; Gyro's trellis rows have 160, and
|
||||
before the fix they went through it anyway. Gyro-S then answered with loops, garbage or nothing: on an RTX PRO 5000, 0 of
|
||||
8 test questions were answered right before the fix and 8 of 8 after it (greedy and with the app's sampling). Now those
|
||||
rows use the full-warp kernel, and the identity gate (10 prompts, 4K and 20K prompts) is identical to the run with the fused
|
||||
kernel off. If Gyro misbehaves on an RTX 50 card, update the engine.
|
||||
| Comparison | Result |
|
||||
| --- | --- |
|
||||
| Gyro-M against Q2_0 | KL 0.159 against 0.312 |
|
||||
| Gyro-M against IQ3_XXS | KL 0.157, level |
|
||||
| Gyro-S against IQ2_XS | KL 0.268 against 0.236 (13% worse), with 24% fewer bytes |
|
||||
| Reasoning set, Gyro-M against Q2_0 | 49/60 against 57/60 |
|
||||
|
||||
> **Measured on:** RTX 3060 12 GB, Core Ultra 7, Ubuntu, CUDA 13.4, default flags (MTP draft 4, auto cache), engine
|
||||
> 0.2.2, the pinned Gyro files, medians of story, code and facts prompts (200 tokens each) over 5 interleaved pairs of
|
||||
> whole engine starts.
|
||||
|
||||
| Decode | tok/s | Against Q2_0 on the same card (65.5 and 64.5 tok/s) |
|
||||
| --- | ---: | --- |
|
||||
| Gyro-S | 62.5 | 7% slower |
|
||||
| Gyro-M | 54.8 | 16% slower |
|
||||
|
||||
On an R9700, Gyro-M decodes about 21% slower than Q2_0 (our evaluation, 2026-10). Gyro prompts are 7 to 9% slower than
|
||||
Q2_0's. What is left of Gyro-M's gap is mostly its dense weights: its Q8_0 matrix-vector kernels take 26% of the GPU
|
||||
time in a decode profile.
|
||||
|
||||
## Platforms
|
||||
|
||||
| Platform | Status |
|
||||
| --- | --- |
|
||||
| NVIDIA (CUDA, Linux and Windows) | Supported, including RTX 50 cards (sm_120). |
|
||||
| AMD (HIP) | Experimental. The authors validated gfx1151. We ran gfx1201 with output identical to rc2's. |
|
||||
| Intel Arc (SYCL, Linux, built from source) | Runs both files. See [Intel Arc (SYCL)](#intel-arc-sycl). |
|
||||
| Apple Silicon (Metal) | Runs both files. See [On a Mac](#on-a-mac). |
|
||||
|
||||
If Gyro misbehaves on an RTX 50 card, update the engine.
|
||||
|
||||
**Prompt speed and the ggml build.** The trellis prompt kernels (MMQ) are agentionai's, from their llama.cpp fork (MIT).
|
||||
Strata builds against upstream ggml, which does not have them, so it adds them itself at configure time:
|
||||
`cmake/strata_ggml_tq_overlay.cmake` writes a copy of the pinned ggml-cuda headers into the build folder with the three
|
||||
trellis types routed like IQ2_XS (the tile loader is the fork's, `third_party/ggml-tq/`); only Strata's own MMQ library sees
|
||||
the copy, every other type is compiled from the untouched files (identity gate below). The ready-made engines read Gyro
|
||||
prompts through it as fast as rc2 does, and bit for bit the same. If the pinned ggml changes and an anchor of the script
|
||||
no longer matches, the overlay switches itself off with a message and prompts take the slower FP16 path. A build against the
|
||||
fork (`-DSTRATA_GGML_DIR=<agentionai/llama.cpp>`, as rc2 does) uses the fork's own instances instead.
|
||||
Strata builds against upstream ggml, which does not have them, so it adds them at configure time.
|
||||
`cmake/strata_ggml_tq_overlay.cmake` writes a copy of the pinned ggml-cuda headers into the build folder with the
|
||||
three trellis types routed like IQ2_XS. The tile loader is the fork's (`third_party/ggml-tq/`). Only Strata's own MMQ
|
||||
library sees the copy; every other type is compiled from the untouched files. The ready-made engines read Gyro prompts
|
||||
through it. If the pinned ggml changes and an anchor of the script no longer matches, the overlay switches itself off
|
||||
with a message and prompts take the slower FP16 path. A build against the fork (`-DSTRATA_GGML_DIR=<agentionai/llama.cpp>`)
|
||||
uses the fork's own instances instead.
|
||||
|
||||
**Decode kernels.** The trellis decode kernels keep the 4 KB codebook in shared memory (`STRATA_IQ_STAGE_GRID=0` reads it
|
||||
from the global table again) and take four rows per warp. The sums are added in the same order as the unstaged kernels,
|
||||
so answers are bit for bit the same. A Tesla P100 (compute capability 6.0) runs its own variant, `STRATA_TQ_STAGE`.
|
||||
|
||||
## Intel Arc (SYCL)
|
||||
|
||||
Gyro-S and Gyro-M run on Intel Arc cards on Linux, with the SYCL engine that setup builds from this source ([INTEL.md](INTEL.md)).
|
||||
Setup lists them for an Arc like for any card and warns, never refuses, when VRAM and RAM together are short. Tested on one
|
||||
card: an Arc A750 (8 GB, i915 driver) in a PC with a Ryzen 5 5600X (6 cores, AVX2) and 62 GB of RAM, on the first-release files of Gyro-S and
|
||||
Gyro-M (the same size as the pinned ones, a different SHA-256). The pinned Gyro-S passes the same parity run (16 cases, 0 failures) and
|
||||
answers correctly through the server; it was not compared with CUDA.
|
||||
Not tested by us: a B-series card (supported since 2026-10-22; `strata launch` says it is new and how to report a problem), two Arc cards (the second-card expert helper refuses a Gyro pack and says so), an Arc on the xe driver. The Battlemage AOT build
|
||||
(`bmg-g31`) compiles with the trellis kernels; it was not run.
|
||||
Gyro-S and Gyro-M run on Intel Arc cards on Linux, with the SYCL engine that setup builds from source
|
||||
([INTEL.md](INTEL.md)). Setup lists them for an Arc like for any card and warns, never refuses, when VRAM and RAM
|
||||
together are short.
|
||||
|
||||
The trellis decode kernels do the same integer and float operations in the same order as the CUDA ones (int8 codebook, q8_1 activations),
|
||||
the Hadamard rotation of the expert input and of the SwiGLU output runs on the card, and the prompt takes the grouped XMX path,
|
||||
which now decodes the trellis types into its tiles. Experts in the card's cache run on the card; the other misses go to the CPU pool or
|
||||
over the PCIe link, as with every pack.
|
||||
**Tested by us:** one Arc A750 (8 GB, i915 driver) in a PC with a Ryzen 5 5600X (6 cores, AVX2) and 62 GB of RAM. It ran
|
||||
the first-release files of Gyro-S and Gyro-M (same size as the pinned ones, different SHA-256). The pinned Gyro-S passes
|
||||
the same parity run (16 cases, 0 failures) and answers correctly through the server. It was not compared with CUDA.
|
||||
|
||||
**Checks.** `tq_sycl_parity <pack> <model.gguf> [layer ...]` (also `--synthetic [--hadamard] GU/DOWN` and `--bench GU/DOWN`) takes real
|
||||
experts of the file (layers 0-3, 10, 20, 30 and 47, two experts each) and compares the card with a float reference and with the CPU path:
|
||||
**Not tested by us:** a B-series card (`strata launch` says it is new and how to report a problem), two Arc cards (the
|
||||
second-card expert helper refuses a Gyro pack and says so) and an Arc on the xe driver. The Battlemage AOT build
|
||||
(`bmg-g31`) compiles with the trellis kernels but was not run.
|
||||
|
||||
- the decode kernels equal a host replay of their own arithmetic **in every bit** (0 of 3,840 gate/up values and 0 of 7,680 down values
|
||||
differ, Gyro-S and Gyro-M, all 16 cases each); the prompt dequantizers equal the fp16 codebook in every bit (0 of 4.9 million values);
|
||||
the card's Hadamard rotation equals the CPU's bit for bit (the fp16 form within 3e-4);
|
||||
- against the float reference the card is within 1.7-2.0e-2 relative L1 error (the int8 codebook; 3e-2 is the bound every expert type
|
||||
gets), the CPU path within 0.9e-2. The card and the CPU path therefore agree within tolerance, not bit for bit: the CPU path uses an
|
||||
int16 codebook and Q8_0 activations, the card the int8 codebook and q8_1, as on CUDA.
|
||||
The trellis decode kernels do the same integer and float operations in the same order as the CUDA ones (int8 codebook,
|
||||
q8_1 activations). The Hadamard rotation of the expert input and of the SwiGLU output runs on the card, and the prompt
|
||||
takes the grouped XMX path, which decodes the trellis types into its tiles. Experts in the card's cache run on the card.
|
||||
Misses go to the CPU pool or over the PCIe link, as with every pack.
|
||||
|
||||
**Against CUDA.** The same Gyro-S file on an RTX 3060 (the f-022-int engine) and on the A750, both with 64 cached experts, no PCIe share and
|
||||
no CPU prompt share, ten prompts of 26-100 tokens, 48 greedy tokens with the MTP draft: 7 of the 10 prompts give the same 48 tokens. The other three (prompts 5, 7 and 8) agree for 9, 9 and 6 tokens and then differ at a near-tie. The first-token KL is 1.3e-4 to 1.1e-2 and the top token is always the same. The two SYCL prompt paths (oneMKL and XMX) also flip prompt 7 against each other, so the prompt-path rounding alone moves it. A 1,406-token prompt gives the same 16 tokens on both.
|
||||
**Checks.** `tq_sycl_parity <pack> <model.gguf> [layer ...]` (also `--synthetic [--hadamard] GU/DOWN` and
|
||||
`--bench GU/DOWN`) takes real experts of the file (layers 0 to 3, 10, 20, 30 and 47, two experts each) and compares the
|
||||
card with a float reference and with the CPU path.
|
||||
|
||||
**Measured** (A750, setup's config: 32K context, 300 MiB VRAM reserve, MTP draft of 4, PCIe probe 10.4 GB/s, median of 5 interleaved pairs):
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| Decode kernels against a host replay of their own arithmetic | Equal in every bit (0 of 3,840 gate/up and 0 of 7,680 down values differ; Gyro-S and Gyro-M, 16 cases each) |
|
||||
| Prompt dequantizers against the fp16 codebook | Equal in every bit (0 of 4.9 million values) |
|
||||
| Card's Hadamard rotation against the CPU's | Bit for bit (the fp16 form within 3e-4) |
|
||||
| Card against the float reference | Within 1.7 to 2.0e-2 relative L1 error (3e-2 is the bound every expert type gets) |
|
||||
| CPU path against the float reference | Within 0.9e-2 |
|
||||
|
||||
| | before | now |
|
||||
| --- | ---: | ---: |
|
||||
| Gyro-S prompt, 1,406 tokens | 40.2 tok/s | **82.0 tok/s** (5/5 pairs) |
|
||||
| Gyro-S prompt, 2,377 tokens | | 101.9 tok/s |
|
||||
| Gyro-S prompt, 20,000 tokens | | 195 tok/s |
|
||||
| Gyro-S decode | 13.7 tok/s | **15.8 tok/s** (5/5 pairs) |
|
||||
| Gyro-M decode (8K context) | | 14.2 tok/s (12.6-14.4), prompt 62 tok/s at 1,406 tokens |
|
||||
The card and the CPU path agree within tolerance, not bit for bit. The CPU path uses an int16 codebook and Q8_0
|
||||
activations, the card the int8 codebook and q8_1, as on CUDA.
|
||||
|
||||
The decode speed is the CPU's and the link's: with 8 GB the card holds 410 of the 24,576 experts (the dense weights, the draft layer,
|
||||
the KV cache and the prompt buffers take the rest), so about 15 experts per layer and token come from the CPU pool
|
||||
(5 workers, AVX2, 8 GB/s: the trellis decode is compute bound, 48 cycles a block) and about 5 over the link. Both move about 10 experts a
|
||||
millisecond, so the link takes a bigger share than for the i-quants: for packs with trellis experts the engine scales the share with a 12 GB/s
|
||||
reference instead of 20 (0.48 instead of 0.29 here; fixed 0.45 and 0.60 measured the same; 0.40 gained only 7%). The card's kernels are not the
|
||||
limit: 20 microseconds an expert (53-68 GB/s of weights) against 18 for IQ3_S, 15 experts take 0.3 ms.
|
||||
**Against CUDA.** Gyro-S on an RTX 3060 and on the A750, both with 64 cached experts, no PCIe share and no CPU prompt
|
||||
share, ten prompts of 26 to 100 tokens, 48 greedy tokens with the MTP draft: 7 of the 10 prompts give the same 48 tokens.
|
||||
The other three agree for 9, 9 and 6 tokens and then differ at a near-tie. The first-token KL is 1.3e-4 to 1.1e-2 and
|
||||
the top token is always the same. A 1,406-token prompt gives the same 16 tokens on both.
|
||||
|
||||
Smaller prompt chunks leave room for more cached experts and cost prompt speed: `--prefill 1024` gives 1,090 slots and +13% decode on a
|
||||
1,406-token prompt but -24% prompt speed, `--prefill 512` 1,429 slots, -36%. The default stays `--prefill auto`.
|
||||
> **Measured on:** Arc A750 8 GB, Ryzen 5 5600X, 62 GB RAM, setup's config (32K context, 300 MiB VRAM reserve, MTP draft
|
||||
> of 4, PCIe probe 10.4 GB/s), median of 5 interleaved pairs, engine 0.2.2 era.
|
||||
|
||||
**Gyro-M on 8 GB.** Its dense weights are bigger (2.9 GiB of native projections against 1.9): with setup's 32K context the card is 750 MiB
|
||||
short of one cache slot and the engine stops with that sentence. It starts with `--max-context 8192 --vram-reserve-mib 100 --prefill 1024`
|
||||
(412 slots). A card with 12 GB or more does not need that.
|
||||
| | Result |
|
||||
| --- | ---: |
|
||||
| Gyro-S prompt, 1,406 tokens | 82.0 tok/s |
|
||||
| Gyro-S prompt, 2,377 tokens | 101.9 tok/s |
|
||||
| Gyro-S prompt, 20,000 tokens | 195 tok/s |
|
||||
| Gyro-S decode | 15.8 tok/s |
|
||||
| Gyro-M decode (8K context) | 14.2 tok/s (12.6 to 14.4); prompt 62 tok/s at 1,406 tokens |
|
||||
|
||||
**Memory.** Gyro-S needs about 27.6 GB of VRAM and RAM together; the n-gram table (28 GB) stays on the SSD, so on an Arc setup does not
|
||||
put `--ple-io ram` in the config for Gyro (it does for the other packs on an i915 card). One run used about 24 GB of RAM for the
|
||||
expert arena. Keep the file on a fast SSD.
|
||||
Decode speed is the CPU's and the link's. With 8 GB the card holds 410 of the 24,576 experts, so about 15 experts per
|
||||
layer and token come from the CPU pool (5 workers, AVX2, 8 GB/s; the trellis decode is compute bound) and about 5 over
|
||||
the link. For packs with trellis experts the engine scales the PCIe share with a 12 GB/s reference instead of 20 (0.48
|
||||
instead of 0.29 here). The card's kernels are not the limit: 20 microseconds an expert, against 18 for IQ3_S.
|
||||
|
||||
## Decode speed on NVIDIA (engine 0.2.2)
|
||||
Smaller prompt chunks leave room for more cached experts and cost prompt speed. On the 1,406-token prompt,
|
||||
`--prefill 1024` gives 1,090 slots, +13% decode and -24% prompt speed. `--prefill 512` gives 1,429 slots and -36%
|
||||
prompt speed. The default stays `--prefill auto`.
|
||||
|
||||
The trellis decode kernels read the codebook (2048 two-byte entries, 8 random lookups for every 16 weights) from the global
|
||||
table. On an RTX 3060 that made the gate/up and down kernels wait on the L1 cache, not on memory: 18 L1 sectors for every
|
||||
lookup request, the card's L1 at 64% busy while the DRAM was 27% busy. The kernels now keep the 4 KB codebook in shared memory
|
||||
(`STRATA_IQ_STAGE_GRID=0` reads it in place again) and take four rows per warp, 32 rows per thread block, so the copy is shared
|
||||
and no lane idles in the 160-dot gate/up rows or the 40-dot down rows. The sums are added in the same order as before, so the
|
||||
answers are bit for bit the old ones: the identity gate (10 prompts, 4096 and 20000 token prompts) is identical on Gyro-S,
|
||||
Gyro-M, Q2_0 and IQ3_XXS, and the Q2_0 and IQ3_XXS kernels did not change. A Tesla P100 (compute capability 6.0) runs its own variant of the same idea, STRATA_TQ_STAGE (Gyro-S +30% there, 5 pairs).
|
||||
**Gyro-M on 8 GB.** Its dense weights are bigger (2.9 GiB of native projections against 1.9). With setup's 32K context
|
||||
the card is 750 MiB short of one cache slot and the engine stops with that message. It starts with
|
||||
`--max-context 8192 --vram-reserve-mib 100 --prefill 1024` (412 slots). A card with 12 GB or more does not need that.
|
||||
|
||||
RTX 3060 12 GB, Core Ultra 7, Ubuntu, CUDA 13.4, default flags (MTP draft 4, auto cache), 5 interleaved pairs of whole engine
|
||||
starts, medians of the story, code and facts prompts (200 tokens each), the pinned Gyro files, engine 0.2.1 against 0.2.2.
|
||||
**Memory.** Gyro-S needs about 27.6 GB of VRAM and RAM together. The n-gram table (28 GB) stays on the SSD, so on an Arc
|
||||
setup does not put `--ple-io ram` in the config for Gyro (it does for the other packs on an i915 card). One run used
|
||||
about 24 GB of RAM for the expert arena.
|
||||
|
||||
| | decode before, tok/s | after | gain | pairs faster | 4K / 20K prompt |
|
||||
| --- | ---: | ---: | ---: | --- | --- |
|
||||
| Gyro-S | 54.3 | 62.5 | +15.2% | 5 of 5 (10 of 10 requests per prompt) | unchanged (0.996, 1.000) |
|
||||
| Gyro-M | 50.2 | 54.8 | +9.1% | 5 of 5 (10 of 10) | unchanged (1.004, 0.999) |
|
||||
| Q2_0 (not changed) | 66.4 | 67.1 | 1.010 (noise) | 3 of 5 | unchanged (1.003, 0.999) |
|
||||
## On a Mac
|
||||
|
||||
Against Q2_0 on the same card, before: Gyro-S 54.2 against 64.4 tok/s (16% slower), Gyro-M 50.5 against 65.6 (23% slower).
|
||||
After: Gyro-S 61.1 against 65.5 (7% slower), Gyro-M 54.1 against 64.5 (16% slower). What is left of Gyro-M's gap is mostly its dense weights: in a
|
||||
decode profile its Q8_0 matrix-vector kernels take 26% of the GPU time, and they read as many bytes as the card's memory
|
||||
delivers. The trellis kernels themselves are limited by the shared-memory bank conflicts of the random codebook lookups now. Prompt speed is not changed by this: Gyro prompts stay 7-8% (Gyro-S) and 8-9% (Gyro-M) slower than Q2_0's.
|
||||
Apple Silicon Macs run Gyro-S and Gyro-M with Metal kernels (`metal/kernels`), the same published files and the same
|
||||
Hadamard-folded pack. What a Mac needs, the measured speeds on a 32 GB Mac mini and what was checked are in
|
||||
[MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts). Setup:
|
||||
|
||||
## Credit and licenses
|
||||
```sh
|
||||
./setup.sh --family gyro --model Gyro-S
|
||||
```
|
||||
|
||||
The trellis types, the Hadamard rotation, the CPU and GPU kernels and the pack step were written by Laurent Zuijdwijk
|
||||
(agentionai) in agentionai/Strata `rc2` (9ff02c9, MIT, Strata v0.1.40.3 plus Gyro) and ported here commit by commit with
|
||||
his authorship kept; `third_party/ggml/ggml-common-tq.h` and the MMQ instances come from agentionai/llama.cpp (MIT, the
|
||||
ggml authors). The weights are agentionai's quantization of Qwen3.8-Flash-Next under the Qwen Community License 1.0
|
||||
(<https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF/blob/main/LICENSE>).
|
||||
The app and `strata launch` list it under Gyro models.
|
||||
|
||||
## How it runs (by hand)
|
||||
## Install by hand
|
||||
|
||||
The rest of this page is the authors' own guide for a manual install and their measurements on their machines (RTX 5090,
|
||||
3090, 4090 and a Strix Halo, with Strata rc2). The numbers are theirs; ours are in the next section.
|
||||
Setup does these steps for you. By hand, from a [source checkout](SOURCE_INSTALL.md):
|
||||
|
||||
## 1. Build
|
||||
### 1. Build
|
||||
|
||||
Setup does these steps for you; by hand, from this checkout. For the fast prompt path Strata's GGML dependency is the agentionai llama.cpp fork, which has the trellis types:
|
||||
For the fast prompt path, Strata's GGML dependency can be the agentionai llama.cpp fork, which has the trellis types:
|
||||
|
||||
```sh
|
||||
git clone --depth 1 https://github.com/agentionai/llama.cpp ../agention-llama.cpp
|
||||
# NVIDIA. RTX 50 series: use CUDA 13 (CUDA 12.8 crashes the prompt path on sm_120: issues #220, #224)
|
||||
# NVIDIA. RTX 50 series: use CUDA 13 (CUDA 12.8 crashes the prompt path on sm_120)
|
||||
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \
|
||||
-DCMAKE_CUDA_ARCHITECTURES=120 -DSTRATA_GGML_DIR=$PWD/../agention-llama.cpp
|
||||
# AMD (experimental: kernel parity and an end-to-end run validated on gfx1151; in a non-root container add
|
||||
# --group-add $(getent group render | cut -d: -f3), otherwise HIP finds no device)
|
||||
# AMD (experimental; in a non-root container add --group-add $(getent group render | cut -d: -f3),
|
||||
# otherwise HIP finds no device)
|
||||
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_HIP=ON -DSTRATA_PREFILL_MMQ=ON \
|
||||
-DCMAKE_HIP_ARCHITECTURES=gfx1201 -DSTRATA_GGML_DIR=$PWD/../agention-llama.cpp
|
||||
cmake --build build -j
|
||||
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## 2. Download and pack
|
||||
### 2. Download and pack
|
||||
|
||||
```sh
|
||||
.venv/bin/pip install huggingface_hub hf_transfer
|
||||
@@ -183,11 +213,10 @@ HF_HUB_ENABLE_HF_TRANSFER=1 .venv/bin/hf download agentionai/Qwen3.8-Flash-Next-
|
||||
build/native_expert_parity --load-check packs/gyro-s models/Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf
|
||||
```
|
||||
|
||||
The pack is 1.5 GB (dense weights, tokenizer, the Hadamard and expert tables); the experts are read from the
|
||||
GGUF. Keep the GGUF on a fast local SSD: the n-gram table is read from it for every token.
|
||||
`--load-check` must end with `0 failures`.
|
||||
The pack is 1.5 GB (dense weights, tokenizer, the Hadamard and expert tables). The experts are read from the GGUF, so
|
||||
keep it on a fast local SSD. `--load-check` must end with `0 failures`.
|
||||
|
||||
## 3. Draft head
|
||||
### 3. Draft head
|
||||
|
||||
```sh
|
||||
.venv/bin/python tools/mtp_fetch.py fetch --out mtp
|
||||
@@ -196,7 +225,7 @@ GGUF. Keep the GGUF on a fast local SSD: the n-gram table is read from it for ev
|
||||
cp data/draft_vocab.bin mtp/rt/draft_vocab.bin
|
||||
```
|
||||
|
||||
## 4. Serve
|
||||
### 4. Serve
|
||||
|
||||
Save as `strata-gyro-s.json` at the repository root:
|
||||
|
||||
@@ -223,147 +252,126 @@ Save as `strata-gyro-s.json` at the repository root:
|
||||
.venv/bin/python -m serve.server --engine strata --config strata-gyro-s.json --port 8080
|
||||
```
|
||||
|
||||
The web interface is at `http://127.0.0.1:8080`; API clients use `http://127.0.0.1:8080/v1` (OpenAI) or
|
||||
`/v1/messages` (Anthropic). `--expert-cache auto` sizes the expert cache to the free VRAM: on a 32 GB card at
|
||||
128k context it holds 23,464 of 24,576 experts (22.9 GiB). On smaller cards it holds fewer and streams the rest
|
||||
from RAM; nothing else changes.
|
||||
The web interface is at `http://127.0.0.1:8080`. API clients use `http://127.0.0.1:8080/v1` (OpenAI) or `/v1/messages`
|
||||
(Anthropic). `--expert-cache auto` sizes the expert cache to the free VRAM. On a 32 GB card at 128K context it holds
|
||||
23,464 of 24,576 experts (22.9 GiB). On smaller cards it holds fewer and streams the rest from RAM.
|
||||
|
||||
## Measured here (the port, 2026-10-15/16)
|
||||
## Numerics
|
||||
|
||||
One AMD Radeon AI PRO R9700 (gfx1201, ROCm 7), one card of four, exclusive box lock, `--prefill auto --spec 4 --spec-min-p 0.5
|
||||
--mtp --kv int8`, 22K context, interleaved pairs (the arm that goes first alternates), medians, the files of the first
|
||||
Gyro release (2026-10-07). A = agentionai/Strata rc2 9ff02c9, B = this port.
|
||||
- **Decode** (Strata's own expert kernels) reads the trellis weights through an int8 copy of the codebook with one
|
||||
global scale (`tq_lut_i8`, relative RMS error 0.85% of the codebook's standard deviation, far below the quantization
|
||||
error itself), against q8_1 activations.
|
||||
- **Prompt processing** decodes them differently. The FP16 path uses the fp16 codebook (`tq_lut_f16`, what GGML's own
|
||||
dequantizer reads). The MMQ path uses the llama.cpp fork's kernels, which read the int8 codebook as well.
|
||||
- Decode and prompt results therefore agree within tolerance, not bit for bit. `native_expert_parity` checks one expert
|
||||
on the decode kernels against GGML's float dequantizer and a float SwiGLU: relative L1 error below 3e-2, the bound
|
||||
every expert type gets there. Its "bitwise equal" lines compare Strata's decode-once and per-entry decode kernels with
|
||||
each other, not with a float reference.
|
||||
- Strata's own CPU kernels for the trellis types (`src/kernels/cpu/tq_*.cpp`, AVX2 and AVX-512, picked at run time)
|
||||
take about 45% of ggml-cpu's time per expert on AVX2 and decode a block once for all the tokens of a verify window
|
||||
(`build/tq_cpu_parity --bench` measures them on your CPU). `STRATA_NO_TQ_KERNELS=1` goes back to ggml-cpu.
|
||||
`STRATA_TQ_ISA=avx2` or `scalar` caps the instruction set.
|
||||
|
||||
| B, ggml of | file | decode tok/s A / B (B/A) | pairs B faster | prompt 4K ms A / B | prompt 20K ms A / B |
|
||||
## Known limits
|
||||
|
||||
- The prompt helper on a second GPU does not run for a Gyro pack. Setup of two cards still runs the layer split and the
|
||||
decode peer. The prompt peer says it is not used for a Hadamard-folded pack, and the prompt stays on the first card.
|
||||
Prompt processing on a peer GPU over P2P is not validated end to end.
|
||||
- Token parity against llama.cpp has not been measured.
|
||||
- `strata --tokens-file ... --max-new N` (benchmark mode) decodes N tokens past the end of the answer. Compare speeds
|
||||
only up to the answer's end.
|
||||
- AMD: the trellis kernels pass parity on gfx1151 (ROCm 7.2.1) and a short end-to-end run there generated correctly
|
||||
(28 to 31 tokens/s). There was no speed tuning or RDNA4 speed run. An AMD GPU other than gfx1151 runs with a warning.
|
||||
- Gyro-M: 29.9 GiB of experts do not all fit a 32 GB card next to the KV cache. `auto` streams the rest.
|
||||
|
||||
## Credit and licenses
|
||||
|
||||
The trellis types, the Hadamard rotation, the CPU and GPU kernels and the pack step were written by Laurent Zuijdwijk
|
||||
(agentionai) in agentionai/Strata `rc2` (MIT) and ported here commit by commit with his authorship kept.
|
||||
`third_party/ggml/ggml-common-tq.h` and the MMQ instances come from agentionai/llama.cpp (MIT, the ggml authors). The
|
||||
weights are agentionai's quantization of Qwen3.8-Flash-Next under the Qwen Community License 1.0
|
||||
(<https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF/blob/main/LICENSE>).
|
||||
|
||||
## History / older measurements
|
||||
|
||||
### Engine 0.2.1 against 0.2.2 (decode kernels)
|
||||
|
||||
> **Measured on:** RTX 3060 12 GB, Core Ultra 7, Ubuntu, CUDA 13.4, default flags, 5 interleaved pairs of whole engine
|
||||
> starts, the pinned Gyro files.
|
||||
|
||||
| | Decode before, tok/s | After | Gain | 4K / 20K prompt |
|
||||
| --- | ---: | ---: | ---: | --- |
|
||||
| Gyro-S | 54.3 | 62.5 | +15.2% | unchanged (0.996, 1.000) |
|
||||
| Gyro-M | 50.2 | 54.8 | +9.1% | unchanged (1.004, 0.999) |
|
||||
| Q2_0 (not changed) | 66.4 | 67.1 | 1.010 (noise) | unchanged (1.003, 0.999) |
|
||||
|
||||
Against Q2_0 before the change: Gyro-S 54.2 against 64.4 tok/s (16% slower), Gyro-M 50.5 against 65.6 (23% slower).
|
||||
Gyro-S gained +30% on a Tesla P100 from the `STRATA_TQ_STAGE` variant (5 pairs).
|
||||
|
||||
### Arc A750 before the XMX prompt path
|
||||
|
||||
> **Measured on:** Arc A750 8 GB, setup's config, median of 5 interleaved pairs.
|
||||
|
||||
| | Before | Later |
|
||||
| --- | ---: | ---: |
|
||||
| Gyro-S prompt, 1,406 tokens | 40.2 tok/s | 82.0 tok/s |
|
||||
| Gyro-S decode | 13.7 tok/s | 15.8 tok/s |
|
||||
|
||||
### The port against agentionai's rc2 (2026-10-15/16)
|
||||
|
||||
> **Measured on:** one AMD Radeon AI PRO R9700 (gfx1201, ROCm 7), one card of four, `--prefill auto --spec 4
|
||||
> --spec-min-p 0.5 --mtp --kv int8`, 22K context, interleaved pairs, medians, the first Gyro release files (2026-10-07).
|
||||
> A = agentionai/Strata rc2, B = this port (Strata 0.1.45 base).
|
||||
|
||||
| B's ggml | File | Decode tok/s A / B | Pairs B faster | Prompt 4K ms A / B | Prompt 20K ms A / B |
|
||||
| --- | --- | ---: | --- | ---: | ---: |
|
||||
| agentionai fork | Gyro-S | 85.4 / 89.7 (1.051) | 5/5 | 5418 / 5313 (0.98) | 14358 / 13859 (0.97) |
|
||||
| agentionai fork | Gyro-M | 72.7 / 82.7 (1.136) | 5/5 | 5183 / 5520 (1.07) | 13421 / 14164 (1.06) |
|
||||
| upstream + overlay (the ready-made engine) | Gyro-S | 85.3 / 89.6 (1.050) | 5/5 | 5413 / 5298 (0.98) | 14372 / 13854 (0.96) |
|
||||
| upstream + overlay | Gyro-M | 72.6 / 82.6 (1.138) | 5/5 | 5682 / 5524 (0.97) | 15005 / 14160 (0.94) |
|
||||
| upstream, no overlay (FP16 prompt path) | Gyro-S | 85.3 / 89.6 (1.050) | 5/5 | 5425 / 6939 (1.28) | 14364 / 21677 (1.51) |
|
||||
| agentionai fork | Gyro-S | 85.4 / 89.7 | 5/5 | 5418 / 5313 | 14358 / 13859 |
|
||||
| agentionai fork | Gyro-M | 72.7 / 82.7 | 5/5 | 5183 / 5520 | 13421 / 14164 |
|
||||
| upstream + overlay (the ready-made engine) | Gyro-S | 85.3 / 89.6 | 5/5 | 5413 / 5298 | 14372 / 13854 |
|
||||
| upstream + overlay | Gyro-M | 72.6 / 82.6 | 5/5 | 5682 / 5524 | 15005 / 14160 |
|
||||
| upstream, no overlay (FP16 prompt path) | Gyro-S | 85.3 / 89.6 | 5/5 | 5425 / 6939 | 14364 / 21677 |
|
||||
|
||||
Decode is faster than rc2's because the port sits on the newer Strata (0.1.45); the same engine change makes IQ3_XXS 7% faster
|
||||
(rc2 86.9, port 93.1 tok/s, 2 pairs). Prompts of the ISTA files gained 11-16% from it too (IQ3_XXS 4K 6184 to 5480 ms, 20K
|
||||
16587 to 13864 ms). Gyro-M's first run against rc2 showed its prompts 5-7% slower; in the second rc2's own Gyro-M prompts
|
||||
took 15.0 s instead of 13.4 s (the 92 GB file's page-cache state), the port's stayed at 14.2 s, and Gyro-M's prompts are 3-6%
|
||||
faster than rc2's: that difference was rc2's baseline, not a regression. Without the overlay (plain upstream ggml) the trellis
|
||||
prompt kernels are missing and prompts go through the FP16 path: 28% (4K) to 51% (20K) slower than rc2, and a different
|
||||
rounding. Decode is unaffected either way.
|
||||
Without the overlay, prompts are 28% (4K) to 51% (20K) slower than rc2's. Decode is unaffected. The identity gate (10
|
||||
prompts, 4096 and 20000 token prompts) was identical between rc2 and the port for Gyro-S and Gyro-M, and between the
|
||||
split base and the port for Q2_0 and IQ3_XXS, on AMD gfx1201 and NVIDIA sm_86. The pack step writes byte-identical files
|
||||
to rc2's tools for Gyro-S.
|
||||
|
||||
Identity (the fixed gate flags, 10 prompts, exact_pp 4096 and 20000 tokens, same card for both arms):
|
||||
### The authors' measurements (rc2)
|
||||
|
||||
- Q2_0 and IQ3_XXS, f-split against the port (upstream ggml): 10/10 identical, residual dumps and outputs IDENTICAL.
|
||||
- Gyro-S and Gyro-M, rc2 against the port with the fork's ggml and with upstream + overlay: 10/10 identical; exact_pp 4096 /
|
||||
20000 residual dumps and outputs IDENTICAL (run alone; a run beside other engines on the box once showed a different dump).
|
||||
- Q2_0 and IQ3_XXS, f-split against the port with the overlay on, on AMD gfx1201 and on NVIDIA sm_86: IDENTICAL.
|
||||
- The pack step (`tools/iq_pack.py --compat-bf16`) writes byte-identical files to rc2's tools for Gyro-S.
|
||||
> **Measured on:** the authors' machines, agentionai/Strata rc2, 2026-10-02 to 04; greedy or the stated sampling, MTP
|
||||
> draft `--spec 4 --spec-min-p 0.5`, int8 KV. llama.cpp is the agentionai fork with the CUDA trellis fixes. The numbers
|
||||
> are theirs.
|
||||
|
||||
## Measured (the authors', rc2)
|
||||
Decode tokens/s, server protocol (128K context, `auto` cache, thinking off, temperature 0.7, top-k 20, top-p 0.95,
|
||||
min-p 0.05, 512 new tokens, 3 repetitions):
|
||||
|
||||
All runs: greedy or the stated sampling, same prompt tokens as llama.cpp, MTP draft `--spec 4 --spec-min-p 0.5`,
|
||||
int8 KV. "llama.cpp" is the agentionai fork with the CUDA trellis fixes (in its `main` since 2026-10-04).
|
||||
| GPU and file | Experts cached | Prose | JSON | Code | Copy | Prefill 4K | Prefill 16K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| RTX 5090 32 GB, Gyro-S, Strata | 23,463 | 131.7 | 200.1 | 195.2 | 201.8 | 3,456 | 4,883 |
|
||||
| RTX 5090, Gyro-S, llama.cpp with MTP draft | | 116.8 | 213.0 | 179.0 | 220.7 | 2,520 | 2,471 |
|
||||
| RTX 5090, Gyro-S, llama.cpp no draft | | 121.8 | 123.2 | 122.5 | 121.2 | | |
|
||||
| RTX 5090, Gyro-M, Strata | 18,007 | 113.8 | 170.8 | 153.0 | 175.7 | 3,369 | 4,854 |
|
||||
| RTX 3090 24 GB, Gyro-S, Strata | 15,756 | 31.8 | 45.2 | 42.2 | 45.1 | 1,402 | 1,863 |
|
||||
| RTX 3090, Gyro-M, Strata | 11,811 | 27.8 | 39.4 | 34.4 | 37.7 | 1,363 | 1,910 |
|
||||
| RTX 3090, Gyro-S, llama.cpp `-ncmoe 10`, 32K, no draft | | 16.0 | 16.2 | 16.2 | 16.5 | 468 | 479 |
|
||||
|
||||
### RTX 5090 (32 GB), Gyro-S, all experts on the GPU (2026-10-03)
|
||||
Single prompts of 34 to 177 tokens, 384 new tokens, `--max-context 16384`, Gyro-S, RTX 5090, all experts cached
|
||||
(2026-10-03): prose 139.9, JSON 221.3, code 209.0 tok/s. Gyro-M on the same card with llama.cpp needs `-ncmoe 8` and
|
||||
decodes at 70.8 tok/s without a draft. A single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3,
|
||||
JSON 76.3 tok/s. On 24 GB the CPU matters (the box had a Zen 2 EPYC, AVX2 only). These Strata rows predate Strata's
|
||||
own trellis CPU kernels.
|
||||
|
||||
Decode tokens/s, single prompts of 34-177 tokens, 384 new tokens, `--max-context 16384` (all 24,576 experts cached):
|
||||
Smaller expert caches, emulated on the RTX 5090 (2026-10-02, before the int8 decode kernels). A real 8 or 16 GB card is
|
||||
a slower GPU, often on a slower link: read these rows as "it stays usable", not as a prediction for a specific card.
|
||||
|
||||
| | prose | JSON | code |
|
||||
|---|---:|---:|---:|
|
||||
| Strata | 139.9 | 221.3 | 209.0 |
|
||||
|
||||
Server, rc2's `gyro_setup.sh` config (128k context, 23,463 experts cached), thinking off, temperature 0.7,
|
||||
top-k 20, top-p 0.95, min-p 0.05, 512 new tokens, 3 repetitions:
|
||||
|
||||
| | prose | JSON | code | copy |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Strata | 131.7 | 200.1 | 195.2 | 201.8 |
|
||||
| llama.cpp, MTP draft | 116.8 | 213.0 | 179.0 | 220.7 |
|
||||
| llama.cpp, no draft | 121.8 | 123.2 | 122.5 | 121.2 |
|
||||
|
||||
Long real-text prompts, prefill tokens/s: 4k tokens 3,456 (llama.cpp 2,520), 16k tokens 4,883 (llama.cpp 2,471).
|
||||
|
||||
### RTX 5090, Gyro-M (2026-10-03)
|
||||
|
||||
The experts (29.9 GiB) do not all fit next to the KV cache; `auto` caches 18,007 of 24,576. Same server protocol:
|
||||
prose 113.8, JSON 170.8, code 153.0, copy 175.7 tokens/s; prefill 4k 3,369, 16k 4,854 tokens/s. llama.cpp needs
|
||||
`-ncmoe 8` for this file on a 32 GB card and decodes at 70.8 tokens/s without a draft.
|
||||
|
||||
### RTX 3090 (24 GB), 2026-10-03
|
||||
|
||||
Same server protocol, 128k context, `auto` cache:
|
||||
|
||||
| | experts cached | prose | JSON | code | copy | prefill 4k | prefill 16k |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| Gyro-S | 15,756 | 31.8 | 45.2 | 42.2 | 45.1 | 1,402 | 1,863 |
|
||||
| Gyro-M | 11,811 | 27.8 | 39.4 | 34.4 | 37.7 | 1,363 | 1,910 |
|
||||
| llama.cpp, Gyro-S, `-ncmoe 10`, 32k context, no draft | (10 layers on the CPU) | 16.0 | 16.2 | 16.2 | 16.5 | 468 (`-ncmoe 11`) | 479 |
|
||||
|
||||
On a 24 GB card, cache misses run on the CPU, so the CPU matters: this box had a Zen 2 EPYC (AVX2, no AVX-512).
|
||||
The llama.cpp row is the build before the CUDA trellis fixes; those added about 3 % at 24 GB, where the CPU layers
|
||||
dominate.
|
||||
|
||||
The Strata rows predate Strata's own CPU kernels for the trellis types (`src/kernels/cpu/tq_*.cpp`, AVX2 and
|
||||
AVX-512, picked at run time). Before them a cache miss ran ggml-cpu's `vec_dot`, one token at a time; the new kernels
|
||||
take about 45 % of its time per expert on AVX2 and decode a block once for all the tokens of a verify window
|
||||
(`build/tq_cpu_parity --bench` measures them on your CPU). `STRATA_NO_TQ_KERNELS=1` goes back to ggml-cpu;
|
||||
`STRATA_TQ_ISA=avx2` or `scalar` caps the instruction set (for comparisons).
|
||||
|
||||
### Smaller caches (RTX 5090, 2026-10-02, before the int8 decode kernels)
|
||||
|
||||
| Expert cache | prose | JSON | code |
|
||||
|---|---:|---:|---:|
|
||||
| Expert cache | Prose | JSON | Code |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| 24,576 slots (all experts) | 116.4 | 194.3 | 173.5 |
|
||||
| 22,000 slots | 111.4 | 175.7 | 166.1 |
|
||||
| 9,000 slots (about a 16 GB card's room) | 81.4 | 74.0 | 96.8 |
|
||||
| 3,000 slots (about an 8 GB card's room) | 48.0 | 52.4 | 57.9 |
|
||||
|
||||
The smaller caches were emulated on the 5090 (fast GPU, PCIe 5.0 x16 at ~51 GB/s). A real 8 or 16 GB card is a
|
||||
slower GPU, often on a slower link: read those rows as "it stays usable", not as a prediction for a specific card.
|
||||
The decoded output was read and is fluent and correct; token-level parity against llama.cpp has not been
|
||||
checked. A real chat at 128k context worked through the web interface.
|
||||
|
||||
### Two GPUs (2026-10-04, 2× RTX 4090, no NVLink/P2P)
|
||||
|
||||
- **Layer split** (Strata splits the layers across both cards automatically) works with Gyro, including the
|
||||
idle-card prompt helper (`STRATA_PREFILL_HELP=1`). Repeated runs give identical output.
|
||||
- **Decode with a second card** (`--peer-device 1` or `--expert-cache-device1 auto`) works and matches single-GPU output.
|
||||
- **Prompt processing on the peer card** needs GPU-to-GPU access (NVLink or data-centre GPUs); without it Strata keeps
|
||||
the prompt on the first GPU and logs `no P2P`.
|
||||
|
||||
Single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3, JSON 76.3 tokens/s.
|
||||
|
||||
## Numerics
|
||||
|
||||
- **Decode** (Strata's own expert kernels) reads the trellis weights through an int8 copy of the codebook with one
|
||||
global scale (`tq_lut_i8`, relative RMS error 0.85 % of the codebook's standard deviation, far below the
|
||||
quantization error itself), against q8_1 activations.
|
||||
- **Prompt processing** decodes them otherwise: the FP16 path through the fp16 codebook (`tq_lut_f16`, what GGML's
|
||||
own dequantizer reads); the MMQ path through the llama.cpp fork's kernels, which read the int8 codebook as well
|
||||
on the fork's current `main`.
|
||||
- So decode and prompt results agree within tolerance, not bit for bit. `native_expert_parity` checks one expert
|
||||
on the decode kernels against GGML's float dequantizer and a float SwiGLU: relative L1 error below 3e-2, the
|
||||
bound every expert type gets there. Its "bitwise equal" lines compare Strata's decode-once and per-entry decode
|
||||
kernels with each other, not with a float reference.
|
||||
|
||||
## Known limits
|
||||
|
||||
- **Not in this port** (rc2 has them; see the list at the end of this page): the prompt helper on a second GPU for a Gyro
|
||||
pack. Setup of two cards still runs the layer split and the decode peer; the prompt peer says it is not used for a
|
||||
Hadamard-folded pack and the prompt stays on the first card. Gyro-L is not supported.
|
||||
- **Source builds:** the SYCL tree has the trellis kernels (Intel Arc, above), and so has the Metal engine (On a Mac, below).
|
||||
|
||||
- **Not measured yet:** token parity against llama.cpp.
|
||||
- **Benchmark mode only:** `strata --tokens-file ... --max-new N` decodes N tokens past the end of the answer;
|
||||
compare speeds only up to the answer's end.
|
||||
- **AMD:** the trellis kernels pass parity on gfx1151 (ROCm 7.2.1) and a short end-to-end run there generated
|
||||
correctly (~28-31 tokens/s); no speed tuning or RDNA4 run yet.
|
||||
- **Gyro-M:** 29.9 GiB of experts do not all fit a 32 GB card next to the KV cache; `auto` streams the rest.
|
||||
- **Not validated end to end yet** (Strata runs them and logs a warning): an AMD GPU other than gfx1151, and
|
||||
prompt processing on a peer GPU over P2P.
|
||||
|
||||
## On a Mac
|
||||
|
||||
Apple Silicon Macs run Gyro-S and Gyro-M from 0.2.2 (Metal kernels in `metal/kernels`, the same published files, the same
|
||||
Hadamard-folded pack). What a Mac needs, the measured speeds on a 32 GB Mac mini and what was checked are in
|
||||
[MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts). Setup: `./setup.sh --family gyro --model Gyro-S` (the app and
|
||||
`strata launch` list it under Gyro models).
|
||||
Two RTX 4090 without NVLink or P2P (2026-10-04): the layer split works with Gyro, including the idle-card prompt helper
|
||||
(`STRATA_PREFILL_HELP=1`), and repeated runs give identical output. Decode with a second card (`--peer-device 1` or
|
||||
`--expert-cache-device1 auto`) matches single-GPU output. Prompt processing on the peer card needs GPU-to-GPU access
|
||||
(NVLink or data-centre GPUs). Without it Strata keeps the prompt on the first GPU and logs `no P2P`.
|
||||
+85
-58
@@ -1,88 +1,115 @@
|
||||
# Image generation (Qwen-Image-2.1-Turbo)
|
||||
|
||||
Strata's server can also make images: `POST /v1/images/generations`, OpenAI's images API. The model is
|
||||
[Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) (8 steps, CFG 1) in
|
||||
This page is for people who want Strata's server to make images through `POST /v1/images/generations`, OpenAI's
|
||||
images API. It is optional, off unless you ask for it, and separate from reading pictures in chat (for that, use
|
||||
`strata launch --vision on` with a supported model and reader).
|
||||
|
||||
It needs a [source checkout](SOURCE_INSTALL.md) and build tools. The steps below are not part of a normal app install.
|
||||
|
||||
The model is [Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) (8 steps, CFG 1) in
|
||||
[Unsloth's GGUF quants](https://huggingface.co/unsloth/Qwen-Image-2.1-Turbo-GGUF), run by
|
||||
[stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) (`sd-cli`), which setup compiles for your GPU.
|
||||
It is optional and off unless you ask for it.
|
||||
|
||||
The model is under the **Qwen Research License**, not Apache 2.0: read
|
||||
It is under the **Qwen Research License**, not Apache 2.0. Read
|
||||
[its license](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo/blob/main/LICENSE) before you use the images
|
||||
commercially.
|
||||
|
||||
## Set it up
|
||||
## Quick start
|
||||
|
||||
./setup.sh --setup --images qwen-image (Linux)
|
||||
START-HERE.bat --setup --images qwen-image (Windows)
|
||||
```sh
|
||||
./setup.sh --setup --images qwen-image # Linux
|
||||
START-HERE.bat --setup --images qwen-image # Windows
|
||||
```
|
||||
|
||||
or, for an install that is already set up, without running the whole setup again:
|
||||
For an install that is already set up, without running the whole setup again:
|
||||
|
||||
.venv/bin/python tools/image/install.py --config strata-<model>.json
|
||||
```sh
|
||||
.venv/bin/python tools/image/install.py --config strata-<model>.json
|
||||
```
|
||||
|
||||
Setup compiles `sd-cli` (stable-diffusion.cpp at a pinned commit) into `engine-image/`, downloads three files into
|
||||
the data folder's `models/qwen-image-2.1-turbo/` and checks each against its SHA-256, and writes an `"images"` section
|
||||
into the model's run config.
|
||||
Then request an image:
|
||||
|
||||
| piece | file | size |
|
||||
|---|---|---|
|
||||
| denoiser (by GPU memory) | `qwen-image-2.1-turbo-Q4_K_M.gguf` up to 8 GB, `Q6_K` up to 12 GB, `Q8_0` above | 4.2 / 6.3 / 7.6 GB |
|
||||
| text encoder | `Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf` | 5.1 GB |
|
||||
```sh
|
||||
curl http://127.0.0.1:8080/v1/images/generations -H "Content-Type: application/json" \
|
||||
-d '{"prompt": "a cartoon sloth mascot waving, flat vector illustration", "size": "1024x1024", "seed": 42}'
|
||||
```
|
||||
|
||||
## What setup does
|
||||
|
||||
Setup compiles `sd-cli` (stable-diffusion.cpp at a pinned commit) into `engine-image/`, downloads three files into the
|
||||
data folder's `models/qwen-image-2.1-turbo/`, checks each against its SHA-256 and writes an `"images"` section into the
|
||||
model's run config.
|
||||
|
||||
| Piece | File | Size |
|
||||
| --- | --- | --- |
|
||||
| Denoiser, chosen by GPU memory | `qwen-image-2.1-turbo-Q4_K_M.gguf` up to 8 GB, `Q6_K` up to 12 GB, `Q8_0` above | 4.2 / 6.3 / 7.6 GB |
|
||||
| Text encoder | `Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf` | 5.1 GB |
|
||||
| VAE | `qwen_image_2.1_vae_bf16.safetensors` | 0.7 GB |
|
||||
|
||||
Options: `--image-quant Q4_K_M|Q6_K|Q8_0` (a bigger one than recommended is kept, with a warning),
|
||||
`--image-backend cuda|hip|vulkan|cpu`, `--image-gpu-policy auto|keep|shrink|unload`, `--images no` (takes the section
|
||||
out of the config).
|
||||
### Setup options
|
||||
|
||||
Backends: CUDA on NVIDIA (needs the CUDA Toolkit), HIP on AMD cards with their own memory (Linux, the same ROCm as the
|
||||
engine), Vulkan on AMD APUs/iGPUs (Radeon 780M, Strix Point/Halo), Intel Arc and AMD under Windows. A Linux PC without
|
||||
the Vulkan development files gets the LunarG Vulkan SDK unpacked into `third_party/` (no sudo); on Windows install the
|
||||
Vulkan SDK first. On Vulkan setup adds `--vae-tiling --vae-tile-overlap 0.25`: an untiled 1024x1024 VAE decode on a Radeon 780M is one
|
||||
GPU job long enough to trip the kernel driver's job timeout (no image), and a quarter's overlap decodes in half the
|
||||
time of sd.cpp's default half overlap.
|
||||
| Option | Effect |
|
||||
| --- | --- |
|
||||
| `--image-quant Q4_K_M\|Q6_K\|Q8_0` | Picks the denoiser. A bigger one than recommended is kept, with a warning. |
|
||||
| `--image-backend cuda\|hip\|vulkan\|cpu` | Picks the backend. |
|
||||
| `--image-gpu-policy auto\|keep\|shrink\|unload` | Sets `images.gpu_policy` ([below](#sharing-the-gpu-with-the-text-model)). |
|
||||
| `--images no` | Takes the section out of the config. |
|
||||
|
||||
## Use it
|
||||
### Backends
|
||||
|
||||
curl http://127.0.0.1:8080/v1/images/generations -H "Content-Type: application/json" \
|
||||
-d '{"prompt": "a cartoon sloth mascot waving, flat vector illustration", "size": "1024x1024", "seed": 42}'
|
||||
| Backend | Used for |
|
||||
| --- | --- |
|
||||
| CUDA | NVIDIA. Needs the CUDA Toolkit. |
|
||||
| HIP | AMD cards with their own memory, on Linux, with the same ROCm as the engine. |
|
||||
| Vulkan | AMD APUs and iGPUs (Radeon 780M, Strix Point and Halo), Intel Arc, and AMD under Windows. |
|
||||
| CPU | Fallback. |
|
||||
|
||||
| field | |
|
||||
|---|---|
|
||||
| `prompt` | the description (required) |
|
||||
| `size` | `WIDTHxHEIGHT`, each side a multiple of 32 from 256 to 2048 (`max_side`); default and `auto`: `1024x1024` |
|
||||
| `n` | images, 1 to 4 (`max_n`); image *i* uses seed + *i* |
|
||||
| `response_format` | `b64_json` (default) or `url` (`/v1/images/files/<random name>.png`; the server keeps the newest 64) |
|
||||
| `seed` | Strata's: a whole number for a repeatable image (default: random; the answer says which) |
|
||||
| `steps` | Strata's: 8 (Turbo's schedule). Another count uses stable-diffusion.cpp's own schedule, which Turbo was not made for |
|
||||
A Linux PC without the Vulkan development files gets the LunarG Vulkan SDK unpacked into `third_party/` (no sudo). On
|
||||
Windows, install the Vulkan SDK first. On Vulkan, setup adds `--vae-tiling --vae-tile-overlap 0.25`. An untiled
|
||||
1024x1024 VAE decode on a Radeon 780M is one GPU job long enough to trip the kernel driver's job timeout, and the
|
||||
quarter overlap decodes in half the time of sd.cpp's default half overlap.
|
||||
|
||||
The answer is OpenAI's shape (`created`, `data[].b64_json` or `data[].url`, `revised_prompt` = the prompt) plus
|
||||
## Request fields
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `prompt` | The description (required). |
|
||||
| `size` | `WIDTHxHEIGHT`, each side a multiple of 32 from 256 to 2048 (`max_side`). Default and `auto`: `1024x1024`. |
|
||||
| `n` | Images, 1 to 4 (`max_n`). Image *i* uses seed + *i*. |
|
||||
| `response_format` | `b64_json` (default) or `url` (`/v1/images/files/<random name>.png`; the server keeps the newest 64). |
|
||||
| `seed` | Strata's: a whole number for a repeatable image. Default: random; the answer says which. |
|
||||
| `steps` | Strata's: 8 (Turbo's schedule). Another count uses stable-diffusion.cpp's own schedule, which Turbo was not made for. |
|
||||
|
||||
The answer has OpenAI's shape (`created`, `data[].b64_json` or `data[].url`, `revised_prompt` = the prompt) plus
|
||||
`data[].seed` and a `strata` object with the seconds, steps and backend. The settings are the model card's for
|
||||
stable-diffusion.cpp: 8 steps, CFG 1, euler, Turbo's own sigmas, `--diffusion-fa`. An image from the API is the PNG
|
||||
plain `sd-cli` writes with the same arguments, byte for byte.
|
||||
that plain `sd-cli` writes with the same arguments, byte for byte.
|
||||
|
||||
## Sharing the GPU with the text model
|
||||
|
||||
One job at a time: an image job takes the same turn a chat request takes, so a chat request waits for the image and
|
||||
the other way round. What the text model does meanwhile is the config's `images.gpu_policy`:
|
||||
One job runs at a time. An image job takes the same turn a chat request takes, so a chat request waits for the image
|
||||
and the other way round. The config's `images.gpu_policy` decides what the text model does meanwhile.
|
||||
|
||||
| policy | the text model during an image job |
|
||||
|---|---|
|
||||
| `auto` (default) | `keep` when the image job fits next to it (an APU always), else `shrink` when the engine was started with `--vram-elastic`, else `unload` |
|
||||
| `keep` | stays loaded and idle |
|
||||
| `shrink` | gives VRAM from its expert cache (as `POST /v1/vram`) and takes it back afterwards; the conversation stays |
|
||||
| `unload` | its conversation is saved to a file, the engine is stopped, and the next start restores the conversation (`"reload": "eager"` starts it again right after the image; default `lazy`: with the next chat request) |
|
||||
| Policy | The text model during an image job |
|
||||
| --- | --- |
|
||||
| `auto` (default) | `keep` when the image job fits next to it (an APU always), else `shrink` when the engine was started with `--vram-elastic`, else `unload`. |
|
||||
| `keep` | Stays loaded and idle. |
|
||||
| `shrink` | Gives VRAM from its expert cache (as `POST /v1/vram`) and takes it back afterwards. The conversation stays. |
|
||||
| `unload` | Its conversation is saved to a file and the engine is stopped. The next start restores the conversation. `"reload": "eager"` starts it again right after the image; the default `lazy` starts it with the next chat request. |
|
||||
|
||||
Each image runs its own `sd-cli` process, so no image memory stays on the GPU between images. Loading the three
|
||||
files takes a few seconds per image from the page cache.
|
||||
Each image runs its own `sd-cli` process, so no image memory stays on the GPU between images. Loading the three files
|
||||
takes a few seconds per image from the page cache.
|
||||
|
||||
## Speed
|
||||
## History / older measurements
|
||||
|
||||
Seconds per image through the API, Q8_0, 8 steps (one `sd-cli` process per image, loading included):
|
||||
> **Measured on:** seconds per image through the API, Q8_0, 8 steps, one `sd-cli` process per image with loading
|
||||
> included. Strata version not recorded.
|
||||
|
||||
| GPU | backend | 1024x1024 | 512x512 |
|
||||
|---|---|---|---|
|
||||
| RTX A4000 16 GB | CUDA | 27 | 8.8 |
|
||||
| Radeon 780M (iGPU) | HIP | 131 | 29 |
|
||||
| Radeon 780M (iGPU) | Vulkan | 179 | 43 |
|
||||
| GPU | Backend | 1024x1024 | 512x512 |
|
||||
| --- | --- | --- | --- |
|
||||
| RTX A4000 16 GB | CUDA | 27 s | 8.8 s |
|
||||
| Radeon 780M (iGPU) | HIP | 131 s | 29 s |
|
||||
| Radeon 780M (iGPU) | Vulkan | 179 s | 43 s |
|
||||
|
||||
On the 780M HIP is faster but runs through the ROCm path whose queue evictions reset this GPU on current firmware
|
||||
(a driver/firmware defect, not this feature; no reset was seen in these image runs); Vulkan is setup's default there, `--image-backend hip` picks HIP.
|
||||
On the 780M, HIP is faster but runs through the ROCm path whose queue evictions reset this GPU on the firmware tested
|
||||
(a driver or firmware defect, not this feature; no reset was seen in these image runs). Vulkan is setup's default
|
||||
there. `--image-backend hip` picks HIP.
|
||||
+95
-373
@@ -1,426 +1,148 @@
|
||||
# Installing Strata
|
||||
# Install Strata
|
||||
|
||||
Everything about installing, starting, updating and configuring Strata, on Windows, Linux and Mac, with an NVIDIA
|
||||
or an AMD graphics card or an Apple Silicon Mac. The short version is in the [README](../README.md#install). An AI
|
||||
coding assistant can do all of this for you with [AI_SETUP.md](AI_SETUP.md).
|
||||
This page is for anyone installing or updating Strata (version 0.2.5). Use the **desktop app** for a graphical
|
||||
interface, or the **Terminal** for `strata launch` and coding tools. Both use released engine packages. A normal
|
||||
install does not need a source checkout or a startup `.bat` file.
|
||||
|
||||
There are three ways to install it. Take the first one unless you have a reason not to:
|
||||
## Quick start
|
||||
|
||||
1. **[The app](#the-app):** a desktop program with the engine inside. Install it, pick a model, chat.
|
||||
2. **[The Terminal](#the-terminal):** one line in a terminal, then `strata launch`. For scripts, servers and AI agents.
|
||||
3. **[From source](#from-source-windows):** `START-HERE.bat` on Windows, `./setup.sh` on Linux and Mac. It builds or
|
||||
fetches the engine itself.
|
||||
| I want | Do this |
|
||||
| --- | --- |
|
||||
| A window with chat | Download the installer from the [latest release](https://github.com/Niko1221/Strata/releases/latest) ([steps](#the-app)) |
|
||||
| The terminal, or my coding tool | Run the one-line install, then `strata launch` ([steps](#the-terminal)) |
|
||||
| DeepSeek V4.1-Flash | Pick it in the app, or run `strata launch --family deepseek41` ([model page](MODELS.md#deepseek-v41-flash)) |
|
||||
|
||||
> **On this page:** [What you need](#what-you-need) · [The app](#the-app) · [The Terminal](#the-terminal) ·
|
||||
> [From source: Windows](#from-source-windows) · [Linux](#from-source-linux) ([build from git](#build-from-git-on-linux-to-try-a-pull-request-or-change-the-code)) · [Mac](#from-source-mac) ·
|
||||
> [AMD cards](#amd-cards) · [Several cards](#two-or-three-cards) · [Docker](#docker-linux) ·
|
||||
> [Older CPUs](#older-cpus-experimental) · [Updating](#updating) · [Where things are stored](#where-things-are-stored) ·
|
||||
> [Setup's questions](#setups-questions) · [Tuning](#tuning-for-your-pc) · [All options](#options-without-questions)
|
||||
[Requirements](#what-you-need) · [Desktop app](#the-app) · [Terminal](#the-terminal) ·
|
||||
[Update](#updating) · [Files and folders](#where-things-are-stored) · [Build from source](SOURCE_INSTALL.md)
|
||||
|
||||
## What you need
|
||||
|
||||
| | |
|
||||
| Computer | Guide |
|
||||
| --- | --- |
|
||||
| **GPU** | **NVIDIA** RTX 20, 30, 40 or 50 series, **12 GB VRAM or more** (8 GB runs, slowly). Measured on an RTX 5070 and an RTX 3090; RTX 20 (Turing, since 0.1.27) was tested by a contributor on an RTX 2070. **AMD** Radeon RX 7900 XT / XTX, RX 9070 / 9070 XT and Radeon AI PRO R9700 (validated), RX 7800 XT / 7700 XT and RX 9060 XT (validated by their owners), RX 6800 / 6900 series (community-reported), with 12 GB of VRAM or more. See [AMD cards](#amd-cards). |
|
||||
| **RAM** | Enough for the size you pick ([which model](MODELS.md#pick-by-memory)); **64 GB** runs every size. A big GPU makes up for less RAM - the [low-RAM mode](MODELS.md#a-big-graphics-card-and-little-ram). |
|
||||
| **CPU** | x86-64 with AVX2 (any Intel/AMD desktop CPU from the last ~8 years). AVX-512 (Ryzen 7000/9000) is a bit faster. Older CPUs without AVX2 are experimental and slow: [Older CPUs](#older-cpus-experimental). |
|
||||
| **Disk** | The app and its engine take about 1-3 GB. ~70-80 GB free for the model, ~6 GB for the MTP layer (+1 GB with images). Q2_0 on an AVX-512 CPU also writes a one-time ~40 GB copy of its experts for the fast CPU kernel. On Linux with an AMD card, ROCm takes ~10 GB more when setup installs it. An NVMe SSD is strongly recommended: it makes the first start much faster. |
|
||||
| **OS** | Windows 10/11, Linux (Ubuntu 22.04/24.04 and Debian 12/13 get everything installed automatically; the app also runs on Fedora 44), or a Mac with Apple Silicon (M1 or newer) and macOS 15 or newer ([Mac](MAC.md)). |
|
||||
| **Driver** | **NVIDIA:** a current driver, version 580 or newer ([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA App). **AMD:** on Linux the kernel's amdgpu driver (no ROCm install needed); on Windows a current AMD Software: Adrenalin Edition driver ([amd.com/support](https://www.amd.com/en/support)). |
|
||||
| Windows 10/11 or Linux x86-64 with an NVIDIA RTX GPU | Use the app or the Terminal. The engine for your card is downloaded for you. [Older NVIDIA cards](OLDER_GPUS.md) use the CUDA 12 engine. DeepSeek V4.1-Flash runs on RTX 30, 40 and 50 cards in the app and the Terminal. |
|
||||
| Windows or Linux x86-64 with a supported AMD Radeon | [AMD guide](AMD_HIP.md), including the card-to-package table. DeepSeek V4.1-Flash runs on the Radeon AI PRO R9700 under Linux, in the app and the Terminal. |
|
||||
| Linux x86-64 with Intel Arc | [Intel Arc guide](INTEL_ARC.md); install the driver/compute runtime and check model preparation requirements |
|
||||
| Apple Silicon Mac, macOS 15 or newer | [Mac guide](MAC.md). Intel Macs are not supported by the Mac package. |
|
||||
| AMD Ryzen AI Max / Strix Halo | [Strix Halo guide](STRIX_HALO.md) |
|
||||
|
||||
The driver is the only thing you install yourself. With the app, the engine is inside the installer. With the
|
||||
Terminal, `strata launch` fetches the engine for your card from the signed update feed, and nothing is compiled.
|
||||
Installing from source sets up the rest as well, the first time: Python 3.12 if you have none (for your user
|
||||
account, no admin), a private Python environment in `.venv/`, the Strata engine, the model and the MTP draft layer.
|
||||
On NVIDIA it uses the ready-made engine for RTX
|
||||
20/30/40/50 and NVIDIA's CUDA libraries from pip (~0.4 GB); if no ready-made engine fits your PC, it offers to install
|
||||
the build tools (Visual Studio Build Tools + CUDA Toolkit on Windows, `build-essential` + CUDA on Ubuntu) and compiles
|
||||
the engine for your GPU (asks first; 20-40 minutes once). More: [details](DETAILS.md#before-you-start).
|
||||
Memory and disk needs depend on the **model, quantization, context and enabled features**. Check
|
||||
[which model fits](MODELS.md#pick-by-memory); download size is not the same as RAM use. Unified memory counts once.
|
||||
Use an SSD and leave room for helper files and model preparation. The Terminal's `--plan` prints what is missing.
|
||||
|
||||
The normal Python install uses `requirements.txt` and does not install CMake or Ninja. If setup has to compile an
|
||||
engine or image encoder, it uses existing CMake/Ninja tools when available and installs missing pinned helpers from
|
||||
`requirements-build.txt` into `.venv`. An older `.venv` keeps any build helpers it already has; setup does not remove
|
||||
packages during an update.
|
||||
Install a graphics driver that supports the package you choose. CUDA 13 packages require NVIDIA driver 580 or newer;
|
||||
older NVIDIA cards use CUDA 12. AMD and Intel requirements are in their guides. Packaged GPU runtimes do not replace
|
||||
hardware drivers. Models and updates need an internet connection to download.
|
||||
|
||||
## The app
|
||||
|
||||
The app is the easiest way. It has the engine inside, so there is nothing to build and no setup window.
|
||||
1. Open the [latest release](https://github.com/Niko1221/Strata/releases/latest).
|
||||
2. Download the one installer for your system. There is one per system, not one per graphics card:
|
||||
- **Windows:** run `Strata-0.2.6-windows-x64-setup.exe`. It installs for your user and needs no administrator rights.
|
||||
- **Linux:** make `Strata-0.2.6-linux-x86_64.AppImage` executable, then open it. If FUSE is unavailable, run the
|
||||
downloaded file with `--appimage-extract-and-run`. [Linux help](TROUBLESHOOTING.md).
|
||||
- **Mac:** open `Strata-0.2.6-macos-arm64.dmg`, drag Strata into Applications, then open it. [Mac help](MAC.md).
|
||||
3. On Windows and Linux the installer has no engine inside. On first start the app shows the engine for your
|
||||
graphics card with its size and one **Download** button, with progress and resume. If you already downloaded the
|
||||
engine file, choose **I already have the file** and it installs without internet. The Mac file keeps its small
|
||||
Metal engine inside.
|
||||
4. Choose a model and its storage folder, let it download, then chat. DeepSeek V4.1-Flash is in the model list on
|
||||
the supported cards ([model page](MODELS.md#deepseek-v41-flash)); it has three sizes, an 8K context and a **Speed**
|
||||
choice, Faster or Full quality.
|
||||
|
||||
1. Download the installer for your system from the [releases page](https://github.com/Niko1221/Strata/releases/latest).
|
||||
The page says which installer is for which graphics card (NVIDIA or AMD). Take the one that matches yours.
|
||||
- **Windows 10 / 11:** run the `.exe`. It installs for your user and needs no administrator rights. Windows may
|
||||
say "Windows protected your PC", because the installer is not code-signed yet. Choose **More info**, then
|
||||
**Run anyway**. The installer asks for the folder (by default `%LOCALAPPDATA%\Programs\Strata`): on a PC
|
||||
with little space on C:, pick a folder on another drive that you can write to (not Program Files, which needs
|
||||
an administrator). Updates from inside the app go into the same folder. The engine inside the app
|
||||
(`%LOCALAPPDATA%\Strata\components`) stays on C:; `STRATA_COMPONENTS_DIR` moves it, and the models go to the
|
||||
folder you choose in the app.
|
||||
- **Linux:** the `.AppImage`. Make it runnable (right-click, Properties, Permissions, "Allow executing file as
|
||||
program", or `chmod +x`), then double-click it. It needs FUSE 2: `sudo apt install libfuse2` on Ubuntu 22.04,
|
||||
`sudo apt install libfuse2t64` on Ubuntu 24.04 and newer, `sudo dnf install fuse fuse-libs` on Fedora. Without
|
||||
FUSE you can still start it with `--appimage-extract-and-run` after the file name. The NVIDIA build needs
|
||||
driver 580 or newer.
|
||||
- **Mac:** the `.dmg`. Open it, drag Strata onto Applications, then open Strata from there. It needs Apple
|
||||
Silicon and macOS 15 or newer. If macOS says the app is "damaged", the file is fine: see [MAC.md](MAC.md).
|
||||
2. Start Strata. The first start is a little slower on Linux and Mac, because the engine inside the app is copied
|
||||
into your home folder.
|
||||
3. The first screen reads your PC and recommends a model that fits. Press the install button. The download is
|
||||
big (40-70 GB). If it stops, it continues where it stopped.
|
||||
4. Chat. **Closing the window stops Strata and the engine.**
|
||||
The `strata-engine-…` and `strata-control-…` archives on the release page are components, not installers. The
|
||||
installer is the same file for a first install and for an update. There is no Intel desktop installer yet; Intel Arc
|
||||
runs on Linux only, through the [Terminal](INTEL_ARC.md).
|
||||
|
||||
**Update:** Settings, then Updates, then **Check for updates**. The buttons **Update Strata** and **Update engine**
|
||||
appear only when something is new. Your models and saved settings are kept. Unload the model first: an engine that
|
||||
is running cannot be replaced. **Roll back** in the same place brings the previous engine back.
|
||||
The first launch can take time to check and prepare large model files. Watch its progress and logs; there is no fixed
|
||||
startup time for every model or computer. Closing the desktop app stops its managed engine.
|
||||
|
||||
**Where things are:** the app's settings and chats are in the `.strata` folder in your user folder, the engine
|
||||
in `%LOCALAPPDATA%\Strata\components` (Linux and Mac: `~/.local/share/strata/components`), and the models in the
|
||||
`Strata` folder in your Documents (you can change the model folder in the app). Removing the app does not remove
|
||||
them. **For a bug report:** Settings, Engine, Diagnostics, View logs, **Save logs**.
|
||||
|
||||
On a Mac pictures are on by default and read on the GPU ([MAC.md](MAC.md)). Everything else about pictures is in [Setup's questions](#setups-questions).
|
||||
The provider order recommended here is **ISTA-DASLab, Unsloth, Gyro, then others**, when a compatible pack fits.
|
||||
Current app recommendations can differ; choose the model explicitly when you want a particular provider.
|
||||
|
||||
## The Terminal
|
||||
|
||||
The Terminal is the `strata` command. It uses the same engine, models and saved settings as the app, and it is the
|
||||
way for scripts, servers and AI agents. Linux (x86-64), Mac (Apple Silicon) and Windows (x64) are supported.
|
||||
Install once, then use `strata launch`. The Terminal includes Python and downloads the engine for your computer.
|
||||
|
||||
Linux and Mac:
|
||||
**Linux / Mac**
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
```
|
||||
|
||||
Windows (PowerShell):
|
||||
**Windows PowerShell**
|
||||
|
||||
```powershell
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
```
|
||||
|
||||
The installer finds out which computer this is, downloads the Terminal archive, checks its SHA-256, unpacks it into
|
||||
your home folder (`~/.local/share/strata/terminal`, or `%LOCALAPPDATA%\Strata\terminal` on Windows; no
|
||||
administrator rights), checks the release's signed update feed with the Terminal it just unpacked (if that check
|
||||
fails it removes what it unpacked), and puts `strata` on your PATH. On Windows, open a new terminal window
|
||||
afterwards. Then:
|
||||
On Windows, open a **new terminal** afterwards. On Linux and Mac, if `strata` is not found, use the full path printed
|
||||
by the installer (normally `~/.local/bin/strata`) or add that folder to your shell's PATH.
|
||||
|
||||
```sh
|
||||
strata launch # guided screens: model, context, pictures, who may connect, then serve
|
||||
strata launch --model q2_0 --yes --no-tui # the same without questions (scripts, agents)
|
||||
strata launch
|
||||
```
|
||||
|
||||
`strata launch` downloads the engine for your graphics card (signed, and test-started before it is used) and the
|
||||
model, starts serving on `http://127.0.0.1:8080` and shows the monitor. It never opens a browser unless you add
|
||||
`--open`. Its flags and screens: [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md).
|
||||
Follow the screens to choose a model, context, storage folder and access settings. The monitor opens after startup.
|
||||
Use the address printed by Strata; the default local endpoint is `http://127.0.0.1:8080`.
|
||||
The Terminal opens a browser only when you add `--open`.
|
||||
|
||||
**Update:** `strata update --check` says what would change, `strata update` installs the new engine (it is test-started
|
||||
first and switched only if that works; the previous engine is kept), `strata components rollback` goes back. It
|
||||
refuses while a model is being served. **Check your PC and the installed engine:** `strata doctor`.
|
||||
For a script or AI agent, inspect a plan before downloading. This example selects ISTA-DASLab Q2_0 explicitly:
|
||||
|
||||
To read the install script before you run it, download it first (`curl -fsSL <address> -o install.sh`), read it,
|
||||
then run `sh install.sh`.
|
||||
|
||||
## From source: Windows
|
||||
|
||||
1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or
|
||||
`git clone` it).
|
||||
2. Double-click **`START-HERE.bat`**.
|
||||
3. Answer a few questions - or just press Enter each time for the recommended choice ([the questions](#setups-questions)).
|
||||
|
||||
Then it downloads everything (the model is ~70 GB, so the first time takes a while - you can stop and it picks up
|
||||
where it left off) and **starts the model**. Your browser opens the Strata app at `http://127.0.0.1:8080`.
|
||||
|
||||
> **While the model starts, your PC can be slow or stop responding for 1-3 minutes** (longest the first time): Strata
|
||||
> loads 35-55 GB into your RAM and locks part of it for the graphics card. That's normal - wait, and don't close the
|
||||
> window. The window tells you what it is doing.
|
||||
|
||||
**Next time**, just double-click `START-HERE.bat` again: it starts right away (30-90 s to load the model), nothing is
|
||||
downloaded twice. Close its window to stop the model. `SETUP.bat` (the same as `START-HERE.bat --setup`) installs
|
||||
another model or changes the settings. Starting Strata from Task Scheduler at logon needs two task settings, or the
|
||||
start is 24x slower: [Running it at startup](DETAILS.md#running-it-at-startup-task-scheduler).
|
||||
|
||||
## From source: Linux
|
||||
|
||||
```bash
|
||||
./setup.sh
|
||||
```sh
|
||||
strata launch --family qwen --model q2_0 --plan
|
||||
strata launch --family qwen --model q2_0 --yes --no-tui
|
||||
```
|
||||
|
||||
The same questions, the same automatic install (it uses `sudo apt` for Python and, only if it has to compile, for
|
||||
the build tools), and the same start: `http://127.0.0.1:8080`. Later runs of `./setup.sh` (or `./run-<model>.sh`)
|
||||
start the model directly; `./setup.sh --setup` installs another model or changes the settings. Other distributions,
|
||||
WSL and compiling: [details](DETAILS.md#linux).
|
||||
`--no-tui` leaves the server running in the background. Check it, connect a tool, or stop it:
|
||||
|
||||
On Nix/NixOS, you can use the flake:
|
||||
|
||||
```bash
|
||||
nix develop -c ./setup.sh
|
||||
```sh
|
||||
strata launch --status
|
||||
strata connect
|
||||
strata launch --stop
|
||||
```
|
||||
|
||||
### Build from git on Linux (to try a pull request or change the code)
|
||||
|
||||
You do not need the app or the AppImage for this: the source install builds the engine on your PC and keeps
|
||||
everything in the checkout, so you can edit, rebuild and test.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Niko1221/Strata.git && cd Strata
|
||||
./setup.sh # first run: .venv, the engine build (build/, or build-hip/ on a Radeon), the model, then it serves
|
||||
```
|
||||
|
||||
- **What it needs:** `git`, a C++ compiler and `cmake` (setup installs the build tools through `sudo apt` on
|
||||
Debian and Ubuntu; elsewhere install them first), and the CUDA Toolkit for NVIDIA or ROCm with the HIP compiler for
|
||||
a Radeon card ([AMD_HIP.md](AMD_HIP.md)). Python goes into `.venv` of the checkout.
|
||||
- **After you change the code:** run `./setup.sh` again. It compiles only what changed and starts the model. To
|
||||
update without starting: `git pull`, then `./update.sh`. By hand, the build folder is an ordinary CMake build:
|
||||
`cmake --build build --target strata -j8` (`build-hip` for a Radeon card; the options are in
|
||||
[AMD_HIP.md](AMD_HIP.md)).
|
||||
- **To try a pull request:** `git fetch origin pull/<number>/head:pr-<number> && git checkout pr-<number>`, then
|
||||
`./setup.sh`. `git checkout main` goes back; the next `./setup.sh` rebuilds.
|
||||
- **Settings for the engine** (for example `STRATA_HIPBLASLT_TUNING=/path/to/table.txt`) go into the `"env"` object of
|
||||
`strata-<model>.json`, which `./setup.sh` writes next to the checkout. `./run-<model>.sh` starts the model with it.
|
||||
- **The Terminal monitor from a checkout:** the monitor needs one extra package, which the source install does not
|
||||
put into `.venv`:
|
||||
|
||||
```bash
|
||||
.venv/bin/pip install -r requirements-monitor.txt
|
||||
STRATA_PYTHON="$PWD/.venv/bin/python3" bin/strata monitor --url http://127.0.0.1:8080
|
||||
```
|
||||
|
||||
(`bin/strata` is the same `strata` command as the Terminal's: `bin/strata --help` lists the rest.)
|
||||
- **Only compile, without the installer:** the engine is a normal CMake project. For NVIDIA:
|
||||
`cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native &&
|
||||
cmake --build build --target strata -j8`. The options, and the flags setup passes, are in `setup.py` (search for
|
||||
`-DSTRATA_`).
|
||||
|
||||
## From source: Mac
|
||||
|
||||
On an Apple Silicon Mac with macOS 15 or newer, install Apple's Command Line Tools once (`xcode-select --install`),
|
||||
then run `./setup.sh` in the Strata folder. It builds the Mac (Metal) engine on your Mac and asks the same
|
||||
questions. The app and the Terminal do not build anything. All of it, and what fits which memory size, is in
|
||||
[MAC.md](MAC.md).
|
||||
|
||||
## AMD cards
|
||||
|
||||
The app and the Terminal pick the AMD engine for a Radeon card by themselves. From source the steps are the same as with NVIDIA: `START-HERE.bat` on Windows, `./setup.sh` on Linux. Setup finds the Radeon
|
||||
card and chooses the AMD (HIP) engine by itself on a PC with no NVIDIA card Strata can use; `--backend hip` chooses
|
||||
it on a PC that has both. Supported cards: RX 7900 XT / XTX (gfx1100), RX 7800 XT / 7700 XT (gfx1101), RX 9060 XT
|
||||
(gfx1200), RX 9070 / 9070 XT and Radeon AI PRO R9700 (gfx1201), and the RX 6800 / 6900 series (gfx1030). Integrated
|
||||
Radeon GPUs are listed as not supported.
|
||||
|
||||
On Linux setup uses a system ROCm 7 when there is one, or installs ROCm into `.venv` from AMD's wheels (~10 GB, no
|
||||
sudo), and compiles the engine on your PC for the card (10-20 minutes, once; it needs a C++ compiler and git:
|
||||
`sudo apt install build-essential git`). Several AMD cards share the model with `--gpus`, as on NVIDIA.
|
||||
|
||||
What differs from NVIDIA for now: `--calibrate` is NVIDIA-only, and Unsloth's 4-bit model needs an NVIDIA card. Measurements per card, the
|
||||
build by hand and the tuning tables: [AMD_HIP.md](AMD_HIP.md).
|
||||
|
||||
## Two or three cards
|
||||
|
||||
**Two or three NVIDIA cards?** Just run `START-HERE.bat`: it lists your cards, says which ones Strata can use, and
|
||||
asks whether to share the model across them (recommended when two can). An install made on one card asks once at
|
||||
its next start. Or choose yourself: `START-HERE.bat --gpus 0,2` (both, remembered), `--gpus all`, or `--gpu 0` (one
|
||||
card, this start only). Each card keeps the experts of its own layers, and prompts flow through the cards in a
|
||||
pipeline: on an RTX 5080 + RTX 3090 prompts were read 18-20% faster than on the 5080 alone, decoding on par.
|
||||
Every card must be an RTX 20 series or newer with 8 GB or more. See [MULTI_GPU.md](MULTI_GPU.md).
|
||||
|
||||
**Several AMD cards:** setup takes one card (the one with the most VRAM, or `--gpu N`) unless you name more:
|
||||
`--gpus 1,0` splits the model's layers across them, the faster card goes last and is the Main card (`--gpu-roles 1=main,0=helper` says it yourself; docs/MULTI_GPU.md; numbers as setup lists them). A
|
||||
split pays only when no single card holds the model's experts ([AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201)).
|
||||
|
||||
## Docker (Linux)
|
||||
|
||||
Ready-made images for NVIDIA (CUDA 13 and CUDA 12), AMD (RX 9000 / R9700 and RX 7000) and Intel Arc cards, with the Terminal
|
||||
inside: `docker run` one command, the model downloads on the first start into a volume, nothing is compiled on your computer.
|
||||
Which image for which card, the commands, Docker Compose, the API key, updating and troubleshooting: [DOCKER.md](DOCKER.md).
|
||||
Building from source: `docker build -t strata:src .` (CUDA 13; [DOCKER.md](DOCKER.md#build-it-yourself)).
|
||||
|
||||
## Older CPUs (experimental)
|
||||
|
||||
Strata's ready-made engine needs AVX2 (Intel Haswell 2013, AMD Zen 2017 or newer). Since 0.1.39 an older CPU - AVX
|
||||
only (Sandy Bridge / Ivy Bridge, Xeon E5 v1/v2, AMD Bulldozer) or SSE4.2 only (Nehalem / Westmere, Xeon X5600) - runs
|
||||
as an **experimental** build that setup compiles on that PC. Run setup as usual: it says the CPU has no AVX2, warns
|
||||
that this is experimental and slow, and compiles the engine (10-20 minutes, once; Windows needs the Visual Studio Build
|
||||
Tools and the CUDA Toolkit, which setup offers to install). It does not stop. The ready-made engine still refuses such
|
||||
a CPU, and says so.
|
||||
|
||||
What differs in that build (`STRATA_ISA_FLOOR=avx` or `none`, chosen from the CPU):
|
||||
|
||||
- ggml-cpu is compiled for AVX or SSE4.2 instead of AVX2, and the CPU's share of the experts runs on its kernels.
|
||||
Strata's own AVX2 / AVX-512 kernels stay in the program but are used only on a CPU that has them.
|
||||
- Only the i-quant models' packs run (every model setup installs on such a CPU); the AVX-512 Q2_0 pack does not.
|
||||
- Speed: the GPU part is unchanged and the CPU part is slower, so a GPU with more VRAM (more experts cached there)
|
||||
helps most. With the older paths forced on a Ryzen 5 7600 + RTX 5070 and the expert cache held at 1500 slots, greedy
|
||||
decode went from 26 to 11-17 tok/s with the AVX build and to 3.5-3.8 tok/s with the SSE4.2 build (Coder and IQ3_XXS,
|
||||
10 prompts each, one run). An old Xeon with DDR3 will be slower than these.
|
||||
- Untested on a real old CPU by us. It was checked here by forcing the older paths on a Ryzen 5 7600
|
||||
(`STRATA_FORCE_ISA=avx` or `sse`): the answers were sane and passed the engine's checks, but not the same token for
|
||||
token as the normal build (ggml-cpu rounds the CPU experts differently, as the AVX2 kernel does from the AVX-512
|
||||
one). Contributors ran earlier versions of the same approach on a Xeon E5-2680 (2.87 tok/s with a GTX 1080 Ti)
|
||||
and on Xeon E5-2687W / X5690 machines.
|
||||
- AMD cards: Linux only (setup compiles the AMD engine there anyway); NVIDIA on Windows and Linux.
|
||||
- Two-socket Xeons: memory on the other socket is slow; keep the RAM on one CPU if you can.
|
||||
|
||||
By hand: `cmake ... -DSTRATA_ISA_FLOOR=avx` (or `none`) builds it; `STRATA_ISA_FLOOR=avx` set for setup builds it on
|
||||
any PC (to try it). Problems and results on real hardware are welcome as GitHub issues.
|
||||
If a model is already running with different settings, add `--restart` to replace that run; `--yes` alone does not
|
||||
restart it. A restart interrupts current answers. See [all launch options](TERMINAL_LAUNCH.md),
|
||||
[advanced Terminal commands](TERMINAL.md), and [agent setup](AI_SETUP.md).
|
||||
|
||||
## Updating
|
||||
|
||||
**The app:** Settings, Updates, **Check for updates**, then **Update Strata** or **Update engine** (see [the app](#the-app)).
|
||||
**Desktop app:** open Settings and find the update controls, check for updates, and use the offered app or engine update.
|
||||
Unload running models before replacing the engine. Your model files and settings are kept.
|
||||
|
||||
**The Terminal:** `strata update --check`, then `strata update` (see [the Terminal](#the-terminal)). Add `--app` to
|
||||
download the app's installer too. Models and saved tuning presets are never downloaded, replaced or deleted by an
|
||||
update. How the signed update feed works: [packaging/UPDATES.md](../packaging/UPDATES.md). What Strata downloads and
|
||||
how it checks it: [SAFETY.md](SAFETY.md).
|
||||
**Terminal:**
|
||||
|
||||
**From source: `UPDATE.bat`** (Linux: `./update.sh`) updates Strata without starting the model - for when the GPU is busy with
|
||||
something else, or you just want the new version ready. In a `git clone` it runs `git pull`, then does what
|
||||
`START-HERE.bat` does before a start: the engine (a new ready-made one when the new version needs it; on Linux a
|
||||
compiled engine is compiled again when its source changed), the Python packages, and each installed model's settings
|
||||
and draft subset. The model files are not touched (at most a new engine is downloaded) and no question is asked.
|
||||
Close the model's window first (a running engine cannot be replaced); start the model later with `START-HERE.bat` as
|
||||
usual. In a copy that was downloaded as a zip it says to download the new zip (below): it cannot fetch new files
|
||||
itself.
|
||||
```sh
|
||||
strata update --check
|
||||
strata update
|
||||
strata --version
|
||||
```
|
||||
|
||||
Or by hand: download the new version and unzip it anywhere (or `git pull`), then run `START-HERE.bat` (Linux:
|
||||
`./setup.sh`) in it. The model files are kept in a `Strata-data` folder next to your Strata folder, so a new copy
|
||||
finds them and sets itself up the same way - nothing big is downloaded again. On Linux after a `git pull`, setup
|
||||
compiles the engine again when its source changed (a minute or two for the changed files); if that compile fails, it
|
||||
says so and starts the engine you had.
|
||||
Stop models first. The next Terminal command uses the updated Control component. The app, Control and engine have
|
||||
separate version numbers, so they can differ.
|
||||
|
||||
Use `strata components rollback` to return to the previous engine. See [update details](../packaging/UPDATES.md).
|
||||
Source checkouts have a [separate update workflow](SOURCE_INSTALL.md#updating).
|
||||
|
||||
## Where things are stored
|
||||
|
||||
- **Your chats: only in your browser.** The Chat tab keeps the conversation, its settings and the API key you typed
|
||||
in the browser's local storage (`strata.*` keys) - not on the server and not in the Strata folder. Pictures are not
|
||||
kept, only their names. Another browser or a private window starts empty; clearing the site's data deletes them.
|
||||
- **How the model starts:** `strata-<model>.json` in the Strata folder (context, GPUs, host, API key, ...), written
|
||||
by setup; next to it `run-<model>.bat` / `.sh`, the log `strata-<model>.log` and, when you use "Use for other
|
||||
apps too", `strata-<model>.shared-settings.json`. Running setup again for the same model (another context, say)
|
||||
rewrites the keys setup writes (`exe`, `args`, `port`, `gpu`, `host`, `api_key`, `vision`, ...) and keeps the
|
||||
ones you added (`sampling`, `mcp_servers`, `mcp`, `cors_origins`, ...); the earlier file is kept as
|
||||
`strata-<model>.json.bak` (0.1.39). Engine options you added to `"args"` by hand are not carried over: setup names
|
||||
them, and you add them again.
|
||||
- **The model files** (`models/`, `packs/`, `mtp/`, 70-120 GB): in **`Strata-data` next to the Strata folder**, or
|
||||
wherever `--data-dir` put them.
|
||||
- **Where that data folder is:** `%APPDATA%\Strata\settings.json` on Windows, `~/.config/strata/settings.json` on
|
||||
Linux ([details](DETAILS.md#before-you-start)).
|
||||
- **The program itself:** `.venv/`, `engine/` and `third_party/` in the Strata folder.
|
||||
- **Where github.com cannot be reached:** a build from source downloads llama.cpp's source (ggml, gguf-py) from
|
||||
GitHub at the commit `LLAMA_CPP_COMMIT` in `setup.py` names. Put that commit's tree in `third_party/llama.cpp`
|
||||
yourself (from a mirror or another PC) and setup uses it instead (#585).
|
||||
| Item | Default location |
|
||||
| --- | --- |
|
||||
| Desktop chats and host state | `~/.strata` (or `STRATA_HOME`) |
|
||||
| Engine components, Windows | `%LOCALAPPDATA%\Strata\components` |
|
||||
| Engine components, Linux / Mac | `$XDG_DATA_HOME/strata/components`, normally `~/.local/share/strata/components` |
|
||||
| Terminal files | The `terminal` folder beside `components`; the installer prints the exact path |
|
||||
| Models | The folder chosen in the app or launch screens; an existing model library is reused |
|
||||
|
||||
### Keeping the models on another disk
|
||||
The desktop app normally proposes `Documents/Strata/Models` for a new library. The Terminal discovers saved
|
||||
folders from earlier installs. Use the path shown by the app or `strata models list`, not an assumed folder name.
|
||||
`--data-dir` changes where new downloads go; it does not move existing models unless you add `--move-data`.
|
||||
See [model discovery](TERMINAL_LAUNCH.md#models-you-already-have).
|
||||
|
||||
The model files are the only big part (70 to 190 GB). Put them where there is room; the app and the Terminal use the same folder.
|
||||
## More options
|
||||
|
||||
- **In the app:** the first screen shows the model folder with a button to change it, and Settings > Storage changes it
|
||||
later. Pick a folder you can save files in; a whole drive (`D:\`) becomes `D:\Strata`, which Strata makes and says so.
|
||||
Change folder then asks what to do with your models: move them (copied file by file and checked before the old copies are removed; you can stop at any time),
|
||||
keep them where they are and also use the new folder, or start empty (the old files are never deleted). From the Terminal the same move is
|
||||
`strata models move NEW_FOLDER` (add `--plan` to see what it would do first).
|
||||
- **In the Terminal:** `strata launch --data-dir D:\Strata` once. The folder is remembered for the next `strata launch`
|
||||
and `strata models list`. Until you give it one, the Terminal uses the folder the app chose; the app keeps its own setting
|
||||
(Settings > Storage) and does not read the Terminal's. `--move-data` moves the models you already have into it; without it they stay where they
|
||||
are and the new folder gets the new downloads. Use your own files in place with `strata launch --model-path FOLDER` (a
|
||||
folder of shards, a folder above it, or one `.gguf` file); a folder that holds several sizes needs the one `.gguf` file.
|
||||
- **Drives that are not NTFS** (exFAT, FAT32, a network share) work: Strata writes straight into `<folder>\models` and makes no
|
||||
link there.
|
||||
- **What stays on C:** the engine (`%LOCALAPPDATA%\Strata\components`, 1 to 2 GB per version; `STRATA_COMPONENTS_DIR` moves it),
|
||||
the app itself (the installer asks), your settings, chat history, tuning results and logs (`%LOCALAPPDATA%\Strata`,
|
||||
`%APPDATA%\Strata`; a few MB). Nothing else large: downloads, the prepared model data (`packs`) and the draft layer (`mtp`) are
|
||||
written in the model folder, and so are the half-finished files of an interrupted download.
|
||||
- **Docker:** `-v /mnt/big/strata:/data` (a folder on the big disk) instead of the volume `strata-data`; everything above lives in it.
|
||||
|
||||
## Setup's questions
|
||||
|
||||
These are the questions of the installer from source (`START-HERE.bat`, `./setup.sh`). The app's first screen and
|
||||
`strata launch` ask the same things in their own look.
|
||||
|
||||
- **Which model and size?** The original, Swift 1.5, the Coder or Unsloth's 4-bit, and Q2_0, IQ2_XS, IQ3_XXS or
|
||||
IQ3_S - see [which model](MODELS.md).
|
||||
- **How much context?** How much text it can keep in mind at once (it suggests one for your card). 384K and
|
||||
512K (experimental) extend the model past its trained 262K by rope scaling - the setup turns it on itself (yarn and a
|
||||
covering factor; `--rope-scaling`/`--rope-scale` override) ([details](DETAILS.md): "Context extension past
|
||||
262K").
|
||||
- **Images?** Whether it should also read pictures. Pictures are on by default for models that can read them. The
|
||||
image encoder (0.9 GB) runs on the CPU when your card has less than 24 GB (about 3-13 s per picture, nothing taken
|
||||
from the GPU) and on the GPU from 24 GB (0.1-0.5 s per picture, ~1.4 GB of VRAM kept free for it). The rule is the
|
||||
same for NVIDIA and AMD cards; `--vision gpu` and `--vision cpu` choose the reader yourself, `--vision no` turns
|
||||
pictures off. On a Radeon AI PRO R9700 the GPU reader took 57 ms for a 260-token picture and the CPU reader 3.1 s.
|
||||
A Mac reads on its GPU (0.45 s for a 260-token picture on an M4). [Details](DETAILS.md#images-vision).
|
||||
- **Experimental speed projection?** Off unless you say yes - [read what it does](DETAILS.md#experimental-speed-projection-experimental-off-by-default)
|
||||
first. It changes how the model answers, and only the original model offers it.
|
||||
|
||||
With `--yes` setup takes the recommended answer to every question.
|
||||
|
||||
## Tuning for your PC
|
||||
|
||||
Every PC is different: `START-HERE.bat --calibrate` (Linux: `./setup.sh --calibrate`) measures a few engine settings
|
||||
on yours and keeps the fastest (about 15-30 minutes, longer on a slow card; on an RTX 5070 with a Ryzen 5 7600 it made the Coder 7% faster).
|
||||
It keeps a setting only when it is more than 3% faster, and the result is remembered per PC and model, so updates
|
||||
keep it. Measuring the CPU worker count needs a fresh engine, so the model is loaded more than once: the PC is
|
||||
busy, and can stop responding for a minute or two, once per restart. When it finishes it **starts the model**,
|
||||
so the server is already running when it returns - do not start it a second time.
|
||||
NVIDIA cards for now. [What it measures](DETAILS.md#double-click-start-herebat).
|
||||
|
||||
## Options without questions
|
||||
|
||||
For the Terminal, `strata launch --help` lists every flag; the most used are in [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md). These are the options of the installer from source:
|
||||
|
||||
```
|
||||
START-HERE.bat --setup install another model, or change context / images
|
||||
SETUP.bat the same (double-click it)
|
||||
START-HERE.bat --model IQ2_XS --context 32768 --vision yes --yes no questions
|
||||
START-HERE.bat --gguf-dir D:\models\IQ2_XS use GGUF files you already have
|
||||
START-HERE.bat --inspect D:\models\some.gguf what a GGUF really holds and whether Strata runs it (no download)
|
||||
START-HERE.bat --data-dir E:\Strata-data keep the model files somewhere else
|
||||
START-HERE.bat --port 8081 another port
|
||||
START-HERE.bat --gpu 1 another GPU (setup picks the one with the most VRAM)
|
||||
START-HERE.bat --gpus 0,2 several GPUs sharing the model
|
||||
START-HERE.bat --vram-reserve-mib 2048 leave 2 GB of VRAM free for other programs (remembered)
|
||||
START-HERE.bat --no-browser do not open the chat page when the model is ready (remembered;
|
||||
--browser undoes it)
|
||||
START-HERE.bat --setup --backend hip the AMD engine on a PC that also has an NVIDIA card
|
||||
START-HERE.bat --setup --host 0.0.0.0 --api-key <secret> reachable from other devices, with a key
|
||||
START-HERE.bat --calibrate tune the engine for this PC (about 15-30 minutes, longer on a slow card), then start
|
||||
START-HERE.bat --check only check this PC
|
||||
```
|
||||
|
||||
**Leaving VRAM for other programs (#493):** Strata fills the graphics card's free VRAM with experts (the expert cache)
|
||||
and leaves `--vram-reserve-mib` MiB free: 700 by default. For a game, a 3D program or another model beside it, leave
|
||||
more: `START-HERE.bat --vram-reserve-mib 2048` (Linux: `./setup.sh --vram-reserve-mib 2048`) writes it into the
|
||||
model's `strata-<model>.json` and starts it; at setup (`--setup --vram-reserve-mib 2048`) it goes into the new config.
|
||||
By hand: add `"--vram-reserve-mib", "2048"` to the config's `"args"` list and restart. The expert cache is then that
|
||||
much smaller, so answers can be a little slower. The engine sizes its cache from the VRAM free when it starts: what
|
||||
another program already holds then is left alone anyway; the reserve is room for what it takes later.
|
||||
|
||||
**An AMD card that also drives a Linux desktop (#560 #516):** with the default reserve the expert cache fills the
|
||||
card, and when the desktop (the compositor, a browser, a new app) needs more VRAM, the amdgpu driver moves GPU memory
|
||||
to system RAM, which the model's experts already fill: the OOM killer then ends the desktop session (KWin, plasmashell)
|
||||
or the compositor fails ("Failed to pin framebuffer"). `./setup.sh --vram-reserve-mib 3072` fixed it in both reports
|
||||
(about 2.3 GB fewer experts in VRAM, a few % of speed). Setup and the server window say so on such a PC.
|
||||
|
||||
**A card under 8 GB (#496):** when the default reserve leaves the expert cache too little room, the engine lowers the
|
||||
reserve (down to 300 MiB) until it fits, and warns if the card then ends nearly full. If the start stops with "no VRAM
|
||||
is left for the expert cache", that log line says how much is short; an 8K context and `--draft-vocab en` at setup
|
||||
free the most.
|
||||
|
||||
**Model files downloaded by hand, or from a mirror (#495):** setup's step 5 prints the folder it expects them in
|
||||
(`Strata-data\models\<SIZE>\`, e.g. `Strata-data\models\IQ3_XXS\`): put them there with their original names, or
|
||||
point setup at them with `--gguf-dir`. To see what a file really holds before using or downloading it, `--inspect`
|
||||
reads only its headers (a few MB, also over the network: a file, a folder, a URL, `ms:owner/repo` for ModelScope or
|
||||
`hf:owner/repo`): every weight group's real bits per weight and storage types, and whether it is one of the files setup
|
||||
installs (under any name), stored like one of them, or not one Strata runs (another architecture, or experts in a
|
||||
format no Strata kernel reads). A name says little: Unsloth's `UD-IQ3_XXS` stores its routed experts at 3.22 bits per
|
||||
weight, `UD-Q4_K_XL` at 5.10, the original `Q2_0` at 2.25 and `IQ3_S` at 3.33. To let setup download from a Hugging Face mirror itself, set `HF_ENDPOINT`
|
||||
first (Windows: `set HF_ENDPOINT=https://hf-mirror.com`, Linux: `export HF_ENDPOINT=https://hf-mirror.com`): the same
|
||||
pinned revisions and checks apply, and the MTP draft layer comes from there too.
|
||||
|
||||
**From ModelScope (mainland China):** every repository setup downloads from (the GSQ-RCO models, Swift 1.5, the
|
||||
Coder, Unsloth's files, and the original checkpoint the MTP draft layer is taken from) is on
|
||||
[ModelScope](https://www.modelscope.cn) under the same name, with the same files. Setup never switches to it by
|
||||
itself (a download that fails says how): use `--source modelscope` (`STRATA_SOURCE=modelscope`). ModelScope serves a
|
||||
repository's current files (no pinned revision), so each file is checked against the SHA-256 ModelScope publishes for
|
||||
it (self-attested: a changed upstream repository would carry its own hash), and the MTP tensors against the
|
||||
pinned checkpoint's own hashes, as from Hugging Face; setup says so when you choose it. Measured on a PC in mainland
|
||||
China on 2026-10-05: 11-14 MB/s; the Q2_0 and IQ3_S files matched their published SHA-256, and Q2_0's shard 1 matched
|
||||
the copy from hf-mirror.com.
|
||||
|
||||
On Linux the same options go to `./setup.sh`. `START-HERE.bat --help` lists them all. The server's own settings
|
||||
(sharing the GPU with games, MCP tools, CORS, API keys, the API itself) are in the [details](DETAILS.md#using-it).
|
||||
- [Docker](DOCKER.md)
|
||||
- [Build or run from a source checkout](SOURCE_INSTALL.md)
|
||||
- [Multiple GPUs](MULTI_GPU.md)
|
||||
- [Troubleshooting and logs](TROUBLESHOOTING.md)
|
||||
- [Model compatibility](MODELS.md)
|
||||
+419
-932
File diff suppressed because it is too large.
Load diff
+117
-117
@@ -1,97 +1,74 @@
|
||||
# Intel Arc (experimental)
|
||||
# Intel Arc on Linux
|
||||
|
||||
Strata 0.1.39 includes an **experimental Intel Arc engine**: Strata's own engine ported to SYCL (Intel oneAPI).
|
||||
maxfridbe wrote it in [#423](https://github.com/Niko1221/Strata/pull/423), with fixes from the people testing it.
|
||||
The code is in `sycl/`, and the port's own notes, measurements and maintenance procedure are in
|
||||
[docs/INTEL.md](INTEL.md). This page covers what you need, how to build it, and what has been tested.
|
||||
Strata runs on Intel Arc cards on Linux through its own engine ported to SYCL (Intel oneAPI). This page is for people with
|
||||
an Arc A-series (Alchemist) or B-series (Battlemage) card. It covers the released Terminal route, what you need, how to
|
||||
build the engine from source, and which cards were tested. The port's workflow and measurements are in [INTEL.md](INTEL.md).
|
||||
|
||||
**Experimental means:** the Intel engine is still young and Linux only. When the port arrived (0.1.39) the maintainers had no
|
||||
Intel GPU and every result on Arc hardware came from the community. Since 0.1.40.2 the maintainers also test on an Arc Pro B70
|
||||
(32 GB, `xe`) and an Arc A750 (8 GB, `i915`); the results from those two cards are marked in the table below. Other cards
|
||||
are still community reports. The NVIDIA and AMD engines are unchanged: the Intel build is a separate CMake target, off by default.
|
||||
maxfridbe wrote the port in [#423](https://github.com/Niko1221/Strata/pull/423). The code is in `sycl/`. Intel Arc is an
|
||||
official platform since 0.1.40.2. The maintainers test on an Arc Pro B70 (32 GB, `xe` driver) and an Arc A750 (8 GB, `i915`
|
||||
driver); results from other cards are community reports. The Mac and Windows builds have no Intel engine.
|
||||
|
||||
**Where the engine comes from.** On Linux, `strata update` (Terminal) and Settings > Update (app) download the Intel engine package, `linux-x86_64-sycl`, for a PC whose graphics card is made by Intel, when the release feed lists one (`strata update --check` shows what it would install). One package serves Alchemist (A-series) and Battlemage (B-series) cards: the GPU code is compiled from SPIR-V at the first start, because the A-series cards have no FP64 units and the engine's FP64 kernels cannot be built ahead of time for them. It carries oneAPI's runtime libraries; the Level Zero loader and the compute runtime stay the PC's own (see "What you need" below), and the user must be in the `render` group (`strata doctor` says so). The package is built by `packaging/build_sycl_linux.sh`. It was tested on an Arc A750 only. The release zips (the ones `START-HERE.bat` and `setup.sh` use) have no Intel engine; to build one from source, follow the steps below. The Docker route is a ready-made image, `ghcr.io/niko1221/strata:0.2.3-sycl`, which is published only for a release that carries a Linux Intel engine package; with none, [build the image yourself](DOCKER.md#build-it-yourself) from the engine you built (the image needs prepared model files; see [Docker: Intel Arc](DOCKER.md#intel-arc)). The Mac and Windows builds have no Intel engine.
|
||||
## Quick start
|
||||
|
||||
## What has been run, and by whom
|
||||
Install the [Terminal](INSTALL.md#the-terminal), then:
|
||||
|
||||
| | Hardware | Result | Reported in |
|
||||
|---|---|---|---|
|
||||
| Port author | Arc Pro B70 32 GB, Ubuntu 24.04 | Coder IQ1_M: 70-78 tok/s decode, ~790 tok/s prompt; IQ2_XS: 51-64 tok/s; 256K context measured | #423, [INTEL.md](INTEL.md) |
|
||||
| Community | 2x Arc Pro B70, `--layer-split` | Flash-Next IQ3_XXS 66 tok/s decode, 394 tok/s prompt (with the `stage_room` fix that is now in 0.1.39) | #423 |
|
||||
| Community | Arc Pro B50 16 GB | Coder IQ1_M ~23 tok/s, IQ2_XS ~25-27 tok/s, up to 128K | #423 |
|
||||
| Community | Arc B580 12 GB, WSL2 | IQ2_XS ~21 tok/s, Coder ~15 tok/s; **device loss also seen** | #423 |
|
||||
| Community | 2x Arc Pro B60 24 GB, `--layer-split` | Coder IQ1_M: 56.6 tok/s decode, 428 tok/s prompt (2,129 tokens); Flash-Next IQ2_XS: 58.6-61.3 tok/s decode, 436 tok/s prompt; all experts in VRAM; 8K context | [bench/results/2026-10-04-community-2x-arc-pro-b60](../bench/results/2026-10-04-community-2x-arc-pro-b60/README.md) |
|
||||
| Community | one Arc Pro B60 24 GB | Coder IQ1_M with 4,042 of 12,288 experts mirrored in RAM: 11.9 tok/s decode (needs the ring-wait fix from the same report) | same |
|
||||
| Strata maintainers (0.1.40.2 and later) | Arc Pro B70 32 GB (xe) and Arc A750 8 GB (i915), test machines | Flash-Next IQ3_S on the B70 with 11.5k of 24.6k experts resident: 10-prompt gate passes, prompt 980-1,000 tok/s on 4K tokens, decode 30 (prose) to 41 (code) tok/s; the xe aliased-pages fix and the `NO_HOST` requirement are in [INTEL.md](INTEL.md) | INTEL.md, "Arc Pro B70 with a model that does not fit" |
|
||||
```sh
|
||||
strata update --check # shows the Intel engine package the feed would install
|
||||
strata doctor --live # checks the driver, the render group and the card
|
||||
strata launch --plan # shows what would happen, changes nothing
|
||||
strata launch
|
||||
```
|
||||
|
||||
The 0.1.39 port re-migrates 0.1.38's port onto the 0.1.39 engine sources (the #606 NaN fix, the #649 verify
|
||||
trace, the new prompt paths). 0.1.39's `sycl/` did not compile against 0.1.39's own engine sources (#784: the
|
||||
`ThreadAffinity` type of #626 and the layer range of `NativeDense::load` from #559), and its ring waits never saw a
|
||||
slow layer as still running (#866, #867). Both are fixed in 0.1.40, and the two B60 reports below were measured with
|
||||
exactly those two fixes on top of 0.1.39. **No one has run an unpatched 0.1.39 port on an Arc**, and the other rows were
|
||||
measured on earlier versions.
|
||||
On Linux, `strata update` (Terminal) and Settings > Update (app) download the Intel engine package `linux-x86_64-sycl` for a PC
|
||||
whose graphics card is made by Intel. One package serves Alchemist and Battlemage: the GPU code is compiled from SPIR-V at the
|
||||
first start, because the A-series cards have no FP64 units. The package carries oneAPI's runtime libraries. The Level Zero
|
||||
loader and the compute runtime stay the PC's own (see [What you need](#what-you-need)), and your user must be in the `render`
|
||||
group (`strata doctor` says so). It was tested on an Arc A750 only. The package is built by `packaging/build_sycl_linux.sh`.
|
||||
|
||||
The two B60 rows ran `6f32ec0` plus two small `sycl/` fixes (the compile fix and the ring-wait fix), AOT `bmg-g21`, on Ubuntu 24.04 with
|
||||
`xe`, Level Zero V2, NEO 26.09.37435.12 and oneAPI 2026.1.1, without Docker. Host: Ryzen 5 5600, 64 GB RAM, PCIe 3.0 x8 per card.
|
||||
Full flags, per-request timings and engine logs are in the report linked in the table.
|
||||
**Model preparation is separate.** `strata launch` reuses compatible prepared local files, but it does not run the Intel
|
||||
source or container setup for you. When a model needs preparation, follow the [source workflow](INTEL.md#setup). Installing the
|
||||
engine package alone does not prove the selected model is ready; check a loaded server with a real request.
|
||||
|
||||
## What was tested here (0.1.39)
|
||||
|
||||
- **Compiles:** Ubuntu 24.04 (WSL2), Intel oneAPI DPC++ 2026.1.1 + oneMKL 2026.1. The whole `sycl/` project
|
||||
builds with 0 errors: the `strata` engine plus all 157 targets (the kernel parity tests and benches). The build is
|
||||
SPIR-V (JIT). The AOT build (`STRATA_SYCL_AOT`) was not built here, because it needs `ocloc`.
|
||||
- **Kernel parity tests on a CPU** (`ONEAPI_DEVICE_SELECTOR=opencl:cpu`, Intel's OpenCL CPU runtime, AMD Ryzen 5 7600):
|
||||
14 of 25 pass. That is the same set that PR #423's own 0.1.38 port passes on that device: the failures are tight
|
||||
float tolerances on the CPU's math (rel 3e-6 against a 1e-6 limit), model fixtures that are not present, and two
|
||||
tests that time out on a CPU. This is a check that the kernels compile and compute, not a test of an Arc.
|
||||
- **Setup:** `--backend sycl` warns, then hands over to `sycl/setup_intel.py` on Linux (unit tests:
|
||||
`tools/test_setup_sycl.py`).
|
||||
- **Unchanged:** the CUDA engine's greedy output, checked byte-identical against the gated 0.1.39 build (Q2_0 and Coder).
|
||||
The HIP build is not affected (the option is off by default).
|
||||
|
||||
**Not tested by anyone yet:** Windows (no native build path; see below), Arc on WSL2 for the 0.1.39 port, integrated Arc
|
||||
GPUs (the B390 / Panther Lake in #515; the shared-memory planning does not exist yet), and pictures on a Battlemage card
|
||||
(the first picture on the Arc Pro B70 reset the GPU; they are off by default there, [INTEL.md](INTEL.md#pictures-on-an-arc-2026-10-19-branch-f-021-arcvis)).
|
||||
Pictures work on the A750 (on by default, read on the CPU). The Alchemist A-series was tested later, on an A750 (rows above and
|
||||
[INTEL.md](INTEL.md), "Arc A750 and the other Alchemist cards").
|
||||
Docker: the ready-made image is `ghcr.io/niko1221/strata:0.2.3-sycl`, published only for a release that carries a Linux Intel
|
||||
engine package. Otherwise [build the image yourself](DOCKER.md#build-it-yourself). The image needs prepared model files
|
||||
([Docker: Intel Arc](DOCKER.md#intel-arc)).
|
||||
|
||||
## What you need
|
||||
|
||||
- **Linux**, e.g. Ubuntu 24.04, with Intel's GPU driver (the `xe` or `i915` kernel driver plus the compute runtime /
|
||||
Level Zero; on Ubuntu, `intel-opencl-icd libze1 libze-intel-gpu1`, or Intel's
|
||||
[client GPU guide](https://dgpu-docs.intel.com/driver/client/overview.html)).
|
||||
- **Intel oneAPI**: the DPC++ compiler (`icpx`, 2025.3 or newer; 2026.1 is what was built here) and **oneMKL**. About 5 GB.
|
||||
- `cmake` 3.24+, `ninja`, `git` (the build fetches ggml unless you point `STRATA_GGML_DIR` at a llama.cpp checkout),
|
||||
Python 3.
|
||||
- For `./setup.sh --backend sycl` today: **Docker**, and nothing else from the list above except the GPU driver.
|
||||
`sycl/setup_intel.py` runs the engine in the `strata-sycl-dev` image built from `sycl/tools/Dockerfile`, and the
|
||||
engine is built in that image too ("Build with Docker" below). On Ubuntu: `sudo apt install docker.io git`, then
|
||||
`sudo usermod -aG docker,render $USER` and log in again. Note that the Dockerfile starts from a community
|
||||
llama.cpp SYCL image (`ghcr.io/snailium/...`), not an Intel or Strata image.
|
||||
- VRAM: the port keeps the experts on the card (`--stream-experts`). A 32 GB card holds the Coder IQ1_M or IQ2_XS.
|
||||
Smaller cards mirror part of the experts in RAM and are slower.
|
||||
- **Linux**, for example Ubuntu 24.04, with Intel's GPU driver: the `xe` or `i915` kernel driver plus the compute runtime and
|
||||
Level Zero. On Ubuntu: `intel-opencl-icd libze1 libze-intel-gpu1`, or follow Intel's
|
||||
[client GPU guide](https://dgpu-docs.intel.com/driver/client/overview.html).
|
||||
- For a source build without Docker: **Intel oneAPI** with the DPC++ compiler (`icpx`, 2025.3 or newer) and **oneMKL**
|
||||
(about 5 GB), `cmake` 3.24+, `ninja`, `git`, Python 3. The build fetches ggml unless you point `STRATA_GGML_DIR` at a
|
||||
llama.cpp checkout.
|
||||
- For `./setup.sh --backend sycl`: **Docker** and the GPU driver, nothing else from the list above. `sycl/setup_intel.py`
|
||||
builds and runs the engine in the `strata-sycl-dev` image from `sycl/tools/Dockerfile`, which starts from a community
|
||||
llama.cpp SYCL image (`ghcr.io/snailium/...`), not an Intel or Strata image. On Ubuntu:
|
||||
`sudo apt install docker.io git`, then `sudo usermod -aG docker,render $USER` and log in again.
|
||||
- VRAM: the port keeps the experts on the card (`--stream-experts`). A 32 GB card holds the Coder IQ1_M or IQ2_XS. Smaller
|
||||
cards mirror part of the experts in RAM and are slower.
|
||||
|
||||
## Build with Docker (what `setup --backend sycl` uses)
|
||||
|
||||
Two commands, the same as in [INTEL.md](INTEL.md), "How to build it" (about 10 minutes the first time). From the checkout,
|
||||
called `Strata` here:
|
||||
About 10 minutes the first time. From the checkout, called `Strata` here:
|
||||
|
||||
```sh
|
||||
cd Strata
|
||||
docker build --pull -t strata-sycl-dev sycl/tools # once: ~6 minutes, a 13 GB image (--pull: INTEL.md says why)
|
||||
docker build --pull -t strata-sycl-dev sycl/tools # once: about 6 minutes, a 13 GB image
|
||||
H=$(basename "$PWD")
|
||||
# Arc B-series: bmg-g31 (Pro B70), bmg-g21 (B580, B570, Pro B60). Arc A-series (Alchemist): leave out AOT (JIT) and use build-sycl
|
||||
docker run --rm -u $(id -u):$(id -g) -v "$PWD/..:/work" -e AOT=bmg-g31 -e REPO=/work/$H \
|
||||
-e BUILD_DIR=/work/$H/build-sycl-aot strata-sycl-dev "cd /work/$H && bash sycl/tools/build.sh strata"
|
||||
```
|
||||
|
||||
Then `./setup.sh --backend sycl` (below). Do not run `python3 sycl/setup_intel.py` by hand: on Ubuntu it fails with
|
||||
`externally-managed-environment`; `setup.sh` runs it in its own `.venv`.
|
||||
`--pull` matters: [INTEL.md](INTEL.md#how-to-build-it) says why. Then run `./setup.sh --backend sycl`. Do not run
|
||||
`python3 sycl/setup_intel.py` by hand: on Ubuntu it fails with `externally-managed-environment`; `setup.sh` runs it in its own
|
||||
`.venv`.
|
||||
|
||||
## Build without Docker (Linux)
|
||||
## Build without Docker
|
||||
|
||||
This builds the engine for running it by hand ("How to run it by hand" in [INTEL.md](INTEL.md)); `setup` itself uses the
|
||||
Docker build above. Install oneAPI from Intel's apt repository (this is what was used here):
|
||||
This builds the engine for running it by hand ([INTEL.md](INTEL.md#how-to-run-it-by-hand)). `setup` itself uses the Docker build.
|
||||
Install oneAPI from Intel's apt repository:
|
||||
|
||||
```sh
|
||||
wget -qO- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB \
|
||||
@@ -102,8 +79,7 @@ sudo apt update
|
||||
sudo apt install intel-oneapi-compiler-dpcpp-cpp intel-oneapi-mkl-devel ninja-build cmake git
|
||||
```
|
||||
|
||||
Then build the engine. Either command works: the first goes through the top-level CMake option, the second
|
||||
configures the `sycl/` project directly.
|
||||
Build the engine through the top-level CMake option or by configuring the `sycl/` project directly:
|
||||
|
||||
```sh
|
||||
source /opt/intel/oneapi/setvars.sh
|
||||
@@ -112,16 +88,14 @@ cmake -S . -B build-sycl -G Ninja -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=ic
|
||||
cmake --build build-sycl --target strata
|
||||
```
|
||||
|
||||
Options:
|
||||
| Option | Effect |
|
||||
| --- | --- |
|
||||
| `-DSTRATA_SYCL_AOT=bmg-g31` | Compile ahead of time for Arc Pro B70. Needs `ocloc` (Intel's `intel-ocloc` package). Without it the first start JIT-compiles every kernel (about 47 s). |
|
||||
| `-DSTRATA_SYCL_AOT=bmg-g21` | The same for B580, B570 and Pro B60 |
|
||||
| `-DSTRATA_SYCL_PARITY=OFF` | Skip the kernel tests (on by default). Run them with `ctest --test-dir build-sycl` on the card. |
|
||||
|
||||
- `-DSTRATA_SYCL_AOT=bmg-g31` (Arc Pro B70) or `bmg-g21` (B580 / B570 / Pro B60) compiles the GPU code ahead of
|
||||
time. This needs `ocloc` (Intel's `intel-ocloc` package). Without it, the first start JIT-compiles every kernel,
|
||||
which takes about 47 s.
|
||||
- `-DSTRATA_SYCL_PARITY=OFF` skips the kernel tests (on by default). Run them with
|
||||
`ctest --test-dir build-sycl` on the card.
|
||||
|
||||
`setup_intel.py` looks for `build-sycl-aot/strata` or `build-sycl/strata` in the checkout. With the
|
||||
top-level option the engine is at `build-sycl/sycl/strata`, so either use `-S sycl` or set `STRATA_SYCL_BIN`.
|
||||
`setup_intel.py` looks for `build-sycl-aot/strata` or `build-sycl/strata` in the checkout. With the top-level option the
|
||||
engine is at `build-sycl/sycl/strata`, so use `-S sycl` or set `STRATA_SYCL_BIN`.
|
||||
|
||||
## Setup and running
|
||||
|
||||
@@ -129,50 +103,76 @@ top-level option the engine is at `build-sycl/sycl/strata`, so either use `-S sy
|
||||
./setup.sh --backend sycl [setup's usual options, e.g. --model IQ2_XS --context 32768]
|
||||
```
|
||||
|
||||
This prints the experimental warning and continues with `sycl/setup_intel.py` (in setup's own Python environment).
|
||||
That script finds the Arc in sysfs, uses the SYCL engine you built, and writes the config and `run-<model>.sh`.
|
||||
On a card under 12 GB or an `i915` card (the A-series) it writes the small-card config ([INTEL.md](INTEL.md), "Arc A750
|
||||
and the other Alchemist cards"). It still downloads and packs the model
|
||||
the usual way. To run the engine by hand (no Docker), see "How to run it by hand" in [INTEL.md](INTEL.md).
|
||||
Setup continues with `sycl/setup_intel.py`, which finds the Arc in sysfs, uses the SYCL engine you built, and writes the
|
||||
config and `run-<model>.sh`. On a card under 12 GB or an `i915` card (the A-series) it writes the small-card config
|
||||
([INTEL.md](INTEL.md#arc-a750-and-the-other-alchemist-cards)). It downloads and packs the model the usual way. Windows has no
|
||||
path: `setup --backend sycl` on Windows stops and points here.
|
||||
|
||||
Things that matter on an Arc (details in INTEL.md):
|
||||
Things that matter on an Arc (details in [INTEL.md](INTEL.md)):
|
||||
|
||||
- **Do not ask for more VRAM than the card has.** On the `xe` driver, an allocation past VRAM can push buffers into
|
||||
RAM until the machine runs out of memory and stalls. Leave about 1.5 GB free.
|
||||
- **Do not ask for more VRAM than the card has.** On the `xe` driver an allocation past VRAM can push buffers into RAM until
|
||||
the machine runs out of memory and stalls. Leave about 1.5 GB free.
|
||||
- `SYCL_CACHE_PERSISTENT=0`: the persistent JIT cache crashed on Xe2 during the first compile.
|
||||
- Two cards: `ONEAPI_DEVICE_SELECTOR=level_zero:*` (the image pins `level_zero:0`; `strata-sycl.sh` now passes the
|
||||
variable through) and `--layer-split`.
|
||||
- **`--layer-split auto` on two equal cards (0.1.44, #1440):** when several splits hold all the experts and predict the
|
||||
same speed, the search now takes the one that leaves its fullest card the most VRAM to spare (within 0.2% of the best
|
||||
predicted time, `STRATA_SPLIT_HEADROOM_TOL`). It used to take the first one tried, which filled the second card to its
|
||||
last MiB. A full card under `xe` moves buffers to system memory instead of failing: 2x B70 with IQ3_S, auto picked
|
||||
K=24 (~11 tok/s) where `--layer-split 30` ran ~54. The startup line names both splits when the choice moves.
|
||||
`STRATA_SPLIT_HEADROOM=0` keeps the old pick.
|
||||
- **Arc Pro B60:** the PCI id is `8086:e211` (`lspci -nn`, the kernel's `xe` id list files it with the BMG-G21 cards), `sycl-ls` prints
|
||||
`Intel(R) Arc(TM) Pro B60 Graphics 20.1.0`, and `ocloc ids bmg-g21` prints 20.1.0, so `-DSTRATA_SYCL_AOT=bmg-g21` is the right target.
|
||||
`setup_intel.py` knows both B60 ids, `e211` and `e221` (both seen on B60 cards). On a `--layer-split` the startup line "N experts are neither in VRAM nor mirrored"
|
||||
counts the other card's layers; the lines `100% of the experts resident` that follow are the ones to read. Set `STRATA_MIRROR_MIB=0` there,
|
||||
or the pinned mirror is allocated and not used.
|
||||
- **`setvars.sh` and `set -u`:** `source /opt/intel/oneapi/setvars.sh` in a shell with `set -u` stops at `OCL_ICD_FILENAMES: unbound variable`
|
||||
(oneAPI 2026.1.1). Source it before `set -u`, or run `set +u` around it.
|
||||
- Two cards: `ONEAPI_DEVICE_SELECTOR=level_zero:*` (the image pins `level_zero:0`; `strata-sycl.sh` passes the variable
|
||||
through) and `--layer-split`.
|
||||
- `--layer-split auto` on two equal cards takes the split that leaves the fullest card the most VRAM to spare, within 0.2% of
|
||||
the best predicted time (`STRATA_SPLIT_HEADROOM_TOL`). `STRATA_SPLIT_HEADROOM=0` turns that off. A full card under `xe` moves
|
||||
buffers to system memory instead of failing. The startup line names both splits when the choice moves.
|
||||
- **Arc Pro B60:** PCI id `8086:e211` (`lspci -nn`), `sycl-ls` prints `Intel(R) Arc(TM) Pro B60 Graphics 20.1.0`, so
|
||||
`-DSTRATA_SYCL_AOT=bmg-g21` is the right target. `setup_intel.py` knows both B60 ids, `e211` and `e221`. On a `--layer-split`
|
||||
the startup line "N experts are neither in VRAM nor mirrored" counts the other card's layers; read the following
|
||||
`100% of the experts resident` lines. Set `STRATA_MIRROR_MIB=0` there, or the pinned mirror is allocated and not used.
|
||||
- **`setvars.sh` and `set -u`:** `source /opt/intel/oneapi/setvars.sh` in a shell with `set -u` stops at
|
||||
`OCL_ICD_FILENAMES: unbound variable` (oneAPI 2026.1.1). Source it before `set -u`.
|
||||
- **Without Docker:** `sycl/serve/strata-sycl.sh` needs Docker. Natively, `source setvars.sh`, export `SYCL_CACHE_PERSISTENT=0`,
|
||||
`ZES_ENABLE_SYSMAN=1` and `STRATA_VERIFY_DEVICE_PLAN=1`, then run `build-sycl-aot/strata` or `sycl/serve/server_intel.py` with an `exe` that does this.
|
||||
A normal user is not in the `render` group on Ubuntu, and then `sycl-ls` shows only the CPU device.
|
||||
- `STRATA_VERIFY_NO_HOST=1` (set by `strata-sycl.sh` unless the config's `"env"` says otherwise) is needed on `xe`
|
||||
cards when part of the experts is not in VRAM (the host-to-GPU flag stores are not visible there, NaN logits
|
||||
otherwise) and is left off on the `i915` Arc A-series: see [INTEL.md](INTEL.md). The config's `"env"` block and any
|
||||
`STRATA_*`, `ONEAPI_*`, `UR_*`, `IGC_*`, `SYCL_*` or `ZES_*` variable of the server's environment reach the engine
|
||||
through `strata-sycl.sh` (for example `STRATA_MIRROR_MIB`, `IGC_EnableDPEmulation`).
|
||||
`ZES_ENABLE_SYSMAN=1` and `STRATA_VERIFY_DEVICE_PLAN=1`, then run `build-sycl-aot/strata` or `sycl/serve/server_intel.py`
|
||||
with an `exe` that does this. A normal user is not in the `render` group on Ubuntu, and then `sycl-ls` shows only the CPU.
|
||||
- `STRATA_VERIFY_NO_HOST=1` (set by `strata-sycl.sh` unless the config's `"env"` says otherwise) is needed on `xe` cards when part
|
||||
of the experts is not in VRAM, and is left off on the `i915` A-series ([INTEL.md](INTEL.md)). The config's `"env"` block and
|
||||
any `STRATA_*`, `ONEAPI_*`, `UR_*`, `IGC_*`, `SYCL_*` or `ZES_*` variable of the server's environment reach the engine
|
||||
through `strata-sycl.sh`.
|
||||
|
||||
## Tested cards
|
||||
|
||||
> **Measured on:** the hardware in each row, Linux. Versions are the engine version of that report, not the current release.
|
||||
|
||||
| Who | Hardware | Result | Reported in |
|
||||
| --- | --- | --- | --- |
|
||||
| Port author | Arc Pro B70 32 GB, Ubuntu 24.04, 0.1.31-sycl | Coder IQ1_M: 70-78 tok/s decode, about 790 tok/s prompt; IQ2_XS: 51-64 tok/s; 256K context | #423, [INTEL.md](INTEL.md) |
|
||||
| Community | 2x Arc Pro B70, `--layer-split` | Flash-Next IQ3_XXS 66 tok/s decode, 394 tok/s prompt | #423 |
|
||||
| Community | Arc Pro B50 16 GB | Coder IQ1_M about 23 tok/s, IQ2_XS about 25-27 tok/s, up to 128K | #423 |
|
||||
| Community | Arc B580 12 GB, WSL2 | IQ2_XS about 21 tok/s, Coder about 15 tok/s; device loss also seen | #423 |
|
||||
| Community | 2x Arc Pro B60 24 GB, `--layer-split`, Ryzen 5 5600, 64 GB, PCIe 3.0 x8 per card | Coder IQ1_M: 56.6 tok/s decode, 428 tok/s prompt (2,129 tokens); Flash-Next IQ2_XS: 58.6-61.3 tok/s decode, 436 tok/s prompt; all experts in VRAM; 8K context | [report](../bench/results/2026-10-04-community-2x-arc-pro-b60/README.md) |
|
||||
| Community | One Arc Pro B60 24 GB | Coder IQ1_M with 4,042 of 12,288 experts mirrored in RAM: 11.9 tok/s decode | same |
|
||||
| Maintainers | Arc Pro B70 32 GB (xe), 0.1.40.2 and later | Flash-Next IQ3_S with 11.5k of 24.6k experts resident: 10-prompt gate passes, prompt 980-1,000 tok/s on 4K tokens, decode 30 (prose) to 41 (code) tok/s | [INTEL.md](INTEL.md#arc-pro-b70-with-a-model-that-does-not-fit-flash-next-iq3_s) |
|
||||
| Maintainers | Arc A750 8 GB (i915) | Flash-Next IQ3_XXS about 10-19 tok/s decode | [INTEL.md](INTEL.md#arc-a750-and-the-other-alchemist-cards) |
|
||||
|
||||
Two cards on `xe` (2x Arc Pro B70, IQ3_S, 0.1.44): `--layer-split auto` picked a split at about 11 tok/s where
|
||||
`--layer-split 30` ran about 54. The auto choice now prefers the split with the most VRAM headroom (above).
|
||||
|
||||
**Not tested by anyone:** Windows (no native build path), Arc on WSL2 with the current engine, integrated Arc GPUs (the B390 /
|
||||
Panther Lake in #515; shared-memory planning does not exist), and pictures on a Battlemage card with a GPU reader. Pictures work
|
||||
on the A750 (on by default, read on the CPU). See [INTEL.md](INTEL.md#pictures-on-an-arc).
|
||||
|
||||
## Windows
|
||||
|
||||
There is no Windows path yet. `setup --backend sycl` on Windows stops and points here. oneAPI exists for Windows,
|
||||
but `sycl/CMakeLists.txt` uses GCC-style flags (`-mavx512f`, `-fp-model=precise`, `-qmkl`) and the runner is a
|
||||
bash/Docker script, so a native Windows build would need work. Nobody has tried it. WSL2 with an Arc has been
|
||||
used by one tester (the B580 row above), but setup cannot detect the card there, because it reads `/sys/class/drm`,
|
||||
which WSL2 does not have.
|
||||
There is no Windows path. `setup --backend sycl` on Windows stops and points here. oneAPI exists for Windows, but
|
||||
`sycl/CMakeLists.txt` uses GCC-style flags (`-mavx512f`, `-fp-model=precise`, `-qmkl`) and the runner is a bash/Docker script.
|
||||
Nobody has tried a native build. WSL2 with an Arc was used by one tester (the B580 row above), but setup cannot detect the card
|
||||
there, because it reads `/sys/class/drm`, which WSL2 does not have.
|
||||
|
||||
## Reporting a problem
|
||||
|
||||
Open an issue with: the card, the driver version, `sycl-ls` output, the oneAPI version, the model and flags, and the
|
||||
engine's stderr. Results from real cards are what move this from experimental to supported.
|
||||
Open an issue with the card, the driver version, the `sycl-ls` output, the oneAPI version, the model and flags, and the engine's
|
||||
stderr. Results from real cards are what move more cards from tested to validated.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
The first port (0.1.39) compiled with 0 errors on Ubuntu 24.04 (WSL2) with oneAPI DPC++ 2026.1.1 and oneMKL 2026.1: the
|
||||
`strata` engine plus 157 targets, as SPIR-V (JIT). Kernel parity tests on a CPU (`ONEAPI_DEVICE_SELECTOR=opencl:cpu`, Ryzen 5
|
||||
7600) passed 14 of 25, the same set as the 0.1.38 port; the failures were tight float tolerances, missing model fixtures and two
|
||||
CPU timeouts. That was a compile-and-compute check, not a test of an Arc. The CUDA engine's greedy output stayed byte-identical.
|
||||
|
||||
The two B60 rows ran commit `6f32ec0` plus two small `sycl/` fixes, AOT `bmg-g21`, Ubuntu 24.04, `xe`, Level Zero V2, NEO
|
||||
26.09.37435.12 and oneAPI 2026.1.1, without Docker. No one ran an unpatched 0.1.39 port on an Arc; the other rows were measured on
|
||||
earlier versions.
|
||||
+51
-31
@@ -1,55 +1,60 @@
|
||||
# Running Strata behind llama-swap
|
||||
|
||||
[llama-swap](https://github.com/mostlygeek/llama-swap) puts several model servers behind one address and starts the one a
|
||||
request names, stopping the others to free the GPU. Strata works as one of its models. This page is what we needed to
|
||||
get it running; it was tested with llama-swap's default stop behaviour, Strata 0.1.39 (Coder IQ1_M on a Tesla V100) and
|
||||
clients reaching the machine by its host name.
|
||||
[llama-swap](https://github.com/mostlygeek/llama-swap) puts several model servers behind one address, starts the one a
|
||||
request names and stops the others to free the GPU. This page is for people who already run llama-swap and want
|
||||
Strata as one of its models. It needs a [source checkout](SOURCE_INSTALL.md), not the desktop installer.
|
||||
|
||||
## The entry
|
||||
## Quick start
|
||||
|
||||
Set Strata up as usual first (`./setup.sh --no-start ...`), then point llama-swap at the server directly. The start script
|
||||
setup writes (`run-<model>.sh`) fixes the port at 8080, so call `serve/server.py` with llama-swap's port instead:
|
||||
1. Set up Strata without starting it: `./setup.sh --no-start`. This writes `strata-<model>.json` in the checkout.
|
||||
2. Add an entry to llama-swap's config. The start script that setup writes (`run-<model>.sh`) fixes the port at 8080,
|
||||
so call `serve/server.py` with llama-swap's port instead:
|
||||
|
||||
```yaml
|
||||
healthCheckTimeout: 300 # Strata takes ~1.5-2 min to load; give it room
|
||||
healthCheckTimeout: 300 # model loading takes minutes; give it room
|
||||
|
||||
models:
|
||||
strata:
|
||||
cmd: >-
|
||||
sh -c 'cd /path/to/Strata && exec .venv/bin/python serve/server.py
|
||||
--engine strata --config strata-coder-iq1_m.json --port ${PORT}'
|
||||
env: ["STRATA_ALLOWED_HOSTS=my-server"] # the host name your clients use -- see below
|
||||
--engine strata --config strata-<model>.json --port ${PORT}'
|
||||
env: ["STRATA_ALLOWED_HOSTS=my-server"] # the host name your clients use, see below
|
||||
checkEndpoint: /health
|
||||
ttl: 1800
|
||||
```
|
||||
|
||||
- `exec` makes the Python server the process llama-swap stops. On llama-swap's stop (SIGTERM) the server stops its engine:
|
||||
in our test the engine was gone and the V100's memory freed within 2 s.
|
||||
- `checkEndpoint: /health` -- Strata's `/health` answers only once the model is loaded. Measured load time on our machine
|
||||
(model files on a SATA SSD): 80 s, 105 s while another download was using the disk.
|
||||
- If other models share the GPU, put them in a llama-swap group or let swapping unload them; Strata needs the card's
|
||||
memory to itself.
|
||||
3. If clients reach llama-swap by a host name, keep the `STRATA_ALLOWED_HOSTS` line and set it to that name
|
||||
([Host not allowed](#host-is-not-allowed)).
|
||||
|
||||
## The trap: "Host ... is not allowed"
|
||||
## The entry
|
||||
|
||||
The first request through llama-swap failed at once with:
|
||||
- `exec` makes the Python server the process llama-swap stops. On SIGTERM the server stops its engine and the
|
||||
GPU memory is freed.
|
||||
- `checkEndpoint: /health` works because Strata's `/health` answers only once the model is loaded.
|
||||
- Strata needs the card's memory to itself. If other models share the GPU, put them in a llama-swap group or let
|
||||
swapping unload them.
|
||||
|
||||
## Host is not allowed
|
||||
|
||||
A client that reaches the proxy by name (`http://my-server:8040`) gets:
|
||||
|
||||
```text
|
||||
403 Host 'my-server:8040' is not allowed (DNS rebinding protection)
|
||||
```
|
||||
|
||||
Strata only answers to the host names it knows, so a web page cannot reach it by pointing a DNS name at 127.0.0.1.
|
||||
llama-swap passes the client's Host header through, so a client that reaches the proxy by name (`http://my-server:8040`)
|
||||
is refused. Requests by IP address and to `localhost` always pass, which is why it can work in a quick test and fail
|
||||
from another machine. Any one of these fixes it:
|
||||
llama-swap passes the client's Host header through. Requests by IP address and to `localhost` always pass, so a quick
|
||||
local test can work while the same setup fails from another machine. Any one of these fixes it:
|
||||
|
||||
- `STRATA_ALLOWED_HOSTS=my-server` in the entry's `env` (comma-separated names; ports are ignored), or
|
||||
- `"allowed_hosts": ["my-server"]` in `strata-<model>.json`, or
|
||||
- an `"api_key"`, which turns the check off -- then every client must send the key.
|
||||
| Fix | Notes |
|
||||
| --- | --- |
|
||||
| `STRATA_ALLOWED_HOSTS=my-server` in the entry's `env` | Comma-separated names; ports are ignored. |
|
||||
| `"allowed_hosts": ["my-server"]` in `strata-<model>.json` | Same effect, stored in the config. |
|
||||
| An `"api_key"` in the config | Turns the host check off. Every client must then send the key. |
|
||||
|
||||
## Strata's own pages through llama-swap
|
||||
|
||||
llama-swap forwards `/upstream/<model>/...` unchanged, so Strata's status and health pages stay reachable, and calling one
|
||||
llama-swap forwards `/upstream/<model>/...` unchanged. Strata's status and health pages stay reachable, and calling one
|
||||
also loads the model without sending a chat request:
|
||||
|
||||
```bash
|
||||
@@ -57,10 +62,25 @@ curl http://my-server:8040/upstream/strata/health
|
||||
curl http://my-server:8040/upstream/strata/v1/status
|
||||
```
|
||||
|
||||
## Claude Code and other Anthropic-API clients: the thinking level
|
||||
## Claude Code and other Anthropic-API clients
|
||||
|
||||
Claude Code (and the Claude Agent SDK) send `"output_config": {"effort": "high"}` with every request unless told
|
||||
otherwise, so Strata thinks at its highest level (seen by logging the requests). `CLAUDE_CODE_EFFORT_LEVEL=medium` (or
|
||||
`low`) in the client's environment changes it. One measurement, for scale only: on one real coding task in our setup,
|
||||
`medium` finished in 6.5 min against 9.7 min at `high`, both correct; `low` took 10.1 min (more trial-and-fix steps).
|
||||
One run each -- not a general result.
|
||||
Claude Code and the Claude Agent SDK send `"output_config": {"effort": "high"}` with every request unless told
|
||||
otherwise, so Strata thinks at its highest level. Set `CLAUDE_CODE_EFFORT_LEVEL=medium` (or `low`) in the client's
|
||||
environment to change it.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
> **Measured on:** Tesla V100, Coder IQ1_M, Strata 0.1.39, llama-swap default stop behaviour, model files on a SATA SSD.
|
||||
|
||||
| Item | Result |
|
||||
| --- | --- |
|
||||
| Model load time | 80 s; 105 s while another download used the disk |
|
||||
| Engine gone and V100 memory freed after llama-swap's stop | within 2 s |
|
||||
|
||||
> **Measured on:** one real coding task, Strata 0.1.39 setup, one run per level. This is not a general result.
|
||||
|
||||
| `CLAUDE_CODE_EFFORT_LEVEL` | Time to a correct result |
|
||||
| --- | --- |
|
||||
| `medium` | 6.5 min |
|
||||
| `high` | 9.7 min |
|
||||
| `low` | 10.1 min (more trial-and-fix steps) |
|
||||
+241
-222
@@ -1,301 +1,320 @@
|
||||
# Strata on a Mac (Apple Silicon, Metal)
|
||||
|
||||
Strata runs on Apple Silicon Macs through its own engine on Metal. This is the same engine as on a PC. The NVIDIA
|
||||
kernels are replaced by Metal ones (`metal/`), and the host code and the server are the PC's. There are two ways in:
|
||||
This page is for people with an Apple Silicon Mac (M1 or newer). Strata runs there on its own Metal engine: the NVIDIA kernels
|
||||
are replaced by Metal ones (`metal/`), and the host code and the server are the PC's. Intel Macs cannot run it.
|
||||
|
||||
- **The Strata app (`.dmg`) and the Terminal (`strata launch`).** The Metal engine comes ready-made: the `.dmg` has it
|
||||
inside, and the Terminal fetches it as an engine component. Nothing is built on your Mac and you need no Command Line
|
||||
Tools ([below](#the-desktop-app-and-strata-launch-on-a-mac)). macOS 15 or newer, as for the source build. If macOS says the
|
||||
app is "damaged", see [Troubleshooting](TROUBLESHOOTING.md#the-desktop-app-and-the-terminal); how the builds are signed:
|
||||
DeepSeek V4.1-Flash does not run on a Mac yet. It runs in the Terminal on NVIDIA RTX 30 and 50 cards and on the Radeon AI PRO
|
||||
R9700 ([MODELS.md](MODELS.md#deepseek-v41-flash)).
|
||||
|
||||
> **On this page:** [What you need](#what-you-need) · [Install](#install) · [Defaults](#defaults-and-options) ·
|
||||
> [Gyro](#gyro-s-and-gyro-m-agentionais-trellis-experts) · [Measured](#measured-m5-max-128-gb) · [Limits](#limits-this-version) ·
|
||||
> [Switches](#switches) · [How it works](#how-it-works) · [History](#history--older-measurements)
|
||||
|
||||
## Quick start
|
||||
|
||||
There are two ways in:
|
||||
|
||||
- **The Strata app (`.dmg`) and the Terminal (`strata launch`).** The Metal engine comes ready-made: the `.dmg` has it inside and
|
||||
the Terminal fetches it as an engine component. Nothing is built on your Mac and you need no Command Line Tools. If macOS says
|
||||
the app is "damaged", see [Troubleshooting](TROUBLESHOOTING.md#the-desktop-app-and-the-terminal). How the builds are signed:
|
||||
[packaging/mac/README.md](../packaging/mac/README.md).
|
||||
- **From source: `./setup.sh`.** It sets things up the way `START-HERE.bat` does on Windows and builds the engine on your Mac
|
||||
(about 1-15 minutes).
|
||||
(about 1 to 15 minutes).
|
||||
|
||||
**Pictures:** on by default in the app, the Terminal and the source build. The reader (`strata-vision`) comes in the engine package
|
||||
and runs on the Mac's GPU through Metal ([Limits](#limits-this-version)). The NVIDIA, AMD and Intel engines are unchanged.
|
||||
Pictures are on by default in the app, the Terminal and the source build ([Limits](#limits-this-version)). The NVIDIA, AMD and
|
||||
Intel engines are unchanged.
|
||||
|
||||
**Status:** new. It is measured on three Macs: an M5 Max with 128 GB, where the model stays in memory; and an M4 Mac mini
|
||||
with 32 GB and an M1 Max with 34 GB, where the experts stream from the SSD (the M1 Max: 7.9 tok/s decode, 186 tok/s on a
|
||||
9,940-token prompt, Q2_0, the expert cache at 64 slots, as reported with the kernel fix #2348). Other chips are untested, and reports help
|
||||
**Status:** new. It was measured on three Macs: an M5 Max with 128 GB, where the model stays in memory, and an M4 Mac mini with
|
||||
32 GB and an M1 Max with 34 GB, where the experts stream from the SSD. Other chips are untested, and reports help
|
||||
([COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md)).
|
||||
|
||||
## What you need
|
||||
|
||||
- **An Apple Silicon Mac (M1 or newer).** Intel Macs cannot run it.
|
||||
- **An Apple Silicon Mac (M1 or newer).**
|
||||
- **macOS 15 or newer.** The engine keeps its buffers resident through a residency set, which macOS 15 introduced.
|
||||
- **The Command Line Tools:** `xcode-select --install`. That is the compiler. Xcode itself is not needed.
|
||||
- **Enough unified memory for the pack.** The GPU maps the model's main file and keeps it in memory while the model runs. Metal lets one
|
||||
program keep about 3/4 of the memory resident (107.5 GB of 128 GB, measured on the M5 Max).
|
||||
- **For a source build only:** Command Line Tools (`xcode-select --install`).
|
||||
- **Enough unified memory for the pack.** The GPU maps the model's main file and keeps it in memory while the model runs. Metal
|
||||
lets one program keep about 3/4 of the memory resident (107.5 GB of 128 GB on the M5 Max).
|
||||
|
||||
| Pack | The file the GPU maps | Needs about (with the prompt buffers, KV cache, draft layer) | Recommended from |
|
||||
|---|---|---|---|
|
||||
| Pack | The file the GPU maps | Needs about (with prompt buffers, KV, draft layer) | Recommended from |
|
||||
| --- | --- | --- | --- |
|
||||
| Q2_0 | 37.6 GB | 46 GB | 64 GB |
|
||||
| IQ3_XXS | 47.0 GB | 55 GB | 74 GB (where IQ3_S does not fit) |
|
||||
| IQ3_S (recommended; the default from 96 GB) | 54.8 GB | 63 GB | 96 GB |
|
||||
| UD-Q4_K_XL (Unsloth, recommended: higher quality) | 82.6 GB | 91 GB | 128 GB |
|
||||
| UD-Q5_K_XL, UD-Q6_K_XL (Unsloth) | 103.9 / 114.8 GB | stream from the SSD on 128 GB | 128 GB, streamed |
|
||||
| Gyro-S (agentionai, one 58.5 GB file) | 29.7 GB (the file's other 28.8 GB is the n-gram table) | 38 GB | 48 GB; streams below that (measured on 32 GB) |
|
||||
| Gyro-M (agentionai, one 92.0 GB file) | 63.2 GB (+ the 28.8 GB table) | 71 GB | 96 GB; streams below that (not measured on a Mac) |
|
||||
| UD-Q4_K_XL (Unsloth; higher quality) | 82.6 GB | 91 GB | 128 GB |
|
||||
| UD-Q5_K_XL, UD-Q6_K_XL (Unsloth) | 103.9 / 114.8 GB | streams from the SSD on 128 GB | 128 GB, streamed |
|
||||
| Gyro-S (agentionai, one 58.5 GB file) | 29.7 GB (the other 28.8 GB is the n-gram table) | 38 GB | 48 GB; streams below that |
|
||||
| Gyro-M (agentionai, one 92.0 GB file) | 63.2 GB (+ the 28.8 GB table) | 71 GB | 96 GB; streams below that |
|
||||
|
||||
- **The n-gram table:** each pack also has a 28.8 GB n-gram table (the model's second file). It is read from the SSD a few rows at a time and does not count against the memory above.
|
||||
- **Disk:** the download is 66-84 GB, plus about 7 GB for the pack and the draft layer. A Mac that streams also needs room for the experts' copy (below): 34-40 GB.
|
||||
- **A Mac whose memory cannot hold the pack (32-48 GB):** the engine streams the experts from the SSD instead. It decides this by itself when the experts are more than 60% of Metal's share.
|
||||
- It keeps the most-used experts in an arena: Metal's share, or the memory macOS reports as available less what it keeps for macOS, whichever is smaller. It keeps 7 GB, or 15% of the RAM when that is more, and it also keeps the n-gram table's share. It counts as free only what macOS can give without swapping (free and purgeable memory and the clean file cache, not other programs' idle memory). It checks again just before it takes the arena, and fills the arena in 1.5 GB steps, checking again before each one; it stops early rather than push macOS below what it keeps. On a Mac with 64 GB or more the arena, the Metal buffers and the host memory together also stay under 60% of the RAM (see "The engine's whole anonymous footprint" below).
|
||||
- It reads the rest as each token needs them, and reads ahead the experts the next layer will probably use.
|
||||
- Setup writes a second copy of the experts for such a Mac (`<model>.experts`, 40 GB for IQ3_XXS). In it each expert's three matrices sit together, so the SSD reads an expert with one request instead of three. On the 32 GB M4 that is about 17% faster decoding and 30% faster prompts, with the same tokens. Setup writes it only when the disk has room for it plus 10 GB; otherwise it warns, and the engine reads the model file as before.
|
||||
- Setup configures such a Mac without the MTP draft layer: one token per step is faster there.
|
||||
- Measured at 32 GB (below). Smaller Macs are untested.
|
||||
- **Setup recommends, and never forces.** It recommends the pack by memory; a bigger pack chosen with `--model` is kept, with a warning about what it risks.
|
||||
- **The n-gram table.** Each pack also has a 28.8 GB n-gram table (the model's second file). It is read from the SSD a few rows at
|
||||
a time and does not count against the memory above.
|
||||
- **Disk.** The original Q2_0 and IQ3 packs download roughly 66 to 84 GB, plus preparation and draft data. Unsloth packs are much
|
||||
larger and Gyro uses different files. A streamed model can also need a separate experts file (34 to 40 GB for the examples
|
||||
below). Check the selected pack rather than treating these sizes as a universal limit.
|
||||
- **Setup recommends and never forces.** It recommends the pack by memory. A bigger pack chosen with `--model` is kept, with a
|
||||
warning about what it risks.
|
||||
|
||||
### A Mac whose memory cannot hold the pack (32 to 48 GB)
|
||||
|
||||
The engine streams the experts from the SSD. It decides this itself when the experts are more than 60% of Metal's share.
|
||||
|
||||
- It keeps the most-used experts in an arena and reads the rest as each token needs them, with read-ahead for the next layer's
|
||||
likely experts. The arena is Metal's share, or the memory macOS reports as available less what it keeps for macOS (7 GB, or 15%
|
||||
of the RAM when that is more), whichever is smaller. It counts as free only what macOS can give without swapping, fills the
|
||||
arena in 1.5 GB steps and stops early rather than push macOS below its reserve.
|
||||
- On a Mac with 64 GB or more, the arena, the Metal buffers and the host memory together stay under 60% of the RAM, and the
|
||||
filled arena is wired (`mlock`) so macOS does not compress it while the GPU is idle. `STRATA_METAL_WIRE_ARENA=0` turns the
|
||||
wiring off. `STRATA_METAL_ARENA_GB` sets the arena yourself, with a warning when it is over the cap. The engine prints the
|
||||
planned footprint (`planned resident footprint ... cap ...`) and the footprint after the fill.
|
||||
- Setup writes a second copy of the experts (`<model>.experts`, 40 GB for IQ3_XXS) in which each expert's three matrices sit
|
||||
together, so the SSD reads an expert with one request. On the 32 GB M4 that is about 17% faster decoding and 30% faster prompts
|
||||
with the same tokens. Setup writes it only when the disk has room for it plus 10 GB, otherwise it warns, and it does not write
|
||||
it on a Mac with 96 GB or more.
|
||||
- Setup configures such a Mac without the draft layer (one token per step is faster there). A streaming Mac with 96 GB or more
|
||||
keeps the draft layer and writes `--spec 3`.
|
||||
- Prompts stream in chunks of 4096 on a streaming Mac with 96 GB or more, and 1024 on a smaller one. `STRATA_METAL_PROMPT_CHUNK=N`
|
||||
sets it.
|
||||
- The model's file must be on the internal SSD. An external disk is untested and likely much slower.
|
||||
|
||||
## Install
|
||||
|
||||
```
|
||||
For everyday use follow [the app or Terminal installation](INSTALL.md). The commands below are the source installation, which
|
||||
builds the Metal engine locally.
|
||||
|
||||
```sh
|
||||
git clone https://github.com/Niko1221/Strata && cd Strata
|
||||
./setup.sh # asks the size; --yes takes the recommended answers
|
||||
```
|
||||
|
||||
What the first run does (each step is skipped when it is already done):
|
||||
|
||||
1. **Python.** It needs Python 3.10 or newer, but macOS ships 3.9. If that is all your Mac has, `setup.sh` downloads a standalone Python 3.12 (25 MB, pinned by SHA-256) into `.python`. No sudo, no Homebrew.
|
||||
1. **Python.** It needs Python 3.10 or newer, but macOS ships 3.9. If that is all your Mac has, `setup.sh` downloads a standalone
|
||||
Python 3.12 (25 MB, pinned by SHA-256) into `.python`. No sudo, no Homebrew.
|
||||
2. **Checks your Mac:** the chip, the memory, the macOS version, the Command Line Tools and the free disk.
|
||||
3. **Python packages** into `.venv`. CMake and Ninja come from pip too, installed just before the build in step 4.
|
||||
4. **Builds the engine** (`build-metal/strata`). It uses Apple's metal-cpp headers (pinned by SHA-256) and llama.cpp's ggml at the commit the PC build pins.
|
||||
3. **Python packages** into `.venv`. CMake and Ninja come from pip too, installed just before the build.
|
||||
4. **Builds the engine** (`build-metal/strata`) with Apple's metal-cpp headers (pinned by SHA-256) and llama.cpp's ggml at the
|
||||
commit the PC build pins.
|
||||
5. **Downloads the model.** The download resumes if interrupted. `--gguf-dir` uses files you already have.
|
||||
6. **Prepares the model:** the pack (seconds), and the MTP draft layer (about 5 GB of the original checkpoint). On a Mac that streams: the experts' copy instead of the draft layer (about 3 minutes on the M4, checked byte for byte against the model file).
|
||||
6. **Prepares the model:** the pack (seconds) and the MTP draft layer (about 5 GB of the original checkpoint). On a Mac that
|
||||
streams, the experts' copy takes the draft layer's place (about 3 minutes on the M4).
|
||||
7. **Writes `strata-<model>.json` and `run-<model>.sh`, then starts the server.**
|
||||
|
||||
The config uses the Mac defaults: `--spec 2` and the Q8_0 hyper-connection read (below). After that, `./setup.sh` or `./run-q2_0.sh` starts the model directly. The server is the same as on a PC:
|
||||
- the OpenAI-compatible API on `http://127.0.0.1:8080/v1`;
|
||||
- the Anthropic API on `/v1/messages`;
|
||||
- the chat page on `http://127.0.0.1:8080/`.
|
||||
After that, `./setup.sh` or `./run-q2_0.sh` starts the model directly. The server is the same as on a PC: the OpenAI-compatible
|
||||
API on `http://127.0.0.1:8080/v1`, the Anthropic API on `/v1/messages`, and the chat page on `http://127.0.0.1:8080/`.
|
||||
|
||||
Options: `--model Q2_0|IQ3_XXS|IQ3_S|UD-Q4_K_XL|UD-Q5_K_XL|UD-Q6_K_XL` (the UD files are Unsloth's), `--vision yes|no`, `--context N`, `--port`, `--host` / `--api-key`, `--draft-vocab`, `--no-browser`,
|
||||
`--data-dir`, `--models-dir`, `--gguf-dir`, `--setup` (another model or new settings), `--no-start`, `--check` (only
|
||||
check the Mac), `--build` (build the engine again).
|
||||
| Option | Meaning |
|
||||
| --- | --- |
|
||||
| `--model Q2_0\|IQ3_XXS\|IQ3_S\|UD-Q4_K_XL\|UD-Q5_K_XL\|UD-Q6_K_XL\|Gyro-S\|Gyro-M` | The pack. The `UD-` files are Unsloth's. |
|
||||
| `--family qwen\|unsloth\|gyro` | The model family. |
|
||||
| `--vision yes\|no\|gpu\|cpu` | Picture reader. |
|
||||
| `--context N`, `--port`, `--host`, `--api-key` | Server settings. |
|
||||
| `--draft-vocab`, `--no-browser` | Draft vocabulary; do not open the browser. |
|
||||
| `--data-dir`, `--models-dir`, `--gguf-dir` | Where files go, or where yours are. |
|
||||
| `--setup`, `--no-start`, `--check`, `--build` | Choose another model or settings; install only; only check the Mac; build the engine again. |
|
||||
|
||||
### The desktop app and `strata launch` on a Mac
|
||||
|
||||
The Strata app (and the Terminal's `strata launch`) do not build anything on your Mac. The Metal engine comes ready-made, as an engine component: from the update feed, or inside the app's installer. The model install is `setup.py` in its Control mode (`STRATA_CONTROL_INSTALL=1`): it downloads (or uses with `--gguf-dir`) and prepares the model files and writes a run config whose engine is `component:strata-engine`. It does not run `metal/setup_mac.py`, needs no Command Line Tools, and writes the same Metal flags (`--mmap-experts`, `--spec 2`, no draft layer when the experts stream from the SSD).
|
||||
The app and `strata launch` build nothing. The Metal engine comes ready-made as an engine component, from the update feed or
|
||||
inside the app's installer. The model install is `setup.py` in its Control mode (`STRATA_CONTROL_INSTALL=1`). It downloads, or
|
||||
uses with `--gguf-dir`, and prepares the model files, and writes a run config whose engine is `component:strata-engine`. It does
|
||||
not run `metal/setup_mac.py`, needs no Command Line Tools, and writes the same Metal flags.
|
||||
|
||||
The model list and the recommendation judge a Mac by its unified memory with the rules of `strata_control/mac_fit.py`, the module `setup_mac.py` uses too: Q2_0, IQ3_XXS and IQ3_S are offered (the Unsloth many-shard files need the repack of `./setup.sh` for now); a pack that does not stay in memory is listed as "read from the SSD", and a Mac under 16 GB of unified memory is told the model does not fit.
|
||||
The model list and the recommendation judge a Mac by its unified memory with the rules of `strata_control/mac_fit.py`, the module
|
||||
`setup_mac.py` uses too. Q2_0, IQ3_XXS, IQ3_S and the Gyro packs are offered. Unsloth's many-shard files need the source
|
||||
installer's repack for now. A pack that does not stay in memory is listed as "read from the SSD", and a Mac under 16 GB of unified
|
||||
memory is told the model does not fit.
|
||||
|
||||
## Measured (M5 Max, 128 GB, macOS 27.2)
|
||||
The ready-made Mac engine is in the app and in the Terminal's engine component, not in the release zips. The zips (for
|
||||
`START-HERE.bat` and `setup.sh`) have no Mac engine: `./setup.sh` builds it.
|
||||
|
||||
**Decode**, in tokens per second over 200 generated tokens, with greedy answers. The table gives two prompts: a short question
|
||||
("prompt1") and a story.
|
||||
## Defaults and options
|
||||
|
||||
The config uses the Mac defaults: `--spec 2` with MTP (best on mixed text; an explicit `--spec N` in the config's `args` is kept)
|
||||
and the Q8_0 hyper-connection read. `STRATA_HC_Q8=0` in the config's `env` gives the exact BF16 read. The Mac engine is
|
||||
deterministic: two runs give identical numbers.
|
||||
|
||||
### Gyro-S and Gyro-M (agentionai's trellis experts)
|
||||
|
||||
The Mac engine runs [Gyro](GYRO.md). The routed experts are trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a
|
||||
Hadamard-rotated basis, read from the published file unchanged. The kernels are in `metal/kernels/quant.metal`, `moe.metal` and
|
||||
`moe_mm.metal`. Setup offers Gyro-S and Gyro-M next to the other sizes (`./setup.sh --family gyro --model Gyro-S`), never as the
|
||||
default, and says what the choice risks on your Mac's memory. The 28.8 GB n-gram table is inside the file and is read from the SSD
|
||||
like every other pack's, so Gyro-S needs 58.5 GB of disk and about 38 GB of memory to stay resident.
|
||||
|
||||
Gyro-M runs on the same kernels (checked tensor by tensor) but no Mac tested here could hold it. `metal/tests/gyro_parity.sh`
|
||||
checks the decoder against the format definition and the expert kernels against a double-precision reference.
|
||||
`STRATA_METAL_GYRO_TILES=0` turns off the dense-weight tiles.
|
||||
|
||||
## Measured (M5 Max, 128 GB)
|
||||
|
||||
**Measured on:** Apple M5 Max, 128 GB, macOS 27.2, Qwen3.8-Flash-Next packs, greedy, 200 generated tokens, Strata engine version
|
||||
not recorded. Decode tok/s for a short question and a story ("prompt1 / story").
|
||||
|
||||
| | Q2_0 | IQ3_XXS | IQ3_S |
|
||||
|---|---|---|---|
|
||||
| Setup's defaults (`--spec 2` with MTP, the suffix drafter on) | **119.5 / 108.2** | **108.9 / 97.0** | - |
|
||||
| --- | --- | --- | --- |
|
||||
| Setup's defaults (`--spec 2` with MTP, suffix drafter on) | **119.5 / 108.2** | **108.9 / 97.0** | - |
|
||||
| `--spec 2` with MTP, suffix drafter off | 114.0 / 107.9 | 103.5 / 96.9 | 98.2 / 94.1 |
|
||||
| No MTP (one token per step) | 87.4 / 87.3 | 78.7 / 78.6 | 74.5 / 74.3 |
|
||||
|
||||
The output with MTP is the same as without it, token for token, in every run. MTP only proposes tokens, and the
|
||||
full model checks every one of them.
|
||||
The output with MTP is the same as without it, token for token. MTP only proposes tokens, and the full model checks each one.
|
||||
|
||||
**Prompt reading** (time to the first token), on the needle-test prompts below:
|
||||
Prompt reading (time to first token), on the needle-test prompts:
|
||||
|
||||
| | 4K tokens | 32K tokens |
|
||||
|---|---|---|
|
||||
| Q2_0 | ~1,430 tok/s | ~1,460 tok/s |
|
||||
| IQ3_XXS | ~1,260 tok/s | ~1,240 tok/s |
|
||||
| IQ3_S | ~1,320 tok/s | ~1,140-1,190 tok/s |
|
||||
| --- | --- | --- |
|
||||
| Q2_0 | about 1,430 tok/s | about 1,460 tok/s |
|
||||
| IQ3_XXS | about 1,260 tok/s | about 1,240 tok/s |
|
||||
| IQ3_S | about 1,320 tok/s | about 1,140-1,190 tok/s |
|
||||
|
||||
The first prompt after the Mac starts is slower, while macOS reads the model file (IQ3_S: 413 tok/s on the first 4K
|
||||
prompt, cold).
|
||||
The first prompt after the Mac starts is slower while macOS reads the model file (IQ3_S: 413 tok/s on the first 4K prompt).
|
||||
llama.cpp on the same Mac with Q2_0: 56.6 tok/s decode on the story, 887 / 809 tok/s on 4K / 32K prompts.
|
||||
|
||||
**For comparison**, llama.cpp on the same Mac with Q2_0: 56.6 tok/s decode on the story, and 887 / 809 tok/s on 4K / 32K prompts.
|
||||
|
||||
### Unsloth's UD-Q4/Q5/Q6_K_XL on the M5 Max
|
||||
|
||||
IQ3_S and UD-Q4_K_XL are the recommended sizes. Setup offers IQ3_S as the default and names UD-Q4_K_XL as the
|
||||
higher-quality choice where it fits. Decode is tokens per second over 200 tokens (greedy, MTP); prompts are the warm rate
|
||||
of the second request on (the first request after the start pays a few seconds of first-use cost).
|
||||
**Unsloth packs on the same Mac.** IQ3_S and UD-Q4_K_XL are the recommended sizes. Decode is tok/s over 200 tokens (greedy, MTP);
|
||||
prompts are the warm rate from the second request on.
|
||||
|
||||
| | IQ3_S | UD-Q4_K_XL | UD-Q5_K_XL (streamed) | UD-Q6_K_XL (streamed) |
|
||||
|---|---|---|---|---|
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Quality against Q8_0 (top-1 / KL) | 87.9% / 0.142 | 92.8% / 0.055 | not measured | not measured |
|
||||
| Decode, short question, no MTP / MTP | 75.4 / 101.9 | 60.7 / 94.0 | - | - |
|
||||
| Decode, story, no MTP / MTP | 74.8 / 94.7 | 59.7 / 73.7 | 28 / 46-51 (`--spec 3`) | not measured / 47-51 (`--spec 3`) |
|
||||
| Prompt 4K / 32K | ~1,180 / ~1,290 | ~1,150 / ~1,290 | ~1,070 / ~1,110 | ~1,020 / ~1,020 |
|
||||
| Prompt 4K / 32K | about 1,180 / 1,290 | about 1,150 / 1,290 | about 1,070 / 1,110 | about 1,020 / 1,020 |
|
||||
| llama.cpp Metal, decode (story / code) | 51.1 / 44.1 | 48.4 / 47.3 | 45.1 / 43.2 | does not run (Insufficient Memory) |
|
||||
| llama.cpp Metal, prompt 4K / 32K | 791 / 730 | 821 / 739 | 729 / 693 | - |
|
||||
|
||||
- **UD-Q5 and UD-Q6 cannot stay resident:** their repacked file (103.9 and 114.8 GB) is bigger than the largest buffer Metal gives (86.6 GB), so the experts stream from the SSD into an arena of about 65-67 GB on the 128 GB M5 Max (about 68% of Q5's experts, 62% of Q6's; it was 80-86 GB before the footprint cap below, and the decode and prompt speeds did not change: Q6 story 51 tok/s, 15K prompt 1,020 tok/s at both sizes).
|
||||
- **The cost is a fixed hand-shake per layer** (the engine reads the router's choice and makes sure the experts are in the arena), not the SSD. A window of 2 tokens costs 38 ms against ~17 resident, so a longer window pays: Q5 story 28 tok/s without MTP, 41.8 with `--spec 2`, 50.6 with `--spec 3`, 46 with `--spec 4`. Setup writes `--spec 3` and keeps the MTP draft layer for a streaming Mac with 96 GB or more.
|
||||
- **The engine's whole anonymous footprint is limited to 60% of the RAM on a Mac with 64 GB or more (82.4 GB of the M5 Max's 137.4 GB): the arena, every Metal buffer and the host memory the process holds, not only the Metal buffers.** macOS keeps a floor of file cache (`vm.vm_page_filecache_min`: 37.2 GB here) and compresses anonymous memory instead of dropping cache below it. The arena is anonymous, and the GPU wires it only while a command buffer runs, so while it is filled it is ordinary pageable memory. Two kernel panics on the M5 Max (the watchdog daemon starved for 92 s) had the engine at 95 and 90 GiB resident and the compressor at 2-4 million pages. Measured again with a memory watchdog every second: an old-build arena of 100 GB compressed 19 GB of itself in 7 seconds and swapped 24,000 pages; the old automatic arena (69 GB) with a 40 GB file write and a 16 GB program beside it took the compressor from 21,000 to 4.07 million pages in 10 seconds (pressure "warn"). With 30% left (footprint 94.5 GB) and the file write alone, 12,000 pages were swapped out; with 35% left, 316; with 40% left and the write and the 16 GB program, none and the compressor at 0.3 million pages. The old rule (arena + Metal buffers under 67%) counted 7.5 GB of buffers where the process held 11-15 GB, so its real total was about 100 GB. A resident pack is different (clean file pages that macOS drops, never compresses): UD-Q4_K_XL resident (wired 91.7 GB) took the same 40 GB write and 16 GB program with no swap-out and nothing compressed, so resident packs are limited to 70% of the RAM as before. The arena is also wired (`mlock`) once it is filled: unwired it is ordinary pageable memory whenever the GPU is idle (the GPU's own wiring ends a few seconds after its last command buffer; a Metal residency set did not hold it), and a 20 GB file written beside the idle engine took the arena into the compressor and froze the Mac (a third watchdog panic, in a control run of this work). Wired, the same write at full speed raised the compressor by 0.7 million pages (the engine's non-arena memory), swapped out 408 pages and changed nothing else. `STRATA_METAL_WIRE_ARENA=0` turns it off; it is on for Macs with 64 GB or more. The engine prints the planned total (`planned resident footprint ... cap ...`) and the footprint after the fill. `STRATA_METAL_ARENA_GB` still sets the arena as you say, with a warning when it is over the cap.
|
||||
- **The experts' copy** (`<model>.experts`, 98 GB for Q5) did not pay on the M5 Max: decode 50.3 against 50.2 tok/s without it, 4K prompt 741 against 685 (noisy). Setup does not write it on a Mac with 96 GB or more.
|
||||
- **Prompts stream in chunks of 4096** on a streaming Mac with 96 GB or more (the engine's own default there; 1024 on a smaller one): every chunk makes one pass over the experts that are not in the arena, so a longer chunk reads them fewer times. UD-Q5, warm, 2 interleaved rounds: chunks of 1024 gave 700 tok/s on the 4K prompt, 2048 gave 920 and 4096 gave 1,100 (32K: 710, 970, 1,100); the decode did not change. `STRATA_METAL_PROMPT_CHUNK=N` sets it. On a resident model 4096 instead of 2048 gained only 2-4% (Q2_0), so that default stays.
|
||||
UD-Q5 and UD-Q6 cannot stay resident: their repacked file (103.9 and 114.8 GB) is bigger than the largest buffer Metal gives
|
||||
(86.6 GB). The experts stream from the SSD into an arena of about 65 to 67 GB. The cost is a fixed hand-shake per layer, not the
|
||||
SSD, so a longer window pays (Q5 story: 28 tok/s without MTP, 41.8 with `--spec 2`, 50.6 with `--spec 3`).
|
||||
|
||||
### M4 Mac mini, 32 GB, experts streamed from the SSD
|
||||
|
||||
The arena was 11.5-14.7 GB. Tokens per second, one token per step (no MTP), medians of 5 interleaved runs:
|
||||
**Measured on:** Apple M4 Mac mini, 32 GB, experts streamed from the SSD (arena 11.5 to 14.7 GB), one token per step (no MTP),
|
||||
medians of 5 interleaved runs, tok/s.
|
||||
|
||||
| | Q2_0 | IQ3_XXS | llama.cpp, CPU only (Q2_0 / IQ3_XXS) |
|
||||
|---|---|---|---|
|
||||
| --- | --- | --- | --- |
|
||||
| Decode, short question (200 tokens) | **14.1** | **10.0** | 6.3 / 4.1 |
|
||||
| Decode, story (200 tokens) | **13.5** | **9.2** | 6.3 / 4.1 |
|
||||
| Decode after a 6K-token document | **13.1** | **7.7** | 5.1 / 2.8 |
|
||||
| Prompt, 4K tokens | **93.6** | **65.8** | 21.2 / 8.7 |
|
||||
|
||||
- The greedy answers equal the M5 Max's (model in memory) on 10 of 10 test prompts, for both packs.
|
||||
- No memory was swapped out during the runs; the least memory left to macOS was 7.1 GB.
|
||||
- MTP was slower here (Q2_0 story 12.1-12.3 with it against 12.3-12.8 without). The draft layer's 1.2 GB do more as expert cache.
|
||||
The greedy answers equal the M5 Max's (model in memory) on 10 of 10 test prompts. MTP was slower here, because the draft layer's
|
||||
1.2 GB do more as expert cache. The arena's size matters most: with 5.7 to 9.3 GB (a Mac with less memory free) the same M4 read
|
||||
about 8.3 tok/s on a story. On an M1 Max with 34 GB (Q2_0, expert cache at 64 slots, kernel fix #2348) decode was 7.9 tok/s and a 9,940-token prompt read at 186 tok/s.
|
||||
|
||||
**The experts' copy** (`<model>.experts`, one SSD request per expert): IQ3_XXS after the 6K-token document, the arena held at 12 GB, medians of 5 interleaved runs each:
|
||||
**Measured on:** Apple M4 Mac mini, 32 GB, Gyro-S, experts streamed (arena 11 to 12 GB of the 25.8 GB of experts), no experts' copy,
|
||||
one token per step, no draft layer. Decode story, 120 tokens, 5 interleaved pairs, medians: **9.14 tok/s** (8.70 with the
|
||||
dense-weight tiles off). Prompt of 4,065 tokens: **109 tok/s**. The engine held 18.6 GB of the 32 GB. Against the CUDA engine on
|
||||
the same Gyro-S file (5 text segments, 4,995 positions, RTX 3060) the Mac is as far from either CUDA run as the two CUDA runs are
|
||||
from each other: top-1 agreement 95.4% against 95.8%, mean KL 0.0207 against 0.0186.
|
||||
|
||||
| | Decode | The 6K prompt | SSD wait per token |
|
||||
|---|---|---|---|
|
||||
| The model file (three reads per expert) | 7.37 tok/s | 64.7 tok/s | 46.6 ms |
|
||||
| The experts' copy (one read per expert) | **8.62 tok/s** | **84.1 tok/s** | 31.5 ms |
|
||||
## Limits (this version)
|
||||
|
||||
- The tokens are identical in all 10 runs.
|
||||
- The 4K-token prompt (3 interleaved runs each, default arena): 72.9 tok/s with the copy against 62.5 with the model file (+17%), the same tokens.
|
||||
- 32K-token prompts through the server, with the copy: 29,768 tokens at 89.2 tok/s (5.6 minutes to the first token), 33,803 tokens at 84.4 tok/s. The code word was found both times.
|
||||
- A 34-minute server run with the copy (47 requests: 32K and 6K needles, short questions, a story, code; context 34,816) got every answer right and did not slow down over time. Decode was 7.2-7.7 tok/s on short answers and 9.9-10.4 on the story; the 6K prompts ran at 80-90 tok/s. macOS's memory pressure was never critical: one 2-second "warn" sample while the arena filled at the start. About 90 MB were swapped out in those same 2 seconds and nothing after; three restarts afterwards (smaller arenas, less memory free) swapped nothing.
|
||||
- The SSD alone measures the same: 2.97 GB/s for one 1.75 MB read per expert against 2.30 GB/s for three 0.58 MB reads (6 threads, uncached).
|
||||
- The arena's size matters most. With the model file, a 7.5 GB arena gave 5.5 tok/s, 10.5 GB gave 6.7 and 12 GB gave 7.5. The read-ahead depth, the I/O threads and the eviction order each moved decode by less than 3%.
|
||||
- The Q2_0 copy is not measured yet. The numbers in the table above are with the model file.
|
||||
- **Measured again on the release build, with less memory free:** the M4 had 13-17 GB available when the engine started (another user's programs held the rest), so the arena was 5.7 GB (Q2_0) and 9.3 GB (IQ3_XXS, with the experts' copy). One token per step, medians of 3: Q2_0 story 8.3 tok/s, 4K prompt 80; IQ3_XXS story 8.3, 4K prompt 89. That is the arena's size effect above, not a regression: with a 6.6 GB arena the engine used to stop at the first prompt ("no free slot"); a small arena now gives the prompt's 1.6 GB of buffers their own allocation, also when no chunk of the arena is big enough for them (on a 24 GB M2 whose arena shrank below the prompt's needs, every prompt stopped with "no arena region for the prompt buffers"), and a 26-minute server run with a 5.4 GB arena (29 requests, 6K and 31K needles found) was clean.
|
||||
- With the default arena (12.1-12.2 GB here), the experts' copy gives 8.35-8.48 tok/s decode and 82.5-82.8 tok/s on the prompt. Nothing was swapped out. macOS's memory pressure stayed "normal", except for one 2-second "warn" sample while the arena fills; keeping 7 GB for macOS shows the same sample.
|
||||
- Those runs and the server run below kept 5 GB for macOS. The default is now 7 GB with the stepped fill: after the 6K document, 7.3 and 6.7 tok/s decode (arenas 9.3 and 7.6 GB, as much as was free), the same tokens, the memory pressure "normal" throughout and nothing swapped out.
|
||||
- **Images (vision):** on by default. The app and the Terminal use the `strata-vision` that sits beside `strata` in the engine
|
||||
package; `./setup.sh` builds it and asks "Should the model also read images?" (default yes). With `--gpu` it reads on the Mac's
|
||||
GPU (ggml's Metal backend), without it on the CPU. A Mac always gets the GPU reader, whatever its memory. The footprint is about
|
||||
1.18 GB on the GPU and 1.05 GB on the CPU. `--vision cpu` is still allowed and prints a note. The CPU reader is slow because
|
||||
ggml has no vector kernel for BF16 on ARM. If the GPU reader cannot start, the server prints one line and starts it on the CPU.
|
||||
Checked on one Mac mini M4 only; the M5 Max is untested for pictures.
|
||||
|
||||
**Why `--spec 2`:** measured with MTP on the same two prompts.
|
||||
**Measured on:** Mac mini M4, 32 GB, Qwen3.8-Flash-Next BF16 projector.
|
||||
|
||||
| Window | Q2_0 (prompt1 / story) | IQ3_XXS (prompt1 / story) |
|
||||
|---|---|---|
|
||||
| Picture | GPU reader | CPU reader |
|
||||
| --- | --- | --- |
|
||||
| 260 tokens (640x400) | 0.45 s | 50.7 s |
|
||||
| 972 tokens (1600x1200) | 2.4 s | 194 s |
|
||||
|
||||
- **Context:** measured up to 32K (needles, KL). Setup recommends 32K below 96 GB and 64K from 96 GB. Up to the model's 262K can
|
||||
be set, but it is not measured.
|
||||
- **One request at a time.** `--parallel` (batch slots) is not ported. Several clients queue.
|
||||
- **Not ported:** the control vector (`--control-vector-scaled`) and the PC tuning (`--calibrate`).
|
||||
- **Streaming** is measured at 32 GB only.
|
||||
- **Families:** the original Qwen3.8-Flash-Next, Unsloth's UD-Q4/Q5/Q6_K_XL and agentionai's Gyro-S are measured. Setup repacks
|
||||
Unsloth's shards into one file for the GPU; a Gyro file is used as it is. The Swift and Coder releases are not set up on Mac
|
||||
yet, and Qwen3.6 does not run on the Mac engine.
|
||||
|
||||
### Quality
|
||||
|
||||
- **The Q8_0 hyper-connection read** is the default. Teacher-forced over 28 prompts (code, reasoning, chat, six languages,
|
||||
documents), about 6,000 positions per set, BF16 reference: mean KL 0.0007 and top-1 agreement 98.9% on the model's own text,
|
||||
99.1% on another model's text, perplexity unchanged. For scale, IQ3_XXS against Q2_0 is a KL of 0.094.
|
||||
- **The fast prompt path** against reading the prompt through decode windows: mean KL 0.0004, top-1 99.4 to 99.6%.
|
||||
- **Needles:** a code word hidden in 8K, 16K and 32K tokens at 10/50/90% depth was found 9 of 9 with Q2_0 and with IQ3_XXS.
|
||||
- **Same answers as on a PC:** against the CUDA engine on a Tesla P100 (IQ3_XXS and IQ3_S) the first 200 tokens of a reply on the
|
||||
short question are identical; against the HIP engine on a Radeon AI PRO R9700 (Q2_0), 36 of 39 first tokens are identical. The
|
||||
differences are near-ties between the two best tokens, from a different floating-point order.
|
||||
|
||||
## Switches
|
||||
|
||||
For testing. All are off unless set; the defaults are described above.
|
||||
|
||||
| Variable | Effect |
|
||||
| --- | --- |
|
||||
| `STRATA_HC_Q8=0` | The BF16 hyper-connection read (exact) instead of Q8_0 |
|
||||
| `STRATA_METAL_PROMPT=window` | Read the prompt through decode windows (slow, exact fallback; about 190 tok/s) |
|
||||
| `STRATA_METAL_PROMPT_CHUNK=N` | Prompt chunk size (default 2048 resident; see streaming above) |
|
||||
| `STRATA_METAL_MTP_Q8=0` | The draft layer's matrices in BF16 instead of Q8_0 |
|
||||
| `STRATA_METAL_MTP_CHAIN=0` | One GPU submission per draft step instead of one per round |
|
||||
| `STRATA_METAL_FUSE=0`, `STRATA_METAL_LAZY_FOLD=0` | Turn off two decode fusions (same tokens, slower) |
|
||||
| `STRATA_METAL_PARTIAL=1`, `STRATA_METAL_ROUTE=1` | Two decode variants that measured no faster. `ROUTE` is also wrong on an M1 Max (stale routing logits) |
|
||||
| `STRATA_METAL_PROMPT_EXPERTS=v3` | The M5 layout for the prompt's expert GEMMs. Fails on an M1 Max, so it is off there (default v2) |
|
||||
| `STRATA_METAL_STREAM=1` / `=0` | Stream the experts from the SSD even when the pack fits / never (the default decides by size) |
|
||||
| `STRATA_METAL_CONTIG=0` | Streaming: read the model file even when the experts' copy is there |
|
||||
| `STRATA_METAL_ARENA_GB=N`, `STRATA_METAL_KEEP_FREE_GB=N` | Streaming: the arena's size, or the memory left to macOS |
|
||||
| `STRATA_METAL_LEAVE_SHARE=0.4` | Streaming, 64 GB or more: the share of the RAM left to macOS when the footprint cap is set |
|
||||
| `STRATA_METAL_WIRE_ARENA=0` | Streaming, 64 GB or more: leave the filled arena pageable (default: wired) |
|
||||
| `STRATA_METAL_IGNORE_REPACK=1` | Start although a model repack or setup's preparation is running (the engine refuses by default) |
|
||||
| `STRATA_METAL_KEEP_HOST_WEIGHTS=1` | Streaming: keep the host code's own copies of the weights (2 to 3 GB) |
|
||||
| `STRATA_METAL_TRACE=1`, `STRATA_METAL_PROFILE=1`, `STRATA_METAL_KPROF=T` | Per-window traces, a time breakdown, per-kernel GPU times |
|
||||
|
||||
## How it works
|
||||
|
||||
The code is in `metal/` (the engine's side in `metal/src/engine`, the kernels in `metal/kernels`).
|
||||
|
||||
- **Host code:** the PC host code compiles unchanged for macOS. `metal/include/metal_compat` maps the CUDA runtime calls it makes
|
||||
onto a small Metal runtime (`metal/src/gpu`).
|
||||
- **Decode and prompt work** run in Metal kernels: one command buffer per verify window, experts read in place from the mapped
|
||||
model file, hyper-connections, GDN state and QSA attention on the GPU, and a whole MTP draft round in one command buffer.
|
||||
- **The PC builds are byte-identical** with this code present. Every shared file's change is behind `STRATA_GPU_METAL` or `__APPLE__`.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
**Why `--spec 2`.** Measured on: Apple M5 Max, 128 GB, decode tok/s with MTP on the two prompts (prompt1 / story).
|
||||
|
||||
| Window | Q2_0 | IQ3_XXS |
|
||||
| --- | --- | --- |
|
||||
| `--spec 2` | 111 / 106 | 102 / 95 |
|
||||
| `--spec 3` | 116 / 92 | 108 / 95 |
|
||||
| `--spec 4` | 111 / 81 | 108 / 83 |
|
||||
| `--spec 5-6` | slower | slower |
|
||||
|
||||
`--spec 2` is the best default on mixed text. An explicit `--spec N` in the config's `args` is kept.
|
||||
**The experts' copy.** Measured on: Apple M4 Mac mini, 32 GB, IQ3_XXS after a 6K-token document, arena held at 12 GB, medians of 5
|
||||
interleaved runs, identical tokens in all 10 runs.
|
||||
|
||||
### Gyro-S and Gyro-M (agentionai's trellis experts)
|
||||
| | Decode | The 6K prompt | SSD wait per token |
|
||||
| --- | --- | --- | --- |
|
||||
| The model file (three reads per expert) | 7.37 tok/s | 64.7 tok/s | 46.6 ms |
|
||||
| The experts' copy (one read per expert) | **8.62 tok/s** | **84.1 tok/s** | 31.5 ms |
|
||||
|
||||
From 0.2.2 the Mac engine runs [Gyro](GYRO.md): the routed experts are trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a
|
||||
Hadamard-rotated basis, read from the published file unchanged. The kernels are ours (`metal/kernels/quant.metal`, `moe.metal`,
|
||||
`moe_mm.metal`); the codebook and the decoder are the CPU engine's, the rotation is applied to the gate/up and down inputs as on CUDA.
|
||||
Setup offers Gyro-S and Gyro-M next to the other sizes (`./setup.sh --family gyro --model Gyro-S`), never as the default, and says
|
||||
what the choice risks on your Mac's memory. The 28.8 GB n-gram table is inside the file and is read from the SSD like every other
|
||||
pack's, so Gyro-S needs 58.5 GB of disk and about 38 GB of memory to stay resident (29.7 GB of weights).
|
||||
The 4K prompt read 72.9 tok/s with the copy against 62.5 without. A 34-minute server run with the copy (47 requests, context
|
||||
34,816) got every answer right and did not slow down. With the model file, an arena of 7.5 GB gave 5.5 tok/s, 10.5 GB gave 6.7 and
|
||||
12 GB gave 7.5. On the 128 GB M5 Max the copy for Q5 did not pay (decode 50.3 against 50.2 tok/s). The SSD alone measures 2.97 GB/s
|
||||
for one 1.75 MB read per expert against 2.30 GB/s for three 0.58 MB reads. The Q2_0 copy was not measured.
|
||||
|
||||
**Measured on the M4 Mac mini, 32 GB** (the experts stream from the SSD; no experts' copy, because the copy would need 26 GB more
|
||||
disk than this Mac had; one token per step, no draft layer; the arena was 11-12 GB of the 25.8 GB of experts):
|
||||
**Prompt chunk on a streaming Mac.** Measured on: M5 Max, UD-Q5, warm, 2 interleaved rounds. Chunks of 1024, 2048 and 4096 gave 700,
|
||||
920 and 1,100 tok/s on the 4K prompt (32K: 710, 970, 1,100). Decode did not change. On a resident model 4096 instead of 2048 gained
|
||||
only 2 to 4% (Q2_0), so that default stays.
|
||||
|
||||
| | Gyro-S |
|
||||
|---|---|
|
||||
| Decode, story, 120 tokens (5 interleaved pairs, medians) | **9.14 tok/s** (9.07-9.15) |
|
||||
| Decode, with the dense-weight tiles turned off (`STRATA_METAL_GYRO_TILES=0`) | 8.70 tok/s (8.69-8.73); same tokens in all 10 runs |
|
||||
| Prompt, 4,065 tokens (3 runs) | **109 tok/s** (108-110) |
|
||||
| Memory the engine held | 18.6 GB of the 32 GB |
|
||||
**Memory footprint cap.** Measured on: M5 Max, 128 GB. Before the cap, a streaming arena of 100 GB compressed 19 GB of itself in 7
|
||||
seconds, and the kernel watchdog panicked three times. With 40% of the RAM left to macOS and the arena wired, a 40 GB file write
|
||||
beside a 16 GB program caused no swap-out. A resident pack (UD-Q4_K_XL, wired 91.7 GB) took the same load with nothing compressed,
|
||||
so resident packs are limited to 70% of the RAM.
|
||||
|
||||
Where the time goes: a decode window of two tokens takes 111 ms. The routed experts are about 40 ms of it, the dense weights
|
||||
(Gyro keeps its attention, state-space and shared-expert matrices in Q5_K and Q6_K: 3.6 GB read a token)
|
||||
about 66 ms before the tile change (a run with those kernels skipped). The kernel for one expert layer (ten experts, one token) takes
|
||||
278 us on the M4, 37.5 GB/s; IQ3_XXS takes 333 us and Q2_0 166 us. The trellis decode is limited by latency and ALU, not by memory:
|
||||
32-bit funnel shifts for the stream bits, fetching the next unit while decoding the current one, one lane a row with the unit index
|
||||
folded away and the codebook in constant memory were all tried and none beat the plain kernel, so it is the plain kernel with the
|
||||
tile that measured fastest. The prompt kernels (fp16 matrix units) are on par with IQ3_XXS: 68 ms against 62 ms a layer for
|
||||
1,024 tokens.
|
||||
**Gyro-S kernel profile.** Measured on: M4 Mac mini, 32 GB. A decode window of two tokens takes 111 ms: the routed experts about
|
||||
40 ms, the dense weights (Q5_K and Q6_K, 3.6 GB read a token) about 66 ms before the tile change. The kernel for one expert layer
|
||||
(ten experts, one token) takes 278 us at 37.5 GB/s; IQ3_XXS takes 333 us and Q2_0 166 us. The trellis decode is limited by latency
|
||||
and ALU, not by memory. The prompt kernels run 68 ms a layer against 62 ms for IQ3_XXS at 1,024 tokens.
|
||||
|
||||
**Checked against the CPU engine's decoder, on the real file:** the decoder equals the format's definition written out bit by bit
|
||||
(614,400 sampled blocks a layer, none differ); the Hadamard rotation equals the CPU's bit for bit (f32 and f16); the expert
|
||||
kernels reproduce a double-precision reference to 1e-7 relative on real experts of layers 0 and 47 of Gyro-S and of layers 0, 23
|
||||
and 47 of Gyro-M (the 92 GB file was read by range requests, 16 experts a layer: it does not fit this Mac's disk). Run it with
|
||||
`metal/tests/gyro_parity.sh`.
|
||||
**Quality against the CUDA engine, Gyro-S file.** `tools/quant_quality.py`, teacher-forced over 4,995 positions, CUDA = RTX 3060.
|
||||
|
||||
**Against the CUDA engine, same file** (the pinned Gyro-S; `tools/quant_quality.py`, teacher-forced over the 5 text segments of
|
||||
`data/quant_quality/v1`, 4,995 positions; CUDA = an RTX 3060 with the engine of this release, run twice: experts on the GPU, experts on the CPU):
|
||||
|
||||
| Reference -> run | top-1 agreement | mean KL | KL p99 | perplexity of the text |
|
||||
|---|---|---|---|---|
|
||||
| CUDA GPU experts -> CUDA CPU experts (the noise floor: two numerics of one engine) | 95.80 % | 0.0186 | 0.195 | 4.462 / 4.477 |
|
||||
| CUDA GPU experts -> Mac | 95.46 % | 0.0207 | 0.241 | 4.486 / 4.477 |
|
||||
| CUDA CPU experts -> Mac | 95.38 % | 0.0187 | 0.272 | 4.486 / 4.462 |
|
||||
|
||||
The Mac is as far from either CUDA run as the two CUDA runs are from each other (the CUDA GPU path quantizes the activations to q8_1 and
|
||||
reads an int8 copy of the codebook, the CPU path an int16 one; the Mac multiplies fp16 codebook values with the float activations).
|
||||
|
||||
### Quality
|
||||
|
||||
- **The Q8_0 hyper-connection read is the default on Mac.** It keeps a Q8_0 copy of the hyper-connection matrices instead of BF16. `STRATA_HC_Q8=0` in the config's `env` gives the exact BF16 read.
|
||||
- **How the KL was measured:** teacher-forced over 28 prompts (code, reasoning, chat, six languages, documents), about 6,000 positions per set. The reference is BF16.
|
||||
- On the model's own text: mean KL 0.0007, top-1 agreement 98.9%.
|
||||
- On another model's text: mean KL 0.0007, top-1 agreement 99.1%.
|
||||
- Perplexity of the text is unchanged (×0.9995 and ×1.0006).
|
||||
- For scale, IQ3_XXS against Q2_0 is a KL of 0.094.
|
||||
- The Mac engine is deterministic: two runs give identical numbers.
|
||||
- **The fast prompt path** (the prompt kernels) against reading the prompt through decode windows: mean KL 0.0004, top-1 99.4-99.6%.
|
||||
- **Needles:** a code word hidden in 8K, 16K and 32K tokens of text, at 10/50/90% depth. Found 9 of 9 with Q2_0 and with IQ3_XXS, both with Q8_0 and with BF16. With the fast prompt path (4K-32K) the answers are the same.
|
||||
- **Same answers as on a PC:**
|
||||
- against the CUDA engine on a Tesla P100 (IQ3_XXS and IQ3_S), the first 200 tokens of a reply are identical on the short question;
|
||||
- on the story, the first IQ3_S tokens differ at token 1, a near-tie (margin 0.12). Following the CUDA engine's tokens, 4 of 200 picks differ, all at margins 0.06-0.21;
|
||||
- against the HIP engine on a Radeon AI PRO R9700 (Q2_0), 36 of 39 first tokens are identical.
|
||||
- The differences are near-ties between the two best tokens, from a different floating-point order.
|
||||
|
||||
## Limits (this version)
|
||||
|
||||
- **Images (vision):** on by default. The app and the Terminal use the `strata-vision` that sits beside `strata` in the engine package;
|
||||
`./setup.sh` builds it itself and asks "Should the model also read images?" (default yes). One program does both jobs: with `--gpu` it
|
||||
reads on the Mac's GPU (ggml's Metal backend, the shaders are built into the program), without it on the CPU. A Mac always gets the GPU
|
||||
reader, whatever its memory. The 24 GB rule of the PC cards does not apply: the Mac has no separate card memory to keep free, and the
|
||||
reader's footprint is about the same on either side (1.18 GB on the GPU, 1.05 GB on the CPU, mostly the 0.9 GB projector file).
|
||||
Measured on a Mac mini M4 (32 GB), Qwen3.8-Flash-Next's BF16 projector:
|
||||
|
||||
| Picture | GPU reader | CPU reader |
|
||||
|---|---|---|
|
||||
| 260 tokens (640x400) | 0.45 s | 50.7 s |
|
||||
| 972 tokens (1600x1200) | 2.4 s | 194 s |
|
||||
|
||||
`--vision cpu` is still allowed and prints that note. The CPU reader is slow because ggml has no vector kernel for BF16 on ARM (a build
|
||||
for newer chips, with the BF16 and int8 instructions, took the same 50.7 s). If the GPU reader cannot start, the server prints one line
|
||||
and starts the same program on the CPU. With Q2_0 streaming from the SSD on that M4, the STRATA 42 test picture (red circle, blue square,
|
||||
green triangle, the text "STRATA 42") was described correctly with both readers. Checked on one Mac only; the M5 Max is untested for pictures.
|
||||
- **Context:** measured up to 32K (needles, KL). Setup recommends 32K below 96 GB and 64K from 96 GB. Up to the model's 262K can be set, but it is not measured.
|
||||
- **One request at a time.** `--parallel` (batch slots) is not ported. Several clients queue.
|
||||
- **Not ported:** the control vector (`--control-vector-scaled`) and the PC tuning (`--calibrate`).
|
||||
- **Streaming:** measured at 32 GB only. The model's file must be on the internal SSD; an external disk is untested and likely much slower.
|
||||
- **Families:** the original Qwen3.8-Flash-Next, Unsloth's UD-Q4/Q5/Q6_K_XL and agentionai's Gyro-S are measured (above). Setup repacks Unsloth's shards into one file for the GPU; a Gyro file is used as it is. Gyro-M runs on the same kernels (checked tensor by tensor) but no Mac here could hold it. The Swift and Coder releases are not set up on Mac yet, and Qwen3.6 does not run on the Mac engine.
|
||||
- **The ready-made Mac engine is in the app and in the Terminal's engine component, not in the release zips.** The zips
|
||||
(for `START-HERE.bat` and `setup.sh`) have no Mac engine: `./setup.sh` builds it.
|
||||
|
||||
## Switches (for testing; all off unless set, the defaults are above)
|
||||
|
||||
| Variable | Effect |
|
||||
|---|---|
|
||||
| `STRATA_HC_Q8=0` | The BF16 hyper-connection read (exact) instead of Q8_0 |
|
||||
| `STRATA_METAL_PROMPT=window` | Read the prompt through decode windows (the slow, exact fallback; ~190 tok/s) |
|
||||
| `STRATA_METAL_PROMPT_CHUNK=N` | Prompt chunk size (default 2048) |
|
||||
| `STRATA_METAL_MTP_Q8=0` | The draft layer's matrices in BF16 instead of Q8_0 |
|
||||
| `STRATA_METAL_MTP_CHAIN=0` | One GPU submission per draft step instead of one per round |
|
||||
| `STRATA_METAL_FUSE=0`, `STRATA_METAL_LAZY_FOLD=0` | Turn off two decode fusions (same tokens, slower) |
|
||||
| `STRATA_METAL_PARTIAL=1`, `STRATA_METAL_ROUTE=1` | Two decode variants that measured no faster here (same tokens) |
|
||||
| `STRATA_METAL_STREAM=1` / `=0` | Stream the experts from the SSD even when the pack fits / never (the default decides by size) |
|
||||
| `STRATA_METAL_CONTIG=0` | Streaming: read the model file even when the experts' copy is there |
|
||||
| `STRATA_METAL_ARENA_GB=N`, `STRATA_METAL_KEEP_FREE_GB=N` | Streaming: the arena's size, or the memory left to macOS when the arena is sized |
|
||||
| `STRATA_METAL_LEAVE_SHARE=0.4` | Streaming, Macs with 64 GB or more: the share of the RAM left to macOS and its file cache when the footprint cap is set (for measuring) |
|
||||
| `STRATA_METAL_WIRE_ARENA=0` | Streaming, Macs with 64 GB or more: leave the filled arena pageable (default: wired with mlock) |
|
||||
| `STRATA_METAL_IGNORE_REPACK=1` | Start the engine although a model repack or setup's preparation is running beside it (the engine refuses by default: it froze this Mac) |
|
||||
| `STRATA_METAL_KEEP_HOST_WEIGHTS=1` | Streaming: keep the host code's own copies of the weights (2-3 GB) instead of giving them back to macOS |
|
||||
| `STRATA_METAL_TRACE=1`, `STRATA_METAL_PROFILE=1`, `STRATA_METAL_KPROF=T` | Per-window traces, a time breakdown, per-kernel GPU times |
|
||||
|
||||
**Two opt-ins an M1 Max cannot run (measured on an M1 Max, macOS 26.6.2, Q2_0):** `STRATA_METAL_PROMPT_EXPERTS=v3` - the
|
||||
prompt's expert GEMMs with the M5's layout, `mm_gu3` - loses the threadgroup once a prompt spreads over more than 40
|
||||
threadgroups of it (35 pass, 40 fail; 190 tiles and the engine's tile both fail, one tile passes), so it is off there (the
|
||||
default is v2). `STRATA_METAL_ROUTE=1` - the router GEMV and `moe_route` fused into one dispatch - routes with stale
|
||||
logits and gives a different top-10 per run (two identical runs differ in ids, weights and plan; the logits themselves
|
||||
stay bitwise): the fusion hands the routing to the last threadgroup of the GEMV through a device counter, and this
|
||||
Metal's device atomics are relaxed-only, with no acquire to pair with the writers' fence. Both are off by default. The
|
||||
`no faster` measurement in the table is from the Macs above; on an M1 Max `ROUTE` is also wrong.
|
||||
|
||||
## How it works
|
||||
|
||||
The code is in `metal/` (the engine's side in `metal/src/engine`, the kernels in `metal/kernels`). In short:
|
||||
- **Host code:** the PC host code compiles unchanged for macOS. `metal/include/metal_compat` maps the CUDA runtime calls it makes onto a small Metal runtime (`metal/src/gpu`).
|
||||
- **The decode and prompt work** run in Metal kernels (`metal/kernels`, `metal/src/engine`):
|
||||
- one command buffer per verify window;
|
||||
- the experts are read in place from the mapped model file;
|
||||
- the hyper-connections, the GDN state and the QSA attention stay on the GPU;
|
||||
- a whole MTP draft round runs in one command buffer.
|
||||
- **The PC builds are byte-identical** with this code present. Every shared file's change is behind `STRATA_GPU_METAL` or `__APPLE__`, and the CUDA and HIP objects are compared after each change.
|
||||
| Reference -> run | top-1 agreement | mean KL | KL p99 | perplexity |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| CUDA GPU experts -> CUDA CPU experts (noise floor) | 95.80% | 0.0186 | 0.195 | 4.462 / 4.477 |
|
||||
| CUDA GPU experts -> Mac | 95.46% | 0.0207 | 0.241 | 4.486 / 4.477 |
|
||||
| CUDA CPU experts -> Mac | 95.38% | 0.0187 | 0.272 | 4.486 / 4.462 |
|
||||
+13
-8
@@ -1,13 +1,14 @@
|
||||
# Strata's MCP server: let your AI assistant install and run Strata
|
||||
|
||||
`tools/strata_mcp.py` is an [MCP](https://modelcontextprotocol.io) server for the AI assistant you already use
|
||||
(Claude Code, Claude Desktop, Cursor, VS Code, Codex ...). Add it once, then ask in plain words:
|
||||
This page is for people who use an AI assistant and want it to install, start, stop and check Strata. `tools/strata_mcp.py`
|
||||
is an [MCP](https://modelcontextprotocol.io) server for the assistant you already use (Claude Code, Claude Desktop,
|
||||
Cursor, VS Code, Codex ...). Add it once ([Quick start](#add-it)), then ask in plain words:
|
||||
|
||||
- "Install Strata for this PC."
|
||||
- "Start Strata." / "Stop Strata."
|
||||
- "Is Strata running? How much VRAM does it use?"
|
||||
- "Which Strata model fits my PC?"
|
||||
- "Strata didn't start - what does the log say?"
|
||||
- "Strata didn't start. What does the log say?"
|
||||
- "How fast is Strata on this PC?" (a short speed test)
|
||||
- "How do I connect Open WebUI / the OpenAI SDK / Claude Code to Strata?"
|
||||
|
||||
@@ -21,6 +22,9 @@ direction: *your assistant* manages Strata.
|
||||
|
||||
## Add it
|
||||
|
||||
This is an optional **source-checkout tool**. It is not required to install the released app or Terminal.
|
||||
For normal agent-assisted installation, use [AI_SETUP.md](AI_SETUP.md).
|
||||
|
||||
You need a copy of Strata: the zip from GitHub, or `git clone https://github.com/Niko1221/Strata`. Below,
|
||||
`C:\Users\you\Strata` (Windows) and `/home/you/Strata` (Linux) stand for that folder. Use `python3` on Linux. On
|
||||
Windows, use `py` if `python` is not on your PATH.
|
||||
@@ -110,9 +114,10 @@ The command is `python <Strata folder>/tools/strata_mcp.py`. The transport is st
|
||||
| Tool | What it does | Arguments (all optional) |
|
||||
| --- | --- | --- |
|
||||
| `strata_status` | Whether a model is running: the model, engine version, context, busy or idle, VRAM and RAM in use. What is installed: models, start scripts, engine version, data folder, downloads. An install in progress. This PC's hardware: OS, CPU, RAM, GPUs, page file, free disk. The model and context setup recommends for this PC. | `port`, `hardware` (true/false) |
|
||||
| `strata_models` | Every family and size setup offers, with download size, RAM needed, whether it fits this PC, and whether it is installed. Also the context lengths and the recommendation. | - |
|
||||
| `strata_install` | Runs setup without questions, in the background (see below). Without `confirm: true` it only returns the plan. A second call shows the progress. `cancel: true` stops the install. | `family` (qwen, swift, coder, unsloth), `model` (Q2_0, IQ2_XS, IQ3_XXS, IQ3_S, IQ1_M, UD-IQ4_XS, UD-Q4_K_XL), `context` (8192 ... 524288), `vision` (no, yes, gpu, cpu), `kv` (int8, q4_0, k8v4), `gpu` (a number) or `gpus` ("0,2", "all"), `backend` (auto, cuda, hip), `low_ram`, `port`, `data_dir`, `confirm`, `cancel` |
|
||||
| `strata_start` | Starts an installed model in the background and waits until it answers. If it is not ready after `wait_seconds` (default 50), it returns "loading". | `model` (an id from `strata_status`, e.g. `q2_0`, `coder-iq1_m`; default: the most recently used), `port`, `gpu`, `wait_seconds` (0-600) |
|
||||
| `strata_models` | Every family and size setup offers (qwen, swift, coder, unsloth, gyro, nvfp4, and Qwen3.6-35B-A3B), with download size, RAM needed, whether it fits this PC, and whether it is installed. Also the context lengths and the recommendation. | - |
|
||||
| `strata_remove` | Deletes an installed model's own files (GGUFs, pack folder, run config) and nothing another installed model still uses. Without `confirm: true` it only returns the plan. Stop the model first. | `model` (required), `confirm` |
|
||||
| `strata_install` | Runs setup without questions, in the background (see below). Without `confirm: true` it only returns the plan. A second call shows the progress. `cancel: true` stops the install. | `family` and `model` (the ones `strata_models` lists), `context` (1024 ... 2097152; usual sizes 8192 ... 524288), `vision` (no, yes, gpu, cpu), `kv` (int8, q4_0, k8v4), `gpu` (a number) or `gpus` ("0,2", "all"), `backend` (auto, cuda, hip), `low_ram` (auto, on, off, resident, mmap), `port`, `data_dir`, `gguf_dir` (a folder with the model's GGUF files, checked and not downloaded again), `confirm`, `cancel` |
|
||||
| `strata_start` | Starts an installed model in the background and waits until it answers. If it is not ready after `wait_seconds` (default 50), it returns "loading". | `model` (an id from `strata_status`, e.g. `q2_0`, `coder-iq1_m`, `swift-iq3_xxs`; default: the most recently used), `port`, `gpu`, `wait_seconds` (0-600) |
|
||||
| `strata_stop` | Unloads the model through the server's own API, then ends the server it started. | `port`, `force` |
|
||||
| `strata_logs` | The last lines of the server log, the engine log or the setup log. | `source` (server, engine, setup), `model`, `lines` (1-500) |
|
||||
| `strata_benchmark` | Sends a short fixed request (greedy, thinking off) and reports output and prompt tokens/s. | `port`, `max_tokens` (16-512) |
|
||||
@@ -130,7 +135,7 @@ Every result is JSON with a one-line `summary`. It is sent both as text and as M
|
||||
- the context setup recommends for the GPU's VRAM;
|
||||
- pictures on where the model can read them (the reader runs on the CPU under 24 GB of VRAM and on the GPU from 24 GB, on NVIDIA and AMD cards; a Mac reads on its GPU).
|
||||
|
||||
The first call returns the plan: the download size (58-111 GB), the free disk space, the RAM check, and the exact
|
||||
The first call returns the plan: the download size, the free disk space, the RAM check, and the exact
|
||||
setup command. The assistant should show you the plan and call again with `confirm: true` once you agree.
|
||||
|
||||
It then runs the same steps `START-HERE.bat` / `setup.sh` run:
|
||||
@@ -164,7 +169,7 @@ VRAM and pinned RAM itself, as when you close the window. Then the server ends:
|
||||
|
||||
While a request is running, the tool refuses unless you pass `force: true`.
|
||||
|
||||
**Status.** The tool reads `GET /health`, `/v1/status`, `/status` and, on older servers, `/metrics` on 127.0.0.1.
|
||||
**Status.** The tool reads `GET /health`, `/v1/status`, `/status` and `/metrics` on 127.0.0.1.
|
||||
If a config of the folder sets an `api_key`, the tool sends that key with its own requests. It never shows the key.
|
||||
|
||||
## Safety
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# Strata media and video guide
|
||||
|
||||
For people and AI tools creating an article, overview or video. Last reviewed: 25 October 2026, for Strata 0.2.5.
|
||||
Use [AI_REFERENCE.md](AI_REFERENCE.md) for product facts and [README.md](../README.md) for the public introduction.
|
||||
|
||||
## Short description
|
||||
|
||||
Strata runs supported AI models on your own computer. Use its desktop app to chat and work with code and images,
|
||||
use its terminal app, or connect a coding tool. One installer per system (Windows, Linux, Mac). Strata's code is free and open source under the MIT license.
|
||||
|
||||
Suggested opening: **“Run powerful AI on your own computer. Chat, code and work with images, locally.”**
|
||||
|
||||
## A short video outline
|
||||
|
||||
1. **Show the desktop app.** A chat and a visible result explain the product quickly.
|
||||
2. **Show the choice of interfaces.** The desktop app for a visual workspace; the terminal app for keyboard-driven use.
|
||||
3. **Show the three model families.** DeepSeekV4.1-Flash-1017, Qwen3.8-Flash-Next and Qwen3.6-35B-A3B.
|
||||
Explain the [availability limits](AI_REFERENCE.md#model-families): DeepSeek V4.1-Flash is in the app and the Terminal since 0.2.5, on NVIDIA RTX 30, 40 and 50 and the Radeon AI PRO R9700 (Linux).
|
||||
4. **Choose a model for the machine.** Prefer ISTA-DASLab, then Unsloth, Gyro and other providers where compatible.
|
||||
Gyro is a memory-saving option, currently slower. State the model, quantization and hardware used in any live demo.
|
||||
5. **Connect a familiar tool.** Examples include Claude Code, Codex CLI, OpenCode, oh-my-pi, pi.dev and Hermes Agent.
|
||||
Show a real successful connection if demonstrating one.
|
||||
6. **Close with the two useful links.** [Download Strata](https://github.com/Niko1221/Strata/releases/latest)
|
||||
and [support development](https://buymeacoffee.com/strataengine).
|
||||
|
||||
This is an outline, not evidence that any particular model, workflow or benchmark was tested for the video.
|
||||
|
||||
## Existing assets
|
||||
|
||||
Use the existing assets with accurate captions. These screenshots demonstrate the interfaces; they do not
|
||||
establish a current default model or universal performance.
|
||||
|
||||
| Asset | File | Suggested caption / context |
|
||||
| --- | --- | --- |
|
||||
| Logo, light background | [strata-lockup-light.svg](../apps/host/public/brand/strata-lockup-light.svg) | Strata |
|
||||
| Logo, dark background | [strata-lockup-dark.svg](../apps/host/public/brand/strata-lockup-dark.svg) | Strata |
|
||||
| Desktop app | [app-chat.png](media/app-chat.png) | Chat, files and a live preview in one window. The capture shows a Gyro model; do not relabel it as an ISTA or DeepSeek run. |
|
||||
| Terminal app | [terminal-monitor.png](media/terminal-monitor.png) | Model activity and hardware use. The capture shows a Gyro-S session; its live counters are not a standardized benchmark. |
|
||||
| Coding demo thumbnail | [pagoda-preview.webp](media/pagoda-preview.webp) | A voxel garden generated locally. |
|
||||
| Coding demo video | [Pagoda.mp4](https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4) | Existing 49-second demo, documented with an RTX 5070, IQ3_S and 128K context. Keep it separate from the GUI and TUI screenshots. |
|
||||
| Engine explanation | [how-it-works.svg](media/how-it-works.svg), [guess-and-check.svg](media/guess-and-check.svg) | Illustrations of the documented Flash-Next engine path, not proof that every model uses the same path. |
|
||||
|
||||
## Accuracy rules
|
||||
|
||||
- Use actual footage or label illustrations. Do not invent a successful install, generated answer or speed reading.
|
||||
- Keep the provider preference distinct from measured performance. The maintainer's collaboration with ISTA-DASLab
|
||||
explains the recommendation order; it does not prove a benchmark result.
|
||||
- DeepSeek V4.1-Flash runs in the desktop app and in the Terminal (0.2.5), on NVIDIA RTX 30, 40 and 50 and the Radeon AI PRO R9700 on Linux. Take its sets (1.75, 2.00 and 2.25 bpw), sizes and speeds from
|
||||
[MODELS.md](MODELS.md#deepseek-v41-flash) and name the hardware. Do not invent quants or memory figures.
|
||||
- Do not promise “any PC,” “every model,” or identical results across GPUs. Follow the model and hardware guides.
|
||||
- For speed comparisons, use a source report with hardware, model, quantization, engine version and workload.
|
||||
Keep reasoning tokens, decode speed, prompt speed and time to first answer distinct.
|
||||
- Local inference, internet downloads and connected tools are separate. Refer to [SAFETY.md](SAFETY.md) for privacy claims.
|
||||
- Strata's MIT license does not replace the licenses of model weights, dependencies or other creators' work.
|
||||
Use the [credits and licensing guide](HOW_IT_WORKS.md#credits).
|
||||
|
||||
## Links and contact
|
||||
|
||||
- Project: [Niko1221/Strata](https://github.com/Niko1221/Strata)
|
||||
- Downloads: [Releases](https://github.com/Niko1221/Strata/releases/latest)
|
||||
- Support: [Buy Me a Coffee](https://buymeacoffee.com/strataengine)
|
||||
- Sponsorships, hardware partnerships and collaborations: [[email protected]](mailto:[email protected])
|
||||
+264
-199
@@ -1,251 +1,265 @@
|
||||
# Which model? Sizes, versions and what fits
|
||||
|
||||
Strata's main model is [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), in several **sizes**
|
||||
(the same model, compressed more or less) and several **versions** (the original, a coding version, a fine-tune, other
|
||||
makers' compressions). It also runs a smaller model, [Qwen3.6-35B-A3B](#qwen36-35b-a3b). The desktop app, the Terminal
|
||||
(`strata launch`) and the installer recommend one for your PC; this page explains the choice. Back to the
|
||||
[README](../README.md).
|
||||
This page helps you choose a model and check that your computer can run it. It covers DeepSeek V4.1-Flash,
|
||||
Qwen3.8-Flash-Next (and its variants) and Qwen3.6-35B-A3B. For installing, see [INSTALL.md](INSTALL.md).
|
||||
|
||||
> **On this page:** [Pick by memory](#pick-by-memory) · [Speed](#how-fast-is-each-size) · [The sizes](#the-sizes) ·
|
||||
> [Will it fit?](#will-it-fit) · [The versions](#the-versions) · [Qwen3.6](#qwen36-35b-a3b) ·
|
||||
> [Adding another model](#adding-or-switching-models)
|
||||
> **On this page:** [Quick start](#quick-start) · [Pick by memory](#pick-by-memory) ·
|
||||
> [DeepSeek V4.1-Flash](#deepseek-v41-flash) · [Qwen3.8-Flash-Next](#qwen38-flash-next) ·
|
||||
> [Qwen3.6-35B-A3B](#qwen36-35b-a3b) · [Adding or switching models](#adding-or-switching-models) ·
|
||||
> [History](#history--older-measurements)
|
||||
|
||||
## Quick start
|
||||
|
||||
1. Find your memory in [Pick by memory](#pick-by-memory) and choose the first provider that fits.
|
||||
2. Print the plan. Nothing is downloaded or changed:
|
||||
|
||||
```sh
|
||||
strata launch --plan --family qwen --model iq2_xs
|
||||
```
|
||||
|
||||
3. Start it with `strata launch` (guided), or run the same command without `--plan` and add `--yes --no-tui`.
|
||||
|
||||
**Provider order:** ISTA-DASLab, then Unsloth, then Gyro, then others, using the first compatible pack that fits the
|
||||
model and your hardware. ISTA-DASLab is the default recommendation. Gyro saves memory and is slower. Swift 1.5, NVFP4
|
||||
and OrcaRouter are other options. An explicit choice by the user always wins. The order applies within a model family:
|
||||
Qwen3.6 has Unsloth packs only, and DeepSeek has the IST-DAS sets. The order is a recommendation. It says nothing about which
|
||||
provider wins a given quality or speed comparison.
|
||||
|
||||
Setup may preselect something else: a Swift or Gyro size at a low memory tier, or Unsloth on Strix Halo. Read the plan
|
||||
and pass `--family` and `--model` when you want a particular provider.
|
||||
|
||||
## Pick by memory
|
||||
|
||||
The tiers count RAM and VRAM (the graphics card's memory) together, the same as the [README](../README.md#the-other-sizes).
|
||||
`strata launch`, setup and the install tool of the MCP server pre-select by them; the desktop app does from 0.2.1. Every
|
||||
other size stays a choice. Strata leaves 5 GB for Windows and the system, so the sum is RAM + VRAM minus 5 GB: a PC with a card of 12 GB and 32 GB of RAM is in the 40 GB tier (32 + 12 - 5 = 39), and a PC with a 16 GB card and 32 GB of RAM is too, not in the 48 GB tier (#3425).
|
||||
Figures are the model's weights in RAM plus VRAM. Leave memory for the OS and other apps. Unified memory counts once.
|
||||
They are estimates; a given context may still not fit.
|
||||
|
||||
| Your memory (RAM + VRAM) | Pre-selected | Notes |
|
||||
| --- | --- | --- |
|
||||
| **32 GB** | **Gyro-S**, or the **Coder** for code | The Coder needs 32 GB of RAM and keeps half of the experts, so it is weaker outside code ([why](#coder)). Gyro-S has lower quality than the 2-3-bit sizes, and the 2-bit Q2_0 decodes faster |
|
||||
| **40 GB** | **Swift 1.5** IQ2_XS (39.2 GB) | Gyro-M (35 GB) is the other choice: close to IQ3_XXS in quality, but it writes answers 16-21% slower than Q2_0. Gyro runs on NVIDIA, AMD and Intel Arc cards and, from 0.2.2, on Macs (Gyro-S measured on a 32 GB Mac mini) |
|
||||
| **48 GB** | **IQ3_XXS** (47.0 GB) | Slower, better |
|
||||
| **64 GB** | **IQ3_S** (54.8 GB), the original. Swift 1.5's IQ3_S is an equal choice | The slowest, the best: it matches the full model on the published tests |
|
||||
| **96 GB or more** | [Unsloth's UD-IQ4_XS to Q8_0](#unsloths-files-ud-iq4_xs-to-q8_0), 4 to 8 bit | UD-IQ4_XS keeps every expert in RAM from about 80 GB of RAM, Q8_0 from about 136 GB. Up to 192 GB, the 5-8-bit files fit in RAM. Setup still pre-selects IQ3_S here, except on a Strix Halo |
|
||||
**Qwen3.8-Flash-Next, ISTA-DASLab original packs**
|
||||
|
||||
With less memory than a row says, Strata's [low-RAM mode](#a-big-graphics-card-and-little-ram) starts by itself and the
|
||||
model runs slower, much slower with a small card. The figures above are RAM + VRAM; [Will it fit?](#will-it-fit) counts
|
||||
RAM alone, for the sizes that keep their experts in RAM.
|
||||
|
||||
Not sure? Take what the installer pre-selects. The Coder fits a 32 GB PC, but it keeps only 256 of the 512 experts,
|
||||
chosen on code data, so it is weaker outside code and in languages other than English. For general use, take a full
|
||||
model (Q2_0, IQ2_XS or IQ3_S) whenever your memory allows.
|
||||
|
||||
## How fast is each size
|
||||
|
||||
These are older numbers, kept because they compare the sizes with each other. Newer engines are faster on some cards
|
||||
(each change is listed with its own measurement in the [details](DETAILS.md)); the 0.2.0 engine (0.1.45.3) has not been
|
||||
re-measured on the whole table. Measured on an RTX 5070 (12 GB), a Ryzen 5 7600 and 64 GB of RAM, engine 0.1.26 (the
|
||||
Q2_0 row: engine 0.1.36, which made Q2_0 prompts faster):
|
||||
|
||||
| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| **Q2_0** | 94 tokens/s | 76 tokens/s | 2,650 tokens/s |
|
||||
| **IQ2_XS** | 79 tokens/s | 63 tokens/s | 2,090 tokens/s |
|
||||
| **IQ3_XXS** | 62 tokens/s | 49 tokens/s | 1,750 tokens/s |
|
||||
| **IQ3_S** | 53 tokens/s | 46 tokens/s | 1,620 tokens/s |
|
||||
| **Coder** (IQ1_M) | 55 tokens/s | 43 tokens/s | 2,180 tokens/s |
|
||||
|
||||
Measured on an AMD RX 9070 XT (16 GB), a Ryzen 9 3900X and 47 GB of RAM (Linux, setup's own install, 2026-10-02, an engine
|
||||
from before 0.1.40; Q2_0 is above setup's RAM estimate for 47 GB and was installed with `--model Q2_0 --yes`):
|
||||
|
||||
| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| **Q2_0** | 60 tokens/s | 48 tokens/s | 1,160 tokens/s |
|
||||
| **IQ2_XS** | 52 tokens/s | 36 tokens/s | 1,110 tokens/s |
|
||||
| **Coder** (IQ1_M) | 44 tokens/s | 33 tokens/s | 1,420 tokens/s |
|
||||
|
||||
- **Writes answers** = how fast the reply appears (tokens per second; a token is about ¾ of a word).
|
||||
- **Reads your prompt** = how fast it takes in what you send (long documents, code, chat history), measured on a
|
||||
32K-token prompt; a 4K prompt reads at 910-1,580 tokens/s. A 32K prompt takes about 15 seconds with Q2_0.
|
||||
|
||||
A card with more VRAM is faster, because more of the model fits on the GPU. For an RTX 3090 (24 GB) the
|
||||
[details](DETAILS.md#other-gpus-estimated) give an estimate, not a measurement. All measurements, long-context numbers
|
||||
and estimates for other cards are in the [details](DETAILS.md#speed-measured). The AMD measurements per card (RX 9070 XT,
|
||||
Radeon AI PRO R9700, RX 7800 XT, RX 9060 XT, RX 6900 XT) are in [AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201).
|
||||
|
||||
Every PC is different: `START-HERE.bat --calibrate` measures a few engine settings on yours and keeps the fastest
|
||||
(about 15-30 minutes, longer on a slow card; on the PC above it made the Coder 7% faster; NVIDIA cards for now).
|
||||
`strata launch` and the app have their own tuning step (quick: about 20 minutes). Measured Strata on your own
|
||||
PC? See [Community benchmark results](COMMUNITY_BENCHMARKS.md) for a report template and how to share your results
|
||||
in a pull request.
|
||||
|
||||
## The sizes
|
||||
|
||||
| Model | RAM+VRAM Requirements | Speed | Quality |
|
||||
| Size | RAM + VRAM for the model | Speed | Quality |
|
||||
| --- | ---: | --- | --- |
|
||||
| **Q2_0** | 37.6 GB | fastest | good |
|
||||
| **IQ2_XS** | 39.2 GB | fast | better (**recommended** at 40 GB, as Swift 1.5) |
|
||||
| **IQ3_XXS** | 47.0 GB | slower | great |
|
||||
| **IQ3_S** | 54.8 GB | slowest | best: matches the full model on the published tests |
|
||||
| Q2_0 | 37.6 GB | fastest | good |
|
||||
| IQ2_XS | 39.2 GB | fast | better |
|
||||
| IQ3_XXS | 47.0 GB | slower | great |
|
||||
| IQ3_S | 54.8 GB | slowest | best; matches the full model on the published tests |
|
||||
|
||||
The download is 66-76 GB for the three smaller sizes ([details](DETAILS.md#which-model)); the first start also
|
||||
fetches the MTP draft layer (~6 GB, +1 GB with images).
|
||||
- **Less memory, coding only:** [Coder](#coder) runs around 32 GB of RAM with a suitable GPU.
|
||||
- **Less memory, general use:** compare Unsloth, then [Gyro](#gyro-s-and-gyro-m-agentionai), which uses the least memory and is slower.
|
||||
- **More memory:** [Unsloth's 4 to 8 bit files](#unsloths-files-ud-iq4_xs-to-q8_0) need 72 to 136 GB for all experts in RAM.
|
||||
- **A smaller model:** [Qwen3.6-35B-A3B](#qwen36-35b-a3b), 32 to 52 GB of RAM.
|
||||
- **A very large model:** [DeepSeek V4.1-Flash](#deepseek-v41-flash), 12 GB graphics card, 46 to 64 GB of RAM and 257 GB of disk.
|
||||
|
||||
## Will it fit?
|
||||
If RAM is short, setup can read experts from the SSD instead ([below](#will-it-fit)). That works and is much slower.
|
||||
|
||||
Shard 1 is the part of the model that gets loaded when it starts: its experts go into your **RAM**, the rest onto
|
||||
your graphics card (the second shard, a 29 GB lookup table, stays on the SSD). So it fits when your **RAM is at least
|
||||
the experts + about 10 GB** for Windows and your other programs - the experts are most of shard 1: 34 GB for Q2_0,
|
||||
35.5 for IQ2_XS, 43 for IQ3_XXS, 50 for IQ3_S, 23 for the Coder (the dense weights in shard 1 go to the graphics
|
||||
card). With 64 GB of RAM every size fits (IQ3_S with little else open); with 48 GB, Q2_0 and IQ2_XS. A bigger graphics card makes it faster, but it doesn't lower the RAM needed
|
||||
- except in the low-RAM mode below.
|
||||
## DeepSeek V4.1-Flash
|
||||
|
||||
### A big graphics card and little RAM
|
||||
DeepSeek V4.1-Flash-1017 runs in the **desktop app** (pick it in the model list) and in the **Terminal** (`strata launch`).
|
||||
It uses its own engine, a separate download that the app and `strata launch` fetch for you.
|
||||
|
||||
On a PC whose RAM cannot hold the experts beside the system, setup maps them from the model's files instead and keeps
|
||||
in RAM only what the graphics card does not hold (the [low-RAM mode](DETAILS.md#speed-measured), chosen by setup). For
|
||||
example, a 32 GB PC with a 24 GB GPU runs Q2_0, IQ2_XS and the Coder this way, and a 32 GB PC with a 12-16 GB GPU the
|
||||
Coder. With a small card most experts then come from the SSD and it is much slower (setup says so).
|
||||
`START-HERE.bat --setup --low-ram on|off` overrides the choice.
|
||||
| Set (IST-DAS) | Download | Quality against the original model |
|
||||
| --- | ---: | --- |
|
||||
| 1.75 bpw | 239 GB | perplexity +21%; code and agent text +8.9% and +10.2%; weaker on plain prose |
|
||||
| 2.00 bpw | 257 GB | perplexity +16%; code and agent text +8.5% and +5.9% |
|
||||
| 2.25 bpw | 274 GB | perplexity +7.7% to +8.3%; code and agent text +3.5% and +2.5% |
|
||||
|
||||
## The versions
|
||||
The download includes the shared n-gram (Engram) table and the DSpark drafter. Pick 2.25 for coding and agent work if
|
||||
you have the disk. Perplexity is measured on a fixed set of 4,995 test texts; lower is closer to the original.
|
||||
|
||||
**Pictures.** The versions in setup's menu read pictures, and pictures are on by default wherever a model can
|
||||
read them. The picture reader runs on the CPU when the graphics card has less than 24 GB (a few seconds per
|
||||
picture, nothing taken from the card) and on the graphics card from 24 GB (about 1-1.4 GB of VRAM; 0.1-0.5 s per
|
||||
picture). The rule is the same on NVIDIA and AMD cards. `--vision off` (or "No" in the app and in `strata launch`) turns
|
||||
pictures off; `--vision cpu` or `--vision gpu` picks the reader. On AMD, `strata-vision-hip` is the card's reader; if it
|
||||
cannot start, Strata uses the CPU reader by itself ([AMD_HIP.md](AMD_HIP.md)). The Mac reads pictures on its GPU
|
||||
([MAC.md](MAC.md)). On an Intel Arc the rule is the same, with two differences: an Arc of 24 GB or more keeps the
|
||||
CPU reader unless you ask for `--vision gpu` (the Intel GPU reader has only run on an 8 GB A750), and a Battlemage card (B580, B70)
|
||||
has pictures off until you pass `--vision yes` ([INTEL.md](INTEL.md#pictures-on-an-arc-2026-10-19-branch-f-021-arcvis)). More: [Images](DETAILS.md#images-vision).
|
||||
**Supported hardware**
|
||||
|
||||
### Qwen3.8-Flash-Next
|
||||
| Hardware | Systems |
|
||||
| --- | --- |
|
||||
| NVIDIA RTX 30, 40 and 50 series, 12 GB or more | Windows, Linux |
|
||||
| AMD Radeon AI PRO R9700 | Linux |
|
||||
|
||||
The original, in all four sizes. With images, and with the
|
||||
[experimental speed projection](DETAILS.md#experimental-speed-projection-experimental-off-by-default) as an option.
|
||||
Tested on an RTX 3060 (Linux), an RTX 5070 (Windows) and four R9700 cards (Linux). The RTX 30 kernels were also checked
|
||||
on an RTX A4000. Windows on RTX 30 and the RX 9070 are built but were not run. Mac and Strix Halo do not run it yet.
|
||||
|
||||
**What you need:** a graphics card with 12 GB or more, 46 GB of RAM at least (64 GB is the tested comfortable size),
|
||||
an NVMe SSD, and the disk space above. Setup warns when disk or RAM is short and does not stop you.
|
||||
|
||||
```sh
|
||||
strata launch --family deepseek41 --model ist-das-2.00bpw --plan
|
||||
strata launch --family deepseek41 --model ist-das-2.00bpw
|
||||
```
|
||||
|
||||
`--model-path DIR` starts from files you already have. The context is 8192 tokens (8K), the size it was checked at.
|
||||
|
||||
**Speed: Faster or Full quality.** The app asks, and the Terminal takes `--speed faster|full` (or `--full-quality`). Faster uses 5 of the
|
||||
model's 6 experts per word, a small quality change that is usually not noticeable. It is the default on a PC with 16 GB of graphics
|
||||
memory or less. Full quality uses all 6. It needs DeepSeek engine 0.1.50 or newer, which the app and `strata launch` fetch.
|
||||
|
||||
**Measured on:** IST-DAS 2.00 bpw, Strata 0.2.4 (before the Speed choice).
|
||||
|
||||
| Hardware | Output | Prompt (8K) |
|
||||
| --- | ---: | ---: |
|
||||
| RTX 5070 12 GB, Ryzen 5 7600, 64 GB RAM | about 5 tokens/s (code) | 231 tokens/s |
|
||||
| RTX 3060 12 GB, 46 GB RAM | 6 to 9 tokens/s | about 103 tokens/s |
|
||||
| 4x Radeon AI PRO R9700 | about 31 tokens/s; 55 to 69 with DSpark | about 360 tokens/s (557 at 32K) |
|
||||
|
||||
- The 2.25 bpw set reads 13% more data per token and is a little slower.
|
||||
- A slow SSD lowers every number, because 12 GB cards spend most of each token waiting on the SSD and the CPU.
|
||||
- DSpark (a small draft model) raised the RTX 3060 PC from 6.1-6.7 to 8.2-8.7 tokens/s and the R9700 box from about 31 to 55-69. It gave no gain on the RTX 5070 PC with a 6-core CPU, so `--dspark auto` leaves the drafter unloaded there.
|
||||
- Answers can differ between runs with the same prompt, because the experts on the card change from moment to moment.
|
||||
|
||||
<a id="the-versions"></a><a id="the-sizes"></a>
|
||||
|
||||
## Qwen3.8-Flash-Next
|
||||
|
||||
Four ISTA-DASLab sizes are the original model; the versions below are variants or other providers' packs of it.
|
||||
Every version in setup's menu reads pictures. [Picture rules](#pictures).
|
||||
|
||||
### Will it fit?
|
||||
|
||||
Shard 1 is loaded at start: its experts go into **RAM**, the rest onto the graphics card. The second shard (a 29 GB
|
||||
lookup table) stays on the SSD. The model fits when **RAM is at least the experts plus about 10 GB** for the OS and
|
||||
other programs.
|
||||
|
||||
| Size | Experts in shard 1 |
|
||||
| --- | ---: |
|
||||
| Q2_0 | 34 GB |
|
||||
| IQ2_XS | 35.5 GB |
|
||||
| IQ3_XXS | 43 GB |
|
||||
| IQ3_S | 50 GB |
|
||||
| Coder | 23 GB |
|
||||
|
||||
With 64 GB of RAM every size fits (IQ3_S with little else open). With 48 GB, Q2_0 and IQ2_XS fit. A bigger graphics
|
||||
card makes it faster and does not lower the RAM needed, except in low-RAM mode.
|
||||
|
||||
<a id="a-big-graphics-card-and-little-ram"></a>
|
||||
|
||||
**Big graphics card, little RAM.** When RAM cannot hold the experts beside the system, setup maps them from the model
|
||||
files and keeps in RAM only what the card does not hold. A 32 GB PC with a 24 GB GPU runs Q2_0, IQ2_XS and the Coder
|
||||
this way, and a 32 GB PC with a 12 to 16 GB GPU runs the Coder. With a small card most experts come from the SSD and it
|
||||
is much slower; setup says so. A source checkout can force the mode with `--low-ram on|off`
|
||||
([SOURCE_INSTALL.md](SOURCE_INSTALL.md)). `strata launch` has no `--low-ram` flag.
|
||||
|
||||
The download is 66 to 76 GB for the three smaller sizes ([details](DETAILS.md#which-model)). The first start also
|
||||
fetches the MTP draft layer (about 6 GB, 1 GB more with pictures).
|
||||
|
||||
### Pictures
|
||||
|
||||
Pictures are on by default wherever a model can read them. The reader runs on the CPU when the card has less than
|
||||
24 GB (a few seconds per picture, no VRAM used) and on the card from 24 GB (about 1 to 1.4 GB of VRAM, 0.1 to 0.5 s per
|
||||
picture). The rule is the same on NVIDIA and AMD.
|
||||
|
||||
| Setting | Effect |
|
||||
| --- | --- |
|
||||
| `--vision off` | no pictures (also "No" in the app and in `strata launch`) |
|
||||
| `--vision cpu` / `--vision gpu` | choose the reader |
|
||||
|
||||
- **AMD:** `strata-vision-hip` is the card's reader. If it cannot start, Strata uses the CPU reader ([AMD_HIP.md](AMD_HIP.md)).
|
||||
- **Mac:** pictures run on the GPU ([MAC.md](MAC.md)).
|
||||
- **Intel Arc:** an Arc with 24 GB or more keeps the CPU reader unless you pass `--vision gpu`. A Battlemage card (B580, B70) has pictures off until you pass `--vision on` ([INTEL.md](INTEL.md)).
|
||||
|
||||
More: [Images](DETAILS.md#images-vision).
|
||||
|
||||
### Coder
|
||||
|
||||
**[Coder](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF)** - ISTA-DASLab's coding version:
|
||||
it keeps 256 of each layer's 512 experts, chosen on code (with agentic and image) data, and drops the rest (91% of
|
||||
the full model's SWE-bench Verified score, 99% of LiveCodeBench, by its authors). One size (IQ1_M: its experts stored
|
||||
like IQ3_S): shard 1 is **29.6 GB**, so it fits a PC with **32 GB of RAM**, runs 262K context on 64 GB, and reads long
|
||||
prompts the fastest of all. With half the experts gone it is weaker outside code and in languages other than
|
||||
English, Chinese and other CJK text included (#438: Chinese answers came out wrong or looping where English was
|
||||
fine). For general use, or whenever your RAM allows, take a size that keeps every expert: Q2_0, IQ2_XS or IQ3_S.
|
||||
More: [details](DETAILS.md#or-the-coder-half-the-experts-for-code).
|
||||
**[Coder](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF)** is ISTA-DASLab's coding version.
|
||||
It keeps 256 of each layer's 512 experts, chosen on code (with agentic and image data), and drops the rest. Its authors
|
||||
report 91% of the full model's SWE-bench Verified score and 99% of LiveCodeBench. There is one size, IQ1_M (experts
|
||||
stored like IQ3_S); shard 1 is **29.6 GB**. A 32 GB RAM computer also needs enough GPU memory and room for buffers;
|
||||
read the launch plan.
|
||||
|
||||
```
|
||||
START-HERE.bat --setup --family coder
|
||||
It is weaker outside code and in languages other than English, Chinese and other CJK text included. For general use,
|
||||
or whenever RAM allows, take a size that keeps every expert. More: [details](DETAILS.md#or-the-coder-half-the-experts-for-code).
|
||||
|
||||
```sh
|
||||
strata launch --family coder --model IQ1_M
|
||||
```
|
||||
|
||||
### Swift 1.5
|
||||
|
||||
**[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** - a fine-tune by UkisAI that
|
||||
thinks much shorter before answering, so you get the answer sooner, with about the same quality. Same speed per
|
||||
token, and about the same RAM as the same size of the original. Its own license applies (see its page).
|
||||
More: [details](DETAILS.md#or-swift-15-a-fine-tune-that-thinks-shorter).
|
||||
**[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** is a fine-tune by UkisAI that
|
||||
thinks much shorter before answering, with about the same quality. Speed per token and RAM match the same size of the
|
||||
original. Its own license applies. More: [details](DETAILS.md#or-swift-15-a-fine-tune-that-thinks-shorter).
|
||||
|
||||
```
|
||||
START-HERE.bat --setup --family swift --model IQ2_XS
|
||||
```sh
|
||||
strata launch --family swift --model IQ2_XS
|
||||
```
|
||||
|
||||
<a id="unsloth-ud-iq4_xs"></a><a id="unsloth-ud-q4_k_xl-experimental"></a>
|
||||
|
||||
### Unsloth's files: UD-IQ4_XS to Q8_0
|
||||
|
||||
The fourth version in setup's menu (`--family unsloth`) is **Unsloth's quantizations** of the same model, from ~4 to 8
|
||||
bits. They keep the original's experts at more bits than the 2-3-bit sizes above, so they need more RAM. Setup keeps
|
||||
every expert in RAM where they and 16 GB more fit (the fastest way); with less RAM it keeps as many as fit and reads
|
||||
the rest from the SSD while it answers (much slower), and for the bigger files it asks first. Pictures are on by default
|
||||
(the CPU reader under 24 GB of graphics memory, the card's reader from 24 GB).
|
||||
`--family unsloth` is Unsloth's 4 to 8 bit quantizations of the same model. They keep the experts at more bits than
|
||||
the sizes above, so they need more RAM. Setup keeps every expert in RAM when they fit with 16 GB to spare. With less RAM
|
||||
it keeps as many as fit and reads the rest from the SSD (much slower), and for the bigger files it asks first.
|
||||
|
||||
| Size | Download | Experts | RAM for all of them in RAM | Same top token as Q8_0 | AMD R9700 (32 GB): story / code / 32K prompt |
|
||||
| --- | ---: | ---: | ---: | ---: | --- |
|
||||
| **UD-IQ4_XS** (setup's first) | 94 GB | 55 GiB | ~72 GB | 88.9% | - |
|
||||
| **UD-Q4_K_XL** | 111 GB | 72 GiB | ~88 GB | 93.1% | 63 / 87 tok/s / 1,110-1,320 tok/s |
|
||||
| **UD-Q5_K_XL** | 158 GB | 92 GiB | ~108 GB | 94.6% | 57-60 / 82 / 810-1,260 |
|
||||
| **UD-Q6_K_XL** | 169 GB | 102 GiB | ~118 GB | 94.6% | 54 / 76 / 1,020 |
|
||||
| **Q8_0** | 188 GB | 120 GiB | ~136 GB | 100% (the reference) | 53-57 / 66-74 / 1,230-1,300 |
|
||||
| Size | Download | Experts | RAM for all experts | Same top token as Q8_0 |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| **UD-IQ4_XS** (setup's first) | 94 GB | 55 GiB | ~72 GB | 88.9% |
|
||||
| **UD-Q4_K_XL** | 111 GB | 72 GiB | ~88 GB | 93.1% |
|
||||
| **UD-Q5_K_XL** | 158 GB | 92 GiB | ~108 GB | 94.6% |
|
||||
| **UD-Q6_K_XL** | 169 GB | 102 GiB | ~118 GB | 94.6% |
|
||||
| **Q8_0** | 188 GB | 120 GiB | ~136 GB | 100% (the reference) |
|
||||
|
||||
For comparison, IQ3_S on the same card: 75 / 95 tok/s, 87.7% the same top token as Q8_0. The speed figures come from
|
||||
engines older than 0.1.45.3 (the conditions are in [QUANTS.md](QUANTS.md)). The quality numbers, the
|
||||
speed bound and NVIDIA/AMD/Intel details: [QUANTS.md](QUANTS.md); the files: [UD-Q4_K_XL](UNSLOTH_Q4.md),
|
||||
[UD-Q6_K_XL](UNSLOTH_Q6.md).
|
||||
For comparison, IQ3_S matches the top token of Q8_0 87.7% of the time. Quality numbers, the speed bound and
|
||||
NVIDIA, AMD and Intel details are in [QUANTS.md](QUANTS.md); the files are described in [UD-Q4_K_XL](UNSLOTH_Q4.md) and
|
||||
[UD-Q6_K_XL](UNSLOTH_Q6.md). Older AMD speeds are in [History](#history--older-measurements).
|
||||
|
||||
```
|
||||
START-HERE.bat --setup --family unsloth --model UD-Q5_K_XL
|
||||
```sh
|
||||
strata launch --family unsloth --model UD-Q5_K_XL
|
||||
```
|
||||
|
||||
<a id="gyro"></a>
|
||||
|
||||
### Gyro-S and Gyro-M (agentionai)
|
||||
|
||||
The fifth version in setup's menu (`--family gyro`) is **agentionai's Gyro quantizations** of the same model: trellis-coded
|
||||
experts in a rotated basis. They are the smallest options and never setup's default. The download includes the
|
||||
28 GB n-gram table; the memory figure is VRAM and RAM together.
|
||||
`--family gyro` is agentionai's Gyro quantization: trellis-coded experts in a rotated basis. They are the smallest
|
||||
options and never setup's default. The download includes the 28 GB n-gram table. The memory figure is VRAM and RAM together.
|
||||
|
||||
| Size | Download | Memory it needs (VRAM + RAM) | In plain words |
|
||||
| Size | Download | Memory (VRAM + RAM) | In plain words |
|
||||
| --- | ---: | ---: | --- |
|
||||
| **Gyro-S** | 58.5 GB | 27.6 GB | the smallest option, for PCs with little memory; lower quality |
|
||||
| **Gyro-M** | 92.0 GB | 35 GB | about IQ3_XXS quality with about 20% less GPU memory; decodes 16-21% slower than Q2_0 (RTX 3060 12 GB, R9700) |
|
||||
| **Gyro-M** | 92.0 GB | 35 GB | about IQ3_XXS quality with about 20% less GPU memory |
|
||||
|
||||
Gyro runs on NVIDIA and AMD (experimental) cards and on Intel Arc cards on Linux (tested on an Arc A750; the Arc engine is built from this release's source, see [INTEL.md](INTEL.md)) and, from 0.2.2, on Apple Silicon Macs ([MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts): Gyro-S is measured on a 32 GB Mac mini, Gyro-M only tensor by tensor). Gyro reads pictures (the same rule as above). On NVIDIA and AMD it needs
|
||||
engine 0.1.45.1 or newer; if the engine is older, the app says "This model needs a newer engine" and offers Update
|
||||
engine. Gyro gives correct output on RTX 50 cards with engine 0.1.45.3. Details, measurements and credit:
|
||||
[GYRO.md](GYRO.md).
|
||||
Gyro runs on NVIDIA and AMD (experimental), on Intel Arc under Linux, and on Apple Silicon Macs. It reads pictures.
|
||||
On an RTX 3060 12 GB and an R9700, Gyro-M decodes 16 to 21% slower than Q2_0. Details, measurements and credit:
|
||||
[GYRO.md](GYRO.md). Intel Arc: [INTEL_ARC.md](INTEL_ARC.md). Mac: [MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts).
|
||||
|
||||
```
|
||||
START-HERE.bat --setup --family gyro --model Gyro-S
|
||||
```sh
|
||||
strata launch --family gyro --model Gyro-S
|
||||
```
|
||||
|
||||
<a id="nvfp4"></a>
|
||||
<a id="nvfp4"></a><a id="nvfp4-qad"></a>
|
||||
|
||||
### NVFP4: three versions (NVIDIA cards)
|
||||
### NVFP4 (NVIDIA cards)
|
||||
|
||||
The sixth version in setup's menu (`--family nvfp4`) keeps the experts in NVIDIA's 4-bit NVFP4 format. There are three
|
||||
versions of it. The app shows them by who made the weights:
|
||||
`--family nvfp4` keeps the experts in NVIDIA's 4-bit NVFP4 format. There are three versions:
|
||||
|
||||
| App name | Setup name (`--model`) | What it is | Download |
|
||||
| App name | `--model` | What it is | Download |
|
||||
| --- | --- | --- | ---: |
|
||||
| **NVFP4 (Strata's version)** | `NVFP4` | Strata's own conversion (GPTQ; the ready-made pack is sergqwer's strata-nvfp4). The best quality of the three | 128 GB |
|
||||
| **NVFP4 (NVIDIA's version)** | `NVFP4-NVIDIA` | NVIDIA's own NVFP4 checkpoint (plain rounding), converted for Strata. The same size and speed as the first | 128 GB |
|
||||
| **NVFP4 (NVIDIA, trained for FP4)** | `NVFP4-QAD` | NVIDIA's retrained 4-bit model, slightly faster on short prompts ([below](#nvfp4-qad)) | 102 GB (+110 GB of disk to prepare) |
|
||||
| **NVFP4 (Strata's version)** | `NVFP4` | Strata's own conversion (GPTQ; the ready pack is sergqwer's strata-nvfp4). Best quality of the three | 128 GB |
|
||||
| **NVFP4 (NVIDIA's version)** | `NVFP4-NVIDIA` | NVIDIA's own checkpoint (plain rounding), converted for Strata. Same size and speed as the first | 128 GB |
|
||||
| **NVFP4 (NVIDIA, trained for FP4)** | `NVFP4-QAD` | NVIDIA's retrained model, slightly faster on short prompts | 102 GB (+110 GB of disk to prepare) |
|
||||
|
||||
All three need about 80 GB of RAM to keep every expert in RAM, run on NVIDIA cards only, and read pictures (the
|
||||
same rule as above). RTX 50 cards have FP4 tensor cores; other NVIDIA cards run NVFP4 at 8-bit precision. The first two
|
||||
need engine 0.1.44 or newer, the third 0.1.45 or newer. Details and measurements: [NVFP4.md](NVFP4.md).
|
||||
All three need about 80 GB of RAM to keep every expert in RAM, run on NVIDIA cards only and read pictures. RTX 50 cards
|
||||
have FP4 tensor cores; other NVIDIA cards run NVFP4 at 8-bit precision. Details and measurements: [NVFP4.md](NVFP4.md).
|
||||
|
||||
```
|
||||
START-HERE.bat --setup --family nvfp4 --model NVFP4
|
||||
```
|
||||
**NVFP4-QAD** is NVIDIA's [Qwen3.8-Flash-Next-FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local),
|
||||
retrained for 4-bit experts (quantization-aware distillation). Setup downloads NVIDIA's files as they are (102 GB) and
|
||||
turns them into a Strata pack on your PC once (20 to 40 minutes, about 110 GB more disk while it runs; the 102 GB can be
|
||||
deleted afterwards). Strata does not redistribute NVIDIA's weights. Pictures use the original model's reader file, which
|
||||
setup downloads with it. Licenses: NVIDIA Open Model Agreement and Qwen's Community License.
|
||||
More: [NVFP4.md](NVFP4.md#nvidias-fp4-local-checkpoint).
|
||||
|
||||
<a id="nvfp4-qad"></a>
|
||||
|
||||
#### NVIDIA's retrained 4-bit model (NVFP4-QAD)
|
||||
|
||||
NVIDIA retrained Qwen3.8-Flash-Next for 4-bit experts (quantization-aware distillation) and published it as
|
||||
[Qwen3.8-Flash-Next-FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local). In setup's NVFP4 family it is
|
||||
the size **NVFP4-QAD**. It is not a ready-made pack: setup downloads NVIDIA's files as they are (102 GB) and **turns them
|
||||
into a Strata pack on your PC** (once, 20-40 minutes; about 110 GB more disk while it runs, and the 102 GB can be deleted
|
||||
afterwards). Nothing of NVIDIA's weights is redistributed by Strata.
|
||||
|
||||
| Size | Download | Memory it needs | In plain words |
|
||||
| --- | ---: | ---: | --- |
|
||||
| **NVFP4-QAD** (app: NVFP4 (NVIDIA, trained for FP4)) | 102 GB (+110 GB of disk to prepare) | like NVFP4: ~80 GB of RAM keeps every expert in RAM | NVIDIA's FP4, trained for FP4 - slightly faster on short prompts than the other two NVFP4 versions (Strata's version has the best quality) |
|
||||
|
||||
NVIDIA cards only. It reads pictures (through the original model's picture reader file, which setup downloads with it). Its licenses are the NVIDIA Open Model Agreement and Qwen's Community License.
|
||||
Technical details and measurements: [NVFP4.md](NVFP4.md#nvidias-fp4-local-checkpoint).
|
||||
|
||||
```
|
||||
START-HERE.bat --setup --family nvfp4 --model NVFP4-QAD
|
||||
```sh
|
||||
strata launch --family nvfp4 --model NVFP4
|
||||
```
|
||||
|
||||
### OrcaRouter Uncensored IQ3_XXS
|
||||
|
||||
For **OrcaRouter's Flash-Next Uncensored IQ3_XXS**, see the [manual compatibility setup](ORCA.md). It needs an
|
||||
explicit packing conversion and is not an installer menu option.
|
||||
For **OrcaRouter's Flash-Next Uncensored IQ3_XXS**, see the [manual compatibility setup](ORCA.md). It needs an explicit
|
||||
packing conversion and is not an installer menu option.
|
||||
|
||||
## Qwen3.6-35B-A3B
|
||||
|
||||
A smaller model from the same maker: [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (35 billion
|
||||
parameters, 3 billion active per token, 256 experts, Apache 2.0 license). Strata runs [Unsloth's GGUFs with the MTP
|
||||
draft layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) on its own engine path, chosen by the file. It is
|
||||
in the app's model list and in setup (`--family qwen36`). It runs on NVIDIA and AMD cards; the Intel Arc engine and the
|
||||
Mac engine do not run it. It reads pictures and serves several requests at once.
|
||||
parameters, 3 billion active per token, 256 experts, Apache 2.0). Strata runs
|
||||
[Unsloth's GGUFs with the MTP draft layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) on its own engine
|
||||
path, chosen by the file. It is in the app's model list and in setup (`--family qwen36`). It runs on NVIDIA and AMD
|
||||
cards; the Intel Arc engine and the Mac engine do not run it. It reads pictures and serves several requests at once.
|
||||
|
||||
| Size | File | RAM setup plans for | In plain words |
|
||||
| --- | ---: | ---: | --- |
|
||||
@@ -254,24 +268,75 @@ Mac engine do not run it. It reads pictures and serves several requests at once.
|
||||
| **Q8_0** | 37.8 GB | 52 GB | 8-bit, the closest to the full model |
|
||||
|
||||
The picture reader file adds 0.9 GB. The engine keeps every expert it can on the graphics card and computes the rest on
|
||||
the CPU. Measured with engine 0.1.45 and UD-Q4_K_XL: decode 117.8 tokens/s on a Radeon AI PRO R9700 (32 GB) and 59.7
|
||||
tokens/s on an RTX 3060 (12 GB, most experts on the CPU). Pictures, requests at once, the full measurements and the
|
||||
limits: [QWEN36.md](QWEN36.md).
|
||||
the CPU.
|
||||
|
||||
```
|
||||
START-HERE.bat --setup --family qwen36 --model UD-Q4_K_XL
|
||||
**Measured on:** UD-Q4_K_XL, decode only.
|
||||
|
||||
| Hardware | Decode |
|
||||
| --- | ---: |
|
||||
| Radeon AI PRO R9700 (32 GB) | 117.8 tokens/s |
|
||||
| RTX 3060 (12 GB), most experts on the CPU | 59.7 tokens/s |
|
||||
|
||||
Pictures, parallel requests, full measurements and limits: [QWEN36.md](QWEN36.md).
|
||||
|
||||
```sh
|
||||
strata launch --family qwen36 --model UD-Q4_K_XL
|
||||
```
|
||||
|
||||
## Adding or switching models
|
||||
|
||||
- **In the app:** the Models page lists every model with its size and whether it fits your computer; pick one and press
|
||||
the install button.
|
||||
- **In the Terminal:** `strata launch` has a Model step with the same list (and "Use model files I already have").
|
||||
`strata models list` shows what is installed; `strata models import <folder or .gguf> --plan` shows what a set of files
|
||||
you already have would be, without changing anything.
|
||||
- **With the installer:** `SETUP.bat` (the same as `START-HERE.bat --setup`; on Linux `./setup.sh --setup`) adds another
|
||||
model any time.
|
||||
- **In the app:** the Models page lists every model with its size and whether it fits your computer. Pick one and press the install button.
|
||||
- **In the Terminal:** `strata launch` has a Model step with the same list and "Use model files I already have".
|
||||
`strata models list` shows what is installed. `strata models import <folder or .gguf> --plan` shows what a set of
|
||||
files you already have would become, without changing anything.
|
||||
- **From a source checkout only:** `SETUP.bat` or `./setup.sh --setup` adds another model ([SOURCE_INSTALL.md](SOURCE_INSTALL.md)).
|
||||
|
||||
Files another model shares are not downloaded again (the Coder uses the original's shard 2 and vision encoder). With more
|
||||
than one model installed, `START-HERE.bat` asks which one to start; `run-<model>.bat` (Linux: `run-<model>.sh`) starts one
|
||||
directly.
|
||||
Shared files are reused. In the Terminal, choose a model in `strata launch`, pass an explicit family and model, or use a
|
||||
saved `--preset`. Add `--restart` when replacing a running launch. Generated `run-<model>` scripts belong to a source
|
||||
checkout, not to the installed app.
|
||||
|
||||
## Speed on your own PC
|
||||
|
||||
`--tune quick` and `--tune thorough` measure settings for the selected model, and the app offers tuning too. Results
|
||||
depend on the backend and machine. All speed measurements, long-context numbers and estimates for other cards are in
|
||||
[DETAILS.md](DETAILS.md#speed-measured). AMD measurements per card are in [AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201).
|
||||
Measured Strata on your own PC? See [Community benchmark results](COMMUNITY_BENCHMARKS.md).
|
||||
|
||||
<a id="how-fast-is-each-size"></a>
|
||||
|
||||
## History / older measurements
|
||||
|
||||
These numbers compare the sizes with each other. Newer engines are faster on some cards, and the tables were not
|
||||
re-measured in full for the current release.
|
||||
|
||||
**Measured on:** RTX 5070 (12 GB), Ryzen 5 7600, 64 GB RAM; ISTA-DASLab sizes; engine 0.1.26 (Q2_0 row: engine 0.1.36).
|
||||
|
||||
| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt (32K) |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| **Q2_0** | 94 tokens/s | 76 tokens/s | 2,650 tokens/s |
|
||||
| **IQ2_XS** | 79 tokens/s | 63 tokens/s | 2,090 tokens/s |
|
||||
| **IQ3_XXS** | 62 tokens/s | 49 tokens/s | 1,750 tokens/s |
|
||||
| **IQ3_S** | 53 tokens/s | 46 tokens/s | 1,620 tokens/s |
|
||||
| **Coder** (IQ1_M) | 55 tokens/s | 43 tokens/s | 2,180 tokens/s |
|
||||
|
||||
**Measured on:** AMD RX 9070 XT (16 GB), Ryzen 9 3900X, 47 GB RAM, Linux; engine before 0.1.40, 2026-10-02. Q2_0 is
|
||||
above setup's RAM estimate for 47 GB and was installed with `--model Q2_0 --yes`.
|
||||
|
||||
| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt (32K) |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| **Q2_0** | 60 tokens/s | 48 tokens/s | 1,160 tokens/s |
|
||||
| **IQ2_XS** | 52 tokens/s | 36 tokens/s | 1,110 tokens/s |
|
||||
| **Coder** (IQ1_M) | 44 tokens/s | 33 tokens/s | 1,420 tokens/s |
|
||||
|
||||
- A token is about three quarters of a word. A 4K prompt reads at 910 to 1,580 tokens/s on the RTX 5070; a 32K prompt takes about 15 seconds with Q2_0.
|
||||
- The older NVIDIA `--calibrate` run on the RTX 5070 PC raised Coder speed by 7%.
|
||||
|
||||
**Measured on:** AMD Radeon AI PRO R9700 (32 GB), Unsloth files, engines before 0.1.45.3 (conditions in [QUANTS.md](QUANTS.md)).
|
||||
|
||||
| Size | Story | Code | 32K prompt |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| **UD-Q4_K_XL** | 63 tokens/s | 87 tokens/s | 1,110 to 1,320 tokens/s |
|
||||
| **UD-Q5_K_XL** | 57 to 60 | 82 | 810 to 1,260 |
|
||||
| **UD-Q6_K_XL** | 54 | 76 | 1,020 |
|
||||
| **Q8_0** | 53 to 57 | 66 to 74 | 1,230 to 1,300 |
|
||||
| *IQ3_S, same card* | 75 | 95 | - |
|
||||
+235
-466
@@ -1,532 +1,301 @@
|
||||
# Strata on two or more GPUs (layer split)
|
||||
|
||||
One model can run across several NVIDIA cards in one PC. The layers are split into contiguous ranges, one per GPU:
|
||||
the first card runs layers 0 to K-1, the next card runs K onward, and so on; the last card also runs the output head
|
||||
and the draft (MTP) layer. Each card keeps an expert cache for **its own layers only**, so two cards hold about twice
|
||||
the experts one card holds - for the Coder model on a 16 GB + 24 GB pair, nearly all of them, which is where the
|
||||
speed comes from (decode then barely touches the CPU pool).
|
||||
This page is for people with two or more graphics cards of the same make in one PC. A layer split puts one model on
|
||||
several cards. NVIDIA is described here; AMD limits and measurements are in [AMD_HIP.md](AMD_HIP.md).
|
||||
|
||||
This is pipeline (layer) parallelism, not tensor parallelism: a token crosses from one card to the next once per
|
||||
verify window (a few hundred KB through pinned RAM), not twice per layer. No NVLink or peer-to-peer access is
|
||||
needed; cards on x4 or x1 slots work, and the PCIe share of each card is probed on its own link. A `--pcie-frac` you give
|
||||
is every card's share and skips those probes; there is no per-card setting yet.
|
||||
|
||||
**Two models on two cards (the app).** A layer split puts one model on several cards. The other way to use two cards is two
|
||||
models, one per card, each answering on its own. In the app, turn on **Run multiple models** on the Models page: the card
|
||||
list then says which card a loaded model uses ("In use by ..."), starts that card as Off for the next model, and
|
||||
explains a failed start on a card that is taken. Measured on a PC with Radeon AI PRO R9700 cards (two of four, Linux): Q2_0 on
|
||||
one card and IQ3_XXS on another both loaded (33 s and 59 s with the files cached, 114 s the first time), both answered chats
|
||||
(about 60 tokens/s each), the Monitor listed both, and unloading one left the other answering. This was run with the AMD build only;
|
||||
the same app code drives NVIDIA cards, but we have not run two models on two NVIDIA cards.
|
||||
|
||||
**One make of card per engine.** An engine drives one make of card (NVIDIA, AMD or Intel). The app, `strata launch` and setup
|
||||
offer only the cards the engine can drive, never split a model across two makes, say in one plain line when the other make has
|
||||
the bigger card, and say that two models at once on two makes is not supported.
|
||||
> **On this page:** [Using it](#using-it) · [Main, Helper, Off](#main-helper-off) · [How the split works](#how-the-split-works) ·
|
||||
> [Tuning options](#tuning-options) · [Pipeline windows](#pipeline-windows) · [Two models on two cards](#two-models-on-two-cards) ·
|
||||
> [DeepSeek](#deepseek-v41-flash) · [Limits](#limits) · [History](#history--older-measurements)
|
||||
|
||||
## Using it
|
||||
|
||||
**Nothing to type.** `START-HERE.bat` (Linux: `./setup.sh`) lists your NVIDIA cards and says for each one whether
|
||||
Strata can use it:
|
||||
**Desktop app:** choose Main, Helper or Off for each card. **Terminal:** run `strata launch` and use the Cards step, or name
|
||||
the cards:
|
||||
|
||||
```sh
|
||||
strata launch --gpu 0,1
|
||||
strata launch --gpu-roles 1=main,0=helper # cards you do not name are Off
|
||||
```
|
||||
Your NVIDIA GPUs:
|
||||
GPU 0: NVIDIA GeForce RTX 5080, 16 GB VRAM - can be used
|
||||
GPU 1: NVIDIA GeForce GTX 1080 Ti, 11 GB VRAM - not supported - older than the RTX 20 series (compute capability 6.1; Strata needs 7.5 or newer)
|
||||
GPU 2: NVIDIA GeForce RTX 3090, 24 GB VRAM - can be used
|
||||
...
|
||||
1) RTX 3090 + RAM: fits: needs 44 GB of the 70 GB available (recommended)
|
||||
2) RTX 3090 + RTX 5080 + RAM (layer split): fits: needs 44 GB of the 84 GB available
|
||||
prompts are read about 20% faster; the cards share the layers (docs/MULTI_GPU.md)
|
||||
3) RTX 5080 + RAM: fits: needs 44 GB of the 66 GB available
|
||||
Which GPUs? [1]:
|
||||
|
||||
Choose a model that fits the combined memory. A split can add capacity without adding speed.
|
||||
|
||||
**Source checkout only:** `START-HERE.bat` and `./setup.sh` have their own card picker. Their flags are not `strata launch` flags:
|
||||
|
||||
| Source setup flag | Meaning |
|
||||
| --- | --- |
|
||||
| `--gpus 0,2` | These cards together, numbered as `nvidia-smi` numbers them. Strata puts the faster card last. Remembered. |
|
||||
| `--gpus all` | Every card that can share the model. |
|
||||
| `--gpu 0` | One card (at a start: for that start only). |
|
||||
| `--layer-split auto` | The default. Or the first layer of each later card, for example `18` or `16,32`. |
|
||||
| `--gpu-roles 1=main,0=helper` | Roles, saved. |
|
||||
|
||||
Or edit an existing run config (`strata-*.json`) and restart. Merge these fields into the config; this is not a complete config:
|
||||
|
||||
```json
|
||||
{
|
||||
"gpu": [0, 2],
|
||||
"layer_split": "auto"
|
||||
}
|
||||
```
|
||||
|
||||
Docker: `GPU_ROLES=1=main,0=helper` is passed as `--gpu-roles` ([DOCKER.md](DOCKER.md)).
|
||||
|
||||
## Main, Helper, Off
|
||||
|
||||
Every card has one of three roles:
|
||||
Every card has one of three roles.
|
||||
|
||||
| Role | What it does |
|
||||
|---|---|
|
||||
| **Main** | Runs the **last** stage of the split: the output head, the draft layer and the check of its guesses. A prompt chunk waits on it. Exactly one card is Main. |
|
||||
| **Helper** | Holds a share of the layers and of the expert cache, so more of the model fits on graphics cards instead of being read from the drive. Prompts are read about 20% faster. |
|
||||
| --- | --- |
|
||||
| **Main** | Runs the last stage of the split: the output head, the draft layer and the check of its guesses. A prompt chunk waits on it. Exactly one card is Main. |
|
||||
| **Helper** | Holds a share of the layers and of the expert cache, so more of the model sits on graphics cards instead of being read from the drive. The effect on speed depends on the cards and the workload. |
|
||||
| **Off** | Strata does not touch the card (a game, the screen, a card that is too small). |
|
||||
|
||||
The Main card is the **last** card in the engine's order, not the first. The first card runs the first layers and, by default,
|
||||
the picture reader. (Older versions of this page said "the first is the main one"; that has not been true since 0.1.41.1.)
|
||||
The Main card is the last card in the engine's order. The first card runs the first layers and, by default, the picture reader.
|
||||
|
||||
**Recommended roles** (the same rule in the app, `strata launch` and setup; `strata_control/gpu_roles.py`): the faster card
|
||||
is Main (speed is multiprocessors x max clock, the engine's own measure; cards within 5% are the same card, and the one with
|
||||
more memory wins, then the lower number; two identical cards: the highest number is Main, so the engine order stays 0, 1 as it always was; a card with clearly more memory beats a faster one that is at least half as fast).
|
||||
Every other card starts as Helper, except a card with **8 GB or less** when a bigger card is there (Off; with only 8 GB cards, say two GTX 1080, the second one is a Helper), built-in graphics beside a card with memory of
|
||||
its own (Off) and a card the engine cannot run (Off). Every role is one click (the app), one key (`strata launch`: M, H, O;
|
||||
R goes back to the recommendation) or one line (below).
|
||||
**Recommended roles** (the same rule in the app, `strata launch` and setup; `strata_control/gpu_roles.py`):
|
||||
|
||||
Only a few things are warned about, in one short line each: built-in graphics as Main, a card the engine cannot run, too
|
||||
little memory for the chosen setup (part of the model would be read from the drive). **Every card Off** is allowed. The CUDA,
|
||||
HIP and SYCL engines cannot start without a card, so the server then says so in one line and stops; an engine that can run on
|
||||
the processor (the Mac engine) says "Performance is degraded on CPU only." and goes on.
|
||||
- The faster card is Main. Speed is multiprocessors x max clock. Cards within 5% count as the same card, and the one with more memory wins, then the lower number. Of two identical cards the highest number is Main. A card with clearly more memory beats a faster one that is at least half as fast.
|
||||
- Every other card starts as Helper, except a card with 8 GB or less when a bigger card is present (Off; with only 8 GB cards the second one is a Helper), built-in graphics beside a card with memory of its own (Off), and a card the engine cannot run (Off).
|
||||
- A role is one click (app), one key (`strata launch`: M, H, O; R restores the recommendation) or one line (above).
|
||||
|
||||
**Setting roles**
|
||||
Only a few things are warned about, one short line each: built-in graphics as Main, a card the engine cannot run, and too
|
||||
little memory for the chosen setup. All cards Off is allowed. The CUDA, HIP and SYCL engines cannot start without a card and
|
||||
say so; the Mac engine says "Performance is degraded on CPU only." and goes on.
|
||||
|
||||
```
|
||||
strata launch --gpu-roles 1=main,0=helper cards you do not name are Off
|
||||
./setup.sh --gpu-roles 1=main,0=helper (START-HERE.bat --gpu-roles ... on Windows); saved
|
||||
strata launch the Cards step: M Main, H Helper, O Off, R recommended, D details
|
||||
```
|
||||
|
||||
or in the run settings (`strata-*.json`), which is what the app writes:
|
||||
**In the run config** (what the app writes). Merge these fields into the existing config:
|
||||
|
||||
```json
|
||||
"gpu_roles": {"1": "main", "0": "helper", "2": "off"},
|
||||
"gpu": [0, 1],
|
||||
"gpu_order": "as_given"
|
||||
{
|
||||
"gpu_roles": {"1": "main", "0": "helper", "2": "off"},
|
||||
"gpu": [0, 1],
|
||||
"gpu_order": "as_given"
|
||||
}
|
||||
```
|
||||
|
||||
`gpu` is the same choice for older engines and the Monitor: the Helpers (slowest first), then the Main card, with the Off
|
||||
cards left out. The server reads `gpu_roles`; when `gpu` was edited by hand and no longer matches, `gpu` wins and the log says
|
||||
so (a settings file that names its cards keeps them). Nothing is rewritten on upgrade: a file with only `"gpu": [1, 0]` shows
|
||||
as Main = the last card, Helper = the first, the rest Off. The recommendation itself is not saved, so a card you add later is
|
||||
recommended again; a card you add after choosing roles starts Off. Docker: `GPU_ROLES=1=main,0=helper` is passed as `--gpu-roles`.
|
||||
`gpu` is the same choice for older engines and the Monitor: the Helpers (slowest first), then the Main card, Off cards left out.
|
||||
The server reads `gpu_roles`. If `gpu` was edited by hand and no longer matches, `gpu` wins and the log says so. A file with only
|
||||
`"gpu": [1, 0]` shows as Main = the last card, Helper = the first, the rest Off. The recommendation is not saved, so a card you
|
||||
add later is recommended again; a card you add after choosing roles starts Off.
|
||||
|
||||
**One rule for which cards count** (`strata_control/gpu_fit.py` keeps the memory arithmetic): not an integrated chip beside
|
||||
a real card, at least 4 GB of graphics memory to be listed, 8 GB or less starts Off beside a bigger card. The arithmetic is the app's `fitOn`;
|
||||
each extra card of a split costs an estimated 2 GB of its own buffers.
|
||||
**Which cards count** (`strata_control/gpu_fit.py` holds the memory arithmetic): not an integrated chip beside a real card; at
|
||||
least 4 GB of graphics memory to be listed; 8 GB or less starts Off beside a bigger card. Each extra card of a split costs an
|
||||
estimated 2 GB of its own buffers.
|
||||
|
||||
**Choosing the cards without roles** (at setup or at any start):
|
||||
**Not supported** (setup says so and names the cards that can be used):
|
||||
|
||||
```
|
||||
--gpus 0,2 these cards together, as nvidia-smi numbers them; Strata puts the faster card last. Remembered.
|
||||
--gpus all every card that can share the model
|
||||
--gpu 0 one card (at a start: for that start only)
|
||||
--layer-split auto (default) or the first layer of each later card, e.g. 18 or 16,32
|
||||
```
|
||||
- A card older than the RTX 20 series (compute capability below 7.5). Pascal and Volta cards are admitted when you name them
|
||||
with `--gpus` (or `--gpu`, `--cuda 12`) and then run the experimental CUDA 12 engine ([OLDER_GPUS.md](OLDER_GPUS.md)).
|
||||
- A card with less than 8 GB of VRAM together with others, because each card holds a copy of the dense weights and its own
|
||||
prompt buffers. If you name it with `--gpus`, setup states the risk and asks (`--yes` goes ahead).
|
||||
- Intel GPUs in a split, and a mix of NVIDIA and AMD cards. An engine drives one make of card. The app, `strata launch` and setup
|
||||
offer only the cards the engine can drive, and say so when the other make has the bigger card. AMD cards share a model among
|
||||
themselves: `./setup.sh --backend hip --gpus 1,0` ([AMD_HIP.md](AMD_HIP.md)).
|
||||
|
||||
**Not supported** (setup says so and names the cards that can be used instead):
|
||||
- a card older than the RTX 20 series (compute capability below 7.5: GTX 10 and older), unless you name it: Pascal
|
||||
and Volta cards (Tesla P100 / P40, GTX 10, V100) are admitted when you choose them with `--gpus` (or `--gpu`,
|
||||
`--cuda 12`) and then run the experimental CUDA 12 engine, see [OLDER_GPUS.md](OLDER_GPUS.md). Two Tesla P40s
|
||||
(`"gpu": [0, 1]` in the config) ran the layer split in a community benchmark: IQ2_XS decode 19.6 tok/s on one card,
|
||||
34.8 on both (#1028, experimental, one report);
|
||||
- a card with less than 8 GB of VRAM, together with others (each card holds a copy of the dense weights and its
|
||||
own prompt buffers) - unless you name it with `--gpus`: then setup says the risk and asks (`--yes` with the named
|
||||
cards goes ahead);
|
||||
- Intel GPUs, and a mix of NVIDIA and AMD cards. (AMD cards share a model among themselves: `./setup.sh --backend
|
||||
hip --gpus 1,0`, see [AMD_HIP.md](AMD_HIP.md).)
|
||||
## How the split works
|
||||
|
||||
Or edit an existing config (`strata-*.json`), then restart:
|
||||
The layers are cut into contiguous ranges, one per card. The first card runs layers 0 to K-1, the next runs K onward, and the
|
||||
last card also runs the output head and the draft (MTP) layer. Each card keeps an expert cache for its own layers only, so two
|
||||
cards hold about twice the experts one card holds.
|
||||
|
||||
```json
|
||||
"gpu": [0, 2],
|
||||
"layer_split": "auto"
|
||||
```
|
||||
This is pipeline parallelism, not tensor parallelism. A token crosses from one card to the next once per verify window (a few
|
||||
hundred KB through pinned RAM). No NVLink or peer-to-peer access is needed, cards on x4 or x1 slots work, and each card's PCIe
|
||||
share is probed on its own link. A `--pcie-frac` you give applies to every card and skips those probes.
|
||||
|
||||
`"layer_split"` is `"auto"` (placed by each card's free VRAM) or the **first layer of each later card**: one rising
|
||||
number per card after the first, not a count of layers per card. With 4 cards and a 48-layer model, `"24,36,42"`
|
||||
(or `[24, 36, 42]`) puts layers 0-23 on the first card, 24-35 on the second, 36-41 on the third and 42-47 on the last.
|
||||
The server checks it before the start and says what is wrong (0.1.39, #644).
|
||||
**What each card holds**
|
||||
|
||||
**Card order with `"auto"`** (NVIDIA, #1352): the faster card (multiprocessors x max clock) goes **last** - the last stage runs the head, the draft layer and the verify, and a prompt chunk waits on it (a 4070 Ti SUPER + 5060 Ti read a 6K prompt in 50 s one way round and 15 s the other). Cards whose scores are within 5% of each other (two of the same model) keep your order; a manual split keeps it too. `"gpu_order": "as_given"` keeps the order you wrote under `"auto"` as well. The engine log line `layer split: card order ...` says when it changed.
|
||||
- Every card: a copy of the dense weights, its own session state (the KV cache of the full context), its verify window, its
|
||||
prompt-path buffers, and an expert cache for its layers filled from the expert profile.
|
||||
- The last card: also the output head and the draft layer (about 0.8 GB).
|
||||
- Host RAM: the expert arena once, shared by all cards. The CPU pool computes whatever no card holds.
|
||||
|
||||
**An unequal pair under `"auto"` (#2644).** The search prices a decode window, and it can give almost every layer to the card it
|
||||
rates faster. A card with one or two layers holds a handful of experts and clamps the prompt chunk of the whole pipeline
|
||||
(#1616), so the search then asks for at least 3 layers per card - which on a 4060 Ti 16 GB + 3060 12 GB (48 layers, IQ3_S)
|
||||
was K=3: the 3060 ran 3 layers, 12 GB idle, 452 tok/s of prefill against 802 at 28 layers on the 3060 (the reporter's sweep of
|
||||
`"layer_split"` 12, 18, 24, 28, 32: 582, 693, 762, 802, 803 tok/s; the 3060's 360 GB/s of memory bandwidth against the 4060 Ti's
|
||||
288 counts for more than its SMs). On two cards the engine now first asks for a placement in which each card owns at least its
|
||||
share of the layers by its room for experts, and the log line says which rule chose the split. Measured on 2x RTX A4000 made
|
||||
unequal (stage 0 priced at 0.57 of the speed with 3.7 GiB less room, `STRATA_SPLIT_SPEED_SCALE=0.57` and `--vram-reserve-mib 4400`; 14.5K-token
|
||||
prompt, IQ3_XXS, 5 interleaved pairs): the old rule chose K=3 (3 layers on stage 0), the new one K=20; prompt reading 945 -> 1,441 tok/s (+52%,
|
||||
5/5 pairs), decode 50.2 -> 65.9 tok/s (+31%, 5/5). Equal cards and three or more GPUs are unchanged.
|
||||
On x16 + x4 rigs the link-miss scaling (#2007, from a 4x gap) can also move `auto` off the balanced placement (2x RTX 3090, x16 + x4:
|
||||
K=31 with miss cost x4.20, 114-119 tok/s decode against 130-136 at K=27, #2418): `STRATA_SPLIT_UNIFORM=1` (opt-in, with `"gpu_order": "as_given"`)
|
||||
prices every link the same, as before 0.1.41.
|
||||
Prefill that is flat in the prompt length (no gain from 4K to 16K) is the sign of an idle card; **on an unequal pair sweep
|
||||
`"layer_split"` once** - a 4K-only benchmark cannot see it.
|
||||
Prompts are read in chunks that flow through the cards in turn: while a later card reads chunk c, the first card reads chunk c+1.
|
||||
Conversation checkpoints save and restore every card's state.
|
||||
|
||||
**Skip the split when the first card holds everything** (opt-in, 0.1.31): `"split_skip_if_fits": true` in the config
|
||||
(engine flag `--split-skip-if-fits`, with `--layer-split auto`) runs on the first card alone when it holds every
|
||||
profiled expert plus the context's KV, the draft layer and the reserve, and says so in the log; otherwise the split
|
||||
stays. On an R9700 32 GB + RX 9070 XT the R9700 holds all of the Coder's experts: with the flag 4K prompts read at
|
||||
1,776 tok/s instead of 1,244 (split) and decode runs at ~60 tok/s instead of ~51 (16K prompts ~5% slower than split).
|
||||
**`"layer_split"`** is `"auto"` (placed by each card's free VRAM) or the first layer of each later card: one rising number per
|
||||
card after the first, not a count per card. With four cards and a 48-layer model, `"24,36,42"` puts layers 0-23 on the first
|
||||
card, 24-35 on the second, 36-41 on the third and 42-47 on the last. The server checks the value before the start and says what
|
||||
is wrong.
|
||||
|
||||
**Short prompts on a split (0.1.32, #340).** 0.1.30 gave each card's prompt path a loan from its own expert cache,
|
||||
refilled after every request; on cards that hold nearly all their experts that cost short prompts up to a third of
|
||||
their speed. 0.1.32 refills all cards at once, uses a smaller streaming ring on a split, and lets a card with free
|
||||
VRAM keep its own prompt buffers - the same output as 0.1.31, measured on an R9700 + RX 9070 XT: 2K prompts 993 ->
|
||||
1,265 tok/s, 16K 1,852 -> 1,950, decode unchanged. `STRATA_SPLIT_OWN=1` (opt-in) gives every card its own buffers:
|
||||
2K 1,450 and 16K 2,227 tok/s there, but a full card then keeps a different set of experts resident, so the output
|
||||
differs from the default's (stable and coherent); `STRATA_SPLIT_OWN=auto` does that only where the buffers are at
|
||||
most 12% of each card's VRAM.
|
||||
|
||||
**The idle card can help one-chunk prompts (opt-in, `STRATA_PREFILL_HELP=1`).** A prompt that fits one chunk runs the stages one after the other, so while
|
||||
one card reads its layers the other idles. With it on, each stage hands a share of its streamed experts to the idle card: it
|
||||
streams them over its own PCIe link into its own (lent) prompt buffers, computes their rows on the MMQ path and sends
|
||||
them back - `--peer-device`'s peer streaming, without P2P (the activations and the rows go through mapped host memory,
|
||||
read by copy kernels, so they do not queue behind the expert blobs on either card's copy engine). The share falls with
|
||||
the prompt (0.41 of the streamed experts at 1.5K tokens, 0.32 at 3K) and is off from ~3.3K tokens, where the stages
|
||||
overlap anyway and no share paid. Measured on 2x RTX 3090 (UD-Q4_K_XL, no P2P), prompt tok/s without / with: 1.1K 496 /
|
||||
723, 1.5K 674 / 833, 2K 878 / 1,128, 2.5K 1,017 / 1,296, 3K 1,260 / 1,440; 4K and 8K unchanged, decode unchanged. It
|
||||
costs no VRAM (the idle card's own prompt buffers) and ~110 KB of mapped host memory per token of the largest chunk it
|
||||
helped (~360 MB at 3.3K tokens). The rows it computes round like a different MMQ grouping, so the output is not
|
||||
bit-identical to the default's (it is repeatable: same prompt, same output), which is why it is **opt-in**:
|
||||
`STRATA_PREFILL_HELP=1` turns it on. Native packs on the MMQ prompt path only (not with the fused prompt kernels,
|
||||
`STRATA_PF_FUSED=1`, nor with `--peer-device`). With it on, `STRATA_PREFILL_HELP_FRAC=f` fixes the share.
|
||||
On two Radeon AI PRO R9700 (each limited to 16 GB with `--vram-reserve-mib 17084`, Q2_0 pack, layer split auto, 5 interleaved pairs,
|
||||
noise arm within 0.2%), `STRATA_PREFILL_HELP=1` gave prompt tok/s without -> with: 700 tokens 638 -> 672 (x1.053, 5/5), 1.5K 839 -> 809
|
||||
(x0.964), 3K 1,105 -> 929 (x0.840), 6K 1,284 -> 1,211 (x0.943): the PCIe 4.0 x16 links of this box (28.4 GB/s each) make the share lose
|
||||
from 1.5K tokens up, so it stays off (`STRATA_PREFILL_HELP_FRAC` was not tuned for it).
|
||||
|
||||
The engine flags behind it: `--layer-split K1[,K2..]|auto` and `--split-device D1[,D2..]` (the later stages'
|
||||
devices; default the next visible ones). `--layer-split K --split-device 0` runs both stages on one card sharing
|
||||
everything - the bit-exact check of the hand-off, not a speed mode.
|
||||
|
||||
**The draft layer's prompt K/V in batches on a split (opt-in, `STRATA_SPLIT_MTP_BATCH=1`).** With `--mtp` the draft
|
||||
layer sits on the last card, and each prompt chunk ends by filling that layer's K/V for the chunk's rows. On one card the
|
||||
prompt path does it in batches (`Prefill::draft_kv`, a few large matrix products per chunk); on a split the drafter's own
|
||||
pass always did it, one graph launch and four device copies per group of 8 rows (1,024 groups per 8,192-token chunk).
|
||||
With the variable set, the last card's prompt path batches it as it does on one card. Restart-only; default off, so the
|
||||
default start is unchanged. It needs a `--native` pack (the GGUF-form token table) and a paged or ring (`--kv-resident`)
|
||||
draft K/V (`STRATA_MTP_BATCH_RING=0` sends a ring back to the old pass); otherwise it says why once and keeps the old pass:
|
||||
`strata serve: draft layer prompt K/V batched on CUDA<d> ...` or `strata serve: STRATA_SPLIT_MTP_BATCH=1 declined, the
|
||||
drafter's own pass runs: <reason>`. The draft layer's K/V come out of Q8_1 x Q8_0 MMQ instead of its own mmvq, so the
|
||||
drafts, and how many are accepted, can move; the target's tokens are decided by the verify window.
|
||||
`tools/split_mtp_batch_parity.py` compares the greedy texts of the two arms. Measured on one machine (2x RTX 3080 20 GB
|
||||
at 220 W, Xeon E5-2696 v4, UD-Q4_K_XL, `--layer-split 23`, `--batch-mtp`, two slots, `--prefill auto:16384`), two full
|
||||
restarts per arm in the order off, on, off, on, the same binary in every arm: pooled medians of the prompt read were 1,996
|
||||
tok/s without and 2,129 with at 25K tokens (+6.7%), 2,448 and 2,553 at 51K (+4.3%), 2,846 and 2,940 at 104K (+3.3%); all 8
|
||||
on/off ratios of medians are above 1 (1.03 to 1.08), and the two off restarts differ by at most 2% at 25K and 0.3% at the
|
||||
other sizes. The host's "after each chunk" time per request fell from 0.9-1.3 s to 0.1-0.25 s (one restart per arm). Greedy
|
||||
output (`--adapt-every 0 --suffix-draft 0`, five prompts of 256 tokens and one 25.5K-token prompt) was identical in 6 of 6
|
||||
prompts in all five comparisons, the two runs inside one process (A/A) included, and the drafts accepted were 826 of 1,210
|
||||
in every run, so no difference in acceptance was visible. Decode is not shown to be unchanged: the medians with the variable
|
||||
set were about 3% lower (pooled solo 82.6 -> 80.2 tok/s, two streams 93.2 -> 90.8), and the same configuration restarted
|
||||
twice moved up to 4.7%, so two restarts per arm can neither confirm nor exclude it; the change touches only the prompt path.
|
||||
|
||||
**Each card loads only its own layers' dense weights** (0.1.39, PR #639) with explicit split points (`--layer-split
|
||||
27`, not `auto`): every card used to keep a full copy (~3.4 GB for the Coder) though its stage reads only its own
|
||||
layers, and the VRAM it frees goes to that card's expert cache (2x MI50 16 GB, Coder: 8,819 -> 10,626 experts in
|
||||
VRAM, decode 39.2 -> 41.7 tok/s). Opt-in for now, on AMD and NVIDIA alike: `STRATA_STAGE_TRIM=1` (please report how
|
||||
it goes).
|
||||
A card holding more experts can change which experts run on the GPU, so the output can differ slightly from a run
|
||||
without it.
|
||||
|
||||
**The resident RAM mode works on a split** (`--resident-experts`, the low-RAM mode). The RAM copy of the experts
|
||||
leaves out the ones every card's cache holds, not only the first card's, and when the rest does not fit whole it keeps
|
||||
the hottest by the expert profile over all the layers. An adaptive swap copies the evicted expert back into RAM from
|
||||
the card that owns its layer. Before, `--resident-experts` with a split ran as `--mmap-experts`, and setup recommended
|
||||
one card in the low-RAM mode; with an engine that has it (`RESIDENT_SPLIT_ENGINE` in setup.py), setup keeps the cards
|
||||
together and the experts no card holds in RAM. The same copy, sized by a RAM budget (`--resident-budget-gib`, the
|
||||
Unsloth UD-Q4_K_XL and UD-IQ4_XS configs), is kept on a split as well ([UNSLOTH_Q4.md](UNSLOTH_Q4.md)). Swift 1.5 IQ3_XXS at
|
||||
160K (q4_0 KV, `--prefill 4096`, `--spec 4` with the stock draft layer), RTX 4060 Ti (layers 0-19) + RTX 5080 (20-47),
|
||||
i9-14900KF, 32 GB of RAM, Windows 11, four greedy prompts at a time, decode tok/s:
|
||||
|
||||
| | first four prompts | after three more rounds |
|
||||
|---|---|---|
|
||||
| 5080 alone, `--resident-experts` | 25 | 29 |
|
||||
| split, `--mmap-experts` (what `--resident-experts` became on a split) | 32 | 64 |
|
||||
| split, `--resident-experts` (22 GiB of experts locked in RAM) | 71 | 69 |
|
||||
|
||||
The split with `--mmap-experts` catches up once the OS file cache holds the experts, on a PC with nothing else
|
||||
running; the resident copy is there from the first request and stays locked when other programs need the RAM. With
|
||||
`--pcie-frac 0 --adapt-every 0` the split's greedy output is the same with either mode.
|
||||
|
||||
**The prompt path's loan on a split and its streamed ring (`STRATA_SPLIT_RING`).** Under a split the prompt path borrows
|
||||
the tail slots of each card's expert cache for its buffers (they are given back after the prompt), and the resident RAM
|
||||
copy keeps those experts too, so a borrowed expert is not read back from the model file during a prompt. How many slots
|
||||
are borrowed depends on the prompt path's streamed ring: a split uses 96 ring slots when at least 75% of the (layer,
|
||||
expert) pairs sit in some card's cache, and `STRATA_SPLIT_RING=N` sets N slots (`0` goes back to the pinned-share rule).
|
||||
The ring used to be chosen after the RAM copy's regions were sized, so the regions were sized for a different ring than
|
||||
the prompt path then used: with `STRATA_SPLIT_RING=384` it borrowed 5,396 slots and the copy kept 4,662, and the
|
||||
other 734 were read from the model file in every chunk. The ring is now chosen first (`Prefill::set_ring_override`
|
||||
says it must be set before the buffers are counted). The start-up line `strata serve: lend sizing: the prompt path
|
||||
borrows N slots (ring R slots at chunk C), K of them keep their experts in RAM too` shows both numbers, and a
|
||||
`WARNING` follows when K is below N. Without a split, or without `--resident-experts`, nothing changes. With a split
|
||||
the order of the two steps changed for every ring choice, the 96-slot rule included: its regions used to be sized for
|
||||
the larger default ring, so there the copy probably kept more slots than were borrowed (not measured). The 5,396 /
|
||||
4,662 figures were measured on the test machine (2x RTX 3080 20 GB at 220 W, Xeon E5-2696 v4, UD-Q4_K_XL,
|
||||
`--layer-split 23`, resident RAM mode, `--prefill auto:16384`, 2 slots), one restart per arm, with `STRATA_SPLIT_RING=384`
|
||||
and `STRATA_PREFILL_RING` unset. Before the change a 104K-token prompt made 413 blob reads and read 10.3 GB of experts
|
||||
from the model file per request; after it, 0, with all 5,396 borrowed slots kept in RAM, for 2.15 GiB more pinned RAM
|
||||
(13.61 -> 15.76 GiB). 25K and 51K prompts read nothing from the file in either case. With the override unset and
|
||||
`STRATA_PREFILL_RING=384` the lend lines are identical before and after (5,396 / 5,396, 15.76 GiB). No speed gain is
|
||||
shown: the medians of three 104K reads were 2,803 tok/s before and 2,907 after, but the spread before is 2,540-2,897
|
||||
(its first read faulted 4.2 GB in from NVMe) and there is no A/A restart. Here the page cache absorbed the 10 GB per
|
||||
request; with less spare page cache it would be disk traffic. Not measured: both ring variables unset.
|
||||
|
||||
**A separate VRAM reserve for the later cards:** `--vram-reserve-later-mib N` (default: `--vram-reserve-mib`'s value).
|
||||
The card that drives the monitors needs more headroom than one that drives none; with the display on the last card,
|
||||
`--vram-reserve-mib 300 --vram-reserve-later-mib 1800` gives the first card's cache that VRAM.
|
||||
|
||||
**auto** tries every placement (all of them for two, three or, since 0.1.40, four cards; proportional to the free VRAM beyond that) and
|
||||
keeps the one whose caches would hold the most of the expert profile, hottest pairs weighted most; ties go to the
|
||||
placement that leaves the fullest card the most room. The startup log prints the choice:
|
||||
**`auto`** tries every placement for two, three or four cards (proportional to free VRAM beyond that) and keeps the one whose
|
||||
caches hold the most of the expert profile, hottest pairs weighted most. Ties go to the placement that leaves the fullest card
|
||||
the most room. The startup log prints the choice:
|
||||
|
||||
```
|
||||
strata generate: layer split auto: K=19 - the caches hold 11767 of 12288 profiled pairs (fullest device 100%)
|
||||
strata serve: layer split: layers 0-18 (CUDA0), 19-47 (CUDA1), one hand-off per window
|
||||
```
|
||||
|
||||
## What each card holds
|
||||
**Card order with `auto`:** the faster card goes last, because the last stage runs the head, the draft layer and the verify, and
|
||||
a prompt chunk waits on it. Cards within 5% keep your order, and so does a manual split. `"gpu_order": "as_given"` keeps your
|
||||
order under `auto` as well. The log line `layer split: card order ...` says when the order changed.
|
||||
|
||||
- **every card**: a copy of the dense weights (~3.4 GB for the Coder), its own session state (the KV cache of the full
|
||||
context), its verify window and its prompt-path buffers, and an expert cache for its layers filled from the profile;
|
||||
- **the last card**: also the output head and the draft layer (~0.8 GB);
|
||||
- **host RAM**: the expert arena once, shared by all cards (the CPU pool computes whatever no card holds).
|
||||
**Unequal cards under `auto`.** The search asks for at least 3 layers per card and, on two cards, first for a placement in which
|
||||
each card owns at least its share of the layers by its room for experts. The log line says which rule chose the split. Prefill
|
||||
that stays flat from 4K to 16K tokens is the sign of an idle card. On an unequal pair, sweep `"layer_split"` once, because a
|
||||
4K-only benchmark cannot see it. On x16 plus x4 rigs the link-miss scaling can move `auto` off the balanced placement;
|
||||
`STRATA_SPLIT_UNIFORM=1` (with `"gpu_order": "as_given"`) prices every link the same.
|
||||
|
||||
Prompts are read in chunks that flow through the cards in turn; while a later card reads chunk c, the first card
|
||||
already reads chunk c+1. (0.1.44, AMD gfx12: the auto chunk on a split stays at 8192, not the single-card 16384: a 16K prompt is then two chunks
|
||||
that overlap instead of one, a 32K one four instead of two. Same tokens; 2x / 3x / 4x R9700 prompt tok/s at 16K +36% / +45% / +53%,
|
||||
32K +18% / +21% / +29%, 64K +4% / +12% / +17%, 8K unchanged. `STRATA_SPLIT_CHUNK16K=1` keeps 16384.) Conversation checkpoints save and restore every card's state; the adaptive expert swaps copy
|
||||
into the card that owns the layer.
|
||||
## Tuning options
|
||||
|
||||
## Overlapped hand-off (`STRATA_SPLIT_OVERLAP=1`, opt-in)
|
||||
These are engine flags and environment variables. Put flags in the config's `"args"` and variables in its `"env"` object, then
|
||||
restart. Defaults are off unless the table says otherwise.
|
||||
|
||||
By default each verify window runs the stages one after another: the host syncs stage 1, then stages and launches
|
||||
stage 2's graph, and so on. `STRATA_SPLIT_OVERLAP=1` (stages on different cards only) gives the hand-off a ready
|
||||
flag: the earlier stage's GPU writes the hand-off with volatile stores and raises the flag (`handoff_publish`), the
|
||||
next stage's graph is staged and launched on a helper thread while the earlier stage is still running and waits for
|
||||
the flag on its GPU (`wait_flag_ge`), and the host goes straight from serving one stage's layers to the next, syncing
|
||||
the earlier stage only once the chain is done. It takes one stream sync and one graph launch per window off the
|
||||
critical path. Batch windows (several conversations) keep the serial order. It is off beside `--pipeline-windows`, whose loop overlaps the stages its own way.
|
||||
| Setting | What it does | Notes |
|
||||
| --- | --- | --- |
|
||||
| `"split_skip_if_fits": true` (flag `--split-skip-if-fits`) | Runs on the first card alone when it holds every profiled expert plus the context's KV, the draft layer and the reserve. Otherwise the split stays. | Opt-in. Needs `--layer-split auto`. |
|
||||
| `STRATA_SPLIT_OWN=1` or `auto` | Gives every card its own prompt buffers. `auto` does it only where the buffers are at most 12% of each card's VRAM. | Output differs from the default's (stable and coherent). |
|
||||
| `STRATA_PREFILL_HELP=1` | The idle card computes a share of the streamed experts of a one-chunk prompt. `STRATA_PREFILL_HELP_FRAC=f` fixes the share. | Helps prompts up to about 3.3K tokens on slow links only; not bit-identical; native packs on the MMQ prompt path only. |
|
||||
| `STRATA_SPLIT_MTP_BATCH=1` | The last card fills the draft layer's prompt K/V in batches, as on one card. | Needs a `--native` pack and a paged or ring draft K/V; says why once if it declines. |
|
||||
| `STRATA_STAGE_TRIM=1` | Each card loads only its own layers' dense weights (needs explicit `--layer-split K`, not `auto`). The freed VRAM goes to that card's expert cache. | Output can differ slightly. |
|
||||
| `--vram-reserve-later-mib N` | A separate VRAM reserve for the later cards (default: `--vram-reserve-mib`). Use it when the display sits on the last card. | `--vram-reserve-mib 300 --vram-reserve-later-mib 1800` gives the first card that VRAM. |
|
||||
| `--prefill-pipe R` (or `STRATA_PREFILL_PIPE=R`) | Reads each prompt in the chunk count with the least pipeline time for b/a = R tokens. `1` evens out today's chunk count. | Opt-in. `./setup.sh --calibrate` measures it per PC and writes it to the config. |
|
||||
| `STRATA_SPLIT_RING=N` | Sets the prompt path's streamed ring to N slots (`0`: the pinned-share rule). A split uses 96 when at least 75% of the (layer, expert) pairs sit in some card's cache. | The startup line `lend sizing: ...` shows the result. |
|
||||
| `STRATA_SPLIT_CHUNK16K=1` | Keeps the 16384-token auto chunk on AMD gfx12 splits (default 8192). | |
|
||||
| `STRATA_SPLIT_OVERLAP=1` | Overlapped hand-off between stages on different cards. | Helps AMD; no gain on NVIDIA. See [below](#overlapped-hand-off). |
|
||||
| `--split-device D1[,D2..]` | The later stages' devices (default: the next visible ones). `--layer-split K --split-device 0` runs both stages on one card; a bit-exact hand-off check, not a speed mode. | |
|
||||
|
||||
Measured on 2x RX 7900 XTX (gfx1100, ROCm, Linux), Ryzen 9 9900X, 52 GB RAM, Swift 1.5 IQ3_XXS, 128K context, 8-bit
|
||||
KV, layers 0-25 / 26-47; 512-token greedy answers, median of 8 runs:
|
||||
**The resident RAM mode works on a split.** `--resident-experts` (the low-RAM mode) leaves out of the RAM copy the experts any
|
||||
card's cache holds. An adaptive swap copies the evicted expert back into RAM from the card that owns its layer. A RAM budget
|
||||
(`--resident-budget-gib`) works on a split as well ([UNSLOTH_Q4.md](UNSLOTH_Q4.md)). With `--pcie-frac 0 --adapt-every 0` the
|
||||
split's greedy output is the same in either mode.
|
||||
|
||||
| | serial (default) | `STRATA_SPLIT_OVERLAP=1` | one card |
|
||||
|---|---|---|---|
|
||||
| decode, code prompt | 53 tok/s | 70.1 tok/s | ~70 tok/s |
|
||||
| decode, prose prompt | 45 tok/s | 55.9 tok/s | ~68 tok/s |
|
||||
**Under WDDM (Windows and WSL2)** only 8 GiB of the expert arena is pinned on a multi-GPU run; the rest streams through the
|
||||
pinned staging ring. Linux pins the whole arena. `STRATA_ARENA_PIN_GIB=N` pins at most N GiB, `0` the whole arena, on any OS.
|
||||
|
||||
Greedy output is byte-identical with the flag at 0 and 1; the quiz set scored the same (47/51). On NVIDIA it
|
||||
gives nothing: 2x RTX A4000 (IQ3_XXS, 5 interleaved pairs, 0.2.1 engine), decode story +0.3 %, code +2.1 %, facts -0.3 % against
|
||||
+2.8 % of noise, prompts +0.1 %, with the same tokens. A sync and a graph launch cost less there. The second card then reads about 94 %
|
||||
busy in `nvidia-smi` (a kernel waiting for the flag), against 58 % without the flag, with no more tokens per second.
|
||||
### Overlapped hand-off
|
||||
|
||||
The reading stage's wait for the flag is bounded on the GPU: `STRATA_SPLIT_WAIT_MS` (default 30000, 0 = no bound).
|
||||
If the flag does not arrive in time, the wait marks an error word and returns, the window's graph completes, and the
|
||||
host fails that window with an error naming the setting (the server then restarts the engine) instead of using a
|
||||
stale hand-off. Normally the host's own per-layer wait (20 s, #267) ends a stalled window first and releases every
|
||||
GPU wait, the hand-off's included. Test hook: `STRATA_TEST_HANDOFF_DROP=N` withholds the flag in the first stage's
|
||||
N-th window, as if the publish were lost.
|
||||
## Prompt chunks sized for the pipeline (`--prefill-pipe`, opt-in; setup's `--calibrate` sets it)
|
||||
By default each verify window runs the stages one after another. `STRATA_SPLIT_OVERLAP=1` (stages on different cards only) gives
|
||||
the hand-off a ready flag, so the next stage's graph is launched while the earlier stage still runs and waits on the flag on
|
||||
its GPU. It takes one stream sync and one graph launch per window off the critical path. Batch windows keep the serial order, and
|
||||
it is off beside `--pipeline-windows`. The wait is bounded by `STRATA_SPLIT_WAIT_MS` (default 30000, `0` = no bound); on timeout
|
||||
the window fails with an error naming the setting and the server restarts the engine.
|
||||
|
||||
`--prefill auto` takes the largest chunk the buffers allow (up to 8192), which is what one card wants: a chunk costs a
|
||||
fixed pass over its layers' experts (b: dequantized, or streamed over PCIe where they are not in VRAM) plus a part per
|
||||
token (a), so few large chunks are cheapest. A layer split reads a prompt as a pipeline - stage s reads chunk c while
|
||||
stage s+1 reads chunk c-1 - and then takes about sum_i (b + a C_i) + (S - 1) max_i (b + a C_i) for chunks C_i over S
|
||||
stages: large or uneven chunks keep the later cards waiting (a 10.7K-token prompt in a 7680 and a 3011 chunk over
|
||||
four stages: the first stage carried 24 of the 36 s), small ones pay b again and again.
|
||||
## Pipeline windows
|
||||
|
||||
`--prefill-pipe R` reads each prompt in the chunk count with the least pipeline time for b/a = R tokens, when that
|
||||
beats today's split by 3% or more: every count from the fewest the buffers allow is tried, each chunk evened out on the
|
||||
256-token grid (at least 512, at most the buffers' chunk; keep `--prefill auto` so long prompts may use large chunks).
|
||||
b/a differs by an order of magnitude between rigs - about 700 tokens on four GP100 dies (every expert in VRAM, slow
|
||||
arithmetic), about 5000 on two RTX 3090 Ti (part of the experts streamed per chunk, fast arithmetic) - and a wrong one
|
||||
costs: a fixed 1024 read 19-24% slower on the two RTX 3090 Ti and 22-36% slower on two RTX 4090 (reported in the PR by
|
||||
MistyMoonR). So it is neither guessed nor learned from the prompts the engine serves (that made a prompt's chunks, and
|
||||
so its output bits, depend on how busy the PC happened to be): like the PCIe share and the draft floor, it is a
|
||||
setting of the PC that setup's `--calibrate` measures once (`tools/calibrate.py`: on a layer split, the prompt read
|
||||
speed of a 2K- and an 8K-token prompt - the same text every time, after a header of its own so nothing is reused -
|
||||
with the chunks off, evened out and b/a 768, 1536, 3072 and 6144; the best is kept only when it reads faster than off
|
||||
in each of three paired rounds and by more than 3% in the median) and writes into the config. Measured so: four GP100
|
||||
dies keep b/a 768 (355-356 against 219-220 tok/s in every round, +62%); two RTX 3090 Ti keep it off (nothing read
|
||||
faster than off, b/a 768 20% slower). A first version that read a different stretch of text per value and confirmed in
|
||||
two rounds picked 1536 on the RTX 3090 Ti pair from one fast read - where the experts are streamed, the read speed
|
||||
depends on the text. `--prefill-pipe 1` needs no
|
||||
b/a: it is `STRATA_PREFILL_EQUAL=1`'s rule (#693) - today's chunk count, evened out on the 256-token grid, and only
|
||||
when today's last chunk has 1024 tokens or more (3072 with `STRATA_PREFILL_CPU_SHARE` set). A chunk that size streams
|
||||
every expert its card does not hold, a shorter one only the experts it routes to, so a prompt just past the chunk size
|
||||
reads as one full chunk and a nearly free tail, and evening that out would add a second full pass over the experts:
|
||||
the first version evened every split out, and on two RTX 4090 MistyMoonR's calibration read 1,601 tok/s with it
|
||||
against 2,129 off. 0 (the default) is today's split;
|
||||
`STRATA_PREFILL_PIPE=<R>` is the same as the flag where the config cannot carry it. The chunk geometry changes the
|
||||
rounding, which is why it is opt-in.
|
||||
`--pipeline-windows 2` lets the first card start the next verify window while the last card still verifies this one. The next
|
||||
window is a guess: that this window is accepted whole and that its bonus token is the one the draft layer predicts. If the guess
|
||||
holds, half of the next window is already done. If not, the first card restores its state and builds the window from the real
|
||||
tokens. The tokens are the serial loop's, up to CPU rounding.
|
||||
|
||||
A prompt segment (the prompt between two of its cuts: the system-prompt root, the last turn, a `pin=`) is planned once,
|
||||
on its whole length and R, and every run of it reads in that chunk. So its chunks, and the loan its buffers are laid
|
||||
out for, are the same whether it is read in one run, in pieces between the windows of decoding `--batch` slots, or goes
|
||||
on after a `BYIELD` or from a periodic checkpoint taken inside it (the slot and the checkpoint keep the plan; the read
|
||||
goes on in it when it continues the same segment, else it plans again and says so in the log). The output bits then
|
||||
match as far as the experts the cards hold do (`--adapt-swaps 0` for byte-identical repeats, as without the pipe).
|
||||
Before this, a read beside decoding slots re-planned every piece, and one that went on after a `BYIELD` re-planned what
|
||||
was left. A chunk the pipe planned takes no `STRATA_PREFILL_CPU_SHARE`: the stages read
|
||||
such chunks at the same time, and which stage got the CPU pool first would decide which experts the CPU computed. The
|
||||
DONE line ends with `chunk=` (the first chunk of the request's longest batched segment) and `experts_streamed=` (the
|
||||
expert blobs its prompt copied to the cards), and the INFO line carries `prefill_cap`, `prefill_stages`,
|
||||
`stream_all_min` and `prefill_pipe`. One line in the log says what a plan chose:
|
||||
- **Default:** on for AMD RDNA4 (gfx1200 / gfx1201) with `--serve`, the draft layer, and a split of two or more stages when the
|
||||
drafter can sit on the first card, otherwise three or more. The log says "is on by default for a layer split of N stages".
|
||||
`--pipeline-windows 0` or `STRATA_PIPELINE_AUTO=0` turns it off. NVIDIA builds keep it opt-in (`--pipeline-windows 2`).
|
||||
- **Draft layer placement:** with the pipeline on, the draft layer loads on the first card, whose window has finished when the
|
||||
chain starts. `STRATA_MTP_STAGE0=0` keeps the old place.
|
||||
- **Off, with one log line saying why:** `--batch` slots, `--peer-device`, the helper caches (`--expert-cache-device1..3`, which
|
||||
`--remote-expert-opt` builds on), a split onto one GPU, no draft layer, and on three or more stages `--pipeline-windows 1` and
|
||||
`--adapt-async 1`. A request with repetition penalties or coupled draft sampling decodes serially.
|
||||
- **Cost:** a second verify window on each card, 160 MiB less expert cache per card, plus two copies of the first card's
|
||||
recurrent state (about 3 MiB per GDN layer it runs).
|
||||
- **Config:** `"args": [ ..., "--pipeline-windows", "2" ]` with `"layer_split": "20"`.
|
||||
|
||||
strata prefill: 10058 tokens in 1280-token chunks over 4 stages (--prefill-pipe 706: b/a)
|
||||
`--remote-expert-opt` does something only with a helper cache; on a plain layer split it is inert.
|
||||
The `STRATA_PIPELINE_*` tuning and test variables are read only with `STRATA_PIPELINE_DEBUG=1`.
|
||||
|
||||
Four GP100 dies (PH402 SKU 200, `--layer-split 12,24,36`, Flash-Next IQ3_XXS, 64K context), prompt tok/s, fresh
|
||||
prompts, two interleaved rounds (each within 0.2%), with b/a 1024 (the same chunks as 700-890, measured on this rig):
|
||||
## Two models on two cards
|
||||
|
||||
| Prompt tokens | `--prefill auto` | `--prefill 2048` | `--prefill auto` + `--prefill-pipe 1024` (chunk) |
|
||||
The other way to use two cards is two models, one per card, each answering on its own. In the app, turn on **Run multiple
|
||||
models** on the Models page. The card list then says which card a loaded model uses ("In use by ..."), starts that card as Off
|
||||
for the next model, and explains a failed start on a card that is taken. Two models at once on two makes is not supported.
|
||||
|
||||
**Measured on:** a PC with two of four Radeon AI PRO R9700 cards, Linux, AMD build, Q2_0 on one card and IQ3_XXS on the other.
|
||||
Both loaded (33 s and 59 s with the files cached, 114 s the first time) and answered chats at about 60 tokens/s each. Unloading
|
||||
one left the other answering. The same app code drives NVIDIA cards, but two models on two NVIDIA cards were not run.
|
||||
|
||||
## DeepSeek V4.1-Flash
|
||||
|
||||
DeepSeek V4.1-Flash runs in the app and the Terminal and spreads its experts and its dense weights over several cards. It does not use
|
||||
the Qwen layer split described above. On four Radeon AI PRO R9700 cards it reaches about 31 tokens/s. Hardware, sets and speeds:
|
||||
[MODELS.md](MODELS.md#deepseek-v41-flash).
|
||||
|
||||
## Limits
|
||||
|
||||
- Every card needs compute capability 7.5 (RTX 20 or newer), unless you opt into the CUDA 12 engine for older cards.
|
||||
- Works across cards: images (`--vision`), control vectors, KV streaming (`--kv-resident`) and mid-prompt checkpoints
|
||||
(`--prompt-cache-every`). Each card keeps its own part.
|
||||
- The older helper-GPU caches (`--expert-cache-device1..3`, [SECOND_GPU.md](SECOND_GPU.md)) take the visible GPUs no stage runs on.
|
||||
A weak helper can make decoding slower, because its per-layer round trip costs more than the CPU pool needs for those experts.
|
||||
- `--mmap-experts` needs a canonical pack (`experts.bin`); a native (IQ) pack says so at start.
|
||||
- The prompt path has its own buffers on every card (1.5 GB each at the default 2048-token chunk; `--prefill 1024` halves that).
|
||||
An explicit `--expert-cache` on the first card is capped to leave room for them.
|
||||
- More cards pay off when the model's routed experts do not fit the faster ones. An extra slow card costs its own round per
|
||||
window and can make a good pair slower.
|
||||
- For several conversations at once, `--batch N --batch-groups G --trim-stage-weights` pipelines them through the cards
|
||||
([BATCHING.md](BATCHING.md)).
|
||||
|
||||
## History / older measurements
|
||||
|
||||
Dated numbers behind the settings above. Each row is one machine; none is a claim about yours.
|
||||
|
||||
<a id="measured"></a>
|
||||
|
||||
**Layer split, Coder model.** Measured on: RTX 5080 + RTX 3090, Ryzen 9 9950X3D, Coder, 32K context, engine 0.1.20-era
|
||||
(`bench/results/2026-09-29-layer-split/`).
|
||||
|
||||
| Setup | Prompt 16K / 28K tok/s | Decode story / code tok/s |
|
||||
| --- | --- | --- |
|
||||
| 5080 alone | 1,726-2,017 / 1,970 | 83-87 / 88-105 |
|
||||
| 5080 + 3090, best split (K=26) | 2,039 / 2,357 | 84 / 110 |
|
||||
| 5080 + 3090, auto (K=22) | 2,037 / 2,073 | 80 / 109 |
|
||||
|
||||
A third card (RTX 2080 Ti) made that pair slower: 68 / 90 tok/s decode.
|
||||
|
||||
**Option measurements**
|
||||
|
||||
| Setting | Measured on | Result |
|
||||
| --- | --- | --- |
|
||||
| Unequal pair, `auto` rule | 2x RTX A4000 made unequal, IQ3_XXS, 14.5K prompt, 5 interleaved pairs | prompt 945 -> 1,441 tok/s (+52%), decode 50.2 -> 65.9 (+31%) |
|
||||
| Unequal pair, hand sweep of `"layer_split"` | 4060 Ti 16 GB + 3060 12 GB, IQ3_S | K=12/18/24/28/32: 582 / 693 / 762 / 802 / 803 prompt tok/s; the old rule chose K=3: 452 |
|
||||
| x16 + x4 link scaling | 2x RTX 3090, x16 + x4 | K=31: 114-119 tok/s decode, K=27: 130-136 |
|
||||
| `split_skip_if_fits` | R9700 32 GB + RX 9070 XT, Coder | 4K prompts 1,244 -> 1,776 tok/s, decode ~51 -> ~60; 16K prompts ~5% slower |
|
||||
| `STRATA_SPLIT_OWN=1` | R9700 + RX 9070 XT | 2K prompts 1,450 and 16K 2,227 tok/s (default 1,265 and 1,950) |
|
||||
| `STRATA_PREFILL_HELP=1` | 2x RTX 3090, UD-Q4_K_XL, no P2P | prompt tok/s without / with: 1.1K 496 / 723, 2K 878 / 1,128, 3K 1,260 / 1,440; 4K and 8K unchanged |
|
||||
| `STRATA_PREFILL_HELP=1` | 2x R9700 (16 GB each), Q2_0, PCIe 4.0 x16 | 700 tokens 638 -> 672; 1.5K 839 -> 809; 3K 1,105 -> 929; slower from 1.5K, so it stays off |
|
||||
| `STRATA_SPLIT_MTP_BATCH=1` | 2x RTX 3080 20 GB, Xeon E5-2696 v4, UD-Q4_K_XL, `--layer-split 23` | prompt read +6.7% at 25K, +4.3% at 51K, +3.3% at 104K; decode not shown unchanged (about 3% lower, within restart noise) |
|
||||
| `STRATA_STAGE_TRIM=1` | 2x MI50 16 GB, Coder | experts in VRAM 8,819 -> 10,626; decode 39.2 -> 41.7 tok/s |
|
||||
| Prompt chunk on a split, AMD gfx12 | 2x / 3x / 4x R9700 | 16K prompt +36% / +45% / +53%; 32K +18% / +21% / +29%; 64K +4% / +12% / +17% |
|
||||
| `STRATA_SPLIT_OVERLAP=1` | 2x RX 7900 XTX, Swift 1.5 IQ3_XXS, 128K, median of 8 | code decode 53 -> 70.1 tok/s (one card ~70); prose 45 -> 55.9 (one card ~68) |
|
||||
| `STRATA_SPLIT_OVERLAP=1` | 2x RTX A4000, IQ3_XXS | +0.3% story, +2.1% code, -0.3% facts, inside the noise |
|
||||
| Two Tesla P40, layer split | community report, IQ2_XS (#1028), experimental | decode 19.6 tok/s on one card, 34.8 on both |
|
||||
|
||||
**Resident RAM mode on a split.** Measured on: RTX 4060 Ti (layers 0-19) + RTX 5080 (20-47), i9-14900KF, 32 GB RAM, Windows 11,
|
||||
Swift 1.5 IQ3_XXS, 160K context, four greedy prompts at a time, decode tok/s.
|
||||
|
||||
| Setup | First four prompts | After three more rounds |
|
||||
| --- | ---: | ---: |
|
||||
| 5080 alone, `--resident-experts` | 25 | 29 |
|
||||
| split, `--mmap-experts` | 32 | 64 |
|
||||
| split, `--resident-experts` (22 GiB locked in RAM) | 71 | 69 |
|
||||
|
||||
**Prompt chunks for the pipeline (`--prefill-pipe`).** Prompt tok/s, fresh prompts, two interleaved rounds per row.
|
||||
|
||||
| Prompt tokens | `--prefill auto` | `--prefill 2048` | `--prefill-pipe 1024` (chunk) |
|
||||
| ---: | ---: | ---: | ---: |
|
||||
| 2,013 | 180.9 | 181.4 | 228.0 (1024) |
|
||||
| 3,865 | 207.6 | 276.5 | 312.4 (1024) |
|
||||
| 10,041 | 261.1 | 433.0 | 432.6 (2048) |
|
||||
| 28,998 | 475.7 | 554.1 | 559.7 (3328) |
|
||||
|
||||
Two RTX 3090 Ti (CUDA 12.9 build for sm_86, two stages, Swift 1.5 IQ3_XXS, 1M context, 13,371 of 24,576 experts in
|
||||
VRAM), prompt tok/s, fresh prompts, two starts per arm, each reading the four prompts twice (the cold first pass after
|
||||
a card was freed is left out); b/a 5836 and 6432 (timed on the rig), against 1024:
|
||||
Measured on: four GP100 dies (PH402 SKU 200), `--layer-split 12,24,36`, Flash-Next IQ3_XXS, 64K context. On 2x RTX 3090 Ti
|
||||
(Swift 1.5 IQ3_XXS, 1M context) calibration kept it off, and a fixed 1024 chunk read 19-24% slower there and 22-36% slower on
|
||||
2x RTX 4090 (community report).
|
||||
|
||||
| Prompt tokens | `--prefill auto` | + b/a 5836 / 6432 (chunk) | for comparison: b/a 1024 |
|
||||
| ---: | ---: | ---: | ---: |
|
||||
| 2,035 | 694-815 | 698-816 (today's) | 560-562 |
|
||||
| 3,887 | 1,127-1,207 | 1,115-1,227 (today's) | 880-885 |
|
||||
| 10,063 | 1,437-1,553 | 1,654-1,681 (2 x 5120) | 1,109-1,113 |
|
||||
| 29,019 | 2,532-2,603 | 2,566-2,620 (today's) | 2,064-2,089 |
|
||||
**Pipeline windows.** Decode tok/s.
|
||||
|
||||
## Limits (for now)
|
||||
| Measured on | Result |
|
||||
| --- | --- |
|
||||
| 2 cards, R9700, code, serial loop -> draft layer on the first card | 103 -> 125 (+21%); 3 cards +19%, 4 cards +22% |
|
||||
| RTX 4060 Ti + RTX 5080, i9-14900KF, 32 GB, Swift 1.5 IQ3_XXS, 160K, greedy 500 tokens | mean 72.4 -> 84.0 (+16%): Python 90.4 -> 103.1, English prose 63.0 -> 71.1, copy-heavy edit 90.5 -> 117.9 |
|
||||
| 4x RX 7900 XT, Flash-Next GSQ-RCO IQ3_S, 262K, `--layer-split 12,24,36` | 4K 59.4 -> 67.6; 32K 51.8 -> 65.6 |
|
||||
|
||||
- **Works across cards** (bench/results/2026-09-29-layer-split-limits):
|
||||
- images (`--vision`): each card keeps its own image-position table;
|
||||
- control vectors and the experimental speed projection: each card holds the vector's tables, switched on and
|
||||
off per request on all of them;
|
||||
- KV streaming (`--kv-resident`): each card streams the KV of its own session;
|
||||
- mid-prompt checkpoints (`--prompt-cache-every`): each card saves its part of a checkpoint when it has read that
|
||||
chunk;
|
||||
- the older helper-GPU caches (`--expert-cache-remote`, docs/SECOND_GPU.md): they take the visible GPUs no stage
|
||||
runs on, and hold only experts no stage's cache holds. On the test rig, a 2080 Ti helper made decoding slower,
|
||||
as it did without a split: its per-layer round trip costs more than the CPU pool needs for those experts.
|
||||
- `--mmap-experts` needs a canonical pack (`experts.bin`), with or without a split; a native (IQ) pack says so at
|
||||
start.
|
||||
- The prompt path has its own buffers on every card (1.5 GB each at the default 2048-token chunk; `--prefill 1024`
|
||||
halves that) instead of borrowing cache slots as one card does. An explicit `--expert-cache` on the first card is
|
||||
capped to leave room for them.
|
||||
- Under WDDM (Windows, and WSL2) only 8 GiB of the expert arena is pinned (more, mapped into two GPU contexts,
|
||||
leaves WDDM refusing allocations); the rest streams through the pinned staging ring. A Linux driver has no such
|
||||
limit, so there the whole arena is pinned (since 0.1.31; the cap cost a 4090 + 3060 split two thirds of its
|
||||
prompt speed, #253). `STRATA_ARENA_PIN_GIB=N` pins at most N GiB, `0` the whole arena, on any OS.
|
||||
- Every card needs compute capability 7.5 (RTX 20 or newer). The pre-sm_80 QSA scorer path is fp32 FMAs, so a
|
||||
Turing card runs the same kernels instead of the tensor-core prompt attention.
|
||||
|
||||
## Measured
|
||||
|
||||
The Coder on an RTX 5080 + RTX 3090 (Ryzen 9 9950X3D), 32K context; details in
|
||||
`bench/results/2026-09-29-layer-split/`:
|
||||
|
||||
| | Prompt 16K / 28K tok/s | Decode story / code tok/s |
|
||||
|---|---|---|
|
||||
| 5080 alone | 1,726-2,017 / 1,970 | 83-87 / 88-105 |
|
||||
| 5080 + 3090, best split (K=26) | 2,039 / 2,357 | 84 / 110 |
|
||||
| 5080 + 3090, auto (K=22) | 2,037 / 2,073 | 80 / 109 |
|
||||
|
||||
- **Prompts gain the most** (+18-20%): each card reads its own layers of the chunk while the other reads the next.
|
||||
- **Decode is on par with the faster card alone**, and ahead on code. Once both caches hold nearly every routed
|
||||
expert, the per-layer GPU time decides.
|
||||
- **Correctness:** one GPU is byte-identical to 0.1.20, and the hand-off itself is bit-exact.
|
||||
|
||||
**Which cards and in what order:**
|
||||
- Put the fastest card first; auto gives it as many layers as its cache allows.
|
||||
- Leave out a much slower card when two already hold the model. An RTX 2080 Ti as a third card made the 5080 +
|
||||
3090 pair slower (68 / 90 tok/s decode): every extra card costs its own round per window.
|
||||
- More cards pay off when the model's routed experts do not fit the faster ones.
|
||||
|
||||
## One conversation with both cards busy (`--pipeline-windows`, default on two or more AMD gfx12 stages)
|
||||
|
||||
**Default on AMD RDNA4 (gfx1200 / gfx1201).** With `--serve`, the draft layer and a layer split of **two or more stages**,
|
||||
`--pipeline-windows 2` is on unless you say otherwise (the log says "is on by default for a layer split of N stages"; two
|
||||
stages since 0.1.44, three or more since 0.1.43). It is no quality trade: the tokens are the serial loop's up to CPU rounding
|
||||
(below). `--pipeline-windows 0` or `STRATA_PIPELINE_AUTO=0` turns it off; `--pipeline-windows 2` asks for it on any split.
|
||||
The same options that switch it off when you ask for it (`--batch`, `--peer-device`, the helper caches, no draft layer,
|
||||
`--adapt-async 1`) leave it off without a line when it is only the default. NVIDIA builds keep it opt-in.
|
||||
|
||||
**Where the draft layer sits (0.1.44).** The draft layer's chain used to run on the last card, beside the last stage's window.
|
||||
On 2x / 4x R9700 the two streams then slowed each other, in most engine starts and for the whole life of that engine: the
|
||||
last stage's window took 27 ms instead of 13 ms while a chain ran (14.9 ms in the lucky starts, which also decoded 25% faster;
|
||||
the 0.1.43 notes' "128.9 does not reproduce" was this). With the pipeline on, the engine now loads the draft layer on the
|
||||
**first** card, whose window has finished when the chain starts: the chain's source rows are copied across once per chain and the
|
||||
draft head's rows once at load. Same tokens; `STRATA_MTP_STAGE0=0` keeps the old place. Exact-flags A/B, 5 pairs, decode tok/s
|
||||
(serial loop -> now): 2 cards code 103 -> 125 (+21%), 4K prompt +8%, prose equal; 3 cards code +19%, 4K +12%, prose +2%; 4 cards
|
||||
code +22%, prose +4%, 4K +15%.
|
||||
|
||||
With a split the cards take turns on a verify window: the first card runs its layers and hands off, then waits while
|
||||
the last card runs the rest, the head and the draft. `--pipeline-windows 2` lets the first card start the **next**
|
||||
window while the last card still verifies this one. The next window is a guess: that this window is accepted whole
|
||||
and that its bonus token is the one the draft layer predicts (the draft layer is run on through the drafts of the
|
||||
window in flight). When the guess holds, half of the next window is already done; when it does not, the first card
|
||||
puts its state back (a copy of its recurrent state taken while the window ran) and the next window is built from the
|
||||
real tokens. A guessed window is only started when the draft layer's estimate says it is likely to be kept. Windows
|
||||
copied from earlier context (`--suffix-draft`) are guessed past as well, which is where edits that copy text gain
|
||||
most. `--pipeline-windows 1` overlaps only the short prompt reads that go through the verify windows
|
||||
(`--short-read`).
|
||||
|
||||
Two or more cards, `--serve`. With three or more stages every stage but the last is a front stage: the guessed
|
||||
window follows the verified one through them one card behind (each front stage keeps its own pair of recurrent-state
|
||||
copies and puts its state back on a wrong guess), and only the last stage waits for the verdict. `--pipeline-windows 1`
|
||||
and the asynchronous tier stay two-stage. In the config:
|
||||
|
||||
```
|
||||
"args": [ ..., "--pipeline-windows", "2" ],
|
||||
"layer_split": "20"
|
||||
```
|
||||
|
||||
- **Cost**: a second verify window on each card, 160 MiB more kept out of each card's expert cache, plus two copies
|
||||
of the first card's recurrent state (about 3 MiB per GDN layer it runs) on the first card with `2` (on every front
|
||||
card with three or more stages; only the first card's room is kept out of its cache, the others allocate theirs after).
|
||||
- **Same text**: the last card only ever runs windows that are verified, and every window row computes what it
|
||||
would in any other window, so the tokens are the serial loop's. With `STRATA_IQ_MT_MIN=1 --pcie-frac 0
|
||||
--adapt-every 0` the greedy output is identical bit for bit to the serial loop's with the same expert caches. The pipeline keeps
|
||||
its VRAM out of the caches, so against a run without the flag a few experts move from a card to the CPU, which
|
||||
rounds them differently, and a near-tie can flip (a serial run given the same caches through `--vram-reserve-mib`
|
||||
matches it exactly).
|
||||
- **Off, with one line in the log saying why**, with `--batch` slots, `--peer-device`, the helper caches
|
||||
(`--expert-cache-device1..3`, which `--remote-expert-opt` builds on), a split onto one GPU (`--split-device 0`), or no
|
||||
draft layer; on three or more stages also `--pipeline-windows 1` and `--adapt-async 1`. A request with repetition penalties (`penalty_last_n`) or coupled
|
||||
draft sampling decodes serially.
|
||||
- **With the resident RAM mode's asynchronous swaps** (`--adapt-async 1`, [DETAILS.md](DETAILS.md)) a round's steps
|
||||
advance between the verified windows. Each card's copies are queued by the decode loop itself while that card has
|
||||
no window in flight, after every window that may still read what they overwrite has finished; the moves into RAM
|
||||
wait the same way.
|
||||
- **Measured** (Swift 1.5 IQ3_XXS, 160K context, q4_0 KV, the stock draft layer, RTX 4060 Ti (layers 0-19) +
|
||||
RTX 5080 (20-47), i9-14900KF, 32 GB of RAM with the resident RAM mode on the split (#848); greedy, 500 tokens, two
|
||||
interleaved pairs of three rounds, decode tok/s): Python code 90.4 -> 103.1, C code 76.6 -> 81.6, English prose
|
||||
63.0 -> 71.1, Italian prose 41.6 -> 46.3, a copy-heavy edit (a 5 KB file back with a rename) 90.5 -> 117.9; mean
|
||||
72.4 -> 84.0 (+16%). It pays when the windows are GPU-bound: with the experts read through the OS file cache
|
||||
(`--mmap-experts` on that 32 GB PC) the file reads dominate and it measured no faster.
|
||||
- **Measured on four stages** (Flash-Next GSQ-RCO IQ3_S, 262K context, int8 KV, layer split 12,24,36 on 4x RX 7900 XT,
|
||||
every expert in VRAM; greedy, reasoning off, 256 tokens, repository-text prompts): decode 59.4 -> 67.6 tok/s at 4K
|
||||
(median of 3) and 51.8 -> 65.6 at 32K. A guessed window that holds costs ~15 ms against ~43 ms for a fresh one; about
|
||||
a third of the guesses held. The greedy text was identical to the serial loop's on all four prompts (`STRATA_IQ_MT_MIN=1
|
||||
--pcie-frac 0 --adapt-every 0`), and with every guess forced wrong (`STRATA_PIPELINE_FORCE_MISS=1`).
|
||||
- **HIP**: the draft layer's per-row `gr_read` used the last stage's session scratch (`ss.block.gr`), which that stage's
|
||||
verifier also uses; a chain launched at a verdict and the next window's last stage then ran on the card at once and
|
||||
the window's head read a clobbered mix (wrong tokens after a guess that held). The draft layer now has its own scratch.
|
||||
|
||||
The `STRATA_PIPELINE_*` tuning and test variables (THETA, FORCE_MISS, SWITCH, LOG, TRACE, SPEC_DEPTH and the like) are read only with
|
||||
`STRATA_PIPELINE_DEBUG=1`. `--pipeline-windows` and `--adapt-async 1` combine: the engine turns the asynchronous tier
|
||||
off beside `--pipeline-windows 2` only when `STRATA_PIPELINE_ADAPT_ASYNC=0` is set. What switches either one off is
|
||||
printed once at start ("is off: ..."). `--remote-expert-opt` does something only with a helper cache
|
||||
(`--expert-cache-device1..3`); on a plain layer split it is inert, and setup no longer writes it there (#1447).
|
||||
|
||||
Measured on 2x RTX 3090 (sm_86, 250 W limits; GPU0 PCIe 4.0 x16, GPU1 x4, no NVLink; Ryzen 7 9800X3D, 32 GB RAM),
|
||||
Qwen3.8-Flash-Next GSQ-RCO IQ3_S, `--resident-experts`, KV int8, `--spec 4 --mtp`, `"layer_split": "29"` for the
|
||||
`--pipeline-windows` rows (without `--remote-expert-opt`); the others are setup's config (`auto`). Decode is the mean of
|
||||
runs 2-6 of 1,500-token coding replies at temperature 0.6, prefill one cold 19.9K-token prompt; one run per arm unless
|
||||
noted (reported by adambenhassen, #1447; not repeated on our boxes):
|
||||
**Several settings on one rig.** Measured on: 2x RTX 3090 (250 W, x16 + x4, no NVLink), Ryzen 7 9800X3D, 32 GB RAM, Flash-Next
|
||||
GSQ-RCO IQ3_S, `--resident-experts`, int8 KV, `--spec 4 --mtp`; decode is the mean of 1,500-token coding replies, prefill one
|
||||
cold 19.9K prompt; one run per arm (community report #1447, not repeated here).
|
||||
|
||||
| Arm | Decode tok/s | Prefill tok/s |
|
||||
|---|---:|---:|
|
||||
| setup defaults (two runs) | 137.0 / 136.9 | 1798 |
|
||||
| `--adapt-async 1` (two runs) | 145.2 / 143.7 | 1800 / 1797 |
|
||||
| `--adapt-async 1`, no `--remote-expert-opt` | 145.0 | 1807 |
|
||||
| `STRATA_ADAPT_LAG=2` | 142.8 | 1800 |
|
||||
| `STRATA_EXCHANGE_ROTATE=1` | 139.6 | 1800 |
|
||||
| `STRATA_PF_FUSED=1` | 137.2 | 1913 |
|
||||
| `--pipeline-windows 2` | 136.7 | 2097 |
|
||||
| `STRATA_SPEC_COUPLED=1` | 137.6 | 1804 |
|
||||
| `STRATA_SPEC_PROB=1` | 137.2 | 1788 |
|
||||
| `STRATA_SPEC_COUPLED=1` + `STRATA_SPEC_GUMBEL=1` | 136.5 | 1805 |
|
||||
| async + lag 2 + rotate + pf_fused | 145.0 | 1829 |
|
||||
| pw2 + lag 2 + rotate + pf_fused | 135.9 | 2296 |
|
||||
| pw2 + async + lag 2 + rotate + pf_fused | 142.0 | 2302 |
|
||||
|
||||
The last row keeps most of the asynchronous tier's decode gain and the pipeline's +28% prefill; no stalls in any arm.
|
||||
These are opt-in settings on one rig, not defaults.
|
||||
|
||||
## Several conversations at once
|
||||
|
||||
With a layer split, `--batch N --batch-groups G --trim-stage-weights` decodes several conversations together and
|
||||
pipelines them through the cards: see [BATCHING.md](BATCHING.md).
|
||||
| --- | ---: | ---: |
|
||||
| setup defaults | 137.0 | 1,798 |
|
||||
| `--adapt-async 1` | 144.5 | 1,799 |
|
||||
| `--pipeline-windows 2` | 136.7 | 2,097 |
|
||||
| `STRATA_PF_FUSED=1` | 137.2 | 1,913 |
|
||||
| pipeline-windows 2 + async + lag 2 + rotate + pf_fused | 142.0 | 2,302 |
|
||||
+198
-154
@@ -1,201 +1,155 @@
|
||||
# NVFP4 routed experts
|
||||
|
||||
This page is for people who run, convert or test Qwen3.8-Flash-Next packs whose routed experts are NVFP4. They need an
|
||||
NVIDIA card. For the choice between the three versions, see [MODELS.md](MODELS.md#nvfp4).
|
||||
|
||||
The engine can serve a Qwen3.8-Flash-Next whose routed experts are NVFP4, as NVIDIA's ModelOpt stores them: 64-value
|
||||
blocks of E2M1 codes with four UE4M3 sub-block scales, plus one FP32 `weight_scale_2` per expert and projection.
|
||||
Everything else (attention, GDN, shared experts, routers, the head) stays in the formats the native path already
|
||||
reads. Measured on Windows 11, RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB DDR5-5600.
|
||||
Everything else (attention, GDN, shared experts, routers, the head) stays in the formats the native path already reads.
|
||||
|
||||
**The three NVFP4 versions in setup and the app.** `NVFP4` is "NVFP4 (Strata's version)", `NVFP4-NVIDIA` is "NVFP4
|
||||
(NVIDIA's version)", and `NVFP4-QAD` is "NVFP4 (NVIDIA, trained for FP4)" ([the short guide](MODELS.md#nvfp4)). All three
|
||||
read pictures, through the original model's picture reader file that setup downloads with them (see "Images" below and
|
||||
the FP4-Local section). The other sections of this page apply to all three unless they say otherwise.
|
||||
## Quick start
|
||||
|
||||
## Getting a model
|
||||
Choose a ready-made pack with the Terminal. Manual conversion below needs a source checkout.
|
||||
|
||||
```sh
|
||||
strata launch --family nvfp4 --model NVFP4
|
||||
```
|
||||
|
||||
| Size in setup and the app | Name | Source |
|
||||
| --- | --- | --- |
|
||||
| `NVFP4` | NVFP4 (Strata's version) | GPTQ-quantized pack, `Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata` |
|
||||
| `NVFP4-NVIDIA` | NVFP4 (NVIDIA's version) | `nvidia/Qwen3.8-Flash-Next-NVFP4` quantizes the routed experts the same way (NVFP4, group 16; other layers BF16, PLE table FP8, MTP experts FP8). Setup offers it as the ready-made pack `Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata`. |
|
||||
| `NVFP4-QAD` | NVFP4 (NVIDIA, trained for FP4) | `nvidia/Qwen3.8-Flash-Next-FP4-Local`, converted on your PC ([below](#nvidias-fp4-local-checkpoint)) |
|
||||
|
||||
All three read pictures through the original model's picture reader file that setup downloads with them
|
||||
([Images](#images)). The other sections of this page apply to all three unless they say otherwise. Setup runs a pack in
|
||||
the RAM-budget mode (`--resident-budget-gib`, the RAM less 24 GB, as for Unsloth's files) when the RAM is below the 68 GB
|
||||
of experts plus room.
|
||||
|
||||
## Getting a model by hand
|
||||
|
||||
```sh
|
||||
python tools/nvfp4_convert.py --model <ModelOpt NVFP4 checkpoint dir> --outfile <model>.gguf
|
||||
python tools/iq_pack.py --gguf <model>.gguf --out <pack dir>
|
||||
```
|
||||
|
||||
- `nvfp4_convert.py` runs llama.cpp's own converter (`third_party/llama.cpp`, the commit setup pins) with the type
|
||||
policy the engine needs. The experts are repacked without loss: `tools/nvfp4_verify.py` compares them with the
|
||||
policy the engine needs. The experts are repacked without loss. `tools/nvfp4_verify.py` compares them with the
|
||||
checkpoint bit for bit, and `tools/nvfp4_verify_gguf.py` checks the GGUF.
|
||||
- The small projections the BF16 kernels read (routers, SSM gates, indexer, PLE and hyper-connection projections)
|
||||
are written as BF16.
|
||||
- Everything served natively goes to Q8_0: attention, `ssm_out`, the shared experts, the output head, and the
|
||||
token embedding.
|
||||
- The small projections the BF16 kernels read (routers, SSM gates, indexer, PLE and hyper-connection projections) are
|
||||
written as BF16.
|
||||
- Everything served natively goes to Q8_0: attention, `ssm_out`, the shared experts, the output head and the token
|
||||
embedding.
|
||||
- The 51.2e9-value PLE table is left out.
|
||||
- `iq_pack.py` writes `experts.bin` for an NVFP4 GGUF by itself. Each blob is `[gate | up | down]` plus a 16-byte
|
||||
tail `{s_gate, s_up, s_down, 0}`. The GGUF has no room for these scales, so the engine reads NVFP4 experts from
|
||||
- `iq_pack.py` writes `experts.bin` for an NVFP4 GGUF by itself. Each blob is `[gate | up | down]` plus a 16-byte tail
|
||||
`{s_gate, s_up, s_down, 0}`. The GGUF has no room for these scales, so the engine reads NVFP4 experts from
|
||||
`experts.bin` only.
|
||||
|
||||
Run it like an IQ pack, with the GGUF as its own dense-weight source and the PLE table from an IQ GGUF's PLE shard:
|
||||
|
||||
```
|
||||
```sh
|
||||
strata --pack <pack dir> --native <model>.gguf --native-dense-gguf <model>.gguf \
|
||||
--ple-gguf <an IQ model's shard with per_layer_token_embd> --mtp <mtp/rt> \
|
||||
--expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 ...
|
||||
```
|
||||
|
||||
The NVFP4 checkpoints change only the routed experts. The PLE table, the MTP layer and the expert profile of the
|
||||
original model therefore work as they are.
|
||||
The NVFP4 checkpoints change only the routed experts. The PLE table, the MTP layer and the expert profile of the original
|
||||
model therefore work as they are. The conversion was tested end to end with
|
||||
`jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4`. The NVIDIA checkpoint takes the same path but was
|
||||
not run here as downloaded.
|
||||
|
||||
Tested end to end with `jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4`. By its
|
||||
`hf_quant_config.json`, NVIDIA's `nvidia/Qwen3.8-Flash-Next-NVFP4` quantizes the routed experts the same way
|
||||
(NVFP4, group 16; the other layers BF16, the PLE table FP8, the MTP experts FP8), so it takes the same path. It has
|
||||
not been run here as downloaded. Setup offers it as a ready-made pack, the size `NVFP4-NVIDIA`.
|
||||
## Expert size and the PCIe share
|
||||
|
||||
## Measured
|
||||
An NVFP4 expert blob is 2.76 MB, against 1.51 MB for IQ2_XS. The cache holds half as many experts, and a miss costs twice
|
||||
the bytes over PCIe. The PCIe share of the missed experts is therefore 0.25 for an NVFP4 pack, not the usual 0.55. At
|
||||
0.55 the copy kernel was 40% of all GPU kernel time (nsys, 300-token decode). Measurements are in
|
||||
[History](#history--older-measurements).
|
||||
|
||||
RTX 5090, 64K context, int8 K/V, `--vram-reserve-mib 1500`:
|
||||
|
||||
| | NVFP4 pack | IQ2_XS, same engine |
|
||||
| --- | ---: | ---: |
|
||||
| experts in RAM | 63.3 GiB | 33.0 GiB |
|
||||
| expert cache slots | ~8,350 | ~17,300 |
|
||||
| decode, 400-token answer | 114-116 tokens/s | 132-139 tokens/s |
|
||||
| a 32K prompt | 3,900 tokens/s | 5,600 tokens/s |
|
||||
|
||||
An NVFP4 expert blob is 2.76 MB, against 1.51 MB for IQ2_XS: the cache holds half as many, and a miss costs twice
|
||||
the bytes over PCIe. The PCIe share of the missed experts is therefore 0.25 for an NVFP4 pack, not the usual 0.55.
|
||||
At 0.55 the copy kernel was 40% of all GPU kernel time (nsys, 300-token decode). A sweep at 262K context gave the
|
||||
following decode rates:
|
||||
|
||||
| PCIe share | short prompt | 32K prompt |
|
||||
| --- | ---: | ---: |
|
||||
| 0.55 | 103 tokens/s | 105 |
|
||||
| 0.40 | 110 | 117 |
|
||||
| 0.25 | 117 | 124 |
|
||||
| 0.10 | 112 | 118 |
|
||||
|
||||
## Measured in Strata (2026-10-13)
|
||||
|
||||
The numbers above are the fork author's, on his RTX 5090 and 128 GB. These were measured here, on the ready-made GPTQ
|
||||
pack (`--family nvfp4 --model NVFP4`).
|
||||
|
||||
Quality, the quant yardstick (`tools/quant_quality.py`, 4,995 teacher-forced positions of code, docs, English, other
|
||||
languages and an agent transcript, against Unsloth's Q8_0; RTX 3060 box):
|
||||
|
||||
| | same top token as Q8_0 | mean KL | KL p99 |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| NVFP4 GPTQ, experts on the CPU | 92.09% | 0.070 | 1.12 |
|
||||
| NVFP4 GPTQ, 1,500 experts on the GPU | 91.79% | 0.069 | 1.07 |
|
||||
| the same with `STRATA_ROUTE_TAIL_SKIP=7` | 88.59% | 0.118 | 1.76 |
|
||||
| NVFP4 GPTQ with the GSQ-RCO IQ4_NL n-gram table instead of the FP8 one | 92.15% | 0.069 | 1.07 |
|
||||
| for comparison, the same tool: Unsloth UD-Q5_K_XL | 94.55% | 0.038 | 0.64 |
|
||||
| Unsloth UD-Q4_K_XL | 93.09% | 0.056 | - |
|
||||
| Unsloth UD-IQ4_XS | 88.91% | 0.101 | 1.60 |
|
||||
| GSQ-RCO IQ3_S | 87.69% | 0.144 | 1.98 |
|
||||
|
||||
The route tail skip moves the answers by more than the quantization itself, so it is off by default for NVFP4
|
||||
experts. `STRATA_ROUTE_TAIL_SKIP=7` turns it on: +7-25% decode on the RTX 3060, +47-55% on the RTX 5070 (64 GB).
|
||||
|
||||
Decode experts (`nvfp4_decode_bench`, 64 distinct experts per call, the share of the card's DRAM bandwidth):
|
||||
|
||||
| tokens per expert | RTX 5070 | RTX 3060 | Tesla P100 |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| 1 | 0.294 ms (89%) | 0.554 ms (89%) | 0.910 ms |
|
||||
| 2 | 0.298 ms (88%) | 0.596 ms | 1.247 ms |
|
||||
| 4 | 0.335 ms (79%) | 0.766 ms | 1.954 ms |
|
||||
|
||||
Each weight is decoded once for all the tokens of a verify window (before the port's kernel work: 2.26 ms on the 3060
|
||||
at 4 tokens), bit for bit the same results.
|
||||
|
||||
Prompt experts on the RTX 5070 (`mmq_expert_bench`, one launch of 16 experts with 625 rows each, gate/up + down): w4a8
|
||||
1.52 ms, w4a4x2 (`STRATA_PREFILL_NVFP4=w4a4x2`) 1.13 ms, w4a4x2 with down too (`STRATA_PREFILL_X2_MIN_K=640`)
|
||||
0.995 ms, UD-IQ4_XS's experts 1.20 ms. First-token KL against the exact FP16 path (24 prompts, 1.5-8K tokens): w4a8
|
||||
0.0068, w4a4x2 0.0056, w4a4x2 with down 0.0044. On a 64 GB PC the prompt waits for the experts read from the SSD,
|
||||
so the modes give the same prompt speed there; w4a8 stays the default until a 96 GB+ RTX 50 PC measures the others.
|
||||
On RTX 20-40 the NVFP4 prompt products are 1.35-1.6x slower than IQ3_S / IQ4_NL ones at 32K: NVFP4's 16-value scales
|
||||
are applied in FP32 after every int8 step, which only Blackwell's FP4 tensor cores avoid.
|
||||
|
||||
Below its RAM need (68 GB of experts plus room) setup runs the pack in the RAM-budget mode (`--resident-budget-gib`,
|
||||
the RAM less 24 GB), as for Unsloth's files. Mapped experts (`--mmap-experts`) on a 64 GB Windows PC filled the RAM.
|
||||
|
||||
Images: the packs keep Qwen's vision tower unquantized, so the original's image encoder reads pictures for them
|
||||
(`--vision`; strata-vision opens the dense GGUF for the vocabulary). Checked on an RTX 5070 and an RTX 3060 against
|
||||
IQ3_XXS's answers on the same picture (a number in the picture read, two shapes with their colours and sides). The
|
||||
encoder on the GPU takes ~1.1 GiB of VRAM: on the 3060 at 8K context the cache held 1,293 experts with images against
|
||||
1,734 without. Pictures are on by default; setup and the app pick the CPU reader under 24 GB of graphics memory and the
|
||||
GPU reader from 24 GB (`--vision cpu` / `--vision gpu` change it).
|
||||
`STRATA_ROUTE_TAIL_SKIP` is off by default for NVFP4 experts, because it moves the answers by more than the quantization
|
||||
itself. `STRATA_ROUTE_TAIL_SKIP=7` turns it on.
|
||||
|
||||
## Where the global scales go
|
||||
|
||||
Each scale multiplies its own projection's FP32 output, as llama.cpp applies `.scale`:
|
||||
`h = silu(s_gate * (G x)) * (s_up * (U x))`, `y = s_down * (D h)`.
|
||||
|
||||
The scales are ~1e-4 each. Folding `s_down` into up is algebraically the same, but it leaves the hidden near 1e-5.
|
||||
The hidden is then rounded to Q8_0 (CPU) or q8_1 (GPU) blocks with an FP16 scale, and `amax / 127` lands deep in
|
||||
FP16's subnormals: one expert's output was off by 2-12% instead of 1.1% (`nvfp4_avx512_parity --expert`). The FP16
|
||||
prompt path had the same problem in its dequantized up weights (~1e-8). Any future fold must be checked against
|
||||
FP16's range.
|
||||
The scales are about 1e-4 each. Folding `s_down` into up is algebraically the same, but it leaves the hidden near 1e-5.
|
||||
The hidden is then rounded to Q8_0 (CPU) or q8_1 (GPU) blocks with an FP16 scale, and `amax / 127` lands deep in FP16's
|
||||
subnormals. One expert's output was off by 2 to 12% instead of 1.1% (`nvfp4_avx512_parity --expert`). The FP16 prompt
|
||||
path had the same problem in its dequantized up weights (about 1e-8). Any future fold must be checked against FP16's
|
||||
range.
|
||||
|
||||
## Prompt path precision
|
||||
|
||||
`STRATA_PREFILL_NVFP4` picks how the batched prompt path multiplies the experts:
|
||||
|
||||
| mode | kernel | activations | one product vs FP64 (`mmq_nvfp4_parity`) |
|
||||
|---|---|---|---|
|
||||
| Mode | Kernel | Activations | One product against FP64 (`mmq_nvfp4_parity`) |
|
||||
| --- | --- | --- | --- |
|
||||
| `w4a8` (default) | llama.cpp MMQ, int8 tensor cores | q8_1, float block scales | 0.53% |
|
||||
| `w4a4` | llama.cpp MMQ, Blackwell FP4 MMA (sm_120a only) | NVFP4 | 8.6% |
|
||||
| `w4a4x2` | Blackwell FP4 MMA | NVFP4 | not in this table; first-token KL in [History](#history--older-measurements) |
|
||||
| `fp16` | dequantize + cuBLAS FP16 GEMM | FP16 | reference |
|
||||
|
||||
`src/prefill/mmq_nvfp4_w4a8.cu` compiles mmq.cuh's int8 NVFP4 path with Blackwell's FP4 MMA hidden from both the
|
||||
device code and the host's tile config, so it is the same on every architecture; decode already multiplies at this
|
||||
precision. The FP4 MMA exists only in the arch-specific target, so `src/prefill/mmq_nvfp4_w4a4.cu` (the FP4 x FP4
|
||||
kernel and its FP4 activation quantizer) is the one unit CMake builds with `12x` -> `12xa` (`strata_mmq_w4a4`);
|
||||
the rest of the engine keeps the architectures as given and runs on sm_121 too. A card without that unit's image
|
||||
(another architecture, or sm_121 against a 120a build) falls back to `w4a8` with a note.
|
||||
`src/prefill/mmq_nvfp4_w4a8.cu` compiles mmq.cuh's int8 NVFP4 path with Blackwell's FP4 MMA hidden from both the device
|
||||
code and the host's tile config, so it is the same on every architecture. Decode already multiplies at this precision.
|
||||
The FP4 MMA exists only in the arch-specific target, so `src/prefill/mmq_nvfp4_w4a4.cu` (the FP4 x FP4 kernel and its FP4
|
||||
activation quantizer) is the one unit CMake builds with `12x` to `12xa` (`strata_mmq_w4a4`). The rest of the engine keeps
|
||||
the architectures as given and runs on sm_121 too. A card without that unit's image (another architecture, or sm_121
|
||||
against a 120a build) falls back to `w4a8` with a note.
|
||||
|
||||
First-token KL divergence against `fp16`, 8 prompts (code and prose, 1K-8K tokens). The noise floor is the same
|
||||
`fp16` path cut into 4096-token chunks instead of 8192: another summation order, same arithmetic.
|
||||
> **Measured on:** first-token KL divergence against `fp16`, 8 prompts (code and prose, 1K to 8K tokens). Hardware and
|
||||
> Strata version not recorded. The noise floor is the same `fp16` path cut into 4096-token chunks instead of 8192.
|
||||
|
||||
| mode | KL mean | KL median | KL max | top-1 agreement | prefill, 4-8K prompts |
|
||||
|---|---|---|---|---|---|
|
||||
| Mode | KL mean | KL median | KL max | Top-1 agreement | Prefill, 4-8K prompts |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| noise floor | 0.00023 | 0.00003 | 0.0016 | 8/8 | - |
|
||||
| `w4a8` | 0.0018 | 0.0010 | 0.0096 | 8/8 | 2,503 tok/s |
|
||||
| `w4a4` | 0.0080 | 0.0027 | 0.040 | 7/8 | 2,904 tok/s |
|
||||
| `fp16` | reference | | | | 1,905 tok/s |
|
||||
|
||||
`w4a8` is the default. It is 4.5x closer to the reference than `w4a4`, at 86% of its speed, and it is the precision
|
||||
every generated token already runs at. `fp16` is the choice when the prompt must be read exactly.
|
||||
`w4a8` is the default. It is 4.5x closer to the reference than `w4a4`, at 86% of its speed, and it is the precision every
|
||||
generated token already runs at. `fp16` is the choice when the prompt must be read exactly.
|
||||
|
||||
`STRATA_DUMP_MOE_INPUT` + `mmq_nvfp4_parity --real` test one product on a real prompt's rows: layer 20, its 12
|
||||
busiest experts, max|x|/rms 4 median and 9 at most. The errors:
|
||||
`STRATA_DUMP_MOE_INPUT` with `mmq_nvfp4_parity --real` tests one product on a real prompt's rows: layer 20, its 12
|
||||
busiest experts, max|x|/rms 4 median and 9 at most.
|
||||
|
||||
| mode | gate/up | down (heavier SwiGLU tails) |
|
||||
| Mode | Gate/up error | Down error (heavier SwiGLU tails) |
|
||||
| --- | ---: | ---: |
|
||||
| `w4a8` | 0.46% | 0.90% |
|
||||
| `w4a4` | 7.2% | 8.4% |
|
||||
| `fp16` | 0.017% | 0.021% |
|
||||
|
||||
On RTX 20 to 40 cards the NVFP4 prompt products are 1.35 to 1.6x slower than IQ3_S / IQ4_NL ones at 32K, because
|
||||
NVFP4's 16-value scales are applied in FP32 after every int8 step, which only Blackwell's FP4 tensor cores avoid.
|
||||
|
||||
## CPU experts
|
||||
|
||||
`src/kernels/cpu/nvfp4_avx512.cpp` computes the CPU pool's NVFP4 rows in 512-bit lanes. Each 64-value block is
|
||||
decoded once for every token of a verify window: `vpdpbusd` on |E2M1| codes, with the sign moved onto the
|
||||
activation. The arithmetic is ggml-cpu's `ggml_vec_dot_nvfp4_q8_0`, with the float additions ordered differently
|
||||
(worst 4.5e-7 relative, `nvfp4_avx512_parity`). `STRATA_NO_NVFP4_512=1` falls back to ggml-cpu.
|
||||
`src/kernels/cpu/nvfp4_avx512.cpp` computes the CPU pool's NVFP4 rows in 512-bit lanes. Each 64-value block is decoded
|
||||
once for every token of a verify window: `vpdpbusd` on |E2M1| codes, with the sign moved onto the activation. The
|
||||
arithmetic is ggml-cpu's `ggml_vec_dot_nvfp4_q8_0`, with the float additions ordered differently (worst 4.5e-7 relative,
|
||||
`nvfp4_avx512_parity`). `STRATA_NO_NVFP4_512=1` falls back to ggml-cpu.
|
||||
|
||||
A CPU without AVX-512 (Zen 1-3, Intel Core 4th-10th and 12th-14th gen, Core Ultra, Xeon E5 v3/v4) runs the same rows in 256-bit lanes
|
||||
(`src/kernels/cpu/nvfp4_avx2.cpp`): ggml's AVX2 arithmetic step for step, so every row is bit-equal to ggml-cpu's
|
||||
(`nvfp4_cpu_parity`, 0 of 1280x9 / 2560x9 rows differ on a Ryzen 5 5600X, a Ryzen 7 255 and a Core Ultra 7 265).
|
||||
`STRATA_NO_NVFP4_256=1` falls back to ggml-cpu. An NVFP4 pack is a native pack (its folder holds `native_experts.txt`),
|
||||
so the engine starts on such a CPU; the AVX-512 requirement belongs to the canonical Q2_0 pack only (#2347: a `--pack`
|
||||
folder without `native_experts.txt` is read as that pack, and the engine now says so before it refuses).
|
||||
A CPU without AVX-512 (Zen 1 to 3, Intel Core 4th to 10th and 12th to 14th gen, Core Ultra, Xeon E5 v3/v4) runs the same
|
||||
rows in 256-bit lanes (`src/kernels/cpu/nvfp4_avx2.cpp`): ggml's AVX2 arithmetic step for step, so every row is bit-equal
|
||||
to ggml-cpu's (`nvfp4_cpu_parity`, 0 of 1280x9 and 2560x9 rows differ on a Ryzen 5 5600X, a Ryzen 7 255 and a Core Ultra
|
||||
7 265). `STRATA_NO_NVFP4_256=1` falls back to ggml-cpu. An NVFP4 pack is a native pack (its folder holds
|
||||
`native_experts.txt`), so the engine starts on such a CPU. The AVX-512 requirement belongs to the canonical Q2_0 pack
|
||||
only; a `--pack` folder without `native_experts.txt` is read as that pack, and the engine says so before it refuses.
|
||||
|
||||
| tokens | ggml-cpu | AVX-512 | speedup |
|
||||
|---|---|---|---|
|
||||
> **Measured on:** one expert's gate+up (1280 rows of 2560, cache-resident), AVX-512 CPU, Strata version not recorded.
|
||||
> The pool itself is bound by DRAM.
|
||||
|
||||
| Tokens | ggml-cpu | AVX-512 | Speedup |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | 0.102 ms | 0.058 ms | 1.76x |
|
||||
| 4 | 0.410 ms | 0.125 ms | 3.29x |
|
||||
| 7 | 0.719 ms | 0.196 ms | 3.66x |
|
||||
|
||||
These are one expert's gate+up (1280 rows of 2560, cache-resident). The pool itself is bound by DRAM.
|
||||
> **Measured on:** the pool in DRAM terms (`nvfp4_avx512_parity --bw THREADS TOKENS`: 1536 experts, 4.2 GB), GB/s of
|
||||
> weights read. Tokens is the tokens an expert serves in one verify window (1 to 2 in decode).
|
||||
|
||||
The pool in DRAM terms (`nvfp4_avx512_parity --bw THREADS TOKENS`: 1536 experts, 4.2 GB, GB/s of weights read; tokens = tokens an
|
||||
expert serves in one verify window, 1-2 in decode):
|
||||
|
||||
| CPU (threads) | kernel | 1 thread, 1 tok | all cores, 1 tok | 2 tok | 4 tok |
|
||||
|---|---|---|---|---|---|
|
||||
| CPU (threads) | Kernel | 1 thread, 1 tok | All cores, 1 tok | 2 tok | 4 tok |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| Ryzen 7 255, Zen 4 (8) | AVX-512 | 15.4 | 58.6 | 57.6 | 47.7 |
|
||||
| | AVX2 rows | 13.9 | 58.9 | 58.1 | 42.1 |
|
||||
| | ggml-cpu | 12.6 | 58.7 | 44.8 | 22.4 |
|
||||
@@ -204,18 +158,25 @@ expert serves in one verify window, 1-2 in decode):
|
||||
| Core Ultra 7 265, DDR5-5600 (20) | AVX2 rows | 16.0 | 74.2 | 73.0 | 60.6 |
|
||||
| | ggml-cpu | 10.3 | 74.1 | 62.7 | 37.1 |
|
||||
|
||||
Without AVX-512 a core reads ~10% slower (13.9 vs 15.4 GB/s), but a pool that reads DRAM runs at the same rate up to two tokens a
|
||||
window and ~12% below at four. The CPU's memory decides, not its vector width.
|
||||
Without AVX-512 a core reads about 10% slower (13.9 against 15.4 GB/s). A pool that reads DRAM runs at the same rate up
|
||||
to two tokens a window and about 12% below at four. The CPU's memory decides, not its vector width.
|
||||
|
||||
## Images
|
||||
|
||||
The packs keep Qwen's vision tower unquantized, so the original's image encoder reads pictures for them (`--vision`;
|
||||
strata-vision opens the dense GGUF for the vocabulary). Pictures are on by default. Setup and the app pick the CPU reader
|
||||
under 24 GB of graphics memory and the GPU reader from 24 GB (`--vision cpu` and `--vision gpu` change it). The encoder on
|
||||
the GPU takes about 1.1 GiB of VRAM.
|
||||
|
||||
## Other GPUs
|
||||
|
||||
Only the optional `w4a4` needs Blackwell. The default `w4a8` is ggml's int8 MMQ (sm_75 and up), and decode uses the
|
||||
engine's own q8_1 kernels, which decode UE4M3/E2M1 in software. The engine was built for sm_75/86/89 as PTX and run
|
||||
on the RTX 5090 with the host answering as that generation. On those builds `mmq_nvfp4_parity` and
|
||||
`nvfp4_expert_gpu_parity` gave results identical to sm_120.
|
||||
engine's own q8_1 kernels, which decode UE4M3/E2M1 in software. The engine was built for sm_75/86/89 as PTX and run on an
|
||||
RTX 5090 with the host answering as that generation. On those builds `mmq_nvfp4_parity` and `nvfp4_expert_gpu_parity` gave
|
||||
results identical to sm_120.
|
||||
|
||||
HIP: the decode kernels use no CUDA-only intrinsics. The NVFP4 MMQ instance and the W4A8 unit are CUDA-only and
|
||||
guarded (`GGML_USE_HIP`). Neither was built for HIP here.
|
||||
HIP: the decode kernels use no CUDA-only intrinsics. The NVFP4 MMQ instance and the W4A8 unit are CUDA-only and guarded
|
||||
(`GGML_USE_HIP`). Neither was built for HIP. Setup does not offer the NVFP4 packs on AMD and Intel.
|
||||
|
||||
## Tests
|
||||
|
||||
@@ -227,26 +188,109 @@ guarded (`GGML_USE_HIP`). Neither was built for HIP here.
|
||||
## NVIDIA's FP4-Local checkpoint
|
||||
|
||||
`nvidia/Qwen3.8-Flash-Next-FP4-Local` (setup size **NVFP4-QAD**) is NVIDIA's own retrained 4-bit model: ModelOpt 0.47
|
||||
safetensors, experts retrained by quantization-aware distillation. Strata reads it by converting it on the user's PC;
|
||||
nothing of NVIDIA's weights is redistributed (licenses: NVIDIA Open Model Agreement and Qwen Community License 1.0).
|
||||
safetensors, experts retrained by quantization-aware distillation. Strata reads it by converting it on the user's PC.
|
||||
Nothing of NVIDIA's weights is redistributed (licenses: NVIDIA Open Model Agreement and Qwen Community License 1.0).
|
||||
|
||||
What the checkpoint holds, and what `tools/nvfp4_local_pack.py` does with it:
|
||||
|
||||
| In the checkpoint | Becomes |
|
||||
| --- | --- |
|
||||
| routed experts: NVFP4 (E2M1, E4M3 scale per 16, F32 scale per tensor); the scales stored in NVIDIA's "CSF" form (`.nvfp4_csf_fixed` U8 + `.nvfp4_csf_exceptions` U32) | the scales are decoded to the plain E4M3 matrix (byte-exact; the format is the one vLLM's loader documents), then repacked exactly like every other NVFP4 pack: `experts.bin` with the same 36-byte super-blocks and the 16-byte scale tail. Checked: 304 complete experts and the whole `experts.bin` (SHA-256) equal the route through a full GGUF |
|
||||
| attention, GDN projections, shared experts, indexer: MXFP8 (E4M3 values, E8M0 scale per 32) | dequantized exactly, then Q8_0 (same 32-value blocks; +0.6% RMS on top of MXFP8's own ~3%), or BF16 for the small projections the BF16 kernels read |
|
||||
| n-gram (PLE) table, 26.8 GiB: NVFP4 rows of 160 values | a one-tensor GGUF of 90-byte rows (80 B codes + 10 E4M3 scales, one F32 for the table), read by the new PLE format `NVFP4` (`strata.ple.format = nvfp4x16`) in `ngram.cpp`; same row size as IQ4_NL |
|
||||
| MTP head: NVFP4 experts, BF16 rest | a Q2_0 draft layer built with `mtp_pack.py` / `mtp_rt.py` (88.7% of drafts accepted over 225 test requests) |
|
||||
| vision tower | not converted. Pictures are read by the original model's picture reader (the shared `mmproj` file that setup downloads, as for the other NVFP4 versions); the dense GGUF supplies the vocabulary |
|
||||
| Routed experts: NVFP4 (E2M1, E4M3 scale per 16, F32 scale per tensor); the scales stored in NVIDIA's "CSF" form (`.nvfp4_csf_fixed` U8 + `.nvfp4_csf_exceptions` U32) | The scales are decoded to the plain E4M3 matrix (byte-exact; the format is the one vLLM's loader documents), then repacked exactly like every other NVFP4 pack: `experts.bin` with the same 36-byte super-blocks and the 16-byte scale tail. Checked: 304 complete experts and the whole `experts.bin` (SHA-256) equal the route through a full GGUF. |
|
||||
| Attention, GDN projections, shared experts, indexer: MXFP8 (E4M3 values, E8M0 scale per 32) | Dequantized exactly, then Q8_0 (same 32-value blocks; +0.6% RMS on top of MXFP8's own ~3%), or BF16 for the small projections the BF16 kernels read. |
|
||||
| N-gram (PLE) table, 26.8 GiB: NVFP4 rows of 160 values | A one-tensor GGUF of 90-byte rows (80 B codes + 10 E4M3 scales, one F32 for the table), read by the PLE format `NVFP4` (`strata.ple.format = nvfp4x16`) in `ngram.cpp`; same row size as IQ4_NL. |
|
||||
| MTP head: NVFP4 experts, BF16 rest | A Q2_0 draft layer built with `mtp_pack.py` / `mtp_rt.py` (88.7% of drafts accepted over 225 test requests). |
|
||||
| Vision tower | Not converted. Pictures are read by the original model's picture reader (the shared `mmproj` file that setup downloads); the dense GGUF supplies the vocabulary. |
|
||||
|
||||
`python tools/nvfp4_local_pack.py prepare --model <checkpoint> --out <dir>` runs every step (setup does this): the n-gram
|
||||
table, the dense GGUF (no experts), the pack (experts written straight from the checkpoint) and the draft layer. It needs
|
||||
about 110 GB of free disk beside the 102 GB download, which can be deleted afterwards. On a Threadripper PRO with the files
|
||||
on NVMe it took about 12 minutes.
|
||||
about 110 GB of free disk beside the 102 GB download, which can be deleted afterwards. On a Threadripper PRO with the
|
||||
files on NVMe it took about 12 minutes.
|
||||
|
||||
Measured on an RTX PRO 5000 72 GB (Blackwell), 128K context, int8 KV, every expert in VRAM (`tools/strata_bench.py`):
|
||||
decode 159 tokens/s (median of chat answers, 137-216), 4K prompt 3,981 tokens/s, 16K 4,813, 32K 4,916 (first token 6.6 s),
|
||||
agent session 151 tokens/s; the engine holds 69 GB of RAM beside 72 GB of VRAM. Quality is in the table below
|
||||
(`tools/quant_quality.py`, set v1, against Unsloth's Q8_0): top-1 90.5%, mean KL 0.085, perplexity 3.95 (Q8_0: 4.05).
|
||||
This is a retrained model, so its distance to the original's Q8_0 includes real weight changes, not only rounding.
|
||||
> **Measured on:** RTX PRO 5000 72 GB (Blackwell), 128K context, int8 KV, every expert in VRAM
|
||||
> (`tools/strata_bench.py`). Strata version not recorded.
|
||||
|
||||
| Measure | Result |
|
||||
| --- | --- |
|
||||
| Decode (median of chat answers) | 159 tok/s (137 to 216) |
|
||||
| Prompt, 4K / 16K / 32K | 3,981 / 4,813 / 4,916 tok/s (first token at 32K: 6.6 s) |
|
||||
| Agent session | 151 tok/s |
|
||||
| Memory | 69 GB of RAM beside 72 GB of VRAM |
|
||||
|
||||
Quality (`tools/quant_quality.py`, set v1, against Unsloth's Q8_0): top-1 90.5%, mean KL 0.085, perplexity 3.95 (Q8_0:
|
||||
4.05). This is a retrained model, so its distance to the original's Q8_0 includes real weight changes, not only rounding.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
### The fork author's measurements
|
||||
|
||||
> **Measured on:** Windows 11, RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB DDR5-5600, 64K context, int8 K/V,
|
||||
> `--vram-reserve-mib 1500`, the fork author's engine.
|
||||
|
||||
| | NVFP4 pack | IQ2_XS, same engine |
|
||||
| --- | ---: | ---: |
|
||||
| Experts in RAM | 63.3 GiB | 33.0 GiB |
|
||||
| Expert cache slots | about 8,350 | about 17,300 |
|
||||
| Decode, 400-token answer | 114 to 116 tok/s | 132 to 139 tok/s |
|
||||
| A 32K prompt | 3,900 tok/s | 5,600 tok/s |
|
||||
|
||||
PCIe share sweep at 262K context (decode tok/s):
|
||||
|
||||
| PCIe share | Short prompt | 32K prompt |
|
||||
| --- | ---: | ---: |
|
||||
| 0.55 | 103 | 105 |
|
||||
| 0.40 | 110 | 117 |
|
||||
| 0.25 | 117 | 124 |
|
||||
| 0.10 | 112 | 118 |
|
||||
|
||||
### Measured in Strata (2026-10-13)
|
||||
|
||||
> **Measured on:** the ready-made GPTQ pack (`--family nvfp4 --model NVFP4`), Strata 0.1.44 era. Quality on an RTX 3060
|
||||
> box with `tools/quant_quality.py`: 4,995 teacher-forced positions of code, docs, English, other languages and an agent
|
||||
> transcript, against Unsloth's Q8_0.
|
||||
|
||||
| | Same top token as Q8_0 | Mean KL | KL p99 |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| NVFP4 GPTQ, experts on the CPU | 92.09% | 0.070 | 1.12 |
|
||||
| NVFP4 GPTQ, 1,500 experts on the GPU | 91.79% | 0.069 | 1.07 |
|
||||
| The same with `STRATA_ROUTE_TAIL_SKIP=7` | 88.59% | 0.118 | 1.76 |
|
||||
| NVFP4 GPTQ with the GSQ-RCO IQ4_NL n-gram table instead of the FP8 one | 92.15% | 0.069 | 1.07 |
|
||||
| For comparison, Unsloth UD-Q5_K_XL | 94.55% | 0.038 | 0.64 |
|
||||
| Unsloth UD-Q4_K_XL | 93.09% | 0.056 | - |
|
||||
| Unsloth UD-IQ4_XS | 88.91% | 0.101 | 1.60 |
|
||||
| GSQ-RCO IQ3_S | 87.69% | 0.144 | 1.98 |
|
||||
|
||||
`STRATA_ROUTE_TAIL_SKIP=7` gave +7 to 25% decode on the RTX 3060 and +47 to 55% on the RTX 5070 (64 GB).
|
||||
|
||||
> **Measured on:** decode experts (`nvfp4_decode_bench`, 64 distinct experts per call), share of the card's DRAM bandwidth
|
||||
> in brackets.
|
||||
|
||||
| Tokens per expert | RTX 5070 | RTX 3060 | Tesla P100 |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| 1 | 0.294 ms (89%) | 0.554 ms (89%) | 0.910 ms |
|
||||
| 2 | 0.298 ms (88%) | 0.596 ms | 1.247 ms |
|
||||
| 4 | 0.335 ms (79%) | 0.766 ms | 1.954 ms |
|
||||
|
||||
Each weight is decoded once for all the tokens of a verify window, bit for bit the same results. Before the kernel work,
|
||||
the RTX 3060 took 2.26 ms at 4 tokens.
|
||||
|
||||
> **Measured on:** prompt experts on an RTX 5070 (`mmq_expert_bench`, one launch of 16 experts with 625 rows each,
|
||||
> gate/up + down); first-token KL against the exact FP16 path over 24 prompts of 1.5K to 8K tokens.
|
||||
|
||||
| Mode | Time | First-token KL |
|
||||
| --- | ---: | ---: |
|
||||
| `w4a8` (default) | 1.52 ms | 0.0068 |
|
||||
| `w4a4x2` (`STRATA_PREFILL_NVFP4=w4a4x2`) | 1.13 ms | 0.0056 |
|
||||
| `w4a4x2` with down too (`STRATA_PREFILL_X2_MIN_K=640`) | 0.995 ms | 0.0044 |
|
||||
| UD-IQ4_XS's experts | 1.20 ms | - |
|
||||
|
||||
On a 64 GB PC the prompt waits for the experts read from the SSD, so the modes give the same prompt speed there. `w4a8`
|
||||
stays the default until a 96 GB or larger RTX 50 PC measures the others.
|
||||
|
||||
> **Measured on:** RTX 5070 and RTX 3060, images with an NVFP4 pack against IQ3_XXS's answers on the same picture (a
|
||||
> number in the picture read, two shapes with their colours and sides).
|
||||
|
||||
The answers matched. On the 3060 at 8K context the cache held 1,293 experts with images (GPU encoder) against 1,734
|
||||
without.
|
||||
|
||||
Mapped experts (`--mmap-experts`) on a 64 GB Windows PC filled the RAM, which is why setup uses the RAM-budget mode below
|
||||
the RAM need.
|
||||
+73
-63
@@ -1,12 +1,22 @@
|
||||
# NVIDIA Tesla V100 / Titan V (Volta, sm_70): the community build
|
||||
# NVIDIA Tesla V100 / Titan V (Volta, sm_70)
|
||||
|
||||
Strata's ready-made engine and the support matrix start at compute capability 7.5 (RTX 20). A V100 (7.0) runs the same
|
||||
engine compiled for it, through the experimental build that issue #236 added. It is **community-tested, not part of the
|
||||
supported cards**: no ready-made engine is published for it and the numbers below come from one machine.
|
||||
This page is for people running Strata on a V100 or Titan V. Volta uses the experimental CUDA 12 engine, which current releases
|
||||
provide as an engine component through the app installer or the Terminal ([INSTALL.md](INSTALL.md)). CUDA 13 cannot run Volta.
|
||||
How setup selects the CUDA 12 engine and the driver requirements are in [OLDER_GPUS.md](OLDER_GPUS.md). Speeds below were
|
||||
reported on the named machines. They are not a guarantee for another V100 setup.
|
||||
|
||||
## Build
|
||||
> **On this page:** [Quick start](#quick-start) · [What runs on Volta](#what-runs-on-volta) · [Opt-in switches](#opt-in-switches) ·
|
||||
> [Checking the kernels](#checking-and-benchmarking-the-volta-kernels) · [Limits](#limits) · [History](#history--older-measurements)
|
||||
|
||||
CUDA 13 dropped Volta, so the build needs a **CUDA 12.x** toolkit (12.8 was used) and a host GCC that toolkit accepts.
|
||||
## Quick start
|
||||
|
||||
1. Install the app or Terminal for your system ([INSTALL.md](INSTALL.md)). Setup picks the CUDA 12 engine when the model runs on
|
||||
a Volta card.
|
||||
2. To choose it by hand from a source checkout: `START-HERE.bat --setup --gpu 0 --cuda 12` (Linux: `./setup.sh`).
|
||||
|
||||
### Build from source
|
||||
|
||||
CUDA 13 dropped Volta, so the build needs a CUDA 12.x toolkit (12.8 was used) and a host GCC that toolkit accepts.
|
||||
|
||||
```sh
|
||||
cmake -S . -B build -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF \
|
||||
@@ -14,53 +24,49 @@ cmake -S . -B build -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF \
|
||||
cmake --build build --target strata -j
|
||||
```
|
||||
|
||||
Setup does the same by itself for a model on a V100 (the experimental CUDA 12 engine, in `engine-cuda12/`: ready-made
|
||||
on Windows, compiled with a CUDA 12.x toolkit on Linux; [OLDER_GPUS.md](OLDER_GPUS.md)). For a PC with a V100 and a newer card, a build for both architectures (`-DCMAKE_CUDA_ARCHITECTURES="70;86"`) is the
|
||||
natural choice (a community report in issue #509 ran an RTX 3080 beside a V100 that way; it was not repeated here). On the
|
||||
measuring PC a Quadro RTX 4000 (sm_75) sits beside the V100s and was hidden with `CUDA_VISIBLE_DEVICES` because the sm_70-only
|
||||
build has no code for it.
|
||||
A source checkout can fetch or build its CUDA 12 engine; `--build` requests compilation with the toolkit (see
|
||||
[OLDER_GPUS.md](OLDER_GPUS.md)). For a PC with a V100 and a newer card, build for both architectures
|
||||
(`-DCMAKE_CUDA_ARCHITECTURES="70;86"`). A sm_70-only build has no code for a card such as a Quadro RTX 4000 (sm_75); hide it with
|
||||
`CUDA_VISIBLE_DEVICES`.
|
||||
|
||||
## What runs on Volta
|
||||
|
||||
The flag only lowers the floor (CMake and the runtime device check) and replaces two intrinsics Volta lacks. It does not
|
||||
select "old" kernels everywhere: which kernel runs is decided per architecture when the engine is built.
|
||||
The flag lowers the floor (CMake and the runtime device check) and replaces two intrinsics Volta lacks. Which kernel runs is
|
||||
decided per architecture when the engine is built.
|
||||
|
||||
| Part of the prompt path | On sm_70 |
|
||||
| --- | --- |
|
||||
| MoE experts | opt-in (`STRATA_PF_WMMA=1`, only when every visible card is compute capability 7+; MMQ otherwise): `gemm_iq_f16_grouped`: a group of experts per launch on the FP16 tensor cores, the weights dequantized in shared memory (+10% prompt speed against MMQ's `dp4a` kernels, [bench](../bench/results/2026-10-07-v100-prompt-experts/README.md); FP16 activations instead of MMQ's q8_1, so the answers differ) |
|
||||
| MoE experts | MMQ by default. Opt-in `STRATA_PF_WMMA=1` (only when every visible card is compute capability 7 or newer): `gemm_iq_f16_grouped`, a group of experts per launch on the FP16 tensor cores. FP16 activations instead of MMQ's q8_1, so answers differ. |
|
||||
| Dense projections (dequantized weights) | FP16 tensor-core GEMMs (cuBLAS / CUTLASS `s884`) |
|
||||
| BF16 projections (hyper-connection, router, indexer, ...) | converted to FP16 and run on the FP16 tensor cores (#655, #540; `STRATA_BF16_TC=0`: cuBLAS BF16, an FP32 SIMT kernel on Volta) |
|
||||
| QSA attention for decode and verify windows (and prompts with `STRATA_PROMPT_ATTN_OLD=1`) | #540's kernel (fewer shuffles, bit-exact; `STRATA_ATTN_PRE75=0`: the one other cards run) |
|
||||
| BF16 projections (hyper-connection, router, indexer) | converted to FP16 and run on the FP16 tensor cores (`STRATA_BF16_TC=0`: cuBLAS BF16, an FP32 SIMT kernel on Volta) |
|
||||
| QSA attention for decode and verify windows | the #540 kernel, fewer shuffles, bit-exact (`STRATA_ATTN_PRE75=0`: the one other cards run) |
|
||||
| QSA prompt attention, int8 / FP16 / K8V4 KV | `prompt_attn_v70_kernel` on `mma.m8n8k4` (`STRATA_PROMPT_ATTN_OLD=1`: the decode kernel, one query at a time) |
|
||||
| QSA prompt attention, Q4_0 KV | the decode kernel |
|
||||
| QSA block scores | the warp kernel (the tensor-core scorer needs sm_80) |
|
||||
|
||||
| Part of a decode / verify window | On sm_70 |
|
||||
| Part of a decode or verify window | On sm_70 |
|
||||
| --- | --- |
|
||||
| Routed experts in VRAM | gfx906's expert mode 8: the codebook grid and the group's activations in shared memory, SwiGLU fused (`STRATA_EXP_MODE=0`: the CUDA layout other cards run) |
|
||||
| Dense 2-4 column GEMVs and the head | the interleaved `native_mmvq_il` with a Volta rows table (`STRATA_MMVQ_IL=0`: `native_mmvq`) |
|
||||
| Hyper-connection read, up projection | gfx906's latency-hidden up kernel (`STRATA_GR_FAST=0`: the plain one) |
|
||||
| Routed experts in VRAM | expert mode 8 (codebook grid and the group's activations in shared memory, SwiGLU fused). Opt-in, see below. |
|
||||
| Dense 2-4 column GEMVs and the head | the interleaved `native_mmvq_il` with a Volta rows table. Opt-in, see below. |
|
||||
| Hyper-connection read, up projection | the latency-hidden up kernel. Opt-in, see below. |
|
||||
|
||||
All three are bitwise the kernels they replace; on a V100-SXM2 they take a verify window's GPU work from 22.8 to 21.2 ms
|
||||
([bench/results/2026-10-07-v100-decode-kernels](../bench/results/2026-10-07-v100-decode-kernels/README.md)).
|
||||
## Opt-in switches
|
||||
|
||||
## Measured
|
||||
None of these is on by default. Each was measured by its contributor on a V100, not by the maintainers.
|
||||
|
||||
One V100-PCIE-32GB, i9-7960X, Unsloth `UD-IQ4_XS` (not a setup model), int8 KV, MTP `--spec 2`; prompt read speed of random-word prompts, tokens/s,
|
||||
with and without the Volta attention kernel (`STRATA_PROMPT_ATTN_OLD=1`):
|
||||
| Switch | What it does |
|
||||
| --- | --- |
|
||||
| `STRATA_SM70_TABLE=1` | Turns on the three decode kernels in the second table above together. Bitwise the default kernels' output. Individual switches `STRATA_EXP_MODE`, `STRATA_MMVQ_IL` and `STRATA_GR_FAST` still apply on top. |
|
||||
| `STRATA_FUSE_DECODE=1` | The decode GEMV / MoE fusions of #1097. Bitwise the unfused kernels. |
|
||||
| `STRATA_PF_WMMA=1` | The FP16 tensor-core prompt experts; also takes IQ4_NL down projections. |
|
||||
| `STRATA_PF_WMMA2=1` | A second prompt-expert kernel for IQ2_S gate/up and IQ4_NL down (#2356). |
|
||||
| `STRATA_ARGMAX_ROWS_SM70=1` | The multi-block greedy pick on sm_7x (#2357), bitwise the one-block pick. |
|
||||
| `STRATA_GR_NORM_FAST=1` | The register-split hyper-connection norm (#2358). |
|
||||
| `STRATA_Q8_GATHER_WIDE=0` | Turns off the 8-wide int8 KV gather (#1095) that the experimental build uses below compute capability 7.5. |
|
||||
| `STRATA_GDN_REC_WARP=2` | The chain-split DeltaNet recurrence of the prompt (#1098). Changes the FP32 summation order, so the default stays the pipelined kernel. `1` is the bit-exact sync-free variant; `STRATA_GDN_CHUNK=1` the chunked path. |
|
||||
|
||||
| Prompt tokens | old attention path | `prompt_attn_v70_kernel` |
|
||||
| --- | ---: | ---: |
|
||||
| 7,194 | 1,010 | 1,164 |
|
||||
| 28,650 | 1,063 | 1,251 |
|
||||
| 114,338 | 973 | 1,123 |
|
||||
|
||||
Decode was 38-51 tok/s in every run and is not affected. The profile, the parity numbers, the needle test and the limits are in
|
||||
[bench/results/2026-10-03-v100-prompt-attn](../bench/results/2026-10-03-v100-prompt-attn/README.md).
|
||||
|
||||
What the kernel does and does not guarantee: against an FP64 reference its error is about 2-3e-6 (output scale 3.6; the FP32 kernel it replaces: 2e-6; 5e-6 before the hi and lo halves got separate accumulator chains); 15 of 15
|
||||
needles were found; greedy output was identical on the 28,650- and 114,338-token prompts and differed late in the answer on the 7,194-token one (another
|
||||
summation order).
|
||||
Volta's tensor cores take FP16 only. A model's BF16 weights are converted, and a weight outside FP16's range would saturate (none
|
||||
did in the check in the history section).
|
||||
|
||||
## Checking and benchmarking the Volta kernels
|
||||
|
||||
@@ -69,37 +75,41 @@ cmake -S . -B build -DSTRATA_PARITY_PROMPT_ATTN=ON ... && cmake --build build --
|
||||
CUDA_VISIBLE_DEVICES=0 ./build/qsa_prompt_attn_parity 32768 2048 5 # synthetic, no model; exit 0 = PASS
|
||||
```
|
||||
|
||||
It compares the tensor-core attention with the FP32 kernel and an FP64 host reference and times both. The Q4_0 cases are skipped below
|
||||
sm_80.
|
||||
It compares the tensor-core attention with the FP32 kernel and an FP64 host reference and times both. The Q4_0 cases are skipped
|
||||
below sm_80.
|
||||
|
||||
## Limits
|
||||
|
||||
- A layer split over two V100 was not measured with this kernel. The cards work in turn on a single request, so a split adds expert-cache room, not prompt speed.
|
||||
- sm_60 (Pascal) is covered by the same flag, and community runs exist: two Tesla P40 with a layer split (#1028, bench
|
||||
[2026-10-05-community-2x-p40](../bench/results/2026-10-05-community-2x-p40/README.md)), a Tesla P100 alone and with an RTX 2070 SUPER as
|
||||
the expert-helper card (#1069: 30.5 tok/s alone, 35.8-38.4 with the helper), and a V100 + P100 pair (#1079: +6-11% decode from 0.1.39 to
|
||||
0.1.40 with the P100 as helper cache). One reporter's numbers each; this page's own kernel measurements are V100 only.
|
||||
- Volta's tensor cores take FP16 only: a model's BF16 weights are converted, and a weight outside FP16's range would saturate (none did in
|
||||
the check above).
|
||||
- A layer split over two V100 was not measured with this kernel. The cards work in turn on a single request, so a split adds
|
||||
expert-cache room, not prompt speed.
|
||||
- Pascal (sm_60) is covered by the same flag. Community runs exist for it; this page's own kernel measurements are V100 only.
|
||||
|
||||
## The decode kernels measured on a V100 (opt-in: `STRATA_SM70_TABLE=1`)
|
||||
## History / older measurements
|
||||
|
||||
The interleaved 2-4 column matvec with a Volta rows table, expert mode 8 and the latency-hidden norm/up (bitwise the default
|
||||
kernels' output, GPU work per verify window 22.8 -> 21.2 ms on a V100-SXM2 in the contributor's measurement) are used on
|
||||
sm_70 only when `STRATA_SM70_TABLE=1` is set. They become the default once the contributor has confirmed them on a V100 with
|
||||
the 0.1.41 code. Without the variable sm_70 runs what 0.1.40.3 ran. The single switches (`STRATA_EXP_MODE`, `STRATA_MMVQ_IL`,
|
||||
`STRATA_GR_FAST`) still apply on top.
|
||||
**Prompt attention kernel.** Measured on: one V100-PCIE-32GB, i9-7960X, Unsloth UD-IQ4_XS, int8 KV, MTP `--spec 2`, random-word
|
||||
prompts, prompt read speed in tok/s with and without the Volta kernel (`STRATA_PROMPT_ATTN_OLD=1`). Decode was 38-51 tok/s in
|
||||
every run and is not affected. Report: [bench/results/2026-10-03-v100-prompt-attn](../bench/results/2026-10-03-v100-prompt-attn/README.md).
|
||||
|
||||
## More opt-in switches (CUDA 12 experimental build)
|
||||
| Prompt tokens | old attention path | `prompt_attn_v70_kernel` |
|
||||
| --- | ---: | ---: |
|
||||
| 7,194 | 1,010 | 1,164 |
|
||||
| 28,650 | 1,063 | 1,251 |
|
||||
| 114,338 | 973 | 1,123 |
|
||||
|
||||
None of these is on by default. Each was measured by its contributor on a V100, not by the maintainers.
|
||||
Against an FP64 reference the kernel's error is about 2-3e-6 (the FP32 kernel it replaces: 2e-6). 15 of 15 needles were found.
|
||||
Greedy output was identical on the 28,650- and 114,338-token prompts and differed late in the answer on the 7,194-token one.
|
||||
|
||||
| Switch | What it does | Contributor's number |
|
||||
| --- | --- | --- |
|
||||
| `STRATA_FUSE_DECODE=1` | the decode GEMV / MoE operator fusions of #1097 (gate/up + SwiGLU in one kernel, SwiGLU + quantizer in one kernel, all activation images of a vector in one pass, two BF16 projections in one kernel); bitwise the unfused kernels | 4-16 us per token saved, not visible end to end |
|
||||
| `STRATA_PF_WMMA=1` | the FP16 tensor-core prompt experts (above); also takes IQ4_NL down projections (K=640, #1771) | +10% prompt speed (#1401) |
|
||||
| `STRATA_PF_WMMA2=1` | a second prompt-expert kernel for IQ2_S gate/up and IQ4_NL down (#2356) | +2.5-4.3% prompt speed |
|
||||
| `STRATA_ARGMAX_ROWS_SM70=1` | the multi-block greedy pick on sm_7x (#2357), bitwise the one-block pick | +0.7-0.8% decode |
|
||||
| `STRATA_GR_NORM_FAST=1` | the register-split hyper-connection norm on CUDA (#2358) | +0.5-0.6% decode |
|
||||
| `STRATA_Q8_GATHER_WIDE=0` | turns off the 8-wide int8 KV gather (#1095) that the experimental build uses below compute capability 7.5 | +11% on the gather kernel |
|
||||
| `STRATA_GDN_REC_WARP=2` | the chain-split DeltaNet recurrence of the prompt on a V100 (#1098); changes the summation order (FP32 level), so the default stays the pipelined kernel; `1` is the bit-exact sync-free variant, `STRATA_GDN_CHUNK=1` the chunked path | +6-9% on the recurrence |
|
||||
**Other measurements**
|
||||
|
||||
| Measured on | Result |
|
||||
| --- | --- |
|
||||
| V100-SXM2, contributor's measurement | decode kernels (`STRATA_SM70_TABLE=1`): GPU work per verify window 22.8 -> 21.2 ms ([report](../bench/results/2026-10-07-v100-decode-kernels/README.md)) |
|
||||
| V100, `STRATA_PF_WMMA=1` | prompt +10% against MMQ ([report](../bench/results/2026-10-07-v100-prompt-experts/README.md)) |
|
||||
| V100, `STRATA_PF_WMMA2=1` | prompt +2.5-4.3% |
|
||||
| V100, `STRATA_ARGMAX_ROWS_SM70=1` / `STRATA_GR_NORM_FAST=1` | decode +0.7-0.8% / +0.5-0.6% |
|
||||
| V100, `STRATA_Q8_GATHER_WIDE=0` | +11% on the gather kernel |
|
||||
| V100, `STRATA_GDN_REC_WARP=2` | +6-9% on the recurrence |
|
||||
| V100, `STRATA_FUSE_DECODE=1` | 4-16 us per token saved, not visible end to end |
|
||||
| Tesla P100 alone and with an RTX 2070 SUPER as expert helper (#1069) | 30.5 tok/s alone, 35.8-38.4 with the helper |
|
||||
| Two Tesla P40, layer split (#1028) | [report](../bench/results/2026-10-05-community-2x-p40/README.md) |
|
||||
| V100 + P100 pair (#1079) | +6-11% decode from engine 0.1.39 to 0.1.40 with the P100 as helper cache |
|
||||
+224
-224
@@ -1,51 +1,61 @@
|
||||
# Older GPUs (experimental)
|
||||
|
||||
Strata's supported cards are NVIDIA RTX 20 / 30 / 40 / 50 and the AMD cards in [AMD_HIP.md](AMD_HIP.md). The cards
|
||||
below run through **opt-in, experimental** paths that community members wrote and measured on their own machines. The
|
||||
maintainers have one of these cards, a Tesla P100, and test on it; they have none of the others: each path is
|
||||
compile-checked and unit-tested here, and the ready-made engines and their output stay exactly as they were. Numbers are
|
||||
the reporters' own, on one machine each, unless the text says the maintainers measured them.
|
||||
This page is for people with an NVIDIA card older than the RTX 20 series, an older AMD card, or a Turing card they want to tune.
|
||||
For a normal install use the app installer for your hardware or the [Terminal](INSTALL.md#the-terminal). Pascal and Volta use the
|
||||
CUDA 12 engine package. The `--cuda`, `--gpus`, `START-HERE.bat` and `UPDATE.bat` examples below are
|
||||
[source-setup](SOURCE_INSTALL.md) options, not Terminal launch flags.
|
||||
|
||||
Strata's supported cards are NVIDIA RTX 20 / 30 / 40 / 50 and the AMD cards in [AMD_HIP.md](AMD_HIP.md). The cards below run
|
||||
through opt-in, experimental paths that community members wrote and measured. The maintainers own one of these cards, a Tesla
|
||||
P100, and test on it. Every other path is compile-checked and unit-tested, and the ready-made engines and their output are
|
||||
unchanged. Numbers are the reporters' own, on one machine each, unless the text says the maintainers measured them.
|
||||
|
||||
> **On this page:** [Support matrix](#support-matrix) · [CUDA 12 engine](#nvidia-the-cuda-12-engine) ·
|
||||
> [Tesla P100](#tesla-p100-sm_60) · [GP100 Q8_0 kernel](#pascal-gp100-a-q8_0-decode-kernel-opt-in) ·
|
||||
> [Tesla P40](#tesla-p40-sm_61-what-was-measured-and-changed) · [Turing switches](#quadro-rtx-8000-and-other-turing-cards) ·
|
||||
> [AMD](#amd-building-gfx906-and-gfx1012) · [History](#history--older-measurements)
|
||||
|
||||
## Support matrix
|
||||
|
||||
### NVIDIA
|
||||
|
||||
| Cards | Compute capability | How it runs | What is different on it | Reported |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Tesla P100 | 6.0 | the CUDA 12 engine | `__dp4a` emulated (bit-exact); BF16 projections through fp32; opt-in: `STRATA_Q8_SM60=1` runs the Q8_0 dense projections and head on a GP100 kernel ([below](#pascal-gp100-a-q8_0-decode-kernel-opt-in)) | 2x P100, IQ3_S, engine 0.1.39: prompt 526-532 tok/s, decode 28-32 tok/s at 128K prompt tokens, 3 of 3 needle checks at 122K ([report](../bench/results/2026-10-06-community-2x-p100/README.md), #1157); PH402 (4 GP100 dies), IQ3_XXS: decode 28.3 -> 38.9 tok/s with `STRATA_Q8_SM60=1` and a Q8_0 dense shard |
|
||||
| Tesla P40 / P4, GTX 10 series | 6.1 | the CUDA 12 engine | BF16 projections through fp32 (cuBLAS has no BF16 GEMM there, #395); the dense weights dequantized by a wide kernel, expert mode 8 and its own interleaved-matvec table; opt-in `STRATA_DENSE_F32=1`: the large dense GEMMs as `cublasSgemm` ([below](#tesla-p40-sm_61-what-was-measured-and-changed)) | P40, IQ3_S, engine 0.1.30: prompt 217-374 tok/s, decode 30-33 tok/s (#395); 0.1.43.1 with the sm_61 changes: prompt 255 tok/s at 565 tokens (was 205), decode 37-38 tok/s (was 34-35); with `STRATA_DENSE_F32=1` 464-474 tok/s at 7K tokens (was 360-432) |
|
||||
| Tesla V100, Titan V | 7.0 | the CUDA 12 engine | BF16 projections on the FP16 tensor cores (#655, #540); the prompt attention on `mma.m8n8k4` (#600); a leaner attention kernel (#540) | V100-PCIE-32GB, UD-IQ4_XS: prompt 1,123-1,251 tok/s (#600); V100 32GB, IQ2_XS: prompt +22% from #540 |
|
||||
| RTX 20 (Turing) | 7.5 | **supported**, the ready-made engine | opt-in: `STRATA_BF16_TC=1` runs the BF16 projections on the FP16 tensor cores | RTX 2080 Ti, Q2_0: prompt +15-18% (#655) |
|
||||
| Cards | Compute capability | How it runs | What is different on it |
|
||||
| --- | --- | --- | --- |
|
||||
| Tesla P100 | 6.0 | the CUDA 12 engine | `__dp4a` emulated (bit-exact); BF16 projections through fp32; seven scheduling switches on by default ([below](#tesla-p100-sm_60)); opt-in `STRATA_Q8_SM60=1` ([below](#pascal-gp100-a-q8_0-decode-kernel-opt-in)) |
|
||||
| Tesla P40 / P4, GTX 10 series | 6.1 | the CUDA 12 engine | BF16 projections through fp32 (cuBLAS has no BF16 GEMM there); dense weights dequantized by a wide kernel; expert mode 8 and its own interleaved-matvec table; opt-in `STRATA_DENSE_F32=1` ([below](#tesla-p40-sm_61-what-was-measured-and-changed)) |
|
||||
| Tesla V100, Titan V | 7.0 | the CUDA 12 engine | BF16 projections on the FP16 tensor cores; prompt attention on `mma.m8n8k4`; a leaner attention kernel. See [NVIDIA_V100.md](NVIDIA_V100.md) |
|
||||
| RTX 20 (Turing) | 7.5 | **supported**, the ready-made engine | opt-in `STRATA_BF16_TC=1` runs the BF16 projections on the FP16 tensor cores |
|
||||
|
||||
A card needs enough VRAM to be useful: 12 GB or more is recommended, as for every card (a 2 GB GT 1030 is compute
|
||||
capability 6.1 too, and cannot hold any of the model).
|
||||
A card needs enough VRAM to be useful. 12 GB or more is recommended, as for every card. A 2 GB GT 1030 is compute capability 6.1
|
||||
too and cannot hold any of the model.
|
||||
|
||||
### AMD
|
||||
|
||||
| Cards | Architecture | How it runs | Reported |
|
||||
| --- | --- | --- | --- |
|
||||
| RX 6800 / 6900 series | gfx1030 | setup (`--backend hip`), unvalidated; #540's attention kernel is the default there, with 8 cells per step and DPP lane exchanges (bit-exact; `STRATA_ATTN_PRE75=0` runs the standard kernel) | [AMD_HIP.md](AMD_HIP.md#rdna2-gfx1030); RX 6900 XT, IQ3_S: prompts +4-6% alone, +7-12% with #835 ([bench/results/2026-10-04-rdna2-pre75-attention](../bench/results/2026-10-04-rdna2-pre75-attention/README.md)) |
|
||||
| RX 6700 XT | gfx1031 | setup (`--backend hip`), unvalidated (#524) | used daily by its reporter, one card |
|
||||
| RX 5500 XT (RDNA1) | gfx1012 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1012` (HIP 5.7 or 7) | 8 GB card, IQ3_S, 8K prompt: 15.3 tok/s decode (#442) |
|
||||
| RX 5700 XT (RDNA1) | gfx1010 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1010`, on ROCm 7.14's `gfx101X-dgpu` wheels; needs `ROCR_VISIBLE_DEVICES=0` in a PC that also has an AMD iGPU | 8 GB card, Coder IQ1_M, 32K context: prompt 115 / 144 / 149 tok/s and decode 18.0 / 20.3 / 23.3 tok/s at 4K / 16K / 30K prompt tokens, 6 of 6 needle checks at 8K and 30K, 61 of 61 ctest on the card, 607 of 12,288 experts in VRAM |
|
||||
| RX 5700 and the 6 GB RX 5600 (RDNA1) | gfx1010 | as above | the same Navi 10 silicon as the RX 5700 XT, so the same build; a 6 GB card leaves ~50 expert slots, expect the RX 5500 XT's range, not this one's |
|
||||
| Radeon PRO V520 / Pro 5600M (RDNA1) | gfx1011 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1011` | untested on hardware: it is the same RDNA1 ISA as gfx1010 (Navi 12), and the engine builds for it with 0 errors on the same wheels |
|
||||
| Instinct MI50 / MI60, Radeon VII | gfx906 (wave64) | built by hand: `-DSTRATA_HIP_GFX906=ON` | 2x MI50, Coder IQ1_M, 128K context: decode 50.1 / 47.8 / 45.7 tok/s at 4K / 32K / 128K prompt tokens, prompt ~520 tok/s (#677) |
|
||||
| Cards | Architecture | How it runs |
|
||||
| --- | --- | --- |
|
||||
| RX 6800 / 6900 series | gfx1030 | setup (`--backend hip`), unvalidated. The #540 attention kernel is the default there (`STRATA_ATTN_PRE75=0` runs the standard kernel). See [AMD_HIP.md](AMD_HIP.md#rdna2-gfx1030) |
|
||||
| RX 6700 XT | gfx1031 | setup (`--backend hip`), unvalidated; used daily by its reporter on one card |
|
||||
| RX 5500 XT (RDNA1) | gfx1012 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1012` (HIP 5.7 or 7) |
|
||||
| RX 5700 XT (RDNA1) | gfx1010 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1010` on ROCm 7.14's `gfx101X-dgpu` wheels; needs `ROCR_VISIBLE_DEVICES=0` on a PC that also has an AMD iGPU |
|
||||
| RX 5700, 6 GB RX 5600 (RDNA1) | gfx1010 | as above; a 6 GB card leaves about 50 expert slots |
|
||||
| Radeon PRO V520 / Pro 5600M (RDNA1) | gfx1011 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1011`; builds with 0 errors, untested on hardware |
|
||||
| Instinct MI50 / MI60, Radeon VII | gfx906 (wave64) | built by hand: `-DSTRATA_HIP_GFX906=ON` |
|
||||
|
||||
## NVIDIA: the CUDA 12 engine
|
||||
|
||||
CUDA 13 dropped Pascal and Volta: it cannot compile for them. Setup therefore keeps a **second engine**, built with
|
||||
CUDA 12.9 and `-DSTRATA_EXPERIMENTAL_SM60=ON`, in its own folder (`engine-cuda12\`, beside `engine\`). One engine runs
|
||||
per model, so the choice is made per model, by the oldest card that model runs on:
|
||||
CUDA 13 dropped Pascal and Volta and cannot compile for them. Setup therefore keeps a second engine, built with CUDA 12.9 and
|
||||
`-DSTRATA_EXPERIMENTAL_SM60=ON`, in its own folder (`engine-cuda12\`, beside `engine\`). One engine runs per model, so the
|
||||
choice is made per model, by the oldest card that model runs on:
|
||||
|
||||
- **Every card the model uses is RTX 20 or newer:** the ready-made CUDA 13 engine, as always.
|
||||
- **A card is Pascal or Volta:** the CUDA 12 engine. Setup says so (`CUDA 12: sm_70 is older than CUDA 13 supports
|
||||
...`). On Windows it downloads `strata-windows-x64-cuda12.zip` with NVIDIA's CUDA 12 libraries (from pip, like the
|
||||
CUDA 13 ones); on Linux, or with `--build`, it compiles the engine with a CUDA 12.x toolkit.
|
||||
- Every card the model uses is RTX 20 or newer: the ready-made CUDA 13 engine.
|
||||
- A card is Pascal or Volta: the CUDA 12 engine. Setup says so (`CUDA 12: sm_70 is older than CUDA 13 supports ...`).
|
||||
|
||||
Release builds deliver engine components through the signed feed (`strata update`). For a source checkout, `--build` compiles with
|
||||
a CUDA 12.x toolkit ([source installation](SOURCE_INSTALL.md)).
|
||||
|
||||
### Opting in
|
||||
|
||||
An older card is used only when you choose it; a PC with a newer card keeps recommending the newer one.
|
||||
An older card is used only when you choose it. A PC with a newer card keeps recommending the newer one.
|
||||
|
||||
| You | Setup |
|
||||
| --- | --- |
|
||||
@@ -53,22 +63,21 @@ An older card is used only when you choose it; a PC with a newer card keeps reco
|
||||
| name the card: `START-HERE.bat --setup --gpu 1`, or `--gpus 0,1` with a newer card | uses it; the model gets the CUDA 12 engine |
|
||||
| `--cuda 12` (or `STRATA_CUDA=12`) | the CUDA 12 engine for this model, on any card |
|
||||
| `--cuda 13` | the CUDA 13 engine even with an older card (a warning: it has no code for that card) |
|
||||
| `STRATA_EXPERIMENTAL_SM60=1` | the older cards are listed as usable (the setting from #295 still works) |
|
||||
| `STRATA_EXPERIMENTAL_SM60=1` | the older cards are listed as usable |
|
||||
|
||||
The choice is kept in the model's config (`"cuda": 12`): its starts and `UPDATE.bat` keep it, and other models keep
|
||||
their own engine. Setting the model up again chooses again (by its cards; add `--cuda 12` to keep a forced choice). A Pascal / Volta card added to a model at a start (`--gpus`) moves that model to the
|
||||
CUDA 12 engine.
|
||||
The choice is kept in the model's config (`"cuda": 12`). Starts and `UPDATE.bat` keep it, and other models keep their own engine.
|
||||
Setting the model up again chooses again (add `--cuda 12` to keep a forced choice). A Pascal or Volta card added to a model at a
|
||||
start (`--gpus`) moves that model to the CUDA 12 engine.
|
||||
|
||||
`--cuda 12` is also the way to run Strata with an NVIDIA driver older than 580: CUDA 12 needs 528 or newer on Windows
|
||||
(527.41, NVIDIA's minor-version compatibility) and 525 on Linux; setup's "driver too old" stop says so. Such old
|
||||
drivers were not tested here.
|
||||
`--cuda 12` is also the way to run Strata with an NVIDIA driver older than 580. The CUDA 12 engine needs driver 528 or newer on
|
||||
Windows and 525 on Linux (`CUDA12_MIN_DRIVER` in `setup.py`); setup's "driver too old" stop says so. The Docker `cuda12` image
|
||||
asks for 575 ([DOCKER.md](DOCKER.md)). Drivers that old were not tested here.
|
||||
|
||||
### Mixed cards
|
||||
|
||||
A model that shares a V100 with an RTX 30 / 40 card runs both on the CUDA 12 engine (it has code for sm_60 to sm_89,
|
||||
plus PTX). An RTX 50 card (sm_120) in a CUDA 12 engine is a warning, not a stop: CUDA 12.8 and newer compile for it,
|
||||
but engines built with 12.8 crashed on long prompts there (#220, #224). Keep the RTX 50 card on its own model (`--gpu
|
||||
N`), where it runs the CUDA 13 engine.
|
||||
A model that shares a V100 with an RTX 30 / 40 card runs both on the CUDA 12 engine (code for sm_60 to sm_89, plus PTX). An RTX
|
||||
50 card (sm_120) in a CUDA 12 engine is a warning, not a stop: engines built with 12.8 crashed on long prompts there (#220,
|
||||
#224). Keep the RTX 50 card on its own model (`--gpu N`), where it runs the CUDA 13 engine.
|
||||
|
||||
### Building it yourself
|
||||
|
||||
@@ -79,71 +88,167 @@ cmake -S . -B build-cuda12 -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF -DST
|
||||
cmake --build build-cuda12 --target strata -j
|
||||
```
|
||||
|
||||
Setup does the same when it compiles: it looks for the newest CUDA 12.x toolkit (`STRATA_NVCC=<path to nvcc>` picks
|
||||
one, #601; on glibc 2.43 use 12.8, see [TROUBLESHOOTING.md](TROUBLESHOOTING.md)).
|
||||
Setup does the same when it compiles: it looks for the newest CUDA 12.x toolkit. `STRATA_NVCC=<path to nvcc>` picks one. On
|
||||
glibc 2.43 use 12.8 ([TROUBLESHOOTING.md](TROUBLESHOOTING.md)). On GCC 12.3 with nvcc (openEuler 24.03, CUDA 12.8) one report
|
||||
needed `-D_BITS_OPT_RANDOM_H` added to the host flags (#1074, not reproduced here).
|
||||
|
||||
On GCC 12.3 with nvcc (openEuler 24.03, CUDA 12.8, 2x V100) the build needed `-D_BITS_OPT_RANDOM_H` added to the host flags (#1074; one
|
||||
report, not reproduced here). The `size_t` error in `vmm.hpp` that the same report hit is fixed in 0.1.40.2.
|
||||
### What the flag changes
|
||||
|
||||
### What the flag changes, and what it does not
|
||||
`-DSTRATA_EXPERIMENTAL_SM60=ON` lowers the runtime floor to compute capability 6.0 and compiles the older cards' code paths into
|
||||
that build only: the Volta prompt attention, the #540 attention kernel (used below sm_75), and the Pascal BF16 path. The
|
||||
ready-made CUDA 13 engine has none of them, so its kernels and output are unchanged.
|
||||
|
||||
`-DSTRATA_EXPERIMENTAL_SM60=ON` lowers the runtime floor to compute capability 6.0 and compiles the older cards' code
|
||||
paths into that build only: the Volta prompt attention (#600), #540's attention kernel (used below sm_75), and the
|
||||
Pascal BF16 path. The ready-made CUDA 13 engine has none of them, so its kernels and its output are unchanged. The
|
||||
FP16 path for BF16 projections is in every build but runs by default only below sm_75; RTX 20 owners can try it with
|
||||
`STRATA_BF16_TC=1` (`=0` turns it off on a V100). It is not bitwise the same as cuBLAS's BF16 kernel (FP16 tensor-core
|
||||
sums round differently): #540 measured a mean KL of 8.4e-3 on the next-token distribution of 24 code prompts on a V100
|
||||
(the same top-1 in 23), the size of other summation-order changes; #655 a worst relative difference of 3.5e-5 per
|
||||
product on an RTX 2080 Ti.
|
||||
The FP16 path for BF16 projections is in every build but runs by default only below sm_75. RTX 20 owners can try it with
|
||||
`STRATA_BF16_TC=1` (`=0` turns it off on a V100). It is not bitwise the same as cuBLAS's BF16 kernel, because FP16 tensor-core
|
||||
sums round differently.
|
||||
|
||||
### Tesla P100 (sm_60): seven decode switches on by default (engine 0.1.45.3)
|
||||
### Switches by card
|
||||
|
||||
On a Tesla P100 the engine now turns on a set of seven switches that change only how the work is scheduled, so the answers
|
||||
stay bit for bit the same: `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_SIDE_MIXER=2`, `STRATA_FUSE_WAIT=2`,
|
||||
`STRATA_B6_MMVQ_ROWS=1`, `STRATA_FUSE_DECODE=1` and `STRATA_GR_UP75=1`. `STRATA_SM60_PERF=0` turns the whole set off, and a
|
||||
switch you set yourself, to any value, is kept. The set is used when every visible card has compute capability 6.0.
|
||||
Measured by the maintainers on a Tesla P100 16 GB, IQ3_XXS, default flags, 8 interleaved pairs of fresh engines: decode 36.56
|
||||
-> 41.76 tokens/s (+14.2%, faster in 8 of 8 pairs; a run against itself gave 0.975), with the same number of expert-cache
|
||||
slots; the identity gate (10 prompts and 4K/20K prompt dumps) is identical with the set on. The full list, the RTX 30 set
|
||||
and the per-switch results: [DETAILS.md](DETAILS.md#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-decode-switches-on-by-default).
|
||||
| Variable | Card | Default | What it does |
|
||||
| --- | --- | --- | --- |
|
||||
| `STRATA_SM60_PERF=0` | P100 | set of 7 on | Turns off the seven scheduling switches below. |
|
||||
| `STRATA_Q8_SM60=1` | GP100 | off | Q8_0 dense projections and head on a GP100 kernel. |
|
||||
| `STRATA_IQ_STAGE_TABLES=1` | Pascal | off | IQ3_XXS / IQ3_S tables staged in shared memory (bitwise the same output). |
|
||||
| `STRATA_DEQUANT_WIDE=0` | sm_61 | on | Back to the old dequantization. |
|
||||
| `STRATA_SM61_TABLE=0` | sm_61 | on | Back to the CUDA expert layout and `native_mmvq`. |
|
||||
| `STRATA_DENSE_F32=1` | Pascal | off | Large dense GEMMs as `cublasSgemm`. |
|
||||
| `STRATA_BF16_TC=0\|1` | sm_7x | on below sm_75 | BF16 projections on the FP16 tensor cores. |
|
||||
| `STRATA_PROMPT_ATTN_OLD=1` | Volta | off | The decode kernel for prompts instead of the Volta kernel. |
|
||||
| `STRATA_ATTN_PRE75=0` | sm_70, gfx103x | on | Turns the #540 attention kernel off. `=1` turns it on for another wave32 AMD card. |
|
||||
|
||||
### Pascal GP100: a Q8_0 decode kernel (opt-in)
|
||||
[NVIDIA_V100.md](NVIDIA_V100.md) has the V100 build, its switches and the parity test.
|
||||
|
||||
GP100 (P100, PH402) is the one Pascal chip without `__dp4a`, and the decode kernels are written for the cards that have
|
||||
it: on a GP100 die the dense projections of a verify window read their weights at 40-80 GB/s of its ~730, decoding K-
|
||||
and i-quant blocks (Q6_K, IQ4_XS, Q4_K, IQ3_S, ...) one int at a time. Two changes, both measured on a PH402 only:
|
||||
## Tesla P100 (sm_60)
|
||||
|
||||
- **`STRATA_Q8_SM60=1`** packs every Q8_0 dense matrix and the head into the planes of `STRATA_Q8_PACKED` (any shape)
|
||||
and runs them on `src/kernels/cuda/q8_sm60.cuh`: a warp (half or quarter warp for narrow rows) per row, 16-byte
|
||||
weight loads, the window's q8_1 activations copied once per workgroup into shared memory as an int8 plane, the dot
|
||||
on `STRATA_DP4A` (vmad), a shuffle sum - no barrier per row. Shapes it declines (a 6144-wide row with 7-8 columns)
|
||||
keep the usual kernel. Not bitwise the exact kernels (the per-32 products are added in another order). The packed
|
||||
copies cost their size in VRAM beside the GGUF layout the prompt path reads (Flash-Next: 2.9 GiB of dense
|
||||
projections over the stages and 0.6 GiB for the head).
|
||||
- The GSQ-RCO packs have no Q8_0 dense matrices, so the kernel needs a copy of the model whose dense projections and head are
|
||||
Q8_0: `python tools/q8_dense_gguf.py <model>-00001-of-00002.gguf <new folder>` writes one: every shard that holds
|
||||
such matrices is converted (Swift 1.5 keeps them in both of its shards) and the rest are hard-linked beside them
|
||||
(+3.5 GiB; the experts keep their offsets, so the native pack is reused). Point `--native` and
|
||||
`--ple-gguf` at the new folder. Requantizing costs at most 0.4% of a tensor's |w|max (Q8_0 is finer than the
|
||||
sources).
|
||||
- **Opt-in `STRATA_IQ_STAGE_TABLES=1`**: IQ3_XXS's and IQ3_S's codebooks and sign-mask tables are staged in shared memory by
|
||||
the grouped expert kernels (IQ3_XXS's ported from shinbunbun/llama-cpp-p100-patches 29 and 30, IQ3_S's the same for its
|
||||
sign bytes; bitwise the same output). IQ3_S was the one gate/up format whose codebook those kernels still read from
|
||||
global memory: the contributor measured a verify window's layer of IQ3_S experts (20 experts, 24 entries) 24% faster on a
|
||||
GP100 die (517 -> 393 us with a Q2_0 down projection, 492 -> 369 us with IQ4_NL), and Swift 1.5 IQ3_XXS (13 of its 48
|
||||
layers are IQ3_S) 3.2% faster at 1M context (39.3 -> 40.5 tok/s); IQ3_XXS's tables alone were within noise (+0-4%). The
|
||||
maintainers' interleaved A/Bs on a Tesla P100 (PCIe, IQ3_XXS pack and IQ3_S pack, 5-6 pairs each) showed no end-to-end
|
||||
change (decode 0.97-1.03x, inside the noise of that box), so it is off by default (the contributor's version had it on
|
||||
for compute capability 6.0). On an RTX 3090 Ti, forced on, the same IQ3_S layers took 3-13% longer (those kernels already
|
||||
read at 490-665 GB/s there).
|
||||
- Neither change is measured on compute capability 6.1 (P40, P4, GTX 10 series), which has `__dp4a`. Both build for
|
||||
sm_61, and on a card with `__dp4a` the Q8_0 kernel's dot is the hardware instruction (`tools/q8_sm60_check.cu`
|
||||
passes on an RTX 3090 Ti); numbers from a 6.1 card with `STRATA_Q8_SM60=1` or `STRATA_IQ_STAGE_TABLES=1` would
|
||||
show whether either helps there.
|
||||
On a Tesla P100 the engine turns on seven switches that change only how the work is scheduled, so answers stay bit for bit the
|
||||
same: `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_SIDE_MIXER=2`, `STRATA_FUSE_WAIT=2`, `STRATA_B6_MMVQ_ROWS=1`,
|
||||
`STRATA_FUSE_DECODE=1` and `STRATA_GR_UP75=1`. `STRATA_SM60_PERF=0` turns the whole set off, and a switch you set yourself, to
|
||||
any value, is kept. The set is used when every visible card has compute capability 6.0. The full list and per-switch results are
|
||||
in [DETAILS.md](DETAILS.md#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-decode-switches-on-by-default).
|
||||
|
||||
Measured on a PH402 SKU 200 (two boards, four GP100 dies of 48 SMs and 32 GB HBM2), application clocks locked at
|
||||
1050 MHz, Windows 11, driver 581.80 (TCC), CUDA 12.9, engine 0.1.40.2; Flash-Next IQ3_XXS, 32K, `--layer-split 12,24,36
|
||||
--prefill 2048`; greedy, 256 tokens, medians of 2-3 interleaved starts, decode tok/s:
|
||||
## Pascal GP100: a Q8_0 decode kernel (opt-in)
|
||||
|
||||
GP100 (P100, PH402) is the one Pascal chip without `__dp4a`. Its dense projections read their weights at 40-80 GB/s of about
|
||||
730, decoding K- and i-quant blocks one int at a time. Two opt-in changes, both measured on a PH402 only:
|
||||
|
||||
- **`STRATA_Q8_SM60=1`** packs every Q8_0 dense matrix and the head into the planes of `STRATA_Q8_PACKED` and runs them on
|
||||
`src/kernels/cuda/q8_sm60.cuh`: a warp per row, 16-byte weight loads, activations copied once per workgroup into shared memory,
|
||||
the dot on `STRATA_DP4A`, no barrier per row. Shapes it declines keep the usual kernel. It is not bitwise the exact kernels.
|
||||
The packed copies cost their size in VRAM (Flash-Next: 2.9 GiB of dense projections over the stages and 0.6 GiB for the head).
|
||||
- The GSQ-RCO packs have no Q8_0 dense matrices, so the kernel needs a copy whose dense projections and head are Q8_0:
|
||||
`python tools/q8_dense_gguf.py <model>-00001-of-00002.gguf <new folder>` writes one (+3.5 GiB; experts keep their offsets,
|
||||
so the native pack is reused). Point `--native` and `--ple-gguf` at the new folder.
|
||||
- **`STRATA_IQ_STAGE_TABLES=1`** stages IQ3_XXS and IQ3_S codebooks in shared memory in the grouped expert kernels. Bitwise the
|
||||
same output. Interleaved A/Bs on a Tesla P100 showed no end-to-end change, so it is off by default. On an RTX 3090 Ti, forced on,
|
||||
the same layers took 3-13% longer.
|
||||
|
||||
Neither change is measured on compute capability 6.1, which has `__dp4a`. Two practical notes for GP100 cards: the default
|
||||
application clock is 759 MHz and a layer split leaves each die idle most of a window, so lock the clocks
|
||||
(`nvidia-smi -i <dies> -ac 715,1050` as administrator). And a second engine pinned to the same logical CPUs halves this one's
|
||||
decode; keep two engines on different logical CPUs.
|
||||
|
||||
`tools/q8_sm60_check.cu` checks the kernel against a CPU reference on these shapes (1-8 columns).
|
||||
|
||||
## Tesla P40 (sm_61): what was measured and changed
|
||||
|
||||
Three things run differently on compute capability 6.1 by default and a fourth is opt-in. Each gives the same bits as what it
|
||||
replaces.
|
||||
|
||||
| Change | What it replaces | Switch |
|
||||
| --- | --- | --- |
|
||||
| Wide dequantization (a thread per 8 values, 16-bit loads, 16-byte stores) for Q4_K / Q5_K / Q6_K / IQ4_XS | a thread per 32 values with byte loads | on; `STRATA_DEQUANT_WIDE=0` |
|
||||
| Expert mode 8 for the experts in VRAM (decode and verify windows) | the CUDA layout | on; `STRATA_SM61_TABLE=0` (or `STRATA_EXP_MODE=0`) |
|
||||
| The interleaved small-batch matvec with a P40 table (`kIlRows_61`) | `native_mmvq` | on; `STRATA_SM61_TABLE=0` (or `STRATA_MMVQ_IL=0`) |
|
||||
| The large dense GEMMs as `cublasSgemm` on FP32 copies of the FP16 operands | cuBLAS's FP16-operand kernels (4.3 TFLOPS instead of 7.6 for most chunk sizes) | **opt-in**, `STRATA_DENSE_F32=1` |
|
||||
|
||||
The GEMM route is opt-in because its equality is a property of the cuBLAS build, not of the code. It was measured with cuBLAS
|
||||
12.4.5.8 only. Run `gemm_f32_parity` (a build with `-DSTRATA_BUILD_TESTS=ON`; exit 0 = every size bitwise) with your toolkit's
|
||||
library before you set it.
|
||||
|
||||
`STRATA_GR_FAST=1` is slower on the P40 below 5 tokens, and `STRATA_PREFILL_CPU_SHARE=auto` and `STRATA_ADAPT_OVERLAP=1` gave
|
||||
nothing there.
|
||||
|
||||
Two things to know when measuring on such a card:
|
||||
|
||||
- **Temperature.** A P40 has no fan of its own. At 89 C the driver reports `SW Thermal Slowdown: Active` and the clock falls from
|
||||
1,531 to about 1,300 MHz (`nvidia-smi -q -d PERFORMANCE`). Start every arm of an A/B at the same temperature, or the later arm loses.
|
||||
- **Greedy answers are not identical run to run on a CPU without AVX-512**, even with `--prompt-cache 0 --adapt-swaps 0
|
||||
--pcie-frac 0`. Add `STRATA_IQ_MT_MIN=1` (and `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_ROUTE_TAIL_SKIP=0`) to compare two builds'
|
||||
answers byte for byte.
|
||||
|
||||
Profiling switches: `STRATA_PREFILL_TIMING=1`, `STRATA_DECODE_TIMING=1`, `STRATA_VERIFY_PROFILE=1`.
|
||||
|
||||
## Quadro RTX 8000 and other Turing cards
|
||||
|
||||
Turing (sm_75) has FP16 tensor cores but no native BF16 or TF32 tensor cores. Two existing, opt-in switches are worth comparing
|
||||
on your own workload:
|
||||
|
||||
- `STRATA_BF16_TC=1` converts BF16 projection operands to FP16 and uses FP16 tensor cores with FP32 accumulation. Default on sm_75
|
||||
is off; `=0` restores the cuBLAS BF16 path. Values outside FP16 range clamp to +-65504, and differently ordered sums can round
|
||||
differently.
|
||||
- `STRATA_SELECT_SIMT=1` uses the tiled FP32 QSA block scorer on the pre-sm_80 CUDA path instead of the warp scorer. `=0` keeps
|
||||
the warp scorer. It does not enable TF32 and its summation order differs.
|
||||
|
||||
Set the variables before starting the engine, or in the model config's `env` object, then restart. Merge these fields into the
|
||||
existing config; this is not a complete run config:
|
||||
|
||||
```json
|
||||
{
|
||||
"env": {
|
||||
"STRATA_BF16_TC": "1",
|
||||
"STRATA_SELECT_SIMT": "1"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`STRATA_BF16_TC=1` improved prompt throughput by 20-34% on one Quadro RTX 8000 and changed replies in 3 of 9 paired requests, so
|
||||
use it only if different rounding is acceptable. It did not improve decode. Numbers are in the history section. These switches are
|
||||
existing code ([gemm.cu](../src/prefill/gemm.cu), [qsa_select.cu](../src/kernels/cuda/qsa_select.cu)), not new kernels.
|
||||
|
||||
## AMD: building gfx906 and gfx1012
|
||||
|
||||
Setup does not build these. Build by hand and run `serve/server.py` with a config, as on any other card.
|
||||
|
||||
- **gfx906** (MI50 / MI60 / Radeon VII, wave64): a separate opt-in build, `-DSTRATA_HIP_GFX906=ON
|
||||
-DCMAKE_HIP_ARCHITECTURES=gfx906` (not `STRATA_ENABLE_HIP`). Current ROCm no longer ships gfx906 libraries; the reporter used a
|
||||
community ROCm 7.14 image. Recipe, kernels and measurements:
|
||||
[AMD_HIP.md](AMD_HIP.md#gfx906-instinct-mi50--mi60-radeon-vii-wave64-built-from-source).
|
||||
- **gfx1012** (RX 5500 XT): the wave32 backend, `-DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1012`. HIP 5.7 (Ubuntu's
|
||||
packages) works: older hipBLAS is used through rocBLAS, legacy HIP names and the missing `__syncwarp` are version-gated, and
|
||||
RDNA1's missing signed dot4 uses llama.cpp's SDWA sequence (`-DSTRATA_GFX1012_PORTABLE_DOT=ON`: the portable one).
|
||||
|
||||
## Reports welcome
|
||||
|
||||
These paths stay experimental until more people run them. A report with the card, the driver or ROCm version, the model and the
|
||||
engine log (`strata-<model>.log`) in an issue helps. [COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md) has the format.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
Dated numbers. Each row is one machine and one engine version, as labelled. They are not a promise for your card.
|
||||
|
||||
**Community reports by card**
|
||||
|
||||
| Card | Measured on | Result |
|
||||
| --- | --- | --- |
|
||||
| 2x Tesla P100 | IQ3_S, engine 0.1.39 ([report](../bench/results/2026-10-06-community-2x-p100/README.md), #1157) | prompt 526-532 tok/s; decode 28-32 tok/s at 128K prompt tokens; 3 of 3 needle checks at 122K |
|
||||
| PH402 (4 GP100 dies) | IQ3_XXS, engine 0.1.40.2 | decode 28.3 -> 38.9 tok/s with `STRATA_Q8_SM60=1` and a Q8_0 dense shard |
|
||||
| Tesla P40 | IQ3_S, engine 0.1.30 (#395) | prompt 217-374 tok/s; decode 30-33 tok/s |
|
||||
| Tesla P40 | IQ3_S, engine 0.1.43.1 with the sm_61 changes | prompt 255 tok/s at 565 tokens (was 205); decode 37-38 tok/s (was 34-35); with `STRATA_DENSE_F32=1` 464-474 tok/s at 7K tokens (was 360-432) |
|
||||
| V100-PCIE-32GB | UD-IQ4_XS (#600) | prompt 1,123-1,251 tok/s |
|
||||
| V100 32GB | IQ2_XS (#540) | prompt +22% |
|
||||
| RTX 2080 Ti | Q2_0 (#655) | prompt +15-18% with `STRATA_BF16_TC=1` |
|
||||
| RX 6900 XT | IQ3_S ([report](../bench/results/2026-10-04-rdna2-pre75-attention/README.md)) | prompts +4-6% alone, +7-12% with #835 |
|
||||
| RX 5500 XT, 8 GB | IQ3_S, 8K prompt (#442) | decode 15.3 tok/s |
|
||||
| RX 5700 XT, 8 GB | Coder IQ1_M, 32K context | prompt 115 / 144 / 149 tok/s and decode 18.0 / 20.3 / 23.3 tok/s at 4K / 16K / 30K; 6 of 6 needle checks; 607 of 12,288 experts in VRAM |
|
||||
| 2x MI50 | Coder IQ1_M, 128K context (#677) | decode 50.1 / 47.8 / 45.7 tok/s at 4K / 32K / 128K; prompt about 520 tok/s |
|
||||
|
||||
**Tesla P100 default switch set.** Measured on: Tesla P100 16 GB, IQ3_XXS, default flags, engine 0.1.45.3, 8 interleaved pairs of
|
||||
fresh engines: decode 36.56 -> 41.76 tok/s (+14.2%, faster in 8 of 8 pairs), same number of expert-cache slots. The identity gate
|
||||
(10 prompts and 4K / 20K prompt dumps) is identical with the set on.
|
||||
|
||||
**GP100 Q8_0 kernel.** Measured on: PH402 SKU 200 (two boards, four GP100 dies, 48 SMs and 32 GB HBM2 each), application clocks
|
||||
locked at 1050 MHz, Windows 11, driver 581.80 (TCC), CUDA 12.9, engine 0.1.40.2, Flash-Next IQ3_XXS, 32K, `--layer-split
|
||||
12,24,36 --prefill 2048`, greedy, 256 tokens, medians of 2-3 interleaved starts, decode tok/s.
|
||||
|
||||
| | ~10.7K-token document | the same, cached | short reasoning | code | mean |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
@@ -151,140 +256,35 @@ Measured on a PH402 SKU 200 (two boards, four GP100 dies of 48 SMs and 32 GB HBM
|
||||
| Q8_0 dense shard, `STRATA_Q8_PACKED=1` | 27.5 | 31.0 | 34.4 | 31.8 | 31.1 |
|
||||
| Q8_0 dense shard, `STRATA_Q8_SM60=1` | 37.5 | 38.4 | 41.2 | 38.5 | 38.9 |
|
||||
|
||||
GPU time per verify window (`STRATA_VERIFY_PROFILE`): dense projections 19.2 -> 11.2 ms, head 4.25 -> 2.1 ms; the
|
||||
draft policy then picks longer windows (2.10 -> 2.31 tokens per window). Prompt reads are unchanged (437-438 tok/s at
|
||||
10.7K). `tools/q8_sm60_check.cu` checks the kernel against a CPU reference on these shapes (1-8 columns) and times it.
|
||||
GPU time per verify window: dense projections 19.2 -> 11.2 ms, head 4.25 -> 2.1 ms. Prompt reads are unchanged (437-438 tok/s at
|
||||
10.7K). Locking clocks at 1050 MHz took one case from 25.0 to 29.1 tok/s at 10.7K context.
|
||||
|
||||
Two things that mattered as much on this card: its default application clock is 759 MHz, and a layer split leaves
|
||||
each die idle most of a window, so the dies decode below 1050 MHz unless the clocks are locked
|
||||
(`nvidia-smi -i <dies> -ac 715,1050` as administrator; 25.0 -> 29.1 tok/s at 10.7K context). And a second engine on
|
||||
the same PC that pins its threads to the same logical CPUs halves this one's decode while it runs (the host thread
|
||||
is time-sliced); keeping the two on different logical CPUs (SMT siblings) avoided it.
|
||||
**Tesla P40 sm_61 kernels.** Measured on: one Tesla P40 24 GB in a 20-vCPU VM (Xeon E5-2695 v4, 98 GB RAM), CUDA 12.4 (cuBLAS
|
||||
12.4.5.8), IQ3_S with MTP (`--spec 4`), int8 KV, `--max-context 262144`, engine 0.1.43.1 ([report](../bench/results/2026-10-12-p40-sm61-kernels/README.md)).
|
||||
|
||||
A/B switches: `STRATA_DEQUANT_WIDE=0`, `STRATA_SM61_TABLE=0` (sm_61: the kernels 0.1.43.1 ran), `STRATA_DENSE_F32=1` (Pascal, opt-in),
|
||||
`STRATA_Q8_SM60=0|1` (Pascal GP100, opt-in), `STRATA_BF16_TC=0|1`, `STRATA_PROMPT_ATTN_OLD=1` (the decode kernel for prompts), `STRATA_ATTN_PRE75=0`
|
||||
(#540's kernel off; on gfx103x with HIP that kernel is the default, see the AMD table above, and `=1` turns it on for
|
||||
another wave32 AMD card). [NVIDIA_V100.md](NVIDIA_V100.md) has the V100 build, its measurements and the parity test.
|
||||
| Change | Result on the P40 |
|
||||
| --- | --- |
|
||||
| Wide dequantization | Q6_K 10240 x 2560: 7.2 -> 0.33 ms (10 -> 227 GB/s); Q4_K 5.2x, Q5_K 3.8x, IQ4_XS 3.6x |
|
||||
| Expert mode 8 | an 18-expert call at 2 tokens: IQ3_S 390 -> 255 us, IQ3_XXS 345 -> 252, IQ2_S 290 -> 238, IQ4_XS 328 -> 261 |
|
||||
| Interleaved matvec table | IQ4_XS 2560 x 10240 at 2 columns 112 -> 65 us; Q6_K 12288 x 2560 at 4 columns 189 -> 127 us |
|
||||
| `STRATA_DENSE_F32=1` | 8.4-9.8 TFLOPS at every size from 512 tokens; qkv at 7,041 tokens 85 -> 45 ms |
|
||||
|
||||
### Tesla P40 (sm_61): what was measured and changed
|
||||
Through the server, medians of 3 requests, two runs per build alternated, card at 65 C. By default: a 565-token prompt 203 / 205
|
||||
-> 256 / 255 tok/s, decode 35.0 / 34.2 -> 38.2 / 37.2 tok/s. With `STRATA_DENSE_F32=1`: a 7,047-token prompt 430 / 364 -> 474 / 473
|
||||
tok/s. Of a 7K-token prompt, 36% is the dense projections' GEMMs, 28% the experts, 13% the hyper-connection read and 12%
|
||||
attention.
|
||||
|
||||
One Tesla P40 24 GB in a 20-vCPU VM (Xeon E5-2695 v4, 98 GB RAM), CUDA 12.4 (cuBLAS 12.4.5.8), IQ3_S with MTP
|
||||
(`--spec 4`), int8 KV, `--max-context 262144`, engine 0.1.43.1; the numbers and every run are in
|
||||
[bench/results/2026-10-12-p40-sm61-kernels](../bench/results/2026-10-12-p40-sm61-kernels/README.md).
|
||||
`STRATA_PREFILL_TIMING=1`, `STRATA_DECODE_TIMING=1` and `STRATA_VERIFY_PROFILE=1` show a card that is the bottleneck
|
||||
everywhere: the prompt's GPU timeline equals its wall time (waiting for expert copies 0.2%, host grouping 0.2% of a
|
||||
7K-token prompt) and a decode window is 49 ms of GPU stages with 1-3 ms of CPU experts. Of a 7K-token prompt, 36% is
|
||||
the dense projections' GEMMs, 28% the experts (ggml's MMQ, which has its own Pascal dp4a configuration), 13% the
|
||||
hyper-connection read and 12% the attention.
|
||||
**Quadro RTX 8000 (sm_75).** Measured on: one 48 GiB Quadro RTX 8000, 260 W limit, Xeon W-2295, CUDA 12.4, GCC 13, 2026-10-07,
|
||||
source build based on v0.1.40.3 with other local changes; IQ3_S with MTP (`--spec 4`), `--max-context 262144`, INT8 KV with
|
||||
`--kv-resident 32768`, `--pcie-frac 0.20`, `--expert-cache 19000`, `--prefill auto`, prompt cache and adaptive swaps off. Three
|
||||
seeded synthetic prompts per size, 128 output tokens, median engine-reported rates.
|
||||
|
||||
Three things run differently on compute capability 6.1 by default and a fourth is opt-in. Each gives the same bits as
|
||||
what it replaces:
|
||||
|
||||
| Change | What it replaces | Measured on the P40 | Switch |
|
||||
| --- | --- | --- | --- |
|
||||
| Wide dequantization (a thread per 8 values, 16-bit loads, 16-byte stores) for Q4_K / Q5_K / Q6_K / IQ4_XS | a thread per 32 values with byte loads: 0.5 s of every prompt chunk, whatever its length | Q6_K 10240 x 2560: 7.2 -> 0.33 ms (10 -> 227 GB/s); Q4_K 5.2x, Q5_K 3.8x, IQ4_XS 3.6x; `dequant_wide_parity` 360 cases bitwise | on; `STRATA_DEQUANT_WIDE=0` |
|
||||
| Expert mode 8 for the experts in VRAM (decode and verify windows) | the CUDA layout | an 18-expert call at 2 tokens: IQ3_S 390 -> 255 us, IQ3_XXS 345 -> 252, IQ2_S 290 -> 238, IQ4_XS 328 -> 261; `native_expert_bench` bitwise | on; `STRATA_SM61_TABLE=0` (or `STRATA_EXP_MODE=0`) |
|
||||
| The interleaved small-batch matvec with a table measured on the P40 (`kIlRows_61`) | `native_mmvq` | IQ4_XS 2560 x 10240 at 2 columns 112 -> 65 us, Q6_K 12288 x 2560 at 4 columns 189 -> 127 us, the Q5_K head at 3 columns 2,545 -> 1,766 us; `mmvq_il_parity` bitwise | on; `STRATA_SM61_TABLE=0` (or `STRATA_MMVQ_IL=0`) |
|
||||
| The large dense GEMMs (qkv, query, gate, output projections) as `cublasSgemm` on FP32 copies of the FP16 operands | cuBLAS's FP16-operand kernels, which run at 4.3 TFLOPS instead of 7.6 for most chunk sizes (a 7K-token prompt reads in 16.3 or 19.5 s by its exact token count) | 8.4-9.8 TFLOPS at every size from 512 tokens; qkv at 7,041 tokens 85 -> 45 ms; `gemm_f32_parity` 5,201 chunk sizes bitwise | **opt-in**, `STRATA_DENSE_F32=1` |
|
||||
|
||||
The GEMM route is opt-in because its equality is a property of the cuBLAS build, not of the code: it was measured with
|
||||
cuBLAS 12.4.5.8 only. Run `gemm_f32_parity` (a build with `-DSTRATA_BUILD_TESTS=ON`; exit 0 = every size bitwise) with
|
||||
your toolkit's library before you set it.
|
||||
|
||||
Through the server, medians of 3 requests per run, two runs per build alternated, each run started with the card at
|
||||
65 C (see below). By default: a 565-token prompt 203 / 205 -> 256 / 255 tok/s, decode after it 35.0 / 34.2 -> 38.2 /
|
||||
37.2 tok/s, a 7,047-token prompt 1-3% faster at the same token count. With `STRATA_DENSE_F32=1` as well: the
|
||||
7,047-token prompt 430 / 364 -> 474 / 473 tok/s (the old build reads it at ~361 or ~431 tok/s depending on whether the
|
||||
chunk has more than 7,040 tokens; with the route every request read at 464-474), decode after it 32.1 / 33.4 -> 35.9 /
|
||||
34.0. With `STRATA_DEQUANT_WIDE=0 STRATA_SM61_TABLE=0` and the route off the new build measures as the old one (210 /
|
||||
365 tok/s, 33.3 tok/s).
|
||||
|
||||
Measured and **not** changed: the latency-hidden hyper-connection read (`STRATA_GR_FAST=1`) is slower on the P40 below
|
||||
5 tokens (82 -> 88 us at one); `STRATA_PREFILL_CPU_SHARE=auto` and `STRATA_ADAPT_OVERLAP=1` gave nothing.
|
||||
|
||||
Two things to know when measuring on such a card:
|
||||
|
||||
- **Temperature.** A P40 has no fan of its own. This one reaches 87-89 C in a few minutes of work; at 89 C the driver
|
||||
reports `SW Thermal Slowdown: Active` and the clock falls from 1,531 to ~1,300 MHz (`nvidia-smi -q -d PERFORMANCE`,
|
||||
`nvidia-smi --query-gpu=temperature.gpu,clocks.gr --format=csv -l 2`). Start every arm of an A/B at the same
|
||||
temperature, or the later arm loses.
|
||||
- **Greedy answers are not identical run to run on a CPU without AVX-512**, even with `--prompt-cache 0
|
||||
--adapt-swaps 0 --pcie-frac 0`: an expert computed on the CPU rounds differently for a lone token than in a window
|
||||
of several (`native_gu_mt_min`), and how many drafts a window holds depends on timing. Add `STRATA_IQ_MT_MIN=1`
|
||||
(and `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_ROUTE_TAIL_SKIP=0`) to compare two builds' answers byte for byte.
|
||||
|
||||
### Quadro RTX 8000: measurements of existing switches
|
||||
|
||||
The Quadro RTX 8000 is Turing (sm_75): it has FP16 tensor cores, but no native BF16 or TF32 tensor cores. Two
|
||||
**existing, opt-in** CUDA switches are worth comparing on your own workload:
|
||||
|
||||
- `STRATA_BF16_TC=1` converts BF16 projection operands to FP16 and uses FP16 tensor cores with FP32 accumulation.
|
||||
The default on sm_75 is off; `=0` restores the cuBLAS BF16 path. The switch applies to 7.x cards, not the native
|
||||
BF16 path on sm_80 and newer. Conversion clamps finite values outside FP16's range to ±65504; small values and
|
||||
differently ordered sums can round differently.
|
||||
- `STRATA_SELECT_SIMT=1` uses the tiled FP32 QSA block scorer on the pre-sm_80 CUDA path instead of the default
|
||||
warp scorer. `=0` keeps the warp scorer. It does not enable TF32 on Turing, and its summation order differs.
|
||||
|
||||
The existing paths are in [gemm.cu](../src/prefill/gemm.cu) and [qsa_select.cu](../src/kernels/cuda/qsa_select.cu).
|
||||
|
||||
Set the environment before starting the engine, or set the switches in the model config's `env` object, then
|
||||
restart. For example, this config fragment opts into both (keep the config's other settings):
|
||||
|
||||
```json
|
||||
"env": {
|
||||
"STRATA_BF16_TC": "1",
|
||||
"STRATA_SELECT_SIMT": "1"
|
||||
}
|
||||
```
|
||||
|
||||
This section records an investigation, **not a new kernel or a change to defaults**. Measurements on 2026-10-07
|
||||
used one 48 GiB Quadro RTX 8000 (physical GPU 3), a 260 W power limit, a Xeon W-2295, CUDA 12.4 and GCC 13.
|
||||
The source-built engine was based on v0.1.40.3 with other local changes integrated; unrelated opt-in changes
|
||||
were disabled in these arms. These are not measurements of an otherwise clean documentation-only checkout,
|
||||
not results at the launcher's 200 W setting, and not a speed promise for every sm_75 card.
|
||||
|
||||
The model was IQ3_S with MTP (`--spec 4`), `--max-context 262144`, INT8 KV with `--kv-resident 32768`,
|
||||
`--pcie-frac 0.20`, a fixed `--expert-cache 19000` request and `--prefill auto` (8,192-token ceiling).
|
||||
The engine reported 21,189 cache slots including the prompt loan. Prompt caching and adaptive swaps were off
|
||||
(`--prompt-cache 0 --adapt-swaps 0`). Each size used three fresh, seeded synthetic prompts and 128 output tokens
|
||||
per request; the table gives median engine-reported rates, not end-to-end server latency.
|
||||
|
||||
| Existing switch settings | 537 tokens: prompt / decode tok/s | 4,057 tokens: prompt / decode tok/s | 32,057 tokens: prompt / decode tok/s |
|
||||
| Switches | 537 tokens: prompt / decode tok/s | 4,057 tokens | 32,057 tokens |
|
||||
| --- | --- | --- | --- |
|
||||
| BF16_TC=0, SELECT_SIMT=0 | 493 / 86.8 | 1,055 / 79.9 | 1,135 / 70.6 |
|
||||
| BF16_TC=1, SELECT_SIMT=0 | 661 / 83.7 | 1,262 / 75.4 | 1,416 / 68.4 |
|
||||
| BF16_TC=1, SELECT_SIMT=1 | 679 / 87.1 | 1,290 / 78.2 | 1,501 / 71.6 |
|
||||
|
||||
BF16_TC improved prompt throughput by about 20–34%, but did not improve decode in these comparisons. Adding
|
||||
SELECT_SIMT improved the 32K prompt rate by about 6% over BF16_TC alone; this is not a universal decode win.
|
||||
BF16_TC changed replies in 3 of the 9 paired requests, so use it only if different rounding is acceptable.
|
||||
All five basic arithmetic/JSON checks passed in each arm, which is a smoke check, **not a model-quality study**.
|
||||
A separate synthetic QSA scorer check at 131,072 context tokens matched selected IDs for all 256 queries and
|
||||
passed its FP64 accuracy gate. That check does not prove identical selections on arbitrary prompts or model quality.
|
||||
|
||||
Two other existing tuning options did not justify a recommendation on this machine:
|
||||
|
||||
- `--prefill auto:16384`, with BF16_TC on and SELECT_SIMT off, increased the prompt's cache loan from 4.38 to
|
||||
7.38 GiB for only about 0.6% more 32K prompt throughput (1,425 versus 1,416 tok/s). Just 24 MiB of VRAM remained
|
||||
at the last graph capture. Keep the 8,192 ceiling rather than spend that memory margin for this small gain.
|
||||
- `STRATA_PREFILL_CPU_SHARE=auto` gave no gain at the two tested sizes (537 and 4,057 tokens): prompt rates were
|
||||
647 and 1,253 tok/s versus 661 and 1,262 with BF16_TC alone; decode was also lower. Larger prompts were not
|
||||
tested for this setting.
|
||||
|
||||
## AMD: building gfx906 and gfx1012
|
||||
|
||||
Setup does not build these; build by hand and run `serve/server.py` with a config, as on any other card.
|
||||
|
||||
- **gfx906** (MI50 / MI60 / Radeon VII, wave64): a separate opt-in build, `-DSTRATA_HIP_GFX906=ON
|
||||
-DCMAKE_HIP_ARCHITECTURES=gfx906` (not `STRATA_ENABLE_HIP`). Current ROCm no longer ships gfx906 libraries; the
|
||||
reporter used a community ROCm 7.14 image. Recipe, kernels and measurements: [AMD_HIP.md](AMD_HIP.md#gfx906-instinct-mi50--mi60-radeon-vii-wave64-built-from-source).
|
||||
- **gfx1012** (RX 5500 XT): the wave32 backend, `-DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1012`. HIP 5.7
|
||||
(Ubuntu's packages) works: older hipBLAS (0.x) is used through rocBLAS, the legacy HIP names and the missing
|
||||
`__syncwarp` are version-gated, and RDNA1's missing signed dot4 uses llama.cpp's SDWA sequence
|
||||
(`-DSTRATA_GFX1012_PORTABLE_DOT=ON`: the portable one).
|
||||
|
||||
## Reports welcome
|
||||
|
||||
These paths stay experimental until more people run them. A report with the card, the driver / ROCm version, the
|
||||
model and the engine log (`strata-<model>.log`) in an issue helps; [COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md)
|
||||
has the format for measurements.
|
||||
`SELECT_SIMT` added about 6% to the 32K prompt rate over BF16_TC alone. `--prefill auto:16384` gained about 0.6% at 32K for 3 GiB
|
||||
more prompt-cache loan, and `STRATA_PREFILL_CPU_SHARE=auto` gave no gain, so neither is recommended. All five basic arithmetic and
|
||||
JSON checks passed in each arm; that is a smoke check, not a quality study.
|
||||
+50
-41
@@ -1,25 +1,30 @@
|
||||
# OrcaRouter IQ3_XXS compatibility
|
||||
|
||||
Validated on Linux with an RTX 5090 (32 GB), Ryzen 9 9950X3D and 128 GB RAM on 2026-09-27.
|
||||
This page is for people who want to run `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF` (**IQ3_XXS**, two shards,
|
||||
85.20 GB) with Strata. It is a manual conversion workflow from a [source checkout](SOURCE_INSTALL.md). OrcaRouter is
|
||||
not an installer menu entry, and there is no `--family orca` command.
|
||||
|
||||
The target is `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, **IQ3_XXS** (two shards, 85.20 GB total).
|
||||
It has Strata's Qwen4Exp geometry, but ordinary quantization also compresses the hyperconnection,
|
||||
small attention and PLE projections which Strata's kernels read as BF16.
|
||||
## Quick start
|
||||
|
||||
`tools/iq_pack.py --compat-bf16` expands only those projections, rounding to BF16 using nearest-even.
|
||||
This adds rounding relative to the GGUF's dequantized weights; it does not recover the original BF16
|
||||
checkpoint or its quality. Expert weights, native attention weights, embeddings and the 28.8 GB PLE
|
||||
lookup table remain unchanged. The latter stays on disk. The pack records its conversions in
|
||||
`compat-bf16.json`.
|
||||
1. Build Strata normally, then pack the model ([Preparation](#preparation)).
|
||||
2. Prepare the optional draft head (MTP) with the same section's commands.
|
||||
3. Save the config from [Local server](#local-server) and start the server.
|
||||
|
||||
The native loader retains this pack's converted PLE key. Its existing native Q2_0 key path remains
|
||||
available to the original model. Hugging Face snapshot symlinks are supported; all split shards must
|
||||
finish downloading before packing. Pass the snapshot filename, not its hash-named blob target.
|
||||
## How it works
|
||||
|
||||
The model has Strata's Qwen4Exp geometry. Ordinary quantization also compresses the hyperconnection, small attention
|
||||
and PLE projections, which Strata's kernels read as BF16. `tools/iq_pack.py --compat-bf16` expands only those
|
||||
projections, rounding to BF16 with nearest-even. This adds rounding relative to the GGUF's dequantized weights. It does
|
||||
not recover the original BF16 checkpoint or its quality.
|
||||
|
||||
Expert weights, native attention weights, embeddings and the 28.8 GB PLE lookup table stay unchanged, and the lookup
|
||||
table stays on disk. The pack records its conversions in `compat-bf16.json`. Hugging Face snapshot symlinks work. All
|
||||
split shards must finish downloading before packing. Pass the snapshot filename, not its hash-named blob target.
|
||||
|
||||
## Preparation
|
||||
|
||||
Build Strata normally using the pinned llama.cpp dependency. Python needs numpy, regex and gguf-py
|
||||
from that dependency (`STRATA_GGUF_PY` can point to its `gguf-py` directory).
|
||||
Python needs numpy, regex and gguf-py from the pinned llama.cpp dependency. `STRATA_GGUF_PY` can point to its
|
||||
`gguf-py` directory.
|
||||
|
||||
```sh
|
||||
.venv/bin/python tools/iq_pack.py \
|
||||
@@ -27,11 +32,10 @@ from that dependency (`STRATA_GGUF_PY` can point to its `gguf-py` directory).
|
||||
--out packs/orca-iq3_xxs --compat-bf16
|
||||
```
|
||||
|
||||
Use this model's own tokenizer exported into the pack. Do not share another model's `dense.bin`,
|
||||
and do not rename the Orca files to impersonate one of setup's GSQ-RCO models.
|
||||
Use this model's own tokenizer, exported into the pack. Do not share another model's `dense.bin`, and do not rename
|
||||
the Orca files to pass as one of setup's GSQ-RCO models.
|
||||
|
||||
Strata's persistent server also uses the MTP runtime (optional: without `--mtp` it drafts by lookup only), prepared by
|
||||
the existing tools:
|
||||
The persistent server can use the MTP runtime. It is optional: without `--mtp` the server drafts by lookup only.
|
||||
|
||||
```sh
|
||||
.venv/bin/python tools/mtp_fetch.py fetch --out mtp
|
||||
@@ -40,15 +44,15 @@ the existing tools:
|
||||
cp data/draft_vocab.bin mtp/rt/draft_vocab.bin
|
||||
```
|
||||
|
||||
This uses the original model's draft head, as setup does for Swift. The target model verifies its
|
||||
proposals; draft acceptance and performance must be measured for this fine-tune.
|
||||
This uses the original model's draft head, as setup does for Swift. The target model verifies every proposal. Measure
|
||||
draft acceptance and speed for this fine-tune; the original model's numbers do not carry over.
|
||||
|
||||
## Local server
|
||||
|
||||
Save the following as `strata-orca-iq3_xxs.json` at the repository root, replacing both `/path/to/`
|
||||
entries with the **same first shard**. Orca's PLE table is in shard 1. Start with a 32K context and
|
||||
512-token prefill chunks; the expert cache sizes itself to available VRAM. The expert arena alone
|
||||
needs about 49.8 GiB of available system RAM, plus draft/runtime buffers and other applications.
|
||||
Save the following as `strata-orca-iq3_xxs.json` at the repository root. Replace both `/path/to/` entries with the
|
||||
**same first shard**, because Orca's PLE table is in shard 1. The config uses a 32K context and 512-token prefill
|
||||
chunks. The expert cache sizes itself to the available VRAM. The expert arena alone needs about 49.8 GiB of free system
|
||||
RAM, plus draft and runtime buffers and your other applications.
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -70,27 +74,32 @@ needs about 49.8 GiB of available system RAM, plus draft/runtime buffers and oth
|
||||
}
|
||||
```
|
||||
|
||||
Run from the repository root:
|
||||
Run it from the repository root:
|
||||
|
||||
```sh
|
||||
.venv/bin/python -m serve.server --engine strata --config strata-orca-iq3_xxs.json --port 8080
|
||||
```
|
||||
|
||||
The web interface is at `http://127.0.0.1:8080`; API clients use `http://127.0.0.1:8080/v1`.
|
||||
Vision is not configured in this text-only example. Performance of the original GSQ-RCO model
|
||||
does not establish this fine-tune's speed or accuracy.
|
||||
The web interface is at `http://127.0.0.1:8080`. API clients use `http://127.0.0.1:8080/v1`. This example is text-only;
|
||||
vision is not configured.
|
||||
|
||||
## Scope and validation
|
||||
## Scope
|
||||
|
||||
- IQ3_XXS is the target of this change; other Orca quantizations are not validated.
|
||||
- IQ3_M additionally uses Q5_0 expert down matrices, which the native GPU expert path does not support.
|
||||
- The installer model menu is unchanged. This is an explicit local packing workflow.
|
||||
- Focused conversion and split-file tests: `.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`.
|
||||
- All eight packing tests, 17 server tests and the GPU gated-residual parity check passed.
|
||||
- The complete model packed successfully: 460 tensors converted, 1.39 GiB of converted BF16 weights;
|
||||
1.43 GiB total packed dense weights. The 48 expert layers remain IQ3_XXS gate/up and IQ4_NL down.
|
||||
- Real server checks passed: arithmetic, Python code, translation, a longer explanation, multi-turn context
|
||||
reuse (25 prompt tokens reused), streaming cancellation and a subsequent correct response. The engine
|
||||
PID stayed unchanged. These are smoke tests, not a quality benchmark or a full-context stress test.
|
||||
- With the configuration above, a 111-token explanation generated at 77.7 tokens/s (engine decode timing);
|
||||
total request time was 1.77 seconds. This single short measurement is not a general throughput claim.
|
||||
- IQ3_XXS is the only validated Orca quantization.
|
||||
- IQ3_M uses Q5_0 expert down matrices, which the native GPU expert path does not support.
|
||||
- The installer model menu does not list this model.
|
||||
- Packing tests: `.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
> **Measured on:** Linux, RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB RAM, 2026-09-27, the configuration above.
|
||||
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| Packing | 460 tensors converted, 1.39 GiB of converted BF16 weights, 1.43 GiB total packed dense weights; the 48 expert layers stay IQ3_XXS gate/up and IQ4_NL down |
|
||||
| Tests | all eight packing tests, 17 server tests and the GPU gated-residual parity check passed |
|
||||
| Server smoke tests | arithmetic, Python code, translation, a longer explanation, multi-turn reuse (25 prompt tokens reused), streaming cancellation and a correct answer afterwards; the engine PID stayed unchanged |
|
||||
| Decode speed | 111-token explanation at 77.7 tokens/s (engine decode timing), 1.77 s total request time |
|
||||
|
||||
These are smoke tests and one short measurement, not a quality benchmark, a full-context stress test or a general
|
||||
throughput claim.
|
||||
+207
-158
@@ -1,23 +1,51 @@
|
||||
# Qwen3.6-35B-A3B: pictures, several requests at once and speed
|
||||
|
||||
Strata runs [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (35B mixture of experts, 3B active, 256 experts, top-8) from
|
||||
[Unsloth's GGUFs with the MTP layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) (UD-Q4_K_XL, UD-Q6_K_XL, Q8_0) on its own
|
||||
engine path (`src/qwen35`; `strata` switches to it from the file's architecture). Pick it in the app's model list, in `strata launch` or in setup (`--family qwen36`). Besides text it
|
||||
reads **pictures** and serves **several requests at once**, the same two options the Flash-Next models have.
|
||||
This page is for people who run [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (35B mixture of
|
||||
experts, 3B active, 256 experts, top-8) with Strata and want to know how pictures, parallel requests and the prompt
|
||||
chunk behave. For sizes and memory, see [MODELS.md](MODELS.md).
|
||||
|
||||
Measured on an AMD Radeon AI PRO R9700 32 GB (HIP, Linux, UD-Q4_K_XL, every expert in VRAM) and on an RTX 3060 12 GB (CUDA, 20 threads,
|
||||
UD-Q4_K_XL in the hybrid mode: 22-37 of 256 experts per layer in VRAM, the rest on the CPU), llama.cpp 972b5fc on the same files.
|
||||
Strata runs [Unsloth's GGUFs with the MTP layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)
|
||||
(UD-Q4_K_XL, UD-Q6_K_XL, Q8_0) on its own engine path (`src/qwen35`; `strata` switches to it from the file's
|
||||
architecture). It runs on NVIDIA and AMD cards. The Intel Arc and Mac engines do not run it.
|
||||
|
||||
## Quick start
|
||||
|
||||
Pick it in the app's model list, in `strata launch` or in setup:
|
||||
|
||||
```sh
|
||||
strata launch --family qwen36 --model ud-q4_k_xl
|
||||
strata launch --family qwen36 --model ud-q4_k_xl --vision on --parallel 2
|
||||
```
|
||||
|
||||
| Option | Effect |
|
||||
| --- | --- |
|
||||
| `--vision on\|off\|gpu\|cpu` | Terminal: pictures and which reader. Source setup uses `--vision yes\|no\|gpu\|cpu` and `--vision-tokens N`. |
|
||||
| `--parallel N` | How many requests are answered at once, 1 to 8. Source setup has the same option. |
|
||||
| `--prefill 512` | Engine flag in the config's `args`, not a `strata launch` option: the prompt chunk size ([Prompt chunk](#prompt-chunk)). |
|
||||
|
||||
## Pictures
|
||||
|
||||
The vision encoder is Unsloth's `mmproj-F16.gguf` (0.9 GB, pinned revision, SHA-256 checked by setup): a 27-layer ViT with a 2x2 merger,
|
||||
the same projector family as the Flash-Next models' (`qwen3vl_merger`; only its output width differs: 2048 here, 2560 there). Strata runs
|
||||
it with the same helper (`strata-vision`, llama.cpp's `mtmd`), so the encoder choices are the Flash-Next ones
|
||||
([Images](DETAILS.md#images-vision)). Pictures are on by default, the same rule as for every model on NVIDIA and AMD cards:
|
||||
the CPU reader under 24 GB of graphics memory (a few seconds per picture, no VRAM), the card's reader from 24 GB (GPU encoder,
|
||||
~1.2 GB of VRAM at up to 1,024 image tokens; on AMD it is `strata-vision-hip`, with the CPU reader as its fallback).
|
||||
`--vision yes|no|gpu|cpu` and `--vision-tokens N` work as there; what you ask for is kept, with a note. The Mac engine does
|
||||
not run this model.
|
||||
The vision encoder is Unsloth's `mmproj-F16.gguf` (0.9 GB, pinned revision, SHA-256 checked by setup): a 27-layer ViT
|
||||
with a 2x2 merger, the same projector family as the Flash-Next models' (`qwen3vl_merger`; only its output width
|
||||
differs: 2048 here, 2560 there). Strata runs it with the same helper (`strata-vision`, llama.cpp's `mtmd`), so the
|
||||
encoder choices are the Flash-Next ones ([Images](DETAILS.md#images-vision)).
|
||||
|
||||
Pictures are on by default, by the same rule as for every model on NVIDIA and AMD cards:
|
||||
|
||||
- The CPU reader runs under 24 GB of graphics memory. It takes a few seconds per picture and no VRAM.
|
||||
- The card's reader runs from 24 GB (GPU encoder, about 1.2 GB of VRAM at up to 1,024 image tokens). On AMD it is
|
||||
`strata-vision-hip`, with the CPU reader as its fallback.
|
||||
|
||||
Your explicit choice is kept, with a note. A picture becomes up to 1,024 tokens of the context (the CPU default is 300).
|
||||
The engine puts the encoder's rows where the prompt has `<|image_pad|>` tokens and gives every row its 2-D position,
|
||||
interleaved M-RoPE as llama.cpp does for this model (rotary sections 11 / 11 / 10).
|
||||
|
||||
Pictures work with several requests at once, with the MTP drafts and with the conversation cache. The same conversation
|
||||
sent again continues from the held state when its pictures are the same (matched by the server's key for each
|
||||
picture). A different picture starts that conversation again.
|
||||
|
||||
> **Measured on:** RTX 3060 12 GB (CUDA) and an R9700 box (32 threads) and a 3060 box (10 threads), UD-Q4_K_XL, Strata
|
||||
> 0.1.45.
|
||||
|
||||
| Encoder | Time per picture (640x400 = 260 tokens / 800x450 = 350 tokens) | VRAM |
|
||||
| --- | --- | --- |
|
||||
@@ -25,179 +53,200 @@ not run this model.
|
||||
| CPU, 32 threads (R9700 box) | 0.9 s / 1.1 s | none |
|
||||
| CPU, 10 threads (3060 box) | 1.6 s / 2.1 s | none |
|
||||
|
||||
A picture becomes up to 1,024 tokens of the context (the CPU default is 300). The engine puts the encoder's rows where the prompt has
|
||||
`<|image_pad|>` tokens and gives every row its 2-D position, interleaved M-RoPE exactly as llama.cpp does for this model (rotary sections
|
||||
11 / 11 / 10, a picture's rows at (t, h, w) = (p, p + row, p + column), the text after it continuing at p + max(columns, rows)).
|
||||
## Several requests at once
|
||||
|
||||
**Checked against llama.cpp** (its `mtmd` and its own graph, the encoder on the CPU on both sides, the same prompt ids, the answer
|
||||
llama.cpp gives fed to both so the logits are compared step by step, 24 steps; the "floor" is llama.cpp's CPU backend against its own
|
||||
HIP run, the noise two correct engines have on this model):
|
||||
`"parallel": N` in the model's config (`strata-qwen36-<size>.json`) or `--parallel N` makes the engine keep N
|
||||
conversations open (`--batch N`, up to 8) and decode them together. Every window carries one token of each
|
||||
conversation, so the dense weights, the head and every routed expert that conversations share are read once for all of
|
||||
them. The server speaks the same engine protocol as for Flash-Next ([BATCHING.md](BATCHING.md)).
|
||||
|
||||
| Picture and question | Prompt tokens | First token | Mean KL, llama.cpp HIP vs Strata | Floor (llama.cpp CPU vs HIP) |
|
||||
A new request waits only for the read of its own prompt, which runs between the decode windows of the others. A
|
||||
conversation continues in the slot that holds its start, or the engine copies the longer held start from another slot.
|
||||
A request that runs alone (the solo path, with its MTP drafts) moves into a batch slot when another arrives, and back
|
||||
when it is alone again, and gives the same text either way.
|
||||
`STRATA_Q36_READ_WINDOWS=4` lets the decoding slots produce more tokens during a long prompt read, at the cost of a
|
||||
slower read. The default is one decode window per chunk.
|
||||
|
||||
**MTP drafts.** A request that has the engine to itself keeps its MTP drafts. With several requests the batch window has
|
||||
no drafts unless `--batch-mtp` is in the config's `args` (opt-in). Each slot's window then also carries the proposal its
|
||||
MTP block made, one per slot, up to the rows to spare in the 8-row window.
|
||||
|
||||
**What a slot costs.** Each slot keeps its own cache in VRAM: 1.5 GB at a context of 32K (fp32 keys and values of the 10
|
||||
attention layers and the MTP block, 45 KB per token) and 0.07 GB of recurrent state. Setup writes the context for one
|
||||
conversation, so N slots take N times that from the room the experts would have. On a card that does not hold every
|
||||
expert (12 and 16 GB, the hybrid mode) the expert slots lose that room, and a batch reads more distinct experts from RAM.
|
||||
|
||||
Setup offers `--parallel` where the sessions do not eat the experts' room, says the price in the same breath and asks
|
||||
"How many requests at once?" Enter keeps one at a time, and any number you give is kept.
|
||||
|
||||
| Card type | Slots setup offers |
|
||||
| --- | --- |
|
||||
| Every expert still fits | Up to 3 to 4 (3 on a 32 GB card at 32K) |
|
||||
| Hybrid mode | The count whose sessions take at most 35% of the room the experts would have. 12 GB: 4 slots at 8K, none at 32K. 16 GB: 2 slots at 32K. |
|
||||
|
||||
The engine says what the sessions take before it allocates them and refuses, with the numbers, a count that cannot fit.
|
||||
|
||||
In the hybrid mode the text of a request can differ a little between running alone and running beside others. The
|
||||
experts that sit on the GPU or on the CPU change with the routing of all the requests, and the two compute with
|
||||
different rounding. With the slot tables fixed (`STRATA_Q36_FIXED_SLOTS=1`, a test setting) every slot equals its solo
|
||||
run there too.
|
||||
|
||||
## Prompt chunk
|
||||
|
||||
The MoE prompt kernels read every expert once per chunk, so a bigger chunk feeds them more rows per expert. With every
|
||||
expert in VRAM the engine takes the largest of 4096 / 2048 / 1024 that fits beside the experts, the 0.77 GiB reserve
|
||||
and the sessions' caches.
|
||||
|
||||
| Situation | Chunk |
|
||||
| --- | --- |
|
||||
| Every expert in VRAM, 8K to 64K context | 4096 |
|
||||
| Every expert in VRAM, 100K context | 2048 |
|
||||
| Every expert in VRAM, 131K context | 512 |
|
||||
| `--batch 2..4` | 1024 at most, so the other slots are not held for a whole chunk |
|
||||
| `--batch 8` | 512 |
|
||||
| Hybrid mode | 2048; 1024 or 512 on a card with 7.5 or 5.5 GiB free |
|
||||
|
||||
In the hybrid mode a bigger chunk takes expert slots (an RTX 3060 at chunk 4096 read prompts 5% faster and decoded 13%
|
||||
slower). `--prefill 512` (or `STRATA_Q36_CHUNK=512`) gives the 512 size, and then the text is the base engine's, bit
|
||||
for bit. The chunk size changes the GEMM tiles the BLAS picks and with them the last bits of the prompt state. This is
|
||||
the size of the llama.cpp CPU-against-HIP difference on the same file, not a quality loss. A prompt shorter than the
|
||||
chunk (every chat turn after the first) is one chunk either way.
|
||||
|
||||
## Speed and exactness
|
||||
|
||||
Every change on the decode path gives the old kernels' bits. `STRATA_Q36_VERIFY=1 strata ... --no-graph` runs the old
|
||||
kernel beside each new one on the real activations and compares them bit for bit. `STRATA_Q36_OLD_KERNELS=1` switches
|
||||
the new kernels off. Gate/up for Q4_K experts uses the V2K kernels (`STRATA_EXPERT_V2K`, the default on gfx12).
|
||||
|
||||
> **Measured on:** AMD Radeon AI PRO R9700 32 GB (HIP gfx1201), UD-Q4_K_XL, Strata 0.1.45 against the engine before it,
|
||||
> 5 interleaved pairs, medians; llama.cpp 972b5fc HIP on the same file (`-fa 1`).
|
||||
|
||||
| | Before | 0.1.45 | llama.cpp |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| Decode tg128 | 83.2 tok/s | 117.8 | 81.9 to 82.1 |
|
||||
| Prompt pp4096 | 3,453 | 6,220 (chunk 4096) | 3,794 (`-ub 2048`) / 3,042 (`-ub 512`) |
|
||||
|
||||
More numbers are in [History](#history--older-measurements).
|
||||
|
||||
## Limits
|
||||
|
||||
- Eight slots at most (the window holds 8 rows). `--batch-mtp` shares those rows between slots and proposals.
|
||||
- A sampling request in a batch decodes one token per window without proposals.
|
||||
- `logit_bias` is not available on this engine path. The server refuses a request that has it (as it does with
|
||||
`--batch` for Flash-Next).
|
||||
- A slot that stopped being alone goes back to the solo path only through the server's own hand-over. The MTP block's
|
||||
cache has holes for the tokens a slot decoded in batch windows, so the drafts of a later solo stretch are a little
|
||||
worse. The text is not affected.
|
||||
- Intel Arc (SYCL) and Mac do not run this family.
|
||||
|
||||
## Testing
|
||||
|
||||
The gates are in `tools/qwen35/` and talk to the engine through its serve protocol (no server needed). Run them under
|
||||
the box's GPU lock.
|
||||
|
||||
| Script | Checks |
|
||||
| --- | --- |
|
||||
| `q36_solo_gate.py` + `q36_cmp_gate.py` | The 10 parity prompts x 128 greedy tokens through `GEN`, compared with another build's. |
|
||||
| `q36_batch_gate.py --slots N --waves --long 2500` | Every slot's tokens equal its solo tokens, slots reused, a long prompt read beside decoding slots. |
|
||||
| `q36_proto_edge.py` | `max_new` 1, `BSTOP`, a busy slot, `STOP` during a prompt read, held-state reuse, sampled requests, the context's end. |
|
||||
| `q36_interleave_probe.py` | What the decoding slots see while a long prompt is read. |
|
||||
| `q36_vision_cmp.py` (with `q36_mtmd_dump.cpp` built against llama.cpp) | The picture comparison against llama.cpp. |
|
||||
| `q36_vision_proto.py` | Pictures through `GENI` / `BGENI` (solo, in a batch, the conversation cache). |
|
||||
| `q36_vision_serve.py`, `q36_serve_par.py`, `q36_serve_handoff.py` | The same through the server. |
|
||||
| `q36_par_bench.py` | The aggregate-throughput A/B against `llama-server -np N`. |
|
||||
|
||||
## History / older measurements
|
||||
|
||||
### Pictures checked against llama.cpp
|
||||
|
||||
> **Measured on:** R9700 32 GB (HIP), UD-Q4_K_XL, Strata 0.1.45. llama.cpp's `mtmd` and its own graph, the encoder on the
|
||||
> CPU on both sides, the same prompt ids, llama.cpp's answer fed to both, logits compared step by step over 24 steps.
|
||||
> The floor is llama.cpp's CPU backend against its own HIP run.
|
||||
|
||||
| Picture and question | Prompt tokens | First token | Mean KL, llama.cpp HIP vs Strata | Floor |
|
||||
| --- | ---: | --- | ---: | ---: |
|
||||
| 640x400, text in the image | 288 | same | 0.029 | 0.056 |
|
||||
| 640x400, list the shapes | 294 | same | 0.035 | 0.039 |
|
||||
| 800x450, text in the image | 378 | same | 0.021 | 0.031 |
|
||||
| 800x450, describe the shapes | 381 | same | 0.0005 | 0.0019 |
|
||||
| 1024x1024, the title (1,024 image tokens, read in 3 chunks) | 1,042 | same | 0.0022 | 0.0026 |
|
||||
| two pictures in one request, the text in each | 655 | differs (a near tie: the floor's KL at this step is 0.16 too) | 0.014 | 0.022 |
|
||||
| 1024x1024, the title (1,024 image tokens, 3 chunks) | 1,042 | same | 0.0022 | 0.0026 |
|
||||
| two pictures in one request | 655 | differs (a near tie; the floor's KL at this step is 0.16) | 0.014 | 0.022 |
|
||||
|
||||
The answers are the same text as llama.cpp's: `STRATA 42` (with llama.cpp's own spelling ` STRATA42`), `CODE 4217`, `Red circle / Blue square
|
||||
/ Green triangle`, `A red square is positioned on the left ... A blue circle is positioned on the right ...`. With the picture options on,
|
||||
a request without pictures gives the same tokens as without them (10 prompts x 128 greedy tokens, identical).
|
||||
The answers are the same text as llama.cpp's. With the picture options on, a request without pictures gives the same
|
||||
tokens as without them (10 prompts x 128 greedy tokens).
|
||||
|
||||
Pictures work with several requests at once, with the MTP drafts and with the conversation cache: the same conversation sent again (a chat
|
||||
app resends its pictures) continues from the held state when its pictures are the same (they are matched by the server's key for each
|
||||
picture); a different picture starts that conversation again.
|
||||
### Aggregate decode speed, several requests at once
|
||||
|
||||
## Several requests at once
|
||||
> **Measured on:** same prompts, greedy, 128 new tokens each, wall time from first request to last answer, 5
|
||||
> interleaved pairs, medians, Strata 0.1.45 with `--batch 8 --mtp 0` against `llama-server -np N`.
|
||||
|
||||
`"parallel": N` in the model's config (`strata-qwen36-<size>.json`, or `--parallel N` at setup) makes the engine keep N conversations open
|
||||
(`--batch N`, up to 8) and decode them together: every window carries one token of each conversation, so the dense weights, the head and
|
||||
every routed expert two conversations share are read once for all of them. The server needs no change: it speaks the same engine protocol
|
||||
as for Flash-Next ([BATCHING.md](BATCHING.md)). A request that arrives while another is read waits only for that read: the prompt of a
|
||||
new request is read **between** the decode windows of the others, and a conversation continues in the slot that holds its start (or the
|
||||
engine copies the longer held start from another slot). A 3,008-token prompt read in 0.95 s (3,200 tok/s, 6 chunks) beside three decoding
|
||||
slots: each of them paused at most 160 ms (one chunk) instead of the whole read, and produced 20 tokens meanwhile (one window per chunk,
|
||||
the default); `STRATA_Q36_READ_WINDOWS=4` gives them 65 tokens during a 1.14 s read (more of the time, a slower read).
|
||||
A request that runs alone (the solo path, with its MTP drafts) and is moved into a batch slot when another arrives, and back when it is
|
||||
alone again, gives the text it gives running alone (the server's hand-over reuses the held state).
|
||||
| Card | Requests | Strata tok/s | llama.cpp | Ratio |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| R9700 32 GB, UD-Q4_K_XL | 1 | 83.9 | 73.8 | 1.14 |
|
||||
| | 2 | 173.5 | 107.4 | 1.61 |
|
||||
| | 4 | 252.0 | 165.0 | 1.53 |
|
||||
| | 8 | 325.4 | 208.1 | 1.56 |
|
||||
| RTX 3060 12 GB, hybrid (llama.cpp `-ncmoe 28 -t 8`, the best split of a sweep) | 1 | 64.0 | 41.1 | 1.56 |
|
||||
| | 2 | 85.1 | 52.5 | 1.62 |
|
||||
| | 4 | 99.3 | 63.0 | 1.58 |
|
||||
|
||||
Aggregate decode speed, N requests at once, same prompts, greedy, 128 new tokens each, wall time from the first request to the last
|
||||
answer, **R9700 32 GB, UD-Q4_K_XL, llama.cpp `llama-server -np N`** (5 interleaved pairs, median; Strata `--batch 8 --mtp 0`):
|
||||
All pairs were faster for Strata. Every slot's tokens equal the tokens of the same request alone.
|
||||
|
||||
| Requests at once | Strata | llama.cpp | Ratio | Pairs faster |
|
||||
| ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | 83.9 tok/s | 73.8 | 1.14 | 5/5 |
|
||||
| 2 | 173.5 | 107.4 | 1.61 | 5/5 |
|
||||
| 4 | 252.0 | 165.0 | 1.53 | 5/5 |
|
||||
| 8 | 325.4 | 208.1 | 1.56 | 5/5 |
|
||||
### Several requests with the base engine, and with `--batch-mtp`
|
||||
|
||||
**RTX 3060 12 GB, UD-Q4_K_XL in the hybrid mode** (each engine with the slots it needs: `--batch N` takes room from the expert slots; llama.cpp
|
||||
`llama-server -np N -ncmoe 28 -t 8`, the best split of a sweep on this card; 5 interleaved pairs, medians; the box was shared with other
|
||||
jobs between runs, never during one):
|
||||
> **Measured on:** R9700 32 GB, 5 interleaved pairs, Strata 0.1.45 (before the speed work below).
|
||||
|
||||
| Requests at once | Strata | llama.cpp | Ratio | Pairs faster |
|
||||
| ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | 64.0 tok/s | 41.1 | 1.56 | 5/5 |
|
||||
| 2 | 85.1 | 52.5 | 1.62 | 5/5 |
|
||||
| 4 | 99.3 | 63.0 | 1.58 | 5/5 |
|
||||
| Requests | `--mtp 0` | `--mtp 2 --batch-mtp` | Ratio |
|
||||
| ---: | ---: | ---: | ---: |
|
||||
| 1 | 83.7 tok/s | 134.0 | 1.60 |
|
||||
| 2 | 173.1 | 207.0 | 1.20 |
|
||||
| 4 | 251.0 | 275.0 | 1.10 |
|
||||
| 8 | 324.0 | 320.5 | 0.99 (no row to spare: no proposals) |
|
||||
|
||||
Every slot's tokens equal the tokens of the same request alone (the base engine's), at 2, 4 and 8 slots, with a 2.5K-token prompt read beside
|
||||
decoding slots, with slots reused for later requests, with pictures and with sampling (same seed). Through the server: four concurrent chat
|
||||
requests give the texts of the same four one after the other.
|
||||
> **Measured on:** hybrid cards, UD-Q4_K_XL, medians of 5 interleaved pairs, `--mtp 2`; "together" is the total tokens
|
||||
> per second of N simultaneous requests against one request at a time with the draft layer.
|
||||
|
||||
**MTP drafts.** A request that has the engine to itself keeps its MTP drafts (the server sends it alone while nothing else runs). With
|
||||
several requests the batch window has no drafts unless `--batch-mtp` is in the config's `args` (opt-in): every slot's window then also
|
||||
carries the proposal its MTP block made (one per slot, up to the rows to spare in the 8-row window) and a confirmed proposal moves two
|
||||
tokens in the window. Same prompts, 5 interleaved pairs, R9700:
|
||||
|
||||
| Requests at once | `--mtp 0` | `--mtp 2 --batch-mtp` | Ratio | Pairs faster |
|
||||
| ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | 83.7 tok/s | 134.0 | 1.60 | 5/5 |
|
||||
| 2 | 173.1 | 207.0 | 1.20 | 5/5 |
|
||||
| 4 | 251.0 | 275.0 | 1.10 | 5/5 |
|
||||
| 8 | 324.0 | 320.5 | 0.99 | 0/5 (no row to spare: no proposals) |
|
||||
|
||||
The text is the same with and without it (every slot equal to its solo tokens).
|
||||
|
||||
**What a slot costs.** Each slot keeps its own cache in VRAM: 1.5 GB at a context of 32K (fp32 keys and values of the 10 attention layers
|
||||
and the MTP block, 45 KB per token) and 0.07 GB of recurrent state. Setup writes the context for one conversation, so N slots take N times
|
||||
that from the room the experts would have: on a card that does not hold every expert (12 and 16 GB, the hybrid mode) that is room the
|
||||
expert slots lose, and a batch reads more distinct experts from RAM. Setup **offers `--parallel`** where the sessions do not eat the experts' room: up to 3-4 where every expert still fits (3 on a 32 GB card at
|
||||
32K), and on a card that does not (the hybrid mode) the count whose sessions take at most 35% of the room the experts would have, so it
|
||||
depends on the context you chose (12 GB: 4 slots at 8K, none at 32K; 16 GB: 2 slots at 32K). It says the price in the same breath, asks
|
||||
"How many requests at once?" (Enter keeps one at a time), and any number you give is kept. Measured on the hybrid cards (UD-Q4_K_XL, medians of 5
|
||||
interleaved pairs, `--mtp 2` as setup writes it; "together" = total tokens per second of N simultaneous requests, against ONE request at a time
|
||||
with the draft layer):
|
||||
|
||||
| card, context | slots | one request alone, vs 1 slot | N together, vs one at a time |
|
||||
| Card, context | Slots | One request alone, against 1 slot | N together, against one at a time |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| RTX 3060 12 GB, 8K | 2 / 4 / 8 | -3.5% / -4.3% / -12% | 0.99x / 1.15x / 1.10x |
|
||||
| RTX A4000 16 GB, 32K | 2 / 4 | -10.6% / -29.9% | 1.03x / 0.76x |
|
||||
| Tesla P100 16 GB, 32K | 2 / 4 | -7% / -23% | 0.98x / 0.81x |
|
||||
|
||||
Without the draft layer (`--mtp 0`) the same slots give 1.26-1.64x together (3060 at 8K: 4 slots 1.64x for -5.5% alone), but a single request
|
||||
with drafts is already about 1.6x as fast as one without, so on a small card the main gain is that a second user starts at once; the
|
||||
speed gain is small. On a card that holds every expert (R9700 32 GB, 8K prompts) 4 slots run 1.45x one-at-a-time-with-drafts (1.59x with
|
||||
`--batch-mtp`). The engine says what the sessions take before it allocates them and refuses (with the numbers) a count that cannot fit.
|
||||
Without the draft layer (`--mtp 0`) the same slots give 1.26x to 1.64x together (3060 at 8K: 4 slots 1.64x for -5.5%
|
||||
alone). A single request with drafts is already about 1.6x as fast as one without, so on a small card the main gain is
|
||||
that a second user starts at once. On a card that holds every expert (R9700 32 GB, 8K prompts), 4 slots run 1.45x
|
||||
one-at-a-time-with-drafts (1.59x with `--batch-mtp`).
|
||||
|
||||
In the hybrid mode the text of a request can differ a little between running alone and running beside others: the experts that sit on the
|
||||
GPU or on the CPU change with the routing of all the requests, and the two compute with different rounding. With the slot tables fixed (`STRATA_Q36_FIXED_SLOTS=1`, a test setting) every slot equals its
|
||||
solo run there too, with and without `--batch-mtp`.
|
||||
### Prompt read beside decoding slots
|
||||
|
||||
## Speed (0.1.45)
|
||||
> **Measured on:** R9700 32 GB, Strata 0.1.45, three decoding slots.
|
||||
|
||||
Decode and prompt kernels of this engine path were rewritten for speed; **every change on the decode path gives the old kernels' bits**
|
||||
(`STRATA_Q36_VERIFY=1 strata ... --no-graph` runs the old kernel beside each new one on the real activations and compares them bit for bit:
|
||||
0 mismatches in 18,360 / 8,187 / 6,399 comparisons on the R9700 and the RTX 3060; `STRATA_Q36_OLD_KERNELS=1` switches them all off). The
|
||||
10 parity prompts (greedy 128) give the base engine's tokens, plain and with `--mtp 2`. Where the prompt's chunk size changes, the numbers
|
||||
change a little: see "Prompt chunk".
|
||||
A 3,008-token prompt was read in 0.95 s (3,200 tok/s, 6 chunks). Each decoding slot paused at most 160 ms (one chunk)
|
||||
and produced 20 tokens meanwhile. With `STRATA_Q36_READ_WINDOWS=4` they produced 65 tokens during a 1.14 s read.
|
||||
|
||||
**What was slow** (rocprofv3, R9700, UD-Q4_K_XL, plain decode, 11.25 ms of kernel time per token): the big Q8_0 products (attention, GDN
|
||||
projections, head: 1.9 GB per token) run at 530 GB/s, the memory bound. The rest did not: the shared expert's and the attention k/v products
|
||||
(140 per token) at 18 us, the Q4_K gate/up kernel at 250 GB/s, `k_moe_out` (one block) 17 us, `k_group` (one thread walking 256 experts) 9 us.
|
||||
Now: those products one warp per row (a launch of 2048 warps took 21 us on gfx1201 where 1024 warps took 5 us, so the grid is capped), shared
|
||||
gate+up+SwiGLU+q8_1 in one launch, the MoE end with 1024 threads and the next layer's q8_1 image, add+norm+q8_1 in one launch, router top-k and
|
||||
grouping in one launch, the router and the shared gate in one product, Q4_K gate/up on the V2K kernels (`STRATA_EXPERT_V2K`, gfx12 default).
|
||||
The prompt's GDN conv runs in tiles of 16 tokens side by side and the delta rule loads one token ahead with two barriers per token (both exact).
|
||||
### Speed work in 0.1.45
|
||||
|
||||
R9700 32 GB (p620, HIP gfx1201), UD-Q4_K_XL, base = the engine before this work, 5 interleaved pairs, 3 repetitions each, medians; llama.cpp
|
||||
972b5fc HIP on the same file (`-fa 1`, best of `-ub` 512 / 2048):
|
||||
> **Measured on:** R9700 32 GB (HIP gfx1201), UD-Q4_K_XL, 5 interleaved pairs, 3 repetitions each, medians; base = the
|
||||
> engine before 0.1.45.
|
||||
|
||||
| | base | 0.1.45 | ratio | pairs faster | llama.cpp |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| decode tg128 | 83.2 tok/s | 117.8 | 1.42 | 5/5 | 81.9-82.1 |
|
||||
| prompt pp4096 | 3,453 | 6,220 (chunk 4096) | 1.80 | 5/5 | 3,794 (-ub 2048) / 3,042 (-ub 512) |
|
||||
| `--mtp 2`, 10 prompts greedy 128 | 167.7 | 183.7 | 1.10 | 3/3 | 129 (draft-mtp, n-max 2, earlier run) |
|
||||
| 1 / 2 / 4 / 8 requests at once (`--batch 8 --mtp 0`) | 85.2 / 182.4 / 271.9 / 361.1 | 122.4 / 205.1 / 303.3 / 386.9 | 1.44 / 1.12 / 1.12 / 1.07 | 5/5 | |
|
||||
| | Base | 0.1.45 | Ratio |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| `--mtp 2`, 10 prompts greedy 128 | 167.7 tok/s | 183.7 | 1.10 |
|
||||
| 1 / 2 / 4 / 8 requests (`--batch 8 --mtp 0`) | 85.2 / 182.4 / 271.9 / 361.1 | 122.4 / 205.1 / 303.3 / 386.9 | 1.44 / 1.12 / 1.12 / 1.07 |
|
||||
| Prompt pp4096 at chunk 512 / 1024 / 2048 / 4096 | | 3,453 / 4,786 / 5,766 / 6,223 | |
|
||||
|
||||
UD-Q6_K_XL on the same card (every main layer's experts in VRAM, so the prompt chunk stays 512): decode 80.4 -> 106.9 tok/s (1.33), pp4096
|
||||
2,160 -> 2,200 (llama.cpp 3,486 at -ub 2048, 2,738 at -ub 512; decode 81.1), `--mtp 2` 154.7 -> 167.3. `--chunk 2048` there moves the experts
|
||||
to the hybrid placement (205 of 256 per layer in VRAM): prompt 3,500 tok/s, decode 94 - an option, not the default.
|
||||
UD-Q6_K_XL on the same card (every main layer's experts in VRAM, chunk stays 512): decode 80.4 to 106.9 tok/s, pp4096
|
||||
2,160 to 2,200 (llama.cpp 3,486 at `-ub 2048`, 2,738 at `-ub 512`), `--mtp 2` 154.7 to 167.3. `--chunk 2048` there moves
|
||||
the experts to the hybrid placement (205 of 256 per layer in VRAM): prompt 3,500 tok/s, decode 94. That is an option,
|
||||
not the default.
|
||||
|
||||
RTX 3060 12 GB (P2003, CUDA sm_86, 20 threads), UD-Q4_K_XL in the hybrid mode (37 of 256 experts per layer in VRAM): the GPU part of a
|
||||
token is the dense Q8_0 at the card's bandwidth (6 ms of the 17) and the rest waits for the CPU pool (9 ms of experts), so only the small
|
||||
kernels' share is gained: decode tg128 58.5 -> 59.7 tok/s (+2%), pp2048 1,958 -> 2,022 (+3.3%), 10/10 identical, plain and `--mtp 2`
|
||||
(84.3 -> 84.7). llama.cpp 972b5fc CUDA (`-ncmoe 26 -t 8 -ub 2048`): pp2048 1,328, tg128 48-52.
|
||||
> **Measured on:** RTX 3060 12 GB (CUDA sm_86, 20 threads), UD-Q4_K_XL in the hybrid mode (37 of 256 experts per layer
|
||||
> in VRAM), Strata 0.1.45.
|
||||
|
||||
**Prompt chunk (the default changed).** The MoE prompt kernels read every expert once per chunk, so a bigger chunk feeds them more rows per
|
||||
expert: R9700 pp4096 at chunk 512 / 1024 / 2048 / 4096 = 3,453 / 4,786 / 5,766 / 6,223 tok/s. With every expert in VRAM the engine now takes the
|
||||
largest of 4096 / 2048 / 1024 that fits beside the experts, the 0.77 GiB reserve and the sessions' caches without giving the packed decode
|
||||
copies back (8K-64K context: 4096, 100K: 2048, 131K: 512; `--batch 2..4`: 1024 at most so the other slots are not held for a whole chunk,
|
||||
`--batch 8`: 512). The hybrid mode keeps 2048 (a bigger chunk takes expert slots: RTX 3060 chunk 4096 reads prompts 5% faster and decodes 13%
|
||||
slower) and goes down to 1024 / 512 on a card with 7.5 / 5.5 GiB free. **`--prefill 512` (or `STRATA_Q36_CHUNK=512`) gives the old size, and
|
||||
then the text is the base engine's, bit for bit.** The chunk size changes the GEMM tiles the BLAS picks and with them the last bits of the
|
||||
prompt state: KL against the 512 chunk on six real 1.5-3.2K-token documents (48 teacher-forced rows each): chunk 1024 4.0e-3, 2048 4.2e-3,
|
||||
4096 3.6e-3 (per prompt 2.3e-3 .. 5.9e-3), top-1 agreement 44-48 of 48 rows - the size of the llama.cpp CPU-vs-HIP difference on the same
|
||||
file (4.2e-3), not a quality loss. Greedy 64 tokens after those six documents: the default chunk gives the base engine's text for 2 of 6,
|
||||
`--prefill 512` for 6 of 6. A prompt shorter than the chunk (every chat turn after the first) is one chunk either way: unchanged.
|
||||
The maintainers keep the new default's reference outputs (six documents, greedy 64) and check later changes against them.
|
||||
Decode tg128 58.5 to 59.7 tok/s (+2%), pp2048 1,958 to 2,022 (+3.3%), 10/10 prompts identical, plain and with `--mtp 2`.
|
||||
llama.cpp 972b5fc CUDA (`-ncmoe 26 -t 8 -ub 2048`): pp2048 1,328, tg128 48 to 52. The GPU part of a token is the dense
|
||||
Q8_0 at the card's bandwidth and the rest waits for the CPU pool, so only the small kernels' share is gained.
|
||||
|
||||
**Tried and dropped.** The MTP drafts' head over a subset of the vocabulary (`--draft-vocab N`, kept as an opt-in; the answer never
|
||||
changes): the first 16K-131K rows gain 5-8% on English and code but lose 10-15% on Chinese / Russian / Arabic / Hindi (70% of a Chinese
|
||||
prompt's token ids are above 65,536), and a candidate list built from the tokens seen so far loses on both. The shared expert on a second
|
||||
stream (captured into the decode graph) decodes at 36 tok/s instead of 117 with this ROCm. Chunked (WY) GDN and the PCIe-assisted hybrid decode
|
||||
were not attempted.
|
||||
|
||||
## Not done / limits
|
||||
|
||||
- Eight slots at most (the window holds 8 rows); `--batch-mtp` shares those rows between slots and proposals.
|
||||
- A sampling request in a batch decodes one token per window without proposals. `logit_bias` is not available on this engine path: the
|
||||
server refuses a request that has it (as it does with `--batch` for Flash-Next).
|
||||
- A slot that stopped being alone goes back to the solo path only through the server's own hand-over; the MTP block's cache has holes for
|
||||
the tokens a slot decoded in batch windows (the drafts of a later solo stretch are a little worse; the text is not affected).
|
||||
- Intel Arc (SYCL) does not run this family.
|
||||
|
||||
## Testing
|
||||
|
||||
The gates are in `tools/qwen35/` and talk to the engine through its serve protocol (no server needed); run them under the box's GPU lock:
|
||||
|
||||
- `q36_solo_gate.py` + `q36_cmp_gate.py`: the 10 parity prompts x 128 greedy tokens through `GEN`, compared with another build's.
|
||||
- `q36_batch_gate.py --slots N --waves --long 2500`: every slot's tokens equal its solo tokens, slots reused, a long prompt read beside
|
||||
decoding slots. `q36_proto_edge.py`: `max_new` 1, `BSTOP`, a busy slot, `STOP` during a prompt read, held-state reuse, sampled requests,
|
||||
the context's end. `q36_interleave_probe.py`: what the decoding slots see while a long prompt is read.
|
||||
- `q36_vision_cmp.py` (with `q36_mtmd_dump.cpp` built against llama.cpp): the picture comparison above; `q36_vision_proto.py`: pictures through
|
||||
`GENI` / `BGENI` (solo, in a batch, the conversation cache); `q36_vision_serve.py` and `q36_serve_par.py` / `q36_serve_handoff.py`:
|
||||
the same through the server.
|
||||
- `q36_par_bench.py`: the aggregate-throughput A/B against `llama-server -np N`.
|
||||
KL against the 512 chunk on six real 1.5K to 3.2K-token documents (48 teacher-forced rows each): chunk 1024 4.0e-3,
|
||||
2048 4.2e-3, 4096 3.6e-3, top-1 agreement 44 to 48 of 48 rows. Greedy 64 tokens after those documents: the default chunk
|
||||
gives the base engine's text for 2 of 6, `--prefill 512` for 6 of 6.
|
||||
@@ -0,0 +1,76 @@
|
||||
# Strata guides
|
||||
|
||||
**New here? Start with [Install Strata](INSTALL.md).** Download the one installer for your system (Windows, Linux or Mac), or install the Terminal with one line. Both use released engine packages. This documentation is for version 0.2.5.
|
||||
|
||||
## Use Strata
|
||||
|
||||
| I want to… | Guide |
|
||||
| --- | --- |
|
||||
| Install or update | [Desktop app and Terminal](INSTALL.md) |
|
||||
| Choose a model and check memory | [Models and quantizations](MODELS.md) |
|
||||
| Run DeepSeek V4.1-Flash (app and Terminal) | [DeepSeek V4.1-Flash](MODELS.md#deepseek-v41-flash) |
|
||||
| Start from the command line | [Terminal launch](TERMINAL_LAUNCH.md) |
|
||||
| Connect a coding tool | [Tool connections](TERMINAL_LAUNCH.md#connect-your-apps-strata-connect) |
|
||||
| Fix an install or startup problem | [Troubleshooting](TROUBLESHOOTING.md) |
|
||||
| Check privacy, downloads and network access | [Safety and privacy](SAFETY.md) |
|
||||
|
||||
## Hardware
|
||||
|
||||
Use the guide for your GPU and operating system. Model and feature support can differ between backends.
|
||||
|
||||
| Hardware | Guide |
|
||||
| --- | --- |
|
||||
| NVIDIA RTX 20 / 30 / 40 / 50 | [Install](INSTALL.md#what-you-need) |
|
||||
| AMD Radeon (RX and AI PRO R9700) | [AMD / HIP](AMD_HIP.md) |
|
||||
| AMD Ryzen AI Max / Strix Halo | [Strix Halo](STRIX_HALO.md) |
|
||||
| Intel Arc on Linux | [Intel Arc](INTEL_ARC.md) |
|
||||
| Apple Silicon | [Mac / Metal](MAC.md) |
|
||||
| More than one GPU | [Multiple GPUs](MULTI_GPU.md) |
|
||||
| Older GPUs | [Experimental hardware](OLDER_GPUS.md) |
|
||||
|
||||
## Developers and contributors
|
||||
|
||||
- [Build or run from a source checkout](SOURCE_INSTALL.md)
|
||||
- [Docker](DOCKER.md)
|
||||
- [Contribution rules](../CONTRIBUTING.md)
|
||||
- [Community benchmarks](COMMUNITY_BENCHMARKS.md) and the [standard benchmark protocol](BENCHMARK_PROTOCOL.md)
|
||||
- [How the engine works](HOW_IT_WORKS.md), [technical reference](DETAILS.md) and [Terminal reference](TERMINAL.md)
|
||||
- [App development](../apps/host/README.md) and [packaging](../packaging/README.md)
|
||||
|
||||
## AI tools and media
|
||||
|
||||
- [Install for a user](AI_SETUP.md)
|
||||
- [Verified facts, model availability and API reference](AI_REFERENCE.md)
|
||||
- [Articles and videos](MEDIA_KIT.md)
|
||||
- [Repository instructions for agents](../AGENTS.md)
|
||||
|
||||
## Which instructions apply?
|
||||
|
||||
`strata launch` belongs to the installed Terminal. `START-HERE.bat`, `SETUP.bat`, `UPDATE.bat` and `setup.sh`
|
||||
belong to a **source checkout**. Raw engine commands are a third interface. Their flags are not interchangeable:
|
||||
Terminal launch uses `--gpu` and `--vision on|off`; source setup uses `--gpus` and `--vision yes|no`.
|
||||
|
||||
Release notes, design proposals and benchmark reports describe their named versions and machines. They remain
|
||||
available as historical evidence; use the guides above for a new installation. A published result on one GPU
|
||||
does not establish support or speed on every GPU.
|
||||
|
||||
## Writing a guide page
|
||||
|
||||
- Installation steps live in [INSTALL.md](INSTALL.md), source steps in [SOURCE_INSTALL.md](SOURCE_INSTALL.md). The public READMEs stay short.
|
||||
- Check command examples against the parser and filenames against the release assets.
|
||||
- A guide describes the current release. Version history, fixed bugs and superseded settings go in a closing
|
||||
**History / older measurements** section, in `bench/results`, or in the release notes. Do not mix them into instructions.
|
||||
|
||||
Every guide page has this shape:
|
||||
|
||||
| Part | Content |
|
||||
| --- | --- |
|
||||
| Title | `# Name of the task or topic` |
|
||||
| Purpose | One to three lines: what the page is for and who it is for |
|
||||
| Quick start or Steps | The shortest working path, commands in code blocks |
|
||||
| Details | `##` sections; option and flag lists as tables; short paragraphs |
|
||||
| History / older measurements | Optional, last. Dated numbers only |
|
||||
|
||||
Label every measurement the same way, directly above the number or table:
|
||||
|
||||
> **Measured on:** hardware (GPU, CPU, RAM), model or set, Strata version, workload.
|
||||
+97
-194
@@ -1,227 +1,130 @@
|
||||
# CUDA1–3 expert experiment
|
||||
# CUDA1-3 expert caches and the peer tier (experimental)
|
||||
|
||||
CUDA0 keeps the dense layers, KV/state, MTP and its existing expert cache. Up to
|
||||
three independent expert caches fill CUDA1, CUDA2 and CUDA3 in that order. Each
|
||||
expert has one owner: CUDA0, one secondary GPU, or the CPU pool. Each secondary
|
||||
GPU computes its rows while the CPU pool works; the results return through
|
||||
private pinned host buffers before CUDA0 continues the layer. The GPU launches
|
||||
can overlap, but each layer waits for all three.
|
||||
This page is for people who tune the engine by hand and want a second, third or fourth card to hold extra experts. These are
|
||||
experimental engine settings. For ordinary multi-GPU use in the app or `strata launch`, see [MULTI_GPU.md](MULTI_GPU.md). The
|
||||
settings go into the run config (`strata-*.json`) as `"args"` entries; they are engine flags, not `strata launch` flags.
|
||||
|
||||
The shipped profile ranks 8,000 `(layer, expert)` pairs. CUDA0 takes its usual
|
||||
`--expert-cache auto` allocation. With only CUDA1, it takes the first remaining
|
||||
pairs. When CUDA2 or CUDA3 is enabled, the remaining profiled pairs are assigned
|
||||
round robin to the secondary GPUs. The ranked list is then extended with all
|
||||
other pairs in expert-then-layer order and distributed in the same way. This
|
||||
fills the additional VRAM and lets all secondary GPUs see frequent experts, but
|
||||
unranked experts may hardly ever be routed. The startup
|
||||
log reports slots and GiB per card; each request reports the number of expert
|
||||
entries actually computed on each secondary GPU. Remote outputs are packed
|
||||
before returning over PCIe/USB4: the request log compares MiB transferred
|
||||
with the previous full-row transfer. The input and routing metadata use pinned
|
||||
host staging. This reduces transfer traffic and launch overhead, but does not
|
||||
release the expert arena in system RAM or significantly change VRAM use.
|
||||
> **On this page:** [How it works](#how-it-works) · [Helper caches](#helper-caches) · [Faster helper decode](#faster-helper-decode---remote-expert-opt) ·
|
||||
> [Peer tier](#peer-tier---peer-device) · [Resident RAM mode](#the-resident-ram-mode-with-the-helper-caches) ·
|
||||
> [Choosing the primary device](#choosing-the-primary-device-on-windows-strata_primary_device) · [History](#history--older-measurements)
|
||||
|
||||
The optional `--gpu-placement layer` gives each layer to exactly one secondary
|
||||
GPU (layer number modulo the number of enabled cards). It fills that card with
|
||||
the most frequently routed experts of its layers first, then the unranked ones.
|
||||
This can reduce USB4 device switches and synchronization when several secondary
|
||||
cards serve the same layer in the default `stripe` mode. All three cards still
|
||||
keep their requested expert slots; the primary CUDA0 cache and CPU fallback
|
||||
continue to work. The log counts active layer launches on each card, so the
|
||||
tradeoff can be measured alongside tokens per second. This placement is
|
||||
experimental and may be slower if a given layer needs more than one card's
|
||||
compute capacity.
|
||||
## How it works
|
||||
|
||||
The experiment requires the corresponding visible CUDA devices and a CUDA
|
||||
enabled build. Each card must keep at least 512 MiB free. Tiers must be enabled
|
||||
in order. Under WDDM (Windows, WSL2), CUDA registration of the host expert arena is capped at
|
||||
8 GiB to leave room for the contexts and MTP on CUDA0 (`STRATA_ARENA_PIN_GIB` overrides it; on Linux the whole
|
||||
arena is registered). The rest remains
|
||||
available to the CPU pool; the PCIe expert path is available only for the
|
||||
registered layers. Without secondary GPUs, the original uncapped registration
|
||||
behavior applies.
|
||||
CUDA0 keeps the dense layers, the KV and state, the draft layer (MTP) and its own expert cache. Up to three independent expert
|
||||
caches fill CUDA1, CUDA2 and CUDA3 in that order. Each expert has one owner: CUDA0, one helper card, or the CPU pool. Each
|
||||
helper computes its rows while the CPU pool works, and the results return through private pinned host buffers before CUDA0
|
||||
continues the layer. Each layer waits for all helpers.
|
||||
|
||||
On an existing Linux installation, apply the patch to its source tree, then run:
|
||||
- No peer-to-peer access is needed. Prompt prefill still uses CUDA0; helpers serve decode, including draft verification.
|
||||
- Each helper card must keep at least 512 MiB free. Enable the tiers in order (CUDA1 before CUDA2).
|
||||
- The startup log reports slots and GiB per card, and each request reports how many expert entries each helper computed.
|
||||
Compare identical requests at one, two and four GPUs. VRAM use alone does not show useful offload.
|
||||
- Longer contexts reserve more KV on CUDA0 and shrink its automatic expert cache.
|
||||
- Under WDDM (Windows, WSL2) only 8 GiB of the host expert arena is registered with CUDA when helpers are on
|
||||
(`STRATA_ARENA_PIN_GIB` overrides it). On Linux the whole arena is registered.
|
||||
- The CUDA0 expert-cache GPU hit path gives outputs that diverge from cache-off runs. Treat performance as experimental until the
|
||||
generated tokens are validated.
|
||||
|
||||
```sh
|
||||
./setup.sh --setup --gpu1-experts 5000 --gpu2-experts 5000 --gpu3-experts 5000
|
||||
```
|
||||
## Helper caches
|
||||
|
||||
On Windows, from PowerShell in the installation folder:
|
||||
| Engine flag | Meaning |
|
||||
| --- | --- |
|
||||
| `--expert-cache-device1 N\|auto` | Fill CUDA1 with N experts, or as many as fit. |
|
||||
| `--expert-cache-device2 N\|auto` | Fill CUDA2 with more experts. |
|
||||
| `--expert-cache-device3 N\|auto` | Fill CUDA3 with more experts. |
|
||||
| `--expert-cache-remote-placement stripe\|layer` | `stripe` (default) spreads the ranked expert list over the helpers. `layer` gives each layer to one helper (layer number modulo the number of cards), which can reduce device switches on USB4 but may be slower. |
|
||||
|
||||
```powershell
|
||||
.\START-HERE.bat --setup --gpu1-experts 5000 --gpu2-experts 5000 --gpu3-experts 5000
|
||||
```
|
||||
The shipped profile ranks 8,000 `(layer, expert)` pairs. CUDA0 takes its usual `--expert-cache auto` share. With only CUDA1 it
|
||||
takes the first remaining pairs. With CUDA2 or CUDA3 the remaining ranked pairs go round robin to the helpers, then all other
|
||||
pairs in expert-then-layer order, so the extra VRAM fills even though unranked experts may rarely be routed.
|
||||
|
||||
To test the layer placement instead, add `--gpu-placement layer` to the setup
|
||||
command. To go back, run the same command with `--gpu-placement stripe`. After
|
||||
the new engine is built, the mode can also be changed without a rebuild: edit
|
||||
`--expert-cache-remote-placement` in `strata-iq3_xxs.json` from `layer` to
|
||||
`stripe` or vice versa and restart the server. If the argument is absent, the
|
||||
default is `stripe`.
|
||||
`auto` fills each helper from its assigned ranking using actual aligned expert bytes and free VRAM, keeping the 512 MiB allowance.
|
||||
It sizes capacity at startup, not throughput, and a card's truncated tail is not moved to another card. Explicit numbers still work.
|
||||
|
||||
Choose the same model, size, context and image settings as your current
|
||||
installation when prompted. Setup recompiles the engine, updates its server
|
||||
configuration, and starts it. Later starts keep those settings. To tune the
|
||||
slots without recompiling, edit the values after `--expert-cache-device1`,
|
||||
`--expert-cache-device2` and `--expert-cache-device3` in the stored
|
||||
`strata-iq3_xxs.json`, then restart with `.\run-iq3_xxs.bat`.
|
||||
|
||||
Compare identical requests at one, two and four GPUs, preferably with several
|
||||
repeats. VRAM use alone does not show useful offload: compare the per-request
|
||||
CUDA1–3 counts and tokens per second. More GPU contexts and synchronization
|
||||
may lower the speed. Prompt prefill still uses CUDA0; secondary GPUs serve
|
||||
decode, including MTP verification. No peer-to-peer access is required.
|
||||
|
||||
Longer contexts reserve more KV/state memory on CUDA0, which reduces its
|
||||
automatic expert cache. On a 64 GB PC `setup.py` limits IQ3_XXS to 128K even
|
||||
if 262K was selected; the secondary caches still require the host expert arena.
|
||||
To try 262K on 64 GB, choose Q2_0 or IQ2_XS instead and check that CUDA0 still
|
||||
has enough free VRAM after loading. Benchmark long prompts separately from
|
||||
short decode requests.
|
||||
|
||||
The existing warning about the CUDA0 expert-cache GPU hit path still applies:
|
||||
its outputs diverge from cache-off runs. Treat performance as experimental
|
||||
until the generated tokens have been validated.
|
||||
|
||||
## Optional helper decode optimization
|
||||
|
||||
`--remote-expert-opt` (`--serve` only) optimizes the CUDA1-3 helper caches
|
||||
above. The engine's default is off; since 0.1.39b setup adds it to a config on
|
||||
two or more GPUs (`--gpus`, or "use both" at start). It acts only when a helper
|
||||
cache is configured; a layer split runs exactly as before. To leave it out:
|
||||
setup's `--no-remote-expert-opt`, or `"remote_expert_opt": false` in the
|
||||
model's `strata-*.json` (kept when setup runs again). Measured by the PR's
|
||||
author: dual RTX 4090 +63% mixed / +132% code decode over the plain helper
|
||||
path; RTX 5090 + 4090 +28% / +63%. The primary cache avoids admitting experts already held
|
||||
by a helper, and helpers replace cold experts with frequently routed CPU
|
||||
misses using their existing same-layer slots. Each helper reduces its expert
|
||||
outputs to a weighted partial sum on its GPU before returning one vector per
|
||||
token. Tokens with no CPU expert work skip CPU activation quantization.
|
||||
|
||||
For an existing server configuration with CUDA0 as the primary and CUDA1 as a
|
||||
helper, use these engine arguments alongside the model and profile arguments:
|
||||
Example engine arguments beside the model and profile arguments:
|
||||
|
||||
```text
|
||||
--expert-cache auto --expert-cache-device1 auto --remote-expert-opt
|
||||
```
|
||||
|
||||
`--expert-cache-device1`, `--expert-cache-device2` and
|
||||
`--expert-cache-device3` now also accept `auto`: fill each helper from its
|
||||
assigned ranking using actual aligned expert bytes and free VRAM, retaining
|
||||
the existing 512 MiB allowance. Explicit numeric budgets still work. This is
|
||||
startup capacity sizing, not throughput balancing; with several helpers a
|
||||
card's truncated candidate tail is not redistributed to another card.
|
||||
To change slot counts without recompiling, edit the values in the stored config and restart.
|
||||
|
||||
The optimization uses the existing host scheduling and pinned-host transport,
|
||||
not P2P or tensor parallelism. Each layer still waits for its participating
|
||||
helpers. It changes floating-point summation order, so enabled output is not
|
||||
claimed to be bitwise identical. Only two-card CUDA operation has been measured
|
||||
by its author; a community machine measured two cards on HIP (2x RX 6900 XT,
|
||||
PCIe 4.0 x8 each, IQ3_S): there the expert plan's PCIe share had been taking
|
||||
experts the helper already held, which made the primary card wait 43 ms per
|
||||
verify window instead of 19 (decode 40 instead of 66 tok/s). #854 keeps a
|
||||
helper's experts out of that share; before it, `--pcie-frac 0` avoided the
|
||||
cost ([bench/results/2026-10-04-rdna2-helper-pcie-share](../bench/results/2026-10-04-rdna2-helper-pcie-share/README.md)).
|
||||
Three/four cards have not been validated. Without the switch, the
|
||||
existing decode path remains in use. This does not optimize the separate
|
||||
`--peer-device` path below or change its existing incompatibility with helper
|
||||
caches.
|
||||
**Size a helper against the card's real free VRAM.** On Windows `auto` reads the process's WDDM budget. It cannot see the
|
||||
desktop, the vision encoder (a separate process), a streaming host or other apps. A helper auto-sized on a card that also drives
|
||||
the display over-commits, and WDDM backs the excess with system RAM. Use explicit counts on a card that shares its VRAM.
|
||||
|
||||
## Faster helper decode (`--remote-expert-opt`)
|
||||
|
||||
`--remote-expert-opt` (serve mode only) optimizes the helper caches. The primary cache avoids admitting experts a helper already
|
||||
holds, helpers replace cold experts with frequently routed CPU misses, each helper reduces its expert outputs to a weighted
|
||||
partial sum on its GPU before returning one vector per token, and tokens with no CPU expert work skip CPU activation
|
||||
quantization. It does nothing without a helper cache, and a plain layer split runs as before.
|
||||
|
||||
- Setup adds it to a config with two or more GPUs and a helper cache. To leave it out, use setup's `--no-remote-expert-opt` or
|
||||
`"remote_expert_opt": false` in the config (kept when setup runs again).
|
||||
- It changes the floating-point summation order, so output is not claimed to be bitwise identical.
|
||||
- It switches `--pipeline-windows` off ([MULTI_GPU.md](MULTI_GPU.md#pipeline-windows)). Keep one of the two.
|
||||
- Only two-card CUDA operation has been measured by its author. Three and four cards are not validated.
|
||||
- It does not apply to the `--peer-device` path below.
|
||||
|
||||
## Peer tier (`--peer-device`)
|
||||
|
||||
`--peer-device N` puts a second adaptive expert cache on CUDA device N. It
|
||||
takes the ranked pairs CUDA0's cache does not hold, as many as fit. The peer
|
||||
computes the rows of its own experts, for decode windows and for prompt
|
||||
chunks; the activations and the results cross NVLink or another P2P path. The
|
||||
tier adapts while the server runs, like the primary cache. It is an
|
||||
alternative to the CUDA1-3 caches above, not a third tier beside them.
|
||||
`--peer-device N` puts a second adaptive expert cache on CUDA device N. It takes the ranked pairs CUDA0's cache does not hold, as
|
||||
many as fit. The peer computes the rows of its own experts for decode windows and for prompt chunks. Activations and results cross
|
||||
NVLink or another peer-to-peer path. It is an alternative to the helper caches above, not a third tier beside them.
|
||||
|
||||
- `--peer-device N` (default off): enable the tier on CUDA device N (N >= 1).
|
||||
- `--peer-reserve-mib M` (default 600): leave M MiB free on the peer card; the
|
||||
cache takes what remains. The prompt-path buffers need this headroom.
|
||||
Keep a few hundred MiB free on both cards beyond this: the prompt path loads its kernels and cuBLAS on first use
|
||||
(a start with 5 MiB left on the peer and 485 MiB on the primary stopped at the first prompt with "cuBLAS status 14").
|
||||
- Whether the two cards can really exchange data is tested once at start (#2666): two RTX A4000 behind a host
|
||||
bridge reported peer access, enabled it, and `cudaMemcpyPeerAsync` then copied nothing, so the peer computed on
|
||||
zeros and the first prompt came back NaN. A test copy each way now decides; if it arrives wrong the log says so
|
||||
and the prompt path uses the host route (the one without P2P: activations and rows through mapped host memory).
|
||||
`STRATA_PEER_NO_P2P=1` asks for that route by hand.
|
||||
- `--peer-slots N` (default 0): cap the tier at N experts; 0 = as many as fit.
|
||||
- `--peer-adapt-swaps N` (default -1): swaps per adaptive round on the peer;
|
||||
-1 uses the primary's `--adapt-swaps`.
|
||||
- `--peer-prefill-rows N` (default -1): the share of each prompt chunk's rows
|
||||
the peer computes; -1 is half of chunk x top-k, 0 keeps prompt rows on the
|
||||
primary.
|
||||
| Flag | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `--peer-device N` | off | Enable the tier on CUDA device N (N >= 1). |
|
||||
| `--peer-reserve-mib M` | 600 | VRAM left free on the peer card. Keep a few hundred MiB free on both cards beyond this, because the prompt path loads its kernels and cuBLAS on first use. |
|
||||
| `--peer-slots N` | 0 | Cap the tier at N experts; 0 means as many as fit. |
|
||||
| `--peer-adapt-swaps N` | -1 | Swaps per adaptive round on the peer; -1 uses the primary's `--adapt-swaps`. |
|
||||
| `--peer-prefill-rows N` | -1 | Share of each prompt chunk's rows the peer computes; -1 is half of chunk x top-k, 0 keeps prompt rows on the primary. |
|
||||
|
||||
Through the server, list both cards in the config's `"gpu"` (e.g. `[0, 1]`, numbered as nvidia-smi numbers them)
|
||||
and add `--peer-device 1` to its `"args"`; the number is the card's position in that list (with `"gpu": [2, 0]`,
|
||||
`--peer-device 1` is nvidia-smi's card 0). Several GPUs in `"gpu"` are otherwise a layer split: the server adds
|
||||
`--layer-split` only when `--peer-device` is not among the args.
|
||||
Whether the two cards can really exchange data is tested once at start with a copy each way. If the copy arrives wrong, the log
|
||||
says so and the prompt path uses the host route (activations and rows through mapped host memory). `STRATA_PEER_NO_P2P=1` asks
|
||||
for that route by hand.
|
||||
|
||||
`--peer-device` requires `--expert-profile` and an enabled expert cache, and
|
||||
the device must be visible; it refuses otherwise. It also refuses
|
||||
`--layer-split` (a different second-GPU mode: use one or the other) and
|
||||
`--expert-cache-device1..3` (the peer tier already caches experts on that card):
|
||||
Through the server, list both cards in the config's `"gpu"` (for example `[0, 1]`, numbered as `nvidia-smi` numbers them) and add
|
||||
`--peer-device 1` to `"args"`. The number is the card's position in that list (with `"gpu": [2, 0]`, `--peer-device 1` is
|
||||
`nvidia-smi`'s card 0). Several GPUs in `"gpu"` are otherwise a layer split; the server adds `--layer-split` only when
|
||||
`--peer-device` is not among the args.
|
||||
|
||||
strata generate: --peer-device cannot be combined with --layer-split (use one or the other)
|
||||
strata generate: --peer-device cannot be combined with --expert-cache-device1..3 (the peer tier already caches experts there)
|
||||
`--peer-device` requires `--expert-profile` and an enabled expert cache, and the device must be visible. It refuses
|
||||
`--layer-split` and `--expert-cache-device1..3`:
|
||||
|
||||
Without `--peer-device` the binary is unchanged; its output is byte-identical
|
||||
to the release. With `--peer-device` and the same expert set split across the
|
||||
two cards, the generated tokens are byte-identical to the single-GPU run under
|
||||
the exactness gate.
|
||||
```text
|
||||
strata generate: --peer-device cannot be combined with --layer-split (use one or the other)
|
||||
strata generate: --peer-device cannot be combined with --expert-cache-device1..3 (the peer tier already caches experts there)
|
||||
```
|
||||
|
||||
With a peer the prompt path keeps the MMQ path; the fused int8 prompt path is
|
||||
not yet combined with the peer's rows. Mapped host buffers gain
|
||||
`cudaHostAllocPortable` only with a peer, since only then does a second
|
||||
context write them. The tier size is manual for now (`--peer-reserve-mib`,
|
||||
`--peer-slots`); automatic sizing on small cards wants the buffer lending of
|
||||
#216 and is a follow-up.
|
||||
With a peer, the prompt path keeps the MMQ path (the fused int8 prompt path is not combined with the peer's rows). The tier size
|
||||
is manual (`--peer-reserve-mib`, `--peer-slots`). With the same expert set split across the two cards, generated tokens are
|
||||
byte-identical to the single-GPU run under the exactness gate.
|
||||
|
||||
## The resident RAM mode with the helper caches
|
||||
|
||||
`--resident-experts` (the low-RAM mode) works with the helper caches: the
|
||||
experts a helper holds are left out of the RAM copy as a split stage's are, and
|
||||
the helpers' refills read through the source - the RAM copy when it holds the
|
||||
bytes, the files otherwise. Measured here on an RX 6800 (gfx1030, 16 GB) + RX
|
||||
6600 (gfx1032, 8 GB), Windows 11, an i5-11400F and 32 GB of RAM (the 6800 alone:
|
||||
~37.5 decode tok/s, the same worker configuration): the 6600's 3.4-3.8k-expert
|
||||
cache lifted decode to ~40.6 / 44.2 tok/s on 8K-token requests at 3800 slots,
|
||||
with the CPU pool's share of the experts down from ~2.9 to ~0.4 entries per
|
||||
layer-window. The helper's rows cost about 0.3-0.6 ms per layer on the pool's
|
||||
critical path - usually less than the CPU rows and page faults they replace,
|
||||
which is why the 6600 (about 2-3x the 6800's cost per row) still pays off as a
|
||||
cache tier.
|
||||
|
||||
**Size a helper against the card's *real* free VRAM.** `--expert-cache-device1
|
||||
auto` reads `cudaMemGetInfo`, which on Windows reports the *process's* WDDM
|
||||
budget: it cannot see the desktop, the vision encoder (a separate process), a
|
||||
streaming host or other apps on that card. A helper auto-sized on a card that
|
||||
also drives the display or the encoder over-commits, WDDM backs the excess with
|
||||
system RAM ("shared GPU memory"), and that memory leaves the machine's RAM - on
|
||||
a 32 GB PC it pushed the resident complement into the pagefile and made the
|
||||
helper read its own cache over PCIe. An explicit count
|
||||
(`--expert-cache-device1 3800` here, of ~4100 that fit an empty card) kept the
|
||||
cache in real VRAM; freeing another app's sticky allocation on the same card
|
||||
was worth about the same again. Prefer explicit counts on a card that shares
|
||||
its VRAM with anything else.
|
||||
`--resident-experts` (the low-RAM mode) works with the helper caches. The experts a helper holds are left out of the RAM copy,
|
||||
and the helpers' refills read through the source: the RAM copy when it holds the bytes, the model files otherwise.
|
||||
|
||||
## Choosing the primary device on Windows (`STRATA_PRIMARY_DEVICE`)
|
||||
|
||||
The engine runs the model on the device the runtime numbers 0. Windows' HIP
|
||||
runtime enumerates the cards in a fixed order that `HIP_VISIBLE_DEVICES` only
|
||||
filters - it cannot reorder - so on a PC whose faster card comes second (an RX
|
||||
6800 after an RX 6600, say) a run with both visible could only put the model on
|
||||
the slow one. `STRATA_PRIMARY_DEVICE=N` picks another visible ordinal as the
|
||||
primary; unset or 0 is upstream's behavior.
|
||||
The engine runs the model on the device the runtime numbers 0. Windows' HIP runtime enumerates cards in a fixed order that
|
||||
`HIP_VISIBLE_DEVICES` only filters, so when the faster card comes second, the model can only land on the slower one.
|
||||
`STRATA_PRIMARY_DEVICE=N` picks another visible ordinal as primary. Unset or 0 keeps the default.
|
||||
|
||||
- the helper caches (`--expert-cache-device1..3`) take the visible devices
|
||||
other than the primary, in order;
|
||||
- the peer tier's cross-device checks use the same ordinal;
|
||||
- a layer split is refused with this set (its later stages must be > 0, and
|
||||
stage 0 is the primary);
|
||||
- NVIDIA cards do not need it: `CUDA_VISIBLE_DEVICES` with
|
||||
`CUDA_DEVICE_ORDER=PCI_BUS_ID` already reorders.
|
||||
- The helper caches (`--expert-cache-device1..3`) take the visible devices other than the primary, in order.
|
||||
- The peer tier uses the same ordinal.
|
||||
- A layer split is refused when this is set.
|
||||
- NVIDIA cards do not need it: `CUDA_VISIBLE_DEVICES` with `CUDA_DEVICE_ORDER=PCI_BUS_ID` already reorders.
|
||||
|
||||
Measured on RX 6800 + RX 6600, Windows 11; the report accompanies this change.
|
||||
## History / older measurements
|
||||
|
||||
Each row is one machine, reported by the author of the change or a community member.
|
||||
|
||||
| Setting | Measured on | Result |
|
||||
| --- | --- | --- |
|
||||
| `--remote-expert-opt` | dual RTX 4090 (PR author) | +63% mixed, +132% code decode over the plain helper path |
|
||||
| `--remote-expert-opt` | RTX 5090 + RTX 4090 (PR author) | +28% mixed, +63% code |
|
||||
| Helper PCIe share on HIP | 2x RX 6900 XT, PCIe 4.0 x8 each, IQ3_S (community) | before the fix, decode 40 tok/s with the primary waiting 43 ms per verify window; after, 66 tok/s and 19 ms. `--pcie-frac 0` avoided the cost ([report](../bench/results/2026-10-04-rdna2-helper-pcie-share/README.md)) |
|
||||
| Resident RAM mode with a helper | RX 6800 (16 GB) + RX 6600 (8 GB), Windows 11, i5-11400F, 32 GB RAM | RX 6800 alone ~37.5 tok/s; with a 3,800-slot helper ~40.6 / 44.2 tok/s on 8K-token requests; CPU pool share of experts down from ~2.9 to ~0.4 entries per layer-window |
|
||||
@@ -0,0 +1,179 @@
|
||||
# Install and build from a source checkout
|
||||
|
||||
Use this guide to work on Strata's code or maintain a source-based installation. It is for developers.
|
||||
To just use Strata (version 0.2.5), install the [desktop app or the Terminal](INSTALL.md). Installed apps do not use `START-HERE.bat`.
|
||||
|
||||
The repository includes `START-HERE.bat`, `SETUP.bat`, `UPDATE.bat`, `setup.sh` and `update.sh`.
|
||||
Run them from a cloned or extracted **source folder**. They may install dependencies, prepare model files and
|
||||
acquire or build an engine. Check [model requirements](MODELS.md) before downloading.
|
||||
|
||||
## Quick start
|
||||
|
||||
| I use | Run in the source folder |
|
||||
| --- | --- |
|
||||
| Windows | `START-HERE.bat` ([steps](#from-source-windows)) |
|
||||
| Linux | `./setup.sh` ([steps](#from-source-linux)) |
|
||||
| Mac (Apple Silicon) | `./setup.sh` ([steps](#from-source-mac)) |
|
||||
| A source build to test a change | `./setup.sh --build --no-start` ([steps](#build-from-git-on-linux-to-try-a-pull-request-or-change-the-code)) |
|
||||
| An update | `UPDATE.bat` or `./update.sh` ([steps](#updating)) |
|
||||
|
||||
## From source: Windows
|
||||
|
||||
1. Clone the [repository](https://github.com/Niko1221/Strata) or download and extract its source archive.
|
||||
2. Open the source folder and run `START-HERE.bat`.
|
||||
3. Choose a supported model, storage folder and settings. Keep the window open while setup works.
|
||||
|
||||
For an explicit original ISTA-DASLab Flash-Next pack, from Command Prompt in that folder:
|
||||
|
||||
```bat
|
||||
START-HERE.bat --family qwen --model Q2_0
|
||||
```
|
||||
|
||||
Choose a pack that fits your machine; Q2_0 is an example, not a universal minimum-memory option.
|
||||
In PowerShell, prefix the wrapper with `.\`, for example `.\START-HERE.bat`.
|
||||
`SETUP.bat` opens setup again to add a model or change settings. The wrapper can reuse an existing install;
|
||||
first-start and later load times depend on the pack, disk, RAM and backend. A model can take several minutes to load.
|
||||
|
||||
## From source: Linux
|
||||
|
||||
Clone or extract the source, enter its folder, then run:
|
||||
|
||||
```bash
|
||||
./setup.sh
|
||||
```
|
||||
|
||||
On Debian and Ubuntu, the wrapper can offer to install missing system packages through `sudo apt`.
|
||||
Other distributions need the equivalent dependencies installed first. Python dependencies use the checkout's
|
||||
`.venv`. NVIDIA source builds need the CUDA toolkit; AMD builds need HIP/ROCm. See [Linux details](DETAILS.md#linux).
|
||||
|
||||
Nix/NixOS users can enter the supplied development environment:
|
||||
|
||||
```bash
|
||||
nix develop -c ./setup.sh
|
||||
```
|
||||
|
||||
### Build from git on Linux (to try a pull request or change the code)
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Niko1221/Strata.git
|
||||
cd Strata
|
||||
./setup.sh --build --no-start
|
||||
```
|
||||
|
||||
`--build` requests a source engine rather than a ready-made one. `--no-start` prevents serving;
|
||||
this is still setup and can download and prepare a model. Inspect the printed paths and generated config before
|
||||
claiming to have tested a code change. A released engine cannot validate changes to your local C++ files.
|
||||
|
||||
For a direct NVIDIA build, with the required compiler and CUDA toolkit installed:
|
||||
|
||||
```bash
|
||||
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native
|
||||
cmake --build build --target strata -j8
|
||||
```
|
||||
|
||||
Run against the resulting binary and the matching model config. Use the build instructions for
|
||||
[HIP](AMD_HIP.md), [SYCL](INTEL_ARC.md) or [Metal](MAC.md) when changing those backends.
|
||||
Follow [contribution validation requirements](../CONTRIBUTING.md) before requesting review.
|
||||
|
||||
For the Terminal monitor from a checkout:
|
||||
|
||||
```bash
|
||||
.venv/bin/pip install -r requirements-monitor.txt
|
||||
STRATA_PYTHON="$PWD/.venv/bin/python3" bin/strata monitor --url http://127.0.0.1:8080
|
||||
```
|
||||
|
||||
## From source: Mac
|
||||
|
||||
Use an Apple Silicon Mac with macOS 15 or newer. Install Apple's Command Line Tools once:
|
||||
|
||||
```bash
|
||||
xcode-select --install
|
||||
```
|
||||
|
||||
Then run `./setup.sh` in the source folder. macOS uses its own setup implementation and builds the Metal engine;
|
||||
its options and model preparation differ from Windows/Linux. Read [MAC.md](MAC.md#install) and
|
||||
`./setup.sh --help`. The released app and Terminal do not require a compiler.
|
||||
|
||||
## AMD cards
|
||||
|
||||
Use `START-HERE.bat` on Windows or `./setup.sh` on Linux. Source setup detects supported Radeon cards;
|
||||
`--backend hip` selects HIP explicitly on a mixed-vendor PC. Build dependencies, architecture support and
|
||||
measurements are in [AMD_HIP.md](AMD_HIP.md). Ryzen AI Max uses the [Strix Halo guide](STRIX_HALO.md).
|
||||
|
||||
Do not assume every AMD card, quantization or feature has been validated. Follow the exact backend guide.
|
||||
Intel Arc is a separate Linux SYCL path with its own [model preparation instructions](INTEL_ARC.md).
|
||||
|
||||
## Two or three cards
|
||||
|
||||
Source setup uses `--gpus` to select multiple cards of the same supported backend:
|
||||
|
||||
```bash
|
||||
./setup.sh --gpus 0,1
|
||||
```
|
||||
|
||||
Windows equivalent: `START-HERE.bat --gpus 0,1`. The installed Terminal instead uses
|
||||
`strata launch --gpu 0,1`. Main/Helper roles, memory limits and measurements are in [MULTI_GPU.md](MULTI_GPU.md).
|
||||
Several cards are not always faster than one; measure the workload you care about.
|
||||
|
||||
## Docker (Linux)
|
||||
|
||||
Use [DOCKER.md](DOCKER.md) for released images, GPU access and the container build instructions.
|
||||
A container is a separate installation route. Intel model preparation and host driver requirements still apply.
|
||||
|
||||
## Older CPUs (experimental)
|
||||
|
||||
Ready-made x86 engines require AVX2. Source setup can build experimental AVX or SSE4.2 paths on older CPUs;
|
||||
this does not make every model pack compatible. Q2_0's CPU requirements differ from the i-quants.
|
||||
See the [CPU and build notes](DETAILS.md) and [older GPU guide](OLDER_GPUS.md) for the relevant hardware.
|
||||
Record the actual CPU, compiler, model and validation results when reporting an experimental build.
|
||||
|
||||
## Updating
|
||||
|
||||
Stop the running source engine before replacing it. Run `UPDATE.bat` on Windows or `./update.sh` on Linux.
|
||||
In a Git checkout the updater can pull source changes, update dependencies and refresh installed configs without
|
||||
starting the model. A source ZIP cannot pull Git history: extract a newer source archive instead.
|
||||
Preserve your configs and data location when moving between checkouts.
|
||||
|
||||
If you changed engine code, rebuild it and verify the executable path again. The presence of updated source files
|
||||
is not evidence that a running engine contains those changes. Desktop and Terminal component updates are covered
|
||||
separately in [INSTALL.md](INSTALL.md#updating).
|
||||
|
||||
## Where things are stored
|
||||
|
||||
Source setup writes model run configs (`strata-<model>.json`), launch scripts (`run-<model>.bat` / `.sh`) and
|
||||
logs in the checkout. Models, packs and draft data normally go in `Strata-data` beside it, or the location selected
|
||||
with `--data-dir`. Dependencies and build products stay in the source installation.
|
||||
|
||||
These are source paths. The desktop app and installed Terminal use the [release data locations](INSTALL.md#where-things-are-stored).
|
||||
Back up a config before editing it: setup can rewrite the fields it owns. Do not publish API keys or private paths in logs.
|
||||
|
||||
## Setup's questions
|
||||
|
||||
Setup detects the machine and offers compatible choices. The list varies by platform, available memory and model
|
||||
catalog; it is not a fixed two-model menu. Read the proposed download, memory use and settings before accepting.
|
||||
For documentation recommendations, prefer ISTA-DASLab when a compatible pack fits, then Unsloth, Gyro and others.
|
||||
Executable defaults may differ, so select the family explicitly when that choice matters.
|
||||
|
||||
## Tuning for your PC
|
||||
|
||||
Use source setup's `--calibrate` only where that backend supports it. Terminal tuning instead uses
|
||||
`strata launch --tune quick` or `--tune thorough`; those are separate interfaces. Tuning can take a substantial
|
||||
amount of time and should use the same hardware and model configuration as the intended workload.
|
||||
|
||||
## Options without questions
|
||||
|
||||
Run the relevant wrapper with `--help` for the complete options on that operating system. Common source examples:
|
||||
|
||||
```bat
|
||||
START-HERE.bat --family qwen --model IQ3_XXS --context 32768 --vision no
|
||||
START-HERE.bat --setup
|
||||
```
|
||||
|
||||
```bash
|
||||
./setup.sh --family qwen --model IQ3_XXS --context 32768 --vision no
|
||||
./setup.sh --no-start
|
||||
```
|
||||
|
||||
`--yes` accepts setup's recommended answers and can start large downloads. Use it only after choosing the pack
|
||||
and storage location. Keep source flags (`--gpus`, `--vision yes|no`, numeric `--context`) separate from
|
||||
[Terminal flags](TERMINAL_LAUNCH.md) and raw engine options.
|
||||
+138
-112
@@ -1,26 +1,47 @@
|
||||
# Strix Halo (Ryzen AI Max, gfx1151)
|
||||
|
||||
Strata runs on the Ryzen AI Max "Strix Halo" APUs (Radeon 8060S / 8050S, RDNA3.5, `gfx1151`, one pool of LPDDR5X shared by the
|
||||
CPU and the GPU) through the same HIP backend as the Radeon cards in [AMD_HIP.md](AMD_HIP.md). **Status: experimental.**
|
||||
Everything here was built and measured on one machine, the maintainers' Strix Halo box (Ryzen AI Max+ 395, 128 GB,
|
||||
Ubuntu 26.04, kernel 7.0, ROCm 7.14.1). Setup (`setup.sh`, `START-HERE.bat`) recognizes it from 0.1.40, by its PCI id (`1002:1586`, one id for the 8060S, 8050S and 8040S) and by
|
||||
`gfx1151` exactly: a Strix Point (890M / 880M, gfx1150), Krackan (860M / 840M, gfx1152) or Phoenix / Hawk Point (780M / 760M, gfx1103) Radeon is
|
||||
named as what it is and not supported. On Linux setup compiles the engine for gfx1151 itself (the steps below are the same, by hand). The 0.2.0-beta release also has ready-made packages: `Strata-0.2.0-beta-linux-amd-strixhalo.AppImage` (engine `halo`, gfx1151, ROCm 7.14.1) and, on Windows, `Strata-0.2.0-beta-windows-hip-rdna3.exe` (engine `rdna3`: gfx1100, gfx1101, gfx1102 and gfx1151, ROCm 10.2.0a20260930); any app installer plus **Update engine** (or `strata update`) fetches the same package. What was run: both packages were built, linked and scanned, and the Linux package refuses to start on a card of another family (tried on an RX 9000 card); the maintainers have no Strix Halo and ran neither on one. The measurements on this page come from source builds on Linux (the maintainers' earlier box and community machines). Nobody has run a Strix Halo on Windows. The packaged engines do not carry the hipBLASLt table of section 3 yet (planned for 0.2.1), so their prompt reading is slower than the numbers below until then. A Windows zip of the 0.1.x START-HERE route without gfx1151 code is refused with a message, never used. Setup counts the GPU's memory as the BIOS carve-out plus the shared memory
|
||||
(the GTT pool; on Windows half the RAM) and recommends UD-IQ4_XS from about 80 GB of memory; it changes no BIOS or kernel setting.
|
||||
This page is for people with a Ryzen AI Max "Strix Halo" PC (Radeon 8060S / 8050S, RDNA 3.5, `gfx1151`, one pool of LPDDR5X
|
||||
shared by the CPU and the GPU). Strata runs there through the same HIP backend as the Radeon cards in [AMD_HIP.md](AMD_HIP.md).
|
||||
**Status: experimental.**
|
||||
|
||||
**Windows start-up (0.2.1.2).** On a Strix Halo under Windows the engine no longer registers the expert arena with the GPU driver: registering it took all free VRAM (91 GiB before, 0 after, on a 64 GB carve-out + 32 GB shared PC), the expert cache got no slots and the engine stopped with "the embedding buffer failed" (0.2.1 and earlier). The arena stays in RAM, held by the working-set lock, and the cache gets its normal slots. `STRATA_ARENA_PIN_GIB=N` (or `0`) registers it as before, `STRATA_UMA_REGISTER=1` too. Linux is unchanged.
|
||||
DeepSeek V4.1-Flash does not run on Strix Halo yet. It runs in the app and the Terminal on NVIDIA RTX 30, 40 and 50 cards and on the Radeon AI
|
||||
PRO R9700 ([MODELS.md](MODELS.md#deepseek-v41-flash)).
|
||||
|
||||
What the engine does differently on this chip:
|
||||
> **On this page:** [Quick start](#quick-start) · [What the engine does differently](#what-the-engine-does-differently) ·
|
||||
> [1. Toolchain](#1-the-toolchain-no-root-needed) · [2. Engine](#2-the-engine) · [3. Tuning table](#3-the-prompt-table-and-running) ·
|
||||
> [4. Defaults](#4-what-is-on-by-default-on-gfx1151) · [5. Opt-in](#5-what-is-not-on-by-default-it-changes-bits) ·
|
||||
> [5b. Desktop and RAM](#5b-the-desktop-the-ram-and-a-discrete-card-beside-the-strix-halo-linux) · [6. Measured](#6-measured) ·
|
||||
> [History](#history--older-measurements)
|
||||
|
||||
- **Unified memory.** The GPU's "VRAM" is the GTT pool of the same RAM. `device_free_bytes()` (the figure `--expert-cache auto` and
|
||||
the memory plan use) counts what Linux can give back (`MemAvailable` less 6 GiB for the system, `STRATA_UMA_HEADROOM_GIB`
|
||||
changes the 6) instead of only `MemFree`, per device, for any integrated GPU on Linux (this applies to a DGX Spark too).
|
||||
## Quick start
|
||||
|
||||
Use the [current release](https://github.com/Niko1221/Strata/releases/latest). You do not need to build ROCm for a packaged install.
|
||||
|
||||
| System | What to install | Engine variant |
|
||||
| --- | --- | --- |
|
||||
| Linux | `Strata-0.2.6-linux-x86_64.AppImage` (it downloads the engine on first start), or `strata launch` | `linux-x86_64-hip-halo` |
|
||||
| Windows | `Strata-0.2.6-windows-x64-setup.exe` (it downloads the engine on first start), or `strata launch` | `windows-x86_64-hip-rdna3` |
|
||||
|
||||
Packaged engines carry the matching hipBLASLt tuning table for gfx1151 and select it on their own. Setup recognizes `gfx1151` and
|
||||
PCI ID `1002:1586`; other integrated Radeon chips have separate support limits. Unified memory is counted once. Setup does not
|
||||
change BIOS or kernel settings. The manual Linux toolchain below is for reproduction and builds, not a prerequisite for the app.
|
||||
|
||||
## What the engine does differently
|
||||
|
||||
- **Unified memory.** The GPU's "VRAM" is the GTT pool of the same RAM. `device_free_bytes()` (the figure `--expert-cache auto`
|
||||
and the memory plan use) counts what Linux can give back (`MemAvailable` less 6 GiB for the system; `STRATA_UMA_HEADROOM_GIB`
|
||||
changes the 6) for any integrated GPU on Linux, a DGX Spark included.
|
||||
- **gfx11 matrix cores.** The prompt attention, the block scorer, the prompt GEMMs and the prompt experts have WMMA kernels for
|
||||
gfx1100 / 1101 / 1102 / 1103 / 1150 / 1151. A gfx11 part outside that list (gfx1152) is not given them: it takes the portable path.
|
||||
- **The hipBLASLt table** `tools/hip/gfx1151-hipblaslt-100401.txt` (ROCm 7.14.1's hipBLASLt 1.x `100401`), including the prompt shapes
|
||||
at `--prefill 16384` (T 16383 / 16384). Setup's own ROCm for gfx1151 (the 7.14.0a20260608 wheels, hipBLASLt 1.4.0) takes
|
||||
`tools/hip/gfx1151-hipblaslt-100400.txt`: the same shapes, calibrated against that library (docs/AMD_HIP.md, Tuning table).
|
||||
- **The expert cache and the RAM budget share the RAM.** For a model with a RAM budget (UD-IQ4_XS, UD-Q4_K_XL) setup writes a number for `--expert-cache` (the RAM less 24 GB less the budget), not `auto`: `auto` counts all the memory the OS can give back at start, and with a 55 GiB budget and a desktop open it took the room the OS needed (#1715, a global OOM on a 128 GB box). The number is in the config's `args`; `--expert-cache N` there sets it. An engine run by hand should pass one too.
|
||||
gfx1100 / 1101 / 1102 / 1103 / 1150 / 1151. A gfx11 part outside that list (gfx1152) takes the portable path.
|
||||
- **The hipBLASLt table.** `tools/hip/gfx1151-hipblaslt-100401.txt` is for ROCm 7.14.1's hipBLASLt (`100401`), including the prompt
|
||||
shapes at `--prefill 16384`. Setup's own ROCm for gfx1151 (the 7.14.0a20260608 wheels, hipBLASLt 1.4.0) takes
|
||||
`tools/hip/gfx1151-hipblaslt-100400.txt`, calibrated against that library.
|
||||
- **The expert cache and the RAM budget share the RAM.** For a model with a RAM budget (UD-IQ4_XS, UD-Q4_K_XL) setup writes a number
|
||||
for `--expert-cache` (the RAM less 24 GB less the budget), not `auto`, because `auto` counts all memory the OS can give back at
|
||||
start and can take the room the OS needs. The number is in the config's `args`. An engine run by hand should pass one too.
|
||||
- **Windows.** The engine does not register the expert arena with the GPU driver. The arena stays in RAM, held by the
|
||||
working-set lock, and the expert cache gets its normal slots. `STRATA_ARENA_PIN_GIB=N` (or `0`) or `STRATA_UMA_REGISTER=1`
|
||||
registers it. Linux registers the whole arena by default.
|
||||
|
||||
## 1. The toolchain (no root needed)
|
||||
|
||||
@@ -37,18 +58,17 @@ export PATH=$SDK/bin:$SDK/lib/llvm/bin:$PATH
|
||||
export LD_LIBRARY_PATH=$SDK/lib:$SDK/lib/rocm_sysdeps/lib:$SDK/lib/llvm/lib:${LD_LIBRARY_PATH:-}
|
||||
```
|
||||
|
||||
You also need CMake 3.24+, Ninja, git and a C++ host toolchain with libstdc++ headers (`sudo apt install build-essential cmake ninja-build git`).
|
||||
The maintainers' box has no system compiler: they point ROCm's clang at a user-space copy of gcc 15 with
|
||||
`--gcc-toolchain=<dir>/usr` (added to `CMAKE_HIP_FLAGS` below; leave it out when the system has one).
|
||||
You also need CMake 3.24+, Ninja, git and a C++ host toolchain with libstdc++ headers
|
||||
(`sudo apt install build-essential cmake ninja-build git`). If the system has no compiler, point ROCm's clang at a user-space gcc
|
||||
with `--gcc-toolchain=<dir>/usr` in `CMAKE_HIP_FLAGS`.
|
||||
|
||||
The kernel's amdgpu driver and `/dev/kfd` access (your user in the `render` and `video` groups) are all the GPU side needs.
|
||||
To let the GPU use most of a 128 GB machine the maintainers' box boots with `ttm.pages_limit=29360128 ttm.page_pool_size=29360128`
|
||||
(a 112 GiB GTT; the default is about half of the RAM) and `amd_iommu=off`; both are host settings the engine does not need to run,
|
||||
only to have the room.
|
||||
The GPU side needs the kernel's amdgpu driver and `/dev/kfd` access (your user in the `render` and `video` groups). To let the GPU
|
||||
use most of a 128 GB machine, the maintainers' box boots with `ttm.pages_limit=29360128 ttm.page_pool_size=29360128` (a 112 GiB GTT;
|
||||
the default is about half of the RAM) and `amd_iommu=off`. Both are host settings the engine does not need to run, only to have the room.
|
||||
|
||||
## 2. The engine
|
||||
|
||||
llama.cpp's `ggml` sources are pinned by this repository: CMake fetches the revision it needs. For an offline build, point
|
||||
llama.cpp's `ggml` sources are pinned by this repository, and CMake fetches the revision it needs. For an offline build, point
|
||||
`STRATA_GGML_DIR` at a checkout of exactly that revision.
|
||||
|
||||
```sh
|
||||
@@ -62,50 +82,52 @@ cmake -S . -B build-halo -G Ninja -DCMAKE_BUILD_TYPE=Release \
|
||||
cmake --build build-halo --target strata -j 8 # about 25 minutes the first time
|
||||
```
|
||||
|
||||
`gfx1151` is "unvalidated" in the build's own wording (it warns). `STRATA_PREFILL_MMQ=ON` builds ggml's MMQ kernels the prompt
|
||||
experts need. The tests: `cmake --build build-halo -j 8` builds them all, `ctest --test-dir build-halo` runs them;
|
||||
`hip_prefill_hcd_exact_parity` and `hip_prefill_hipblaslt_gemm` need the table (next section) and skip without it.
|
||||
The build warns that `gfx1151` is "unvalidated". `STRATA_PREFILL_MMQ=ON` builds ggml's MMQ kernels the prompt experts need. To test:
|
||||
`cmake --build build-halo -j 8` builds the tests and `ctest --test-dir build-halo` runs them. `hip_prefill_hcd_exact_parity` and
|
||||
`hip_prefill_hipblaslt_gemm` need the table (next section) and skip without it.
|
||||
|
||||
## 3. The prompt table, and running
|
||||
|
||||
Hand the engine the table (`setup` does it for the supported cards; for gfx1151 you do it): without it the prompt's dense matrix
|
||||
products use plain hipBLAS and are slower.
|
||||
A packaged engine finds its table itself. For a hand-built engine, hand it the table, because without it the prompt's dense matrix
|
||||
products use plain hipBLAS and are slower:
|
||||
|
||||
```sh
|
||||
export STRATA_HIPBLASLT_TUNING=$PWD/tools/hip/gfx1151-hipblaslt-100401.txt
|
||||
```
|
||||
|
||||
The engine accepts it only for the architecture and the hipBLASLt version named on the table's second line
|
||||
The engine accepts a table only for the architecture and hipBLASLt version named on its second line
|
||||
(`STRATA_HIPBLASLT_TUNING_V1 gfx1151 100401`: ROCm 7.14.1) and says so at start-up (`prefill gemm: hipBLASLt tuning enabled (90 rows,
|
||||
gfx1151, version 100401)`). Then run `strata` with the same arguments as on any other card (the model packs, `--serve`, the
|
||||
settings of [DETAILS.md](DETAILS.md)); a pack of the Unsloth UD-IQ4_XS or UD-Q4_K_XL GGUFs is what the numbers below used.
|
||||
gfx1151, version 100401)`). Then run `strata` with the same arguments as on any other card (the model packs, `--serve`, the settings
|
||||
of [DETAILS.md](DETAILS.md)). The numbers below used packs of the Unsloth UD-IQ4_XS or UD-Q4_K_XL GGUFs.
|
||||
|
||||
## 4. What is on by default on gfx1151
|
||||
|
||||
At start the engine looks at GPU 0; on a gfx1151 it sets the switches below unless you set them yourself
|
||||
(`STRATA_GFX1151_DEFAULTS=0` turns the whole table off, and the start-up line names what was set). **Every one of them gave
|
||||
byte-identical output on the maintainers' box** (the same token ids, 4K / 32K / 64K and four models, and a bitwise harness per kernel):
|
||||
they change speed, not answers. Nothing here applies to CUDA or to another AMD architecture.
|
||||
At start the engine looks at GPU 0. On a gfx1151 it sets the switches below unless you set them yourself.
|
||||
`STRATA_GFX1151_DEFAULTS=0` turns the whole table off, and the start-up line names what was set. Every one of them gave
|
||||
byte-identical output on the maintainers' box (same token ids at 4K / 32K / 64K and on four models, plus a bitwise harness per
|
||||
kernel): they change speed, not answers. Nothing here applies to CUDA or to another AMD architecture.
|
||||
|
||||
| Switch | What it does |
|
||||
|---|---|
|
||||
| `STRATA_GDN_HEAD`, `STRATA_GDN_PP=2`, `STRATA_GDN_CONVL2`, `STRATA_GDN_NOY` | the prompt's gated-delta-net recurrence: four lanes per column, two columns per lane, the convolution and the norms fused |
|
||||
| `STRATA_HCD_EXACT` | the hyper-connection down projection (N 320, K 10240) in a WMMA kernel that reproduces hipBLASLt solution 1176 / 1177's summation order (`hip_prefill_hcd_exact_parity` checks it bit for bit); it runs only where the table makes hipBLASLt pick one of those two, and says once if it cannot |
|
||||
| `STRATA_CVEC_FUSE`, `STRATA_PF_PAD` | a steered layer's write, control vector and next norm in one pass; padded GEMM row strides |
|
||||
| `STRATA_TOPK_REG_MIN_BLOCKS=4096` | the prompt's QSA top-k uses the register kernel from 16K cells of context instead of 28K (the original kernel is slower there; the selected cells are identical, `qsa_topk_parity` checks it) |
|
||||
| `STRATA_Q8_PACKED`, `STRATA_Q6_PACKED`, `STRATA_MMVF_ROWS`, `STRATA_ATTN_LANECELL` | decode layouts and kernels (packed Q8_0 / Q6_K weights, 4-row BF16 GEMV, one-cell-per-thread attention scores) |
|
||||
| `STRATA_EXPERT_V2`, `STRATA_TSUM`, `STRATA_LFUSE` | the grouped decode experts (IQ3_S / IQ4_NL), single-butterfly warp sums, fewer launches around the shared expert and the KV append |
|
||||
| `STRATA_GDN_SPLIT`, `STRATA_QFUSE`, `STRATA_PLE_BATCH` | the GDN step over four blocks per head, activation images written by their producers, the PLE key / value projections of a verify window at once |
|
||||
| `STRATA_SH_STREAM=1` | the shared expert on a second stream beside the routed experts. It is off by default on other AMD cards (#816), but it pays on gfx1151: decode +1.8% (UD-IQ4_XS) and +6.7% (UD-Q4_K_XL) at 8K |
|
||||
| --- | --- |
|
||||
| `STRATA_GDN_HEAD`, `STRATA_GDN_PP=2`, `STRATA_GDN_CONVL2`, `STRATA_GDN_NOY` | The prompt's gated-delta-net recurrence: four lanes per column, two columns per lane, convolution and norms fused. |
|
||||
| `STRATA_HCD_EXACT` | The hyper-connection down projection (N 320, K 10240) in a WMMA kernel that reproduces hipBLASLt solution 1176 / 1177's summation order (`hip_prefill_hcd_exact_parity` checks it bit for bit). It runs only where the table makes hipBLASLt pick one of those two, and says once if it cannot. |
|
||||
| `STRATA_CVEC_FUSE`, `STRATA_PF_PAD` | A steered layer's write, control vector and next norm in one pass; padded GEMM row strides. |
|
||||
| `STRATA_TOPK_REG_MIN_BLOCKS=4096` | The prompt's QSA top-k uses the register kernel from 16K cells of context instead of 28K. The selected cells are identical (`qsa_topk_parity`). |
|
||||
| `STRATA_Q8_PACKED`, `STRATA_Q6_PACKED`, `STRATA_MMVF_ROWS`, `STRATA_ATTN_LANECELL` | Decode layouts and kernels: packed Q8_0 / Q6_K weights, 4-row BF16 GEMV, one-cell-per-thread attention scores. |
|
||||
| `STRATA_EXPERT_V2`, `STRATA_TSUM`, `STRATA_LFUSE` | The grouped decode experts (IQ3_S / IQ4_NL), single-butterfly warp sums, fewer launches around the shared expert and the KV append. |
|
||||
| `STRATA_GDN_SPLIT`, `STRATA_QFUSE`, `STRATA_PLE_BATCH` | The GDN step over four blocks per head, activation images written by their producers, the PLE key / value projections of a verify window at once. |
|
||||
| `STRATA_SH_STREAM=1` | The shared expert on a second stream beside the routed experts. Off by default on other AMD cards, but it pays on gfx1151 (decode +1.8% UD-IQ4_XS, +6.7% UD-Q4_K_XL at 8K). |
|
||||
|
||||
Two further defaults change bits. `STRATA_HC_Q8` reads the hyper-connection weights as 8-bit (`STRATA_HC_Q8=0` restores the BF16
|
||||
read), and on Windows the prompt is read in chunks of 8192 tokens (`STRATA_GFX1151_CHUNK=0` keeps 6144). About `STRATA_HC_Q8`: it
|
||||
halves the hyper-connection weight bytes (13.29 -> 7.15 MB per read) and gave +8.4% decode on `json_extract`, at the cost of answers
|
||||
that are not bit-identical to the BF16 read. How far they move: [AMD_HIP.md](AMD_HIP.md) ("Strix Halo: 8-bit hyper-connection weights").
|
||||
|
||||
## 5. What is not on by default (it changes bits)
|
||||
|
||||
**Five of the switches below (`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA`, `STRATA_SELECT_WMMA`) are one opt-in on an integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"). It was measured on a Radeon 780M only, where it is the default.** `STRATA_PF_FUSED`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well. `STRATA_HC_Q8` is on by default on gfx1151 (below); `STRATA_HC_Q8=0` turns it off.
|
||||
|
||||
These are faster on gfx1151 and **round differently** from the default path (a different summation order or a half-precision
|
||||
intermediate), so they are opt-in and the maintainers gate them with KL against the default (`STRATA_PF_SWITCH_MIN_T=4096` keeps
|
||||
the short prompts on the default numerics, since these pay on long ones). The maintainers' fast configuration sets all of
|
||||
them:
|
||||
These are faster on gfx1151 and round differently from the default path (a different summation order or a half-precision
|
||||
intermediate), so they are opt-in. The maintainers gate them with KL against the default. `STRATA_PF_SWITCH_MIN_T=4096` keeps
|
||||
short prompts on the default numerics, since these pay on long ones. The maintainers' fast configuration sets all of them:
|
||||
|
||||
```sh
|
||||
export STRATA_PF_FUSED=1 # the prompt experts on the matrix cores (IQ2 / IQ3 / IQ4_XS gate/up, Q2_0 / IQ4_NL / Q8_0 down)
|
||||
@@ -115,64 +137,64 @@ export STRATA_PA_FAST=1 # the prompt attention with single FP16 q and p
|
||||
export STRATA_HIP_WMMA=1 # the prompt attention on matrix cores
|
||||
export STRATA_SELECT_WMMA=1 # the block scorer on matrix cores
|
||||
export STRATA_PF_SWITCH_MIN_T=4096
|
||||
export STRATA_PREFILL_STREAM_MIN=128 # prompt reads of 128-1,023 tokens on the fused experts too (below)
|
||||
export STRATA_PREFILL_STREAM_MIN=128 # prompt reads of 128-1,023 tokens on the fused experts too
|
||||
```
|
||||
|
||||
`STRATA_HC_Q8` (on by default on gfx1151 since 0.2.2; `STRATA_HC_Q8=0` restores the bf16 read) measured on the Strix Halo (IQ3_S, gfx1151, 2026-10-21, MTP 4 drafts / min-p 0.5): **+8.4%**
|
||||
on `json_extract` at fixed windows (53.55 -> 58.03 tok/s), at the cost of **94 of 250 output tokens differing** from
|
||||
the BF16 read (first at 60). It halves the hyper-connection weight bytes (13.29 -> 7.15 MB per read; cold read
|
||||
93.8 -> 20.3 us, 142 -> 352 GB/s). The answers are not bit-identical to the bf16 read; how far they move is in docs/AMD_HIP.md ("Strix Halo: 8-bit hyper-connection weights").
|
||||
`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA` and `STRATA_SELECT_WMMA` together are one opt-in on an
|
||||
integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"), measured on a Radeon 780M only, where
|
||||
it is the default. `STRATA_PF_FUSED`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well.
|
||||
|
||||
`STRATA_PF_FUSED=1` reaches only chunks of `STRATA_PREFILL_STREAM_MIN` tokens or more (1,024 by default, the floor measured on
|
||||
a discrete card, where a smaller chunk does not pay for streaming every expert); a smaller read runs its experts through MMQ.
|
||||
With every expert in the unified memory nothing streams, and MMQ is the slow part: an agent's turn - a tool result of ~1,000
|
||||
tokens on a cached conversation - read in 3.1-6.0 s at 1,024 and 0.9-2.5 s at 128 (C2T8, `--no-prefill-borrow`; 12-turn
|
||||
replay 3.4 -> 2.1 s per turn). Its output is the fused path's, which reads of 1,024 tokens or more already take; it is not
|
||||
under `STRATA_PF_SWITCH_MIN_T`, which gates only the hyper-connection and padding switches. A UD-Q4_K_XL or other K-quant
|
||||
pack takes the fused experts only with `STRATA_PF_FUSED_KQ=1` as well.
|
||||
|
||||
`STRATA_PF_F16=1` (gfx11 only; add it to the list above) runs the covered prompt experts on the F16 matrix cores instead of
|
||||
the int8 ones: K-quant gate/up and Q5_1 down (with `STRATA_PF_FUSED_KQ=1`), IQ2 / IQ3 / IQ4_XS gate/up, Q2_0 / IQ4_NL / Q8_0
|
||||
down. One layer, 16,384 tokens: Q4_K / Q5_1 85.3 -> 53.5 ms, IQ3_S / IQ4_NL gate/up 11.65 -> 9.87 and down 6.11 -> 4.71 ms.
|
||||
It rounds differently from the int8 kernels. First-token KL against a run with no switches, 12 natural-text prompts of
|
||||
1.5K-24K tokens (Strata's own docs and sources), top-1 the same in 12 of 12: UD-Q4_K_XL mean 0.0068 (median 0.0021) against
|
||||
0.0031 (0.0024) for `STRATA_PF_FUSED=1 STRATA_PF_FUSED_KQ=1` without it; UD-IQ4_XS 0.0085 (0.0036) against 0.0068 (0.0032). One
|
||||
1.5K prompt carries most of the Q4_K_XL mean (0.049).
|
||||
|
||||
`STRATA_REFILL_PARTS=1` (opt-in, exact; HIP and CUDA, a GGUF read in place) refills the expert-cache slots a prompt borrowed straight from the GGUF mapping (an expert's gate, up and down tensors) instead of through an assembled 4 MB copy per expert: the same bytes in the slot (checked against the assembled blob on 162 slots), 0.1 to 0.3 s less at the end of a prompt; gfx1151, UD-Q4_K_XL and UD-IQ4_XS: a 4K-8K prompt +2.3% to +3.5%, 32K +1.1%. It is not in the table above because only gfx1151 has measured it.
|
||||
|
||||
Two engine flags help on long contexts and are not Strix-specific: `--mtp-window 8192` (the draft layer attends to the last 8,192
|
||||
cells: 64K output +3.5%, 32K -0.4%) and `--spec`, `--lookup-chain`, `--mtp-q4` (see [DETAILS.md](DETAILS.md)).
|
||||
| Switch | What it does |
|
||||
| --- | --- |
|
||||
| `STRATA_PREFILL_STREAM_MIN=N` | `STRATA_PF_FUSED=1` reaches only chunks of N tokens or more (1,024 by default, the floor measured on a discrete card). With every expert in unified memory nothing streams and MMQ is the slow part, so a smaller N (128) speeds up an agent's turn (a tool result of about 1,000 tokens on a cached conversation: 3.1 to 6.0 s at 1,024 against 0.9 to 2.5 s at 128). |
|
||||
| `STRATA_PF_FUSED_KQ=1` | A UD-Q4_K_XL or other K-quant pack takes the fused experts only with this as well. |
|
||||
| `STRATA_PF_F16=1` (gfx11 only) | Runs the covered prompt experts on the F16 matrix cores instead of the int8 ones. It rounds differently from the int8 kernels. |
|
||||
| `STRATA_REFILL_PARTS=1` (exact; HIP and CUDA, a GGUF read in place) | Refills the expert-cache slots a prompt borrowed straight from the GGUF mapping instead of through an assembled 4 MB copy per expert. Only gfx1151 has measured it. |
|
||||
| `--mtp-window 8192` | Not Strix-specific: the draft layer attends to the last 8,192 cells. |
|
||||
| `--spec`, `--lookup-chain`, `--mtp-q4` | See [DETAILS.md](DETAILS.md). |
|
||||
|
||||
## 5b. The desktop, the RAM, and a discrete card beside the Strix Halo (Linux)
|
||||
|
||||
The Strix Halo chip usually draws your desktop, and its graphics memory is your RAM. Two things follow, and the engine now takes both into account.
|
||||
The Strix Halo chip usually draws your desktop, and its graphics memory is your RAM. The engine takes both into account.
|
||||
|
||||
- **One memory, counted once.** The expert cache of an integrated GPU lives in the same RAM as the expert arena. The automatic cache size is the smaller of
|
||||
what the GPU reports free and what the OS reports available, less a reserve for the desktop (the larger of 8 GiB and 8% of the RAM; the start-up line says so
|
||||
when it changes the size). `STRATA_UMA_RESERVE_GIB=N` sets the reserve (`0` = the old sizing, nothing kept back); `--expert-cache N` is never changed.
|
||||
- **A card of its own beside the chip** (for example a Radeon AI PRO R9700). A card of its own is the recommended Main card and the chip stays Off in the card roles.
|
||||
If you name the chip in a layer split anyway (`--gpus 0,1`, or the config's `"gpu": [0, 1]`), the split runs as asked, the chip is placed on the first layers and
|
||||
never on the last stage (the output head, the draft layer and the check of its guesses), the picture reader is started on the card of its own, and the
|
||||
start prints one plain line about it. `"gpu_order": "as_given"` and a manual `layer_split` keep your order. `setup --gpus all` leaves the chip out
|
||||
when two cards of their own are present; name it by number to use it.
|
||||
- **The CPU.** On Linux with built-in Radeon graphics the engine starts at nice +5, so the desktop keeps its share of the CPU while the engine's workers spin.
|
||||
- **One memory, counted once.** The automatic expert cache size is the smaller of what the GPU reports free and what the OS reports
|
||||
available, less a reserve for the desktop (the larger of 8 GiB and 8% of the RAM; the start-up line says so when it changes the
|
||||
size). `STRATA_UMA_RESERVE_GIB=N` sets the reserve (`0` = nothing kept back). `--expert-cache N` is never changed.
|
||||
- **A card of its own beside the chip** (for example a Radeon AI PRO R9700). That card is the recommended Main card and the chip
|
||||
stays Off in the card roles. If you name the chip in a layer split anyway (`--gpus 0,1`, or the config's `"gpu": [0, 1]`), the
|
||||
split runs as asked, the chip takes the first layers and never the last stage, the picture reader starts on the card of its own,
|
||||
and the start prints one plain line about it. `"gpu_order": "as_given"` and a manual `layer_split` keep your order.
|
||||
`setup --gpus all` leaves the chip out when two cards of their own are present; name it by number to use it.
|
||||
- **The CPU.** On Linux with built-in Radeon graphics the engine starts at nice +5, so the desktop keeps its share of the CPU.
|
||||
`STRATA_ENGINE_NICE=0` turns it off, `STRATA_ENGINE_NICE=N` sets another value.
|
||||
|
||||
What is not known yet: how much of the screen freeze a long compute job on the same chip causes, and whether registering the whole expert arena with the driver
|
||||
(the Linux default on gfx1151; Windows uses the unregistered arena since 0.2.1.2) hurts when the GTT is small. `STRATA_ARENA_PIN_GIB=none` runs the unregistered
|
||||
arena on any OS; reports with `amdgpu_top` and `dmesg` are welcome.
|
||||
Not known yet: how much of the screen freeze a long compute job on the same chip causes, and whether registering the whole expert
|
||||
arena with the driver (the Linux default on gfx1151) hurts when the GTT is small. `STRATA_ARENA_PIN_GIB=none` runs the unregistered
|
||||
arena on any OS. Reports with `amdgpu_top` and `dmesg` are welcome.
|
||||
|
||||
## 6. Measured
|
||||
|
||||
Ryzen AI Max+ 395 (128 GB, gfx1151), the maintainers' fast configuration of section 5 on top of the defaults of section 4, `--spec 4`,
|
||||
`--mtp`, `--lookup-chain 3`, `--mtp-q4 all`, `--prefill 16384`, int8 KV, all experts in the unified memory. Prompt tok/s is
|
||||
(prompt tokens - 1) / prompt time, output tok/s is the tokens / decode time; medians of 3 runs, one fresh process per run, the two
|
||||
engines interleaved. "Before" is the maintainers' earlier engine branch (the same switches set by hand), "now" is this release.
|
||||
Every pair printed the same token ids.
|
||||
**Measured on:** a Strix Halo PC (gfx1151, Windows), IQ3_S model, Strata 0.2.3 (engine 0.1.48), 3 pairs against 0.2.2, from
|
||||
[release-notes-0.2.3.md](release-notes-0.2.3.md). Answer speed +6% to +19% (tool calls +19%, code +18%, 64K prompt +13%, 8K prompt
|
||||
+12%, 4K prompt +8%, JSON +6%, agent chat +9%). Prompt speed +5% to +14% (8K +14%, 64K +5%, 4K unchanged because it is one chunk).
|
||||
Prompt speed at 4K, 8K and 64K is 685, 783 and 749 tokens per second. This is specific evidence, not validation of every APU or
|
||||
quantization.
|
||||
|
||||
The Linux numbers from the maintainers' Ryzen AI Max+ 395 (128 GB, Ubuntu 26.04, kernel 7.0, ROCm 7.14.1) are in the history
|
||||
section below.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
<a id="measured"></a>
|
||||
|
||||
**Measured on:** Ryzen AI Max+ 395, 128 GB, gfx1151, Linux, Strata 0.1.40 code, the maintainers' fast configuration of section 5 on
|
||||
top of the defaults of section 4, `--spec 4`, `--mtp`, `--lookup-chain 3`, `--mtp-q4 all`, `--prefill 16384`, int8 KV, all experts
|
||||
in unified memory. Prompt tok/s is (prompt tokens - 1) / prompt time. Medians of 3 runs, one fresh process per run, two engines
|
||||
interleaved, every pair printed the same token ids. "Before" is the maintainers' earlier engine branch with the same switches set
|
||||
by hand.
|
||||
|
||||
| Model | Context | Prompt before -> now | Output before -> now |
|
||||
|---|---|---|---|
|
||||
| --- | --- | --- | --- |
|
||||
| UD-IQ4_XS | 8K | 1,212 -> 1,293 (+6.7%) | 51.7 -> 53.8 (+4.0%) |
|
||||
| UD-IQ4_XS | 128K | 1,241 -> 1,320 (+6.4%) | 49.3 -> 51.4 (+4.2%) |
|
||||
| UD-Q4_K_XL | 8K | 1,136 -> 1,169 (+2.9%) | 50.6 -> 52.6 (+3.9%) |
|
||||
@@ -183,29 +205,33 @@ Every pair printed the same token ids.
|
||||
| IQ3_XXS | 128K | 1,259 -> 1,252 (-0.6%) | 48.4 -> 50.9 (+5.1%) |
|
||||
|
||||
UD-IQ4_XS at 4K / 32K / 64K (5-6 interleaved pairs): prompt +9.2% / +6.0% / +5.3%, output +5.0% / +3.9% / +4.6%.
|
||||
IQ3_XXS at 8K was measured again with 6 interleaved pairs (an earlier 3-run set had read 5% low while other jobs shared the GPU): prompt equal, output +5.6%. With the defaults off the same build's prompt is 8% lower (1,137), so the table helps there too.
|
||||
|
||||
### Re-measured on the merged 0.1.40 code
|
||||
|
||||
The same box and method, no environment switches at all (the arch defaults only), on the final merged tree. 4 interleaved pairs per cell,
|
||||
medians; "j" is the build of the table above (the Aurora port before the other 0.1.40 work was merged), "merged" is the release
|
||||
code. Every run printed the same token ids, and the ids also match `strata-int29` (the v4.2 config) on UD-IQ4_XS and UD-Q4_K_XL at
|
||||
a8 / 4K / 64K.
|
||||
**Same box, merged 0.1.40 code, no environment switches (arch defaults only).** 4 interleaved pairs per cell, medians. "j" is the
|
||||
build of the table above, "merged" is the 0.1.40 release code. Every run printed the same token ids. Prompt numbers at 8K swing by
|
||||
more than the differences because other jobs shared the box, so read them as unchanged.
|
||||
|
||||
| Model | Context | Prompt j -> merged | Output j -> merged |
|
||||
|---|---|---|---|
|
||||
| UD-IQ4_XS | 8K | 1,260 -> 1,159 (-8.0%, spread 1,070-1,320) | 53.94 -> 53.14 (-1.5%) |
|
||||
| --- | --- | --- | --- |
|
||||
| UD-IQ4_XS | 8K | 1,260 -> 1,159 | 53.94 -> 53.14 (-1.5%) |
|
||||
| UD-IQ4_XS | 128K | 1,317 -> 1,299 (-1.4%) | 51.38 -> 50.72 (-1.3%) |
|
||||
| UD-Q4_K_XL | 8K | 1,163 -> 1,104 (-5.0%, spread 890-1,240) | 52.66 -> 50.47 (-4.2%) |
|
||||
| UD-Q4_K_XL | 8K | 1,163 -> 1,104 | 52.66 -> 50.47 (-4.2%) |
|
||||
| UD-Q4_K_XL | 128K | 1,252 -> 1,211 (-3.3%) | 43.28 -> 42.64 (-1.5%) |
|
||||
| IQ3_S | 8K | 1,283 -> 1,265 (-1.4%) | 59.81 -> 59.10 (-1.2%) |
|
||||
| IQ3_S | 128K | 1,311 -> 1,310 (-0.1%) | 42.74 -> 42.09 (-1.5%) |
|
||||
| IQ3_XXS | 8K | 1,127 -> 1,212 (+7.6%) | 51.64 -> 50.95 (-1.3%) |
|
||||
| IQ3_XXS | 128K | 1,252 -> 1,250 (-0.1%) | 50.88 -> 50.22 (-1.3%) |
|
||||
|
||||
The prompt numbers at 8K swing by more than the differences (other jobs share this box), so read them as unchanged. The output
|
||||
column moved by a steady -1.2% to -1.5% (-4% on UD-Q4_K_XL at 8K, where the default arm also had slow runs): since 0.1.39 the
|
||||
shared-expert stream fork is off by default on HIP (#816). It overlaps the shared expert with the routed experts on a second
|
||||
stream and does help on gfx1151, so 0.1.40 turns it on there by default (section 4). It changes no bits (same ids). 4 interleaved pairs at 8K, merged
|
||||
code with the fork off -> on: UD-IQ4_XS output 53.1 -> 54.05 (+1.8%), UD-Q4_K_XL 49.5 -> 52.8 (+6.7%, the default arm ranged 46.0-52.1),
|
||||
prompt unchanged. With the fork on, the merged code's output is at or above the j build's (IQ4_XS 54.05 vs 53.94).
|
||||
With the shared-expert stream (`STRATA_SH_STREAM=1`, now a default) at 8K, 4 pairs: UD-IQ4_XS output 53.1 -> 54.05 (+1.8%),
|
||||
UD-Q4_K_XL 49.5 -> 52.8 (+6.7%), prompt unchanged.
|
||||
|
||||
**Other switch measurements** (Measured on: the same Ryzen AI Max+ 395)
|
||||
|
||||
| Switch | Result |
|
||||
| --- | --- |
|
||||
| `STRATA_HC_Q8` (IQ3_S, 2026-10-21, MTP 4 drafts, min-p 0.5) | `json_extract` 53.55 -> 58.03 tok/s (+8.4%); 94 of 250 output tokens differ from the BF16 read (first at 60); cold read 93.8 -> 20.3 us (142 -> 352 GB/s) |
|
||||
| `STRATA_PF_F16=1`, one layer, 16,384 tokens | Q4_K / Q5_1 85.3 -> 53.5 ms; IQ3_S / IQ4_NL gate/up 11.65 -> 9.87 ms and down 6.11 -> 4.71 ms |
|
||||
| `STRATA_PF_F16=1`, first-token KL against no switches, 12 prompts of 1.5K-24K tokens, top-1 same in 12 of 12 | UD-Q4_K_XL mean 0.0068 (median 0.0021) against 0.0031 (0.0024) without it; UD-IQ4_XS 0.0085 (0.0036) against 0.0068 (0.0032) |
|
||||
| `STRATA_REFILL_PARTS=1`, UD-Q4_K_XL and UD-IQ4_XS | 4K-8K prompt +2.3% to +3.5%, 32K +1.1% |
|
||||
| `--mtp-window 8192` | 64K output +3.5%, 32K -0.4% |
|
||||
| 12-turn agent replay, `STRATA_PREFILL_STREAM_MIN=128`, C2T8, `--no-prefill-borrow` | 3.4 -> 2.1 s per turn |
|
||||
| Defaults off, IQ3_XXS 8K | prompt 8% lower (1,137) |
|
||||
+195
-80
@@ -1,116 +1,231 @@
|
||||
# `strata launch`: set up, serve and watch, in the terminal
|
||||
# `strata launch`: set up, serve and watch
|
||||
|
||||
One command for the whole path: choose a model, tune it, start serving, and watch it. It is the same questions as the one-click setup (`START-HERE.bat` / `setup.sh`),
|
||||
in the Terminal's look, with presets, an Advanced table, an access choice and a Connect view for your coding apps. Text previews of every screen are in
|
||||
This page is for anyone who uses the installed Terminal. One command chooses a model, tunes it, starts serving and
|
||||
opens a monitor. It also connects your coding apps. Text previews of the screens are in
|
||||
[docs/terminal-previews/](terminal-previews/) (`launch-*.txt`).
|
||||
|
||||
## Quick start
|
||||
|
||||
1. Install the Terminal with the one-line command in [INSTALL.md](INSTALL.md#the-terminal).
|
||||
2. Run `strata launch` and follow the screens, or run a command without screens:
|
||||
|
||||
```sh
|
||||
strata launch --plan --family qwen --model q2_0 # what would happen; nothing changes
|
||||
strata launch --family qwen --model q2_0 --yes --no-tui # download, serve, print the address, return
|
||||
strata launch --status # what is serving now
|
||||
strata launch --stop
|
||||
```
|
||||
|
||||
`strata launch --help` lists every flag with examples. An unknown model or flag gets a plain message with the closest match.
|
||||
|
||||
## Install (one line)
|
||||
|
||||
Linux and Mac:
|
||||
|
||||
```sh
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh # Linux, macOS
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex # Windows PowerShell
|
||||
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
|
||||
```
|
||||
|
||||
It finds out which computer this is, downloads the Terminal archive named in the release folder's `install-manifest.txt`, checks its SHA-256, unpacks it into your
|
||||
home (`~/.local/share/strata/terminal`, `%LOCALAPPDATA%\Strata\terminal`; no administrator rights), checks the release's signed feed with the Terminal it just unpacked
|
||||
(`strata update --check`; if that fails it removes what it unpacked), puts `strata` on your PATH and prints `Installed. Run: strata launch`. The engine (the part that talks
|
||||
to your graphics card) comes with the first model: `strata launch` fetches the one for your card from the same release folder, signed, and test-starts it before using it.
|
||||
The release folder is the latest GitHub release of Niko1221/Strata (`https://github.com/Niko1221/Strata/releases/latest/download`). The script only fetches and checks; read it first if you like (it is the same file as `install.sh` / `install.ps1` in the repository's `packaging` folder and in the release). Set `STRATA_FEED` to another folder to try a local one; `packaging/make_local_feed.py` builds a test-signed one. How far to trust a piped installer is in [SECURITY.md](../SECURITY.md).
|
||||
Windows PowerShell (then open a new terminal):
|
||||
|
||||
## The screens (`strata launch`)
|
||||
```powershell
|
||||
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
|
||||
```
|
||||
|
||||
1. **Start**: saved presets (name, model, context, parallel requests, tuned or not, last used) or "Set up from the beginning" or "Use model files I already have".
|
||||
2. **Model**: the catalog of the app and `setup.py` (one source), with download size, what it needs (graphics memory + RAM) and "fits / tight / does not fit this computer".
|
||||
A warning, never a block. The n-gram table stays on your SSD and is not counted. `o` = your own files (a folder or a .gguf; checked with the fingerprint table).
|
||||
*(Cards: only with two or more usable graphics cards. Each card gets a role: M Main, H Helper, O Off, R back to the recommendation, D details. `--gpu-roles 1=main,0=helper` does the same from the command line; see [MULTI_GPU.md](MULTI_GPU.md).)*
|
||||
3. **Folder**: where model files are stored, free space shown. For a model that is already installed (also one the desktop app installed) it starts where the model is, says
|
||||
"Already downloaded" and lists only what is really missing with its size (the draft layer, about 5.2 GB; the picture reader, 0.9 GB). It never says "download the model" when only
|
||||
a helper file is missing.
|
||||
More in [INSTALL.md](INSTALL.md#the-terminal). The installer needs no administrator rights and puts `strata` on your PATH.
|
||||
How far to trust a piped installer is in [SECURITY.md](../SECURITY.md). The engine (the part that talks to your graphics
|
||||
card) comes with the first model: `strata launch` fetches the one for your card from the same release, checks its signature
|
||||
and test-starts it before using it.
|
||||
|
||||
## The screens
|
||||
|
||||
1. **Start:** saved presets (name, model, context, parallel requests, tuned or not, last used), "Set up from the beginning"
|
||||
or "Use model files I already have".
|
||||
2. **Model:** the catalog the app and `setup.py` share, with download size, what it needs (graphics memory and RAM) and
|
||||
"fits / tight / does not fit this computer". This is a warning, never a block. The n-gram table stays on your SSD and is
|
||||
not counted. `o` picks your own files (a folder or a `.gguf`).
|
||||
3. **Folder:** where model files are stored, with free space. For a model that is already installed, including one the
|
||||
desktop app installed, it starts where the model is and lists only what is missing (the draft layer, about 5.2 GB; the
|
||||
picture reader, 0.9 GB).
|
||||
4. **Context** (8K to 512K, with the extra memory each costs) and the memory format (`m`: 8-bit or 4-bit).
|
||||
5. **Pictures**: only for models that can read them. They are **on by default** (Yes is preselected; `--vision off` or "No" turns them off). It says what it adds (about 0.9 GB:
|
||||
the picture-reader file, downloaded with the model through setup's Control mode, the same path the app's "Read pictures" uses). The reader runs on the CPU
|
||||
for a card under 24 GB (a few seconds per picture, nothing taken from the card) and on the graphics card from 24 GB (about 1-1.4 GB kept free), on NVIDIA and AMD cards; the
|
||||
rule lives in one place, `strata_control/vision_reader.py`, for setup, `strata launch` and the app's catalog. `--vision gpu` / `--vision cpu` (or the Pictures step) switch it per model. It needs a picture reader beside the engine in the engine package; with an
|
||||
engine that has none, pictures default to off and the screen says so in plain words. A model that cannot read pictures stays off with no question.
|
||||
6. **Parallel requests**: "fit to your card" (12 GB: 1, 20: 2, 24: 3, 32+: 4) or 1 to 4; a warning if it will not fit.
|
||||
7. **Tuning**: quick (about 20 minutes) / thorough (up to 2 hours) / skip (`strata tune auto`; the "answers may change very slightly" tick is off). The screens suggest quick the first
|
||||
time (it is done once and kept); the command line skips tuning unless you add `--tune`.
|
||||
8. **Advanced** (optional): every engine setting with default, tuned value and your value; edit, reset, add a setting by name (checked against the engine's own list; a warning, not a refusal);
|
||||
the exact command (it includes the context, memory format and picture choice you made above; it has its own scroll area: Tab, then the arrows) and the matching `strata launch` flags.
|
||||
9. **Access**: this computer only (127.0.0.1), your network (LAN address, key), Cloudflare quick tunnel (needs `cloudflared`), Tailscale (needs it installed and connected); port; API key
|
||||
(optional, recommended beyond this computer: Strata makes one for you, and one keypress (`t`) turns it off. Without a key the server still starts and warns that anyone who can reach the port can use the model and its tools; `--no-api-key` does the same without the screens, and on a Cloudflare tunnel the warning is stronger. This computer only never needs a key).
|
||||
10. **Save as preset** and launch. Then downloads (with progress), engine start, and the monitor. In a small window the summary shrinks so the three choices stay on screen,
|
||||
and the key line says what Enter will do. No web page opens by itself; only `--open` opens one.
|
||||
5. **Pictures:** only for models that can read them. They are on by default (`--vision off` or "No" turns them off). The
|
||||
reader adds about 0.9 GB. It runs on the CPU for a card under 24 GB and on the graphics card from 24 GB (about 1 to 1.4 GB kept
|
||||
free). The rule is in `strata_control/vision_reader.py`. `--vision gpu` or `--vision cpu` overrides it per model. If the
|
||||
engine package has no picture reader, pictures default to off and the screen says so.
|
||||
6. **Parallel requests:** "fit to your card" (12 GB: 1, 20: 2, 24: 3, 32 and more: 4) or a number from 1 to 8, with a warning
|
||||
if it will not fit. Check the effective slots after startup.
|
||||
7. **Tuning:** quick (about 20 minutes), thorough (up to 2 hours) or skip. The screens suggest quick the first time; the
|
||||
command line skips tuning unless you add `--tune`.
|
||||
8. **Advanced** (optional): every engine setting with its default, tuned value and your value. You can edit, reset, or add a
|
||||
setting by name (checked against the engine's own list, with a warning instead of a refusal). It also shows the exact command and
|
||||
the matching `strata launch` flags.
|
||||
9. **Access:** this computer only (127.0.0.1), your network (LAN address and key), a Cloudflare quick tunnel (needs `cloudflared`)
|
||||
or Tailscale (needs it installed and connected). Also the port and an API key. Beyond this computer a key is recommended:
|
||||
Strata makes one, and `t` turns it off. Without a key the server still starts and warns that anyone who can reach the port
|
||||
can use the model and its tools; `--no-api-key` does the same without screens, and the warning is stronger on a Cloudflare tunnel.
|
||||
10. **Save as preset** and launch. Downloads show progress, then the engine starts and the monitor opens. No web page opens
|
||||
unless you pass `--open`.
|
||||
|
||||
Keys: arrows choose, Enter goes on, Esc goes back, `?` explains the screen, `q` leaves. Every screen says in plain words what it means and what to do next.
|
||||
Keys: arrows choose, Enter goes on, Esc goes back, `?` explains the screen, `q` leaves.
|
||||
|
||||
With two or more usable graphics cards the Model step also asks for a role per card (M main, H helper, O off). On the command
|
||||
line, `--gpu-roles 1=main,0=helper` does the same ([MULTI_GPU.md](MULTI_GPU.md)).
|
||||
|
||||
## Flags
|
||||
|
||||
```sh
|
||||
strata launch --preset "IQ3_XXS 128K" # a saved preset (opens the monitor)
|
||||
strata launch --family qwen --model iq3_xxs --context 128k --parallel 4 --vision off --tune quick \
|
||||
--access lan --port 8080 --save-preset "Office" --no-tui
|
||||
strata launch --model-path /data/models/IQ3_XXS --no-tui # files you already have
|
||||
strata launch --preset Office --set prefill=65536 --set STRATA_PIN_BY_NEED=1 --no-tui
|
||||
strata launch --family qwen --model q2_0 --context 64k --restart --no-tui
|
||||
strata launch --list-presets
|
||||
strata launch --show-key
|
||||
```
|
||||
|
||||
| Flag | What it does |
|
||||
| --- | --- |
|
||||
| `--preset NAME` | Start a saved preset. Other flags change that start only; add `--save-preset` to keep them. |
|
||||
| `--family`, `--model` | The model family and size, for example `qwen`, `q2_0`. Any case. |
|
||||
| `--model-path PATH` | Use model files you already have: a folder of shards or one `.gguf`. |
|
||||
| `--data-dir DIR`, `--move-data` | Where new downloads go. `--move-data` also moves the models you already have. |
|
||||
| `--context SIZE` | `32k`, `64k`, `128k`, `256k` or a number of tokens. |
|
||||
| `--kv FORMAT` | Conversation memory format: `int8` (default), `q4_0` (half the memory), `k8v4`, `fp16`. |
|
||||
| `--vision off\|on\|gpu\|cpu` | Picture reading. Default: on when the model and the engine can. |
|
||||
| `--parallel N` | Requests answered at the same time, 1 to 8. Default: fit to your card. |
|
||||
| `--mtp on\|off` | The draft layer. `off` saves about 1 GB of graphics memory and is slower. Default: on. |
|
||||
| `--drafting auto\|mtp\|dflash2\|off` | Which drafter guesses ahead. `dflash2` is for Qwen3.8-Flash-Next only and downloads a 1.2 GB drafter once. If it cannot run, MTP is used and the start says why. |
|
||||
| `--gpu LIST`, `--gpu-roles LIST` | Which cards to use, and a role per card. |
|
||||
| `--tune quick\|thorough\|skip`, `--tune-allow-small`, `--retune` | Tuning. `--retune` measures again; a model tuned before for the same choices on this computer reuses the stored result. |
|
||||
| `--set KEY=VALUE` | An advanced engine setting or `STRATA_*` variable (repeatable). Unknown names get a warning. |
|
||||
| `--pool-cores LIST` | Experimental: pin CPU workers to these logical processors, for example `2-14:2` ([DETAILS.md](DETAILS.md)). |
|
||||
| `--access local\|lan\|cloudflare\|tailscale`, `--port N` | Who can reach the server. |
|
||||
| `--api-key KEY`, `--api-key-env NAME`, `--new-key`, `--no-api-key` | The key clients must send. |
|
||||
| `--save-preset NAME`, `--list-presets`, `--delete-preset NAME` | Presets. |
|
||||
| `--no-tui` | No screens: serve, print the address and return. |
|
||||
| `--yes` | No questions. It never restarts a running model. |
|
||||
| `--restart` | Stop a model that is serving other choices (or holds the port), then start these. |
|
||||
| `--open` | Open the server's web page once it serves. |
|
||||
| `--plan` | Show what would happen and exit. It lists the model, context, picture setting and what would download (0 when everything is here). |
|
||||
| `--feed URL\|DIR` | The signed update feed. Default: `STRATA_UPDATE_FEED` or the project's. |
|
||||
| `--output human\|json` | Output format. |
|
||||
| `--status`, `--stop`, `--show-key` | Show, stop, or print the key of the running launch (add `--preset NAME` to pick one). |
|
||||
|
||||
Presets are small files in `launch-presets/` beside the profiles folder. A run uses the shared profile `launch-<name>` and the shared
|
||||
tuning presets (`strata presets`), so the app sees them too.
|
||||
|
||||
**Launching while a model is serving.** The same choices restart nothing and print the running addresses. Different choices
|
||||
(another context, pictures, access, or another model on the same port) make the screens ask once, "Restart the model with these
|
||||
settings? Answers in progress stop." With `--no-tui` the command refuses with one line naming the running model, its port and settings. Add
|
||||
`--restart` or run `strata launch --stop` first. A different `--port` runs beside the first server; two big models may not fit in graphics memory together, and the launch says so.
|
||||
|
||||
## DeepSeek V4.1-Flash
|
||||
|
||||
DeepSeek V4.1-Flash runs in the Terminal and in the app, on NVIDIA RTX 30, 40 and 50 (Windows and Linux) and on the Radeon AI PRO R9700 (Linux). Hardware, sizes and speeds are in [MODELS.md](MODELS.md#deepseek-v41-flash).
|
||||
|
||||
```sh
|
||||
strata launch --plan --family deepseek41 --model ist-das-2.00bpw
|
||||
strata launch --family deepseek41 --model ist-das-2.00bpw
|
||||
```
|
||||
|
||||
| Model name | Download |
|
||||
| --- | ---: |
|
||||
| `ist-das-1.75bpw` | 239 GB |
|
||||
| `ist-das-2.00bpw` | 257 GB |
|
||||
| `ist-das-2.25bpw` | 274 GB |
|
||||
| `ud-q8_k_xl` | 508 GB |
|
||||
|
||||
- The context is 8192 tokens (8K), the size the engine was checked at.
|
||||
- `--speed faster|full` picks how many of the model's 6 experts it uses per word. Faster uses 5 (a small quality change, usually not noticeable) and is the default on a PC with 16 GB of graphics memory or less and on PCs whose graphics share the computer's memory. Full uses all 6. `--full-quality` is the same as `--speed full`. Both need DeepSeek engine 0.1.50 or newer, which `strata launch` fetches. The engine's own flag is `--experts N`.
|
||||
- It answers one request at a time. `--parallel` above 1 is ignored with a note.
|
||||
- It cannot read pictures, so pictures are off and `--vision on` is ignored with a note.
|
||||
- `--tune` runs DeepSeek's own short tuner (drafter on or off, CPU threads), not the Qwen tuner.
|
||||
- `--model-path DIR` starts from the shards you already have.
|
||||
- `--plan` shows where the model goes on your cards and how much is held in RAM.
|
||||
|
||||
## After the start: the monitor
|
||||
|
||||
The Strata Monitor with the new **Overview** (key 1: model state bar, speed, GPU load, VRAM, temperature, power, PCIe, CPU, disk, context fill, experts in VRAM, system RAM, recent requests; read from the
|
||||
server's `/metrics`, the same data as the browser Monitor), the other five views as before, and **7 Connect** (the seventh view: the arrow keys reach it like the others, and every view's hint says 1-7): the address, the key (hidden until `k`), and the setup for each app
|
||||
(`c` copy, `w` write it for you after a backup, `h` check it: one prompt and one tool call sent the way that app sends them). The list keeps two things apart: **Vendor tested** (the app
|
||||
version its maker tested the setup with: history) and **This server** (Not checked / Passed / Failed for your server, after `h`). `q` asks: keep serving in the background, or stop
|
||||
(stopping unloads the model and frees the graphics memory).
|
||||
The monitor has seven views. Press `1` to `7` or use the arrow keys.
|
||||
|
||||
In the classic Windows console (the Windows 10 default window) the monitor first switches that window's font to Cascadia Mono when it is installed (the console font API: no admin, nothing saved, gone with the window) and draws the full look; otherwise it draws a solid-block look: █ bars with a thin ▄ track, ▄ / █ sparklines, square boxes and the 16 palette colours (`•` instead of `●` when the font is not Consolas). It always shows one line: "For the full look, use Windows Terminal (built into Windows 11)". Windows Terminal, Linux and macOS are never touched. `STRATA_GLYPHS=unicode|ascii|solid` forces a look. With two or more graphics cards the Overview tiles show the busiest card (load), all cards (memory), the hottest card and the total power, a "Cards" strip lists each card (up to 4, then "press 5" for the Hardware view), and the CPU tile says "2 CPUs · 64 cores" from the server's new `sockets` figure.
|
||||
| Key | View |
|
||||
| --- | --- |
|
||||
| 1 | Overview: model state, speed, GPU load, VRAM, temperature, power, PCIe, CPU, disk, context fill, experts in VRAM, system RAM, recent requests. It reads the server's `/metrics`, like the browser Monitor. |
|
||||
| 2 to 6 | Serving, Requests, Engine, Hardware, Logs |
|
||||
| 7 | Connect: the address, the key (hidden until `k`) and the setup for each app |
|
||||
|
||||
## Without the screens (flags)
|
||||
In Connect, `c` copies a setup, `w` writes it after a backup and `h` checks it by sending one prompt and one tool call the way that
|
||||
app sends them. The list keeps two columns apart. **Vendor tested** is the app version its maker tested the setup with.
|
||||
**This server** is Not checked, Passed or Failed for your server after `h`.
|
||||
|
||||
```sh
|
||||
strata launch --preset "IQ3_XXS 128K" # a saved preset (opens the monitor)
|
||||
strata launch --model iq3_xxs --context 128k --parallel 4 --vision off --tune quick \
|
||||
--access lan --port 8080 --save-preset "Office" --no-tui # serve and print the address
|
||||
strata launch --model-path /data/models/IQ3_XXS --no-tui # files you already have
|
||||
strata launch --preset Office --set prefill=65536 --set STRATA_PIN_BY_NEED=1 --no-tui
|
||||
strata launch --model q2_0 --pool-cores 2-14:2 --no-tui # experimental: pin the CPU workers to these logical processors (see DETAILS.md)
|
||||
strata launch --plan --model q2_0 # what would happen; nothing changes
|
||||
strata launch --model q2_0 --yes --no-tui # no questions (for one: download on a computer with no engine yet)
|
||||
strata launch --model q2_0 --open # also open the server's page in your browser once it serves
|
||||
strata launch --model q2_0 --context 64k --restart --no-tui # a model is already serving other choices (or holds the port): stop it and start these
|
||||
strata launch --list-presets | --delete-preset NAME | --status | --stop [--preset NAME] | --show-key
|
||||
```
|
||||
`q` asks whether to keep serving in the background or stop. Stopping unloads the model and frees graphics memory.
|
||||
|
||||
`strata launch --help` lists everything with examples. An unknown model or flag gets a plain message with the closest match. Presets are small files in `launch-presets/` beside the profiles
|
||||
folder; the run uses the shared profile `launch-<name>` and the shared tuning presets (`strata presets`), so the app sees them too. Flags next to `--preset` change that start only
|
||||
(add `--save-preset` to keep them). `--plan` shows the model (also for your own files, which it recognises), the context, the picture setting and what would be downloaded and how big (0 when everything is already here). A model that was tuned before for the same choices on the same computer is not measured again by `--tune quick`; the stored result is used and the run says so. Add `--retune` to measure again (the model of the same launch is stopped for the measuring and starts again with the result).
|
||||
With two or more cards the Overview shows the busiest card's load, all cards' memory, the hottest card and the total power, and a
|
||||
"Cards" strip lists up to 4 cards (`5` opens Hardware). The CPU tile shows sockets and cores.
|
||||
|
||||
**A launch while a model is already serving.** The same choices: nothing restarts, the running addresses are printed. Different choices (another context, pictures, access, or another model on the same port): the screens ask once, "Restart the model with these settings? Answers in progress stop.", with Restart and Cancel; `--no-tui` refuses with one line naming the running model, its port and settings, and says to add `--restart` or run `strata launch --stop` first. `--yes` alone never restarts. With `--restart` the old server is stopped first and the new one starts on the same port. A different port given with `--port` runs beside the first one; two big models may not fit in the graphics memory together, and the launch says so.
|
||||
**Windows console.** In the classic Windows 10 console the monitor switches that window's font to Cascadia Mono when it is installed
|
||||
and draws the full look. Otherwise it draws a solid-block look and shows one line suggesting Windows Terminal. Nothing is saved and
|
||||
Windows Terminal, Linux and macOS are never touched. `STRATA_GLYPHS=unicode|ascii|solid` forces a look.
|
||||
|
||||
## Connect your apps: `strata connect`
|
||||
|
||||
`strata connect` lists OpenCode, Codex CLI, Claude Code, oh-my-pi (omp), pi, Hermes Agent, OpenClaw and the DeepSeek Harness (built into the app). `strata connect opencode` prints the file and
|
||||
environment for the running server (address, key, model name and context filled in); `--write` writes it after asking and keeps a backup of anything it replaces; `--check` sends one prompt and one tool call
|
||||
the way that app does and says whether each worked. With `--output json` (`strata connect opencode --check --output json`) it prints one JSON object with the same file and environment fields plus `checks` (one entry per check: `name`, `ok`, `detail`, `seconds`) and `checked_server`; `ok` is true only when both checks passed. The exit code is 0 when both worked and 1 when one failed or the server could not be reached; in that case the object also carries an `error` that says which of the two it was. Each configuration was tested for real against a Strata server at 32K context (the app versions are on screen); the known limits are printed in plain
|
||||
words (for example: Claude Code needs `CLAUDE_CODE_MAX_CONTEXT_TOKENS`; OpenCode needs `limit.input`; Hermes refuses windows under 64K unless set up as the config does; OpenClaw needs Node 24.16+).
|
||||
`strata connect` lists OpenCode, Codex CLI, Claude Code, oh-my-pi (`omp`), pi, Hermes Agent, OpenClaw and the DeepSeek Harness
|
||||
(built into the app).
|
||||
|
||||
## Boxes the installer cannot set up
|
||||
```sh
|
||||
strata connect opencode # print the file and environment, with address, key, model and context filled in
|
||||
strata connect opencode --write # write it after asking; anything replaced is backed up
|
||||
strata connect opencode --check # send one prompt and one tool call the way that app does
|
||||
strata connect opencode --check --output json
|
||||
```
|
||||
|
||||
`setup.py` sets up NVIDIA (CUDA) and AMD (HIP) cards. For an Intel Arc card its Linux path is a container build that the Terminal does not run: if the model's files and their prepared pack are
|
||||
already on the disk, `strata launch` writes the run settings itself and says so; otherwise it says which file is missing. Nothing is downloaded or compiled in that case.
|
||||
| Flag | What it does |
|
||||
| --- | --- |
|
||||
| `--url URL`, `--key KEY`, `--key-env NAME` | The server address and key. Default: the running launch, else `http://127.0.0.1:8080`. |
|
||||
| `--model NAME`, `--context SIZE` | The model name the server reports and the context it started with. Default: the running launch's, else 32k. |
|
||||
| `--preset NAME` | Take address, key and model from a saved preset. |
|
||||
| `--yes`, `--show-key` | Do not ask before `--write`; print the real key instead of a masked one. |
|
||||
|
||||
With `--output json` the object has the file and environment fields, `checks` (one entry per check with `name`, `ok`, `detail`,
|
||||
`seconds`) and `checked_server`. `ok` is true only when both checks passed. The exit code is 0 when both worked and 1 otherwise;
|
||||
then `error` says which check failed or that the server could not be reached.
|
||||
|
||||
Each configuration was tested against a Strata server at 32K context. Known limits are printed in plain words, for example:
|
||||
Claude Code needs `CLAUDE_CODE_MAX_CONTEXT_TOKENS`; OpenCode needs `limit.input`; Hermes refuses windows under 64K unless set up
|
||||
as the config does; OpenClaw needs Node 24.16 or newer.
|
||||
|
||||
## Models you already have
|
||||
|
||||
`strata models list` looks in the same places the screens and `strata launch` look (one function, `strata_control/discovery.py`): the folder you name with `--install-dir` or
|
||||
STRATA_INSTALL_DIR (then only that one), the Terminal's own folder, the folders earlier installs remembered in settings.json, and the desktop app's install folder, which the Terminal
|
||||
shares (Windows `%LOCALAPPDATA%\Strata\install`; Linux and macOS `$XDG_DATA_HOME/strata/install`, by default `~/.local/share/strata/install`). The data folder is the one you set, else the
|
||||
app's (`app-settings.json` beside its install folder, else a `Documents/Strata` that holds models), else Strata's default. `strata models import PATH --plan` knows the Gyro-S and Gyro-M files
|
||||
(TQK6/TQK7 and TQ2_T experts) and every file set setup installs, also under another name when the header matches, and says which run config, prepared pack, draft layer and picture reader
|
||||
already exist for them. Nothing is copied or moved.
|
||||
`strata models list` looks in these places (one function, `strata_control/discovery.py`):
|
||||
|
||||
- the folder you name with `--install-dir` or `STRATA_INSTALL_DIR` (then only that one)
|
||||
- the Terminal's own folder
|
||||
- folders earlier installs remembered in `settings.json`
|
||||
- the desktop app's install folder, which the Terminal shares: `%LOCALAPPDATA%\Strata\install` on Windows,
|
||||
`$XDG_DATA_HOME/strata/install` (normally `~/.local/share/strata/install`) on Linux and macOS
|
||||
|
||||
The data folder is the one you set, else the app's (`app-settings.json` beside its install folder, else a `Documents/Strata` that holds
|
||||
models), else Strata's default.
|
||||
|
||||
`strata models import PATH --plan` recognises the Gyro-S and Gyro-M files (TQK6/TQK7 and TQ2_T experts) and every file set setup
|
||||
installs, also under another name when the header matches. It says which run config, prepared pack, draft layer and picture reader
|
||||
already exist. Nothing is copied or moved.
|
||||
|
||||
## When there is no engine for this computer
|
||||
|
||||
`strata launch` does not refuse. It says, in plain words, "Strata couldn't find an engine for this computer (the reason). The model will download, but it may not run until you install the right
|
||||
engine (next step: ...)", asks whether to go on (Enter means yes; `--yes` skips the question; nothing to download means no question), downloads the model, and ends with a message that names the
|
||||
engine problem and the next step. The exit code is 0 and nothing is started.
|
||||
`strata launch` does not refuse. It says "Strata couldn't find an engine for this computer" with the reason, asks whether to go on
|
||||
(Enter means yes; `--yes` skips the question), downloads the model, and ends with a message that names the problem and the next step.
|
||||
The exit code is 0 and nothing is started.
|
||||
|
||||
## Cards of different makes
|
||||
## Intel Arc and cards of different makes
|
||||
|
||||
One engine drives the cards of one make (CUDA: NVIDIA, HIP: AMD). Only those cards are offered for running and splitting; a model is never split across makes, and two models at the same
|
||||
time, one on each make, is not supported yet. When the other make has the bigger card you get one line: "This computer also has an AMD R9700 (32 GB). The AMD build of Strata can use it - it fits bigger models."
|
||||
- **Intel Arc:** the Terminal downloads the released Linux SYCL engine, but its model preparation for Arc is a container build that it does not
|
||||
run. If the model's files and prepared pack are already on disk, `strata launch` writes the run settings itself. Otherwise it reports the preparation problem. Use
|
||||
[Intel setup](INTEL_ARC.md) for the preparation route.
|
||||
- **Mixed makes:** one engine drives the cards of one make (CUDA: NVIDIA, HIP: AMD). Only those cards are offered, and a model is never
|
||||
split across makes. Two models at once, one per make, is not supported. If the other make has the bigger card, you get one line saying so.
|
||||
|
||||
## Limits
|
||||
|
||||
* The model download uses `setup.py` in its control mode (no pip, no engine fetch, no compiler), as the desktop app does; it needs internet and room for the files.
|
||||
* `strata update`'s test start of an engine runs `<engine> --version`; the SYCL (Intel) package's engine does not answer that flag (exit 2), so on an Intel box the engine package has to be copied into the
|
||||
components folder until that is fixed.
|
||||
* The Windows console has no way for the test scripts to drive the screens; they were checked by pilot tests and by hand.
|
||||
- The model download uses `setup.py` in its control mode (no pip, no engine fetch, no compiler), as the desktop app does. It needs internet and room for the files.
|
||||
- The Linux SYCL package updates through `strata update`. Do not copy it into the components folder by hand.
|
||||
@@ -1,8 +1,11 @@
|
||||
# Terminal quick start
|
||||
|
||||
New here? `strata launch` sets up a model, serves it and opens the monitor in one guided flow: [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md).
|
||||
This page is for people who already have a model and want to read a running server, rerun it from a saved profile, or measure it.
|
||||
To set up and serve a new model, use [`strata launch`](TERMINAL_LAUNCH.md).
|
||||
|
||||
These commands use the `strata` command that the one-line Terminal install puts on your PATH ([how to install](TERMINAL_LAUNCH.md#install-one-line)). The Terminal carries its own Python, so there is nothing else to install. In a source checkout, type `python strata.py` (or `python3 strata.py`) in place of `strata`, from the repository root. The commands below read, prepare and rerun a model you already have; they do not download one. The complete options, permissions and limits are in [TERMINAL.md](TERMINAL.md).
|
||||
The commands use the `strata` command from the [Terminal install](INSTALL.md#the-terminal), which carries its own Python.
|
||||
In a source checkout, type `python strata.py` in place of `strata`, from the repository root. These commands read, prepare
|
||||
and rerun a model you already have; they do not download one. All options, permissions and limits are in [TERMINAL.md](TERMINAL.md).
|
||||
|
||||
## Read a running server
|
||||
|
||||
@@ -18,7 +21,7 @@ The interactive monitor comes with the installed Terminal. Open the one-screen O
|
||||
strata monitor --url http://127.0.0.1:8080
|
||||
```
|
||||
|
||||
The Overview puts serving state, prompt and answer speed, batch/queue pressure, four selected GPU rows, requests and activity on one screen. Press `1`–`6` for detail (`1`–`7` in the monitor that `strata launch` opens, where `7` is the Connect view), `Space` to freeze the display, `r` to refresh, and `q` to detach. Detach leaves the server running. On a machine without a terminal, use `status --json` or `monitor --once --json`.
|
||||
The Overview puts serving state, prompt and answer speed, batch/queue pressure, four selected GPU rows, requests and activity on one screen. Press `1`–`7` for the views (`7` is Connect), `Space` to freeze the display, `r` to refresh, and `q` to detach. Detach leaves the server running. On a machine without a terminal, use `status --json` or `monitor --once --json`.
|
||||
|
||||
For a remote server, use HTTPS or a local SSH tunnel. A saved connection stores only the credential environment variable's **name**:
|
||||
|
||||
|
||||
+109
-246
@@ -1,279 +1,142 @@
|
||||
# Something went wrong?
|
||||
# Troubleshooting
|
||||
|
||||
The common problems and what to do, for the desktop app, the Terminal (`strata launch`) and the installer. Back to the [README](../README.md#something-went-wrong). The full table of
|
||||
error messages, with the older engine fixes, is in the [details](DETAILS.md#troubleshooting).
|
||||
This page is for anyone whose install or startup fails. Find your symptom below. Each entry gives one fix.
|
||||
Start from the route you installed: [desktop app or Terminal](INSTALL.md), or [source checkout](SOURCE_INSTALL.md).
|
||||
The normal release uses the app or `strata launch` and needs no `.bat` file. Detailed engine errors are in the
|
||||
[engine reference](DETAILS.md#troubleshooting).
|
||||
|
||||
**Quick check.** Run these and keep the output for a bug report:
|
||||
|
||||
```sh
|
||||
strata --version
|
||||
strata doctor --live
|
||||
strata launch --status
|
||||
```
|
||||
|
||||
## The desktop app and the Terminal
|
||||
|
||||
**A warning says Strata "couldn't find an engine for this computer".**
|
||||
The engine is the part that talks to your graphics card. Strata only warns and still downloads the
|
||||
model, and the model files stay on your computer, but the model may not run until the right engine is installed. The usual
|
||||
reasons are that no engine is installed yet and the update source has none for your card, that the engine found is for
|
||||
another make of card, or that the graphics driver is too old. In the app, open Settings, then Engine. In the Terminal, the
|
||||
message names the next step: run `strata update` and start again. If your computer has no NVIDIA or AMD graphics card
|
||||
(only an Intel UHD or Iris chip, an AMD processor graphics chip, a virtual machine, or Windows on ARM), Strata says so and
|
||||
there is nothing to install: it needs a computer that has one. The model files stay where they are.
|
||||
**I cannot find `START-HERE.bat` or `SETUP.bat`.**
|
||||
Open the installed Strata app, or run `strata launch`. Those wrappers are in the source repository and are not the
|
||||
startup path for the app. For a fresh desktop install, download the one installer for your system
|
||||
(`Strata-0.2.6-windows-x64-setup.exe`, `Strata-0.2.6-linux-x86_64.AppImage` or `Strata-0.2.6-macos-arm64.dmg`), not a
|
||||
`strata-engine-…` archive.
|
||||
|
||||
**"This model needs a newer engine." (the app), or a button called Update engine.**
|
||||
Some models need an engine newer than the one installed: Gyro needs 0.1.45.1 or newer, NVFP4 needs 0.1.44, and NVFP4 (NVIDIA,
|
||||
trained for FP4) needs 0.1.45. In the app, open Settings, then Engine, and press Update engine. It asks first, keeps your
|
||||
model files and settings, and wants running models unloaded. In the Terminal: `strata update --check` says what would
|
||||
change, `strata update` installs it (it switches to the new engine only after a test start). If the engine you
|
||||
installed is the newest and the message stays, the update source has no newer engine for your card yet.
|
||||
**`strata` is not found.**
|
||||
On Windows, open a new PowerShell window after installing. On Linux or Mac, use the full path the installer printed,
|
||||
normally `~/.local/bin/strata`, and add `~/.local/bin` to PATH if needed. Check `strata --version`.
|
||||
|
||||
**On a Mac: "Strata is damaged and can't be opened", "cannot be opened", or "unidentified developer".**
|
||||
Nothing is wrong with the file. macOS marks everything a browser downloads, and only lets a marked program run if Apple
|
||||
has checked it. Strata's release builds are checked by Apple ("notarised"): drag Strata onto Applications and open it; the
|
||||
first time, macOS asks "Are you sure you want to open it?" and you press Open. A test build that is not notarised says
|
||||
"damaged". Do not move it to the Trash. Open the Terminal app and remove the mark, with the line that matches what you have
|
||||
(drag the file from Finder into the Terminal window to type its path):
|
||||
**Strata could not find an engine for this computer.**
|
||||
Run `strata doctor --live` and `strata update --check`. Check the selected GPU, driver and
|
||||
[backend guide](INSTALL.md#what-you-need). Supported paths include Apple Silicon and Intel Arc on Linux; do not
|
||||
assume every machine without an NVIDIA/AMD discrete card is unsupported. A model download can finish even when
|
||||
no engine starts. Verify `strata launch --status` and `/health` before calling the installation successful.
|
||||
|
||||
xattr -dr com.apple.quarantine ~/Downloads/<the .dmg file> # the .dmg, before opening it
|
||||
xattr -dr com.apple.quarantine /Applications/Strata.app # the app, after dragging it over
|
||||
xattr -dr com.apple.quarantine <the folder you unzipped> # an unzipped Terminal folder
|
||||
**This model needs a newer engine.**
|
||||
Unload running models, then use the app's engine update or `strata update`. Minimum engine versions vary by model and
|
||||
backend. Check the model guide and the installed engine, not only the app version. If the feed has no compatible engine,
|
||||
report the missing combination.
|
||||
|
||||
Then open it again. If macOS says only "the developer cannot be verified", you can instead try to open it once, then open
|
||||
System Settings > Privacy & Security and press "Open Anyway" next to the message about Strata. Installs made with
|
||||
`curl ... | sh` carry no mark and need none of this. To see which kind you have: `spctl -a -vv /Applications/Strata.app`
|
||||
answers "accepted" and "Notarized Developer ID" for a notarised build. (How the builds are signed: `packaging/mac/README.md`.)
|
||||
**An update says it succeeded but `strata --version` is still old.**
|
||||
The Terminal switches to the updated Control on the next command, so run `strata --version` again. If it is still old,
|
||||
run the one-line installer once more; it keeps models and settings. The app, Control and engine have separate version
|
||||
numbers; see [updating](INSTALL.md#updating) and the [release notes](https://github.com/Niko1221/Strata/releases).
|
||||
|
||||
**Checking what is installed: `strata doctor`.**
|
||||
It is a read-only check of the installed engine parts (it changes nothing): which engine and helper files a saved model
|
||||
needs, and whether they are there. `strata doctor --live` also asks the computer which graphics cards it sees and compares
|
||||
them with the cards the model is set to use; `--output json` prints the same as data. Put its output into a bug report.
|
||||
**A new engine misbehaves after an update.**
|
||||
Stop the model, then use the app's rollback control or `strata components rollback`. Source-checkout engines use a
|
||||
separate rollback workflow, `python setup.py --rollback-engine`, described in [source setup](SOURCE_INSTALL.md).
|
||||
|
||||
**Sending logs for a bug report.**
|
||||
In the app: Settings, Engine, Diagnostics, View logs, then Save logs. It writes one text file,
|
||||
`strata-diagnostics-<date>.txt`, into your Downloads folder and shows where. It holds the engine's messages, the failed
|
||||
steps and the end of each model's server log. In the Terminal, send the output of `strata doctor`. If the Linux app window
|
||||
crashes or will not open, run the AppImage with `--save-logs` in a terminal instead; it writes the crash log and the other
|
||||
logs to one file in Downloads. If the Windows app closes right after you start it, it shows a message with the reason and writes
|
||||
`desktop-crash-<date>.log` in `%LOCALAPPDATA%\Strata\logs`; Save logs includes it, and so can you, by hand. Look through a log before you send it, and never send an API key.
|
||||
**The Mac app cannot be opened.**
|
||||
Check that the download is for Apple Silicon and that macOS is 15 or newer. Follow
|
||||
[Mac installation](MAC.md#the-desktop-app-and-strata-launch-on-a-mac). A warning alone does not establish that a file
|
||||
is safe or intact: verify the download source and release before changing quarantine settings. Release signing and
|
||||
test-build handling are described in [Mac packaging](../packaging/mac/README.md).
|
||||
|
||||
**The Linux AppImage does not open.**
|
||||
Make the downloaded file executable. It may need FUSE 2 (`libfuse2` on Ubuntu 22.04, `libfuse2t64` on newer Ubuntu;
|
||||
package names vary by distribution). Without FUSE, run the file with `--appimage-extract-and-run`. For a crash log,
|
||||
run it with `--save-logs`; see the logs section below.
|
||||
|
||||
## While installing or starting
|
||||
|
||||
**My PC froze, or got very slow, the first time Strata started.**
|
||||
That's normal while it starts, most of all the first time. Strata loads 35-55 GB into your RAM, locks part of it for
|
||||
the graphics card, and works out how much of the model fits on your GPU. The mouse can freeze for a few minutes.
|
||||
**Wait, and don't close the window.** The next starts are much faster. Still frozen after 10 minutes? Restart the
|
||||
PC, close other programs (browsers use a lot of RAM) and try again. If it keeps happening, pick a smaller size (Q2_0
|
||||
or IQ2_XS).
|
||||
**A download stopped.**
|
||||
Retry it in the app, or rerun the same `strata launch` command. Completed model-download steps are reused. Check
|
||||
free space and the error message. For an interrupted Terminal installer, rerun the install command. Only a source
|
||||
checkout uses `START-HERE.bat` / `./setup.sh` to resume source setup.
|
||||
|
||||
**The engine stops while it starts, on a small graphics card or a PC with little RAM. Can I turn the draft layer (MTP) off?**
|
||||
Yes. The draft layer is what makes answers faster, and it takes about 1 GB of graphics memory (plus some RAM and a memory
|
||||
copy that grows with the context). Without it the expert cache gets that memory back and answers are slower. Turn it off
|
||||
for one model: in the app, Models, Expert settings, "Draft layer (MTP)": Off; in the Terminal,
|
||||
`strata launch --model q2_0 --mtp off` (older Terminals: `--set mtp=off --set spec-min-p=off`); by hand, add `"mtp": false` to the
|
||||
model's `strata-<model>.json`. The choice is kept per model and applied on every start.
|
||||
**Startup takes a long time or the computer becomes slow.**
|
||||
Large models need time to check files, prepare packs and fill memory. A first launch can take many minutes; there is
|
||||
no fixed startup time. Watch progress and the engine log. If memory pressure is high, close memory-heavy programs and try a
|
||||
smaller model or context. DeepSeek V4.1-Flash needs 46 GB of RAM at least ([model page](MODELS.md#deepseek-v41-flash));
|
||||
`strata launch --plan --family deepseek41 --model ist-das-2.00bpw` shows what your computer is missing.
|
||||
|
||||
**I changed `strata-<model>.json` by hand and the change came back.**
|
||||
Engine options you add to `args` by hand (control vectors, `--spec`, `--vram-reserve-mib 0`, anything the app does not offer)
|
||||
are kept. The file has two helper keys for this. `user_args` lists your options; every start and every setup run puts them
|
||||
over the ones the app generates, and for the same option your value wins. `args_written` is a copy of `args` as the app last
|
||||
wrote it, so an option you added or changed in `args` is noticed and moved into `user_args` the next time the app or setup
|
||||
touches the file. To undo one, delete it from both `args` and `user_args`. A value you save in the app's Expert settings
|
||||
("GPU memory for other apps") is put in `user_args` for you.
|
||||
**The NVIDIA driver is too old.**
|
||||
Use the driver required by the engine package. CUDA 13 needs driver 580 or newer; older cards use the
|
||||
[CUDA 12 route](OLDER_GPUS.md). After updating the driver and restarting if needed, open the app or rerun
|
||||
`strata launch`.
|
||||
|
||||
Two things the app still chooses itself: the context size, KV type, picture reading and draft layer. Set them in the Models
|
||||
screen, or with `"mtp": false` (draft layer). The app keeps the earlier file as `strata-<model>.json.bak`.
|
||||
**Port 8080 is already in use, or I changed a running model's settings.**
|
||||
Check `strata launch --status`. To replace your existing launch, rerun the intended command with `--restart`;
|
||||
it stops current answers. To run beside it, choose another port, for example `--port 8081`, if memory permits.
|
||||
Do not stop an unrelated program just because it holds the port. `--yes` alone does not restart a running model.
|
||||
|
||||
**The app keeps more graphics memory free than I want (decode is slower than in 0.1.45).**
|
||||
Without a value of yours the app keeps 1536 MiB free (2048 with picture reading) on a card that drives a display, and 512 MiB
|
||||
(700 with pictures) on a card that has no display attached, such as a second GPU with the screen on the integrated one. It
|
||||
reads that from the driver (nvidia-smi `display_active`, or the Windows adapter that has the desktop); when it cannot tell, it
|
||||
assumes a display. Your own value is never raised or lowered: set "GPU memory for other apps" in Expert settings, or
|
||||
`--vram-reserve-mib 0` in `args`. The engine's own warning about low free memory stays; a card that also draws the desktop
|
||||
can freeze or slow to a few tokens a second with a very small reserve.
|
||||
**The engine stops on a small GPU or low-memory computer.**
|
||||
Try a smaller pack or context. The draft layer also uses memory. In the app, set the model's Draft layer to Off;
|
||||
in the Terminal, an explicit example is:
|
||||
|
||||
**It stopped while downloading or installing.**
|
||||
Run `START-HERE.bat` (Linux: `./setup.sh`) again. It continues where it stopped.
|
||||
```sh
|
||||
strata launch --family qwen --model q2_0 --context 32k --mtp off
|
||||
```
|
||||
|
||||
**It says the NVIDIA driver is too old.**
|
||||
Update it (NVIDIA App or [nvidia.com/drivers](https://www.nvidia.com/drivers)), restart the PC, and run
|
||||
`START-HERE.bat` again.
|
||||
Use `--restart` if replacing an active run. Turning MTP off can slow decoding. Keep some VRAM free for the display;
|
||||
setting the reserve to zero is not a general speed fix.
|
||||
|
||||
**It says port 8080 is already in use.**
|
||||
Strata is already running. Look for its window. Or another program uses the port: `START-HERE.bat --port 8081`.
|
||||
Only an address that is really taken says "already in use". Any other reason (Windows keeps the port reserved, or
|
||||
the config's `"host"` is not an address of this PC) is printed as what the OS said, with what to change (#769).
|
||||
**Pictures do not work.**
|
||||
The model, backend and installed engine package must include a compatible picture reader. Inspect the startup
|
||||
error and the model guide. `--vision off` disables reading pictures; it is different from optional
|
||||
[image generation](IMAGES.md). In the Terminal, use `on|off|gpu|cpu`, not source setup's `yes|no` spelling.
|
||||
|
||||
**"Windows blocked the Strata engine (or the image encoder) ... Smart App Control".**
|
||||
On a clean Windows 11, Smart App Control refuses programs that are not signed, and Strata's `strata.exe` and
|
||||
`strata-vision.exe` are not (#735). Turn it off (Windows Security > App & browser control > Smart App Control
|
||||
settings; it cannot be turned back on without a reset of Windows), or, if only the image encoder is blocked, run
|
||||
`START-HERE.bat --setup --vision no`.
|
||||
|
||||
**A new engine misbehaves after an update.**
|
||||
An update keeps the engine it replaced in `engine\.previous` (one generation, about 210 MiB).
|
||||
`python setup.py --rollback-engine` puts it back (and keeps the newer one there: run it again to go forward) (#670).
|
||||
|
||||
**UPDATE.bat / update.sh say "git pull did not succeed", or "has no commit in common" (#1276).**
|
||||
The repository's history was cleaned up on 2026-10-06, so a clone made before that cannot be updated with `git pull`.
|
||||
The 0.1.40.2 scripts handle it: when no tracked file is edited they keep your old commits in the branch
|
||||
`pre-cleanup-backup` and move the clone to the new history; otherwise they print the two commands for you. Your models,
|
||||
settings and engine are untracked files and are never touched. An older UPDATE.bat / update.sh (before 0.1.40.2) cannot
|
||||
do this itself: run these once in the Strata folder, then use UPDATE.bat as usual (add `git stash` first if `git status`
|
||||
lists edited files):
|
||||
|
||||
git fetch origin
|
||||
git branch pre-cleanup-backup
|
||||
git checkout -B main origin/main
|
||||
|
||||
**Python or the build tools could not be installed.**
|
||||
Install what it names (links are printed), then run it again. Everything already done is kept.
|
||||
|
||||
**Linux: the engine does not compile (`unsupported GNU version`, or `exception specification is incompatible` for
|
||||
`cospi`/`sinpi`/`rsqrt`).** Two known mismatches between the CUDA toolkit and a new Linux (#601):
|
||||
- gcc newer than 14 (Ubuntu 26.04's default 15): CUDA 12.x and 13.0 refuse it. Install g++-14 and run
|
||||
`CXX=g++-14 ./setup.sh` (CC, CXX and CUDAHOSTCXX name the build's compilers; setup passes them to CMake itself,
|
||||
so an already-configured build folder picks them up too, #1645).
|
||||
- glibc 2.43 with CUDA 12.9: the toolkit's math headers clash with glibc's, already in CMake's first test compile.
|
||||
Use CUDA 12.8 or 13.x instead. Setup takes the newest toolkit it finds; `STRATA_NVCC=/usr/local/cuda-12.8/bin/nvcc
|
||||
./setup.sh` makes it use that one (only that one).
|
||||
|
||||
**The first start takes minutes.**
|
||||
It is reading 34-55 GB into RAM; the second start is faster while the files are in the OS cache. Started from Task
|
||||
Scheduler, it can be 24x slower: see [Running it at startup](DETAILS.md#running-it-at-startup-task-scheduler).
|
||||
On Linux the engine now asks the kernel to read the model files ahead (a cold start went from ~920 s to 70 s on one
|
||||
PC); `STRATA_READ_AHEAD=0` turns that off. On Linux with transparent huge pages on `always`, the arena no longer asks
|
||||
for `MADV_HUGEPAGE` on top (a fragmented machine spent minutes compacting memory, #771); `STRATA_NO_ARENA_THP=1` skips
|
||||
that request on any setting.
|
||||
|
||||
**Setup says the engine does not match GitHub's checksum.**
|
||||
Setup checks the downloaded ready-made engine against the SHA-256 GitHub publishes. On a mismatch the file is corrupt or was changed on the way, so setup deletes it and stops; run setup again to download it afresh. If GitHub gives no checksum (offline, rate limited, your own mirror), setup warns and installs anyway. `STRATA_SKIP_SHA256=1` skips the check, only if you insist.
|
||||
**Windows blocks an executable.**
|
||||
Verify the official release and the component named in the message. Feed signatures and Windows code-signing are
|
||||
different checks. If only the picture reader is blocked, `--vision off` may let text requests run. Do not disable
|
||||
system-wide protection as the routine installation fix; include the blocked file and exact message in a bug report.
|
||||
|
||||
## While it answers
|
||||
|
||||
**It's very slow and the disk light keeps blinking.**
|
||||
Your PC is out of free RAM. Close other programs, or pick a smaller size (Q2_0 or IQ2_XS).
|
||||
**It is slow and the disk stays busy.**
|
||||
The chosen model may exceed available RAM/VRAM and be using the file tier. Close other programs, check memory in
|
||||
the monitor, and choose a smaller model or context. Compare performance only with the same hardware and workload.
|
||||
|
||||
**An answer stopped with "the engine stopped unexpectedly".**
|
||||
Usually not enough RAM (on Linux the system then stops the engine). Just send your message again: Strata starts the
|
||||
engine by itself. If it keeps happening, close other programs or pick a smaller size.
|
||||
**The prompt exceeds the context.**
|
||||
Start a new chat, shorten the input, or use the app's Summarize and continue action. A larger context needs more
|
||||
memory. In the Terminal, restart your chosen model with an explicit `--context`, such as `64k`; preserve its family,
|
||||
model and other choices. Do not look for `SETUP.bat` in an app install.
|
||||
|
||||
**On Linux the engine stops with "exit code -9", but nothing ran out of RAM.**
|
||||
Check `ulimit -R`. If it prints 0 (or any number), the desktop session sets a real-time CPU limit: the audio system (PipeWire) does that when rtkit is not installed, and every program started from the desktop inherits it. The kernel can then end a program with signal 9 and no message. Install `rtkit`, or start Strata outside the session's limits, for example `systemd-run --user --collect -p LimitRTTIME=infinity -- strata serve ...`. Strata prints a warning at start when it sees this limit.
|
||||
**On Linux the engine exits with code -9.**
|
||||
Check system logs for an out-of-memory kill. Also check `ulimit -R`: a desktop-session real-time limit can cause a
|
||||
kill even when memory is available. The engine prints a warning for that condition; see the
|
||||
[detailed troubleshooting table](DETAILS.md#troubleshooting) before changing system limits.
|
||||
|
||||
**It says the prompt exceeds the context.**
|
||||
The conversation is longer than the context you chose. Start a new chat, or run `SETUP.bat` and pick more
|
||||
context.
|
||||
**A hand-edited engine option disappears.**
|
||||
The run config tracks explicit options in `user_args` and the generated baseline in `args_written`. Prefer the
|
||||
app's Expert settings or Terminal `--set` for supported overrides. Context, KV, picture reading and drafting are
|
||||
launch choices; set them through the app or the corresponding `strata launch` flags. Inspect the effective config
|
||||
rather than assuming a saved preference was applied.
|
||||
|
||||
**It is slower than the tables.**
|
||||
The monitor plugged into the graphics card and other GPU programs take VRAM from the expert cache; RAM running below
|
||||
its rated speed (enable EXPO/XMP in the BIOS) slows the CPU half.
|
||||
## Source-checkout problems
|
||||
|
||||
**An earlier reply that came back empty is gone from the chat history (0.1.40, #843).**
|
||||
The server leaves out finished assistant turns that have no text, so the next reply does not copy the empty ones.
|
||||
`STRATA_KEEP_EMPTY_TURNS=1` in the server's environment renders them as before.
|
||||
Python/venv, CUDA compiler versions, ROCm builds, `UPDATE.bat`, `update.sh` and generated `run-<model>` scripts
|
||||
belong to [SOURCE_INSTALL.md](SOURCE_INSTALL.md). Use the backend guide for build prerequisites. The packaged
|
||||
Terminal carries Python; a missing system Python is not a reason to install a compiler for the normal release.
|
||||
|
||||
**A Turing card (RTX 20) reads prompts above ~90K tokens differently (0.1.40, #743).**
|
||||
The prompt's top-k selection takes a wider kernel there, with the same ids. `STRATA_TOPK_STREAM=0` restores the old one.
|
||||
If `git pull` fails in an old clone, back up your local work and follow the recovery steps the source update script prints.
|
||||
Do not overwrite local branches or edits just to make `git pull` succeed.
|
||||
|
||||
**Decode is several times slower with `--expert-cache N` than with `auto` (#781, #831).**
|
||||
An explicit N is a byte budget that is not checked against the free VRAM once the slots are written. On a card it
|
||||
fills (the log says `0 MiB of VRAM free with everything loaded - LOW`), Windows pages the GPU's memory and decode drops
|
||||
from 100 to 14 tok/s with only a few hundred slots too many. Use `--expert-cache auto`, or a smaller N.
|
||||
## Logs and bug reports
|
||||
|
||||
**A low-RAM PC (about 32 GB) stalls with `--resident-budget-gib N` (#649).**
|
||||
The budget is pinned (locked) in RAM, which leaves the OS and the CPU workers little room. `STRATA_RESIDENT_PIN=0` keeps
|
||||
it pageable; a smaller budget works too.
|
||||
- **App:** Settings → Engine → Diagnostics → View logs → Save logs. The diagnostic file goes to Downloads.
|
||||
- **Terminal:** include `strata --version`, `strata doctor --live`, `strata launch --status`, and the relevant log excerpt.
|
||||
- **Windows app crash:** inspect `%LOCALAPPDATA%\Strata\logs` for `desktop-crash-<date>.log`.
|
||||
- **Linux app crash:** start the AppImage with `--save-logs` to collect diagnostics.
|
||||
|
||||
**Decode got slower in 0.1.40 on a 32 GB PC with `--resident-budget-gib N`, and the drive is read hard (#1194, #1116, #1085).**
|
||||
The start line "the file tier reads unbuffered" means the experts outside the RAM copy are read straight from the drive.
|
||||
0.1.40 chose that whenever the free RAM could not hold all of them; but the same few come back token after token, and
|
||||
a cache a twentieth of their size serves most of the repeats. 0.1.40.2 asks for that much (at least 1.5 GiB beside the
|
||||
4 GiB headroom) and reads through the cache otherwise, which measured 7% to 118% faster decode with a third to a sixth of
|
||||
the drive traffic on a 20 to 32 GB box. On 0.1.40 / 0.1.40.1: `STRATA_UNBUFFERED_LOAD=0` does the same, `=1` forces the
|
||||
unbuffered reads.
|
||||
|
||||
**Linux: the start is OOM-killed while the engine allocates the page-locked expert copy (#1250).**
|
||||
Page-locked pages come from the driver in one go and cannot wait for the kernel to give back file cache, so with the
|
||||
model files' cache in the way (little really free RAM, much "available") the host could run out. 0.1.40.2 gives back the
|
||||
cached pages of the model files it mapped before it page-locks the copy; when the really free RAM still does not cover it,
|
||||
it takes the pages in 1 GiB steps while the RAM available (cgroup limits included) stays above the headroom, page-locks
|
||||
what it has, and keeps the rest resident but pageable with a warning ("only N of M GiB could be page-locked"): the answers
|
||||
are the same, only the copies of the pageable part to the GPU go through the CPU. `STRATA_PIN_GUARD=0` turns this off,
|
||||
`STRATA_PIN_RESERVE_GIB=N` sets the RAM that must stay free (default: `STRATA_RESIDENT_HEADROOM_GIB`, 4 GiB).
|
||||
|
||||
**The Windows display driver resets, then the PC blue-screens (0x141, then 0x116 in `nvlddmkm`) during an answer (#961).**
|
||||
This is the NVIDIA driver, not Strata: the report shows the same wedge with other CUDA programs on that driver (610.88,
|
||||
RTX 4080 SUPER). Try another driver (a Studio one, or an older one) and a lower power limit; if it only happens with
|
||||
`--spec 4`, `--spec 0` avoids the verify window while you wait for a driver fix.
|
||||
|
||||
**An RTX 50 card (a source build with CUDA 13.2) answers with nonsense, or reads prompts wrongly (#892, #968).**
|
||||
CUDA 13.2.0 and 13.2.1's compiler (nvcc 13.2.51) miscompile some of the engine's kernels for sm_120: on our RTX 5070 the IQ2_S and IQ3_S
|
||||
products are wrong (relative error 0.5 to 1.0 in the tests) with 13.2 and right with 13.0. The ready-made engine is built
|
||||
with 13.0. If you compile it yourself, use CUDA 13.0, 13.1 or 13.2.2 (13.2.2, nvcc build 13.2.86, fixes it; an older one can sit next to a newer: `STRATA_NVCC=<path to its nvcc>`);
|
||||
setup warns when it finds 13.2.0 or 13.2.1 for such a card, and takes an older 13.x when one is installed.
|
||||
|
||||
**Pictures are refused, or slow.**
|
||||
"this server was started without the vision encoder": the model was set up for text only (someone chose "No", or the
|
||||
engine has no picture reader) - run setup again with `--vision yes` (or `--vision gpu` / `--vision cpu`), or switch pictures
|
||||
on in the app or on the Pictures step of `strata launch`. Pictures are on by default wherever a model can read them. The
|
||||
reader runs on the CPU under 24 GB of graphics memory and on the card from 24 GB, on NVIDIA and AMD cards. Pictures that take
|
||||
several seconds (about 3 s at 300 image tokens on 8 cores, more with more tokens) are read by the encoder on the CPU;
|
||||
`--vision gpu` (NVIDIA and AMD, ~1.4 GB of VRAM) makes it 0.1-0.5 s. On AMD the card's reader is `strata-vision-hip`; if it
|
||||
cannot start, the server prints "the GPU picture reader did not start (...); pictures are read on the CPU" and goes on with
|
||||
the CPU reader. On a Mac the reader runs on the GPU; `--vision cpu` takes about 50 s per picture there. On Intel Arc, pictures are off by default for now (`--vision yes` turns them on).
|
||||
|
||||
## AMD cards
|
||||
|
||||
**"No AMD GPU found (the amdgpu driver's KFD topology is empty)" on Linux.**
|
||||
The kernel's amdgpu driver is not loaded for the card. Integrated Radeon GPUs are listed as not supported; setup
|
||||
lists every card it found and whether Strata can use it.
|
||||
|
||||
**The engine stops at start with the card's name, its architecture and the build's list.**
|
||||
The engine was compiled for another card (for example after moving the Strata folder to another PC). Run
|
||||
`./setup.sh --setup --backend hip`: it compiles the engine for this card's architecture.
|
||||
|
||||
**On Windows the engine exits with `0xC0000005` in `amdhip64_7.dll` before it prints anything (#654).**
|
||||
Two things in your environment can cause it, and the server now repairs both before it starts the engine (it says so in
|
||||
the log): `HIP_PATH`, `HIP_DEVICE_LIB_PATH` or `LLVM_PATH` naming a ROCm folder that no longer exists (a deleted build),
|
||||
and a `TEMP`/`TMP` folder the engine cannot create files in (AMD's runtime compiles its first kernels through temporary
|
||||
files). If you start `strata.exe` by hand, fix them yourself: `set HIP_PATH=`, and point `TEMP` and `TMP` at a normal
|
||||
folder such as `C:\Temp`.
|
||||
|
||||
**Windows AMD: the driver resets (VIDEO_ENGINE_TIMEOUT_DETECTED / screen flicker / the engine dies mid-answer).**
|
||||
Windows gives a GPU about 2 seconds to answer; when a graphics card does not, it resets the driver (a TDR): the screen
|
||||
flickers, the engine dies, and the Event Viewer shows `VIDEO_ENGINE_TIMEOUT_DETECTED` (#613, with #579 and #541 on
|
||||
Linux). Strata 0.1.39 and newer say "the GPU stopped responding" in the log. It was seen with KV streaming on
|
||||
(`--kv-resident 32768`, which setup turns on from 64K context) while reading a long prompt on a gfx1201 card. Try these
|
||||
in order, one at a time, and tell us what changed:
|
||||
1. **Send us the evidence.** The last ~80 lines of `strata-<model>.log` in the Strata folder (the server window shows
|
||||
the same lines), your AMD driver version (AMD Software > System), your card and Windows version, the Strata
|
||||
version, and the exact steps (context size, how long the prompt was, which app sent it). If Windows wrote a
|
||||
dump, the newest `.dmp` in `C:\Windows\LiveKernelReports\WATCHDOG` (or `C:\Windows\Minidump`) helps; it holds no
|
||||
chats, only the driver state.
|
||||
2. **Find the step that hangs:** add `"STRATA_PF_STEP_SYNC": "1"` to the `"env"` block of `strata-<model>.json`
|
||||
and restart. The prompt path then waits for the GPU after each step and logs any step over 250 ms, so the log names
|
||||
the step. It is slower; use it to diagnose, then take it out.
|
||||
3. **`"STRATA_KV_HOST_DMA": "1"`** in the same `"env"` block: no GPU kernel writes the RAM copy of the K/V during a
|
||||
prompt, it is copied by DMA instead. Same answers.
|
||||
4. **No KV streaming:** run setup again and pick 32K context (or remove `--kv-resident` and its number from
|
||||
`"args"` in the JSON, with a context that fits the VRAM).
|
||||
5. **A smaller prompt chunk:** change `--prefill` in `"args"` (for example `--prefill 512`): each GPU launch then does
|
||||
less work between two checks by Windows. Prompts are read more slowly.
|
||||
6. **Optional, your decision: give Windows more time (`TdrDelay`).** The default is 2 seconds. In the registry,
|
||||
`HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers`, a DWORD `TdrDelay` of `10` (seconds)
|
||||
lets a long GPU job finish before Windows resets the driver; restart the PC. It hides the symptom and does not fix
|
||||
a GPU that has really hung, and a hung GPU then freezes the screen for those 10 seconds. Strata changes no Windows
|
||||
setting itself; to undo it, delete the value.
|
||||
7. **Driver:** use the newest AMD Software release for your card (Adrenalin, not a Windows Update driver) and restart
|
||||
after installing. If it started after a driver update, tell us both versions; going back one version is a fair
|
||||
test.
|
||||
|
||||
**Large pinned host allocations fail on ROCm although RAM is free.**
|
||||
See [AMD_HIP.md](AMD_HIP.md#model-and-serving-configuration): the mapped expert mode avoids the full pinned arena.
|
||||
|
||||
**"verify: timed out at layer N" or "no progress for 60 s" on AMD (Linux), mostly with `--mmap-experts`.**
|
||||
Two reports fixed it with `"GPU_PINNED_MIN_XFER_SIZE": "1048576"` (RX 6800) or `"HSA_USERPTR_FOR_PAGED_MEM": "0"` (two
|
||||
R9700) in the `env` block of the server JSON, then a restart. Experimental, one machine each; on a systemd unit use
|
||||
`MemoryMax`, not `MemoryHigh`. Details: [AMD_HIP.md](AMD_HIP.md#linux-verify-timeouts-while-the-kernel-reclaims-host-memory-experimental-workarounds).
|
||||
|
||||
## Still stuck?
|
||||
|
||||
Look in the [full troubleshooting table](DETAILS.md#troubleshooting), or open an
|
||||
[issue](https://github.com/Niko1221/Strata/issues) and attach the log: in the app, the file Save logs writes (see
|
||||
[above](#the-desktop-app-and-the-terminal)); in the Terminal, the output of `strata doctor`; with the installer,
|
||||
`strata-<model>.log` from the Strata folder. Say which one you use (app, Terminal or the installer), your Strata and
|
||||
engine versions and your card.
|
||||
Include model and quantization, OS, GPU/VRAM, RAM, driver, install route, exact command, expected result and actual
|
||||
error. Remove API keys and private data before sharing. Follow [CONTRIBUTING.md](../CONTRIBUTING.md).
|
||||
+184
-178
@@ -1,60 +1,52 @@
|
||||
# Unsloth UD-Q4_K_XL
|
||||
|
||||
A regular size since MODEL-UPDATE Phase 2 (it was experimental): setup offers it from engine 0.1.32
|
||||
([below](#setup)); the manual workflow after that section works with engine 0.1.31 or newer. First validated on
|
||||
Windows 10, an RTX 5070 (12 GB), 64 GB of RAM and an AVX-512 CPU (2026-10-01); since then on an R9700 (AMD, every
|
||||
expert in RAM) and an RTX A4000 (62 GB, the RAM budget) - quality, speed and images in [QUANTS.md](QUANTS.md).
|
||||
This page is for people who want to run Unsloth's 4-bit Qwen3.8-Flash-Next file (UD-Q4_K_XL) with Strata, and for
|
||||
people who pack it by hand. It also covers UD-IQ4_XS and plain Q4_0 files. For UD-Q5_K_XL, UD-Q6_K_XL and Q8_0, see
|
||||
[UNSLOTH_Q6.md](UNSLOTH_Q6.md). Quality, speed and images across the Unsloth sizes are in [QUANTS.md](QUANTS.md).
|
||||
|
||||
The target is Unsloth's 4-bit quantization of the same model the other packs use:
|
||||
[unsloth/Qwen3.8-Flash-Next-GGUF, `UD-Q4_K_XL`, revision `38bb39e`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q4_K_XL).
|
||||
"UD-Q4_K_XL" is a mix of formats, not Q4_K everywhere: the routed experts are Q4_K (Q5_K in layer 2) for gate/up and
|
||||
Q5_1 (Q8_0 in layers 2, 4, 30, 46, 47) for down; the token embedding, the output head, the attention and shared-expert
|
||||
projections and the PLE key are Q8_0; the 28.8 GB PLE table is IQ4_NL. The routed experts are 71.7 GiB, about twice
|
||||
the 3-bit models'.
|
||||
"UD-Q4_K_XL" is a mix of formats, not Q4_K everywhere:
|
||||
|
||||
## Setup
|
||||
- Routed experts: Q4_K (Q5_K in layer 2) for gate/up, Q5_1 (Q8_0 in layers 2, 4, 30, 46, 47) for down. They are
|
||||
71.7 GiB, about twice the 3-bit models'.
|
||||
- Token embedding, output head, attention and shared-expert projections, and the PLE key: Q8_0.
|
||||
- The 28.8 GB PLE table: IQ4_NL.
|
||||
|
||||
`START-HERE.bat --setup` (Linux: `./setup.sh --setup`) and choose **Qwen3.8-Flash-Next (Unsloth)**; or directly:
|
||||
## Quick start
|
||||
|
||||
Run `strata launch` and choose **Qwen3.8-Flash-Next (Unsloth)**, or:
|
||||
|
||||
```sh
|
||||
START-HERE.bat --setup --family unsloth --model UD-Q4_K_XL
|
||||
strata launch --family unsloth --model UD-Q4_K_XL
|
||||
```
|
||||
|
||||
What setup does differently for this model:
|
||||
## What setup does
|
||||
|
||||
- Every expert in RAM where its 72 GiB of experts and 16 GiB more fit the RAM (~88 GB: no `--resident-budget-gib`;
|
||||
the rule is sergqwer's fork's, MIT), else the RAM budget below. It needs 48 GB of RAM or more (with less it asks,
|
||||
default no; `--model UD-Q4_K_XL --yes` installs it anyway) and engine 0.1.32 or newer (checked before anything is
|
||||
downloaded). NVIDIA or AMD: it ran on an R9700 too (#429's question is gone). One GPU by default in the budget mode
|
||||
(the tested setup); on several, see the next point. No experimental speed projection (not tested with it). Images
|
||||
(#967) are on by default on an NVIDIA card of 16 GB or more and on AMD (CPU encoder) where every expert fits in RAM;
|
||||
run end to end on CUDA and HIP (QUANTS.md).
|
||||
- Several GPUs, with an engine that keeps the resident RAM copy on a layer split (0.1.40 and newer, #642): the RAM
|
||||
budget is kept on the split. Each card caches the experts of its own layers, and the budget holds the hottest of
|
||||
the rest by the whole expert profile (the experts no card holds), so any RAM works as on one card. Setup asks (one
|
||||
GPU stays the default; `--gpus 0,1`, "2" at setup or "y" when a start offers both cards takes the split), and a
|
||||
start on several GPUs keeps `--resident-budget-gib`. Measured on an RTX 3090 (x16) + RTX 3080 Ti (x4), 91 GB of RAM,
|
||||
budget 64 GiB, engine 0.1.41: decode 45 tok/s against 38 on the 3090 alone (the auto split gives the x4 card two
|
||||
layers); 57.6 GiB of experts in RAM, no file reads per request.
|
||||
- Several GPUs with an older engine (#498): when the RAM holds the GGUF files and 24 GB more (~135 GB of RAM) and two or more cards can
|
||||
share it, setup asks (one GPU stays the default; `--gpus 0,1` takes the split). The split runs **without** the RAM budget: all 77 GB of experts are loaded into RAM from the GGUFs at
|
||||
start, the files pass through the OS file cache while they load, and the config gets `"gpu": [0, 1]` and
|
||||
`"layer_split": "auto"`. Measured on 2x RTX 3090 with 165 GiB (#498): decode 31 tok/s on one card with the budget,
|
||||
64-78 tok/s split (55 tok/s at a 128K prompt), with `MemAvailable` never under 68 GiB. With less RAM, `--gpus`
|
||||
keeps one GPU and says so, and `START-HERE.bat --gpus 0,1` on an installed UD-Q4_K_XL stops with the reason
|
||||
(it used to keep the budget, and the engine exited with code 2).
|
||||
- It downloads the four shards below from the pinned revision `38bb39e` (resumable, like the other models), then
|
||||
checks each one's size and SHA-256 against the table below; the check takes a few minutes once and is remembered
|
||||
in the file's finish mark. A file with the wrong hash is deleted, so the next run downloads it again.
|
||||
- It packs with `--compat-bf16` and never writes `experts.bin` (`--low-ram` does not apply).
|
||||
- The RAM budget (below ~88 GB of RAM) is the PC's RAM less 24 GB: `--resident-budget-gib 40` on 64 GB, at most all 71 GiB of experts on
|
||||
a PC that asks for a budget. From a 64K context up, where the KV cache moves to RAM, its size comes out of the budget. Setup's
|
||||
`--resident-budget-gib N` sets another one (a bigger one is kept, with a note on what it risks).
|
||||
- It recommends an 8K context on a GPU under 14 GB (every GB of KV cache is a GB less of cached experts).
|
||||
- **Where the experts live.** Every expert goes into RAM when its 72 GiB plus 16 GiB more fit (about 88 GB of RAM). On a
|
||||
PC with less RAM, or on a unified-memory APU, setup keeps a RAM budget: the hottest experts in RAM, the rest read
|
||||
from the GGUF on the SSD. It needs 48 GB of RAM or more. With less it asks (default no), and
|
||||
`--model UD-Q4_K_XL --yes` installs it anyway.
|
||||
- **The RAM budget** is the PC's RAM less 24 GB (`--resident-budget-gib 40` on 64 GB), at most all 71 GiB of experts. From
|
||||
a 64K context up, where the KV cache moves to RAM, its size comes out of the budget. `--resident-budget-gib N` at
|
||||
setup sets another one; a bigger one is kept, with a note on what it risks.
|
||||
- **Download.** The four shards below from the pinned revision `38bb39e`, resumable. Setup checks each one's size and
|
||||
SHA-256 against the table below once and remembers it in the file's finish mark. A file with the wrong hash is
|
||||
deleted, so the next run downloads it again.
|
||||
- **Packing.** `--compat-bf16`, and never an `experts.bin` (`--low-ram` does not apply).
|
||||
- **Defaults it writes** into the run config's `env`: `STRATA_HC_Q8=1` on CUDA (the GGUF's own Q8_0 hyper-connection
|
||||
projections, closer to the file and faster), and `STRATA_ROUTE_TAIL_SKIP=0` where every expert is in RAM. UD-Q4_K_XL
|
||||
keeps `STRATA_MMQ_KQ` opt-in.
|
||||
- **Context.** It recommends 8K on a GPU under 14 GB, because every GB of KV cache is a GB less of cached experts.
|
||||
- **GPUs.** NVIDIA or AMD. One GPU is the default in the budget mode. With several GPUs, each card caches the experts of
|
||||
its own layers and the budget holds the hottest of the rest by the whole expert profile. Setup asks; `--gpus 0,1`
|
||||
takes the split, and a start on several GPUs keeps `--resident-budget-gib`.
|
||||
- **Pictures.** The reader is the CPU one below 24 GB of VRAM and the GPU one from 24 GB, when the package has a reader.
|
||||
No experimental speed projection is used with this file.
|
||||
|
||||
## The files
|
||||
|
||||
Four shards, 111,334,654,784 bytes (111.3 GB) together. Shard 1 holds only the metadata; layer 11's down projection is
|
||||
Four shards, 111,334,654,784 bytes (111.3 GB) together. Shard 1 holds only the metadata. Layer 11's down projection is
|
||||
in shard 2 and its gate/up in shard 3, which this engine handles (`native_experts.txt` v4).
|
||||
|
||||
| File | Bytes | SHA-256 |
|
||||
@@ -64,23 +56,22 @@ in shard 2 and its gate/up in shard 3, which this engine handles (`native_expert
|
||||
| `Qwen3.8-Flash-Next-UD-Q4_K_XL-00003-of-00004.gguf` | 49,376,141,504 | `56758f40269cad5cd9b0d3d6fbae0f40f6d5be6de49e4ab392dbe83157d9cbd3` |
|
||||
| `Qwen3.8-Flash-Next-UD-Q4_K_XL-00004-of-00004.gguf` | 12,087,983,520 | `753bda48b98ba4f1636134a90a967de1b2d3908a236c026e464777342e53510a` |
|
||||
|
||||
Keep the four files together in one folder under these names (the engine finds shards 2-4 from shard 1's name; a
|
||||
missing shard is an error that names it). Setup checks them itself; by hand, check them before packing
|
||||
(`sha256sum -c`, or `Get-FileHash` on Windows).
|
||||
Hugging Face snapshot symlinks work if you pass the snapshot's file name, not the hash-named blob it points to.
|
||||
Keep the four files together in one folder under these names. The engine finds shards 2 to 4 from shard 1's name, and a
|
||||
missing shard is an error that names it. By hand, check the files before packing (`sha256sum -c`, or `Get-FileHash` on
|
||||
Windows). Hugging Face snapshot symlinks work if you pass the snapshot's file name, not the hash-named blob it points to.
|
||||
|
||||
## What it needs
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Disk | 111.3 GB for the four files, 1.4 GB for the pack, ~6 GB for the MTP draft layer if you have none yet. **No `experts.bin`**: the engine reads the experts from the GGUF files in place. An NVMe SSD matters: the experts that fit neither VRAM nor the RAM budget are read from it for every token. |
|
||||
| RAM | 64 GB measured. The RAM budget (below) holds the most-used experts; the rest come from the SSD through the OS file cache. |
|
||||
| GPU | 12 GB measured (RTX 5070): after the weights, the draft layer and the buffers, the expert cache held 1,280 of the 24,576 experts (3.7 GiB) at 4K context. |
|
||||
| Disk | 111.3 GB for the four files, 1.4 GB for the pack, about 6 GB for the MTP draft layer if you have none yet. There is **no `experts.bin`**: the engine reads the experts from the GGUF files in place. An NVMe SSD matters, because experts that fit neither VRAM nor the RAM budget are read from it for every token. |
|
||||
| RAM | 48 GB minimum; about 88 GB for every expert in RAM. The RAM budget holds the most-used experts and the rest come from the SSD through the OS file cache. |
|
||||
| GPU | 12 GB works: after the weights, the draft layer and the buffers, the expert cache held 1,280 of the 24,576 experts (3.7 GiB) at 4K context on an RTX 5070. |
|
||||
|
||||
## The pack (by hand)
|
||||
|
||||
Build Strata (or install 0.1.31+), then, from the repository root, with gguf-py from the pinned llama.cpp
|
||||
(setup installs it; `STRATA_GGUF_PY` can point to its `gguf-py` folder):
|
||||
Build Strata, then from the repository root, with gguf-py from the pinned llama.cpp (setup installs it;
|
||||
`STRATA_GGUF_PY` can point to its `gguf-py` folder):
|
||||
|
||||
```sh
|
||||
.venv/bin/python tools/iq_pack.py \
|
||||
@@ -88,36 +79,40 @@ Build Strata (or install 0.1.31+), then, from the repository root, with gguf-py
|
||||
--out packs/ud-q4_k_xl --compat-bf16
|
||||
```
|
||||
|
||||
About 30 seconds. It writes `index.txt`, `dense.bin` (1.4 GB), `native_experts.txt`, the tokenizer, `conversions.json`
|
||||
and `compat-bf16.json`. **`--compat-bf16` is required**: this file stores 195 small projections (the hyper-connection
|
||||
up/down matrices, `output_hc_*` and the PLE value) as Q8_0, while the engine reads them as BF16. Without the flag the
|
||||
packer refuses and writes nothing; with it, they are dequantized and rounded to BF16 (nearest-even). Do not add
|
||||
`--experts-bin`: it would write a 77 GB copy of the experts that this mode does not need.
|
||||
It takes about 30 seconds and writes `index.txt`, `dense.bin` (1.4 GB), `native_experts.txt`, the tokenizer,
|
||||
`conversions.json` and `compat-bf16.json`. **`--compat-bf16` is required.** This file stores 195 small projections (the
|
||||
hyper-connection up/down matrices, `output_hc_*` and the PLE value) as Q8_0, while the engine reads them as BF16.
|
||||
Without the flag the packer refuses and writes nothing. With it, they are dequantized and rounded to BF16
|
||||
(nearest-even). Do not add `--experts-bin`: it would write a 77 GB copy of experts that this mode does not need.
|
||||
|
||||
## The numbers: what is native and what is converted
|
||||
|
||||
The routed experts, the token embedding, the output head, the attention and shared-expert projections, the PLE key and
|
||||
the PLE table are used as the GGUF stores them (the engine has GPU kernels and CPU (ggml) paths for Q4_K, Q5_K, Q5_1 and
|
||||
Q8_0). The small tensors the engine reads as floats are converted, and `conversions.json` lists every one with its
|
||||
source/destination type, method, whether it is exact, the largest absolute error and the source bytes' SHA-256:
|
||||
264 F32 tensors (routers, injections, SSM gates) whose values are already BF16's are stored as BF16 exactly; the 195
|
||||
Q8_0 projections above are rounded to BF16 (largest absolute error 0.0144, in `hc_ffn_up`); the PLE convolution (F32)
|
||||
is narrowed to F16 when the engine loads it (largest error 3e-8). That is the compatibility Unsloth's file needs; it is
|
||||
not the original BF16 checkpoint. The output matches llama.cpp's on the same file:
|
||||
[Quality](#quality-against-llamacpp-on-the-same-file).
|
||||
the PLE table are used as the GGUF stores them. The engine has GPU kernels and CPU (ggml) paths for Q4_K, Q5_K, Q5_1 and
|
||||
Q8_0.
|
||||
|
||||
The small tensors the engine reads as floats are converted. `conversions.json` lists every one with its source and
|
||||
destination type, method, whether it is exact, the largest absolute error and the source bytes' SHA-256:
|
||||
|
||||
- 264 F32 tensors (routers, injections, SSM gates) whose values are already BF16's are stored as BF16 exactly.
|
||||
- The 195 Q8_0 projections above are rounded to BF16 (largest absolute error 0.0144, in `hc_ffn_up`).
|
||||
- The PLE convolution (F32) is narrowed to F16 when the engine loads it (largest error 3e-8).
|
||||
|
||||
That is the compatibility Unsloth's file needs. It is not the original BF16 checkpoint. The output matches llama.cpp's on
|
||||
the same file ([Quality](#quality-against-llamacpp-on-the-same-file)).
|
||||
|
||||
## The MTP draft layer
|
||||
|
||||
The draft layer is the base model's, the same one the other packs use. If setup installed Strata, you have it
|
||||
(`mtp/rt` in `Strata-data`). Otherwise build it as [docs/ORCA.md](ORCA.md#preparation) shows (`tools/mtp_fetch.py`,
|
||||
The draft layer is the base model's, the same one the other packs use. If setup installed Strata, you have it (`mtp/rt`
|
||||
in `Strata-data`). Otherwise build it as [ORCA.md](ORCA.md#preparation) shows (`tools/mtp_fetch.py`,
|
||||
`tools/mtp_pack.py`, `tools/mtp_rt.py`, then copy `data/draft_vocab.bin` to `mtp/rt/`).
|
||||
|
||||
## The server
|
||||
|
||||
Save as `strata-ud-q4_k_xl.json` at the repository root (or next to setup's other `strata-*.json`), with `/path/to/`
|
||||
pointing at **shard 1**. `--ple-gguf` is not needed: the engine finds the shard that holds the PLE table
|
||||
(`per_layer_token_embd.weight`, shard 2) by name. `--resident-budget-gib` turns on the mapped mode with a RAM budget
|
||||
(it implies `--mmap-experts`).
|
||||
(`per_layer_token_embd.weight`, shard 2) by name. `--resident-budget-gib` turns on the mapped mode with a RAM budget (it
|
||||
implies `--mmap-experts`).
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -143,61 +138,35 @@ pointing at **shard 1**. `--ple-gguf` is not needed: the engine finds the shard
|
||||
.venv/bin/python -m serve.server --engine strata --config strata-ud-q4_k_xl.json --port 8080
|
||||
```
|
||||
|
||||
(`engine/strata.exe` is setup's engine on Windows; a self-built one is `build/strata`.) The expert profile is the base
|
||||
model's (`data/expert-profile.bin`); it ranks the same 48 x 512 experts and decides which ones the GPU cache and the RAM
|
||||
`engine/strata.exe` is setup's engine on Windows; a self-built one is `build/strata`. The expert profile is the base
|
||||
model's (`data/expert-profile.bin`). It ranks the same 48 x 512 experts and decides which ones the GPU cache and the RAM
|
||||
budget take first.
|
||||
|
||||
**Choosing the RAM budget N:** your RAM minus 20-24 GB (the OS, the engine itself, the 126 MB router copy, and room for
|
||||
the OS file cache that serves the rest). On 64 GB, 40. The engine checks available RAM and, on Windows, available
|
||||
commit capacity. If N exceeds the smaller value minus 4 GiB headroom, it reduces N to that amount minus a 256 MiB
|
||||
margin and reports which limit applies. Before #403's fix a clamped budget could then fail the safety check.
|
||||
Everything above N comes from the SSD for every token, so N is the setting that matters
|
||||
most; a bigger budget was faster in every measurement (24 / 32 / 40 GiB). When the driver page-locks the whole budget
|
||||
(24 GiB did on this PC, 32 and 40 did not), the GPU also computes a share of the misses over PCIe, as in the resident
|
||||
low-RAM mode; `STRATA_RESIDENT_PIN=0` uses ordinary pageable memory (the CPU then computes every miss).
|
||||
**Choosing the RAM budget N.** Use your RAM minus 20 to 24 GB (the OS, the engine itself, the 126 MB router copy, and
|
||||
room for the OS file cache that serves the rest). On 64 GB that is 40. The engine checks available RAM and, on Windows,
|
||||
available commit capacity. If N exceeds the smaller value minus 4 GiB headroom, it reduces N to that amount minus a
|
||||
256 MiB margin and reports which limit applies. Everything above N comes from the SSD for every token, so N is the
|
||||
setting that matters most; a bigger budget was faster in every measurement (24 / 32 / 40 GiB).
|
||||
|
||||
On Windows, the allocation log reports available physical RAM and commit capacity before allocation and after a
|
||||
failure. Free RAM alone does not guarantee that Windows can commit the requested memory. If the pageable fallback
|
||||
also fails, the warning preserves both the GPU allocation or mapping error and the `malloc` error.
|
||||
When the driver page-locks the whole budget, the GPU also computes a share of the misses over PCIe, as in the resident
|
||||
low-RAM mode. `STRATA_RESIDENT_PIN=0` uses ordinary pageable memory instead, and the CPU then computes every miss. On
|
||||
Windows the allocation log reports available physical RAM and commit capacity before allocation and after a failure,
|
||||
because free RAM alone does not guarantee that Windows can commit the requested memory.
|
||||
|
||||
**Context:** measured at 4K. The KV cache takes VRAM from the expert cache, so a longer context makes decoding slower;
|
||||
8K is a reasonable start on 12 GB. `--kv int8` halves the KV cache's VRAM.
|
||||
|
||||
## What to expect (RTX 5070 12 GB, 64 GB RAM, Windows, one short greedy prompt)
|
||||
|
||||
| RAM budget | Greedy output | Notes |
|
||||
| --- | --- | --- |
|
||||
| 24 GiB | 5.1-5.2 tok/s (2 runs, before the routing prefetch below) | 2.2 GB read from the SSD per verify round (~3.5 tokens) |
|
||||
| 40 GiB | **7-8.5 tok/s** (6 runs: 6.7-8.6, mean 7.9) | 1.1 GB read from the SSD per verify round, ~0.33 GB per token |
|
||||
|
||||
- The first answer comes about 60 s after the engine starts (mostly loading: the RAM budget is copied from the GGUF
|
||||
files at start). A 30-token prompt then took 5-6 s.
|
||||
- Long prompts (engine 0.1.32): a 16K prompt is read at **160 tokens/s** (15,873 tokens in 99 s; 0.1.31: 57 tokens/s,
|
||||
273-282 s). Reading a prompt streams every expert once per 4K chunk, and the third of them outside the RAM budget
|
||||
come from the SSD (~35 GB per chunk), so the prompt path now reads them with 32 threads and up to 128 blobs in
|
||||
flight when the experts are read from the GGUF in place (`STRATA_STAGER_THREADS` / `STRATA_STAGER_RING` override
|
||||
it; 4 and 16 before, still the default for every other model). Between the chunks, the 28.8 GB PLE table's rows
|
||||
take ~5 s per chunk.
|
||||
- With `STRATA_MMQ_KQ=1` the engine multiplies the Q4_K / Q5_K / Q5_1 experts of a prompt with llama.cpp's MMQ
|
||||
kernels (as the other models' formats always are) instead of dequantizing them to FP16: 4x less GPU time for those
|
||||
products, but on this PC the prompt waits for the SSD either way (29 s per 4K chunk with or without). Opt-in: it
|
||||
rounds differently from the FP16 path. On CUDA these kernels are a library next to the engine (`libstrata_kq.so` /
|
||||
`strata_kq.dll`), opened only then, so they take no VRAM from other models ([QUANTS.md](QUANTS.md)); before, they were
|
||||
the build option `-DSTRATA_MMQ_KQUANTS=ON` and took 1-3 expert slots from every model.
|
||||
- The speed varies from run to run (6.7-8.6 tok/s at 40 GiB for the same prompt), with what the OS file cache holds.
|
||||
- For comparison, IQ3_S (all its experts in RAM) writes ~53 tok/s on the same PC ([the speed tables](DETAILS.md#speed-measured)).
|
||||
- Where the time goes (`--stats` on the command line): at 40 GiB about 550-750 ms of each verify round (3.5 tokens) is
|
||||
reading experts from the SSD, ~75 ms the CPU's expert kernels, ~13 ms the GPU.
|
||||
**Context.** The KV cache takes VRAM from the expert cache, so a longer context makes decoding slower. 8K is a reasonable
|
||||
start on 12 GB. `--kv int8` halves the KV cache's VRAM.
|
||||
|
||||
## Quality: against llama.cpp on the same file
|
||||
|
||||
Strata and llama.cpp (the pinned commit `3cf0325`, a CPU build reading the GGUF memory-mapped) were given the same
|
||||
token sequences, and at every position each wrote its 20 most likely next tokens with their log-probabilities: Strata
|
||||
through its verify windows (`STRATA_LOGPOS` with `STRATA_LOGPOS_TOPK=20`), llama.cpp from `llama_get_logits_ith` over
|
||||
one batch. Strata ran with the recommended settings (RAM budget 40 GiB, the MTP draft layer on, greedy,
|
||||
`--adapt-every 100000` so the expert cache does not move during the run). Per position: whether the most likely token is the same (argmax
|
||||
agreement), how many of the 5 / 10 most likely tokens both have, the KL divergence (llama.cpp's distribution against
|
||||
Strata's, over llama.cpp's top 20 plus one bucket for the rest), and the perplexity of the sequence under each.
|
||||
> **Measured on:** Strata 0.1.32 era engine against llama.cpp pinned commit `3cf0325` (CPU build, GGUF memory-mapped).
|
||||
> Strata ran with RAM budget 40 GiB, the MTP draft layer on, greedy and `--adapt-every 100000`. Hardware is not recorded
|
||||
> on this page.
|
||||
|
||||
Both were given the same token sequences, and at every position each wrote its 20 most likely next tokens with their
|
||||
log-probabilities. Strata used its verify windows (`STRATA_LOGPOS` with `STRATA_LOGPOS_TOPK=20`), llama.cpp used
|
||||
`llama_get_logits_ith` over one batch. Per position: whether the most likely token is the same (argmax agreement), how
|
||||
many of the 5 and 10 most likely tokens both have, the KL divergence (llama.cpp's distribution against Strata's, over
|
||||
llama.cpp's top 20 plus one bucket for the rest), and the perplexity of the sequence under each.
|
||||
|
||||
| Token set | Positions | Argmax agreement | Top-5 / top-10 overlap | KL | Perplexity Strata / llama.cpp |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
@@ -207,42 +176,44 @@ Strata's, over llama.cpp's top 20 plus one bucket for the rest), and the perplex
|
||||
| Real text after a 16K prompt: the last 510 tokens of 16,384 tokens of this repository's docs and code | 510 | 91.2% | 88.3% / 89.9% | 0.035 | 4.27 / 4.29 |
|
||||
| The same 511 positions with only 512 tokens before them | 511 | 89.2% | 86.0% / 86.3% | 0.061 | 8.99 / 8.87 |
|
||||
|
||||
For comparison, [eddoursul/Strata](https://github.com/eddoursul/Strata) reported 98.2-99.3% (code), 95.6-95.8%
|
||||
For comparison, [eddoursul/Strata](https://github.com/eddoursul/Strata) reported 98.2 to 99.3% (code), 95.6 to 95.8%
|
||||
(thinking) and 97.7% (after a 16K prompt) for the same file against llama.cpp on an RTX 3090.
|
||||
|
||||
**Where they differ:** at near-ties. Grouped by llama.cpp's own gap between its two most likely tokens, Strata picks
|
||||
the same token at every position of the code and thinking answers where that gap is 0.2 or more (695 of 695) and at
|
||||
every position after the 16K prompt where it is 0.5 or more (144 of 144 in the continuation, 254 of 254 in the real
|
||||
text; 50 of 54 and 117 of 121 between 0.2 and 0.5). Text after a long document has many more near-ties than a short
|
||||
answer (58 of the continuation's 256 positions and 135 of the real text's 510 have a gap under 0.2, against 13 of the
|
||||
code answer's 400), which is why its agreement is lower. It is not the long context itself: the same real-text
|
||||
positions after only 512 tokens agree a little less (89.2%), and the perplexities match within 3% in every set. For
|
||||
scale, Strata's own greedy run and its teacher-forced rerun of the continuation pick different tokens at 6% of the
|
||||
positions (93.8% the same: other window sizes, other experts in VRAM), the same kind of near-tie flips. The 16K
|
||||
prompts were read by Strata's batched prompt path (15,872 tokens), the rest through the verify windows. These numbers
|
||||
are the FP16 prompt path's (the released engine's); with the MMQ prompt path (`-DSTRATA_MMQ_KQUANTS=ON`, above) the
|
||||
The two differ at near-ties. Where llama.cpp's own gap between its two most likely tokens is 0.2 or more, Strata picks
|
||||
the same token at every position of the code and thinking answers (695 of 695). After the 16K prompt it does so at every
|
||||
position where the gap is 0.5 or more (144 of 144 in the continuation, 254 of 254 in the real text). Text after a long
|
||||
document has many more near-ties than a short answer (58 of the continuation's 256 positions and 135 of the real text's
|
||||
510 have a gap under 0.2, against 13 of the code answer's 400), which is why its agreement is lower. The perplexities
|
||||
match within 3% in every set. The 16K prompts were read by Strata's batched prompt path (15,872 tokens), the rest
|
||||
through the verify windows. These numbers are the FP16 prompt path's. With the MMQ prompt path (`STRATA_MMQ_KQ=1`) the
|
||||
continuation after the 16K prompt agrees at 92.2% (KL 0.029, three runs, identical).
|
||||
|
||||
To repeat it: Strata's side is `STRATA_LOGPOS=<file>` with `STRATA_LOGPOS_TOPK=20` (engine 0.1.32) on a serve
|
||||
engine with `--short-read` covering the positions to compare; llama.cpp's side was a short program over
|
||||
`llama_decode` / `llama_get_logits_ith` writing the same top 20 per position.
|
||||
To repeat it, set `STRATA_LOGPOS=<file>` with `STRATA_LOGPOS_TOPK=20` on a serve engine with `--short-read` covering the
|
||||
positions to compare. llama.cpp's side was a short program over `llama_decode` and `llama_get_logits_ith` writing the
|
||||
same top 20 per position.
|
||||
|
||||
## Opt-ins and switches (environment variables)
|
||||
|
||||
| | |
|
||||
| Variable | Effect |
|
||||
| --- | --- |
|
||||
| `STRATA_LOOKAHEAD=0` | Turns off the routing-aware prefetch (on by default in this mode): while the CPU works on a layer, a thread applies the next layer's router to this layer's input and asks the OS to read the predicted experts' pages. Pages only, the answers are the same. About half of the SSD reads were predicted; mean +14% (6.9 -> 7.9 tok/s). `STRATA_LOOKAHEAD_K` sets the experts per token (default 10). |
|
||||
| `STRATA_LOOKAHEAD=0` | Turns off the routing-aware prefetch (on by default in this mode). While the CPU works on a layer, a thread applies the next layer's router to this layer's input and asks the OS to read the predicted experts' pages. Pages only; the answers are the same. `STRATA_LOOKAHEAD_K` sets the experts per token (default 10). |
|
||||
| `STRATA_MMQ_KQ=1` | Multiplies the Q4_K / Q5_K / Q5_1 experts of a prompt with llama.cpp's MMQ kernels instead of dequantizing them to FP16. It rounds differently from the FP16 path, so it is opt-in for this file. On CUDA the kernels are a library next to the engine (`libstrata_kq.so` / `strata_kq.dll`), opened only then, so they take no VRAM from other models ([QUANTS.md](QUANTS.md)). |
|
||||
| `STRATA_KQ256=1` | Multi-token AVX2 kernels for the Q4_K / Q5_1 / Q8_0 experts. Bit-exact with ggml's, but measured no faster, so off. |
|
||||
| `STRATA_PARTIAL_PIN=1` | Registers the hottest part of the RAM budget (up to `STRATA_PARTIAL_PIN_GIB`, default 24) with the GPU driver, so the GPU computes a share of the misses over PCIe (`--pcie-frac`). Measured no faster on this PC, and it changes the numerics of those experts (GPU kernels instead of the CPU's), so off. |
|
||||
| `STRATA_FETCH_THREADS=N` | Threads that read the experts from the GGUF while it answers (default 8; 16 was no faster). The prompt path has its own: `STRATA_STAGER_THREADS` (default 32 here) and `STRATA_STAGER_RING` (128). |
|
||||
| `STRATA_PARTIAL_PIN=1` | Registers the hottest part of the RAM budget (up to `STRATA_PARTIAL_PIN_GIB`, default 24) with the GPU driver, so the GPU computes a share of the misses over PCIe (`--pcie-frac`). Measured no faster, and it changes the numerics of those experts, so off. |
|
||||
| `STRATA_FETCH_THREADS=N` | Threads that read the experts from the GGUF while it answers (default 8). The prompt path has its own: `STRATA_STAGER_THREADS` (default 32 here) and `STRATA_STAGER_RING` (128). |
|
||||
| `STRATA_RESIDENT_PIN=0` | Ordinary pageable memory for the RAM budget (see above). |
|
||||
|
||||
## UD-IQ4_XS (setup from 0.1.39, #621)
|
||||
## UD-IQ4_XS
|
||||
|
||||
Unsloth's `UD-IQ4_XS` at the same revision is in setup too, as a regular choice (not experimental; the Unsloth
|
||||
family's first size and its default): `START-HERE.bat --setup --family unsloth --model UD-IQ4_XS` (engine 0.1.38 or
|
||||
newer). It sits between IQ3_S and UD-Q4_K_XL: its routed experts are IQ3_S gate/up (IQ4_XS
|
||||
in one layer) with IQ4_NL downs (Q8_0 in five layers), 59.5 GB of them; the dense side (Q8_0 projections, the IQ4_NL
|
||||
PLE table, a Q6_K head) is UD-Q4_K_XL's. Three shards, 93.7 GB:
|
||||
Unsloth's `UD-IQ4_XS` at the same revision is in setup as a regular choice, the Unsloth family's first size:
|
||||
|
||||
```sh
|
||||
strata launch --family unsloth --model UD-IQ4_XS
|
||||
```
|
||||
|
||||
It sits between IQ3_S and UD-Q4_K_XL. Its routed experts are IQ3_S gate/up (IQ4_XS in one layer) with IQ4_NL downs (Q8_0
|
||||
in five layers), 59.5 GB of them. The dense side (Q8_0 projections, the IQ4_NL PLE table, a Q6_K head) is UD-Q4_K_XL's.
|
||||
Three shards, 93.7 GB:
|
||||
|
||||
| File | Bytes | SHA-256 |
|
||||
| --- | ---: | --- |
|
||||
@@ -250,37 +221,38 @@ PLE table, a Q6_K head) is UD-Q4_K_XL's. Three shards, 93.7 GB:
|
||||
| `Qwen3.8-Flash-Next-UD-IQ4_XS-00002-of-00003.gguf` | 49,835,229,856 | `577a38a2392b40ca2193cea502e1d92f60b8cd370675d308e0ec21885d9daaa7` |
|
||||
| `Qwen3.8-Flash-Next-UD-IQ4_XS-00003-of-00003.gguf` | 43,836,407,744 | `d4634e6d84f0ebb0940be15c90d3790bf6464e3dea3a1cddc567dc0e83ad8833` |
|
||||
|
||||
Setup treats it like UD-Q4_K_XL (the list above): the same RAM budget (your RAM less 24 GB; at most all 55 GiB of its
|
||||
experts, so a PC with ~80 GB of RAM or more holds all of them), the pack with `--compat-bf16`, no `experts.bin`, one
|
||||
GPU by default (several keep the budget, as above). It does not ask on AMD: its experts' formats have prompt kernels there too. Images are an option (asked,
|
||||
off by default; NVIDIA): the image path has no restriction for this pack, which uses the original model's image
|
||||
encoder; not yet run with images.
|
||||
Setup treats it like UD-Q4_K_XL: the same RAM budget rule (your RAM less 24 GB, at most all 55 GiB of its experts, so a
|
||||
PC with about 80 GB of RAM or more holds all of them), the pack with `--compat-bf16`, no `experts.bin`, and one GPU by
|
||||
default. It uses the original model's image encoder.
|
||||
|
||||
What is known so far: it packs and runs on a Strix Halo (AMD gfx1151, 128 GB unified memory), where prompts read at
|
||||
~820 tokens/s at 8K and 64K and the answers passed the retrieval checks at 8K and 64K. It has **not** been measured on
|
||||
NVIDIA yet; please report what you see. One fidelity note that applies to every `--compat-bf16` pack (UD-Q4_K_XL too):
|
||||
the pack rounds the file's Q8_0 hyper-connection projections to BF16, and on the Strix Halo that rounding put
|
||||
perplexity 6-9% above llama.cpp's on the same file (teacher-forced, short context). Reading the Q8_0 values instead
|
||||
closes most of that gap; it is measured on AMD only and not in this release.
|
||||
It packs and runs on a Strix Halo (AMD gfx1151, 128 GB unified memory), where prompts read at about 820 tokens/s at 8K
|
||||
and 64K and the answers passed the retrieval checks at 8K and 64K. It has not been measured on NVIDIA; please report
|
||||
what you see.
|
||||
|
||||
One fidelity note applies to every `--compat-bf16` pack, UD-Q4_K_XL included. The pack rounds the file's Q8_0
|
||||
hyper-connection projections to BF16, and on the Strix Halo that put perplexity 6 to 9% above llama.cpp's on the same
|
||||
file (teacher-forced, short context). `STRATA_HC_Q8=1`, which setup writes on CUDA for the UD files, reads the file's Q8_0
|
||||
values instead of the rounded BF16. The 6 to 9% gap was measured on AMD, where setup does not write it.
|
||||
|
||||
## Q4_0 files
|
||||
|
||||
A plain `llama-quantize` Q4_0 file of this model runs the same way: pack shard 1 with `--compat-bf16` as above and
|
||||
point `--native` at it. Tested:
|
||||
A plain `llama-quantize` Q4_0 file of this model runs the same way: pack shard 1 with `--compat-bf16` as above and point
|
||||
`--native` at it. Tested:
|
||||
[bartowski/Qwen3.8-Flash-Next-GGUF, `Qwen3.8-Flash-Next-Q4_0`, revision `928589f`](https://huggingface.co/bartowski/Qwen3.8-Flash-Next-GGUF/tree/928589fdb66c6ff07f22ac561e3fbce76553548f/Qwen3.8-Flash-Next-Q4_0)
|
||||
(3 shards, 100.6 GB). Its routed experts are Q4_0 for gate/up and for down, except down in layers 0-5, which is Q4_1;
|
||||
the PLE table and the token embedding are Q4_0. The pack serves 302 tensors from the GGUF as stored and converts 362
|
||||
(3 shards, 100.6 GB). Its routed experts are Q4_0 for gate/up and for down, except down in layers 0 to 5, which is Q4_1.
|
||||
The PLE table and the token embedding are Q4_0. The pack serves 302 tensors from the GGUF as stored and converts 362
|
||||
(168 exact, 194 rounded to BF16, largest absolute error 0.0156; 1.23 GiB).
|
||||
|
||||
The Q4_0 expert kernels centre the codes (q - 8) before the integer dot product, as ggml-cpu's Q4_0 x Q8_0 dot does,
|
||||
instead of llama.cpp's CUDA form (uncentred codes, minus 8 x the Q8_1 block sum in float); Q4_1 follows the Q5_1
|
||||
kernel without the fifth bit. `native_expert_parity` on this file (layers 0 and 5: Q4_0/Q4_1, layers 20 and 47:
|
||||
Q4_0/Q4_0): relative error against the float product 1.1-1.3e-2 on the CPU and the GPU alike, CPU vs GPU 1.1e-7 for
|
||||
Q4_0/Q4_0 and 5e-4 to 1.1e-3 with Q4_1 down.
|
||||
instead of llama.cpp's CUDA form (uncentred codes, minus 8 x the Q8_1 block sum in float). Q4_1 follows the Q5_1 kernel
|
||||
without the fifth bit. `native_expert_parity` on this file (layers 0 and 5: Q4_0/Q4_1, layers 20 and 47: Q4_0/Q4_0)
|
||||
gives a relative error against the float product of 1.1 to 1.3e-2 on the CPU and the GPU alike, and CPU against GPU of
|
||||
1.1e-7 for Q4_0/Q4_0 and 5e-4 to 1.1e-3 with Q4_1 down.
|
||||
|
||||
Measured on one RTX 3060 12 GB (PCIe 3.0 x16), Xeon E5-2696 v4, 121 GiB DDR4-2400, Linux, CUDA 12.4; every expert in
|
||||
RAM, `--max-context 131072 --kv int8 --kv-resident 20480`, expert cache `auto` (1,960 slots, 5.06 GiB), prompt chunk
|
||||
`auto` (6,144 tokens); greedy, `reasoning_effort` medium, one request at a time:
|
||||
> **Measured on:** one RTX 3060 12 GB (PCIe 3.0 x16), Xeon E5-2696 v4, 121 GiB DDR4-2400, Linux, CUDA 12.4. Every
|
||||
> expert in RAM, `--max-context 131072 --kv int8 --kv-resident 20480`, expert cache `auto` (1,960 slots, 5.06 GiB),
|
||||
> prompt chunk `auto` (6,144 tokens), greedy, `reasoning_effort` medium, one request at a time. Strata version not
|
||||
> recorded.
|
||||
|
||||
| Prompt | Prompt read | Output |
|
||||
| --- | --- | --- |
|
||||
@@ -289,28 +261,62 @@ RAM, `--max-context 131072 --kv int8 --kv-resident 20480`, expert cache `auto` (
|
||||
| 24,461 tokens | 787 tok/s | 24.0 tok/s |
|
||||
| 67,934 tokens (a fact hidden in the middle: found) | 737 tok/s | 26.4 tok/s |
|
||||
|
||||
A coding prompt was answered correctly. Not measured: agreement with llama.cpp on this file (as
|
||||
[above](#quality-against-llamacpp-on-the-same-file) for UD-Q4_K_XL), AMD, Windows, and bartowski's Swift-1.5 Q4_0,
|
||||
whose header shows the same formats.
|
||||
A coding prompt was answered correctly. Not measured: agreement with llama.cpp on this file, AMD, Windows, and
|
||||
bartowski's Swift-1.5 Q4_0, whose header shows the same formats.
|
||||
|
||||
## Scope and validation
|
||||
|
||||
- UD-Q4_K_XL and UD-IQ4_XS at revision `38bb39e` are targeted, and plain Q4_0 files ([above](#q4_0-files)). Other
|
||||
quantizations use formats this engine may not have kernels for; the engine checks every layer's formats at start
|
||||
and refuses an unsupported one by name.
|
||||
- Tests: the packer's synthetic 4-shard and conversion tests (`.venv/bin/python -m unittest discover -s tools -p
|
||||
test_iq_pack.py`); CTests `gguf_split_test`, `expert_layout_test`, `native_expert_parity_*` (the three real expert
|
||||
format pairs against ggml-cpu, the Q5_1 min term, Q8_0 rows, Q4_0/Q4_0 and Q4_0/Q4_1), `prefill_mmq_kquant_test` (with `-DSTRATA_MMQ_KQUANTS=ON`: the
|
||||
prompt path's MMQ products for Q4_K / Q5_K / Q5_1 / Q8_0 against ggml's dequantized weights); the in-place mode against `experts.bin` on the Coder
|
||||
(identical tokens and logits).
|
||||
- Real runs: greedy answers to a coding prompt (correct) at every budget and setting above, identical across them;
|
||||
the same tokens as llama.cpp at 97.5-99% of the positions of short greedy answers and 90-91% after a 16K prompt,
|
||||
differing mostly at near-ties (above). Not yet: sampled decoding, long conversations. Please report what you see.
|
||||
- UD-Q4_K_XL and UD-IQ4_XS at revision `38bb39e` are targeted, and plain Q4_0 files. Other quantizations use formats this
|
||||
engine may not have kernels for. The engine checks every layer's formats at start and refuses an unsupported one by
|
||||
name.
|
||||
- Tests: the packer's synthetic 4-shard and conversion tests
|
||||
(`.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`) and the CTests `gguf_split_test`,
|
||||
`expert_layout_test`, `native_expert_parity_*` (the three real expert format pairs against ggml-cpu, the Q5_1 min term,
|
||||
Q8_0 rows, Q4_0/Q4_0 and Q4_0/Q4_1). With `-DSTRATA_MMQ_KQUANTS=ON`, `prefill_mmq_kquant_test` checks the prompt path's
|
||||
MMQ products for Q4_K / Q5_K / Q5_1 / Q8_0 against ggml's dequantized weights.
|
||||
- Not validated: sampled decoding and long conversations. Please report what you see.
|
||||
|
||||
## Credits
|
||||
|
||||
The split-artifact loader, the expert formats' GPU kernels and the packing of Unsloth's files follow
|
||||
[eddoursul/Strata](https://github.com/eddoursul/Strata) (PR #245), which ran these files first. The Q8_0 kernels, the
|
||||
per-role shard column and the PLE convolution's F32 -> F16 fix come from @gopinath87607's PR #255; per-role shard names
|
||||
per-role shard column and the PLE convolution's F32 to F16 fix come from @gopinath87607's PR #255. Per-role shard names
|
||||
were also proposed by @jagsan-cyber in PR #247. The quantization is [Unsloth](https://huggingface.co/unsloth)'s; the
|
||||
model is Qwen's ([license](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)).
|
||||
|
||||
## History / older measurements
|
||||
|
||||
### RAM budget, speed on a 12 GB card
|
||||
|
||||
> **Measured on:** RTX 5070 12 GB, 64 GB RAM, Windows 10, AVX-512 CPU, UD-Q4_K_XL, one short greedy prompt, first
|
||||
> validation (2026-10-01, engine 0.1.31/0.1.32).
|
||||
|
||||
| RAM budget | Greedy output | Notes |
|
||||
| --- | --- | --- |
|
||||
| 24 GiB | 5.1 to 5.2 tok/s (2 runs, before the routing prefetch) | 2.2 GB read from the SSD per verify round (about 3.5 tokens) |
|
||||
| 40 GiB | 7 to 8.5 tok/s (6 runs: 6.7 to 8.6, mean 7.9) | 1.1 GB read from the SSD per verify round, about 0.33 GB per token |
|
||||
|
||||
- The first answer comes about 60 s after the engine starts (the RAM budget is copied from the GGUF files at start). A
|
||||
30-token prompt then took 5 to 6 s.
|
||||
- A 16K prompt was read at 160 tokens/s (15,873 tokens in 99 s) on engine 0.1.32, up from 57 tokens/s (273 to 282 s) on
|
||||
0.1.31, after the prompt path began reading experts with 32 threads and up to 128 blobs in flight.
|
||||
- The routing-aware prefetch added a mean of +14% (6.9 to 7.9 tok/s); about half of the SSD reads were predicted.
|
||||
- With `STRATA_MMQ_KQ=1` the products took 4x less GPU time, but on this PC the prompt waited for the SSD either way (29 s
|
||||
per 4K chunk).
|
||||
- Where the time goes at 40 GiB (`--stats`): about 550 to 750 ms of each verify round (3.5 tokens) reading experts from
|
||||
the SSD, about 75 ms the CPU's expert kernels, about 13 ms the GPU.
|
||||
- For comparison, IQ3_S (all its experts in RAM) wrote about 53 tok/s on the same PC
|
||||
([the speed tables](DETAILS.md#speed-measured)).
|
||||
|
||||
### Two GPUs with the RAM budget
|
||||
|
||||
> **Measured on:** RTX 3090 (x16) + RTX 3080 Ti (x4), 91 GB RAM, budget 64 GiB, engine 0.1.41.
|
||||
|
||||
Decode 45 tok/s against 38 on the 3090 alone (the auto split gives the x4 card two layers); 57.6 GiB of experts in RAM,
|
||||
no file reads per request.
|
||||
|
||||
> **Measured on:** 2x RTX 3090, 165 GiB RAM, an older engine that ran the split without the RAM budget (all 77 GB of
|
||||
> experts loaded into RAM at start).
|
||||
|
||||
Decode 31 tok/s on one card with the budget, 64 to 78 tok/s split (55 tok/s at a 128K prompt), with `MemAvailable` never
|
||||
under 68 GiB.
|
||||
+62
-87
@@ -1,24 +1,27 @@
|
||||
# Unsloth UD-Q6_K_XL
|
||||
|
||||
**In setup since MODEL-UPDATE Phase 2** (`START-HERE.bat --setup --family unsloth --model UD-Q6_K_XL`, engine 0.1.44 or
|
||||
newer: every expert in RAM from ~118 GB, else it asks and keeps a RAM budget; quality, speed and images in
|
||||
[QUANTS.md](QUANTS.md); UD-Q5_K_XL and Q8_0 likewise). The manual workflow below still works. This is Unsloth's 6-bit
|
||||
file of the same model the other packs use:
|
||||
[unsloth/Qwen3.8-Flash-Next-GGUF, `UD-Q6_K_XL`, revision `38bb39e`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL).
|
||||
Build it for engine 0.1.38 or newer (earlier engines read `per_layer_token_embd.weight` in IQ4_NL, Q5_0 or FP8 only,
|
||||
below). UD-Q4_K_XL's manual workflow asks nothing of setup, and neither does this one.
|
||||
This page is for people who want to run Unsloth's 6-bit Qwen3.8-Flash-Next file with Strata, and for people who pack it
|
||||
by hand. UD-Q5_K_XL and Q8_0 are set up the same way. Quality, speed and images across the Unsloth sizes are in
|
||||
[QUANTS.md](QUANTS.md). UD-Q4_K_XL, with the full setup details, is in [UNSLOTH_Q4.md](UNSLOTH_Q4.md).
|
||||
|
||||
> **Status (MODEL-UPDATE Phase 1).** The Q6_K gate/up kernels are part of every build now (they were the opt-in
|
||||
> `-DSTRATA_Q6K_EXPERTS=ON`): measured to leave other models' expert-cache slots unchanged ([QUANTS.md](QUANTS.md)). The
|
||||
> MMQ prompt kernels for Q6_K come with the K-quant library and `STRATA_MMQ_KQ=1`; without it the prompt takes the FP16
|
||||
> path. The file runs on the standard engine (AMD R9700: the quality yardstick in [QUANTS.md](QUANTS.md)).
|
||||
The file is [unsloth/Qwen3.8-Flash-Next-GGUF, `UD-Q6_K_XL`, revision `38bb39e`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL),
|
||||
the same model the other packs use.
|
||||
|
||||
## Quick start
|
||||
|
||||
```sh
|
||||
strata launch --family unsloth --model UD-Q6_K_XL
|
||||
```
|
||||
|
||||
Setup keeps every expert in RAM from about 118 GB of RAM. Below that it asks and keeps a RAM budget, as for UD-Q4_K_XL
|
||||
([What setup does](UNSLOTH_Q4.md#what-setup-does)). For UD-Q5_K_XL and UD-Q6_K_XL it writes `STRATA_MMQ_KQ=1`, the
|
||||
K-quant MMQ prompt kernels, into the run config.
|
||||
|
||||
## What is in the file
|
||||
|
||||
Six shards, 169,165,382,688 bytes (169.2 GB) together ([sizes and SHA-256 from the HF tree API](https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL);
|
||||
shards 1, 2 and 6 were also checked bit for bit against their Hugging Face LFS hashes after download). Shard 1 holds
|
||||
only metadata; shard 2 only the output head (`output.weight`, `output_hc_*`, all Q8_0 but `output_hc_norm`, F32);
|
||||
shard 3 is exactly the PLE table, nothing else.
|
||||
Six shards, 169,165,382,688 bytes (169.2 GB) together ([sizes and SHA-256 from the HF tree API](https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL)).
|
||||
Shard 1 holds only metadata. Shard 2 holds only the output head (`output.weight`, `output_hc_*`, all Q8_0 but
|
||||
`output_hc_norm`, F32). Shard 3 is exactly the PLE table.
|
||||
|
||||
| File | Bytes | SHA-256 |
|
||||
| --- | --- | --- |
|
||||
@@ -29,30 +32,24 @@ shard 3 is exactly the PLE table, nothing else.
|
||||
| `Qwen3.8-Flash-Next-UD-Q6_K_XL-00005-of-00006.gguf` | 49,419,317,344 | `9948e81ae8368144b135a7403d8c0f2cf9a6c2f9f692313865855b2832e696c9` |
|
||||
| `Qwen3.8-Flash-Next-UD-Q6_K_XL-00006-of-00006.gguf` | 14,838,313,984 | `e4ae255234f42b18012f6d6eec5bf615b57479b7e0a0233c47436a321ffeec3f` |
|
||||
|
||||
Formats, read from the shard headers with `tools/gguf_reader.py` (all six shards verified: shard 4 = blocks 0-20,
|
||||
520 tensors, shard 5 = blocks 20-41, 530 tensors, and per layer the types above):
|
||||
Formats, read from the shard headers with `tools/gguf_reader.py`:
|
||||
|
||||
- **Routed experts:** `ffn_gate_exps` and `ffn_up_exps` are Q6_K (seen in every layer of shard 6: blocks 41-47);
|
||||
`ffn_down_exps` is Q8_0 (per layer 891.3 MB at 640×2560×512, against Q6_K's 688.1 MB at 2560×640×512 for gate or
|
||||
up). Gate and up are Q6_K in 47 of 48 layers and Q8_0 in layer 2; down is Q8_0 in every layer (the pack's
|
||||
`native_experts.txt`: 47 rows of ggml types 14/8, and layer 2 is 8/8). Q6_K gate/up is native: the
|
||||
engine multiplies them directly on the GPU in every path check below, and the CPU has the ggml block layout (210
|
||||
bytes per 256 elements); the per-role layout in `native_experts.txt` v4 handles shards whose boundaries cut
|
||||
through a layer like UD-Q4_K_XL's layer 11 case does.
|
||||
- **Everything else:** attention and shared-expert projections, SSM, hyper-connections, the PLE key/value, the token
|
||||
embedding and the output head are Q8_0, or F32/BF16 where the engine reads floats — the same pattern
|
||||
- **Routed experts.** `ffn_gate_exps` and `ffn_up_exps` are Q6_K in 47 of 48 layers and Q8_0 in layer 2. `ffn_down_exps`
|
||||
is Q8_0 in every layer (891.3 MB per layer at 640x2560x512, against Q6_K's 688.1 MB for gate or up). The pack's
|
||||
`native_experts.txt` has 47 rows of ggml types 14/8, and layer 2 is 8/8. The engine multiplies Q6_K gate/up natively on
|
||||
the GPU, and the CPU has the ggml block layout (210 bytes per 256 elements). The per-role layout in
|
||||
`native_experts.txt` v4 handles shards whose boundaries cut through a layer.
|
||||
- **Everything else.** Attention and shared-expert projections, SSM, hyper-connections, the PLE key/value, the token
|
||||
embedding and the output head are Q8_0, or F32/BF16 where the engine reads floats. This is the same pattern
|
||||
[UD-Q4_K_XL](UNSLOTH_Q4.md) handles with `tools/iq_pack.py --compat-bf16`.
|
||||
- **The PLE table** (`per_layer_token_embd.weight`, [160, 320001536]): **Q8_0**, its own shard 3. A row is
|
||||
5 blocks of 32 elements at 34 bytes = 170 bytes, and the whole table is 54.4 GB, against the 28.8 GB IQ4_NL of
|
||||
UD-Q4_K_XL. Engine 0.1.38 reads Q8_0 rows natively: `PleTable` accepts the type, the row reader is already
|
||||
parameterized by bytes-per-row (its selftest runs 90, 110 and 170 byte rows), and no GPU kernel changes —
|
||||
`gather` outputs floats. (UD-Q4_K_XL's table is IQ4_NL; the FP8 table of [tools/ple_fp8_pack.py](../tools/ple_fp8_pack.py)
|
||||
needs the original checkpoint, which this file is not a substitute for. Converting this Q8_0 table to FP8 would
|
||||
trade table precision away for 6% smaller rows; Q8_0 read directly has none that conversion's error.)
|
||||
- **The PLE table** (`per_layer_token_embd.weight`, [160, 320001536]) is **Q8_0**, in its own shard 3. A row is 5 blocks
|
||||
of 32 elements at 34 bytes, 170 bytes in all, and the whole table is 54.4 GB (UD-Q4_K_XL's IQ4_NL table is 28.8 GB).
|
||||
The engine reads Q8_0 rows natively. The FP8 table of [tools/ple_fp8_pack.py](../tools/ple_fp8_pack.py) needs the
|
||||
original checkpoint, which this file is not a substitute for.
|
||||
|
||||
Q6_K gate/up costs 210 bytes per 256 elements, against Q4_K's 144, so the routed experts' share of the file grows
|
||||
about 45% per element read; expected decode speed accordingly lower than UD-Q4_K_XL's on the same hardware. The
|
||||
table row gather grows 170/90 = 1.89x over IQ4_NL.
|
||||
Q6_K gate/up costs 210 bytes per 256 elements, against Q4_K's 144, so the routed experts' share of the file grows about
|
||||
45% per element read. Decode is accordingly slower than UD-Q4_K_XL's on the same hardware. The table row gather grows
|
||||
170/90 = 1.89x over IQ4_NL.
|
||||
|
||||
## The pack (by hand)
|
||||
|
||||
@@ -64,22 +61,18 @@ Same workflow as [UD-Q4_K_XL](UNSLOTH_Q4.md#the-pack-by-hand), from the reposito
|
||||
--out packs/ud-q6_k_xl --compat-bf16
|
||||
```
|
||||
|
||||
**`--compat-bf16` is required** for the same reason as there (the 195 small Q8_0 projections the engine reads as
|
||||
BF16). Do not add `--experts-bin`; the experts stay in the GGUF files. Built here end to end on this file
|
||||
(`packs/ud-q6_k_xl`, the packer's own tests cover the shard and conversion logic via
|
||||
`.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`): 1079 tensors indexed, 303 served natively, 460
|
||||
tensors converted (264 exact, 196 rounded, max abs error 0.0144 - the same profile as UD-Q4_K_XL's), 195
|
||||
compat-BF16 tensors (1.20 GiB), arena 1.38 GiB, and `native_experts.txt` v4 with the per-role shard column on
|
||||
layers 20 and 41.
|
||||
**`--compat-bf16` is required**, for the same reason as there (the 195 small Q8_0 projections the engine reads as BF16).
|
||||
Do not add `--experts-bin`; the experts stay in the GGUF files. The pack on the finished file has 1079 tensors indexed,
|
||||
303 served natively, 460 converted (264 exact, 196 rounded, max abs error 0.0144), 195 compat-BF16 tensors (1.20 GiB),
|
||||
an arena of 1.38 GiB, and `native_experts.txt` v4 with the per-role shard column on layers 20 and 41. The packer's tests:
|
||||
`.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`.
|
||||
|
||||
## The server
|
||||
|
||||
Like UD-Q4_K_XL's [config](UNSLOTH_Q4.md#the-server), with the Q6 paths and names. `--ple-gguf` is not needed: the
|
||||
engine finds shard 3 by the table's name. On a machine whose RAM holds the ~110 GB of routed experts plus the rest
|
||||
(499 GB total, ~325 GB free, measured on the machine this was written on), there is no RAM budget and no
|
||||
`--expert-cache` gymnastics: drop `--resident-budget-gib` and `--expert-cache-device1` from the Q4 config. The
|
||||
54.4 GB table is mmap'd (the default) — do not switch it to `--ple-io ram` unless the RAM really is there to lock
|
||||
it. `mtp/rt` is the base model's draft layer as always.
|
||||
Like UD-Q4_K_XL's [config](UNSLOTH_Q4.md#the-server), with the Q6 paths and names. `--ple-gguf` is not needed: the engine
|
||||
finds shard 3 by the table's name. On a machine whose RAM holds the roughly 110 GB of routed experts plus the rest, drop
|
||||
`--resident-budget-gib` and `--expert-cache-device1` from the Q4 config. The 54.4 GB table is mapped (the default). Do
|
||||
not switch it to `--ple-io ram` unless the RAM really is there to lock it. `mtp/rt` is the base model's draft layer.
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -99,46 +92,28 @@ it. `mtp/rt` is the base model's draft layer as always.
|
||||
}
|
||||
```
|
||||
|
||||
On 64 GB machines treat the Q4 numbers below as the starting point: the RAM budget matters the same way, and
|
||||
everything it leaves out comes from the SSD, faster now per expert (Q6_K) and per table row (170 bytes).
|
||||
On a 64 GB machine, start from UD-Q4_K_XL's RAM-budget advice. The budget matters the same way, and everything it leaves
|
||||
out comes from the SSD.
|
||||
|
||||
## What the engine did not have, and has now
|
||||
## Kernels and tests
|
||||
|
||||
UD-Q4_K_XL's expert formats (gate/up Q4_K, down Q5_1) were in every path; Q6_K gate/up was not. This branch adds
|
||||
it (the PLE Q8_0 reader is the other half):
|
||||
- Decode windows: the grouped expert kernels (`native_expert_grouped`) take Q6_K gate/up (a pinned copy of llama.cpp's
|
||||
`vec_dot_q6_K_q8_1` behind `Fmt<14>`). Per-token MMVQ (`native_mmvq`) has Q6_K too.
|
||||
- Prompt path: llama.cpp's MMQ prompt kernels instantiate Q6_K in the K-quant library (`STRATA_MMQ_KQ=1`). Without it
|
||||
the prompt takes the FP16 path.
|
||||
- Tests: `native_grouped_parity` loops Q6_K/Q8_0 pairs, `prefill_mmq_kquant_test` runs Q6_K gate/up products at
|
||||
1280x2560 against a double-precision reference, and `native_expert_parity --synthetic q6_K/q8_0` checks GPU and CPU
|
||||
against ggml's dequantized weights. `ple_reader_test --selftest` runs 90, 110 and 170-byte PLE rows, and
|
||||
`ple_q8_parity` matches ggml's reference dequantizer bit for bit through both Mmap and Direct.
|
||||
- Not measured on this page: quality against llama.cpp on this file ([QUANTS.md](QUANTS.md) has the Unsloth quality
|
||||
comparison).
|
||||
|
||||
- **Decode windows:** the grouped expert kernels (`native_expert_grouped`) take gate/up Q6_K — a pinned copy of
|
||||
llama.cpp's `vec_dot_q6_K_q8_1` behind `Fmt<14>`, and the dequantizer (`dequantize_q6_K` folded onto the kernel's
|
||||
32 threads). Per-token MMVQ (`native_mmvq`) already had Q6_K.
|
||||
- **The prompt path:** llama.cpp's MMQ prompt kernels instantiate Q6_K (the K-quant library, `STRATA_MMQ_KQ=1`; it builds
|
||||
`template-instances/mmq-instance-q6_k.cu` alongside q4_k/q5_k/q5_1; `moe_mmq` accepts and dispatches it).
|
||||
- **The gates:** `native_expert_supported` took Q6_K/Q8_0 pairs (gate/up Q8_0 as in layer 2 was already in the
|
||||
lists) — the engine previously refused the whole pack with "no GPU kernels for".
|
||||
## History / older measurements
|
||||
|
||||
Tests: `native_grouped_parity` loops Q6_K/Q8_0 pairs; `prefill_mmq_kquant_test` runs Q6_K gate/up products at
|
||||
1280×2560 against a double-precision reference; `native_expert_parity --synthetic q6_K/q8_0` checks GPU and CPU
|
||||
against ggml's dequantized weights. All three are built (CUDA, `86;120`). They have not been run on a GPU: the
|
||||
card was serving the file below.
|
||||
> **Measured on:** Linux, RTX PRO 4500 Blackwell (compute capability 12.0) plus a second GPU, Threadripper PRO 5975WX
|
||||
> (AVX2, no AVX-512), 30 expert-pool workers, 499 GB RAM, 2026-10-04. One code-review session.
|
||||
|
||||
## Status: what is verified and what is not
|
||||
|
||||
Verified here (test clone, Linux, NVIDIA GPU, source build):
|
||||
|
||||
- The six files' sizes and SHA-256 (HF tree API; shards 1, 2, 6 additionally against the downloaded blobs).
|
||||
- The tensor inventory of shards 1, 2 and 6, read from the headers (`tools/gguf_reader.py`); shards 4 and 5's
|
||||
tensor counts and per-layer expert types, from the pack's `conversions.json` and `native_experts.txt` v4.
|
||||
- Q8_0 PLE rows: `ple_reader_test --selftest` runs 90, 110 and 170-byte rows (all OK); `ple_q8_parity` on a
|
||||
synthetic Q8_0 table (`tools/ple_q8_fixture.py`) matches ggml's reference dequantizer bit for bit through both
|
||||
Mmap and Direct (`max_abs 0.000e+00`). Both still pass on this branch with the Q6_K expert work in.
|
||||
- The pack on the finished file (numbers above), and the engine binary builds (CUDA `86;120`, MMQ K-quants on).
|
||||
- Q6_K expert source support: decode-side MMVQ was already there; the grouped verify-window kernels, the MMQ
|
||||
prompt kernels and the `native_expert_supported` gate were not, and are in this branch (section above).
|
||||
|
||||
The server has run this file. On 2026-10-04, Linux, an RTX PRO 4500 Blackwell (compute capability 12.0) plus a
|
||||
second GPU, a Threadripper PRO 5975WX (AVX2, no AVX-512), 30 expert-pool workers and 499 GB of RAM, engine 0.1.39
|
||||
loaded 101.75 GiB of experts (8.43 GiB/s) and served one review. The cold read was 14,915 tokens at 3,176 tok/s.
|
||||
Fourteen turns generated 5,799 tokens in 74.1 s (78.3 tok/s) and finished at 44,160 tokens of context; drafts were
|
||||
accepted 3,485 of 4,406 (79.1%). The same machine on engine 0.1.38, before this branch was rebased onto 0.1.39,
|
||||
read 249,193 tokens cold at 2,802 tok/s and then generated 1,079 tokens at 69.0 tok/s. Quality against llama.cpp
|
||||
on this file has not been measured, so the page stays experimental. The three Q6_K parity programs above are the
|
||||
remaining GPU check.
|
||||
| Engine | Experts loaded | Prompt | Decode | Drafts accepted |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 0.1.39 | 101.75 GiB at 8.43 GiB/s | 14,915 tokens cold at 3,176 tok/s | 5,799 tokens in 74.1 s (78.3 tok/s) over 14 turns, ending at 44,160 tokens of context | 3,485 of 4,406 (79.1%) |
|
||||
| 0.1.38 | | 249,193 tokens cold at 2,802 tok/s | 1,079 tokens at 69.0 tok/s | |
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2023 LobeHub
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Model logos
|
||||
|
||||
Small Qwen and DeepSeek marks from [LobeHub Icons](https://github.com/lobehub/lobe-icons), used to identify model families. Downloaded 23 October 2026; SVG artwork is unchanged.
|
||||
|
||||
- [Qwen source](https://github.com/lobehub/lobe-icons/blob/master/packages/static-svg/icons/qwen-color.svg)
|
||||
- [DeepSeek source](https://github.com/lobehub/lobe-icons/blob/master/packages/static-svg/icons/deepseek-color.svg)
|
||||
|
||||
The icon library is [MIT licensed](LICENSE), copyright 2023 LobeHub. Model names and brand marks belong to their respective owners.
|
||||
@@ -0,0 +1 @@
|
||||
<svg height="1em" style="flex:none;line-height:1" viewBox="0 0 24 24" width="1em" xmlns="http://www.w3.org/2000/svg"><title>DeepSeek</title><path d="M23.748 4.482c-.254-.124-.364.113-.512.234-.051.039-.094.09-.137.136-.372.397-.806.657-1.373.626-.829-.046-1.537.214-2.163.848-.133-.782-.575-1.248-1.247-1.548-.352-.156-.708-.311-.955-.65-.172-.241-.219-.51-.305-.774-.055-.16-.11-.323-.293-.35-.2-.031-.278.136-.356.276-.313.572-.434 1.202-.422 1.84.027 1.436.633 2.58 1.838 3.393.137.093.172.187.129.323-.082.28-.18.552-.266.833-.055.179-.137.217-.329.14a5.526 5.526 0 01-1.736-1.18c-.857-.828-1.631-1.742-2.597-2.458a11.365 11.365 0 00-.689-.471c-.985-.957.13-1.743.388-1.836.27-.098.093-.432-.779-.428-.872.004-1.67.295-2.687.684a3.055 3.055 0 01-.465.137 9.597 9.597 0 00-2.883-.102c-1.885.21-3.39 1.102-4.497 2.623C.082 8.606-.231 10.684.152 12.85c.403 2.284 1.569 4.175 3.36 5.653 1.858 1.533 3.997 2.284 6.438 2.14 1.482-.085 3.133-.284 4.994-1.86.47.234.962.327 1.78.397.63.059 1.236-.03 1.705-.128.735-.156.684-.837.419-.961-2.155-1.004-1.682-.595-2.113-.926 1.096-1.296 2.746-2.642 3.392-7.003.05-.347.007-.565 0-.845-.004-.17.035-.237.23-.256a4.173 4.173 0 001.545-.475c1.396-.763 1.96-2.015 2.093-3.517.02-.23-.004-.467-.247-.588zM11.581 18c-2.089-1.642-3.102-2.183-3.52-2.16-.392.024-.321.471-.235.763.09.288.207.486.371.739.114.167.192.416-.113.603-.673.416-1.842-.14-1.897-.167-1.361-.802-2.5-1.86-3.301-3.307-.774-1.393-1.224-2.887-1.298-4.482-.02-.386.093-.522.477-.592a4.696 4.696 0 011.529-.039c2.132.312 3.946 1.265 5.468 2.774.868.86 1.525 1.887 2.202 2.891.72 1.066 1.494 2.082 2.48 2.914.348.292.625.514.891.677-.802.09-2.14.11-3.054-.614zm1-6.44a.306.306 0 01.415-.287.302.302 0 01.2.288.306.306 0 01-.31.307.303.303 0 01-.304-.308zm3.11 1.596c-.2.081-.399.151-.59.16a1.245 1.245 0 01-.798-.254c-.274-.23-.47-.358-.552-.758a1.73 1.73 0 01.016-.588c.07-.327-.008-.537-.239-.727-.187-.156-.426-.199-.688-.199a.559.559 0 01-.254-.078c-.11-.054-.2-.19-.114-.358.028-.054.16-.186.192-.21.356-.202.767-.136 1.146.016.352.144.618.408 1.001.782.391.451.462.576.685.914.176.265.336.537.445.848.067.195-.019.354-.25.452z" fill="#4D6BFE"></path></svg>
|
||||
|
After Width: | Height: | Size: 2.1 KiB |
@@ -0,0 +1 @@
|
||||
<svg height="1em" style="flex:none;line-height:1" viewBox="0 0 24 24" width="1em" xmlns="http://www.w3.org/2000/svg"><title>Qwen</title><path d="M19.713 2.825h-5.907c0-.251-.066-.503-.196-.728L12.819.728A1.46 1.46 0 0 0 11.559 0h-1.58c-.52 0-1 .275-1.26.728l-2.42 4.19a1.45 1.45 0 0 0 0 1.453l.791 1.37c.26.448.74.727 1.26.727h11.36c.52 0 1-.275 1.26-.728l.79-1.37a1.45 1.45 0 0 0 0-1.451l-.79-1.37a1.46 1.46 0 0 0-1.26-.728zM.195 9.905l2.951 5.115a1.46 1.46 0 0 0-.531.531l-.791 1.37a1.45 1.45 0 0 0 0 1.452l.79 1.37c.26.448.74.727 1.26.727h4.84c.52 0 1-.275 1.26-.728l.79-1.37a1.45 1.45 0 0 0 0-1.451L5.086 7.083a1.46 1.46 0 0 0-1.26-.728h-1.58c-.52 0-1 .276-1.26.728l-.79 1.37a1.45 1.45 0 0 0 0 1.452m15.888 13.367 2.951-5.115c.217.125.469.196.728.196h1.582c.52 0 1-.275 1.26-.728l.79-1.37a1.45 1.45 0 0 0 0-1.451l-2.42-4.191a1.46 1.46 0 0 0-1.259-.728h-1.582c-.52 0-1 .276-1.259.728l-5.678 9.838a1.45 1.45 0 0 0 0 1.452l.79 1.37c.26.448.74.727 1.26.727h1.582c.52 0 1-.275 1.26-.728z" fill="#082dff"></path></svg>
|
||||
|
After Width: | Height: | Size: 1016 B |
@@ -1,7 +1,7 @@
|
||||
# Engine component manifest (schema 2)
|
||||
|
||||
One format for every engine package: Windows CUDA 13 and 12, Windows HIP, Linux CUDA, Linux HIP, Linux SYCL and macOS
|
||||
Metal. Control (`strata_control/engine_component.py`) reads it, checks it, and only then starts the engine.
|
||||
This page is for developers who build or check engine packages. It defines the manifest that every engine package carries:
|
||||
Windows CUDA 13 and 12, Windows HIP, Linux CUDA, Linux HIP, Linux SYCL and macOS Metal. Control (`strata_control/engine_component.py`) reads it, checks it, and only then starts the engine.
|
||||
`packaging/package_engine.py` writes it. A package you make locally is unsigned and loads. A package in the release feed is signed with the
|
||||
feed key (an Ed25519 key the owner keeps offline), and `strata update` / "Update engine" install it only when that signature verifies
|
||||
([UPDATES.md](UPDATES.md)). The installers carry an engine of their own, so a new install needs no download to start a model.
|
||||
@@ -24,7 +24,7 @@ A package folder is immutable: a new build is a new folder.
|
||||
|---|---|---|
|
||||
| `schema_version` | 2 | schema 1 packages (no `archs`, `min_driver`, `build_id`, `signature`; `cuda_archs` instead of `archs`) still load |
|
||||
| `id`, `name` | `"strata-engine"` | the component name; anything else is refused |
|
||||
| `version` | string | the engine release, e.g. `0.1.45`; the newest compatible version wins |
|
||||
| `version` | string | the engine release, e.g. `0.1.50`; the newest compatible version wins |
|
||||
| `platform` | string | `windows-x86_64`, `linux-x86_64`, `macos-arm64` (must match this PC) |
|
||||
| `backend` | string | `cuda`, `hip`, `sycl`, `metal`, `cpu`; must be a combination `BACKEND_MATRIX` commits to, and the one this PC's GPU needs |
|
||||
| `archs` | list of strings | what the build contains. CUDA: `sm_86` (machine code for that card), `compute_89` (PTX, runs on that generation or newer). HIP: `gfx1201`. SYCL and Metal: a descriptive tag (`intel-arc`, `apple-gpu`). Required for every non-`cpu` backend. Enforced for CUDA against `nvidia-smi` compute capability; recorded for the others |
|
||||
@@ -40,7 +40,7 @@ A package folder is immutable: a new build is a new folder.
|
||||
Example (shortened):
|
||||
|
||||
```json
|
||||
{"schema_version": 2, "id": "strata-engine", "name": "strata-engine", "version": "0.1.45",
|
||||
{"schema_version": 2, "id": "strata-engine", "name": "strata-engine", "version": "0.1.50",
|
||||
"platform": "linux-x86_64", "backend": "cuda", "archs": ["sm_86"], "min_driver": {"nvidia": "580.65"},
|
||||
"build_id": "be885026f96a-cuda13.0", "entrypoint": "strata",
|
||||
"files": [{"path": "strata", "sha256": "<64 hex>", "bytes": 191000000}], "signature": null}
|
||||
@@ -59,7 +59,7 @@ In this order, each refusal names the cause; none of them starts anything and no
|
||||
6. on POSIX the entrypoint is executable.
|
||||
|
||||
If a package fails (5) or (4) and no other package fits this PC, the start ends with `EngineIntegrityError`
|
||||
(exit code 5): "The engine package failed its integrity check, so it was not started: 0.1.45 linux-x86_64-cuda:
|
||||
(exit code 5): "The engine package failed its integrity check, so it was not started: 0.1.50 linux-x86_64-cuda:
|
||||
libcudart.so.13 does not match its recorded SHA-256. Repair or reinstall the engine component with Update engine in the app's
|
||||
Settings or `strata update` in the Terminal". A package
|
||||
for another GPU or an older driver is `EngineMissing` with its reason. A tampered newer package never falls back silently
|
||||
@@ -83,29 +83,35 @@ UTF-8). It contains every file hash, so one signature covers the whole package.
|
||||
|
||||
## Building a package
|
||||
|
||||
python packaging/package_engine.py --src <build folder> --backend cuda --version 0.1.45 \
|
||||
python packaging/package_engine.py --src <build folder> --backend cuda --version 0.1.50 \
|
||||
--out <components>/engine --arch sm_86 --arch compute_89 --min-driver nvidia=580.65 \
|
||||
[--build-id <id>] [--variant cuda13] [--platform linux-x86_64] [--include 'rocm/**' --include strata-vision.exe | --tree]
|
||||
[--build-id <id>] [--variant cuda13] [--platform linux-x86_64] [--include 'rocm/**' --include strata-vision.exe | --tree] [--require-dsv41]
|
||||
python packaging/package_engine.py --verify <package folder> # full hash check
|
||||
|
||||
Without `--include`/`--tree` it takes the entrypoint, the libraries beside it, `BUILD.json` and `LICENSE*`. The build id
|
||||
Without `--include`/`--tree` it takes the entrypoint, the libraries beside it, `BUILD.json`, `LICENSE*`, the picture reader
|
||||
`strata-vision` when the build has one, and the DeepSeek engine `strata-dsv41[.exe]` when it sits beside `strata`.
|
||||
`--require-dsv41` refuses a package without `strata-dsv41`. The build id
|
||||
defaults to the checkout's commit (`-dirty` when the tree has changes). It prints the size, file count and the SHA-256 of
|
||||
the manifest file (which holds every file's hash), used as the package's digest in notes.
|
||||
|
||||
## What each target's package holds
|
||||
|
||||
Variants in the release: Windows `cuda13`, `cuda12`, `rx9070`, `rdna3`, `rdna2`; Linux `cuda13`, `cuda12`, `rx9070`, `rx7900`, `halo`, `rx6000`, plus
|
||||
the SYCL and Metal packages. The packages that also carry `strata-dsv41` (DeepSeek V4.1-Flash, engine 0.1.50) are Windows `cuda13`, Linux `cuda13`
|
||||
and Linux `rx9070`. Build `strata-dsv41` with `packaging/build_dsv41_cuda.sh` or `packaging/build_dsv41_windows.bat` and copy it beside `strata` before packaging.
|
||||
|
||||
| Package | Built on | Contents | Notes |
|
||||
|---|---|---|---|
|
||||
| `windows-x86_64` cuda, variant `cuda13` | the owner PC, the same cmake line as the release zips (CUDA 13.0, `75;86;89;120` real + `120` PTX, `STRATA_PORTABLE=ON`) | `strata.exe`, `strata_kq.dll`, `strata-vision.exe` (built for this variant's archs by `packaging/build_vision_windows.bat cuda13 <stage>`, which writes `vision_archs` into BUILD.json), `strata-dsv41.exe` (DeepSeek-V4.1, `packaging/build_dsv41_windows.bat`, sm_86 + sm_120; `--require-dsv41`), `BUILD.json` | `min_driver` nvidia 580.65 |
|
||||
| `windows-x86_64` cuda, variant `cuda12` | the owner PC (CUDA 12.9, `60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same four files, with its own `strata-vision.exe` (`packaging/build_vision_windows.bat cuda12 <stage>`: sm_60 .. sm_89; the CUDA 13 reader has no Pascal code, #3669) | experimental, `min_driver` nvidia 575.51; the CUDA 13 build wins when both fit |
|
||||
| `windows-x86_64` hip | the owner PC, `tools/hip/build_windows.bat` (ROCm 10.2 nightly wheels) then `tools/hip/package_windows.py` | `strata.exe`, `strata-device.exe`, the ROCm DLLs and the rocBLAS/hipBLASLt kernels of the zip (`--tree`) | no Windows AMD test machine yet; with `STRATA_NO_VISION` unset it also builds the two picture readers (`strata-vision.exe` on the CPU, `strata-vision-hip.exe` on the GPU) |
|
||||
| `windows-x86_64` cuda, variant `cuda13` | the owner PC, the same cmake line as the release zips (CUDA 13.0, `75;86;89;120` real + `120` PTX, `STRATA_PORTABLE=ON`) | `strata.exe`, `strata_kq.dll`, `strata-vision.exe` (built for this variant's archs by `packaging/build_vision_windows.bat cuda13 <stage>`, which writes `vision_archs` into BUILD.json), `BUILD.json` | `min_driver` nvidia 580.65 |
|
||||
| `windows-x86_64` cuda, variant `cuda12` | the owner PC (CUDA 12.9, `60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same four files, with its own `strata-vision.exe` (`packaging/build_vision_windows.bat cuda12 <stage>`: sm_60 .. sm_89; the CUDA 13 reader has no Pascal code) | experimental, `min_driver` nvidia 575.51; the CUDA 13 build wins when both fit |
|
||||
| `windows-x86_64` hip | the owner PC, `tools/hip/build_windows.bat` (ROCm 10.2 nightly wheels) then `tools/hip/package_windows.py` | `strata.exe`, `strata-device.exe`, the ROCm DLLs and the rocBLAS/hipBLASLt kernels of the zip (`--tree`) | with `STRATA_NO_VISION` unset it also builds the two picture readers (`strata-vision.exe` on the CPU, `strata-vision-hip.exe` on the GPU) |
|
||||
| `linux-x86_64` cuda, variant `cuda13` | a Linux box with the CUDA 13 toolkit, `VARIANT=cuda13 packaging/build_cuda_linux.sh` | `strata`, `libstrata_kq.so`, `strata-vision` (built by the same script for the same archs, `BUILD_VISION=0` for a text-only package), `libcudart/libcublas/libcublasLt` (13.x) | the GPU driver's `libcuda` stays the PC's |
|
||||
| `linux-x86_64` cuda, variant `cuda12` | a Linux box with CUDA 12.9, `VARIANT=cuda12 packaging/build_cuda_linux.sh` (`60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same files with the 12.x libraries (the reader has sm_60 / 61 / 70 too) | experimental, `min_driver` nvidia 575.51; the Pascal and Volta package (the 0.2.1 one lacked sm_60 / 61 / 70, #3503); `package_engine.py` refuses a cuda12 package without them |
|
||||
| `linux-x86_64` cuda, variant `cuda12` | a Linux box with CUDA 12.9, `VARIANT=cuda12 packaging/build_cuda_linux.sh` (`60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same files with the 12.x libraries (the reader has sm_60 / 61 / 70 too) | experimental, `min_driver` nvidia 575.51; the Pascal and Volta package; `package_engine.py` refuses a cuda12 package without them |
|
||||
| `linux-x86_64` hip | a box with ROCm (the Python wheels are enough) | `strata`, the HIP runtime, hipBLAS/hipBLASLt/rocBLAS, comgr + libLLVM, `rocblas/library` and `hipblaslt/library` for the listed gfx archs; `packaging/build_hip_linux.sh` builds it with both picture readers |
|
||||
| `linux-x86_64` hip, variant `rx6000` (RX 6000, gfx1030 / gfx1031) | `VARIANT=rx6000 packaging/build_hip_linux.sh` in the jammy chroot, ROCm 7.13.0a20260515 `gfx103X-all` wheels | the same set as the other HIP packages, with `rocblas/library` for gfx1030 / gfx1031; no `hipblaslt/library` kernels exist for them and no `tuning/` table | built, not tested on a real RX 6000 card (the manifest notes say so). `windows-x86_64` hip, variant `rdna2`: the same archs from `tools/hip/build_windows.bat` with `STRATA_HIP_ARCHS=gfx1030;gfx1031`, packaged with `--variant rdna2`. A PC with an RX 7000 / 9000 card beside an RX 6000 one never takes it (`engine_component.hip_preference`) |
|
||||
| `linux-x86_64` hip, gfx1103 (Radeon 780M / Phoenix iGPU) | `packaging/build_hip_gfx1103.sh` on the mini PC (TheRock gfx110X-all SDK) | the same set as the gfx1200/1201 package, with `rocblas/library` and `hipblaslt/library` for gfx1103 (600 MiB) | experimental, unvalidated: the iGPU resets now and then (MES REMOVE_QUEUE); never kill the engine mid-request; run configs set `STRATA_NO_ARENA_THP=1`. `engine_component.resolve` picks a HIP package only when its `archs` list a gfx target the Linux kfd topology reports |
|
||||
| `linux-x86_64` sycl, `archs` `intel-arc` | a Linux box with oneAPI DPC++ 2026.1 and `patchelf`, `packaging/build_sycl_linux.sh` (JIT from SPIR-V; ocloc cannot build the Alchemist FP64 kernels ahead of time, so one JIT package serves A-series and B-series) | `strata`, libsycl, the Unified Runtime adapters, the oneMKL SYCL libraries, libimf / libsvml / libirng / libintlc (about 650 MiB); every ELF file has RUNPATH `$ORIGIN` (Intel's libraries ship without it), so `env -i ./strata --version` runs | the Level Zero loader and compute runtime are the PC's driver; the user must be in the `render` group (`strata doctor` warns). No Windows package: `BACKEND_MATRIX` allows sycl on `linux-x86_64` only |
|
||||
| `macos-arm64` metal | an Apple Silicon Mac | `strata`, `strata-vision` (`packaging/build_vision_mac.sh`; `package_engine.py` refuses a macOS package without it, #3697), `metal/kernels/*.metal`, `metal/engine-msl/*.metal`, `metal/ggml-common.h`, `metal/data/*.bin` | the Metal engine compiles its kernel sources when it starts; since S1 it finds them beside the executable (`metal/src/resource_path.h`: environment variable, then `<exe dir>/metal/`, then the compiled-in source path) |
|
||||
| `macos-arm64` metal | an Apple Silicon Mac | `strata`, `strata-vision` (`packaging/build_vision_mac.sh`; `package_engine.py` refuses a macOS package without it), `metal/kernels/*.metal`, `metal/engine-msl/*.metal`, `metal/ggml-common.h`, `metal/data/*.bin` | the Metal engine compiles its kernel sources when it starts; it finds them beside the executable (`metal/src/resource_path.h`: environment variable, then `<exe dir>/metal/`, then the compiled-in source path) |
|
||||
|
||||
`packaging/collect_libs.py` gathers the Linux libraries (an `ldd` closure with the dlopen'd extras named, kernel data filtered by
|
||||
GPU architecture) into the folder `package_engine.py` packages; libraries the distribution provides are listed in
|
||||
@@ -119,16 +125,14 @@ with the package). `packaging/stage_tuning.py` does it for the Linux build scrip
|
||||
`package_engine.py` refuses an AMD stage without `hipblaslt_version` or with a table for another version or architecture. The server's start
|
||||
(`serve/server.py with_component_engine`, rule in `strata_control/hipblaslt_tuning.py`) picks the table of the card and sets
|
||||
`STRATA_HIPBLASLT_TUNING` for the engine; a version mismatch, an unreadable card or a missing version are one log line and plain hipBLAS.
|
||||
`STRATA_HIPBLASLT_TABLE=on|off` (or `"hipblaslt_table"` in the run config) overrides; unset it follows `AUTO_USES_TABLE`, which is off until the
|
||||
owner decides, because a table changes the rounding of the prompt's GEMMs (docs/AMD_HIP.md, Tuning table). Tests: `tests/terminal/test_hipblaslt_tuning.py`.
|
||||
`STRATA_HIPBLASLT_TABLE=on|off` (or `"hipblaslt_table"` in the run config) overrides; unset it follows `AUTO_USES_TABLE`, which is off by default, because a table changes the rounding of the prompt's GEMMs (docs/AMD_HIP.md, Tuning table). Tests: `tests/terminal/test_hipblaslt_tuning.py`.
|
||||
|
||||
## The picture reader must match the package
|
||||
|
||||
`package_engine.py` (the command line, not the library call) checks the reader before it packages:
|
||||
|
||||
* a CUDA package with `strata-vision` needs `vision_archs` in BUILD.json, and every `sm_NN` of the package must be in it. A reader reused from an
|
||||
older build was the cause of #3669 (the cuda12 package on a Pascal card: the engine ran, the reader stopped with "device kernel image is
|
||||
invalid"). `--allow-stale-vision` packages it anyway and prints which archs are missing;
|
||||
older build fails on cards it was not built for. `--allow-stale-vision` packages it anyway and prints which archs are missing;
|
||||
* a macOS package without `strata-vision` is refused (`--no-picture-reader` ships a text-only Mac package on purpose);
|
||||
* a CUDA package with no reader is allowed: it answers text only, the app says "the engine installed here does not include the picture reader
|
||||
yet" and keeps Read pictures off.
|
||||
@@ -146,3 +150,10 @@ and the CPU reader as `vision.cpu_exe`, and the GPU reader does not start (the c
|
||||
busy) the server starts the CPU reader by itself and prints one line. Both programs have RUNPATH `$ORIGIN` on Linux; on Windows the
|
||||
ROCm DLLs they import sit beside the programs or in `rocm/bin` (the engine's PATH entry, `lib_dirs`). `strata_control/vision_reader.py`
|
||||
holds the choice; `tools/test_vision_reader.py` the tests.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
- **Stale picture reader (#3669):** a cuda12 package reused a reader built for newer cards. On a Pascal card the engine ran and the reader stopped with
|
||||
"device kernel image is invalid". `package_engine.py` now checks `vision_archs`.
|
||||
- **Missing sm_60 / 61 / 70 (#3503):** the first cuda12 package lacked Pascal and Volta code. `package_engine.py` now refuses a cuda12 package without them.
|
||||
- **Mac without a reader (#3697):** a macOS package once shipped without `strata-vision`. It is refused unless `--no-picture-reader` is given.
|
||||
+112
-114
@@ -1,42 +1,51 @@
|
||||
# Strata packaging: targets, recipes and locks
|
||||
|
||||
Revision: Strata 0.2.0. This directory holds the per-target build recipes and the machine-readable
|
||||
manifests. **It describes what can be built where, and it is explicit about what has *not* been
|
||||
built or tested.** Nothing here is a support claim.
|
||||
This page is for developers who build the desktop app, the Terminal archive and the engine packages. To install Strata, use
|
||||
[docs/INSTALL.md](../docs/INSTALL.md). It describes the current release (app and Terminal 0.2.5, Control and engines 0.1.50).
|
||||
How packages are signed and published is in [UPDATES.md](UPDATES.md), and the package format is in [ENGINE-COMPONENT.md](ENGINE-COMPONENT.md).
|
||||
|
||||
The installers carry the engine: the desktop installers hold an engine package for their platform and graphics card, and the first
|
||||
start copies it into the per-user component folder after checking every file against its manifest. A new install therefore starts a
|
||||
model without a download; later engines come from the signed feed ([UPDATES.md](UPDATES.md)). The Terminal install is the other
|
||||
route: one line, and the first model brings the engine for that PC from the same release.
|
||||
Release assets show what was published; the platform guides and retained reports show what was tested. A published package does not
|
||||
qualify every GPU, model or feature on its platform.
|
||||
|
||||
## Quick start
|
||||
|
||||
| I want to | See |
|
||||
| --- | --- |
|
||||
| Build the desktop app | [Desktop app](#desktop-app) |
|
||||
| Build an engine package | [ENGINE-COMPONENT.md](ENGINE-COMPONENT.md#building-a-package) |
|
||||
| Build the Terminal archive | [The Terminal archive](#the-terminal-archive) |
|
||||
| Sign and publish a release | [UPDATES.md](UPDATES.md#releasing) |
|
||||
| Test an update against a local feed | [UPDATES.md](UPDATES.md#testing-locally) |
|
||||
|
||||
## The target matrix
|
||||
|
||||
A backend name alone is not a target. Each row is an OS x CPU ABI x backend tuple, and a row is
|
||||
`qualified` only after the acceptance checklist has actually run on it.
|
||||
Each row is an OS, CPU ABI and backend. A row is `qualified` only after the acceptance checklist
|
||||
([below](#what-a-platform-must-produce-before-it-is-qualified)) has run on it.
|
||||
|
||||
| Target tuple | App artifact | Engine package | Status |
|
||||
| Target | App artifact | Engine package recipe | Status |
|
||||
| --- | --- | --- | --- |
|
||||
| `macos-arm64-metal` | `.app` + `.dmg` (Tauri) | Metal `strata`, bundled | **built.** The app's self-test passed from inside the mounted DMG and the engine's 17 kernel-parity suites pass on an Apple M4 GPU. A large model does not fit the 16 GiB test Mac. Carries `strata-vision` (Metal), built by `packaging/build_vision_mac.sh`. Signing and notarising: `packaging/mac/sign-and-notarize.sh` (see [mac/README.md](mac/README.md)); it has been run with stand-ins for Apple's service, not yet against the real one. |
|
||||
| Terminal `darwin-arm64` | self-contained `.tar.gz` | any | **built and verified.** Runs from a moved directory in a clean environment with no usable system Python. |
|
||||
| `windows-x86_64-cuda` | NSIS `.exe` (Tauri) | `cuda13` and `cuda12` packages | **built and tested** on a Windows PC with an NVIDIA card: silent install with a PATH of System32 only, a real model, an agent chat. |
|
||||
| `windows-x86_64-hip` | NSIS `.exe` (Tauri) | `tools/hip/build_windows.bat` | **built; never run on a real AMD card.** |
|
||||
| `linux-x86_64-cuda` | AppImage (Tauri) | `setup.py`; built in a glibc 2.35 chroot | **cuda13 built and tested** (RTX 3060 and RTX A4000: app in the browser, a real model, a picture). cuda12 built. |
|
||||
| `linux-x86_64-hip` | AppImage (Tauri) | `packaging/build_hip_linux.sh` | **RX 9000 (gfx1200 / gfx1201) built and tested** on a Radeon AI PRO R9700. **RX 7000 (gfx1100 / 1101 / 1102) built; not run on an RX 7000 card.** **RX 6000 (gfx1030 / 1031, 0.2.2, engine package `rx6000` only) built; not tested on a real RX 6000 card.** |
|
||||
| `linux-x86_64-sycl` | | `packaging/build_sycl_linux.sh` (one JIT package for Alchemist and Battlemage) | **engine package built; tested on an Arc A750** (Intel Arc is an official platform since 0.1.40.2, docs/INTEL_ARC.md). Arc B-series (Battlemage) cards are supported and untested by us: no B-series card was available for the package (the B70 tests of 2026-10 were with the source build), and the Terminal says so once and tells the person how to report a problem. `sycl/setup_intel.py` (Docker) remains the source-build route. |
|
||||
| `windows-x86_64-sycl` | none | none | **gated.** No native Windows SYCL path exists upstream; it needs a separate feasibility gate, not a recipe. |
|
||||
| `macos-x86_64-*` | none | none | **unsupported.** The macOS engine is Apple-Silicon only (`metal/CMakeLists.txt` requires Apple Silicon for the Metal runtime). |
|
||||
| `macos-arm64-metal` | `.app` and `.dmg` (Tauri) | Metal `strata`, bundled with `strata-vision` (`packaging/build_vision_mac.sh`) | Built. Self-test passes from the mounted DMG. Inference measurements are in [MAC.md](../docs/MAC.md). Signing: `packaging/mac/sign-and-notarize.sh` ([mac/README.md](mac/README.md)). |
|
||||
| `darwin-arm64` Terminal | self-contained `.tar.gz` | any | Built and verified. Runs from a moved directory with no system Python. |
|
||||
| `windows-x86_64-cuda` | NSIS `.exe` (Tauri) | `cuda13` and `cuda12` | Built and tested on a Windows PC with an NVIDIA card: silent install with a System32-only PATH, a real model, an agent chat. |
|
||||
| `windows-x86_64-hip` | NSIS `.exe` (Tauri) | `tools/hip/build_windows.bat` | Published. Strix Halo inference is recorded in [STRIX_HALO.md](../docs/STRIX_HALO.md). This does not qualify every discrete Radeon card ([AMD_HIP.md](../docs/AMD_HIP.md)). |
|
||||
| `linux-x86_64-cuda` | AppImage (Tauri) | `packaging/build_cuda_linux.sh`, built in a glibc 2.35 chroot | `cuda13` built and tested on RTX 3060 and RTX A4000 (app in the browser, a real model, a picture). `cuda12` built. |
|
||||
| `linux-x86_64-hip` | AppImage (Tauri) | `packaging/build_hip_linux.sh` | `rx9070` (gfx1200 / gfx1201) built and tested on a Radeon AI PRO R9700. `rx7900` (gfx1100 / 1101 / 1102) built, not run on an RX 7000 card. `rx6000` (gfx1030 / 1031) built, not tested on a real RX 6000 card. |
|
||||
| `linux-x86_64-sycl` | none | `packaging/build_sycl_linux.sh` (one JIT package for Alchemist and Battlemage) | Engine package built and tested on an Arc A750. Battlemage is supported and untested by us. |
|
||||
| `windows-x86_64-sycl` | none | none | Gated. No native Windows SYCL path exists upstream. |
|
||||
| `macos-x86_64-*` | none | none | Unsupported. The macOS engine is Apple Silicon only. |
|
||||
|
||||
**Cross-compilation is not attempted for the desktop app.** Tauri bundles are per-OS and the
|
||||
Windows installer must be produced on Windows. What *is* portable is
|
||||
everything below the carrier: the Python contracts, the owner transport, the checks, and the engine
|
||||
build recipes.
|
||||
**DeepSeek V4.1-Flash.** Three packages carry engine 0.1.50 with the DeepSeek engine `strata-dsv41`: Windows NVIDIA `cuda13`,
|
||||
Linux NVIDIA `cuda13` and Linux AMD `rx9070`. All other packages carry engine 0.1.50. Build `strata-dsv41` with
|
||||
`packaging/build_dsv41_cuda.sh` (Linux) or `packaging/build_dsv41_windows.bat` (Windows), then package it with
|
||||
`package_engine.py --require-dsv41`. The DeepSeek engine can also be a separate download (feed member `dsv41`, see [UPDATES.md](UPDATES.md#the-feed)).
|
||||
|
||||
Cross-compilation is not attempted for the desktop app. Tauri bundles are per OS, and the Windows installer is built on Windows.
|
||||
The Python contracts, the owner transport, the checks and the engine build recipes are portable.
|
||||
|
||||
## The private Node runtime
|
||||
|
||||
The desktop bundle ships a pinned standalone Node beside the App Host, because the packaging
|
||||
requirement is that an end user supplies no runtime. It is **not committed**: it is a ~117 MB
|
||||
platform binary, it cannot be reviewed in a diff, and it is an input rather than source. Fetch and
|
||||
verify it instead:
|
||||
The desktop bundle ships a pinned standalone Node beside the App Host, so an end user supplies no runtime. It is not committed
|
||||
(about 117 MB, an input rather than source). Fetch and verify it:
|
||||
|
||||
```sh
|
||||
apps/desktop/scripts/fetch-node.sh # host platform
|
||||
@@ -44,11 +53,9 @@ apps/desktop/scripts/fetch-node.sh --target linux-x64
|
||||
apps/desktop/scripts/fetch-node.sh --check # verify what is installed
|
||||
```
|
||||
|
||||
The script refuses an archive whose SHA-256 does not match the digest recorded for that target. The
|
||||
digests are taken from nodejs.org's own `SHASUMS256.txt` for the pinned version, so they are not
|
||||
derived from the same response as the payload. **To bump the Node version: change
|
||||
`STRATA_NODE_VERSION`, add the new digests, and re-run `--check`** — the script will refuse an
|
||||
unknown target's archive rather than installing it unverified.
|
||||
The script refuses an archive whose SHA-256 differs from the digest recorded for that target. The digests come from nodejs.org's
|
||||
`SHASUMS256.txt` for the pinned version. To bump the version, change `STRATA_NODE_VERSION`, add the new digests and run `--check`.
|
||||
`fetch-node.ps1` is the Windows equivalent.
|
||||
|
||||
## Desktop app
|
||||
|
||||
@@ -56,104 +63,95 @@ unknown target's archive rather than installing it unverified.
|
||||
cd apps/desktop
|
||||
./scripts/fetch-node.sh # the sidecar (once)
|
||||
npm install --prefix ../host && npm --prefix ../host run build # the UI
|
||||
cargo tauri build --bundles app # -> Strata.app
|
||||
|
||||
# macOS DMG (headless: Tauri's default step needs a Finder session)
|
||||
./scripts/make-dmg.sh
|
||||
|
||||
# Windows (on Windows): produces an NSIS per-user installer
|
||||
cargo tauri build --bundles nsis
|
||||
|
||||
# Linux (on Linux): AppImage plus a deb
|
||||
cargo tauri build --bundles appimage deb
|
||||
cargo tauri build --bundles app # macOS: Strata.app
|
||||
./scripts/make-dmg.sh # macOS DMG (headless; Tauri's default step needs a Finder session)
|
||||
cargo tauri build --bundles nsis # Windows, on Windows: per-user NSIS installer
|
||||
cargo tauri build --bundles appimage deb # Linux, on Linux
|
||||
```
|
||||
|
||||
Windows installer policy is set in `tauri.conf.json`: `installMode: currentUser`, so the installer
|
||||
is **per-user and needs no elevation**. Uninstall retention (checklist P10) is a property of *where
|
||||
the app keeps its data*, not of an installer flag: models, profiles and sessions live outside the
|
||||
install directory (`~/.strata` and the platform state directories), so removing the application
|
||||
cannot remove them. The packaging config was validated against this Tauri version's schema — an
|
||||
earlier draft used `perMachine`/`oneClick`/`deleteAppDataOnUninstall`, which `tauri-utils` 2.10 does
|
||||
not declare and which are rejected outright rather than ignored.
|
||||
The Windows installer is per-user and needs no elevation (`installMode: currentUser` in `tauri.conf.json`). Models, profiles and
|
||||
sessions live outside the install directory (`~/.strata` and the platform state directories), so uninstalling the app does not remove them.
|
||||
|
||||
**The `dmg`/`app` targets were rebuilt and the packaged self-test re-run (8/8) after the Windows and
|
||||
Linux targets were added**, so the added targets do not affect the macOS artifact.
|
||||
Build boxes build without a signing key: the default `tauri.conf.json` has `createUpdaterArtifacts: false`. Signing happens on the
|
||||
release PC ([UPDATES.md](UPDATES.md#releasing)).
|
||||
|
||||
## Strata Control and the engine in the installed app
|
||||
|
||||
The installer carries two things that start a model, and neither fetches anything when it runs.
|
||||
The installer carries Control, and on a Mac the Metal engine too. Neither fetches anything when it runs.
|
||||
|
||||
**Control** (`<install>/control`, built by `apps/desktop/scripts/stage-control.py`): the same bundle as the Terminal
|
||||
archive, the pinned CPython 3.12.15, the hash-locked wheels, `strata_control`, `strata_cli` and `serve`, plus a
|
||||
`BUNDLE.json` with the interpreter and lock digests. The host finds it beside itself (or through `STRATA_CONTROL_DIR`),
|
||||
runs it on its own interpreter (`PYTHONNOUSERSITE=1`, no `PYTHONHOME`/`PYTHONPATH`) and starts a model with
|
||||
`strata profile create|use-config` + `strata runtime start`, so the engine runs under the owner runtime (loopback TCP and a
|
||||
256-bit token on Windows, a private socket elsewhere; a Job Object / process group; leases) and a second client (the
|
||||
Terminal) attaches to the same instance.
|
||||
**Control** (`<install>/control`, built by `apps/desktop/scripts/stage-control.py`) is the same bundle as the Terminal archive:
|
||||
the pinned CPython 3.12.15, the hash-locked wheels, `strata_control`, `strata_cli` and `serve`, plus a `BUNDLE.json` with the
|
||||
interpreter and lock digests. The host finds it beside itself or through `STRATA_CONTROL_DIR`, runs it on its own interpreter
|
||||
(`PYTHONNOUSERSITE=1`, no `PYTHONHOME` or `PYTHONPATH`) and starts a model with `strata profile create|use-config` and
|
||||
`strata runtime start`. The engine then runs under the owner runtime (loopback TCP and a 256-bit token on Windows, a private
|
||||
socket elsewhere; a Job Object or process group; leases), and a second client such as the Terminal can attach to the same instance.
|
||||
|
||||
The app starts each engine with `runtime start --die-with-pid <host>`, so the engine ends with the host, and the host ends
|
||||
with the shell (a kill-on-close Job Object around the app on Windows; the host watching its parent everywhere). The app is a
|
||||
single instance: a second launch focuses the open window. One app can run several engines (profiles `app-1`, `app-2`, ...).
|
||||
docs/CLOSING.md has every close path and how each is tested.
|
||||
The app starts each engine with `runtime start --die-with-pid <host>`, so the engine ends with the host. The host ends with the shell
|
||||
(a kill-on-close Job Object on Windows; the host watches its parent everywhere). The app is a single instance: a second launch focuses
|
||||
the open window. One app can run several engines (profiles `app-1`, `app-2`, ...). Every close path is in [docs/CLOSING.md](../docs/CLOSING.md).
|
||||
|
||||
**The engine** comes with the installer (`<install>/engine-bundle/<package folder>`, the same layout as below); the first start checks it against its own manifest and copies it into the per-user component folder, where Control finds it like any engine that `strata update` installed. It is a component in the per-user component folder
|
||||
(`%LOCALAPPDATA%\Strata\components`, `$XDG_DATA_HOME/strata/components`, or `STRATA_COMPONENTS_DIR`):
|
||||
**The engine.** The Windows and Linux installers carry no engine. On the first start the app shows the engine for the card with its size
|
||||
and one Download button (with progress and resume), or installs an engine file the person already has ("I already have the file").
|
||||
The Mac dmg keeps its small Metal engine in `<install>/engine-bundle/<package folder>`. Either way the engine ends up in the per-user
|
||||
component folder, after the package is checked against its manifest (`%LOCALAPPDATA%\Strata\components`, `$XDG_DATA_HOME/strata/components`, or `STRATA_COMPONENTS_DIR`),
|
||||
where Control finds it like any engine that `strata update` installed:
|
||||
|
||||
components/engine/<version>-<platform>-<backend>/
|
||||
engine-component.json schema 2: id/name "strata-engine", version, platform, backend, archs, min_driver,
|
||||
build_id, min_runtime (optional), entrypoint, files [{path, sha256, bytes}],
|
||||
signature (null for a local package, set by the release feed); full table and the checks in ENGINE-COMPONENT.md
|
||||
components/engine/<version>-<platform>-<backend>[-<variant>]/
|
||||
engine-component.json the manifest (see ENGINE-COMPONENT.md)
|
||||
strata[.exe] *.dll | *.so the binary and the GPU runtime libraries beside it
|
||||
|
||||
A run config names it as `"exe": "component:strata-engine"`; at start `strata_control/engine_component.py` picks the package
|
||||
for this PC (platform, then the backend its GPU needs: nvidia-smi / rocminfo / sycl-ls, or `STRATA_ENGINE_BACKEND`; CUDA
|
||||
architectures, `min_driver` and `min_runtime` are checked; the newest version wins), checks every file's size and
|
||||
SHA-256 (a start hashes the whole package; a mismatch is `EngineIntegrityError` and nothing starts) and puts the package
|
||||
folder first on the library path.
|
||||
Nothing matching is `EngineMissing`: `strata serve` ends with exit 5 and "No engine for this PC was found in ... The app never downloads an engine
|
||||
while it starts or serves", and no fetch. That is the start itself. Everything before it is a warning, not a refusal: `strata launch` says in plain
|
||||
words that it could not find an engine for this computer, asks whether to go on (Enter means yes), downloads the model and ends with the next step
|
||||
(exit code 0, nothing started), and the app shows the same warning with an "Update engine" button in Settings. `packaging/package_engine.py` makes a local package from a build folder.
|
||||
A run config names it as `"exe": "component:strata-engine"`. At start, `strata_control/engine_component.py` picks the package for this PC
|
||||
(platform, then the backend its GPU needs, found with nvidia-smi, rocminfo or sycl-ls, or `STRATA_ENGINE_BACKEND`), checks every
|
||||
file's size and SHA-256 and puts the package folder first on the library path. A mismatch is `EngineIntegrityError` and nothing starts.
|
||||
|
||||
**No acquisition at run time.** `STRATA_NO_ACQUIRE=1` (set by the Tauri shell for the host, and for Control by the host)
|
||||
makes `strata_control/acquisition.py` refuse pip, downloads and compiles; on the host side `server/acquisition.mjs`
|
||||
refuses the llama.cpp and cloudflared fetches (a connector placed in `components/cloudflared/` is used as it is). Models the
|
||||
user chooses to download are a separate, explicit action. `apps/host/tests/no-acquire.test.mjs` runs the host under
|
||||
`tests/helpers/acquire-spy.mjs` and fails on any outbound request or installer process;
|
||||
`apps/desktop/scripts/installed-flow.mjs` drives an installed app end to end (stub engine anywhere, real engine with `--components`).
|
||||
When no package matches, `strata serve` ends with exit 5 and "No engine for this PC was found". It never fetches. Before that point
|
||||
everything is a warning: `strata launch` says it could not find an engine, asks whether to go on, downloads the model and ends with
|
||||
the next step (exit 0, nothing started). The app shows the same warning with an "Update engine" button in Settings.
|
||||
`packaging/package_engine.py` makes a local package from a build folder.
|
||||
|
||||
## What every platform must produce before it can be called qualified
|
||||
**No acquisition at run time.** `STRATA_NO_ACQUIRE=1` (set by the Tauri shell for the host, and by the host for Control) makes
|
||||
`strata_control/acquisition.py` refuse pip, downloads and compiles. On the host side `server/acquisition.mjs` refuses the llama.cpp
|
||||
and cloudflared fetches (a connector placed in `components/cloudflared/` is used as it is). Models the user chooses to download are a
|
||||
separate, explicit action. `apps/host/tests/no-acquire.test.mjs` fails on any outbound request or installer process.
|
||||
`apps/desktop/scripts/installed-flow.mjs` drives an installed app end to end.
|
||||
|
||||
## The Terminal archive
|
||||
|
||||
The Terminal bundles a private Python and its dependencies for `windows-x86_64`, `linux-x86_64` and `darwin-arm64`
|
||||
(`package_terminal.py --target ...`, built on the target OS).
|
||||
|
||||
- The interpreter is python-build-standalone CPython 3.12.15, digest-checked at build time.
|
||||
- Packages come from `packaging/locks/runtime-<target>-cp312.txt` and are installed with
|
||||
`pip install --require-hashes --no-deps --only-binary=:all:` at build time only. `lock_wheels.py` makes the lock; review its diff when a pin changes.
|
||||
- Wrappers: `bin/strata.cmd` and `bin/strata.ps1` on Windows, `bin/strata` on POSIX. `START-HERE.bat`, `SETUP.bat`, `UPDATE.bat`, `setup.sh`
|
||||
and `update.sh` use a `python/` folder beside them when there is one.
|
||||
- `smoke_terminal.py <archive>` extracts an archive into a clean environment (a path with a space, a PATH with no Python) and runs
|
||||
`--version`, `doctor`, `profile list`, `install` and `serve --engine mock`.
|
||||
- Check each archive's `BUNDLE.json` and lock digests. An old archive is not upgraded when the source lock changes.
|
||||
|
||||
## What a platform must produce before it is qualified
|
||||
|
||||
1. The app launches and the window serves the supplied UI from the host's origin.
|
||||
2. The owner transport works: one owner per installation, a second client attaches, a stale owner
|
||||
can be recovered, and no client can stop a process it does not own (R03/R04).
|
||||
2. The owner transport works: one owner per installation, a second client attaches, a stale owner can be recovered, and no client can stop a process it does not own.
|
||||
3. Real inference on a supported pack, with the engine build hash and the model hashes recorded.
|
||||
4. `docs/work/QUALIFICATION-KIT.md` run end to end, with raw evidence retained.
|
||||
4. `docs/work/QUALIFICATION-KIT.md` run end to end, with raw evidence kept.
|
||||
|
||||
## Honest gaps in this directory
|
||||
## Release checks and gaps
|
||||
|
||||
* There is no CI. Every recipe here is written to be run by hand; making them automated is separate
|
||||
work.
|
||||
* Signing needs the owner's credentials, which are never in this tree. The update feed and every engine package are signed with the feed key
|
||||
(`feed_tool.py`, [UPDATES.md](UPDATES.md)); the app installers with the Tauri updater key. The Mac app and Terminal archive are signed and
|
||||
notarised on a Mac that holds the Developer ID certificate, with `packaging/mac/sign-and-notarize.sh` ([mac/README.md](mac/README.md)); for a release, `--updater-archive` also packs the notarised app as `Strata.app.tar.gz` for the in-app Mac update (the one `tauri build` makes holds the unsigned app), signed with the Tauri key on the PC and added with `feed_tool.py set-app-platform`. No
|
||||
Windows code-signing certificate is configured in this tree; Windows may warn about an installer that is not signed.
|
||||
* No font or asset redistribution evidence has been gathered (D05). The app currently ships the
|
||||
brand assets from the supplied UI ZIP, which is fine for an internal candidate and not sufficient
|
||||
for redistribution.
|
||||
* The terminal distribution bundles a private Python runtime and its dependencies for `windows-x86_64`,
|
||||
`linux-x86_64` and `darwin-arm64` (`package_terminal.py --target ...`, built on the target OS). The interpreter is
|
||||
the pinned python-build-standalone CPython 3.12.15 (digest-checked at build time); the packages come from
|
||||
`packaging/locks/runtime-<target>-cp312.txt`, installed with `pip install --require-hashes --no-deps
|
||||
--only-binary=:all:` at BUILD time only (`lock_wheels.py` makes the lock; review its diff when a pin changes).
|
||||
Wrappers: `bin/strata.cmd` / `bin/strata.ps1` on Windows, `bin/strata` on POSIX; START-HERE.bat, SETUP.bat,
|
||||
UPDATE.bat, setup.sh and update.sh use a `python/` folder beside them when there is one.
|
||||
`smoke_terminal.py <archive>` extracts an archive into a clean environment (a path with a space, a PATH with no
|
||||
Python) and runs `--version`, `doctor`, `profile list`, `install` and `serve --engine mock`. The macOS archive
|
||||
predates the lock: build it again on a Mac to get the hash-pinned install (the lock is already here).
|
||||
* `.gitignore` coverage for build output was added late. `apps/desktop/src-tauri/target` and the
|
||||
Node sidecar were briefly committed, which left ~324 MB in the git object store. The files are
|
||||
untracked now and the ignore rules are in place, but the history is not rewritten: that is a
|
||||
deliberate choice, since rewriting published history is worse than a large local repository. A
|
||||
developer cloning this branch gets the current tree, and the sidecar arrives via
|
||||
`fetch-node.sh`.
|
||||
- CI covers scripts, Terminal checks, Metal builds and Docker images ([.github/workflows](../.github/workflows)). It does not replace native desktop
|
||||
packaging, signing or real-GPU acceptance on every target. Some recipes still run by hand.
|
||||
- Signing needs the owner's credentials, which are never in this tree. The feed and every engine package are signed with the feed key
|
||||
(`feed_tool.py`); the app installers with the Tauri updater key ([UPDATES.md](UPDATES.md)). The Mac app and Terminal archive are signed and
|
||||
notarised on a Mac that holds the Developer ID certificate. No Windows code-signing certificate is configured, so Windows may warn about the installer.
|
||||
- Check the licenses and provenance of bundled fonts and assets when packaging. The early D05 checklist lacked redistribution evidence;
|
||||
treat that as an open checklist finding, not an audit of every release asset.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
- **Tauri config:** an early draft used `perMachine`, `oneClick` and `deleteAppDataOnUninstall`. The Tauri schema rejects them; the
|
||||
current config uses `installMode: currentUser`.
|
||||
- **Repository size:** `apps/desktop/src-tauri/target` and the Node sidecar were briefly committed, which left about 324 MB in the git object
|
||||
store. They are untracked and ignored now. History was not rewritten.
|
||||
- **Mac packaging tests:** the first DMG self-test (8/8) and the engine's 17 kernel-parity suites ran on an Apple M4 GPU in a 16 GiB Mac, with service stand-ins for signing.
|
||||
Later inference measurements on larger Macs are in [MAC.md](../docs/MAC.md).
|
||||
- **Intel Arc:** official since 0.1.40.2. The 2026-10 B70 tests used the source build, not the engine package. `sycl/setup_intel.py` (Docker) remains the source-build route.
|
||||
+41
-22
@@ -1,10 +1,14 @@
|
||||
# Updates: the signed feed, the component updater, the app updater
|
||||
|
||||
Everything that downloads software on a user's PC lives behind one explicit step: `strata update` in the Terminal, or
|
||||
"Update engine" / "Update Strata" in the app's Settings. Starting a model and serving never download anything
|
||||
This page is for developers who build, sign and publish releases. Users update with [INSTALL.md](../docs/INSTALL.md#updating).
|
||||
It covers the three updated parts, the feed format, what a PC checks, and the release steps.
|
||||
|
||||
Everything that downloads software on a user's PC sits behind one explicit step: `strata update` in the Terminal, or
|
||||
"Update engine" or "Update Strata" in the app's Settings. Starting a model and serving never download anything
|
||||
(`STRATA_NO_ACQUIRE` stays on for them; the update step is a separate Control process that is allowed to). Engines are
|
||||
separate from the app and ship more often: an app-only update downloads 0 engine bytes (the update installers are built without the engine), an engine update downloads only the
|
||||
package for that PC's GPU. **Models and saved tuning presets are never downloaded, replaced or deleted by any update.**
|
||||
separate from the app and ship more often. An app-only update downloads no engine bytes, because the Windows and Linux installers are built
|
||||
without the engine. An engine update downloads only the package for that PC's GPU.
|
||||
**Models and saved tuning presets are never downloaded, replaced or deleted by any update.**
|
||||
|
||||
## The three things that are updated
|
||||
|
||||
@@ -19,33 +23,41 @@ Two separate private keys, both kept by the owner, neither in git, neither the s
|
||||
* **feed key**: Ed25519, signs `strata-feed.json` and every engine package manifest.
|
||||
* **Tauri updater key**: minisign (what `tauri signer generate` makes), signs the app installers.
|
||||
|
||||
<a id="the-feed"></a>
|
||||
## The feed (`strata-feed.json` + `strata-feed.json.sig`)
|
||||
|
||||
`strata-feed.json.sig` is `{"alg": "ed25519", "key_id": "<16 hex>", "value": "<base64>"}` over the **exact bytes** of
|
||||
`strata-feed.json` (no canonicalisation). The feed:
|
||||
`strata-feed.json` (no canonicalisation). The annotated example below illustrates the format; it is not a current signed feed.
|
||||
Comments are for this document only; actual feed files must be strict JSON.
|
||||
|
||||
```json
|
||||
```jsonc
|
||||
{
|
||||
"schema_version": 1, "feed": "strata-updates", "channel": "stable",
|
||||
"sequence": 12, // goes up with every feed; an older one than the last accepted is refused
|
||||
"generated_at": "2026-10-20T10:00:00Z", "expires_at": "2027-10-20T10:00:00Z", // expires_at may be null
|
||||
"app": { // exactly Tauri's static updater JSON, plus sha256/size per platform
|
||||
"version": "0.2.0", "notes": "...", "pub_date": "2026-10-20T10:00:00Z",
|
||||
"platforms": {"windows-x86_64": {"url": "app/Strata_0.2.0_x64-setup.exe", "signature": "<minisign, base64>", "sha256": "...", "size": 71000000}}
|
||||
"version": "0.2.5", "notes": "...", "pub_date": "2026-10-20T10:00:00Z",
|
||||
"platforms": {"windows-x86_64": {"url": "app/Strata-0.2.6-windows-x64-setup.exe", "signature": "<minisign, base64>", "sha256": "...", "size": 71000000}}
|
||||
},
|
||||
"control": {"version": "0.1.48", "packages": [{"platform": "linux-x86_64", "format": "tar.gz", "url": "control/...tar.gz",
|
||||
"control": {"version": "0.1.50", "packages": [{"platform": "linux-x86_64", "format": "tar.gz", "url": "control/...tar.gz",
|
||||
"size": 0, "sha256": "...", "installed_bytes": 0, "entry": "strata.py"}]},
|
||||
"engine": {"packages": [{
|
||||
"version": "0.1.48", "platform": "linux-x86_64", "backend": "cuda", "variant": "cuda13",
|
||||
"version": "0.1.50", "platform": "linux-x86_64", "backend": "cuda", "variant": "cuda13",
|
||||
"archs": ["sm_75", "sm_86", "sm_89", "sm_120", "compute_120"], "min_driver": {"nvidia": "580.65"}, "min_runtime": null,
|
||||
"build_id": "be885026f96a-cuda13.0", "format": "tar.gz", "url": "engine/strata-engine-0.1.48-linux-x86_64-cuda-cuda13.tar.gz",
|
||||
"build_id": "be885026f96a-cuda13.0", "format": "tar.gz", "url": "engine/strata-engine-0.1.50-linux-x86_64-cuda-cuda13.tar.gz",
|
||||
"size": 505568970, // bytes of the archive
|
||||
"sha256": "...", // of the archive
|
||||
"manifest_sha256": "...", // of the package's engine-component.json (it holds every file's hash)
|
||||
"installed_bytes": 840893419, "notes": ""}]}
|
||||
"installed_bytes": 840893419, "notes": ""}]},
|
||||
"dsv41": {"packages": []} // optional: the separate DeepSeek engine packages, same entry shape as "engine"
|
||||
}
|
||||
```
|
||||
|
||||
The DeepSeek V4.1-Flash engine `strata-dsv41` reaches a PC in one of two ways. The engine package carries it beside `strata`
|
||||
(`package_engine.py --require-dsv41`), or it is a separate package in the feed's `dsv41` member, packed with
|
||||
`pack-engine --prefix strata-dsv41`. The separate package installs in `<data>/engine-dsv41/` beside the engine, never inside it
|
||||
(`strata_control/dsv41_engine.py`), and is fetched when DeepSeek is chosen. `STRATA_DSV41_EXE` points setup at a build of your own.
|
||||
|
||||
`url` is relative to the feed's own location (or an absolute `https://` address; a loopback `http://` is allowed so a local test
|
||||
feed works; a local feed folder may only name files inside itself).
|
||||
For a GitHub release the assets are flat: pack with `--url-prefix ""` and upload every file beside the feed.
|
||||
@@ -98,7 +110,7 @@ On an offline machine, outside any git checkout:
|
||||
npx tauri signer generate -w D:\strata-keys\tauri-updater.key # asks for a password; prints the public key
|
||||
|
||||
Back both up (a lost feed key means every installed PC needs a build with a new key; a lost Tauri key means the same for the app).
|
||||
Then, in git, once, in the commit that ships the first signed release (done for 0.2.0, key id 5d1bc23abbeffd0d):
|
||||
Then, in git, once, in the commit that ships the first signed release (the release key id is `5d1bc23abbeffd0d`):
|
||||
|
||||
* the `keys` entry from `feed-key.pub.json` into `strata_control/update_keys.json`;
|
||||
* the Tauri public key into `apps/desktop/src-tauri/tauri.conf.json` (`plugins.updater.pubkey`) and into `update_keys.json` as `app_minisign_public`
|
||||
@@ -110,7 +122,7 @@ without their key being part of a release build. A build whose `update_keys.json
|
||||
feed with "this build trusts no update key".
|
||||
A key rotation is a release that carries both public keys; the old private key signs that release, the new one signs the next.
|
||||
|
||||
## Releasing (when the owner says go)
|
||||
## Releasing
|
||||
|
||||
Keys live only on the release PC (`feed-key.json`, `tauri-updater.key`; the Tauri key has no password). Every build box builds WITHOUT a key:
|
||||
the default `tauri.conf.json` has `createUpdaterArtifacts: false`, so `tauri build` needs no signing key and makes the same installer bytes.
|
||||
@@ -119,7 +131,7 @@ All signing happens on the release PC, on the finished files.
|
||||
**What is built (per OS)**
|
||||
|
||||
* *The installer*, one per OS (0.2.5). The Windows and Linux ones have no engine inside: built with `--config src-tauri/tauri.update.conf.json`, which drops `engine-bundle`. The Mac dmg keeps its small Metal engine (one Mac variant).
|
||||
Windows: `Strata-0.2.5-windows-x64-setup.exe`; Linux: `Strata-0.2.5-linux-x86_64.AppImage`; Mac: `Strata-0.2.5-macos-arm64.dmg` for people and
|
||||
Windows: `Strata-0.2.6-windows-x64-setup.exe`; Linux: `Strata-0.2.6-linux-x86_64.AppImage`; Mac: `Strata-0.2.6-macos-arm64.dmg` for people and
|
||||
`Strata.app.tar.gz` as the updater archive (the updater cannot install from a dmg). The Windows and Linux files are both the first download and the
|
||||
file `latest.json` names; the first start of the app downloads the engine for the card (the first-run card), or installs a file the person already has.
|
||||
Why: an app update must not carry an engine, and the right engine depends on the card. Engines come per card from the signed feed
|
||||
@@ -129,22 +141,22 @@ All signing happens on the release PC, on the finished files.
|
||||
|
||||
python packaging/feed_tool.py sign-package <engine package folder> --key <feed-key.json> # every engine package (the Mac app bundles its UNSIGNED copy)
|
||||
python packaging/feed_tool.py pack-engine <engine package folder> --out RELEASE/engine --url-prefix=
|
||||
python packaging/feed_tool.py pack-control <control folder> --version 0.1.48 --platform P --out RELEASE/control --url-prefix= # --format zip for windows-x86_64
|
||||
python packaging/feed_tool.py pack-control <control folder> --version 0.1.50 --platform P --out RELEASE/control --url-prefix= # --format zip for windows-x86_64
|
||||
# --url-prefix= (not --url-prefix "") because PowerShell 5.1 drops an empty argument. The Control version is the VERSION file
|
||||
# (0.1.48), not the app version: a Control package newer than the installed one is installed by `strata update`.
|
||||
# (0.1.50), not the app version: a Control package newer than the installed one is installed by `strata update`.
|
||||
# Pack Linux and Mac packages on Linux / the Mac or on Windows with this tool: it sets the execute bit from the file header either way.
|
||||
cd apps/desktop && npx tauri build --bundles nsis --config src-tauri/tauri.update.conf.json # Windows installer (no engine inside)
|
||||
# Linux: tauri build --bundles appimage (no engine inside) is the AppImage
|
||||
# on the release PC, per updater artifact (Windows setup .exe, Linux .AppImage, Mac Strata.app.tar.gz):
|
||||
npx tauri signer sign -f <path to tauri-updater.key> <artifact> # writes <artifact>.sig
|
||||
python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target windows-x86_64 --version 0.2.5 --archive Strata-0.2.5-windows-x64-setup.exe --sig Strata-0.2.5-windows-x64-setup.exe.sig --notes "..."
|
||||
python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target linux-x86_64 --version 0.2.5 --archive Strata-0.2.5-linux-x86_64.AppImage --sig Strata-0.2.5-linux-x86_64.AppImage.sig
|
||||
python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target windows-x86_64 --version 0.2.5 --archive Strata-0.2.6-windows-x64-setup.exe --sig Strata-0.2.6-windows-x64-setup.exe.sig --notes "..."
|
||||
python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target linux-x86_64 --version 0.2.5 --archive Strata-0.2.6-linux-x86_64.AppImage --sig Strata-0.2.6-linux-x86_64.AppImage.sig
|
||||
python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target darwin-aarch64 --version 0.2.5 --archive Strata.app.tar.gz --sig Strata.app.tar.gz.sig
|
||||
python packaging/feed_tool.py build-feed --out RELEASE --sequence N --key <feed-key.json> --engine RELEASE/engine --control RELEASE/control --app-json RELEASE/tauri-latest.json
|
||||
python packaging/feed_tool.py build-feed --out RELEASE --sequence N --key <feed-key.json> --engine RELEASE/engine --control RELEASE/control --app-json RELEASE/tauri-latest.json [--dsv41 RELEASE/dsv41]
|
||||
python packaging/feed_tool.py verify-feed RELEASE --pub <feed-key.pub.json>
|
||||
|
||||
`set-app-platform` checks each `.sig` against the public key in `tauri.conf.json` over the exact bytes, so a file signed with another key, or changed after
|
||||
signing, stops the release. The full installers are not in `latest.json` and need no Tauri signature (they are downloaded by hand).
|
||||
signing, stops the release. The Mac dmg is not in `latest.json` (the updater uses `Strata.app.tar.gz`) and needs no Tauri signature; it is downloaded by hand.
|
||||
|
||||
### The Mac app
|
||||
|
||||
@@ -163,7 +175,7 @@ Mach-O in the app with the Developer ID and notarises and staples it, which chan
|
||||
The Windows installer and the Linux AppImage need no extra step: their `.sig` files from `tauri build` are the ones to pass to `set-app-platform`. Re-signing the Mac app after the archive
|
||||
was made (a second `sign-and-notarize.sh` run) makes a new archive: sign and add it again.
|
||||
|
||||
`build-feed` also writes `RELEASE/install-manifest.txt`, the file the one-line installers read (`install.sh` L76, `install.ps1` L45): one line per Control package,
|
||||
`build-feed` also writes `RELEASE/install-manifest.txt`, the file the one-line installers read: one line per Control package,
|
||||
`platform version url sha256 size`, taken from the `*.entry.json` files in `RELEASE/control`. The Control package is the Terminal that `install.sh` unpacks (it must hold `bin/strata`).
|
||||
`install.sh` unpacks with `tar -xzf` and `install.ps1` with `Expand-Archive`, so `windows-x86_64` must be packed with `--format zip` and the other platforms as `tar.gz`;
|
||||
`build-feed` leaves a pack in the wrong format out of the manifest and says so. Pack Control for every platform you want the one-line install to cover (`install.sh` knows `linux-x86_64` and `macos-arm64`, `install.ps1` `windows-x86_64`);
|
||||
@@ -185,3 +197,10 @@ Everything above runs against a folder or a loopback server: `tests/terminal/tes
|
||||
`tests/terminal/test_updater.py` (end to end against a loopback HTTP feed with stub engines: update, test-start, switch; a failing test start
|
||||
keeps the old engine; an interrupted download resumes; rollback; Control; the app installer's minisign check), `apps/host/tests/component-updates.test.mjs`
|
||||
(the Settings buttons). For a real engine on a box, build a package with `packaging/package_engine.py`, serve it from a local feed made by `packaging/make_local_feed.py`, and run `strata update --feed` against it.
|
||||
|
||||
## History / older measurements
|
||||
|
||||
- **Keys:** the feed key and the Tauri updater key were created once and the feed key's public half was committed in `strata_control/update_keys.json`
|
||||
with the first signed release (0.2.0-beta). Terminals from before that build trust no key and refuse every feed.
|
||||
- **Engine-less installers:** an app update once carried an engine. The Windows and Linux installers now drop `engine-bundle`
|
||||
(`tauri.update.conf.json`), so one `latest.json` entry per OS serves every card variant.
|
||||
@@ -539,6 +539,13 @@ def tool_arguments(raw) -> dict:
|
||||
return {"arguments": json.dumps(raw, ensure_ascii=False)}
|
||||
|
||||
|
||||
def require_messages(messages: list) -> None:
|
||||
"""A chat request with no message at all is a 400 (OpenAI: "empty array. Expected an array with minimum length 1"); the
|
||||
template would otherwise render just the assistant turn and the model would answer nothing."""
|
||||
if not messages:
|
||||
raise ValueError("messages must be a non-empty list: send at least one message")
|
||||
|
||||
|
||||
def openai_to_messages(req: dict) -> tuple[list[dict], list[dict] | None, dict]:
|
||||
"""OpenAI Chat Completions -> (template messages, template tools, template kwargs)."""
|
||||
messages = []
|
||||
|
||||
+4
-3
@@ -61,7 +61,7 @@ ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tools"))
|
||||
sys.path.insert(0, str(ROOT)) # run as a script (run-<model>.bat) as well as a module
|
||||
from serve.frontend import (THINK_END, ChatTemplate, Event, OutputParser, anthropic_to_messages, # noqa: E402
|
||||
forced_call, images_of, literal_tags, mark_think_literals, openai_to_messages,
|
||||
forced_call, images_of, literal_tags, mark_think_literals, openai_to_messages, require_messages,
|
||||
tool_choice_of, unmark_think_literals)
|
||||
from serve.mcp import McpCancelled, hub_from_config # noqa: E402
|
||||
from serve import runconfig # noqa: E402
|
||||
@@ -5516,7 +5516,7 @@ class Service:
|
||||
"engine_generated": last.get("generated"),
|
||||
"prompt_ms": last.get("prompt_ms"), "decode_ms": last.get("decode_ms"),
|
||||
"decode_tok_s": round(last["generated"] / (last["decode_ms"] / 1000), 1)
|
||||
if n and last.get("generated") and last.get("decode_ms") else None,
|
||||
if n and (last.get("generated") or 0) >= 2 and last.get("decode_ms") else None,
|
||||
"hit_rate": hit_rate, "pcie_share": pcie_share, "ram_blobs": last.get("ram_blobs"),
|
||||
"file_blobs": last.get("file_blobs"), "file_mb": last.get("file_mb"),
|
||||
# #457: the speculative drafts from the DONE line (None: the engine did not say)
|
||||
@@ -5650,7 +5650,7 @@ def answer_speed(last: dict, generated: int, wall_s: float | None = None) -> flo
|
||||
Only an engine that sent no DONE falls back to the server's own clock from the first token, and a clock younger than
|
||||
RATE_MIN_SPAN_S says nothing (it printed 1000000 tok/s for a one-token reply). Prompt reading is never in it."""
|
||||
produced, decode_ms = last.get("generated"), last.get("decode_ms")
|
||||
if produced and decode_ms and decode_ms > 0:
|
||||
if produced and produced >= 2 and decode_ms and decode_ms > 0: # one token has no first-to-last span (a 1-token probe printed 1,250 tok/s)
|
||||
return round(produced / (decode_ms / 1000), 1)
|
||||
if generated and wall_s is not None and wall_s >= RATE_MIN_SPAN_S:
|
||||
return round(generated / wall_s, 1)
|
||||
@@ -7136,6 +7136,7 @@ def make_handler(svc: Service):
|
||||
req = svc.with_shared(req, "openai")
|
||||
bias_api.normalize(req.get("logit_bias"), len(getattr(svc.tok, "tokens", ())) or None)
|
||||
messages, tools, kw = openai_to_messages(req)
|
||||
require_messages(messages) # an empty conversation is a 400, as OpenAI answers it, not a reply to nothing
|
||||
self._no_local_images(messages)
|
||||
if tool_choice_of(req.get("tool_choice"))[0] == "none": # as the Responses route: no tools are offered
|
||||
tools = None
|
||||
|
||||
@@ -28,6 +28,7 @@ class AnswerSpeed(unittest.TestCase):
|
||||
self.assertEqual(answer_speed({**self.REAL, "drafts_offered": 500}, 73), 65.9)
|
||||
|
||||
def test_a_one_token_reply_has_no_absurd_rate(self):
|
||||
self.assertIsNone(answer_speed({"generated": 1, "decode_ms": 0.8}, 1, None)) # a 1-token probe: 1 / 0.8 ms was 1,250 tok/s
|
||||
self.assertIsNone(answer_speed({}, 1, 1e-6))
|
||||
self.assertIsNone(answer_speed({}, 1, None))
|
||||
|
||||
|
||||
+10
-1
@@ -3,7 +3,7 @@ import copy
|
||||
from pathlib import Path
|
||||
import unittest
|
||||
|
||||
from serve.frontend import ChatTemplate, anthropic_to_messages, openai_to_messages
|
||||
from serve.frontend import ChatTemplate, anthropic_to_messages, openai_to_messages, require_messages
|
||||
from serve.server import ByteTokenizer, MockEngine, Service
|
||||
|
||||
|
||||
@@ -340,3 +340,12 @@ class FencedJsonCallInWrapper(unittest.TestCase):
|
||||
other = [{"name": "other", "parameters": {}}]
|
||||
calls, text = self.run_parser(raw, len(raw), False, tools=other)
|
||||
self.assertEqual((calls, text), ([], raw))
|
||||
|
||||
|
||||
class EmptyConversation(unittest.TestCase):
|
||||
def test_no_message_is_an_error_a_message_is_not(self):
|
||||
with self.assertRaises(ValueError):
|
||||
require_messages(openai_to_messages({"messages": []})[0])
|
||||
with self.assertRaises(ValueError):
|
||||
require_messages(openai_to_messages({})[0])
|
||||
require_messages(openai_to_messages({"messages": [{"role": "user", "content": "hi"}]})[0])
|
||||
@@ -7847,6 +7847,18 @@ def dsv41_engine_dir(eng: Path) -> Path:
|
||||
return (home() if CONTROL else Path(eng).parent) / DSV41_ENGINE_DIR
|
||||
|
||||
|
||||
def local_feed(feed: str | None = None) -> bool:
|
||||
"""Is the update feed this run reads a local one (a folder, a file or a loopback address, as a test or an offline copy uses)? Then
|
||||
nothing is downloaded from GitHub and its being slow or down must not stop the run. `feed` is --feed, else STRATA_UPDATE_FEED."""
|
||||
src = (feed or os.environ.get("STRATA_UPDATE_FEED") or "").strip()
|
||||
if not src:
|
||||
return False
|
||||
if not re.match(r"^[a-z][a-z0-9+.-]*://", src, re.I) or src.lower().startswith("file://"):
|
||||
return True # a path (a Windows C:\... one too)
|
||||
host = (urllib.parse.urlparse(src).hostname or "").lower()
|
||||
return host in ("localhost", "127.0.0.1", "::1") or host.startswith("127.")
|
||||
|
||||
|
||||
def github_reachable(timeout: float = 10.0) -> bool:
|
||||
"""Does GitHub answer within `timeout` seconds (any HTTP answer counts; no answer, a refused or timed-out connection does not)? One quick check
|
||||
instead of minutes of retries when the connection is down (a SYN that is never answered)."""
|
||||
@@ -8143,7 +8155,8 @@ def install_dsv41(a, family, gpu, hip, amd, cuda_tk, multi, port, ram, adopted,
|
||||
step(4, "the DeepSeek engine")
|
||||
eng, meta, lib_dirs = ROOT / "engine", {"version": None}, []
|
||||
exe = dsv41_exe(eng)
|
||||
if exe is None and not getattr(a, "dsv41_package", None) and not dsv41_engine_pending(eng) and not github_reachable():
|
||||
if (exe is None and not getattr(a, "dsv41_package", None) and not dsv41_engine_pending(eng) and not local_feed(getattr(a, "feed", None))
|
||||
and not github_reachable()):
|
||||
fail("Strata can't reach GitHub to download the DeepSeek engine.",
|
||||
"Check the internet connection and run the same command again. " + dsv41_manual_steps(eng))
|
||||
if exe is None and not getattr(a, "build", False):
|
||||
@@ -8842,6 +8855,11 @@ def main() -> int:
|
||||
else "the amdgpu driver's KFD topology is empty)."))
|
||||
for g in amd:
|
||||
say(f" GPU {g['index']}: {g['name']}, {amd_mem_text(g)} - " + (amd_problem(g) or "can be used"))
|
||||
if not WIN: # WSL: the Linux HIP engines hang there; a warning, the run goes on
|
||||
from strata_control.engine_component import wsl_amd_note
|
||||
wsl_note = wsl_amd_note("hip", wsl=is_wsl())
|
||||
if wsl_note:
|
||||
warn(wsl_note)
|
||||
usable = [g for g in amd if amd_problem(g) is None]
|
||||
if not WIN and (amd or amd_pci_devices()): # a warning only: recommend, never force
|
||||
acc = amd_device_access_problem()
|
||||
|
||||
@@ -50,6 +50,15 @@ inline int prefill_chunk_default(bool own_default, double gpu_share, bool gguf_c
|
||||
return file_experts ? big : 2048;
|
||||
}
|
||||
|
||||
// --mmap-experts auto copies all the CPU lane's experts into RAM at load (no demand store) only when they fit: lane_bytes of experts,
|
||||
// ram_avail the RAM this process can still get (~0: unknown -> no), ram_margin left to the rest, commit_room what Windows still lets it
|
||||
// commit beside its margin (1e300: no such limit, Linux). All or none: a lane that does not fit both leaves the demand store as it was.
|
||||
// Measured 1 x R9700 + 251 GB, 2.00 pack: decode +10-25% (the copies sit on huge pages, the store's page-locked arena does not).
|
||||
inline bool copy_all_fits(double lane_bytes, uint64_t ram_avail, double ram_margin, double commit_room) {
|
||||
if (!(lane_bytes > 0.0) || ram_avail == ~uint64_t{0}) return false;
|
||||
return lane_bytes + ram_margin <= (double) ram_avail && lane_bytes <= commit_room;
|
||||
}
|
||||
|
||||
// DS4_TOPK=N (a test, not for users): over the flag and the rule; returns the count to use
|
||||
inline int experts_test_override(int chosen, int model_k) {
|
||||
if (const char* v = std::getenv("DS4_TOPK")) {
|
||||
|
||||
+35
-3
@@ -1978,7 +1978,7 @@ struct Model::Impl {
|
||||
// most (4 x R9700: 8192 214, 1024 370) the small one wins; with the experts copied into RAM (--mmap-experts off) 4096 beats 8192 by 7-9%
|
||||
// (H100 32 GiB: 2048 502, 4096 538, 8192 495). A GGUF pack always has file_experts set (copied only by --mmap-experts off).
|
||||
const bool own_default = !(par || o.tp > 1); // (a single-card run: no --threads, tp 1 - a launch with several cards passes --tp 1; one card passes neither)
|
||||
const bool gguf_copy = pk.gguf && o.mmap_experts == 0;
|
||||
const bool gguf_copy = pk.gguf && (o.mmap_experts == 0 || (store_gb() <= 0.0 && auto_copy_all));
|
||||
uint64_t vmin_bytes = ~0ull;
|
||||
for (const Dev& dv : devs) vmin_bytes = std::min<uint64_t>(vmin_bytes, (uint64_t) dv.total);
|
||||
if (devs.empty()) vmin_bytes = 0;
|
||||
@@ -4275,12 +4275,18 @@ struct Model::Impl {
|
||||
// (auto: none - copies that all fit were measured no faster than the page cache: the CPU lane's rows are
|
||||
// compute-bound, ~46 GB/s from either on 2 Xeon sockets, NUMA parts included; a part copied cost decode speed
|
||||
// on 64 GB and on 32 GB of RAM)
|
||||
double budget = o.mmap_experts == 0 ? std::min(all, can) : 0.0;
|
||||
double budget = o.mmap_experts == 0 ? std::min(all, can) : auto_copy_all && all <= can ? all : 0.0; // (auto: all or none)
|
||||
if (const char* v = std::getenv("DS4_COPY_GB")) budget = std::atof(v) * 1e9;
|
||||
if (o.mmap_experts < 0 && auto_copy_all && all > can)
|
||||
std::fprintf(stderr, "ds4: the CPU lane's experts (%.1f GB) do not all fit the RAM this process can get now (%.1f GB beside %.0f GB for "
|
||||
"the rest): read in place from the model's files (--mmap-experts off copies what fits)\n", all / 1e9, can / 1e9, kRamMargin / 1e9);
|
||||
if (o.mmap_experts == 0 && all > can)
|
||||
std::fprintf(stderr, "ds4: --mmap-experts off: the CPU lane's experts (%.1f GB) do not all fit the RAM this "
|
||||
"process can get (%.1f GB beside %.0f GB for the rest) - the most-routed copied, the "
|
||||
"others read in place\n", all / 1e9, can / 1e9, kRamMargin / 1e9);
|
||||
if (auto_copy_all && budget > 0)
|
||||
std::fprintf(stderr, "ds4: --mmap-experts auto: the CPU lane's experts (%.1f GB) fit the RAM this process can get (%.1f GB beside %.0f GB for "
|
||||
"the rest): copied in at load, no demand store (--mmap-experts on keeps them in place)\n", all / 1e9, can / 1e9, kRamMargin / 1e9);
|
||||
if (budget <= 0) {
|
||||
if (o.verbose)
|
||||
std::fprintf(stderr, "ds4: the CPU lane's experts (%.1f GB) stay in place: the RAM this process can get "
|
||||
@@ -4565,6 +4571,22 @@ struct Model::Impl {
|
||||
// than they are is never filled), never less than the old 48 GB. A PC whose free RAM holds them all (the 251 GB R9700 box:
|
||||
// 115 GB of CPU-lane experts, 1 card, 2.00 set: 12.6 -> 21.4 tokens/s with the whole set resident, misses 15.4 -> 1.5 a token)
|
||||
// gets every one; the RAM that the room below leaves (free - 12 GB) is what bounds it elsewhere.
|
||||
// Whether the experts the GPUs will not hold (the CPU lane's: the pack's less the GPUs' share) all fit the RAM this process can get beside
|
||||
// kRamMargin, so that --mmap-experts auto copies them in (a copied expert is the CPU lane's own memory: no page cache to lose to the
|
||||
// Engram tables and the files' churn, no store indirection or miss path). Measured, 1 x R9700 + 251 GB, 2.00 pack, warm: decode +10-15%.
|
||||
// --mmap-experts on/off and --ssd-cache N stand as given.
|
||||
bool cpu_lane_fits_ram() const {
|
||||
if (!pk.gguf || o.mmap_experts >= 0 || o.ssd_cache_gb >= 0.0 || std::getenv("DS4_STORE_GB") || std::getenv("DS4_COPY_GB")) return false;
|
||||
if (const char* v = std::getenv("DS4_COPY_ALL")) return std::atoi(v) != 0 && ram_avail() != ~uint64_t{0};
|
||||
const uint64_t avail = ram_avail();
|
||||
if (avail == ~uint64_t{0}) return false;
|
||||
double exp_b = 0, room = 0, other_b = 0;
|
||||
const double share = gpu_expert_share(&exp_b, &room, &other_b);
|
||||
if (exp_b <= 0) return false;
|
||||
const double lane = exp_b * (1.0 - std::min(1.0, share));
|
||||
return copy_all_fits(lane, avail, (double) kRamMargin, commit_room()); // (Windows: the commit room too, not only the free RAM)
|
||||
}
|
||||
mutable bool auto_copy_all = false; // (set by store_gb(): the auto size is the CPU lane's experts copied in, no demand store)
|
||||
double store_cap_gb() const {
|
||||
double b = 0;
|
||||
for (const auto& kt : pk.t)
|
||||
@@ -4577,7 +4599,10 @@ struct Model::Impl {
|
||||
double gb = o.ssd_cache_gb;
|
||||
if (const char* v = std::getenv("DS4_STORE_GB")) gb = std::atof(v);
|
||||
if (!pk.gguf) gb = 0.0;
|
||||
else if (gb < 0.0) { // auto: the RAM free now less 12 GB for the system and the owner's programs, all the experts at most
|
||||
else if (gb < 0.0 && cpu_lane_fits_ram()) { // auto, and the RAM holds every CPU-lane expert beside kRamMargin: they are copied in at load
|
||||
auto_copy_all = true; // (copy_experts), the demand store would only be a second, smaller copy of them
|
||||
gb = 0.0;
|
||||
} else if (gb < 0.0) { // auto: the RAM free now less 12 GB for the system and the owner's programs, all the experts at most
|
||||
const uint64_t avail = ram_avail();
|
||||
avail0_ = avail;
|
||||
const double room = avail == ~uint64_t{0} ? 0.0 : (double) avail / 1e9 - 12.0;
|
||||
@@ -4608,6 +4633,13 @@ struct Model::Impl {
|
||||
void init_store() {
|
||||
const double gb = store_gb_now();
|
||||
if (gb <= 0.0 || L.empty() || L[0].lay.fmt != gg::kFmtMXFP4 || !file_experts) return;
|
||||
if (o.ssd_cache_gb < 0.0 && !std::getenv("DS4_STORE_GB")) { // (the auto size: say when it holds less than the CPU lane's experts)
|
||||
double exp_b = 0;
|
||||
const double share = gpu_expert_share(&exp_b), lane_gb = exp_b * (1.0 - std::min(1.0, share)) / 1e9;
|
||||
if (lane_gb > gb + 1.0 && o.verbose)
|
||||
std::fprintf(stderr, "ds4: RAM store: the auto size (%.1f GB) holds less than the CPU lane's experts (%.1f GB): the hottest are kept, the "
|
||||
"rest is read from the SSD as the prompt picks it (--ssd-cache N sets the size)\n", gb, lane_gb);
|
||||
}
|
||||
std::vector<ExpertStore::LayerGeo> geo((size_t) g.nl); // (the trunk's layers: the drafter's bigger experts stay in place)
|
||||
for (const Layer& l : L) {
|
||||
if (l.il >= g.nl) continue;
|
||||
|
||||
@@ -709,7 +709,7 @@ struct Options {
|
||||
/// Stop when the model emits an end-of-turn token (<|endoftext|> 248044, <|im_end|> 248046, or --eos-ids).
|
||||
bool stop_eos = false;
|
||||
std::vector<int64_t> eos_ids = {248044, 248046};
|
||||
bool spec_split = false; ///< opt-in split verify window (the overlap study: exact, ~7% slower)
|
||||
bool spec_split = false; ///< opt-in split verify window (the overlap study: ~7% slower; not bit-exact against no-split, the batch shape of its windows differs)
|
||||
/// --serve, multi-GPU layer split: "K" or "K1,K2,.." (the first layer of each later stage) or "auto" (placed
|
||||
/// from each GPU's free VRAM); empty = one GPU
|
||||
std::string layer_split;
|
||||
|
||||
@@ -7,7 +7,7 @@ SCHEMA_VERSION = 1
|
||||
|
||||
#: The version people are told: the Strata app's own (apps/desktop/src-tauri/Cargo.toml `version`; tests/terminal/test_tuiqa_version.py keeps the
|
||||
#: two equal, except for a Terminal-only release such as 0.2.3.1 or 0.2.4, whose app stays 0.2.3). `version()` below is the files' build number, which the engine-compatibility checks compare, so it stays numeric.
|
||||
PRODUCT_VERSION = "0.2.5"
|
||||
PRODUCT_VERSION = "0.2.6"
|
||||
|
||||
|
||||
def version() -> str:
|
||||
|
||||
@@ -666,8 +666,22 @@ class Runner:
|
||||
def _cli(self, *args: str) -> list[str]:
|
||||
return [self.python, str(ROOT / "strata.py"), *args]
|
||||
|
||||
def warn_wsl_amd(self) -> None:
|
||||
"""Under WSL the Linux AMD engines hang (no /dev/kfd, librocdxg never completes a GPU command): say so plainly with one next step,
|
||||
and go on (a warning, never a refusal)."""
|
||||
try:
|
||||
from strata_control import engine_component as ec
|
||||
note = ec.wsl_amd_note("hip") and ec.wsl_amd_note(ec.detect_backend()[0]) # the cheap file test first: no GPU tool is run on other systems
|
||||
except Exception: # noqa: BLE001 - a nicety, never a reason to stop
|
||||
return
|
||||
if note:
|
||||
self.emit("engine", note, level="warn")
|
||||
if note not in self.notices:
|
||||
self.notices.append(note)
|
||||
|
||||
# ---- 1. the engine
|
||||
def ensure_engine(self) -> None:
|
||||
self.warn_wsl_amd()
|
||||
if self.multi_file(): # DeepSeek: its own engine package (setup fetches it); the Strata engine is not used
|
||||
self.emit("engine", "DeepSeek runs on its own engine package, so the Strata engine is not downloaded for it.", 100)
|
||||
return
|
||||
|
||||
@@ -58,7 +58,7 @@ def request_speed(r: dict) -> float | None:
|
||||
out, ms = _num(r.get("output_tokens")), _num(r.get("decode_ms"))
|
||||
if r.get("decode_tok_s") is not None and _num(r.get("decode_tok_s")) is not None:
|
||||
return _num(r.get("decode_tok_s"))
|
||||
return round(out / (ms / 1000.0), 1) if out and ms and ms > 0 else None
|
||||
return round(out / (ms / 1000.0), 1) if out and out >= 2 and ms and ms > 0 else None # one token has no decode span (a probe showed 1,250 tok/s)
|
||||
|
||||
|
||||
def _cards(hw: dict, static: dict) -> list[dict]:
|
||||
|
||||
@@ -524,6 +524,22 @@ def _drm_vendor_makes(root: str = "/sys/class/drm") -> set[str]:
|
||||
return {backend for vendor, backend in (("0x10de", "cuda"), ("0x1002", "hip"), ("0x8086", "sycl")) if vendor in vendors}
|
||||
|
||||
|
||||
def wsl_amd_note(backend: str | None, dev_root: str = "/dev", wsl: bool | None = None) -> str | None:
|
||||
"""Under WSL (the GPU is reached through /dev/dxg, there is no /dev/kfd) the Linux AMD (HIP) engines hang: librocdxg 1.2.2
|
||||
never completes a barrier-AND packet, so the first GPU command never returns. A plain sentence with one next step for that
|
||||
case, else None. A warning only: the caller goes on (recommend, never force)."""
|
||||
if backend != "hip":
|
||||
return None
|
||||
if wsl is None:
|
||||
import platform
|
||||
import sys
|
||||
wsl = sys.platform.startswith("linux") and "microsoft" in platform.uname().release.lower()
|
||||
if not wsl or not Path(dev_root, "dxg").exists() or Path(dev_root, "kfd").exists():
|
||||
return None
|
||||
return ("The AMD engines do not run under WSL yet: the driver layer WSL uses for the card never finishes a GPU command, so the engine "
|
||||
"would hang at its first step. Use the Strata app or Terminal for Windows, which run the AMD engine directly.")
|
||||
|
||||
|
||||
def _drm_vendor_backend(root: str = "/sys/class/drm") -> str | None:
|
||||
"""Linux without the vendor tools on PATH (a HIP box that runs ROCm from Python wheels has no rocminfo): the PCI
|
||||
vendor of the display adapters decides. NVIDIA 0x10de -> cuda, AMD 0x1002 -> hip, Intel 0x8086 -> sycl, in that
|
||||
|
||||
@@ -55,7 +55,8 @@ class TemplateTests(unittest.TestCase):
|
||||
@unittest.skipUnless(MAKENSIS.is_file(), "NSIS (the Tauri bundler's copy) is not installed here")
|
||||
class MigrationTests(unittest.TestCase):
|
||||
def run_nsis(self, body: str, work: Path):
|
||||
block = re.search(r"; STRATA-MIGRATE-BEGIN.*?; STRATA-MIGRATE-END", TEMPLATE.read_text(encoding="utf-8"), re.S).group(0)
|
||||
template = TEMPLATE.read_text(encoding="utf-8")
|
||||
block = chr(10).join(re.search(rf"; STRATA-{name}-BEGIN.*?; STRATA-{name}-END", template, re.S).group(0) for name in ("MIGRATE", "STOP", "CLEAN"))
|
||||
script = work / "t.nsi"
|
||||
script.write_text('Unicode true\nOutFile "t.exe"\nRequestExecutionLevel user\nSilentInstall silent\n' + block + "\nSection\n" + body + "\nSectionEnd\n", encoding="utf-8")
|
||||
done = subprocess.run([str(MAKENSIS), "/V1", str(script)], cwd=work, capture_output=True, text=True)
|
||||
@@ -109,5 +110,60 @@ class MigrationTests(unittest.TestCase):
|
||||
self.assertFalse((old / "strata-desktop.exe").exists())
|
||||
|
||||
|
||||
def test_the_uninstaller_stops_only_the_processes_started_from_the_install_folder(self):
|
||||
ping = Path(os.environ.get("SystemRoot", r"C:\Windows")) / "System32" / "PING.EXE"
|
||||
if not ping.is_file():
|
||||
self.skipTest("no PING.EXE to stand in for a running Strata process")
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
base = Path(tmp)
|
||||
inst, other = base / "Programs" / "Strata", base / "Elsewhere"
|
||||
for folder in (inst / "node", other):
|
||||
folder.mkdir(parents=True)
|
||||
mine = inst / "node" / "node.exe"
|
||||
theirs = other / "node.exe"
|
||||
for target in (mine, theirs):
|
||||
target.write_bytes(ping.read_bytes())
|
||||
procs = [subprocess.Popen([str(p), "-t", "127.0.0.1"], stdout=subprocess.DEVNULL) for p in (mine, theirs)]
|
||||
try:
|
||||
self.run_nsis(f'!insertmacro StrataStopOwnProcesses "{inst}"', base)
|
||||
procs[0].wait(timeout=20)
|
||||
self.assertIsNotNone(procs[0].poll(), "the process from the install folder was stopped")
|
||||
self.assertIsNone(procs[1].poll(), "a process of the same name elsewhere is left running")
|
||||
finally:
|
||||
for proc in procs:
|
||||
if proc.poll() is None:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
|
||||
|
||||
def test_files_the_app_made_in_its_own_folders_are_removed_and_the_data_beside_them_is_not(self):
|
||||
"""Strix, standard user: after the app had run, Python bytecode caches under control\\ kept the program folder alive."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
base = Path(tmp)
|
||||
inst = base / "Programs" / "Strata"
|
||||
touch(inst, ["strata-desktop.exe", "control/python/python.exe", "control/python/Lib/__pycache__/argparse.cpython-312.pyc",
|
||||
"control/serve/__pycache__/server.cpython-312.pyc", "host/server/index.mjs", "host/logs/made-at-run.txt", "node/node.exe",
|
||||
"harness/BUNDLE.json", "components/engine/keep.txt", "runtime/keep.json"])
|
||||
self.run_nsis(f'!insertmacro StrataRemoveAppFolders "{inst}"', base)
|
||||
for folder in ("control", "host", "node", "harness"):
|
||||
self.assertFalse((inst / folder).exists(), folder)
|
||||
for name in ("components/engine/keep.txt", "runtime/keep.json", "strata-desktop.exe"):
|
||||
self.assertTrue((inst / name).is_file(), f"kept: {name}")
|
||||
|
||||
def test_the_uninstaller_clears_those_folders_before_the_last_rmdir(self):
|
||||
text = TEMPLATE.read_text(encoding="utf-8")
|
||||
uninstall = text[text.index("Section Uninstall"):]
|
||||
self.assertIn('!insertmacro StrataRemoveAppFolders "$INSTDIR"', uninstall)
|
||||
self.assertLess(uninstall.index("StrataRemoveAppFolders"), uninstall.index('RMDir "$INSTDIR"'))
|
||||
self.assertNotRegex(uninstall, r'RMDir /r "\$INSTDIR"')
|
||||
|
||||
|
||||
class StopOrderTests(unittest.TestCase):
|
||||
def test_the_uninstaller_stops_them_before_it_removes_files(self):
|
||||
text = TEMPLATE.read_text(encoding="utf-8")
|
||||
uninstall = text[text.index("Section Uninstall"):]
|
||||
self.assertLess(uninstall.index("StrataStopOwnProcesses"), uninstall.index(r'Delete "$INSTDIR\${MAINBINARYNAME}.exe"'))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -7,6 +7,7 @@
|
||||
|
||||
using strata::ds4::experts_per_word;
|
||||
using strata::ds4::experts_test_override;
|
||||
using strata::ds4::copy_all_fits;
|
||||
|
||||
static int fails = 0;
|
||||
#define CHECK_EQ(a, b) \
|
||||
@@ -107,6 +108,24 @@ int main() {
|
||||
CHECK_EQ(prefill_chunk_default(false, 0.30, false, false, false, 24 * GiB), 2048);
|
||||
// the rule takes no --max-context: a 128K session on a 12 GB card gets the same chunk, because the chunk's buffers and the KV are
|
||||
// allocated whole at load for that context and a chunk that does not fit there is halved before the first prompt (4096 -> 2048 -> 1024 ...)
|
||||
// copy_all_fits: the 251 GB R9700 box (108 GB lane) copies; a 64 GB PC, a 125 GB box with the 2.25 pack's lane, an unknown RAM do not
|
||||
{
|
||||
const double GB = 1e9, margin = 8.0 * (1ull << 30);
|
||||
CHECK_EQ(copy_all_fits(108 * GB, (uint64_t) (236 * GB), margin, 1e300), 1);
|
||||
CHECK_EQ(copy_all_fits(108 * GB, (uint64_t) (110 * GB), margin, 1e300), 0); // inside the margin
|
||||
CHECK_EQ(copy_all_fits(108 * GB, (uint64_t) (116 * GB), margin, 1e300), 0); // 108 + 8.6 > 116
|
||||
CHECK_EQ(copy_all_fits(108 * GB, (uint64_t) (118 * GB), margin, 1e300), 1);
|
||||
CHECK_EQ(copy_all_fits(108 * GB, (uint64_t) (101 * GB), margin, 1e300), 0); // the 125 GB box, free RAM 101 GB
|
||||
CHECK_EQ(copy_all_fits(92 * GB, ~uint64_t{0}, margin, 1e300), 0); // RAM not known
|
||||
CHECK_EQ(copy_all_fits(0, (uint64_t) (236 * GB), margin, 1e300), 0); // nothing on the CPU lane
|
||||
// Windows: the commit room (RAM + page file beside its margin) must hold the lane too, whatever the free RAM says
|
||||
CHECK_EQ(copy_all_fits(108 * GB, (uint64_t) (236 * GB), margin, 90 * GB), 0);
|
||||
CHECK_EQ(copy_all_fits(108 * GB, (uint64_t) (236 * GB), margin, 108 * GB), 1);
|
||||
// the owner-PC class: 64 GB RAM, a 12 GB card (a lane of ~120 GB), 40 GB free, a 90 GB commit limit
|
||||
CHECK_EQ(copy_all_fits(120 * GB, (uint64_t) (40 * GB), margin, 60 * GB), 0);
|
||||
// a 192 GB Windows PC with a 4 GB page file: 150 GB free RAM but only 100 GB of commit room: no
|
||||
CHECK_EQ(copy_all_fits(108 * GB, (uint64_t) (150 * GB), margin, 100 * GB), 0);
|
||||
}
|
||||
if (fails == 0) std::printf("experts rule: all checks passed\n");
|
||||
return fails == 0 ? 0 : 1;
|
||||
}
|
||||
@@ -42,7 +42,7 @@ class OldLauncherIsToldTheTruth(unittest.TestCase):
|
||||
for env in ({"STRATA_NO_ACQUIRE": "1", "STRATA_HANDOVER": "1"}, {}):
|
||||
done = self.run_version(tree, **env)
|
||||
self.assertEqual(done.returncode, 0, done.stderr)
|
||||
self.assertTrue(done.stdout.startswith("strata 0.2.5 (build "), done.stdout)
|
||||
self.assertTrue(done.stdout.startswith("strata 0.2.6 (build "), done.stdout)
|
||||
elsewhere = self.staged_tree(parent="terminal") # not the updater's staging folder
|
||||
self.assertEqual(self.run_version(elsewhere, STRATA_NO_ACQUIRE="1").returncode, 0)
|
||||
|
||||
|
||||
@@ -21,24 +21,24 @@ def first(path, pattern):
|
||||
|
||||
class VersionBump(unittest.TestCase):
|
||||
def test_the_terminal_is_0_2_4_2_and_control_is_0_1_49_2_while_the_app_stays_0_2_3(self):
|
||||
# 0.2.5: Terminal, app and release are 0.2.5; engine and Control are 0.1.50 for every Qwen and DeepSeek package (metal included)
|
||||
self.assertEqual(strata_cli.PRODUCT_VERSION, "0.2.5")
|
||||
self.assertEqual(strata_cli.version(), "0.1.50") # the CMake project version: Control
|
||||
self.assertEqual(first("CMakeLists.txt", r"^project\(strata VERSION ([0-9.]+) "), "0.1.50")
|
||||
# 0.2.6: Terminal and app are 0.2.6; Control and the DeepSeek engine are 0.1.51 (the CMake project version); the Qwen engine packages and Metal stay 0.1.50 (no C++ change since 0.2.5)
|
||||
self.assertEqual(strata_cli.PRODUCT_VERSION, "0.2.6")
|
||||
self.assertEqual(strata_cli.version(), "0.1.51") # the CMake project version: Control
|
||||
self.assertEqual(first("CMakeLists.txt", r"^project\(strata VERSION ([0-9.]+) "), "0.1.51")
|
||||
self.assertIn('STRATA_VERSION="0.1.50-metal"', (ROOT / "metal" / "CMakeLists.txt").read_text(encoding="utf-8"))
|
||||
|
||||
def test_every_place_the_app_version_lives_says_0_2_3(self):
|
||||
self.assertEqual(first("apps/desktop/src-tauri/Cargo.toml", r'^version = "([^"]+)"'), "0.2.5")
|
||||
self.assertEqual(first("apps/desktop/src-tauri/Cargo.toml", r'^version = "([^"]+)"'), "0.2.6")
|
||||
lock = (ROOT / "apps/desktop/src-tauri/Cargo.lock").read_text(encoding="utf-8")
|
||||
self.assertRegex(lock, r'name = "strata-desktop"\r?\nversion = "0.2.5"')
|
||||
self.assertEqual(json.loads((ROOT / "apps/desktop/package.json").read_text(encoding="utf-8"))["version"], "0.2.5")
|
||||
self.assertRegex(lock, r'name = "strata-desktop"\r?\nversion = "0.2.6"')
|
||||
self.assertEqual(json.loads((ROOT / "apps/desktop/package.json").read_text(encoding="utf-8"))["version"], "0.2.6")
|
||||
host = json.loads((ROOT / "apps/host/package.json").read_text(encoding="utf-8"))
|
||||
self.assertEqual(host["version"], "0.2.5")
|
||||
self.assertEqual(host["version"], "0.2.6")
|
||||
locked = json.loads((ROOT / "apps/host/package-lock.json").read_text(encoding="utf-8"))
|
||||
self.assertEqual((locked["version"], locked["packages"][""]["version"]), ("0.2.5", "0.2.5"))
|
||||
self.assertEqual((locked["version"], locked["packages"][""]["version"]), ("0.2.6", "0.2.6"))
|
||||
|
||||
def test_the_documents_that_state_the_current_version_agree(self):
|
||||
self.assertIn("Version 0.2.5, engine 0.1.50.", (ROOT / "AGENTS.md").read_text(encoding="utf-8"))
|
||||
self.assertIn("Version 0.2.6, engine 0.1.51.", (ROOT / "AGENTS.md").read_text(encoding="utf-8"))
|
||||
for name in ("README.md", "README.de.md", "README.es.md", "README.fr.md", "README.ja.md", "README.pt-BR.md", "README.zh-CN.md"):
|
||||
text = (ROOT / name).read_text(encoding="utf-8")
|
||||
self.assertRegex(text, r"Strata-0[.]2[.][0-9]+-", name) # (the restyled READMEs state the app version through the download buttons; their owner moves them to the release)
|
||||
@@ -47,7 +47,7 @@ class VersionBump(unittest.TestCase):
|
||||
def test_version_prints_the_product_and_the_build(self):
|
||||
done = subprocess.run([sys.executable, str(ROOT / "strata.py"), "--version"], capture_output=True, text=True, cwd=str(ROOT),
|
||||
env={**os.environ, "STRATA_NO_HANDOVER": "1"})
|
||||
self.assertTrue(done.stdout.startswith("strata 0.2.5 (build 0.1.50;"), done.stdout)
|
||||
self.assertTrue(done.stdout.startswith("strata 0.2.6 (build 0.1.51;"), done.stdout)
|
||||
|
||||
|
||||
class UpdateFromTheLastRelease(test_updater.UpdaterCase):
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
"""The Linux AMD (HIP) engines hang under WSL (/dev/dxg present, /dev/kfd absent: librocdxg never completes a barrier-AND packet).
|
||||
Setup and launch say so in plain words with one next step, as a warning, and never refuse."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from strata_cli.launch import settings as S
|
||||
from strata_cli.launch.runner import Runner
|
||||
from strata_control import engine_component as ec
|
||||
|
||||
|
||||
def dev(root: str, dxg: bool = True, kfd: bool = False) -> str:
|
||||
for name, on in (("dxg", dxg), ("kfd", kfd)):
|
||||
if on:
|
||||
Path(root, name).write_text("", encoding="utf-8")
|
||||
return root
|
||||
|
||||
|
||||
class WslAmdNote(unittest.TestCase):
|
||||
def test_wsl_with_dxg_and_no_kfd_on_hip_gets_one_plain_sentence_with_a_next_step(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
note = ec.wsl_amd_note("hip", dev(d), wsl=True)
|
||||
self.assertIn("do not run under WSL yet", note)
|
||||
self.assertIn("Windows", note) # the one next step: the Windows app or Terminal
|
||||
|
||||
def test_nothing_to_say_elsewhere(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
self.assertIsNone(ec.wsl_amd_note("hip", dev(d), wsl=False)) # a real Linux box
|
||||
self.assertIsNone(ec.wsl_amd_note("cuda", dev(d), wsl=True)) # NVIDIA works under WSL
|
||||
self.assertIsNone(ec.wsl_amd_note(None, dev(d), wsl=True))
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
self.assertIsNone(ec.wsl_amd_note("hip", dev(d, dxg=True, kfd=True), wsl=True)) # a /dev/kfd means the native driver
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
self.assertIsNone(ec.wsl_amd_note("hip", dev(d, dxg=False), wsl=True)) # no /dev/dxg: not the WSL GPU bridge
|
||||
|
||||
def test_launch_warns_and_goes_on(self):
|
||||
events = []
|
||||
r = Runner(S.LaunchSettings(family="qwen", model="Q2_0", context=32768), catalog=mock.Mock(), emit=events.append)
|
||||
with mock.patch.object(ec, "wsl_amd_note", side_effect=lambda backend, *a, **k: "WSL note" if backend == "hip" else None), \
|
||||
mock.patch.object(ec, "detect_backend", return_value=("hip", "test")):
|
||||
r.warn_wsl_amd()
|
||||
self.assertEqual([(e.level, e.text) for e in events], [("warn", "WSL note")])
|
||||
self.assertIn("WSL note", r.notices)
|
||||
|
||||
def test_launch_is_silent_when_the_backend_is_not_hip(self):
|
||||
events = []
|
||||
r = Runner(S.LaunchSettings(family="qwen", model="Q2_0", context=32768), catalog=mock.Mock(), emit=events.append)
|
||||
with mock.patch.object(ec, "wsl_amd_note", side_effect=lambda backend, *a, **k: "WSL note" if backend == "hip" else None), \
|
||||
mock.patch.object(ec, "detect_backend", return_value=("cuda", "test")):
|
||||
r.warn_wsl_amd()
|
||||
self.assertEqual(events, [])
|
||||
|
||||
def test_setup_warns_in_the_hip_branch_and_does_not_fail_on_its_own(self):
|
||||
text = (Path(__file__).resolve().parents[2] / "setup.py").read_text(encoding="utf-8")
|
||||
i = text.index("wsl_amd_note(\"hip\", wsl=is_wsl())")
|
||||
self.assertIn("warn(wsl_note)", text[i:i + 200])
|
||||
self.assertLess(i, text.index("no AMD GPU Strata can use"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+25
-15
@@ -1,20 +1,30 @@
|
||||
# DFlash2 acceptance experiment (research tooling, not shipped)
|
||||
# DFlash2 acceptance experiment
|
||||
|
||||
Measures how many tokens per round the incoai DFlash2 drafter for Qwen3.8-Flash-Next would get on OUR quantized target, offline, from the hidden states
|
||||
the engine dumps. Results and method: phaseA-tests/notes-dflash2.md. All scripts keep the paths of the run (C:\dflash2-work on the owner PC, C:\strata-test\dflash2
|
||||
on the test box): edit them before reuse.
|
||||
Research tooling, not shipped with the app. It measures how many tokens per round the incoai DFlash2 drafter for
|
||||
Qwen3.8-Flash-Next would accept on Strata's quantized target. The measurement is offline, from hidden states the
|
||||
engine dumps. It is for developers who want to repeat or extend the experiment.
|
||||
|
||||
Engine side (both off unless the variable is set):
|
||||
The scripts keep the paths of the original run (`C:\dflash2-work` on the development PC, `C:\strata-test\dflash2` on
|
||||
the test box). Edit the paths before you reuse them. The method and results are in the project's internal notes
|
||||
and are not part of this tree.
|
||||
|
||||
- `STRATA_TAP_FILE=<file>` (+ `STRATA_TAP_LAYERS`, default `5,15,25,35,45`): `Verifier::record_window` copies each verified row's completed layer output (the mean of
|
||||
the 4 hyper-connection streams, the quantity the drafter was trained on) and `Verifier::run` appends `pos, token, taps` records.
|
||||
- `STRATA_SPEC_T_CYCLE=1,2,...,8` (CLI bench only): the verify window size cycles; the report line `window cost (ms/round)` gives the cost of each width.
|
||||
## Engine variables
|
||||
|
||||
Scripts:
|
||||
Both are off unless set.
|
||||
|
||||
- `mk_prompts.py` builds chat-template prompt ids with the server's own tokenizer and template.
|
||||
- `orch.py <prompt> <max_new> [gen,genf,genfm,tap]` drives `run_one.ps1` on the test box: free generation (ids, MTP statistics), the same text with MTP only, and the
|
||||
teacher-forced tap dump with oracle drafts (`--spec 8 --spec-oracle --spec-follow`, a 2-token prompt: a native pack never starts the speculative loop from a 1-token prompt).
|
||||
- `dflash2_ref.py` is a plain-torch port of z-lab's `DFlash2DraftModel` (checked against it by `test_port.py`), with the target's embedding and head dequantized from the GGUF.
|
||||
- `run_eval.py` (walk, a few block sizes), `run_all2.py` (every position, resumable: per-anchor correctness and the selector's confidences), `analyze2.py` and
|
||||
`final_tables.py` (policies, speed estimates from the measured window costs).
|
||||
| Variable | Effect |
|
||||
| --- | --- |
|
||||
| `STRATA_TAP_FILE=<file>` | Writes `pos, token, taps` records. `Verifier::record_window` copies each verified row's completed layer output (the mean of the 4 hyper-connection streams, the quantity the drafter was trained on) and `Verifier::run` appends it. |
|
||||
| `STRATA_TAP_LAYERS` | Layers to tap, default `5,15,25,35,45`. Used with `STRATA_TAP_FILE`. |
|
||||
| `STRATA_SPEC_T_CYCLE=1,2,...,8` | CLI bench only. The verify window size cycles; the report line `window cost (ms/round)` gives the cost of each width. |
|
||||
|
||||
## Scripts
|
||||
|
||||
| Script | Purpose |
|
||||
| --- | --- |
|
||||
| `mk_prompts.py` | Builds chat-template prompt ids with the server's own tokenizer and template. |
|
||||
| `orch.py <prompt> <max_new> [gen,genf,genfm,tap]` | Drives `run_one.ps1` on the test box: free generation (ids and MTP statistics), the same text with MTP only, and the teacher-forced tap dump with oracle drafts (`--spec 8 --spec-oracle --spec-follow`, a 2-token prompt, because a native pack never starts the speculative loop from a 1-token prompt). |
|
||||
| `dflash2_ref.py` | Plain-torch port of z-lab's `DFlash2DraftModel`, with the target's embedding and head dequantized from the GGUF. `test_port.py` checks it against the original. |
|
||||
| `run_eval.py` | Walks the tap file with a few block sizes. |
|
||||
| `run_all2.py` | Evaluates every position, resumable: per-anchor correctness and the selector's confidences. |
|
||||
| `analyze2.py`, `final_tables.py` | Policies, and speed estimates from the measured window costs. |
|
||||
@@ -975,6 +975,8 @@ class Strata:
|
||||
except Exception: # noqa: BLE001 - unknown stays unknown: the app then keeps its safety default
|
||||
pass
|
||||
hw["gpus"] = nv + amd
|
||||
for g in hw["gpus"]: # each make numbers its cards from 0 (the engine drives one make): a stable id tells two cards apart
|
||||
g.setdefault("id", f"{g.get('vendor') or 'gpu'}:{g.get('index')}")
|
||||
try: # how fast each card is (SMs x clock): the app and the Terminal pick the Main card by it
|
||||
from strata_control import gpu_roles
|
||||
gpu_roles.add_speeds(hw["gpus"])
|
||||
|
||||
@@ -347,6 +347,13 @@ class NoNetwork(Install):
|
||||
self.assertIn("Check the internet connection and run the same command again.", out)
|
||||
self.assertFalse([c for c in self.calls if c[0] == "get"])
|
||||
|
||||
def test_a_local_feed_needs_no_github(self):
|
||||
"""A run on a local feed (a folder, a loopback address) downloads nothing from GitHub: a slow or down GitHub must not stop it."""
|
||||
self.github_up = False
|
||||
with mock.patch.dict("os.environ", {"STRATA_UPDATE_FEED": str(self.root / "feed")}):
|
||||
rc, out = self.install(engine=False)
|
||||
self.assertNotIn("can't reach GitHub", out)
|
||||
|
||||
def test_a_package_the_person_has_needs_no_github(self):
|
||||
self.github_up = False
|
||||
self.install(engine=False) # (the first run only creates the folders)
|
||||
@@ -377,6 +384,15 @@ class GithubCheck(unittest.TestCase):
|
||||
with mock.patch.object(setup, "_urlopen", return_value=mock.MagicMock()):
|
||||
self.assertTrue(setup.github_reachable(1))
|
||||
|
||||
def test_a_local_feed_is_told_from_the_signed_one(self):
|
||||
for feed in ("C:/feeds/f", "/tmp/feed", "file:///tmp/feed", "http://127.0.0.1:8099/feed.json", "https://localhost/f"):
|
||||
self.assertTrue(setup.local_feed(feed), feed)
|
||||
for feed in (None, "", "https://github.com/Niko1221/Strata/releases/latest/download/feed.json"):
|
||||
with mock.patch.dict("os.environ", {"STRATA_UPDATE_FEED": ""}):
|
||||
self.assertFalse(setup.local_feed(feed), feed)
|
||||
with mock.patch.dict("os.environ", {"STRATA_UPDATE_FEED": "/tmp/x"}):
|
||||
self.assertTrue(setup.local_feed()) # the environment names it when --feed does not
|
||||
|
||||
|
||||
class OfficialSets(unittest.TestCase):
|
||||
"""The two official IST-DAS sets (ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF): the catalog, the pins, the pack recipe, the fit words."""
|
||||
|
||||
@@ -811,6 +811,9 @@ class MixedGraphics(unittest.TestCase):
|
||||
self.assertTrue(by["nvidia"]["usable"])
|
||||
self.assertFalse(by["amd"]["usable"])
|
||||
self.assertIn("NVIDIA card", by["amd"]["problem"])
|
||||
# the RTX and the Ryzen iGPU both say index 0 (each make counts from 0), so the report gives each a stable id of its own
|
||||
self.assertEqual((by["nvidia"]["index"], by["amd"]["index"]), (0, 0))
|
||||
self.assertEqual(sorted(g["id"] for g in hw["gpus"]), ["amd:0", "nvidia:0"])
|
||||
rec = s.recommend({**hw, "ram_gb": 92.0})
|
||||
self.assertEqual((rec["backend"], rec["gpu"]["name"]), ("cuda", "NVIDIA GeForce RTX 5090"))
|
||||
# with no NVIDIA card the integrated GPU stays what it was
|
||||
|
||||
Reference in new issue
Block a user