mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-07 14:21:06 -10:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Niko1221
|
0c2e63e930 | ||
|
Niko1221
|
888751a0e1 | ||
|
Niko1221
|
af1c91f245 | ||
|
Niko1221
|
14b22c51c1 | ||
|
Vistawizard
|
be028948f4 | ||
|
Mirtraxxx
|
470b3d7c21 | ||
|
Mirtraxxx
|
be0e0c94e3 | ||
|
Mirtraxxx
|
42810ee8bb | ||
|
btechnet
|
0f8b5d4938 | ||
|
Niko1221
|
07b4a6263a | ||
|
Niko1221
|
e550970773 |
No files matched your search
@@ -1,332 +1,140 @@
|
||||
<h1 align="center">Qwen3.8-Flash-Next (125B MoE) on a 12-24 GB NVIDIA GPU + 64 GB RAM: one-click install for Windows / Linux</h1>
|
||||
<h1 align="center">Strata</h1>
|
||||
|
||||
<p align="center"><b>Strata</b>: an inference engine built for exactly one model and exactly this kind of PC</p>
|
||||
<p align="center"><b>Run a 125-billion-parameter AI model on a normal gaming PC</b><br>
|
||||
one NVIDIA card (12-24 GB) + 64 GB of RAM · Windows or Linux · one click to install</p>
|
||||
|
||||
A 125-billion-parameter mixture-of-experts model (6B active per token) on a **normal gaming PC**: one NVIDIA card with
|
||||
12-24 GB of VRAM, 64 GB of RAM and a desktop CPU. Strata keeps the always-used part of the model on the GPU, all 24,576
|
||||
experts in RAM, computes the experts the GPU does not hold on the CPU at the same time, and checks several tokens per
|
||||
pass with the model's own multi-token-prediction (MTP) layer. The text it produces is identical to plain greedy decoding.
|
||||
Strata runs **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** - a large, smart AI model that
|
||||
normally needs a server - on your own PC. It writes its answers at **60-95 tokens per second** (a token is about ¾
|
||||
of a word): faster than you can read.
|
||||
|
||||
It serves an **OpenAI-** and **Anthropic-compatible API on localhost**, so any chat app, coding agent or script that
|
||||
talks to those APIs can use it. Optionally it also **reads images** (screenshots, photos, scanned pages) with the
|
||||
model's own vision encoder.
|
||||
- **Private:** everything runs on your PC. Nothing is sent anywhere.
|
||||
- **Works with your apps:** chat apps, coding agents and scripts that speak the OpenAI or Anthropic API just work.
|
||||
- **Sees pictures** too, if you want (screenshots, photos, scanned pages).
|
||||
- **Free and open source.**
|
||||
|
||||
> **Jump to:** [Speed](#speed-measured) · [Other GPUs](#other-gpus-estimated) · [Which model?](#which-model) ·
|
||||
> [Windows](#windows) · [Linux](#linux) · [Using it](#using-it) · [Images](#images-vision) ·
|
||||
> [Troubleshooting](#troubleshooting) · [How it works](#how-it-works)
|
||||
> **Jump to:** [Is my PC enough?](#is-my-pc-enough) · [Install](#install-3-steps) · [How fast?](#how-fast-is-it) ·
|
||||
> [Which model?](#which-model-should-i-pick) · [Using it](#using-it) · [Problems?](#something-went-wrong) ·
|
||||
> [How it works](#how-does-it-work) · [All the details](docs/DETAILS.md)
|
||||
|
||||
---
|
||||
|
||||
## Speed (measured)
|
||||
## Is my PC enough?
|
||||
|
||||
RTX 5070 **12 GB**, Ryzen 5 7600 (6 cores), 64 GB DDR5-5200, Windows. One code-agent prompt per length, 256 generated
|
||||
tokens, MTP speculative decoding on. "262K" is the model's full context window (a 259,943-token prompt).
|
||||
|
||||
### Prompt processing (tokens/s)
|
||||
|
||||
| Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 389 | 539 | 571 | 561 | 543 | 496 |
|
||||
| **IQ2_XS** | 332 | 463 | 495 | 486 | 472 | 437 |
|
||||
| **IQ3_XXS** | 285 | 410 | 435 | 427 | 414 | - |
|
||||
|
||||
### Output (tokens/s)
|
||||
|
||||
| Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 88.7 | 94.6 | 87.5 | 76.4 | 65.1 | 56.3 |
|
||||
| **IQ2_XS** | 82.0 | 78.0 | 65.3 | 63.7 | 52.0 | 48.0 |
|
||||
| **IQ3_XXS** | 64.6 | 65.6 | 57.3 | 54.4 | 44.8 | - |
|
||||
|
||||
IQ3_XXS at 262K is not measured: with its 43 GB of experts, a 260K-token context brings a 64 GB PC to its memory
|
||||
limit. Use up to 128K with IQ3_XXS on 64 GB.
|
||||
|
||||
Time to first token is prompt length / prompt speed: about 7 s at 4K, 55 s at 32K, 4 minutes at 128K and 9 minutes at
|
||||
262K. The raw numbers: `bench/results/`. The [paper](docs/paper/Strata-Paper.pdf) explains every number.
|
||||
|
||||
## Other GPUs (estimated)
|
||||
|
||||
Not measured - estimated from the runs above (same CPU and 64 GB RAM): the GPU part scaled by memory bandwidth, the CPU
|
||||
part by how many more experts the card's VRAM holds. Treat as **±20%**. Numbers are *prompt / output* tokens/s.
|
||||
|
||||
| GPU | Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| RTX 5060 Ti 16GB | Q2_0 | ~341 / ~80 | ~472 / ~87 | ~501 / ~81 | ~492 / ~72 | ~476 / ~62 | ~435 / ~53 |
|
||||
| | IQ2_XS | ~291 / ~80 | ~406 / ~77 | ~434 / ~63 | ~426 / ~62 | ~413 / ~51 | ~383 / ~47 |
|
||||
| | IQ3_XXS | ~249 / ~66 | ~359 / ~65 | ~381 / ~56 | ~374 / ~54 | ~363 / ~45 | - |
|
||||
| RTX 3090 24GB | Q2_0 | ~355 / ~128 | ~491 / ~140 | ~521 / ~130 | ~512 / ~115 | ~495 / ~100 | ~453 / ~85 |
|
||||
| | IQ2_XS | ~303 / ~131 | ~422 / ~128 | ~451 / ~103 | ~444 / ~102 | ~430 / ~85 | ~398 / ~78 |
|
||||
| | IQ3_XXS | ~260 / ~106 | ~374 / ~103 | ~396 / ~89 | ~390 / ~85 | ~378 / ~71 | - |
|
||||
|
||||
More VRAM matters more than a faster GPU: every extra GB holds ~700 more experts, and every expert on the GPU is one the
|
||||
CPU does not have to compute. A 3090's 24 GB takes most of the CPU work away.
|
||||
|
||||
## Which model?
|
||||
|
||||
All three are [ISTA-DASLab's GSQ-RCO quantizations](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
of [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).
|
||||
|
||||
| Model | Download | RAM it uses | Speed | Quality |
|
||||
| --- | ---: | ---: | --- | --- |
|
||||
| **Q2_0** | 66 GB | ~34 GB experts + ~6 GB | fastest | good |
|
||||
| **IQ2_XS** | 68 GB | ~36 GB experts + ~6 GB | close to Q2_0 | a bit better |
|
||||
| **IQ3_XXS** | 76 GB | ~43 GB experts + ~6 GB | slower (more CPU work) | best |
|
||||
|
||||
With 64 GB of RAM all three fit (close the browser for IQ3_XXS, and keep its context at 128K or less). With 48 GB only Q2_0 / IQ2_XS may fit. 32 GB is not enough.
|
||||
|
||||
### Or: Swift 1.5 (a fine-tune that thinks shorter)
|
||||
|
||||
The setup's first question also offers **[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)**,
|
||||
UkisAI's fine-tune of Qwen3.8-Flash-Next, trained to reach the answer with much less thinking (its authors: 63% fewer
|
||||
thinking tokens, 1.8x sooner answers, under 1% accuracy loss). Same architecture, the same three sizes, its own
|
||||
vision encoder; Strata runs it at the same speed (4K, IQ2_XS: 465 prompt / 78.7 output tokens/s, vs 467 / 78.3 for
|
||||
the original). Its authors recommend **IQ2_XS** (their Q2_0 is marked experimental). Its license is the Swift Open
|
||||
License 1.0 - read it on the model page.
|
||||
|
||||
Our small check (8 reasoning questions, default thinking, IQ2_XS): both models got **8/8**; Swift used **1,234**
|
||||
output tokens in 28 s, the original **2,682** in 46 s - most of the difference from one question the original
|
||||
thought about for 1,524 tokens. Not a benchmark, but consistent with the claim.
|
||||
|
||||
```
|
||||
START-HERE.bat --setup --family swift --model IQ2_XS
|
||||
```
|
||||
|
||||
## Before you start
|
||||
|
||||
You need **only an NVIDIA driver** (version 580 or newer; update it with the NVIDIA App or from
|
||||
[nvidia.com/drivers](https://www.nvidia.com/drivers)). Everything else is installed for you the first time.
|
||||
|
||||
| | |
|
||||
| You need | |
|
||||
| --- | --- |
|
||||
| GPU | NVIDIA **RTX 30, 40 or 50 series**, **12 GB VRAM or more** (8 GB runs, slowly). Measured on an RTX 5070; RTX 30/40 are untested. |
|
||||
| RAM | **64 GB** recommended (see the table above). |
|
||||
| CPU | x86-64 with AVX2 (any Intel/AMD desktop CPU from the last ~8 years). AVX-512 (Ryzen 7000/9000) is a bit faster. |
|
||||
| Disk | ~70-80 GB free for the model, ~6 GB for the MTP layer (+1 GB with images). **Q2_0 on an AVX-512 CPU** also writes a one-time ~40 GB copy of its experts for the fast CPU kernel. An NVMe SSD is strongly recommended. |
|
||||
| OS | Windows 10/11, or Linux (Ubuntu 22.04/24.04 get everything installed automatically). |
|
||||
| **Graphics card** | NVIDIA RTX 30, 40 or 50 series with **12 GB of VRAM or more** |
|
||||
| **Memory (RAM)** | **64 GB** |
|
||||
| **Free disk space** | ~80 GB (an SSD makes the first start much faster) |
|
||||
| **System** | Windows 10/11, or Linux |
|
||||
|
||||
What the first start installs, all inside this folder (`.venv/`, `engine/`, `third_party/`, `models/`, `packs/`, `mtp/`):
|
||||
Python 3.12 if you have none (for your user account, no admin), a private Python environment, NVIDIA's CUDA libraries
|
||||
(from pip, ~0.4 GB), the ready-made Strata engine for RTX 30/40/50, the model and the MTP draft layer. If no
|
||||
ready-made engine fits your PC, it offers to install the build tools (Visual Studio Build Tools + CUDA Toolkit on
|
||||
Windows, `build-essential` + CUDA on Ubuntu) and compiles the engine for your GPU (asks first; 20-40 minutes once).
|
||||
That's it. The only thing you install yourself is a current **NVIDIA driver**
|
||||
([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA App). Everything else - Python, the engine, the
|
||||
model - is set up for you.
|
||||
|
||||
---
|
||||
## Install (3 steps)
|
||||
|
||||
## Windows
|
||||
**Windows**
|
||||
|
||||
### Double-click `START-HERE.bat`
|
||||
1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it).
|
||||
2. Double-click **`START-HERE.bat`**.
|
||||
3. Answer 4 questions - or just press Enter each time for the recommended choice:
|
||||
- **Which model?** The original, or Swift 1.5 (a version that thinks shorter and answers sooner)
|
||||
- **Which size?** Q2_0, IQ2_XS or IQ3_XXS - see [which model](#which-model-should-i-pick)
|
||||
- **How much context?** How much text it can keep in mind at once (it suggests one for your card)
|
||||
- **Images?** Whether it should also read pictures
|
||||
|
||||
**The first time** it asks four questions and does the rest:
|
||||
Then it downloads everything (the model is ~70 GB, so the first time takes a while - you can stop and it picks up
|
||||
where it left off) and **starts the model**. Your browser opens a chat page at `http://127.0.0.1:8080`.
|
||||
|
||||
1. **Which model?** Qwen3.8-Flash-Next (the original) or Swift 1.5 (the fine-tune that thinks shorter).
|
||||
2. **Which size?** Q2_0, IQ2_XS or IQ3_XXS (it recommends one for your RAM).
|
||||
3. **How much context?** 8K to 256K tokens (it recommends one for your VRAM).
|
||||
4. **Images?** yes / no (see [Images](#images-vision)).
|
||||
**Next time**, just double-click `START-HERE.bat` again: it starts right away, nothing is downloaded twice. Close its
|
||||
window to stop the model.
|
||||
|
||||
Then it downloads and prepares everything (the model is 66-76 GB, so the first start takes a while; an interrupted
|
||||
download continues where it stopped) and **starts the model**: your browser opens `http://127.0.0.1:8080`, a small
|
||||
page that shows it is running and lets you chat. The API is at `http://127.0.0.1:8080/v1` for your apps.
|
||||
**Linux:** run `./setup.sh` - same questions, same result.
|
||||
|
||||
**Every time after that**, `START-HERE.bat` just starts the model (30-90 s to load 34-43 GB into RAM). Nothing is
|
||||
downloaded again. Closing the window stops the model.
|
||||
## How fast is it?
|
||||
|
||||
```
|
||||
START-HERE.bat --setup install another model, or change context / images
|
||||
START-HERE.bat --model IQ2_XS --context 32768 --vision yes --yes no questions
|
||||
START-HERE.bat --gguf-dir D:\models\IQ2_XS use GGUF files you already have
|
||||
START-HERE.bat --port 8081 another port
|
||||
```
|
||||
Measured on an RTX 5070 (12 GB), a Ryzen 5 7600 and 64 GB of RAM:
|
||||
|
||||
With more than one model installed, it asks which one to start. `run-<model>.bat` starts a model directly.
|
||||
| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| **Q2_0** | 95 tokens/s | 65 tokens/s | 539 tokens/s |
|
||||
| **IQ2_XS** | 78 tokens/s | 52 tokens/s | 463 tokens/s |
|
||||
| **IQ3_XXS** | 66 tokens/s | 45 tokens/s | 410 tokens/s |
|
||||
|
||||
### Chat in the terminal (optional)
|
||||
- **Writes answers** = how fast the reply appears (tokens per second).
|
||||
- **Reads your prompt** = how fast it takes in what you send (long documents, code, chat history).
|
||||
|
||||
```
|
||||
.venv\Scripts\python chat.py
|
||||
```
|
||||
A card with more VRAM is faster, because more of the model fits on the GPU: an RTX 3090 (24 GB) should do roughly
|
||||
100-140 tokens per second. All measurements, long-context numbers and estimates for other cards are in the
|
||||
[details](docs/DETAILS.md#speed-measured).
|
||||
|
||||
---
|
||||
## Which model should I pick?
|
||||
|
||||
## Linux
|
||||
**The size** (the same model, compressed more or less):
|
||||
|
||||
```bash
|
||||
./setup.sh
|
||||
```
|
||||
| Size | Download | Speed | Quality | Pick it if... |
|
||||
| --- | ---: | --- | --- | --- |
|
||||
| **Q2_0** | 66 GB | fastest | good | you want speed |
|
||||
| **IQ2_XS** | 68 GB | fast | better | you want a good all-rounder (**recommended**) |
|
||||
| **IQ3_XXS** | 76 GB | slower | best | you want the best answers (uses 43 GB of your 64 GB RAM) |
|
||||
|
||||
The same questions, the same automatic install (it uses `sudo apt` for Python and, only if it has to compile,
|
||||
for the build tools), and the same start: `http://127.0.0.1:8080`. Later runs of `./setup.sh` (or `./run-<model>.sh`)
|
||||
start the model directly. Options as on Windows (`./setup.sh --setup`, `--model Q2_0 --yes`, `--gguf-dir /data/Q2_0`).
|
||||
Terminal chat: `.venv/bin/python chat.py`.
|
||||
**The version:**
|
||||
|
||||
---
|
||||
- **Qwen3.8-Flash-Next** - the original.
|
||||
- **[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** - a fine-tune by UkisAI
|
||||
that thinks much shorter before answering, so you get the answer sooner, with about the same quality. Same speed per
|
||||
token. Its own license applies (see its page).
|
||||
|
||||
Not sure? Take **IQ2_XS**. You can add another one later with `START-HERE.bat --setup`.
|
||||
|
||||
## Using it
|
||||
|
||||
The server listens on `http://127.0.0.1:8080` (change with `--port` in setup, or edit the run script).
|
||||
- **Chat in the browser:** `http://127.0.0.1:8080` - a simple chat page (it opens by itself when the model starts).
|
||||
- **Chat in the terminal:** `.venv\Scripts\python chat.py`
|
||||
- **Your apps and coding agents:** add it as an "OpenAI-compatible" provider with base URL
|
||||
**`http://127.0.0.1:8080/v1`**, any API key and any model name. Apps that use Anthropic's API: `http://127.0.0.1:8080/v1/messages`.
|
||||
- **Thinking:** the model thinks before it answers. Choose **off, low, medium or high** - in the chat page menu, with
|
||||
`/think low` in `chat.py`, or with your app's "reasoning effort" setting. Off is fastest; high is best for hard questions.
|
||||
- **Pictures:** in the chat page click **Picture**; in `chat.py` type `/image <path>`; in apps just attach them.
|
||||
- **From your phone or another PC:** see the [details](docs/DETAILS.md#using-it) (set an API key first).
|
||||
|
||||
| API | Endpoint |
|
||||
**Good to know:** it answers one request at a time. The first message of a chat is read in full (about 1 minute per
|
||||
30,000 tokens); after that it keeps the conversation and reads only what is new, so follow-ups start in seconds.
|
||||
|
||||
## Something went wrong?
|
||||
|
||||
| What you see | What to do |
|
||||
| --- | --- |
|
||||
| OpenAI Chat Completions (stream and non-stream, tools) | `POST /v1/chat/completions` |
|
||||
| Anthropic Messages (stream and non-stream, tools) | `POST /v1/messages` |
|
||||
| Model list / health | `GET /v1/models`, `GET /health` |
|
||||
| `the NVIDIA driver is too old` | Update the driver (NVIDIA App or nvidia.com/drivers), restart the PC, run `START-HERE.bat` again. |
|
||||
| It stopped during download or setup | Run `START-HERE.bat` again - it continues where it stopped. |
|
||||
| `port 8080 is already in use` | Strata is already running - look for its window. |
|
||||
| The first start takes minutes | Normal: it loads 35-43 GB into RAM. The next start is faster. |
|
||||
| Slow, and the disk light is busy | Not enough free RAM: close other programs (browsers use a lot), or pick Q2_0 / IQ2_XS. |
|
||||
| "prompt exceeds the context" | The conversation is longer than the context you chose: run `START-HERE.bat --setup` and pick more. |
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
|
||||
"model": "strata", "messages": [{"role": "user", "content": "Write a haiku about GPUs."}], "max_tokens": 512 }'
|
||||
```
|
||||
More in the [full troubleshooting table](docs/DETAILS.md#troubleshooting). Still stuck? Open an issue and attach
|
||||
`strata-<model>.log` from this folder.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
|
||||
r = client.chat.completions.create(model="strata", messages=[{"role": "user", "content": "Hello!"}])
|
||||
print(r.choices[0].message.content)
|
||||
```
|
||||
## How does it work?
|
||||
|
||||
- **Thinking levels: none, low, medium, high.** The model thinks before it answers (streamed as
|
||||
`reasoning_content`, Anthropic: `thinking` blocks). Choose how much per request - in the chat page (the "Thinking"
|
||||
menu), in `chat.py` (`/think low`), or over the API:
|
||||
A model this big doesn't fit on a gaming graphics card. Strata splits the work between the parts of your PC:
|
||||
|
||||
| API | how |
|
||||
| --- | --- |
|
||||
| OpenAI | `"reasoning_effort": "none" \| "low" \| "medium" \| "high"` (also `"reasoning": {"effort": ...}`, or `"chat_template_kwargs": {"enable_thinking": false}`) |
|
||||
| Anthropic | `"output_config": {"effort": "low" \| "medium" \| "high"}`, `"thinking": {"type": "disabled"}`, or `"thinking": {"type": "enabled", "budget_tokens": N}` (under 2K = low, under 8K = medium, more = high) |
|
||||
<p align="center"><img src="docs/paper/tiers.svg" width="700" alt="how Strata splits the model between GPU, RAM and SSD"></p>
|
||||
|
||||
Without a setting the model uses its own default, **high**. `none` answers at once (fastest); `low` keeps the thinking
|
||||
short. The levels are instructions the model was trained with, not a hard token limit: on easy questions all three
|
||||
think briefly, on hard ones `high` thinks longest and is most accurate.
|
||||
- **Chat apps.** Any app with an "OpenAI-compatible" provider works: base URL `http://127.0.0.1:8080/v1`, any API key.
|
||||
- **Context.** Chosen in setup (8K-262K). Requests longer than that are refused, never silently cut.
|
||||
- **From other devices / the internet.** The server listens on your PC only (`127.0.0.1`). To reach it from elsewhere,
|
||||
put a tunnel in front of it, for example [cloudflared](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/do-more-with-tunnels/trycloudflare/):
|
||||
`cloudflared tunnel --url http://127.0.0.1:8080`. **Set a key first**, or anyone with the link can use your PC:
|
||||
add `"api_key": "some-long-secret"` to `strata-<model>.json` (or set the `STRATA_API_KEY` environment variable);
|
||||
clients then send it as their API key.
|
||||
- **The GPU** runs the part of the model that is used for every word, plus the "experts" it needs most often.
|
||||
- **The RAM** holds all 24,576 experts, and **the CPU** computes the few the GPU doesn't have - at the same time as the GPU.
|
||||
- **The SSD** holds a big lookup table; the model reads a few rows of it per word.
|
||||
- **A small helper inside the model guesses the next words**, and Strata checks several guesses at once. That makes
|
||||
it 1.6-1.8x faster than going word by word - and the answer is exactly the same.
|
||||
|
||||
**Current limits (v1):** one request at a time; greedy decoding (temperature is ignored); every request processes its
|
||||
whole prompt again (no conversation cache yet, so long chats have a long time-to-first-token); images only when set up
|
||||
with them (below); no video.
|
||||
The full story is in the [paper](docs/paper/Strata-Paper.pdf) and the [details](docs/DETAILS.md).
|
||||
|
||||
---
|
||||
## Credits
|
||||
|
||||
## Images (vision)
|
||||
|
||||
The model has a vision encoder: [`mmproj-Qwen3.8-Flash-Next-BF16.gguf`](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
(0.9 GB, a 27-layer ViT plus the projector into the language model). It is **optional**: say yes when the setup asks
|
||||
"Images?", or run it again with `--vision gpu` (or `--vision cpu`). The setup downloads the encoder, builds a small
|
||||
helper (`strata-vision`, from llama.cpp's `mtmd` library) and adds it to your start script. Nothing else changes.
|
||||
|
||||
| Encoder on | Time per picture | Cost |
|
||||
| --- | --- | --- |
|
||||
| **GPU** (recommended) | **0.1-0.5 s** (up to 1,024 image tokens) | ~1.4 GB of VRAM is kept free for it, so the expert cache is smaller: text output is a few % slower (table below) |
|
||||
| CPU | 10-30 s (pictures are scaled down to ~300 image tokens) | nothing on the GPU |
|
||||
|
||||
A picture becomes up to 1,024 tokens of the context (a 640x480 photo: 300). The same picture sent again, as chat apps
|
||||
do on every turn, is encoded only once.
|
||||
|
||||
### Sending a picture
|
||||
|
||||
**Terminal chat:** type `/image <path to a picture>`, press Enter, then type your question.
|
||||
|
||||
```
|
||||
you> /image C:\Users\me\Pictures\receipt.jpg
|
||||
(picture attached: receipt.jpg - now type your question)
|
||||
you> What is the total on this receipt?
|
||||
```
|
||||
|
||||
**OpenAI API** (an `image_url` part: a `data:` URL, an `http(s)://` URL or a local file path):
|
||||
|
||||
```python
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
|
||||
img = base64.b64encode(open("photo.jpg", "rb").read()).decode()
|
||||
r = client.chat.completions.create(model="strata", messages=[{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img}"}},
|
||||
{"type": "text", "text": "What is in this picture?"}]}])
|
||||
print(r.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Anthropic API:** an `image` block with a `base64` (or `url`) source, as usual.
|
||||
|
||||
JPEG, PNG, BMP and GIF work (WebP does not). Chat apps with image upload work the same way.
|
||||
|
||||
### Speed with images on (4K context, measured)
|
||||
|
||||
| Model | Prompt tok/s, images off | Prompt tok/s, images on | Output tok/s, images off | Output tok/s, images on |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 541 | 531 | 90.7 | 86.9 |
|
||||
| **IQ2_XS** | 468 | 458 | 77.1 | 74.0 |
|
||||
| **IQ3_XXS** | 411 | 401 | 64.7 | 59.5 |
|
||||
|
||||
"Off" is the published setup; "on" runs with the encoder loaded on the GPU and its VRAM kept free, so ~1,000 fewer
|
||||
experts fit in VRAM and the CPU computes a few more per token: 2-8% slower. For text, turning images on changes
|
||||
nothing else (the output is bit-identical when the VRAM is the same).
|
||||
|
||||
A question about a picture (a 640x480 newspaper page = 300 image tokens, a 328-token prompt; the whole request, encoder
|
||||
on the GPU, measured through the API):
|
||||
|
||||
| Model | Answer ("MEN WALK ON MOON") | Picture encoded | Prompt | Output |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 1.9 s | ~0.1 s | 207 tok/s | 65-73 tok/s |
|
||||
| **IQ2_XS** | 2.3 s | ~0.1 s | 173 tok/s | 50-60 tok/s |
|
||||
| **IQ3_XXS** | 2.7 s | ~0.1 s | 144 tok/s | 41-50 tok/s |
|
||||
|
||||
(Short prompts run below the 4K prompt speed: a 2,048-token chunk is where the prompt path is efficient. Short answers
|
||||
run below the long-output speed: the first rounds have no draft yet.)
|
||||
|
||||
**How it works inside:** the encoder turns the picture into rows of the same width as the model's word embeddings;
|
||||
Strata puts them where the prompt has `<|image_pad|>` tokens and gives each one its 2-D position (row and column in the
|
||||
picture; the model uses interleaved M-RoPE). Answers match llama.cpp's multimodal implementation token for token on our
|
||||
test images.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | What to do |
|
||||
| --- | --- |
|
||||
| `the NVIDIA driver is too old` | Update the driver (NVIDIA App or nvidia.com/drivers), restart, run `START-HERE.bat` again. |
|
||||
| Python or the build tools could not be installed | Install what it names (links are printed), then run it again. Everything already done is kept. |
|
||||
| `port 8080 is already in use` | Strata is already running (look for its window), or another program uses the port: `START-HERE.bat --port 8081`. |
|
||||
| `cudaHostRegister ... out of memory` in the log | Normal on Windows: the engine pins the experts in per-layer slices instead. Only a problem if the load then fails. |
|
||||
| The first start takes minutes | It is reading 34-43 GB into RAM; the second start is faster while the files are in the OS cache. |
|
||||
| Slow output, disk light busy | Not enough free RAM: close other programs, or choose Q2_0 / IQ2_XS. |
|
||||
| `prompt ... exceeds the context` | The request is longer than the context you chose: run setup again with a bigger `--context`. |
|
||||
| Slower than the tables | The monitor plugged into the GPU and other GPU programs take VRAM from the expert cache; RAM running below its rated speed (enable EXPO/XMP in the BIOS) slows the CPU half. |
|
||||
| `this server was started without the vision encoder` | The model was set up for text only: run setup again with `--vision gpu`. |
|
||||
| A picture is refused or `cannot read the image` | Use JPEG, PNG, BMP or GIF (not WebP). |
|
||||
| Pictures are slow (10-30 s) | The encoder runs on the CPU: run setup again with `--vision gpu` (needs ~1.4 GB of VRAM). |
|
||||
| Anything else | The engine log is `strata-<model>.log` in this folder. |
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
<p align="center"><img src="docs/paper/tiers.svg" width="760" alt="memory tiers"></p>
|
||||
|
||||
- **GPU (VRAM):** attention and DeltaNet mixers, the gated-residual weights, routers, shared experts, output head, the MTP
|
||||
draft layer, the KV cache, and an **expert cache** that fills the rest of VRAM with the most-used experts (it adapts to
|
||||
the conversation while you chat).
|
||||
- **RAM:** all 24,576 experts, pinned. The CPU computes the experts that are not on the GPU **in place**, at the same time
|
||||
as the GPU works on the cached ones (AVX-512 / AVX2 kernels, ggml's for the i-quants).
|
||||
- **SSD:** the 28.8 GB n-gram table, read a few rows per token through the OS cache.
|
||||
- **Speculation:** the model's own MTP layer drafts up to 3 tokens; one pass over all 48 layers checks them. 2.4-3.2
|
||||
tokens per pass on average.
|
||||
- **Prompts** are processed in 2,048-token chunks with the experts streamed to the GPU over PCIe.
|
||||
|
||||
The full story, with measurements, bottlenecks and what comes next: **[docs/paper/Strata-Paper.pdf](docs/paper/Strata-Paper.pdf)**.
|
||||
|
||||
---
|
||||
|
||||
## Credits and licenses
|
||||
|
||||
- Model: [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team; quantizations:
|
||||
[ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF).
|
||||
Swift 1.5: [ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
by UkisAI. Their licenses apply to the weights.
|
||||
- [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT): the i-quant formats, the GPU dot products and
|
||||
dequantizers transcribed in `src/kernels/cuda/iq_kernels.cu`, the CPU backend linked for the i-quant experts, the
|
||||
`mtmd` library behind the image encoder (`tools/vision/`), and `gguf-py` used by the tools. See
|
||||
`third_party/ggml/LICENSE`.
|
||||
- Ideas from [Splash](https://github.com/incoai/splash), [ninfer](https://github.com/Neroued/ninfer) and
|
||||
[HyperQwen](https://github.com/syv-ai/HyperQwen); references in the paper.
|
||||
- Model: [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team; compressed versions by
|
||||
[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF);
|
||||
[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF) by UkisAI. Their licenses apply
|
||||
to the model files.
|
||||
- Built with parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT). Ideas from
|
||||
[Splash](https://github.com/incoai/splash), [ninfer](https://github.com/Neroued/ninfer) and
|
||||
[HyperQwen](https://github.com/syv-ai/HyperQwen). More in the [details](docs/DETAILS.md#credits-and-licenses).
|
||||
+337
@@ -0,0 +1,337 @@
|
||||
# Strata - the details
|
||||
|
||||
The technical side of Strata: every measured number, the API, images, all settings and how the engine works.
|
||||
New here? Start with the [README](../README.md) - it has everything you need to install and use it.
|
||||
|
||||
> **On this page:** [Speed](#speed-measured) · [Other GPUs](#other-gpus-estimated) · [Which model?](#which-model) ·
|
||||
> [Requirements](#before-you-start) · [Windows](#windows) · [Linux](#linux) · [API](#using-it) · [Images](#images-vision) ·
|
||||
> [Troubleshooting](#troubleshooting) · [How it works](#how-it-works)
|
||||
|
||||
---
|
||||
|
||||
## Speed (measured)
|
||||
|
||||
RTX 5070 **12 GB**, Ryzen 5 7600 (6 cores), 64 GB DDR5-5200, Windows. One code-agent prompt per length, 256 generated
|
||||
tokens, MTP speculative decoding on. "262K" is the model's full context window (a 259,943-token prompt).
|
||||
|
||||
### Prompt processing (tokens/s)
|
||||
|
||||
| Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 389 | 539 | 571 | 561 | 543 | 496 |
|
||||
| **IQ2_XS** | 332 | 463 | 495 | 486 | 472 | 437 |
|
||||
| **IQ3_XXS** | 285 | 410 | 435 | 427 | 414 | - |
|
||||
|
||||
### Output (tokens/s)
|
||||
|
||||
| Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 88.7 | 94.6 | 87.5 | 76.4 | 65.1 | 56.3 |
|
||||
| **IQ2_XS** | 82.0 | 78.0 | 65.3 | 63.7 | 52.0 | 48.0 |
|
||||
| **IQ3_XXS** | 64.6 | 65.6 | 57.3 | 54.4 | 44.8 | - |
|
||||
|
||||
IQ3_XXS at 262K is not measured: with its 43 GB of experts, a 260K-token context brings a 64 GB PC to its memory
|
||||
limit. Use up to 128K with IQ3_XXS on 64 GB.
|
||||
|
||||
Time to first token is prompt length / prompt speed: about 7 s at 4K, 55 s at 32K, 4 minutes at 128K and 9 minutes at
|
||||
262K. The raw numbers: [`bench/results/`](../bench/results/). The [paper](paper/Strata-Paper.pdf) explains every number.
|
||||
|
||||
## Other GPUs (estimated)
|
||||
|
||||
Not measured - estimated from the runs above (same CPU and 64 GB RAM): the GPU part scaled by memory bandwidth, the CPU
|
||||
part by how many more experts the card's VRAM holds. Treat as **±20%**. Numbers are *prompt / output* tokens/s.
|
||||
|
||||
| GPU | Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| RTX 5060 Ti 16GB | Q2_0 | ~341 / ~80 | ~472 / ~87 | ~501 / ~81 | ~492 / ~72 | ~476 / ~62 | ~435 / ~53 |
|
||||
| | IQ2_XS | ~291 / ~80 | ~406 / ~77 | ~434 / ~63 | ~426 / ~62 | ~413 / ~51 | ~383 / ~47 |
|
||||
| | IQ3_XXS | ~249 / ~66 | ~359 / ~65 | ~381 / ~56 | ~374 / ~54 | ~363 / ~45 | - |
|
||||
| RTX 3090 24GB | Q2_0 | ~355 / ~128 | ~491 / ~140 | ~521 / ~130 | ~512 / ~115 | ~495 / ~100 | ~453 / ~85 |
|
||||
| | IQ2_XS | ~303 / ~131 | ~422 / ~128 | ~451 / ~103 | ~444 / ~102 | ~430 / ~85 | ~398 / ~78 |
|
||||
| | IQ3_XXS | ~260 / ~106 | ~374 / ~103 | ~396 / ~89 | ~390 / ~85 | ~378 / ~71 | - |
|
||||
|
||||
More VRAM matters more than a faster GPU: every extra GB holds ~700 more experts, and every expert on the GPU is one the
|
||||
CPU does not have to compute. A 3090's 24 GB takes most of the CPU work away.
|
||||
|
||||
## Which model?
|
||||
|
||||
All three are [ISTA-DASLab's GSQ-RCO quantizations](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
of [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).
|
||||
|
||||
| Model | Download | RAM it uses | Speed | Quality |
|
||||
| --- | ---: | ---: | --- | --- |
|
||||
| **Q2_0** | 66 GB | ~34 GB experts + ~6 GB | fastest | good |
|
||||
| **IQ2_XS** | 68 GB | ~36 GB experts + ~6 GB | close to Q2_0 | a bit better |
|
||||
| **IQ3_XXS** | 76 GB | ~43 GB experts + ~6 GB | slower (more CPU work) | best |
|
||||
|
||||
With 64 GB of RAM all three fit (close the browser for IQ3_XXS, and keep its context at 128K or less). With 48 GB only Q2_0 / IQ2_XS may fit. 32 GB is not enough.
|
||||
|
||||
### Or: Swift 1.5 (a fine-tune that thinks shorter)
|
||||
|
||||
The setup's first question also offers **[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)**,
|
||||
UkisAI's fine-tune of Qwen3.8-Flash-Next, trained to reach the answer with much less thinking (its authors: 63% fewer
|
||||
thinking tokens, 1.8x sooner answers, under 1% accuracy loss). Same architecture, the same three sizes, its own
|
||||
vision encoder; Strata runs it at the same speed (4K, IQ2_XS: 465 prompt / 78.7 output tokens/s, vs 467 / 78.3 for
|
||||
the original). Its authors recommend **IQ2_XS** (their Q2_0 is marked experimental). Its license is the Swift Open
|
||||
License 1.0 - read it on the model page.
|
||||
|
||||
Our small check (8 reasoning questions, default thinking, IQ2_XS): both models got **8/8**; Swift used **1,234**
|
||||
output tokens in 28 s, the original **2,682** in 46 s - most of the difference from one question the original
|
||||
thought about for 1,524 tokens. Not a benchmark, but consistent with the claim.
|
||||
|
||||
```
|
||||
START-HERE.bat --setup --family swift --model IQ2_XS
|
||||
```
|
||||
|
||||
## Before you start
|
||||
|
||||
You need **only an NVIDIA driver** (version 580 or newer; update it with the NVIDIA App or from
|
||||
[nvidia.com/drivers](https://www.nvidia.com/drivers)). Everything else is installed for you the first time.
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| GPU | NVIDIA **RTX 30, 40 or 50 series**, **12 GB VRAM or more** (8 GB runs, slowly). Measured on an RTX 5070; RTX 30/40 are untested. |
|
||||
| RAM | **64 GB** recommended (see the table above). |
|
||||
| CPU | x86-64 with AVX2 (any Intel/AMD desktop CPU from the last ~8 years). AVX-512 (Ryzen 7000/9000) is a bit faster. |
|
||||
| Disk | ~70-80 GB free for the model, ~6 GB for the MTP layer (+1 GB with images). **Q2_0 on an AVX-512 CPU** also writes a one-time ~40 GB copy of its experts for the fast CPU kernel. An NVMe SSD is strongly recommended. |
|
||||
| OS | Windows 10/11, or Linux (Ubuntu 22.04/24.04 get everything installed automatically). |
|
||||
|
||||
What the first start installs, all inside this folder (`.venv/`, `engine/`, `third_party/`, `models/`, `packs/`, `mtp/`):
|
||||
Python 3.12 if you have none (for your user account, no admin), a private Python environment, NVIDIA's CUDA libraries
|
||||
(from pip, ~0.4 GB), the ready-made Strata engine for RTX 30/40/50, the model and the MTP draft layer. If no
|
||||
ready-made engine fits your PC, it offers to install the build tools (Visual Studio Build Tools + CUDA Toolkit on
|
||||
Windows, `build-essential` + CUDA on Ubuntu) and compiles the engine for your GPU (asks first; 20-40 minutes once).
|
||||
|
||||
---
|
||||
|
||||
## Windows
|
||||
|
||||
### Double-click `START-HERE.bat`
|
||||
|
||||
**The first time** it asks four questions and does the rest:
|
||||
|
||||
1. **Which model?** Qwen3.8-Flash-Next (the original) or Swift 1.5 (the fine-tune that thinks shorter).
|
||||
2. **Which size?** Q2_0, IQ2_XS or IQ3_XXS (it recommends one for your RAM).
|
||||
3. **How much context?** 8K to 256K tokens (it recommends one for your VRAM).
|
||||
4. **Images?** yes / no (see [Images](#images-vision)).
|
||||
|
||||
Then it downloads and prepares everything (the model is 66-76 GB, so the first start takes a while; an interrupted
|
||||
download continues where it stopped) and **starts the model**: your browser opens `http://127.0.0.1:8080`, a small
|
||||
page that shows it is running and lets you chat. The API is at `http://127.0.0.1:8080/v1` for your apps.
|
||||
|
||||
**Every time after that**, `START-HERE.bat` just starts the model (30-90 s to load 34-43 GB into RAM). Nothing is
|
||||
downloaded again. Closing the window stops the model.
|
||||
|
||||
```
|
||||
START-HERE.bat --setup install another model, or change context / images
|
||||
START-HERE.bat --model IQ2_XS --context 32768 --vision yes --yes no questions
|
||||
START-HERE.bat --gguf-dir D:\models\IQ2_XS use GGUF files you already have
|
||||
START-HERE.bat --port 8081 another port
|
||||
```
|
||||
|
||||
With more than one model installed, it asks which one to start. `run-<model>.bat` starts a model directly.
|
||||
|
||||
### Chat in the terminal (optional)
|
||||
|
||||
```
|
||||
.venv\Scripts\python chat.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Linux
|
||||
|
||||
```bash
|
||||
./setup.sh
|
||||
```
|
||||
|
||||
The same questions, the same automatic install (it uses `sudo apt` for Python and, only if it has to compile,
|
||||
for the build tools), and the same start: `http://127.0.0.1:8080`. Later runs of `./setup.sh` (or `./run-<model>.sh`)
|
||||
start the model directly. Options as on Windows (`./setup.sh --setup`, `--model Q2_0 --yes`, `--gguf-dir /data/Q2_0`).
|
||||
Terminal chat: `.venv/bin/python chat.py`.
|
||||
|
||||
---
|
||||
|
||||
## Using it
|
||||
|
||||
The server listens on `http://127.0.0.1:8080` (change with `--port` in setup, or edit the run script).
|
||||
|
||||
| API | Endpoint |
|
||||
| --- | --- |
|
||||
| OpenAI Chat Completions (stream and non-stream, tools) | `POST /v1/chat/completions` |
|
||||
| Anthropic Messages (stream and non-stream, tools) | `POST /v1/messages` |
|
||||
| Model list / health | `GET /v1/models`, `GET /health` |
|
||||
| What the model is doing right now | `GET /status` |
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
|
||||
"model": "strata", "messages": [{"role": "user", "content": "Write a haiku about GPUs."}], "max_tokens": 512 }'
|
||||
```
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
|
||||
r = client.chat.completions.create(model="strata", messages=[{"role": "user", "content": "Hello!"}])
|
||||
print(r.choices[0].message.content)
|
||||
```
|
||||
|
||||
- **Thinking levels: none, low, medium, high.** The model thinks before it answers (streamed as
|
||||
`reasoning_content`, Anthropic: `thinking` blocks). Choose how much per request - in the chat page (the "Thinking"
|
||||
menu), in `chat.py` (`/think low`), or over the API:
|
||||
|
||||
| API | how |
|
||||
| --- | --- |
|
||||
| OpenAI | `"reasoning_effort": "none" \| "low" \| "medium" \| "high"` (also `"reasoning": {"effort": ...}`, or `"chat_template_kwargs": {"enable_thinking": false}`) |
|
||||
| Anthropic | `"output_config": {"effort": "low" \| "medium" \| "high"}`, `"thinking": {"type": "disabled"}`, or `"thinking": {"type": "enabled", "budget_tokens": N}` (under 2K = low, under 8K = medium, more = high) |
|
||||
|
||||
Without a setting the model uses its own default, **high**. `none` answers at once (fastest); `low` keeps the thinking
|
||||
short. The levels are instructions the model was trained with, not a hard token limit: on easy questions all three
|
||||
think briefly, on hard ones `high` thinks longest and is most accurate.
|
||||
- **Streaming.** With `"stream": true` everything arrives as it is made: the thinking, the answer, and tool calls
|
||||
(the tool's name first, then its arguments piece by piece, like OpenAI and Anthropic do). While the model reads a
|
||||
long prompt the stream sends keep-alives, so agents do not time out; the server window prints progress every
|
||||
15 s, and `GET /status` says what it is doing (`reading the prompt`, `answering`, tokens so far). Closing the
|
||||
connection or pressing stop in your app really stops the model, so the next request starts at once.
|
||||
- **Chat apps.** Any app with an "OpenAI-compatible" provider works: base URL `http://127.0.0.1:8080/v1`, any API key.
|
||||
- **Context.** Chosen in setup (8K-262K). Requests longer than that are refused, never silently cut.
|
||||
- **From other devices / the internet.** The server listens on your PC only (`127.0.0.1`). To reach it from elsewhere,
|
||||
put a tunnel in front of it, for example [cloudflared](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/do-more-with-tunnels/trycloudflare/):
|
||||
`cloudflared tunnel --url http://127.0.0.1:8080`. **Set a key first**, or anyone with the link can use your PC:
|
||||
add `"api_key": "some-long-secret"` to `strata-<model>.json` (or set the `STRATA_API_KEY` environment variable);
|
||||
clients then send it as their API key.
|
||||
|
||||
**Conversation cache.** A request that continues a chat reads only the part after what the engine already holds: the
|
||||
live session, or one of the checkpoints it keeps in RAM (up to 6, ~118 MB each, taken at the start of each new
|
||||
assistant turn and every 16K prompt tokens). A checkpoint is used only when the prompt starts with exactly its tokens
|
||||
and pictures. Engine options: `--prompt-cache N` (0 = off), `--prompt-cache-every N`, `--turn-token ID`.
|
||||
|
||||
**Current limits (v1):** one request at a time, and one conversation cached at a time (switching between two chats
|
||||
re-reads the other one); greedy decoding (temperature is ignored); images only when set up with them (below); no
|
||||
video.
|
||||
|
||||
---
|
||||
|
||||
## Images (vision)
|
||||
|
||||
The model has a vision encoder: [`mmproj-Qwen3.8-Flash-Next-BF16.gguf`](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
(0.9 GB, a 27-layer ViT plus the projector into the language model). It is **optional**: say yes when the setup asks
|
||||
"Images?", or run it again with `--vision gpu` (or `--vision cpu`). The setup downloads the encoder, builds a small
|
||||
helper (`strata-vision`, from llama.cpp's `mtmd` library) and adds it to your start script. Nothing else changes.
|
||||
|
||||
| Encoder on | Time per picture | Cost |
|
||||
| --- | --- | --- |
|
||||
| **GPU** (recommended) | **0.1-0.5 s** (up to 1,024 image tokens) | ~1.4 GB of VRAM is kept free for it, so the expert cache is smaller: text output is a few % slower (table below) |
|
||||
| CPU | 10-30 s (pictures are scaled down to ~300 image tokens) | nothing on the GPU |
|
||||
|
||||
A picture becomes up to 1,024 tokens of the context (a 640x480 photo: 300). The same picture sent again, as chat apps
|
||||
do on every turn, is encoded only once.
|
||||
|
||||
### Sending a picture
|
||||
|
||||
**Terminal chat:** type `/image <path to a picture>`, press Enter, then type your question.
|
||||
|
||||
```
|
||||
you> /image C:\Users\me\Pictures\receipt.jpg
|
||||
(picture attached: receipt.jpg - now type your question)
|
||||
you> What is the total on this receipt?
|
||||
```
|
||||
|
||||
**OpenAI API** (an `image_url` part: a `data:` URL, an `http(s)://` URL or a local file path):
|
||||
|
||||
```python
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
|
||||
img = base64.b64encode(open("photo.jpg", "rb").read()).decode()
|
||||
r = client.chat.completions.create(model="strata", messages=[{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img}"}},
|
||||
{"type": "text", "text": "What is in this picture?"}]}])
|
||||
print(r.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Anthropic API:** an `image` block with a `base64` (or `url`) source, as usual.
|
||||
|
||||
JPEG, PNG, BMP, GIF, WebP, TIFF and AVIF work (the last ones are converted to PNG first; agents such as omp send
|
||||
WebP). Chat apps with image upload work the same way.
|
||||
|
||||
### Speed with images on (4K context, measured)
|
||||
|
||||
| Model | Prompt tok/s, images off | Prompt tok/s, images on | Output tok/s, images off | Output tok/s, images on |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 541 | 531 | 90.7 | 86.9 |
|
||||
| **IQ2_XS** | 468 | 458 | 77.1 | 74.0 |
|
||||
| **IQ3_XXS** | 411 | 401 | 64.7 | 59.5 |
|
||||
|
||||
"Off" is the published setup; "on" runs with the encoder loaded on the GPU and its VRAM kept free, so ~1,000 fewer
|
||||
experts fit in VRAM and the CPU computes a few more per token: 2-8% slower. For text, turning images on changes
|
||||
nothing else (the output is bit-identical when the VRAM is the same).
|
||||
|
||||
A question about a picture (a 640x480 newspaper page = 300 image tokens, a 328-token prompt; the whole request, encoder
|
||||
on the GPU, measured through the API):
|
||||
|
||||
| Model | Answer ("MEN WALK ON MOON") | Picture encoded | Prompt | Output |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 1.9 s | ~0.1 s | 207 tok/s | 65-73 tok/s |
|
||||
| **IQ2_XS** | 2.3 s | ~0.1 s | 173 tok/s | 50-60 tok/s |
|
||||
| **IQ3_XXS** | 2.7 s | ~0.1 s | 144 tok/s | 41-50 tok/s |
|
||||
|
||||
(Short prompts run below the 4K prompt speed: a 2,048-token chunk is where the prompt path is efficient. Short answers
|
||||
run below the long-output speed: the first rounds have no draft yet.)
|
||||
|
||||
**How it works inside:** the encoder turns the picture into rows of the same width as the model's word embeddings;
|
||||
Strata puts them where the prompt has `<|image_pad|>` tokens and gives each one its 2-D position (row and column in the
|
||||
picture; the model uses interleaved M-RoPE). Answers match llama.cpp's multimodal implementation token for token on our
|
||||
test images.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | What to do |
|
||||
| --- | --- |
|
||||
| `the NVIDIA driver is too old` | Update the driver (NVIDIA App or nvidia.com/drivers), restart, run `START-HERE.bat` again. |
|
||||
| Python or the build tools could not be installed | Install what it names (links are printed), then run it again. Everything already done is kept. |
|
||||
| `port 8080 is already in use` | Strata is already running (look for its window), or another program uses the port: `START-HERE.bat --port 8081`. |
|
||||
| `cudaHostRegister ... out of memory` in the log | Normal on Windows: the engine pins the experts in per-layer slices instead. Only a problem if the load then fails. |
|
||||
| The first start takes minutes | It is reading 34-43 GB into RAM; the second start is faster while the files are in the OS cache. |
|
||||
| Slow output, disk light busy | Not enough free RAM: close other programs, or choose Q2_0 / IQ2_XS. |
|
||||
| `prompt ... exceeds the context` | The request is longer than the context you chose: run setup again with a bigger `--context`. |
|
||||
| Slower than the tables | The monitor plugged into the GPU and other GPU programs take VRAM from the expert cache; RAM running below its rated speed (enable EXPO/XMP in the BIOS) slows the CPU half. |
|
||||
| `this server was started without the vision encoder` | The model was set up for text only: run setup again with `--vision gpu`. |
|
||||
| A picture is refused or `cannot read the image` | The file is not a picture Pillow can open (JPEG, PNG, WebP, GIF, BMP, TIFF, AVIF work). |
|
||||
| Pictures are slow (10-30 s) | The encoder runs on the CPU: run setup again with `--vision gpu` (needs ~1.4 GB of VRAM). |
|
||||
| A request never finishes (older engines, mostly with pictures) | Run `START-HERE.bat` once to get engine 0.1.2 or newer. It keeps a real margin of VRAM free: the log says `... MiB of VRAM free with everything loaded` and warns when it is close to 0. |
|
||||
| Anything else | The engine log is `strata-<model>.log` in this folder. |
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
<p align="center"><img src="paper/tiers.svg" width="760" alt="memory tiers"></p>
|
||||
|
||||
- **GPU (VRAM):** attention and DeltaNet mixers, the gated-residual weights, routers, shared experts, output head, the MTP
|
||||
draft layer, the KV cache, and an **expert cache** that fills the rest of VRAM with the most-used experts (it adapts to
|
||||
the conversation while you chat).
|
||||
- **RAM:** all 24,576 experts, pinned. The CPU computes the experts that are not on the GPU **in place**, at the same time
|
||||
as the GPU works on the cached ones (AVX-512 / AVX2 kernels, ggml's for the i-quants).
|
||||
- **SSD:** the 28.8 GB n-gram table, read a few rows per token through the OS cache.
|
||||
- **Speculation:** the model's own MTP layer drafts up to 3 tokens; one pass over all 48 layers checks them. 2.4-3.2
|
||||
tokens per pass on average.
|
||||
- **Prompts** are processed in 2,048-token chunks with the experts streamed to the GPU over PCIe.
|
||||
|
||||
The full story, with measurements, bottlenecks and what comes next: **[docs/paper/Strata-Paper.pdf](paper/Strata-Paper.pdf)**.
|
||||
|
||||
---
|
||||
|
||||
## Credits and licenses
|
||||
|
||||
- Model: [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team; quantizations:
|
||||
[ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF).
|
||||
Swift 1.5: [ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
by UkisAI. Their licenses apply to the weights.
|
||||
- [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT): the i-quant formats, the GPU dot products and
|
||||
dequantizers transcribed in `src/kernels/cuda/iq_kernels.cu`, the CPU backend linked for the i-quant experts, the
|
||||
`mtmd` library behind the image encoder (`tools/vision/`), and `gguf-py` used by the tools. See
|
||||
`third_party/ggml/LICENSE`.
|
||||
- Ideas from [Splash](https://github.com/incoai/splash), [ninfer](https://github.com/Neroued/ninfer) and
|
||||
[HyperQwen](https://github.com/syv-ai/HyperQwen); references in the paper.
|
||||
@@ -46,6 +46,8 @@ public:
|
||||
/// The prompt's length: prefill() skips the cells the attention window can never reach again.
|
||||
void set_prompt_len(int64_t n) { prompt_len_ = n; }
|
||||
uint64_t vram_bytes() const { return vram_; }
|
||||
/// The draft layer's K/V state (read-only: --serve's STRATA_STATE_HASH check hashes it)
|
||||
const QsaState& kv_state() const { return st_; }
|
||||
/// The main model's embedding and head, and the verify window's final residuals (T rows, hc*n_embd each).
|
||||
bool bind(const WeightTable& wt, const NativeHead* head, const float* window_R, std::string& err);
|
||||
|
||||
|
||||
@@ -23,7 +23,10 @@
|
||||
#include "strata/kernels/cpu/native_expert.hpp"
|
||||
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <condition_variable>
|
||||
#include <cstdint>
|
||||
#include <mutex>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
@@ -136,10 +139,19 @@ public:
|
||||
repark = ms_repark_;
|
||||
}
|
||||
|
||||
/// **A PARKED WORKER SPINS FOR THIS LONG, THEN SLEEPS.** The park is a `_mm_pause` spin because a layer's
|
||||
/// batches are microseconds apart and a wake-up from the OS costs more than that. But a spin that never ends
|
||||
/// keeps every worker's core at 100% while the engine waits for a request - issue #4, "CPU 50% even when
|
||||
/// doing nothing" (7 of the 5700X's 16 threads). Inside a request the gaps between batches are far below
|
||||
/// this, so the token path still never sleeps; between requests the workers block on `sleep_cv_`.
|
||||
static constexpr std::chrono::milliseconds kSpinBeforeSleep{20};
|
||||
|
||||
private:
|
||||
void worker(int id);
|
||||
void drain(int id, ExpertScratch& scratch);
|
||||
void run_phase(int mode, int n_tasks);
|
||||
/// Bump `epoch_`, and wake the workers that went to sleep. Every publish goes through here.
|
||||
void publish();
|
||||
|
||||
int n_ = 0;
|
||||
bool host_works_ = true;
|
||||
@@ -169,6 +181,11 @@ private:
|
||||
alignas(64) std::atomic<uint32_t> parked_{0};
|
||||
alignas(64) std::atomic<uint32_t> epoch_{0};
|
||||
alignas(64) std::atomic<bool> stop_{false};
|
||||
// The sleep after `kSpinBeforeSleep`. `sleepers_` is how `publish` knows whether anyone needs waking, so the
|
||||
// token path pays one uncontended load per publish and never takes the mutex while the workers spin.
|
||||
alignas(64) std::atomic<uint32_t> sleepers_{0};
|
||||
std::mutex sleep_mu_;
|
||||
std::condition_variable sleep_cv_;
|
||||
std::vector<std::thread> threads_;
|
||||
std::vector<ExpertScratch> scratch_; // one per worker: no allocation, no false sharing of the hot data
|
||||
// run_split state: mode 0 = whole experts, 1 = gate/up row parts, 2 = down row parts
|
||||
|
||||
@@ -62,6 +62,9 @@ public:
|
||||
/// K/V from them. The prefill stream is synchronized before the call.
|
||||
std::function<bool(const float* R_rows, int64_t T, int64_t pos0, std::string& err)> on_chunk;
|
||||
|
||||
/// Checked before every chunk: true stops the prompt early (`run` returns false with err "cancelled").
|
||||
std::function<bool()> should_stop;
|
||||
|
||||
/// The vision path: HOST rows (n_embd floats) indexed by absolute position, read in place of the token
|
||||
/// embedding where non-null (an image's <|image_pad|> cells). Null (default): every position embeds its token.
|
||||
const float* const* embd_rows = nullptr;
|
||||
|
||||
+137
-2
@@ -270,11 +270,127 @@ class OutputParser:
|
||||
"""Incremental parser of the model's text. Feed deltas; get events. A tag split across deltas is held back
|
||||
until it is complete, so clients never see `<tool_` or `</thi`."""
|
||||
|
||||
def __init__(self, thinking: bool = True, tools: list[dict] | None = None):
|
||||
def __init__(self, thinking: bool = True, tools: list[dict] | None = None, stream_tools: bool = False):
|
||||
self.state = "reasoning" if thinking else "content"
|
||||
self.buf = ""
|
||||
self.lead = False
|
||||
self.schemas = {t.get("name"): t for t in tools or []}
|
||||
# stream_tools: a tool call is also reported while it is being written - "tool_start" (its name and id) as
|
||||
# soon as the name is known, then "tool_args" pieces of its JSON arguments (string parameters character by
|
||||
# character; other types whole, once complete) - before the final "tool_call". Without it, a client sees
|
||||
# nothing until the call is complete, which for a large file write can be many minutes.
|
||||
self.stream_tools = stream_tools
|
||||
self._reset_scan()
|
||||
|
||||
def _reset_scan(self):
|
||||
self.sp = 0 # how much of self.buf (the call body) the scanner has consumed
|
||||
self.ss = "name" # name -> between -> str|raw -> ... -> done
|
||||
self.scall = None # the ToolCall being streamed (its id is reused by the final event)
|
||||
self.sfirst = True
|
||||
self.sval_started = False
|
||||
self.sdeclared = {}
|
||||
|
||||
def _scan(self) -> list[Event]:
|
||||
"""Advance the streaming view of the call body in self.buf (see stream_tools)."""
|
||||
out = []
|
||||
|
||||
def args(s):
|
||||
if s:
|
||||
out.append(Event("tool_args", s, call=self.scall))
|
||||
while True:
|
||||
rest = self.buf[self.sp:]
|
||||
if self.ss == "name":
|
||||
a = rest.find("<function=")
|
||||
b = rest.find(">", a + 10) if a >= 0 else -1
|
||||
if b < 0:
|
||||
return out
|
||||
name = rest[a + 10:b]
|
||||
self.scall = ToolCall(name=name, arguments={})
|
||||
props = ((self.schemas.get(name) or {}).get("parameters") or {}).get("properties") or {}
|
||||
self.sdeclared = {k: (v or {}).get("type") for k, v in props.items()}
|
||||
out.append(Event("tool_start", call=self.scall))
|
||||
args("{")
|
||||
self.sp += b + 1
|
||||
self.ss = "between"
|
||||
elif self.ss == "between":
|
||||
stripped = rest.lstrip()
|
||||
self.sp += len(rest) - len(stripped)
|
||||
if stripped.startswith("<parameter="):
|
||||
b = stripped.find(">")
|
||||
if b < 0:
|
||||
return out
|
||||
pname = stripped[11:b]
|
||||
args(("" if self.sfirst else ",") + json.dumps(pname) + ":")
|
||||
self.sfirst = False
|
||||
self.sp += b + 1
|
||||
if self.sdeclared.get(pname) == "string":
|
||||
args('"')
|
||||
self.ss, self.sval_started = "str", False
|
||||
else:
|
||||
self.ss = "raw"
|
||||
elif stripped.startswith("</function>"):
|
||||
args("}")
|
||||
self.sp += len("</function>")
|
||||
self.ss = "done"
|
||||
else:
|
||||
return out # a tag still arriving (or trailing text): wait
|
||||
elif self.ss == "str":
|
||||
if not self.sval_started:
|
||||
if not rest:
|
||||
return out
|
||||
if rest[0] == "\n": # the value's leading newline is not part of it
|
||||
self.sp += 1
|
||||
rest = rest[1:]
|
||||
self.sval_started = True
|
||||
end = rest.find("</parameter>")
|
||||
if end >= 0:
|
||||
value = rest[:end]
|
||||
if value.endswith("\n"):
|
||||
value = value[:-1]
|
||||
args(json.dumps(value)[1:-1] + '"')
|
||||
self.sp += end + len("</parameter>")
|
||||
self.ss = "between"
|
||||
continue
|
||||
safe = len(rest) - self._hold(rest, ("</parameter>",))
|
||||
if safe > 0 and rest[safe - 1] == "\n": # may be the trailing newline before </parameter>
|
||||
safe -= 1
|
||||
if safe > 0:
|
||||
args(json.dumps(rest[:safe])[1:-1])
|
||||
self.sp += safe
|
||||
return out
|
||||
elif self.ss == "raw":
|
||||
end = rest.find("</parameter>")
|
||||
if end < 0:
|
||||
return out
|
||||
value = rest[:end]
|
||||
if value.startswith("\n"):
|
||||
value = value[1:]
|
||||
if value.endswith("\n"):
|
||||
value = value[:-1]
|
||||
try:
|
||||
v = json.loads(value)
|
||||
except ValueError:
|
||||
v = value
|
||||
args(json.dumps(v, ensure_ascii=False))
|
||||
self.sp += end + len("</parameter>")
|
||||
self.ss = "between"
|
||||
else:
|
||||
return out
|
||||
|
||||
def _close_scan(self) -> list[Event]:
|
||||
"""A streamed call that ended without a clean </function>: close its JSON so clients can still parse it."""
|
||||
out = []
|
||||
if self.scall is None or self.ss == "done":
|
||||
return out
|
||||
tail = ""
|
||||
if self.ss == "str":
|
||||
tail += '"'
|
||||
elif self.ss == "raw":
|
||||
tail += json.dumps(self.buf[self.sp:].strip("\n"))
|
||||
tail += "}"
|
||||
out.append(Event("tool_args", tail, call=self.scall))
|
||||
self.ss = "done"
|
||||
return out
|
||||
|
||||
def _hold(self, text: str, tags: tuple[str, ...]) -> int:
|
||||
"""Length of the longest suffix of `text` that is a proper prefix of one of `tags`."""
|
||||
@@ -325,17 +441,36 @@ class OutputParser:
|
||||
self.state = "call"
|
||||
else:
|
||||
i = self.buf.find(CALL_END)
|
||||
if self.stream_tools:
|
||||
if i >= 0:
|
||||
whole, self.buf = self.buf, self.buf[:i] # scan only the body
|
||||
out += self._scan()
|
||||
out += self._close_scan()
|
||||
self.buf = whole
|
||||
else:
|
||||
out += self._scan()
|
||||
if i < 0:
|
||||
return out
|
||||
body = self.buf[:i]
|
||||
self.buf = self.buf[i + len(CALL_END):]
|
||||
name = body.strip()[len("<function="):].split(">", 1)[0]
|
||||
out.append(Event("tool_call", call=parse_tool_call(body, self.schemas.get(name))))
|
||||
call = parse_tool_call(body, self.schemas.get(name))
|
||||
if self.scall is not None:
|
||||
call.id = self.scall.id
|
||||
out.append(Event("tool_call", call=call))
|
||||
self._reset_scan()
|
||||
self.state, self.lead = "content", True
|
||||
|
||||
def finish(self) -> list[Event]:
|
||||
"""End of generation: flush whatever is held (an unterminated tool call is returned as content)."""
|
||||
out = []
|
||||
if self.state == "call" and self.stream_tools and self.scall is not None:
|
||||
out += self._scan() # the output ended inside a call that was already announced
|
||||
out += self._close_scan()
|
||||
out.append(Event("tool_call", call=self.scall))
|
||||
self.buf = ""
|
||||
self._reset_scan()
|
||||
return out
|
||||
if self.buf:
|
||||
kind = {"reasoning": "reasoning", "content": "content"}.get(self.state, "content")
|
||||
text = self.buf if self.state != "call" else CALL_START + self.buf
|
||||
|
||||
+272
-39
@@ -6,8 +6,9 @@
|
||||
Endpoints: POST /v1/chat/completions (OpenAI, stream and non-stream), POST /v1/messages (Anthropic, stream and
|
||||
non-stream), GET /v1/models, GET /health. One sequence at a time behind a FIFO (plan: one resident sequence).
|
||||
Images (optional, when the config has a "vision" entry): OpenAI image_url parts and Anthropic image blocks (base64
|
||||
data, http(s) URLs or local file paths; JPEG/PNG/BMP/GIF) go through `strata-vision` (the model's mmproj file) and
|
||||
reach the engine as embeddings (`GENI`).
|
||||
data, http(s) URLs or local file paths) go through `strata-vision` (the model's mmproj file) and reach the engine as
|
||||
embeddings (`GENI`). JPEG/PNG/BMP/GIF go straight in; WebP, TIFF, AVIF, ... (agents like omp send WebP) are
|
||||
converted to PNG first with Pillow.
|
||||
Requests whose prompt plus max tokens exceed the engine's context are REJECTED with 400, never truncated.
|
||||
|
||||
The engine boundary is `Engine.generate(prompt_ids, max_new, sampling, cancel) -> iterator of token ids`.
|
||||
@@ -21,6 +22,7 @@ import base64
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
@@ -77,38 +79,79 @@ class StrataEngine:
|
||||
self.proc = subprocess.Popen([exe, "--serve", *args], cwd=cwd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
|
||||
stderr=self.log, text=True, encoding="utf-8", bufsize=1, env=env)
|
||||
self.max_context = 0
|
||||
self.can_stop = False # the engine honours a STOP line mid-request (READY <ctx> stop)
|
||||
self.last = {}
|
||||
for line in self.proc.stdout:
|
||||
if line.startswith("READY"):
|
||||
self.max_context = int(line.split()[1])
|
||||
f = line.split()
|
||||
self.max_context = int(f[1])
|
||||
self.can_stop = "stop" in f[2:]
|
||||
break
|
||||
if self.max_context <= 0:
|
||||
raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else ""))
|
||||
# the engine's stdout on a thread, so a request can wait with a timeout (heartbeats, cancel checks)
|
||||
self.lines: queue.Queue = queue.Queue()
|
||||
threading.Thread(target=self._pump, daemon=True).start()
|
||||
|
||||
def _pump(self):
|
||||
for line in self.proc.stdout:
|
||||
self.lines.put(line)
|
||||
self.lines.put(None)
|
||||
|
||||
def _parse_done(self, line):
|
||||
f = line.split()
|
||||
self.last = {"generated": int(f[1]), "prompt_tokens": int(f[2]), "prompt_ms": float(f[3]),
|
||||
"decode_ms": float(f[4]), "finish": f[5]}
|
||||
|
||||
def generate(self, ids, max_new, sampling, cancel, embeddings=None):
|
||||
"""Yields token ids, and None as a heartbeat every 10 s while the engine is quiet (reading a long prompt):
|
||||
the HTTP layer turns it into an SSE comment, which keeps clients' watchdogs calm and notices a client that
|
||||
has gone. A consumer that stops early (or `cancel`) makes the engine STOP, so it does not run to max_new."""
|
||||
head = f"GENI {int(max_new)} {embeddings}" if embeddings else f"GEN {int(max_new)}"
|
||||
self.proc.stdin.write(f"{head} {','.join(str(int(t)) for t in ids)}\n")
|
||||
self.proc.stdin.flush()
|
||||
done = False
|
||||
try:
|
||||
for line in self.proc.stdout:
|
||||
while True:
|
||||
try:
|
||||
line = self.lines.get(timeout=10)
|
||||
except queue.Empty:
|
||||
if cancel.is_set():
|
||||
return
|
||||
yield None
|
||||
continue
|
||||
if line is None:
|
||||
done = True
|
||||
raise RuntimeError("the engine process ended")
|
||||
if line.startswith("T "):
|
||||
if not cancel.is_set():
|
||||
yield int(line[2:])
|
||||
if cancel.is_set():
|
||||
return
|
||||
yield int(line[2:])
|
||||
elif line.startswith("PP "): # prompt progress, one per chunk: also a heartbeat (the
|
||||
if cancel.is_set(): # lines reset the 10 s wait, so without this a long prompt
|
||||
return # would send no keep-alives at all)
|
||||
yield None
|
||||
elif line.startswith("DONE"):
|
||||
f = line.split()
|
||||
self.last = {"generated": int(f[1]), "prompt_tokens": int(f[2]), "prompt_ms": float(f[3]),
|
||||
"decode_ms": float(f[4]), "finish": f[5]}
|
||||
self._parse_done(line)
|
||||
done = True
|
||||
return
|
||||
elif line.startswith("ERR"):
|
||||
done = True
|
||||
raise ValueError(line[4:].strip())
|
||||
raise RuntimeError("the engine process ended")
|
||||
finally:
|
||||
if not done: # the consumer stopped early: drain to DONE
|
||||
for line in self.proc.stdout:
|
||||
if line.startswith("DONE") or line.startswith("ERR"):
|
||||
if not done: # the consumer stopped early: stop the engine, drain to DONE
|
||||
if self.can_stop:
|
||||
try:
|
||||
self.proc.stdin.write("STOP\n")
|
||||
self.proc.stdin.flush()
|
||||
except OSError:
|
||||
pass
|
||||
while True:
|
||||
line = self.lines.get()
|
||||
if line is None or line.startswith("ERR"):
|
||||
break
|
||||
if line.startswith("DONE"):
|
||||
self._parse_done(line)
|
||||
break
|
||||
|
||||
def close(self):
|
||||
@@ -155,9 +198,37 @@ class Vision:
|
||||
return Path(path).read_bytes()
|
||||
raise ValueError("an image must be a data: URL, an http(s) URL or a local file path")
|
||||
|
||||
@staticmethod
|
||||
def normalize(data: bytes) -> bytes:
|
||||
"""The formats strata-vision's decoder (stb_image) reads pass through; anything else is converted to PNG."""
|
||||
if data[:3] == b"\xff\xd8\xff" or data[:8] == b"\x89PNG\r\n\x1a\n" or data[:2] == b"BM" or \
|
||||
data[:6] in (b"GIF87a", b"GIF89a"):
|
||||
return data
|
||||
try:
|
||||
import io
|
||||
from PIL import Image
|
||||
except ImportError:
|
||||
raise ValueError("this image format needs Pillow (python -m pip install pillow); JPEG, PNG, BMP and "
|
||||
"GIF work without it") from None
|
||||
try:
|
||||
im = Image.open(io.BytesIO(data))
|
||||
im.load()
|
||||
except Exception as e:
|
||||
raise ValueError(f"the image could not be read ({e})") from None
|
||||
if im.mode in ("RGBA", "LA", "P") and "transparency" in im.info or im.mode in ("RGBA", "LA"):
|
||||
im = im.convert("RGBA")
|
||||
bg = Image.new("RGB", im.size, (255, 255, 255)) # transparent areas become white, not black
|
||||
bg.paste(im, mask=im.split()[-1])
|
||||
im = bg
|
||||
elif im.mode != "RGB":
|
||||
im = im.convert("RGB")
|
||||
out = io.BytesIO()
|
||||
im.save(out, format="PNG")
|
||||
return out.getvalue()
|
||||
|
||||
def encode(self, source: str) -> tuple[Path, int]:
|
||||
"""-> (embeddings file, number of image tokens)."""
|
||||
data = self.load(source)
|
||||
data = self.normalize(self.load(source))
|
||||
key = hashlib.sha256(data).hexdigest()[:32]
|
||||
with self.lock:
|
||||
if key in self.cache:
|
||||
@@ -245,6 +316,8 @@ class Service:
|
||||
self.fifo = threading.Lock()
|
||||
self.embeddings = threading.local() # the current request's image embeddings file (GENI)
|
||||
self.api_key = "" # when set, /v1/* needs it (Bearer or x-api-key)
|
||||
self.status = {"busy": False, "queued": 0} # GET /status: what the model is doing right now
|
||||
self.status_lock = threading.Lock()
|
||||
self.stop_ids = set(tokenizer.encode(IM_END, parse_special=True) +
|
||||
tokenizer.encode("<|endoftext|>", parse_special=True))
|
||||
|
||||
@@ -258,7 +331,12 @@ class Service:
|
||||
raise ValueError("this server was started without the vision encoder (run setup again and choose "
|
||||
"'vision'), so it cannot read images")
|
||||
pad = self.tok.encode(IMAGE_PAD, parse_special=True)[0]
|
||||
encoded = [self.vision.encode(src) for src in images]
|
||||
# Encode only while the engine is idle: the engine and the image encoder (a separate process) must not
|
||||
# run on the GPU at the same time - an encode during a running request left that request stuck at
|
||||
# "reading the prompt" with CPU and GPU busy, for good (reproduced). So encoding takes its turn in the
|
||||
# same FIFO as the requests.
|
||||
with self.fifo:
|
||||
encoded = [self.vision.encode(src) for src in images]
|
||||
out, k = [], 0
|
||||
for t in ids: # one <|image_pad|> per image -> one per image token
|
||||
if t == pad and k < len(encoded):
|
||||
@@ -279,29 +357,116 @@ class Service:
|
||||
f"({self.engine.max_context}); requests are never truncated")
|
||||
return ids, kwargs.get("enable_thinking", True) is not False
|
||||
|
||||
def _note(self, n, evs):
|
||||
with self.status_lock:
|
||||
s = self.status
|
||||
s["generated"] = n
|
||||
if s.get("first_token") is None:
|
||||
s["first_token"] = time.time()
|
||||
for ev in evs:
|
||||
if ev.kind == "reasoning":
|
||||
s["phase"] = "thinking"
|
||||
elif ev.kind == "content":
|
||||
s["phase"] = "answering"
|
||||
elif ev.kind == "tool_start":
|
||||
s["phase"], s["tool"] = f"writing a tool call: {ev.call.name}", ev.call.name
|
||||
elif ev.kind == "tool_call":
|
||||
s["phase"] = "tool call complete"
|
||||
s["tail"] = (s["tail"] + (ev.text or ""))[-600:]
|
||||
|
||||
def _progress(self, last_print, every=1.0):
|
||||
"""A progress line in the server window every `every` seconds while a request runs."""
|
||||
now = time.time()
|
||||
if now - last_print < every:
|
||||
return last_print
|
||||
with self.status_lock:
|
||||
s = dict(self.status)
|
||||
el = now - s.get("started", now)
|
||||
if s.get("first_token") is None:
|
||||
print(f"[strata] reading the prompt: {s.get('prompt_tokens', 0)} tokens, {el:.0f} s so far", flush=True)
|
||||
else:
|
||||
rate = s["generated"] / max(1e-6, now - s["first_token"])
|
||||
print(f"[strata] {s['phase']}: {s['generated']} of max {s.get('max_tokens')} tokens, {rate:.1f} tok/s, "
|
||||
f"{el:.0f} s", flush=True)
|
||||
return now
|
||||
|
||||
def run(self, ids, thinking, tools, max_new, sampling, cancel) -> Iterator[tuple[str, object]]:
|
||||
"""Yields ("event", Event) as text arrives, then ("done", {"finish": .., "completion_tokens": ..})."""
|
||||
parser, detok, n, finish = OutputParser(thinking=thinking, tools=tools), Detokenizer(self.tok), 0, "length"
|
||||
parser = OutputParser(thinking=thinking, tools=tools, stream_tools=True)
|
||||
detok, n, finish = Detokenizer(self.tok), 0, "length"
|
||||
raw_ids = [] # every generated id (STRATA_DEBUG: dump raw model text)
|
||||
emb = getattr(self.embeddings, "path", None)
|
||||
with self.status_lock:
|
||||
self.status["queued"] += 1
|
||||
try:
|
||||
with self.fifo:
|
||||
with self.status_lock:
|
||||
self.status["queued"] -= 1
|
||||
self.status.update(busy=True, phase="reading the prompt", prompt_tokens=len(ids), generated=0,
|
||||
started=time.time(), first_token=None, tool=None, tail="", max_tokens=max_new)
|
||||
last_print = time.time()
|
||||
gen = self.engine.generate(ids, max_new, sampling, cancel, embeddings=emb) if emb else \
|
||||
self.engine.generate(ids, max_new, sampling, cancel)
|
||||
for t in gen:
|
||||
n += 1
|
||||
if t in self.stop_ids:
|
||||
finish = "stop"
|
||||
break
|
||||
for ev in parser.feed(detok.push(t)):
|
||||
yield "event", ev
|
||||
try:
|
||||
for t in gen:
|
||||
if t is None: # heartbeat while the engine is quiet
|
||||
last_print = self._progress(last_print)
|
||||
yield "ping", None
|
||||
continue
|
||||
n += 1
|
||||
if t in self.stop_ids:
|
||||
finish = "stop"
|
||||
raw_ids.append(t)
|
||||
break
|
||||
raw_ids.append(t)
|
||||
evs = parser.feed(detok.push(t))
|
||||
self._note(n, evs)
|
||||
last_print = self._progress(last_print)
|
||||
for ev in evs:
|
||||
yield "event", ev
|
||||
if cancel.is_set():
|
||||
finish = "cancel"
|
||||
finally:
|
||||
gen.close() # STOP+drain to THIS request's DONE while still holding the
|
||||
# fifo, so a stop-token break can't leave the shared engine
|
||||
# queue mid-drain for the next request to read as its own DONE
|
||||
except GeneratorExit: # the client disconnected mid-stream
|
||||
finish = "disconnect"
|
||||
raise
|
||||
finally:
|
||||
if emb:
|
||||
Path(emb).unlink(missing_ok=True)
|
||||
with self.status_lock:
|
||||
if self.status.get("busy"):
|
||||
now = time.time()
|
||||
el = now - self.status.get("started", now)
|
||||
ft = self.status.get("first_token")
|
||||
rate = n / max(1e-6, now - ft) if ft else 0.0
|
||||
print(f"[strata] done: {n} tokens in {el:.0f} s ({rate:.1f} tok/s) "
|
||||
f"({finish}, cancel={cancel.is_set()})", flush=True)
|
||||
if os.environ.get("STRATA_DEBUG") and raw_ids:
|
||||
print(f"[strata] raw: {self.tok.decode(raw_ids)!r}", flush=True)
|
||||
self.status["busy"] = False
|
||||
for ev in parser.finish():
|
||||
yield "event", ev
|
||||
yield "done", {"finish": finish, "completion_tokens": n}
|
||||
|
||||
|
||||
def _debug_req(api, req, messages, tools, max_new, thinking, prompt_tokens):
|
||||
"""One compact line per request while diagnosing blank/empty turns. Set STRATA_DEBUG=1 to enable."""
|
||||
if not os.environ.get("STRATA_DEBUG"):
|
||||
return
|
||||
last = messages[-1] if messages else {}
|
||||
body = last.get("content")
|
||||
if isinstance(body, list):
|
||||
body = " ".join(p.get("text", "") for p in body if isinstance(p, dict))
|
||||
preview = (str(body or "")[:80]).replace("\n", " ")
|
||||
print(f"[strata] req {api}: msgs={len(messages)} tools={len(tools or [])} "
|
||||
f"max_tokens_raw={req.get('max_tokens')!r}/{req.get('max_completion_tokens')!r} "
|
||||
f"max_new={max_new} thinking={thinking} stream={bool(req.get('stream'))} "
|
||||
f"prompt_tokens={prompt_tokens} last={last.get('role')!r}:{preview!r}", flush=True)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------ OpenAI
|
||||
def openai_chunks(svc: Service, req: dict, ids, thinking, tools, max_new, cancel):
|
||||
cid, created = "chatcmpl-" + uuid.uuid4().hex[:24], int(time.time())
|
||||
@@ -312,20 +477,32 @@ def openai_chunks(svc: Service, req: dict, ids, thinking, tools, max_new, cancel
|
||||
|
||||
yield chunk({"role": "assistant", "content": ""})
|
||||
calls = 0
|
||||
streamed = {} # tool call id -> index, for calls sent piece by piece
|
||||
for kind, x in svc.run(ids, thinking, tools, max_new, req, cancel):
|
||||
if kind == "event":
|
||||
if kind == "ping":
|
||||
yield None
|
||||
elif kind == "event":
|
||||
ev: Event = x
|
||||
if ev.kind == "reasoning" and ev.text:
|
||||
yield chunk({"reasoning_content": ev.text})
|
||||
elif ev.kind == "content" and ev.text:
|
||||
yield chunk({"content": ev.text})
|
||||
elif ev.kind == "tool_start":
|
||||
streamed[ev.call.id] = calls
|
||||
calls += 1
|
||||
yield chunk({"tool_calls": [{"index": streamed[ev.call.id], "id": ev.call.id, "type": "function",
|
||||
"function": {"name": ev.call.name, "arguments": ""}}]})
|
||||
elif ev.kind == "tool_args":
|
||||
yield chunk({"tool_calls": [{"index": streamed[ev.call.id], "function": {"arguments": ev.text}}]})
|
||||
elif ev.kind == "tool_call" and ev.call.id in streamed:
|
||||
continue
|
||||
elif ev.kind == "tool_call":
|
||||
yield chunk({"tool_calls": [{"index": calls, "id": ev.call.id, "type": "function",
|
||||
"function": {"name": ev.call.name,
|
||||
"arguments": json.dumps(ev.call.arguments, ensure_ascii=False)}}]})
|
||||
calls += 1
|
||||
else:
|
||||
finish = "tool_calls" if calls and x["finish"] == "stop" else x["finish"]
|
||||
finish = "tool_calls" if calls and x["finish"] == "stop" else {"cancel": "stop"}.get(x["finish"], x["finish"])
|
||||
last = chunk({}, finish)
|
||||
last["usage"] = {"prompt_tokens": len(ids), "completion_tokens": x["completion_tokens"],
|
||||
"total_tokens": len(ids) + x["completion_tokens"]}
|
||||
@@ -333,13 +510,22 @@ def openai_chunks(svc: Service, req: dict, ids, thinking, tools, max_new, cancel
|
||||
|
||||
|
||||
def openai_collect(chunks) -> dict:
|
||||
content, reasoning, calls, last = [], [], [], None
|
||||
content, reasoning, by_index, last = [], [], {}, None
|
||||
for c in chunks:
|
||||
if c is None: # a heartbeat
|
||||
continue
|
||||
d = c["choices"][0]["delta"]
|
||||
content.append(d.get("content") or "")
|
||||
reasoning.append(d.get("reasoning_content") or "")
|
||||
calls += [{k: v for k, v in tc.items() if k != "index"} for tc in d.get("tool_calls") or []]
|
||||
for tc in d.get("tool_calls") or []: # streamed calls arrive in pieces: merge them by index
|
||||
cur = by_index.setdefault(tc.get("index", len(by_index)), {"id": None, "type": "function",
|
||||
"function": {"name": "", "arguments": ""}})
|
||||
cur["id"] = tc.get("id") or cur["id"]
|
||||
fn = tc.get("function") or {}
|
||||
cur["function"]["name"] += fn.get("name") or ""
|
||||
cur["function"]["arguments"] += fn.get("arguments") or ""
|
||||
last = c
|
||||
calls = [by_index[i] for i in sorted(by_index)]
|
||||
msg = {"role": "assistant", "content": "".join(content) or None}
|
||||
if "".join(reasoning):
|
||||
msg["reasoning_content"] = "".join(reasoning)
|
||||
@@ -361,12 +547,25 @@ def anthropic_events(svc: Service, req: dict, ids, thinking, tools, max_new, can
|
||||
def close():
|
||||
return ("content_block_stop", {"type": "content_block_stop", "index": index})
|
||||
|
||||
streamed = set()
|
||||
for kind, x in svc.run(ids, thinking, tools, max_new, req, cancel):
|
||||
if kind == "ping":
|
||||
yield None
|
||||
continue
|
||||
if kind == "event":
|
||||
ev: Event = x
|
||||
want = {"reasoning": "thinking", "content": "text", "tool_call": "tool_use"}[ev.kind]
|
||||
if ev.kind != "tool_call" and not ev.text:
|
||||
if ev.kind == "tool_args":
|
||||
yield "content_block_delta", {"type": "content_block_delta", "index": index,
|
||||
"delta": {"type": "input_json_delta", "partial_json": ev.text}}
|
||||
continue
|
||||
if ev.kind == "tool_call" and ev.call.id in streamed:
|
||||
continue
|
||||
want = {"reasoning": "thinking", "content": "text", "tool_call": "tool_use", "tool_start": "tool_use"}[ev.kind]
|
||||
if ev.kind not in ("tool_call", "tool_start") and not ev.text:
|
||||
continue
|
||||
if ev.kind == "tool_start":
|
||||
streamed.add(ev.call.id)
|
||||
used_tool = True
|
||||
if open_kind != want or want == "tool_use":
|
||||
if open_kind is not None:
|
||||
yield close()
|
||||
@@ -383,6 +582,8 @@ def anthropic_events(svc: Service, req: dict, ids, thinking, tools, max_new, can
|
||||
elif want == "text":
|
||||
yield "content_block_delta", {"type": "content_block_delta", "index": index,
|
||||
"delta": {"type": "text_delta", "text": ev.text}}
|
||||
elif ev.kind == "tool_start":
|
||||
pass # its input follows as tool_args pieces
|
||||
else:
|
||||
used_tool = True
|
||||
yield "content_block_delta", {"type": "content_block_delta", "index": index, "delta": {
|
||||
@@ -391,7 +592,7 @@ def anthropic_events(svc: Service, req: dict, ids, thinking, tools, max_new, can
|
||||
if open_kind is not None:
|
||||
yield close()
|
||||
stop = "tool_use" if used_tool and x["finish"] == "stop" else \
|
||||
{"stop": "end_turn", "length": "max_tokens"}[x["finish"]]
|
||||
{"stop": "end_turn", "length": "max_tokens", "cancel": "end_turn"}[x["finish"]]
|
||||
yield "message_delta", {"type": "message_delta", "delta": {"stop_reason": stop, "stop_sequence": None},
|
||||
"usage": {"output_tokens": x["completion_tokens"]}}
|
||||
yield "message_stop", {"type": "message_stop"}
|
||||
@@ -399,7 +600,10 @@ def anthropic_events(svc: Service, req: dict, ids, thinking, tools, max_new, can
|
||||
|
||||
def anthropic_collect(events) -> dict:
|
||||
msg, blocks = None, []
|
||||
for name, e in events:
|
||||
for item in events:
|
||||
if item is None: # a heartbeat
|
||||
continue
|
||||
name, e = item
|
||||
if name == "message_start":
|
||||
msg = e["message"]
|
||||
elif name == "content_block_start":
|
||||
@@ -410,8 +614,11 @@ def anthropic_collect(events) -> dict:
|
||||
b["text"] += d["text"]
|
||||
elif d["type"] == "thinking_delta":
|
||||
b["thinking"] += d["thinking"]
|
||||
else:
|
||||
b["input"] = json.loads(d["partial_json"])
|
||||
else: # input_json_delta pieces: parsed when complete
|
||||
b["_json"] = b.get("_json", "") + d["partial_json"]
|
||||
elif name == "content_block_stop" and blocks and "_json" in blocks[-1]:
|
||||
b = blocks[-1]
|
||||
b["input"] = json.loads(b.pop("_json") or "{}")
|
||||
elif name == "message_delta":
|
||||
msg["stop_reason"] = e["delta"]["stop_reason"]
|
||||
msg["usage"]["output_tokens"] = e["usage"]["output_tokens"]
|
||||
@@ -457,6 +664,17 @@ def make_handler(svc: Service):
|
||||
elif path == "/health":
|
||||
self._json(200, {"status": "ok", "max_context": svc.engine.max_context, "model": svc.model,
|
||||
"images": svc.vision is not None, "api_key": bool(svc.api_key)})
|
||||
elif path == "/status":
|
||||
with svc.status_lock:
|
||||
s = dict(svc.status)
|
||||
now = time.time()
|
||||
if s.get("busy"):
|
||||
s["elapsed_s"] = round(now - s["started"], 1)
|
||||
if s.get("first_token"):
|
||||
s["tokens_per_s"] = round(s["generated"] / max(1e-6, now - s["first_token"]), 1)
|
||||
for k in ("started", "first_token"):
|
||||
s.pop(k, None)
|
||||
self._json(200, s)
|
||||
elif path == "/v1/models":
|
||||
if self._authorized():
|
||||
self._json(200, {"object": "list", "data": [{"id": svc.model, "object": "model"}]})
|
||||
@@ -486,7 +704,10 @@ def make_handler(svc: Service):
|
||||
def _openai(self, req):
|
||||
messages, tools, kw = openai_to_messages(req)
|
||||
max_new = int(req.get("max_completion_tokens") or req.get("max_tokens") or 1024)
|
||||
if max_new <= 0: # a non-positive budget (some clients send -1) means "unset"
|
||||
max_new = 1024
|
||||
ids, thinking = svc.prepare(messages, tools, kw, max_new)
|
||||
_debug_req("openai", req, messages, tools, max_new, thinking, len(ids))
|
||||
cancel = threading.Event()
|
||||
chunks = openai_chunks(svc, req, ids, thinking, tools, max_new, cancel)
|
||||
if not req.get("stream"):
|
||||
@@ -494,28 +715,40 @@ def make_handler(svc: Service):
|
||||
self._sse()
|
||||
try:
|
||||
for c in chunks:
|
||||
self.wfile.write(b"data: " + json.dumps(c, ensure_ascii=False).encode() + b"\n\n")
|
||||
if c is None:
|
||||
self.wfile.write(b": keep-alive\n\n") # an SSE comment: clients ignore it
|
||||
else:
|
||||
self.wfile.write(b"data: " + json.dumps(c, ensure_ascii=False).encode() + b"\n\n")
|
||||
self.wfile.flush()
|
||||
self.wfile.write(b"data: [DONE]\n\n")
|
||||
except OSError:
|
||||
cancel.set() # client went away: stop at the next step
|
||||
cancel.set() # client went away: stop the engine
|
||||
chunks.close()
|
||||
|
||||
def _anthropic(self, req):
|
||||
messages, tools, kw = anthropic_to_messages(req)
|
||||
max_new = int(req.get("max_tokens") or 1024)
|
||||
if max_new <= 0: # a non-positive budget (some clients send -1) means "unset"
|
||||
max_new = 1024
|
||||
ids, thinking = svc.prepare(messages, tools, kw, max_new)
|
||||
_debug_req("anthropic", req, messages, tools, max_new, thinking, len(ids))
|
||||
cancel = threading.Event()
|
||||
events = anthropic_events(svc, req, ids, thinking, tools, max_new, cancel)
|
||||
if not req.get("stream"):
|
||||
return self._json(200, anthropic_collect(events))
|
||||
self._sse()
|
||||
try:
|
||||
for name, e in events:
|
||||
self.wfile.write(f"event: {name}\n".encode() + b"data: " +
|
||||
json.dumps(e, ensure_ascii=False).encode() + b"\n\n")
|
||||
for item in events:
|
||||
if item is None:
|
||||
self.wfile.write(b": keep-alive\n\n")
|
||||
else:
|
||||
name, e = item
|
||||
self.wfile.write(f"event: {name}\n".encode() + b"data: " +
|
||||
json.dumps(e, ensure_ascii=False).encode() + b"\n\n")
|
||||
self.wfile.flush()
|
||||
except OSError:
|
||||
cancel.set()
|
||||
events.close()
|
||||
|
||||
return Handler
|
||||
|
||||
@@ -546,7 +779,7 @@ def main() -> int:
|
||||
ap.add_argument("--api-key", default=os.environ.get("STRATA_API_KEY", ""),
|
||||
help="require this key on /v1/* (Authorization: Bearer ... or x-api-key); also $STRATA_API_KEY")
|
||||
a = ap.parse_args()
|
||||
cfg = json.loads(Path(a.config).read_text(encoding="utf-8")) if a.config else {}
|
||||
cfg = json.loads(Path(a.config).read_text(encoding="utf-8-sig")) if a.config else {} # Notepad adds a BOM
|
||||
try: # before the minutes of loading: is the port free?
|
||||
Server((a.host, a.port), BaseHTTPRequestHandler).server_close()
|
||||
except OSError:
|
||||
|
||||
@@ -54,8 +54,8 @@ PREBUILT_ASSET = "strata-windows-x64.zip" if WIN else "strata-linux-x64.zip"
|
||||
# the CUDA libraries the ready-made engine loads (the same CUDA 13.0 it is built with), from NVIDIA's pip packages
|
||||
CUDA_WHEELS = ["nvidia-cublas==13.0.2.14", "nvidia-cuda-runtime==13.0.96"]
|
||||
MIN_DRIVER = 580 # CUDA 13.0
|
||||
MIN_ENGINE = (0, 1, 1) # the ready-made engine that reads split models (Swift 1.5)
|
||||
PY_PACKAGES = ["numpy", "jinja2", "regex", "pyyaml", "tqdm", "requests", "cmake", "ninja"]
|
||||
MIN_ENGINE = (0, 1, 2) # split models (Swift 1.5), STOP, cache sized after the slots are written
|
||||
PY_PACKAGES = ["numpy", "jinja2", "regex", "pyyaml", "tqdm", "requests", "cmake", "ninja", "pillow"]
|
||||
|
||||
MODELS = {
|
||||
"Q2_0": {"about": "2-bit, the fastest", "download_gb": 66.4, "ram_gb": 48, "arena_gb": 34.0},
|
||||
@@ -82,9 +82,9 @@ FAMILIES = {
|
||||
"license": "Swift Open License 1.0: https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF"},
|
||||
}
|
||||
MMPROJ = "mmproj-Qwen3.8-Flash-Next-BF16.gguf"
|
||||
# the image encoder on the GPU: ~0.9 GB of weights + ~0.3 GB of work buffers at 1024 image tokens, kept free of
|
||||
# expert slots (the engine's default reserve is 700 MiB)
|
||||
VISION = {"gpu": {"max_tokens": 1024, "reserve_mib": 2100},
|
||||
# the image encoder on the GPU (~1.2 GB at 1024 image tokens) warms up before the engine starts, so the engine
|
||||
# sizes its expert slots around it and the default reserve (700 MiB) is enough; engines before 0.1.2 need more
|
||||
VISION = {"gpu": {"max_tokens": 1024, "reserve_mib": 700},
|
||||
"cpu": {"max_tokens": 300, "reserve_mib": 700}}
|
||||
EXE = "strata.exe" if WIN else "strata"
|
||||
VEXE = "strata-vision.exe" if WIN else "strata-vision"
|
||||
@@ -572,7 +572,7 @@ def installed_configs():
|
||||
|
||||
|
||||
def start(cfg_path: Path, port: int | None, open_browser=True) -> int:
|
||||
cfg = json.loads(cfg_path.read_text(encoding="utf-8"))
|
||||
cfg = json.loads(cfg_path.read_text(encoding="utf-8-sig"))
|
||||
missing = [p for p in [cfg["exe"], *[a for a in cfg["args"] if a.endswith(".gguf")]] if not Path(p).exists()]
|
||||
if missing:
|
||||
fail(f"{cfg_path.name} refers to missing files: {missing[0]}", "run it again with --setup to repair")
|
||||
@@ -631,7 +631,7 @@ def main() -> int:
|
||||
return start(have[0], None)
|
||||
say()
|
||||
for i, c in enumerate(have, 1):
|
||||
say(f" {i}) {json.loads(c.read_text(encoding='utf-8')).get('model_name', c.stem)}")
|
||||
say(f" {i}) {json.loads(c.read_text(encoding='utf-8-sig')).get('model_name', c.stem)}")
|
||||
say(f" {len(have) + 1}) install another model / change settings")
|
||||
pick = int(ask("Which one?", [str(i) for i in range(1, len(have) + 2)], "1", a.yes))
|
||||
if pick <= len(have):
|
||||
@@ -725,7 +725,7 @@ def main() -> int:
|
||||
|
||||
# ---- 3. python packages
|
||||
step(3, "Python packages")
|
||||
pip_install(PY_PACKAGES, "numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja")
|
||||
pip_install(PY_PACKAGES, "numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja, pillow")
|
||||
|
||||
# ---- 4. the engine
|
||||
step(4, "the Strata engine")
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <fstream>
|
||||
#include <thread>
|
||||
@@ -263,8 +264,14 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
|
||||
if (native && d.nact_multi.size() < (size_t) MAXT * kNativeActBytes) d.nact_multi.resize((size_t) MAXT * kNativeActBytes);
|
||||
if (d.job_of.size() != (size_t) d.n_expert) d.job_of.assign((size_t) d.n_expert, (int16_t) -1);
|
||||
if (d.jobs_multi.size() < (size_t) (n_tok * k)) d.jobs_multi.resize((size_t) (MAXT * k));
|
||||
static const bool ptrace = std::getenv("STRATA_POOL_TRACE") != nullptr;
|
||||
auto pt = [&](const char* what, long long a = -1) {
|
||||
if (ptrace) { std::fprintf(stderr, "pool trace: layer %lld %s %lld\n", (long long) d.layers, what, a); std::fflush(stderr); }
|
||||
};
|
||||
const auto c0 = std::chrono::steady_clock::now();
|
||||
pt("begin");
|
||||
d.src->begin_layer(d.layers, ids, n_tok * k);
|
||||
pt("begun");
|
||||
if (!d.usage.empty())
|
||||
for (int64_t i = 0; i < n_tok * k; ++i)
|
||||
if (ids[i] >= 0 && ids[i] < d.n_expert) d.usage[(size_t) d.layers * (size_t) d.n_expert + (size_t) ids[i]] += 1.0f;
|
||||
@@ -348,7 +355,9 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
|
||||
P.counts[1] = entries;
|
||||
P.counts[2] = fetches;
|
||||
std::atomic_thread_fence(std::memory_order_seq_cst);
|
||||
pt("publish", fetches);
|
||||
if (P.publish) P.publish(P.ctx);
|
||||
pt("fetch", fetches);
|
||||
if (P.fetch) P.fetch(P.ctx, dma_src, P.pcie_mode != 0 ? 0 : fetches, (size_t) bb); // the copy engine, beside the CPU's work
|
||||
} else {
|
||||
for (int64_t i = 0; i < n; ++i) {
|
||||
@@ -409,9 +418,11 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
|
||||
++d.multi_entries;
|
||||
}
|
||||
const auto c3 = std::chrono::steady_clock::now();
|
||||
pt("run", njobs);
|
||||
if (native) d.pool->run_split_multi_native(lay.fmt[(size_t) d.layers], d.jobs_multi.data(), njobs);
|
||||
else d.pool->run_split_multi(d.jobs_multi.data(), njobs);
|
||||
const auto c4 = std::chrono::steady_clock::now();
|
||||
pt("ran");
|
||||
auto ms = [](auto a, auto b) { return std::chrono::duration<double, std::milli>(b - a).count(); };
|
||||
d.ms_plan += ms(c0, c1);
|
||||
d.ms_actq += ms(c1, c2);
|
||||
|
||||
@@ -49,11 +49,12 @@ std::vector<int> physical_cores(bool skip_first) {
|
||||
#else
|
||||
cpu_set_t set;
|
||||
CPU_ZERO(&set);
|
||||
if (sched_getaffinity(0, sizeof set, &set) == 0)
|
||||
if (sched_getaffinity(0, sizeof set, &set) == 0) {
|
||||
for (int i = 0; i < CPU_SETSIZE; ++i)
|
||||
if (CPU_ISSET(i, &set)) cores.push_back(i);
|
||||
else
|
||||
} else {
|
||||
for (unsigned i = 0; i < std::thread::hardware_concurrency(); ++i) cores.push_back((int) i);
|
||||
}
|
||||
#endif
|
||||
if (skip_first && !cores.empty()) cores.erase(cores.begin());
|
||||
return cores;
|
||||
@@ -127,10 +128,23 @@ ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works) : host_works_(h
|
||||
ExpertPool::~ExpertPool() {
|
||||
stop_.store(true, std::memory_order_release);
|
||||
// Bump the epoch so a PARKED worker notices the stop flag rather than sleeping through it.
|
||||
epoch_.fetch_add(1, std::memory_order_release);
|
||||
publish();
|
||||
for (auto& t : threads_) t.join();
|
||||
}
|
||||
|
||||
void ExpertPool::publish() {
|
||||
// Both sides are seq_cst, and that is the whole lost-wakeup argument: a worker going to sleep does
|
||||
// `sleepers_++` and then reads `epoch_`, the host does `epoch_++` and then reads `sleepers_`. In one total
|
||||
// order at least one of them sees the other's write - the worker sees the new epoch and does not sleep, or
|
||||
// the host sees the sleeper and notifies under the mutex the worker holds until it is inside `wait`.
|
||||
// On x86 the fetch_add is a locked xadd either way, so this costs the token path nothing.
|
||||
epoch_.fetch_add(1, std::memory_order_seq_cst);
|
||||
if (sleepers_.load(std::memory_order_seq_cst) != 0) {
|
||||
std::lock_guard<std::mutex> lk(sleep_mu_);
|
||||
sleep_cv_.notify_all();
|
||||
}
|
||||
}
|
||||
|
||||
void ExpertPool::worker(int id) {
|
||||
uint32_t seen = 0;
|
||||
// ARRIVE at the park before the first wait, so `parked_ == n_` is true from construction. Counting only
|
||||
@@ -147,9 +161,22 @@ void ExpertPool::worker(int id) {
|
||||
// - a locked read-modify-write, five workers against one cache line - so the workers spent their wait
|
||||
// invalidating each other's caches and the very line the host writes to publish work. The counter was
|
||||
// diagnostic and nothing branched on it. See the note on the atomics in pool.hpp.
|
||||
//
|
||||
// After `kSpinBeforeSleep` with no work the worker sleeps instead (issue #4). The clock is read once
|
||||
// every 1024 pauses, so the spin itself is unchanged.
|
||||
const auto parked_at = std::chrono::steady_clock::now();
|
||||
uint32_t spins = 0;
|
||||
while (epoch_.load(std::memory_order_acquire) == seen) {
|
||||
if (stop_.load(std::memory_order_relaxed)) return;
|
||||
_mm_pause();
|
||||
if ((++spins & 1023u) != 0) continue;
|
||||
if (std::chrono::steady_clock::now() - parked_at < kSpinBeforeSleep) continue;
|
||||
std::unique_lock<std::mutex> lk(sleep_mu_);
|
||||
sleepers_.fetch_add(1, std::memory_order_seq_cst);
|
||||
sleep_cv_.wait(lk, [&] {
|
||||
return epoch_.load(std::memory_order_seq_cst) != seen || stop_.load(std::memory_order_relaxed);
|
||||
});
|
||||
sleepers_.fetch_sub(1, std::memory_order_relaxed);
|
||||
}
|
||||
if (stop_.load(std::memory_order_acquire)) return;
|
||||
seen = epoch_.load(std::memory_order_relaxed);
|
||||
@@ -245,7 +272,7 @@ void ExpertPool::run_phase(int mode, int n_tasks) {
|
||||
njobs_ = n_tasks;
|
||||
head_.store(0, std::memory_order_relaxed);
|
||||
done_.store(0, std::memory_order_relaxed);
|
||||
epoch_.fetch_add(1, std::memory_order_release);
|
||||
publish();
|
||||
if (host_works_) drain(-1, host_scratch_);
|
||||
while (done_.load(std::memory_order_acquire) != (uint32_t) n_tasks) _mm_pause();
|
||||
while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause();
|
||||
@@ -356,7 +383,7 @@ void ExpertPool::run(ExpertJob* jobs, int n) {
|
||||
mode_ = 0;
|
||||
head_.store(0, std::memory_order_relaxed);
|
||||
done_.store(0, std::memory_order_relaxed);
|
||||
epoch_.fetch_add(1, std::memory_order_release); // release: jobs_/njobs_ are visible before the bump
|
||||
publish(); // a release (seq_cst): jobs_/njobs_ are visible before the bump
|
||||
|
||||
// ---- **THE HOST DRAINS TOO (R2.2), INSTEAD OF SPINNING ON `done_`.**
|
||||
//
|
||||
|
||||
@@ -261,6 +261,8 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e
|
||||
int32_t prev[2] = {ss.ple_prev[0], ss.ple_prev[1]};
|
||||
|
||||
for (int64_t c0 = 0; c0 < n; c0 += m.T) {
|
||||
if (should_stop && should_stop()) { err = "cancelled"; return false; }
|
||||
if (std::getenv("STRATA_TRACE")) { std::fprintf(stderr, "strata trace: prompt chunk %lld of %lld\n", (long long) c0, (long long) n); std::fflush(stderr); }
|
||||
const int64_t T = std::min(m.T, n - c0), p0 = pos0 + c0;
|
||||
++stats_.chunks;
|
||||
// ---- embeddings, broadcast to the four streams
|
||||
|
||||
+534
-40
@@ -48,6 +48,10 @@
|
||||
#include <algorithm>
|
||||
#include <iostream>
|
||||
#include <thread>
|
||||
#include <atomic>
|
||||
#include <condition_variable>
|
||||
#include <deque>
|
||||
#include <mutex>
|
||||
#include <charconv>
|
||||
#include <cmath>
|
||||
#include <cstdio>
|
||||
@@ -231,6 +235,19 @@ struct Options {
|
||||
/// The vision path: keep a per-cell (t, h, w) rotary position table so --serve can take GENI requests.
|
||||
bool vision = false;
|
||||
int adapt_swaps = 96;
|
||||
/// --serve: how many conversation checkpoints to keep between requests (0 = every request reads its whole
|
||||
/// prompt again, the v0.1.2 behaviour). One is the GDN recurrence of the 36 layers, the QSA indexer tails and
|
||||
/// the PLE history (~118 MB of host RAM); the KV cache itself is positional and stays where it is.
|
||||
int prompt_cache = 6;
|
||||
/// --serve: also keep a checkpoint every N freshly read prompt tokens (0 = only at the last turn boundary)
|
||||
int64_t prompt_cache_every = 16384;
|
||||
/// --serve: the token that opens a chat turn (<|im_start|>). The last one in a prompt is where the chat's
|
||||
/// history ends and the new assistant turn begins, which is the checkpoint the next request can reuse.
|
||||
int64_t turn_token = 248045;
|
||||
/// --serve: a text part of the prompt of at most N tokens (a chat message, the assistant header, a short tool
|
||||
/// result) goes through the verify windows, S tokens at a time, instead of the batched prompt path (0 = always
|
||||
/// the batched path)
|
||||
int64_t short_read = 64;
|
||||
};
|
||||
|
||||
void usage() {
|
||||
@@ -288,6 +305,12 @@ void usage() {
|
||||
" --shared-late A/B: shared expert after the CPU pool (default: overlapped with it)\n"
|
||||
" --keep-canonical A/B: also load canonical copies of natively served tensors (more VRAM)\n"
|
||||
" --vision --serve takes images too (GENI requests; embeddings from strata-vision)\n"
|
||||
" --prompt-cache N --serve: keep N conversation checkpoints between requests (default 6, ~118 MB\n"
|
||||
" of RAM each; 0 = read every prompt from the start)\n"
|
||||
" --prompt-cache-every N --serve: also checkpoint every N fresh prompt tokens (default 16384, 0 = off)\n"
|
||||
" --turn-token ID --serve: the token that opens a chat turn (default 248045, <|im_start|>)\n"
|
||||
" --short-read N --serve: read at most N fresh text tokens through the decode windows instead\n"
|
||||
" of the batched prompt path (default 64, 0 = off)\n"
|
||||
" --no-token-graph A/B: two graphs per layer (the host launches each) instead of one per token\n"
|
||||
" --no-fused-gr A/B: the six-kernel hyper-connection read and a separate write (native)\n"
|
||||
" --prefill CHUNK batched prompt processing in chunks of CHUNK tokens (needs --native)\n"
|
||||
@@ -422,6 +445,15 @@ void drive_pool_multi(void* user, const float* x_f, const int32_t* ids, int64_t
|
||||
++t->calls;
|
||||
}
|
||||
|
||||
/// STRATA_TRACE=1: the VRAM left at a step of the startup (finds what fills the card after the cache is sized)
|
||||
void mem_mark(const char* where) {
|
||||
static const bool on = std::getenv("STRATA_TRACE") != nullptr;
|
||||
if (!on) return;
|
||||
size_t free_b = 0, total_b = 0;
|
||||
cudaMemGetInfo(&free_b, &total_b);
|
||||
std::fprintf(stderr, "strata trace: %lld MiB free after %s\n", (long long) (free_b >> 20), where);
|
||||
}
|
||||
|
||||
int argmax(const std::vector<float>& v) {
|
||||
int best = 0;
|
||||
for (size_t i = 1; i < v.size(); ++i)
|
||||
@@ -429,6 +461,84 @@ int argmax(const std::vector<float>& v) {
|
||||
return best;
|
||||
}
|
||||
|
||||
// ---- --serve's conversation cache. A chat or an agent sends the whole conversation again with every request, and
|
||||
// reading it again is what made a long session wait minutes for every turn. What a sequence leaves behind splits in
|
||||
// two, and only one half needs copying:
|
||||
// * POSITIONAL state - the KV cache of the 12 QSA layers and their pooled indexer keys, the draft layer's KV. A
|
||||
// cell is written once for its position and read only by later positions (the block scores take `dead` for the
|
||||
// block being filled, never its pooled row), so rewinding to a position just means writing from there again.
|
||||
// * RUNNING state - the 36 GDN recurrences and conv histories, each QSA layer's indexer tail (the unfinished
|
||||
// block's raw keys) and the PLE's normalized history. Each describes "everything so far" and cannot be
|
||||
// rewound, so a checkpoint is a copy of exactly these: ~118 MB, the same set the verifier snapshots to roll
|
||||
// back rejected drafts.
|
||||
// A checkpoint is only valid while the positional cells below it still hold ITS tokens, so the serve loop keeps
|
||||
// just the checkpoints that are prefixes of the tokens the session holds now.
|
||||
struct ImgKey {
|
||||
int64_t start = 0; ///< the image's first <|image_pad|> position
|
||||
uint64_t hash = 0; ///< its embeddings and grid: the pad tokens alone are the same for every picture
|
||||
bool operator==(const ImgKey& o) const { return start == o.start && hash == o.hash; }
|
||||
};
|
||||
|
||||
struct ConvCheckpoint {
|
||||
std::vector<int32_t> ids; ///< the tokens this state has consumed
|
||||
std::vector<ImgKey> imgs; ///< the images among them
|
||||
std::vector<uint8_t> gdn, ple, tails;
|
||||
};
|
||||
|
||||
uint64_t fnv1a(const void* data, size_t n, uint64_t h = 1469598103934665603ull) {
|
||||
const uint8_t* p = (const uint8_t*) data;
|
||||
for (size_t i = 0; i < n; ++i) { h ^= p[i]; h *= 1099511628211ull; }
|
||||
return h;
|
||||
}
|
||||
|
||||
struct ConvStateSizes {
|
||||
size_t gdn = 0, ple = 0, tail = 0;
|
||||
};
|
||||
|
||||
ConvStateSizes conv_state_sizes(const strata::core::ModelGeometry& g) {
|
||||
ConvStateSizes z;
|
||||
z.gdn = (size_t) g.n_gdn_layers() *
|
||||
((size_t) g.ssm_state_size * (size_t) g.ssm_v_heads * (size_t) g.ssm_state_size +
|
||||
(size_t) g.ssm_conv_channels * (size_t) (g.ssm_d_conv - 1)) * sizeof(float);
|
||||
z.ple = (size_t) strata::kernels::NG_HIST * (size_t) strata::kernels::NG_HC_DIM * sizeof(float);
|
||||
z.tail = (size_t) (strata::kernels::qsa_real_shapes().idx_block - 1) * (size_t) g.idx_key_dim * sizeof(float);
|
||||
return z;
|
||||
}
|
||||
|
||||
/// Copies the running state out. The caller has synchronized the device.
|
||||
bool checkpoint_save(ConvCheckpoint& c, const strata::core::SessionState& ss, const strata::core::ModelGeometry& g) {
|
||||
const ConvStateSizes z = conv_state_sizes(g);
|
||||
c.gdn.resize(z.gdn);
|
||||
c.ple.resize(ss.ple_hist != nullptr ? z.ple : 0);
|
||||
c.tails.resize(z.tail * (size_t) g.n_qsa_layers());
|
||||
if (cudaMemcpy(c.gdn.data(), ss.gdn_state, z.gdn, cudaMemcpyDeviceToHost) != cudaSuccess) return false;
|
||||
if (!c.ple.empty() && cudaMemcpy(c.ple.data(), ss.ple_hist, z.ple, cudaMemcpyDeviceToHost) != cudaSuccess)
|
||||
return false;
|
||||
for (int64_t i = 0; i < g.n_qsa_layers(); ++i)
|
||||
if (cudaMemcpy(c.tails.data() + (size_t) i * z.tail, ss.qsa_states[i].idx_tail, z.tail, cudaMemcpyDeviceToHost) !=
|
||||
cudaSuccess)
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/// Puts a checkpoint's running state back; the positional cells below it are the caller's to guarantee.
|
||||
bool checkpoint_restore(const ConvCheckpoint& c, strata::core::SessionState& ss, const strata::core::ModelGeometry& g) {
|
||||
const ConvStateSizes z = conv_state_sizes(g);
|
||||
if (c.gdn.size() != z.gdn || c.tails.size() != z.tail * (size_t) g.n_qsa_layers()) return false;
|
||||
if (cudaMemcpy(ss.gdn_state, c.gdn.data(), z.gdn, cudaMemcpyHostToDevice) != cudaSuccess) return false;
|
||||
if (!c.ple.empty() && cudaMemcpy(ss.ple_hist, c.ple.data(), z.ple, cudaMemcpyHostToDevice) != cudaSuccess)
|
||||
return false;
|
||||
for (int64_t i = 0; i < g.n_qsa_layers(); ++i)
|
||||
if (cudaMemcpy(ss.qsa_states[i].idx_tail, c.tails.data() + (size_t) i * z.tail, z.tail, cudaMemcpyHostToDevice) !=
|
||||
cudaSuccess)
|
||||
return false;
|
||||
// the PLE's token window is the last two tokens, OLDEST FIRST, -1 where there is none (as session_zero leaves it)
|
||||
const size_t L = c.ids.size();
|
||||
ss.ple_prev[0] = L >= 2 ? c.ids[L - 2] : -1;
|
||||
ss.ple_prev[1] = L >= 1 ? c.ids[L - 1] : -1;
|
||||
return cudaDeviceSynchronize() == cudaSuccess;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
int main(int argc, char** argv) {
|
||||
@@ -544,6 +654,10 @@ int main(int argc, char** argv) {
|
||||
else if (a == "--pcie-mode") o.pcie_mode = next("--pcie-mode");
|
||||
else if (a == "--serve") o.serve = true;
|
||||
else if (a == "--vision") o.vision = true;
|
||||
else if (a == "--prompt-cache") o.prompt_cache = std::max(0, std::atoi(next("--prompt-cache")));
|
||||
else if (a == "--prompt-cache-every") o.prompt_cache_every = std::max(0LL, std::atoll(next("--prompt-cache-every")));
|
||||
else if (a == "--turn-token") o.turn_token = std::atoll(next("--turn-token"));
|
||||
else if (a == "--short-read") o.short_read = std::max(0LL, std::atoll(next("--short-read")));
|
||||
else if (a == "--no-spec-split") o.spec_split = false;
|
||||
else if (a == "--eos-ids") {
|
||||
std::string e;
|
||||
@@ -1003,6 +1117,7 @@ int main(int argc, char** argv) {
|
||||
// ---- R4's slot storage. Allocated AFTER the weights and the session, so `cudaMemGetInfo` inside `open`
|
||||
// sees the memory this process actually has left rather than the card's idle figure - and refuses with both
|
||||
// numbers if the slots do not fit, instead of handing back a cache smaller than it was asked for.
|
||||
mem_mark("the weights, the session and the drafter");
|
||||
strata::core::ExpertCache xcache;
|
||||
std::vector<std::pair<int32_t, int32_t>> profile;
|
||||
if (!o.expert_profile.empty()) {
|
||||
@@ -1018,6 +1133,7 @@ int main(int argc, char** argv) {
|
||||
std::fprintf(stderr, "strata generate: profile %s: %zu ranked pairs, built for %lld slots\n",
|
||||
o.expert_profile.c_str(), profile.size(), (long long) pslots);
|
||||
}
|
||||
const bool auto_cache = o.expert_cache < 0;
|
||||
if (o.expert_cache < 0) {
|
||||
size_t free_b = 0, total_b = 0;
|
||||
cudaMemGetInfo(&free_b, &total_b);
|
||||
@@ -1053,15 +1169,51 @@ int main(int argc, char** argv) {
|
||||
o.expert_cache = (int) sized_slots.size();
|
||||
}
|
||||
if (o.expert_cache > 0) {
|
||||
const bool ok = sized_slots.empty()
|
||||
? xcache.open(o.expert_cache, g.n_layers, g.n_expert, (int64_t) strata::kernels::cpu::expert_layout().max_blob, err)
|
||||
: xcache.open_sized(sized_slots, g.n_layers, g.n_expert, err);
|
||||
if (!ok) {
|
||||
std::fprintf(stderr, "strata generate: %s\n", err.c_str());
|
||||
return 1;
|
||||
// With `--expert-cache auto` the reserve must still be free once the slots are WRITTEN: under WDDM an
|
||||
// allocation is not resident until it is touched, and the free figure read before it can be ~1 GB too
|
||||
// high. A cache sized from it filled the card to 0 MiB, the driver then paged, and a request that needed a
|
||||
// page back while the verify graph spun on a host flag never finished. So the slots are zeroed and the
|
||||
// free figure read again; while it is short of the reserve the cache is reopened smaller.
|
||||
for (int attempt = 0;; ++attempt) {
|
||||
const bool ok = sized_slots.empty()
|
||||
? xcache.open(o.expert_cache, g.n_layers, g.n_expert, (int64_t) strata::kernels::cpu::expert_layout().max_blob, err)
|
||||
: xcache.open_sized(sized_slots, g.n_layers, g.n_expert, err);
|
||||
if (!ok) {
|
||||
std::fprintf(stderr, "strata generate: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
if (!auto_cache || attempt >= 6) break;
|
||||
cudaMemset(xcache.device_slot(0), 0, (size_t) xcache.bytes());
|
||||
cudaDeviceSynchronize();
|
||||
size_t free_b = 0, total_b = 0;
|
||||
cudaMemGetInfo(&free_b, &total_b);
|
||||
const int64_t want = (int64_t) o.vram_reserve_mib << 20;
|
||||
if ((int64_t) free_b >= want - (64ll << 20)) break;
|
||||
// short by (want - free); a figure of 0 only says "at least", so then give back a quarter as well
|
||||
int64_t give = want - (int64_t) free_b + (64ll << 20);
|
||||
if (free_b < ((size_t) 16 << 20)) give = std::max<int64_t>(give, xcache.bytes() / 4);
|
||||
const int64_t keep_bytes = xcache.bytes() - give;
|
||||
std::fprintf(stderr, "strata generate: only %lld MiB free once the slots are written (reserve %d MiB); "
|
||||
"shrinking the expert cache\n", (long long) (free_b >> 20), o.vram_reserve_mib);
|
||||
xcache.close();
|
||||
if (keep_bytes <= 0) { o.expert_cache = 0; sized_slots.clear(); break; }
|
||||
if (!sized_slots.empty()) {
|
||||
int64_t used = 0;
|
||||
size_t keep = 0;
|
||||
while (keep < sized_slots.size() && used + (sized_slots[keep] + 255) / 256 * 256 <= keep_bytes)
|
||||
used += (sized_slots[keep++] + 255) / 256 * 256;
|
||||
sized_slots.resize(keep);
|
||||
o.expert_cache = (int) keep;
|
||||
} else {
|
||||
o.expert_cache = (int) (keep_bytes / (int64_t) strata::kernels::cpu::expert_layout().max_blob);
|
||||
}
|
||||
if (o.expert_cache <= 0) { o.expert_cache = 0; sized_slots.clear(); break; }
|
||||
}
|
||||
}
|
||||
if (o.expert_cache > 0) {
|
||||
std::fprintf(stderr, "strata generate: expert cache %lld slots, %.2f GiB of VRAM; policy is\n",
|
||||
(long long) xcache.slots(), xcache.gib());
|
||||
mem_mark("opening the expert cache");
|
||||
xcache.set_per_layer_admission(o.expert_cache_per_layer);
|
||||
// **ROUND 328: THE HIT PATH IS PROVABLY WRONG, AND THIS SAYS SO OUT LOUD RATHER THAN LETTING IT
|
||||
// CORRUPT A RUN QUIETLY.** With the cache on, the generated tokens DIVERGE from the cache-off run:
|
||||
@@ -1109,12 +1261,13 @@ int main(int argc, char** argv) {
|
||||
// **AND ONE SLOT IS READ BACK AND COMPARED.** A residency table that is right about indices and wrong
|
||||
// about bytes produces a plausible token, which is this project's most expensive failure mode; the
|
||||
// cache's own `verify_slot` is the check and it costs one 1.38 MB D2H at startup.
|
||||
if (!xcache.verify_slot(xcache.slot_of(profile[0].first, profile[0].second),
|
||||
if (prefilled > 0 && !xcache.verify_slot(xcache.slot_of(profile[0].first, profile[0].second),
|
||||
srcp->blob(profile[0].first, profile[0].second), err,
|
||||
(int64_t) strata::kernels::cpu::expert_layout().blob_bytes(profile[0].first))) {
|
||||
std::fprintf(stderr, "strata generate: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
mem_mark("the profile fill");
|
||||
std::fprintf(stderr, "strata generate: pre-filled %lld of %lld slots from the profile; slot 0 verified\n",
|
||||
(long long) prefilled, (long long) want);
|
||||
}
|
||||
@@ -1172,6 +1325,7 @@ int main(int argc, char** argv) {
|
||||
drive.d.hit_done = (void*) hit_done;
|
||||
drive.d.hit_poke = !o.no_hit_poke;
|
||||
drive.d.h_dst.resize((size_t) K);
|
||||
mem_mark("the R4 hit path");
|
||||
std::fprintf(stderr, "strata generate: R4 hit path ON - resident experts are computed on the GPU\n");
|
||||
}
|
||||
// ---- P0.S8: the routing trace. Only meaningful with the pool running, because the ids arrive through
|
||||
@@ -1289,6 +1443,7 @@ int main(int argc, char** argv) {
|
||||
return 2;
|
||||
}
|
||||
|
||||
mem_mark("the expert cache and the graphs");
|
||||
std::fprintf(stderr, "strata generate: session is up; locating the head\n");
|
||||
const strata::core::WeightRef* wo = wt.find("output.weight");
|
||||
if (wo == nullptr) { std::fprintf(stderr, "strata generate: output.weight is missing\n"); return 1; }
|
||||
@@ -1758,11 +1913,15 @@ int main(int argc, char** argv) {
|
||||
//
|
||||
// and each generated token is written to stdout as `T <id>` as soon as its verify window is done, followed by
|
||||
//
|
||||
// DONE <generated> <prompt_tokens> <prompt_ms> <decode_ms> <stop|length>
|
||||
// DONE <generated> <prompt_tokens> <prompt_ms> <decode_ms> <stop|length|cancel> <drafts accepted>
|
||||
// <drafts offered> <prompt tokens reused>
|
||||
//
|
||||
// (`ERR <message>` instead when a request cannot run; `QUIT` ends the process). Every request starts from an
|
||||
// empty sequence (`session_zero`): the prompt goes through the batched prompt path and its last token through
|
||||
// the first verify window - the path all three model files share. Decoding is greedy.
|
||||
// Before that, `RESUME <n>` (n prompt tokens are not read again), `PP <position> <prompt_tokens> <ms> <tok/s>`
|
||||
// after every prompt chunk, and `REUSED <n>` once the prompt is read. (`ERR <message>` instead when a request
|
||||
// cannot run; `STOP` ends the running request at its next step; `QUIT` ends the process.) A request continues
|
||||
// from the live session or the longest conversation checkpoint its prompt starts with (see ConvCheckpoint),
|
||||
// otherwise from an empty sequence (`session_zero`); the rest of the prompt goes through the batched prompt path
|
||||
// and its last token through the first verify window - the path all three model files share. Decoding is greedy.
|
||||
if (o.serve) {
|
||||
if (o.spec < 2 || o.mtp.empty() || o.prefill_chunk <= 0 || thits.d_res == nullptr || host_res.empty()) {
|
||||
std::fprintf(stderr, "strata serve: needs --spec T, --mtp DIR, --prefill CHUNK, --expert-profile P and "
|
||||
@@ -1773,8 +1932,11 @@ int main(int argc, char** argv) {
|
||||
void* borrow = nullptr;
|
||||
uint64_t borrow_bytes = 0;
|
||||
int32_t lend_first = -1;
|
||||
if (!o.no_prefill_borrow && d_res != nullptr) {
|
||||
const uint64_t need = strata::prefill::Prefill::bytes_needed(g, ss, o.prefill_chunk);
|
||||
// a cache too small to lend the prompt path its buffers would make it allocate them on top - on a card the
|
||||
// cache already filled to its reserve, that is the over-subscription the auto sizing avoids - so the
|
||||
// prompt chunk is halved until its buffers fit in the lendable slots (a smaller chunk only reads slower)
|
||||
for (int64_t chunk = o.prefill_chunk; !o.no_prefill_borrow && d_res != nullptr && chunk >= 256; chunk /= 2) {
|
||||
const uint64_t need = strata::prefill::Prefill::bytes_needed(g, ss, chunk);
|
||||
const int64_t blob = (int64_t) strata::kernels::cpu::expert_layout().max_blob;
|
||||
int64_t k = (int64_t) ((need + (uint64_t) blob - 1) / (uint64_t) blob);
|
||||
if (xcache.slot_offsets() != nullptr) {
|
||||
@@ -1783,16 +1945,27 @@ int main(int argc, char** argv) {
|
||||
(uint64_t) (xcache.bytes() - (int64_t) xcache.slot_offsets()[xcache.slots() - k]) < need) ++k;
|
||||
}
|
||||
if (k + 128 <= xcache.slots()) {
|
||||
if (chunk != o.prefill_chunk)
|
||||
std::fprintf(stderr, "strata serve: prompt chunk %lld -> %lld tokens so its buffers fit in the "
|
||||
"expert cache\n", (long long) o.prefill_chunk, (long long) chunk);
|
||||
o.prefill_chunk = chunk;
|
||||
lend_first = (int32_t) (xcache.slots() - k);
|
||||
borrow = xcache.device_slot(lend_first);
|
||||
borrow_bytes = xcache.slot_offsets() ? (uint64_t) (xcache.bytes() - (int64_t) xcache.slot_offsets()[lend_first])
|
||||
: (uint64_t) k * (uint64_t) blob;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (borrow != nullptr)
|
||||
std::fprintf(stderr, "strata serve: the prompt path borrows %lld cache slots (%.2f GiB)\n",
|
||||
(long long) (xcache.slots() - lend_first), (double) borrow_bytes / 1073741824.0);
|
||||
else
|
||||
std::fprintf(stderr, "strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)\n");
|
||||
if (!sp.init(wt, g, ss, srcp, &xcache, host_res.data(), o.prefill_chunk, main_cs, err, borrow, borrow_bytes)) {
|
||||
std::fprintf(stderr, "strata serve: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
mem_mark("the head and the prompt path");
|
||||
strata::core::Verifier ver;
|
||||
strata::core::VerifyHits vh;
|
||||
vh.d_res = thits.d_res;
|
||||
@@ -1803,14 +1976,52 @@ int main(int argc, char** argv) {
|
||||
std::fprintf(stderr, "strata serve: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
mem_mark("the verifier and the drafter's binding");
|
||||
ver.set_split(o.spec_split);
|
||||
ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2
|
||||
: native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0
|
||||
std::vector<int64_t> cur;
|
||||
// ---- the conversation cache (see ConvCheckpoint). `live` is what the session holds right now: the tokens
|
||||
// it has consumed, so a request that starts with exactly them continues without any copy. `checks` are the
|
||||
// saved points; every one of them is a prefix of `live` (the loop drops the rest).
|
||||
std::vector<int32_t> live;
|
||||
std::vector<ImgKey> live_imgs, req_imgs;
|
||||
bool live_ok = false;
|
||||
std::vector<ConvCheckpoint> checks;
|
||||
int64_t pp_total = 0, pp_from = 0, pp_next_check = 0;
|
||||
Clock::time_point pp_t0 = Clock::now();
|
||||
auto imgs_below = [&](const std::vector<ImgKey>& all, int64_t L) {
|
||||
std::vector<ImgKey> v;
|
||||
for (const ImgKey& k : all) if (k.start < L) v.push_back(k);
|
||||
return v;
|
||||
};
|
||||
// a checkpoint of the state after `cur[0, L)`; false only when the copy itself failed
|
||||
auto checkpoint_at = [&](int64_t L) -> bool {
|
||||
if (o.prompt_cache <= 0 || L < 1) return true;
|
||||
for (const ConvCheckpoint& c : checks) if ((int64_t) c.ids.size() == L) return true;
|
||||
ConvCheckpoint c;
|
||||
c.ids.assign(cur.begin(), cur.begin() + L);
|
||||
c.imgs = imgs_below(req_imgs, L);
|
||||
if (cudaDeviceSynchronize() != cudaSuccess || !checkpoint_save(c, ss, g)) return false;
|
||||
checks.push_back(std::move(c));
|
||||
while ((int) checks.size() > o.prompt_cache) checks.erase(checks.begin()); // the oldest goes first
|
||||
return true;
|
||||
};
|
||||
sp.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool {
|
||||
std::vector<int32_t> nxt((size_t) T);
|
||||
for (int64_t t = 0; t < T; ++t) nxt[(size_t) t] = (int32_t) cur[(size_t) (p0 + t + 1)];
|
||||
return mtp.prefill(R_rows, nxt.data(), T, p0, e);
|
||||
if (!mtp.prefill(R_rows, nxt.data(), T, p0, e)) return false;
|
||||
// progress for the server window: PP <position reached> <prompt tokens> <ms> <fresh tokens/s>
|
||||
const int64_t done = p0 + T;
|
||||
const double ms = std::chrono::duration<double, std::milli>(Clock::now() - pp_t0).count();
|
||||
std::printf("PP %lld %lld %.0f %.1f\n", (long long) done, (long long) pp_total, ms,
|
||||
ms > 0.0 ? 1000.0 * (double) (done - pp_from) / ms : 0.0);
|
||||
std::fflush(stdout);
|
||||
if (o.prompt_cache_every > 0 && done >= pp_next_check) {
|
||||
if (!checkpoint_at(done)) { e = "saving a conversation checkpoint failed"; return false; }
|
||||
pp_next_check = done + o.prompt_cache_every;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
drive.d.plan = ver.plan_sink();
|
||||
drive.d.pcie_num = std::max(0, std::min(256, (int) (o.pcie_frac * 256.0 + 0.5)));
|
||||
@@ -1875,7 +2086,52 @@ int main(int argc, char** argv) {
|
||||
for (float& v : drive.d.usage) v *= 0.7f;
|
||||
return true;
|
||||
};
|
||||
std::printf("READY %lld\n", (long long) o.max_context);
|
||||
// stdin is read on its own thread, so a STOP line reaches a request that is still running (the client went
|
||||
// away, or pressed Esc): the flag is checked between prompt chunks and between verify windows.
|
||||
std::atomic<bool> stop_req{false};
|
||||
std::mutex in_mu;
|
||||
std::condition_variable in_cv;
|
||||
std::deque<std::string> in_lines;
|
||||
bool in_eof = false;
|
||||
std::thread([&] {
|
||||
std::string l;
|
||||
while (std::getline(std::cin, l)) {
|
||||
if (!l.empty() && l.back() == '\r') l.pop_back();
|
||||
if (l == "STOP") { stop_req.store(true); continue; }
|
||||
std::lock_guard<std::mutex> lk(in_mu);
|
||||
in_lines.push_back(l);
|
||||
in_cv.notify_one();
|
||||
}
|
||||
std::lock_guard<std::mutex> lk(in_mu);
|
||||
in_eof = true;
|
||||
in_cv.notify_one();
|
||||
}).detach();
|
||||
auto next_line = [&](std::string& out) -> bool {
|
||||
std::unique_lock<std::mutex> lk(in_mu);
|
||||
in_cv.wait(lk, [&] { return !in_lines.empty() || in_eof; });
|
||||
if (in_lines.empty()) return false;
|
||||
out = std::move(in_lines.front());
|
||||
in_lines.pop_front();
|
||||
return true;
|
||||
};
|
||||
sp.should_stop = [&] { return stop_req.load(); };
|
||||
// STRATA_TRACE=1: one stderr line per step of a request (the log shows where a request stops)
|
||||
const bool trace = std::getenv("STRATA_TRACE") != nullptr;
|
||||
auto tr = [&](const char* what, long long a = -1, long long b = -1) {
|
||||
if (!trace) return;
|
||||
std::fprintf(stderr, "strata trace: %s %lld %lld\n", what, a, b);
|
||||
std::fflush(stderr);
|
||||
};
|
||||
{
|
||||
// what is left once everything is allocated: under WDDM a GPU filled to the brim does not fail, it pages -
|
||||
// and a page-in while the verify graph spins on a host flag stalls the request for good
|
||||
size_t free_b = 0, total_b = 0;
|
||||
cudaMemGetInfo(&free_b, &total_b);
|
||||
std::fprintf(stderr, "strata serve: %lld MiB of VRAM free with everything loaded%s\n",
|
||||
(long long) (free_b >> 20), free_b < ((size_t) 128 << 20)
|
||||
? " - LOW: requests may stall; lower --max-context or raise --vram-reserve-mib" : "");
|
||||
}
|
||||
std::printf("READY %lld stop\n", (long long) o.max_context); // "stop": this engine honours STOP
|
||||
std::fflush(stdout);
|
||||
std::string line;
|
||||
int64_t rounds = 0;
|
||||
@@ -1888,9 +2144,9 @@ int main(int argc, char** argv) {
|
||||
bool mrope_identity = true;
|
||||
std::vector<float> img_rows;
|
||||
std::vector<const float*> row_ptr;
|
||||
while (std::getline(std::cin, line)) {
|
||||
if (!line.empty() && line.back() == '\r') line.pop_back();
|
||||
while (next_line(line)) {
|
||||
if (line == "QUIT") break;
|
||||
stop_req.store(false); // a STOP that arrived between requests is stale
|
||||
const bool geni = line.rfind("GENI ", 0) == 0;
|
||||
if (!geni && line.rfind("GEN ", 0) != 0) {
|
||||
std::printf("ERR expected: GEN <max_new> <id,id,...> or GENI <max_new> <file> <id,id,...>\n");
|
||||
@@ -1911,6 +2167,7 @@ int main(int argc, char** argv) {
|
||||
continue;
|
||||
}
|
||||
const int64_t n = (int64_t) ids.size();
|
||||
req_imgs.clear();
|
||||
if (geni && !o.vision) { std::printf("ERR this engine was started without --vision\n"); continue; }
|
||||
if (geni || !mrope_identity) {
|
||||
// positions for every cell this request can reach; the identity again for a text request
|
||||
@@ -1952,6 +2209,12 @@ int main(int argc, char** argv) {
|
||||
if (ids[(size_t) i] != kImagePad) { put(i, p, p, p); ++p; ++i; continue; }
|
||||
if (k >= imgs.size()) { ve = "the prompt has more images than the embeddings file"; break; }
|
||||
const Img& im = imgs[k++];
|
||||
{ // what the conversation cache compares: a picture is its grid and its embeddings
|
||||
const int64_t grid[3] = {im.n, im.nx, im.ny};
|
||||
uint64_t h = fnv1a(grid, sizeof grid);
|
||||
h = fnv1a(img_rows.data() + im.off, (size_t) im.n * (size_t) g.n_embd * sizeof(float), h);
|
||||
req_imgs.push_back({i, h});
|
||||
}
|
||||
for (int64_t j = 0; j < im.n && ve.empty(); ++j)
|
||||
if (i + j >= n || ids[(size_t) (i + j)] != kImagePad)
|
||||
ve = "image " + std::to_string(k) + " has " + std::to_string((long long) im.n) +
|
||||
@@ -1968,7 +2231,9 @@ int main(int argc, char** argv) {
|
||||
if (ve.empty() && n > 0 && ids[(size_t) (n - 1)] == kImagePad) ve = "the prompt cannot end in an image";
|
||||
for (int64_t c = n; ve.empty() && c < cells; ++c) put(c, p + (c - n), p + (c - n), p + (c - n));
|
||||
}
|
||||
tr("positions built", (long long) img_rows.size());
|
||||
cudaDeviceSynchronize();
|
||||
tr("device idle");
|
||||
if (ve.empty() && cudaMemcpy(d_mrope, mrope_host.data(), mrope_host.size() * sizeof(int32_t),
|
||||
cudaMemcpyHostToDevice) != cudaSuccess)
|
||||
ve = "the image position upload failed";
|
||||
@@ -1994,36 +2259,183 @@ int main(int argc, char** argv) {
|
||||
if (bad) { std::printf("ERR a token id is outside the vocabulary\n"); continue; }
|
||||
cur = ids;
|
||||
const Clock::time_point r0 = Clock::now();
|
||||
strata::core::session_zero(ss, g, nullptr, main_cs);
|
||||
cudaStreamSynchronize(main_stream);
|
||||
// ---- where this request starts reading: the live session, or a checkpoint, whose tokens AND pictures are
|
||||
// exactly the start of this prompt - at most n - 1 of them, the last token is always the first window
|
||||
auto starts_with = [&](const std::vector<int32_t>& pre, const std::vector<ImgKey>& pre_imgs) -> bool {
|
||||
const int64_t L = (int64_t) pre.size();
|
||||
if (L < 1 || L > n - 1) return false;
|
||||
for (int64_t i = 0; i < L; ++i)
|
||||
if ((int32_t) ids[(size_t) i] != pre[(size_t) i]) return false;
|
||||
return imgs_below(req_imgs, L) == pre_imgs;
|
||||
};
|
||||
int64_t resume = 0;
|
||||
bool from_live = false;
|
||||
if (o.prompt_cache > 0) {
|
||||
if (live_ok && starts_with(live, live_imgs)) { resume = (int64_t) live.size(); from_live = true; }
|
||||
for (const ConvCheckpoint& c : checks)
|
||||
if ((int64_t) c.ids.size() > resume && starts_with(c.ids, c.imgs)) {
|
||||
resume = (int64_t) c.ids.size();
|
||||
from_live = false;
|
||||
}
|
||||
}
|
||||
// this request rewrites every cell from `resume` on, so a checkpoint past it (or not on this prompt's
|
||||
// path) no longer has its cells; the ones kept are prefixes of both the old tokens and the new
|
||||
checks.erase(std::remove_if(checks.begin(), checks.end(), [&](const ConvCheckpoint& c) {
|
||||
return (int64_t) c.ids.size() > resume || !starts_with(c.ids, c.imgs);
|
||||
}), checks.end());
|
||||
live_ok = false; // until this request has finished, the session is in between
|
||||
int64_t reread_to = -1; // STRATA_CKPT_REREAD only: read [0, reread_to) again instead of restoring
|
||||
if (resume == 0) {
|
||||
strata::core::session_zero(ss, g, nullptr, main_cs);
|
||||
cudaStreamSynchronize(main_stream);
|
||||
checks.clear();
|
||||
} else if (!from_live) {
|
||||
const ConvCheckpoint* c = nullptr;
|
||||
for (const ConvCheckpoint& k : checks) if ((int64_t) k.ids.size() == resume) c = &k;
|
||||
static const bool reread = std::getenv("STRATA_CKPT_REREAD") != nullptr;
|
||||
if (reread && c != nullptr) {
|
||||
// THE CHECK OF THE CHECKPOINT: instead of restoring it, read its tokens again from position 0 in
|
||||
// one run (below, with the prompt path's slots lent like any read) - the same chunks the request
|
||||
// that saved it read them in, when that request started at 0. With the VRAM expert set fixed
|
||||
// (--adapt-swaps 0) the answer must match the restored one token for token; anything the
|
||||
// checkpoint missed shows up as a difference.
|
||||
strata::core::session_zero(ss, g, nullptr, main_cs);
|
||||
cudaStreamSynchronize(main_stream);
|
||||
reread_to = resume;
|
||||
std::fprintf(stderr, "strata serve: STRATA_CKPT_REREAD: reading %lld tokens again instead of "
|
||||
"restoring\n", (long long) resume);
|
||||
} else if (c == nullptr || !checkpoint_restore(*c, ss, g)) {
|
||||
std::printf("ERR restoring a conversation checkpoint failed\n");
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
tr("request", n, geni ? 1 : 0);
|
||||
mtp.set_prompt_len(n);
|
||||
const int64_t read_from = reread_to > 0 ? 0 : resume;
|
||||
pp_total = n;
|
||||
pp_from = read_from;
|
||||
pp_t0 = r0;
|
||||
pp_next_check = reread_to > 0 ? INT64_MAX : resume + o.prompt_cache_every;
|
||||
std::printf("RESUME %lld\n", (long long) resume); // before reading: this many prompt tokens are reused
|
||||
std::fflush(stdout);
|
||||
// A SHORT PART OF THE PROMPT - the new message of a chat that continues from a checkpoint, the assistant
|
||||
// header - goes through the verify windows, S tokens at a time, as decode reads them. The batched path
|
||||
// costs ~300 ms per run however few tokens it has (it streams every expert the chunk routes to that is
|
||||
// not in VRAM over PCIe), and it borrows slots it must refill after (~180 ms); a window costs ~16 ms a
|
||||
// token, with the misses on the CPU. Each part below is decided on its own, so a long first message is
|
||||
// read batched and its header still goes through the windows. Picture rows need the batched path.
|
||||
// STRATA_CKPT_REREAD compares a restored checkpoint with a batched re-read, so it keeps every read batched.
|
||||
static const bool no_short = std::getenv("STRATA_CKPT_REREAD") != nullptr;
|
||||
auto windows_ok = [&](int64_t a, int64_t b) -> bool {
|
||||
if (no_short || b - a > o.short_read) return false;
|
||||
if (sp.embd_rows != nullptr)
|
||||
for (int64_t i = a; i < b; ++i)
|
||||
if (sp.embd_rows[i] != nullptr) return false;
|
||||
return true;
|
||||
};
|
||||
// tokens [a, b) through the windows: commit all of them, then give the draft layer their residuals
|
||||
auto read_windows = [&](int64_t a, int64_t b, std::string& e) -> bool {
|
||||
std::vector<int32_t> win((size_t) S), outw((size_t) S), nxt((size_t) S);
|
||||
for (int64_t q = a; q < b;) {
|
||||
if (stop_req.load()) { e = "cancelled"; return false; }
|
||||
const int T = (int) std::min<int64_t>(S, b - q);
|
||||
for (int t = 0; t < T; ++t) {
|
||||
win[(size_t) t] = (int32_t) cur[(size_t) (q + t)];
|
||||
nxt[(size_t) t] = (int32_t) cur[(size_t) (q + t + 1)];
|
||||
}
|
||||
drive.d.layers = 0;
|
||||
drive.d.experts = 0;
|
||||
drive.d.failed = false;
|
||||
if (!ver.run(T, win.data(), q, &drive_pool_multi, &drive, outw.data(), e) || drive.d.failed) {
|
||||
if (drive.d.failed && drive.d.fail) e = drive.d.fail;
|
||||
return false;
|
||||
}
|
||||
if (!ver.commit(T, e) || !mtp.prefill(ver.final_R_all(), nxt.data(), T, q, e)) return false;
|
||||
q += T;
|
||||
}
|
||||
const double ms = std::chrono::duration<double, std::milli>(Clock::now() - pp_t0).count();
|
||||
std::printf("PP %lld %lld %.0f %.1f\n", (long long) b, (long long) pp_total, ms,
|
||||
ms > 0.0 ? 1000.0 * (double) (b - pp_from) / ms : 0.0);
|
||||
std::fflush(stdout);
|
||||
return true;
|
||||
};
|
||||
// the batched path's slots are lent just before its first run and given back (refilled) before a window
|
||||
// reads - so the windows always see the whole expert cache - or once the prompt is read
|
||||
std::vector<std::pair<int32_t, int32_t>> lent_now;
|
||||
apply_pending(true);
|
||||
if (lend_first >= 0) {
|
||||
auto lend = [&] {
|
||||
if (lend_first < 0 || !lent_now.empty()) return;
|
||||
for (size_t i = 0; i < host_res.size(); ++i)
|
||||
if (host_res[i] >= lend_first) {
|
||||
lent_now.emplace_back((int32_t) i, host_res[i]);
|
||||
host_res[i] = strata::core::kNotResident;
|
||||
}
|
||||
cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice);
|
||||
}
|
||||
if (n > 1 && !sp.run(ids.data(), n - 1, 0, err)) {
|
||||
std::fprintf(stderr, "strata serve: %s\n", err.c_str());
|
||||
std::printf("ERR %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
for (const auto& [i, slot] : lent_now) {
|
||||
const uint8_t* b = srcp->blob(i / g.n_expert, i % g.n_expert);
|
||||
if (b == nullptr || !xcache.fill_slot_blocking(slot, b, err,
|
||||
(int64_t) strata::kernels::cpu::expert_layout().blob_bytes(i / g.n_expert))) {
|
||||
};
|
||||
auto refill = [&](std::string& e) -> bool {
|
||||
if (lent_now.empty()) return true;
|
||||
tr("refill start", (long long) lent_now.size());
|
||||
for (const auto& [i, slot] : lent_now) {
|
||||
const uint8_t* b = srcp->blob(i / g.n_expert, i % g.n_expert);
|
||||
if (b == nullptr || !xcache.fill_slot_blocking(slot, b, e,
|
||||
(int64_t) strata::kernels::cpu::expert_layout().blob_bytes(i / g.n_expert)))
|
||||
return false;
|
||||
host_res[(size_t) i] = slot;
|
||||
}
|
||||
cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice);
|
||||
lent_now.clear();
|
||||
return true;
|
||||
};
|
||||
apply_pending(true);
|
||||
bool cancelled = false;
|
||||
tr("prompt start", n - 1);
|
||||
// The prompt is read in two parts when it has a turn boundary past `resume`: up to the last <|im_start|>
|
||||
// (the conversation so far), a checkpoint there, then the new turn's header. The next request of the same
|
||||
// chat renders the same history - but not always the same header or the thinking of this reply - so that
|
||||
// checkpoint is the one it reuses.
|
||||
int64_t turn_at = -1;
|
||||
if (o.prompt_cache > 0 && o.turn_token >= 0)
|
||||
for (int64_t i = n - 1; i > resume; --i)
|
||||
if (ids[(size_t) i] == o.turn_token) { turn_at = i; break; }
|
||||
int64_t at = read_from;
|
||||
for (const int64_t to : {reread_to, turn_at, n - 1}) {
|
||||
if (to <= at) continue;
|
||||
const bool win = windows_ok(at, to);
|
||||
if (win && !refill(err)) {
|
||||
std::printf("ERR refilling a lent slot failed: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
host_res[(size_t) i] = slot;
|
||||
if (!win) lend();
|
||||
const auto tsp = Clock::now();
|
||||
const bool sp_ok = win ? read_windows(at, to, err) : sp.run(ids.data() + at, to - at, at, err);
|
||||
if (trace) {
|
||||
std::fprintf(stderr, "strata trace: read %lld tokens (%s) in %.1f ms\n", (long long) (to - at),
|
||||
win ? "windows" : "batched",
|
||||
std::chrono::duration<double, std::milli>(Clock::now() - tsp).count());
|
||||
std::fflush(stderr);
|
||||
}
|
||||
if (!sp_ok) {
|
||||
if (!stop_req.load()) {
|
||||
std::fprintf(stderr, "strata serve: %s\n", err.c_str());
|
||||
std::printf("ERR %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
cancelled = true; // stopped while reading the prompt: refill the lent slots below, then DONE cancel
|
||||
break;
|
||||
}
|
||||
at = to;
|
||||
if (to == turn_at && !checkpoint_at(to)) {
|
||||
std::printf("ERR saving a conversation checkpoint failed\n");
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
if (!lent_now.empty())
|
||||
cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice);
|
||||
if (!refill(err)) {
|
||||
std::printf("ERR refilling a lent slot failed: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
tr("prompt done (slots refilled)");
|
||||
const double prompt_ms = std::chrono::duration<double, std::milli>(Clock::now() - r0).count();
|
||||
std::printf("REUSED %lld\n", (long long) resume); // the prompt is read; the first window comes next
|
||||
std::fflush(stdout);
|
||||
// the verify windows: the first holds the last prompt token alone
|
||||
int64_t p = n - 1;
|
||||
int32_t x = (int32_t) ids[(size_t) (n - 1)];
|
||||
@@ -2031,9 +2443,15 @@ int main(int argc, char** argv) {
|
||||
std::vector<float> dprob((size_t) S, 0.0f);
|
||||
bool first_window = true;
|
||||
int64_t produced_n = 0;
|
||||
int64_t draft_offered = 0, draft_accepted = 0;
|
||||
// what the session holds once this request is done: the prompt read so far, then every committed token
|
||||
std::vector<int32_t> consumed;
|
||||
consumed.reserve((size_t) (n + max_new + S));
|
||||
for (int64_t i = 0; i < n - 1; ++i) consumed.push_back((int32_t) ids[(size_t) i]);
|
||||
const char* finish = "length";
|
||||
const Clock::time_point d0 = Clock::now();
|
||||
while (produced_n < max_new) {
|
||||
if (cancelled) finish = "cancel";
|
||||
while (!cancelled && produced_n < max_new) {
|
||||
int T = S;
|
||||
if (o.spec_min_p > 0.0) {
|
||||
T = 1;
|
||||
@@ -2047,6 +2465,7 @@ int main(int argc, char** argv) {
|
||||
drive.d.experts = 0;
|
||||
drive.d.failed = false;
|
||||
apply_pending(false);
|
||||
tr("window", p, T);
|
||||
if (!ver.run(T, window.data(), p, &drive_pool_multi, &drive, outv.data(), err) || drive.d.failed) {
|
||||
std::printf("ERR %s\n", drive.d.failed && drive.d.fail ? drive.d.fail : err.c_str());
|
||||
return 1;
|
||||
@@ -2062,6 +2481,10 @@ int main(int argc, char** argv) {
|
||||
std::printf("ERR %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
// the window's first a + 1 tokens are in the session now (the last output is not: it is next x)
|
||||
for (int i = 0; i <= a; ++i) consumed.push_back(window[(size_t) i]);
|
||||
draft_offered += T - 1;
|
||||
draft_accepted += a;
|
||||
first_window = false;
|
||||
bool eos = false;
|
||||
for (int i = 0; i <= a && produced_n < max_new && !eos; ++i) {
|
||||
@@ -2083,16 +2506,85 @@ int main(int argc, char** argv) {
|
||||
return 1;
|
||||
}
|
||||
if (eos) { finish = "stop"; break; }
|
||||
if (stop_req.load()) { finish = "cancel"; break; }
|
||||
x = outv[(size_t) a];
|
||||
p += a + 1;
|
||||
}
|
||||
const double decode_ms = std::chrono::duration<double, std::milli>(Clock::now() - d0).count();
|
||||
std::printf("DONE %lld %lld %.1f %.1f %s\n", (long long) produced_n, (long long) n, prompt_ms, decode_ms,
|
||||
finish);
|
||||
if (!cancelled) {
|
||||
// a prompt stopped halfway leaves the session somewhere between two chunks: nothing to continue from
|
||||
// (the checkpoints taken while reading it are still good)
|
||||
live.swap(consumed);
|
||||
live_imgs = imgs_below(req_imgs, (int64_t) live.size());
|
||||
live_ok = o.prompt_cache > 0;
|
||||
}
|
||||
static const bool state_hash = std::getenv("STRATA_STATE_HASH") != nullptr;
|
||||
if (state_hash && live_ok) {
|
||||
// DEBUG: a fingerprint of every part of the session over the positions it holds ([0, L)), and
|
||||
// separately of what lies past them in the last KV page (stale cells, fine unless something reads them)
|
||||
cudaDeviceSynchronize();
|
||||
const int64_t L = (int64_t) live.size();
|
||||
const strata::kernels::QsaShapes qs = [&] {
|
||||
strata::kernels::QsaShapes s = strata::kernels::qsa_real_shapes();
|
||||
s.n_head_kv = g.n_head_kv; s.head_dim = g.head_dim; s.idx_dim = g.idx_key_dim;
|
||||
return s;
|
||||
}();
|
||||
auto hash_dev = [&](const void* p, size_t bytes, uint64_t h) {
|
||||
std::vector<uint8_t> b(bytes);
|
||||
if (bytes) cudaMemcpy(b.data(), p, bytes, cudaMemcpyDeviceToHost);
|
||||
return fnv1a(b.data(), b.size(), h);
|
||||
};
|
||||
// the cells [c0, c1) of one int8 K or V pool ([page][kv_head][page_size][head_dim]), bytes per value `w`
|
||||
auto hash_cells = [&](const void* pool, int64_t per_cell, int64_t c0, int64_t c1, uint64_t h) {
|
||||
const int64_t ps = qs.page_size;
|
||||
for (int64_t pg = c0 / ps; pg * ps < c1; ++pg)
|
||||
for (int64_t hd = 0; hd < qs.n_head_kv; ++hd) {
|
||||
const int64_t a = std::max(c0, pg * ps) - pg * ps, e = std::min(c1, (pg + 1) * ps) - pg * ps;
|
||||
const size_t off = (size_t) (((pg * qs.n_head_kv + hd) * ps + a) * per_cell);
|
||||
h = hash_dev((const uint8_t*) pool + off, (size_t) ((e - a) * per_cell), h);
|
||||
}
|
||||
return h;
|
||||
};
|
||||
const ConvStateSizes z = conv_state_sizes(g);
|
||||
uint64_t h_gdn = hash_dev(ss.gdn_state, z.gdn, 1469598103934665603ull);
|
||||
uint64_t h_ple = hash_dev(ss.ple_hist, z.ple, 1469598103934665603ull);
|
||||
uint64_t h_tail = 1469598103934665603ull, h_pool = h_tail, h_kv = h_tail, h_stale = h_tail;
|
||||
const int64_t kvb = qs.head_dim, scb = (qs.head_dim / 64) * 2;
|
||||
const int64_t end_cell = std::min<int64_t>(((L + qs.page_size - 1) / qs.page_size) * qs.page_size,
|
||||
ss.qsa_states[0].max_cells);
|
||||
for (int64_t i = 0; i < g.n_qsa_layers(); ++i) {
|
||||
const strata::core::QsaState& st = ss.qsa_states[i];
|
||||
h_tail = hash_dev(st.idx_tail, z.tail, h_tail);
|
||||
h_pool = hash_dev(st.idx_pooled, (size_t) (L / qs.idx_block) * qs.idx_dim * 4, h_pool);
|
||||
for (const auto& [pool, w] : {std::pair<const void*, int64_t>{st.k_q, kvb}, {st.v_q, kvb},
|
||||
{st.k_scale, scb}, {st.v_scale, scb}}) {
|
||||
h_kv = hash_cells(pool, w, 0, L, h_kv);
|
||||
h_stale = hash_cells(pool, w, L, end_cell, h_stale);
|
||||
}
|
||||
}
|
||||
const strata::core::QsaState& ms = mtp.kv_state();
|
||||
uint64_t h_mtp = 1469598103934665603ull;
|
||||
const int64_t mL = std::min<int64_t>(L, ms.max_cells);
|
||||
for (const auto& [pool, w] : {std::pair<const void*, int64_t>{ms.k_q, kvb}, {ms.v_q, kvb},
|
||||
{ms.k_scale, scb}, {ms.v_scale, scb}})
|
||||
if (pool != nullptr) h_mtp = hash_cells(pool, w, 0, mL, h_mtp);
|
||||
std::fprintf(stderr, "strata serve: STATE_HASH L=%lld gdn=%016llx ple=%016llx tail=%016llx pooled=%016llx "
|
||||
"kv=%016llx mtp=%016llx stale=%016llx ple_prev=%d,%d\n", (long long) L,
|
||||
(unsigned long long) h_gdn, (unsigned long long) h_ple, (unsigned long long) h_tail,
|
||||
(unsigned long long) h_pool, (unsigned long long) h_kv, (unsigned long long) h_mtp,
|
||||
(unsigned long long) h_stale, ss.ple_prev[0], ss.ple_prev[1]);
|
||||
}
|
||||
// DONE <generated> <prompt> <prompt ms> <decode ms> <finish> <drafts accepted> <drafts offered> <reused>
|
||||
std::printf("DONE %lld %lld %.1f %.1f %s %lld %lld %lld\n", (long long) produced_n, (long long) n, prompt_ms,
|
||||
decode_ms, finish, (long long) draft_accepted, (long long) draft_offered, (long long) resume);
|
||||
std::fflush(stdout);
|
||||
std::fprintf(stderr, "strata serve: %lld prompt tokens in %.0f ms (%.1f tok/s), %lld generated in %.0f ms "
|
||||
"(%.1f tok/s)\n", (long long) n, prompt_ms, prompt_ms > 0 ? 1000.0 * n / prompt_ms : 0.0,
|
||||
(long long) produced_n, decode_ms, decode_ms > 0 ? 1000.0 * produced_n / decode_ms : 0.0);
|
||||
const int64_t fresh = n - resume;
|
||||
std::fprintf(stderr, "strata serve: prompt %lld tokens = %lld reused + %lld read in %.0f ms (%.1f tok/s), "
|
||||
"%lld generated in %.0f ms (%.1f tok/s), drafts accepted %lld of %lld, %zu checkpoints%s\n",
|
||||
(long long) n, (long long) resume, (long long) fresh, prompt_ms,
|
||||
prompt_ms > 0 ? 1000.0 * fresh / prompt_ms : 0.0, (long long) produced_n, decode_ms,
|
||||
decode_ms > 0 ? 1000.0 * produced_n / decode_ms : 0.0, (long long) draft_accepted,
|
||||
(long long) draft_offered, checks.size(), cancelled ? " (cancelled)" : "");
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@@ -2426,6 +2918,7 @@ int main(int argc, char** argv) {
|
||||
"--expert-cache and the token graph)\n");
|
||||
return 2;
|
||||
}
|
||||
mem_mark("the head and the prompt path");
|
||||
strata::core::Verifier ver;
|
||||
strata::core::VerifyHits vh;
|
||||
vh.d_res = thits.d_res;
|
||||
@@ -2440,6 +2933,7 @@ int main(int argc, char** argv) {
|
||||
std::fprintf(stderr, "strata generate: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
mem_mark("the verifier and the drafter's binding");
|
||||
ver.set_split(o.spec_split);
|
||||
ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2
|
||||
: native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0
|
||||
|
||||
@@ -99,6 +99,30 @@ int main(int argc, char** argv) {
|
||||
if (gg) gguf_free(gg);
|
||||
}
|
||||
if (n_embd <= 0) { std::printf("ERR the vision encoder has no projection_dim\n"); std::fflush(stdout); return 1; }
|
||||
// Warm up at the LARGEST picture before READY: the encoder's GPU work buffers are allocated now, not at the first
|
||||
// real picture. The server starts this process before the engine, so the engine sizes its expert cache from
|
||||
// what is really left; allocating ~1 GB later, on a GPU the engine has filled, made Windows page GPU memory and
|
||||
// the engine crawl to a standstill. (A square image well above any cap; mtmd scales it to the token limit.)
|
||||
{
|
||||
const uint32_t side = 2048;
|
||||
std::vector<unsigned char> rgb((size_t) side * side * 3, 128);
|
||||
mtmd_bitmap* bm = mtmd_bitmap_init(side, side, rgb.data());
|
||||
mtmd_input_chunks* chunks = mtmd_input_chunks_init();
|
||||
const std::string marker = mtmd_default_marker();
|
||||
mtmd_input_text txt{marker.c_str(), marker.size(), false, true};
|
||||
const mtmd_bitmap* bms[1] = {bm};
|
||||
int warm_tokens = 0;
|
||||
if (bm && mtmd_tokenize(ctx, chunks, &txt, bms, 1) == 0) {
|
||||
for (size_t c = 0; c < mtmd_input_chunks_size(chunks); ++c) {
|
||||
const mtmd_input_chunk* ch = mtmd_input_chunks_get(chunks, c);
|
||||
if (mtmd_input_chunk_get_type(ch) == MTMD_INPUT_CHUNK_TYPE_IMAGE && mtmd_encode_chunk(ctx, ch) == 0)
|
||||
warm_tokens = (int) mtmd_input_chunk_get_n_tokens(ch);
|
||||
}
|
||||
}
|
||||
std::fprintf(stderr, "strata-vision: warmed up at %d image tokens\n", warm_tokens);
|
||||
mtmd_input_chunks_free(chunks);
|
||||
if (bm) mtmd_bitmap_free(bm);
|
||||
}
|
||||
std::printf("READY %d\n", n_embd);
|
||||
std::fflush(stdout);
|
||||
|
||||
|
||||
Reference in new issue
Block a user