mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-07 14:21:06 -10:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Niko1221
|
59b0696208 | ||
|
Niko1221
|
0223d32663 | ||
|
Niko1221
|
d0017a7c58 | ||
|
Niko1221
|
0c2e63e930 | ||
|
Niko1221
|
888751a0e1 | ||
|
Niko1221
|
af1c91f245 | ||
|
Niko1221
|
14b22c51c1 | ||
|
Vistawizard
|
be028948f4 | ||
|
Mirtraxxx
|
470b3d7c21 | ||
|
Mirtraxxx
|
be0e0c94e3 | ||
|
Mirtraxxx
|
42810ee8bb | ||
|
btechnet
|
0f8b5d4938 | ||
|
Niko1221
|
07b4a6263a | ||
|
Niko1221
|
e550970773 | ||
|
Niko1221
|
6801c77ff6 |
No files matched your search
@@ -1,313 +1,142 @@
|
||||
<h1 align="center">Qwen3.8-Flash-Next (125B MoE) on a 12-24 GB NVIDIA GPU + 64 GB RAM: one-click install for Windows / Linux</h1>
|
||||
<h1 align="center">Strata</h1>
|
||||
|
||||
<p align="center"><b>Strata</b>: an inference engine built for exactly one model and exactly this kind of PC</p>
|
||||
<p align="center"><b>Run a 125-billion-parameter AI model on a normal gaming PC</b><br>
|
||||
one NVIDIA card (12-24 GB) + 64 GB of RAM · Windows or Linux · one click to install</p>
|
||||
|
||||
A 125-billion-parameter mixture-of-experts model (6B active per token) on a **normal gaming PC**: one NVIDIA card with
|
||||
12-24 GB of VRAM, 64 GB of RAM and a desktop CPU. Strata keeps the always-used part of the model on the GPU, all 24,576
|
||||
experts in RAM, computes the experts the GPU does not hold on the CPU at the same time, and checks several tokens per
|
||||
pass with the model's own multi-token-prediction (MTP) layer. The text it produces is identical to plain greedy decoding.
|
||||
Strata runs **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** - a large, smart AI model that
|
||||
normally needs a server - on your own PC. It writes its answers at **60-95 tokens per second** (a token is about ¾
|
||||
of a word): faster than you can read.
|
||||
|
||||
It serves an **OpenAI-** and **Anthropic-compatible API on localhost**, so any chat app, coding agent or script that
|
||||
talks to those APIs can use it. Optionally it also **reads images** (screenshots, photos, scanned pages) with the
|
||||
model's own vision encoder.
|
||||
- **Private:** everything runs on your PC. Nothing is sent anywhere.
|
||||
- **Works with your apps:** chat apps, coding agents and scripts that speak the OpenAI or Anthropic API just work.
|
||||
- **Sees pictures** too, if you want (screenshots, photos, scanned pages).
|
||||
- **Free and open source.**
|
||||
|
||||
> **Jump to:** [Speed](#speed-measured) · [Other GPUs](#other-gpus-estimated) · [Which model?](#which-model) ·
|
||||
> [Windows](#windows) · [Linux](#linux) · [Using it](#using-it) · [Images](#images-vision) ·
|
||||
> [Troubleshooting](#troubleshooting) · [How it works](#how-it-works)
|
||||
> **Jump to:** [Is my PC enough?](#is-my-pc-enough) · [Install](#install-3-steps) · [How fast?](#how-fast-is-it) ·
|
||||
> [Which model?](#which-model-should-i-pick) · [Using it](#using-it) · [Problems?](#something-went-wrong) ·
|
||||
> [How it works](#how-does-it-work) · [All the details](docs/DETAILS.md)
|
||||
|
||||
---
|
||||
|
||||
## Speed (measured)
|
||||
## Is my PC enough?
|
||||
|
||||
RTX 5070 **12 GB**, Ryzen 5 7600 (6 cores), 64 GB DDR5-5200, Windows. One code-agent prompt per length, 256 generated
|
||||
tokens, MTP speculative decoding on. "262K" is the model's full context window (a 259,943-token prompt).
|
||||
|
||||
### Prompt processing (tokens/s)
|
||||
|
||||
| Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 389 | 539 | 571 | 561 | 543 | 496 |
|
||||
| **IQ2_XS** | 332 | 463 | 495 | 486 | 472 | 437 |
|
||||
| **IQ3_XXS** | 285 | 410 | 435 | 427 | 414 | - |
|
||||
|
||||
### Output (tokens/s)
|
||||
|
||||
| Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 88.7 | 94.6 | 87.5 | 76.4 | 65.1 | 56.3 |
|
||||
| **IQ2_XS** | 82.0 | 78.0 | 65.3 | 63.7 | 52.0 | 48.0 |
|
||||
| **IQ3_XXS** | 64.6 | 65.6 | 57.3 | 54.4 | 44.8 | - |
|
||||
|
||||
IQ3_XXS at 262K is not measured: with its 43 GB of experts, a 260K-token context brings a 64 GB PC to its memory
|
||||
limit. Use up to 128K with IQ3_XXS on 64 GB.
|
||||
|
||||
Time to first token is prompt length / prompt speed: about 7 s at 4K, 55 s at 32K, 4 minutes at 128K and 9 minutes at
|
||||
262K. The raw numbers: `bench/results/`. The [paper](docs/paper/Strata-Paper.pdf) explains every number.
|
||||
|
||||
## Other GPUs (estimated)
|
||||
|
||||
Not measured - estimated from the runs above (same CPU and 64 GB RAM): the GPU part scaled by memory bandwidth, the CPU
|
||||
part by how many more experts the card's VRAM holds. Treat as **±20%**. Numbers are *prompt / output* tokens/s.
|
||||
|
||||
| GPU | Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| RTX 5060 Ti 16GB | Q2_0 | ~341 / ~80 | ~472 / ~87 | ~501 / ~81 | ~492 / ~72 | ~476 / ~62 | ~435 / ~53 |
|
||||
| | IQ2_XS | ~291 / ~80 | ~406 / ~77 | ~434 / ~63 | ~426 / ~62 | ~413 / ~51 | ~383 / ~47 |
|
||||
| | IQ3_XXS | ~249 / ~66 | ~359 / ~65 | ~381 / ~56 | ~374 / ~54 | ~363 / ~45 | - |
|
||||
| RTX 3090 24GB | Q2_0 | ~355 / ~128 | ~491 / ~140 | ~521 / ~130 | ~512 / ~115 | ~495 / ~100 | ~453 / ~85 |
|
||||
| | IQ2_XS | ~303 / ~131 | ~422 / ~128 | ~451 / ~103 | ~444 / ~102 | ~430 / ~85 | ~398 / ~78 |
|
||||
| | IQ3_XXS | ~260 / ~106 | ~374 / ~103 | ~396 / ~89 | ~390 / ~85 | ~378 / ~71 | - |
|
||||
|
||||
More VRAM matters more than a faster GPU: every extra GB holds ~700 more experts, and every expert on the GPU is one the
|
||||
CPU does not have to compute. A 3090's 24 GB takes most of the CPU work away.
|
||||
|
||||
## Which model?
|
||||
|
||||
All three are [ISTA-DASLab's GSQ-RCO quantizations](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
of [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).
|
||||
|
||||
| Model | Download | RAM it uses | Speed | Quality |
|
||||
| --- | ---: | ---: | --- | --- |
|
||||
| **Q2_0** | 66 GB | ~34 GB experts + ~6 GB | fastest | good |
|
||||
| **IQ2_XS** | 68 GB | ~36 GB experts + ~6 GB | close to Q2_0 | a bit better |
|
||||
| **IQ3_XXS** | 76 GB | ~43 GB experts + ~6 GB | slower (more CPU work) | best |
|
||||
|
||||
With 64 GB of RAM all three fit (close the browser for IQ3_XXS, and keep its context at 128K or less). With 48 GB only Q2_0 / IQ2_XS may fit. 32 GB is not enough.
|
||||
|
||||
## Before you start
|
||||
|
||||
You need **only an NVIDIA driver** (version 580 or newer; update it with the NVIDIA App or from
|
||||
[nvidia.com/drivers](https://www.nvidia.com/drivers)). Everything else is installed for you the first time.
|
||||
|
||||
| | |
|
||||
| You need | |
|
||||
| --- | --- |
|
||||
| GPU | NVIDIA **RTX 30, 40 or 50 series**, **12 GB VRAM or more** (8 GB runs, slowly). Measured on an RTX 5070; RTX 30/40 are untested. |
|
||||
| RAM | **64 GB** recommended (see the table above). |
|
||||
| CPU | x86-64 with AVX2 (any Intel/AMD desktop CPU from the last ~8 years). AVX-512 (Ryzen 7000/9000) is a bit faster. |
|
||||
| Disk | ~70-80 GB free for the model, ~6 GB for the MTP layer (+1 GB with images). **Q2_0 on an AVX-512 CPU** also writes a one-time ~40 GB copy of its experts for the fast CPU kernel. An NVMe SSD is strongly recommended. |
|
||||
| OS | Windows 10/11, or Linux (Ubuntu 22.04/24.04 get everything installed automatically). |
|
||||
| **Graphics card** | NVIDIA RTX 30, 40 or 50 series with **12 GB of VRAM or more** |
|
||||
| **Memory (RAM)** | **64 GB** |
|
||||
| **Free disk space** | ~80 GB (an SSD makes the first start much faster) |
|
||||
| **System** | Windows 10/11, or Linux |
|
||||
|
||||
What the first start installs, all inside this folder (`.venv/`, `engine/`, `third_party/`, `models/`, `packs/`, `mtp/`):
|
||||
Python 3.12 if you have none (for your user account, no admin), a private Python environment, NVIDIA's CUDA libraries
|
||||
(from pip, ~0.4 GB), the ready-made Strata engine for RTX 30/40/50, the model and the MTP draft layer. If no
|
||||
ready-made engine fits your PC, it offers to install the build tools (Visual Studio Build Tools + CUDA Toolkit on
|
||||
Windows, `build-essential` + CUDA on Ubuntu) and compiles the engine for your GPU (asks first; 20-40 minutes once).
|
||||
That's it. The only thing you install yourself is a current **NVIDIA driver**
|
||||
([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA App). Everything else - Python, the engine, the
|
||||
model - is set up for you.
|
||||
|
||||
---
|
||||
## Install (3 steps)
|
||||
|
||||
## Windows
|
||||
**Windows**
|
||||
|
||||
### Double-click `START-HERE.bat`
|
||||
1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it).
|
||||
2. Double-click **`START-HERE.bat`**.
|
||||
3. Answer 4 questions - or just press Enter each time for the recommended choice:
|
||||
- **Which model?** The original, or Swift 1.5 (a version that thinks shorter and answers sooner)
|
||||
- **Which size?** Q2_0, IQ2_XS, IQ3_XXS or IQ3_S - see [which model](#which-model-should-i-pick)
|
||||
- **How much context?** How much text it can keep in mind at once (it suggests one for your card)
|
||||
- **Images?** Whether it should also read pictures
|
||||
|
||||
**The first time** it asks three questions and does the rest:
|
||||
Then it downloads everything (the model is ~70 GB, so the first time takes a while - you can stop and it picks up
|
||||
where it left off) and **starts the model**. Your browser opens a chat page at `http://127.0.0.1:8080`.
|
||||
|
||||
1. **Which model?** Q2_0, IQ2_XS or IQ3_XXS (it recommends one for your RAM).
|
||||
2. **How much context?** 8K to 256K tokens (it recommends one for your VRAM).
|
||||
3. **Images?** yes / no (see [Images](#images-vision)).
|
||||
**Next time**, just double-click `START-HERE.bat` again: it starts right away, nothing is downloaded twice. Close its
|
||||
window to stop the model.
|
||||
|
||||
Then it downloads and prepares everything (the model is 66-76 GB, so the first start takes a while; an interrupted
|
||||
download continues where it stopped) and **starts the model**: your browser opens `http://127.0.0.1:8080`, a small
|
||||
page that shows it is running and lets you chat. The API is at `http://127.0.0.1:8080/v1` for your apps.
|
||||
**Linux:** run `./setup.sh` - same questions, same result.
|
||||
|
||||
**Every time after that**, `START-HERE.bat` just starts the model (30-90 s to load 34-43 GB into RAM). Nothing is
|
||||
downloaded again. Closing the window stops the model.
|
||||
## How fast is it?
|
||||
|
||||
```
|
||||
START-HERE.bat --setup install another model, or change context / images
|
||||
START-HERE.bat --model IQ2_XS --context 32768 --vision yes --yes no questions
|
||||
START-HERE.bat --gguf-dir D:\models\IQ2_XS use GGUF files you already have
|
||||
START-HERE.bat --port 8081 another port
|
||||
```
|
||||
Measured on an RTX 5070 (12 GB), a Ryzen 5 7600 and 64 GB of RAM:
|
||||
|
||||
With more than one model installed, it asks which one to start. `run-<model>.bat` starts a model directly.
|
||||
| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| **Q2_0** | 95 tokens/s | 65 tokens/s | 539 tokens/s |
|
||||
| **IQ2_XS** | 78 tokens/s | 52 tokens/s | 463 tokens/s |
|
||||
| **IQ3_XXS** | 66 tokens/s | 45 tokens/s | 410 tokens/s |
|
||||
| **IQ3_S** | 54 tokens/s | 42 tokens/s | 374 tokens/s |
|
||||
|
||||
### Chat in the terminal (optional)
|
||||
- **Writes answers** = how fast the reply appears (tokens per second).
|
||||
- **Reads your prompt** = how fast it takes in what you send (long documents, code, chat history).
|
||||
|
||||
```
|
||||
.venv\Scripts\python chat.py
|
||||
```
|
||||
A card with more VRAM is faster, because more of the model fits on the GPU: an RTX 3090 (24 GB) should do roughly
|
||||
100-140 tokens per second. All measurements, long-context numbers and estimates for other cards are in the
|
||||
[details](docs/DETAILS.md#speed-measured).
|
||||
|
||||
---
|
||||
## Which model should I pick?
|
||||
|
||||
## Linux
|
||||
**The size** (the same model, compressed more or less):
|
||||
|
||||
```bash
|
||||
./setup.sh
|
||||
```
|
||||
| Size | Download | Speed | Quality | Pick it if... |
|
||||
| --- | ---: | --- | --- | --- |
|
||||
| **Q2_0** | 66 GB | fastest | good | you want speed |
|
||||
| **IQ2_XS** | 68 GB | fast | better | you want a good all-rounder (**recommended**) |
|
||||
| **IQ3_XXS** | 76 GB | slower | great | you want better answers (uses 43 GB of your 64 GB RAM) |
|
||||
| **IQ3_S** | 84 GB | slowest | best: matches the full model on the published tests | you want the very best answers (original model only; uses 50 GB of your 64 GB RAM, so close other big programs) |
|
||||
|
||||
The same three questions, the same automatic install (it uses `sudo apt` for Python and, only if it has to compile,
|
||||
for the build tools), and the same start: `http://127.0.0.1:8080`. Later runs of `./setup.sh` (or `./run-<model>.sh`)
|
||||
start the model directly. Options as on Windows (`./setup.sh --setup`, `--model Q2_0 --yes`, `--gguf-dir /data/Q2_0`).
|
||||
Terminal chat: `.venv/bin/python chat.py`.
|
||||
**The version:**
|
||||
|
||||
---
|
||||
- **Qwen3.8-Flash-Next** - the original.
|
||||
- **[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** - a fine-tune by UkisAI
|
||||
that thinks much shorter before answering, so you get the answer sooner, with about the same quality. Same speed per
|
||||
token. Its own license applies (see its page).
|
||||
|
||||
Not sure? Take **IQ2_XS**. You can add another one later with `START-HERE.bat --setup`.
|
||||
|
||||
## Using it
|
||||
|
||||
The server listens on `http://127.0.0.1:8080` (change with `--port` in setup, or edit the run script).
|
||||
- **Chat in the browser:** `http://127.0.0.1:8080` - a simple chat page (it opens by itself when the model starts).
|
||||
- **Chat in the terminal:** `.venv\Scripts\python chat.py`
|
||||
- **Your apps and coding agents:** add it as an "OpenAI-compatible" provider with base URL
|
||||
**`http://127.0.0.1:8080/v1`**, any API key and any model name. Apps that use Anthropic's API: `http://127.0.0.1:8080/v1/messages`.
|
||||
- **Thinking:** the model thinks before it answers. Choose **off, low, medium or high** - in the chat page menu, with
|
||||
`/think low` in `chat.py`, or with your app's "reasoning effort" setting. Off is fastest; high is best for hard questions.
|
||||
- **Pictures:** in the chat page click **Picture**; in `chat.py` type `/image <path>`; in apps just attach them.
|
||||
- **From your phone or another PC:** see the [details](docs/DETAILS.md#using-it) (set an API key first).
|
||||
|
||||
| API | Endpoint |
|
||||
**Good to know:** it answers one request at a time. The first message of a chat is read in full (about 1 minute per
|
||||
30,000 tokens); after that it keeps the conversation and reads only what is new, so follow-ups start in seconds.
|
||||
|
||||
## Something went wrong?
|
||||
|
||||
| What you see | What to do |
|
||||
| --- | --- |
|
||||
| OpenAI Chat Completions (stream and non-stream, tools) | `POST /v1/chat/completions` |
|
||||
| Anthropic Messages (stream and non-stream, tools) | `POST /v1/messages` |
|
||||
| Model list / health | `GET /v1/models`, `GET /health` |
|
||||
| `the NVIDIA driver is too old` | Update the driver (NVIDIA App or nvidia.com/drivers), restart the PC, run `START-HERE.bat` again. |
|
||||
| It stopped during download or setup | Run `START-HERE.bat` again - it continues where it stopped. |
|
||||
| `port 8080 is already in use` | Strata is already running - look for its window. |
|
||||
| The first start takes minutes | Normal: it loads 35-43 GB into RAM. The next start is faster. |
|
||||
| Slow, and the disk light is busy | Not enough free RAM: close other programs (browsers use a lot), or pick Q2_0 / IQ2_XS. |
|
||||
| "prompt exceeds the context" | The conversation is longer than the context you chose: run `START-HERE.bat --setup` and pick more. |
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
|
||||
"model": "strata", "messages": [{"role": "user", "content": "Write a haiku about GPUs."}], "max_tokens": 512 }'
|
||||
```
|
||||
More in the [full troubleshooting table](docs/DETAILS.md#troubleshooting). Still stuck? Open an issue and attach
|
||||
`strata-<model>.log` from this folder.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
|
||||
r = client.chat.completions.create(model="strata", messages=[{"role": "user", "content": "Hello!"}])
|
||||
print(r.choices[0].message.content)
|
||||
```
|
||||
## How does it work?
|
||||
|
||||
- **Thinking levels: none, low, medium, high.** The model thinks before it answers (streamed as
|
||||
`reasoning_content`, Anthropic: `thinking` blocks). Choose how much per request - in the chat page (the "Thinking"
|
||||
menu), in `chat.py` (`/think low`), or over the API:
|
||||
A model this big doesn't fit on a gaming graphics card. Strata splits the work between the parts of your PC:
|
||||
|
||||
| API | how |
|
||||
| --- | --- |
|
||||
| OpenAI | `"reasoning_effort": "none" \| "low" \| "medium" \| "high"` (also `"reasoning": {"effort": ...}`, or `"chat_template_kwargs": {"enable_thinking": false}`) |
|
||||
| Anthropic | `"output_config": {"effort": "low" \| "medium" \| "high"}`, `"thinking": {"type": "disabled"}`, or `"thinking": {"type": "enabled", "budget_tokens": N}` (under 2K = low, under 8K = medium, more = high) |
|
||||
<p align="center"><img src="docs/paper/tiers.svg" width="700" alt="how Strata splits the model between GPU, RAM and SSD"></p>
|
||||
|
||||
Without a setting the model uses its own default, **high**. `none` answers at once (fastest); `low` keeps the thinking
|
||||
short. The levels are instructions the model was trained with, not a hard token limit: on easy questions all three
|
||||
think briefly, on hard ones `high` thinks longest and is most accurate.
|
||||
- **Chat apps.** Any app with an "OpenAI-compatible" provider works: base URL `http://127.0.0.1:8080/v1`, any API key.
|
||||
- **Context.** Chosen in setup (8K-262K). Requests longer than that are refused, never silently cut.
|
||||
- **From other devices / the internet.** The server listens on your PC only (`127.0.0.1`). To reach it from elsewhere,
|
||||
put a tunnel in front of it, for example [cloudflared](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/do-more-with-tunnels/trycloudflare/):
|
||||
`cloudflared tunnel --url http://127.0.0.1:8080`. **Set a key first**, or anyone with the link can use your PC:
|
||||
add `"api_key": "some-long-secret"` to `strata-<model>.json` (or set the `STRATA_API_KEY` environment variable);
|
||||
clients then send it as their API key.
|
||||
- **The GPU** runs the part of the model that is used for every word, plus the "experts" it needs most often.
|
||||
- **The RAM** holds all 24,576 experts, and **the CPU** computes the few the GPU doesn't have - at the same time as the GPU.
|
||||
- **The SSD** holds a big lookup table; the model reads a few rows of it per word.
|
||||
- **A small helper inside the model guesses the next words**, and Strata checks several guesses at once. That makes
|
||||
it 1.6-1.8x faster than going word by word - and the answer is exactly the same.
|
||||
|
||||
**Current limits (v1):** one request at a time; greedy decoding (temperature is ignored); every request processes its
|
||||
whole prompt again (no conversation cache yet, so long chats have a long time-to-first-token); images only when set up
|
||||
with them (below); no video.
|
||||
The full story is in the [paper](docs/paper/Strata-Paper.pdf) and the [details](docs/DETAILS.md).
|
||||
|
||||
---
|
||||
## Credits
|
||||
|
||||
## Images (vision)
|
||||
|
||||
The model has a vision encoder: [`mmproj-Qwen3.8-Flash-Next-BF16.gguf`](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
(0.9 GB, a 27-layer ViT plus the projector into the language model). It is **optional**: say yes when the setup asks
|
||||
"Images?", or run it again with `--vision gpu` (or `--vision cpu`). The setup downloads the encoder, builds a small
|
||||
helper (`strata-vision`, from llama.cpp's `mtmd` library) and adds it to your start script. Nothing else changes.
|
||||
|
||||
| Encoder on | Time per picture | Cost |
|
||||
| --- | --- | --- |
|
||||
| **GPU** (recommended) | **0.1-0.5 s** (up to 1,024 image tokens) | ~1.4 GB of VRAM is kept free for it, so the expert cache is smaller: text output is a few % slower (table below) |
|
||||
| CPU | 10-30 s (pictures are scaled down to ~300 image tokens) | nothing on the GPU |
|
||||
|
||||
A picture becomes up to 1,024 tokens of the context (a 640x480 photo: 300). The same picture sent again, as chat apps
|
||||
do on every turn, is encoded only once.
|
||||
|
||||
### Sending a picture
|
||||
|
||||
**Terminal chat:** type `/image <path to a picture>`, press Enter, then type your question.
|
||||
|
||||
```
|
||||
you> /image C:\Users\me\Pictures\receipt.jpg
|
||||
(picture attached: receipt.jpg - now type your question)
|
||||
you> What is the total on this receipt?
|
||||
```
|
||||
|
||||
**OpenAI API** (an `image_url` part: a `data:` URL, an `http(s)://` URL or a local file path):
|
||||
|
||||
```python
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
|
||||
img = base64.b64encode(open("photo.jpg", "rb").read()).decode()
|
||||
r = client.chat.completions.create(model="strata", messages=[{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img}"}},
|
||||
{"type": "text", "text": "What is in this picture?"}]}])
|
||||
print(r.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Anthropic API:** an `image` block with a `base64` (or `url`) source, as usual.
|
||||
|
||||
JPEG, PNG, BMP and GIF work (WebP does not). Chat apps with image upload work the same way.
|
||||
|
||||
### Speed with images on (4K context, measured)
|
||||
|
||||
| Model | Prompt tok/s, images off | Prompt tok/s, images on | Output tok/s, images off | Output tok/s, images on |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 541 | 531 | 90.7 | 86.9 |
|
||||
| **IQ2_XS** | 468 | 458 | 77.1 | 74.0 |
|
||||
| **IQ3_XXS** | 411 | 401 | 64.7 | 59.5 |
|
||||
|
||||
"Off" is the published setup; "on" runs with the encoder loaded on the GPU and its VRAM kept free, so ~1,000 fewer
|
||||
experts fit in VRAM and the CPU computes a few more per token: 2-8% slower. For text, turning images on changes
|
||||
nothing else (the output is bit-identical when the VRAM is the same).
|
||||
|
||||
A question about a picture (a 640x480 newspaper page = 300 image tokens, a 328-token prompt; the whole request, encoder
|
||||
on the GPU, measured through the API):
|
||||
|
||||
| Model | Answer ("MEN WALK ON MOON") | Picture encoded | Prompt | Output |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 1.9 s | ~0.1 s | 207 tok/s | 65-73 tok/s |
|
||||
| **IQ2_XS** | 2.3 s | ~0.1 s | 173 tok/s | 50-60 tok/s |
|
||||
| **IQ3_XXS** | 2.7 s | ~0.1 s | 144 tok/s | 41-50 tok/s |
|
||||
|
||||
(Short prompts run below the 4K prompt speed: a 2,048-token chunk is where the prompt path is efficient. Short answers
|
||||
run below the long-output speed: the first rounds have no draft yet.)
|
||||
|
||||
**How it works inside:** the encoder turns the picture into rows of the same width as the model's word embeddings;
|
||||
Strata puts them where the prompt has `<|image_pad|>` tokens and gives each one its 2-D position (row and column in the
|
||||
picture; the model uses interleaved M-RoPE). Answers match llama.cpp's multimodal implementation token for token on our
|
||||
test images.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | What to do |
|
||||
| --- | --- |
|
||||
| `the NVIDIA driver is too old` | Update the driver (NVIDIA App or nvidia.com/drivers), restart, run `START-HERE.bat` again. |
|
||||
| Python or the build tools could not be installed | Install what it names (links are printed), then run it again. Everything already done is kept. |
|
||||
| `port 8080 is already in use` | Strata is already running (look for its window), or another program uses the port: `START-HERE.bat --port 8081`. |
|
||||
| `cudaHostRegister ... out of memory` in the log | Normal on Windows: the engine pins the experts in per-layer slices instead. Only a problem if the load then fails. |
|
||||
| The first start takes minutes | It is reading 34-43 GB into RAM; the second start is faster while the files are in the OS cache. |
|
||||
| Slow output, disk light busy | Not enough free RAM: close other programs, or choose Q2_0 / IQ2_XS. |
|
||||
| `prompt ... exceeds the context` | The request is longer than the context you chose: run setup again with a bigger `--context`. |
|
||||
| Slower than the tables | The monitor plugged into the GPU and other GPU programs take VRAM from the expert cache; RAM running below its rated speed (enable EXPO/XMP in the BIOS) slows the CPU half. |
|
||||
| `this server was started without the vision encoder` | The model was set up for text only: run setup again with `--vision gpu`. |
|
||||
| A picture is refused or `cannot read the image` | Use JPEG, PNG, BMP or GIF (not WebP). |
|
||||
| Pictures are slow (10-30 s) | The encoder runs on the CPU: run setup again with `--vision gpu` (needs ~1.4 GB of VRAM). |
|
||||
| Anything else | The engine log is `strata-<model>.log` in this folder. |
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
<p align="center"><img src="docs/paper/tiers.svg" width="760" alt="memory tiers"></p>
|
||||
|
||||
- **GPU (VRAM):** attention and DeltaNet mixers, the gated-residual weights, routers, shared experts, output head, the MTP
|
||||
draft layer, the KV cache, and an **expert cache** that fills the rest of VRAM with the most-used experts (it adapts to
|
||||
the conversation while you chat).
|
||||
- **RAM:** all 24,576 experts, pinned. The CPU computes the experts that are not on the GPU **in place**, at the same time
|
||||
as the GPU works on the cached ones (AVX-512 / AVX2 kernels, ggml's for the i-quants).
|
||||
- **SSD:** the 28.8 GB n-gram table, read a few rows per token through the OS cache.
|
||||
- **Speculation:** the model's own MTP layer drafts up to 3 tokens; one pass over all 48 layers checks them. 2.4-3.2
|
||||
tokens per pass on average.
|
||||
- **Prompts** are processed in 2,048-token chunks with the experts streamed to the GPU over PCIe.
|
||||
|
||||
The full story, with measurements, bottlenecks and what comes next: **[docs/paper/Strata-Paper.pdf](docs/paper/Strata-Paper.pdf)**.
|
||||
|
||||
---
|
||||
|
||||
## Credits and licenses
|
||||
|
||||
- Model: [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team; quantizations:
|
||||
[ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF).
|
||||
Their licenses apply to the weights.
|
||||
- [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT): the i-quant formats, the GPU dot products and
|
||||
dequantizers transcribed in `src/kernels/cuda/iq_kernels.cu`, the CPU backend linked for the i-quant experts, the
|
||||
`mtmd` library behind the image encoder (`tools/vision/`), and `gguf-py` used by the tools. See
|
||||
`third_party/ggml/LICENSE`.
|
||||
- Ideas from [Splash](https://github.com/incoai/splash), [ninfer](https://github.com/Neroued/ninfer) and
|
||||
[HyperQwen](https://github.com/syv-ai/HyperQwen); references in the paper.
|
||||
- Model: [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team; compressed versions by
|
||||
[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF);
|
||||
[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF) by UkisAI. Their licenses apply
|
||||
to the model files.
|
||||
- Built with parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT). Ideas from
|
||||
[Splash](https://github.com/incoai/splash), [ninfer](https://github.com/Neroued/ninfer) and
|
||||
[HyperQwen](https://github.com/syv-ai/HyperQwen). More in the [details](docs/DETAILS.md#credits-and-licenses).
|
||||
+340
@@ -0,0 +1,340 @@
|
||||
# Strata - the details
|
||||
|
||||
The technical side of Strata: every measured number, the API, images, all settings and how the engine works.
|
||||
New here? Start with the [README](../README.md) - it has everything you need to install and use it.
|
||||
|
||||
> **On this page:** [Speed](#speed-measured) · [Other GPUs](#other-gpus-estimated) · [Which model?](#which-model) ·
|
||||
> [Requirements](#before-you-start) · [Windows](#windows) · [Linux](#linux) · [API](#using-it) · [Images](#images-vision) ·
|
||||
> [Troubleshooting](#troubleshooting) · [How it works](#how-it-works)
|
||||
|
||||
---
|
||||
|
||||
## Speed (measured)
|
||||
|
||||
RTX 5070 **12 GB**, Ryzen 5 7600 (6 cores), 64 GB DDR5-5200, Windows. One code-agent prompt per length, 256 generated
|
||||
tokens, MTP speculative decoding on. "262K" is the model's full context window (a 259,943-token prompt).
|
||||
|
||||
### Prompt processing (tokens/s)
|
||||
|
||||
| Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 389 | 539 | 571 | 561 | 543 | 496 |
|
||||
| **IQ2_XS** | 332 | 463 | 495 | 486 | 472 | 437 |
|
||||
| **IQ3_XXS** | 285 | 410 | 435 | 427 | 414 | - |
|
||||
| **IQ3_S** | 260 | 374 | 397 | - | 378 | - |
|
||||
|
||||
### Output (tokens/s)
|
||||
|
||||
| Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 88.7 | 94.6 | 87.5 | 76.4 | 65.1 | 56.3 |
|
||||
| **IQ2_XS** | 82.0 | 78.0 | 65.3 | 63.7 | 52.0 | 48.0 |
|
||||
| **IQ3_XXS** | 64.6 | 65.6 | 57.3 | 54.4 | 44.8 | - |
|
||||
| **IQ3_S** | 51.2 | 54.4 | 51.1 | - | 42.2 | - |
|
||||
|
||||
IQ3_XXS and IQ3_S at 262K are not measured: with their 43 / 50 GB of experts, a 260K-token context brings a 64 GB PC
|
||||
to its memory limit. Use up to 128K with them on 64 GB (setup caps it). IQ3_S (engine 0.1.4 or newer) is only published
|
||||
for the original model, not for Swift 1.5.
|
||||
|
||||
Time to first token is prompt length / prompt speed: about 7 s at 4K, 55 s at 32K, 4 minutes at 128K and 9 minutes at
|
||||
262K. The raw numbers: [`bench/results/`](../bench/results/). The [paper](paper/Strata-Paper.pdf) explains every number.
|
||||
|
||||
## Other GPUs (estimated)
|
||||
|
||||
Not measured - estimated from the runs above (same CPU and 64 GB RAM): the GPU part scaled by memory bandwidth, the CPU
|
||||
part by how many more experts the card's VRAM holds. Treat as **±20%**. Numbers are *prompt / output* tokens/s.
|
||||
|
||||
| GPU | Model | 1K | 4K | 32K | 64K | 128K | 262K |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| RTX 5060 Ti 16GB | Q2_0 | ~341 / ~80 | ~472 / ~87 | ~501 / ~81 | ~492 / ~72 | ~476 / ~62 | ~435 / ~53 |
|
||||
| | IQ2_XS | ~291 / ~80 | ~406 / ~77 | ~434 / ~63 | ~426 / ~62 | ~413 / ~51 | ~383 / ~47 |
|
||||
| | IQ3_XXS | ~249 / ~66 | ~359 / ~65 | ~381 / ~56 | ~374 / ~54 | ~363 / ~45 | - |
|
||||
| RTX 3090 24GB | Q2_0 | ~355 / ~128 | ~491 / ~140 | ~521 / ~130 | ~512 / ~115 | ~495 / ~100 | ~453 / ~85 |
|
||||
| | IQ2_XS | ~303 / ~131 | ~422 / ~128 | ~451 / ~103 | ~444 / ~102 | ~430 / ~85 | ~398 / ~78 |
|
||||
| | IQ3_XXS | ~260 / ~106 | ~374 / ~103 | ~396 / ~89 | ~390 / ~85 | ~378 / ~71 | - |
|
||||
|
||||
More VRAM matters more than a faster GPU: every extra GB holds ~700 more experts, and every expert on the GPU is one the
|
||||
CPU does not have to compute. A 3090's 24 GB takes most of the CPU work away.
|
||||
|
||||
## Which model?
|
||||
|
||||
All three are [ISTA-DASLab's GSQ-RCO quantizations](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
of [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).
|
||||
|
||||
| Model | Download | RAM it uses | Speed | Quality |
|
||||
| --- | ---: | ---: | --- | --- |
|
||||
| **Q2_0** | 66 GB | ~34 GB experts + ~6 GB | fastest | good |
|
||||
| **IQ2_XS** | 68 GB | ~36 GB experts + ~6 GB | close to Q2_0 | a bit better |
|
||||
| **IQ3_XXS** | 76 GB | ~43 GB experts + ~6 GB | slower (more CPU work) | best |
|
||||
|
||||
With 64 GB of RAM all three fit (close the browser for IQ3_XXS, and keep its context at 128K or less). With 48 GB only Q2_0 / IQ2_XS may fit. 32 GB is not enough.
|
||||
|
||||
### Or: Swift 1.5 (a fine-tune that thinks shorter)
|
||||
|
||||
The setup's first question also offers **[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)**,
|
||||
UkisAI's fine-tune of Qwen3.8-Flash-Next, trained to reach the answer with much less thinking (its authors: 63% fewer
|
||||
thinking tokens, 1.8x sooner answers, under 1% accuracy loss). Same architecture, the same three sizes, its own
|
||||
vision encoder; Strata runs it at the same speed (4K, IQ2_XS: 465 prompt / 78.7 output tokens/s, vs 467 / 78.3 for
|
||||
the original). Its authors recommend **IQ2_XS** (their Q2_0 is marked experimental). Its license is the Swift Open
|
||||
License 1.0 - read it on the model page.
|
||||
|
||||
Our small check (8 reasoning questions, default thinking, IQ2_XS): both models got **8/8**; Swift used **1,234**
|
||||
output tokens in 28 s, the original **2,682** in 46 s - most of the difference from one question the original
|
||||
thought about for 1,524 tokens. Not a benchmark, but consistent with the claim.
|
||||
|
||||
```
|
||||
START-HERE.bat --setup --family swift --model IQ2_XS
|
||||
```
|
||||
|
||||
## Before you start
|
||||
|
||||
You need **only an NVIDIA driver** (version 580 or newer; update it with the NVIDIA App or from
|
||||
[nvidia.com/drivers](https://www.nvidia.com/drivers)). Everything else is installed for you the first time.
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| GPU | NVIDIA **RTX 30, 40 or 50 series**, **12 GB VRAM or more** (8 GB runs, slowly). Measured on an RTX 5070; RTX 30/40 are untested. |
|
||||
| RAM | **64 GB** recommended (see the table above). |
|
||||
| CPU | x86-64 with AVX2 (any Intel/AMD desktop CPU from the last ~8 years). AVX-512 (Ryzen 7000/9000) is a bit faster. |
|
||||
| Disk | ~70-80 GB free for the model, ~6 GB for the MTP layer (+1 GB with images). **Q2_0 on an AVX-512 CPU** also writes a one-time ~40 GB copy of its experts for the fast CPU kernel. An NVMe SSD is strongly recommended. |
|
||||
| OS | Windows 10/11, or Linux (Ubuntu 22.04/24.04 get everything installed automatically). |
|
||||
|
||||
What the first start installs, all inside this folder (`.venv/`, `engine/`, `third_party/`, `models/`, `packs/`, `mtp/`):
|
||||
Python 3.12 if you have none (for your user account, no admin), a private Python environment, NVIDIA's CUDA libraries
|
||||
(from pip, ~0.4 GB), the ready-made Strata engine for RTX 30/40/50, the model and the MTP draft layer. If no
|
||||
ready-made engine fits your PC, it offers to install the build tools (Visual Studio Build Tools + CUDA Toolkit on
|
||||
Windows, `build-essential` + CUDA on Ubuntu) and compiles the engine for your GPU (asks first; 20-40 minutes once).
|
||||
|
||||
---
|
||||
|
||||
## Windows
|
||||
|
||||
### Double-click `START-HERE.bat`
|
||||
|
||||
**The first time** it asks four questions and does the rest:
|
||||
|
||||
1. **Which model?** Qwen3.8-Flash-Next (the original) or Swift 1.5 (the fine-tune that thinks shorter).
|
||||
2. **Which size?** Q2_0, IQ2_XS or IQ3_XXS (it recommends one for your RAM).
|
||||
3. **How much context?** 8K to 256K tokens (it recommends one for your VRAM).
|
||||
4. **Images?** yes / no (see [Images](#images-vision)).
|
||||
|
||||
Then it downloads and prepares everything (the model is 66-76 GB, so the first start takes a while; an interrupted
|
||||
download continues where it stopped) and **starts the model**: your browser opens `http://127.0.0.1:8080`, a small
|
||||
page that shows it is running and lets you chat. The API is at `http://127.0.0.1:8080/v1` for your apps.
|
||||
|
||||
**Every time after that**, `START-HERE.bat` just starts the model (30-90 s to load 34-43 GB into RAM). Nothing is
|
||||
downloaded again. Closing the window stops the model.
|
||||
|
||||
```
|
||||
START-HERE.bat --setup install another model, or change context / images
|
||||
START-HERE.bat --model IQ2_XS --context 32768 --vision yes --yes no questions
|
||||
START-HERE.bat --gguf-dir D:\models\IQ2_XS use GGUF files you already have
|
||||
START-HERE.bat --port 8081 another port
|
||||
```
|
||||
|
||||
With more than one model installed, it asks which one to start. `run-<model>.bat` starts a model directly.
|
||||
|
||||
### Chat in the terminal (optional)
|
||||
|
||||
```
|
||||
.venv\Scripts\python chat.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Linux
|
||||
|
||||
```bash
|
||||
./setup.sh
|
||||
```
|
||||
|
||||
The same questions, the same automatic install (it uses `sudo apt` for Python and, only if it has to compile,
|
||||
for the build tools), and the same start: `http://127.0.0.1:8080`. Later runs of `./setup.sh` (or `./run-<model>.sh`)
|
||||
start the model directly. Options as on Windows (`./setup.sh --setup`, `--model Q2_0 --yes`, `--gguf-dir /data/Q2_0`).
|
||||
Terminal chat: `.venv/bin/python chat.py`.
|
||||
|
||||
---
|
||||
|
||||
## Using it
|
||||
|
||||
The server listens on `http://127.0.0.1:8080` (change with `--port` in setup, or edit the run script).
|
||||
|
||||
| API | Endpoint |
|
||||
| --- | --- |
|
||||
| OpenAI Chat Completions (stream and non-stream, tools) | `POST /v1/chat/completions` |
|
||||
| Anthropic Messages (stream and non-stream, tools) | `POST /v1/messages` |
|
||||
| Model list / health | `GET /v1/models`, `GET /health` |
|
||||
| What the model is doing right now | `GET /status` |
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
|
||||
"model": "strata", "messages": [{"role": "user", "content": "Write a haiku about GPUs."}], "max_tokens": 512 }'
|
||||
```
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
|
||||
r = client.chat.completions.create(model="strata", messages=[{"role": "user", "content": "Hello!"}])
|
||||
print(r.choices[0].message.content)
|
||||
```
|
||||
|
||||
- **Thinking levels: none, low, medium, high.** The model thinks before it answers (streamed as
|
||||
`reasoning_content`, Anthropic: `thinking` blocks). Choose how much per request - in the chat page (the "Thinking"
|
||||
menu), in `chat.py` (`/think low`), or over the API:
|
||||
|
||||
| API | how |
|
||||
| --- | --- |
|
||||
| OpenAI | `"reasoning_effort": "none" \| "low" \| "medium" \| "high"` (also `"reasoning": {"effort": ...}`, or `"chat_template_kwargs": {"enable_thinking": false}`) |
|
||||
| Anthropic | `"output_config": {"effort": "low" \| "medium" \| "high"}`, `"thinking": {"type": "disabled"}`, or `"thinking": {"type": "enabled", "budget_tokens": N}` (under 2K = low, under 8K = medium, more = high) |
|
||||
|
||||
Without a setting the model uses its own default, **high**. `none` answers at once (fastest); `low` keeps the thinking
|
||||
short. The levels are instructions the model was trained with, not a hard token limit: on easy questions all three
|
||||
think briefly, on hard ones `high` thinks longest and is most accurate.
|
||||
- **Streaming.** With `"stream": true` everything arrives as it is made: the thinking, the answer, and tool calls
|
||||
(the tool's name first, then its arguments piece by piece, like OpenAI and Anthropic do). While the model reads a
|
||||
long prompt the stream sends keep-alives, so agents do not time out; the server window prints progress every
|
||||
15 s, and `GET /status` says what it is doing (`reading the prompt`, `answering`, tokens so far). Closing the
|
||||
connection or pressing stop in your app really stops the model, so the next request starts at once.
|
||||
- **Chat apps.** Any app with an "OpenAI-compatible" provider works: base URL `http://127.0.0.1:8080/v1`, any API key.
|
||||
- **Context.** Chosen in setup (8K-262K). Requests longer than that are refused, never silently cut.
|
||||
- **From other devices / the internet.** The server listens on your PC only (`127.0.0.1`). To reach it from elsewhere,
|
||||
put a tunnel in front of it, for example [cloudflared](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/do-more-with-tunnels/trycloudflare/):
|
||||
`cloudflared tunnel --url http://127.0.0.1:8080`. **Set a key first**, or anyone with the link can use your PC:
|
||||
add `"api_key": "some-long-secret"` to `strata-<model>.json` (or set the `STRATA_API_KEY` environment variable);
|
||||
clients then send it as their API key.
|
||||
|
||||
**Conversation cache.** A request that continues a chat reads only the part after what the engine already holds: the
|
||||
live session, or one of the checkpoints it keeps in RAM (up to 6, ~118 MB each, taken at the start of each new
|
||||
assistant turn and every 16K prompt tokens). A checkpoint is used only when the prompt starts with exactly its tokens
|
||||
and pictures. Engine options: `--prompt-cache N` (0 = off), `--prompt-cache-every N`, `--turn-token ID`.
|
||||
|
||||
**Current limits (v1):** one request at a time, and one conversation cached at a time (switching between two chats
|
||||
re-reads the other one); greedy decoding (temperature is ignored); images only when set up with them (below); no
|
||||
video.
|
||||
|
||||
---
|
||||
|
||||
## Images (vision)
|
||||
|
||||
The model has a vision encoder: [`mmproj-Qwen3.8-Flash-Next-BF16.gguf`](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
(0.9 GB, a 27-layer ViT plus the projector into the language model). It is **optional**: say yes when the setup asks
|
||||
"Images?", or run it again with `--vision gpu` (or `--vision cpu`). The setup downloads the encoder, builds a small
|
||||
helper (`strata-vision`, from llama.cpp's `mtmd` library) and adds it to your start script. Nothing else changes.
|
||||
|
||||
| Encoder on | Time per picture | Cost |
|
||||
| --- | --- | --- |
|
||||
| **GPU** (recommended) | **0.1-0.5 s** (up to 1,024 image tokens) | ~1.4 GB of VRAM is kept free for it, so the expert cache is smaller: text output is a few % slower (table below) |
|
||||
| CPU | 10-30 s (pictures are scaled down to ~300 image tokens) | nothing on the GPU |
|
||||
|
||||
A picture becomes up to 1,024 tokens of the context (a 640x480 photo: 300). The same picture sent again, as chat apps
|
||||
do on every turn, is encoded only once.
|
||||
|
||||
### Sending a picture
|
||||
|
||||
**Terminal chat:** type `/image <path to a picture>`, press Enter, then type your question.
|
||||
|
||||
```
|
||||
you> /image C:\Users\me\Pictures\receipt.jpg
|
||||
(picture attached: receipt.jpg - now type your question)
|
||||
you> What is the total on this receipt?
|
||||
```
|
||||
|
||||
**OpenAI API** (an `image_url` part: a `data:` URL, an `http(s)://` URL or a local file path):
|
||||
|
||||
```python
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
|
||||
img = base64.b64encode(open("photo.jpg", "rb").read()).decode()
|
||||
r = client.chat.completions.create(model="strata", messages=[{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img}"}},
|
||||
{"type": "text", "text": "What is in this picture?"}]}])
|
||||
print(r.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Anthropic API:** an `image` block with a `base64` (or `url`) source, as usual.
|
||||
|
||||
JPEG, PNG, BMP, GIF, WebP, TIFF and AVIF work (the last ones are converted to PNG first; agents such as omp send
|
||||
WebP). Chat apps with image upload work the same way.
|
||||
|
||||
### Speed with images on (4K context, measured)
|
||||
|
||||
| Model | Prompt tok/s, images off | Prompt tok/s, images on | Output tok/s, images off | Output tok/s, images on |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 541 | 531 | 90.7 | 86.9 |
|
||||
| **IQ2_XS** | 468 | 458 | 77.1 | 74.0 |
|
||||
| **IQ3_XXS** | 411 | 401 | 64.7 | 59.5 |
|
||||
|
||||
"Off" is the published setup; "on" runs with the encoder loaded on the GPU and its VRAM kept free, so ~1,000 fewer
|
||||
experts fit in VRAM and the CPU computes a few more per token: 2-8% slower. For text, turning images on changes
|
||||
nothing else (the output is bit-identical when the VRAM is the same).
|
||||
|
||||
A question about a picture (a 640x480 newspaper page = 300 image tokens, a 328-token prompt; the whole request, encoder
|
||||
on the GPU, measured through the API):
|
||||
|
||||
| Model | Answer ("MEN WALK ON MOON") | Picture encoded | Prompt | Output |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| **Q2_0** | 1.9 s | ~0.1 s | 207 tok/s | 65-73 tok/s |
|
||||
| **IQ2_XS** | 2.3 s | ~0.1 s | 173 tok/s | 50-60 tok/s |
|
||||
| **IQ3_XXS** | 2.7 s | ~0.1 s | 144 tok/s | 41-50 tok/s |
|
||||
|
||||
(Short prompts run below the 4K prompt speed: a 2,048-token chunk is where the prompt path is efficient. Short answers
|
||||
run below the long-output speed: the first rounds have no draft yet.)
|
||||
|
||||
**How it works inside:** the encoder turns the picture into rows of the same width as the model's word embeddings;
|
||||
Strata puts them where the prompt has `<|image_pad|>` tokens and gives each one its 2-D position (row and column in the
|
||||
picture; the model uses interleaved M-RoPE). Answers match llama.cpp's multimodal implementation token for token on our
|
||||
test images.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | What to do |
|
||||
| --- | --- |
|
||||
| `the NVIDIA driver is too old` | Update the driver (NVIDIA App or nvidia.com/drivers), restart, run `START-HERE.bat` again. |
|
||||
| Python or the build tools could not be installed | Install what it names (links are printed), then run it again. Everything already done is kept. |
|
||||
| `port 8080 is already in use` | Strata is already running (look for its window), or another program uses the port: `START-HERE.bat --port 8081`. |
|
||||
| `cudaHostRegister ... out of memory` in the log | Normal on Windows: the engine pins the experts in per-layer slices instead. Only a problem if the load then fails. |
|
||||
| The first start takes minutes | It is reading 34-43 GB into RAM; the second start is faster while the files are in the OS cache. |
|
||||
| Slow output, disk light busy | Not enough free RAM: close other programs, or choose Q2_0 / IQ2_XS. |
|
||||
| `prompt ... exceeds the context` | The request is longer than the context you chose: run setup again with a bigger `--context`. |
|
||||
| Slower than the tables | The monitor plugged into the GPU and other GPU programs take VRAM from the expert cache; RAM running below its rated speed (enable EXPO/XMP in the BIOS) slows the CPU half. |
|
||||
| `this server was started without the vision encoder` | The model was set up for text only: run setup again with `--vision gpu`. |
|
||||
| A picture is refused or `cannot read the image` | The file is not a picture Pillow can open (JPEG, PNG, WebP, GIF, BMP, TIFF, AVIF work). |
|
||||
| Pictures are slow (10-30 s) | The encoder runs on the CPU: run setup again with `--vision gpu` (needs ~1.4 GB of VRAM). |
|
||||
| A request never finishes (older engines, mostly with pictures) | Run `START-HERE.bat` once to get engine 0.1.2 or newer. It keeps a real margin of VRAM free: the log says `... MiB of VRAM free with everything loaded` and warns when it is close to 0. |
|
||||
| Anything else | The engine log is `strata-<model>.log` in this folder. |
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
<p align="center"><img src="paper/tiers.svg" width="760" alt="memory tiers"></p>
|
||||
|
||||
- **GPU (VRAM):** attention and DeltaNet mixers, the gated-residual weights, routers, shared experts, output head, the MTP
|
||||
draft layer, the KV cache, and an **expert cache** that fills the rest of VRAM with the most-used experts (it adapts to
|
||||
the conversation while you chat).
|
||||
- **RAM:** all 24,576 experts, pinned. The CPU computes the experts that are not on the GPU **in place**, at the same time
|
||||
as the GPU works on the cached ones (AVX-512 / AVX2 kernels, ggml's for the i-quants).
|
||||
- **SSD:** the 28.8 GB n-gram table, read a few rows per token through the OS cache.
|
||||
- **Speculation:** the model's own MTP layer drafts up to 3 tokens; one pass over all 48 layers checks them. 2.4-3.2
|
||||
tokens per pass on average.
|
||||
- **Prompts** are processed in 2,048-token chunks with the experts streamed to the GPU over PCIe.
|
||||
|
||||
The full story, with measurements, bottlenecks and what comes next: **[docs/paper/Strata-Paper.pdf](paper/Strata-Paper.pdf)**.
|
||||
|
||||
---
|
||||
|
||||
## Credits and licenses
|
||||
|
||||
- Model: [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team; quantizations:
|
||||
[ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF).
|
||||
Swift 1.5: [ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)
|
||||
by UkisAI. Their licenses apply to the weights.
|
||||
- [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT): the i-quant formats, the GPU dot products and
|
||||
dequantizers transcribed in `src/kernels/cuda/iq_kernels.cu`, the CPU backend linked for the i-quant experts, the
|
||||
`mtmd` library behind the image encoder (`tools/vision/`), and `gguf-py` used by the tools. See
|
||||
`third_party/ggml/LICENSE`.
|
||||
- Ideas from [Splash](https://github.com/incoai/splash), [ninfer](https://github.com/Neroued/ninfer) and
|
||||
[HyperQwen](https://github.com/syv-ai/HyperQwen); references in the paper.
|
||||
@@ -46,6 +46,8 @@ public:
|
||||
/// The prompt's length: prefill() skips the cells the attention window can never reach again.
|
||||
void set_prompt_len(int64_t n) { prompt_len_ = n; }
|
||||
uint64_t vram_bytes() const { return vram_; }
|
||||
/// The draft layer's K/V state (read-only: --serve's STRATA_STATE_HASH check hashes it)
|
||||
const QsaState& kv_state() const { return st_; }
|
||||
/// The main model's embedding and head, and the verify window's final residuals (T rows, hc*n_embd each).
|
||||
bool bind(const WeightTable& wt, const NativeHead* head, const float* window_R, std::string& err);
|
||||
|
||||
|
||||
@@ -23,6 +23,9 @@ struct ExpertLayout {
|
||||
/// Plan v0.3 P6: per layer, the absolute offsets of the gate / up / down tensors in the model's shard 1, so
|
||||
/// the arena can be filled from the GGUF itself when the pack has no experts.bin (3 x n_layers, 0 = unknown).
|
||||
std::vector<uint64_t> gguf_off;
|
||||
/// Per layer, the GGUF file (a name beside the --native shard) that holds its experts when the model's
|
||||
/// shards split the layers (Swift's GGUFs: layers 13-47 in shard 2). Empty = the --native shard itself.
|
||||
std::vector<std::string> gguf_file;
|
||||
uint64_t max_blob = BLOB;
|
||||
uint64_t total = 0; ///< experts.bin size
|
||||
|
||||
|
||||
@@ -23,7 +23,10 @@
|
||||
#include "strata/kernels/cpu/native_expert.hpp"
|
||||
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <condition_variable>
|
||||
#include <cstdint>
|
||||
#include <mutex>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
@@ -136,10 +139,19 @@ public:
|
||||
repark = ms_repark_;
|
||||
}
|
||||
|
||||
/// **A PARKED WORKER SPINS FOR THIS LONG, THEN SLEEPS.** The park is a `_mm_pause` spin because a layer's
|
||||
/// batches are microseconds apart and a wake-up from the OS costs more than that. But a spin that never ends
|
||||
/// keeps every worker's core at 100% while the engine waits for a request - issue #4, "CPU 50% even when
|
||||
/// doing nothing" (7 of the 5700X's 16 threads). Inside a request the gaps between batches are far below
|
||||
/// this, so the token path still never sleeps; between requests the workers block on `sleep_cv_`.
|
||||
static constexpr std::chrono::milliseconds kSpinBeforeSleep{20};
|
||||
|
||||
private:
|
||||
void worker(int id);
|
||||
void drain(int id, ExpertScratch& scratch);
|
||||
void run_phase(int mode, int n_tasks);
|
||||
/// Bump `epoch_`, and wake the workers that went to sleep. Every publish goes through here.
|
||||
void publish();
|
||||
|
||||
int n_ = 0;
|
||||
bool host_works_ = true;
|
||||
@@ -169,6 +181,11 @@ private:
|
||||
alignas(64) std::atomic<uint32_t> parked_{0};
|
||||
alignas(64) std::atomic<uint32_t> epoch_{0};
|
||||
alignas(64) std::atomic<bool> stop_{false};
|
||||
// The sleep after `kSpinBeforeSleep`. `sleepers_` is how `publish` knows whether anyone needs waking, so the
|
||||
// token path pays one uncontended load per publish and never takes the mutex while the workers spin.
|
||||
alignas(64) std::atomic<uint32_t> sleepers_{0};
|
||||
std::mutex sleep_mu_;
|
||||
std::condition_variable sleep_cv_;
|
||||
std::vector<std::thread> threads_;
|
||||
std::vector<ExpertScratch> scratch_; // one per worker: no allocation, no false sharing of the hot data
|
||||
// run_split state: mode 0 = whole experts, 1 = gate/up row parts, 2 = down row parts
|
||||
|
||||
@@ -62,6 +62,9 @@ public:
|
||||
/// K/V from them. The prefill stream is synchronized before the call.
|
||||
std::function<bool(const float* R_rows, int64_t T, int64_t pos0, std::string& err)> on_chunk;
|
||||
|
||||
/// Checked before every chunk: true stops the prompt early (`run` returns false with err "cancelled").
|
||||
std::function<bool()> should_stop;
|
||||
|
||||
/// The vision path: HOST rows (n_embd floats) indexed by absolute position, read in place of the token
|
||||
/// embedding where non-null (an image's <|image_pad|> cells). Null (default): every position embeds its token.
|
||||
const float* const* embd_rows = nullptr;
|
||||
|
||||
+137
-2
@@ -270,11 +270,127 @@ class OutputParser:
|
||||
"""Incremental parser of the model's text. Feed deltas; get events. A tag split across deltas is held back
|
||||
until it is complete, so clients never see `<tool_` or `</thi`."""
|
||||
|
||||
def __init__(self, thinking: bool = True, tools: list[dict] | None = None):
|
||||
def __init__(self, thinking: bool = True, tools: list[dict] | None = None, stream_tools: bool = False):
|
||||
self.state = "reasoning" if thinking else "content"
|
||||
self.buf = ""
|
||||
self.lead = False
|
||||
self.schemas = {t.get("name"): t for t in tools or []}
|
||||
# stream_tools: a tool call is also reported while it is being written - "tool_start" (its name and id) as
|
||||
# soon as the name is known, then "tool_args" pieces of its JSON arguments (string parameters character by
|
||||
# character; other types whole, once complete) - before the final "tool_call". Without it, a client sees
|
||||
# nothing until the call is complete, which for a large file write can be many minutes.
|
||||
self.stream_tools = stream_tools
|
||||
self._reset_scan()
|
||||
|
||||
def _reset_scan(self):
|
||||
self.sp = 0 # how much of self.buf (the call body) the scanner has consumed
|
||||
self.ss = "name" # name -> between -> str|raw -> ... -> done
|
||||
self.scall = None # the ToolCall being streamed (its id is reused by the final event)
|
||||
self.sfirst = True
|
||||
self.sval_started = False
|
||||
self.sdeclared = {}
|
||||
|
||||
def _scan(self) -> list[Event]:
|
||||
"""Advance the streaming view of the call body in self.buf (see stream_tools)."""
|
||||
out = []
|
||||
|
||||
def args(s):
|
||||
if s:
|
||||
out.append(Event("tool_args", s, call=self.scall))
|
||||
while True:
|
||||
rest = self.buf[self.sp:]
|
||||
if self.ss == "name":
|
||||
a = rest.find("<function=")
|
||||
b = rest.find(">", a + 10) if a >= 0 else -1
|
||||
if b < 0:
|
||||
return out
|
||||
name = rest[a + 10:b]
|
||||
self.scall = ToolCall(name=name, arguments={})
|
||||
props = ((self.schemas.get(name) or {}).get("parameters") or {}).get("properties") or {}
|
||||
self.sdeclared = {k: (v or {}).get("type") for k, v in props.items()}
|
||||
out.append(Event("tool_start", call=self.scall))
|
||||
args("{")
|
||||
self.sp += b + 1
|
||||
self.ss = "between"
|
||||
elif self.ss == "between":
|
||||
stripped = rest.lstrip()
|
||||
self.sp += len(rest) - len(stripped)
|
||||
if stripped.startswith("<parameter="):
|
||||
b = stripped.find(">")
|
||||
if b < 0:
|
||||
return out
|
||||
pname = stripped[11:b]
|
||||
args(("" if self.sfirst else ",") + json.dumps(pname) + ":")
|
||||
self.sfirst = False
|
||||
self.sp += b + 1
|
||||
if self.sdeclared.get(pname) == "string":
|
||||
args('"')
|
||||
self.ss, self.sval_started = "str", False
|
||||
else:
|
||||
self.ss = "raw"
|
||||
elif stripped.startswith("</function>"):
|
||||
args("}")
|
||||
self.sp += len("</function>")
|
||||
self.ss = "done"
|
||||
else:
|
||||
return out # a tag still arriving (or trailing text): wait
|
||||
elif self.ss == "str":
|
||||
if not self.sval_started:
|
||||
if not rest:
|
||||
return out
|
||||
if rest[0] == "\n": # the value's leading newline is not part of it
|
||||
self.sp += 1
|
||||
rest = rest[1:]
|
||||
self.sval_started = True
|
||||
end = rest.find("</parameter>")
|
||||
if end >= 0:
|
||||
value = rest[:end]
|
||||
if value.endswith("\n"):
|
||||
value = value[:-1]
|
||||
args(json.dumps(value)[1:-1] + '"')
|
||||
self.sp += end + len("</parameter>")
|
||||
self.ss = "between"
|
||||
continue
|
||||
safe = len(rest) - self._hold(rest, ("</parameter>",))
|
||||
if safe > 0 and rest[safe - 1] == "\n": # may be the trailing newline before </parameter>
|
||||
safe -= 1
|
||||
if safe > 0:
|
||||
args(json.dumps(rest[:safe])[1:-1])
|
||||
self.sp += safe
|
||||
return out
|
||||
elif self.ss == "raw":
|
||||
end = rest.find("</parameter>")
|
||||
if end < 0:
|
||||
return out
|
||||
value = rest[:end]
|
||||
if value.startswith("\n"):
|
||||
value = value[1:]
|
||||
if value.endswith("\n"):
|
||||
value = value[:-1]
|
||||
try:
|
||||
v = json.loads(value)
|
||||
except ValueError:
|
||||
v = value
|
||||
args(json.dumps(v, ensure_ascii=False))
|
||||
self.sp += end + len("</parameter>")
|
||||
self.ss = "between"
|
||||
else:
|
||||
return out
|
||||
|
||||
def _close_scan(self) -> list[Event]:
|
||||
"""A streamed call that ended without a clean </function>: close its JSON so clients can still parse it."""
|
||||
out = []
|
||||
if self.scall is None or self.ss == "done":
|
||||
return out
|
||||
tail = ""
|
||||
if self.ss == "str":
|
||||
tail += '"'
|
||||
elif self.ss == "raw":
|
||||
tail += json.dumps(self.buf[self.sp:].strip("\n"))
|
||||
tail += "}"
|
||||
out.append(Event("tool_args", tail, call=self.scall))
|
||||
self.ss = "done"
|
||||
return out
|
||||
|
||||
def _hold(self, text: str, tags: tuple[str, ...]) -> int:
|
||||
"""Length of the longest suffix of `text` that is a proper prefix of one of `tags`."""
|
||||
@@ -325,17 +441,36 @@ class OutputParser:
|
||||
self.state = "call"
|
||||
else:
|
||||
i = self.buf.find(CALL_END)
|
||||
if self.stream_tools:
|
||||
if i >= 0:
|
||||
whole, self.buf = self.buf, self.buf[:i] # scan only the body
|
||||
out += self._scan()
|
||||
out += self._close_scan()
|
||||
self.buf = whole
|
||||
else:
|
||||
out += self._scan()
|
||||
if i < 0:
|
||||
return out
|
||||
body = self.buf[:i]
|
||||
self.buf = self.buf[i + len(CALL_END):]
|
||||
name = body.strip()[len("<function="):].split(">", 1)[0]
|
||||
out.append(Event("tool_call", call=parse_tool_call(body, self.schemas.get(name))))
|
||||
call = parse_tool_call(body, self.schemas.get(name))
|
||||
if self.scall is not None:
|
||||
call.id = self.scall.id
|
||||
out.append(Event("tool_call", call=call))
|
||||
self._reset_scan()
|
||||
self.state, self.lead = "content", True
|
||||
|
||||
def finish(self) -> list[Event]:
|
||||
"""End of generation: flush whatever is held (an unterminated tool call is returned as content)."""
|
||||
out = []
|
||||
if self.state == "call" and self.stream_tools and self.scall is not None:
|
||||
out += self._scan() # the output ended inside a call that was already announced
|
||||
out += self._close_scan()
|
||||
out.append(Event("tool_call", call=self.scall))
|
||||
self.buf = ""
|
||||
self._reset_scan()
|
||||
return out
|
||||
if self.buf:
|
||||
kind = {"reasoning": "reasoning", "content": "content"}.get(self.state, "content")
|
||||
text = self.buf if self.state != "call" else CALL_START + self.buf
|
||||
|
||||
+275
-40
@@ -6,8 +6,9 @@
|
||||
Endpoints: POST /v1/chat/completions (OpenAI, stream and non-stream), POST /v1/messages (Anthropic, stream and
|
||||
non-stream), GET /v1/models, GET /health. One sequence at a time behind a FIFO (plan: one resident sequence).
|
||||
Images (optional, when the config has a "vision" entry): OpenAI image_url parts and Anthropic image blocks (base64
|
||||
data, http(s) URLs or local file paths; JPEG/PNG/BMP/GIF) go through `strata-vision` (the model's mmproj file) and
|
||||
reach the engine as embeddings (`GENI`).
|
||||
data, http(s) URLs or local file paths) go through `strata-vision` (the model's mmproj file) and reach the engine as
|
||||
embeddings (`GENI`). JPEG/PNG/BMP/GIF go straight in; WebP, TIFF, AVIF, ... (agents like omp send WebP) are
|
||||
converted to PNG first with Pillow.
|
||||
Requests whose prompt plus max tokens exceed the engine's context are REJECTED with 400, never truncated.
|
||||
|
||||
The engine boundary is `Engine.generate(prompt_ids, max_new, sampling, cancel) -> iterator of token ids`.
|
||||
@@ -21,6 +22,7 @@ import base64
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
@@ -77,38 +79,79 @@ class StrataEngine:
|
||||
self.proc = subprocess.Popen([exe, "--serve", *args], cwd=cwd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
|
||||
stderr=self.log, text=True, encoding="utf-8", bufsize=1, env=env)
|
||||
self.max_context = 0
|
||||
self.can_stop = False # the engine honours a STOP line mid-request (READY <ctx> stop)
|
||||
self.last = {}
|
||||
for line in self.proc.stdout:
|
||||
if line.startswith("READY"):
|
||||
self.max_context = int(line.split()[1])
|
||||
f = line.split()
|
||||
self.max_context = int(f[1])
|
||||
self.can_stop = "stop" in f[2:]
|
||||
break
|
||||
if self.max_context <= 0:
|
||||
raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else ""))
|
||||
# the engine's stdout on a thread, so a request can wait with a timeout (heartbeats, cancel checks)
|
||||
self.lines: queue.Queue = queue.Queue()
|
||||
threading.Thread(target=self._pump, daemon=True).start()
|
||||
|
||||
def _pump(self):
|
||||
for line in self.proc.stdout:
|
||||
self.lines.put(line)
|
||||
self.lines.put(None)
|
||||
|
||||
def _parse_done(self, line):
|
||||
f = line.split()
|
||||
self.last = {"generated": int(f[1]), "prompt_tokens": int(f[2]), "prompt_ms": float(f[3]),
|
||||
"decode_ms": float(f[4]), "finish": f[5]}
|
||||
|
||||
def generate(self, ids, max_new, sampling, cancel, embeddings=None):
|
||||
"""Yields token ids, and None as a heartbeat every 10 s while the engine is quiet (reading a long prompt):
|
||||
the HTTP layer turns it into an SSE comment, which keeps clients' watchdogs calm and notices a client that
|
||||
has gone. A consumer that stops early (or `cancel`) makes the engine STOP, so it does not run to max_new."""
|
||||
head = f"GENI {int(max_new)} {embeddings}" if embeddings else f"GEN {int(max_new)}"
|
||||
self.proc.stdin.write(f"{head} {','.join(str(int(t)) for t in ids)}\n")
|
||||
self.proc.stdin.flush()
|
||||
done = False
|
||||
try:
|
||||
for line in self.proc.stdout:
|
||||
while True:
|
||||
try:
|
||||
line = self.lines.get(timeout=10)
|
||||
except queue.Empty:
|
||||
if cancel.is_set():
|
||||
return
|
||||
yield None
|
||||
continue
|
||||
if line is None:
|
||||
done = True
|
||||
raise RuntimeError("the engine process ended")
|
||||
if line.startswith("T "):
|
||||
if not cancel.is_set():
|
||||
yield int(line[2:])
|
||||
if cancel.is_set():
|
||||
return
|
||||
yield int(line[2:])
|
||||
elif line.startswith("PP "): # prompt progress, one per chunk: also a heartbeat (the
|
||||
if cancel.is_set(): # lines reset the 10 s wait, so without this a long prompt
|
||||
return # would send no keep-alives at all)
|
||||
yield None
|
||||
elif line.startswith("DONE"):
|
||||
f = line.split()
|
||||
self.last = {"generated": int(f[1]), "prompt_tokens": int(f[2]), "prompt_ms": float(f[3]),
|
||||
"decode_ms": float(f[4]), "finish": f[5]}
|
||||
self._parse_done(line)
|
||||
done = True
|
||||
return
|
||||
elif line.startswith("ERR"):
|
||||
done = True
|
||||
raise ValueError(line[4:].strip())
|
||||
raise RuntimeError("the engine process ended")
|
||||
finally:
|
||||
if not done: # the consumer stopped early: drain to DONE
|
||||
for line in self.proc.stdout:
|
||||
if line.startswith("DONE") or line.startswith("ERR"):
|
||||
if not done: # the consumer stopped early: stop the engine, drain to DONE
|
||||
if self.can_stop:
|
||||
try:
|
||||
self.proc.stdin.write("STOP\n")
|
||||
self.proc.stdin.flush()
|
||||
except OSError:
|
||||
pass
|
||||
while True:
|
||||
line = self.lines.get()
|
||||
if line is None or line.startswith("ERR"):
|
||||
break
|
||||
if line.startswith("DONE"):
|
||||
self._parse_done(line)
|
||||
break
|
||||
|
||||
def close(self):
|
||||
@@ -155,9 +198,37 @@ class Vision:
|
||||
return Path(path).read_bytes()
|
||||
raise ValueError("an image must be a data: URL, an http(s) URL or a local file path")
|
||||
|
||||
@staticmethod
|
||||
def normalize(data: bytes) -> bytes:
|
||||
"""The formats strata-vision's decoder (stb_image) reads pass through; anything else is converted to PNG."""
|
||||
if data[:3] == b"\xff\xd8\xff" or data[:8] == b"\x89PNG\r\n\x1a\n" or data[:2] == b"BM" or \
|
||||
data[:6] in (b"GIF87a", b"GIF89a"):
|
||||
return data
|
||||
try:
|
||||
import io
|
||||
from PIL import Image
|
||||
except ImportError:
|
||||
raise ValueError("this image format needs Pillow (python -m pip install pillow); JPEG, PNG, BMP and "
|
||||
"GIF work without it") from None
|
||||
try:
|
||||
im = Image.open(io.BytesIO(data))
|
||||
im.load()
|
||||
except Exception as e:
|
||||
raise ValueError(f"the image could not be read ({e})") from None
|
||||
if im.mode in ("RGBA", "LA", "P") and "transparency" in im.info or im.mode in ("RGBA", "LA"):
|
||||
im = im.convert("RGBA")
|
||||
bg = Image.new("RGB", im.size, (255, 255, 255)) # transparent areas become white, not black
|
||||
bg.paste(im, mask=im.split()[-1])
|
||||
im = bg
|
||||
elif im.mode != "RGB":
|
||||
im = im.convert("RGB")
|
||||
out = io.BytesIO()
|
||||
im.save(out, format="PNG")
|
||||
return out.getvalue()
|
||||
|
||||
def encode(self, source: str) -> tuple[Path, int]:
|
||||
"""-> (embeddings file, number of image tokens)."""
|
||||
data = self.load(source)
|
||||
data = self.normalize(self.load(source))
|
||||
key = hashlib.sha256(data).hexdigest()[:32]
|
||||
with self.lock:
|
||||
if key in self.cache:
|
||||
@@ -245,6 +316,8 @@ class Service:
|
||||
self.fifo = threading.Lock()
|
||||
self.embeddings = threading.local() # the current request's image embeddings file (GENI)
|
||||
self.api_key = "" # when set, /v1/* needs it (Bearer or x-api-key)
|
||||
self.status = {"busy": False, "queued": 0} # GET /status: what the model is doing right now
|
||||
self.status_lock = threading.Lock()
|
||||
self.stop_ids = set(tokenizer.encode(IM_END, parse_special=True) +
|
||||
tokenizer.encode("<|endoftext|>", parse_special=True))
|
||||
|
||||
@@ -258,7 +331,12 @@ class Service:
|
||||
raise ValueError("this server was started without the vision encoder (run setup again and choose "
|
||||
"'vision'), so it cannot read images")
|
||||
pad = self.tok.encode(IMAGE_PAD, parse_special=True)[0]
|
||||
encoded = [self.vision.encode(src) for src in images]
|
||||
# Encode only while the engine is idle: the engine and the image encoder (a separate process) must not
|
||||
# run on the GPU at the same time - an encode during a running request left that request stuck at
|
||||
# "reading the prompt" with CPU and GPU busy, for good (reproduced). So encoding takes its turn in the
|
||||
# same FIFO as the requests.
|
||||
with self.fifo:
|
||||
encoded = [self.vision.encode(src) for src in images]
|
||||
out, k = [], 0
|
||||
for t in ids: # one <|image_pad|> per image -> one per image token
|
||||
if t == pad and k < len(encoded):
|
||||
@@ -279,29 +357,116 @@ class Service:
|
||||
f"({self.engine.max_context}); requests are never truncated")
|
||||
return ids, kwargs.get("enable_thinking", True) is not False
|
||||
|
||||
def _note(self, n, evs):
|
||||
with self.status_lock:
|
||||
s = self.status
|
||||
s["generated"] = n
|
||||
if s.get("first_token") is None:
|
||||
s["first_token"] = time.time()
|
||||
for ev in evs:
|
||||
if ev.kind == "reasoning":
|
||||
s["phase"] = "thinking"
|
||||
elif ev.kind == "content":
|
||||
s["phase"] = "answering"
|
||||
elif ev.kind == "tool_start":
|
||||
s["phase"], s["tool"] = f"writing a tool call: {ev.call.name}", ev.call.name
|
||||
elif ev.kind == "tool_call":
|
||||
s["phase"] = "tool call complete"
|
||||
s["tail"] = (s["tail"] + (ev.text or ""))[-600:]
|
||||
|
||||
def _progress(self, last_print, every=1.0):
|
||||
"""A progress line in the server window every `every` seconds while a request runs."""
|
||||
now = time.time()
|
||||
if now - last_print < every:
|
||||
return last_print
|
||||
with self.status_lock:
|
||||
s = dict(self.status)
|
||||
el = now - s.get("started", now)
|
||||
if s.get("first_token") is None:
|
||||
print(f"[strata] reading the prompt: {s.get('prompt_tokens', 0)} tokens, {el:.0f} s so far", flush=True)
|
||||
else:
|
||||
rate = s["generated"] / max(1e-6, now - s["first_token"])
|
||||
print(f"[strata] {s['phase']}: {s['generated']} of max {s.get('max_tokens')} tokens, {rate:.1f} tok/s, "
|
||||
f"{el:.0f} s", flush=True)
|
||||
return now
|
||||
|
||||
def run(self, ids, thinking, tools, max_new, sampling, cancel) -> Iterator[tuple[str, object]]:
|
||||
"""Yields ("event", Event) as text arrives, then ("done", {"finish": .., "completion_tokens": ..})."""
|
||||
parser, detok, n, finish = OutputParser(thinking=thinking, tools=tools), Detokenizer(self.tok), 0, "length"
|
||||
parser = OutputParser(thinking=thinking, tools=tools, stream_tools=True)
|
||||
detok, n, finish = Detokenizer(self.tok), 0, "length"
|
||||
raw_ids = [] # every generated id (STRATA_DEBUG: dump raw model text)
|
||||
emb = getattr(self.embeddings, "path", None)
|
||||
with self.status_lock:
|
||||
self.status["queued"] += 1
|
||||
try:
|
||||
with self.fifo:
|
||||
with self.status_lock:
|
||||
self.status["queued"] -= 1
|
||||
self.status.update(busy=True, phase="reading the prompt", prompt_tokens=len(ids), generated=0,
|
||||
started=time.time(), first_token=None, tool=None, tail="", max_tokens=max_new)
|
||||
last_print = time.time()
|
||||
gen = self.engine.generate(ids, max_new, sampling, cancel, embeddings=emb) if emb else \
|
||||
self.engine.generate(ids, max_new, sampling, cancel)
|
||||
for t in gen:
|
||||
n += 1
|
||||
if t in self.stop_ids:
|
||||
finish = "stop"
|
||||
break
|
||||
for ev in parser.feed(detok.push(t)):
|
||||
yield "event", ev
|
||||
try:
|
||||
for t in gen:
|
||||
if t is None: # heartbeat while the engine is quiet
|
||||
last_print = self._progress(last_print)
|
||||
yield "ping", None
|
||||
continue
|
||||
n += 1
|
||||
if t in self.stop_ids:
|
||||
finish = "stop"
|
||||
raw_ids.append(t)
|
||||
break
|
||||
raw_ids.append(t)
|
||||
evs = parser.feed(detok.push(t))
|
||||
self._note(n, evs)
|
||||
last_print = self._progress(last_print)
|
||||
for ev in evs:
|
||||
yield "event", ev
|
||||
if cancel.is_set():
|
||||
finish = "cancel"
|
||||
finally:
|
||||
gen.close() # STOP+drain to THIS request's DONE while still holding the
|
||||
# fifo, so a stop-token break can't leave the shared engine
|
||||
# queue mid-drain for the next request to read as its own DONE
|
||||
except GeneratorExit: # the client disconnected mid-stream
|
||||
finish = "disconnect"
|
||||
raise
|
||||
finally:
|
||||
if emb:
|
||||
Path(emb).unlink(missing_ok=True)
|
||||
with self.status_lock:
|
||||
if self.status.get("busy"):
|
||||
now = time.time()
|
||||
el = now - self.status.get("started", now)
|
||||
ft = self.status.get("first_token")
|
||||
rate = n / max(1e-6, now - ft) if ft else 0.0
|
||||
print(f"[strata] done: {n} tokens in {el:.0f} s ({rate:.1f} tok/s) "
|
||||
f"({finish}, cancel={cancel.is_set()})", flush=True)
|
||||
if os.environ.get("STRATA_DEBUG") and raw_ids:
|
||||
print(f"[strata] raw: {self.tok.decode(raw_ids)!r}", flush=True)
|
||||
self.status["busy"] = False
|
||||
for ev in parser.finish():
|
||||
yield "event", ev
|
||||
yield "done", {"finish": finish, "completion_tokens": n}
|
||||
|
||||
|
||||
def _debug_req(api, req, messages, tools, max_new, thinking, prompt_tokens):
|
||||
"""One compact line per request while diagnosing blank/empty turns. Set STRATA_DEBUG=1 to enable."""
|
||||
if not os.environ.get("STRATA_DEBUG"):
|
||||
return
|
||||
last = messages[-1] if messages else {}
|
||||
body = last.get("content")
|
||||
if isinstance(body, list):
|
||||
body = " ".join(p.get("text", "") for p in body if isinstance(p, dict))
|
||||
preview = (str(body or "")[:80]).replace("\n", " ")
|
||||
print(f"[strata] req {api}: msgs={len(messages)} tools={len(tools or [])} "
|
||||
f"max_tokens_raw={req.get('max_tokens')!r}/{req.get('max_completion_tokens')!r} "
|
||||
f"max_new={max_new} thinking={thinking} stream={bool(req.get('stream'))} "
|
||||
f"prompt_tokens={prompt_tokens} last={last.get('role')!r}:{preview!r}", flush=True)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------ OpenAI
|
||||
def openai_chunks(svc: Service, req: dict, ids, thinking, tools, max_new, cancel):
|
||||
cid, created = "chatcmpl-" + uuid.uuid4().hex[:24], int(time.time())
|
||||
@@ -312,20 +477,32 @@ def openai_chunks(svc: Service, req: dict, ids, thinking, tools, max_new, cancel
|
||||
|
||||
yield chunk({"role": "assistant", "content": ""})
|
||||
calls = 0
|
||||
streamed = {} # tool call id -> index, for calls sent piece by piece
|
||||
for kind, x in svc.run(ids, thinking, tools, max_new, req, cancel):
|
||||
if kind == "event":
|
||||
if kind == "ping":
|
||||
yield None
|
||||
elif kind == "event":
|
||||
ev: Event = x
|
||||
if ev.kind == "reasoning" and ev.text:
|
||||
yield chunk({"reasoning_content": ev.text})
|
||||
elif ev.kind == "content" and ev.text:
|
||||
yield chunk({"content": ev.text})
|
||||
elif ev.kind == "tool_start":
|
||||
streamed[ev.call.id] = calls
|
||||
calls += 1
|
||||
yield chunk({"tool_calls": [{"index": streamed[ev.call.id], "id": ev.call.id, "type": "function",
|
||||
"function": {"name": ev.call.name, "arguments": ""}}]})
|
||||
elif ev.kind == "tool_args":
|
||||
yield chunk({"tool_calls": [{"index": streamed[ev.call.id], "function": {"arguments": ev.text}}]})
|
||||
elif ev.kind == "tool_call" and ev.call.id in streamed:
|
||||
continue
|
||||
elif ev.kind == "tool_call":
|
||||
yield chunk({"tool_calls": [{"index": calls, "id": ev.call.id, "type": "function",
|
||||
"function": {"name": ev.call.name,
|
||||
"arguments": json.dumps(ev.call.arguments, ensure_ascii=False)}}]})
|
||||
calls += 1
|
||||
else:
|
||||
finish = "tool_calls" if calls and x["finish"] == "stop" else x["finish"]
|
||||
finish = "tool_calls" if calls and x["finish"] == "stop" else {"cancel": "stop"}.get(x["finish"], x["finish"])
|
||||
last = chunk({}, finish)
|
||||
last["usage"] = {"prompt_tokens": len(ids), "completion_tokens": x["completion_tokens"],
|
||||
"total_tokens": len(ids) + x["completion_tokens"]}
|
||||
@@ -333,13 +510,22 @@ def openai_chunks(svc: Service, req: dict, ids, thinking, tools, max_new, cancel
|
||||
|
||||
|
||||
def openai_collect(chunks) -> dict:
|
||||
content, reasoning, calls, last = [], [], [], None
|
||||
content, reasoning, by_index, last = [], [], {}, None
|
||||
for c in chunks:
|
||||
if c is None: # a heartbeat
|
||||
continue
|
||||
d = c["choices"][0]["delta"]
|
||||
content.append(d.get("content") or "")
|
||||
reasoning.append(d.get("reasoning_content") or "")
|
||||
calls += [{k: v for k, v in tc.items() if k != "index"} for tc in d.get("tool_calls") or []]
|
||||
for tc in d.get("tool_calls") or []: # streamed calls arrive in pieces: merge them by index
|
||||
cur = by_index.setdefault(tc.get("index", len(by_index)), {"id": None, "type": "function",
|
||||
"function": {"name": "", "arguments": ""}})
|
||||
cur["id"] = tc.get("id") or cur["id"]
|
||||
fn = tc.get("function") or {}
|
||||
cur["function"]["name"] += fn.get("name") or ""
|
||||
cur["function"]["arguments"] += fn.get("arguments") or ""
|
||||
last = c
|
||||
calls = [by_index[i] for i in sorted(by_index)]
|
||||
msg = {"role": "assistant", "content": "".join(content) or None}
|
||||
if "".join(reasoning):
|
||||
msg["reasoning_content"] = "".join(reasoning)
|
||||
@@ -361,12 +547,25 @@ def anthropic_events(svc: Service, req: dict, ids, thinking, tools, max_new, can
|
||||
def close():
|
||||
return ("content_block_stop", {"type": "content_block_stop", "index": index})
|
||||
|
||||
streamed = set()
|
||||
for kind, x in svc.run(ids, thinking, tools, max_new, req, cancel):
|
||||
if kind == "ping":
|
||||
yield None
|
||||
continue
|
||||
if kind == "event":
|
||||
ev: Event = x
|
||||
want = {"reasoning": "thinking", "content": "text", "tool_call": "tool_use"}[ev.kind]
|
||||
if ev.kind != "tool_call" and not ev.text:
|
||||
if ev.kind == "tool_args":
|
||||
yield "content_block_delta", {"type": "content_block_delta", "index": index,
|
||||
"delta": {"type": "input_json_delta", "partial_json": ev.text}}
|
||||
continue
|
||||
if ev.kind == "tool_call" and ev.call.id in streamed:
|
||||
continue
|
||||
want = {"reasoning": "thinking", "content": "text", "tool_call": "tool_use", "tool_start": "tool_use"}[ev.kind]
|
||||
if ev.kind not in ("tool_call", "tool_start") and not ev.text:
|
||||
continue
|
||||
if ev.kind == "tool_start":
|
||||
streamed.add(ev.call.id)
|
||||
used_tool = True
|
||||
if open_kind != want or want == "tool_use":
|
||||
if open_kind is not None:
|
||||
yield close()
|
||||
@@ -383,6 +582,8 @@ def anthropic_events(svc: Service, req: dict, ids, thinking, tools, max_new, can
|
||||
elif want == "text":
|
||||
yield "content_block_delta", {"type": "content_block_delta", "index": index,
|
||||
"delta": {"type": "text_delta", "text": ev.text}}
|
||||
elif ev.kind == "tool_start":
|
||||
pass # its input follows as tool_args pieces
|
||||
else:
|
||||
used_tool = True
|
||||
yield "content_block_delta", {"type": "content_block_delta", "index": index, "delta": {
|
||||
@@ -391,7 +592,7 @@ def anthropic_events(svc: Service, req: dict, ids, thinking, tools, max_new, can
|
||||
if open_kind is not None:
|
||||
yield close()
|
||||
stop = "tool_use" if used_tool and x["finish"] == "stop" else \
|
||||
{"stop": "end_turn", "length": "max_tokens"}[x["finish"]]
|
||||
{"stop": "end_turn", "length": "max_tokens", "cancel": "end_turn"}[x["finish"]]
|
||||
yield "message_delta", {"type": "message_delta", "delta": {"stop_reason": stop, "stop_sequence": None},
|
||||
"usage": {"output_tokens": x["completion_tokens"]}}
|
||||
yield "message_stop", {"type": "message_stop"}
|
||||
@@ -399,7 +600,10 @@ def anthropic_events(svc: Service, req: dict, ids, thinking, tools, max_new, can
|
||||
|
||||
def anthropic_collect(events) -> dict:
|
||||
msg, blocks = None, []
|
||||
for name, e in events:
|
||||
for item in events:
|
||||
if item is None: # a heartbeat
|
||||
continue
|
||||
name, e = item
|
||||
if name == "message_start":
|
||||
msg = e["message"]
|
||||
elif name == "content_block_start":
|
||||
@@ -410,8 +614,11 @@ def anthropic_collect(events) -> dict:
|
||||
b["text"] += d["text"]
|
||||
elif d["type"] == "thinking_delta":
|
||||
b["thinking"] += d["thinking"]
|
||||
else:
|
||||
b["input"] = json.loads(d["partial_json"])
|
||||
else: # input_json_delta pieces: parsed when complete
|
||||
b["_json"] = b.get("_json", "") + d["partial_json"]
|
||||
elif name == "content_block_stop" and blocks and "_json" in blocks[-1]:
|
||||
b = blocks[-1]
|
||||
b["input"] = json.loads(b.pop("_json") or "{}")
|
||||
elif name == "message_delta":
|
||||
msg["stop_reason"] = e["delta"]["stop_reason"]
|
||||
msg["usage"]["output_tokens"] = e["usage"]["output_tokens"]
|
||||
@@ -457,6 +664,17 @@ def make_handler(svc: Service):
|
||||
elif path == "/health":
|
||||
self._json(200, {"status": "ok", "max_context": svc.engine.max_context, "model": svc.model,
|
||||
"images": svc.vision is not None, "api_key": bool(svc.api_key)})
|
||||
elif path == "/status":
|
||||
with svc.status_lock:
|
||||
s = dict(svc.status)
|
||||
now = time.time()
|
||||
if s.get("busy"):
|
||||
s["elapsed_s"] = round(now - s["started"], 1)
|
||||
if s.get("first_token"):
|
||||
s["tokens_per_s"] = round(s["generated"] / max(1e-6, now - s["first_token"]), 1)
|
||||
for k in ("started", "first_token"):
|
||||
s.pop(k, None)
|
||||
self._json(200, s)
|
||||
elif path == "/v1/models":
|
||||
if self._authorized():
|
||||
self._json(200, {"object": "list", "data": [{"id": svc.model, "object": "model"}]})
|
||||
@@ -486,7 +704,10 @@ def make_handler(svc: Service):
|
||||
def _openai(self, req):
|
||||
messages, tools, kw = openai_to_messages(req)
|
||||
max_new = int(req.get("max_completion_tokens") or req.get("max_tokens") or 1024)
|
||||
if max_new <= 0: # a non-positive budget (some clients send -1) means "unset"
|
||||
max_new = 1024
|
||||
ids, thinking = svc.prepare(messages, tools, kw, max_new)
|
||||
_debug_req("openai", req, messages, tools, max_new, thinking, len(ids))
|
||||
cancel = threading.Event()
|
||||
chunks = openai_chunks(svc, req, ids, thinking, tools, max_new, cancel)
|
||||
if not req.get("stream"):
|
||||
@@ -494,28 +715,40 @@ def make_handler(svc: Service):
|
||||
self._sse()
|
||||
try:
|
||||
for c in chunks:
|
||||
self.wfile.write(b"data: " + json.dumps(c, ensure_ascii=False).encode() + b"\n\n")
|
||||
if c is None:
|
||||
self.wfile.write(b": keep-alive\n\n") # an SSE comment: clients ignore it
|
||||
else:
|
||||
self.wfile.write(b"data: " + json.dumps(c, ensure_ascii=False).encode() + b"\n\n")
|
||||
self.wfile.flush()
|
||||
self.wfile.write(b"data: [DONE]\n\n")
|
||||
except OSError:
|
||||
cancel.set() # client went away: stop at the next step
|
||||
cancel.set() # client went away: stop the engine
|
||||
chunks.close()
|
||||
|
||||
def _anthropic(self, req):
|
||||
messages, tools, kw = anthropic_to_messages(req)
|
||||
max_new = int(req.get("max_tokens") or 1024)
|
||||
if max_new <= 0: # a non-positive budget (some clients send -1) means "unset"
|
||||
max_new = 1024
|
||||
ids, thinking = svc.prepare(messages, tools, kw, max_new)
|
||||
_debug_req("anthropic", req, messages, tools, max_new, thinking, len(ids))
|
||||
cancel = threading.Event()
|
||||
events = anthropic_events(svc, req, ids, thinking, tools, max_new, cancel)
|
||||
if not req.get("stream"):
|
||||
return self._json(200, anthropic_collect(events))
|
||||
self._sse()
|
||||
try:
|
||||
for name, e in events:
|
||||
self.wfile.write(f"event: {name}\n".encode() + b"data: " +
|
||||
json.dumps(e, ensure_ascii=False).encode() + b"\n\n")
|
||||
for item in events:
|
||||
if item is None:
|
||||
self.wfile.write(b": keep-alive\n\n")
|
||||
else:
|
||||
name, e = item
|
||||
self.wfile.write(f"event: {name}\n".encode() + b"data: " +
|
||||
json.dumps(e, ensure_ascii=False).encode() + b"\n\n")
|
||||
self.wfile.flush()
|
||||
except OSError:
|
||||
cancel.set()
|
||||
events.close()
|
||||
|
||||
return Handler
|
||||
|
||||
@@ -546,7 +779,7 @@ def main() -> int:
|
||||
ap.add_argument("--api-key", default=os.environ.get("STRATA_API_KEY", ""),
|
||||
help="require this key on /v1/* (Authorization: Bearer ... or x-api-key); also $STRATA_API_KEY")
|
||||
a = ap.parse_args()
|
||||
cfg = json.loads(Path(a.config).read_text(encoding="utf-8")) if a.config else {}
|
||||
cfg = json.loads(Path(a.config).read_text(encoding="utf-8-sig")) if a.config else {} # Notepad adds a BOM
|
||||
try: # before the minutes of loading: is the port free?
|
||||
Server((a.host, a.port), BaseHTTPRequestHandler).server_close()
|
||||
except OSError:
|
||||
@@ -580,7 +813,9 @@ def main() -> int:
|
||||
engine = StrataEngine(cfg["exe"], cfg["args"], cwd=cfg.get("cwd"), log=cfg.get("log"), env=env)
|
||||
else:
|
||||
engine, vision = MockEngine(tok, a.script), None
|
||||
svc = Service(engine, tok, ChatTemplate(ROOT / "serve/chat_template.jinja"),
|
||||
# the model's own chat template (exported with its tokenizer), else the original model's
|
||||
tpl = tpath / "chat_template.jinja"
|
||||
svc = Service(engine, tok, ChatTemplate(tpl if tpl.exists() else ROOT / "serve/chat_template.jinja"),
|
||||
model_name=cfg.get("model_name", "qwen3.8-flash-next"), vision=vision)
|
||||
svc.api_key = a.api_key or cfg.get("api_key", "")
|
||||
httpd = serve(svc, host=a.host, port=a.port)
|
||||
|
||||
@@ -3,15 +3,15 @@
|
||||
|
||||
START-HERE.bat (Windows) / ./setup.sh (Linux) - they install Python if needed and run this file
|
||||
|
||||
The first time it asks three questions - which model, how much context, and whether the model should also read
|
||||
images - then installs everything and starts the model on http://127.0.0.1:8080 (OpenAI- and Anthropic-compatible
|
||||
The first time it asks four questions - which model (the original Qwen3.8-Flash-Next or the Swift 1.5 fine-tune),
|
||||
which size, how much context, and whether the model should also read images - then installs everything and starts the model on http://127.0.0.1:8080 (OpenAI- and Anthropic-compatible
|
||||
API; a small page there shows that it runs). Every later start skips straight to running the model: nothing that
|
||||
is already downloaded, installed or prepared is done again.
|
||||
|
||||
What the first run does (each step is skipped when it is already done):
|
||||
|
||||
1. checks your PC: NVIDIA GPU and driver, RAM, CPU, free disk space
|
||||
2. asks the three questions
|
||||
2. asks the questions
|
||||
3. installs the Python packages it needs into .venv (numpy, jinja2, ..., and NVIDIA's CUDA libraries)
|
||||
4. gets the Strata engine: a ready-made build for RTX 30/40/50 cards (no compiler needed); if none fits your PC,
|
||||
it installs the build tools (asks first) and compiles the engine for your GPU
|
||||
@@ -19,7 +19,7 @@ What the first run does (each step is skipped when it is already done):
|
||||
6. prepares the model for Strata and fetches the MTP draft layer (~5 GB, from the original Qwen checkpoint)
|
||||
7. writes run-<model>.bat / run-<model>.sh and starts the model
|
||||
|
||||
Options: --model Q2_0|IQ2_XS|IQ3_XXS, --context 32768, --vision yes|no|gpu|cpu, --port 8080, --yes (recommended
|
||||
Options: --family qwen|swift, --model Q2_0|IQ2_XS|IQ3_XXS|IQ3_S, --context 32768, --vision yes|no|gpu|cpu, --port 8080, --yes (recommended
|
||||
answers, no questions), --setup (install another model / change settings instead of starting), --no-start,
|
||||
--models-dir DIR, --gguf-dir DIR (use GGUF files you already have), --build (compile instead of the ready-made
|
||||
engine), --check (only check this PC).
|
||||
@@ -54,20 +54,41 @@ PREBUILT_ASSET = "strata-windows-x64.zip" if WIN else "strata-linux-x64.zip"
|
||||
# the CUDA libraries the ready-made engine loads (the same CUDA 13.0 it is built with), from NVIDIA's pip packages
|
||||
CUDA_WHEELS = ["nvidia-cublas==13.0.2.14", "nvidia-cuda-runtime==13.0.96"]
|
||||
MIN_DRIVER = 580 # CUDA 13.0
|
||||
PY_PACKAGES = ["numpy", "jinja2", "regex", "pyyaml", "tqdm", "requests", "cmake", "ninja"]
|
||||
MIN_ENGINE = (0, 1, 3) # conversation cache, idle pool sleeps, Linux pool fix (v0.1.3)
|
||||
PY_PACKAGES = ["numpy", "jinja2", "regex", "pyyaml", "tqdm", "requests", "cmake", "ninja", "pillow"]
|
||||
|
||||
MODELS = {
|
||||
"Q2_0": {"about": "2-bit, the fastest", "download_gb": 66.4, "ram_gb": 48, "arena_gb": 34.0},
|
||||
"IQ2_XS": {"about": "2-bit i-quant, a little better quality, close in speed", "download_gb": 68.0, "ram_gb": 48,
|
||||
"arena_gb": 35.5},
|
||||
"IQ3_XXS": {"about": "3-bit i-quant, the best quality, slower (more CPU work per token)", "download_gb": 75.8,
|
||||
"IQ3_XXS": {"about": "3-bit i-quant, better quality, slower (more CPU work per token)", "download_gb": 75.8,
|
||||
"ram_gb": 60, "arena_gb": 42.9},
|
||||
# the original model only (Swift 1.5 has no IQ3_S): matches the full BF16 model on the published benchmarks
|
||||
"IQ3_S": {"about": "3.5-bit i-quant, the best quality (matches the full model), the slowest; needs a 64 GB PC "
|
||||
"with little else running", "download_gb": 83.6, "ram_gb": 62, "arena_gb": 50.3,
|
||||
"families": ("qwen",)},
|
||||
}
|
||||
CONTEXTS = [8192, 32768, 65536, 131072, 262144]
|
||||
# The model families: the same architecture, weights in the same three GSQ-RCO sizes, different files.
|
||||
FAMILIES = {
|
||||
"qwen": {"title": "Qwen3.8-Flash-Next", "by": "Qwen; GSQ-RCO quants by ISTA-DASLab",
|
||||
"about": "the original model",
|
||||
"hf": "https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/main/{q}/",
|
||||
"file": "Qwen3.8-Flash-Next-GSQ-RCO-{q}-0000{i}-of-00002.gguf", "tag": "",
|
||||
"mmproj_hf": "https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/main/",
|
||||
"mmproj": "mmproj-Qwen3.8-Flash-Next-BF16.gguf", "name": "qwen3.8-flash-next"},
|
||||
"swift": {"title": "Swift 1.5", "by": "UkisAI's fine-tune of Qwen3.8-Flash-Next",
|
||||
"about": "thinks much shorter (-63% thinking tokens, 1.8x sooner answers by its authors' numbers)",
|
||||
"hf": "https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/main/",
|
||||
"file": "Swift-Qwen3.8-Flash-Next-GSQ-RCO-{q}-0000{i}-of-00002.gguf", "tag": "swift-",
|
||||
"mmproj_hf": "https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/main/",
|
||||
"mmproj": "mmproj-Swift-Qwen3.8-Flash-Next-BF16.gguf", "name": "swift-1.5",
|
||||
"license": "Swift Open License 1.0: https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF"},
|
||||
}
|
||||
MMPROJ = "mmproj-Qwen3.8-Flash-Next-BF16.gguf"
|
||||
# the image encoder on the GPU: ~0.9 GB of weights + ~0.3 GB of work buffers at 1024 image tokens, kept free of
|
||||
# expert slots (the engine's default reserve is 700 MiB)
|
||||
VISION = {"gpu": {"max_tokens": 1024, "reserve_mib": 2100},
|
||||
# the image encoder on the GPU (~1.2 GB at 1024 image tokens) warms up before the engine starts, so the engine
|
||||
# sizes its expert slots around it and the default reserve (700 MiB) is enough; engines before 0.1.2 need more
|
||||
VISION = {"gpu": {"max_tokens": 1024, "reserve_mib": 700},
|
||||
"cpu": {"max_tokens": 300, "reserve_mib": 700}}
|
||||
EXE = "strata.exe" if WIN else "strata"
|
||||
VEXE = "strata-vision.exe" if WIN else "strata-vision"
|
||||
@@ -384,8 +405,13 @@ def get_prebuilt(url_base, gpu, vision) -> Path | None:
|
||||
eng = ROOT / "engine"
|
||||
info = eng / "BUILD.json"
|
||||
if info.exists() and (eng / EXE).exists():
|
||||
ok("ready-made engine already installed")
|
||||
return eng
|
||||
meta = json.loads(info.read_text())
|
||||
ver = tuple(int(x) for x in str(meta.get("version", "0")).split(".")[:3] if x.isdigit())
|
||||
if meta.get("source") == "local" or ver >= MIN_ENGINE:
|
||||
ok("ready-made engine already installed")
|
||||
return eng
|
||||
say(f" Updating the ready-made engine ({meta.get('version')} -> {'.'.join(map(str, MIN_ENGINE))} or newer) ...")
|
||||
info.unlink()
|
||||
if not url_base:
|
||||
return None
|
||||
z = ROOT / "engine" / PREBUILT_ASSET
|
||||
@@ -404,6 +430,11 @@ def get_prebuilt(url_base, gpu, vision) -> Path | None:
|
||||
with zipfile.ZipFile(z) as f:
|
||||
f.extractall(tmp)
|
||||
meta = json.loads((tmp / "BUILD.json").read_text())
|
||||
if tuple(int(x) for x in str(meta.get("version", "0")).split(".")[:3] if x.isdigit()) < MIN_ENGINE:
|
||||
warn(f"the ready-made engine at {base} is version {meta.get('version')}; this setup needs "
|
||||
f"{'.'.join(map(str, MIN_ENGINE))}: compiling instead")
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
return None
|
||||
archs = [int(a) for a in meta.get("archs", [])]
|
||||
arch = int(gpu["arch"])
|
||||
if arch not in archs and not (meta.get("ptx") and arch > max(archs)):
|
||||
@@ -429,6 +460,42 @@ def get_prebuilt(url_base, gpu, vision) -> Path | None:
|
||||
return eng
|
||||
|
||||
|
||||
def update_installed_engine(url_base) -> None:
|
||||
"""An installed ready-made engine older than MIN_ENGINE is replaced before the model starts, so a plain
|
||||
START-HERE.bat on an existing install picks up a new release. If that cannot happen (no internet, the model
|
||||
still running, no ready-made engine for this GPU) the installed engine is kept and starts as before."""
|
||||
eng = ROOT / "engine"
|
||||
info = eng / "BUILD.json"
|
||||
if not info.exists() or not (eng / EXE).exists():
|
||||
return
|
||||
meta_text = info.read_text()
|
||||
meta = json.loads(meta_text)
|
||||
ver = tuple(int(x) for x in str(meta.get("version", "0")).split(".")[:3] if x.isdigit())
|
||||
if meta.get("source") == "local" or ver >= MIN_ENGINE:
|
||||
return
|
||||
try: # a running engine cannot be replaced (Windows keeps it locked)
|
||||
for x in (EXE, VEXE):
|
||||
if (eng / x).exists():
|
||||
with open(eng / x, "r+b"):
|
||||
pass
|
||||
except OSError:
|
||||
warn(f"engine {meta.get('version')} is in use: close the model window and run this again to update it")
|
||||
return
|
||||
gpu = gpu_info()
|
||||
new = None
|
||||
if gpu is not None:
|
||||
try:
|
||||
new = get_prebuilt(url_base, gpu, "gpu")
|
||||
except Exception as e: # a failed download must not stop the model from starting
|
||||
warn(f"updating the engine failed ({e})")
|
||||
if new is None:
|
||||
if not info.exists():
|
||||
info.write_text(meta_text) # get_prebuilt drops it before downloading: put it back
|
||||
warn(f"could not update the engine: starting the installed {meta.get('version')}")
|
||||
return
|
||||
pip_install(CUDA_WHEELS, "NVIDIA CUDA libraries (cuBLAS, CUDA runtime; ~0.4 GB)")
|
||||
|
||||
|
||||
def install_build_tools(gpu, yes):
|
||||
"""The compiler and the CUDA toolkit, installed for the user (asks once). Returns (nvcc, vcvars)."""
|
||||
nvcc, cuda_v = find_nvcc()
|
||||
@@ -545,7 +612,7 @@ def installed_configs():
|
||||
|
||||
|
||||
def start(cfg_path: Path, port: int | None, open_browser=True) -> int:
|
||||
cfg = json.loads(cfg_path.read_text(encoding="utf-8"))
|
||||
cfg = json.loads(cfg_path.read_text(encoding="utf-8-sig"))
|
||||
missing = [p for p in [cfg["exe"], *[a for a in cfg["args"] if a.endswith(".gguf")]] if not Path(p).exists()]
|
||||
if missing:
|
||||
fail(f"{cfg_path.name} refers to missing files: {missing[0]}", "run it again with --setup to repair")
|
||||
@@ -577,6 +644,7 @@ def write_run_script(model, cfg_path, port):
|
||||
# ------------------------------------------------------------------------------------------------ main
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--family", choices=list(FAMILIES), help="qwen = Qwen3.8-Flash-Next, swift = Swift 1.5")
|
||||
ap.add_argument("--model", choices=list(MODELS))
|
||||
ap.add_argument("--context", type=int)
|
||||
ap.add_argument("--vision", choices=["yes", "no", "none", "gpu", "cpu"],
|
||||
@@ -598,12 +666,14 @@ def main() -> int:
|
||||
|
||||
# ---- 0. already installed: just start it
|
||||
have = installed_configs()
|
||||
if have and not (a.setup or a.model or a.check or a.no_start):
|
||||
if have and not (a.setup or a.model or a.family or a.check or a.no_start):
|
||||
if not a.build:
|
||||
update_installed_engine(a.prebuilt)
|
||||
if len(have) == 1:
|
||||
return start(have[0], None)
|
||||
say()
|
||||
for i, c in enumerate(have, 1):
|
||||
say(f" {i}) {c.stem[len('strata-'):].upper()}")
|
||||
say(f" {i}) {json.loads(c.read_text(encoding='utf-8-sig')).get('model_name', c.stem)}")
|
||||
say(f" {len(have) + 1}) install another model / change settings")
|
||||
pick = int(ask("Which one?", [str(i) for i in range(1, len(have) + 2)], "1", a.yes))
|
||||
if pick <= len(have):
|
||||
@@ -640,17 +710,33 @@ def main() -> int:
|
||||
|
||||
# ---- 2. the questions
|
||||
step(2, "your choices")
|
||||
names = list(MODELS)
|
||||
fams = list(FAMILIES)
|
||||
if a.family:
|
||||
family = a.family
|
||||
else:
|
||||
for i, f in enumerate(fams, 1):
|
||||
d = FAMILIES[f]
|
||||
say(f" {i}) {d['title']:20s} {d['by']} - {d['about']}")
|
||||
family = fams[int(ask("Which model?", [str(i) for i in range(1, len(fams) + 1)], "1", a.yes)) - 1]
|
||||
fam = FAMILIES[family]
|
||||
ok(f"model: {fam['title']}")
|
||||
if fam.get("license"):
|
||||
say(f" Its license: {fam['license']}")
|
||||
say()
|
||||
names = [m for m in MODELS if family in MODELS[m].get("families", FAMILIES)]
|
||||
if a.model and a.model not in names:
|
||||
fail(f"{fam['title']} has no {a.model} model file", "choose one of: " + ", ".join(names))
|
||||
for i, m in enumerate(names, 1):
|
||||
d = MODELS[m]
|
||||
fit = "" if ram >= d["ram_gb"] else f" <- needs {d['ram_gb']} GB RAM, you have {ram:.0f}"
|
||||
say(f" {i}) {m:8s} {d['about']}; download {d['download_gb']:.0f} GB, uses ~{d['arena_gb']:.0f} GB of RAM{fit}")
|
||||
rec = "3" if ram >= 60 else "1"
|
||||
model = a.model or names[int(ask("Which model?", ["1", "2", "3"], rec, a.yes)) - 1]
|
||||
rec = str(names.index("IQ3_XXS") + 1) if ram >= 60 else "1"
|
||||
model = a.model or names[int(ask("Which size?", [str(i) for i in range(1, len(names) + 1)], rec, a.yes)) - 1]
|
||||
if ram < MODELS[model]["ram_gb"] - 4:
|
||||
fail(f"{model} needs about {MODELS[model]['ram_gb']} GB of RAM; this PC has {ram:.0f} GB",
|
||||
"choose Q2_0 or IQ2_XS, or add RAM")
|
||||
ok(f"model: {model}")
|
||||
ok(f"size: {model}")
|
||||
tag = fam["tag"] + model # names of the pack, config and start script
|
||||
rec_ctx = 32768 if gpu["vram_gb"] < 14 else 65536 if gpu["vram_gb"] < 20 else 131072
|
||||
if a.context:
|
||||
ctx = a.context
|
||||
@@ -661,8 +747,9 @@ def main() -> int:
|
||||
for i, c in enumerate(CONTEXTS, 1):
|
||||
say(f" {i}) {c // 1024}K tokens" + (" (recommended for your GPU)" if c == rec_ctx else ""))
|
||||
ctx = CONTEXTS[int(ask("Context?", [str(i) for i in range(1, 6)], str(CONTEXTS.index(rec_ctx) + 1), a.yes)) - 1]
|
||||
if model == "IQ3_XXS" and ram < 90 and ctx > 131072:
|
||||
warn("IQ3_XXS with a 262K context needs more than 64 GB of RAM (43 GB of experts + the context): using 128K")
|
||||
if model in ("IQ3_XXS", "IQ3_S") and ram < 90 and ctx > 131072:
|
||||
warn(f"{model} with a 262K context needs more than 64 GB of RAM ({MODELS[model]['arena_gb']:.0f} GB of experts "
|
||||
"+ the context): using 128K")
|
||||
ctx = 131072
|
||||
ok(f"context: {ctx} tokens")
|
||||
if a.vision:
|
||||
@@ -673,17 +760,17 @@ def main() -> int:
|
||||
say(" download and keeps ~1.4 GB of VRAM free for the image encoder, so text is a few % slower.")
|
||||
vision = "gpu" if ask("Do you want images?", ["y", "n"], "n", a.yes) == "y" else "none"
|
||||
ok("images: " + {"none": "off", "gpu": "on", "cpu": "on (encoder on the CPU)"}[vision])
|
||||
models_dir = Path(a.gguf_dir) if a.gguf_dir else Path(a.models_dir) / model
|
||||
shards = [models_dir / f"Qwen3.8-Flash-Next-GSQ-RCO-{model}-0000{i}-of-00002.gguf" for i in (1, 2)]
|
||||
models_dir = Path(a.gguf_dir) if a.gguf_dir else Path(a.models_dir) / tag
|
||||
shards = [models_dir / fam["file"].format(q=model, i=i) for i in (1, 2)]
|
||||
have_model = all(s.exists() and (done(s) or a.gguf_dir) for s in shards)
|
||||
need = (0 if a.gguf_dir or have_model else MODELS[model]["download_gb"]) + 8 + \
|
||||
(40 if model == "Q2_0" and avx512 else 0) + (1 if vision != "none" else 0)
|
||||
(40 if model == "Q2_0" and avx512 and family == "qwen" else 0) + (1 if vision != "none" else 0)
|
||||
if free_gb(models_dir) < need:
|
||||
fail(f"not enough free disk space in {models_dir}: need ~{need:.0f} GB", "use --models-dir on a bigger drive")
|
||||
|
||||
# ---- 3. python packages
|
||||
step(3, "Python packages")
|
||||
pip_install(PY_PACKAGES, "numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja")
|
||||
pip_install(PY_PACKAGES, "numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja, pillow")
|
||||
|
||||
# ---- 4. the engine
|
||||
step(4, "the Strata engine")
|
||||
@@ -702,15 +789,15 @@ def main() -> int:
|
||||
ok(f"engine: {eng / EXE}")
|
||||
|
||||
# ---- 5. the model files
|
||||
step(5, f"downloading {model}")
|
||||
step(5, f"downloading {fam['title']} {model}")
|
||||
if not a.gguf_dir:
|
||||
for s in shards:
|
||||
if s.exists() and done(s):
|
||||
ok(f"{s.name} already downloaded")
|
||||
continue
|
||||
# shard 2 is the same file for all three models: reuse one that is already here
|
||||
# the original's shard 2 is the same file for all three sizes: reuse one that is already here
|
||||
other = [p for p in Path(a.models_dir).glob("*/Qwen3.8-Flash-Next-GSQ-RCO-*-00002-of-00002.gguf") if done(p)]
|
||||
if s.name.endswith("00002-of-00002.gguf") and other and not s.exists():
|
||||
if family == "qwen" and s.name.endswith("00002-of-00002.gguf") and other and not s.exists():
|
||||
try:
|
||||
os.link(other[0], s)
|
||||
mark(s)
|
||||
@@ -718,24 +805,24 @@ def main() -> int:
|
||||
continue
|
||||
except OSError:
|
||||
pass
|
||||
download(HF + f"{model}/{s.name}", s)
|
||||
download(fam["hf"].format(q=model) + s.name, s)
|
||||
for s in shards:
|
||||
if not s.exists():
|
||||
fail(f"missing {s}")
|
||||
ok("model files present")
|
||||
mmproj = Path(a.models_dir) / MMPROJ
|
||||
mmproj = Path(a.models_dir) / fam["mmproj"]
|
||||
if vision != "none":
|
||||
if not mmproj.exists() and a.gguf_dir and (Path(a.gguf_dir) / MMPROJ).exists():
|
||||
mmproj = Path(a.gguf_dir) / MMPROJ
|
||||
if not mmproj.exists() and a.gguf_dir and (Path(a.gguf_dir) / fam["mmproj"]).exists():
|
||||
mmproj = Path(a.gguf_dir) / fam["mmproj"]
|
||||
else:
|
||||
download(HF + MMPROJ, mmproj, "vision encoder")
|
||||
download(fam["mmproj_hf"] + fam["mmproj"], mmproj, "vision encoder")
|
||||
ok(f"vision encoder: {mmproj}")
|
||||
|
||||
# ---- 6. the pack and the MTP draft layer
|
||||
step(6, "preparing the model for Strata")
|
||||
pack = ROOT / "packs" / model.lower()
|
||||
pack = ROOT / "packs" / tag.lower()
|
||||
env = dict(os.environ, STRATA_GGUF_PY=str(llama / "gguf-py"))
|
||||
if model == "Q2_0" and avx512:
|
||||
if model == "Q2_0" and avx512 and family == "qwen":
|
||||
# the Q2_0 experts repacked for the AVX-512 kernel (the measured speed): a one-time ~40 GB conversion
|
||||
if not (pack / "index.txt").exists() or not (pack / "experts.bin").exists(): # index.txt is written last
|
||||
say(" Converting the Q2_0 experts for the AVX-512 kernel (one time, ~40 GB written, 2-5 min) ...")
|
||||
@@ -765,7 +852,12 @@ def main() -> int:
|
||||
|
||||
# ---- 7. the start script
|
||||
step(7, "writing the start script")
|
||||
args = ["--pack", str(pack), "--native", str(shards[0]), "--ple-gguf", str(shards[1]),
|
||||
sys.path.insert(0, str(ROOT / "tools"))
|
||||
from gguf_reader import GGUFFile # the PLE table's shard: shard 2 (original) or 1 (Swift)
|
||||
ple = next((s for s in shards if any(t.name == "per_layer_token_embd.weight" for t in GGUFFile(s).tensors)), None)
|
||||
if ple is None:
|
||||
fail("the model has no per_layer_token_embd tensor (is this a Qwen3.8-Flash-Next GGUF?)")
|
||||
args = ["--pack", str(pack), "--native", str(shards[0]), "--ple-gguf", str(ple),
|
||||
"--expert-profile", str(ROOT / "data" / "expert-profile.bin"), "--expert-cache", "auto",
|
||||
"--prefill", "2048", "--spec", "4", "--spec-min-p", "0.5", "--mtp", str(rt),
|
||||
"--max-context", str(ctx)]
|
||||
@@ -774,16 +866,16 @@ def main() -> int:
|
||||
if vision != "none":
|
||||
args += ["--vision", "--vram-reserve-mib", str(VISION[vision]["reserve_mib"])]
|
||||
cfg = {"exe": str(eng / EXE), "args": args, "cwd": str(ROOT), "tokenizer": str(pack / "tokenizer"),
|
||||
"model_name": f"qwen3.8-flash-next-{model.lower()}", "log": str(ROOT / f"strata-{model.lower()}.log"),
|
||||
"model_name": f"{fam['name']}-{model.lower()}", "log": str(ROOT / f"strata-{tag.lower()}.log"),
|
||||
"lib_dirs": lib_dirs, "port": a.port}
|
||||
if vision != "none":
|
||||
cfg["vision"] = {"exe": str(eng / VEXE), "mmproj": str(mmproj), "model": str(shards[0]),
|
||||
"gpu": vision == "gpu", "max_tokens": VISION[vision]["max_tokens"]}
|
||||
if vision == "cpu":
|
||||
cfg["vision"]["threads"] = max(1, (os.cpu_count() or 8) // 2)
|
||||
cfg_path = ROOT / f"strata-{model.lower()}.json"
|
||||
cfg_path = ROOT / f"strata-{tag.lower()}.json"
|
||||
cfg_path.write_text(json.dumps(cfg, indent=1), encoding="utf-8")
|
||||
script = write_run_script(model, cfg_path, a.port)
|
||||
script = write_run_script(tag, cfg_path, a.port)
|
||||
ok(f"start script: {script.name}")
|
||||
|
||||
say()
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <fstream>
|
||||
#include <thread>
|
||||
@@ -263,8 +264,14 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
|
||||
if (native && d.nact_multi.size() < (size_t) MAXT * kNativeActBytes) d.nact_multi.resize((size_t) MAXT * kNativeActBytes);
|
||||
if (d.job_of.size() != (size_t) d.n_expert) d.job_of.assign((size_t) d.n_expert, (int16_t) -1);
|
||||
if (d.jobs_multi.size() < (size_t) (n_tok * k)) d.jobs_multi.resize((size_t) (MAXT * k));
|
||||
static const bool ptrace = std::getenv("STRATA_POOL_TRACE") != nullptr;
|
||||
auto pt = [&](const char* what, long long a = -1) {
|
||||
if (ptrace) { std::fprintf(stderr, "pool trace: layer %lld %s %lld\n", (long long) d.layers, what, a); std::fflush(stderr); }
|
||||
};
|
||||
const auto c0 = std::chrono::steady_clock::now();
|
||||
pt("begin");
|
||||
d.src->begin_layer(d.layers, ids, n_tok * k);
|
||||
pt("begun");
|
||||
if (!d.usage.empty())
|
||||
for (int64_t i = 0; i < n_tok * k; ++i)
|
||||
if (ids[i] >= 0 && ids[i] < d.n_expert) d.usage[(size_t) d.layers * (size_t) d.n_expert + (size_t) ids[i]] += 1.0f;
|
||||
@@ -348,7 +355,9 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
|
||||
P.counts[1] = entries;
|
||||
P.counts[2] = fetches;
|
||||
std::atomic_thread_fence(std::memory_order_seq_cst);
|
||||
pt("publish", fetches);
|
||||
if (P.publish) P.publish(P.ctx);
|
||||
pt("fetch", fetches);
|
||||
if (P.fetch) P.fetch(P.ctx, dma_src, P.pcie_mode != 0 ? 0 : fetches, (size_t) bb); // the copy engine, beside the CPU's work
|
||||
} else {
|
||||
for (int64_t i = 0; i < n; ++i) {
|
||||
@@ -409,9 +418,11 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
|
||||
++d.multi_entries;
|
||||
}
|
||||
const auto c3 = std::chrono::steady_clock::now();
|
||||
pt("run", njobs);
|
||||
if (native) d.pool->run_split_multi_native(lay.fmt[(size_t) d.layers], d.jobs_multi.data(), njobs);
|
||||
else d.pool->run_split_multi(d.jobs_multi.data(), njobs);
|
||||
const auto c4 = std::chrono::steady_clock::now();
|
||||
pt("ran");
|
||||
auto ms = [](auto a, auto b) { return std::chrono::duration<double, std::milli>(b - a).count(); };
|
||||
d.ms_plan += ms(c0, c1);
|
||||
d.ms_actq += ms(c1, c2);
|
||||
@@ -545,13 +556,28 @@ LoadStats load_experts_gguf(const std::string& gguf, uint8_t* dst, const strata:
|
||||
const auto t0 = std::chrono::steady_clock::now();
|
||||
std::atomic<int64_t> next{0};
|
||||
std::atomic<bool> bad{false};
|
||||
// a layer's experts may sit in another shard of the model (native_experts.txt v3): a name beside `gguf`
|
||||
const size_t cut = gguf.find_last_of("/\\");
|
||||
const std::string dir = cut == std::string::npos ? std::string() : gguf.substr(0, cut + 1);
|
||||
auto file_of = [&](int64_t l) -> std::string {
|
||||
if (lay.gguf_file.empty() || lay.gguf_file[(size_t) l].empty()) return gguf;
|
||||
return dir + lay.gguf_file[(size_t) l];
|
||||
};
|
||||
auto worker = [&]() {
|
||||
std::ifstream f(gguf, std::ios::binary);
|
||||
if (!f) { bad = true; return; }
|
||||
std::ifstream f;
|
||||
std::string open_name;
|
||||
std::vector<uint8_t> buf;
|
||||
for (;;) {
|
||||
const int64_t l = next.fetch_add(1);
|
||||
if (l >= lay.n_layers || bad) break;
|
||||
const std::string name = file_of(l);
|
||||
if (name != open_name) {
|
||||
f.close();
|
||||
f.clear();
|
||||
f.open(name, std::ios::binary);
|
||||
if (!f) { bad = true; return; }
|
||||
open_name = name;
|
||||
}
|
||||
const auto& fm = lay.fmt[(size_t) l];
|
||||
const uint64_t blob = lay.bytes[(size_t) l];
|
||||
const uint64_t per[3] = {fm.up_off, fm.up_off, blob - fm.down_off};
|
||||
|
||||
@@ -113,6 +113,11 @@ bool expert_layout_load(const std::string& pack_dir, int64_t n_layers, int64_t n
|
||||
L.gguf_off[(size_t) (3 * l)] = go;
|
||||
L.gguf_off[(size_t) (3 * l + 1)] = uo;
|
||||
L.gguf_off[(size_t) (3 * l + 2)] = dox;
|
||||
std::string file; // v3: the shard that holds this layer (a file name beside --native)
|
||||
if (ss >> file) {
|
||||
if (L.gguf_file.empty()) L.gguf_file.assign((size_t) n_layers, std::string());
|
||||
L.gguf_file[(size_t) l] = file;
|
||||
}
|
||||
}
|
||||
L.fmt[(size_t) l] = f;
|
||||
L.offset[(size_t) l] = off;
|
||||
|
||||
@@ -49,11 +49,12 @@ std::vector<int> physical_cores(bool skip_first) {
|
||||
#else
|
||||
cpu_set_t set;
|
||||
CPU_ZERO(&set);
|
||||
if (sched_getaffinity(0, sizeof set, &set) == 0)
|
||||
if (sched_getaffinity(0, sizeof set, &set) == 0) {
|
||||
for (int i = 0; i < CPU_SETSIZE; ++i)
|
||||
if (CPU_ISSET(i, &set)) cores.push_back(i);
|
||||
else
|
||||
} else {
|
||||
for (unsigned i = 0; i < std::thread::hardware_concurrency(); ++i) cores.push_back((int) i);
|
||||
}
|
||||
#endif
|
||||
if (skip_first && !cores.empty()) cores.erase(cores.begin());
|
||||
return cores;
|
||||
@@ -127,10 +128,23 @@ ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works) : host_works_(h
|
||||
ExpertPool::~ExpertPool() {
|
||||
stop_.store(true, std::memory_order_release);
|
||||
// Bump the epoch so a PARKED worker notices the stop flag rather than sleeping through it.
|
||||
epoch_.fetch_add(1, std::memory_order_release);
|
||||
publish();
|
||||
for (auto& t : threads_) t.join();
|
||||
}
|
||||
|
||||
void ExpertPool::publish() {
|
||||
// Both sides are seq_cst, and that is the whole lost-wakeup argument: a worker going to sleep does
|
||||
// `sleepers_++` and then reads `epoch_`, the host does `epoch_++` and then reads `sleepers_`. In one total
|
||||
// order at least one of them sees the other's write - the worker sees the new epoch and does not sleep, or
|
||||
// the host sees the sleeper and notifies under the mutex the worker holds until it is inside `wait`.
|
||||
// On x86 the fetch_add is a locked xadd either way, so this costs the token path nothing.
|
||||
epoch_.fetch_add(1, std::memory_order_seq_cst);
|
||||
if (sleepers_.load(std::memory_order_seq_cst) != 0) {
|
||||
std::lock_guard<std::mutex> lk(sleep_mu_);
|
||||
sleep_cv_.notify_all();
|
||||
}
|
||||
}
|
||||
|
||||
void ExpertPool::worker(int id) {
|
||||
uint32_t seen = 0;
|
||||
// ARRIVE at the park before the first wait, so `parked_ == n_` is true from construction. Counting only
|
||||
@@ -147,9 +161,22 @@ void ExpertPool::worker(int id) {
|
||||
// - a locked read-modify-write, five workers against one cache line - so the workers spent their wait
|
||||
// invalidating each other's caches and the very line the host writes to publish work. The counter was
|
||||
// diagnostic and nothing branched on it. See the note on the atomics in pool.hpp.
|
||||
//
|
||||
// After `kSpinBeforeSleep` with no work the worker sleeps instead (issue #4). The clock is read once
|
||||
// every 1024 pauses, so the spin itself is unchanged.
|
||||
const auto parked_at = std::chrono::steady_clock::now();
|
||||
uint32_t spins = 0;
|
||||
while (epoch_.load(std::memory_order_acquire) == seen) {
|
||||
if (stop_.load(std::memory_order_relaxed)) return;
|
||||
_mm_pause();
|
||||
if ((++spins & 1023u) != 0) continue;
|
||||
if (std::chrono::steady_clock::now() - parked_at < kSpinBeforeSleep) continue;
|
||||
std::unique_lock<std::mutex> lk(sleep_mu_);
|
||||
sleepers_.fetch_add(1, std::memory_order_seq_cst);
|
||||
sleep_cv_.wait(lk, [&] {
|
||||
return epoch_.load(std::memory_order_seq_cst) != seen || stop_.load(std::memory_order_relaxed);
|
||||
});
|
||||
sleepers_.fetch_sub(1, std::memory_order_relaxed);
|
||||
}
|
||||
if (stop_.load(std::memory_order_acquire)) return;
|
||||
seen = epoch_.load(std::memory_order_relaxed);
|
||||
@@ -245,7 +272,7 @@ void ExpertPool::run_phase(int mode, int n_tasks) {
|
||||
njobs_ = n_tasks;
|
||||
head_.store(0, std::memory_order_relaxed);
|
||||
done_.store(0, std::memory_order_relaxed);
|
||||
epoch_.fetch_add(1, std::memory_order_release);
|
||||
publish();
|
||||
if (host_works_) drain(-1, host_scratch_);
|
||||
while (done_.load(std::memory_order_acquire) != (uint32_t) n_tasks) _mm_pause();
|
||||
while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause();
|
||||
@@ -356,7 +383,7 @@ void ExpertPool::run(ExpertJob* jobs, int n) {
|
||||
mode_ = 0;
|
||||
head_.store(0, std::memory_order_relaxed);
|
||||
done_.store(0, std::memory_order_relaxed);
|
||||
epoch_.fetch_add(1, std::memory_order_release); // release: jobs_/njobs_ are visible before the bump
|
||||
publish(); // a release (seq_cst): jobs_/njobs_ are visible before the bump
|
||||
|
||||
// ---- **THE HOST DRAINS TOO (R2.2), INSTEAD OF SPINNING ON `done_`.**
|
||||
//
|
||||
|
||||
@@ -268,6 +268,28 @@ __device__ __forceinline__ float vec_dot_iq4_nl_q8_1(const void* __restrict__ vb
|
||||
return d * sumi;
|
||||
}
|
||||
|
||||
// IQ4_XS: 256 values as 8 sub-blocks of 32 (6-bit scale each); one call covers one sub-block (iqs = 4 * sub-block),
|
||||
// and `bq8_1` is the super-block's first q8_1 block, so the call's activation is bq8_1[iqs / 4]. The GSQ-RCO IQ3_S
|
||||
// file keeps one layer's routed gate/up experts in this format.
|
||||
__device__ __forceinline__ float vec_dot_iq4_xs_q8_1(const void* __restrict__ vbq, const block_q8_1* __restrict__ bq8_1,
|
||||
const int& kbx, const int& iqs) {
|
||||
const block_iq4_xs* bq4 = (const block_iq4_xs*) vbq + kbx;
|
||||
int sumi = 0;
|
||||
#pragma unroll
|
||||
for (int j = 0; j < 4; ++j) {
|
||||
const int aux_q4 = get_int_b4(bq4->qs, iqs + j);
|
||||
const int2 v = get_int_from_table_16(aux_q4, kvalues_iq4nl);
|
||||
const int u0 = get_int_b4(bq8_1[iqs / 4].qs, j + 0);
|
||||
const int u1 = get_int_b4(bq8_1[iqs / 4].qs, j + 4);
|
||||
sumi = ggml_cuda_dp4a(v.x, u0, sumi);
|
||||
sumi = ggml_cuda_dp4a(v.y, u1, sumi);
|
||||
}
|
||||
const int ls = ((bq4->scales_l[iqs / 8] >> (iqs & 0x04)) & 0x0F) | (((bq4->scales_h >> (iqs / 2)) & 0x03) << 4);
|
||||
sumi *= ls - 32;
|
||||
const float d = __half2float(bq4->d) * __low2float(bq8_1[iqs / 4].ds);
|
||||
return d * sumi;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- the formats
|
||||
// qk = values per block, ipb = dot calls per block (qi / vdr), step = the iqs stride between calls.
|
||||
template<int TY> struct Fmt;
|
||||
@@ -281,6 +303,8 @@ template<> struct Fmt<20> { static constexpr int qk = 32, ipb = 2, step = 2;
|
||||
__device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq4_nl_q8_1(v, y, kbx, iqs); } };
|
||||
template<> struct Fmt<21> { static constexpr int qk = 256, ipb = 8, step = 2;
|
||||
__device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq3_s_q8_1(v, y, kbx, iqs); } };
|
||||
template<> struct Fmt<23> { static constexpr int qk = 256, ipb = 8, step = 4;
|
||||
__device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq4_xs_q8_1(v, y, kbx, iqs); } };
|
||||
template<> struct Fmt<22> { static constexpr int qk = 256, ipb = 8, step = 2;
|
||||
__device__ static float dot(const void* v, const block_q8_1* y, int kbx, int iqs) { return vec_dot_iq2_s_q8_1(v, y, kbx, iqs); } };
|
||||
template<> struct Fmt<29> { static constexpr int qk = 256, ipb = 8, step = 1;
|
||||
@@ -618,6 +642,7 @@ void iq_mmvq(int t, const void* w, const void* x_q8_1, float* y, int n_in, int n
|
||||
case 20: mmvq_kernel<20><<<grid, block, 0, s>>>(W, rb, X, y, n_in, n_out, ncols); break;
|
||||
case 21: mmvq_kernel<21><<<grid, block, 0, s>>>(W, rb, X, y, n_in, n_out, ncols); break;
|
||||
case 22: mmvq_kernel<22><<<grid, block, 0, s>>>(W, rb, X, y, n_in, n_out, ncols); break;
|
||||
case 23: mmvq_kernel<23><<<grid, block, 0, s>>>(W, rb, X, y, n_in, n_out, ncols); break;
|
||||
case 29: mmvq_kernel<29><<<grid, block, 0, s>>>(W, rb, X, y, n_in, n_out, ncols); break;
|
||||
case 42: mmvq_kernel<42><<<grid, block, 0, s>>>(W, rb, X, y, n_in, n_out, ncols); break;
|
||||
default: std::fprintf(stderr, "iq_mmvq: type %d is not supported\n", t); std::exit(1);
|
||||
@@ -700,6 +725,7 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long
|
||||
case 18: native_gu_kernel<18><<<ggu, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break;
|
||||
case 21: native_gu_kernel<21><<<ggu, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break;
|
||||
case 22: native_gu_kernel<22><<<ggu, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break;
|
||||
case 23: native_gu_kernel<23><<<ggu, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break;
|
||||
case 29: native_gu_kernel<29><<<ggu, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break;
|
||||
case 42: native_gu_kernel<42><<<ggu, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_tok, X, L, gate, up); break;
|
||||
default: std::fprintf(stderr, "native_expert_grouped: gate/up type %d\n", L.gu_type); std::exit(1);
|
||||
@@ -711,6 +737,7 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long
|
||||
const dim3 gd((unsigned) ((L.n_embd + 7) / 8), (unsigned) cap_groups);
|
||||
switch (L.d_type) {
|
||||
case 20: native_down_kernel<20><<<gd, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_dst, hq, L, out); break;
|
||||
case 23: native_down_kernel<23><<<gd, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_dst, hq, L, out); break;
|
||||
case 42: native_down_kernel<42><<<gd, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_dst, hq, L, out); break;
|
||||
default: std::fprintf(stderr, "native_expert_grouped: down type %d\n", L.d_type); std::exit(1);
|
||||
}
|
||||
|
||||
@@ -244,12 +244,14 @@ __global__ void ident_hits_kernel(const int32_t* __restrict__ ids, int n, int32_
|
||||
if (i == 0) *count = n;
|
||||
}
|
||||
|
||||
__global__ void gather_rows_kernel(const uint4* __restrict__ src, long long row16, const int32_t* __restrict__ ids,
|
||||
long long n, uint4* __restrict__ dst) {
|
||||
const long long total = n * row16;
|
||||
// E = the widest element the row size divides into (16, 4 or 1 bytes): a Q6_K head row of 2560 values is 2100 bytes
|
||||
template<typename E>
|
||||
__global__ void gather_rows_kernel(const E* __restrict__ src, long long row_e, const int32_t* __restrict__ ids,
|
||||
long long n, E* __restrict__ dst) {
|
||||
const long long total = n * row_e;
|
||||
for (long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x; i < total; i += (long long) gridDim.x * blockDim.x) {
|
||||
const long long r = i / row16, o = i - r * row16;
|
||||
dst[i] = src[(long long) ids[r] * row16 + o];
|
||||
const long long r = i / row_e, o = i - r * row_e;
|
||||
dst[i] = src[(long long) ids[r] * row_e + o];
|
||||
}
|
||||
}
|
||||
|
||||
@@ -334,8 +336,13 @@ void mtp_select(const float* R_src, int64_t R_stride, const int32_t* ids, const
|
||||
}
|
||||
|
||||
void gather_rows(const uint8_t* src, int64_t row_bytes, const int32_t* ids, int64_t n, uint8_t* dst, void* stream) {
|
||||
if (row_bytes % 16 != 0) { std::fprintf(stderr, "gather_rows: row size must be a multiple of 16\n"); std::exit(1); }
|
||||
gather_rows_kernel<<<48 * 8, 256, 0, (cudaStream_t) stream>>>((const uint4*) src, row_bytes / 16, ids, n, (uint4*) dst);
|
||||
cudaStream_t s = (cudaStream_t) stream;
|
||||
if (row_bytes % 16 == 0)
|
||||
gather_rows_kernel<<<48 * 8, 256, 0, s>>>((const uint4*) src, row_bytes / 16, ids, n, (uint4*) dst);
|
||||
else if (row_bytes % 4 == 0)
|
||||
gather_rows_kernel<<<48 * 8, 256, 0, s>>>((const uint32_t*) src, row_bytes / 4, ids, n, (uint32_t*) dst);
|
||||
else
|
||||
gather_rows_kernel<<<48 * 8, 256, 0, s>>>(src, row_bytes, ids, n, dst);
|
||||
check("gather_rows");
|
||||
}
|
||||
|
||||
|
||||
@@ -164,15 +164,19 @@ bool PleTable::open(const std::string& gguf_path, std::string& err, const PleIoO
|
||||
// row. `GgufFile` parses the header, so `tensor_data` is already correct; this asserts the tensor
|
||||
// exactly fills the file from there, which is what makes the whole arrangement checkable rather than
|
||||
// assumed. A wrong data offset would leave a different remainder.
|
||||
// A shard may hold other tensors too (Swift 1.5's shard 1 holds layers 0-12 and the table): the table must
|
||||
// then fit inside the file at its own offset; alone in its shard (the original's shard 2) it fills it exactly.
|
||||
const uint64_t need = impl_->n_rows * (uint64_t) PLE_ROW_BYTES;
|
||||
const uint64_t have = impl_->file->file_size() - impl_->file->data_start();
|
||||
if (need != have) {
|
||||
const bool alone = impl_->file->tensors().size() == 1;
|
||||
if (alone ? need != have : t->offset + need > have) {
|
||||
char buf[256];
|
||||
std::snprintf(buf, sizeof buf,
|
||||
"PLE table size mismatch: %llu rows x %d B = %llu, but the file holds %llu from "
|
||||
"PLE table size mismatch: %llu rows x %d B = %llu at offset %llu, but the file holds %llu from "
|
||||
"data_start %llu",
|
||||
(unsigned long long) impl_->n_rows, PLE_ROW_BYTES, (unsigned long long) need,
|
||||
(unsigned long long) have, (unsigned long long) impl_->file->data_start());
|
||||
(unsigned long long) t->offset, (unsigned long long) have,
|
||||
(unsigned long long) impl_->file->data_start());
|
||||
err = buf;
|
||||
close();
|
||||
return false;
|
||||
|
||||
@@ -261,6 +261,8 @@ bool Prefill::run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& e
|
||||
int32_t prev[2] = {ss.ple_prev[0], ss.ple_prev[1]};
|
||||
|
||||
for (int64_t c0 = 0; c0 < n; c0 += m.T) {
|
||||
if (should_stop && should_stop()) { err = "cancelled"; return false; }
|
||||
if (std::getenv("STRATA_TRACE")) { std::fprintf(stderr, "strata trace: prompt chunk %lld of %lld\n", (long long) c0, (long long) n); std::fflush(stderr); }
|
||||
const int64_t T = std::min(m.T, n - c0), p0 = pos0 + c0;
|
||||
++stats_.chunks;
|
||||
// ---- embeddings, broadcast to the four streams
|
||||
|
||||
+558
-41
@@ -48,6 +48,10 @@
|
||||
#include <algorithm>
|
||||
#include <iostream>
|
||||
#include <thread>
|
||||
#include <atomic>
|
||||
#include <condition_variable>
|
||||
#include <deque>
|
||||
#include <mutex>
|
||||
#include <charconv>
|
||||
#include <cmath>
|
||||
#include <cstdio>
|
||||
@@ -231,6 +235,19 @@ struct Options {
|
||||
/// The vision path: keep a per-cell (t, h, w) rotary position table so --serve can take GENI requests.
|
||||
bool vision = false;
|
||||
int adapt_swaps = 96;
|
||||
/// --serve: how many conversation checkpoints to keep between requests (0 = every request reads its whole
|
||||
/// prompt again, the v0.1.2 behaviour). One is the GDN recurrence of the 36 layers, the QSA indexer tails and
|
||||
/// the PLE history (~118 MB of host RAM); the KV cache itself is positional and stays where it is.
|
||||
int prompt_cache = 6;
|
||||
/// --serve: also keep a checkpoint every N freshly read prompt tokens (0 = only at the last turn boundary)
|
||||
int64_t prompt_cache_every = 16384;
|
||||
/// --serve: the token that opens a chat turn (<|im_start|>). The last one in a prompt is where the chat's
|
||||
/// history ends and the new assistant turn begins, which is the checkpoint the next request can reuse.
|
||||
int64_t turn_token = 248045;
|
||||
/// --serve: a text part of the prompt of at most N tokens (a chat message, the assistant header, a short tool
|
||||
/// result) goes through the verify windows, S tokens at a time, instead of the batched prompt path (0 = always
|
||||
/// the batched path)
|
||||
int64_t short_read = 64;
|
||||
};
|
||||
|
||||
void usage() {
|
||||
@@ -288,6 +305,12 @@ void usage() {
|
||||
" --shared-late A/B: shared expert after the CPU pool (default: overlapped with it)\n"
|
||||
" --keep-canonical A/B: also load canonical copies of natively served tensors (more VRAM)\n"
|
||||
" --vision --serve takes images too (GENI requests; embeddings from strata-vision)\n"
|
||||
" --prompt-cache N --serve: keep N conversation checkpoints between requests (default 6, ~118 MB\n"
|
||||
" of RAM each; 0 = read every prompt from the start)\n"
|
||||
" --prompt-cache-every N --serve: also checkpoint every N fresh prompt tokens (default 16384, 0 = off)\n"
|
||||
" --turn-token ID --serve: the token that opens a chat turn (default 248045, <|im_start|>)\n"
|
||||
" --short-read N --serve: read at most N fresh text tokens through the decode windows instead\n"
|
||||
" of the batched prompt path (default 64, 0 = off)\n"
|
||||
" --no-token-graph A/B: two graphs per layer (the host launches each) instead of one per token\n"
|
||||
" --no-fused-gr A/B: the six-kernel hyper-connection read and a separate write (native)\n"
|
||||
" --prefill CHUNK batched prompt processing in chunks of CHUNK tokens (needs --native)\n"
|
||||
@@ -333,6 +356,23 @@ void usage() {
|
||||
" 34 GB, and measured 71.97 vs 34.78 ms/token cold vs warm.\n");
|
||||
}
|
||||
|
||||
/// All shards of a split GGUF, from shard 1's path ("...-00001-of-00002.gguf"); just the path when it is not split.
|
||||
std::vector<std::string> model_shards(const std::string& first) {
|
||||
const std::string tag = "-00001-of-";
|
||||
const size_t at = first.rfind(tag);
|
||||
if (at == std::string::npos || first.size() < at + tag.size() + 10) return {first};
|
||||
const int total = std::atoi(first.substr(at + tag.size(), 5).c_str());
|
||||
std::vector<std::string> out;
|
||||
for (int i = 1; i <= total && i <= 99; ++i) {
|
||||
char num[8];
|
||||
std::snprintf(num, sizeof num, "%05d", i);
|
||||
std::string p = first;
|
||||
p.replace(at + 1, 5, num);
|
||||
if (std::ifstream(p, std::ios::binary)) out.push_back(p);
|
||||
}
|
||||
return out.empty() ? std::vector<std::string>{first} : out;
|
||||
}
|
||||
|
||||
bool parse_i64_list(const char* s, std::vector<int64_t>& out, std::string& err) {
|
||||
out.clear();
|
||||
std::string text(s);
|
||||
@@ -405,6 +445,15 @@ void drive_pool_multi(void* user, const float* x_f, const int32_t* ids, int64_t
|
||||
++t->calls;
|
||||
}
|
||||
|
||||
/// STRATA_TRACE=1: the VRAM left at a step of the startup (finds what fills the card after the cache is sized)
|
||||
void mem_mark(const char* where) {
|
||||
static const bool on = std::getenv("STRATA_TRACE") != nullptr;
|
||||
if (!on) return;
|
||||
size_t free_b = 0, total_b = 0;
|
||||
cudaMemGetInfo(&free_b, &total_b);
|
||||
std::fprintf(stderr, "strata trace: %lld MiB free after %s\n", (long long) (free_b >> 20), where);
|
||||
}
|
||||
|
||||
int argmax(const std::vector<float>& v) {
|
||||
int best = 0;
|
||||
for (size_t i = 1; i < v.size(); ++i)
|
||||
@@ -412,6 +461,84 @@ int argmax(const std::vector<float>& v) {
|
||||
return best;
|
||||
}
|
||||
|
||||
// ---- --serve's conversation cache. A chat or an agent sends the whole conversation again with every request, and
|
||||
// reading it again is what made a long session wait minutes for every turn. What a sequence leaves behind splits in
|
||||
// two, and only one half needs copying:
|
||||
// * POSITIONAL state - the KV cache of the 12 QSA layers and their pooled indexer keys, the draft layer's KV. A
|
||||
// cell is written once for its position and read only by later positions (the block scores take `dead` for the
|
||||
// block being filled, never its pooled row), so rewinding to a position just means writing from there again.
|
||||
// * RUNNING state - the 36 GDN recurrences and conv histories, each QSA layer's indexer tail (the unfinished
|
||||
// block's raw keys) and the PLE's normalized history. Each describes "everything so far" and cannot be
|
||||
// rewound, so a checkpoint is a copy of exactly these: ~118 MB, the same set the verifier snapshots to roll
|
||||
// back rejected drafts.
|
||||
// A checkpoint is only valid while the positional cells below it still hold ITS tokens, so the serve loop keeps
|
||||
// just the checkpoints that are prefixes of the tokens the session holds now.
|
||||
struct ImgKey {
|
||||
int64_t start = 0; ///< the image's first <|image_pad|> position
|
||||
uint64_t hash = 0; ///< its embeddings and grid: the pad tokens alone are the same for every picture
|
||||
bool operator==(const ImgKey& o) const { return start == o.start && hash == o.hash; }
|
||||
};
|
||||
|
||||
struct ConvCheckpoint {
|
||||
std::vector<int32_t> ids; ///< the tokens this state has consumed
|
||||
std::vector<ImgKey> imgs; ///< the images among them
|
||||
std::vector<uint8_t> gdn, ple, tails;
|
||||
};
|
||||
|
||||
uint64_t fnv1a(const void* data, size_t n, uint64_t h = 1469598103934665603ull) {
|
||||
const uint8_t* p = (const uint8_t*) data;
|
||||
for (size_t i = 0; i < n; ++i) { h ^= p[i]; h *= 1099511628211ull; }
|
||||
return h;
|
||||
}
|
||||
|
||||
struct ConvStateSizes {
|
||||
size_t gdn = 0, ple = 0, tail = 0;
|
||||
};
|
||||
|
||||
ConvStateSizes conv_state_sizes(const strata::core::ModelGeometry& g) {
|
||||
ConvStateSizes z;
|
||||
z.gdn = (size_t) g.n_gdn_layers() *
|
||||
((size_t) g.ssm_state_size * (size_t) g.ssm_v_heads * (size_t) g.ssm_state_size +
|
||||
(size_t) g.ssm_conv_channels * (size_t) (g.ssm_d_conv - 1)) * sizeof(float);
|
||||
z.ple = (size_t) strata::kernels::NG_HIST * (size_t) strata::kernels::NG_HC_DIM * sizeof(float);
|
||||
z.tail = (size_t) (strata::kernels::qsa_real_shapes().idx_block - 1) * (size_t) g.idx_key_dim * sizeof(float);
|
||||
return z;
|
||||
}
|
||||
|
||||
/// Copies the running state out. The caller has synchronized the device.
|
||||
bool checkpoint_save(ConvCheckpoint& c, const strata::core::SessionState& ss, const strata::core::ModelGeometry& g) {
|
||||
const ConvStateSizes z = conv_state_sizes(g);
|
||||
c.gdn.resize(z.gdn);
|
||||
c.ple.resize(ss.ple_hist != nullptr ? z.ple : 0);
|
||||
c.tails.resize(z.tail * (size_t) g.n_qsa_layers());
|
||||
if (cudaMemcpy(c.gdn.data(), ss.gdn_state, z.gdn, cudaMemcpyDeviceToHost) != cudaSuccess) return false;
|
||||
if (!c.ple.empty() && cudaMemcpy(c.ple.data(), ss.ple_hist, z.ple, cudaMemcpyDeviceToHost) != cudaSuccess)
|
||||
return false;
|
||||
for (int64_t i = 0; i < g.n_qsa_layers(); ++i)
|
||||
if (cudaMemcpy(c.tails.data() + (size_t) i * z.tail, ss.qsa_states[i].idx_tail, z.tail, cudaMemcpyDeviceToHost) !=
|
||||
cudaSuccess)
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/// Puts a checkpoint's running state back; the positional cells below it are the caller's to guarantee.
|
||||
bool checkpoint_restore(const ConvCheckpoint& c, strata::core::SessionState& ss, const strata::core::ModelGeometry& g) {
|
||||
const ConvStateSizes z = conv_state_sizes(g);
|
||||
if (c.gdn.size() != z.gdn || c.tails.size() != z.tail * (size_t) g.n_qsa_layers()) return false;
|
||||
if (cudaMemcpy(ss.gdn_state, c.gdn.data(), z.gdn, cudaMemcpyHostToDevice) != cudaSuccess) return false;
|
||||
if (!c.ple.empty() && cudaMemcpy(ss.ple_hist, c.ple.data(), z.ple, cudaMemcpyHostToDevice) != cudaSuccess)
|
||||
return false;
|
||||
for (int64_t i = 0; i < g.n_qsa_layers(); ++i)
|
||||
if (cudaMemcpy(ss.qsa_states[i].idx_tail, c.tails.data() + (size_t) i * z.tail, z.tail, cudaMemcpyHostToDevice) !=
|
||||
cudaSuccess)
|
||||
return false;
|
||||
// the PLE's token window is the last two tokens, OLDEST FIRST, -1 where there is none (as session_zero leaves it)
|
||||
const size_t L = c.ids.size();
|
||||
ss.ple_prev[0] = L >= 2 ? c.ids[L - 2] : -1;
|
||||
ss.ple_prev[1] = L >= 1 ? c.ids[L - 1] : -1;
|
||||
return cudaDeviceSynchronize() == cudaSuccess;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
int main(int argc, char** argv) {
|
||||
@@ -527,6 +654,10 @@ int main(int argc, char** argv) {
|
||||
else if (a == "--pcie-mode") o.pcie_mode = next("--pcie-mode");
|
||||
else if (a == "--serve") o.serve = true;
|
||||
else if (a == "--vision") o.vision = true;
|
||||
else if (a == "--prompt-cache") o.prompt_cache = std::max(0, std::atoi(next("--prompt-cache")));
|
||||
else if (a == "--prompt-cache-every") o.prompt_cache_every = std::max(0LL, std::atoll(next("--prompt-cache-every")));
|
||||
else if (a == "--turn-token") o.turn_token = std::atoll(next("--turn-token"));
|
||||
else if (a == "--short-read") o.short_read = std::max(0LL, std::atoll(next("--short-read")));
|
||||
else if (a == "--no-spec-split") o.spec_split = false;
|
||||
else if (a == "--eos-ids") {
|
||||
std::string e;
|
||||
@@ -592,7 +723,13 @@ int main(int argc, char** argv) {
|
||||
o.native_ple_key = o.native_moe_combine = o.native_gdn = o.native_router = true;
|
||||
o.native_qsa = o.native_qsa_indexer = o.native_rope = o.native_ple_postops = true;
|
||||
if (o.native_head_gguf.empty()) o.native_head_gguf = o.native_preset;
|
||||
if (o.native_dense_gguf.empty()) o.native_dense_gguf = {o.native_preset, o.ple_gguf};
|
||||
if (o.native_dense_gguf.empty()) {
|
||||
// every shard of the model (<name>-0000N-of-0000M.gguf beside --native), then the PLE shard: a split
|
||||
// may put any layer in any shard (Swift's GGUFs: layers 13-47 in shard 2, the PLE table in shard 1)
|
||||
o.native_dense_gguf = model_shards(o.native_preset);
|
||||
if (std::find(o.native_dense_gguf.begin(), o.native_dense_gguf.end(), o.ple_gguf) == o.native_dense_gguf.end())
|
||||
o.native_dense_gguf.push_back(o.ple_gguf);
|
||||
}
|
||||
// Plan v0.3 (24 Sep): the CPU experts stay on the VNNI kernel. The llama.cpp-CPU-exact q8_0 contract
|
||||
// cost 27.0 vs 17.2 ms/token of pool time and G-C does not need it; `--cpu-oracle-q8-0` still selects it.
|
||||
}
|
||||
@@ -980,6 +1117,7 @@ int main(int argc, char** argv) {
|
||||
// ---- R4's slot storage. Allocated AFTER the weights and the session, so `cudaMemGetInfo` inside `open`
|
||||
// sees the memory this process actually has left rather than the card's idle figure - and refuses with both
|
||||
// numbers if the slots do not fit, instead of handing back a cache smaller than it was asked for.
|
||||
mem_mark("the weights, the session and the drafter");
|
||||
strata::core::ExpertCache xcache;
|
||||
std::vector<std::pair<int32_t, int32_t>> profile;
|
||||
if (!o.expert_profile.empty()) {
|
||||
@@ -995,6 +1133,7 @@ int main(int argc, char** argv) {
|
||||
std::fprintf(stderr, "strata generate: profile %s: %zu ranked pairs, built for %lld slots\n",
|
||||
o.expert_profile.c_str(), profile.size(), (long long) pslots);
|
||||
}
|
||||
const bool auto_cache = o.expert_cache < 0;
|
||||
if (o.expert_cache < 0) {
|
||||
size_t free_b = 0, total_b = 0;
|
||||
cudaMemGetInfo(&free_b, &total_b);
|
||||
@@ -1030,15 +1169,51 @@ int main(int argc, char** argv) {
|
||||
o.expert_cache = (int) sized_slots.size();
|
||||
}
|
||||
if (o.expert_cache > 0) {
|
||||
const bool ok = sized_slots.empty()
|
||||
? xcache.open(o.expert_cache, g.n_layers, g.n_expert, (int64_t) strata::kernels::cpu::expert_layout().max_blob, err)
|
||||
: xcache.open_sized(sized_slots, g.n_layers, g.n_expert, err);
|
||||
if (!ok) {
|
||||
std::fprintf(stderr, "strata generate: %s\n", err.c_str());
|
||||
return 1;
|
||||
// With `--expert-cache auto` the reserve must still be free once the slots are WRITTEN: under WDDM an
|
||||
// allocation is not resident until it is touched, and the free figure read before it can be ~1 GB too
|
||||
// high. A cache sized from it filled the card to 0 MiB, the driver then paged, and a request that needed a
|
||||
// page back while the verify graph spun on a host flag never finished. So the slots are zeroed and the
|
||||
// free figure read again; while it is short of the reserve the cache is reopened smaller.
|
||||
for (int attempt = 0;; ++attempt) {
|
||||
const bool ok = sized_slots.empty()
|
||||
? xcache.open(o.expert_cache, g.n_layers, g.n_expert, (int64_t) strata::kernels::cpu::expert_layout().max_blob, err)
|
||||
: xcache.open_sized(sized_slots, g.n_layers, g.n_expert, err);
|
||||
if (!ok) {
|
||||
std::fprintf(stderr, "strata generate: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
if (!auto_cache || attempt >= 6) break;
|
||||
cudaMemset(xcache.device_slot(0), 0, (size_t) xcache.bytes());
|
||||
cudaDeviceSynchronize();
|
||||
size_t free_b = 0, total_b = 0;
|
||||
cudaMemGetInfo(&free_b, &total_b);
|
||||
const int64_t want = (int64_t) o.vram_reserve_mib << 20;
|
||||
if ((int64_t) free_b >= want - (64ll << 20)) break;
|
||||
// short by (want - free); a figure of 0 only says "at least", so then give back a quarter as well
|
||||
int64_t give = want - (int64_t) free_b + (64ll << 20);
|
||||
if (free_b < ((size_t) 16 << 20)) give = std::max<int64_t>(give, xcache.bytes() / 4);
|
||||
const int64_t keep_bytes = xcache.bytes() - give;
|
||||
std::fprintf(stderr, "strata generate: only %lld MiB free once the slots are written (reserve %d MiB); "
|
||||
"shrinking the expert cache\n", (long long) (free_b >> 20), o.vram_reserve_mib);
|
||||
xcache.close();
|
||||
if (keep_bytes <= 0) { o.expert_cache = 0; sized_slots.clear(); break; }
|
||||
if (!sized_slots.empty()) {
|
||||
int64_t used = 0;
|
||||
size_t keep = 0;
|
||||
while (keep < sized_slots.size() && used + (sized_slots[keep] + 255) / 256 * 256 <= keep_bytes)
|
||||
used += (sized_slots[keep++] + 255) / 256 * 256;
|
||||
sized_slots.resize(keep);
|
||||
o.expert_cache = (int) keep;
|
||||
} else {
|
||||
o.expert_cache = (int) (keep_bytes / (int64_t) strata::kernels::cpu::expert_layout().max_blob);
|
||||
}
|
||||
if (o.expert_cache <= 0) { o.expert_cache = 0; sized_slots.clear(); break; }
|
||||
}
|
||||
}
|
||||
if (o.expert_cache > 0) {
|
||||
std::fprintf(stderr, "strata generate: expert cache %lld slots, %.2f GiB of VRAM; policy is\n",
|
||||
(long long) xcache.slots(), xcache.gib());
|
||||
mem_mark("opening the expert cache");
|
||||
xcache.set_per_layer_admission(o.expert_cache_per_layer);
|
||||
// **ROUND 328: THE HIT PATH IS PROVABLY WRONG, AND THIS SAYS SO OUT LOUD RATHER THAN LETTING IT
|
||||
// CORRUPT A RUN QUIETLY.** With the cache on, the generated tokens DIVERGE from the cache-off run:
|
||||
@@ -1086,12 +1261,13 @@ int main(int argc, char** argv) {
|
||||
// **AND ONE SLOT IS READ BACK AND COMPARED.** A residency table that is right about indices and wrong
|
||||
// about bytes produces a plausible token, which is this project's most expensive failure mode; the
|
||||
// cache's own `verify_slot` is the check and it costs one 1.38 MB D2H at startup.
|
||||
if (!xcache.verify_slot(xcache.slot_of(profile[0].first, profile[0].second),
|
||||
if (prefilled > 0 && !xcache.verify_slot(xcache.slot_of(profile[0].first, profile[0].second),
|
||||
srcp->blob(profile[0].first, profile[0].second), err,
|
||||
(int64_t) strata::kernels::cpu::expert_layout().blob_bytes(profile[0].first))) {
|
||||
std::fprintf(stderr, "strata generate: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
mem_mark("the profile fill");
|
||||
std::fprintf(stderr, "strata generate: pre-filled %lld of %lld slots from the profile; slot 0 verified\n",
|
||||
(long long) prefilled, (long long) want);
|
||||
}
|
||||
@@ -1149,6 +1325,7 @@ int main(int argc, char** argv) {
|
||||
drive.d.hit_done = (void*) hit_done;
|
||||
drive.d.hit_poke = !o.no_hit_poke;
|
||||
drive.d.h_dst.resize((size_t) K);
|
||||
mem_mark("the R4 hit path");
|
||||
std::fprintf(stderr, "strata generate: R4 hit path ON - resident experts are computed on the GPU\n");
|
||||
}
|
||||
// ---- P0.S8: the routing trace. Only meaningful with the pool running, because the ids arrive through
|
||||
@@ -1266,6 +1443,7 @@ int main(int argc, char** argv) {
|
||||
return 2;
|
||||
}
|
||||
|
||||
mem_mark("the expert cache and the graphs");
|
||||
std::fprintf(stderr, "strata generate: session is up; locating the head\n");
|
||||
const strata::core::WeightRef* wo = wt.find("output.weight");
|
||||
if (wo == nullptr) { std::fprintf(stderr, "strata generate: output.weight is missing\n"); return 1; }
|
||||
@@ -1735,11 +1913,15 @@ int main(int argc, char** argv) {
|
||||
//
|
||||
// and each generated token is written to stdout as `T <id>` as soon as its verify window is done, followed by
|
||||
//
|
||||
// DONE <generated> <prompt_tokens> <prompt_ms> <decode_ms> <stop|length>
|
||||
// DONE <generated> <prompt_tokens> <prompt_ms> <decode_ms> <stop|length|cancel> <drafts accepted>
|
||||
// <drafts offered> <prompt tokens reused>
|
||||
//
|
||||
// (`ERR <message>` instead when a request cannot run; `QUIT` ends the process). Every request starts from an
|
||||
// empty sequence (`session_zero`): the prompt goes through the batched prompt path and its last token through
|
||||
// the first verify window - the path all three model files share. Decoding is greedy.
|
||||
// Before that, `RESUME <n>` (n prompt tokens are not read again), `PP <position> <prompt_tokens> <ms> <tok/s>`
|
||||
// after every prompt chunk, and `REUSED <n>` once the prompt is read. (`ERR <message>` instead when a request
|
||||
// cannot run; `STOP` ends the running request at its next step; `QUIT` ends the process.) A request continues
|
||||
// from the live session or the longest conversation checkpoint its prompt starts with (see ConvCheckpoint),
|
||||
// otherwise from an empty sequence (`session_zero`); the rest of the prompt goes through the batched prompt path
|
||||
// and its last token through the first verify window - the path all three model files share. Decoding is greedy.
|
||||
if (o.serve) {
|
||||
if (o.spec < 2 || o.mtp.empty() || o.prefill_chunk <= 0 || thits.d_res == nullptr || host_res.empty()) {
|
||||
std::fprintf(stderr, "strata serve: needs --spec T, --mtp DIR, --prefill CHUNK, --expert-profile P and "
|
||||
@@ -1750,8 +1932,11 @@ int main(int argc, char** argv) {
|
||||
void* borrow = nullptr;
|
||||
uint64_t borrow_bytes = 0;
|
||||
int32_t lend_first = -1;
|
||||
if (!o.no_prefill_borrow && d_res != nullptr) {
|
||||
const uint64_t need = strata::prefill::Prefill::bytes_needed(g, ss, o.prefill_chunk);
|
||||
// a cache too small to lend the prompt path its buffers would make it allocate them on top - on a card the
|
||||
// cache already filled to its reserve, that is the over-subscription the auto sizing avoids - so the
|
||||
// prompt chunk is halved until its buffers fit in the lendable slots (a smaller chunk only reads slower)
|
||||
for (int64_t chunk = o.prefill_chunk; !o.no_prefill_borrow && d_res != nullptr && chunk >= 256; chunk /= 2) {
|
||||
const uint64_t need = strata::prefill::Prefill::bytes_needed(g, ss, chunk);
|
||||
const int64_t blob = (int64_t) strata::kernels::cpu::expert_layout().max_blob;
|
||||
int64_t k = (int64_t) ((need + (uint64_t) blob - 1) / (uint64_t) blob);
|
||||
if (xcache.slot_offsets() != nullptr) {
|
||||
@@ -1760,16 +1945,27 @@ int main(int argc, char** argv) {
|
||||
(uint64_t) (xcache.bytes() - (int64_t) xcache.slot_offsets()[xcache.slots() - k]) < need) ++k;
|
||||
}
|
||||
if (k + 128 <= xcache.slots()) {
|
||||
if (chunk != o.prefill_chunk)
|
||||
std::fprintf(stderr, "strata serve: prompt chunk %lld -> %lld tokens so its buffers fit in the "
|
||||
"expert cache\n", (long long) o.prefill_chunk, (long long) chunk);
|
||||
o.prefill_chunk = chunk;
|
||||
lend_first = (int32_t) (xcache.slots() - k);
|
||||
borrow = xcache.device_slot(lend_first);
|
||||
borrow_bytes = xcache.slot_offsets() ? (uint64_t) (xcache.bytes() - (int64_t) xcache.slot_offsets()[lend_first])
|
||||
: (uint64_t) k * (uint64_t) blob;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (borrow != nullptr)
|
||||
std::fprintf(stderr, "strata serve: the prompt path borrows %lld cache slots (%.2f GiB)\n",
|
||||
(long long) (xcache.slots() - lend_first), (double) borrow_bytes / 1073741824.0);
|
||||
else
|
||||
std::fprintf(stderr, "strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)\n");
|
||||
if (!sp.init(wt, g, ss, srcp, &xcache, host_res.data(), o.prefill_chunk, main_cs, err, borrow, borrow_bytes)) {
|
||||
std::fprintf(stderr, "strata serve: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
mem_mark("the head and the prompt path");
|
||||
strata::core::Verifier ver;
|
||||
strata::core::VerifyHits vh;
|
||||
vh.d_res = thits.d_res;
|
||||
@@ -1780,14 +1976,52 @@ int main(int argc, char** argv) {
|
||||
std::fprintf(stderr, "strata serve: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
mem_mark("the verifier and the drafter's binding");
|
||||
ver.set_split(o.spec_split);
|
||||
ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2
|
||||
: native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0
|
||||
std::vector<int64_t> cur;
|
||||
// ---- the conversation cache (see ConvCheckpoint). `live` is what the session holds right now: the tokens
|
||||
// it has consumed, so a request that starts with exactly them continues without any copy. `checks` are the
|
||||
// saved points; every one of them is a prefix of `live` (the loop drops the rest).
|
||||
std::vector<int32_t> live;
|
||||
std::vector<ImgKey> live_imgs, req_imgs;
|
||||
bool live_ok = false;
|
||||
std::vector<ConvCheckpoint> checks;
|
||||
int64_t pp_total = 0, pp_from = 0, pp_next_check = 0;
|
||||
Clock::time_point pp_t0 = Clock::now();
|
||||
auto imgs_below = [&](const std::vector<ImgKey>& all, int64_t L) {
|
||||
std::vector<ImgKey> v;
|
||||
for (const ImgKey& k : all) if (k.start < L) v.push_back(k);
|
||||
return v;
|
||||
};
|
||||
// a checkpoint of the state after `cur[0, L)`; false only when the copy itself failed
|
||||
auto checkpoint_at = [&](int64_t L) -> bool {
|
||||
if (o.prompt_cache <= 0 || L < 1) return true;
|
||||
for (const ConvCheckpoint& c : checks) if ((int64_t) c.ids.size() == L) return true;
|
||||
ConvCheckpoint c;
|
||||
c.ids.assign(cur.begin(), cur.begin() + L);
|
||||
c.imgs = imgs_below(req_imgs, L);
|
||||
if (cudaDeviceSynchronize() != cudaSuccess || !checkpoint_save(c, ss, g)) return false;
|
||||
checks.push_back(std::move(c));
|
||||
while ((int) checks.size() > o.prompt_cache) checks.erase(checks.begin()); // the oldest goes first
|
||||
return true;
|
||||
};
|
||||
sp.on_chunk = [&](const float* R_rows, int64_t T, int64_t p0, std::string& e) -> bool {
|
||||
std::vector<int32_t> nxt((size_t) T);
|
||||
for (int64_t t = 0; t < T; ++t) nxt[(size_t) t] = (int32_t) cur[(size_t) (p0 + t + 1)];
|
||||
return mtp.prefill(R_rows, nxt.data(), T, p0, e);
|
||||
if (!mtp.prefill(R_rows, nxt.data(), T, p0, e)) return false;
|
||||
// progress for the server window: PP <position reached> <prompt tokens> <ms> <fresh tokens/s>
|
||||
const int64_t done = p0 + T;
|
||||
const double ms = std::chrono::duration<double, std::milli>(Clock::now() - pp_t0).count();
|
||||
std::printf("PP %lld %lld %.0f %.1f\n", (long long) done, (long long) pp_total, ms,
|
||||
ms > 0.0 ? 1000.0 * (double) (done - pp_from) / ms : 0.0);
|
||||
std::fflush(stdout);
|
||||
if (o.prompt_cache_every > 0 && done >= pp_next_check) {
|
||||
if (!checkpoint_at(done)) { e = "saving a conversation checkpoint failed"; return false; }
|
||||
pp_next_check = done + o.prompt_cache_every;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
drive.d.plan = ver.plan_sink();
|
||||
drive.d.pcie_num = std::max(0, std::min(256, (int) (o.pcie_frac * 256.0 + 0.5)));
|
||||
@@ -1852,7 +2086,52 @@ int main(int argc, char** argv) {
|
||||
for (float& v : drive.d.usage) v *= 0.7f;
|
||||
return true;
|
||||
};
|
||||
std::printf("READY %lld\n", (long long) o.max_context);
|
||||
// stdin is read on its own thread, so a STOP line reaches a request that is still running (the client went
|
||||
// away, or pressed Esc): the flag is checked between prompt chunks and between verify windows.
|
||||
std::atomic<bool> stop_req{false};
|
||||
std::mutex in_mu;
|
||||
std::condition_variable in_cv;
|
||||
std::deque<std::string> in_lines;
|
||||
bool in_eof = false;
|
||||
std::thread([&] {
|
||||
std::string l;
|
||||
while (std::getline(std::cin, l)) {
|
||||
if (!l.empty() && l.back() == '\r') l.pop_back();
|
||||
if (l == "STOP") { stop_req.store(true); continue; }
|
||||
std::lock_guard<std::mutex> lk(in_mu);
|
||||
in_lines.push_back(l);
|
||||
in_cv.notify_one();
|
||||
}
|
||||
std::lock_guard<std::mutex> lk(in_mu);
|
||||
in_eof = true;
|
||||
in_cv.notify_one();
|
||||
}).detach();
|
||||
auto next_line = [&](std::string& out) -> bool {
|
||||
std::unique_lock<std::mutex> lk(in_mu);
|
||||
in_cv.wait(lk, [&] { return !in_lines.empty() || in_eof; });
|
||||
if (in_lines.empty()) return false;
|
||||
out = std::move(in_lines.front());
|
||||
in_lines.pop_front();
|
||||
return true;
|
||||
};
|
||||
sp.should_stop = [&] { return stop_req.load(); };
|
||||
// STRATA_TRACE=1: one stderr line per step of a request (the log shows where a request stops)
|
||||
const bool trace = std::getenv("STRATA_TRACE") != nullptr;
|
||||
auto tr = [&](const char* what, long long a = -1, long long b = -1) {
|
||||
if (!trace) return;
|
||||
std::fprintf(stderr, "strata trace: %s %lld %lld\n", what, a, b);
|
||||
std::fflush(stderr);
|
||||
};
|
||||
{
|
||||
// what is left once everything is allocated: under WDDM a GPU filled to the brim does not fail, it pages -
|
||||
// and a page-in while the verify graph spins on a host flag stalls the request for good
|
||||
size_t free_b = 0, total_b = 0;
|
||||
cudaMemGetInfo(&free_b, &total_b);
|
||||
std::fprintf(stderr, "strata serve: %lld MiB of VRAM free with everything loaded%s\n",
|
||||
(long long) (free_b >> 20), free_b < ((size_t) 128 << 20)
|
||||
? " - LOW: requests may stall; lower --max-context or raise --vram-reserve-mib" : "");
|
||||
}
|
||||
std::printf("READY %lld stop\n", (long long) o.max_context); // "stop": this engine honours STOP
|
||||
std::fflush(stdout);
|
||||
std::string line;
|
||||
int64_t rounds = 0;
|
||||
@@ -1865,9 +2144,9 @@ int main(int argc, char** argv) {
|
||||
bool mrope_identity = true;
|
||||
std::vector<float> img_rows;
|
||||
std::vector<const float*> row_ptr;
|
||||
while (std::getline(std::cin, line)) {
|
||||
if (!line.empty() && line.back() == '\r') line.pop_back();
|
||||
while (next_line(line)) {
|
||||
if (line == "QUIT") break;
|
||||
stop_req.store(false); // a STOP that arrived between requests is stale
|
||||
const bool geni = line.rfind("GENI ", 0) == 0;
|
||||
if (!geni && line.rfind("GEN ", 0) != 0) {
|
||||
std::printf("ERR expected: GEN <max_new> <id,id,...> or GENI <max_new> <file> <id,id,...>\n");
|
||||
@@ -1888,6 +2167,7 @@ int main(int argc, char** argv) {
|
||||
continue;
|
||||
}
|
||||
const int64_t n = (int64_t) ids.size();
|
||||
req_imgs.clear();
|
||||
if (geni && !o.vision) { std::printf("ERR this engine was started without --vision\n"); continue; }
|
||||
if (geni || !mrope_identity) {
|
||||
// positions for every cell this request can reach; the identity again for a text request
|
||||
@@ -1929,6 +2209,12 @@ int main(int argc, char** argv) {
|
||||
if (ids[(size_t) i] != kImagePad) { put(i, p, p, p); ++p; ++i; continue; }
|
||||
if (k >= imgs.size()) { ve = "the prompt has more images than the embeddings file"; break; }
|
||||
const Img& im = imgs[k++];
|
||||
{ // what the conversation cache compares: a picture is its grid and its embeddings
|
||||
const int64_t grid[3] = {im.n, im.nx, im.ny};
|
||||
uint64_t h = fnv1a(grid, sizeof grid);
|
||||
h = fnv1a(img_rows.data() + im.off, (size_t) im.n * (size_t) g.n_embd * sizeof(float), h);
|
||||
req_imgs.push_back({i, h});
|
||||
}
|
||||
for (int64_t j = 0; j < im.n && ve.empty(); ++j)
|
||||
if (i + j >= n || ids[(size_t) (i + j)] != kImagePad)
|
||||
ve = "image " + std::to_string(k) + " has " + std::to_string((long long) im.n) +
|
||||
@@ -1945,7 +2231,9 @@ int main(int argc, char** argv) {
|
||||
if (ve.empty() && n > 0 && ids[(size_t) (n - 1)] == kImagePad) ve = "the prompt cannot end in an image";
|
||||
for (int64_t c = n; ve.empty() && c < cells; ++c) put(c, p + (c - n), p + (c - n), p + (c - n));
|
||||
}
|
||||
tr("positions built", (long long) img_rows.size());
|
||||
cudaDeviceSynchronize();
|
||||
tr("device idle");
|
||||
if (ve.empty() && cudaMemcpy(d_mrope, mrope_host.data(), mrope_host.size() * sizeof(int32_t),
|
||||
cudaMemcpyHostToDevice) != cudaSuccess)
|
||||
ve = "the image position upload failed";
|
||||
@@ -1971,36 +2259,183 @@ int main(int argc, char** argv) {
|
||||
if (bad) { std::printf("ERR a token id is outside the vocabulary\n"); continue; }
|
||||
cur = ids;
|
||||
const Clock::time_point r0 = Clock::now();
|
||||
strata::core::session_zero(ss, g, nullptr, main_cs);
|
||||
cudaStreamSynchronize(main_stream);
|
||||
// ---- where this request starts reading: the live session, or a checkpoint, whose tokens AND pictures are
|
||||
// exactly the start of this prompt - at most n - 1 of them, the last token is always the first window
|
||||
auto starts_with = [&](const std::vector<int32_t>& pre, const std::vector<ImgKey>& pre_imgs) -> bool {
|
||||
const int64_t L = (int64_t) pre.size();
|
||||
if (L < 1 || L > n - 1) return false;
|
||||
for (int64_t i = 0; i < L; ++i)
|
||||
if ((int32_t) ids[(size_t) i] != pre[(size_t) i]) return false;
|
||||
return imgs_below(req_imgs, L) == pre_imgs;
|
||||
};
|
||||
int64_t resume = 0;
|
||||
bool from_live = false;
|
||||
if (o.prompt_cache > 0) {
|
||||
if (live_ok && starts_with(live, live_imgs)) { resume = (int64_t) live.size(); from_live = true; }
|
||||
for (const ConvCheckpoint& c : checks)
|
||||
if ((int64_t) c.ids.size() > resume && starts_with(c.ids, c.imgs)) {
|
||||
resume = (int64_t) c.ids.size();
|
||||
from_live = false;
|
||||
}
|
||||
}
|
||||
// this request rewrites every cell from `resume` on, so a checkpoint past it (or not on this prompt's
|
||||
// path) no longer has its cells; the ones kept are prefixes of both the old tokens and the new
|
||||
checks.erase(std::remove_if(checks.begin(), checks.end(), [&](const ConvCheckpoint& c) {
|
||||
return (int64_t) c.ids.size() > resume || !starts_with(c.ids, c.imgs);
|
||||
}), checks.end());
|
||||
live_ok = false; // until this request has finished, the session is in between
|
||||
int64_t reread_to = -1; // STRATA_CKPT_REREAD only: read [0, reread_to) again instead of restoring
|
||||
if (resume == 0) {
|
||||
strata::core::session_zero(ss, g, nullptr, main_cs);
|
||||
cudaStreamSynchronize(main_stream);
|
||||
checks.clear();
|
||||
} else if (!from_live) {
|
||||
const ConvCheckpoint* c = nullptr;
|
||||
for (const ConvCheckpoint& k : checks) if ((int64_t) k.ids.size() == resume) c = &k;
|
||||
static const bool reread = std::getenv("STRATA_CKPT_REREAD") != nullptr;
|
||||
if (reread && c != nullptr) {
|
||||
// THE CHECK OF THE CHECKPOINT: instead of restoring it, read its tokens again from position 0 in
|
||||
// one run (below, with the prompt path's slots lent like any read) - the same chunks the request
|
||||
// that saved it read them in, when that request started at 0. With the VRAM expert set fixed
|
||||
// (--adapt-swaps 0) the answer must match the restored one token for token; anything the
|
||||
// checkpoint missed shows up as a difference.
|
||||
strata::core::session_zero(ss, g, nullptr, main_cs);
|
||||
cudaStreamSynchronize(main_stream);
|
||||
reread_to = resume;
|
||||
std::fprintf(stderr, "strata serve: STRATA_CKPT_REREAD: reading %lld tokens again instead of "
|
||||
"restoring\n", (long long) resume);
|
||||
} else if (c == nullptr || !checkpoint_restore(*c, ss, g)) {
|
||||
std::printf("ERR restoring a conversation checkpoint failed\n");
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
tr("request", n, geni ? 1 : 0);
|
||||
mtp.set_prompt_len(n);
|
||||
const int64_t read_from = reread_to > 0 ? 0 : resume;
|
||||
pp_total = n;
|
||||
pp_from = read_from;
|
||||
pp_t0 = r0;
|
||||
pp_next_check = reread_to > 0 ? INT64_MAX : resume + o.prompt_cache_every;
|
||||
std::printf("RESUME %lld\n", (long long) resume); // before reading: this many prompt tokens are reused
|
||||
std::fflush(stdout);
|
||||
// A SHORT PART OF THE PROMPT - the new message of a chat that continues from a checkpoint, the assistant
|
||||
// header - goes through the verify windows, S tokens at a time, as decode reads them. The batched path
|
||||
// costs ~300 ms per run however few tokens it has (it streams every expert the chunk routes to that is
|
||||
// not in VRAM over PCIe), and it borrows slots it must refill after (~180 ms); a window costs ~16 ms a
|
||||
// token, with the misses on the CPU. Each part below is decided on its own, so a long first message is
|
||||
// read batched and its header still goes through the windows. Picture rows need the batched path.
|
||||
// STRATA_CKPT_REREAD compares a restored checkpoint with a batched re-read, so it keeps every read batched.
|
||||
static const bool no_short = std::getenv("STRATA_CKPT_REREAD") != nullptr;
|
||||
auto windows_ok = [&](int64_t a, int64_t b) -> bool {
|
||||
if (no_short || b - a > o.short_read) return false;
|
||||
if (sp.embd_rows != nullptr)
|
||||
for (int64_t i = a; i < b; ++i)
|
||||
if (sp.embd_rows[i] != nullptr) return false;
|
||||
return true;
|
||||
};
|
||||
// tokens [a, b) through the windows: commit all of them, then give the draft layer their residuals
|
||||
auto read_windows = [&](int64_t a, int64_t b, std::string& e) -> bool {
|
||||
std::vector<int32_t> win((size_t) S), outw((size_t) S), nxt((size_t) S);
|
||||
for (int64_t q = a; q < b;) {
|
||||
if (stop_req.load()) { e = "cancelled"; return false; }
|
||||
const int T = (int) std::min<int64_t>(S, b - q);
|
||||
for (int t = 0; t < T; ++t) {
|
||||
win[(size_t) t] = (int32_t) cur[(size_t) (q + t)];
|
||||
nxt[(size_t) t] = (int32_t) cur[(size_t) (q + t + 1)];
|
||||
}
|
||||
drive.d.layers = 0;
|
||||
drive.d.experts = 0;
|
||||
drive.d.failed = false;
|
||||
if (!ver.run(T, win.data(), q, &drive_pool_multi, &drive, outw.data(), e) || drive.d.failed) {
|
||||
if (drive.d.failed && drive.d.fail) e = drive.d.fail;
|
||||
return false;
|
||||
}
|
||||
if (!ver.commit(T, e) || !mtp.prefill(ver.final_R_all(), nxt.data(), T, q, e)) return false;
|
||||
q += T;
|
||||
}
|
||||
const double ms = std::chrono::duration<double, std::milli>(Clock::now() - pp_t0).count();
|
||||
std::printf("PP %lld %lld %.0f %.1f\n", (long long) b, (long long) pp_total, ms,
|
||||
ms > 0.0 ? 1000.0 * (double) (b - pp_from) / ms : 0.0);
|
||||
std::fflush(stdout);
|
||||
return true;
|
||||
};
|
||||
// the batched path's slots are lent just before its first run and given back (refilled) before a window
|
||||
// reads - so the windows always see the whole expert cache - or once the prompt is read
|
||||
std::vector<std::pair<int32_t, int32_t>> lent_now;
|
||||
apply_pending(true);
|
||||
if (lend_first >= 0) {
|
||||
auto lend = [&] {
|
||||
if (lend_first < 0 || !lent_now.empty()) return;
|
||||
for (size_t i = 0; i < host_res.size(); ++i)
|
||||
if (host_res[i] >= lend_first) {
|
||||
lent_now.emplace_back((int32_t) i, host_res[i]);
|
||||
host_res[i] = strata::core::kNotResident;
|
||||
}
|
||||
cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice);
|
||||
}
|
||||
if (n > 1 && !sp.run(ids.data(), n - 1, 0, err)) {
|
||||
std::fprintf(stderr, "strata serve: %s\n", err.c_str());
|
||||
std::printf("ERR %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
for (const auto& [i, slot] : lent_now) {
|
||||
const uint8_t* b = srcp->blob(i / g.n_expert, i % g.n_expert);
|
||||
if (b == nullptr || !xcache.fill_slot_blocking(slot, b, err,
|
||||
(int64_t) strata::kernels::cpu::expert_layout().blob_bytes(i / g.n_expert))) {
|
||||
};
|
||||
auto refill = [&](std::string& e) -> bool {
|
||||
if (lent_now.empty()) return true;
|
||||
tr("refill start", (long long) lent_now.size());
|
||||
for (const auto& [i, slot] : lent_now) {
|
||||
const uint8_t* b = srcp->blob(i / g.n_expert, i % g.n_expert);
|
||||
if (b == nullptr || !xcache.fill_slot_blocking(slot, b, e,
|
||||
(int64_t) strata::kernels::cpu::expert_layout().blob_bytes(i / g.n_expert)))
|
||||
return false;
|
||||
host_res[(size_t) i] = slot;
|
||||
}
|
||||
cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice);
|
||||
lent_now.clear();
|
||||
return true;
|
||||
};
|
||||
apply_pending(true);
|
||||
bool cancelled = false;
|
||||
tr("prompt start", n - 1);
|
||||
// The prompt is read in two parts when it has a turn boundary past `resume`: up to the last <|im_start|>
|
||||
// (the conversation so far), a checkpoint there, then the new turn's header. The next request of the same
|
||||
// chat renders the same history - but not always the same header or the thinking of this reply - so that
|
||||
// checkpoint is the one it reuses.
|
||||
int64_t turn_at = -1;
|
||||
if (o.prompt_cache > 0 && o.turn_token >= 0)
|
||||
for (int64_t i = n - 1; i > resume; --i)
|
||||
if (ids[(size_t) i] == o.turn_token) { turn_at = i; break; }
|
||||
int64_t at = read_from;
|
||||
for (const int64_t to : {reread_to, turn_at, n - 1}) {
|
||||
if (to <= at) continue;
|
||||
const bool win = windows_ok(at, to);
|
||||
if (win && !refill(err)) {
|
||||
std::printf("ERR refilling a lent slot failed: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
host_res[(size_t) i] = slot;
|
||||
if (!win) lend();
|
||||
const auto tsp = Clock::now();
|
||||
const bool sp_ok = win ? read_windows(at, to, err) : sp.run(ids.data() + at, to - at, at, err);
|
||||
if (trace) {
|
||||
std::fprintf(stderr, "strata trace: read %lld tokens (%s) in %.1f ms\n", (long long) (to - at),
|
||||
win ? "windows" : "batched",
|
||||
std::chrono::duration<double, std::milli>(Clock::now() - tsp).count());
|
||||
std::fflush(stderr);
|
||||
}
|
||||
if (!sp_ok) {
|
||||
if (!stop_req.load()) {
|
||||
std::fprintf(stderr, "strata serve: %s\n", err.c_str());
|
||||
std::printf("ERR %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
cancelled = true; // stopped while reading the prompt: refill the lent slots below, then DONE cancel
|
||||
break;
|
||||
}
|
||||
at = to;
|
||||
if (to == turn_at && !checkpoint_at(to)) {
|
||||
std::printf("ERR saving a conversation checkpoint failed\n");
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
if (!lent_now.empty())
|
||||
cudaMemcpy(d_res, host_res.data(), host_res.size() * sizeof(int32_t), cudaMemcpyHostToDevice);
|
||||
if (!refill(err)) {
|
||||
std::printf("ERR refilling a lent slot failed: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
tr("prompt done (slots refilled)");
|
||||
const double prompt_ms = std::chrono::duration<double, std::milli>(Clock::now() - r0).count();
|
||||
std::printf("REUSED %lld\n", (long long) resume); // the prompt is read; the first window comes next
|
||||
std::fflush(stdout);
|
||||
// the verify windows: the first holds the last prompt token alone
|
||||
int64_t p = n - 1;
|
||||
int32_t x = (int32_t) ids[(size_t) (n - 1)];
|
||||
@@ -2008,9 +2443,15 @@ int main(int argc, char** argv) {
|
||||
std::vector<float> dprob((size_t) S, 0.0f);
|
||||
bool first_window = true;
|
||||
int64_t produced_n = 0;
|
||||
int64_t draft_offered = 0, draft_accepted = 0;
|
||||
// what the session holds once this request is done: the prompt read so far, then every committed token
|
||||
std::vector<int32_t> consumed;
|
||||
consumed.reserve((size_t) (n + max_new + S));
|
||||
for (int64_t i = 0; i < n - 1; ++i) consumed.push_back((int32_t) ids[(size_t) i]);
|
||||
const char* finish = "length";
|
||||
const Clock::time_point d0 = Clock::now();
|
||||
while (produced_n < max_new) {
|
||||
if (cancelled) finish = "cancel";
|
||||
while (!cancelled && produced_n < max_new) {
|
||||
int T = S;
|
||||
if (o.spec_min_p > 0.0) {
|
||||
T = 1;
|
||||
@@ -2024,6 +2465,7 @@ int main(int argc, char** argv) {
|
||||
drive.d.experts = 0;
|
||||
drive.d.failed = false;
|
||||
apply_pending(false);
|
||||
tr("window", p, T);
|
||||
if (!ver.run(T, window.data(), p, &drive_pool_multi, &drive, outv.data(), err) || drive.d.failed) {
|
||||
std::printf("ERR %s\n", drive.d.failed && drive.d.fail ? drive.d.fail : err.c_str());
|
||||
return 1;
|
||||
@@ -2039,6 +2481,10 @@ int main(int argc, char** argv) {
|
||||
std::printf("ERR %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
// the window's first a + 1 tokens are in the session now (the last output is not: it is next x)
|
||||
for (int i = 0; i <= a; ++i) consumed.push_back(window[(size_t) i]);
|
||||
draft_offered += T - 1;
|
||||
draft_accepted += a;
|
||||
first_window = false;
|
||||
bool eos = false;
|
||||
for (int i = 0; i <= a && produced_n < max_new && !eos; ++i) {
|
||||
@@ -2060,16 +2506,85 @@ int main(int argc, char** argv) {
|
||||
return 1;
|
||||
}
|
||||
if (eos) { finish = "stop"; break; }
|
||||
if (stop_req.load()) { finish = "cancel"; break; }
|
||||
x = outv[(size_t) a];
|
||||
p += a + 1;
|
||||
}
|
||||
const double decode_ms = std::chrono::duration<double, std::milli>(Clock::now() - d0).count();
|
||||
std::printf("DONE %lld %lld %.1f %.1f %s\n", (long long) produced_n, (long long) n, prompt_ms, decode_ms,
|
||||
finish);
|
||||
if (!cancelled) {
|
||||
// a prompt stopped halfway leaves the session somewhere between two chunks: nothing to continue from
|
||||
// (the checkpoints taken while reading it are still good)
|
||||
live.swap(consumed);
|
||||
live_imgs = imgs_below(req_imgs, (int64_t) live.size());
|
||||
live_ok = o.prompt_cache > 0;
|
||||
}
|
||||
static const bool state_hash = std::getenv("STRATA_STATE_HASH") != nullptr;
|
||||
if (state_hash && live_ok) {
|
||||
// DEBUG: a fingerprint of every part of the session over the positions it holds ([0, L)), and
|
||||
// separately of what lies past them in the last KV page (stale cells, fine unless something reads them)
|
||||
cudaDeviceSynchronize();
|
||||
const int64_t L = (int64_t) live.size();
|
||||
const strata::kernels::QsaShapes qs = [&] {
|
||||
strata::kernels::QsaShapes s = strata::kernels::qsa_real_shapes();
|
||||
s.n_head_kv = g.n_head_kv; s.head_dim = g.head_dim; s.idx_dim = g.idx_key_dim;
|
||||
return s;
|
||||
}();
|
||||
auto hash_dev = [&](const void* p, size_t bytes, uint64_t h) {
|
||||
std::vector<uint8_t> b(bytes);
|
||||
if (bytes) cudaMemcpy(b.data(), p, bytes, cudaMemcpyDeviceToHost);
|
||||
return fnv1a(b.data(), b.size(), h);
|
||||
};
|
||||
// the cells [c0, c1) of one int8 K or V pool ([page][kv_head][page_size][head_dim]), bytes per value `w`
|
||||
auto hash_cells = [&](const void* pool, int64_t per_cell, int64_t c0, int64_t c1, uint64_t h) {
|
||||
const int64_t ps = qs.page_size;
|
||||
for (int64_t pg = c0 / ps; pg * ps < c1; ++pg)
|
||||
for (int64_t hd = 0; hd < qs.n_head_kv; ++hd) {
|
||||
const int64_t a = std::max(c0, pg * ps) - pg * ps, e = std::min(c1, (pg + 1) * ps) - pg * ps;
|
||||
const size_t off = (size_t) (((pg * qs.n_head_kv + hd) * ps + a) * per_cell);
|
||||
h = hash_dev((const uint8_t*) pool + off, (size_t) ((e - a) * per_cell), h);
|
||||
}
|
||||
return h;
|
||||
};
|
||||
const ConvStateSizes z = conv_state_sizes(g);
|
||||
uint64_t h_gdn = hash_dev(ss.gdn_state, z.gdn, 1469598103934665603ull);
|
||||
uint64_t h_ple = hash_dev(ss.ple_hist, z.ple, 1469598103934665603ull);
|
||||
uint64_t h_tail = 1469598103934665603ull, h_pool = h_tail, h_kv = h_tail, h_stale = h_tail;
|
||||
const int64_t kvb = qs.head_dim, scb = (qs.head_dim / 64) * 2;
|
||||
const int64_t end_cell = std::min<int64_t>(((L + qs.page_size - 1) / qs.page_size) * qs.page_size,
|
||||
ss.qsa_states[0].max_cells);
|
||||
for (int64_t i = 0; i < g.n_qsa_layers(); ++i) {
|
||||
const strata::core::QsaState& st = ss.qsa_states[i];
|
||||
h_tail = hash_dev(st.idx_tail, z.tail, h_tail);
|
||||
h_pool = hash_dev(st.idx_pooled, (size_t) (L / qs.idx_block) * qs.idx_dim * 4, h_pool);
|
||||
for (const auto& [pool, w] : {std::pair<const void*, int64_t>{st.k_q, kvb}, {st.v_q, kvb},
|
||||
{st.k_scale, scb}, {st.v_scale, scb}}) {
|
||||
h_kv = hash_cells(pool, w, 0, L, h_kv);
|
||||
h_stale = hash_cells(pool, w, L, end_cell, h_stale);
|
||||
}
|
||||
}
|
||||
const strata::core::QsaState& ms = mtp.kv_state();
|
||||
uint64_t h_mtp = 1469598103934665603ull;
|
||||
const int64_t mL = std::min<int64_t>(L, ms.max_cells);
|
||||
for (const auto& [pool, w] : {std::pair<const void*, int64_t>{ms.k_q, kvb}, {ms.v_q, kvb},
|
||||
{ms.k_scale, scb}, {ms.v_scale, scb}})
|
||||
if (pool != nullptr) h_mtp = hash_cells(pool, w, 0, mL, h_mtp);
|
||||
std::fprintf(stderr, "strata serve: STATE_HASH L=%lld gdn=%016llx ple=%016llx tail=%016llx pooled=%016llx "
|
||||
"kv=%016llx mtp=%016llx stale=%016llx ple_prev=%d,%d\n", (long long) L,
|
||||
(unsigned long long) h_gdn, (unsigned long long) h_ple, (unsigned long long) h_tail,
|
||||
(unsigned long long) h_pool, (unsigned long long) h_kv, (unsigned long long) h_mtp,
|
||||
(unsigned long long) h_stale, ss.ple_prev[0], ss.ple_prev[1]);
|
||||
}
|
||||
// DONE <generated> <prompt> <prompt ms> <decode ms> <finish> <drafts accepted> <drafts offered> <reused>
|
||||
std::printf("DONE %lld %lld %.1f %.1f %s %lld %lld %lld\n", (long long) produced_n, (long long) n, prompt_ms,
|
||||
decode_ms, finish, (long long) draft_accepted, (long long) draft_offered, (long long) resume);
|
||||
std::fflush(stdout);
|
||||
std::fprintf(stderr, "strata serve: %lld prompt tokens in %.0f ms (%.1f tok/s), %lld generated in %.0f ms "
|
||||
"(%.1f tok/s)\n", (long long) n, prompt_ms, prompt_ms > 0 ? 1000.0 * n / prompt_ms : 0.0,
|
||||
(long long) produced_n, decode_ms, decode_ms > 0 ? 1000.0 * produced_n / decode_ms : 0.0);
|
||||
const int64_t fresh = n - resume;
|
||||
std::fprintf(stderr, "strata serve: prompt %lld tokens = %lld reused + %lld read in %.0f ms (%.1f tok/s), "
|
||||
"%lld generated in %.0f ms (%.1f tok/s), drafts accepted %lld of %lld, %zu checkpoints%s\n",
|
||||
(long long) n, (long long) resume, (long long) fresh, prompt_ms,
|
||||
prompt_ms > 0 ? 1000.0 * fresh / prompt_ms : 0.0, (long long) produced_n, decode_ms,
|
||||
decode_ms > 0 ? 1000.0 * produced_n / decode_ms : 0.0, (long long) draft_accepted,
|
||||
(long long) draft_offered, checks.size(), cancelled ? " (cancelled)" : "");
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@@ -2403,6 +2918,7 @@ int main(int argc, char** argv) {
|
||||
"--expert-cache and the token graph)\n");
|
||||
return 2;
|
||||
}
|
||||
mem_mark("the head and the prompt path");
|
||||
strata::core::Verifier ver;
|
||||
strata::core::VerifyHits vh;
|
||||
vh.d_res = thits.d_res;
|
||||
@@ -2417,6 +2933,7 @@ int main(int argc, char** argv) {
|
||||
std::fprintf(stderr, "strata generate: %s\n", err.c_str());
|
||||
return 1;
|
||||
}
|
||||
mem_mark("the verifier and the drafter's binding");
|
||||
ver.set_split(o.spec_split);
|
||||
ver.set_pcie_mode(o.pcie_mode == "dma" ? 0 : o.pcie_mode == "direct" ? 1 : o.pcie_mode == "kernel" ? 2
|
||||
: native_pack ? 0 : 2); // auto: DMA for the native packs, the copy kernel for Q2_0
|
||||
|
||||
+66
-13
@@ -15,7 +15,12 @@ its GGUF form:
|
||||
With --base: the base (Q2_0) pack's dense.bin, hard-linked - the float tensors are
|
||||
byte-identical in all three model files (checked) - plus extra.bin for tensors that are
|
||||
float here but quantized in the base pack (blk.1.ple_key).
|
||||
tokenizer/ exported from the GGUF (tools/strata_tokenizer.py).
|
||||
tokenizer/ exported from the GGUF (tools/strata_tokenizer.py), with the model's chat template.
|
||||
|
||||
Split files: every shard of the model is read (<name>-0000N-of-0000M.gguf beside --gguf), so the layers may be
|
||||
split anyhow (Swift 1.5's GGUFs put layers 13-47 in shard 2 and the PLE table in shard 1). A layer whose experts
|
||||
are not in shard 1 names its shard in native_experts.txt (v3). Router tensors stored as F32 whose values are
|
||||
exactly BF16 (Swift 1.5) are written as BF16, the form the engine's router takes; anything else is refused.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -34,6 +39,33 @@ sys.path.insert(0, str(HERE))
|
||||
import gguf_reader as G # noqa: E402
|
||||
|
||||
FLOAT = {"BF16", "F32", "F16"}
|
||||
ROUTERS = ("ffn_gate_inp.weight", "ffn_gate_inp_shexp.weight")
|
||||
NOT_IN_PACK = {"per_layer_token_embd.weight"} # the 28.8 GB PLE table: read from its GGUF by the engine
|
||||
|
||||
|
||||
class Model:
|
||||
"""All shards of one model: name -> (GGUFFile, TensorInfo, memmap, shard path)."""
|
||||
|
||||
def __init__(self, first: pathlib.Path):
|
||||
import re
|
||||
m = re.search(r"-(\d{5})-of-(\d{5})\.gguf$", first.name)
|
||||
paths = [first]
|
||||
if m:
|
||||
total = int(m.group(2))
|
||||
paths = [first.with_name(first.name[:m.start()] + "-%05d-of-%05d.gguf" % (i, total))
|
||||
for i in range(1, total + 1)]
|
||||
paths = [p for p in paths if p.exists()]
|
||||
self.paths = paths
|
||||
self.where = {}
|
||||
for p in paths:
|
||||
g = G.GGUFFile(p)
|
||||
mm = np.memmap(p, dtype=np.uint8, mode="r")
|
||||
for t in g.tensors:
|
||||
self.where[t.name] = (g, t, mm, p)
|
||||
|
||||
def bytes(self, name) -> np.ndarray:
|
||||
g, t, mm, _ = self.where[name]
|
||||
return tensor_bytes(mm, g, t)
|
||||
ROLES = ("gate", "up", "down")
|
||||
N_EXPERT = 512
|
||||
ALIGN = 64
|
||||
@@ -59,13 +91,14 @@ def is_expert(name: str) -> bool:
|
||||
return name.startswith("blk.") and name.endswith(("_exps.weight",))
|
||||
|
||||
|
||||
def index_standalone(src, out, g, T, mm) -> int:
|
||||
"""Every non-expert tensor of shard 1: floats into dense.bin as stored, quantized ones native-only."""
|
||||
def index_standalone(src, out, model: Model) -> int:
|
||||
"""Every non-expert tensor of the model: floats into dense.bin as stored (exact-BF16 F32 routers as BF16),
|
||||
quantized ones native-only."""
|
||||
rows, at = [], 0
|
||||
served = 0
|
||||
with open(out / "dense.bin", "wb") as fo:
|
||||
for t in g.tensors:
|
||||
if is_expert(t.name):
|
||||
for name, (g, t, mm, _) in model.where.items():
|
||||
if is_expert(t.name) or t.name in NOT_IN_PACK:
|
||||
continue
|
||||
if len(t.shape) > 2:
|
||||
print("tensor %s has %d dimensions; the index holds two" % (t.name, len(t.shape)))
|
||||
@@ -75,6 +108,13 @@ def index_standalone(src, out, g, T, mm) -> int:
|
||||
if t.type_name in FLOAT:
|
||||
raw = tensor_bytes(mm, g, t).tobytes()
|
||||
kind = {"BF16": "4", "F16": "5", "F32": "2"}[t.type_name]
|
||||
if t.type_name == "F32" and t.name.endswith(ROUTERS):
|
||||
u = np.frombuffer(raw, dtype=np.uint32)
|
||||
if np.count_nonzero(u & 0xFFFF):
|
||||
print("router %s is F32 with values that are not BF16; the engine's router is BF16" % t.name)
|
||||
return 1
|
||||
raw = (u >> 16).astype(np.uint16).tobytes() # the exact BF16 values
|
||||
kind = "4"
|
||||
rows.append([t.name, "0", kind, str(at), str(len(raw)), "0", str(len(raw)), str(ne0), str(ne1),
|
||||
"0", "0", "1"] + ["0"] * 7)
|
||||
fo.write(raw)
|
||||
@@ -167,15 +207,21 @@ def main() -> int:
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
g = G.GGUFFile(src)
|
||||
T = {t.name: t for t in g.tensors}
|
||||
mm = np.memmap(src, dtype=np.uint8, mode="r")
|
||||
model = Model(src)
|
||||
T = {n: w[1] for n, w in model.where.items()}
|
||||
if len(model.paths) > 1:
|
||||
print("model shards: " + ", ".join(p.name for p in model.paths))
|
||||
if a.base:
|
||||
rc = index_from_base(a, src, base, out, g, T, mm)
|
||||
if any(w[3] != src for w in model.where.values() if not w[1].name in NOT_IN_PACK):
|
||||
print("--base needs a model whose tensors are all in shard 1")
|
||||
return 1
|
||||
rc = index_from_base(a, src, base, out, g, {t.name: t for t in g.tensors}, mm)
|
||||
else:
|
||||
rc = index_standalone(src, out, g, T, mm)
|
||||
rc = index_standalone(src, out, model)
|
||||
if rc:
|
||||
return rc
|
||||
if not (out / "tokenizer" / "vocab.json").exists():
|
||||
if not (out / "tokenizer" / "vocab.json").exists() or not (out / "tokenizer" / "chat_template.jinja").exists():
|
||||
subprocess.run([sys.executable, str(HERE / "strata_tokenizer.py"), "--gguf", str(src), "--out", str(out)],
|
||||
check=True) # writes <out>/tokenizer/
|
||||
|
||||
@@ -192,10 +238,17 @@ def main() -> int:
|
||||
layout.append((l, ts[0].type_id, ts[2].type_id, offset, blob, ts))
|
||||
offset += blob * N_EXPERT
|
||||
with open(out / "native_experts.txt", "w", encoding="utf-8", newline="\n") as fo:
|
||||
fo.write("# strata native experts v2: layer gu_type d_type offset blob_bytes gate_off up_off down_off "
|
||||
"(n_expert %d, total %d; the last three are absolute offsets in %s)\n" % (N_EXPERT, offset, src.name))
|
||||
fo.write("# strata native experts v3: layer gu_type d_type offset blob_bytes gate_off up_off down_off [shard] "
|
||||
"(n_expert %d, total %d; absolute offsets in %s, or in the named shard beside it)\n"
|
||||
% (N_EXPERT, offset, src.name))
|
||||
for l, gt, dt, off, blob, ts in layout:
|
||||
fo.write("%d %d %d %d %d %d %d %d\n" % (l, gt, dt, off, blob, *[g.data_start + t.offset for t in ts]))
|
||||
ws = [model.where[t.name] for t in ts]
|
||||
if len({w[3] for w in ws}) != 1:
|
||||
print("layer %d: its gate/up/down tensors are in different shards" % l)
|
||||
return 1
|
||||
gg, shard = ws[0][0], ws[0][3]
|
||||
line = "%d %d %d %d %d %d %d %d" % (l, gt, dt, off, blob, *[gg.data_start + t.offset for t in ts])
|
||||
fo.write(line + ("" if shard == src else " " + shard.name) + "\n")
|
||||
if a.skip_experts or not a.experts_bin:
|
||||
if (out / "experts.bin").exists() and not a.experts_bin:
|
||||
print("note: %s/experts.bin exists; the engine reads it instead of the GGUF" % out)
|
||||
@@ -206,7 +259,7 @@ def main() -> int:
|
||||
return 0
|
||||
with open(path, "wb") as fo:
|
||||
for l, gt, dt, off, blob, ts in layout:
|
||||
parts = [tensor_bytes(mm, g, t).reshape(N_EXPERT, -1) for t in ts]
|
||||
parts = [model.bytes(t.name).reshape(N_EXPERT, -1) for t in ts]
|
||||
chunk = np.concatenate(parts, axis=1) # (512, blob): gate | up | down per expert
|
||||
assert chunk.shape == (N_EXPERT, blob)
|
||||
fo.write(chunk.tobytes())
|
||||
|
||||
@@ -230,6 +230,12 @@ def extract(gguf_path, out_dir) -> dict:
|
||||
sorted(tk.ranks.items(), key=lambda kv: kv[1])), encoding="utf-8")
|
||||
(out / "token_type.json").write_text(json.dumps(tk.token_types), encoding="utf-8")
|
||||
(out / "tokenizer.json").write_text(json.dumps(cfg, ensure_ascii=False, indent=1), encoding="utf-8")
|
||||
# the model's own chat template: fine-tunes change it (Swift 1.5 differs from Qwen3.8-Flash-Next's)
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
from gguf_reader import GGUFFile
|
||||
tpl = GGUFFile(pathlib.Path(gguf_path)).metadata.get("tokenizer.chat_template")
|
||||
if tpl:
|
||||
(out / "chat_template.jinja").write_text(tpl, encoding="utf-8", newline="\n")
|
||||
return cfg
|
||||
|
||||
|
||||
|
||||
@@ -99,6 +99,30 @@ int main(int argc, char** argv) {
|
||||
if (gg) gguf_free(gg);
|
||||
}
|
||||
if (n_embd <= 0) { std::printf("ERR the vision encoder has no projection_dim\n"); std::fflush(stdout); return 1; }
|
||||
// Warm up at the LARGEST picture before READY: the encoder's GPU work buffers are allocated now, not at the first
|
||||
// real picture. The server starts this process before the engine, so the engine sizes its expert cache from
|
||||
// what is really left; allocating ~1 GB later, on a GPU the engine has filled, made Windows page GPU memory and
|
||||
// the engine crawl to a standstill. (A square image well above any cap; mtmd scales it to the token limit.)
|
||||
{
|
||||
const uint32_t side = 2048;
|
||||
std::vector<unsigned char> rgb((size_t) side * side * 3, 128);
|
||||
mtmd_bitmap* bm = mtmd_bitmap_init(side, side, rgb.data());
|
||||
mtmd_input_chunks* chunks = mtmd_input_chunks_init();
|
||||
const std::string marker = mtmd_default_marker();
|
||||
mtmd_input_text txt{marker.c_str(), marker.size(), false, true};
|
||||
const mtmd_bitmap* bms[1] = {bm};
|
||||
int warm_tokens = 0;
|
||||
if (bm && mtmd_tokenize(ctx, chunks, &txt, bms, 1) == 0) {
|
||||
for (size_t c = 0; c < mtmd_input_chunks_size(chunks); ++c) {
|
||||
const mtmd_input_chunk* ch = mtmd_input_chunks_get(chunks, c);
|
||||
if (mtmd_input_chunk_get_type(ch) == MTMD_INPUT_CHUNK_TYPE_IMAGE && mtmd_encode_chunk(ctx, ch) == 0)
|
||||
warm_tokens = (int) mtmd_input_chunk_get_n_tokens(ch);
|
||||
}
|
||||
}
|
||||
std::fprintf(stderr, "strata-vision: warmed up at %d image tokens\n", warm_tokens);
|
||||
mtmd_input_chunks_free(chunks);
|
||||
if (bm) mtmd_bitmap_free(bm);
|
||||
}
|
||||
std::printf("READY %d\n", n_embd);
|
||||
std::fflush(stdout);
|
||||
|
||||
|
||||
Reference in new issue
Block a user