Row error on 4,000 random rows against the checkpoint's BF16 table (Q8_0 0.53%, FP8 2.64%, IQ4_NL 7.60%, matching the
numbers reported in #651), and teacher-forced KL on the Q2_0 model with each table.
- a prompt that fits 0.1.39's auto chunk keeps 0.1.39's ring (Prefill::set_ring_budget's small_max): 4K prompts
were 1-17% slower with the smaller ring (Coder -15%, IQ3_S -5%, IQ3_XXS -2%; 5 interleaved pairs each);
- ring slots are given up for a bigger chunk only where 0.1.39's rule held the chunk under 6144 (ring_cap_for);
from 6144 on the ring keeps its 0.1.39 size (the Coder 6144/384 -> 7936/199 was -12% at 32K);
- the unpinned arm's cap stays 96 (the byte budget made it ~49 on IQ3_S);
- the scan sets its own ring budget instead of set_ring_override, so a layer split's 96-slot override (#340) is no
longer clobbered by the scan.
RTX 5070 vs STRATA_RING_BYTES=0: 4K -0.7..+1.1% (bit-identical); 32K IQ3_XXS +18.5% (cache 1500) / +9.4%, Coder
+3.1%, IQ3_S +0.2%.
server: a request decoding in a slot whose neighbours finished (no other slot busy, nobody waiting, at least 32
tokens allowed) is BSTOPped and continued on the solo path with its MTP drafts - the engine copies the slot's
sessions back (INFO slot_cache=1; at most twice per request; STRATA_PARALLEL_SOLO=0 keeps it in the slot).
engine: INFO batch_slots=N slot_cache=0|1.
tools/batch_interleave_test.py: the back-to-solo path, and a next turn from a slot's turn checkpoint (a client
that drops the reply's thinking).
docs: "parallel": N, what a slot costs and what setup recommends, the server's handling (interleaved prompt reads,
BYIELD, slot conversation cache, back to solo), exactness and its settings, the RTX 5070 measurements (the slots
buy waiting time, not speed, on a 12 GB card), the protocol; DETAILS / README / AI_SETUP mention the option.
INSTALL.md gets the section (the build setup compiles, what differs, the forced-path numbers from a
Ryzen 5 7600 + RTX 5070, what is untested); the CPU rows of INSTALL.md / DETAILS.md and AI_SETUP.md
point to it.
Conflicts with 0.1.39: --trim-stage-weights is now the same switch as STRATA_STAGE_TRIM=1 (PR #639's
implementation, already in 0.1.39); the #533 VRAM command is taken between batch windows only when no slot
is decoding.
--vram-elastic (the config's "vram_elastic": true) allocates the expert cache as one
reserved address range backed by physical segments (CUDA virtual memory management,
512 MiB each, --vram-segment-mib), so every slot keeps its address. Between
requests, the engine command `VRAM <reserve_mib>` gives the last segments back to
the driver until that much VRAM is free (their experts become CPU misses, the prompt
path's loan moves down and its chunk shrinks to fit), or maps them again and puts
the same experts back when more is free. `VRAM` alone: the start's reserve. The
server's POST /v1/vram {"reserve_mib": N | null} sends it after a running request,
keeps it for an engine that is unloaded, and /v1/status shows the cache's size.
Never automatic. One NVIDIA GPU only (no layer split, helper caches, --peer-device
or the resident low-RAM mode); HIP refuses the flag.
RTX 5070, Q2_0 (4.8 GiB cache): reserve 6000 -> 4.3 GiB freed in 78 ms, decode
44 -> 33 tok/s; grown back in 92 ms, the answers token for token the ones before.
Without the flag the cache is the one cudaMalloc it was (same answers with and
without the flag until a VRAM command).
IQ3_XXS on the RTX 5070, temperature 0, 256 tokens, 8 French prompts (topics outside the corpus), 2 English and 2 code
prompts, arms alternating, 2 passes each: drafts accepted in French 0.505 (en) / 0.507 (cjk) -> 0.597 (fr), higher on
all 8 prompts and in each pass; English 0.610 -> 0.629, code 0.772 -> 0.779. Decode speed was not measurable here
(builds ran beside it); the reporter measured 141 -> 158 tok/s in French on an RTX 5090.
Stateless, on the chat path: a Responses request becomes the chat template's
messages, tools and kwargs, runs through Service.run, and its reasoning, text
and tool calls come back as output items and the official typed SSE events
(response.created ... response.completed / incomplete / failed).
- input as a string or items (messages, reasoning, function_call,
function_call_output, custom tool calls); instructions and leading developer
messages make one system message; tool results are put in call order
- function, namespace (Codex's MCP/agent tools) and custom tools; hosted tools
are left out with a log line
- reasoning.effort -> reasoning_effort, max_output_tokens, text.format
json_schema/json_object through the structured output path
- reasoning returned as reasoning_text (and as opaque encrypted_content when
asked) and replayed into the prompt, so Codex's next turn reuses the cache
- errors in the Responses format; same API key, Host and Origin checks as /v1
- a response.in_progress event every 15 s while a long prompt is read
(Codex's idle timeout ignores SSE comments)
- docs: DETAILS.md section with a Codex config.toml
Based on #630 by CC-David-CC (the event assembler, namespace tools, call-id
ordering of results), without its opt-in flag, replay key and crypto
dependency, strict-function and JSON-format refusals, or the second
summary generation pass. Tested with Codex CLI 0.160.0 and openai 2.15.0.
Co-authored-by: David <[email protected]>
The About tab's Model settings card reads GET /config and saves through
POST /config: the sampling defaults, reasoning_budget_tokens,
fit_max_tokens, anthropic_thinking, effort_position, aliases,
idle_unload_s, lazy_load, engine_silence_s, api_monitor, open_browser
and --vram-reserve-mib (serve/runconfig.py). Every value is checked
first (one bad value changes nothing), only the named keys change, every
other key of strata-<model>.json stays as it is (as setup run again
keeps them, #629), the earlier file is kept as .bak, and the change is
used from the next start. POST takes JSON from Strata's own page only,
like /settings, and the API key when one is set; the network, key, MCP
and program keys are not editable there.
- MODELS.md: the RAM rule is the experts + ~10 GB (setup's arena_gb), not shard 1 + 10 GB; the sizes per model.
- DETAILS.md: prompts are read in chunks of up to 8,192 tokens (32,768 opt-in), not 2,048; Q2_0's AVX-512 pack is
~40 GB on disk = 34 GB of experts plus the dense weights.
- README: the NVIDIA speed table says which engine measured which row (Q2_0 0.1.36, the others 0.1.26).
- expert_cache.hpp: the comment that the cache "does not yet compute anything" is gone.
/metrics gains "conversation_cache": the parked conversations, their
bytes, parks/restores/evictions and the last switch, read from the
engine's own log lines ("conversation cache: parked/restored ...") from
the start of its current run on; the budget and slots from its INFO line;
and the prompt tokens reused in the last request and since the start.
The card shows the parked slots and memory when --conversation-cache-mib
is on, the reuse lines always, and how to turn parking on when it is off.
--gpu-stages replays per-layer graphs, which a native (IQ) pack does not capture, and refused with "not captured
with the split". The serve path already times a native window's stages: STRATA_DECODE_TIMING=1 prints the
per-request decode split and STRATA_VERIFY_PROFILE=1 adds the GPU stages. DETAILS.md now says how to read them, and
--gpu-stages on a native pack says to use them.
The decode hit rate is the VRAM share of the experts the pool looked up; the routed experts the GPU reads over PCIe
(--pcie-frac) or another GPU computes are in neither count, so raising --pcie-frac raised the "hit rate" while the
decode got slower. Its meaning stays; now that share is said beside it:
- engine: DriveState::offload_entries counts those entries; the serve line adds "; N more read by the GPU over
PCIe or from another GPU (X% of all M routed)", and the DONE line a 16th field (an older server ignores it).
- server: /metrics' requests get "pcie_share" (None from an older engine), the console's done line "(+X% of the
routed experts over PCIe)", and the Monitor's column is "VRAM hit rate" with "+X% PCIe" after it.
- DETAILS.md says what each means.
No number changes; tokens are unchanged.
The effort's instruction is the first thing the template renders, so a
request that only changes reasoning_effort (or turns thinking off) read
the whole prompt again. With "effort_position": "end" in the run config
the server renders every prompt's start as the default effort's and puts
a low/medium effort in a short system turn right before the answer
(thinking off: just the template's empty thinking block); the rendering
is template-agnostic (history + turn + the generation prompt).
Engine: --tail-role-token ID (default -1, off) moves the turn checkpoint
in front of such a trailing turn, so it holds the conversation and not the
effort text; INFO reports tail_role_token. The server passes the id of
"system" only when the engine binary knows the flag, and says so when it
does not. Without the key nothing changes: the same prompt ids, no flag.
On a 7200 RPM disk the default --ple-io direct's random reads of the n-gram table stalled prompts past the 60 s
watchdog, and the stall report looked like any other engine stall.
- Engine (Linux): at start with --ple-io direct, the table's disk is looked up in sysfs (queue/rotational); on a
rotational one it warns and recommends --ple-io ram or an SSD (not on WSL, whose virtual disk says rotational).
- Stall report (Linux): the engine's threads in state D and their wchan ("threads waiting on the disk (state D): 16
of 35 (blk_io_schedule x16)"), with what that means.
- Setup (Linux): a model on a rotational disk gets --ple-io ram when the RAM holds the table beside the model,
otherwise a warning that recommends an SSD. Nothing changes on an SSD.
- DETAILS.md: --ple-io direct is for SSDs (it reads unbuffered, not through the OS cache), and a troubleshooting row.
A broken state (or a model in a loop) can answer one token forever: #606 streamed 36,689 tokens of "!" for four
minutes, because a request without max_tokens gets the whole remaining context. The server now ends such a reply
at 256 identical tokens in a row with finish "length" and a line in its window saying what to do if it repeats on
a new request. "repeat_stop_tokens": N in strata-<model>.json sets the run (0 = off). Nothing changes for a reply
without such a run.
Kept in the run config's vision.max_tokens (the server already passes it
to the encoder); a setup run again keeps it for the same encoder device.
More than the default is allowed with a note on the encode time. Without
the flag the config is as before. DETAILS.md: vision.mmproj can point to
a Q8_0 mmproj of this model.
- tools/parking_test.py: a follow-up decodes the same tokens whether its conversation stayed live or came back
from the parking cache (every stage's image with a layer split). 4-GPU split, IQ3_S: 3,786 and 9,276-token
conversations restored in 42 / 51 ms, follow-ups identical; without parking the same follow-ups took 3.4 / 6.8 s.
- tools/batch_test.py: --keys to send sampling keys (sampled rows in batch windows).
- docs/BATCHING.md: parking together with batching, the Testing section; docs/DETAILS.md: parking now works with
--layer-split.
A rebinding page cannot send the key, so with one configured the check only
broke tunnels and proxies that pass their own name on (cloudflared does).
Keyless setups keep it, with allowed_hosts; the 403 now also says that
setting an API key turns the check off.
/unload and /load took a plain form POST from any site: they now take JSON
from Strata's own page only, like /settings (_own_page). Without an API key, a
POST to /v1/* that carries an Origin must come from Strata's own page, a page
on localhost or an allowed host name, an origin in trusted_origins or
cors_origins, or a non-web scheme (browser extensions, desktop apps), and then
send JSON; others (and Origin: null) get 403, so a web page can no longer burn
GPU time with a text/plain POST. Requests without an Origin (curl, the SDKs)
are unchanged; with an API key, the key decides.
A web page of another site could point its own name at 127.0.0.1 and reach
the server as the same origin: without an API key it could read /status,
/metrics and /api/requests, change settings and run MCP tools. The server now
answers only Host names it knows: localhost (and *.localhost), any IP address
(rebinding needs a name), the bind address and, bound beyond this PC, the PC's
name, name.local, its LAN addresses and host.docker.internal; any port. No
Host header (HTTP/1.0) passes. "allowed_hosts" in the config and
STRATA_ALLOWED_HOSTS add names (".example.com" for a domain, "*" turns the
check off); trusted_origins' hosts count too. A refusal is a 403 naming the
setting and one log line.
The provider, the context and output limits, the thinking level as a default and as variants, and how
they relate to the context check, fit_max_tokens and reasoning_budget_tokens.
A request whose engine prints nothing for engine_silence_s (default 300 s; while a prompt is read, a chunk may take
3x the previous one's time, the first up to its tokens at 50 tok/s more) ends with EngineSilent (an EngineDied): the
engine is killed and the next request starts it again. The STOP drain, under the request FIFO, is no longer an
untimed wait. "engine_silence_s": 0 keeps the old behaviour. Tests: unit (silence, PP chunks, first chunk, STOP
never acknowledged, 0, config) and over HTTP with a fake engine process (non-stream, stream, stop token).
`--expert-profile-save P` (--serve) writes the adaptive tier's learning as a profile in tools/make_profile.py's
format, byte for byte (read_expert_profile reads it back): every pair ranked - the experts resident in VRAM now
first, then the routing counted since the start (the decayed usage summed at each adapt step, before its decay),
then the profile the engine started from. On QUIT / stdin EOF and every --expert-profile-save-every minutes
(default 10, 0 = exit only) between requests, through <P>.tmp and a rename. Without the flag nothing is counted or
written: `heat` stays empty and every added loop is over it.
Server: "expert_profile_save" (and "expert_profile_save_every") in the config pass the flags, and the next start
begins from the saved file instead of the config's --expert-profile when its header matches that profile's
layers x experts. tests/core/expert_profile_save_test.cpp (CPU only) checks the order and the bytes.
Engine: when the MTP draft head's subset does not fit, it prints what the head needs, the VRAM free, and the smaller
shipped subsets (--draft-vocab cyrillic / en, their MiB from the head's real row size) with how to pick one
(START-HERE.bat / ./setup.sh --draft-vocab en, kept in the model's config). Text only, on the failure path.
Server: a start that ended that way carries the engine's hint (or the same advice for an older engine) in its
"the engine exited before it was ready" error. Setup: a card under 14 GB with the default subset gets a note
recommending --draft-vocab en; nothing is changed, an explicit or saved choice gets no note, and the golden
configs are unchanged.
The server already read drafts_offered / drafts_accepted from the engine's DONE line but only returned them in a
response's timings (draft_n / draft_n_accepted). Each /metrics request entry now carries both (null when the
engine did not report them) and the totals keep their running sums since the server started, so a dashboard such
as llm-visuals can show the acceptance. Additive: no existing field changes.
Claude Code, Claude Desktop, Cursor, VS Code, Codex ... add it over stdio (JSON-RPC 2.0, MCP 2025-06-18, also
2025-03-26 / 2024-11-05); standard library only, so it runs before setup made .venv. Eight tools: strata_status
(running server via /health, /v1/status, /status, /metrics; installed configs, engine, data folder; hardware and
setup's recommendation), strata_models, strata_install (setup --yes --no-start in the background, plan first and
confirm=true to start, progress / cancel on later calls), strata_start (the run script's command, waits for
/health), strata_stop (POST /unload, then ends the server it started), strata_logs, strata_benchmark,
strata_connect_info.
Every argument is checked against a schema and setup's own tables; the only path (data_dir) must be the data
folder, inside the Strata folder, or a new Strata* folder outside system folders. Processes are tracked by pid and
start time in .strata-mcp/, so only processes it started are ever ended; a Strata started elsewhere is only
unloaded. docs/MCP_SERVER.md has every client's config; docs/DETAILS.md links it. Tests use a fake setup and
server (Windows and Linux).
README.md (279 -> ~180 lines): what it is, speed on an NVIDIA and an AMD baseline PC side by side (the AMD
numbers are TODO-AMD placeholders until the RX 9070 XT baseline is measured), what you need with NVIDIA and AMD
as equals, install by asking an AI assistant (docs/AI_SETUP.md, MCP server) or by hand, pick-by-RAM table,
using it, the four common problems, how it works with the two diagrams, Buy Me a Coffee button below the speed
table and at the end. Its anchors other pages use (#which-model-should-i-pick, #install, #using-it,
#something-went-wrong, #how-does-it-work) are kept.
Everything the README had moves, nothing dropped: docs/INSTALL.md (requirements, Windows, Linux, AMD cards,
several cards, Docker, updating, where things are stored, setup's questions incl. rope scaling / images /
speed projection, calibrate, all options), docs/MODELS.md (speed per size, sizes, will it fit, low-RAM, Coder,
Swift 1.5, Unsloth UD-Q4_K_XL, Orca), docs/TROUBLESHOOTING.md (the README's problems plus AMD ones),
docs/HOW_IT_WORKS.md (the longer explanation, credits, license). New docs/AI_SETUP.md (instructions for an AI
agent installing Strata for a user) and AGENTS.md. .github/FUNDING.yml (buy_me_a_coffee: strataengine).
AMD_HIP.md loses "Experimental" in its title and intro; DETAILS.md lists the AMD cards and points to the new pages.
Zuraiz Anjum