A body over 1 MiB was left unread, and its answer was reset on Windows. The limit is now 5 s for the whole body, read one socket read at a time: a body of any size gets its answer, and a client that sends a byte at a time is let go after the 5 s (read() waited for the whole piece, with a new timeout for each byte).
The server answers before reading the body in some cases (401, 403, /load, /unload, a method with no handler) and then closes. Closing on unread bytes makes Windows send a reset, and a client that sends its body after the headers (http.client, urllib, requests) gets WinError 10054 instead of the answer.
After an answer, the body no handler took is now read and dropped, up to 1 MiB and with a 5 s timeout. A larger or chunked body is left unread, as before.
Filter earlier assistant messages with no text, images or tool calls
before rendering the prompt. Keep the final message.
Co-authored-by: jramirez-uab <[email protected]>
Signed-off-by: Arthur031221 <[email protected]>
With tools declared, a <tool_call> in the reasoning is held whole; a declared name becomes a tool call,
anything else (undeclared name, malformed, unfinished, cut by </think>) stays reasoning text. finish() never
makes a call from it. Without tools the parser is byte-identical. Tests: parser (split tags, undeclared,
no tools, unfinished), OpenAI and Anthropic finish reasons, Anthropic block order.
Co-authored-by: kossum <[email protected]>
Maintainer: fatal lines checked before batch routing, 'an earlier window never finished' is fatal, alive() check in generate_batched, tests run on Windows (#748).
Vision.load read whatever an image source named. A network path (\\host\share\x.png, //host/share/x.png,
\\?\UNC\host\..., \??\UNC\host\..., file:////host/share/x.png) made Windows connect to that host and sign in with the
user's NTLM credentials - os.path.isfile alone does it. Such paths are now refused before anything looks at them, and
so is file://host/share/x.png (that names \\host\share; the old code read it as the relative path host/share/x.png).
An http(s) image was read whole, with no cap, inside prepare()'s `with self.fifo`: a huge one filled the memory and a
slow one held every other request for as long as urlopen waited (60 s per read). Now it is read up to 32 MiB
(IMAGE_URL_MAX; a larger Content-Length is refused unread), downloaded before the FIFO is taken, and a URL that cannot
be read is a 400 instead of a dropped connection.
Other sites' pages: with no api_key (the default) _authorized() lets every request in, and /v1/chat/completions and
/v1/messages read the body as JSON whatever its Content-Type; only /settings, /v1/load, /v1/unload and the MCP tools
check the Content-Type and Origin (_own_page). So any web page open in the user's browser could POST a chat request as
text/plain - a CORS "simple" request, no preflight - that names a file on this computer (or a network path) as an
image, and the server read it. The page gets no answer to read (no CORS headers unless cors_origins lists it or "*"),
but the read, and a network path's sign-in, happened. Reproduced over HTTP: answered 200, the file handed to the
encoder. Now a request whose Origin is neither the server's own address nor in trusted_origins may send data: and
http(s) images only; a file path or file:// URL is a 400 that says so. Clients that send no Origin (CLI tools, SDKs,
agents), Strata's own page (it sends data: URLs) and trusted origins are unchanged. As for /settings and MCP, a
DNS-rebinding page, whose Origin matches its Host header, still counts as the server's own.
Tests: each network form is refused with os.path.isfile never called, and local files, file:// and data: URLs still
load; the cap with and without a Content-Length, and a 404; the URL is downloaded with the FIFO free; a foreign-Origin
text/plain request naming a file is a 400 on both APIs, while data: images and no-Origin, own-origin and
trusted-origin requests are answered.
Row error on 4,000 random rows against the checkpoint's BF16 table (Q8_0 0.53%, FP8 2.64%, IQ4_NL 7.60%, matching the
numbers reported in #651), and teacher-forced KL on the Q2_0 model with each table.
The 0.1.39 paragraph said nobody had run the 0.1.39 port on an Arc. Two Arc Pro B60 reports (#867, #870) did, with
the two sycl/ fixes now in 0.1.40 (#784 the compile fix, #866 the ring wait). Say that, and that an unpatched 0.1.39
port did not compile against its own engine sources.
INTEL_ARC.md: "What has been run" rows for 2x and 1x Arc Pro B60, plus notes for the card: PCI id 8086:e211 (bmg-g21), setvars.sh
stopping at OCL_ICD_FILENAMES under `set -u`, running natively without Docker, and the layer-split startup warning that counts the
other card's layers.
sycl/setup_intel.py: the B60 table entry was e221, which the kernel's xe id list files under the BMG-G31 cards (0xE220-0xE223). Both
Arc Pro B60 cards measured here are 8086:e211, so setup named them "Intel GPU e211 (xe)" with a guessed VRAM size. e211 is added;
e221 stays, since there is no card here to say it is wrong. tools/test_setup_sycl.py checks the table.
COMMUNITY_BENCHMARKS.md and bench/results/2026-10-04-community-2x-arc-pro-b60/: the report, in the template's format, with the
server configs, engine logs and per-request timings.
The migration turned `q = cudaStreamQuery(cs)` into `DPCT_CHECK_ERROR(cs->ext_oneapi_empty())`. That macro yields 0 whenever
the call does not throw, whatever the queue holds. The guard `q != 1 && *seq < want` (verify.cpp, session.cpp) therefore
fired on any ring that took more than 2 ms and reported "never rang (graph finished)" while the graph was still running.
With every expert resident the host does not wait per layer (STRATA_VERIFY_NO_HOST), so the path measured on the B70 never
reached it. With host-mirrored experts the first layer whose GPU time passes 2 ms does: layer 15 of the Coder on one Arc Pro
B60, every time.
q is now 0 when the queue is empty and 1 while it is running, as the guard and the trace comment ("1 = still running") expect.
fixups.py gets the same rewrite (entry 8), so a fresh dpct run of session.cpp produces it too.
Measured on one Arc Pro B60 (4,042 of 12,288 Coder IQ1_M experts mirrored in RAM, 128 greedy tokens, AOT bmg-g21): before,
exit 1 at layer 15 in 5 of 5 tries; after, exit 0 in 7 of 7 runs, at 11.2-11.9 tok/s with coherent text.
The sycl port went in as 5ffcce4 ("sycl: port 0.1.39"). Two engine-side changes
were merged into the release branch after it (4f5e6c8, f5410ae) and nothing in
sycl/ was updated for them, so the v0.1.39 tag does not compile:
- #626 (5a8b292) made the thread affinity a value: pin_current_thread()
returns strata::kernels::cpu::ThreadAffinity and restore_thread_affinity()
takes it. sycl/include/strata/core/session.hpp still stored a `long long`
(no conversion exists, and it did not include the header that names the
type); sycl/src/core/session.cpp still assigned the returned struct to it
and passed the `long long` back.
- #559 (9461f5e) added the two layer-range arguments to NativeDense::load.
The sycl out-of-line definition was still the 4-argument one and no longer
matched the declaration in include/strata/core/native_dense.hpp.
Both are fixed by mirroring the CUDA sources: the same member type, the same
`outside()` predicate, and the same place it is applied in the tensor loop.
`pinned` now follows CUDA as well: it is pinned_core.valid, so a pin that
failed is not "restored". Nothing changes when the pin succeeds.
No caller passes layer_lo/layer_hi yet, so `outside()` is never true today:
this restores the build and does not change what the engine computes.
Measured here: Arch Linux, Intel oneAPI DPC++ 2026.0.0 + oneMKL 2026.0,
SPIR-V (JIT), cmake -S sycl -B build-sycl -G Ninja -DCMAKE_C_COMPILER=icx
-DCMAKE_CXX_COMPILER=icpx, then
`cmake --build build-sycl --target strata -- -k 0` -> 0 errors, links. The two
errors are a type and a declaration mismatch, so the compiler version does not
matter.
Not built here: the other targets (kernel parity tests, benches) and the AOT
path. Not run on an Arc in this change.
The full checkpoint stores the table as BF16 (320-byte rows, 102.4 GB), the source of record; the engine gets it as
IQ4_NL (about 8% off per row) or FP8. A bfloat16 is the top half of a float32, so widening is a shift: exact for
every value, no rounding and no scale. One entry in the PLE format list (type BF16, self-describing: no scale, no
metadata) and bf16_dequant_row; the direct reader takes the 320-byte row like any other width, and
PLE_ROW_BYTES_MAX is 320.
ple_reader_test checks the widening is exact over the patterns a lossy path breaks on (subnormals, +-inf, both NaNs);
the generic every-format test covers a synthetic BF16 GGUF through both readers.
Not in this commit: the PR's rename of tools/ple_fp8_pack.py (nothing needs it) and any setup option. Whether a
table format other than IQ4_NL is worth offering is a quality question (see the measurement in the 0.1.40 notes).
Ported by the maintainer from constantindjonkam's PR #586.
A plain llama-quantize Q4_0 file of this model (bartowski's Qwen3.8-Flash-Next-Q4_0 and
Swift-1.5 Q4_0: gate/up Q4_0, down Q4_0 on 42 layers and Q4_1 on 6, a Q4_0 PLE table and
token embedding) was refused at start: no GPU kernels for Q4_0/Q4_1 experts, and the PLE
reader took only IQ4_NL, Q5_0 and FP8. Follows #222 (Q4 to Q6 quantizations).
- Q4_0: Fmt<2> with the codes centred before dp4a (q - 8 as signed bytes), the exact
integer form ggml-cpu's q4_0 x q8_0 dot uses, rather than llama.cpp's CUDA chain
(uncentred codes minus 8 x the q8_1 float sum): on a real Q4_0 expert the CUDA chain
measured 1.4e-2 relative error against the float product, the centred form 5.3e-3.
- Q4_1: Fmt<3>, the Q5_1 chain without the fifth bit and the same min-term choice.
- dq_q4_0 / dq_q4_1 for the prompt path's dequant fallback; row bytes, is_iq and the
GU/D/MMVQ format lists.
- PLE: Q4_0 rows (90 B, as IQ4_NL) through the same reader: one entry in the PLE format list (ple_format_test and
ple_reader_test cover it).
- docs/UNSLOTH_Q4.md: a Q4_0 section with the measured run.
native_expert_parity --synthetic (now in the CMake matrix): q4_0/q4_0 CPU 1.29e-2, GPU
1.29e-2, CPU-GPU 1.1e-7; q4_0/q4_1 CPU-GPU 3.2e-4; GPU dequant bit-exact against
to_float. The existing pairs are unchanged. On bartowski's file (revision 928589f), layers
0 and 5 (Q4_1 down) and 20 and 47: CPU and GPU 1.1-1.3e-2, CPU-GPU 1.1e-7 (Q4_0 down) and
5e-4 to 1.1e-3 (Q4_1 down). Served on one RTX 3060 (all experts in RAM, 131K context):
24,461-token prompt 787 tok/s read, 24.0 tok/s out; 67,934 tokens 737 / 26.4, the hidden
fact found.
Ported onto the format table by the maintainer (the PLE part of the PR is the Q4_0 table entry).
mlock alone brings a cold file in one page at a time from one thread, ~0.47 GiB/s (about two minutes for a 54 GB
Q8_0 table); 16 threads reach ~2 GiB/s on anon761's machine. The pages are then resident and mlock only pins them.
When mlock fails, the pages stay faulted in (the message says they may be reclaimed) instead of being touched again
one by one.
ple_reader_test gets --make-table PATH FORMAT (a big synthetic table, with --rows) and --ram-time PATH (open mapped
and locked, print the seconds) to time it cold.
Ported by the maintainer from anon761's PR #651.
Co-authored-by: ksanislo <[email protected]>
PleTable::open compared rows x row_bytes against the file after the 64-bit product could already have wrapped, and
mapped the tensor before checking it fits. Now: refuse a row count that overflows rows x row_bytes and a data
section that starts past the end of the file, check offset and size without an addition that can wrap
(offset > have, need > have - offset), require a single-tensor shard to start at offset 0 and fill the file, and map
the table only after all of that. close() also resets the FP8 scale and the byte counter.
ple_reader_test --selftest writes tables that claim one row too many, one too few and a wrapping row count, for every
format, and checks both readers refuse each with a message and stay closed.
Checks taken from CC-David-CC's #865 (its opt-in Q8_0 table is #651's; the direct reader reads Q8_0, so the
experimental switch and the mapped-only limit are not needed).
Co-authored-by: CC-David-CC <[email protected]>
Ordinary GGUFs ship per_layer_token_embd.weight in other formats than IQ4_NL: Unsloth's UD-Q6_K_XL and
Swift-1.5 Q4_K_L as Q8_0 (170-byte rows, ~54 GB), a Q5_K_M finetune as Q5_1 (120-byte rows). The engine refused
both at start.
- dequantize_q5_1 in strata/artifact/dequant.hpp (ggml's dequantize_row_q5_1: unsigned 5-bit codes times d plus the
block minimum m); Q8_0 rows use the existing dequantize_q8_0
- two entries in the PLE format list (the table-driven commit before this one): Q5_1 and Q8_0, on both readers;
PLE_ROW_BYTES_MAX grows to 170
- tests: dequant_q5_1_test (synthetic blocks against ggml, no model, no GPU), dequant_bf16_test knows Q5_1,
ple_q5_parity takes Q5_0, Q5_1 or Q8_0 tables and checks both the mapped and the direct reader
Ported onto the format table by the maintainer from anon761's PR #651 (the --ple-io ram fault-in is the next
commit). Quality of the Q8_0 table: ksanislo measured 0.53% error against the BF16 table, FP8 2.66%, IQ4_NL 7.59%.
Co-authored-by: ksanislo <[email protected]>
PleTable had one bool per format (q5_0, fp8) and an if-chain in decode(), open() and format(). Now one list,
ple_formats(), holds one entry per table type: its GGUF type name, the bytes of a 160-value row, whether it
carries a scale and its dequantizer. open(), format(), the row size, the error string and the row buffers all
read it, so the next formats (Q8_0, Q5_1, Q4_0, BF16) are one entry each.
No behaviour change: IQ4_NL, Q5_0 and FP8 decode with the same functions, and the "not IQ4_NL, Q5_0 or FP8 (I8)"
error text is built from the list and reads the same.
ple_reader_test --selftest now writes a PLE-only GGUF for every format in the list and checks both readers
(direct and mapped) against the format's own dequantizer, for single rows and a 16-row token gather.
A prompt loan, a VRAM shrink or an adaptive swap marks experts -1 in the residency table. The zero-doorbell graph
planned from the device table and returned silently on a -1, so the window ran a stale plan (!!!! at 100% residency).
refresh_ar() checks the host table before each window and runs the doorbell graph (its own captured variant) while any
expert is out; resident_plan also writes a mapped error flag the host checks after the window.
Every upload of the expert residency table (host_res -> d_res, and each stage's) was a plain cudaMemcpy: a pageable
copy on the legacy stream, which may return before its DMA has landed, while the kernels that read the table run on
streams created cudaStreamNonBlocking, which do not wait for the legacy stream. A kernel enqueued right after could
read a table half old, half new: an expert computed by both the GPU and the CPU, or by neither. The ~100 KB copy
lands in microseconds, so this rarely fires, but nothing ordered it. The uploads now go through res_put(), which
waits for the copy on the current device's legacy stream only. Same values, same output.
With an engine that keeps --resident-experts on a layer split, setup no longer
steers a low-RAM PC with two cards to one of them:
- at install, the low-RAM mode on several GPUs takes the resident variant by
the same rule as on one card (what no card holds fits the RAM with
LOW_RAM_HEADROOM_GB beside it), without the one-card question;
- at a start, a resident config on several GPUs keeps --resident-experts
(split_mmap leaves it alone);
- a resident config started on a PC with two cards is offered both, "y" by
default, as any other config.
It is gated on MIN_ENGINE >= RESIDENT_SPLIT_ENGINE (0.1.40 here, a guess at
the release that would carry the engine change), so until then setup behaves
as before and the golden configs are unchanged. Two tests run the new paths
with MIN_ENGINE patched; tools/test_setup_*.py pass (260 tests).
--resident-experts (the low-RAM mode's resident variant) ran as plain
--mmap-experts as soon as a layer split was set. It now keeps its RAM copy on
a split:
- the experts every stage's cache holds are left out of the RAM copy, not only
CUDA0's;
- when the rest does not fit whole (#467), the copy keeps the hottest by the
whole expert profile. Each stage only keeps its own layers' share of the
profile, and ranked by CUDA0's share alone the copy held layers 0..K-1's
experts and read every later layer's from the files (4.0 GiB in RAM instead
of 21.2 on the PC below);
- an adaptive swap copies the evicted expert back into RAM from the cache of
the card that owns its layer, on that card's stream. host_res holds slot
numbers of whichever cache owns the layer: reading a later stage's slot out
of CUDA0's cache put another expert's bytes in RAM, and after the first
swaps the answers turned to garbage. (Latent until now: the guard kept the
mode off a split.)
- the decode profile prints every stage's GPU profile, not only CUDA0's.
Remote expert caches (--expert-cache-remote) are still refused with it.
Swift 1.5 IQ3_XXS at 160K, q4_0 KV, RTX 4060 Ti (layers 0-19) + RTX 5080
(20-47), i9-14900KF, 32 GB of RAM, Windows 11, four greedy prompts, decode
tok/s (first round / fourth):
- 5080 alone, --resident-experts: 25 / 29
- split, --mmap-experts (what --resident-experts became): 32 / 64
- split, --resident-experts (22 GiB of experts locked in RAM): 71 / 69
With --pcie-frac 0 --adapt-every 0 the split's greedy output is the same as
--mmap-experts' (4/4 prompts). With adaptive swaps on, 33 varied requests at
temperature 0.6 (thinking on and off, five languages, code, each checked for
degenerate output) exchanged 81,783 experts with the two caches and gave no
degenerate answer.
--vram-reserve-mib is every card's reserve, but the card that drives the
monitors needs more headroom than one that drives none: one value has to fit
the display card, and the other cards then keep that headroom for nothing.
--vram-reserve-later-mib N is the reserve of every card after the first
(default: --vram-reserve-mib's value), in the split search and in each later
card's cache sizing.
RTX 4060 Ti (no display, layers 0-19) + RTX 5080 (display, 20-47), Swift 1.5
IQ3_XXS at 160K: --vram-reserve-mib 1000 gives the 4060 Ti's cache 4551 slots;
--vram-reserve-mib 300 --vram-reserve-later-mib 1800 gives it 4867, and the
5080 keeps 1.3 GB free with everything loaded.
0.1.39 clamped two q8_1 activation quantizers (#606: an fp16 sum or scale past 65504 rounds to inf, the dot products
then read inf * 0 = NaN, and a NaN in the residual ends as one token repeated). Two more store the block unclamped:
- native_swiglu_quantize_q8_1_kernel (native_mmvq.cu), the shared expert's fused SwiGLU + q8_1, on by default
(STRATA_FUSED_SWIGLU_Q81) in decode and the multi-token pass;
- native_gu_fused_kernel (iq_kernels.cu), the routed experts' fused SwiGLU + q8_1 (gfx906 builds, STRATA_EXP_MODE=8).
Both now go through q8_1_finite / q8_1_quant / q8_1_ds as their siblings do: every finite block is stored bit for bit
as before. iq_multi_parity's #606 check covers the shared expert's one (gate = 32, up = x / 32: its activations are x
exactly). rwkeyes found the first one (#606).
With the image encoder on the GPU, the 700 MiB reserve can leave ~200 MiB free on a 12 GB card (UD-IQ4_XS
end-to-end check on an RTX 5070). Setup prints a one-line recommendation; the written config is unchanged.
- "experimental" is per model now: MODELS["UD-Q4_K_XL"] keeps it, the unsloth family and UD-IQ4_XS drop it. The
first menu lists the family without [experimental], with a one-line about (UD-IQ4_XS: a 94 GB download; under
~80 GB of RAM part of its experts are read from the SSD). Its size menu lists UD-IQ4_XS first (the default) and
UD-Q4_K_XL second; an experimental size always sorts last. --model names are unchanged.
- UD-IQ4_XS: no EXPERIMENTAL warning, --check verdict without it. Images are asked (off by default), per model: the
engine's image path has no restriction for this pack, which uses the original model's image encoder; not yet run
with images. UD-Q4_K_XL keeps images off.
- MCP: the fallback tables mirror setup's (per-model experimental/vision), sizes in setup's order, per-size
"experimental" and "images", the vision check per model.
- Docs: README and its six translations, MODELS, UNSLOTH_Q4, AI_SETUP, MCP_SERVER.
- Tests: the menus, the default, --model by name, images, --check, the pack/tokenizer paths; the golden configs
are unchanged.
- a prompt that fits 0.1.39's auto chunk keeps 0.1.39's ring (Prefill::set_ring_budget's small_max): 4K prompts
were 1-17% slower with the smaller ring (Coder -15%, IQ3_S -5%, IQ3_XXS -2%; 5 interleaved pairs each);
- ring slots are given up for a bigger chunk only where 0.1.39's rule held the chunk under 6144 (ring_cap_for);
from 6144 on the ring keeps its 0.1.39 size (the Coder 6144/384 -> 7936/199 was -12% at 32K);
- the unpinned arm's cap stays 96 (the byte budget made it ~49 on IQ3_S);
- the scan sets its own ring budget instead of set_ring_override, so a layer split's 96-slot override (#340) is no
longer clobbered by the scan.
RTX 5070 vs STRATA_RING_BYTES=0: 4K -0.7..+1.1% (bit-identical); 32K IQ3_XXS +18.5% (cache 1500) / +9.4%, Coder
+3.1%, IQ3_S +0.2%.
Added where a config goes to several GPUs (setup --gpus / a multi-GPU install, start --gpus, "use both" at
start). Single-GPU configs are not touched. Opt-out: setup --no-remote-expert-opt, or "remote_expert_opt": false
in the config (a user key, kept on re-runs). The engine uses it only with a helper cache (--expert-cache-device1..3),
so a layer split runs as before. tools/test_setup_remote_opt.py; the 2x16 GB golden configs gain the flag.
Niko1221