With the image encoder on the GPU, the 700 MiB reserve can leave ~200 MiB free on a 12 GB card (UD-IQ4_XS
end-to-end check on an RTX 5070). Setup prints a one-line recommendation; the written config is unchanged.
- "experimental" is per model now: MODELS["UD-Q4_K_XL"] keeps it, the unsloth family and UD-IQ4_XS drop it. The
first menu lists the family without [experimental], with a one-line about (UD-IQ4_XS: a 94 GB download; under
~80 GB of RAM part of its experts are read from the SSD). Its size menu lists UD-IQ4_XS first (the default) and
UD-Q4_K_XL second; an experimental size always sorts last. --model names are unchanged.
- UD-IQ4_XS: no EXPERIMENTAL warning, --check verdict without it. Images are asked (off by default), per model: the
engine's image path has no restriction for this pack, which uses the original model's image encoder; not yet run
with images. UD-Q4_K_XL keeps images off.
- MCP: the fallback tables mirror setup's (per-model experimental/vision), sizes in setup's order, per-size
"experimental" and "images", the vision check per model.
- Docs: README and its six translations, MODELS, UNSLOTH_Q4, AI_SETUP, MCP_SERVER.
- Tests: the menus, the default, --model by name, images, --check, the pack/tokenizer paths; the golden configs
are unchanged.
- a prompt that fits 0.1.39's auto chunk keeps 0.1.39's ring (Prefill::set_ring_budget's small_max): 4K prompts
were 1-17% slower with the smaller ring (Coder -15%, IQ3_S -5%, IQ3_XXS -2%; 5 interleaved pairs each);
- ring slots are given up for a bigger chunk only where 0.1.39's rule held the chunk under 6144 (ring_cap_for);
from 6144 on the ring keeps its 0.1.39 size (the Coder 6144/384 -> 7936/199 was -12% at 32K);
- the unpinned arm's cap stays 96 (the byte budget made it ~49 on IQ3_S);
- the scan sets its own ring budget instead of set_ring_override, so a layer split's 96-slot override (#340) is no
longer clobbered by the scan.
RTX 5070 vs STRATA_RING_BYTES=0: 4K -0.7..+1.1% (bit-identical); 32K IQ3_XXS +18.5% (cache 1500) / +9.4%, Coder
+3.1%, IQ3_S +0.2%.
Added where a config goes to several GPUs (setup --gpus / a multi-GPU install, start --gpus, "use both" at
start). Single-GPU configs are not touched. Opt-out: setup --no-remote-expert-opt, or "remote_expert_opt": false
in the config (a user key, kept on re-runs). The engine uses it only with a helper cache (--expert-cache-device1..3),
so a layer split runs as before. tools/test_setup_remote_opt.py; the 2x16 GB golden configs gain the flag.
`--prefill auto` walked a fixed list of chunk sizes - 32768, 16384, 8192,
6144, ... - and took the first that fitted, with the ring a fixed 384 slots
taken off the top before the chunk was ever considered. Both halves of
that were wrong on a rig like this one.
The list is coarse exactly where it matters. `bytes_needed` is a sum of
(T x positive constant) terms plus a max of such sums, so it rises
monotonically with T, and so does every test the scan applies - which makes
the largest chunk that fits a bisection on the 256-token grid the prompt
path already works on. Seven probes against the list's ten, at one
`bytes_needed` each. It matters because this rig affords ~8,700 tokens and
was handed 8192; 8704 is not on the list.
The order was wrong because chunk and ring spend the same borrowed VRAM,
and a ring slot is worth far more than a chunk token. Measured here (2x
RTX 3060 + 2x RTX 5060, CUDA3 lending at its 90% cap, 120K prompt): 8960
tokens with the 17-slot ring that leaves reads at 963 tok/s, 8192/130 at
1,008, 7168 with the ring full at its 199-slot byte budget at 1,000, 5632/199
at 915. A ring slot is worth ~0.53 tok/s and a chunk token ~0.05, so the 69
slots between a full ring and 8192's 130 are worth more than the 512 chunk
tokens they cost - and once the ring IS full, a smaller chunk buys nothing.
So: the largest chunk that still leaves the ring full, and only a rig where
no chunk can afford one falls back to the old rule with a kRingMin floor.
The room is counted in BYTES, not slots. A ring slot is max_blob, but a
cache slot holds its own layer's blob - 2.15 MiB against a 2.54 MiB max_blob
here. In slots the ring looked 18% cheaper than it is, a chunk whose ring
fitted only after that discount was refused outright, and the scan stopped a
step short. `bytes_from_slots` is the inverse of the existing
`slots_from_bytes`, and `cache_slots_for` factors the latter out for the
serve path's per-participant version. Both inverses clamp at k <= 0: the
lend budget is `min(slots - 128, pct * slots / 100)` and goes negative on a
cache under 128 slots, where `slots - k` is off_[slots] - the end of the
prefix sum, so that cache lends 0 bytes rather than reading past its offsets.
Upstream's per-participant `fits`/`fits_one` and the #448 diagnostic are kept:
the ring room is taken over the smallest participant, `alone` is the chunk
CUDA0 alone would read in, and the small-card warning is unchanged.
Verified on the rig: the scan picks 7680/199 against the old rule's 8960/17,
1018.8 and 1021.1 tok/s on two cold 120K prompts against 963.1 - +5.9%.
Q8_0 picks 5632/101, its 506 MiB budget over a 5,222,400 B blob.
0.1.39b (Niko1221): behind STRATA_RING_BYTES=1 with the byte-budget ring; the default keeps 0.1.39's chunk list
(plan_lend and the serve scan), its log line gains the ring. The INFO fields (the Monitor commit) are not taken.
Default: 0.1.39's 384/1024-slot ring and bytes_needed count, so the default loan and output stay as they were.
The auto chunk scan (the PR's second commit) is not taken.
On a card whose experts mostly run on the CPU the slots cost speed per request (RTX 5070, Q2_0: 11-24 %), so setup
now recommends slots only when the expert cache (every card of a split counted) still holds half of the model's
experts beside them (Q2_0: 24 GB and up, or a split); elsewhere it stays at one at a time and says "parallel N
reduces waiting for several users but costs about 10-25% speed per request on this card". --parallel N is
honoured as asked, as before. BATCHING.md: the table with the change against one at a time.
The ring was 384 slots - a number measured on Q2_0, where a slot is one
expert blob of 1,382,400 B, so 506 MiB. A slot is a whole blob, so on a
pack with bigger blobs the same 384 slots are a different amount of
memory: 1,912 MiB on Q8_0, more than half of an 8 GB card's expert cache,
and what kept that card at a 1024-token chunk when its own buffers would
have fitted 6144. Measured on the 4-way rig, one env var and nothing
else: chunk 1024 -> 6144, prefill 87 -> 402 tok/s.
So the budget is the constant and the slot count is derived from the
pack's max blob: 384 on Q2_0 (pinned) or 96 (not), exactly as before, and
the pack it was tuned on does not move. The fused rule (1024 slots, #136)
is preserved the same way, and `ring_cap()` still bounds the result. This
is also what lets the auto chunk scan treat "the ring is full" as a plain
byte comparison.
Also exports the three accessors the scan needs: bytes_needed_no_ring
(the chunk's own buffers, without the ring), ring_max_slots (the cap in
slots) and ring_slots_for (what the ring resolves to for a chunk, after
the override, the env var and the pinned-share rule - what `init` lays out
and what the INFO line reports).
And one fix on the way: `bytes_needed` counted `carve`'s buffers from
memory and got two wrong - it took `xn` unconditionally, though `carve`
takes it only under STRATA_GR_UNFUSED, and it never took `grs`. Net
T*(D-HC)*4 bytes too many: 42 MB at a 1024-token chunk. The direction is
safe - the prompt path was told it had less room than it did - but it
under-sizes every loan.
server: a request decoding in a slot whose neighbours finished (no other slot busy, nobody waiting, at least 32
tokens allowed) is BSTOPped and continued on the solo path with its MTP drafts - the engine copies the slot's
sessions back (INFO slot_cache=1; at most twice per request; STRATA_PARALLEL_SOLO=0 keeps it in the slot).
engine: INFO batch_slots=N slot_cache=0|1.
tools/batch_interleave_test.py: the back-to-solo path, and a next turn from a slot's turn checkpoint (a client
that drops the reply's thinking).
docs: "parallel": N, what a slot costs and what setup recommends, the server's handling (interleaved prompt reads,
BYIELD, slot conversation cache, back to solo), exactness and its settings, the RTX 5070 measurements (the slots
buy waiting time, not speed, on a 12 GB card), the protocol; DETAILS / README / AI_SETUP mention the option.
INSTALL.md gets the section (the build setup compiles, what differs, the forced-path numbers from a
Ryzen 5 7600 + RTX 5070, what is untested); the CPU rows of INSTALL.md / DETAILS.md and AI_SETUP.md
point to it.
batch_test.py: the engine log in the temp folder, the CUDA DLLs on PATH, ASCII output, --mt-min (empty: the
engine's default kernels, for the invariance check without the exact settings). server: /metrics live.waiting
and live.outside_slots instead of a guessed count.
Windows: pool workers pin with SetThreadGroupAffinity (the processor group was dropped: a 3990X's 63 workers
collapsed to 32 group-local masks); the host thread selects a CPU Set and restores the previous selection.
Linux: the complete, dynamically sized affinity mask is kept and restored (CPU IDs past 63).
pool_affinity_test (Windows) and pool_affinity_linux_test. Squashed from the PR's 8 commits (net diff).
The slots' sessions may take a fifth of the expert cache (~VRAM less 5 GB), up to 4: 2 on 12 GB, 3 on 16 GB, 4
from 24 GB, none below. Without --parallel setup only says it once; with it the config gets "parallel": N and a
count above the recommendation is kept with a note.
- "parallel": N in the run config passes --batch N (a value the engine cannot use is said and passed on: the
engine warns and adjusts); the server uses the slot count the engine reports (INFO batch_slots)
- every running request keeps its own status, rate window and engine figures (history rows are their own);
/metrics live.slots shows each slot (idle with the tokens it holds, reading, decoding with tok/s),
/v1/status concurrency.serving says N
- a conversation's next turn goes to the slot that holds its start, a new one to an empty or the least recently
used slot; a slot left at a stop token keeps what it holds
- a long prompt read (solo or an admission) gives way (BYIELD) to a waiting request with a prompt under half as
long, at most twice; the part read waits in a reserved slot and the request goes on once the other is in
- the engine start and POST /v1/vram are serialized against parallel requests
- serve/test_parallel.py: a scripted engine behind the real StrataEngine and Service
- --slots N is the same as --batch N; a count the engine cannot run (no --serve, 1, more than 8, a split on one
GPU, sessions that do not fit, --batch-groups that do not divide) is a warning and fewer slots or none, never a
refusal; INFO says batch_slots=N when slots run
- a batch window runs over the active slots only (Verifier::run_slot_rows, graphs per row list): an idle slot is
not written, so it keeps the conversation it served
- a request whose prompt starts with what an idle slot holds (all of it, or its turn checkpoint) continues from
the slot's sessions instead of reading the history again
- with slots, a prompt is read one prompt chunk at a time (the same chunks as one run) and the slots decode
between two chunks for half as long as the chunk took (STRATA_BATCH_DECODE_SHARE)
- BYIELD <slot>: the prompt being read stops at its next chunk boundary, the part read is copied into the slot,
YIELDED <slot> <tokens> + DONE cancel; the same request sent again goes on from there with the same chunks
and checkpoints (#656's cooperative preemption, a slot as the park)
Default (no --batch) unchanged.
PR #638 was cut from 0.1.38. 0.1.39's segmented expert cache (#533) keeps its cuMem* code out of HIP builds through
STRATA_USE_HIP, which the gfx906 compat build does not define, so it included <cuda.h>. Its virtual-memory blocks are
now behind STRATA_EC_NO_VMM (STRATA_USE_HIP or STRATA_HIP_GFX906); the CUDA and RDNA preprocessor output is unchanged.
(std::max) keeps windows.h's max macro out of two Windows-only lines (found by a Windows gfx906 compile check).
- layer split: commit stays synchronous on a split (set_commit_async(!multi_gpu), the `|| next_ != nullptr`),
no pre-staging of the next stage's inputs, kDrafterMib 1000 - the split's sizing and order as in 0.1.39
(untested here: one GPU);
- the CPU expert kernels keep cpu_avx512_ok() (F/BW/VL/VNNI/VBMI): #646's AVX-512BW-only check would send a
CPU without VNNI/VBMI to kernels that use them;
- resident_plan_kernel: one thread per window entry up to 128 (64 lost a 7- or 8-token window's last entries
on the all-resident path);
- the host keeps zeroing the GPU's rows under STRATA_VERIFY_DEVICE_PLAN=1 (copy_or_zero_from_mapped copies
every row there).
dpct left the DPCT1106 note but dropped cudaMemGetInfo in generate.cpp's
stage_room, so every later stage of --layer-split sized its expert cache from
0 bytes free and the second card crashed with 'expert cache: no room'.
Reported with the fix by tmking01 in #423 (2x Arc Pro B70); also a fixups.py
entry so a re-migration keeps it.
The SYCL port of PR #423 tracked 0.1.38; 13 of its migrated copies were behind
0.1.39's sources. Re-migrated 0.1.38 and 0.1.39 with sycl/tools/migrate.sh,
normalized, merged with sycl/tools/merge_upstream.py:
- clean merges: expert_cache, expert_source, native_dense, pinned, elementwise,
kv_stream, iq_multi_parity, prefill, verify.hpp; q8_1_finite.hpp new (#606)
- verify.cpp: the #649 trace kept with the port's queue handling
(STRATA_VERIFY_TRACE; the HIP-only coherent/doorbell switches stay HIP-only)
- generate.cpp: the port's --prefill-borrow beside --vram-elastic/--vram-segment-mib
- iq_kernels / native_mmvq: the port's kernels kept, #606's finite q8_1 scale
and sum applied by hand (dpct's conflicts there were kernel-name hashes)
- expert_cache: --vram-elastic (#533) is CUDA-only on the port too (no driver
VMM on Level Zero through dpct); it reports so instead of failing to build
- version 0.1.39
Compiled here with icpx 2026.1.1 (SPIR-V, no Intel GPU on this PC): strata
builds. Not run on an Arc.
docs/OLDER_GPUS.md: Pascal / Volta through the CUDA 12 engine (#395#600#540#655), RTX 20's STRATA_BF16_TC opt-in,
AMD gfx1031 (#524), gfx1012 (#442) and gfx906 (#638, #677) with the reporters' numbers; README, AI_SETUP, AMD_HIP and
NVIDIA_V100 point to it.
Random BF16 rows of the router's shape [512 x 2560], 1..8 tokens: both within 3.3e-7 of the double
sums on a Ryzen 5 7600 (0.53 ms and 0.50 ms per layer for 4 tokens). Two ctest entries, the second with
STRATA_FORCE_ISA=avx (the AVX-only CPU's path). Test program only.
CUDA 13 dropped Pascal and Volta. A model whose GPUs include one (its oldest card decides: one engine runs per model)
now runs the experimental CUDA 12 engine in engine-cuda12/ - the ready-made strata-windows-x64-cuda12.zip with NVIDIA's
CUDA 12 pip libraries, or compiled here with a CUDA 12.x toolkit (Linux; STRATA_NVCC picks one, #601) - and setup says
why. engine/ and every model on it stay as they were.
- the cards are an opt-in: naming one (--gpu / --gpus), --cuda 12, STRATA_EXPERIMENTAL_SM60=1, or a PC with no newer
NVIDIA card and no supported AMD card; a PC with a newer card keeps recommending it
- --cuda 12|13 (or STRATA_CUDA) overrides the choice and is kept, with a warning where it cannot work (recommend,
never force); --cuda 12 also runs with a driver older than 580 (525+), which the stop now mentions
- at a start, a Pascal / Volta card added to a CUDA 13 model moves that model's config to the CUDA 12 engine
- updates look at both engine folders; a compiled CUDA 12 engine builds in build-cuda12/
- an RTX 50 card in a CUDA 12 engine is a warning (CUDA 12.8+ builds it; #220 crashed with 12.8), no longer a stop
The setup golden test (every recommended config on five PCs) is unchanged; tools/test_setup_older_gpus.py covers the rest.
- CMake: STRATA_ISA_FLOOR (empty by default) builds ggml-cpu for AVX (Sandy/Ivy Bridge) or SSE4.2
(Nehalem/Westmere) instead of the build host or AVX2; every other ggml ISA option off. Strata's own
AVX2/AVX-512 kernels stay in the binary behind their CPU checks. Excludes STRATA_PORTABLE. The ggml
block follows the Strata_Dirigo fork's (rwkeyes).
- STRATA_FORCE_ISA=avx2|avx|sse caps the engine's CPU probes, so the older-CPU paths run on any PC
(how this was tested here: no old CPU). cpu_sse42_ok(), isa_floor_build().
- The engine: a normal build still refuses a CPU without AVX2, now pointing at the experimental build;
a floor build checks its own floor (AVX, or SSE4.2 + POPCNT) and warns that it is experimental and
slow (or, on an AVX2 CPU, that the normal build is faster).
- setup: a CPU without AVX2 warns ("experimental, slow") and compiles the engine on the PC with the
floor it has, instead of failing (#595's idea); the prebuilt AVX2 engine is not used there. Recorded
in BUILD.json; a build folder for another floor is configured afresh. STRATA_ISA_FLOOR=avx|none asks
for it on any PC. Windows + AMD has no source build: a clear stop. The MCP server's recommendation
notes it instead of refusing.
- tools/test_setup_oldcpu.py.
Default build: nothing changes (no new CMake definitions, the probes answer as before without the env).
The two IQ3 gate/up formats decode each 32-value half with eight scalar loads
assembled by _mm256_set_epi32. A scratch build that substitutes a constant for
the grid vector shows that assembly is ~half the single-core kernel (IQ3_XXS
248 -> 130 us, IQ3_S 357 -> 140 us), and the two formats are ~58% of the
gate/up lookup work on the Swift 1.5 IQ3_XXS pack.
Add an opt-in AVX2 gather for IQ3_XXS (iq3xxs_grid) and IQ3_S (iq3s_grid),
the mirror of the AVX-512 kernels' STRATA_IQ_GATHER. On the i7-12850HX
(Alder Lake, no AVX-512) it is bit-identical - a direct scalar-vs-gather output
dump matches for all five gate/up formats, native_expert_parity stays green -
and 1.32-1.34x on the two formats isolated, +1.0% serve end to end (on >= off in
5 of 5 interleaved pairs). The pool is flat in the deterministic --stats (the
15-worker pool is not decode-bound), so it is a per-core win that compounds.
Default off, like the AVX-512 one: AVX2 gather throughput is microarchitecture-
dependent. IQ2_XXS / IQ2_XS / IQ2_S use 64-bit grids, where an AVX2 i64 gather
measured a wash, and are unchanged. The AVX-512 path is untouched.
Without a hugetlb pool the Linux arena fell back to a plain anonymous
mapping with 4 KB pages. The CPU expert pool streams whole experts (about
3 MB each for the 4-5 bit GGUFs) out of that arena, so every expert costs
~750 TLB misses.
Map the fallback 2 MiB-aligned and madvise(MADV_HUGEPAGE), so THP in
"madvise" mode (the common distro default) backs it. STRATA_NO_LARGEPAGES=1
keeps 4 KB pages, as on the hugetlb path; the log note says which one it got.
A prompt that fits one chunk runs a layer split's stages one after the other,
so while one card reads its layers the other idles. --peer-device already has
the mechanism for this (PeerPrefill's peer streaming: a share of the experts
the primary streams goes over the peer's own PCIe link and is computed there),
but it needs P2P and is refused with --layer-split.
This makes PeerPrefill usable as a stream-only peer between the stages:
- set_stage_helper: each stage's prompt path gets the next stage's (the last
one the first's); a stream-only PeerPrefill holds no experts (peer == null),
so every use of the peer tier is guarded
- per prompt, bind_stage_helper points it at the idle stage's own lent prompt
buffers (mixed, Xq, GU, H, Hq, Dm, ring, MMQ context): no VRAM of its own
- without P2P the activations, the row table, the group bounds and the result
rows go through mapped host memory, read by copy kernels on the device that
needs them. Copy-engine uploads queued behind the ring's expert blobs (the
grouping tables' reason for mapped memory), which cost ~20 ms per layer and
ate the whole gain; cudaMemcpyPeerAsync without P2P is staged by the driver
- only for one-chunk prompts on the MMQ path (not the fused kernels, not with
--peer-device), with a share that falls with the chunk: min(0.45,
0.5 - T/16384), off below 0.3 (~3.3K tokens) - measured, below
- STRATA_PREFILL_HELP=0 turns it off, STRATA_PREFILL_HELP_FRAC fixes the share
2x RTX 3090 (no P2P), UD-Q4_K_XL, layer split auto, 262K context,
--adapt-swaps 0, prompt tok/s without / with: 1.1K 496 / 723, 1.5K 674 / 833,
2K 878 / 1128, 2.5K 1017 / 1296, 3K 1260 / 1440, 4K 1605 / 1607, 8K 2215 /
2221; decode unchanged. The output is repeatable (same prompt, same answer)
but not bit-identical to STRATA_PREFILL_HELP=0's: the helped rows are
computed in another MMQ grouping.
0.1.39b (Niko1221): opt-in (STRATA_PREFILL_HELP=1), since the helped rows round differently from the default;
merged with #598 (run_impl / next_run_).
Add default-off cache coordination, helper-side weighted reduction and CPU activation quantization on demand. Allow helper cache capacities to be sized automatically at startup.
Port the saved v2.2 implementation onto upstream 0.1.38 without changing the separate peer tier. Document the helper-only scope and current validation limits.
The Coder's router is 256 experts, which the native 512 x 10 router does not take, so a verify window routed each
token with its own projection and top-k. moe_route_window does the window in one multi-column BF16 projection
(bf16_gemv_fp32_mmvf_cols over the existing multi-row kernel) and one n-token top-k; each token's route is bitwise
what moe_route gives it (route_window_parity, new ctest). gfx906 build only, so the CUDA and RDNA paths are
unchanged; STRATA_ROUTE_PER_TOKEN=1 restores the per-token calls.
2x MI50, Coder IQ1_M, 32K, greedy 1,500-token request: verify window 55.5 -> 52.5 ms, generated text identical.
- native_expert_grouped: the AMD layouts handle the IQ packs' gate/up and down types only; Unsloth's Q4_K / Q5_K
gate/up and Q5_1 / Q8_0 down now take the CUDA layout instead of exiting (native_grouped_parity found it).
- fused_gr_read_multi: the K-split gr_down (fixed-order finish) sums in another order than the single-token
fused_gr_read, so it broke gr_parity's multi == single contract; it is now opt-in (STRATA_GR_SPLIT=1). The
latency-hidden norm/up (same sums, same order) run in the default read (STRATA_GR_FAST=0: off).
- mmvq_multi_parity: on gfx906 the wave64 MMVQ is the same exact kernel for every column count, so the negative
control has nothing to find; a second ctest entry runs it on the CUDA layout (STRATA_MMVQ_WAVE=0), where it has
power.
- kv_hybrid_parity: the tensor-core prompt attention refuses on gfx906 as on the RDNA backend.
ctest on 2x MI50 (gfx906): 34 of 34 pass (iq_parity and ple_parity need fixture files and are not counted).
The HIP backend (STRATA_ENABLE_HIP) is wave32 RDNA only, and the engine refuses a wave64 card. This adds a
separate opt-in build for gfx906: the CUDA sources compiled as HIP through a small compat layer, with a CUDA warp
as a logical half of the 64-lane wavefront. With the option off nothing changes for CUDA or the RDNA backend;
every new code path is under STRATA_HIP_GFX906.
Build: -DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906 (implies STRATA_ENABLE_CUDA's targets, refuses
STRATA_ENABLE_HIP). The CUDA runtime/cuBLAS names come from include/strata/platform/hip_compat/ (first on the
include path), CUDA::cudart / CUDA::cublas become aliases of hip::host / roc::hipblas, every .cu (and the ggml MMQ
sources, with GGML_USE_HIP) is compiled as HIP. The native kernels' --use_fast_math (an nvcc flag) is not passed.
Correctness fixes found on the way (each one changed results or hung, not only speed):
- __fmul_rn / __fadd_rn / __fsub_rn contracted into FMA under HIP; they no longer do (as on NVIDIA) - the
elementwise and quantize_act parity tests pass bitwise;
- the device check of the gfx906 build reads the card's gcnArchName (the wave32 check is the RDNA backend's);
- ggml MMQ got NVIDIA compute capability 9.0 for gfx906 and hung the prompt path; it gets the AMD arch code;
- __byte_perm is a scratch-memory array in HIP: the IQ4 table lookup uses llama.cpp's v_perm_b32 sequence
(agent-request decode 29.9 -> 39.2 tok/s on 2x MI50);
- the stage profiler's wall clock runs at 25 MHz on gfx906, not 100 MHz;
- a CUDA toolkit-mismatch check that reads CUDART_VERSION is skipped (no CUDART_VERSION under HIP);
- gdn_rec_parity (cp.async, CUDA occupancy API) is not built for gfx906.
wave64 kernels (gfx906 only; the CUDA layout stays one env switch away for A/B, every one bitwise-checked):
- grouped native experts, STRATA_EXP_MODE=7 (default): gfx906 has no packed byte SIMD, so __vcmpne4/__vsub4
became ~12 word ops each; signs are now dp4a(g ^ m, u) - dp4a(m, u) (exact in integers), grids and the group's
activations sit in LDS, one weight load serves every token of the group. Same k-to-lane order, so the output is
bitwise the old layout's (native_expert_bench, real GGUF rows). Verify window 60.0 -> 51.3 ms.
STRATA_EXP_MODE=2 is the previous layout;
- MMVQ: one wavefront per row with a 64-lane butterfly, no LDS/barrier, same kernel for 1..8 columns
(+2.3% decode; STRATA_MMVQ_WAVE=0: CUDA layout);
- router top-10 in one wavefront (register argmax, no block barriers), bit-identical (router_top10_parity);
- gr_down_multi split along K (8 rows x 5 slices of 2048 per block, fixed-order finish) (+2.9%;
STRATA_GR_SPLIT=0: old);
- hyper-connection read: norm with its loads in flight and xn in registers, up with 8 lanes per row
(51.3 -> 50.2 ms per window; STRATA_GR_FAST=0: old);
- the QSA prompt-attention tensor-core/WMMA kernels are not used on gfx906 (the FP32 kernel runs).
Benches (not ctest; they need a model shard): native_expert_bench (one expert layout against another, bitwise,
with timings) and fused_gr_bench (the fast hyper-connection kernels against the old ones).
The routing-aware prefetch's router dot (bf16_rows_dot_multi) lives in kq_avx2.cpp, so an AVX-only
CPU started fine and died on the first request (SIGILL in vpmovzxwd on a Xeon E5-2687W). It now
takes the AVX2 kernel where the CPU has AVX2, kq_avx1.cpp's AVX dot (mul + add, 128-bit integer
widening) where it has AVX, and the same sums in plain C++ below that. cpu_avx1_ok() probes AVX +
OSXSAVE + the YMM state. An estimate only (which experts to prefetch). A CPU with AVX2 runs exactly
what it ran before.
From rwkeyes' Strata_Dirigo fork (MIT; 0d00a66, e0754d3), with MSVC's /arch:AVX added and
__builtin_memcpy replaced by std::memcpy for MSVC.
#540's attention kernel (bit-exact, +7% prompt on a V100) uses about twice the registers and also runs decode, so it
is not taken for every card: the kernel the ready-made engine runs is restored as it was, and #540's version lives
beside it as attn_chunk_kernel_pre75, compiled only with -DSTRATA_EXPERIMENTAL_SM60=ON and launched on a card below
sm_75 (per device; STRATA_ATTN_PRE75=0 is the A/B arm). The launches pass cap/stride explicitly (the same 0, 0 as
the defaults) so either kernel can be picked through one pointer.
Conflicts with 0.1.39: --trim-stage-weights is now the same switch as STRATA_STAGE_TRIM=1 (PR #639's
implementation, already in 0.1.39); the #533 VRAM command is taken between batch windows only when no slot
is decoding.
STRATA_EXPERIMENTAL_SM60 now reaches every target (add_compile_definitions). prompt_attn_v70_kernel with its launcher,
and Gemm::bf16's fp32 path for Pascal, exist only under it: the ready-made engine (sm_75+) loads no extra kernels for
them, while the experimental build's sm_60/61/70 cards get them.
A native pack's Q2_0 layers (most IQ packs' down projections) ran on Strata's own Q2_0 kernels, which
are AVX2 at least. q2_native_kernels() keeps them there when the CPU has AVX2 and otherwise hands the
layer to ggml-cpu's Q2_0 vec_dot like every other native type (the pool, the dispatcher's activation
quantizer and the parity test). STRATA_KQ256's gate/up kernel needs AVX2 too. native_expert_parity
skips the AVX2 checks on a CPU without AVX2. No change on a CPU with AVX2.
(The engine part of #595; its setup and run-script parts are not in this commit.)
Old path against new kernel on 8.5K / 33K / 99K-token chats, 617 scored positions each, with two controls that change only the rounding
order (BF16 GEMMs without the FP16 detour, another prefill chunk), reruns of both paths (bit-identical), and the kernel's own error on the
model's data (relative 4.4e-7 mean, 4.8e-6 worst row over ~5 M rows).
The experimental sm_70 build (STRATA_EXPERIMENTAL_SM60, #236) had no page. NVIDIA_V100.md says how to build it, which kernel each
part of the prompt path uses on Volta and what the prompt attention kernel was measured at.
bench/results/2026-10-03-v100-prompt-attn: prompt speed with and without the kernel (7K / 28K / 114K tokens), the nsys share of the
attention before and after, the parity numbers, needle recall (15 of 15 with either path) and the greedy-output comparison.
On sm_70 qsa_prompt_attn_batch refused the device and the prompt path fell
back to the decode kernel (FP32 FMAs, one query at a time): 19-25% of the GPU
time of a 32K prompt on a V100.
prompt_attn_v70_kernel is the v1 kernel's structure with the two matrix
products on m8n8k4 (FP16 in, FP32 accumulate): q.k with one dim group (64 dims,
= one int8 scale group) per warp and 8 cells per quad-pair, p.v with 16 dims per
quad-pair. q and p keep their hi + lo FP16 split, the four dim groups' partials
are added in a fixed order (deterministic). Int8, FP16 and K8V4 KV are handled;
Q4_0 KV (mode 4, sm_80+ only upstream) keeps the old kernel. The fragment layout
was measured on a V100 and matches turbomind's SM70_MMA_884.
qsa_prompt_attn_parity (V100): error against FP64 ~5e-6 (output scale ~3.6),
int8 3.0x and fp16 1.3x faster than the FP32 kernel per 2048-query chunk. The
test now skips its Q4_0 cases below sm_80 instead of exiting 2 there.
End to end, 32K prompt, one V100, greedy output identical:
26 s -> 22 s on current main (STRATA_PROMPT_ATTN_OLD=1 is the old path).
PR #655's FP16 tensor-core path for Gemm::bf16, adapted so the ready-made engine stays byte-identical:
- default on for compute capability 7.0-7.4 only (Volta; the experimental STRATA_EXPERIMENTAL_SM60 build), off for
7.5 (RTX 20): STRATA_BF16_TC=1 opts in, =0 turns it off on any 7.x card
- from #540: finite values beyond FP16's range clamp to +-65504; a beta = 1 product (the BF16X2 remainder, in
FP16's subnormal band) keeps cuBLAS; the compute capability is cached per device (a layer split can mix cards)
- from #395: Pascal (6.x; cuBLAS has no BF16 GEMM there) widens both operands to fp32 by an exact shift and runs
cublasSgemm with the same fp32 accumulator, tiled over T in the instance's own buffers
sm_80 and newer, and HIP, run the same cuBLAS / hipBLAS call as before.
Co-Authored-By: Károly Sárdi <[email protected]>
Co-Authored-By: Rafal R@V Prasal <[email protected]>
native_gu_rows() tested cpu_avx512_ok() before the AVX-512 kernel but not cpu_avx2_ok() before the
AVX2 one, and native_down_rows() had the same asymmetry for its two multi-token kernels (kq256 and
the IQ4_NL one). On a CPU without AVX2 a verify window of two or more tokens took an /arch:AVX2 path
and died with 0xC000001D. With the gates it falls through to ggml-cpu's vec_dot, which covers every
native type for any nt. A CPU with AVX2 takes exactly the paths it took before.
(Cherry-picked from #394's 0083bb7 without its generate.cpp part; the startup gate is reworked for the
opt-in STRATA_ISA_FLOOR build in a later commit.)
#391 removed MSVC's BMI2 shlx from the table's constructor; GCC still vectorised the same loop into
AVX2 (vpbroadcastb) and ran it before main() on an AVX-only Xeon E5 (found by rwkeyes' Strata_Dirigo
fork, 6dd37a2, which made it constexpr). constexpr here as well, deriving ksigns_iq2xs[i] (the 7 bits
plus an even-parity bit) so the constant expression does not read a non-constexpr array. Same 128
values; the file now has no runtime initializer.
A namespace-scope static initializer shifted a byte into place with << (8 * k), which is a
variable shift. MSVC compiling this file with /arch:AVX2 emits BMI2 shlx for that - it assumes
anything with AVX2 also has BMI2, true from Haswell on and false of everything before it.
Because the table is static, the constructor runs before main(), so this faulted at PROCESS START:
strata.exe exited 0xC000001D with no output at all, not even for --help. One shlx in the whole
linked binary was enough to make the engine unstartable on any pre-Haswell CPU under MSVC.
Storing the byte directly is the same 64-bit value on the little-endian targets this file is for,
and leaves no variable shift for the compiler to widen.
Found by building the engine for sm_61 on a Sandy Bridge-E (Xeon E5-2680) with
STRATA_EXPERIMENTAL_SM60=ON, which 0.1.31 added (#236). Verified by disassembly: no shlx/shrx/sarx/
mulx/andn/bzhi/pext/pdep anywhere in strata.exe afterwards, and --help now runs.
This is independent of any AVX1 work - it is a plain startup crash on the CPU this was built on.
(cherry picked from commit 3fcaf03840)
Volta has FP16 tensor cores and no BF16 ones as well, so cuBLAS takes the same SIMT path for BF16 products there.
Suggested in review. Not measured on sm_70; STRATA_BF16_TC=0 turns it off.
Turing has no BF16 tensor cores, so cublasGemmEx on CUDA_R_16BF inputs falls back to a SIMT fp32 kernel
(magma_sgemmEx). On an RTX 2080 Ti that was the largest kernel of the prompt path, 20 % of a 4K prompt:
the hyper-connection down / up projections, the router and indexer rows, PLE key / value.
Gemm::bf16 now converts the weight and the activations (in row slices, into the instance's own scratch)
BF16 -> FP16, which is exact for every value inside FP16's range, and runs the existing FP16 GEMM (fp32
accumulate) on the tensor cores. Single-row products keep cuBLAS's GEMV. STRATA_BF16_TC=0/1 overrides;
the default is on for sm_75 only, so other cards run exactly what they ran.
New test gemm_bf16_parity: Gemm::bf16 against cuBLAS's BF16 product over the prompt path's shapes,
within fp32 rounding (worst relative difference 1.7e-5 on sm_75; 0 with the path off).
Qwen3.8-Flash-Next Q2_0, RTX 2080 Ti, i7-6700: prefill 877 / 1134 / 1153 / 1116 / 1009 t/s at
4K / 16K / 32K / 64K / 128K -> 1020 / 1337 / 1366 / 1312 / 1166.
It hung on the B70 until the stager/PLE event-wait fixes; it runs since and reads
Coder prompts faster (40K 1,106 -> 1,160, 128K 897 -> 920, 128K k8v4 1,006 ->
1,041, 256K 760 -> 780 tok/s; same output). It copies every non-resident expert,
so with a big host tier it loses (IQ2_XS, a quarter in the RAM mirror: 2,184
tokens ~560 -> 254 tok/s): generate.cpp hands the non-resident share to the
prompt path, which walks only below 10%. STRATA_PREFILL_STREAM_ALL=1/0 forces it.
Docs: the 40K prompt breakdown, and why the cluster kernels and Q4_0 XMX prompt
attention are not worth porting here; the 2K "regression" was session variance
(old and new builds A/B'd: 784-799 tok/s both).
Tested on the B70: Coder 2,184-token and 40K, IQ2_XS 19- and 2,184-token
outputs identical; IQ2_XS 562 tok/s (stream-all off by its share), Coder 40K
1,160 tok/s.
On CUDA the register kernel holds 33,792 blocks (~135K cells). Past it qsa_block_topk took the original kernel:
256 threads, each reading its own stretch of the scores on all six passes, one shared histogram updated with
atomics. The dispatch follows the capacity (--max-context), so with a context past ~135K that was every prompt
batch and every decode window on a card without the sm_75 active-bound dispatch (#512), whatever the prompt's
length, and on sm_75 every prompt batch past 33,792 blocks and every decode window.
block_topk_wide_kernel is the register kernel's layout (1,024 threads, a histogram per warp, warp scans) with the
keys read from memory:
- the four radix passes: thread t reads blocks t, t + 1024, ... - a histogram has no order, so a warp reads 32
neighbours at a time;
- the count and the output: warp w holds `per` rows of 32 consecutive blocks, and the cells are written in the
order (warp, row, lane); a row without a selected block is skipped.
The selection rule is the original kernel's (radix threshold, ties to the lowest index, cells ascending): identical
ids. One launch per call, no scratch memory, no capacity limit.
Dispatch (CUDA; HIP is unchanged and does not compile the kernel): where the blocks do not fit the registers the
new kernel, except a call that names fewer than 4,608 blocks (a prompt batch below ~18K cells), which keeps the
original kernel: its fixed cost is lower there. A decode window (no block count) takes the new kernel at any
context. The register kernel's range, the sm_75 active-bound dispatch and the sm_90+ cluster path are unchanged.
STRATA_TOPK_OLD=1 keeps the original kernel, as before.
qsa_topk_parity (new, with the kernel parity tests): the dispatched top-k against qsa_block_topk_ref on both sides
of the register kernel's reach, at short contexts under a long capacity and up to 524,288 cells, called as decode
and as the prompt path call it, with continuous scores, 64 score levels and equal scores.
RTX 3060 12 GB (sm_86), 256 queries with their block count under a 262,144-cell capacity (524,288 for the last
two), original -> dispatched:
16K cells 0.199 -> 0.199 ms (the original kernel)
32K 0.845 -> 0.328
131K 4.70 -> 0.97
262K 16.09 -> 1.71
393K 26.02 -> 2.49
524K 37.53 -> 3.24
4 queries without a count (a decode window): 0.031 -> 0.028 ms at 8K, 0.176 -> 0.083 at 131K, 0.309 -> 0.127 at
262K.
IQ3_XXS on that card (--resident-experts, --max-context 262144, --kv int8 --kv-resident 32768),
tools/needle_bench.py, STRATA_TOPK_OLD=1 against the default: tok/s of the prompt, and the prompt path's selection
phase (STRATA_PREFILL_TIMING):
16,471 tokens 1004.5, 1016.2 -> 989.0, 1013.6 195 -> 195 ms
31,970 1043.3, 1055.5 -> 1060.1, 1060.5 900 -> 731
61,333 932.9 -> 1013.8 4,061 -> 2,428
122,755 904.7, 904.2 -> 992.7, 989.6 20,019 -> 9,034
185,729 819.7 -> 954.7 50,055 -> 20,761
243,176 743.3 -> 933.7 96,326 -> 35,544
The needles are found in every run.
3-way merge from the port's "main + PRs" migration to dpct of upstream 0.1.38
(prefill.cpp re-merged with the histogram diff: upstream's peer-device and fused
restructure kept, the port's edits carried over - the stager's GGUF job, the
stream-all gate, the fp16 NaN checks; generate.cpp: the pipelined profile fill
beside upstream's prefetching serial fill; mtp/verify: the SYCL host mapping).
- the PRs' header forks dropped: upstream has them now
- stubs: --peer-device (peer_experts.cpp; open refuses) and the fused int8 prompt
kernels (moe_fused_stub.cpp; MMQ is not built here)
- off on SYCL: the sm_90 cluster greedy sampler and QSA top-k (fixups.py)
- fused_gr: dpct's undeduced kernel names filled in (fixups.py)
- STRATA_GDN_KEYHEAD stays opt-in on the port (upstream's gate is NVIDIA SM counts)
- version 0.1.38
Tested on the B70: build clean; native_grouped, iq_multi, gr, qsa, kv_hybrid,
quantize_act parity pass, qsa_prompt_attn's int8/fp16 pass (its new Q4_0
tensor-core cases are refused: not in the port); Coder short/long, IQ2_XS, 40K
outputs identical; 128K int8 KV streaming 892 / 66.8 tok/s; GR_DOWN_MAX4 neutral,
GDN_KEYHEAD 8% slower (stays off).
The trim was on by default in the HIP builds. It now waits, as on CUDA, for the PR author's re-run of the release
branch on his cards; the default flips after that. --layer-split auto and one GPU were never affected.
"saving a conversation checkpoint failed" was the only text when the checkpoint's
device sync failed - a GPU hang that Windows then resets (VIDEO_ENGINE_TIMEOUT), not a
checkpoint problem. The error now names it: "...: the GPU stopped responding before it
(<error>) - a GPU hang; on Windows the driver is then reset". Error text only.
Two private engines, the same conversation: the baseline keeps it live, the candidate
parks it (an unrelated request in between, a re-rendering client, or two long
conversations taking turns) and restores it. Each turn's greedy tokens must match and
its decode speed must be within --tolerance (15%) of the live turn's. Pure logic
tested without a GPU (tools/test_conversation_restore_speed.py).
#528 did not reproduce on the RTX 5070 with it: Q2_0 37K tokens (--kv-resident 16384)
and IQ3_XXS 50-75K tokens (--kv-resident 32768) decoded at the live turns' speed after
a restore, with the same tokens.
- The per-layer wait for the router ids names itself in a stall report ("waiting for
the GPU (attention, router) at layer"): a GPU stall reads apart from a host one.
- STRATA_PF_STEP_SYNC=1 (opt-in): the compute and copy streams are waited for after the
K/V staging, the K/V append and each layer; a step over 250 ms is logged with its
layer and chunk, and a stall names the step.
- STRATA_KV_HOST_DMA=1 (HIP only, opt-in): with KV streaming the prompt path's append
writes the staging pool and the resident slots only, and one DMA copies the chunk's
blocks to the host copy - no kernel writes host memory over PCIe. Same bytes for
every reader (the staged first block is complete; past the chunk nothing is read
before a later append writes it).
Q2_0 on the 5070: the same tokens with STRATA_PF_STEP_SYNC=1 as without (and as 0.1.38).
STRATA_VERIFY_TRACE=1 records each verify window's handshake on the host (window
start, launch, each layer's ring and service, the final sync, a timeout, the release
and how long the GPU took to drain after it) with the seq / flag / A / B words, and
GPU breadcrumbs: the profiler's stage stamps per layer and token group, written to
mapped memory so they read while the GPU hangs. A timeout prints them before and
after the release; the serve watchdog's stall report (Verifier::diag) prints them too.
HIP only, opt-in, for the reporter's A/B: STRATA_VERIFY_COHERENT=1 allocates the
handshake words and rows with hipHostMallocCoherent; STRATA_DOORBELL_STORE=1 has the
GPU store each step's ring (known at capture) instead of read-modify-writing it over
PCIe. CUDA builds and the default path are unchanged (Q2_0 on the 5070: the same 96
tokens with and without the trace as 0.1.38; a forced stall dumps the breadcrumbs up
to the withheld flag).
docs/AMD_HIP.md's gfx1030 report measured an i7-13700KF (8P + 8E) at 38-42 tok/s with
all 15 workers against 36 with 8, so a CPU with no more E-cores than P-cores keeps the
engine's own count; the E-heavy chips the fork measured (i9-14900KF: 15 of 23 faster)
get the recommendation.
--vram-elastic (the config's "vram_elastic": true) allocates the expert cache as one
reserved address range backed by physical segments (CUDA virtual memory management,
512 MiB each, --vram-segment-mib), so every slot keeps its address. Between
requests, the engine command `VRAM <reserve_mib>` gives the last segments back to
the driver until that much VRAM is free (their experts become CPU misses, the prompt
path's loan moves down and its chunk shrinks to fit), or maps them again and puts
the same experts back when more is free. `VRAM` alone: the start's reserve. The
server's POST /v1/vram {"reserve_mib": N | null} sends it after a running request,
keeps it for an engine that is unloaded, and /v1/status shows the cache's size.
Never automatic. One NVIDIA GPU only (no layer split, helper caches, --peer-device
or the resident low-RAM mode); HIP refuses the flag.
RTX 5070, Q2_0 (4.8 GiB cache): reserve 6000 -> 4.3 GiB freed in 78 ms, decode
44 -> 33 tok/s; grown back in 92 ms, the answers token for token the ones before.
Without the flag the cache is the one cudaMalloc it was (same answers with and
without the flag until a VRAM command).
IQ3_XXS on the RTX 5070, temperature 0, 256 tokens, 8 French prompts (topics outside the corpus), 2 English and 2 code
prompts, arms alternating, 2 passes each: drafts accepted in French 0.505 (en) / 0.507 (cjk) -> 0.597 (fr), higher on
all 8 prompts and in each pass; English 0.610 -> 0.629, code 0.772 -> 0.779. Decode speed was not measurable here
(builds ran beside it); the reporter measured 141 -> 158 tok/s in French on an RTX 5090.
UD-IQ4_XS sits between IQ3_S and UD-Q4_K_XL: IQ3_S gate/up experts (IQ4_XS in one layer) with IQ4_NL downs (Q8_0 in
five layers), 59.5 GB of them, and UD-Q4_K_XL's dense side. The engine's kernels already take these formats; it was
packed (iq_pack.py --compat-bf16) and run on the Strix Halo, not yet on NVIDIA.
- MODELS["UD-IQ4_XS"] in the unsloth family: three shards pinned at the same revision 38bb39e with their sizes and
SHA-256 (the Hub's LFS pointers), the RAM budget like UD-Q4_K_XL (budget, one GPU, no images, --compat-bf16 pack,
no experts.bin), engine 0.1.38 or newer (the fused-layout fix for packs not every layer of which is covered).
- A model can have its own shard pattern and count (model_file / model_shards); budget_model() tells which Unsloth
model a config runs, for the #498 multi-GPU messages and RAM need.
- The AMD question (#429) stays for UD-Q4_K_XL only (nvidia_only): UD-IQ4_XS's formats have AMD prompt kernels.
- --gguf-dir accepts UD-IQ4_XS files (#444's list says so).
- UNSLOTH_Q4.md: its files, what setup does, what is known (Strix Halo), and the BF16 rounding of the hyper-connection
projections that every --compat-bf16 pack has (not changed here). MODELS.md: a paragraph.
- test_setup_unsloth: pins, budget, a mocked install, the engine check, no AMD question, the config read back.
UD-Q4_K_XL's install, its tests and every other model's config are unchanged.
Stateless, on the chat path: a Responses request becomes the chat template's
messages, tools and kwargs, runs through Service.run, and its reasoning, text
and tool calls come back as output items and the official typed SSE events
(response.created ... response.completed / incomplete / failed).
- input as a string or items (messages, reasoning, function_call,
function_call_output, custom tool calls); instructions and leading developer
messages make one system message; tool results are put in call order
- function, namespace (Codex's MCP/agent tools) and custom tools; hosted tools
are left out with a log line
- reasoning.effort -> reasoning_effort, max_output_tokens, text.format
json_schema/json_object through the structured output path
- reasoning returned as reasoning_text (and as opaque encrypted_content when
asked) and replayed into the prompt, so Codex's next turn reuses the cache
- errors in the Responses format; same API key, Host and Origin checks as /v1
- a response.in_progress event every 15 s while a long prompt is read
(Codex's idle timeout ignores SSE comments)
- docs: DETAILS.md section with a Codex config.toml
Based on #630 by CC-David-CC (the event assembler, namespace tools, call-id
ordering of results), without its opt-in flag, replay key and crypto
dependency, strict-function and JSON-format refusals, or the second
summary generation pass. Tested with Codex CLI 0.160.0 and openai 2.15.0.
Co-authored-by: David <[email protected]>
The About tab's Model settings card reads GET /config and saves through
POST /config: the sampling defaults, reasoning_budget_tokens,
fit_max_tokens, anthropic_thinking, effort_position, aliases,
idle_unload_s, lazy_load, engine_silence_s, api_monitor, open_browser
and --vram-reserve-mib (serve/runconfig.py). Every value is checked
first (one bad value changes nothing), only the named keys change, every
other key of strata-<model>.json stays as it is (as setup run again
keeps them, #629), the earlier file is kept as .bak, and the change is
used from the next start. POST takes JSON from Strata's own page only,
like /settings, and the API key when one is set; the network, key, MCP
and program keys are not editable there.
- MODELS.md: the RAM rule is the experts + ~10 GB (setup's arena_gb), not shard 1 + 10 GB; the sizes per model.
- DETAILS.md: prompts are read in chunks of up to 8,192 tokens (32,768 opt-in), not 2,048; Q2_0's AVX-512 pack is
~40 GB on disk = 34 GB of experts plus the dense weights.
- README: the NVIDIA speed table says which engine measured which row (Q2_0 0.1.36, the others 0.1.26).
- expert_cache.hpp: the comment that the cache "does not yet compute anything" is gone.
Private reports through GitHub's private vulnerability reporting; the
server's protections as of 0.1.38 (127.0.0.1 by default, the API key,
the Host and Origin checks, CORS, MCP tools and the request monitor
opt-in), with DETAILS.md for every setting. README links it.
/metrics gains "conversation_cache": the parked conversations, their
bytes, parks/restores/evictions and the last switch, read from the
engine's own log lines ("conversation cache: parked/restored ...") from
the start of its current run on; the budget and slots from its INFO line;
and the prompt tokens reused in the last request and since the start.
The card shows the parked slots and memory when --conversation-cache-mib
is on, the reuse lines always, and how to turn parking on when it is off.
--gpu-stages replays per-layer graphs, which a native (IQ) pack does not capture, and refused with "not captured
with the split". The serve path already times a native window's stages: STRATA_DECODE_TIMING=1 prints the
per-request decode split and STRATA_VERIFY_PROFILE=1 adds the GPU stages. DETAILS.md now says how to read them, and
--gpu-stages on a native pack says to use them.
The decode hit rate is the VRAM share of the experts the pool looked up; the routed experts the GPU reads over PCIe
(--pcie-frac) or another GPU computes are in neither count, so raising --pcie-frac raised the "hit rate" while the
decode got slower. Its meaning stays; now that share is said beside it:
- engine: DriveState::offload_entries counts those entries; the serve line adds "; N more read by the GPU over
PCIe or from another GPU (X% of all M routed)", and the DONE line a 16th field (an older server ignores it).
- server: /metrics' requests get "pcie_share" (None from an older engine), the console's done line "(+X% of the
routed experts over PCIe)", and the Monitor's column is "VRAM hit rate" with "+X% PCIe" after it.
- DETAILS.md says what each means.
No number changes; tokens are unchanged.
CMake links CUDA code with cudart_static by default and the targets also link CUDA::cudart (cudart.lib); MSVC with
CUDA 12.x (the STRATA_EXPERIMENTAL_SM60 build for sm_60/sm_70) then fails with LNK1169. That build (MSVC +
STRATA_EXPERIMENTAL_SM60, no CMAKE_CUDA_RUNTIME_LIBRARY given) now sets CMAKE_CUDA_RUNTIME_LIBRARY Shared, the
reporter's verified fix. The ready-made engine and every other build are unchanged. INSTALL.md: where github.com
cannot be reached, third_party/llama.cpp at the pinned commit can be put there by hand.
The effort's instruction is the first thing the template renders, so a
request that only changes reasoning_effort (or turns thinking off) read
the whole prompt again. With "effort_position": "end" in the run config
the server renders every prompt's start as the default effort's and puts
a low/medium effort in a short system turn right before the answer
(thinking off: just the template's empty thinking block); the rendering
is template-agnostic (history + turn + the generation prompt).
Engine: --tail-role-token ID (default -1, off) moves the turn checkpoint
in front of such a trailing turn, so it holds the conversation and not the
effort text; INFO reports tail_role_token. The server passes the id of
"system" only when the engine binary knows the flag, and says so when it
does not. Without the key nothing changes: the same prompt ids, no flag.
The first mapped start writes the whole arena (25 GB for the Coder) next to the pack. A drive without that much free
space filled up before the write failed. Now the start checks the free space first (the arena plus 2 GiB to spare);
without it, it says so and keeps the arena in RAM for this start. Linux opt-in only; nothing changes without the env.
The trim was measured on 2x MI50 (gfx906) only; nobody has run it on NVIDIA multi-GPU, where CUDA-only paths could
still read another stage's dense weights on a card that no longer holds them. The HIP builds trim with explicit split
points as the PR does (STRATA_STAGE_TRIM=0 keeps the full copies); the CUDA build keeps the full copies unless
STRATA_STAGE_TRIM=1. --layer-split auto and one GPU are unchanged everywhere.
find_nvcc takes the newest toolkit it finds, so with CUDA 12.8 and 12.9 installed, 12.9 was always used - and 12.9's
math headers do not compile with glibc 2.43 (Ubuntu 26.04). STRATA_NVCC=<nvcc> is now the only one considered (a
warning when the file is missing, or when it is too new for the Pascal/Volta build). CUDA_HOME is a candidate like
CUDA_PATH. TROUBLESHOOTING.md: the two build errors and what to do (g++-14 for gcc > 14; CUDA 12.8 or 13.x for glibc
2.43). Without STRATA_NVCC nothing changes.
The pinned expert arena (the default tier) was allocated without looking at the RAM: on Linux the anonymous mapping
succeeds whatever the host has, and a container whose memory limit is below the arena was killed by the OOM killer
part-way through the load, without a message.
- detail::host_available_memory: the file tier's probe (MemAvailable lowered to the tightest cgroup v2 limit's room),
now with its files as parameters (tested on fake trees), cgroup v1's memory controller (memory.limit_in_bytes -
memory.usage_in_bytes), and no cgroup line at all meaning MemAvailable alone (it was "cannot determine", which
refused a --resident-budget-gib on a v1 host). It also returns the tightest hard limit itself.
- ArenaExpertSource::open, after the size check and before the allocation: a hard cgroup limit below the arena is
refused with both numbers (a certain OOM kill); less RAM available than the arena is a warning with the numbers
and what runs with less (--mmap-experts with --resident-budget-gib), and the load goes on. Not for a shared arena.
- file_expert_source_test: v2 limit / max / malformed / missing, v1 limit / unlimited / not mounted, no cgroup.
On a 7200 RPM disk the default --ple-io direct's random reads of the n-gram table stalled prompts past the 60 s
watchdog, and the stall report looked like any other engine stall.
- Engine (Linux): at start with --ple-io direct, the table's disk is looked up in sysfs (queue/rotational); on a
rotational one it warns and recommends --ple-io ram or an SSD (not on WSL, whose virtual disk says rotational).
- Stall report (Linux): the engine's threads in state D and their wchan ("threads waiting on the disk (state D): 16
of 35 (blk_io_schedule x16)"), with what that means.
- Setup (Linux): a model on a rotational disk gets --ple-io ram when the RAM holds the table beside the model,
otherwise a warning that recommends an SSD. Nothing changes on an SSD.
- DETAILS.md: --ple-io direct is for SSDs (it reads unbuffered, not through the OS cache), and a troubleshooting row.
The tokenizer matches <think> and </think> as their special tokens even
in plain text (type 4, as llama.cpp does), so a user quoting "</think>"
handed the model a real end-of-reasoning token and the answer came back
empty. Before the template is rendered the tags in message text (user,
tool, system, assistant history, tool call arguments, tool descriptions)
are swapped for private-use marks; after it, those spans are encoded as
ordinary text (Tokenizer.encode's new plain= spans), the template's own
markers stay special. An assistant turn that opens with its own
<think>...</think> block keeps that block. A prompt without the tags is
encoded exactly as before; count_tokens uses the same path.
hardware_key, the settings file's key for a saved calibration, read the cards with nvidia-smi's list: on an AMD PC
every card was "?|0GB" (two different cards shared one calibration), and on a PC with an NVIDIA card too a HIP index
named that NVIDIA card or stopped setup ("there is no GPU N"). A HIP config's cards now come from amd_gpus, in HIP's
numbering, with the architecture in the name (an RX 7900 XTX and an RX 7900 XT are both gfx1100; a card without a
product name is known by its arch). Setup reuses a saved calibration on Linux HIP as on NVIDIA (setup_calibration);
Windows HIP keeps the defaults for now. The tuning is still offered on NVIDIA only - a `./setup.sh --calibrate` on
HIP runs it by hand, and each control is verified on HIP before setup offers it.
The NVIDIA key and the settings file's format are unchanged. Tests: tools/test_setup_amd.py CalibrationKey (7).
French answers drafted poorly with the shipped subsets: 23.6% of the token occurrences of French text fall outside
draft_vocab_en.bin, and the draft head cannot propose those (no subset adds the accented words and word pieces).
- tools/draft_vocab.py: a coverage mode. --corpus FILE... adds to the base the most frequent tokens of the corpus that
together cover --coverage (default 0.99) of its occurrences, and prints the share outside the base and the new list.
- data/draft_vocab_fr.bin: draft_vocab_en.bin plus 5,686 tokens (46,211 ids), from 10 French Wikipedia articles
(France, Paris, Napoleon Ier, Langue francaise, Union europeenne, Litterature francaise, Cuisine francaise,
Economie de la France, Revolution francaise, Seconde Guerre mondiale; 1.24 M characters, 326,926 tokens) at 0.99:
python tools/draft_vocab.py --gguf <IQ3_XXS shard 1> --base data/draft_vocab_en.bin --corpus <the articles> \
--coverage 0.99 --out data/draft_vocab_fr.bin
Outside the list on that corpus: 0.74% (23.65% with en).
- setup.py: "fr" in DRAFT_VOCABS / --draft-vocab (~151 MiB head on IQ3_S, between en and cyrillic). The sha256 check
knows it as a shipped subset, so a later --draft-vocab cjk/en replaces it and a subset made by hand stays.
- tests: tools/test_draft_vocab.py (the coverage mode), tools/test_setup_draft_vocab.py (fr replaces a shipped subset
and back, keeps the en ids first in their order, its MiB estimate).
The default subset (cjk) is unchanged.
A broken state (or a model in a loop) can answer one token forever: #606 streamed 36,689 tokens of "!" for four
minutes, because a request without max_tokens gets the whole remaining context. The server now ends such a reply
at 256 identical tokens in a row with finish "length" and a line in its window saying what to do if it repeats on
a new request. "repeat_stop_tokens": N in strata-<model>.json sets the run (0 = off). Nothing changes for a reply
without such a run.
Both q8_1 activation quantizers (iq_kernels.cu q8_1_store, native_mmvq.cu native_quantize_q8_1_kernel) stored the
sum of a block's 32 raw activations as fp16: past 65504 (one massive-activation dimension is enough) it rounded to
inf, and d = amax / 127 did past amax = 8.3M. The dot products then read inf * 0 = NaN, and a NaN in the residual
answers one token forever (ggml-org/llama.cpp#23606 is the same defect).
Now both are clamped to the largest finite half (and the values to the int8 range they then exceed). A block that
was finite before is stored bit for bit as before - values in (65504, 65520) round to 65504 anyway, and a finite
block's |x / d| is at most 127 - so the default output only changes where a block overflowed (and was NaN). A NaN
input stays NaN, so STRATA_DBG_NAN still finds it.
iq_multi_parity checks both quantizers against the 0.1.38 formula on a host transcription: every finite block as
before, and three overflowing blocks (sum 96,000, sum -69,984, amax 2e7) stored finite.
On a hybrid CPU (Intel 12th gen+, AMD Zen 5 + Zen 5c) setup writes --pool-workers
N into the run config: the performance cores but the host loop's one, plus half
of the efficiency cores. An E-core runs the expert kernels ~2.2x slower and each
layer waits for its slowest part; Hardin22 measured 15 workers at 165 / 116 tok/s
against 106 / 84 with all 23 on an i9-14900KF (#642). A recommendation only: the
line is in the config, a saved calibration's measured count wins, and a CPU whose
cores are all alike gets the same config as before.
The engine's pool, with every core allowed (the default affinity), now orders the
P-cores first on a hybrid CPU, so a smaller pool runs on the P-cores and the first
E-cores rather than on whatever the OS numbered first (from Hardin22's fork).
Non-hybrid CPUs: unchanged.
Co-Authored-By: Hardin22 <[email protected]>
Kept in the run config's vision.max_tokens (the server already passes it
to the encoder); a setup run again keeps it for the same encoder device.
More than the default is allowed with a note on the encode time. Without
the flag the config is as before. DETAILS.md: vision.mmproj can point to
a Q8_0 mmproj of this model.
- The draft layer's K/V lives on the last stage's GPU (the drafter is loaded there); its save, ring restore and
verification now run with that stage's image under that device, instead of with stage 0's image on GPU 0, where
the ring restore kernel ran against another GPU's memory (seen in Niko1221/Strata#653, which places it the same way).
- Parking with a layer split on one GPU (--split-device 0) is refused at start: untested.
- The whole-session validate refuses an image with stage images; clearer message for stage parts in live.
- --trim-stage-weights says which layers each GPU loads, and fails when index.txt cannot be read.
- conversation_snapshot_test: the whole-session refusal; parking_test.py runs with STRATA_SNAPSHOT_VERIFY=1.
tools/make_profile.py builds the ranking as the base's order, then the
trace's pairs, then the fill. The base is only appended to, so --base can
never move a pair the base already ranks - and the shipped
data/expert-profile.bin ranks all 24,576 pairs, which makes a trace a
no-op:
wrote ...: 24576 ranked pairs (24576 from the base, 0 from the traces, ...)
So the documented way to seed a profile from --dump-routing (rank the
pairs your workload routes most) silently did nothing with the shipped
base; only --no-base changed anything, and that drops the base's ranking
entirely.
Add --reorder: rank the trace's pairs first (most frequent first), then
let the base fill the rest, then the fill. The default is unchanged. The
ranking moves into a rank_profile() so it can be tested without a model;
tools/test_make_profile.py covers the no-op, --reorder, --no-base, the
frequency/tie order and an end-to-end run over a complete base. On this
host's coding trace the same command now puts 5,808 trace pairs first.
--no-browser (on a start or at setup) saves "open_browser": false in the
model's run config; start() and run-<model>.bat/.sh then leave --open
out, and the server ignores --open when the config says so (a start
script written earlier still has it). --browser undoes it. Without the
flag the config, the command and the script are as before.
The adaptive swap loops called srcp->blob() for every swap to collect the spans pin_blobs locks, on every build -
pin_blobs is a no-op except on Linux AMD with STRATA_ARENA_MMAP=1. blob() is not free: it counts reads and the file
tier (--mmap-experts) may assemble the blob, a second time for the copy. pin_blobs_on() now guards the gathering, so
CUDA and the runs without STRATA_ARENA_MMAP do exactly what 0.1.38 did.
"layer_split" is the first layer of each later GPU ("24,36,42" for 4 GPUs), not a count of layers per card. A
value such as "24,16,12,12" failed the engine's check after the start began, and a JSON list reached the engine as
"[24, 16, 12, 12]".
- server.py layer_split_value: "auto" (default), a comma string or a JSON list of one rising number per GPU after the
first, from 2; anything else stops the start with the format and an example for that many GPUs.
- The engine's message says the same, with the value it got.
- MULTI_GPU.md and setup's --layer-split help say it.
The host expert arena is normally read into pinned (locked) RAM. On a machine whose GPUs already hold most
experts, that locks memory the CPU never reads: with the Coder (25 GB arena) on 2x 16 GB GPUs and 32 GB of RAM,
~1 GB stayed available at 128K context, the kernel swapped, and a multi-turn checkpoint took 25-80 s.
With STRATA_ARENA_MMAP=1 (opt-in, Linux; off by default, nothing changes without it):
- the first start writes the arena to the pack's experts.bin (padded by one blob so a whole-slot copy may start at
any expert); later starts map it read-only, not locked, not pinned. The page cache keeps what the CPU pool and
the prompt path read and gives the rest back under pressure; a dropped page is re-read from the file, so no
result depends on what is resident. The GPUs get no mapped alias of it: run with --pcie-frac 0;
- after the VRAM caches are filled, the pages of experts a GPU holds are handed back (MADV_DONTNEED,
STRATA_ARENA_RELEASE=0 keeps them) and the rest are prefetched (MADV_WILLNEED); an adaptive swap releases the
expert that went to VRAM and prefetches the one that came back;
- HIP: ROCclr copies a pageable source of 1 MiB or more by locking its pages in place and keeps them locked, which
undid every release. The engine raises GPU_PINNED_MIN_XFER_SIZE (an explicit setting wins) so such copies
stage through ROCclr's own buffers, and locks an adaptive swap's source pages itself only until the copy lands
(the page ranges are merged first: a copy from a range only part of which is registered fails);
- STRATA_RSS_TRACE=1 prints the process's RssFile at the startup steps (madvise reports success on locked pages
and frees nothing; RssFile is the honest check).
The existing --mmap-experts refuses native packs (the Coder); this path is for them.
Measured on 2x AMD Instinct MI50 16 GB + 32 GB DDR4 (gfx906 port, Coder IQ1_M, 128K context, layer split):
RAM available while serving ~1 GB -> 25 GB; multi-turn re-read of a checkpoint 25-80 s -> <= 3.2 s; decode
unchanged (49.06 vs 49.16 ms per verify window). Not measured on NVIDIA (pin_blobs is a no-op there).
The file tier's routing-aware prefetch (RouterLookahead) copies all layers' ffn_gate_inp.weight from CUDA0's arena.
With CUDA0 trimmed to its own layers it found them missing and turned itself off ("the routers are not BF16 in the
arena"). CUDA0 now keeps the routers (~2.5 MiB a layer) when --mmap-experts is on; the other stages and the runs
without --mmap-experts trim as before.
Without this every stage of a layer split keeps a full copy of the dense weights (~3.4 GB for the Coder),
although it only ever reads its own layers. That VRAM goes to the expert cache instead.
- the canonical arena of each GPU skips the `blk.N.*` tensors of layers outside its range (PLE tensors stay
everywhere), read from the pack's index.txt;
- NativeDense::set_layer_range limits the native projections the next `load` uploads the same way;
- only with explicit split points (`--layer-split 27`); `auto` and a single GPU are unchanged;
STRATA_STAGE_TRIM=0 keeps the full copies;
- one log line per GPU says what it loads, and a later stage logs its free VRAM before the cache is sized.
Measured on 2x AMD Instinct MI50 16 GB (gfx906 port, Coder IQ1_M, 128K context, --layer-split 27):
experts held in VRAM 8,819 -> 10,626 of 12,288, VRAM hit rate 95% -> 98%, decode on a 17K-token agent prompt
39.2 -> 41.7 tok/s. The trimmed tensors are never read by that GPU.
A re-run for an installed model (another --context, say) rewrote
strata-<model>.json from scratch and dropped the user's sampling,
mcp_servers, mcp and other blocks. Setup now carries over every key it
does not write (and env/vision entries of the user's own, such as a Q8_0
mmproj), keeps the earlier file as strata-<model>.json.bak and says what
it kept and which hand-added engine options it did not. A copy set up
like an earlier install carries them over from that install's config.
A first install writes the same config as before.
The native head's cudaMalloc ran after the expert arena had registered ~45 GiB of host pages. WDDM then refused it
on a 16 GB card with the desktop on the iGPU ("native head upload: out of memory" with GiBs free) - the drafter is
loaded before the arena for the same reason. Now the head and the logits buffer load right after the drafter, still
before the expert cache is sized, so the slots are the same. The head's, the drafter's and a split stage's head
failures print "(N MiB of M MiB VRAM free on this GPU)".
setup: a context of 64K or more whose KV streaming does not fit the RAM test now says so (it was silent, and a
regenerated config lost --kv-resident without a word).
The server replied to /load and /unload without reading the request body; on
Windows, closing a socket with unread data sends a TCP reset, so the client saw
a connection reset instead of the answer. Read the (bounded) body first.
Taken from #630 by CC-David-CC.
0.1.38 made the choice before the resident set existed, from the budget asked for, against every shard's bytes
(dense weights and the PLE table included). On a 96 GB PC with UD-Q4_K_XL (budget 72 GiB, 48.94 GiB really
resident) that read 83.0 - 72 - 4 = 7 GiB of room against 103.7 GiB of files, so every refill after a prompt read
the drive: prompts 15-40% slower than 0.1.34.
- platform::file_cache_keeps(avail, arena, read): the decision, unit-tested in platform_memory_test with the
reporter's numbers and the 32/64 GB PCs of #357/#362.
- set_unbuffered: the arena is at most every expert's bytes, and the cache has to keep only the experts outside it.
- recheck_unbuffered, after pin_cache_complement: the same decision from the RAM the copy really holds and the
expert bytes outside it; switches either way (startup, nothing reading) and logs the numbers.
- The 32 and 64 GB PCs still read unbuffered (the profile fill and the copy too, as before).
Same tokens either way: only how the same bytes are read changes.
openai_to_messages / anthropic_to_messages check that tools is a list of
named tool objects (a JSON-string list is decoded, as for messages) and
raise ValueError otherwise; the MCP name set skips non-objects too.
0.1.38's close() already ends the old engine and waits for it (QUIT, terminate, kill, 20 s each) and raises
EngineStuck when it cannot. The PR's second kill and 120 s wait, after which a new engine was started next to one
still alive, is left out; the start retry and the last known context stay.
Seen in long agent sessions (Claude Code, many turns at ~20K tokens): when the engine was ended mid-request (the
60 s watchdog), restart() started the new engine while the old process still held its VRAM. The new one ran out
of memory and exited before READY, `max_context` stayed 0, and from then on every request failed with
"400 ... exceeds the context (0)" until the server was restarted by hand.
- restart() ends the old process first (close(); kill and wait if it is stuck), so its GPU memory is free;
- a start that exits before READY is tried again, 3 times, RESTART_RETRY_S (15 s) apart; `starting` is set
meanwhile, so a request gets the 503 "starting" of #344;
- `known_ctx` keeps the last context the engine reported: after a restart that failed for good, a request is
checked against it and reaches run(), which starts the engine again, instead of a 400 about context 0.
Two tests in RestartWindow (both fail without this change): a start that fails once is retried and the engine
comes up; a request after a failed restart gets 200, not a 400. `python -m unittest serve.test_server`: 120 OK.
- conversation_snapshot_*(..., const QsaState* draft): nullptr saves, validates and restores an image without the
draft layer's K/V. The later stages park that way; the draft ring is saved once, with the first stage's image.
- ConversationKvReuse::stages / ConversationCache::retain(.., stage_kv): every stage's restored K/V is retained, so
its next park copies only what changed, as the first stage's already did.
- conversation_checkpoints_split / _merge: the checkpoints are moved into their stage parts for the capture and
back afterwards (and on restore), instead of copied.
- Tests: conversation_cache_test (split/merge round trip without copies, refused misalignments, per-stage reuse),
conversation_snapshot_test (draft-less images: estimate, validation both ways, A/B/A exactness, incremental reuse).
- tools/parking_test.py: a follow-up decodes the same tokens whether its conversation stayed live or came back
from the parking cache (every stage's image with a layer split). 4-GPU split, IQ3_S: 3,786 and 9,276-token
conversations restored in 42 / 51 ms, follow-ups identical; without parking the same follow-ups took 3.4 / 6.8 s.
- tools/batch_test.py: --keys to send sampling keys (sampled rows in batch windows).
- docs/BATCHING.md: parking together with batching, the Testing section; docs/DETAILS.md: parking now works with
--layer-split.
park_current snapshotted stage 0 only, so a split rejected parking. A parked conversation now holds one image per
later stage (SavedConversation::stage_images, captured from each stage's session on its own GPU); a split
checkpoint is cut into its stage-0 part and its stage parts while parking and put back together on restore; every
stage image is validated before anything is overwritten; the guard is removed.
Patch posted on #559 by crazyaimachine (written by their coding agent), tested on 2 GPUs there and on a 4-GPU
split here.
Co-authored-by: benoit lange <[email protected]>
With --batch, a consumer that left mid-answer (a client disconnect, a tool call or a stop string ending the
response) left the engine writing the abandoned answer's tokens, and the next request read them as its own:
it answered the previous question again. Now the solo request is STOPped and read to its DONE, an admission
to its BADM, and a slot is BSTOPped and freed only at its BDONE. STRATA_BATCH_TRACE=1 logs the control lines.
tools/early_close_test.py reproduces it (solo and with a second request running).
The engine's --serve can now keep up to N conversations open (--batch N) and decode them together, one token
each per verify window; with a layer split, --batch-groups G pipelines the slot groups through the GPUs, and
--trim-stage-weights makes every GPU load only its own layers' dense weights (VRAM back to the expert cache).
The server runs concurrent requests in the slots. See docs/BATCHING.md.
A rebinding page cannot send the key, so with one configured the check only
broke tunnels and proxies that pass their own name on (cloudflared does).
Keyless setups keep it, with allowed_hosts; the 403 now also says that
setting an API key turns the check off.
It keeps 256 of the 512 experts, chosen on code data, so it is weaker outside
code and in other languages; for general use, or when RAM allows, MODELS.md
now points to the full models.
/unload and /load took a plain form POST from any site: they now take JSON
from Strata's own page only, like /settings (_own_page). Without an API key, a
POST to /v1/* that carries an Origin must come from Strata's own page, a page
on localhost or an allowed host name, an origin in trusted_origins or
cors_origins, or a non-web scheme (browser extensions, desktop apps), and then
send JSON; others (and Origin: null) get 403, so a web page can no longer burn
GPU time with a text/plain POST. Requests without an Origin (curl, the SDKs)
are unchanged; with an API key, the key decides.
A web page of another site could point its own name at 127.0.0.1 and reach
the server as the same origin: without an API key it could read /status,
/metrics and /api/requests, change settings and run MCP tools. The server now
answers only Host names it knows: localhost (and *.localhost), any IP address
(rebinding needs a name), the bind address and, bound beyond this PC, the PC's
name, name.local, its LAN addresses and host.docker.internal; any port. No
Host header (HTTP/1.0) passes. "allowed_hosts" in the config and
STRATA_ALLOWED_HOSTS add names (".example.com" for a domain, "*" turns the
check off); trusted_origins' hosts count too. A refusal is a 403 naming the
setting and one log line.
The weight arena is the first large VRAM allocation and its size does not depend on the context, so when it fails
something else holds the card (another program, an engine still exiting). The error now names the CUDA error and the
card's free and total VRAM at that moment; a layer split's stage says the same for its card, with the MiB it needed.
Error paths only.
One line under the start banner: the config's engine options without the model's file paths, and the
server's fields (host:port, an API key set or not, GPUs, draft subset, fit_max_tokens, thinking budget).
The key itself is never printed.
The provider, the context and output limits, the thinking level as a default and as variants, and how
they relate to the context check, fit_max_tokens and reasoning_budget_tokens.
A CUDA 13 build that linked the system's CUDA 12 libcudart.so (a kit with a dangling libcudart.so symlink) read
cudaDeviceProp shifted: sharedMemPerBlockOptin came back as 1, #420's fits() then refused MMQ for every i-quant and
the prompts ran ~4x slower, with a log line printing "1 bytes of shared memory per block".
- ggml_cuda_info() (CUDA) reads cc, smpbo, smpb, nsm, warp size, integrated and cooperative launch with
cudaDeviceGetAttribute, which is ABI-stable. A matched build reads the same values: the same MMQ choices, the
same output.
- The engine warns at start when the loaded CUDA runtime's major version differs from the headers' it was compiled
with, and says to rebuild against one toolkit (-DCUDAToolkit_ROOT).
HIP is unchanged.
Dev::put uploads with a pageable cudaMemcpy, which can return before its DMA lands, and the graph replays and the
timing cases run on a cudaStreamNonBlocking stream that does not wait for the legacy stream: the graph could read
the previous replay's logits (one argmax token off; 5 of 6 replays on an RTX 5060 Ti, now and then on a 5090).
Dev::put now synchronizes the device after its copy, which covers run_graph_case and bench(). Test only; no engine
code.
A reply that ends with finish "length" inside the reasoning has no answer; the server log now says so
and names reasoning_budget_tokens (per request, or in the config for every request). Nothing the client
receives changes.
With the default 700 MiB reserve the expert cache fills an AMD card that also drives a KDE/Wayland desktop. When the
desktop then needs VRAM (the browser --open starts, a new app), amdgpu moves GPU memory to system RAM (GTT), which
the experts already fill: the OOM killer ends kwin_wayland / plasmashell (#560, R9700), or the compositor fails to pin
its framebuffer (#516, 7900 XTX). --vram-reserve-mib 3072 / 4000 fixed it in both reports.
A recommendation, nothing is changed: setup prints the tip on a Linux desktop session (WAYLAND_DISPLAY or DISPLAY) with
the HIP backend and no reserve given, and the server prints it after the start when the engine reports less than
2048 MiB of VRAM free (INFO vram_free_mib) and the reserve is under 3072 - before --open starts the browser.
INSTALL.md and AMD_HIP.md describe the symptom and the fix. No engine code.
installed_configs() and update_install() keep only JSON objects with "exe" and "args"; any other
strata-*.json in the folder gets a one-line warning naming it.
The verify window waits for the adaptive tier's copies since #463; STRATA_ADAPT_NOWAIT=1 starts it as 0.1.37 did (the
non-blocking query), for timing that wait. Default unchanged.
fused_layout() (STRATA_PF_FUSED=1) shrinks GU / H / Xq to the fused kernels' needs whenever
fused_ring() is true, and fused_ring() was true when ANY layer of a native pack was covered. A pack
with an uncovered layer - Unsloth UD-IQ4_XS: 5 layers with Q8_0 down projections - then ran MMQ for
those layers at the full chunk in buffers sized for stream_all_min() - 1 tokens: on gfx1151 an 8K
prompt answered "!!!!..." and a 64K prompt hung (--prefill 2048: an illegal memory access). Such a
pack now keeps MMQ's buffers; its covered layers still take the fused kernels. UD-IQ4_XS 8K
retrieval after the fix: 3/3 correct, 837.9 prompt tok/s. (The CUDA path shares this code.)
#531's peer-aware lookup indexed the residency table with the 512-expert constant NE instead of the model's own
count. The Coder has 256 experts per layer, so from its second layer a resident expert was looked up in the wrong row,
and a prompt long enough for the streamed walk stopped with "an expert is neither resident, streamed nor on the peer"
(found by the 57K real-use check). The 512-expert models index the same row as before.
Setup no longer writes a fixed 300 MiB reserve for cards under 8 GB: the engine's own rule (lower the reserve only as
far as the smallest working cache needs, down to 300) chooses better, and a fixed 300 left an emulated small card with
5 MiB free. Setup keeps the tip (an 8K context, --draft-vocab en). When the engine lowered the reserve and the card ends
below the 256 MiB LOW line, it prints one warning (requests may stall; close other programs that use the GPU, or lower
--max-context) instead of advising a larger reserve, which would leave the cache no room.
peer_experts.cpp calls cudaDeviceCanAccessPeer, cudaDeviceEnablePeerAccess and cudaMemcpyPeerAsync and tests
cudaErrorPeerAccessAlreadyEnabled; the HIP shim had none of them, so the Windows HIP build stopped there. Mapped to
their hip* names.
On an RTX 3060 Laptop (6 GB) the weights, the KV cache and the draft layer left 0.29 GiB, the default 700 MiB reserve
and the draft head took it, the expert cache got 0 slots and the verify window refused to start.
- setup: a single card under 8 GB (nvidia-smi below 7.5 GB; an 8 GB card lists 7.99) gets --vram-reserve-mib 300
unless one is given (images on the GPU keep theirs), marked in the config as setup's own choice so a later setup
decides again. It prints what frees more VRAM if the start still stops: an 8K context, --draft-vocab en, other
programs closed. The draft layer stays (the server needs --mtp and --spec). 8 GB and larger cards: the same config
(golden test).
- engine: a cache too small to lend the prompt path a 256-token chunk (its buffers plus the 128-slot floor; one slot
without --prefill) makes the prompt path allocate its own on top. When the default reserve leaves less than that,
the reserve shrinks to what leaves exactly that cache, down to 300 MiB (what comes after the cache is paid from
the reserve: at 300 an emulated small card ended with 5 MiB free, LOW by the serve check). With no slot at all the start stops with how many
MiB are short, what the session and the draft head take, and the options. A start whose cache could lend that
chunk is sized as before. Checked with a VRAM ballast on the 5070 (IQ3_XXS, 32K): 258 slots needed, 0.71 GiB free
-> "about 333 MiB more", and with --vram-reserve-mib 700 given, "733 MiB more" plus the reserve tip.
- verify's refusal points at that line.
The switch was read with getenv on every launch_multi call, which runs per layer when decode is not captured in a
graph. It is now read once, like the engine's other STRATA_* switches.
STRATA_GR_DOWN_MAX4=1: launches of at most four tokens run the GR down-projection kernels (plain, split and staged)
with per-thread storage for four tokens instead of kFusedGrMaxT, keeping the block dimensions, the tile width, the
accumulation order and the plain/split/staged choice: the same bits. Off by default. RTX PRO 6000 Blackwell, IQ3_S,
MTP 4, 64K: decode +2.1% (code) / +2.4% (prose) with identical output tokens.
From #443; its one-warp MMVQ experiment (STRATA_MMVQ_WARP1, not bitwise), the roofline benches and the benchmark
evidence files are left out.
fused_ring() turns the fused ring sizing off for any --peer-device session (core::peer_portable), but the layer's
fused choice tested the peer's prompt-path state (m.pp), which is null when set_peer declines (no P2P, the buffers do
not fit, --peer-prefill-rows 0). Both now follow the flag, so a peer session's prompt path is MMQ with the MMQ ring
and buffers it was sized for. Without --peer-device nothing changes.
A second visible CUDA GPU (N >= 1) holds an adaptive expert cache beside
CUDA0's, filled with the ranked pairs the primary does not hold. Activations
and result rows cross NVLink or another P2P path; the peer also computes its
experts' rows for prompt chunks (MMQ; the fused path is not combined yet).
Knobs: --peer-device N, --peer-reserve-mib, --peer-slots, --peer-adapt-swaps,
--peer-prefill-rows; sizing stays manual until #216's buffer lending lands.
Default output unchanged: without the flag byte-identical to v0.1.36, with it
byte-identical under the exactness gate. cudaHostAllocPortable only with a
peer. docs/SECOND_GPU.md documents the flags and refused combinations.
Measured on 2x RTX 3090 + NVLink, IQ3_S: prefill +15/+25/+25 %, decode +19/+14/+16 %.
E-9 (Prefill::draft_kv) declines when the drafter's KV is a ring, which it is
with KV streaming (--kv-resident), so every chunk falls back to the E-4 pass in
MtpDrafter::prefill. On gfx1201 / ROCm 7.2.4 that queue (a graph launch and
four device copies per group, ~2,000 groups per 8192-token chunk, one sync)
sometimes never completes: the prompt hangs in hipGraphLaunch or the final sync
until the watchdog ends the engine.
HIP builds now use the per-group loop unless STRATA_MTP_PREFILL_SYNC=0; CUDA is
unchanged. The variable now reads its value (any value, 0 included, used to
select the old loop).
R9700, full IQ3_XXS, --kv-resident 32768, 131K context, mixed load: 0.1.31
hung in round 1 on 2 of 2 tries; this change ran 30 of 30 rounds clean with no
measurable prompt cost.
Calibrated on a Radeon AI PRO R9700 with tune_hipblaslt (24 of the gfx1100
table's 26 shapes). Setup only picks it when the card is gfx1201 and the
installed hipBLASLt is exactly 100202.
With that hipBLASLt the plain hipBLAS path is far off: on the full GSQ-RCO
IQ3_XXS (KV streaming, engine 0.1.31) a fresh 32K prompt reads at 1,177 tok/s
with the table against 638 without it, and a 7K prompt at 1,164 against 656
(median of 4 warm runs per arm, arms alternated); decode is unchanged.
hip_prefill_hipblaslt_gemm passes with it. docs/AMD_HIP.md notes that the
1.4.1 result (0-3%) does not carry over to 1.2.2.
STRATA_NO_LARGEPAGES is documented as a same-run A/B switch for the large-page path, but reserve() only read it in the Windows branch; on Linux the MAP_HUGETLB path ignored it, so the switch silently did nothing. Honor it there too.
The failure message also said "no hugetlb pool configured?" even when a pool exists but is smaller than the mapping: MAP_HUGETLB is all-or-nothing, so a 34 GiB pool cannot serve a 40 GiB arena and the whole mapping falls back to 4 KiB pages. Print the needed 2 MiB page count against vm.nr_hugepages (the Windows branch already names its byte count and error codes).
setup's low-RAM packs have an experts.bin, and with it 0.1.31's tier mapped it: on an emulated 32 GB PC the mapped
reads took the RAM the budget was sized for, and available RAM fell below 1 GB (#286, measured yesterday).
- set_unbuffered also takes experts.bin (paths_ is the one file); read_direct reads one window per blob (a blob is
contiguous there) and merges near ones as for the GGUF.
- blob() hands out the stage buffers for any unbuffered source (staged()), so prefetch, begin_layer and transient
work as with the GGUF in place; the mapped experts.bin path is unchanged otherwise.
IQ2_XS pack with experts.bin, emulated 32 GB PC + 16 GB GPU, 16 GiB budget, 8K prompt + 256 tokens, two runs:
decode 100.2 / 100.6 tok/s, done at 17.2 / 15.5 s, the prompt path's host copies 2.1 s (the GGUF in place: 84-89
tok/s, 21.5-22.1 s, 4.3 s). Greedy tokens identical unbuffered, through the cache and on 0.1.31.
With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside
the budget comes from the drive anyway. Through the mapped GGUF that costs twice:
- the page faults are small requests on the critical path (the prompt path's stager, the profile fill);
- the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was
clamped to 11.65 GiB after the profile fill.
- FileExpertSource::set_unbuffered: chosen by experts_unbuffered (#357) with the budget as the RAM it keeps: whenever
the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in
the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened
FILE_FLAG_NO_BUFFERING | OVERLAPPED.
- copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then
copy it into place.
- Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads
one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE).
Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a
mapped one; 8 MiB reads give 8.6 GB/s.
- The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots.
- warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache,
a second time.
- Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows.
STRATA_UNBUFFERED_LOAD=1/0 forces the choice.
Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31.
The definition gained a defaulted parameter while the header kept the four-parameter declaration. The engine's
caller is in the same file, so it built, but tests/core/expert_layout_test.cpp calls it through the header and
failed to link (LNK2019, unresolved load_experts_gguf). The default now lives in the declaration;
expert_layout_test passes.
A buffered read copies every expert through the file cache. That pays off only when the cache can keep the files
for the next start. On a PC whose RAM cannot hold them beside the arena (a 64 GB PC with IQ2_XS: 24 GiB left for
36.5 GiB of files), every start reads the drive anyway, and the copy is pure cost.
- experts_unbuffered() (pinned.cu): timed random 64 KiB reads tell whether the files are in the cache already
(a cached read takes ~10 us, the drive ~80). Unbuffered is chosen only when they are not and the RAM left beside
the arena could not keep them either. A warm restart that 0.1.31 serves from the cache stays on its FILE* path.
STRATA_UNBUFFERED_LOAD=1/0 forces the choice; the log names the reader and why.
- load_experts_gguf: each chunk's 4 KiB-aligned window is read with FILE_FLAG_NO_BUFFERING into an aligned buffer and
scattered into the blobs, with a handle per role file as expert_gguf_file() resolves it (split shards). 16 readers.
- load_experts_direct (pinned.cu): a pack's experts.bin ranges are read unbuffered straight into the arena. When the
ranges are not 4 KiB-aligned, it falls back to load_experts_ranges.
IQ2_XS without experts.bin, RTX 5090, emulated 64 GB PC (a locked-pages RAM ballast leaves 58 GiB), two rounds:
| | first start: arena / exit | restart: arena / exit |
|---|---|---|
| 0.1.31 | 22.1-25.2 s / 28.2-31.0 s | 12.5-16.0 s / 17.5-20.8 s |
| this | 18.3-20.2 s / 23.1-25.4 s | 6.0-6.1 s / 9.8-9.9 s |
With 128 GB the probe finds the files cached (15 of 16 reads) and the buffered path runs as before. Greedy tokens are
identical unbuffered, buffered and on 0.1.31 main.
--adapt-decay F sets the factor the routing counts are multiplied by after each adaptation (0.7, as before; the same
float, so the default is bit for bit 0.1.37's). #477's routing heat is added before the decay, as before.
From #407; its --adapt-tuned (opt-in re-ranking every 2 rounds, 192 swaps, x0.92 for caches holding 20-60% of the
experts) is left out of this release.
- The engine's `avx2` switch is on for AVX-512 CPUs too, so the widened gate sent an AVX-512 CPU's IQ4_XS gate/up rows
to the new AVX-2 kernel instead of ggml-cpu: other rounding on CPUs that #415 did not mean to change. A format with
only an AVX-2 kernel now takes it on CPUs without AVX-512; everything else is dispatched as before.
- native_expert_parity ran its AVX-512 check for IQ4_XS on an AVX-512 CPU: iq512_rows has no case for it, so the
rows stayed zero and the check failed. Each check runs only for the formats its kernel implements; the AVX-2 one
runs on AVX-512 CPUs too, so this PC tests the new kernel.
IQ4_XS (23) was missing from iq256_supported, so on an AVX-2-only CPU its gate/up expert rows
fell back to ggml's per-token vec_dot while every other i-quant format got the multi-token
kernel that decodes the weights once for the whole verify window.
Fmt32<23> reuses the 16-value codebook IQ4_NL already has. Each of the eight 32-value sub-blocks
carries its own 6-bit scale (two nibbles of scales_l plus two bits of scales_h), used signed as
(ls - 32): the scale folds into the int16 operand of madd_epi16 instead of becoming a float
multiply per sub-block, and the codebook sign is carried the way iq4nl_rows does it (|w| as the
unsigned maddubs operand, w's sign applied to the activation).
The gate in native_expert.cpp and in native_expert_parity cannot be iq512_supported alone:
IQ4_XS has an AVX-2 kernel and no AVX-512 one, so on an AVX-512 CPU the widened gate must still
pick each kernel only for the formats it implements - falling through an empty switch would
leave ff unwritten rather than falling back to ggml-cpu. Widening the parity gate is what makes
the format tested at all: with iq512_supported alone the tool skipped it on every CPU.
The cherry-picked change kept the old `if (wait) cudaEventSynchronize` above its own (a second, no-op wait), and its
serve-mode call site sat four columns left of its block. The same code otherwise.
A native pack leaves the per-token loop at `if (native_pack) { spec_pos = pos;
break; }`, before the site that dumps a row per position. The header was
written at open anyway, so the file claimed 64 or 66 rows and contained none -
and a reader that trusted it reported "0 differing positions" from a file with
no positions in it, which is the one answer that looks like a pass.
The header is now written with the FIRST row. A run that dumps nothing leaves
an empty file, which is what actually happened, and a reader can treat size
zero as "this mode never dumps" instead of as agreement.
Verified on the native pack: the file is now 0 bytes where it was 8. The
non-native path is unchanged apart from WHEN the header lands; it was not
exercised here because no non-native model is set up on this machine.
(cherry picked from commit b87d7dcfe9a0754c0a2b8b9776b7f2ec8980b6a1)
Greedy decode was not reproducible on this box. Repeated identical requests
diverged from about the 28th generated character, and the drafter, the expert
routing and the first verify window were all byte-stable, which left the
adaptive tier as the only thing left that could move a result.
It is. `apply_pending(false)` polls the swap's event with a NON-BLOCKING
cudaEventQuery, so whether the experts an adapt round just copied in were
resident for the next window depended on whether their PCIe copies had landed.
Resident runs the GPU kernel; non-resident is computed on the CPU, and on a
machine without AVX-512 those two kernels round differently enough to move a
top logit by 0.26 - measured at the window right after the swap was issued.
From there the trajectory is a different one, permanently.
Waiting is the fix. The copies were issued during the previous window's
drafting, so the wait is short, and in exchange the residency table is a
function of what was actually copied rather than of how fast the disk was.
With the default --adapt-every 4 left in place, five runs are now byte-identical
across all 24 verify windows and emit the identical 40-token stream; before the
fix three of five agreed and two forked at window 16. --adapt-every 0 (frozen
placement) was already deterministic, which is what identified the tier.
The STRATA_TRACE_ADAPT lines added with the probe are env-gated and off by
default: they report whether a round considered, skipped, or swapped, and
whether the previous round's copies had landed.
No cost measurement is claimed here: this says the trajectory is a function of
the inputs again, not what the wait does to decode throughput.
(cherry picked from commit 1e3888a73b69a332585056af5b86643ff8367242)
native_expert_grouped launched a block row per possible group (cap = window tokens x 10) for gate/up and down,
and SwiGLU and the q8_1 quantization as two kernels over all cap entries. The verify window's PCIe call has no
group at all at pcie_frac 0 and usually none otherwise, yet every layer paid ~17k blocks that started only to
return plus two passes over entries that are not its own.
- gate/up and down stride over the groups (rows y, y + gridDim.y, ...); the PCIe call runs 4 block rows
(kPcieGroupRows), the VRAM call keeps one per possible group. Each group's rows are computed by the same warp
code in the same order, so the results do not depend on the grid.
- SwiGLU + q8_1 in one kernel over the call's own entries [grp_start[0], grp_start[n]); the product is rounded on
its own (__fmul_rn; on HIP a plain product under contract(off)) before the block sum, as when it went through
memory, and the quantizer body is quantize_q8_1_kernel's (factored out, its PTX unchanged).
- STRATA_GROUPED_V1=1 keeps the old launches for A/B timing; native_grouped_parity checks old vs new bitwise
(every gate/up x down format, 0..cap groups, entries from 0 and past 0, grid_groups 1..4 and > cap).
The engine takes the bound on Turing only, so on other cards the test compared the capacity rule with itself.
STRATA_TOPK_ACTIVE_ANY=1 (set by the test) takes the Turing dispatch on any card; the kernels are the same on every
architecture. The CMake block moves above the qsa_select_bench comment it had split.
With --max-context 262144 the QSA score rows have a stride of 65538 blocks; a prefill batch at 131K actual context
fits the register top-k kernel's 33792-block limit, but CUDA dispatch tested the allocated capacity and chose the
reference kernel. The supplied active_blocks bound is now used on sm_75 only; other CUDA cards keep the capacity
rule, HIP its own policy, and decode (no bound) is unchanged. STRATA_TOPK_CAPACITY_GUARD restores the capacity rule.
qsa_topk_active_parity: 16 cases at stride 65538, selected ids bitwise.
Squashed from #512 (its commit and its merge of upstream 0.1.36).
Co-Authored-By: Codex <[email protected]>
PleReader's row_bytes has been a runtime parameter since the FP8 table
(160 B rows) needed it -- not 90-specific anywhere in the actual read/
cache/job-building code (RowCache, Job/Use, the straddle-length formula
in issue()). ngram.cpp's "Q5_0 PLE requires --ple-io mmap" refusal
predates that generalization (or was just never revisited): FP8's 160 B
rows already go through PleIo::Direct with no such gate.
ple_reader_test --selftest now runs its full synthetic suite (decode-
shaped tickets, page-straddling rows, dedup/edges, bulk, two in-flight
tickets, fault injection, keep-alive -- both threaded and caller-thread
modes) at row_bytes=110 as well as the production default 90, before
touching the refusal. Both: OK.
ple_reader_test --gguf against the real OrcaRouter Q4_K_M shard (the
same one from the Q5_0 kernel PR): direct vs mmap over 312 real PLE
tickets -- bit-identical, and direct is ~8.6x faster (461 vs 3959 us/
token) on this SSD, matching the ~2-2.6ms/token mmap cost the original
"sixteen serial page faults" comment in gather() measured for the
IQ4_NL table.
End to end on the running model (engine 0.1.32, RTX 3090): same text/
math/vision outputs as --ple-io mmap (deterministic sampling), with a
consistent decode speedup (~9-12% across three real generations) and
image-prompt PLE throughput up from not practical to measure to 185
tok/s -- the full per-token win is smaller than the isolated 8.6x since
PLE is one part of a token's cost, not all of it.
is_iq()/the GU/D/MMVQ format lists had Q4_K, Q5_K, Q5_1 and Q8_0 already,
but not Q5_0 -- a plain `llama-quantize` "_M"/"_S" tier GGUF (not Unsloth's
dynamic method) commonly puts Q5_0 on down-projections, sometimes gate/up
too, and native_expert_supported() refused those layers outright ("this
engine has no GPU kernels for").
vec_dot_q5_0_q8_1/dq_q5_0 port vecdotq.cuh's integer chain and the
dequantizer verbatim (symmetric format, so -- unlike Q5_1 -- there's no
min-term rounding choice to make). Fmt<6> plus the row-bytes/is_iq/
dq_dispatch entries follow the existing Q5_1 (type 7) pattern.
native_expert_parity --synthetic q4_K/q5_0 and q5_0/q8_0 (both now in the
CMake test matrix): 0 failures, GPU dequant bit-exact against to_float,
gate/up and down rows bit-exact against ggml's own vec_dot. q5_0/q5_0
(Q5_0 in both roles) is left out of the matrix -- stacking the same
lower-precision format on both roles pushes the GPU path to ~3.6% rel
error against the float reference, just over this test's 3% threshold;
that's compounded quantization noise (CPU rel error is the same ~1.25%
baseline as q5_0/q8_0), not a kernel correctness issue, and no quantizer
I've seen actually picks Q5_0 for both roles of the same expert.
Verified end to end on a real model: a community Q4_K_M GGUF (gate/up
Q4_K, down Q5_0 on several layers) that previously failed at layer 6
now loads and serves correctly on an RTX 3090 (engine 0.1.32).
gdn_rec_cols_pipe_kernel is bound by shared memory, not by arithmetic: every thread of a warp needs the same
32 q and k values per token (the k twice), and a warp receives one such broadcast value per clock. The new
kernel runs column c of the three value heads that share a key head (head % 16) in one thread, so every q/k
value it loads feeds three heads and the token's k row is loaded into registers once for both of its uses.
Its inputs come in blocks of 8 tokens, copied to shared memory by cp.async while the block before computes,
and a token needs 2 __syncthreads instead of 5. 64 blocks instead of 192. Per value head and column the same
arithmetic in the same order: gdn_rec_parity checks output, state and FP16 output bit for bit (T from 1 to
4099) and times it: 18.8 -> 13.0 ms per layer at 32768 tokens.
The output norm stored the normalized value in FP32 as well, which nothing reads (the out projection takes
the FP16 copy): 4.65 -> 3.05 ms per layer at 32768 tokens.
Together 845 -> 579 ms per 32768-token block over the 36 DeltaNet layers; in the engine +2.3 % prefill
tokens/s on 32000-token prompts and +2.2 % on 64000-token prompts, the same greedy answers. Measured on 0.1.32
on a 4080 SUPER (80 SMs); on 0.1.31 the same change gave 1.42x, 1.51x and +2.1 % / +2.0 %. CUDA sm_80+
(cp.async) on a card that holds all 64 blocks at once; otherwise, on Turing and with STRATA_GDN_KEYHEAD=0 the
kernel as before.
Squashed from #413 (5 commits: the kernel, gdn_rec_parity and its --bench SM bands; the 48-63 SM exception
added and taken back).
On Turing mode 4 would run as pairs of m16n8k8 MMAs, which no parity run has checked yet; RTX 20 cards keep the old
kernel for --kv q4_0 until one does.
The tensor-core prompt attention (D-1) refused Q4_0 pools, so --kv q4_0 read every prompt through the FP32 per-query
kernel. Mode 4 is mode 1 (int8) with a scale per 32 values: each q4_0 block's 16 code bytes become 32 exact int8
codes (code - 8) at gather, its fp16 scale goes to the per-group scales, which are applied in FP32 as int8's are
(q.k per 32-dim group; p.v folds each group's scale into p, two groups per warp). The p.v fold is bounded by the
chunk's largest |scale|: q4_0 scales are signed (ggml's d = max / -8). The caller's rotation of q and of the output
is unchanged. STRATA_PROMPT_ATTN_Q4=0: the old kernel (A/B).
qsa_prompt_attn_parity gets Q4_0 cases with signed scales spread over two decades, as a pool holds them (32K and
1.5K contexts), against FP64 and the old kernel.
With KV streaming (--kv-resident) the MTP drafter's K/V is a ring of its window (kv_mode 2: page p in slot p % n_slots
over a host copy), and draft_kv refused it, so every prompt chunk fell back to the drafter's own pass - one graph per
few rows. The batched path takes the ring with its own page table and host copy (as a streamed main layer appends);
the cells it writes are those the window can still reach, which the ring holds, and one batch never spans more than
the ring's slots, so no two cells of a batch share a slot. STRATA_MTP_BATCH_RING=0: the drafter's own pass (A/B).
STRATA_DRAFT_TIMING=1 (debug): the pass's time, synced.
Stager::work waited for a buffer's DMA only from the generation's kRing-th job on (j >= kRing); a generation's first
kRing jobs wrote their buffers straight away. The previous generation's last DMAs from those buffers are only
known to be done where something else waits for them: the routing sync before a staged layer's start(), or the
sync at the end of a chunk with an MTP drafter or stages. A stream_all chunk that ends without one (no --mtp), or a
ring entry the routing skipped (its copy is never waited for), leaves a window where the next chunk's first jobs
overwrite a pinned buffer a DMA may still read. The event is now waited for on every job; an event never recorded,
or already complete, returns at once.
IQ2_XS with --mmap-experts --resident-budget-gib 16 (12,144 of 48,668 streamed experts through the stager): the
same tokens on a 32K prompt, prefill 6,491 vs 6,546 ms.
From an audit: the upload of a chunk's rows, the event recorded after it and the wait on the other buffer's event
before the SSD thread refills it were unchecked, so a failed record or wait let the thread write a buffer an
upload might still read. A failure now ends the chunk with the CUDA error.
Layer 1 adds the per-layer embedding to every token of a chunk, so its rows must be in. A later chunk's rows were
already read on a thread during the chunk before; the first chunk's were read synchronously before layer 0, so the
GPU idled while the SSD read them (an nsys trace: ~245 ms of a single-chunk 32K prompt). Now the first chunk's
gather starts on a thread too and lands just before layer 1 (ple_land), which also starts the next chunk's.
The reads themselves went 64 at a time: a 32K prompt is 512,720 rows in 213,358 4 KiB reads, ~620K IOPS at queue
depth 64. --ple-inflight now defaults to 256, where the drive saturates (1024 is no faster). Decode reads 16 rows a
token and is unchanged. The same rows: the same results.
IQ2_XS, RTX 5090: a 32K prompt 5986 / 6013 -> 5843 / 5808 ms, 8K 1714 -> 1624 ms; first-token logits bit-identical.
In the streamed walk every expert put a cross-stream wait (its copy) and an event record (its ring slot back) on the
compute stream around a ~3 us gather. Under WDDM each pair left ~10 us of GPU idle even when the copy had long
landed: an nsys trace of a 32K prompt showed 226 ms of idle between back-to-back gathers while the host was 3-11 ms
ahead of the GPU. Now a group of 16 is gathered by one launch (copy16_group_kernel) after one wait on its last
streamed copy - the copy stream is in order - and one event releases all of its ring slots: the issuer waits on
used[used_of[slot]]. A ring entry the routing skipped inside an open group first gathers what the group holds so
far, so no more than a group's entries are ever held back from the issuer. The same bytes in the same group slots.
STRATA_PREFILL_GROUP_GATHER=0: one at a time, as before.
IQ2_XS, RTX 5090, 262K context: 32K prompt -8.6%, 8K -10%, 2K -12.6%; first-token logits bit-identical at 2K, 8K,
16K and 32K and with a 96-slot ring.
- engine (serve mode): GEN key logprobs=K (0..20) writes "LP logprob id:lp ..."
after each "T id", from the verify window's head logits (Verifier::copy_logits;
before temperature, penalties and sampling); absent key, no LP lines
- server_intel.py: pairs LP lines with their tokens (StrataEngine._pump), passes
logprobs/top_logprobs to the engine, and fills OpenAI's choices[].logprobs
(content / reasoning_content) in streamed chunks and the collected reply; the
stop token is not listed. serve/ itself is unchanged.
Tested: a fake engine on hardac (streamed, not streamed, without logprobs,
stop token excluded); on the B70 with the IQ2_XS: same answer with and without,
"Paris" -0.0019 with top-1 equal to the chosen token, no measurable time cost.
One dpct run of upstream main + the seven PRs, 3-way merged into the files they
touch (sycl/tools/merge_upstream.py); their header changes are forked into
sycl/include until upstream merges them.
- #385 stager waits for a buffer's previous DMA on every job (port: polled seq)
- #463 decode waits for an adaptive swap before reading residency
- #453 drafter's batched K/V on the KV-streaming ring
- #374 first chunk's PLE rows read beside layer 0; port fixes: the next read
starts at c0 + T (the port's first chunk is shorter), and the upload is
marked by a polled sequence number instead of a host event wait, which
deadlocked prompts past ~4K tokens under the Level Zero v2 adapter
- #363 group stride + fused SwiGLU/q8_1 for the PCIe expert call, re-applied
to the port's lane kernels (fp contract off, as upstream's HIP form)
- #407 --adapt-tuned (opt-in), #413 key-head DeltaNet recurrence (opt-in via
STRATA_GDN_KEYHEAD=1: 8% slower on the B70; its NVIDIA gate and SM-holding
parity bench not ported)
Tested on the B70: native_grouped_parity, iq_multi, gr, qsa, kv_hybrid,
quantize_act, qsa_prompt_attn parity pass; Coder 19/2,184-token, IQ2_XS and
40K outputs identical to 0.1.33; Coder 2,184-token prompt 784 -> 825 tok/s;
128K int8 KV streaming 895 / 67.0 tok/s; IQ2_XS decode +1.7% (#363).
Co-authored-by: sergqwer, constantindjonkam, architectds, BlueKingMuch (upstream PR authors)
The probe timed one 1 GiB burst; an x16 PCIe 4 laptop (RTX A3000) read 18.5, 6.9 and 5.8 GB/s
in three starts, and the low reading set pcie_frac (0.12-0.15 where calibrate.py found 0.35).
The same 1 GiB is now four bursts of 256 MiB, each timed on its own (events between them on
the stream; on Windows HIP the host clock, #377), and the fastest is the link's figure: a
disturbance only ever slows a copy, while the median would still follow one lasting through
half the bursts. The probe copies no more than before.
The share was base above 20 GB/s and base * bw / 26 below (19.9 GB/s -> 0.42, 20.0 -> 0.55;
4.0 -> 0.08, 3.9 -> 0). It is now base * min(1, bw / 20): continuous, unchanged from 20 GB/s
up (an x16 PCIe 4/5 link reads ~26-28 GB/s, so the default stays 0.55 there), in proportion to
the link below, so the link's time on its share stays about what the x16 share costs. The rule
is one function for CUDA0 and the layer split's stages.
The log names every burst: "PCIe probe: 28.9 GB/s host->device (best of 28.9 28.9 28.9 28.9)"
(the RTX 5070 here; pcie_frac 0.55 as before).
Noted, not changed: a given --pcie-frac is every stage's share and skips the stages' probes.
The engine already leaves --vram-reserve-mib MiB of VRAM free (default 700). INSTALL.md now says what it does, its
default, and how to set it (the flag, or "--vram-reserve-mib", "2048" in the config's args). Setup's new flag writes
it into the config only when given (one value: it replaces the images' own 700, with a note when it is less than the
GPU image encoder needs); on a start of an installed model it is saved into that model's args, and a new copy set up
from an earlier install keeps it. The default config is unchanged (golden test). Tests in test_setup_choices.py.
The engine refuses --resident-budget-gib with a layer split, so setup always dropped UD-Q4_K_XL to one GPU. With RAM
>= the GGUF size + UNSLOTH_RAM_LEFT_GB (~135 GB) and 2+ usable cards, setup now asks (one GPU stays the default and
--yes keeps it; an explicit --gpus is honoured; an explicit --resident-budget-gib keeps one GPU) and the split writes
"gpu": [..], "layer_split": "auto" without the budget: all the experts loaded into RAM at start, the files through the
OS file cache (the config #498 measured: 2x RTX 3090, 31 -> 64-78 tok/s). Latent bug: START-HERE --gpus 0,1 (or the
start-time offer, or a hand-edited config) on a UD-Q4 config kept the budget and the engine exited with code 2; the
split now drops it, or setup stops before saving when the RAM is too small. docs/UNSLOTH_Q4.md says so. Tests in
tools/test_setup_unsloth.py; the golden configs are unchanged.
hf() takes the host from HF_ENDPOINT (default huggingface.co, a trailing slash dropped); the pinned revisions, the
fallback to main and the SHA-256 checks are unchanged. tools/mtp_fetch.py reads the MTP tensors from the same host.
Step 5 prints the folder setup expects the model files in and how to use files downloaded by hand (that folder with
their original names, or --gguf-dir), and the mirror when one is set. INSTALL.md says the same. Tests: the endpoint
in hf()/hf_unpinned()/mtp_fetch, and step 5's words over a mocked install. The golden configs are unchanged.
The server window said "loading the experts into RAM (about N GB)" whatever the flags. Now: --mmap-experts maps them
from the model files (the OS file cache reads them, nothing is loaded up front; its "experts via mmap" line triggers
the message when the model has no PLE line), --resident-experts copies the ones the GPU does not hold, and
--resident-budget-gib N copies up to N GiB. setup's start banner says "maps" for the plain mapped mode too. Tests:
the words per flag set, and the narrator over a mapped and an arena log.
Every start failure, not only the draft-head case (#474), now carries the engine's own reason: the last 20 lines
this start wrote to the engine log (never an earlier start's). The failed start also closes its pipes and its
handle on the log. Tests: the tail itself, and a fake engine that exits with a reason in its log.
A request whose engine prints nothing for engine_silence_s (default 300 s; while a prompt is read, a chunk may take
3x the previous one's time, the first up to its tokens at 50 tok/s more) ends with EngineSilent (an EngineDied): the
engine is killed and the next request starts it again. The STOP drain, under the request FIFO, is no longer an
untimed wait. "engine_silence_s": 0 keeps the old behaviour. Tests: unit (silence, PP chunks, first chunk, STOP
never acknowledged, 0, config) and over HTTP with a fake engine process (non-stream, stream, stop token).
On Windows, hipMemGetInfo (ROCclr's PAL backend) reports the card's size
minus this process's own allocations; what the desktop and other
programs hold on the card is not subtracted. On a card that also drives
the desktop the engine counts the ~930 MiB those hold as free, and
--expert-cache auto fills the card past what Windows lets the process
keep in VRAM: WDDM moves part of it out to system RAM. On an RX 6800
(HIP SDK 7.2) with setup's configuration: a 9.83 GiB cache, the card at
16132 of its 16368 MiB, 929 MiB more of the engine in system RAM than
with this change, 30.1-31.0 tok/s decode. On a 0.1.30 build,
--vram-reserve-mib 1500 left the card as full (16037 MiB), since the
reserve is taken from the same figure.
Windows gives every process a video memory budget that does account
for the others (DXGI QueryVideoMemoryInfo). On that card it is 799 MiB
below the card's size while the process is small and about 1.8 GiB
below once it holds 11 GiB; allocations past it still succeed.
cudaMemGetInfo, which the HIP shim maps to hipMemGetInfo, now lowers
hipMemGetInfo's free figure by what the budget withholds (the card's
size minus the budget), on Windows only. What is sized from it then
stays within the budget: --expert-cache auto sizes the cache from it,
and the existing check after the slots are written shrinks it ("only
8 MiB free once the slots are written ... shrinking"): an 8.05 GiB
cache, the card at 15230 MiB, 40.6-41.5 tok/s.
Lowering HIP's own figure, instead of taking the budget minus DXGI's
usage, keeps memory the HIP runtime has freed but still caches counted
as free, as HIP counts it.
dxgi.dll is loaded at run time, so nothing new is linked; the adapter
is found by the device's LUID. Without dxgi.dll, the adapter or a
budget below the card's size, hipMemGetInfo's figure stands. The first
query logs the budget once. STRATA_WDDM_BUDGET=0 turns it off. CUDA
and Linux builds are unchanged.
The PCIe probe sets the share of the missed experts the GPU reads over
the link (--pcie-frac): the native default 0.55 from 20 GB/s up, less
on a slower link (#44). On Windows HIP it timed its copies with events
that do not bracket them there: an RX 6800 read between 3,348 and
25,999 GB/s in 24 engine starts, so every link kept the x16 share. A
standalone copy of the probe on that card: its events measure 0.13 ms
for the 1 GiB, the host clock 43 ms (25.1 GB/s).
On Windows HIP the probe now times the same copies on the host, between
two synchronizes (1 GiB, tens of milliseconds): the same RX 6800 on
PCIe 4.0 x16 reads 27.2 GB/s in every run, which keeps 0.55 there. On
a PCIe 4.0 x8 or 3.0 x16 link the share now shrinks as #44 intended.
CUDA and Linux keep the events.
device, expert_source, native_head, prefill gemm, prefill.cpp, generate.cpp
merged from dpct of 8554b6b and 5b734e4 with sycl/tools/merge_upstream.py (the
3-way merge restricted to files upstream changed, dpct kernel-name hashes
canonicalized to the port's). Version 0.1.35.
The ring rule: 1024 slots for the Q2_0 pack's fused experts (P3), 512 for native packs with fused layers.
Quality (teacher-forced vs the FP16 path, Q2_0): fused as close as MMQ at 8K, closer at 32K (phaseA-tests s18-tf).
src/prefill/moe_fused_iq.cu: the Q2_0 path's design (moe_fused.cu: the grouping on the GPU, a pointer per expert -
cache slot or ring slot, no gather -, SwiGLU and H's int8 rounding in the gate/up epilogue) for the raw GGUF blobs of
the native packs. The load stage decodes the i-quant blocks to int8 in shared memory (llama.cpp's codebooks and sign
tables, held in shared memory; a 32-value sub-block per thread per 64 values of K, read one stage ahead with 16-bit
loads and two 256-value blocks ahead into L2), with each sub-block's scale (per 16 values for IQ2_XS / IQ2_S, which
multiply at m16n8k16). The activations are int8 per 32 values in natural order (quantize_act_native); fragments by
ldmatrix; the magic bias 1.5 * 2^23 enters as the mma's C, so a dot becomes a float with one add. A work item is 256
weight rows x 64 routed rows, or 128 x 128 when the experts have 56-112 rows each (STRATA_PF_FUSED_TILE=64|128).
Wiring (prefill.cpp): a native layer takes these kernels when both its formats are covered, else MMQ (IQ1_M keeps the
FP16 path); the 512-slot ring of the fused path now also for a native pack with such layers. sm_75, HIP and the
default (no variable) are unchanged: on IQ2_XS, IQ3_XXS and IQ3_S the default output is 0.1.35's (256/256 greedy
tokens, 4K and 32K).
tests/cuda/prefill_fused_iq_test (exit 77 below sm_80), RTX 5070, 256 tokens x 64 experts against a double-precision
reference from ggml's to_float, rel RMS / worst row - MMQ and fused equal to four digits: IQ2_S/Q2_0 1.489e-2 /
2.93e-2 (fused vs MMQ 2.6e-4), IQ2_XXS/Q2_0 1.464e-2 / 2.96e-2 (1.9e-4), IQ2_XS/IQ4_NL 1.465e-2 / 2.41e-2 (3.5e-5),
IQ3_XXS/IQ4_NL 1.458e-2 / 2.73e-2 (1.6e-4), IQ3_S/IQ4_NL 1.470e-2 / 2.65e-2 (6.5e-5), IQ4_XS/Q2_0 1.446e-2 /
2.72e-2 (4.4e-5); both tile shapes. One layer (512 experts, top 10), ms, MMQ path -> fused at 2048 / 3584 / 8192
tokens: IQ2_S 10.5 -> 5.2, 12.5 -> 9.3, 21.4 -> 17.5; IQ2_XXS 8.8 -> 4.2, 10.7 -> 7.6, 17.9 -> 13.6; IQ3_S 10.6 -> 5.6,
12.3 -> 8.8, 19.8 -> 18.4; IQ3_XXS 10.2 -> 5.1, 11.9 -> 8.3, 19.1 -> 16.5. MMQ's products alone (no gathers) at
3584 / 8192: IQ2_S 8.9 / 17.9, IQ3_S 8.1 / 15.6: the kernels are about even with MMQ's, the gain is the path around them.
(The test syncs the device after its uploads: a cudaMemcpy from pageable memory on the legacy stream can still be
queued on Windows while a non-blocking stream runs - the grouping then read zeroed ids, every pair on expert 0.
prefill_fused_moe_test's timing part has the same pattern.)
End to end, RTX 5070, speed_135.py's settings, greedy, A (MMQ) / B (STRATA_PF_FUSED=1) alternating, 2 pairs, prompt
tok/s: IQ2_XS 4K 1,228 -> 1,378 (+12%; a cold first A run of 590 left out), 32K 2,107 -> 2,170 (+3.0%); IQ3_XXS 4K
1,001 -> 1,020 (+1.8%), 32K 1,745 -> 1,739 (-0.4%); IQ3_S 4K 927 -> 925 (-0.2%), 32K 1,382 -> 1,362 (-1.4%). The IQ3
packs' prompts here wait on the host copies of their unpinned experts (IQ3_XXS 4K: 5,062 of 22,573 streamed, ~3.0 s
of 3.4 s), so faster kernels change little. Each arm repeats itself (256/256; IQ3_XXS 32K B and IQ3_S 32K A split at
162 / 223, the decode's timing); A vs B greedy agreement, identical prefix of 256: IQ2_XS 204 / 88, IQ3_XXS 182 / 89,
IQ3_S 31 / 256.
- qsa_block_topk: calls of up to 16 queries (decode windows) run on a cluster of 8 CTAs per query
(block_topk_cluster_kernel): keys in shared memory, digit histograms pushed through distributed shared memory,
one cluster barrier per radix pass, a parallel digit search, cluster-wide offsets for the ordered emit. The
threshold and the cut at it are the one-CTA kernels' (same tie rule, NaN keys 0). STRATA_QSA_CLUSTER=0: off.
RTX 5070 per call: 21.9 -> 15.6 us at 32K, 58 -> 18 at 128K, 200 -> 22 at 262K.
- sample_tokens greedy without penalties: a cluster of 8 CTAs per row (sampler_greedy_cluster_kernel), four loads
in flight per thread, results merged in CTA 0 (lowest index on ties, NaN never taken, 0 when none).
STRATA_ARGMAX_MULTI=0: off. 39.6 -> 5.9 us per call at 248,320 logits.
- Both launch through cudaLaunchKernelEx with the cluster attribute (captured in the decode graphs) only where
they run: CUDA, compute capability 9+ (STRATA_EMULATE_CC respected), the kernel's own PTX/SASS at sm_90+ (an
older build's JIT copy is a trap), room for one cluster - checked once per device. HIP and sm_75/86/89 keep the
one-CTA kernels.
- decode_cluster_parity (ctest; skips below sm_90): bitwise vs the one-CTA dispatch and the reference kernel over
434 top-k cases (contexts 7..262,144, every tail weight, 1-16 queries, ties / NaN / +-inf / -0 / denormal / all
NaN) and 216 argmax cases, plus a CUDA graph capturing both (first calls inside the capture); --bench timings.
With P3's smaller buffers the 1024-slot ring fits beside an 8192-token chunk at 32K (borrowed 3,153 of 3,690 slots,
under the 90% cap). RTX 5070, Q2_0 (cfg-q2rel), --prefill auto, prompt tok/s at 512 / 1024 slots (2 alternating
pairs): 4K 1,512 / 1,605, 32K 2,485 / 2,660, 128K (KV streaming) 2,346 / 2,392 - there the chunk falls from 8192 to
6144 and is still faster. Against MMQ (one more pair on this build): 4K 1,299 -> 1,609 (+23.9%), 32K 2,187 -> 2,656
(+21.5%), 128K 2,128 -> 2,454 (+15.3%). The ring does not change the numbers (the same output as at 512 slots).
Without STRATA_PF_FUSED nothing changes (384 / 96 slots, cap 512).
In a layout whose chunks run the fused experts (STRATA_PF_FUSED=1, the Q2_0 pack, a streamed chunk) GU holds the
grouping tables, H the int8 H rows and Xq the per-token int8 activations, and Hq nothing - instead of MMQ's FP32 GU / H
and per-slot q8_1 rows: ~100 KB a token less in the MoE set, which then no longer sets the shared region's size (the
GDN set does). A last chunk below stream_all_min() still runs MMQ in the same buffers, so each keeps MMQ's size for
stream_all_min() - 1 tokens. The ring may grow to 1024 slots with the fused experts (STRATA_PREFILL_RING; the default
stays 512). Without STRATA_PF_FUSED nothing changes: the same buffers, ring cap (512) and events.
The fused walk now counts the experts it reads from the cache in experts_resident (it stayed 0): every resident expert
of the layer, routed or not - at a streamed chunk's size nearly all are.
RTX 5070, Q2_0 (cfg-q2rel), --prefill auto, 2 alternating pairs, prompt tok/s (borrowed slots):
4K: MMQ 1,305 (1621), fused 1,517 (1749), fused + P3 1,512 (1534), + 1024-slot ring 1,605 (2046)
32K: MMQ 2,177 (3071), fused 2,453 (3199), fused + P3 2,485 (2641), + 1024-slot ring 2,660 (3153)
128K (--max-context 131072 --kv-resident 32768): MMQ 2,108 (chunk 8192), fused 2,140 (6144), fused + P3 2,346
(8192), + 1024-slot ring 2,392 (6144)
A 16K chunk still does not fit the 12 GB card: at 8192 tokens the loan is 2,641 of 3,690 slots, ~2.1K of them per
8K tokens.
The fused path launches on batches of a layer's streamed experts, so with the 384-slot ring a layer's last batch
waited for slots its own first batch frees. With STRATA_PF_FUSED=1 (and the canonical Q2_0 layout) the pinned-RAM
ring is 512 slots; without it nothing changes (384 / 96 as before; STRATA_PREFILL_RING still overrides).
RTX 5070, Q2_0 (cfg-q2rel), greedy, 3 alternating pairs, prompt tok/s: 4K 1,275 -> 1,516 (+18.9%), 32K 2,176 ->
2,422 (+11.3%); at 384 slots the fused path gave +4-7% / +3%. 1024 slots: 4K +24-27%, but the 32K chunk falls
from 8192 to 6144 (+6%). The default path's output is identical to 0.1.35's on both prompts.
Strata's own kernels (src/prefill/moe_fused.cu): the layer's input rounded to int8 once per token (a scale per 32
values), the (token, k) pairs grouped by expert on the GPU (counts, offsets, a 64-row tile list - no host sync),
then per batch of up to 128 experts one gate/up launch (mma.sync m16n8k32 s8, the 2-bit codes unpacked in
registers, Q2_0's offset folded into the activation blocks' sums, SwiGLU and H's int8 rounding in the epilogue)
and one down launch into the per-slot rows the existing combine reads. Experts are read where they are (cache
slot or streamed ring slot, a pointer per expert): no gather.
Used only on the streamed walk (chunks from stream_all_min()) of the canonical Q2_0 layout on sm_80+ CUDA, with
STRATA_PF_FUSED=1; it reuses the MMQ buffers (Xq, H, GU). Everything else - the default, native/IQ packs, small
chunks, sm_75, HIP - runs the MMQ path unchanged (the old block is only wrapped in the else branch).
tests/cuda/prefill_fused_moe_test (exit 77 below sm_80), RTX 5070: vs a double-precision reference, MMQ rel RMS
1.144e-2 (worst row 4.15e-2), fused 1.144e-2 (4.15e-2); fused vs MMQ 9.0e-5. One layer at 2048 tokens x top 10
over 512 experts: MMQ path 11.12 ms, fused 3.13 ms.
`--expert-profile-save P` (--serve) writes the adaptive tier's learning as a profile in tools/make_profile.py's
format, byte for byte (read_expert_profile reads it back): every pair ranked - the experts resident in VRAM now
first, then the routing counted since the start (the decayed usage summed at each adapt step, before its decay),
then the profile the engine started from. On QUIT / stdin EOF and every --expert-profile-save-every minutes
(default 10, 0 = exit only) between requests, through <P>.tmp and a rename. Without the flag nothing is counted or
written: `heat` stays empty and every added loop is over it.
Server: "expert_profile_save" (and "expert_profile_save_every") in the config pass the flags, and the next start
begins from the saved file instead of the config's --expert-profile when its header matches that profile's
layers x experts. tests/core/expert_profile_save_test.cpp (CPU only) checks the order and the bytes.
In a git clone they run `git pull --ff-only` (a zip copy is told to download the new zip), then `setup.py --update`:
what a plain start does to an install before it starts the model - the Python packages, the ready-made engine when
MIN_ENGINE is newer (a compiled one when its source changed), each model's config upgrades and its draft subset, the
Windows AMD runtime - and nothing else: no question, no model files, no start. Both scripts run from one parsed
block, since the pull can rewrite the running script. Documented in docs/INSTALL.md (Updating) and the README.
Engine: when the MTP draft head's subset does not fit, it prints what the head needs, the VRAM free, and the smaller
shipped subsets (--draft-vocab cyrillic / en, their MiB from the head's real row size) with how to pick one
(START-HERE.bat / ./setup.sh --draft-vocab en, kept in the model's config). Text only, on the failure path.
Server: a start that ended that way carries the engine's hint (or the same advice for an older engine) in its
"the engine exited before it was ready" error. Setup: a card under 14 GB with the default subset gets a note
recommending --draft-vocab en; nothing is changed, an explicit or saved choice gets no note, and the golden
configs are unchanged.
The engine's `strata serve: prompt ...` line now says "= 0 reused + 12288 of 98179 read in ... (cancelled)" with the
rate from the tokens read, and DONE gets a 15th field: the prompt tokens actually read. The server's /metrics history
and totals count reused + read for a cancelled read (prompt_total keeps the whole prompt); an older engine's DONE line
(no such field) is read as before.
The fill from the GGUF (--stream-experts) read each expert's three slices,
copied the blob and waited for the copy before the next read: ~12,000-18,500
experts with neither the SSD nor PCIe kept busy. Now the slots are admitted
in profile order first (same placement), reads go in file order by up to 8
threads into page-locked batches of 64, and a batch's copies overlap the next
batch's reads. STRATA_FILL_SERIAL=1 keeps the serial fill.
Tested on the B70: cold page cache, Coder fill 23.4 GiB 76.2 s -> 18.8 s;
launch to first token Coder 82 s -> 26 s, IQ2_XS 120 s -> 41 s; outputs
identical to the serial fill.
README and docs/MODELS.md said only "Weaker outside coding". #438 measured Chinese answers wrong or looping on the
Coder where English was fine; both now say plainly that this includes Chinese and other CJK text and point to the
sizes that keep every expert (Q2_0, IQ2_XS, IQ3_S).
With Unsloth's UD-IQ3_XXS or UD-Q2_K_XL in the folder, setup either said only "has no IQ3_XXS model file"
(--family unsloth --model IQ3_XXS) or took the UD-IQ3_XXS file for the GSQ-RCO IQ3_XXS by the size in its name.
Now:
- a chosen shard whose name is a quantization setup has no size for (UD-IQ3_XXS, UD-Q2_K_XL, a K-quant) stops
with "<file> is UD-IQ3_XXS, a GGUF Strata cannot run" and what Strata runs: ISTA-DASLab's GSQ-RCO files (and
Swift 1.5's and the Coder's) and Unsloth's UD-Q4_K_XL only;
- a folder without the chosen file but with other GGUFs lists the ones Strata cannot use and the --family/--model
of the ones it can (an empty folder keeps check_shards' "missing");
- a size the family does not have names the families that do (--family qwen --model IQ3_XXS), and with
--gguf-dir also what Strata runs.
The server already read drafts_offered / drafts_accepted from the engine's DONE line but only returned them in a
response's timings (draft_n / draft_n_accepted). Each /metrics request entry now carries both (null when the
engine did not report them) and the totals keep their running sums since the server started, so a dashboard such
as llm-visuals can show the acceptance. Additive: no existing field changes.
- engine (layer splits only): when a stage's cache cannot fund the chunk CUDA0 alone would choose, a WARNING
names it (CUDA index, name, cache slots, the slots the chunk borrows) and points at the helper cache
(--expert-cache-device1, docs/SECOND_GPU.md). The chunk choice itself is unchanged: fits() is split into
fits_one() with the same tests; one GPU never reaches the new log.
- setup: a pair whose second card is under 12 GB beside a first card of 12 GB or more is still offered, but
the first card alone is recommended (the default with --yes and Enter; offer_together asks "n" by default),
with a note. A named --gpus pair is kept. Equal pairs (2x 8/16/24 GB) are unchanged.
- The automatic drop of the small card from an auto split is not done: stages load their weights before the
boundary search and the exact prompt-buffer bytes need the sessions made after it.
tools/calibrate.py read any "gpu" list as a layer split and added --layer-split auto, so a one-card config with a
helper card for the expert tier ("gpu": [0] + --expert-cache-device1) was refused by the engine before it read a
weight, a "0,2" string never got its split, and split_skip_if_fits was ignored. The tuner now builds the arguments
with serve.server.engine_args, the same as a normal start.
When the tuning fails, setup prints the engine's own last error line from the log (only lines written during the
tuning), and says the failure again where it is not lost: before --calibrate starts the model, and in the
"All set." summary of an install that tuned.
rocm_root took /opt/rocm when bin/hipcc and lib/libhipblas.so* were there and the version was 7.0 or newer. A
runtime-only ROCm 7 (no -dev packages) has both, and the engine's cmake configure then failed on the missing
hip-lang package. The system ROCm now also needs lib/cmake/hip-lang/hip-lang-config.cmake (lib, lib64 or the
multiarch lib folder, where CMake looks) and include/hip/hip_runtime.h; without them setup warns, naming what is
missing, and takes the TheRock wheels path, as it already does for a ROCm older than 7.0.
A fresh copy of Strata set itself up like the newest strata-*.json of another Strata folder and crashed with
JSONDecodeError when that file was empty or cut off. previous_config now takes the newest one that parses as a JSON
object; each one that does not gets a one-line warning naming it, and with none the copy is set up as a fresh
install. Every place setup writes a run config now goes through write_config (a .tmp file, then os.replace), so a
setup stopped half-way no longer leaves an empty config behind.
- Windows: the process working set is trimmed before the resident complement's RAM reading, so the experts
the GPU cache pre-fill touched through the mapping count as available again (standby list); one line
prints the reading before and after.
- --resident-experts (soft): when the whole complement does not fit, a second try keeps what fits, by
profile rank, through the #403 budget path (RAM room minus the 256 MiB margin) instead of falling back
to mmap entirely; a WARNING names it. Same file bytes either way, so the answers are unchanged.
- Only the resident mode calls pin_cache_complement: the default and mmap paths are untouched.
Windows looks for an imported DLL in the exe's folder, then System32, then PATH (where the zip's rocm/bin is): a
driver that installs its own amdhip64_7.dll in System32 won, and the bundled rocBLAS/hipBLAS ran on that runtime - an
access violation on a W7900, hipErrorInvalidDeviceFunction on a 7900 XTX, on the first prompt. setup copies
amdhip64_7.dll and amd_comgr.dll from engine/rocm/bin next to the exes after the unzip, before the device probe and
on every start (so a 0.1.34 install is fixed too); package_windows.py ships them there; the engine and strata-device
log which runtime they loaded. rocBLAS/hipBLASLt stay in rocm/bin (their kernel libraries and .kpack are found from
there).
openai_to_messages and anthropic_to_messages iterated req["messages"] directly, so a client that double-encodes
the array as a JSON string had its characters iterated and the request crashed on m.get. The new _object_list
decodes such a string and requires a list of objects; anything else raises ValueError("messages must be a list
of objects"), which the server answers with a 400. A message's tool_calls get the same guard.
- gguf_expert_source: ExpertLayout::gguf_file is per layer and role
(3 * layer + role) since upstream 0.1.31; the port indexed it by layer, so
experts in shard 2 were read from shard 1 at shard-2 offsets (Swift 1.5:
layers 13-47, NaN logits from layer 13). One fd per layer and role now.
- iq_kernels: vec_dot_iq2_xs and Split<17> took their 16-bit codes through
a uint16_t* to an int2 (type punning the device compiler does not honour);
by shifts now. iq_multi_parity checks IQ2_XS against a CPU decode from
ggml's tables (iq2xs_host_tables.inc) and passes.
- prefill: STRATA_DBG_NAN=1 also reports the experts' fp16 GEMM inputs.
Tested on the B70: Swift 1.5 IQ2_XS answers correctly (48 tok/s raw, short
thinking); the original IQ2_XS output is unchanged (65109a, 58 tok/s).
expert_cache.cpp, expert_source.cpp, fused_gr, qsa_prompt_attn, gr_parity and
generate.cpp merged from a fresh dpct of 2ca1b66 and 8554b6b; files that only
differed by dpct run-to-run noise left alone. Conflicts: --resident-cpu-experts
(upstream's resident_cpu_explicit kept with --stream-experts) and the prompt
attention's capability check (the port's XMX dispatch kept). Version 0.1.33.
Tested on the B70: build clean; gr_parity and qsa_prompt_attn_parity pass;
Coder short/long, IQ2_XS and the 40K prompt (borrowing) outputs identical,
78.3 / 75.7 / 58.0 / 68.9 tok/s. setup_intel.py --check and an install run, and
server_intel.py with the mock engine, against the new setup.py / server.py.
The hang was in the streamed-expert prompt path, which borrowing is the only
way to reach on the Coder: the stager threads waited on the copy queue's
events to reuse their 16 page-locked buffers, and under the Level Zero v2
adapter that left another queue's barrier on the same event waiting forever
once the ring wrapped. The copy queue now writes a sequence number into
page-locked memory after each DMA and the stager polls it.
- prefill.cpp: Stager::issued_one / buffer_free; STRATA_PREFILL_SYNC=1 debug
switch (wait and log after every phase)
- generate.cpp: borrowing is the default above a 32K context again
Tested on the B70: 40K prompt with borrowing 1,117-1,144 tok/s, decode 69
(output identical to the run without borrowing); 128K int8/q4_0/k8v4 decode
66.6/62.8/66.8 (from 59.5/53.4/54.4), 256K 55.9/51.7/52.2 (from
48.7/43.2/42.3); Coder and IQ2_XS 32K outputs unchanged.
llama.cpp's tile choice aborts the process ("J_best=0") when no tile size of a type fits the card's shared memory.
The plan now asks the same question for every visible GPU (the experts' gate+up and down shapes, the draft layer's
Q8_0 projections) and keeps a product with no tile on the existing FP16 path, saying which type, card and limits.
Where every product fits (all cards tested here), the plan is the same as before.
#429 (jkuepker): the HIP engine has no prompt kernels for its Q4_K/Q5_K experts and it has not been run on AMD -
--check says so, and setup asks (--model UD-Q4_K_XL --yes tries it). #425 (jctaborda): the finished shards and the
.part files already on the disk count against the download's size.
A non-streamed request wrote nothing until its end, and a streamed one only a keep-alive per prompt chunk (the
first write after a hang-up usually still succeeds), so a dropped request kept the engine reading the whole prompt
or generating to max_tokens while the next request waited. A watcher per request checks the client's socket for EOF
every 0.5 s and cancels; the engine then stops at its next prompt chunk.
dpct of 6ac1a2c and 2ca1b66, normalized, kernel-name hashes canonicalized to
the port's, then git merge-file into sycl/: 49 files, 10 with real conflicts.
- verify: upstream's set_commit_async/wait_commit on the port's deferred
commit (wait_commit = commit_finish); copy_logits on the queue
- fused_gr: upstream's split/staged read variants merged but unused by the
port's read; their per-card self-test segfaults on the B70, so on SYCL it
runs only with STRATA_HC_CHECK=1. The port's split-norm kernel renamed
(upstream has one of the same name); two colliding kernel names renamed
- fixups.py: fused_gr's per-block local-memory query
- hand files: mtp.cpp, verify.cpp, ple_reader.cpp (keep-alive reads + trace)
- qsa_prompt_attn_parity turns on the XMX kernel it checks
- version 0.1.32
Tested on the B70: build clean; Coder (19 / 2,184-token prompts) and IQ2_XS
outputs identical to 0.1.31 at the same speed (78.1 / 75.7, 58.5 tok/s);
40K prompt 1,201 tok/s, decode 65.1; kv_hybrid, qsa_prompt_attn, qsa, gr,
kv_q4, kv_q8, rope, router, sampler, quantize_act parity pass.
Every file outside sycl/ (and docs/INTEL.md) is now upstream's, byte for byte.
The Intel pieces that used to edit shared files move under sycl/ and wrap
upstream from outside:
- sycl/setup_intel.py: runs setup.py's main() with the GPU check, engine
step, RAM rule and config writer swapped for the SYCL port's
- sycl/serve/server_intel.py: serve/server.py plus the Arc's Monitor
readings (sycl/serve/xe_telemetry.py) and the optional model menu
(sycl/serve/web/switcher.js)
- sycl/tools/gpustat.py (was tools/intel/), sycl/.gitignore; migrate.sh
writes its compilation database outside the checkout
- removed: the llama.cpp engine path (serve/engine_llama.py), the README box
Engine sources in sycl/ still track 0.1.31; the 0.1.32 kernels are not
migrated yet.
- fixups.py: default-queue memcpy/memset in the parity tests first waits for
all device queues (cudaMemcpy's legacy-stream semantics), applied to every
src/kernels/*_parity.cpp
- generate.cpp: the post-prompt refill loops sync the queued copies every 256
entries so the ring cannot wrap over in-flight slots
- kv_hybrid_parity: enable STRATA_PROMPT_ATTN_XMX (step 5 needs the XMX path)
Tested on the B70: build clean; qsa_parity, gr_parity OK.
The recommended answers stay the defaults, and --yes / Enter write the same config as before
(tools/test_setup_golden.py, five PCs x five sizes). What a user asks for beyond the recommendation is kept and
its risk said:
- #406: the RAM rule for IQ3_XXS / IQ3_S (experts + KV + 24 GB) is part of the recommended context (the smaller of
it and the GPU's rule), no longer a cap: an explicit --context, a pick in the list or the earlier install's
context is kept with a note ("256K with IQ3_S needs ~78 GB of RAM by setup's estimate; this PC has 64. Kept as
you chose ..."), and the list marks the sizes past it. The low-RAM mode keeps its KV cache in VRAM, so there
the context does not count against RAM.
- #364#384: the low-RAM mode recommends one GPU (the resident variant has no layer split) and asks: 2 = all the
chosen GPUs with the experts read through the OS file cache (--mmap-experts, layer_split auto; faster in both
reports, RAM can fill up during long prompts). An explicit --gpus (or the earlier install's cards) is kept;
--yes keeps one GPU. The GPUs' share counts every card. The one-GPU text no longer sends users to --low-ram off.
At start, --gpus on a --resident-experts config (and offer_together's "y", whose default is now n for such a
config) switches it to --mmap-experts with a note instead of an engine refusal.
- S4: --resident-budget-gib N for UD-Q4_K_XL (default: the RAM less 24 GB, as before); more is kept with a note,
and a given N is not lowered for KV streaming.
- confirm_risk(msg, explicit, yes): one helper for the stops a user may take on - asked with default n, and with
--yes an explicit flag is the consent (--yes alone keeps the stop): UD-Q4_K_XL under 48 GB (--model), the RAM
floor (--model), the paging question (--model, besides --low-ram off), a card under 8 GB named with --gpus.
- --kv-streaming on|off|auto (auto: the RAM test, as before); on past the RAM test with a note, never with k8v4 or
under WSL.
The refusals that stay are the impossible ones: rope none past 262144, disk space, engine version, driver < 580,
no AVX2, a size outside the family.
The low-RAM mode's resident variant has no layer split yet, and the engine refused the pair. When the residency
comes only from setup's soft --resident-experts (no --resident-cpu-experts, no --resident-budget-gib, no remote
caches), it now prints a WARNING - the experts the GPUs do not hold are read through the OS file cache and RAM may
fill up during long prompts - and continues exactly as --mmap-experts: the resident flags are cleared before
anything reads them, so the source, the placement and the per-stage caches are the mapped mode's (#364's hand-edited
config). An explicit --resident-cpu-experts, a budget, or remote caches are still refused.
The explicit --expert-cache reduced under a layer split is now a WARNING that names --vram-reserve-mib.
FileExpertSource::pin_cache_complement read the available RAM twice: the budget was clamped to the first reading
(less the 4 GiB headroom) and the resident-memory check then failed it on a second reading a few MB lower, so a
64 GB PC with setup's --resident-budget-gib 40 could exit at start. With a budget, the check now reuses the reading
the budget was sized from, and a clamped budget stops 256 MiB under the room. An unclamped budget holds the same
experts as before (the lend region is off with a budget, so `budget` there is unused).
If the budget still cannot be kept (no RAM reading, an allocation that fails), generate.cpp warns and serves every
expert the GPU does not hold from the files through the OS file cache (--mmap-experts) instead of exiting; the
complement is left unbuilt on every failure path.
setup.main() with --yes, and with Enter for every question, on 64 GB + 32 GB, 32 GB + 2x24 GB, 47 GB + 2x16 GB,
96 GB + 16 GB and 128 GB + 24 GB, for the recommended size, Q2_0, IQ3_XXS, IQ3_S and UD-Q4_K_XL: the run config
recorded by this setup (tools/test_setup_golden.json), so the choices that follow cannot change the default.
With STRATA_GR_V3=1 "fused GR multi max-T=8 LDS launch and changing graph replay" failed: the test compares the
multi-token read with the single-token kernel bit for bit, and v3 sums `down` in another order (row group x stream
blocks). It also compared `lo`, the default kernels' workspace between down and up, which v3 keeps in shared
memory and never writes. Under v3 the test now checks R_out, rs, inject and mixed within 2e-6 of the largest
reference value and leaves `lo` out; without it, bit for bit as before. Passes both ways.
open_sized seeded every layer's cursor at slot 0, so all layers wrote the same slots and the startup check failed;
the profile fill ended at the first full layer. Now each layer starts in its own range, a full layer skips only
its own pairs, and per-layer mode keeps slots of the largest blob (its ranges ignore the rank a sized slot was cut
for). The default (shared) cache is unchanged.
The host check compared the major version only (>= 7) while the device code traps below sm_75, so an sm_70 card
took the m16n8k8 kernel and hit __trap(). sm_75 and newer: the same choice as before.
START-HERE.bat on a Windows PC whose AMD Radeon is the card Strata can use now
sets it up like an NVIDIA one: no compiler, no ROCm install, only the AMD driver.
- Detection without any AMD software: the display adapters present
(Win32_VideoController: name + PCI id) with each one's 64-bit VRAM size from
the display-class registry (readable without admin; #247/#325's source),
the arch by PCI device id, else by name. An integrated or older Radeon is
listed and refused, as on Linux. The NVIDIA-or-AMD choice is the Linux one:
AMD by itself when no NVIDIA card can be used, asked when both can.
- The engine: strata-windows-x64-hip.zip from the release (this version's,
then the latest; --prebuilt for a local folder), unpacked into engine/ and
checked for this card's arch and version >= 0.1.33; an older one is
replaced on a later start like the NVIDIA zip. Without a published zip setup
says how to make one (tools\hip\build_windows.bat + --prebuilt).
- Before the model download, the engine's own strata-device --list-devices
(with its rocm/bin on PATH) must see the card - otherwise it stops and
points to the AMD driver - and gives the card's HIP ordinal (#325: an
integrated Radeon is HIP's device 0). From then on setup lists the AMD
cards in HIP's order, so "gpu" in the config and --gpu N are HIP numbers.
- The config: backend hip, lib_dirs = engine\rocm\bin, the hipBLASLt table
for the zip's hipBLASLt version (BUILD.json). Images on the CPU and
several AMD cards (--gpus) stay Linux-only for now (said, not silent).
- Texts: setup's description and step 1 name NVIDIA or AMD; START-HERE.bat
needs "an NVIDIA or AMD graphics driver"; the "AMD backend runs on Linux
only" stop is gone.
- An installed HIP engine's BUILD.json ("gfx1201" archs) is no longer read as
an NVIDIA one by get_prebuilt.
Linux is unchanged (KFD detection, TheRock wheels, compiled here).
Tests: tools/test_setup_amd.py WindowsDetection (mocked WMI/registry rows,
the list-devices parser on this PC's real output, the iGPU-first ordinal
match, the driver-missing stop, the zip install/refusals, the dispatch);
test_setup_choices' Linux CPU-encoder case pins WIN=False.
Co-authored-by: Dimitri Vasdekis <[email protected]>
Co-authored-by: isshin <[email protected]>
build_windows.bat builds strata.exe + strata-device.exe (and with `tests` every
HIP test) for gfx1100;gfx1101;gfx1102;gfx1200;gfx1201;gfx1030 with no admin
rights and no AMD GPU: ROCm 10.2.0a20260930 from AMD's TheRock wheels
(whl-next: rocm[libraries,devel,device-<arch>...], the version #325 ran on an
RX 9070 XT) in a private venv (.rocm-win, ~7 GB), ROCm's clang as host and HIP
compiler with MSVC's headers/linker from vcvars, STRATA_PORTABLE (AVX2 ggml).
package_windows.py makes dist/strata-windows-x64-hip.zip (524 MiB, 810 MiB
unpacked): strata.exe (61 MiB), strata-device.exe, BUILD.json (backend hip,
archs, ROCm and hipBLASLt versions, lib_dirs rocm/bin), rocm/bin with the DLLs
the programs import followed through ROCm (amdhip64_7, hipblas, rocblas,
rocsolver, libhipblaslt, libtensilelite-host, origami, rocm_kpack) + amd_comgr
(loaded by name) + the MSVC runtime, rocBLAS/hipBLASLt kernels for these archs,
rocBLAS's kpack device code (found at ../.kpack from rocblas.dll), licenses and
a NOTICE. The user needs only the AMD driver (the HIP runtime uses its
user-mode components).
Checked on this PC's Ryzen iGPU (gfx1036, a test build of the same tree): with
only the zip's rocm/bin on PATH, strata-device lists the card and
hipblasGemmEx BF16 -> FP32 (T=100) matches the CPU (max err 4.7e-7); ctest
52 of 56 pass, the 4 failures the same as #325's RX 9070 XT run (hip_handoff,
and ple_parity / expert_parity / pool_test without the pack fixture).
- prefill gemm: on Windows hipBLAS can return success with the right BF16/FP16
product and still leave hipErrorInvalidValue set (#247 on gfx1201: hc up once
T >= 96, the router), which the next kernel check turns into an exit. That
one stale error is cleared after a GEMM that succeeded; any other error
still stops the engine. Only under __HIPCC__ && _WIN32: on Linux a stale
hipErrorInvalidValue is a real error and keeps being reported (our review
of #247). CUDA compiles a ((void) 0).
- native embedding (HIP builds): when the runtime refuses a device alias for
the mapped table, the table goes into VRAM (as when it cannot be pinned)
instead of failing the start; the pinned copy is freed so the destructor's
one-of-two rule holds (#325's version leaked it and freed dev_owned_ twice).
CUDA keeps the old code path.
- tests/hip/mapped_alias (#325): what a hipHostGetDevicePointer alias is on
this stack. On Windows it is the host pointer itself, and a D2D copy into it
does not land - hip_handoff's timeout there; kernels reading it work (the
engine serves on Windows). Rewritten to read the kernel's output back with
hipMemcpy (the PR's version dereferenced device memory on the host) and to
try the kernel read in both cases. Diagnostic: exits 0, 77 without a device.
Co-authored-by: Dimitri Vasdekis <[email protected]>
Co-authored-by: isshin <[email protected]>
On Windows the AMD cards are found in the display-adapter order, but
HIP_VISIBLE_DEVICES counts the devices the HIP runtime enumerates, and an
integrated Radeon takes HIP's device 0 and pushes the discrete card to 1 - so
the engine was pointed at the iGPU, which the arch check then refuses (found
by dvasdekis on an RX 9070 XT + Ryzen iGPU).
- device_count() and device_summary(): enumerate without device_info's arch
check (the cards this binary has no code for are part of the answer).
- strata-device --list-devices: every device the runtime enumerates, as
"device N: <name>", then "arch gfx1201, 15.9 GiB, wave32" (CUDA: compute
capability and VRAM), then "cannot run: <why>" for a card this binary
cannot run. setup.py parses it (next commit).
- server.py hip_visible(): a one-card config's "hip_ordinal" (written by
#325's branch) wins over "gpu"; a layer split keeps its list.
Checked on this PC's own Ryzen iGPU through TheRock's Windows HIP runtime:
"device 0: AMD Radeon(TM) Graphics / arch gfx1036, 28.1 GiB, wave32 / cannot
run: ... not an architecture this Strata engine was compiled for (...)".
Co-authored-by: Dimitri Vasdekis <[email protected]>
CMake refuses to mix cl.exe with Clang for HIP, so on Windows ROCm's clang++
compiles the host code too (targeting the MSVC ABI, with MSVC's headers and
linker from vcvars). The CUDA->HIP shim is force-included with /FI when the
host front end is MSVC-style (cl / clang-cl) and with -include otherwise, its
path in forward slashes; CMAKE_HIP_ARCHITECTURES may also be space-separated.
On Windows the shim keeps <windows.h>'s min/max macros out (NOMINMAX,
WIN32_LEAN_AND_MEAN).
Taken from jagsan-cyber's #247 as ported by dvasdekis in #325 (2373c75); the
arch lists and the arch check stay 0.1.32's. Built on this PC with ROCm
10.2.0a20260930 (TheRock wheels, the version #325 ran on an RX 9070 XT) for
gfx1100;gfx1101;gfx1102;gfx1200;gfx1201;gfx1030: strata.exe and all 253
targets with the tests. CUDA builds do not read these files.
Co-authored-by: Dimitri Vasdekis <[email protected]>
Claude Code, Claude Desktop, Cursor, VS Code, Codex ... add it over stdio (JSON-RPC 2.0, MCP 2025-06-18, also
2025-03-26 / 2024-11-05); standard library only, so it runs before setup made .venv. Eight tools: strata_status
(running server via /health, /v1/status, /status, /metrics; installed configs, engine, data folder; hardware and
setup's recommendation), strata_models, strata_install (setup --yes --no-start in the background, plan first and
confirm=true to start, progress / cancel on later calls), strata_start (the run script's command, waits for
/health), strata_stop (POST /unload, then ends the server it started), strata_logs, strata_benchmark,
strata_connect_info.
Every argument is checked against a schema and setup's own tables; the only path (data_dir) must be the data
folder, inside the Strata folder, or a new Strata* folder outside system folders. Processes are tracked by pid and
start time in .strata-mcp/, so only processes it started are ever ended; a Strata started elsewhere is only
unloaded. docs/MCP_SERVER.md has every client's config; docs/DETAILS.md links it. Tests use a fake setup and
server (Windows and Linux).
README.md (279 -> ~180 lines): what it is, speed on an NVIDIA and an AMD baseline PC side by side (the AMD
numbers are TODO-AMD placeholders until the RX 9070 XT baseline is measured), what you need with NVIDIA and AMD
as equals, install by asking an AI assistant (docs/AI_SETUP.md, MCP server) or by hand, pick-by-RAM table,
using it, the four common problems, how it works with the two diagrams, Buy Me a Coffee button below the speed
table and at the end. Its anchors other pages use (#which-model-should-i-pick, #install, #using-it,
#something-went-wrong, #how-does-it-work) are kept.
Everything the README had moves, nothing dropped: docs/INSTALL.md (requirements, Windows, Linux, AMD cards,
several cards, Docker, updating, where things are stored, setup's questions incl. rope scaling / images /
speed projection, calibrate, all options), docs/MODELS.md (speed per size, sizes, will it fit, low-RAM, Coder,
Swift 1.5, Unsloth UD-Q4_K_XL, Orca), docs/TROUBLESHOOTING.md (the README's problems plus AMD ones),
docs/HOW_IT_WORKS.md (the longer explanation, credits, license). New docs/AI_SETUP.md (instructions for an AI
agent installing Strata for a user) and AGENTS.md. .github/FUNDING.yml (buy_me_a_coffee: strataengine).
AMD_HIP.md loses "Experimental" in its title and intro; DETAILS.md lists the AMD cards and points to the new pages.
--backend's help, the NVIDIA-or-AMD question and the GPU line drop "experimental" (the Unsloth model, the
speed projection and the Pascal/Volta build keep it); the AMD choice says images run on the CPU (0.1.32)
instead of "no images". setup.sh's header comment likewise.
Re-migrated (dpct of 0.1.29 and 0.1.31, 3-way merge per file): 45 files
changed upstream, 38 conflicts resolved; verify.cpp, mtp.cpp and the new
conversation_state.cpp ported by hand; 7 new upstream files.
Fixed on the way:
- the port's grouped expert kernels stay the default: upstream's new multi
kernels (8 rows a group) ran on the port's 32-row grid and wrote a quarter
of each expert (end-of-text output); STRATA_EXPERT_SPLIT=1 runs them with
their own grid. Lanes per row now selectable (STRATA_GU/DOWN_LANES)
- dpct's unparenthesised __fadd_rn: the native QSA indexer's per-token
pooled key (decode), the native QSA scores and the s2 activation rounding
replaced instead of added - from the first migration; fixups.py fixed
- doorbell waits: fixups.py bounds upstream's new spin spelling again
- prompt-slot borrowing hangs since upstream's rework: off by default
- the Turing PTX branch in qsa_prompt_attn, dp4a.hpp's __nanosleep, the
coupled-draft capture signature, a duplicate kernel name
- gr_parity: the single-token GR read runs the multi kernels (bitwise)
- qsa_parity: a cross-queue copy race in the test itself
Decode 78.2 / 75.7 tok/s (Coder), 58.6 / 64.2 (IQ2_XS); 128K-256K complete.
Two changes to the prompt path's QSA selection (qsa_select.cu), measured on a Radeon AI PRO R9700 (gfx1201) with
--max-context 262144:
1. qsa_block_topk picked its register kernel by max_blocks, a CAPACITY (--max-context / 4 + 2). With a 262144-cell
capacity every prompt, even a 4K one, took the slow reference top-k (up to ~10x slower at 128K+). The prompt path
now passes the batch's active block count (the argument qsa_block_scores already takes); decode passes none and
keeps the capacity rule and the original register width, so it is unchanged. On HIP the register kernel is
instantiated for 66 blocks per thread (contexts up to 270,336 cells, the 262144 --max-context) and, below
~28K cells, where it measured 0.2-0.9x of the reference on gfx1201, the reference is kept. Same ids as the reference
(checked 256/256 at every size).
2. qsa_block_scores_tc on gfx1200/gfx1201: block_scores_wmma_kernel on v_wmma_f32_16x16x16_bf16 (wave32). Each operand
is split into three bf16 slices (hi / mid / lo are exact 8-bit mantissa slices of the fp32, so no scaling is
needed) and the six products of order <= 2 are summed; the hi*hi chain keeps its own accumulator and the five
corrections another, added once at the end (a single shared accumulator rounded the corrections at the big
accumulator's ulp: 8e-7 of the score scale, failing the harness's gate at short context; split: 1.6e-7, the warp
kernel's own error is of the same order). The tail block n_bid is the warp kernel's arithmetic. Like the TF32
kernel it is not bitwise with the warp kernel (a near-tie can select differently). Gated at compile time
(__gfx1200__/__gfx1201__) and run time (gcnArchName); other HIP targets keep the warp kernel; CUDA is unchanged.
qsa_select_bench now builds on HIP, takes the engine's capacity as a 4th argument, checks the scorer against an FP64
reference (gate: no worse than 4x the warp kernel's error, floored at 1e-6 of the score scale), exits non-zero on a
failed gate, and is registered with ctest (32K context under a 262144-cell capacity).
R9700, synthetic bench, 256 queries, capacity 262144: scores 0.81 -> 0.18 ms at 32K, 3.5 -> 0.65 ms at 131K, 6.8 ->
1.5 ms at 262K; top-k 3.9 -> 0.51 ms at 131K, 12.9 -> 1.3 ms at 262K. End to end (Swift 1.5 IQ3_XXS, --kv int8,
prefill timing, same binary with STRATA_SELECT_OLD=1 STRATA_TOPK_OLD=1 as the old arm): 107.7K-token prompt 79.5 s ->
59.5 s (1.34x; qsa select 23.3 s -> 3.4 s); 30.9K 1.02x; below 10K about 1.01x. ctest 44/44.
dp4a through the plain signed v_dot4_i32_i8 (RDNA2 has no sudot4).
setup knows the card and the gfx103X-all TheRock index.
docs carry a community report (RX 6900 XT 16 GB: complete build, 28/28
ctest, 38-42 tok/s decode at 131,072 context, 330-339 tok/s prefill)
docs/AMD_HIP.md: the RDNA4 validation note still said there was no gfx1201 table and none was shipped; it now says
that holds for hipBLASLt 1.4.1 and points to the 1.5.0 table and its measured gain. The "Tuning table" text no longer
says the parity test "exits non-zero when the table does not fit". It describes what the test checks (architecture and
version through the loader, two of the 32 rows against hipBLASEx) and what it does not: a solution id the library
rejects falls back to hipBLASEx and the test still passes, so it says to read STRATA_HIPBLASLT_VERBOSE's
"fallbacks=" summary and to recalibrate for another 1.5.0 build.
tests/hip/prefill_hipblaslt_gemm.cpp: the header comment, the SKIP and missing-row messages and the case comment now
say it is a smoke test over two rows, and it prints "smoke test: 4 cases on 2 of the table's N rows" at the end.
Test behaviour and exit codes are unchanged.
Clarify usage of hip_prefill_hipblaslt_gemm and tuning process for different 1.5.0 builds.
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Both helpers built their BF16 inputs with hip_bfloat16(x).data. Compiled as host C++ (no __HIPCC__) the SDK's
hip_bfloat16 is a struct holding a uint16_t, and the conversion from a float is not the engine's round-to-nearest-even
BF16 encoding (kernels/bf16_bits.hpp has that one).
Observed effect on gfx1201 with ROCm 10.2.0a, parity test with the 100500 table: the T=37 beta=1 BF16 case printed
relative_l2=0 max_abs=0 with the old encoding and relative_l2=3.6e-07 max_abs=2.6e-06 with this one; the T=4096 BF16
case prints 0 either way, and the f16 cases are unchanged. I did not inspect the old encoder's bit patterns directly,
so the exact fault is inferred from that difference, not shown.
Use strata::kernels::bf16_from_f32() in both, so the BF16 rows of gfx1201-hipblaslt-100500.txt were calibrated on
intended inputs and anyone regenerating a table gets the same. No change to the engine or to the f16 path.
tools/hip/gfx1201-hipblaslt-100500.txt: 16 dense GEMM geometries (bf16 and f16) at T=4096 and T=8192, 32 rows,
calibrated on an R9700 with tune_hipblaslt (16 candidates per row, relative L2 <= 1e-4, max abs <= 1e-2, no padding
writes) against rocm-nightly-gfx120x-all 10.2.0a20260914, hipBLASLt 1.5.0 (version number 100500).
The engine refuses a table of another hipBLASLt version, so the existing gfx1201 table in #262 (100401, hipBLASLt
1.4.1) is not used on this ROCm; setup picks tools/hip/<arch>-hipblaslt-<version>.txt by the installed version, so both
can be shipped side by side. Compared with #262's 24 rows this table has the same 24 shapes (with solution ids chosen
for hipBLASLt 1.5.0: 13 of the 24 ids are the same) and 8 more, all bf16 (N=1/K=2560 at T=8192, N=4/K=10240, N=512/K=2560
at T=4096, N=2560/K=2560 at T=4096, and N=10240 with K=2560 and K=320 at T=4096/8192).
docs/AMD_HIP.md lists the shipped tables and how to check one.
With STRATA_ROPE_TABLE unset the table was skipped at run time, yet the compiled default kernels changed their
results: the model-level parity run (Swift IQ2_XS, tools/conversation_cache_parity.py, STRATA_STATE_HASH) gave the
same tokens but other GDN / indexer / K/V state hashes than 0.1.31's (bisected to #280). native_rope's apply,
the indexer's append / append_first / append_blocks and the prompt path's rope_kernel are templates on TAB now;
the host launches <false> whenever no table applies, and that instantiation has no table read at all. Checked:
the same tokens and every state hash as round 1 with the whole branch.
- sycl/include/dpct/LICENSE.TXT: the SYCLomatic helper headers' license
(Apache-2.0 WITH LLVM-exception; their SPDX notices already name it)
- sycl/third_party/ggml/LICENSE: ggml's MIT notice beside the migrated copy
- tools/intel/gpustat.py + gpustat.service: the root sampler the Monitor
tab reads load and VRAM from on xe (finds the card, sizes VRAM from its
BAR; GPUSTAT_PCI / GPUSTAT_VRAM_MB override)
- dpct reports and .yaml sidecars removed and ignored
- docs/INTEL.md: no references to a private test rig; the sampler and the
model_switcher setting documented
Conflicts: setup.py (upstream's reworked low-RAM/paging/rope logic kept, the
Intel backends guarded out of the experts-in-RAM rules as before; meta, the
CUDA/HIP build stamp, only read where the resident variant uses it) and
serve/server.py (upstream's /load, /unload and idle-unload beside the port's
/switcher and request logging). The SYCL copies are refreshed next.
- telemetry: on an Intel Arc (xe) the GPU readings come from a root
sampler's /run/gpustat.json (load, VRAM: xe exposes them to root only)
and sysfs (temperature, power, cap, the PCIe link the card trained at,
capped by the slot). NVML stays first; nothing changes on NVIDIA.
- a run config may name a model_switcher RPC (one model at a time on a
shared GPU): GET/POST /switcher report and request a swap, and the web
app shows a Model menu with the models served on this port; it waits
for /health to name the new model, then reloads. setup.py keeps the
setting when it rewrites a config.
k8v4 (KV in VRAM, no streaming): 995 tok/s prompt / 65.8 decode at 128K,
804 / 51.2 at 256K with 994 experts served from the host mirror.
API: 48-72 tok/s on a short chat answer, prompt included.
The engine sets CUDA_MODULE_LOADING=EAGER (a kernel loaded mid-prompt found no VRAM), so the three extra mmq
instances (+24 MB of binary) were resident for every model: IQ3_S 2261 -> 2260 slots, Q2_0 3678 -> 3675 (S1's A/B,
IQ3_S story decode ~1.5% lower). Measured with the IQ3_S setup config against 0.1.31 (wt-131-r9 build-release):
- slots 2261 / free 5.25 GiB, the same as 0.1.31 again (2260 / 5.24 with the instances);
- engine threads the same (23 at READY, 25 after a request, 0.1.31 likewise): nothing new runs for a normal pack -
the stager's 32 threads are only for a source with file-read blobs, the verify hook only with STRATA_LOGPOS;
- 5 alternating pairs, greedy story / code / third prompt: 0.1.31 39.5 / 44.7 / 32.0, this 40.3 / 45.2 / 33.3
tok/s (story B >= A in 5 of 5).
The UD-Q4_K_XL prompt is SSD-bound with or without MMQ (29 s per 4K chunk either way), so the released engine keeps
FP16 products for it; prefill_mmq_kquant_test is built with the option.
Summary 77.9 tok/s (19-token prompt) / 74.2 after 2,184 tokens; long
contexts with setup's KV streaming: 66.1 tok/s decode after 128K, 54.1
after 256K (all experts in VRAM). The 'not re-measured' caveat is gone.
- the GR norm ran one work-group per token; one per (token, stream) visits
every element in the same order, so rs and xn are bitwise unchanged
(gr_bench checksums equal), GR read 76.7 -> 71.4 us at 6 tokens
- the gate/up expert dots read their 2-byte-aligned weights as aligned
4-byte loads (page-safe) instead of 16-bit pairs; same ints
- grids in local memory (STRATA_GRID_SLM=1) measured slower: off
Decode 77.9 tok/s (19-token prompt), output tokens unchanged.
The gather wrote 97 rows into a 61-row device buffer (and read them back into a 61-row vector); the next launch
reported the overrun as "iq_dequant_f32: invalid argument". The buffer is max(gathered, table) rows now.
The expert down projection (IQ4_NL in 39 of 48 layers) was the largest
expert kernel: ggml reads four uint32 of the codebook from global memory
per lookup and each weight int as two 16-bit loads. Register constants
and aligned 4-byte loads (the third only when unaligned, so page-safe)
produce the same ints: 80.8 -> 43.1 us per call, output tokens identical.
-DSTRATA_IQ4NL_FAST=0 restores ggml's path.
The summary still described llama.cpp as the Intel engine at 23-25 tok/s.
It now compares NVIDIA, the SYCL port (setup's default once built) and
llama.cpp; the stale 'not ported'/'not done' items, setup's KV choice and
the served speed are corrected, and the long-context table is marked as
measured before the decode work.
S5's branch (f-132-q4) compiles the Q4_K, Q5_K and Q5_1 MMQ instances on CUDA and enables them with the private
define STRATA_MMQ_KQUANTS; with both, moe_mmq.cu would have duplicate case labels. STRATA_ORCA_Q4KS_MMQ now adds only
q5_0 (OrcaRouter's Q5_0 expert downs; HIP included) - supported() and run() take Q5_0 under it.
The engine read the 51.2e9-value n-gram table only as IQ4_NL (ISTA-DASLab's shard 2, 90 B/row), which is 8.1% off the
checkpoint's own values per row (correlation 0.996-0.997 on rows from all 128 shards: same table, same order). The
checkpoint stores it as 128 shards of [2500012, 160] F8_E4M3 plus one weight_scale.
- tools/ple_fp8_pack.py copies those bytes, unchanged, into a one-tensor GGUF: per_layer_token_embd.weight
[160, 320001536] as I8, with strata.ple.format = f8_e4m3 and strata.ple.scale. 51.2 GB, streamed.
- PleTable opens either format; the row size (90 / 160) and the decode (IQ4_NL codebook / an E4M3 table times the
scale) follow the file. PleReader takes the row size at open; its cache sizes itself by it. GGUF reader: type I8.
- A PLE-only table (architecture strata-ple) is not added to the dense-projection sources.
- The start line names the table's format.
- ple_fp8_parity: rows through Direct, Mmap and gather_batch match torch's decode of the checkpoint bit for bit.
First-token KL of the IQ4_NL table against the FP8 one: 0.0009 (1K), 0.0013 (2K), 0.0002 (4K), 0.022 (32K); mean 0.0061, top-1 4/4.
Rebased onto 0.1.31 + #317 + #296 (Niko1221): one row size (`rb`) and one decoder in PleTable for the three table
formats - IQ4_NL, #296's Q5_0 (110 B rows, the mapped reader only, as #296 has it) and FP8 (160 B) - and #317's
SSD keep-alive picks its page inside the table at the table's own row size; 0.1.31's shard list for the dense
sources keeps the PLE-only exception; ple_fp8_parity builds with STRATA_BUILD_TESTS beside #296's checks. The
"PLE on" start line stays 0.1.31's.
- The prompt path streams every expert once per 4K chunk; with the GGUF read in place ~35 GB of that comes from
the SSD as page faults, and 4 copy threads with 16 blobs in flight read it at ~0.6 GB/s. When the expert source
has transient (file-read) blobs the stager now defaults to 32 threads and a ring of 4 x threads (the env
overrides stay); every other source keeps 4 / 16. Measured (4K chunk): 71 s -> 34 s (16/64) -> 29 s (32/128);
16K prompt 273-282 s -> 99 s.
- moe_mmq: Q4_K, Q5_K and Q5_1 in supported() and the dispatcher, their mmq instances compiled for CUDA
(STRATA_MMQ_KQUANTS; HIP unchanged). The dispatch follows eddoursul/Strata 7706c1b; main's prompt path already
picks MMQ per layer. GPU time of the experts' products 4x lower (10 s -> 2.4 s per 4K chunk), not visible in
the wall time on this PC (SSD-bound either way: 29 s with and without); after a 16K prompt the continuation
agrees with llama.cpp at 92.2% (KL 0.029) against 90.2-90.6% (0.038) with the FP16 products. The other
models' formats are not affected (their layers' types decide; none uses these three).
- tests/cuda/prefill_mmq_kquant_test: the four expert formats through MMQ against ggml's dequantized weights
(synthetic, several experts per group, permuted rows, a zero row; rel. L2 <= 0.34%), and Q6_K stays uncovered.
A disconnect in the log now names the client that closed it. The
strata-sycl-dev image inherits a health check on port 8080 (llama.cpp's)
that marked the serving container unhealthy forever.
- FAMILIES["unsloth"] (marked [experimental], not the default) and MODELS["UD-Q4_K_XL"] ("budget": True,
48 GB of RAM or more); HF_REVISIONS pins unsloth/Qwen3.8-Flash-Next-GGUF; families may have more than two
shards ("shards": 4).
- The download resumes as before; then verify_sha256 checks each shard's size and SHA-256 once (remembered in its
finish mark; a wrong file is deleted so the next run downloads it again).
- resident_budget_gib: round(RAM) - 24 GiB (40 on 64 GB, the measured setting), at most the 71 GiB of experts, at
least 8; a KV cache streamed to RAM (64K+) comes out of it. --resident-budget-gib replaces the low-RAM mode for
this model (no experts.bin; --low-ram is ignored with a warning); one GPU, no images, no speed projection; 8K
context recommended on GPUs under 14 GB; no --ple-gguf for a 4-shard file (the engine finds the PLE shard).
- The engine version is checked right after the engine step, before the 111 GB download.
- choices_from_config reads a size with a dash (strata-unsloth-ud-q4_k_xl.json).
- tools/test_setup_unsloth.py: pins, budget, SHA-256 handling, config round trip, and setup.main() end to end with
every outside effect mocked (URLs, pack flags, engine args, old engine, too little RAM, two GPUs + images).
- README and docs/UNSLOTH_Q4.md / DETAILS.md describe it.
The model support itself follows eddoursul/Strata (PR #245).
Same token sequences through Strata (40 GiB budget, greedy, adaptation off, STRATA_LOGPOS_TOPK=20 on the verify
windows) and llama.cpp 3cf0325 (CPU, mmap, top 20 from llama_get_logits_ith): argmax agreement, top-5/10 overlap,
KL over the top 20, perplexity. Code 99.0% / thinking 97.5% (the fork, eddoursul/Strata: 98.2-99.3% / 95.6-95.8%);
after a 16K prompt 90.2% on Strata's greedy continuation and 91.2% on real text, the same real-text positions with
512 tokens of context 89.2% - so not a long-context loss: all positions with a top-1/top-2 gap of 0.5 or more agree
(and every code/thinking position from 0.2), perplexities within 3%.
The two real-weight checks were added to every CUDA build with native experts (a full release build compiled them);
they are test programs. docs/ORCA_Q4_K_S.md: the build flag for them, and that 0.1.31's GPU expert kernels take
Q4_K gate/up with Q5_1 or Q8_0 downs (Q5_0 is not among them yet: the engine refuses such a pack at start, naming the layer).
Load three-shard Uncensored weights while preserving Q5_0 PLE rows
and Q4_K/Q5 expert bytes.
Add native CUDA and MMQ paths with real-weight parity checks.
Rebased onto 0.1.31 (Niko1221): 0.1.31 already loads each expert role from its own shard (the per-role shard column
of native_experts.txt, iq_pack.py, expert_source.cpp, native_expert_parity's multi-shard mode) and has its own
Q4_K/Q5_K/Q5_1/Q8_0 GPU expert decoders, so this PR's gguf_role_file / v4 manifest, its iq_pack.py and
test_iq_pack.py changes and its iq_kernels.cu decoders are dropped. Kept: Q5_1 in native_mmvq (type 7 through
0.1.31's iq_mmvq) and in the BF16 dequantizer, the Q5_0 PLE table (mapped reader), the Q4_K/Q5_0/Q5_1 MMQ
instances under the STRATA_ORCA_Q4KS_MMQ option (OFF), ple_q5_parity / q5_projection_parity and the doc.
#272 made Auto the default, which on a hybrid CPU moves the workers (P-core primaries, then SMT siblings, then
E-cores) and cuts their number to P-cores - 1: a speed change nobody has measured here, on the machines most
users have. The default is All - the same cores in the same order as before on every CPU (Windows: the first
logical processor of each core; Linux: one allowed CPU per (package, core)), and the same worker count - in
Options, ExpertPool, physical_cores() and detect_cpu_topology(). --pool-affinity auto | p-cores opts in; the
hybrid line is printed only then. Usage text says so.
- Add detect_cpu_topology() for Windows (GetLogicalProcessorInformationEx) and Linux (/sys/devices/system/cpu/cpu*/cpu_capacity)
- Identify hybrid P/E-core topology and SMT sibling structures
- Prioritize primary P-cores for worker allocation, reserving core 0 for host thread
- Default pool_workers to (p_cores - 1) on hybrid CPUs when unspecified
- Add --pool-affinity auto|p-cores|all CLI option
- Maintain 100% backward compatibility on non-hybrid CPUs and default configs
iq_embed_rows (97 rows gathered out of order, with repeats) and iq_dequant_f32 on a random BF16 table, every value
compared bit for bit with the exact widening (bits << 16); embed_type_supported(30) / iq_supported(30) /
iq_row_bytes(30). ctest native_expert_parity_bf16_embd.
- tools/embd_bf16_pack.py copies embed_tokens (BF16, as the checkpoint ships it) into a one-tensor GGUF; only the
safetensors shard that holds it is needed (3 GB for Qwen/Qwen3.8-Flash-Next).
- --embd-gguf PATH: NativeEmbed reads token_embd from that file; iq_embed_rows and iq_dequant_f32 decode BF16
(embed_type_supported). Mapped host memory as before (1.2 GiB instead of 0.3 for IQ4_XS), no VRAM.
First-token KL of the GGUF's IQ4_XS embedding against the BF16 one: 0.0015 (1K), 0.0089 (2K; the top token changes), 0.0010 (4K), 0.037 (32K); mean 0.012, ~100x the run-to-run noise. On the NVFP4 fork, whose GGUF embedding is Q8_0, the same comparison gave 0.0025. Decode and prompt speed are unchanged (fork); the start takes ~0.2 s more.
Rebased onto 0.1.31 (Niko1221): NativeEmbed::load takes the shard list (one file for --embd-gguf) and checks the
model's architecture since 0.1.31 - the one-tensor file says "strata-embd", so only its tensor is checked there;
the Q4_K/Q5_K/Q5_1/Q8_0 decoders 0.1.31 added stay beside the BF16 one.
- mtp.cpp record_forward: the `if (st_.kv_rot)` branch rotated K and V and then always appended with
kv_append_q4_step - for INT8 + STRATA_KV_ROT=1 into Q4_0 pools the state does not have. The rotation and the
append are separate now: rotate when kv_rot, then append in the state's own format (Q4_0, INT8, FP16).
- conversation_snapshot.cpp: rotated INT8 K/V is its own snapshot format (+16), so a conversation snapshot or prompt
checkpoint taken with the rotation is refused by a state without it and the reverse ("incompatible K/V
geometry"); without the rotation the format is unchanged. conversation_snapshot_test checks both directions.
- generate: says when STRATA_KV_ROT=1 applies; the q4_0 line's comment back.
INT8 K and V can go through the Walsh-Hadamard rotation --kv q4_0 already uses (kv_q4.hpp): it spreads outlier
channels over the 64-value scale groups before quantization; the queries are rotated too (<Hq, Hk> = <q, k>) and the
output rotated back.
- QsaState::kv_rot: Q4_0 always, INT8 with STRATA_KV_ROT=1 (off by default: the same tokens as main).
- Every writer and reader follows it: decode, the verify window, the prompt path, the MTP drafter, and the drafter's
batched prompt pass (Prefill::draft_kv) - which rotated only for Q4_0.
- verify.cpp: the batched window path (qb) never rotated its queries; it did not matter while Q4_0 was kept off it.
First-token KL to fp16 K/V, 4 prompts incl. 32K: NVFP4 pack 0.0066 -> 0.0051; IQ2_XS 0.0022 -> 0.0054. Hence opt-in.
- The table's angles differ from the fast-math cosf/sinf ones in the last bits, so every output moves: off by
default (the kernels compute the angle exactly as 0.1.31), STRATA_ROPE_TABLE=1 turns it on (replaces the PR's
STRATA_ROPE_LEGACY A/B switch).
- The per-device registration was a raw pointer into the session's arena that outlived it: QsaState::owns_rope marks
the state that built the table (not one that borrowed it with share_rope, like the MTP drafter's), and
session_release() - to be called before a session's memory is freed - clears every device's entry pointing at
it (rope_table_release). The latest session of a device now wins, instead of the longest table.
The native decode/verify rope and the indexer's pooled keys computed pos * powf(...) with cosf/sinf under
--use_fast_math (0.0014 rad off at 32K, ~0.02 at 262K), the prompt path with precise float libm, and the float64
table the session builds was read only by the non-native path. rope_table_set registers that table per device
(mrope.hpp); rope_cs reads it, or computes in float64 past its end.
Measured on main with IQ2_XS: first-token KL of the old angles against the table 0.0045 on a 32K prompt (two runs of
main: 0 to 0.00013 on short prompts). On an NVFP4 pack: 0.0063 at 32K and at 125K. Speed unchanged.
STRATA_ROPE_LEGACY=1 brings the old float angles back for A/B.
Rebased onto 0.1.31 (Niko1221): 0.1.31's rope scaling (#84: linear / YaRN, rope_scaled_angle) is in every rope
kernel and in the session's table (build_rope_table with the RopeScaling), so a kernel reads the table only when it
was built with the scaling it was given, and otherwise - or past the table's end - computes the angle exactly as
0.1.31 does (no separate float64 device path); rope_table_set registers the scaling with the table.
For a per-position comparison with another engine on the same tokens (argmax agreement, top-k overlap, KL on the
top K): docs/UNSLOTH_Q4.md compares UD-Q4_K_XL with llama.cpp this way. Off unless the variable is set; the
eight existing columns are unchanged.
Every request that does not continue the live session parks it, and put() never removed older copies of the
same conversation while make_room evicted oldest-first: a subagent's 9 successive turns (each re-rendering the
previous reply) parked 9 copies and pushed the parent conversation out of 4 slots. drop_superseded() now runs
before make_room (park_current) and in put(): an entry whose DEEPEST checkpoint the outgoing chain (or live
state) still holds - the same conversation a turn back, differing only in the stale reply tail - is dropped
(counted apart from evictions, logged). An entry without checkpoints, of the other steering mode, or whose
deepest checkpoint the outgoing chain lacks (another conversation sharing only the system-prompt root) is kept;
an oversized put drops nothing. The cache is opt-in (#189): with it off nothing runs. Tests in
conversation_cache_test (MAIN -> SUB x9 -> MAIN with 4 slots; the kept cases).
With the AMD backend (config backend "hip") telemetry reads the amdgpu driver's sysfs files instead of NVML:
gpu_busy_percent, mem_info_vram_used/_total, and hwmon temp1_input, power1_average (or power1_input) and
power1_cap. The card is found as setup's amd_gpus() numbers it (KFD topology GPU nodes in order, CPU nodes
skipped, drm_render_minor -> /sys/class/drm/renderD<N>/device), which is HIP_VISIBLE_DEVICES' numbering; a
layer split sums/averages as on NVIDIA. Service.free_vram_mib (min_free_vram_mib) uses it on HIP. NVIDIA
paths unchanged. Test: AmdTelemetry (a fake sysfs tree, two cards and a CPU node). Not yet checked on a real
AMD box.
The second (remainder) GEMM changes what the prompt path computes, so it is opt-in: 2 = all but the
hyper-connection's projections, 1 = all. With 0 every lo pointer is null: no extra buffers in carve()/bytes_needed(),
the PLE batch size as before, and every kernel writes exactly the BF16 images it wrote before.
The prompt path fed BF16-rounded activations (2^-9 relative) to the BF16-weight projections, where decode feeds FP32:
the router (a discrete top-10), the indexer (a discrete block selection), SSM alpha/beta, the shared-expert gate, the
PLE key/value and the hyper-connection. Each now takes x as hi + lo, both BF16 (lo = bf16(x - hi)), in two GEMMs
summed in FP32: ~16 mantissa bits, decode's FP32 x to within ~1e-5.
STRATA_PREFILL_BF16X2: 2 (default) all but the hyper-connection, 1 all, 0 off.
IQ2_XS, first-token KL to mode 1: short prompts main 0.0028/0.0069/0.0009, mode 2 0.0014/0.0024/0.0012; a 32K prompt
0.074 and 0.072 (the hyper-connection is what matters there).
A bigger chunk reads a long prompt in a different order of chunks (other expert loans, other rounding of the
batched path), so it stays out of the default: plain --prefill auto picks from 8192 down exactly as 0.1.31 did (the
PR's "no chunk past the context" rule, which also moved the choice below 8192 for small contexts, now applies only
to the new sizes). auto:N (16384 or 32768) or STRATA_PREFILL_AUTO_MAX with auto sets the ceiling; o.prefill_chunk
starts at that ceiling for the buffer estimates. Both kAutoChunks lists - plan_lend and the serve path's per-stage
fits() - take the same sizes under the same rule.
Every chunk streams again each expert it routes to, so a long prompt read in 8192-token chunks crossed PCIe with
nearly all experts of every layer four times per 32K. The auto list now starts at 32768 and 16384, still bounded by
the lend rule and by --max-context; a request lends only what its prompt needs, so short prompts are unaffected.
IQ2_XS, RTX 5090, 64K context, a 32K prompt (3 runs each): 5,624 -> 6,465 tok/s, 40K -> 18K experts streamed.
NVFP4 pack at 262K: 3,535 -> 5,201 tok/s.
fmaxf(NaN, -65504) is -65504, so #281's clamp turned a NaN product into -65504 and the prompt path went on with a
plausible value where it used to show the NaN (STRATA_DBG_NAN and the NaN checks find it there). isnan(x) ? x :
clamp(x): only +-inf (and finite values from 65520 up, which rounded to inf) saturate; everything else is unchanged.
- prefill/kernels.cu: the SwiGLU products that feed the prompt path's FP16 down projection are clamped to FP16's
range (+-65504) before rounding. A token with a massive activation otherwise becomes inf there and NaN after the
GEMM, where decode's q8_1 path (range ~8e6) stays finite.
- native_gdn_preprocess.cu: the unfused gate softplus used logf(1 + expf(v)), which returns 0 for v below ~-16
(1 + e^v rounds to 1); the fused kernels, gdn.cu and verify_kernels.cu already use log1pf(expf(v)). The two paths
now agree.
On ordinary prompts nothing moves: first-token KL to main 0, 0.00013, 0 (the run-to-run noise).
Some SSDs (here a WD_BLACK SN7100 on Windows 11) drop into a power state
after ~250 ms without a command and stall the next reads 50-150 ms; the
NVMe idle timeouts of Windows' power plan do not change it. In decode that
hits the first read after a pause: at the start of a request, or after a
few rounds whose rows all came from the row cache. --ple-io ram, which
avoids the reads altogether, is not available on Windows.
The reader's worker now reads one page of the table (a different one each
time) when no read has gone out for 100 ms, until 60 s after the last
issue; then the SSD may sleep until the next request re-arms it. A
keep-alive read counts when it completes and stays out of the row-read
counters and latencies; the "ple io" line adds "SSD kept awake by N reads
(slowest X ms)". A keep-alive read that fails, or cannot be submitted,
turns the keep-alive off instead of failing the reader.
STRATA_SSD_KEEPALIVE=0 turns it off, =50 shortens the period;
STRATA_SSD_KEEPALIVE_WINDOW sets the window in seconds. Only with the I/O
worker (the default), not with --ple-sync-submit or --ple-io mmap/ram.
The reader's statistics are now copied and reset under the worker's lock
(snapshot()), since the worker may be counting a keep-alive read. The
selftest checks the period, the window, the re-arm by a ticket the row
cache serves, that rows and read counters are untouched, and that the
caller-thread mode has none.
A scan on that PC (reads after pauses of every length): 0 of 211 bursts
stalled after pauses under 200 ms, 35 of 73 after 200 ms and more, 0 of 68
of those with the keep-alive on.
main already has an opt-in "v3" hyper-connection read (#186, STRATA_GR_V3=1, another summation order); #315's
"v2"/"v3" and STRATA_HC_V2 read as the same thing. Now: the plain read (0.1.31's default), split (the norm per
token and stream) and staged (split + the down projection's activations staged by cp.async), kHcPlain/kHcSplit/
kHcStaged, gr_norm_split_kernel / gr_down_staged_kernel, and STRATA_HC_SPLIT: unset or 2 = the newest the card's
check accepts, 1 = at most split, 0 = the plain read. The check compares against the plain read (launch_multi with
kHcPlain) and the single-token read, never against what STRATA_GR_V3 selects; with STRATA_GR_V3=1 that opt-in read
still runs first, unchanged.
`"aliases": [...]` (or a comma-separated string) in strata-<model>.json: /v1/models lists the model with an
`aliases` field and each alias as its own entry (`alias_of`), /props?model= accepts them, and a request naming
an alias is answered under that name (OpenAI, streamed or not, and Anthropic). Any other name is still served
under the model's own name, as before; without aliases the responses are unchanged. Documented in DETAILS.md.
Test: ModelAliases (serve/test_server.py).
The multi-token read (verify windows, the MTP drafter) keeps v1's
arithmetic exactly - every sum over the same elements in the same order -
and only splits the work differently:
- the norm (v2, v3): one block per token and stream instead of one per
token. Each thread visits the elements of its stream in the order it did
in v1, so every sum of squares and rs is v1's; R' * w_norm is recomputed
and scaled once (the same two roundings as v1's store and in-place
scale). On an RTX 4080 SUPER: 0.58 -> 0.23 ms per 4-token round.
- the down projection (v3): v1's rows per block and lane order, but the
activations arrive by cp.async in half-stream tiles, two in flight, in a
bank-conflict-free two-plane layout, and the next tile's weights load
while the current one is used. A tile is 160 = 5 x 32 chunks, so a lane
still accumulates its chunks lane + 32 q in ascending order. Before sm_80
and on HIP the staging is a plain copy (the same bits).
Verifier::init runs fused_gr_check once per card: v1, v2, v3 and the
single-token read on random bf16 weights and inputs (1..8 tokens, with and
without the pending write), every output compared bit for bit; the newest
variant that agrees runs on that card and the log says which. A card that
has not been checked runs v1. STRATA_HC_V2=0 keeps v1, =2 stops at v2.
The down kernels carry as many tokens per launch as the card's shared
memory allows (v3: 10 KB per token), as v1 did.
Rebased onto 0.1.31 (Niko1221): v1 is 0.1.31's default read unchanged - the TILEV 1280/2560 down kernel (sm_75's
small tile), the pre-Volta capacity rule, STRATA_EMULATE_CC - moved into down_chunk()/launch_multi(); main's opt-in
STRATA_GR_V3 path (#186, another summation order) still comes first in fused_gr_read_multi, and the card's check
calls launch_multi(1) directly, so it compares against the default read whatever STRATA_GR_V3 says.
The commit graph writes the GDN state and conv tail, the QSA indexer's tail/dead/pooled/block_pos and the PLE
history; with #284 it may still run when commit() returns. Verifier::commit records commit_done_ after it, and
Verifier::wait_commit() synchronizes that event (not the device; a no-op when nothing is pending, recursing into a
layer split's later stages). A layer split's first stage keeps the synchronous commit (next_ set).
Audited, and where the wait now is:
- the serve decode loop's end (end of turn, max tokens, STOP/cancel): wait before the live session, the state
fingerprint (STRATA_STATE_HASH) and the DONE line - so the next request's park (conversation snapshot capture),
restore, checkpoint restore and zeroing never see a running commit;
- the prompt read by verify windows (serve): wait at its end - the batched prompt path (other streams), prompt
checkpoints (cudaDeviceSynchronize already) and snapshots may follow;
- the start of each serve request: an event wait replaces #284's cudaDeviceSynchronize (covers error paths);
- the end of a generate run: wait right after the decode loop, before the stats and the dumps (#284 synchronized
only later, after the routing dump);
- unchanged and safe: the next window (same stream), the MTP draft (its own stream; reads the window's final rows
and its own K/V, which commit does not write), the adaptive tier thread (expert slots and residency tables only),
STRATA_LOGPOS / STRATA_DUMP_FIRST_LOGITS (read run()'s head logits after run() synchronized), QUIT and /unload
(the Verifier destructor synchronizes its streams before freeing).
Setup forced images off on the AMD backend. hip_vision() now takes --vision cpu: build_engine_hip builds
tools/vision without CUDA beside the HIP engine (build_vision_cpu, again when its source changes) and records
vision/vision_src in BUILD.json; the start config is the CPU encoder's, as on NVIDIA. --vision yes/gpu stay off
with a hint (no HIP encoder build yet); without --vision nothing changes. Test: tools/test_setup_choices.py.
The commit's cudaStreamSynchronize held the host ~0.45 ms a round. The next window follows it on the same stream and
the drafter (its own stream) reads only this window's final rows and its own K/V, so a single-GPU session
(Verifier::set_commit_async) returns at once. Whatever touches the session from another stream or the host afterwards
synchronizes the device: the start of each --serve request (checkpoint restore, zeroing, the prompt path) and the end
of a generate run; checkpoint saves already did. A layer split keeps the wait, as does STRATA_COMMIT_SYNC=1.
IQ2_XS, 400-token answers, 3 runs each: 128.6 -> 135.8 tok/s decode (+5.6%), the same tokens. NVFP4 pack +0.8%.
The engine/CMake part shipped in 0.1.31. Setup now admits compute capability 6.x and 7.0 only when
STRATA_EXPERIMENTAL_SM60=1 is set (gpu_problem; without it the message says so), compiles the engine with
-DSTRATA_EXPERIMENTAL_SM60=ON for cards below sm_75 (engine_defs), and takes the newest CUDA 12.x toolkit
(find_nvcc(below=(13, 0)); CUDA 13 cannot build sm_60/sm_70) - stopping with a hint when there is none, or
when an RTX 50 card (CUDA 13) shares the model with such a card. Without the variable nothing changes.
Test: tools/test_setup_choices.py (mocked toolkits and cards).
It keeps the last 100 requests' prompts, answers and reasoning in memory. Off (the default): begin_request() keeps
nothing, the streams are not wrapped, GET /api/requests and /api-monitor answer 404. Every record write goes through
_note(), which does nothing without a record (the PR's direct self.record[...] writes in the stream error paths
would have failed with the monitor off). A client disconnect in do_POST is marked and then re-raised, as the server
handled it before. The start line says when it is on (and that anyone can read the history without an API key).
test_monitor: turns it on; MonitorOff checks the default; Anthropic requests ask for thinking with "thinking" (#278).
--gguf-dir built exactly <model>-00001-of-00002.gguf and -00002-of-00002.gguf, so a model split or named
differently (-00001-of-00003, Unsloth's ...-00001-of-00004.gguf) was not found. gguf_dir_shards() takes the
published name when it is there (unchanged for the existing models), else the one first shard in the folder
whose name has the size in it, and names shards 1..N from it as the engine and iq_pack do. Help text updated.
Test: tools/test_setup_choices.py (2- and 4-shard names).
The release zip's strata-vision has no sm_75 code (BUILD.json vision_archs 86/89/120), so on an RTX 20 card
setup fell back to compiling the encoder, which fails on most Windows PCs. prebuilt_vision() now chooses the
CPU encoder (the same program without --gpu) with a warning; --build still compiles a GPU one. The next release
zip should build strata-vision with 75-real as well. Test: tools/test_setup_choices.py.
Since 0.1.25 NativeDense treats an IQ3_XXS/IQ4_XS/Q8_0 GGUF key as native, so for a native pack the loader
skipped the pack's key row and then refused the BF16 row iq_pack --compat-bf16 writes for OrcaRouter's IQ3_XXS
key ("native dense: incompatible matrix blk.1.ple_key.weight"). NativeDense::keep_unquantized_ple_key reads the
key row's code_bits from index.txt (WeightTable::index_code_bits) and keeps an unquantized row in the arena;
NativeDense::load does not upload the GGUF key over it, and the PLE uses the BF16 key as before 0.1.25. A
quantized key row (Q2_0, UD-Q4_K_XL packs) is unchanged; the GSQ packs' GGUF keys are BF16 (not eligible), so
they never reach this path. The reporter's fallback (Gen4536). Test: native_dense_ple_key_test (synthetic GGUF
+ pack, CUDA).
A mirror that ignores the Range header answers 200 with the shard's start (its JSON header); cut to the asked
length it passed the size check and the manifest hashed the wrong bytes, so the drafter accepted nothing. get()
now requires HTTP 206 with the requested Content-Range. The 31 MTP tensors' sha256 at the pinned revision
(de4b8e4) ship in SHA256 (two independent fetches agree; 29 re-read from the revision, the two expert tensors'
first/last 16 MiB): a wrong tensor is fetched again once, then refused. An inventory read from another
repository/revision is read again and its tensors are kept only when they hash right. `mtp_fetch.py verify`
(offline, cached by size/mtime) finds a corrupt install; setup then fetches the bad tensors again and rebuilds
the draft layer. From PR #324: a refused engine archive no longer keeps its zip and .done mark.
Tests: tools/test_mtp_fetch.py (fake checkpoint behind a mocked urlopen, a mirror that ignores Range),
test_setup_pins (refused archive).
- The rebase keeps 0.1.31's requirements.txt install in setup.py and the Dockerfile: jsonschema is not added.
- serve/structured.py imports jsonschema on the first json_schema request, not at server start (a server without it
started fine before #334 and still does). json_object never needs it; json_schema without it is checked to be one
JSON object, and the server window says once how to get full schema validation.
- test_structured: the two schema-violation tests skip without jsonschema; new tests for the object-only fallback
and for "import serve.server" not importing jsonschema. docs/DETAILS.md says it is optional.
- StrataEngine.close() (now also the unload and the restart): QUIT and EOF, then terminate, then kill, each with 20 s
as the unload had (#333 went from QUIT straight to kill after 10 s; an engine unpinning tens of GB can need more).
A process that outlives all three raises EngineStuck and stays owned.
- do_POST caught every RuntimeError as a 503 - EngineDied, GpuBusy and ModelBusy are RuntimeErrors too, and any other
failure was turned into a vague 503. Only EngineStuck is caught now (also around POST /unload, and the idle-unload
thread survives it); EngineDied keeps its own answer.
- test_lifecycle: terminate before kill, EngineStuck, an EngineDied on load answered as such.
- The Origin check of /settings and the MCP tools let any page through when the request carried X-Forwarded-For,
CF-Ray or CF-Connecting-IP (headers every reverse proxy or tunnel adds), and compared hosts without their ports
(a page on another local port passed): any web page could then change the shared settings or run MCP tools with the
user's rights. Removed. The Origin must be this server's host:port, or an origin in the config's new
"trusted_origins" (the web app behind a proxy or tunnel); a proxy header alone opens nothing.
- CORS is opt-in: "cors_origins" in the config (an origin, a list, or "*"). Only /v1/* answers with
Access-Control-Allow-Origin (the origin itself, or "*"), never /settings, /unload or the app's files; the
preflight echoes the headers asked for ("*" does not cover Authorization). OPTIONS is answered 204 either way.
- X-Accel-Buffering: no on SSE stays. origins_of() checks both lists at start (scheme://host[:port], no paths;
"*" only for cors_origins). docs/DETAILS.md; tests for the bypasses, the trusted origin, CORS off/on, the header.
- Implement do_OPTIONS (204 No Content) and add Access-Control-Allow-* headers to allow web-based clients (Open WebUI, LibreChat, browser extensions) to query endpoints.
- Update _own_page to recognize reverse proxies and tunnels (Cloudflare Tunnel, Tailscale Funnel, Nginx) via X-Forwarded-Host, CF-Ray, and forwarded headers, preventing 403 CSRF false-positives when serving remotely.
- Add X-Accel-Buffering: no to SSE streaming responses to prevent reverse proxies from buffering real-time token generation.
- Add unit tests in serve/test_server.py covering CORS preflight, CORS headers, and reverse proxy origin checks.
Co-authored-by: Gemini <[email protected]>
"thinking" absent meant no thinking, also for a request that gives reasoning_budget_tokens (#123), which asks for
thinking with a cap (serve.test_server's test_anthropic_stream failed: no thinking at all). Such a request renders
as before. test_anthropic_thinks_only_when_asked pins the five cases.
restart() set ended=False before __init__, which zeroes max_context and blocks until READY, so a request in that
window skipped load() and failed with a 400 "leaves no room to answer in the context (0)". The engine is now
marked alive by __init__ once READY; prepare() answers 503 "the engine is starting" when max_context <= 0. The
output thread is bound to its own process and queue, so a killed engine's thread no longer marks its successor
dead or writes into its queue. Tests: RestartWindow (a fake strata --serve with a held READY).
park_current dropped the retained K/V whenever can_fit failed and fell
back to a full snapshot capture. The gate duplicates what make_room
decides immediately after, with harsher arithmetic: the slot term fires
when make_room enforces the same cap, and the byte term compares the
with-reuse estimate against headroom that ignores the parked image
replacing the retained buffers. make_room already implements the
stated policy - pressure drops the optional retained storage before
evicting any parked conversation (the priority the cache tests pin).
With a full cache the fallback buys nothing: the fresh estimate forces
the same or more eviction, plus a whole-snapshot recapture instead of
the growth - hundreds of ms per park at ~100K tokens where one long
conversation round-trips while short-lived conversations keep the
cache full.
The with-reuse estimate stays uncapped: it counts the retained buffers'
capacity and directories, and put() charges that same true size, so a
capped figure would under-evict and overfill the budget.
- Add set_coupled_draft() and coupled_draft_state() in include/strata/core/coupled_draft.hpp
- Add --coupled-draft and --no-coupled-draft CLI options to src/program/generate.cpp
- Default option to STRATA_SPEC_COUPLED environment variable for full backward compatibility
- Update usage() help text with option descriptions
Isolated benchmark (no inference-path changes) comparing the production
bf16_gemv warp kernel against an mma.sync.m16n8k8 tensor-core variant with
BF16->FP16 conversion, on the real decode GEMV shapes (n_in = 2560). Three
TC variants (W preconverted / inline conversion / fused output) and a full
k-split occupancy sweep; parity against an fp64 reference.
Result: the TC variant is 1.3-2.2x slower on every real shape even at its
best config - the warp kernel is already bandwidth-bound (374 GB/s) and a
single-token GEMV wastes 15/16 of the m16 MMA tile. Keeping the warp
kernel; the benchmark is checked in so this does not have to be measured
twice.
kProfPer 32 -> 33: fused_gr_read_multi writes three stamps per half and the second
half's base was 30, so its up-stamp landed at slot 32 = (l+1)*32 + 0, the next layer's
stage-0 base. Derived columns (gap/head/hc0) also wrapped to ~1.8e19 ns whenever a
slot was unstamped, printing ~1e14 ms per window. A delta now counts only when both
stamps are nonzero and monotonic; the hc0 split replaces the single hc-read0 column
only when the split chain is complete. Metrics-only: no computed model value changes.
Reason recorded in PORTING.md 19.
An existing experts.bin was kept whenever its size matched the layout. A different checkpoint with the same geometry
(a re-quantization or a fine-tune packed into the same folder) has the same size, so the engine silently served the
old experts with the new model's dense weights.
Before keeping the file, the first and last blobs of the first, middle and last layers are now compared with the
GGUF; a mismatch rewrites it. The check reads six blobs, so an unchanged pack is still reused at once.
Test: test_experts_bin_is_kept_only_with_this_ggufs_blobs packs two GGUFs of the same geometry into one folder (it
fails on the previous code).
Rebased onto 0.1.31 (Niko1221): 0.1.31 already refuses an experts.bin whose sidecar (experts.bin.src.json: shard
names and sizes, the hash of native_experts.txt) does not match, so the six-blob comparison runs only when the
sidecar says "reuse" - it catches the case the sidecar cannot (a re-quantized or fine-tuned checkpoint under the same
file names and sizes). It reads each of the six expert blobs on its own instead of concatenating the whole layer.
The test packs the second model under the first one's file name, so it reaches the new check.
The full setup rewrites the model's config and refreshed the MTP folder's subset with `a.draft_vocab or "cjk"`, so
running it again without the flag put the CJK subset back over a cyrillic (or en) choice, and dropped draft_vocab from
the config. saved_draft_vocab() reads the choice the config holds; the flag still wins. tools/test_setup_draft_vocab.py.
The shipped draft subsets hold 142 of the vocabulary's 18,580 Cyrillic tokens, so an answer in Ukrainian, Russian,
Bulgarian or Serbian drafted almost nothing. data/draft_vocab_cyrillic.bin is draft_vocab_en.bin plus every token
whose text is Cyrillic (58,963 ids), built with the repository's own tool:
python tools/draft_vocab.py --gguf <IQ2_XS shard 1> --base data/draft_vocab_en.bin --add cyrillic
- tools/draft_vocab.py: a "cyrillic" script (U+0400-052F, 1C80-1C8F, 2DE0-2DFF, A640-A69F).
- setup.py: --draft-vocab cyrillic; docs/DETAILS.md.
On the NVFP4 fork: 1.4 -> 2.1 tokens per round, 83 -> 109 tok/s in Ukrainian answers, English unchanged.
- --flash-attn on|off|auto (mtmd's flash_attn_type): flash attention keeps K and V in FP16. On the NVFP4 fork the
vision tower's output was 11.2% off with it on the GPU; FP32 attention (CPU) 0.1%.
- On the CPU a CUDA build keeps the GPU unseen (CUDA_VISIBLE_DEVICES=-1 before the runtime starts): it otherwise
opened a context there, 0.4-0.7 GB of VRAM the expert cache loses.
- On the CPU without --threads: one thread per core (mtmd's default is 4); no warm-up encode (~6 s of start).
- serve/server.py: relative exe/mmproj/model paths in the vision config are the config's cwd's, as the engine's are.
- StrataEngine.restart() runs __init__ again (after a crash, an idle unload's reload), which started one more
STRATA_REQUEST_LINES follower each time and printed every request line once per restart: a follower is started
once per log path (the log is opened for append, so the first follower keeps reading the same file).
- The field the follower printed as "ttft" is the engine's prompt read time (the new tokens only, without the queue,
the template, the first decode step): it is named prompt_read.
Found by running Claude Code against the server (claude -p, three tool turns) and by starting a config with a
relative engine path:
- POST /v1/messages/count_tokens answered 404; Claude Code asks it for its context figures. It now renders and
tokenizes the request exactly as /v1/messages would, without running the model.
- A /v1/messages request with no "thinking" (and no effort) got the template's default, which thinks: Claude Code's
helper calls (a ~60-token session title) came back with a thinking block and no text. Anthropic's thinking is
opt-in, so such a request now renders with enable_thinking false. A config's reasoning_effort still applies;
"thinking" set, or an effort, behave as before.
- STRATA_REQUEST_LINES=1: a thread follows the engine log and re-states each request's summary line on stdout, for
supervisors that only see the server's stdout (a tray, llama-swap). Off by default.
- A relative "exe" in the config failed in Windows' CreateProcess (WinError 2): it is made absolute against the
config's cwd, as the engine's cwd already is.
Tests: count_tokens equals the input_tokens /v1/messages reports for the same request; the request line pattern
parses the engine's summary. python -m unittest serve.test_server: 44 OK.
- generate.cpp: the startup line names "(an unnamed GPU)" when cudaGetDevice/cudaGetDeviceProperties fail (the
error is cleared), and the "no code for" message uses the same name; a HIP build prints the gfx architecture
instead of a CUDA compute capability.
- device.cu: device_code_error() returns "" on HIP - cudaFuncAttributes/cudaFuncGetAttributes have no mapping in
hip_compat, and gpu_arch_problem() already stops a HIP run on an architecture the build lacks.
- fused_gr.cu: STRATA_EMULATE_CC also answers the two compute-capability queries there (small tile, the sm_70
capacity rule), as the PR's description promises; with the variable unset nothing changes.
- generate.cpp: before the expert arena loads, the engine names the card and its compute capability and checks that
the build has code for it (device_code_error: cudaFuncGetAttributes on a kernel). A binary built for other GPUs
used to fail at the first kernel, after the whole arena had loaded; now it stops in ~0.3 s and says why.
- include/strata/core/emulate.hpp: STRATA_EMULATE_CC=75|80|86|89 makes the host's device queries (compute
capability, shared memory per block) answer as that generation, for a PTX build of it JIT-compiled on a newer
card - ggml's MMQ config, the QSA kernels' choice, fused_gr's shared memory. Tests only.
- STRATA_QSA_WARP=1|select|attn: the pre-sm_80 QSA kernels on any card (A/B).
STRATA_DUMP_FIRST_LOGITS=<file>: the logits of the first window's first position (the token the prompt predicts),
n_vocab float32, written once per run. Two engine variants run on the same prompt then give a first-token KL
divergence: how far a prompt-path change moves the model's distribution, not only whether its top token changed.
On a layer split the last stage's verifier writes it.
On RDNA4 the router took 39 us per call in decode (16% of the decode GPU time): 512 dependent FP64 adds, ten
selection passes with 20 block barriers, a renormalisation re-reading its weights from global memory. The sum's only
use is inv = (float)(1.0 / sum): all terms are positive with the largest exactly 1, so the serial sum and a tree sum
are within 2^-43 of each other; when the bracket tree*(1 -+ 2^-40) maps to one float, that float is the serial sum's
(1/x and the cast are monotone), otherwise thread 0 runs the serial sum as before. The selection runs on wave 0 from
registers (the argmax with the lowest index on ties is unique), the renormalisation from registers in rank order.
hip_router_fast checks ids and weights bitwise against the portable kernel (65,536 rows with ties, near-ties, NaN
rows, the forced serial path; and 256 experts / k 8). CUDA builds unchanged (HIP-only code).
The v2 tensor-core design with gfx12's v_wmma_f32_16x16x16_f16: one block per (query, KV head), wave w owns
dims [64w, 64w+64) (int8 scale group w), q and p split into FP16 hi + lo parts, int8 codes exact as FP16, scales in
FP32, the four groups' q.k partials added in a fixed order, online softmax over 32-cell chunks. Masked
(non-resident) cells get no weight, as in the decode kernel. FP32-level accuracy, not bitwise equal to the default
kernel, hence opt-in. hip_prompt_attn_wmma = qsa_prompt_attn_parity built for HIP (skips off gfx12).
NVIDIA builds unchanged (all new code is under __HIPCC__ / the HIP test section).
Each work-group owns a 128-column slice of all 324 rows, stages that
slice of xn once (<= 4 KB SLM) and streams the rows through it, 4 lanes
per row; a reduce kernel sums the 80 slice partials in a fixed order.
GR read 108.6 -> 76.5 us at 6 tokens. STRATA_GR_DOWN_SLICED=0 restores
the direct kernel.
Q8_0 and IQ4_NL blocks (34/18 bytes) are 2-byte aligned: the wide kernels
load them through load16_a2, now page-safe (the second chunk only when the
address is unaligned). Q8_0 4-6x, IQ4_NL 640->2560 4x. Q6_K's Q8_1 activation
int4 loads were 4-byte aligned: four int loads instead.
Output identical for 146 of 256 tokens, then a near-tie flips (summation
order); STRATA_MMVQ_WIDE_32=0 restores the old kernels.
- resident_plan groups the entries with one thread per entry (thread 0
alone took 73 us per layer); the plan is the host loop's exactly
- the GR down kernel reads xn from global memory instead of a 60 KB SLM
tile per work-group (occupancy); outputs bitwise equal (gr_bench)
- 16-byte-load wide kernels for Q4_K, Q5_K and IQ4_XS (IQ4 codebook in
registers): IQ4_XS 107 -> 323 GB/s, Q4_K/Q5_K 1.3-1.6x
Output tokens identical in every engine run.
Q6_K blocks are 210 bytes, so the wide kernel's 16-byte ql/qh loads were
2-byte aligned and the B70 split them: 145 GB/s against 640 GB/s for the
same blocks at a 224-byte stride. load16_a2 does two aligned loads and a
shift; with it the wide kernel takes every Q6_K shape and up to 8 columns
(2.6-3.4x in q6k_align_bench). Output tokens identical.
Also a port copy of ple_reader.cpp with STRATA_PLE_TRACE=1, which showed
the 45/39 tok/s two-speed runs are one 226 ms PLE stall landing in either
the prompt or the first decode round (TTFT + decode is the same).
- docs/UNSLOTH_Q4.md: what UD-Q4_K_XL is (Hugging Face revision 38bb39e, the four shards with sizes and SHA-256),
disk/RAM/GPU needs (111.3 GB download, 1.4 GB pack, no experts.bin), the iq_pack.py command and why
--compat-bf16, the numerical policy (native experts/embedding/head/projections/PLE; the converted small tensors
listed in conversions.json), the MTP draft layer, a server config (--native shard 1, --ple-gguf found by name,
--resident-budget-gib N with N = RAM minus 20-24 GB), measured speed on an RTX 5070 / 64 GB PC (7-8.5 tok/s at
40 GiB; ~60 s to the first answer), the switches (STRATA_LOOKAHEAD=0, STRATA_KQ256=1, STRATA_PARTIAL_PIN=1,
STRATA_FETCH_THREADS), scope and validation, credits (eddoursul #245, gopinath87607 #255, jagsan-cyber #247).
Experimental, not in setup.
- docs/DETAILS.md: the mapped low-RAM mode reads a native pack's experts from the GGUF when it has no experts.bin
(checked, identical answers on the Coder); --resident-budget-gib and the routing-aware prefetch; where the tier
counters are (--stats, the server log, /metrics' ram_blobs / file_blobs / file_mb); a pointer from "Which model?".
On the RTX 5070 / 64 GB PC the GPU computing a share of the misses over PCIe measured no faster than the CPU:
7.30 and 7.44 tok/s with 24 and 16 GiB of a 40 GiB budget registered, against 7.05-7.74 unpinned, and
registering costs startup time and driver memory. The default budget mode is now locked-only; the routing-aware
prefetch (multi-run A/B: 6.93 -> 7.91 tok/s mean, identical answers) stays on with the GGUF in place.
Add preliminary native v0.1.26 results, raw run JSON, scrubbed guest hardware facts, and the local API harness. Prompt and memory topology limitations are documented.
- RouterLookahead: when the CPU pool starts layer l, a worker thread applies layer l+1's router (a BF16 host copy
of ffn_gate_inp, 126 MB for 48 layers) to layer l's MoE input - the router's own input - for each token of the
window, takes the top STRATA_LOOKAHEAD_K (default 10), drops the experts the GPU cache (host_res) or the RAM
copy holds, and asks the source to warm the rest. It never waits: a prediction still running makes the next
one skip. STRATA_LOOKAHEAD=0 is the A/B arm.
- ExpertSource::warm / warms(): FileExpertSource with the GGUF in place issues one PrefetchVirtualMemory for the
predicted slices (madvise(WILLNEED) elsewhere). Pages only: blob(), the experts computed and their bytes do not
change (the same answers in every A/B run). Every other source: no-op, and the predictor is not started.
- bf16_rows_dot_multi (kq_avx2.cpp, AVX2): the router read once for all tokens of the window.
- Stats: experts warmed, how many of the decode's file-tier reads had been warmed, the predictor's time and skips.
UD-Q4_K_XL at a 40 GiB budget (single-token-router version): ~48-54% of the file tier's reads had been warmed;
reading time 850-968 -> 630-730 ms per verify round; decode 5.31/6.60 tok/s off vs 7.08/6.79/6.91 on.
- Locking the whole 40 GiB arena in the working set and then registering its prefix made the next device
allocation fail (the prompt path's token id buffer); the registered prefix is page-locked by the driver
already, so only the rest is locked now. 24 GiB registered out of a 40 GiB budget then starts and runs.
- The registered prefix ends at an expert boundary: a blob that began inside it and ran past it was taken as
page-locked by cudaMemcpyAsync and refused (an adaptive refill failed, with 16 GiB registered).
- The prompt path's token id buffer error names the CUDA error.
- kq_avx2 multi-token kernels: bit-exact, but no faster in the engine (the rows phases stayed at 72 ms per
verify round at a 40 GiB budget; a window's expert groups hold ~1.4 tokens and ggml's per-token calls find the
weights in L1), so they are opt-in with STRATA_KQ256=1; native_expert_parity still checks them bit for bit.
Experts without a VRAM slot are read once into pinned host memory and the
device-built verify plan points the expert kernels at them over PCIe. The
plan used to drop non-resident experts, so output with misses diverged.
Forced 1,879-expert miss: decode 2.6 -> 40.9 tok/s, output identical.
Before any later stage loads, split-auto checks whether CUDA0 alone holds every profiled expert pair plus the whole
session (KV of every layer), the drafter + head, the verify windows and the reserve; if so it clears the split (one
GPU) and logs why, else it logs how much of the profile CUDA0 would hold and keeps the split. Default behaviour is
unchanged: the engine flag is opt-in, and the server passes it only for a config with "split_skip_if_fits": true
(and a GPU list). Not with remote expert caches.
Measured on amdbox (Coder IQ1_M, --expert-cache auto --prefill auto, 128 greedy tokens, gpu [1, 0] = R9700 first,
2 starts per arm, warm = the second request of each size):
| arm | 4K warm | 16K warm | decode |
| split (flag off) | 1,244 | 1,897 | 50.5-53.0 |
| --split-skip-if-fits (on) | 1,776 | 1,791 | 59.7-61.8 |
| R9700 alone | 1,787 | 1,801 | 59.9-61.3 |
The flag's run is the R9700-alone run (same tokens, all four requests); with the 9070 XT first it keeps the split
("would hold only 9.84 of the profile's 23.42 GiB"). On this pair the split still reads 16K prompts ~5% faster.
- The expert arena was pinned at most 8 GiB whenever a layer split (or remote experts) ran, on every OS - a
Windows/WDDM workaround that cost a Linux 4090 + 3060 split two thirds of its prompt speed (#253,
bettercallcaleb). The cap now applies only under WDDM (_WIN32, or WSL2 detected by /dev/dxg), for multi_gpu and
expert_cache_remote alike. STRATA_ARENA_PIN_GIB overrides it on any OS: "0" pins the whole arena, N > 0 at most
N GiB (arena_pin_cap_gib(), -1 when unset: the same name and meaning as the S2 branch's helper, file-local here).
- The split's prompt chunk no longer stops the start when its buffers do not fit: a prompt path without a loan
first gets the largest chunk whose buffers (Prefill::bytes_needed) leave 512 MiB free on its device, and an init
that still fails steps down one size (8192 -> 6144 -> ... -> 2048 -> 1536 -> 1024 -> 512): Prefill::reset() frees
what the failed init made, the loans are re-priced, and the failed allocation's sticky error is consumed (else the
next launch check reported "out of memory").
- pinned.cu: the capped note no longer says "for CUDA1".
- docs: MULTI_GPU.md, SECOND_GPU.md.
Measured on amdbox (Linux, R9700 + RX 9070 XT, gpu [1, 0], Coder IQ1_M, 128 greedy tokens; tok/s):
| cache | arm | 4K cold | 4K warm | 16K cold | 16K warm | decode |
| auto (holds ~all) | 8 GiB cap | 678 | 1,355 | 1,709 | 1,880 | 44.5-51.5 |
| | whole arena | 703 | 1,234 | 1,729 | 1,864 | 44.6-51.4 |
| fixed 1500 on CUDA0 | 8 GiB cap | 608 | 1,101 | 1,287 | | 34.5-36.1 |
| | whole arena | 716 | 1,160 | 1,803 | | 34.8-36.0 |
(2 starts per arm for auto, 5 for fixed.) With most experts streamed the whole arena reads a 16K prompt 40%
faster; with every expert already on the cards the two are within noise except 4K warm (-9%: the pinned share
raises the auto lend to 90% and the streamed ring to 384 slots). Identity: the fixed-cache arms gave the same
tokens in all 5 + 5 starts (4K, 16K, 4K again), and the auto arms 2 + 2. The step-down was exercised on the
9070 XT with --expert-cache 3000 --no-prefill-borrow --prefill 8192 ("need 4528 MiB, 2910 MiB free: 2048-token
chunks"; with the init retry alone it went 8192 -> ... -> 2048, and 1536 ran all four requests). Known, also on
main: that own-buffer configuration at 2048 makes hipBLAS pick hipBLASLt kernels whose code objects this ROCm 7.14
lacks (TensileLibrary_..._gfx1201-xnack+.co not found, "launchKernel failed") and the prompt hangs. The WDDM side (Windows, WSL2) is unchanged and not re-measured here.
The copies' page faults read a few clusters each; one PrefetchVirtualMemory call for every role slice about to
be copied lets the memory manager queue large reads at once, and the parallel copies then find the pages
resident or in flight. Only the GGUF-in-place source's prefetch uses it; STRATA_FETCH_PVM=0 is the A/B arm.
- Registering 30 GiB of a 40 GiB RAM tier left the driver unable to page-lock the prompt path's token id buffer
afterwards (RTX 5070, WDDM: "prefill: the token id buffer"); the prefix is now capped at 24 GiB, the size a
whole 24 GiB budget page-locks with, and STRATA_PARTIAL_PIN_GIB sets it.
- A verify window's expert groups hold ~1.4 tokens on average; one token goes to ggml's own dot (the same bits,
without the multi-token kernel's per-token arrays).
Where the 288 ms/token at a 40 GiB budget went (UD-Q4_K_XL, --stats): of the 945 ms CPU pool call per verify
round (3.46 tokens), 871 ms were the dispatch thread assembling missed experts from the mapped GGUF one after
another (~391 blobs, 1.3 GB per round, ~1.5 GB/s: page faults from the SSD); the kernels were 42 ms gate/up and
30 ms down. At 24 GiB, 771 ms/round more went into the PCIe planning, which asked blob() - a file read with the
GGUF in place - of every miss just to learn it was not pinned.
- ExpertSource::prefetch: the dispatch hands the source the layer's missed experts before its jobs loop;
FileExpertSource with the GGUF in place assembles them on STRATA_FETCH_THREADS threads (default 8) so the
SSD sees a queue instead of one fault at a time. A no-op for every other source. Staging buffers being
filled are never reused, and a blob asked while its fill runs waits for it.
- The PCIe planning asks pinned() before blob(): the same decisions and bytes for every source.
- --resident-budget-gib: when the driver refuses to page-lock the whole budget, the locked arena's hottest
prefix (profile order) is registered with cudaHostRegister, as much as it takes (from the whole down in 2 GiB
steps), so the GPU can read those misses over PCIe and the prompt path can DMA them; pinned() answers for
that prefix only. STRATA_PARTIAL_PIN=0 is the A/B arm.
- src/kernels/cpu/kq_avx2.cpp: ggml-cpu's AVX2 vec_dot for Q4_K x Q8_K, Q5_1 x Q8_1 and Q8_0 x Q8_0 with the
token loop inside the block loop - the weight side decoded once per verify window, each token's integer chain
and float operations exactly ggml's, so the results are ggml's to the bit for any group size (no #152 rule).
native_gu_rows / native_down_rows use them for those types (only UD-Q4_K_XL has them); STRATA_NO_KQ256 is the
A/B arm. native_expert_parity checks them bit for bit against vec_dot for 1..8 tokens and times 4 tokens.
- Stats: the decode's file tier alone (blobs, MB, MB/round, reading ms/round and GB/s per thread) and the
prompt path's copies separately.
--kv-resident 32768 keeps the whole KV in pinned host memory and only the attended window in VRAM, so every
expert stays in VRAM at long contexts. B70, decode after the prompt: 128K int8 8.5 -> 38.7 tok/s, 128K q4_0
32.7 -> 35.3, 256K int8 3.7 -> 31.3, 256K q4_0 5.0 -> 31.5; the prompt pays 5-13%. setup.py now turns it on
from 64K for the SYCL engine (its RAM check counts only the KV copy and the engine, the experts being in VRAM)
and keeps INT8 KV, which is the faster of the two with streaming.
setup refused AMD GPU lists at install and at start ("NVIDIA only for now"), although the engine's layer split runs
on two AMD cards (0.1.30, from a hand-written config).
- amd_parse_gpus(): --gpus "1,0" (setup's HIP numbering, the first card the main one) or "all" (every supported
card, the most VRAM first); every card must be in AMD_ARCHS, else a plain message with what can be used.
- install: --backend hip --gpus writes "gpu": [..] + "layer_split" like the NVIDIA path; build_engine_hip compiles
for the set of the chosen cards' archs (CMAKE_HIP_ARCHITECTURES "gfx1100;gfx1201" already worked) and records
it in BUILD.json; an engine whose archs cover the cards is not rebuilt; update_installed_engine keeps every arch.
- rocm_root(archs): cards of two families need a system ROCm (TheRock's wheels hold one family) - said plainly.
- start: --gpus on an AMD install is saved like on NVIDIA (refused when the engine has no code for a card);
--gpus all finds the AMD cards on a PC without NVIDIA cards that can share.
- docs/AMD_HIP.md, README; tools/test_setup_amd.py: lists, "all", refusals, the one-family wheel rule, the arch set
in the build and BUILD.json.
Cut down from #254 and #256 onto 0.1.30's AMD tables:
- setup.py: AMD_ARCHS / AMD_NAMES / ROCM_INDEXES / AMD_CARDS list gfx1101 (TheRock gfx110X-dgpu) and gfx1200
(gfx120X-all); amd_gpus() takes the sysfs root so the detection can be tested.
- cmake/hip_backend.cmake: gfx1101 and gfx1200 build without the "not validated" warning (a status line names
them community-validated); gfx1102 (#192: ctest only) keeps the warning.
- docs/AMD_HIP.md (Community-validated cards: the owners' setups, ctest and numbers) and README.
- tools/test_setup_amd.py: detection on a mocked KFD topology (every family, the CPU node, an iGPU and gfx1102
listed as not supported, product-name fallbacks, the ROCm index per family).
Left out: the PRs' own cmake/device.cu arch checks (0.1.30's compiled_for covers them), the dp4a change (main has
it), math_constants.h, the bench JSONs, the launcher, and both hipBLASLt tables (neither PR measured the prompt speed
without its table).
Co-Authored-By: Joseph Hohertz <[email protected]>
Co-Authored-By: efeisot <[email protected]>
Cut down from #262 (ttio2tech): only the intrinsics and their test.
- HIP's __byte_perm indexes a byte array in private memory (scratch traffic in the IQ4_XS/Q2_0 table
lookups); byte_perm() is the same selection (nibble & 7 of the pair {y:x}) in one v_perm_b32.
- __vsub4, __vsubss4 and __vcmpne4 work on all four byte lanes at once (Hacker's Delight 2-18) instead
of a per-lane loop.
- tests/hip/intrinsics checks them on 1,024 random operands against host references (passes on gfx1201).
Left out: the arch/setup/device.cu parts (0.1.30 already builds and checks gfx1201; main's dp4a covers
gfx1100-1102/1200/1201) and the gfx1201 hipBLASLt table (none is shipped for gfx1201).
A/B vs main on amdbox (Coder IQ1_M, 128 greedy tokens): identity with a fixed cache (--expert-cache 1500
--prefill 2048), 5 + 5 starts per card, 4K / 16K / 4K again: the modal outputs are identical on the R9700
and on the RX 9070 XT (one start of 20 differed from its arm's mode, the known HIP start drift). Speed with
setup's settings (auto cache/chunk), 2 + 2 starts per card, tok/s:
| card | arm | 4K cold | 4K warm | 16K cold | 16K warm | decode |
| R9700 | main | 1,027 | 1,792 | 1,695 | 1,812 | 46.0-52.2 |
| | #262 | 1,023 | 1,786 | 1,681 | 1,808 | 53.0-61.0 |
| RX 9070 XT | main | 781 | 1,018 | 1,421 | 1,517 | 32.7-41.4 |
| | #262 | 780 | 1,017 | 1,424 | 1,517 | 34.9-44.0 |
Decode +15% on the R9700, +5-7% on the 9070 XT; prompts unchanged.
Co-Authored-By: ttio2tech <[email protected]>
setup.py gets a third backend, sycl: on an Intel Arc with build-sycl-aot/strata and the strata-sycl-dev image
present (or --intel-engine strata; --intel-engine llama keeps llama.cpp), it builds the native pack and the
MTP draft layer (reusing an mtp-q2_0.gguf that is already there), writes the engine's arguments with the
container's paths through sycl/serve/strata-sycl.sh, --stream-experts with a reserve that keeps every expert in
VRAM (1,024 MiB up to 32K, 2,048 MiB and 4,096-token chunks above), Q4_0 KV above 64K, and keeps the sampling
block of an earlier config. start() checks the model files through the config's sycl_root; upgrade_config
leaves the SYCL config alone (the wrapper has no version, which read as 0.1.12 and downgraded --prefill).
Tested on the B70: the Coder's config regenerated by setup serves at 38.9 / 30.3 tok/s (first turn / follow-up)
with all 12,288 experts in VRAM.
The serve loop used to abort at exit (139): the Verifier's destructor waited on its queue after the Level
Zero context had begun tearing down. The destructor is now exception-safe and serve mode exits directly once
its requests are done: exit 0.
The Coder on the B70 now runs Strata's own engine behind serve/server.py --engine strata: the wrapper starts
build-sycl-aot/strata inside strata-sycl-dev with the serve pipes attached and the port's run-time switches set.
Through the OpenAI API with temperature 0.6 / top-p 0.95 / top-k 20 / repetition penalty 1.05: 36 tok/s decode
on a first turn, 33 on a follow-up (conversation cache reused), vs 24-26 for llama.cpp on the same card. The
expert reserve has to leave every expert in VRAM (--stream-experts has no host copy): 1,024 MiB at 32K context.
The selection dump measured how much neighbouring prompt positions' QSA selections overlap at 80K tokens: the
union of 8 is 3.3x one selection (16: 5.1x), 12% shared by all 8, so a grouped-gather attention kernel would buy
at most 5-8 s of an 80K prompt. XMX v2 across the 128K/256K matrix: 23-33% slower prompts than the FP32 attention.
- FileExpertSource: a native pack without experts.bin, given the --native shard, maps every GGUF file
native_experts.txt names (after check_experts_gguf) and assembles a blob [gate | up | down] from the
three role slices when asked: into a small pool of buffers for blob() (a buffer is reused once its blob
has not been asked for during 3 layers (begin_layer) or 256 assemblies), or straight into the caller's
memory for copy_blob(). No 30-77 GB experts.bin on the disk. With experts.bin present nothing changes.
- ExpertSource::transient / copy_blob: the prompt path's stager copies a transient blob itself (its queue
holds a whole chunk's experts), everything else keeps using blob().
- --resident-budget-gib N (the resident mode with a budget): the experts the GPU cache does not hold are
copied into RAM in expert-profile order while they fit N GiB (clamped to available RAM minus the
headroom), the rest are read from the files; no lend region then. Without the flag the resident mode
and --mmap-experts are unchanged.
- Counters: blobs served from the RAM copy, from the files, and bytes read from the files; in the stats
table ("expert tiers"), the serve log, and appended to DONE (RAM blobs, file blobs, file MB for the
request), which serve/server.py puts into /metrics' request history (older DONE lines still parse).
- expert_layout_test: the GGUF-in-place source gives the loaded arena's bytes through blob() and
copy_blob(), an assembled blob stays put while in use, and experts.bin still wins when present.
- Dots: llama.cpp's vec_dot_q4_K / q5_K / q8_0_q8_1 transcribed (VDR 2), via eddoursul/Strata
8029fa9 and #255 (gopinath87607, Q8_0), which agree. Q5_1's min term multiplies the sum of the
QUANTIZED activations (ggml-cpu's convention, as the fork does), not llama.cpp CUDA's ds.y.
- Dequantizers dq_q4_k / dq_q5_k / dq_q5_1 / dq_q8_0 (llama.cpp's formulas) for the prompt path's FP16
fallback, f32 dumps and the Q8_0 embedding; iq_row_bytes / is_iq know the four types;
iq_dequant_gu_f16 now refuses an unknown type instead of leaving its output unwritten.
- The grouped gate/up, down and iq_mmvq switches are generated from STRATA_GU_FMTS /
STRATA_D_FMTS / STRATA_MMVQ_FMTS; the existing entries launch the same kernels as before.
- native_expert_supported(gu, d, n_embd, n_ff): generate checks every layer of a native pack right
after expert_layout_load and refuses an unsupported pair by name before anything is allocated
(it used to exit from inside the first verify window).
- Native PLE key: Q8_0 accepted (NativeDense, generate, ple.cu); native_mmvq already runs it.
- native_expert_parity: --synthetic GU/DOWN (random ggml-quantized blobs at 2560/640; port of the
fork's mode), GPU dequant f32/f16 and embedding rows vs ggml to_float bit for bit, split models via
GgufModel, --q5_1-min. CTests: q4_K/q5_1, q4_K/q8_0, q5_K/q8_0, q8_0/q8_0, the Q5_1 min term, and
q6_K refused. RTX 5070: GPU and CPU within 1.25-1.37e-2 of the float reference for all three real
pairs; every dequantized value identical; Q5_1 GPU vs ggml-cpu 7.5e-5 (vs 1.95e-2 for the ds.y
variant); IQ3_XXS/IQ4_NL and IQ2_XS/Q2_0 unchanged.
- Not yet: #242's decode-once traits for the new types (they take the per-entry kernel; see
phaseA-tests/s5_cs3_design.md), prompt MMQ for Q4_K/Q5_K/Q5_1, identity gates.
`gguf_split_test --real SHARD` opens a real model's shards (GgufModel), checks the architecture on the
metadata shard and prints where output / token_embd / the PLE table and key / layer 11's experts are.
`expert_layout_test --real PACK NATIVE_SHARD` checks a real pack's native_experts.txt against its model
(check_experts_gguf; no expert byte is read). Both pass on the Q2_0, IQ3_XXS (v2), IQ3_S, Coder IQ1_M
and Swift IQ2_XS packs and models on the reference PC.
- FORM (eddoursul/Strata's list of which tensor the engine reads as BF16, F16 or F32) decides the
pack form; the conversion itself is upstream's bf16_bytes (gguf-py, finite check, round-to-nearest-
even). By default only lossless conversions happen: F32 whose values are exactly BF16 (low 16 bits
zero; before, only the routers) and F16/BF16 widened to F32. Rounding (inexact F32, Q8_0 hyper-
connection / PLE value / output_hc, F16 -> BF16) still needs --compat-bf16. NaN/Inf are refused
even where the low bits are zero.
- A Q8_0 PLE key stays native (served from the GGUF, like Q2_0); other quantized keys take the BF16
path as before. (The engine's native PLE key learns Q8_0 in the formats change.)
- blk.1.ple_conv1d stored F32 (Q8_0 / UD-Q4_K_XL files) goes into dense.bin as index kind 3 and the
loader narrows it to F16 (#255, gopinath87607), with or without --compat-bf16; a value beyond
F16's range is refused. generate refuses a pack whose conv1d is not F16 (#255's check, which also
accepts the canonical Q2_0 pack's verbatim-F16 kind 0 - #255 as written refuses that pack).
- conversions.json (schema 1): the source shards' names and sizes, and per converted tensor its
shard, source/destination type, method, exact flag, largest absolute error and source sha256.
compat-bf16.json is written as before. Refusals happen before anything is published.
- Byte identity: index.txt, native_experts.txt and dense.bin of Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M and
Swift IQ2_XS are identical to main's packer (Swift's 96 exact F32 routers are now listed in
conversions.json).
- Tests: exact F32->BF16 by default, inexact only with --compat-bf16 (error bound), NaN/+-Inf
refused with and without the flag (previous pack untouched), conv1d kind 3 (bytes, record, F16
kept, overflow refused), Q8_0 projections need the flag and are recorded, Q8_0 key native.
- Each expert role is resolved in the shard that holds it and its offset is absolute in THAT shard
(the first role's data_start was used for all three; #255, gopinath87607). A layer whose roles
are in different shards gets the per-role column `gate,up,down` and the file becomes v4; every
other pack keeps the v3 header and bytes (checked on Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M, Swift
IQ2_XS: index.txt, native_experts.txt and dense.bin identical to main's packer).
- The split keys are checked as the engine does (shard 1 has the metadata; split.count / split.no /
split.tensors.count consistent). The expert table is computed and checked before any file of
the pack changes.
- experts.bin is written under a temporary name and gets experts.bin.src.json (the shards' names
and sizes, the sha256 of native_experts.txt); it is reused only when that matches, never by size.
An experts.bin of another source is refused without --experts-bin (the engine would read it).
- dense.bin and index.txt are published together after the old completion marker
(native_experts.txt) is removed, so a stop part-way never mixes files of two runs.
- .absolute() stays (not .resolve()): symlinked HF snapshot names keep the split naming.
- Tests: a synthetic 4-shard GGUF (shard 1 metadata only, layer 1 split per role; offsets per
shard, experts.bin byte for byte), v3 kept when nothing is split, missing shard, duplicate tensor,
a shard of another split, truncated shard, wrong type (pack untouched), interrupted pack (experts
and index), stale same-size experts.bin. Idea of per-role names also in #247 (jagsan-cyber).
Groundwork for Unsloth's UD-Q4_K_XL (4 shards: shard 1 metadata only, output/token_embd/PLE table in
shard 2, layer 11's down in shard 2 and its gate/up in shard 3). Existing packs load unchanged.
- include/strata/artifact/gguf_split.hpp: gguf_split_paths (from eddoursul/Strata 8029fa9) finds the
shards from any shard's name; a missing shard is an error naming it. It replaces generate.cpp's
model_shards(), which skipped a shard it could not open.
- gguf_reader.hpp: GgufModel (8029fa9) looks a tensor up across the shards and refuses a name in two
shards; the split keys are validated as #255 (gopinath87607) does for the head: shard 1 carries the
architecture, later shards declare split.count / split.no / split.tensors.count consistently, and the
directories add up to split.tensors.count. tensor_payload_bytes / in_bounds for payload checks.
- NativeHead::load / NativeEmbed::load take the shard list: the architecture from the metadata shard,
the tensor from the shard that holds it; the embedding also checks its payload is in the file.
- generate: --native resolves all shards once (native_shards), the head from --native-head-gguf's
family, the model keys from the metadata shard; --ple-gguf defaults to the shard that holds
per_layer_token_embd.weight.
- native_experts.txt: the shard column may be per role, `gate,up,down` (#255's comma form; empty =
the --native shard). The reader accepts v2-v4 and refuses a higher version; an engine older than
this one fails to open a "gate,up,down" file loudly. Fixes the parse's own aliasing hazard
(parts.assign(3, parts[0])).
- load_experts_gguf reads each role from its file, after check_experts_gguf proved every (file,
offset) is that tensor, of the pack's type and dimensions, and inside its file.
- Tests (synthetic, no model, no GPU): gguf_split_test (4 shards, missing shard, duplicate, split keys,
count, truncation); expert_layout_test (copies of the Q2_0/IQ2_XS/IQ3_XXS/IQ3_S/Coder/Swift packs'
native_experts.txt parse field for field as before; v4 column; v5 / 4 / 2 names refused; the arena
from split synthetic shards at 2560/640 with Q4_K/Q5_1 and Q5_K/Q8_0, every byte compared, and
wrong offset / file / type / shape / truncated / missing shard refused).
v2: 64-cell chunks, V packed two cells at a time as 32-byte stores, K^T as 4-byte pairs with INT8 converted on
the load, hi+lo Q in one accumulator per scale group, one accumulator update per chunk. qsa_prompt_attn_parity
passes in every mode and kv_hybrid_parity passes completely with it (its last step needs a tensor-core-class
prompt kernel). 13.4 ms per chunk vs v1's 18.7 and the FP32 fallback's 5.5: the attention is gather-bound, so
it stays opt-in (STRATA_PROMPT_ATTN_XMX=1, or =32 for 32-cell chunks).
INTEL.md: 128K and 256K prompts with int8, q4_0 and k8v4 KV on the B70. All complete; q4_0 keeps every expert
in VRAM at 128K (decode 32.7 tok/s after the prompt); at 256K all three evict experts and decode drops to 4-5.
Without borrowing, the VRAM reserve for the KV state and the chunk buffers evicts ~1,900 experts, the prompt
streams them (790 tok/s at 80,000 tokens) and decode after it runs at 2 tok/s. Borrowing lends ~940 slots and
refills them in ~1 s: 1,062 tok/s and 35-40 tok/s decode. It costs ~1 s per prompt at short contexts (2,184
tokens 610 vs 792 tok/s), so the default is by context: borrow above 32K. --prefill-borrow / --no-prefill-borrow
decide explicitly; layer splits keep their own buffers as before.
-DSTRATA_EXPERIMENTAL_SM60=ON now names Volta (sm_70) beside Pascal (sm_60)
and is a declared option; below sm_60 is refused even then. The runtime check
in device_info refused anything below compute capability 7.5 even in that
build, so the flag produced an engine that could not start on the cards it was
built for: strata_core now gets STRATA_EXPERIMENTAL_SM60=1 with the flag and
the check admits 6.0 and up there. The release engine is unchanged (7.5).
Checked here: configuring for 70-real without the flag still fails with the
7.5 message; with it (CUDA 12.6 - CUDA 13 no longer compiles compute_70, noted
in the comment) it configures and strata-device builds with the define.
- Re-migration merge of upstream 99b4097..0ea2fc3 (GDN recurrence pipelining, block scores read once per window,
the GPU-wide sampler, the draft head's VRAM reserve, correctness fixes); verify.cpp/mtp.cpp ported by hand; two
new fixups (a cudaStream_t cast in the sampler, a const sycl::free in native_head). Parity unchanged (19 of 21).
B70: 2,184-token prompt 765 -> 792 tok/s, decode at that context 43.1 -> 45.2 tok/s, outputs identical.
- The prompt path's stream plan kept GgufExpertSource::blob() pointers for ~1,900 experts per chunk against a
512-slot ring, so blobs were overwritten before they were copied (the first 80k measurement was partly wrong
experts). The stager threads now read (layer, expert) themselves via GgufExpertSource::read_into.
- The stream-all walk hangs in its first large chunk on this card (copy engine stuck on a barrier, any ring size,
either issuer; the old read path too). The port defaults to the per-layer routed-only walk (upstream e258e6e's
STRATA_PREFILL_RING=8, included); STRATA_PREFILL_STREAM_ALL=1 restores it for debugging.
- The PLE reader copy keeps its 64-deep queue. 80,000 tokens now complete: 101 s = 790 tok/s.
Every CPU expert kernel is AVX2 at least (q2_avx2.cpp, iq_avx2.cpp; the
AVX-512 ones are chosen above them) and so is ggml-cpu in the release build,
but nothing checked for AVX2: a pre-Haswell CPU died of an illegal instruction.
cpu_avx2_ok() (AVX, AVX2, FMA, F16C and the OS's XSAVE state) is checked
before the pack's layout load, and the message names the CPU (cpu_name, the
CPUID brand string). STRATA_FORCE_AVX2 is unaffected (it only answers
cpu_avx512_ok).
The #251 report stalled in "reading the prompt, from token 0" and printed the
previous request's verify window as if it were the stuck one.
- Verifier::diag says "(last window, not the current stage)" when the stage
is not a verify window.
- The batched prompt read sets its stage per chunk (preparing, finishing) and
per layer, with the chunk's first position (Progress::chunk, a third
progress_at argument; every other call leaves it -1), so the stall report and
the watchdog line read e.g. "reading the prompt (batched): layer 17 of the
prompt chunk from token 8192".
Text only: nothing computed changes.
A verify window the GPU never finished held the host in an unbounded
cudaStreamSynchronize, and the serve watchdog then aborted the process with
the window's spin kernel (wait_flag_ge / wait_flag_ge_or on a host-mapped
flag) still resident - on Windows the GPU was left "lost" until a power cycle.
- The final wait of Verifier::run polls cudaStreamQuery (the same wait while
the window runs) and gives up after 20 s like the per-layer wait, through
the existing error path.
- Verifier::release_gpu_waits writes UINT32_MAX into the mapped flags M/A/B
(past every ring, so each spin kernel returns; wait_flag_ge_or returns on its
flag too), then waits for the streams up to 5 s; the verifier refuses later
windows. Both timeouts run it before returning their error, and the serve
watchdog and the expert pool's stall aborts run it (progress.hpp
release_gpu_fn, every live verifier) before std::abort.
- Test hook: STRATA_TEST_VERIFY_STALL=N withholds the last layer's flag in the
N-th window; unset it is never taken.
The normal path runs the same kernels in the same order.
When cudaHostRegister refuses the whole expert arena, the sliced fallback
pinned layer after layer until the driver refused one (28 GiB on a 63 GB
single-GPU PC), and WDDM then refused every later device allocation
("native head upload: out of memory"). On Windows the sliced pass now stops
at the adapter's DXGI non-local budget - the process's current usage - 4 GiB
(IDXGIAdapter3::QueryVideoMemoryInfo for the CUDA device's LUID; dxgi.dll is
loaded at run time), or RAM/2 - 8 GiB when that query fails. The arena line
names the cap and where it came from.
STRATA_ARENA_PIN_GIB: unset = the above; 0 = no cap (the old behaviour);
N > 0 = at most N GiB from the start. arena_pin_cap_gib() (pinned.hpp)
reads it, -1 when unset, for the multi-GPU cap in generate.cpp as well.
A machine whose whole-arena registration succeeds is unchanged, and a
caller's cap (multi-GPU) still takes precedence.
This PC's whole registration fails too: IQ3_XXS now pins 24 GiB (32 slices;
budget 30.8 GiB - 2.0 in use - 4) instead of 28 GiB (36).
The GGUF branch of the arena load (a native pack without experts.bin, the
normal install) read through std::ifstream, which MSVC splits into 4095-byte
freads - 0.02 GiB/s on the reporter's PC. It now uses the same fopen/fread
with a 64-bit seek as load_experts_ranges (#89), with an RAII close. Same
chunks, same bytes, same scatter.
Measured here (IQ3_XXS pack without experts.bin, RTX 5070, 64 GB): 1.49-1.60
GiB/s before, 3.17-3.25 GiB/s after (start 45-48 s -> 29-34 s); the generated
tokens are the same.
Chats (and the Chat tab's settings and API key) only in the browser's local storage (strata.* keys), the start
config strata-<model>.json and its siblings in the Strata folder, the model files in Strata-data next to it (or
--data-dir), and where that folder is remembered (%APPDATA%\Strata\settings.json, ~/.config/strata/settings.json).
A request (OpenAI or Anthropic) may set "reasoning_budget_tokens": N, and "reasoning_budget_tokens" in
strata-<model>.json sets a default (a request's own value wins; 0 means no budget). When the thinking reaches N
tokens - at a clean point, no tag held back and no character split across tokens - Service.run stops the engine,
feeds a short wrap-up line and </think> through the detokenizer and parser (the client sees it as the end of the
thinking; it counts as output), and generates again from prompt + what was generated + the wrap-up: a prefix of
what the engine's session already holds, so it continues from it. With no room left to answer it ends with
"length" as before. A bad value is a 400 before anything is sent (a bad config value refuses to start). Off
by default: without the field and the key every request runs exactly as before (one engine pass). Anthropic's
thinking.budget_tokens still only picks the level. Documented in docs/DETAILS.md.
Tests (serve/test_server.py ThinkingBudget, a fake engine that continues after </think>): off by default, the
wrap-up and the continued prompt (the first prompt + the kept thinking + the wrap-up), the Anthropic stream, a
budget the thinking stays under, the config default and a request turning it off, thinking off, no room left,
bad values.
- The ready-made engine: with the default --prebuilt, setup looks at releases/download/v<version>/ first (the
version CMakeLists.txt says, source_version()) and falls back to releases/latest with a message when that
release is not published (yet). An explicit --prebuilt / STRATA_PREBUILT_URL is used as before; an
installed engine new enough is kept without asking the network.
- Hugging Face: every model family's shards and vision encoder come from a fixed commit (HF_REVISIONS, the repos'
`sha` on 2026-09-30), tools/mtp_fetch.py reads the BF16 checkpoint at its commit (STRATA_MTP_REVISION
overrides). A revision a repository no longer has falls back to its current files, with a message. Files
already downloaded keep their names and .done marks, so nothing is downloaded again.
- Python: requirements.txt pins the packages and their dependencies (numpy per Python range, 3.10+), used by
setup's pip step and the Dockerfile. A venv set up before this (bare names in .strata-pip.json) is left as it
is; only packages it never had are installed, pinned.
Tests: tools/test_setup_pins.py (mocked network: the tag URL first, latest as the fallback, an installed engine
kept, a gone revision falling back to main in download() and mtp_fetch, the pip step for fresh, old and bumped
installs).
m16n8k16 needs sm_80, but sm_75 has m16n8k8 with the same A/B/C register mapping, so
each k=16 step becomes two k=8 steps on the fragments as they are already laid out
(a[0]/a[1] with b[0], a[2]/a[3] with b[1]). The sum now rounds twice, so the two paths
do not agree bit for bit.
cp.async does not exist before sm_80, so Turing takes the v1 kernel (launch<1>, same
accuracy, another summation order) instead of launch_i8; the capability probe now
accepts cc >= 7 and a device without cp.async sets g_turing.
Measured on a Quadro RTX 8000, CUDA 12.9, arch 75: qsa_prompt_attn_parity PASS
(FAILURES: 0; vs FP64 new 6.99e-06, old 2.37e-06, output scale 3.62); the int8 kernel
62.3 -> 19.6 ms per chunk (3.19x), fp16 65.0 -> 25.1 ms (2.59x). Engine prefill on
three Turing cards: 9.7K-token prompt 1,050 -> 1,177 tok/s, 42K 1,124 -> 1,251 tok/s;
an 80K needle test answers correctly.
iq_mmvq, native_gu_kernel and native_down_kernel called the format's dot once
per column (or group entry), re-reading the weight words and redoing the grid
lookups and sign unpacking each time. Each dot is now split into load (weight
side) and apply (activation side, same integer chain and float expression), as
native_mmvq.cu's multi-column traits are: the dense kernel takes 1, 2, 4 or 8
columns per pass (8 at a time past 8), the grouped ones 4 entries per pass
(~48 registers; the default verify window, --spec 4 with the split off, fits
in one pass).
Every column keeps the same calls, lane stride, summation order and warp_sum,
so results are bitwise equal to the old kernels, which STRATA_OLD_IQ_MMVQ=1 (or
iq_set_old_kernels) keeps for A/B timing.
Tests: iq_multi_parity (ctest, GPU, synthetic, no model) checks old vs new
bitwise for every format, ncols 1..8 and 11, groups of 0..11 entries; --bench
times both. iq_parity (real rows from tools/iq_fixture.py) now runs the dot for
1 to 8 columns and requires every column of a multi-column call to equal a
one-column call on it bitwise (exact layout, the default). native_expert_parity
runs the GPU expert with both the decode-once and the per-entry kernels on real
blobs and requires the same bits.
The grouped kernels (verify window, MTP layer) staged each entry's activations
as xs_q[k][c * 8 + j], an 8-way bank conflict on every read, and recomputed the
chunk sum hx with eight dp4a for every row. The per-hit kernels (token graph)
read each chunk with 44 byte loads.
New kernels, on by default:
- activations regrouped once per chunk so the codes expand with shift+mask
and hx is summed once per chunk;
- grouped: word-major shared layout xs_w[j][k * NC + c] plus (dx, hx) pairs,
conflict-free on the staging writes and the reads, two rows per warp;
- per hit: nine aligned word loads per activation chunk, uint2 code loads,
16-bit scale loads, one warp per (gate, up) pair or pair of down rows.
s, hx and the float expression dw * dx * (float) (s - hx) are unchanged, so
every output matches the previous kernels bit for bit. STRATA_OLD_GROUPED=1
(or moe_grouped_select_old(1)) selects the previous kernels, and so does any
activation row, scratch or blob arena the wide loads cannot take.
STRATA_GROUPED_PAIR_MIN_HITS=N keeps the previous per-hit kernels below N hits
of capacity. moe_grouped_last_path() reports which kernels the last call ran.
The funnel shift that realigns an activation chunk is written as a 64-bit
shift instead of __funnelshift_r, which nothing else in the tree uses.
tests: s2_expert_grouped_parity runs every entry point with the previous
kernels then the new ones on synthetic Q2_0 blobs and Q8_0 activations, checks
the path taken, and requires output and scratch to match byte for byte, plus a
double-precision host reference. It covers the stride and arena fallbacks and
a 2-byte aligned activation row. --bench times previous against new. It
needs a GPU and is registered in ctest.
Low-RAM mode on a multi-GPU pick dropped to one GPU with only "the low-RAM mode runs on one GPU". It now says
why: the experts plus ~10 GB for the rest (e.g. 43 + 10 = 53 GB) against this PC's RAM, or that --low-ram was
chosen; that the engine does not split layers in that mode (it refuses --resident-experts with a layer split);
and what keeps all the GPUs (more RAM, a smaller size, --low-ram off).
--low-ram off on a PC short of RAM asked "Install it anyway?" with the default n, so an unattended --yes
install always stopped although the user had turned the mode off explicitly. With --low-ram off the default is
y (--yes installs, with a warning); with auto the question and its n default are as before (#125), and the stop
message names --low-ram off --yes. The two blocks are functions now (low_ram_one_gpu_why, confirm_paging);
tests in tools/test_setup_lowram.py with mocked input.
The follow-up to #231: openai_collect kept a call the output ended inside, with its truncated arguments, in
message.tool_calls (finish_reason "stop" or "length"). It now leaves out every call whose arguments are not
JSON when the finish reason is not "tool_calls" - what anthropic_collect does with such a tool_use block. The
streamed answers are unchanged. Tests: the cut call over both whole APIs (at the end of the turn and at the
token limit) and openai_collect directly.
When the model's output ended inside a tool call whose name was already
streamed, finish() closed its JSON and emitted a tool_call event, and the
answer ended with finish_reason "tool_calls" / stop_reason "tool_use", so
clients ran the truncated arguments; an MCP call ran with empty arguments.
The call now stays unfinished: its JSON is not closed and no tool_call
follows its tool_start. Both APIs report "tool_calls" / "tool_use" only when
every streamed call completed, and the non-streamed Anthropic answer leaves
out a call whose input never closed.
Fixes#211
_bpe rescanned every adjacent pair after each merge: O(n^2) on one long pre-token piece (a CJK run, a minified
blob). With the pack's vocabulary a 2,000-character CJK run took 1.6 s and 8K characters tens of seconds;
_bpe_heap takes about 5 ms and 20 ms. It pops the lowest rank, then the lowest position - the order the scan's
strict `<` picks - and skips pairs whose symbols have merged since they were pushed. Words up to HEAP_MIN (64)
symbols run the existing scan unchanged.
Tests (serve/test_detok.py HeapBpe): the heap against the scan word by word and through encode (both
parse_special modes, heap forced on for every word and at the default threshold) over random text with CJK and
Hangul runs, conjoining jamo, combining marks, URLs, minified code and specials - with a synthetic vocabulary
(always) and the pack's (when there is one); an 8K-character CJK run round-trips in well under a second.
Service.run() recorded the finished request's history row and totals, set busy=False and dropped `tail` in a
finally that ran after `with self.fifo:` had released the lock. A request waiting for the fifo could start in
that gap: the old request then recorded the NEW request's status as its own, cleared its busy flag and popped
its tail, and the new request crashed in _note (KeyError 'tail'). The block now runs inside the fifo (an
inner try/finally, which also sees the disconnect), and _note reads the tail defensively. Test: a stream
aborted mid-answer while another request waits, with a fifo that lets the waiter run first on release.
Rows that serve 1 or 2 activations (the near-universal shape of routed
decode) read each weight block twice less often: one 256-bit shuffle
ladder covers two consecutive blocks (lanes are per-128-bit, so the
arithmetic is the SSE ladder run twice and the re-homed halves keep the
original byte order) -> the same madd lane mapping, bit-identical
results. The SSE unpack stays for 3+ tokens: at four accumulators the
extra live unpack vectors spill registers and it measurably loses.
Measured on i7-8700K AVX2, qwen3.8 125B-MoE Q2_0 decode: +10% gate/up and
+5-7% down at 1 token/expert in the standalone harness
(ML-Experiments/bench5/q2_cpu_bench.cpp, bit-exact verified), and -7
ms/verify-window end to end (CPU pool waitCPU ~19 -> ~13 ms), GPU stages
and generation unchanged.
Turing (sm_75) opt-in shared memory is 64 KiB. With the old fixed
TILE=2560 staging the down kernel could hold at most 6 tokens
(2560*6*4 = 60 KiB), so 7+ token windows split into two launches and,
more importantly, a single 40 KiB tile at T=4 filled the SM so only
one block per SM fit next to the resident expert/attention work.
Specialize gr_down_multi_kernel on the per-token tile size:
- sm_75: TILE=1280 -> up to 8 tokens staged in one 40 KiB launch
(kFusedGrMaxT=8 covered), and T-token tiles fit 1-3 blocks/SM
depending on T, so concurrent hc lanes can overlap.
- every other arch keeps TILE=2560 exactly as before.
Per-lane weight chunk order within a tile stays strictly increasing
in (tile, chunk), so the accumulation order for every (expert, row)
is identical to the 2560 path and results are bitwise unchanged.
Measured on RTX 2070 (8700K host, engine 0.1.27, Qwen3.6 125B Q2_0,
29K context, temperature 0, identical benchmark): GDN hc0 down
2.73-2.80 -> 1.88-1.98 ms/window (-30%), QSA hc0 down 0.90-0.93 ->
0.62-0.65, up/norm kernels unchanged, GPU util 91.7 -> 93.5%,
decode tok/s slightly higher on average (within per-run noise;
window is CPU expert-scheduling bound).
The i-quant fixtures were never published, so iq_parity reported ten
"missing fixture" failures on any fresh tree. tools/iq_fixture.py now
generates them deterministically (seeded rows/cols, RandomState) into
the BUILD directory: for each of the ten types it writes the raw block
bytes (sane fp16 scales at each layout's scale offsets, seeded index and
sub-scale bytes) plus the gguf-py dequantized reference - gguf-py is
dequant-only for the i-quants, so no quantizer is needed, and Q2_0 (the
repository's type 42) uses tools/gguf_writer.py's codec. The kernel
type ids in the headers match the vendored gguf-py's enum one for one.
Dependencies: python3 with numpy and the vendored gguf-py - nothing is
pip-installed, and CMake probes the configured interpreter and .venv,
registering the tests only when a usable interpreter is found and
saying so otherwise.
CTest now runs iq_parity (previously built but never registered) with
the fixtures as an explicit CTest fixture: generation completes before
the test even under parallel execution, and a generation failure fails
the run. The missing-fixture diagnostic names both candidate files and
the generating command; the nonzero exit is preserved.
`Gemm::init` collapsed five distinct failures into three bare strings ("cublasCreate
failed", "workspace", "dequant scratch of N MiB"), so an init failure on a tight machine
could not be told apart from a driver or install problem without a debugger.
Each call is now checked through two small helpers that report the operation and the real
status - `cuBLAS status N` or `cudaGetErrorString` - and optionally set
`resource_failure` when the failure means the machine is out of room
(`CUBLAS_STATUS_ALLOC_FAILED`, `cudaErrorMemoryAllocation`). A caller can then retry with
a smaller cache or prompt instead of treating a shortage as fatal.
The parameter is defaulted to nullptr, so every existing caller is source-compatible; the
upstream HIP/hipblaslt block is untouched.
Verifier::window_logprobs writes one line per row of the last verify window's
head: position, target, its log-probability, the top token and its
log-probability, a hit flag, the log-probability of STRATA_LOGPOS_EXTRA
(default 248046, <|im_end|>) and the target's log-probability renormalized
without that token. read_windows calls it before commit when
STRATA_LOGPOS=<path> is set; otherwise the only cost is one null check.
A layer split's earlier stage forwards to the stage that holds the head,
under OnDevice as commit() does.
AMD_HIP.md: the per-arch ROCm wheels, the system-ROCm rule, the table rule, the
arch list and the startup check; an RDNA4 section (ctest 42 pass / 1 skip /
2 non-GPU failures, the arch check, end-to-end 4K/16K numbers on both cards and
with setup's TheRock 7.10 wheels, the 9/9 server smoke on each card, why no
gfx1201 table is shipped, what is not validated); a Tuning table section with
the tune_hipblaslt command at the engine's shapes. setup's no-table message
points there.
On gfx1201 a plain store to mapped pinned memory stays in the GPU's L2 until the
stream is synchronized, so hip_handoff failed ("separate copy/ring timeout") on
both RX 9070 XT and R9700. A probe of the ring forms on both cards, graph and
stream launches, 100 rounds each: plain store 0/100 seen without a sync;
volatile store, plain store + trailing system fence, and a system-scope atomic
release store 100/100 (41-157 us), no stale payloads. The ring now uses the
volatile form doorbell_publish_kernel already uses. The ring kernel is the
--no-publish-kernel fallback; the default fused publish was already volatile.
qsa_prompt_attn_parity linked CUDA::cudart unconditionally, so
-DSTRATA_ENABLE_HIP=ON -DSTRATA_BUILD_TESTS=ON stopped at CMake's generate step
("CUDA::cudart ... target was not found"). The tool compares the tensor-core
kernel that HIP compiles out, so it is built with CUDA only.
- CMake: CMAKE_HIP_ARCHITECTURES may list gfx1100 and gfx1201 (validated) and
gfx1101 / gfx1102 / gfx1200 (build with a warning: same wave32, 64 KiB LDS and
dot4, not run by us; #192, #176). The list reaches the code as STRATA_HIP_ARCHS.
- Runtime: the engine (every GPU a run uses) and strata-device compare the card's
gcnArchName (up to the ':' feature suffix) with the compiled archs and require
wave32, with a message naming the card, its arch and the build's archs, instead
of a later "invalid device function".
- __dp4a uses __builtin_amdgcn_sudot4 (v_dot4_i32_iu8) on all five archs; the
portable fallback stays for other compilers/targets.
- setup.py: AMD_ARCHS gfx1100 + gfx1201; TheRock's wheel index per GPU family
(gfx110X-dgpu, gfx120X-all; the pinned 7.10.0a20251120 is on both), only that
family's libraries on the path, reinstall when the family changes; a system
ROCm older than 7.0 is passed over for the wheels; the engine is rebuilt when
the chosen card's arch is not in its BUILD.json. The hipBLASLt table is
tools/hip/<arch>-hipblaslt-<version>.txt for the card's arch AND the installed
hipBLASLt (read from hipblaslt-version.h, 1.4.1 -> 100401), else none with a
message; before, the highest-named table of the arch was taken whatever the
installed version, which the engine then refused.
conversation_session_sizes() gives one session's running-state sizes: the GDN bytes of its gdn_alloc
rows, the indexer tail/dead/block_pos per owned QSA state; an inconsistent carve is rejected.
Checkpoint save/validate/restore, the snapshot size estimate, capture, validation and restore iterate
the owned QSA ordinals [qsa_ord0, qsa_ord0 + qsa_alloc) instead of every model layer (the spare
pooled row is written back per owned layer only), and a parked snapshot records the carve it was
taken from (layer_lo, layer_hi): a restore into another range is refused before any write.
One GPU (the default full range: gdn_alloc = n_gdn_layers, qsa_ord0 = 0, qsa_alloc = n_qsa_layers)
saves and restores exactly the bytes it did before, in the same order. generate.cpp's
conv_state_sizes uses the same function. The tests' hand-built sessions carry the full-range carve,
and the host-only validation test checks the carved sizes and the rejections.
Conflicts in generate.cpp:
- against #216's session carve: conv_state_sizes takes the session (the carve's signature) and scales
the PR's whole-model GDN size to ss.gdn_alloc rows; checkpoint_save/checkpoint_restore are the PR's
(conversation_state.cpp), which the next commit makes carve-aware.
- STATE_HASH: the per-GDN-layer line over ss.gdn_alloc (guarded for zero rows), the QSA loop over the
owned ordinals [qsa_ord0, qsa_ord0 + qsa_alloc) with the PR's dead= and pooled_full= in it, and
end_cell from ss.max_cells (the primary state's max_cells, and valid with no QSA layer).
- INFO: f-resram's arena_mib (resident bytes under --mmap-experts) and the PR's three
conversation_cache_* fields.
The --conversation-cache-* options sit in the option parser's second chain (6fb466b); their value is
`number`, not `parsed`, which is that chain's flag.
doplxyz's minimal patch from #178 (their commit 0c59582): CMake, the
runtime device check, the signed dot4 intrinsic and setup's AMD_ARCHS
accept gfx1201 beside gfx1100.
On a PC whose VRAM cache plus RAM can hold all experts but whose RAM alone cannot (e.g. 32 GB + a 24 GB GPU with
Q2_0 / IQ2_XS / the Coder), the plain low-RAM mode (--mmap-experts) reads the non-VRAM experts through the OS file
cache, which such a PC keeps giving back to the SSD. The resident variant copies exactly those experts from
experts.bin into RAM once at start:
- engine: --resident-experts = --mmap-experts --resident-cpu-experts, the copy page-locked when the driver allows
(else locked in the working set), 4 GiB headroom, plain mmap with a warning when it does not fit. Built after the
prompt path's lend plan: the lendable slots' experts are kept in RAM too, from the last slot down, as far as RAM
allows. Adaptive swaps stay on: an evicted expert is copied back from its VRAM slot into the RAM place of the one
that replaces it (exchange buffers, committed when the swap lands), so no swap reads the file.
- --resident-cpu-experts (the ROCm arm) no longer requires --adapt-every 0 and gets the lend region too; its copy
stays pageable with 8 GiB headroom as before.
- the PCIe share is gated per source (pcie_layer): the arena answers as before; a page-locked resident copy answers
for every layer. Same bits as --mmap-experts for the same placement with --pcie-frac 0 or STRATA_RESIDENT_PIN=0.
- INFO arena_mib reports the RAM the experts take in the mmap modes (the resident copy, or 0); the serve log and the
CLI report the resident GiB, exchanges and blob reads from the file.
- setup: in the low-RAM mode, resident when RAM >= (experts minus the GPU's estimated share at this context, less the
image encoder's room) + 10 GB; --low-ram resident|mmap forces a variant; needs engine 0.1.30 (else mmap).
- tests: lend-region choice and exchange bookkeeping (CPU only). Default (arena) path unchanged.
Under sampling the draft layer proposed its argmax. With STRATA_SPEC_COUPLED=1 a sampled request's drafts are
sampled with the target's own chain (top_k / top_p / min_p / temperature, penalties over the same history) and the
same Philox uniform the verify window draws for the row that checks the draft: counter = MTP cell + 1 (a draft at
cell p + a + j is checked by row j of the next window, counter p' + j). Verification is still exact-match against
the target's sample, so the output does not depend on the drafts; acceptance should rise.
- sampler.cu: coupled_stage / coupled_penalize / coupled_merge kernels (params, seed, counter and penalty ring from
device memory, so they live in the drafter's CUDA graphs); the split part kernel reused unchanged; the tail
(sampled_tail_warp) templated to also return the pick's probability. The five existing sampler kernels compile to
identical SASS (sm_120).
- mtp.cpp: separate coupled round/step graphs (the argmax graphs are untouched and still used for greedy requests);
subset-aware (sample over the draft head's subset, id <-> subset maps for the penalties); --spec-min-p gates on the
draft's probability of its sampled token under the chain.
- generate.cpp: set_draft_sampling per request (serve and CLI), the penalty history base before each draft (serve).
- coupled_draft.hpp + coupled_draft_test.cpp: the counter / ring arithmetic and a host-only test (907k checks).
Switch off: no allocations, no new launches, the same graphs.
DETAILS.md: what it does, that no flag means no change, the contributors' paired per-position NLL (yarn 2 -0.18 nats vs none and linear 2 at 293K, no separation at 2.7K/32K, needles not discriminating), the explicit-factor pass-through, fixed per run. The stale 'engine 0.1.18' references go; the setup's 384K/512K menu entries and the README say experimental.
rope_scaling_invalid() checks every value with std::isfinite first (a NaN passes every range comparison), factor >= 1, original context > 0, a finite positive freq_scale and mscale; the engine applies it to the resolved config (CLI + model file) and fails at startup with the reason, and native_rope, both native indexer appends and the prefill rope use the same rule instead of three hand-copied lists. RopeScaling::kernel_args() gives the analytic kernels their float constants and returns the identity (1, 0, 0, 0, 1) for none whatever the other knobs hold, as the table builder already ignored them: an unscaled run cannot pick up a stray --yarn-attn-factor on the native path only. none resets the knobs (with a note if some were given). The magnitude-correction note prints whenever mscale != 1, and only then (not for plain linear).
indexer_key_append_kernel skipped the rotation at pos 0, which is only an identity while row 0 of the table is (1, 0). Under YaRN row 0 is (mscale, 0), so dead and pooled[0] stayed the only unscaled keys of a scaled cache and disagreed with the native indexer's spare. The rotation at angle 0 reduces to v * cos_tab[pair] on the rotated dims (the sine is 0 in every scaling), so no partner read and no barrier; unscaled it is v * 1.0f, bit for bit the old value. qsa_parity checks the table spare against the unscaled spare times row 0 (bit-exact), pooled[0] == dead, the native spare's factor, and that YaRN's mscale is observable there.
pooled= hashes rows [0, L/4) of every QSA layer's idx_pooled again, as in 0.1.29 and earlier, so a
state hash from this engine compares with an older one field by field. The PR's extent (those rows plus
the spare row at L/4, which a conversation restore writes back) is pooled_full=; dead= stays. The
cache gates compare pooled and pooled_full; the cache-off gate against an older engine compares pooled
directly and needs pooled_full only as proof that the candidate is this build.
A failed checkpoint save/restore prints its reason to stderr (the ERR line has none). The cache
harness reads its files as UTF-8 (Windows' default code page stopped it at the tokenizer), and the
isolation gate passes absolute fixture paths (the engine runs in the config's cwd).
Two things a layer split was paying for and not using.
* THE SESSION CARVE. A stage carved state for ALL 48 layers whatever slice it ran - every QSA KV pool at
max_cells, every GDN recurrence - the whole-model-sized-buffer disease the chunked-QSA-prefill PR cured
upstream. `session_bytes`/`session_init` take [layer_lo, layer_hi), and a session now owns only that range:
`qsa_states`/`gdn_state` keep GLOBAL ordinal indexing and simply hold fewer rows (`qsa_ord0`/`qsa_alloc`,
`gdn_ord0`/`gdn_alloc`), per-layer consumers subtract `gdn_ord0`, and the model-level uses - the shared RoPE
table, the prefill staging identity, the MTP rope borrow - read `qsa_states[qsa_primary()]`. A range with no
QSA layer still carves ONE primary state so those uses stay valid, and the default full range is
byte-identical to the old carve. On the 2x3060 + 2x5060 rig, CUDA1's [30, 48) session costs 0.93 GiB:
7.83 GiB free before the carve, 6.90 GiB after (`session [30, 48) and the head`).
* PER-STAGE PROMPT LOANS. `no_prefill_borrow = true` for split stages said "each stage's prompt path has its
own buffers" - true, but that is a reason to give each stage its own LOAN, not to make every stage withhold a
chunk-sized reserve from its cache for the whole session, and forced on it also collapsed `--prefill auto` to
2048. Each prompt path now borrows the tail of ITS OWN stage's cache (`PfPart`), so outside the prompt the
whole cache is expert cache: `--prefill auto` is the largest chunk every participant can lend (8192 tokens
here), a participant refills its loan into its own slots and marks only its own layers (a slot refilled into
another cache, or another stage's rows marked resident, is silent and produces plausible tokens), and the
chunk is the largest one EVERY participant can lend rather than one card's answer imposed on the others.
A participant that cannot lend the buffers falls back to the prompt path's own, as a lone one always could;
with a single participant the plan reduces to `plan_lend` exactly, so the single-GPU loan is unchanged from
main - the percentage cap stays an AUTO-chunk rule (an explicit `--prefill` is the operator's number and a
loan of it only has to leave the 128-slot floor). The flat 1 GiB withheld from every stage after the first
becomes what is actually held back after the weights, session and drafter exist: 96 MiB of verify windows,
plus 1000 MiB of drafter and head on the ONE stage that carries them. CUDA0 lends 2162 slots (3.98 GiB),
CUDA1 1993 of its 2653 (3.97 GiB).
* REPORTING. `expert_slots`/`expert_cache_mib` sum every tier: a 4-GPU 256K run reported CUDA0's cache alone
(3327 experts) while the four cards held 13310, so the Monitor tab was wrong by 4x for every multi-GPU
config. `expert_slots_primary`/`expert_cache_primary_mib` keep the old figure. The helper ranks'
ms_begin/ms_wait are cumulative since boot and were printed next to per-request deltas; they take deltas now.
KV-streaming stats iterate the session's OWNED ordinals.
* REFUSALS. `--gpu-only-full` and `--gpu-stages` replay every layer through CUDA0's session, which a split
carves to CUDA0's range, so they are refused with a layer split rather than answering about another stage's
state; the token-graph hit path is disabled under a split (such a graph cannot span stages); and the sessions
are allocated AFTER the split search, because the search prices each placement by what that range's session
will cost on that device (`session_bytes` is pure arithmetic).
Two gates stand between this engine and a 6.0 card, plus one attribute the card does
not honour. None of them needs the kernels rewritten:
* `CMakeLists.txt` refuses to configure below sm_75;
* `__dp4a` (6.1+) appears 29 times in the i-quant kernels (`native_mmvq.cu`,
`s2_expert_grouped.cu`, `iq_kernels.cu`) and `__nanosleep` (7.0+) in the three
doorbell waits (`elementwise.cu`, `verify_kernels.cu`);
* the Turing port sizes `fused_gr_read_multi`'s slice from
`cudaDevAttrMaxSharedMemoryPerBlockOptin`, which on an sm_60 answers 65536 - a card
whose launch validator enforces the 49152 B per-block limit instead. The capacity
comes out 6 tokens and every 6- or 8-token window fails the launch with
"invalid argument".
## The changes
* **`include/strata/kernels/dp4a.hpp` (new)** - `STRATA_DP4A` and `strata_spin_pause()`.
The dp4a fallback is llama.cpp's own (`ggml/src/ggml-cuda/common.cuh`): both operands
read as SIGNED bytes, accumulated into int32 - which is what `__dp4a` does for exactly
these call sites, the kernels here being transcribed from `vecdotq.cuh`/`mmvq.cu`. The
fallback is bit-exact, not approximate; the nanosleep sites only pace single-thread
doorbell waits, so on sm_6x the loop spins. Both are `__CUDA_ARCH__` selections, so a
HIP build takes neither: its `__dp4a` and `__nanosleep` are `hip_compat`'s own.
* **`CMakeLists.txt`** - `-DSTRATA_EXPERIMENTAL_SM60=ON` relaxes the architecture gate,
and the refusal message names it. The UTF-8 BOM stays.
* **`fused_gr.cu`** - below sm_70 the slice capacity comes from
`cudaDevAttrMaxSharedMemoryPerBlock` (49152 B -> 4 tokens) instead of the opt-in. The
Turing port's slicing, its no-opt-in fallback and the HIP expression are unchanged.
On sm_70 and newer this is a no-op: the opt-in is honoured there, so the capacity is the
value it already was, and `STRATA_DP4A` and `strata_spin_pause()` compile to `__dp4a` and
`__nanosleep` exactly as before. Building for a 6.0 card needs a CUDA release that still
targets Pascal (CUDA 13 removed it; 12.x has compute_60), and `CMAKE_CUDA_STANDARD` has
to stay 17 for it: nvcc 12.9 refuses `-std=c++20` when the host compiler is VS2019's `cl`
and silently falls back to C++14, which rejects the nested
`namespace strata::kernels {` declarations.
Rebased onto 0.1.27 (0.1.21 -> 0.1.24 -> 0.1.27). The 0.1.21 form of this patch clamped
the MTP and verify windows instead, because the fused GR read could not launch a 6-token
window at all. Now that `fused_gr_read_multi` slices the batch itself, the clamp, its
`fused_gr_max_window` helper and the two `generate.cpp` call sites are gone, and the full
window runs:
strata verify: window up to 6 tokens, 69.1 MiB of device buffers
strata verify: captured the 6-token window (upload no error, sync no error)
Measured on a 6.0 card (GP100-class, four dies through the layer split, the model's
IQ3_XXS quant, an 883-token prompt, 256 generated, the shipped `--spec 4`): every one of
the 24576 experts resident, 100% decode cache hit rate, 110 tok/s prefill and 21.3 tok/s
decode (warm median). This port on the 0.1.21 base measured 110.6 and 17.9 in the same
session with the same prompt, so the full window is worth about 19% of decode here. One
RTX 3090 Ti with the release engine measures 305 and 79 tok/s for the same model. The
port is about fitting, not speed: decode is bounded by Pascal's per-layer arithmetic, not
by capacity or memory. Only the 6.0 path was exercised on hardware.
Merged by hand onto 0.1.29: SIGTERM takes #153's Ctrl+C path (the timed sleep loop and the second-Ctrl+C kill stay);
SIGINT keeps Python's handler; the image builds RTX 20 (75) too.
Reserve accounted space for later appends without moving prior payloads. Prefer full capture when optional capacity would evict useful conversations, and report the actual additional RAM required. Exercise 4096 small appends and non-mutating admission checks under sanitizers.
Grow snapshot buffers in segments, track the first rewritten token, and charge retained active storage to the existing RAM budget. Refresh running state, partial pages and draft boundary state. Add full-capture versus incremental growth/rewind checks and report snapshot sizes with usable pressure-test budget ranges.
Replay identical requests and require token, known-answer, reuse, resolved-setting and comparable main-state parity. Retain the complete pooled-state diagnostic and add a completed-row fingerprint matching upstream. Add negative controls that remain active under optimized Python.
Move core diagnostics from the dependent NVMe work (4d73bb3) into the shared-core review branch. Read back authoritative bytes and resident ring pages without rewriting state; remove a redundant GPU fixture refill that could mask restore failures. Bound hash workspace and check synchronization/copy errors. Opt-in diagnostics record batched versus token draft prefill.
Validation: current core translation units pass C++ syntax checks; 1188 host transfer checks pass with ASan/UBSan. Real GPU and full-model validation remain pending the benchmark window.
The note fired for every scaled run inside the trained 262144 and its text
named YaRN's magnitude correction even for linear, whose resolved mscale
is exactly 1. Split in two: the in-window warning now only says that no
position needs the extension while the resolved scaling still applies to
every angle, and the magnitude explanation became its own note, printed
only when YaRN is resolved and its magnitude multiplier differs from 1
(yarn factor 1 inside the window stays quiet; linear never sees it). The
main rope scaling line is unchanged, the numbers it prints already carry
the resolved mscale.
Diagnostics only. Checked on the rebased tree: none prints nothing,
linear 2 prints the in-window note without the magnitude line, yarn 2
prints both (mscale 1.069315), yarn factor 1 prints the in-window note
only, and yarn 2 past the trained window prints the magnitude note
without the in-window one.
v0.1.29's new draft_kv site (the 0.1.26 batched draft prompt pass) kept
upstream's float freq_base form through the rebase, which both misses the
scaling on the draft K/V and does not compile against the RopeScaling
signature the other four sites carry. It passes rope_scaling() now, like
the prompt K, Q and indexer-query sites.
Item 8 verification. The parity section now runs all three scaling
configs - none, linear 2 (angle shrink only), yarn 2 (ramp + mscale) -
and "bit-identical" is asserted on the BIT representations (memcmp of
the state words), not float !=, which would pass a NaN-payload or -0.0
flip. Each scaled variant must also differ from the none baseline in
pooled-key bits, so the parity cannot pass because both sides ignore
the scaling. Green on this tree: 0 of 33409 state words differ in
every config; 128 (linear) and 192 (yarn) pooled words move.
The implementation landed with the derived-factor change; per the item's
own scope note this is verification and tests, not a second one. The
table from the item is now pinned case by case - 131072 and 262144 with
yarn and no factor resolve to 1, an explicit --rope-scale 2 stays 2 inside
the window, 393216/524288 keep 1.5/2 - and linear is checked under the
same rule the old fallback applied to it too. On the real engine the
resolved combo runs as "rope scaling yarn, factor 1 (freq_scale 1, mscale
1.000000)" and a greedy 12-token run is token-identical to the stock
baseline - evidence on one path, not a bit-identity claim.
The factor ladder (smallest of 1.5/2/3/4 covering the chosen context) is
gone. The automatic factor is now the final context over the trained
262144, clamped at 1 - max(1.0, final / trained) - computed AFTER the
RAM reductions, so it always matches the context the engine will serve:
512K + IQ3_S under 90 GB of RAM + yarn, no factor: final 128K, factor 1
no reduction: 384K -> 1.5, 512K -> 2
a chosen method inside the trained range: factor 1, not 2
Factor 1 removes the automatic expansion - the trained angles stand - but
it is not a switch for rope as a whole: an explicitly chosen method's
settings keep their defined behavior. A user-supplied --rope-scale is
still kept verbatim when a reduction changes the context; the ok() line
now says where the factor came from (final context / trained, or as
requested), and help, docstring and docs/DETAILS.md drop the
"smallest covering factor" wording.
The triple-check pass found what the menu could not: --context is not
limited to CONTEXTS, and the factor ladder (1.5, 2.0) ended at 512K, so
`--context 600000` died with a bare traceback instead of a setup message.
The ladder extends (3, 4) and a context past its top refuses cleanly,
naming --rope-scale as the way out. The interactive explanation block
also stays quiet under --yes, whose ok() line announces the choice
already.
The comment said the setup adds the flag when the chosen context needs it;
the code called fail(). `START-HERE.bat --setup --context 393216` - the
command the docs give - stopped instead of configuring. Now an omitted
--rope-scaling with a past-trained context resolves the config itself:
yarn (llama.cpp's extension method, recall-tested here at 512K; one
question when interactive, the silent default with --yes) and the smallest
covering factor (1.5 at 384K, 2 at 512K). An explicit none is still
refused for such a context, with the reason and the way out spelled out
instead of a bare stop; explicit linear/yarn/scale pass verbatim; inside
the trained 262144 nothing turns on by itself.
The resolution moved AFTER the RAM reduction (the §4 ordering): the factor
must cover the context the engine will serve, so a 384K choice reduced to
128K by the RAM limit gets no scaling at all, and an explicit selection
that survives the reduction says so in a warning.
The rope config is now two pure functions (covering_factor, resolve_rope)
with its own unit tests (tools/test_setup_rope.py); the engine args take
the resolved values; help, module docstring, the context menu note, README
and docs/DETAILS.md describe the implemented behavior.
The rebase onto 0.1.24 had brought the QSA indexer's C-2 chunk append,
whose kernels rotated the pooled keys inline - cosf/sinf of the unscaled
angle, no mscale - so a scaled prefill would have written indexer keys
under a different rotation than the per-token append the decode path uses
(rope_scaling.hpp's one-rotation rule, broken silently). The batch path
now resolves freq_scale/corr_dims/mscale from the same RopeScaling the
single append reads and rotates through the shared rope_scaled_angle; the
raw freq_base parameter is gone from its signature.
qsa_parity pins the contract the C-2 rework claimed: the batch append's
whole end state (pooled, spare, tail ring, block_pos) is BIT-IDENTICAL to
the sequential single appends', under none and yarn 2 alike, and the
scaling is observable in the batched keys.
The first end-to-end check that a scaled run recalls past the trained end:
8 of 8 probes found the needle at mid-depth, temperature 0 - 1k/32k/128k on
the stock model (arm A), 262k and 512k prompts on linear factor 2 (arm B)
and yarn factor 2 (arm C), 512k being 2.01x the trained 262,144. The yarn
262k probe was repeated green after the rebase onto 0.1.20, next to the
per-arm engine and server logs and the launch recipes.
What it does not establish is written down in the README: one depth, one
needle per length, and no long-document perplexity with and without
scaling at 262K yet - the one open measurement.
The defaults collided with the opt-outs: "none" was both the untouched
--rope-scaling default and the explicit switch-off, and 1.0 the same for
--rope-scale, so qwen4exp.rope.scaling.type/factor from the model file won
even against an explicit flag - the opposite of the stated precedence (and
of docs/DETAILS.md). Now the empty string and 0 mean the flag is absent
(the same sentinel idiom as --rope-freq-base 0 and --yarn-orig-ctx 0), the
model file fills only what no explicit flag named, and `none` / `1` are
the opt-outs they read as.
The resolution moves above the weight pipeline: the config needs only the
CLI and the model file's metadata, its one hard constraint (set before
session_init builds the table and the graphs capture the constants) is
unchanged, and a bad rope key now dies before the shards are scanned.
Checked on the bench machine against a synthetic GGUF carrying
rope.scaling.type yarn / factor 4: the flags absent -> yarn 4; --rope-scaling
none -> no scaling; --rope-scale 1 -> yarn, factor 1, mscale 1; explicit
linear 2 -> linear 2; a GGUF type other than none/linear/yarn dies before
the weight scan. The real artifact (no rope keys) starts and generates
unchanged, and rope_parity stays green.
DETAILS.md gains the 'Context extension' subsection - what linear and yarn do,
the setup path (383K/512K in the context menu pick the factor themselves), the
engine flags with llama.cpp's names and their defaults, why the scaling is fixed
for the whole run (K sits in the cache post-RoPE), and the one honest caveat:
inside the trained range a scaled run is a slightly different model, and the
recall numbers are text-only at one depth. Troubleshooting grows the row for
the refusal a context past 262,144 now raises without the flag; README points
the context question at it.
CONTEXTS grows 383K/512K, labelled in the menu, and the setup refuses a context
past the trained 262,144 without --rope-scaling (the engine would refuse it
too). The factor defaults to the smallest that covers the chosen context
(1.5 at 393K, 2 at 524K) and lands in the engine args like every other flag.
Nine flags, llama.cpp's names: --rope-scaling none|linear|yarn, --rope-scale,
--rope-freq-base, --rope-freq-scale, --yarn-orig-ctx, --yarn-ext-factor,
--yarn-attn-factor, --yarn-beta-fast, --yarn-beta-slow. Ranges die in the
validation block at second zero; the config itself resolves after the model file
has had its say and before session_init builds the table or captures a graph
(rope_scaling.hpp's one-writer rule). Precedence: CLI over the model file's
qwen4exp.rope.* keys over the struct defaults - today's artifact ships no rope
keys, so that channel is a no-op defaults path for future fine-tunes.
A scaled run prints its resolved summary (type, factor, freq_scale, base,
mscale, context vs trained) to stderr, which is what the bench logs quote, and
warns when scaling is on with the context still inside the trained range -
YaRN's magnitude correction applies everywhere, so that run is not the stock
model.
The table path got scaling for free (it lives in the table contents); the three
ANALYTIC paths computed the angle in-kernel and so need the config explicitly.
Each wrapper now takes the RopeScaling and passes the resolved constants -
freq_scale, the corr_dims pair, ext_factor, attn_factor - as kernel arguments.
They are process constants set before session_init, so baking them at CUDA-graph
capture is correct by the same argument as every other kernel argument: only
per-token values must ride device memory, and positions still do (st.pos_dev).
The shared float32 arithmetic is rope_scaled_angle (rope_scaling.hpp), a
line-for-line rope_yarn: one definition for native_rope, the native indexer's
pooled-key rotation and prefill's rope, instead of three transcriptions that
could drift. The spare indexer key keeps its zero angle; under YaRN the mscale
rides in through cos(0) on both pooled keys and indexer queries, so top-k is
unmoved. rope_parity adds the check this whole wiring answers to: native vs
table on the same input, yarn and none alike - which caught the first draft
passing scaling.mscale() into the helper that applies the log term itself, a
double magnitude correction visible as ~7% on every rotated dim.
The artifact pins theta 1e7 with no rope.scaling keys, so the engine had exactly
one rope: inv[i] = theta**(-2i/n_rot), no freq_scale, no correction. Contexts
past the trained 262,144 had no coherent path.
rope_scaling.hpp adds the one process config - none | linear | YaRN, llama.cpp's
types - and build_rope_table grows an overload that folds the whole scaled angle
(ggml's rope_yarn: interpolation, the corr_dims ramp along the pairs, and the
mscale magnitude correction) into the float64 host table. The rotation kernels
cannot tell a scaled table from an unscaled one, so nothing downstream changes:
for the default decode path and the indexer's pooled key, scaling IS the table.
The math is transcribed from the vendored ggml (ggml.c corr_dims, ggml-cuda/
rope.cu rope_yarn/ramp, MIT headers kept), not re-derived; the one deliberate
deviation is bookkeeping - the ramp takes the PAIR index ggml's i0/2 would
produce, because every Strata rope kernel speaks in pairs.
For type None the new builder calls the original loop verbatim, and the parity
test holds it bit-identical; linear(4) is checked bit-exact against a float64
reference and equal row-for-row to none at p/4; YaRN against a float64
transcription of the ggml spec, plus the structural checks a tolerance cannot
fake: cos_tab[0] == mscale (position 0 is angle 0 for every pair), first pair
extrapolates / last interpolates at a far position (cos for the first, sin for
the last - near zero sin IS the angle, which is where interpolation happens),
and an observability assertion that the fixtures see scaling at all.
The rows of a 2,184-token prompt are 27k random 4 KB O_DIRECT reads (114 MB for ~7 MB of rows), all before the
first kernel: 466 ms with the reader's 16 blocking threads. 64 threads (the port's default now, a copy of
src/platform/direct_file.cpp; STRATA_IO_THREADS still overrides) give 330 ms and 128 / 256 the same - the drive
tops out near 85k IOPS. A 256-token first chunk (STRATA_PREFILL_FIRST, 0: off) lets the GPU start while the
rest of the rows are read behind it: 2,184 tokens 3,075 -> 2,860 ms, time to first token ~150 ms less; 8,000
tokens 986 tok/s. The chunk boundary moves rounding in the expert GEMMs, so a long greedy continuation can
diverge late (token 45 of 64 on the test prompt); short prompts are bit-identical.
Measured on the B70, outputs identical before and after each change (docs/INTEL.md "Speed work"):
- every window graph and the drafter's graphs captured at load (STRATA_WARM_GRAPHS=0 restores first-use capture)
- the commit graph is left running while the drafter's round runs on its own queue: decode 38.3 -> 43.1 tok/s at
the 2,184-token context, 46.5 -> 49.7 on a short prompt
- the draft layer's batched prompt pass takes prompts under 64 rows (106 -> 16 ms on 19 tokens)
- one device module per kernel: the first launch in the prompt path 245 -> 1 ms
- the expert dequant writes each thread's run as one vector store: 0.085 -> 0.030 ms per expert; the 2,184-token
prompt 566 -> 682 tok/s, 8,000 tokens 720 -> 841
- SWAR sign compare/subtract in the expert dots, a local-memory resident-plan kernel, a split-K fused down kernel
(bit-identical to the single-token kernel, gr_parity): parity-clean, no measurable decode change
XMX: oneMKL's FP16 GEMMs already run on the XMX units (30-60 TFLOP/s). Two joint_matrix kernels are in the tree,
correct and opt-in because both lose to the paths they would replace on this card: xmx_gemm_iq (fused dequant +
GEMM from the quantized rows; xmx_gemm_bench) and qsa_prompt_attn_xmx (the mma.sync prompt attention's port;
qsa_prompt_attn_parity passes at 1e-6 of scale; STRATA_PROMPT_ATTN_XMX=1).
Profiling: unitrace -q needs Intel's metrics libraries (Dockerfile.metrics) and dev.xe.observation_paranoid=0;
the per-kernel table is in INTEL.md. A trap recorded there: --prefill-until with a native pack drops the prompt
tail (the token loop is skipped), so a "windows for short prompts" routing was invalid and is not in this commit.
The canonical path zeroes session state from put_input at position 0, but a
native pack breaks out of that loop before the first put_input and runs verify
windows instead, leaving cudaMalloc'd state uninitialized. On a reused
allocation that produced NaN/overflow and degenerate identical-token output.
Zero the state explicitly when native_pack is set.
Also removes an unused argmax() helper that emitted a compiler warning, and
ignores the local .pi/ harness directory.
IQ3_S and IQ3_XXS asked for 262K were cut to 128K on any machine under
90 GB of RAM. The limit is now counted: the experts' arena, plus the
context's 8-bit KV (in RAM with KV streaming, ~13.7 KB a token), plus
24 GB of room.
- A 64 GB PC still gets 128K for both models, as before.
- A box with ~84 GB keeps the 262K it asked for. A Colab A100-40G
(83.5 GiB) runs IQ3_S at 262K, with images, in 56 GiB.
Display both rates in t/s with independent graph scales and a darker
prefill curve. Expose prefill_tok_s_mean using the engine's existing
rate, excluding cached tokens.
Co-authored-by: Codex (gpt-6-astra-medium) <[email protected]>
Under WSL2 the NVIDIA driver can pin/map only about 1 GiB of host memory in
total (dmesg: "dxgk: create_existing_sysmem: establish_gpadl failed: -122").
With --expert-cache-device1/2 the startup ran out of it twice:
- CUDA2's context is created after CUDA0's weights and MTP, whose mapped host
buffers have spent the budget: "CUDA2 experts: cudaSetDevice(2) failed: out
of memory". STRATA_EARLY_REMOTE_CONTEXTS=1 now creates every secondary
context up front, like CUDA1's (opt-in: the default order is unchanged).
- With three contexts up, the 260 MiB mapped native embedding no longer fits:
"native embedding: cannot pin 260 MiB". It now falls back to a device copy
on the current GPU instead of failing the start.
Measured in Docker Desktop (WSL2), RTX 5070 Ti + 2x RTX 5060 Ti, Swift 1.5
IQ3_XXS: starts and serves with both helper caches (5000 experts each).
Closing the console window (or a crash, or Task Manager) ends the server without its cleanup, and strata --serve,
strata-vision and the MCP servers kept running with their VRAM and RAM. Each child now goes into a job object that
kills everything in it when its only handle - the server's - closes; what the children start joins it too. The
server itself stays out, so a browser --open starts is not tied to it. A no-op elsewhere.
serve/test_winjob.py: a contained child and its grandchild end when the process that contained them is killed.
The sampled path (penalties, top_k, top_p, min_p, temperature, draw) kept one
row on one SM: sampler_kernel excluded the logits already kept by sweeping the
kept list for every logit of every round, O(k^2 x n_vocab) shared-memory
compares per row, and ran the double-precision tail on all 1,024 threads.
Three implementations now pick the same token:
- Split (new default): stage 1 cuts each row into 4,096-logit blocks (61 per
row at 248,320 logits), each warp keeps the top_k of its 1,024 penalised
logits in registers, and the block merges its four warp lists. Stage 2, one
warp per row, merges the block lists and runs the tail on one warp.
- sampler_one_block_kernel: one block per row, top_k rounds that each take the
logits strictly after the previous pick in the selection order (value
descending, id ascending). It is the fallback when the split cannot run
(stream under graph capture, vocabulary above 262,144, more than 64 rows,
no scratch) and is selectable with STRATA_SAMPLER_ONE_BLOCK=1.
- sampler_kernel, the previous kernel, kept unchanged as the reference and
selectable with STRATA_OLD_SAMPLER=1.
The list, its order and the pick are unchanged: the first k of a union lie
within the first k of each part, and a merge of ordered lists is ordered. The
tail keeps the same arithmetic, with the exps and quotients shared across one
warp and the ordered sums and scans run by lane 0.
The block lists live in a per-stream scratch buffer grown on demand. A
grown-out buffer is retired rather than freed, and a failed grow is
remembered so later calls of that size do not retry cudaMalloc.
sampler_parity is the parity test. Fixtures 16 to 18 pin the whole top_k list
position by position under ties (+inf, -inf, NaN, +0/-0, sentinel id 0,
penalised tokens landing on other tokens' values), tied draws through the whole
chain, and the automatic fallbacks (graph capture, row cap). Each pick is an
integer token id compared for equality with a host reference. ctest runs the
binary once per path: sampler_parity (split), sampler_parity_one_block and
sampler_parity_old, so all three paths are held to the same expected ids.
hip_compat/cuda_runtime.h: map cudaStreamIsCapturing, cudaStreamCaptureStatus
and cudaStreamCaptureStatusNone to their hip equivalents, which the new
capture check needs on HIP builds.
24d4013 put the software prefetch in iq_avx512.cpp, so it only reaches CPUs with AVX-512; the AVX-2 kernels, which is what a Zen 3 / Haswell / Skylake-client box actually runs the i-quant rows on, had none. Same env, same default, same hint: STRATA_IQ_PREFETCH is the distance in bytes, 0 = off, default 2048.
Measured on a Ryzen 7 5700X3D (AVX-2 only, 64 GB DDR4-2666, 4 KB pages - large pages refused) with IQ3_S decode, 4 rounds interleaved, outputs identical:
gate/up phase 13.571 -> 13.024 ms/round (-4.0%)
rows 25.1 -> 26.1 GB/s (+1.0)
whole round 45.54 -> 44.94 ms/round (-1.3%)
and with the pool cold (first round after the arena is built) the round is -2.4%. The same 4 rounds with STRATA_IQ_PREFETCH=0 land within 0.1% of the unpatched binary, so the patch itself costs nothing. native_expert_parity: 0 failures with the prefetch on and off.
The non-temporal hint was measured worse than T0 at the same distance (-0.7% vs -1.3%, +0.5 vs +1.0 GB/s), so this keeps T0. 4096 is a little worse than 1024/2048, which is why the default stays 2048 as in the AVX-512 file.
When a request is cancelled during prompt reading (e.g. client disconnect
or timeout), err is set to 'cancelled'. Because err is declared in main()
and was never cleared between requests, any subsequent request in v0.1.26+
evaluated 'if (!e.empty())' in sp.on_chunk on its first chunk, immediately
failing with 'strata serve: cancelled' and terminating the engine with code 1.
Clearing err at the start of each request, before prompt segment runs,
and on entry to Prefill::run prevents state pollution across requests.
The merge of upstream main (99b4097) left 28 migrated copies behind. They are refreshed by re-migration:
dpct over the old and the merged tree, sycl/tools/normalize.sh (dpct's names, unchanged files, message
serials, fixups.py) on both, then a 3-way merge per file. Copies dpct never produced (verify.cpp, mtp.cpp)
take upstream's K8V4 diff by hand. The PTX gate fixup follows upstream's new __HIPCC__ guard (#elif).
The AOT build never received -cl-fp32-correctly-rounded-divide-sqrt (it went to the spir64 backend, which
an AOT build does not have): quantize_act_parity had its 303k mismatches back. The option now reaches
ocloc through -device "... -options ...". 19 of 22 parity tests pass again; kv_hybrid_parity (new) passes
all but its tensor-core step and is built, not registered.
B70, merged engine, AOT, --stream-experts --no-prefill-borrow --spec 4 --mtp: 2,184-token prompt 566.5
tok/s (was 475), decode 38.3 tok/s at that context (was 42.7; 77% draft acceptance on this text), MTP
prompt pass 38.6 ms (E-9 batched).
bw probe (incompressible data): 4-byte loads stream at 395 GB/s, 16-byte at 596. The Q6_K matvec sits at
130-160 GB/s in every variant tried (lanes per row, rows per warp, unroll, 16-byte loads, software
pipelining): latency-bound on a dependent load->unpack->dp4a chain, not ALU or bandwidth. The 16-byte
kernel is kept for n_out >= 4096 (+13% there, slower on small shapes); rel 5e-8 against the shared one.
Decode 20.80 tok/s (spec 2). The drafter is the lever now: MTP fetch + pack running.
The prompt path collects hundreds of blob pointers per layer and copies from them on other threads later;
the 8-slot ring handed back pointers that were overwritten by then, and its index was not thread-safe.
Slots are now keyed by (layer, expert), reused oldest-first among 512 (~1.4 GB), under a mutex.
Long prompts also hung in the prompt path's streamed-expert handshake (lent cache slots make experts
non-resident, the host supplies them behind a flag the GPU wait kernel does not see). --no-prefill-borrow
keeps every expert resident and removes the handshake: 2,000 tokens 693 tok/s, 2,185 tokens 560 tok/s,
300 tokens 180 tok/s (llama.cpp on the same card: 150 short, 424 at 105k).
Rig: watchdog reports and kills a hung run at the expected time (EXPECT_S/QUIET_S); --spec 2 default
(20.75 tok/s decode vs 16.95 at spec 4: the suffix drafter accepts 9%).
- exit segfault: dpct's static global_memory/constant_memory objects destructed after the default queue
(queue::get_device in the core); heap-allocated, never freed. Exit code 0; unitrace can write its table.
- Dockerfile.unitrace (pti-gpu unitrace in the dev image) + sycl/rank_kernels.py: per-kernel device time.
- Q6KTraits: the column-independent unpack (shifts, masks, saturating byte subtract) moved into load();
acts()/dot() split; a row-blocked kernel (native_mmvq_rowwarp_kernel, STRATA_MMVQ_RPW) kept but off:
with a 300 ms warm-up the shared kernel already does 2560x2560x4 in 48 us (110 GB/s) and the variants tie.
- mmvq_bench: the dense kernel timing harness (warm-up + 400 iterations; a 5 ms run measures the clock ramp).
route<NE> templated on the expert count; native_router_top10_multi_ne dispatches 256/512. Both call sites
(the verify window and the per-token moe_route) take it for the Coder. 64 tokens: 16.25 -> 16.95 tok/s,
same output. Also: Dockerfile.unitrace (per-kernel Level Zero profiling image).
A window routes the same expert for up to 6 tokens; the grouped kernels recomputed the dequantised weights
(grid lookups, sign unpacking, the byte-wise compare/xor/subtract emulations) once per token. Multi<TY>
splits each of the five formats this model uses (iq2_s, iq3_xxs, iq3_s gate/up; iq4_nl, q2_0 down) into
prep (once) and acts + finish (per token); the kernels take entries four at a time. Lanes per row is a
compile-time constant (STRATA_EXPERT_LANES, 8): 32 lanes on an 80-call row was mostly reduction.
Measured with the new NATIVE_BENCH=1 mode of native_expert_parity (10 experts x 4 entries, one layer):
iq3_xxs/iq4_nl 1.467 -> 0.649 ms, iq2_s/q2_0 ~0.68 -> 0.579 ms, iq3_s/iq4_nl 0.750 ms; parity unchanged.
strata::dp4a wraps dpct::dp4a (sycl::ext::oneapi::dot_acc is the same emulation with a header that breaks
the link across translation units).
The -Xs device flag was mis-quoted; fixed. Measured on the B70 with the same 19-token prompt, 32 tokens:
decode 15.17 tok/s, prefill 28.79 tok/s, a 4-token window 73 ms on the card. The persistent JIT cache
(SYCL_CACHE_PERSISTENT=1) is the alternative for the JIT build: 15 MB, second start 15.6 tok/s.
Measured on the B70: 64 tokens in 52.0 s, of which ~47 s is the first window (the runtime JIT-compiles
every kernel on first use: 6 tokens take 47.5 s, 64 take 52.0 s). The output is the model's real answer.
Debug tooling kept in the port: STRATA_VERIFY_EAGER=1 runs the window on the queue instead of as a graph
and turns the stage stamps into host clocks, so STRATA_VERIFY_PROFILE=1 prints ms per stage without a
device timer; STRATA_VERIFY_DEBUG=1 prints the residual after every layer and the round's phase times.
build.sh takes BUILD_DIR and AOT (an ahead-of-time bmg-g31 build removes the JIT cost).
--stream-experts (GgufExpertSource): every expert goes straight from the GGUF into the VRAM cache through a
small ring of staging buffers; no resident host arena, so the engine runs on a 23 GiB machine.
What it took to get from "loads" to "<think>\nThe user wants a Python function":
- the doorbell handshake through host-mapped memory: volatile device loads do not bypass the caches on
Intel; system-scope atomics (strata::sys_load/sys_store) do. A device-side spin is bounded (kSpinMax): an
unbounded one wedged the GT twice when its process died.
- STRATA_VERIFY_NO_HOST=1 + STRATA_VERIFY_DEVICE_PLAN=1: with every expert resident the GPU plans each
layer itself and the host waits for the whole window graph; per-layer ring visibility inside a graph is
not reliable on this platform (measured with sycl/probe).
- -fsycl-default-sub-group-size=32: the kernels are written for 32-lane warps; dpct pinned only 134 of 289.
- cudaMemcpy is synchronous; dpct's default-queue memcpy had no wait, and the streaming ring reuses its
buffers - the expert cache filled from overwritten memory (non-deterministic residuals, NaN by layer 5).
- native_expert_parity hand-ported (dpct cannot parse it without ggml-cpu.h): the GPU native expert
kernel matches the float reference on real IQ1_M rows.
- verify.cpp debug readbacks (STRATA_VERIFY_DEBUG): the per-layer residual ladder that found all of it.
Not yet: speed (0.17 tok/s - the spin budgets), the segfault at exit, the tensor-core kernels.
Rebased onto main (0.1.22): the portable pre-sm_80 QSA scorer and the sm_80
guard on the tensor-core prompt attention are already in main, so this is
the rest of the port.
- the runtime floor: device_info refuses anything older than compute
capability 7 (main's runtime check still requires sm_120, so a binary
built for sm_75 would never start),
- fused_gr: the down kernel carries the tokens in slices that fit the
card's shared-memory opt-in (Turing, 64 KB: 6-token slices; 4 tokens when
the card reports no opt-in, under the 48 KB default), merged on top of
main's per-device opt-in - full 8-token windows no longer fail on Turing
with "invalid argument",
- the setup/CMake floor is 7.5 (RTX 20 / 30 / 40 / 50) without an
experimental flag: CMake's arch refusals, setup.py's gpu_problem and the
GPU hints and error texts,
- docs/MULTI_GPU.md says what a Turing card actually runs.
Verified end-to-end on RTX 2070 (sm_75, 0.1.20 base): model loads, OpenAI/
Anthropic API serving works, greedy output identical to the sm_120 build.
Merged (PR #87) with two changes: the runtime floor is 7.5 exactly (major*10+minor < 75; the PR let Volta 7.0/7.2
through), kept beside main's HIP check; and a card reporting no shared-memory opt-in slices the tokens to what fits the
48 KB default for its own tile (4 tokens on CUDA, all 8 with HIP's smaller tile, so gfx1100 keeps one launch).
The tensor-core prompt-attention kernel is already refused at runtime on AMD
(line 695), but the file unconditionally included <math_constants.h> for
CUDART_INF_F, which is not present on a ROCm-only build. Replace it with the
portable INFINITY macro from <cmath> so the HIP build succeeds without a CUDA
toolkit installed.
The port lives in sycl/ and changes no upstream file: sycl/src and sycl/include hold only the files the
migration touched, sycl/CMakeLists.txt takes everything else from the original tree.
How it was made, reproducibly (sycl/tools/): a dev image (the llama.cpp SYCL image + SYCLomatic 2025.3 +
CUDA 12.8 headers from the pip wheels), migrate.sh (compilation database + dpct over 86 translation
units), fixups.py (every hand fix with its reason, idempotent), build.sh (icpx, ninja).
What dpct got wrong and the script repairs: CUDA's null stream became a null sycl::queue* (strata::q_of
maps it to the default queue); __ldg((const float*) p) lost its cast and read one byte; __fadd_rn lost
parentheses; ggml's lookup tables were threaded through kernel parameters with the wrong table per
template; helper headers renamed since 2025.3; cudaGraphUpload, graph introspection and %globaltimer
have no equivalents. Two compiler flags are load-bearing: -fp-model=precise, and the device compiler's
-cl-fp32-correctly-rounded-divide-sqrt - the Arc's fp32 divide is not correctly rounded by default, and
quantize_act_parity went from 303k mismatches to byte-exact with it.
Not ported: the three kernels with inline PTX (mma.sync, ldmatrix, cp.async) take their existing
pre-sm_80 fallback; the launchers refuse the device. The XMX joint_matrix versions are next.
The engine itself starts on the card, loads the pack, the projections, the PLE table and the 23.4 GiB
expert arena, and is then stopped by the test rig's memory guard: Strata keeps the experts as a resident
host copy, and the rig has 23 GiB of RAM (upstream asks for 32). A streaming fill of the VRAM cache
from the GGUF is the next piece; the whole model fits in the 32 GB card.
Prompt reading scales up with prompt size (150 tok/s at 2.7k, 424 tok/s at 105k), so the full
window costs about five minutes, not fifteen. 30.3 GB in use with 1.5 GB free at 131,072.
About 20 KiB of KV per token (only 12 of 48 layers are full attention). 32k, 64k, 96k and 128k all
load with the whole model on the card, 128k with 1.6 GB to spare; 160k is under the safety margin
and 256k does not fit - asking for it spilled VRAM into host RAM and took the machine down until its
watchdog reset it, so the doc says so in bold.
The generic prompt kernel now templates K and V element types separately, so
mode 3 is mode 1's INT8 K path (codes, scales, i8x2 MMA conversion) composed
with mode 0's fp16 V path: the row's q4_0 blocks dequantize straight into the
fp16 V row at gather. The caller's output Hadamard is unchanged. Pure Q4_0
pools still refuse (whole-format q4 stays on the old kernel).
The v2 cp.async int8 kernel (launch_i8) is untouched; mode 3 uses the generic
launch<>, as fp16 does. Parity's [5/5] flips from a refusal check to a numeric
check against the dequant reference (accuracy-level, not bitwise, like the
kernel's own int8 variant).
K stays INT8 (unrotated - the attention scores stay exactly int8's), V uses the
rotated q4_0 of kv_q4.hpp: 816 B per cell vs int8's 1,056. The K/V quantization
asymmetry is the point: K enters through the softmax scores, V only through a
softmax-weighted average (llama.cpp's q8_K/q4_V hybrids reason the same way).
No new kernels: the append/gather entry points already select K/V by blockIdx.z
and take separate K/V pointers, so the hybrid composes the existing validated
q8/q4 kernels with the unused half's lanes folded onto the used pool (a
bit-identical duplicate write). The fused attention gets KV_MODE 3 (K-side int8
loader, V-side q4); 0.1.22's tensor-core prompt path (qsa_prompt_attn) refuses
the hybrid pools through its !v_q check and the old kernel runs - covered by
the parity test. The MTP drafter stays INT8 (mtp.cpp toggles the format globals
around its state creation, any ring shape); qsa_kv_format exits on a hybrid
state rather than misreporting a layout (the kernels' own unsupported-geometry convention); the hybrid appends pass nullptr for
the host mirror (mode 0 has none). setup.py excludes k8v4 from the whole
auto-streaming decision (the engine refuses --kv-resident with it).
kv_hybrid_parity: bitwise append/gather through a non-identity page table,
step and batched mode-3 attention vs host references (<= 1.1e-07), and the
qsa_prompt_attn refusal - all at machine precision; kv_q8/kv_q4 parity unchanged.
Measured on a RTX 3090, Coder IQ1_M, 118.75k-token prompts, temp 0: decode
82-97 t/s (int8: 75; the band is draft-acceptance variance), prefill unchanged
~1,130-1,170 t/s, 26-needle seed 42 identical to int8's tokens, KV 1.39 GB at
131k (int8: 1.80). At --max-context 204800 on 24 GB: 1,024 t/s prefill,
82-91 t/s decode, 25/26 needles at 197.9k.
Without nvidia-smi, setup now looks for an Intel discrete GPU in sysfs (vendor 0x8086 under xe or
i915, named by PCI id - lspci's database does not know an Arc Pro B70 yet) and takes a path with no
compiler and no CUDA toolkit: download the model, export the tokenizer, find a llama-server
(--llama-server PATH, one on PATH, or the B70 container image via docker), write a config with a
"llama" block and a start-llama.sh, and a run script that uses --engine llama.
The CUDA engine's rules that do not apply are stepped around rather than deleted: the compute
capability and driver-version checks, the experts-in-RAM gates (on llama.cpp the fit is VRAM,
shard 1 = the download minus the 28.8 GB lookup table), the MTP draft download (5 GB the llama
engine cannot use), and the engine's own argument list.
engine_from_config runs the config's start script when nothing answers at the url, so the run
script is one click and closing the server stops the container.
Verified on an Arc Pro B70: setup runs steps 1-7 clean against the downloaded Coder IQ1_M.
docs/INTEL.md has the whole picture and the measurements.
The Strata engine is CUDA. On a GPU without it - written for and measured on an Intel Arc Pro
B70 - the same GGUF weights run through llama.cpp's SYCL backend. serve/engine_llama.py puts
llama-server behind Engine.generate(ids, max_new, sampling, cancel), so the OpenAI and
Anthropic endpoints, streaming, tool calls, MCP and the web app are unchanged above it.
It sends the prompt as token ids (the server's own tokenizer, from the same GGUF) with
return_tokens, streams ids back with llama-server's timings mapped onto Strata's `last`,
heartbeats while a long prompt is read, and stops the server by closing the connection when a
client goes away. Attach to a running server ("llama": {"url"}) or spawn one ("llama": {"exe"}).
Not yet: images (Strata's Vision produces embeddings for the CUDA engine; llama-server takes
the image itself) - start without "vision" and image requests are refused cleanly.
Tests: a scripted llama-server (the wire format observed on b29c606), plus a live round trip
when STRATA_LLAMA_URL is set. Measured on the B70 with Coder IQ1_M: 23-25 tok/s decode,
~150 tok/s prefill, correct code, GPU-bound.
The prompt path's selection at long context was 18% of a 128K prompt. Two changes:
* block scores as a GEMM on tensor cores: rows (query, indexer head) x blocks, K = 128, relu per head summed
in the warp kernel's order; 3xTF32 (hi*hi + hi*lo + lo*hi) keeps FP32-level accuracy; the tail block (the
`dead` key, +1e9) keeps the warp kernel's arithmetic. sm_80+ (TF32 MMA), shared-memory opt-in per device;
STRATA_SELECT_OLD=1 (or an older card): the warp kernel. Not bitwise (another summation order).
* top-k with each query's keys read once into registers (1,024 threads, up to 33 blocks each: contexts up to
~135K), per-warp histograms, block scans - the same selection rule, identical ids (exact; decode uses it
too). STRATA_TOPK_OLD=1: the original kernel.
5070, 256 queries (src/kernels/qsa_select_bench.cpp): 32K 1.18 -> 0.36 ms, 128K 4.68 -> 1.33 ms; synthetic selections
identical 256/256, the register top-k identical to the reference.
Quality gate (Q2_0, vs STRATA_SELECT_OLD=1 + STRATA_TOPK_OLD=1): needles 8K/16K/32K x 5 depths 15/15 with both;
16K continuation 32/32 identical, KL 0.005; 32K top-1 same, KL 0.03 (the size of any FP32-level change, cf. D-1).
Prompt: 32K 1,742 -> 1,797 tok/s (select 897 -> 365 ms), 128K 1,608 -> 1,843 tok/s (select 14.6 -> 4.2 s),
time to first token 82.7 -> 72.3 s.
It says what happens (the experts must stay in RAM, so the OS pages part of them: much slower, maybe no start) and
asks, "no" by default, so an unattended --yes install still stops. Thanks @FatheredPuma81.
The opt-in was set once per process, but it is a per-device setting. The int8 kernel fits the default 48 KB; the
fp16 one (KV of contexts up to 8K) needs 54 KB, so on the second card of a split its launch failed.
Resolved against 0.1.22: E-6's device-plan waits and copies kept (the row copy of the CPU share is used when the
device plan is off - bit-identical, +0.0 on the GPU rows as the pool leaves them); the multi-GPU per-device
shared-memory opt-in and stage pool callback kept. Left out: the session-wide --pcie-frac 0 skip of the PCIe path,
because a request can ask for a PCIe share (pcie_frac=, used by --calibrate) and the window graph is captured once.
--prefill auto sized the chunk only when there were cache slots to lend; with none the buffers stayed at 8,192
tokens and a full card stopped at start with "device buffers ... do not fit". Thanks @sinand99.
The server sent only cvec= with GENI and the engine skipped every other key, so a request with an image decoded
greedily whatever the client asked. Both now pass the text path's keys. Defaults unchanged (greedy unless set).
Thanks @jleaders for pinning it down.
Its MMA and cp.async need sm_80: an sm_75 build stopped with ptxas errors. They now compile to a trap below
sm_80, and qsa_prompt_attn_batch refuses such a device at run time, so the old kernel runs there. RTX 30/40/50:
the same code (parity unchanged: vs FP64 3.17e-6).
The expert-arena load runs at ~0.05 GiB/s (13-14 min to ready) when the serve
is started from Task Scheduler with its registration defaults (Priority 7 /
Below normal, least-privilege token), versus 1.4-1.5 GiB/s (~35 s) from a
terminal or SSH - same binary, args and cache state.
- docs/DETAILS.md: measured table + the two registration settings that fix it
- src/program/generate.cpp: when load_gib_per_second() < 0.2 GiB/s, print a
hint naming this cause under the load line
Expose loaded-model metadata through /models and /v1/models.
Add /props and single-slot /slots responses with existing API-key
enforcement and regression coverage.
Validation: all 30 server tests pass. Pi 0.87.1 discovered the loaded
model with 262144-token context and thinking support, and completed
a chat smoke test successfully.
Fixes#117.
Co-authored-by: Codex (gpt-6-astra-medium) <[email protected]>
The GPU, VRAM, temperature and power cards read all the cards the model runs on: their total (VRAM, power, PCIe
traffic), mean (load) or hottest (temperature), with each card's own reading underneath; the GPU name lists them.
One card: unchanged.
The same copies on the same (legacy) stream, but queued instead of waited on one by one: the same bytes, ordered the
same against earlier work. Q2_0 on a 5070 (1,782 slots): 162 -> 96 ms per prompt, 4K and 32K outputs identical.
The rest (publishing slots in the background while decode starts) would hide the last ~2% of the time to first
token on a 4K prompt; not done. STRATA_REFILL_BLOCKING=1 keeps the old loop.
D-3 measurement: Q2_0, 128K prompt with KV streaming, 5070: staging 1,331 ms of 81.3 s (1.6%); staging only the
selected union could save part of that, so D-3 is not done. The same prompt: 1,144 tok/s on 0.1.21, 1,596 on this
branch (+39.5%), time to first token 115.7 -> 83.4 s.
vswhere -latest picked Visual Studio 2026 (version 18) on a PC that also has 2022, and CUDA 13.1's nvcc stops with
"unsupported Microsoft Visual Studio version". Now only 2019/2022 are looked for; with only a newer one, the 2022
Build Tools are installed as when none is there.
Several users could pick one GPU but not two. Now START-HERE.bat / setup.sh:
* lists every NVIDIA card with "can be used" or why not (older than the RTX 30 series; under 8 GB to share a model);
* asks, when two or more cards can share the model, which to use - the two best together recommended (newest
generation first), every combination and each single card offered; --yes takes the recommendation;
* asks once, at the next start of a model installed on one card, whether to use both from now on (kept);
* --gpus 0,2 / --gpus all / --gpu 0,2 at a start saves the cards for that model; --gpu N is for that start only;
* a card that cannot be used (or is gone) stops with its reason and the command that works instead.
The engine gets code for every chosen card (a local build compiles all their generations; the ready-made one is
checked against each), the context is recommended from the smallest card, and --calibrate measures the split.
One block per (query, KV head) walks its selected cells in chunks of 32 with an online softmax (no split-K
scratch, no merge kernel). q.k and p.v are FP16 MMAs with FP32 accumulation; int8 codes enter exactly, their
scales in FP32; q and p are split into hi + lo halves after an exact power-of-two prescale, so the result keeps
FP32-level accuracy. Int8 KV: each warp owns one 64-dim scale group and gathers its own K/V slices with
double-buffered cp.async. FP16 KV: the first version. Q4_0 KV, or STRATA_PROMPT_ATTN_OLD=1: the old kernel.
qsa_prompt_attn_parity (5070, synthetic): vs FP64 3.2e-6 (old kernel 2.5e-6); int8 4.0x faster (26.6 -> 6.6 ms per
2,048-query chunk at 32K), fp16 1.45x. Q2_0, 32K prompt: attention 5,216 -> 1,318 ms, prompt 1,386 -> 1,646 tok/s
(+18.8%). Not bitwise: the model amplifies any FP32-level change (v1 vs v2 of this kernel, same accuracy, another
summation order, differ as much as old vs new: residual rel L2 mean 0.060 vs 0.061). Needles 8K/16K/32K x 5 depths:
15/15 with both kernels.
Also STRATA_SEL_OVERLAP (debug): how many of the cells 16-query tiles read compared to one query at a time.
A user reported that he could pick a GPU but not two. With v0.1.21 (reproduced on a 5080 + 2080 Ti + 3090 PC):
* START-HERE.bat --gpus 0,2 on an installed model started it on GPU 0 alone: --gpus was saved only with --setup,
and the start passed its first card as --gpu, which overrides the saved choice;
* START-HERE.bat --gpu 0,2 stopped with "argument --gpu: invalid int value";
* an update unzipped next to a layer-split install stopped at step 1: "there is no GPU [0, 2]".
Now the first two start across both cards (the server gets --gpu 0,2 and splits the layers, auto), and an update
sets itself up across the same cards with the saved layer split. --gpu N still runs one card.
Two gaps kept the routing trace one-shot-only:
- drive_pool_multi, the verify-window dispatch the serve path uses for every
decode call, had no trace hook; --dump-routing under --serve opened the file
and wrote nothing. Records now go out per token in drive_pool's format
(layer, k, ids, weights). The multi dispatch fuses the router weights into
the kernel and does not surface them, so records carry unit weights:
make_profile.py ranks pairs by routed frequency, which is the signal that
matters; a one-shot run still records true weights.
- the trace sat in a stdio buffer until teardown, so a crash lost the whole
session's data. It now flushes at every request's DONE.
Verified live: a serve-mode trace grows per request, parses cleanly in
make_profile.py's format (all 48 layers, records per decode position), and
captures real multi-session traffic for expert-profile building.
One 512-thread block per head (48 blocks) walked the chunk. The value columns' recurrences are
independent; only the per-token output norm couples a head's columns. Now 4 blocks of 32 columns per
head (192 blocks, 128 threads per __syncthreads), and a parallel norm kernel with the same warp sums.
Per column the same arithmetic in the same order: residual dumps and continuations byte-identical to
0.1.21 (4K, 20K). Q2_0 32K, back to back: GDN 5118 -> 4264 ms, prompt 1258 -> 1308 tok/s.
STRATA_GDN_REC_HEADS=1: the per-head kernel. (A chunked delta-rule kernel - not bitwise - is the
larger step left.)
After routing, one thread checks the group's entries against the device residency table; when all
are resident it writes the plan the host pool would (same layout, same order) and marks the ring.
The layer's waits on the host (plan, PCIe share, CPU share) then return at once and the CPU rows
are zeros; the host still runs its loop, off the GPU's critical path. STRATA_VERIFY_DEVICE_PLAN=0: off.
A group of <= --spec rows was staged in mapped memory and synced before the next: ~5,500 host round
trips on a 32K prompt. The rows' token / step / position records are now uploaded once per call and
each group is device copies plus a graph on the one stream, one sync at the end (a graph variant
without the mapped staging). The same work in the same order: continuations byte-identical to 0.1.21
(4K, 20K). Q2_0 32K: the draft layer's share 1091 -> 905 ms; prompt 1213 (Phase B) -> 1387 tok/s with
Phase C and D-5. STRATA_MTP_PREFILL_SYNC=1 keeps the old loop.
Agent clients put screenshots in tool results; flattening non-user messages to text made the model blind to them. The template and images_of already handle them; only the conversion was too strict.
The thread that launches a chunk's kernels also issued the expert copies, and waited there for
the host copies of the unpinned blobs (the arena part Windows would not pin): the GPU idled - the
'wait copy' and 'dequant' time of the i-quant prompts. A per-chunk issuer thread now walks the
stream: the same copies in the same order into the same slots, a slot refilled only after the
compute stream recorded it free. STRATA_PREFILL_ISSUER=0 keeps the inline issue.
IQ3_S 32K prompt: 1143 -> 1213 tok/s, 'wait copy' 15.2% -> 7.5%. Exact: residual dumps and
continuations byte-identical to 0.1.21 at 20K (IQ3_S, IQ3_XXS: +16% prompt tok/s with Phase C).
The stager's ring and threads are tunable (STRATA_STAGER_RING / _THREADS); the defaults stay
(more copy threads were slower: 816 tok/s at 6).
Measured on a Ryzen 5 7600 (IQ3_S, 4K context, CLI decode timeline, outputs identical):
gate/up 37.1 -> 36.5 ms/round with a 2 KB prefetch; the gather decode (same results) is 3-5%
slower than the scalar lookups, so it is opt-in (STRATA_IQ_GATHER=1). More pool threads (SMT)
do not help: the pool reads its rows at ~30 GB/s, near DDR5's practical rate through 4 KB pages
- memory-bound, so decode speed now comes from VRAM hits (E-1) rather than CPU kernels.
The 4-tap conv reads inputs, not its own outputs, so 64-token tiles are independent: 160x the
threads of the per-channel walk. Same per-element expression (identical bits: residual dumps and
continuations byte-identical to 0.1.21, also with 1001-token chunks). STRATA_GDN_CONV_SERIAL=1: the walk.
- C-1: the block-score grid spans the batch's largest n_bid + 1 blocks instead of the context's
capacity (the blocks past a query's n_bid returned at once). The graphed decode paths keep the
capacity grid.
- C-2: the indexer's appends of a chunk in three launches (the first cell's spare, one CTA per
completed block, the tail) instead of one launch per token and QSA layer (24K per 2048-token chunk).
The end state is the sequential one's: the same arithmetic per block, only the batch's last block
writes the spare after it. STRATA_INDEXER_PER_TOKEN=1 keeps the old loop.
Exact: residual dumps and continuations byte-identical to 0.1.21 (4K, 20K, and 5K read in chunks of
1001 so blocks straddle chunks). Coder prompt tok/s with C-4: 4K 1063 -> 1255 (+18%), 20K 1216 -> 1373 (+13%).
The conversation cache's reuse is kept per request (GET /metrics shows it),
but the APIs did not report it, so clients showed a cached prompt as 0%:
- OpenAI: usage.prompt_tokens_details.cached_tokens, streamed and not.
- Anthropic: the final usage splits input_tokens from
cache_read_input_tokens (message_start can only know the whole prompt).
- llama.cpp's `timings` block on the last chunk and on a whole reply
(prompt_n, cache_n, prompt/predicted ms and per second), from the
engine's own clock: what llama.cpp's clients already show as speed.
- GET /v1/status (with the API key when one is set): the model and its
window, images, the APIs, whether a request is running, the last
request's timings and the GPU and RAM from the telemetry, for a
front-end that polls its server rather than guessing.
An engine without a clock (MockEngine) reports no timings and 0 cached.
Tests: serve/test_server.py UsageAndStatus.
live.tok_s returned `generated / (now - first_token)`: a cumulative mean whose
first sample is 1/elapsed. Against an engine paced at 25 tokens/s the Speed
card, the header pill and the tok_s sparkline read 27,235.7 tok/s at that first
sample - about 1090x the real rate - and then undershot for the first second of
every answer (one token counted over up to a second of elapsed time). Because
the sparkline scales to the maximum of its 60-sample ring, one such sample also
flattened the Speed graph for a minute.
tok_s is now the rate over the last RATE_WINDOW_S (2 s), with the mean so far
reported only while a request is younger than RATE_MIN_SPAN_S (0.25 s) and its
span floored there, so no sample can diverge. The old formula is kept as
_tok_s_mean() and still reported (live.tok_s_mean, status.tokens_per_s_mean).
The same request fixes a second mis-attribution: a request that ended without a
DONE line (engine death, error, client disconnect) recorded the PREVIOUS
request's generated/decode_ms as its own decode_tok_s. The engine's counters
are now trusted only when its DONE replaced the object captured at request
start, and the row carries the engine's own token count as engine_generated.
Tests: serve/test_server.py::LiveRate paces a mock engine and polls GET /metrics
at 10 ms. Both cases fail without the server.py change (990.0 tok/s recorded
for a request that never finished) and pass with it; the rest of the suite is
unchanged at 28 tests, OK. The engine's own per-request counters - what the
Speed card shows for "last request" - are untouched, and the server's progress
and summary log lines keep their whole-request mean on purpose.
- One batched gather per chunk instead of a launch per token (same arithmetic: identical bits).
- The prompt path's 'embed+steps' time was mostly the host reading the n-gram (PLE) rows from the SSD
with the GPU idle: one thread issued every 90-byte read (~12 us of kernel time each, ~80K reads/s).
DirectFile now only queues a read; a pool issues them (4 threads on Windows, overlapped; 16 blocking
O_DIRECT preads on Linux, which read them one at a time before - perf-review F-2). STRATA_IO_THREADS.
Q2_0 4K prompt: PLE 591 -> 291 ms, 941 -> 1010 tok/s.
- STRATA_PREFILL_TIMING also prints the host's share (chunk setup, the per-chunk callback).
Exactness: the prompt path's final residuals (STRATA_PREFILL_DUMP_R) and the continuation are
byte-identical to 0.1.21 at 4K and 20K tokens, IQ and S2 embedding tables.
Each stage saves its own part of a checkpoint when it has read the chunk that reaches the
position (Prefill::on_stage_chunk, its state synchronized at that chunk's end), by the same
rule the last stage uses; the last stage puts the parts together in on_chunk. They were off
across GPUs because the first stage is a chunk ahead when the last one reports.
- the image-position (mrope) table and the control vector's tables are per device: every
stage's rope and steering kernels read their own device's copy; picture rows go through
the split prompt path (stage 0 embeds them, the later stages take the hand-off)
- KV streaming: each stage's session already streams its own pinned KV copy; no longer forced off
- --mmap-experts: allowed (no pinned arena: every miss the CPU computes, prompts copy on the host)
- --expert-cache-remote with a split: the helpers run on the visible GPUs no stage runs on and
hold only experts no stage's cache holds
It re-decoded every generated id per token: 2 ms per token after 8K tokens, 4 ms after 16K.
Now 0.0005 ms at any length; the text is the same (serve/test_detok.py checks it against the
old algorithm on real and random token streams), and a complete character is no longer held
back behind an incomplete one.
Fitted on the 5080 + 3090 K sweep (18-32): it picks K=26, the measured best (prompt +18-20%
over the 5080 alone), and gives a slow third card (2080 Ti) a single layer.
The server sets CUDA_VISIBLE_DEVICES to the list and adds --layer-split ("layer_split" in the
config, auto by default); the Monitor reads the first (main) card.
build writes source.shard1_sha256 / shard2_sha256 into manifest.json (unless --skip-hash, which is how
setup builds), and nothing read them back: src/, serve/, setup.py, tools/ have 0 readers. A pack
verified against a different shard than it was built from reported tensor MISMATCHes, or passed under
--limit.
A full verify (no --limit) now hashes the shard it is handed and fails naming both digests when they
differ; a manifest without the field, or a --limit run, is unchanged. Shard 2's hash is recorded but not
checked here: verify is not handed that file. tools/test_shards.py: a matching hash passes, a wrong one
fails naming both, an absent hash or --limit skips. On main the wrong-hash case returns 0.
find() is first-match over the tensor directory, so a GGUF that carries one name twice resolves it to
whichever entry came first; the other tensor's bytes are never read and nothing says so. GGUF has no
index that could arbitrate between the two, so refusal is the whole fix: parse() keeps the names it has
seen and throws on the second sighting ("GGUF: duplicate tensor name 'blk.0.attn_q.weight' in <path>"),
as llama.cpp's gguf_init_from_file does.
gguf_reader_test (STRATA_BUILD_TESTS; no model, no GPU) writes a two-tensor GGUF v3 by hand: distinct
names open and find() sees both; the same name twice is refused and the error names the tensor and the
file. On main the second file opens: 3 of 4 checks fail.
setup checked a shard by presence only, and a download by Content-Length only: a shard copied short into
--gguf-dir, a .part renamed by hand or a download an older setup finished without a mark passed as a
model file, and the failure came later from the packer or the engine, at the first tensor past the end,
naming neither the file nor the shortfall.
check_shards() reads each shard's header (not its data) and compares the file's length with what its own
tensor directory needs: "m-00001-of-00002.gguf is short: 1,234 of 1,274 bytes (40 missing)"; a header
that does not parse is refused by name; a missing file stays "missing <path>". The download's "could not
finish" line now says the bytes on disk and the bytes the server announced.
tools/test_shards.py (unittest over a minimal GGUF written by hand; no download): whole shards pass; a
missing, a short and a header-truncated shard each stop setup with the file named.
--layer-split K1[,K2..]|auto with --split-device D1[,D2..]: each later stage runs its layer
range on its own GPU (weights, session, cache, verify window, prompt path). The prompt path
hands each chunk's rows to the next stage, which reads it on a thread while the previous
stage reads the next chunk. Checkpoints carry every stage's session state; adaptive swaps
copy into the layer's own cache; each GPU probes its own PCIe share. "auto" scores every
placement (up to 3 GPUs) by the profile-weighted pairs each cache would hold.
STRATA_EXPERIMENTAL_SM75: lets CMake build for a Turing card (the 2080 Ti A/B).
--split-device D (default: the second visible GPU) runs layers [K, n) on device D with
its own copy of the dense weights, session, head, drafter and an expert cache filled
from the profile's pairs of its layers. Verifier and MtpDrafter switch to the device
they were made on (OnDevice), the pool wrapper swaps the stage's cache pointers, the
penalty history lives with the head, and the s2 LUT is per device.
v1 switches off what does not cross devices yet: checkpoints, adaptive swaps, the PCIe
share and the batched prompt path (prompts read through the verify windows).
--layer-split K runs layers [0, K) and [K, n_layers) as two verify stages on the same device (sharing weights,
session and expert cache). The hand-off carries R, the last layer's pending write and inject, so the next stage
folds it in exactly as the unsplit window does. The GPU shared-memory opt-ins are now per device.
Each layer waits for the helper GPU on the CPU pool's critical path. Measured on RIBPC (Windows): ~0.3 ms
per round trip, more than the fast CPU needs for the same experts. The helper's context is created with
cudaDeviceScheduleSpin (STRATA_REMOTE_SPIN=0: the default), and its input/compact-output staging is mapped:
the kernels read and write it directly, two copies fewer per layer (STRATA_REMOTE_ZEROCOPY=0: copies).
The drafter loader had the same defect as the expert arena: read_file() pulled the whole
dense.bin through std::ifstream, and the 512-expert loop read 64 MiB chunks through the
same 4095-byte-per-call path. Same fix (fopen/fread, _fseeki64/_ftelli64, short-read
check), so the drafter stage stops paying per-read overhead.
On MSVC, std::ifstream::read does not issue one fread per call: basic_filebuf::xsgetn
loops over fread(_Ptr, 1, 4095, _Myfile) (__msvc_filebuf.hpp, _Read_size = 4095).
A 44.7 GB expert arena therefore costs ~11.4 million 4095-byte reads instead of a few
thousand bulk reads, and the load is dominated by per-call overhead.
Read the arena with fopen/fread, an explicit 64-bit seek (_fseeki64) and a short-read
check that fails the load (LoadStats.ok/error) instead of silently returning partial
data. Behaviour on Linux/glibc is unchanged in substance: same bytes, same call sites.
The page referenced its stylesheet, script, sprite, fonts and every API call
with absolute paths (/web/app.css, fetch("/health"), url("/fonts/...")).
Behind a reverse proxy that mounts the app under a sub-path - llama-swap's
/upstream/<model>/, an nginx subpath - those resolve to the proxy root and
404: an unstyled page stuck at Connecting, or a Monitor that polls /metrics
at the proxy root and reports 'Server not reachable'.
Make every same-origin reference relative:
- index.html: (href|src)="/web/... -> web/...
- app.js: fetch("/... -> fetch("..., plus the endpoint strings used outside
a plain fetch(" call - the Monitor's ternary fetch(reqShowAll ?
"/metrics?requests=all" : "/metrics") and the SPRITE constant - because
the quote does not always sit right after 'fetch('
- tokens.css: url("/fonts/... -> url("../fonts/... (fonts have their own
top-level /fonts/ route, one level above the CSS)
Standalone at / resolves the same, so nothing changes there.
Verified through llama-swap's /upstream/<model-id>/ prefix in a browser:
page, stylesheets, fonts, sprite icons, /health, /settings, /metrics, /mcp
and a streamed /v1/chat/completions POST all 200; the Monitor tab connects
and streams live telemetry. Same checks pass at / standalone.
The CMake arch guard already lets RTX 30 (sm_86) / 40 (sm_89) build - its own
comment says the kernels need sm_80 or newer (tf32 mma, bf16 math) - and the
README lists RTX 30/40/50 as supported. But device_info() still refused
anything with cc_major != 12, so a working Ampere/Ada build died at startup
with 'Strata targets sm_120 only'. Relax the runtime check to the kernels'
real floor (compute capability 8.0) and align the stale 120-only comments.
Verified on RTX A2000 12GB + RTX 3080 10GB (both sm_86), CUDA 12.0, gcc-13,
engine 0.1.20 serving daily: TG 21-24 t/s flat 1k-128k, 29.6 t/s code-gen at
81% draft acceptance, PP 375/445 @8k/32k, 8/8 needle retrieval at 113k.
A prompt read from token 0 is also checkpointed at its first turn boundary when that is 2,048 tokens or
more (--prompt-cache-root N, 0 = off). It becomes the chain's root, which #62's retention policy pins,
so a new chat of the same client reads only what follows the system prompt and tools. Without it, #62's
root existed only for system prompts past --prompt-cache-every (16K).
Co-Authored-By: code-martin <[email protected]>
The gate/up row and IQ4_NL down comparisons printed their distance from ggml's vec_dot without affecting the exit code, so a kernel that silently diverged still reported 0 failures. Gate both at 1e-5 (measured 3e-8 and 1e-7, so 100x headroom). Verified: 0 failures at 1e-5, and with the threshold dropped to 1e-9 the gate fires on every layer.
Also corrects the comment above the gate/up dispatch: STRATA_NO_IQ512 drops an AVX-512 CPU to the AVX-2 kernel, not to ggml; ggml's vec_dot needs both switches set.
- Reset cache hits/lookups baseline at start of autoregressive decode to exclude prefill distortion
- Append req_hits and req_look to engine DONE line and log decode hit rate to stderr
- Parse decode hit metrics in server.py and record hit_rate per request in history
- Add Hit rate column to Recent requests table in web monitor
A small MCP client (JSON-RPC 2.0, stdio and Streamable HTTP, standard library
only): servers from the run config ("mcp_servers", or Claude Desktop's
"mcpServers" block) or --mcp-config start with the server; their tools are
offered as <server>__<tool>. A request that sends "strata_mcp": true (the web
app does) gets them: the server runs the calls, feeds the results back and lets
the model continue, up to mcp.max_rounds; the stream carries strata_mcp events.
Other clients are unchanged. Errors, crashes and timeouts become "error: ..."
results; long results are cut at mcp.max_result_chars; Stop cancels a running
tool. GET /mcp lists servers and tools; the web app shows each call as a
collapsible block, a switch in the Sampling drawer and a Monitor card.
The mock engine takes --script more than once (answers in turn).
tools/calibrate.py starts the engine through serve's StrataEngine and measures decode speed with the
three settings that depend on the machine: --pcie-frac and --spec-min-p through new per-request GEN
keys (pcie_frac=, spec_min_p=; the server forwards a request's "strata_tune"), then 2-3 worker counts
(a restart each). A setting is kept only when it beats the product default by more than 3% in an
interleaved re-measurement. Setup offers it after an interactive install and runs it with --calibrate;
the result is saved per PC + model + context in the settings file and re-applied when a config is
written, so updates keep it. The engine's INFO line now says pool_workers, pcie_frac and spec_min_p.
tools/test_calibrate.py tests it with a stand-in engine (no GPU).
On Windows every allocation on the graphics card is also charged to the system commit (RAM + page file).
With the page file off or tiny, the expert cache's one big cudaMalloc failed while the VRAM was free and
the engine stopped. An auto-sized cache now retries at three quarters of the size (up to 8 times) and
says how much commit Windows has left; a failure that remains names the page-file setting. Setup warns
when the page file is under 4 GB; DETAILS has a troubleshooting row. STRATA_TEST_CACHE_FAIL=N fakes N
failed opens to test the retry.
- serve: a verify window samples T rows, but only row 0's penalty history was staged; rows 1..T-1 read
unwritten slots (past the allocation at penalty_last_n=4096). Every row now gets its own history
(strata::kernels::penalty_rows: the consumed tail, the fed-back token, the drafts before that row).
- prompt windows (their picks are discarded) skip the head sampling: Verifier::set_head_sampling.
- sampler: history ids outside [0, n_vocab) are ignored (they set bits past the shared bitmap); sampled
top_k 0 or > 64 means 64 and every row is written (0 used to leave the row unwritten).
- server: a client's top_k 0 or > 64 is sent as 64 (it fell back to the default 20).
- CLI: the speculative loop gets the CLI's sampling (it was always greedy), the per-token path draws
Philox(seed, position) like a verify window, and the effective sampling is printed.
- sampler_parity: fixtures 12-15 (per-row histories T=1..8 x H=1..4096, out-of-vocab ids, the top_k
contract, penalty_rows); outputs are pre-filled so an unwritten row fails.
- setup requires engine 0.1.19.
Found by the engineering performance review (C1, C2).
Bring in 45 upstream commits through f32577e. Update the split-GGUF test fixture with router geometry required by upstream's dynamic expert counts. Preserve the Orca BF16 conversion and native PLE compatibility paths.
Validation: Release CUDA build for sm_120; 8 packing tests; 23 server tests; sampler, gated residual, and KV streaming GPU parity checks.
The retained checkpoints are always a prefix chain - the request loop drops
any checkpoint that is not a prefix of the current prompt, because its
positional cells are gone (the KV cache is one arena, one branch of history
at a time). A radix cache's tree collapsed onto the one branch the session
can hold, and the chain's root is the node a radix cache keeps while its
leaves rotate: in practice the end of the system prompt, which every new
chat of the same client mounts through.
First-in-first-out kept that node only by accident of being first: after
prompt_cache newer checkpoints it was gone, and the next chat re-read the
whole prefix at prefill speed (~30 s per 30K tokens). Now the root is
pinned while the budget allows root and leaf apart (cap >= 2; below that
the old behaviour stands), the rest rotates by least recent use, and a
mount re-dates the checkpoint it mounted through.
The policy is one pure function (conv_cache.hpp) with its scenarios in
conv_cache_test.cpp; checkpoint_save/restore and the state format are
untouched, so STRATA_CKPT_REREAD's token-for-token guarantee holds.
`use_bits` required only a non-null history, but the launch sizes the bitmap
for `penalty_last_n > 0` alone - a caller handing over a stale history buffer
with the penalties disabled made the kernel zero `bits_words` words of a
ZERO-BYTE dynamic shared allocation. Unreachable from the serve path
(set_history passes nullptr when the window is empty) and invisible to
memcheck (the write lands in the unallocated SMEM window), but undefined per
the contract of the public `sample_tokens`. The gate now also requires a
non-empty window (`hlen > 0`, both kernels), and fixture 10 pins a stale
history with last_n = 0 to the no-history result in both kernels.
- A config's "gpu" (or the server's --gpu N) runs the engine and the image
encoder on that card: CUDA_VISIBLE_DEVICES=N with CUDA_DEVICE_ORDER=PCI_BUS_ID,
so N means what nvidia-smi and the Monitor's NVML mean (CUDA's own default,
fastest first, can number the cards differently). The Monitor reads that card.
- setup lists every NVIDIA GPU, picks the one with the most VRAM (or --gpu N),
says so when there is more than one, and saves the choice; START-HERE.bat
--gpu N also overrides it when starting. A number that does not exist stops
with the list of cards. Configs without "gpu" run as before.
- Tests: the child environment; the choice on a simulated 4060 + 5070 PC; end to
end on the real engine with --gpu 0.
The sampled chain applied the repetition/frequency/presence penalties a second
time after the temperature, and ran min_p before top_p: both came from the
order of the case labels in llama.cpp's common/sampling.cpp, not the order its
chain runs (params.samplers: PENALTIES ... TOP_K ... TOP_P, MIN_P ...
TEMPERATURE). With presence_penalty 1.5 at T 0.7 a repeated token was drawn at
2.6% instead of 10.5% (@hendrikp's example). Greedy decoding is unchanged.
sampler_parity: the host reference follows the same order; two new fixtures
check against an independently computed distribution (#53's example over 8,000
draws: 0.1040 vs 0.1050) and that top_p-before-min_p is observable. Both fail on
the old kernel (642 and 91 draws differ).
Builds its long texts from this repository's own files at run time, hides a
code word at the chosen depths and asks through the API. Checked on the Coder:
1K, 32K (10/50/90%) and 128K found.
- POST routes ignore the query string (Claude Code calls /v1/messages?beta=true
and got a 404, which it reports as a model problem); /settings too.
- A system (or late developer) message after the first one becomes a user
message in place: the chat template only takes a system message at the start
and raised "System message must be at the beginning" (the client saw a dropped
connection). Kept in place, not merged into the first, so the conversation
cache still reuses the prompt.
- Tests with the issues' own requests (they fail on the old code); DETAILS:
how to point Claude Code at Strata.
- setup: a third family, "coder" (IQ1_M), with its own expert profile; its
shard 2 and vision encoder are the original's files and are reused; 32 GB of
RAM is now enough to install (the Coder's experts are 23 GB).
- data/expert-profile-coder.bin: the shipped ranking mapped onto the kept
experts through the release's rco-allocation.txt (72% GPU hits, 12 GB card);
tools/make_profile.py takes --n-expert.
- README/DETAILS: the Coder's speeds (prompts 599-1,350 tok/s, output
43-53 tok/s, 1K-262K) and what it is; bench/results/2026-09-28-coder.
- The experimental speed projection can be turned on for the Coder too.
- The model files (models/, packs/, mtp/: 70-120 GB) live in Strata-data next to
the Strata folder, remembered per user (%APPDATA%\Strata\settings.json,
~/.config/strata/settings.json). Updating by unzipping a new copy used to give
an empty folder and a full download again. Model files of this folder and of
earlier Strata folders on the same drive are moved there (a rename) and their
configs repointed; files on another drive are used where they are. A new copy
with no config sets itself up like the last install, reusing the files.
--data-dir chooses another place.
- The start says, before and while it happens, that the PC can be slow or stop
responding for 1-3 minutes while the experts are loaded and locked: setup's
start box, and the server narrates the engine's start from its log (weights,
experts + the warning, expert cache, ready, a "still starting" line every 20 s).
- iq_pack.py: derive n_expert from the router tensor shape (was hardcoded 512)
- expert_layout.cpp: parse the pack header's recorded n_expert instead of the
caller's compile-time canonical geometry; use it for the contiguity stride
- gguf_reader.hpp: Qwen4ExpGuard expert-count checks become presence-only
- generate.cpp: read qwen4exp.expert_count/expert_used_count from the model
file into the runtime geometry; MTP drafter keeps canonical geometry (the
draft is the original 512-expert head); serve gate allows profile-less
caches; K from metadata
- layer.cpp: native fused router stays 512x10-only, generic top-k fallback
- prefill: expert count taken from the geometry everywhere; route_kernel
templatized (512->16, 256->8 registers/lane, else generic router_top10)
pcie_frac's defaults (0.55 native / 0.2 canonical) were measured on a PCIe 4.0 x16 link.
A x8 card in a x8 slot carries about half of that, and the copy engine's share then waits
for DMA that arrived late. Probe the H2D bandwidth once at startup the way the engine uses
it (DMA reads from pinned host memory) and scale the default by it: an RTX 5060 Ti at x8
probes 14.1 GB/s -> pcie_frac 0.29 instead of 0.55. --pcie-frac still wins when given, and
a machine that cannot run the probe keeps the current default.
Agentic clients legitimately request the model's full output cap with a long
prompt (e.g. prompt 131,888 + max_tokens 130,298 on a 262,144 window: rejected
over a 42-token overshoot). Default behaviour is unchanged: without the flag
the server still rejects with 400, like llama.cpp. With --fit-max-tokens the
server clamps max_tokens to the remaining room (max_context - CTX_SLACK -
prompt, floor 1 token) and answers.
Measured on a 262,144-token host: prompt 251,455 + max_tokens 131,072 ->
HTTP 200, completion_tokens 10,681 (= 262,144 - 251,455 - 8), finish length.
Flag off: byte-identical 400. Small requests unaffected in both modes.
<p align="center"><b>Ein KI-Modell mit 125 Milliarden Parametern auf deinem eigenen Gaming-PC</b><br>
NVIDIA- oder AMD-Grafikkarte (ab 12 GB) · Windows oder Linux · kostenlos und Open Source</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Ein Voxel-Pagodengarten, den das Modell in Strata geschrieben hat, läuft im Browser"></a><br>
<sub>Ein Voxel-Pagodengarten aus einem einzigen Prompt, auf einer RTX 5070 mit Strata (IQ3_S, 128K Kontext) ·
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">ganzes Video (49 s)</a></sub></p>
Strata lässt **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** auf einem normalen PC laufen.
Das ist ein großes, kluges KI-Modell, das sonst meist einen Server braucht. Es chattet, schreibt Code, versteht
Bilder und arbeitet mit deinen Apps und Coding-Agents zusammen. Nichts verlässt deinen PC.
## Wie schnell ist es?
Wir haben es auf zwei ganz normalen Gaming-PCs gemessen. Ein Token ist etwa ¾ eines Wortes.
- **Schreibt Antworten:** wie schnell die Antwort in einem kurzen Chat erscheint. 60 Tokens pro Sekunde sind
schneller, als du lesen kannst.
- **Liest deinen Prompt:** wie schnell es aufnimmt, was du schickst (hier ein Dokument, Code oder Chatverlauf mit
NVIDIA: Q2_0 mit Engine 0.1.36, die anderen Zeilen mit 0.1.26 (4K Antworten, 32K Prompts). Die vollständigen
Tabellen stehen in [DETAILS.md](docs/DETAILS.md#speed-measured). Eine Karte mit mehr VRAM ist schneller: Eine
RTX 3090 (24 GB) sollte etwa 100-140 Tokens pro Sekunde schreiben. Lange Chats und andere Karten:
[Tempo jedes Modells](docs/MODELS.md#how-fast-is-each-size), [Ergebnisse aus der Community](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata ist kostenlos. Wenn es auf deinem PC gut läuft, hält ein Kaffee die Arbeit daran am Laufen.</sub></p>
## Was du brauchst
| | |
| --- | --- |
| **Grafikkarte** | **NVIDIA** GeForce RTX 20, 30, 40 oder 50 Serie, oder **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 oder RX 6800 / 6900 Serie. Sie braucht **12 GB VRAM oder mehr**. |
| **RAM** | 32 GB oder mehr. Dein RAM entscheidet, [welches Modell](#welches-modell-soll-ich-nehmen) passt. Mit 64 GB laufen alle Größen. |
| **Festplatte** | Etwa 80 GB frei. Nimm wenn möglich eine SSD: Der erste Start geht dann viel schneller. |
| **System** | Windows 10 / 11 oder Linux und ein aktueller Grafiktreiber von NVIDIA oder AMD. |
Alles andere richtet der Installer ein. Zwei oder drei Karten können sich das Modell teilen ([Multi-GPU](docs/MULTI_GPU.md)).
Experimentell, von Community-Mitgliedern auf ihren eigenen Rechnern geschrieben und getestet:
- **Mein PC ist beim ersten Start von Strata eingefroren.** Das ist normal, während das Modell lädt. Warte und
schließ das Fenster nicht. Nach 10 Minuten immer noch eingefroren? Starte den PC neu, schließ andere Programme und
versuch es noch einmal, oder nimm eine kleinere Größe.
- **Es hat beim Herunterladen oder Installieren aufgehört.** Starte `START-HERE.bat` (oder `./setup.sh`) noch
einmal. Es macht dort weiter, wo es aufgehört hat.
- **Es ist sehr langsam und die Festplattenlampe blinkt ständig, oder es meldet „the engine stopped
unexpectedly“.** Dein PC hat nicht genug freien RAM. Schließ andere Programme (Browser brauchen viel) oder nimm
eine kleinere Größe (Q2_0 oder IQ2_XS).
- **Es meldet, dass Port 8080 schon belegt ist.** Strata läuft bereits. Such nach seinem Fenster.
Weitere Probleme und ihre Lösungen: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Kommst du nicht weiter?
Öffne ein [Issue](https://github.com/Niko1221/Strata/issues) und hänge `strata-<model>.log` aus dem Strata-Ordner
an. Du hast ein Sicherheitsproblem gefunden? Melde es vertraulich: [SECURITY.md](SECURITY.md).
## Wie funktioniert es?
Modelle wie dieses laufen normalerweise auf Servern mit Hunderten Gigabyte Grafikspeicher. Deine Grafikkarte hat
12-24 GB. Strata bringt das Modell trotzdem unter, indem es **die Arbeit auf deinen ganzen PC verteilt**. Stell dir
eine Küche vor: Was du ständig brauchst, bleibt auf der Arbeitsfläche, der Rest wartet in der Vorratskammer.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Die 24,576 Experten des Modells: die meistgenutzten auf der Grafikkarte, alle im RAM, eine Nachschlagetabelle auf der SSD"></p>
- **Das Modell ist ein Team aus 24,576 kleinen Spezialisten („Experten“).** Jedes Wort braucht nur 10 davon.
- **Deine Grafikkarte** hält die paar tausend Experten, die am häufigsten gebraucht werden. **Dein RAM** hält alle,
und **dein Prozessor** rechnet gleichzeitig am Rest. **Deine SSD** hält eine große Nachschlagetabelle.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Ein kleiner Helfer rät die nächsten Wörter; das große Modell prüft alle auf einmal und behält die richtigen"></p>
- **Raten, dann prüfen:** Ein kleiner Helfer rät die nächsten paar Wörter. Das große Modell prüft sie alle auf
einmal. Du bekommst dieselbe Antwort, nur 1.6-1.8x schneller.
- **Lange Texte werden in großen Stücken gelesen** (bis zu 8,192 Tokens auf einmal), mit über 1,000 Tokens pro
Sekunde.
Die längere Erklärung: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Jeder Teil und seine Zahlen:
[die Details](docs/DETAILS.md#how-it-works) und das [Paper](docs/paper/Strata-Paper.pdf).
## Danksagung und Lizenz
Das Modell ist [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) vom Qwen-Team. Komprimiert
wurde es von [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
und Unsloth. Strata nutzt Teile von [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Alle Danksagungen:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata ist Open Source unter der [MIT-Lizenz](LICENSE). Ein
paar Teile und alle Modelle haben eigene Lizenzen ([welche](docs/HOW_IT_WORKS.md#license)).
## Strata unterstützen
Strata ist kostenlos und Open Source. Wenn es dir nützt, kannst du die Entwicklung unterstützen:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><b>Ejecuta un modelo de IA de 125 mil millones de parámetros en tu propio PC gaming</b><br>
Tarjeta gráfica NVIDIA o AMD (12 GB o más) · Windows o Linux · libre y de código abierto</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Un jardín con una pagoda de vóxeles que escribió el modelo de Strata, funcionando en el navegador"></a><br>
<sub>Un jardín con una pagoda de vóxeles, hecho con un solo prompt en una RTX 5070 con Strata (IQ3_S, contexto de 128K) ·
NVIDIA: Q2_0 con el motor 0.1.36, las demás filas con 0.1.26 (respuestas de 4K, prompts de 32K). Las tablas
completas están en [DETAILS.md](docs/DETAILS.md#speed-measured). Una tarjeta con más VRAM es más rápida: una
RTX 3090 (24 GB) debería escribir unos 100-140 tokens por segundo. Chats largos y otras tarjetas:
[velocidad de cada modelo](docs/MODELS.md#how-fast-is-each-size), [resultados de la comunidad](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata es gratis. Si funciona bien en tu PC, un café ayuda a seguir trabajando en él.</sub></p>
## Qué necesitas
| | |
| --- | --- |
| **Tarjeta gráfica** | **NVIDIA** GeForce serie RTX 20, 30, 40 o 50, o **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 o serie RX 6800 / 6900. Necesita **12 GB de VRAM o más**. |
| **RAM** | 32 GB o más. Tu RAM decide [qué modelo](#qué-modelo-elijo) cabe. Con 64 GB funcionan todos los tamaños. |
| **Disco** | Unos 80 GB libres. Usa un SSD si puedes: el primer arranque es mucho más rápido. |
| **Sistema** | Windows 10 / 11 o Linux, y un controlador gráfico actual de NVIDIA o AMD. |
El instalador prepara todo lo demás. Dos o tres tarjetas pueden repartirse el modelo ([multi-GPU](docs/MULTI_GPU.md)).
Experimental, escrito y probado por miembros de la comunidad en sus propios equipos:
- **Intel Arc**, compilado desde el código fuente en Linux: [Intel Arc](docs/INTEL_ARC.md).
- **Procesadores antiguos sin AVX2**: funcionan, pero despacio. [Older CPUs](docs/INSTALL.md#older-cpus-experimental).
La lista completa: [docs/INSTALL.md](docs/INSTALL.md#what-you-need).
## Instalación
### Deja que tu IA lo instale
¿Usas un asistente de programación con IA (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Pega esto en él:
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
```
Revisa tu tarjeta gráfica, tu RAM y tu disco, y elige el modelo que cabe. Luego lo instala, lo arranca y te explica
cómo conectar tus aplicaciones. Las herramientas de IA también pueden instalar, arrancar y detener Strata con su
[servidor MCP](docs/MCP_SERVER.md).
### O hazlo tú mismo
[Descarga Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) y descomprímelo (o usa `git clone`).
**Windows:** haz doble clic en **`START-HERE.bat`**. **Linux:** ejecuta **`./setup.sh`** en la carpeta de Strata.
Los pasos son los mismos para NVIDIA y AMD. El instalador detecta tu tarjeta y prepara el motor adecuado para ella.
Te hace unas pocas preguntas:
- qué modelo y qué tamaño,
- cuánto contexto (cuánto texto tiene presente el modelo),
- si debe entender imágenes.
Pulsa Enter cada vez para usar la respuesta recomendada. Después descarga el modelo (unos 70 GB) y lo arranca. Si la
descarga se detiene, vuelve a ejecutarlo: sigue donde lo dejó. Tu navegador abre la aplicación de Strata en
`http://127.0.0.1:8080`.
> **Mientras el modelo arranca, tu PC puede ir lento o dejar de responder durante 1-3 minutos** (más la primera vez).
> Strata carga 35-55 GB en tu RAM y reserva una parte para la tarjeta gráfica. Es normal. Espera y no cierres la
> ventana. La ventana muestra lo que está haciendo Strata.
**La próxima vez**, vuelve a ejecutar `START-HERE.bat` (o `./setup.sh`). Arranca enseguida y no descarga nada dos
veces. Cierra su ventana para detener el modelo. `UPDATE.bat` (`./update.sh`) actualiza Strata sin arrancarlo.
Actualizaciones, Docker, varias tarjetas, dónde se guardan los archivos y todas las opciones: [docs/INSTALL.md](docs/INSTALL.md).
## ¿Qué modelo elijo?
El instalador te recomienda uno según tu RAM. El mismo modelo viene en varios tamaños, más o menos comprimidos. Los
tamaños pequeños son más rápidos. Los grandes son algo más inteligentes.
| Tu RAM | Elige | Por qué |
| --- | --- | --- |
| **32 GB** | **Coder** | cabe en 32 GB y está hecho para programar (con una tarjeta de 24 GB también funcionan Q2_0 e IQ2_XS) |
| **48 GB** | **IQ2_XS** (o Q2_0, el más rápido) | los tamaños más grandes no caben |
| **64 GB** | **IQ2_XS** (recomendado), o IQ3_XXS / IQ3_S | caben todos los tamaños; IQ3_S es el mejor y el más lento |
| **96 GB o más** | **IQ3_S**, o el UD-IQ4_XS de Unsloth (~4 bits) | espacio para los tamaños más grandes con todo lo demás abierto |
- **[Coder](docs/MODELS.md#coder):** una versión para programar a la que se le quitó la mitad de los expertos. Alcanza
el 91% de la puntuación SWE-bench Verified del modelo completo (medido por sus autores) y cabe en 32 GB de RAM. Es
más flojo fuera del código, también con chino y otros textos CJK (#438). Para eso, elige Q2_0, IQ2_XS o IQ3_S, que
conservan todos los expertos.
- **[Swift 1.5](docs/MODELS.md#swift-15):** un ajuste fino que piensa mucho menos tiempo antes de responder. Recibes
la respuesta antes, con más o menos la misma calidad.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** la versión de ~4 bits de Unsloth, entre IQ3_S y
UD-Q4_K_XL en calidad. Una descarga de 94 GB. Con menos de ~80 GB de RAM, Strata lee una parte desde el SSD
mientras responde, así que ahí es más lento (un SSD NVMe ayuda).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental): el más parecido al modelo
completo. Pero Strata lee la mayor parte desde el SSD mientras responde, así que solo escribe 7-8.5 tokens/s en un
PC con 64 GB.
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** se instala a mano. No está en
el menú del instalador.
Tamaños, descargas y qué cabe dónde: [docs/MODELS.md](docs/MODELS.md). Para añadir otro modelo más adelante, ejecuta
`SETUP.bat` (Linux: `./setup.sh --setup`).
## Cómo usarlo
<p align="center"><img src="docs/media/runpagoda.png" width="900" alt="La pestaña Monitor de la aplicación de Strata junto a un agente de programación"><br>
<sub>El <b>Monitor</b> de la aplicación de Strata (izquierda) mientras un agente de programación escribe el jardín de la pagoda del vídeo (derecha)</sub></p>
- **En el navegador:** abre `http://127.0.0.1:8080`. Tiene **Chat**, un **Monitor** en vivo del modelo y de tu
GPU/CPU/RAM, y **About** con los ajustes y las direcciones.
- **Tus aplicaciones y agentes de programación:** añade un proveedor "compatible con OpenAI" con la URL base
**`http://127.0.0.1:8080/v1`**. Sirve cualquier clave de API y cualquier nombre de modelo.
- Aplicaciones que usan la API de Anthropic: `http://127.0.0.1:8080/v1/messages` (Claude Code:
`ANTHROPIC_BASE_URL=http://127.0.0.1:8080`).
- Codex CLI y otras aplicaciones que usan la API Responses de OpenAI: `/v1/responses`
- **Razonamiento:** elige **off, low, medium o high** en el menú del chat o en el "reasoning effort" de tu aplicación.
Off es lo más rápido. High es lo mejor para preguntas difíciles.
- **Imágenes:** responde que sí a "Images?" en la instalación. Luego haz clic en **Picture** en el chat, o adjunta
imágenes en tu aplicación. Las tarjetas AMD leen imágenes en Linux a través del procesador; en Windows todavía no.
- **Desde tu móvil u otro PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Pon siempre una clave.
- **Una petición cada vez:** por defecto Strata responde una petición y las demás esperan. Para responder varias a la
vez, pon `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). En una tarjeta de 12 GB, cada respuesta va más lenta.
- **Prompts largos:** Strata lee entero el primer mensaje de un chat, más o menos 1 minuto por cada 30,000 tokens. Los
mensajes siguientes empiezan en segundos.
Más: [dónde se guardan tus chats](docs/INSTALL.md#where-things-are-stored), [la API](docs/DETAILS.md#using-it).
## ¿Algo salió mal?
- **Mi PC se congeló la primera vez que arrancó Strata.** Es normal mientras carga el modelo. Espera y no cierres la
ventana. ¿Sigue congelado después de 10 minutos? Reinicia el PC, cierra otros programas y vuelve a intentarlo, o
elige un tamaño más pequeño.
- **Se detuvo durante la descarga o la instalación.** Vuelve a ejecutar `START-HERE.bat` (o `./setup.sh`). Sigue
donde se detuvo.
- **Va muy lento y la luz del disco no deja de parpadear, o dice "the engine stopped unexpectedly".** Tu PC no tiene
suficiente RAM libre. Cierra otros programas (los navegadores usan mucha), o elige un tamaño más pequeño (Q2_0 o
IQ2_XS).
- **Dice que el puerto 8080 ya está en uso.** Strata ya está en marcha. Busca su ventana.
Más problemas y sus soluciones: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). ¿Sigues atascado? Abre un
[issue](https://github.com/Niko1221/Strata/issues) y adjunta `strata-<model>.log` de la carpeta de Strata. ¿Has
encontrado un problema de seguridad? Infórmalo en privado: [SECURITY.md](SECURITY.md).
## ¿Cómo funciona?
Los modelos como este suelen funcionar en servidores con cientos de gigabytes de memoria gráfica. Tu tarjeta gráfica
tiene 12-24 GB. Strata hace que el modelo quepa **repartiendo el trabajo por todo tu PC**. Piensa en una cocina: lo
que usas todo el rato se queda en la encimera, y el resto espera en la despensa.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Los 24,576 expertos del modelo: los más usados en la tarjeta gráfica, todos en la RAM y una tabla de consulta en el SSD"></p>
- **El modelo es un equipo de 24,576 pequeños especialistas ("expertos").** Cada palabra solo necesita 10 de ellos.
- **Tu tarjeta gráfica** guarda los pocos miles de expertos que más se usan. **Tu RAM** los tiene todos, y **tu
procesador** trabaja con el resto al mismo tiempo. **Tu SSD** guarda una gran tabla de consulta.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Un pequeño ayudante adivina las siguientes palabras; el modelo grande las comprueba todas a la vez y se queda con las correctas"></p>
- **Adivinar y comprobar:** un pequeño ayudante adivina las siguientes palabras. El modelo grande las comprueba todas
a la vez. Recibes la misma respuesta, 1.6-1.8x antes.
- **Los textos largos se leen en trozos grandes** (hasta 8,192 tokens cada vez), a más de 1,000 tokens por segundo.
La explicación larga: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Cada parte y sus números:
[los detalles](docs/DETAILS.md#how-it-works) y el [artículo](docs/paper/Strata-Paper.pdf).
## Créditos y licencia
El modelo es [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), del equipo de Qwen. Lo
y Unsloth. Strata usa partes de [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Todos los créditos:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata es de código abierto con la [licencia MIT](LICENSE).
Algunas partes y cada modelo tienen sus propias licencias ([cuáles](docs/HOW_IT_WORKS.md#license)).
## Apoya a Strata
Strata es libre y de código abierto. Si te resulta útil, puedes apoyar su desarrollo:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><b>Faites tourner un modèle d'IA de 125 milliards de paramètres sur votre propre PC de jeu</b><br>
Carte graphique NVIDIA ou AMD (12 Go ou plus) · Windows ou Linux · gratuit et open source</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Un jardin de pagode en voxels écrit par le modèle de Strata, qui tourne dans le navigateur"></a><br>
<sub>Un jardin de pagode en voxels, généré en une seule requête sur une RTX 5070 avec Strata (IQ3_S, contexte de 128K) ·
Strata fait tourner **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** sur un PC normal. C'est
un grand modèle d'IA, très capable, qui demande d'habitude un serveur. Il discute, écrit du code, lit des images et
fonctionne avec vos applications et vos agents de code. Rien ne quitte votre PC.
## Quelle est sa vitesse ?
Nous l'avons mesurée sur deux PC de jeu ordinaires. Un token correspond à environ ¾ d'un mot.
- **Écrit les réponses :** la vitesse à laquelle la réponse s'affiche dans une courte discussion. 60 tokens par seconde, c'est plus rapide que votre lecture.
- **Lit votre prompt :** la vitesse à laquelle il absorbe ce que vous envoyez (ici un document, du code ou un historique de discussion de 32K tokens).
<table>
<tr><th>NVIDIA : RTX 5070 (12 Go), Ryzen 5 7600, 64 Go de RAM</th><th>AMD : RX 9070 XT (16 Go), Ryzen 9 3900X, 47 Go de RAM</th></tr>
<tr><td>
| Taille | Écrit les réponses | Lit votre prompt |
| --- | ---: | ---: |
| **Q2_0** | 94 tokens/s | 2,650 tokens/s |
| **IQ2_XS** | 79 tokens/s | 2,090 tokens/s |
| **IQ3_XXS** | 62 tokens/s | 1,750 tokens/s |
| **IQ3_S** | 53 tokens/s | 1,620 tokens/s |
| **Coder** | 55 tokens/s | 2,180 tokens/s |
</td><td>
| Taille | Écrit les réponses | Lit votre prompt |
| --- | ---: | ---: |
| **Q2_0** | 60 tokens/s | 1,160 tokens/s |
| **IQ2_XS** | 52 tokens/s | 1,110 tokens/s |
| **Coder** | 44 tokens/s | 1,420 tokens/s |
</td></tr>
</table>
NVIDIA : Q2_0 avec le moteur 0.1.36, les autres lignes avec 0.1.26 (réponses de 4K, prompts de 32K). Les tableaux
complets sont dans [DETAILS.md](docs/DETAILS.md#speed-measured). Une carte avec plus de VRAM va plus vite : une
RTX 3090 (24 Go) devrait écrire environ 100-140 tokens par seconde. Longues discussions et autres cartes :
[vitesse de chaque modèle](docs/MODELS.md#how-fast-is-each-size), [résultats de la communauté](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata est gratuit. S'il tourne bien sur votre PC, un café aide à poursuivre le travail.</sub></p>
## Ce qu'il vous faut
| | |
| --- | --- |
| **Carte graphique** | **NVIDIA** GeForce RTX série 20, 30, 40 ou 50, ou **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 ou série RX 6800 / 6900. Il faut **12 Go de VRAM ou plus**. |
| **RAM** | 32 Go ou plus. Votre RAM décide [quel modèle](#quel-modèle-choisir-) tient. Avec 64 Go, toutes les tailles tournent. |
| **Disque** | Environ 80 Go libres. Prenez un SSD si possible : le premier démarrage est bien plus rapide. |
| **Système** | Windows 10 / 11 ou Linux, avec un pilote graphique NVIDIA ou AMD à jour. |
L'installateur s'occupe de tout le reste. Deux ou trois cartes peuvent se partager le modèle ([multi-GPU](docs/MULTI_GPU.md)).
Expérimental, écrit et testé par des membres de la communauté sur leurs propres machines :
- **Réflexion :** choisissez **off, low, medium ou high** dans le menu du chat ou dans le « reasoning effort » de
votre application. Off est le plus rapide. High est le meilleur pour les questions difficiles.
- **Images :** répondez oui à « Images? » pendant l'installation. Ensuite cliquez sur **Picture** dans le chat, ou
joignez des images dans votre application. Les cartes AMD lisent les images sous Linux via le processeur ; sous
Windows, pas encore.
- **Depuis votre téléphone ou un autre PC :** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Définissez
toujours une clé.
- **Une requête à la fois :** par défaut, Strata répond à une requête et les autres attendent. Pour répondre à
plusieurs en même temps, mettez `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). Sur une carte de 12 Go, chaque
réponse devient alors plus lente.
- **Longs prompts :** Strata lit en entier le premier message d'une discussion, environ 1 minute pour 30,000 tokens.
Les messages suivants démarrent en quelques secondes.
Plus d'infos : [où sont stockées vos discussions](docs/INSTALL.md#where-things-are-stored), [l'API](docs/DETAILS.md#using-it).
## Un problème ?
- **Mon PC s'est figé au premier démarrage de Strata.** C'est normal pendant le chargement du modèle. Attendez, et ne
fermez pas la fenêtre. Toujours figé après 10 minutes ? Redémarrez le PC, fermez les autres programmes et
réessayez, ou choisissez une taille plus petite.
- **Il s'est arrêté pendant le téléchargement ou l'installation.** Relancez `START-HERE.bat` (ou `./setup.sh`). Il
reprend là où il s'était arrêté.
- **C'est très lent et le voyant du disque clignote sans arrêt, ou il affiche « the engine stopped unexpectedly ».**
Votre PC n'a pas assez de RAM libre. Fermez les autres programmes (les navigateurs en utilisent beaucoup), ou
choisissez une taille plus petite (Q2_0 ou IQ2_XS).
- **Il dit que le port 8080 est déjà utilisé.** Strata tourne déjà. Cherchez sa fenêtre.
Autres problèmes et leurs solutions : [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Toujours bloqué ? Ouvrez une
[issue](https://github.com/Niko1221/Strata/issues) et joignez `strata-<model>.log` depuis le dossier Strata. Vous
avez trouvé un problème de sécurité ? Signalez-le en privé : [SECURITY.md](SECURITY.md).
## Comment ça marche ?
Les modèles comme celui-ci tournent d'habitude sur des serveurs avec des centaines de gigaoctets de mémoire
graphique. Votre carte graphique a 12-24 Go. Strata fait tenir le modèle en **répartissant le travail sur tout votre
PC**. Pensez à une cuisine : ce que vous utilisez tout le temps reste sur le plan de travail, et le reste attend dans
le placard.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Les 24,576 experts du modèle : les plus sollicités sur la carte graphique, tous en RAM, une table de correspondance sur le SSD"></p>
- **Le modèle est une équipe de 24,576 petits spécialistes (les « experts »).** Chaque mot n'en demande que 10.
- **Votre carte graphique** garde les quelques milliers d'experts les plus utilisés. **Votre RAM** les contient tous,
et **votre processeur** travaille sur les autres en même temps. **Votre SSD** contient une grande table de
correspondance.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Un petit assistant devine les mots suivants ; le grand modèle les vérifie tous d'un coup et garde les bons"></p>
- **Deviner, puis vérifier :** un petit assistant devine les quelques mots suivants. Le grand modèle les vérifie
tous d'un coup. Vous obtenez la même réponse, 1.6-1.8x plus tôt.
- **Les longs textes sont lus par gros morceaux** (jusqu'à 8,192 tokens à la fois), à plus de 1,000 tokens par seconde.
L'explication plus longue : [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Chaque partie et ses chiffres :
[les détails](docs/DETAILS.md#how-it-works) et l'[article](docs/paper/Strata-Paper.pdf).
## Crédits et licence
Le modèle est [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), de l'équipe Qwen. Il a été
compressé par [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
et Unsloth. Strata utilise des parties de [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Tous les
crédits : [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata est open source sous [licence MIT](LICENSE).
Quelques parties et chaque modèle ont leur propre licence ([lesquelles](docs/HOW_IT_WORKS.md#license)).
## Soutenir Strata
Strata est gratuit et open source. S'il vous est utile, vous pouvez soutenir son développement :
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata は無料です。あなたの PC でうまく動いたら、コーヒー 1 杯の支援が開発を続ける力になります。</sub></p>
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><b>Run a 125-billion-parameter AI model on your own gaming PC</b><br>
NVIDIA or AMD graphics card (12 GB or more) · Windows or Linux · free and open source</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="A voxel pagoda garden that Strata's model wrote, running in the browser"></a><br>
<sub>A voxel pagoda garden, 1 shot prompt running on an RTX 5070 with Strata (IQ3_S, 128K context) ·
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">full video (49 s)</a></sub></p>
Strata runs **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** - a large, smart AI model that
normally needs a server - on your own PC. It writes its answers at **60-95 tokens per second** (a token is about ¾
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata is free. If it runs well on your PC, a coffee keeps the work on it going.</sub></p>
- **Qwen3.8-Flash-Next** - the original.
- **[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** - a fine-tune by UkisAI
that thinks much shorter before answering, so you get the answer sooner, with about the same quality. Same speed per
token, and about the same RAM as the same size of the original (no IQ3_S). Its own license applies (see its page).
## What you need
Not sure? Take **IQ2_XS**. You can add another one later with `START-HERE.bat --setup`.
| | |
| --- | --- |
| **Graphics card** | **NVIDIA** GeForce RTX 20, 30, 40 or 50 series, or **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 or RX 6800 / 6900 series. It needs **12 GB of VRAM or more**. |
| **RAM** | 32 GB or more. Your RAM decides [which model](#which-model-should-i-pick) fits. 64 GB runs every size. |
| **Disk** | About 80 GB free. Use an SSD if you can: the first start is much faster. |
| **System** | Windows 10 / 11 or Linux, and a current graphics driver from NVIDIA or AMD. |
The installer sets up everything else. Two or three cards can share the model ([multi-GPU](docs/MULTI_GPU.md)).
Experimental, written and tested by community members on their own machines:
- **Intel Arc**, built from source on Linux: [Intel Arc](docs/INTEL_ARC.md).
- **Older processors without AVX2**: they work, but slowly. [Older CPUs](docs/INSTALL.md#older-cpus-experimental).
The full list: [docs/INSTALL.md](docs/INSTALL.md#what-you-need).
## Install
**You need:** an NVIDIA RTX 30, 40 or 50 card with 12 GB of VRAM or more, enough RAM for the size you pick (above),
~80 GB of free disk space (an SSD makes the first start much faster), and Windows 10/11 or Linux. The only thing you
install yourself is a current **NVIDIA driver** ([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA
App). Everything else - Python, the engine, the model - is set up for you.
### Let your AI set it up
**Windows**
Do you use an AI coding assistant (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Paste this into it:
1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it).
2. Double-click **`START-HERE.bat`**.
3. Answer a few questions - or just press Enter each time for the recommended choice:
- **Which model and size?** The original or Swift 1.5, and Q2_0, IQ2_XS, IQ3_XXS or IQ3_S - see [above](#which-model-should-i-pick)
- **How much context?** How much text it can keep in mind at once (it suggests one for your card)
- **Images?** Whether it should also read pictures
- **Experimental speed projection?** Off unless you say yes - [read what it does](docs/DETAILS.md#experimental-speed-projection-experimental-off-by-default) first
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
```
Then it downloads everything (the model is ~70 GB, so the first time takes a while - you can stop and it picks up
where it left off) and **starts the model**. Your browser opens the Strata app at `http://127.0.0.1:8080`.
It checks your graphics card, RAM and disk and picks the model that fits. Then it installs and starts it and tells
you how to connect your apps. AI tools can also install, start and stop Strata through its
[MCP server](docs/MCP_SERVER.md).
**Next time**, just double-click `START-HERE.bat` again: it starts right away, nothing is downloaded twice. Close its
window to stop the model.
### Or do it yourself
**Linux:** run `./setup.sh` - same questions, same result.
[Download Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it).
**Windows:** double-click **`START-HERE.bat`**. **Linux:** run **`./setup.sh`** in the Strata folder.
The steps are the same for NVIDIA and AMD. The installer finds your card and sets up the right engine for it. It
asks you a few questions:
- which model and which size,
- how much context (how much text the model keeps in mind),
- whether it should read pictures.
Press Enter each time for the recommended answer. Then it downloads the model (about 70 GB) and starts it. If the
download stops, run it again: it continues where it left off. Your browser opens the Strata app at
`http://127.0.0.1:8080`.
> **While the model starts, your PC can be slow or stop responding for 1-3 minutes** (longest the first time).
> Strata loads 35-55 GB into your RAM and locks part of it for the graphics card. This is normal. Wait, and don't
> close the window. The window shows what Strata is doing.
**Next time**, run `START-HERE.bat` (or `./setup.sh`) again. It starts right away and downloads nothing twice. Close
its window to stop the model. `UPDATE.bat` (`./update.sh`) updates Strata without starting it. Updating, Docker,
several cards, where the files go and every option: [docs/INSTALL.md](docs/INSTALL.md).
## Which model should I pick?
The installer recommends one for your RAM. The same model comes in several sizes, compressed more or less. Smaller
sizes are faster. Larger sizes are a bit smarter.
| Your RAM | Take | Why |
| --- | --- | --- |
| **32 GB** | **Coder** | it fits 32 GB, and it is made for code (with a 24 GB card, Q2_0 and IQ2_XS run too) |
| **48 GB** | **IQ2_XS** (or Q2_0, the fastest) | the larger sizes do not fit |
| **64 GB** | **IQ2_XS** (recommended), or IQ3_XXS / IQ3_S | every size fits; IQ3_S is the best and the slowest |
| **96 GB or more** | **IQ3_S**, or Unsloth's UD-IQ4_XS (~4-bit) | room for the largest sizes with everything else open |
- **[Coder](docs/MODELS.md#coder):** a coding version with half of the experts removed. It reaches 91% of the full
model's SWE-bench Verified score (measured by its authors) and fits 32 GB of RAM. It is weaker outside code,
including Chinese and other CJK text (#438). For those, take Q2_0, IQ2_XS or IQ3_S, which keep every expert.
- **[Swift 1.5](docs/MODELS.md#swift-15):** a fine-tune that thinks for a much shorter time before it answers. You
get the answer sooner, at about the same quality.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** Unsloth's ~4-bit version, between IQ3_S and
UD-Q4_K_XL in quality. A 94 GB download. With less than ~80 GB of RAM, Strata reads part of it from the SSD
while it answers, so it is slower there (an NVMe SSD helps).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental): the closest to the full
model. But Strata reads most of it from the SSD while it answers, so it writes only 7-8.5 tokens/s on a 64 GB PC.
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** you set it up by hand. It is
not in the installer's menu.
Sizes, downloads and what fits where: [docs/MODELS.md](docs/MODELS.md). To add another model later, run
`SETUP.bat` (Linux: `./setup.sh --setup`).
## Using it
<p align="center"><img src="docs/media/runpagoda.png" width="900" alt="The Strata app's Monitor tab next to a coding agent"><br>
<sub>The Strata app's <b>Monitor</b> (left) while a coding agent writes the pagoda garden from the video (right)</sub></p>
- **In the browser:** `http://127.0.0.1:8080` - the Strata app (it opens by itself when the model starts): **Chat**, a
live **Monitor** of the model and your GPU/CPU/RAM, and **About** with the settings and addresses.
- **Chat in the terminal:** `.venv\Scripts\python chat.py`
- **Your apps and coding agents:** add it as an "OpenAI-compatible" provider with base URL
**`http://127.0.0.1:8080/v1`**, any API key and any model name. Apps that use Anthropic's API: `http://127.0.0.1:8080/v1/messages`.
- **Thinking:** the model thinks before it answers. Choose **off, low, medium or high** - in the chat page menu, with
`/think low` in `chat.py`, or with your app's "reasoning effort" setting. Off is fastest; high is best for hard questions.
- **Pictures:** in the chat page click **Picture**; in `chat.py` type `/image <path>`; in apps just attach them.
- **From your phone or another PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`, then open the
address the server window prints; see the [details](docs/DETAILS.md#using-it).
- **Experimental speed projection (off by default):** an experimental control vector that setup can turn on; it
changes how the model answers - read [what it does](docs/DETAILS.md#experimental-speed-projection-experimental-off-by-default) first.
- **In the browser:** open`http://127.0.0.1:8080`. It has **Chat**, a live **Monitor** of the model and your
GPU/CPU/RAM, and **About** with the settings and addresses.
- **Your apps and coding agents:** add an "OpenAI-compatible" provider with the base URL
**`http://127.0.0.1:8080/v1`**. Any API key and any model name work.
- Apps that use Anthropic's API: `http://127.0.0.1:8080/v1/messages` (Claude Code:
`ANTHROPIC_BASE_URL=http://127.0.0.1:8080`).
- Codex CLI and other apps that use the OpenAI Responses API: `/v1/responses`
- **Thinking:** choose **off, low, medium or high** in the chat menu or in your app's "reasoning effort". Off is the
fastest. High is best for hard questions.
- **Pictures:** say yes to "Images?" in setup. Then click **Picture** in the chat, or attach pictures in your app.
AMD cards read pictures on Linux through the processor; on Windows they can't yet.
- **From your phone or another PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Always set a key.
- **One request at a time:** by default Strata answers one request, and the others wait. To answer several at once,
set `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). On a 12 GB card this makes each answer slower.
- **Long prompts:** Strata reads the first message of a chat in full, about 1 minute per 30,000 tokens. Follow-up
messages start in seconds.
**Good to know:** it answers one request at a time. The first message of a chat is read in full (about 1 minute per
30,000 tokens); after that it keeps the conversation and reads only what is new, so follow-ups start in seconds.
More: [where your chats are stored](docs/INSTALL.md#where-things-are-stored), [the API](docs/DETAILS.md#using-it).
## Something went wrong?
**My PC froze, or got very slow, the first time Strata started.**
That's normal the first time. Strata loads 35-55 GB into your RAM, locks part of it for the graphics card, and works
out how much of the model fits on your GPU. The mouse can freeze for a few minutes. **Wait, and don't close the
window.** The next starts are much faster. Still frozen after 10 minutes? Restart the PC, close other programs
(browsers use a lot of RAM) and try again. If it keeps happening, pick a smaller size (Q2_0 or IQ2_XS).
-**My PC froze the first time Strata started.** This is normal while it loads the model. Wait, and don't close the
window. Still frozen after 10 minutes? Restart the PC, close other programs and try again, or pick a smaller size.
- **It stopped while downloading or installing.** Run `START-HERE.bat` (or `./setup.sh`) again. It continues where
it stopped.
- **It's very slow and the disk light keeps blinking, or it says "the engine stopped unexpectedly".** Your PC does
not have enough free RAM. Close other programs (browsers use a lot), or pick a smaller size (Q2_0 or IQ2_XS).
- **It says port 8080 is already in use.** Strata is already running. Look for its window.
**It stopped while downloading or installing.**
Run `START-HERE.bat` again. It continues where it stopped.
**It says the NVIDIA driver is too old.**
Update it (NVIDIA App or [nvidia.com/drivers](https://www.nvidia.com/drivers)), restart the PC, and run
`START-HERE.bat` again.
**It says port 8080 is already in use.**
Strata is already running. Look for its window.
**It's very slow and the disk light keeps blinking.**
Your PC is out of free RAM. Close other programs, or pick a smaller size (Q2_0 or IQ2_XS).
**An answer stopped with "the engine stopped unexpectedly".**
Usually not enough RAM (on Linux the system then stops the engine). Just send your message again: Strata starts the
engine by itself. If it keeps happening, close other programs or pick a smaller size.
**It says the prompt exceeds the context.**
The conversation is longer than the context you chose. Start a new chat, or run `START-HERE.bat --setup` and pick more
context.
**Still stuck?** Look in the [full troubleshooting table](docs/DETAILS.md#troubleshooting), or open an issue and
attach `strata-<model>.log` from the Strata folder.
More problems and their fixes: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Still stuck? Open an
[issue](https://github.com/Niko1221/Strata/issues) and attach `strata-<model>.log` from the Strata folder. Found a
security problem? Report it privately: [SECURITY.md](SECURITY.md).
## How does it work?
A model this big doesn't fit on a gaming graphics card. Strata splits the work between the parts of your PC:
Models like this one usually run on servers with hundreds of gigabytes of graphics memory. Your graphics card has
12-24 GB. Strata makes the model fit by **sharing the work across your whole PC**. Think of a kitchen: the things
you use all the time stay on the counter, and the rest waits in the pantry.
<p align="center"><img src="docs/paper/tiers.svg" width="700" alt="how Strata splits the model between GPU, RAM and SSD"></p>
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="The model's 24,576 experts: the busiest on the graphics card, all of them in RAM, a lookup table on the SSD"></p>
- **The GPU** runs the part of the model that is used for every word, plus the "experts" it needs most often.
- **The RAM** holds all 24,576 experts, and **the CPU** computes the few the GPU doesn't have - at the same time as the GPU.
- **The SSD** holds a big lookup table; the model reads a few rows of it per word.
- **A small helper inside the model guesses the next words**, and Strata checks several guesses at once. That makes
it 1.6-1.8x faster than going word by word - and the answer is exactly the same.
- **The model is a team of 24,576 small specialists ("experts").** Each word needs only 10 of them.
- **Your graphics card** keeps the few thousand experts that are used most often. **Your RAM** holds all of them,
and **your processor** works on the rest at the same time. **Your SSD** holds a big lookup table.
The full story is in the [paper](docs/paper/Strata-Paper.pdf) and the [details](docs/DETAILS.md).
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="A small helper guesses the next words; the big model checks them all at once and keeps the right ones"></p>
## Credits
- **Guess, then check:** a small helper guesses the next few words. The big model checks them all at once. You get
the same answer, 1.6-1.8x sooner.
- **Long texts are read in big pieces** (up to 8,192 tokens at a time), at over 1,000 tokens per second.
- Model: [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team; compressed versions by
[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF) by UkisAI. Their licenses apply
to the model files.
- Built with parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT). Ideas from
[Splash](https://github.com/incoai/splash), [ninfer](https://github.com/Neroued/ninfer) and
[HyperQwen](https://github.com/syv-ai/HyperQwen). More in the [details](docs/DETAILS.md#credits-and-licenses).
The longer explanation: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Every part and its numbers:
[the details](docs/DETAILS.md#how-it-works) and the [paper](docs/paper/Strata-Paper.pdf).
## Credits and license
The model is [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team. It was
compressed by [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
and Unsloth. Strata uses parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). All credits:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata is open source under the [MIT License](LICENSE). A few
parts and every model have their own licenses ([which ones](docs/HOW_IT_WORKS.md#license)).
## Support Strata
Strata is free and open source. If it is useful to you, you can support its development:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><b>Rode um modelo de IA de 125 bilhões de parâmetros no seu próprio PC gamer</b><br>
Placa de vídeo NVIDIA ou AMD (12 GB ou mais) · Windows ou Linux · gratuito e de código aberto</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Um jardim com pagode em voxels que o modelo do Strata escreveu, rodando no navegador"></a><br>
<sub>Um jardim com pagode em voxels, feito com um único prompt numa RTX 5070 com o Strata (IQ3_S, contexto de 128K) ·
NVIDIA: Q2_0 com o engine 0.1.36, as outras linhas com 0.1.26 (respostas de 4K, prompts de 32K). As tabelas
completas estão em [DETAILS.md](docs/DETAILS.md#speed-measured). Uma placa com mais VRAM é mais rápida: uma
RTX 3090 (24 GB) deve escrever cerca de 100-140 tokens por segundo. Conversas longas e outras placas:
[velocidade de cada modelo](docs/MODELS.md#how-fast-is-each-size), [resultados da comunidade](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>O Strata é gratuito. Se ele roda bem no seu PC, um café ajuda a manter o trabalho nele.</sub></p>
## Do que você precisa
| | |
| --- | --- |
| **Placa de vídeo** | **NVIDIA** GeForce RTX séries 20, 30, 40 ou 50, ou **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 ou séries RX 6800 / 6900. Ela precisa de **12 GB de VRAM ou mais**. |
| **RAM** | 32 GB ou mais. Sua RAM decide [qual modelo](#qual-modelo-devo-escolher) cabe. Com 64 GB, roda qualquer tamanho. |
| **Disco** | Cerca de 80 GB livres. Use um SSD se puder: a primeira inicialização fica bem mais rápida. |
| **Sistema** | Windows 10 / 11 ou Linux, e um driver de vídeo atualizado da NVIDIA ou da AMD. |
O instalador cuida de todo o resto. Duas ou três placas podem dividir o modelo ([multi-GPU](docs/MULTI_GPU.md)).
Experimental, escrito e testado por membros da comunidade nas próprias máquinas:
- **Placas de vídeo mais antigas** (Tesla P40 / V100, GTX 10, Radeon VII / MI50, RX 6700 XT, RX 5500 XT): [Older GPUs](docs/OLDER_GPUS.md).
- **Intel Arc**, compilado a partir do código-fonte no Linux: [Intel Arc](docs/INTEL_ARC.md).
- **Processadores mais antigos sem AVX2**: funcionam, mas devagar. [Older CPUs](docs/INSTALL.md#older-cpus-experimental).
A lista completa: [docs/INSTALL.md](docs/INSTALL.md#what-you-need).
## Instalação
### Deixe sua IA instalar
Você usa um assistente de programação com IA (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Cole isto nele:
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
```
Ele verifica sua placa de vídeo, RAM e disco e escolhe o modelo que cabe. Depois instala, inicia e explica como
conectar seus apps. Ferramentas de IA também podem instalar, iniciar e parar o Strata pelo
[servidor MCP](docs/MCP_SERVER.md) dele.
### Ou faça você mesmo
[Baixe o Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) e descompacte (ou use `git clone`).
**Windows:** clique duas vezes em **`START-HERE.bat`**. **Linux:** rode **`./setup.sh`** na pasta do Strata.
Os passos são os mesmos para NVIDIA e AMD. O instalador encontra sua placa e configura o engine certo para ela.
Ele faz algumas perguntas:
- qual modelo e qual tamanho,
- quanto contexto (quanto texto o modelo guarda na memória),
- se ele deve entender imagens.
Aperte Enter em cada uma para usar a resposta recomendada. Depois ele baixa o modelo (cerca de 70 GB) e o inicia.
Se o download parar, rode de novo: ele continua de onde parou. Seu navegador abre o app do Strata em
`http://127.0.0.1:8080`.
> **Enquanto o modelo inicia, seu PC pode ficar lento ou parar de responder por 1-3 minutos** (mais na primeira vez).
> O Strata carrega 35-55 GB na sua RAM e reserva parte dela para a placa de vídeo. Isso é normal. Espere e não
> feche a janela. A janela mostra o que o Strata está fazendo.
**Da próxima vez**, rode `START-HERE.bat` (ou `./setup.sh`) de novo. Ele inicia na hora e não baixa nada duas
vezes. Feche a janela para parar o modelo. `UPDATE.bat` (`./update.sh`) atualiza o Strata sem iniciá-lo.
Atualização, Docker, várias placas, onde ficam os arquivos e todas as opções: [docs/INSTALL.md](docs/INSTALL.md).
## Qual modelo devo escolher?
O instalador recomenda um de acordo com sua RAM. O mesmo modelo vem em vários tamanhos, mais ou menos comprimidos.
Os tamanhos menores são mais rápidos. Os maiores são um pouco mais inteligentes.
| Sua RAM | Escolha | Por quê |
| --- | --- | --- |
| **32 GB** | **Coder** | cabe em 32 GB e é feito para código (com uma placa de 24 GB, Q2_0 e IQ2_XS também rodam) |
| **48 GB** | **IQ2_XS** (ou Q2_0, o mais rápido) | os tamanhos maiores não cabem |
| **64 GB** | **IQ2_XS** (recomendado), ou IQ3_XXS / IQ3_S | todos os tamanhos cabem; IQ3_S é o melhor e o mais lento |
| **96 GB ou mais** | **IQ3_S**, ou o UD-IQ4_XS da Unsloth (~4-bit) | espaço para os maiores tamanhos com todo o resto aberto |
- **[Coder](docs/MODELS.md#coder):** uma versão para programação com metade dos experts removida. Ela alcança 91%
da pontuação do modelo completo no SWE-bench Verified (medido pelos autores) e cabe em 32 GB de RAM. É mais fraca
fora do código, inclusive em chinês e outros textos CJK (#438). Para isso, escolha Q2_0, IQ2_XS ou IQ3_S, que
mantêm todos os experts.
- **[Swift 1.5](docs/MODELS.md#swift-15):** um fine-tune que pensa por bem menos tempo antes de responder. Você
recebe a resposta mais cedo, com quase a mesma qualidade.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** a versão de ~4 bits da Unsloth, entre IQ3_S e
UD-Q4_K_XL em qualidade. Um download de 94 GB. Com menos de ~80 GB de RAM, o Strata lê uma parte dele do SSD
enquanto responde, então ali ele fica mais lento (um SSD NVMe ajuda).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental): o mais próximo do modelo
completo. Mas o Strata lê a maior parte dele do SSD enquanto responde, então ele escreve só 7-8.5 tokens/s num PC
com 64 GB.
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** você configura à mão. Ele
não está no menu do instalador.
Tamanhos, downloads e o que cabe onde: [docs/MODELS.md](docs/MODELS.md). Para adicionar outro modelo depois, rode
`SETUP.bat` (Linux: `./setup.sh --setup`).
## Como usar
<p align="center"><img src="docs/media/runpagoda.png" width="900" alt="A aba Monitor do app Strata ao lado de um agente de programação"><br>
<sub>O <b>Monitor</b> do app Strata (à esquerda) enquanto um agente de programação escreve o jardim com pagode do vídeo (à direita)</sub></p>
- **No navegador:** abra `http://127.0.0.1:8080`. Lá tem o **Chat**, um **Monitor** ao vivo do modelo e da sua
GPU/CPU/RAM, e o **About** com as configurações e os endereços.
- **Seus apps e agentes de programação:** adicione um provedor "OpenAI-compatible" com a URL base
**`http://127.0.0.1:8080/v1`**. Qualquer chave de API e qualquer nome de modelo funcionam.
- Apps que usam a API da Anthropic: `http://127.0.0.1:8080/v1/messages` (Claude Code:
`ANTHROPIC_BASE_URL=http://127.0.0.1:8080`).
- Codex CLI e outros apps que usam a OpenAI Responses API: `/v1/responses`
- **Raciocínio:** escolha **off, low, medium ou high** no menu do chat ou no "reasoning effort" do seu app. Off é o
mais rápido. High é o melhor para perguntas difíceis.
- **Imagens:** responda sim para "Images?" na instalação. Depois clique em **Picture** no chat, ou anexe imagens no
seu app. Placas AMD entendem imagens no Linux usando o processador; no Windows, ainda não.
- **Do seu celular ou de outro PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Sempre defina uma chave.
- **Uma requisição por vez:** por padrão, o Strata responde uma requisição e as outras esperam. Para responder
várias ao mesmo tempo, defina `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). Numa placa de 12 GB, isso deixa
cada resposta mais lenta.
- **Prompts longos:** o Strata lê a primeira mensagem de uma conversa inteira, cerca de 1 minuto a cada 30,000
tokens. As mensagens seguintes começam em segundos.
Mais: [onde suas conversas ficam salvas](docs/INSTALL.md#where-things-are-stored), [a API](docs/DETAILS.md#using-it).
## Algo deu errado?
- **Meu PC travou na primeira vez que o Strata iniciou.** Isso é normal enquanto ele carrega o modelo. Espere e não
feche a janela. Ainda travado depois de 10 minutos? Reinicie o PC, feche outros programas e tente de novo, ou
escolha um tamanho menor.
- **Parou durante o download ou a instalação.** Rode `START-HERE.bat` (ou `./setup.sh`) de novo. Ele continua de
onde parou.
- **Está muito lento e a luz do disco não para de piscar, ou aparece "the engine stopped unexpectedly".** Seu PC
não tem RAM livre suficiente. Feche outros programas (navegadores usam muita), ou escolha um tamanho menor (Q2_0
ou IQ2_XS).
- **Diz que a porta 8080 já está em uso.** O Strata já está rodando. Procure a janela dele.
Mais problemas e como resolver: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Ainda sem solução? Abra uma
[issue](https://github.com/Niko1221/Strata/issues) e anexe o `strata-<model>.log` da pasta do Strata. Encontrou um
problema de segurança? Informe em particular: [SECURITY.md](SECURITY.md).
## Como funciona?
Modelos como este normalmente rodam em servidores com centenas de gigabytes de memória de vídeo. Sua placa de vídeo
tem 12-24 GB. O Strata faz o modelo caber **dividindo o trabalho pelo PC inteiro**. Pense numa cozinha: o que você
usa o tempo todo fica na bancada, e o resto espera na despensa.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Os 24,576 experts do modelo: os mais usados na placa de vídeo, todos na RAM, uma tabela de consulta no SSD"></p>
- **O modelo é uma equipe de 24,576 pequenos especialistas ("experts").** Cada palavra precisa de só 10 deles.
- **Sua placa de vídeo** guarda os poucos milhares de experts usados com mais frequência. **Sua RAM** guarda todos
eles, e **seu processador** trabalha no resto ao mesmo tempo. **Seu SSD** guarda uma grande tabela de consulta.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Um pequeno ajudante adivinha as próximas palavras; o modelo grande confere todas de uma vez e fica com as certas"></p>
- **Adivinhar e conferir:** um pequeno ajudante adivinha as próximas palavras. O modelo grande confere todas de
uma vez. Você recebe a mesma resposta, 1.6-1.8x mais cedo.
- **Textos longos são lidos em grandes pedaços** (até 8,192 tokens por vez), a mais de 1,000 tokens por segundo.
A explicação mais longa: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Cada parte e seus números:
[os detalhes](docs/DETAILS.md#how-it-works) e o [artigo](docs/paper/Strata-Paper.pdf).
## Créditos e licença
O modelo é o [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), da equipe Qwen. Ele foi
comprimido por [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
e Unsloth. O Strata usa partes do [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Todos os créditos:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). O Strata é de código aberto sob a [MIT License](LICENSE).
Algumas partes e todos os modelos têm licenças próprias ([quais](docs/HOW_IT_WORKS.md#license)).
## Apoie o Strata
O Strata é gratuito e de código aberto. Se ele é útil para você, você pode apoiar o desenvolvimento:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
- **Correctness runs:** `rib_norepeat.py`, with a fixed cache, no swaps, no PCIe share and no suffix drafts where two
runs are compared byte for byte.
- **Speed runs:** `rib_pp.py`.
- Raw results are in `data/`.
## Correctness
| Check | Result |
|---|---|
| One GPU, this branch vs the 0.1.21-equivalent run (fixed cache) | 10/10 byte-identical |
| Speed projection (control vector), one GPU vs the split on the same GPU (K=24) | 10/10 byte-identical |
| KV streaming (128K context, 32K resident), one GPU vs the split on the same GPU | 10/10 byte-identical |
| Images, one GPU vs the split on the same GPU (three questions about a test picture) | identical answers |
| Speed projection / KV streaming / plain, 5080 + 3090 (K=24): no-repeat under presence 100, the 144 sanity case | all pass, 0 repeats |
| The vector acts across GPUs: 5080 + 3090 with vs without the vector | 8 of 10 answers differ (as on one GPU: 7 of 10) |
| Images, 5080 + 3090 (K=24): "What color is the circle?" / "What number?" / describe | "Red" / "42" / "A red circle, the number 42, and a blue square are displayed on a white background" |
| Mid-prompt checkpoint (every 16,384 tokens): a 19.5K prompt, then another sharing its first 18,600 tokens | one GPU and 5080 + 3090: reused 16,384, answer right |
| 5080 + 3090 (K=30) + a 2080 Ti helper holding 3,000 experts | all pass |
| `--mmap-experts` with the Coder (a native pack) | a clear start-up error: it needs a canonical pack's `experts.bin` (with or without a split) |
Across GPUs the text is not byte-identical to one GPU: experts that one card leaves to the CPU run on a GPU in the
split, and the two round differently.
## Speed
| Run | Prompt 16K / 28K / 60K tok/s | Decode story / code tok/s | Hits |
"text":"The wind didn\u2019t just howl; it screamed. It tore through the iron gallery of the Blackwood Point Light, rattling the glass panes like a prisoner testing the bars of a cell.\n\nElias Thorne was not a man who feared the dark. He was a man who feared the silence.\n\nFor forty years, he had lived in the shadow of the Fresnel lens, a rotating eye of glass and brass that swept the Atlantic with a beam of pure, unyielding light. His job was simple: keep the light burning. Keep the logbook current. Keep the foghorn sounding when the mist rolled in, thick as wool.\n\nBut Elias knew something else. He knew that the light was not just for ships. It was for the lost.\n\nIt was November 14th, 1982. The storm was a Category 5, a monster that had swallowed the horizon. The waves were mountains of black water, crashing against the granite"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":134,
"drafts_offered":180,
"tok_s":80.8,
"text":"```python\nimport csv\nfrom typing import List, Dict\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, str]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False"
"text":"The wind didn\u2019t just howl; it screamed against the glass of Lantern House 42, demanding entry into every crack and crevice that had been sealed with tar in previous years but now wept like old wounds on an open sore skin\u2014like a wound left unhealed by time. Elias knew this rhythm better than he remembered his own name<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":26,
"drafts_offered":76,
"tok_s":49.3,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in my teeth, searching for something to break its rhythm against stone that refused be broken by anything less than God himself had forged into being willing and able withstand eternity\u2019s cruelty without complaint or mercy from start finish until now forevermore if any such concept even existed within human comprehension let alone divine authority which we all knew did not exist but wished it would because wishful thinking is always easier to imagine hopelessly impossible truths as probable lies than vice versa.\n\nElias Thorne was eighty-three years old today, exactly one hundred and forty-seven days since his last heart attack had been logged by Dr Armitage who himself retired seven months ago after receiving a subpoena from the county clerk indicating he might have overstayed professional courtesy boundaries"
"text":"The phrase \"the History Of The Sea\" is inherently ambiguous, as it could refer to several distinct historical narratives: geological formation (how oceans came into existence), maritime navigation and exploration by humans throughout human civilization; or perhaps more specifically in modern times refers often used metaphorically for economic trends like stock market fluctuations (\"sea change\"). However given your request likely intends an academic-style comprehensive overview covering both natural science aspects AND cultural significance through time periods\u2014especially since this kind question appears frequently on exams asking students write essays discussing evolution from primordial origins up present-day impacts including environmental changes due climate crisis etc.\u2014I will craft such broad yet structured piece combining geology + anthropology/ecological impact together while maintaining coherent flow suitable high-level educational context without overly technical jargon but still accurate enough scientifically grounded facts where appropriate within reasonable scope limits typical school assignments expect (~80\u201312% word count range implied implicitly though not stated explicitly here we aim moderate length ~65 words? Wait no user said \u201clong\u201d so probably >3k+ characters minimum safe bet around 7-9 paragraphs total approximating maybe close-to-full-page standard double-spaced equivalent which translates roughly \u22654K chars safely acceptable). Let me proceed accordingly ensuring clarity structure intro body conclusion style formal tone neutral informative balanced perspective acknowledging uncertainties if any minor ones exist e.g exact dates vary depending sources hence use hedging language sparingly only when necessary avoid overclaiming certainty especially regarding speculative theories vs established consensus knowledge base current mainstream understanding circa early-mid"
"text":"The wind didn\u2019t just howl; it screamed. It tore through the iron gallery of the Blackwood Point Light, rattling the glass panes like a prisoner testing the bars of a cell.\n\nElias Thorne was not a man who feared the dark. He was a man who feared the silence.\n\nFor forty years, he had lived in the shadow of the Fresnel lens, a rotating eye of glass and brass that swept the Atlantic with a beam of pure, unyielding light. His job was simple: keep the light burning. Keep the logbook current. Keep the foghorn sounding when the mist rolled in, thick as wool.\n\nBut Elias knew something else. He knew that the light was not just for ships. It was for the lost.\n\nIt was November 14th, 1982. The storm was a Category 5, a monster that had swallowed the horizon. The waves were mountains of black water, crashing against the granite"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":134,
"drafts_offered":180,
"tok_s":81.0,
"text":"```python\nimport csv\nfrom typing import List, Dict\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, str]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False"
"text":"The wind didn\u2019t just howl; it screamed against the glass of Lantern House 42, demanding entry into every crack and crevice that had been sealed with tar in previous years but now wept like old wounds on an open sore skin\u2014like a wound left unhealed by time. Elias knew this rhythm better than he remembered his own name<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":26,
"drafts_offered":76,
"tok_s":48.9,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in my teeth, searching for something to break its rhythm against stone that refused be broken by anything less than God himself had forged into being willing and able withstand eternity\u2019s cruelty without complaint or mercy from start finish until now forevermore if any such concept even existed within human comprehension let alone divine authority which we all knew did not exist but wished it would because wishful thinking is always easier to imagine hopelessly impossible truths as probable lies than vice versa.\n\nElias Thorne was eighty-three years old today, exactly one hundred and forty-seven days since his last heart attack had been logged by Dr Armitage who himself retired seven months ago after receiving a subpoena from the county clerk indicating he might have overstayed professional courtesy boundaries"
"text":"The phrase \"the History Of The Sea\" is inherently ambiguous, as it could refer to several distinct historical narratives: geological formation (how oceans came into existence), maritime navigation and exploration by humans throughout human civilization; or perhaps more specifically in modern times refers often used metaphorically for economic trends like stock market fluctuations (\"sea change\"). However given your request likely intends an academic-style comprehensive overview covering both natural science aspects AND cultural significance through time periods\u2014especially since this kind question appears frequently on exams asking students write essays discussing evolution from primordial origins up present-day impacts including environmental changes due climate crisis etc.\u2014I will craft such broad yet structured piece combining geology + anthropology/ecological impact together while maintaining coherent flow suitable high-level educational context without overly technical jargon but still accurate enough scientifically grounded facts where appropriate within reasonable scope limits typical school assignments expect (~80\u201312% word count range implied implicitly though not stated explicitly here we aim moderate length ~65 words? Wait no user said \u201clong\u201d so probably >3k+ characters minimum safe bet around 7-9 paragraphs total approximating maybe close-to-full-page standard double-spaced equivalent which translates roughly \u22654K chars safely acceptable). Let me proceed accordingly ensuring clarity structure intro body conclusion style formal tone neutral informative balanced perspective acknowledging uncertainties if any minor ones exist e.g exact dates vary depending sources hence use hedging language sparingly only when necessary avoid overclaiming certainty especially regarding speculative theories vs established consensus knowledge base current mainstream understanding circa early-mid"
"text":"The wind didn\u2019t just howl; it screamed. It tore through the gaps in the iron railing of the Lantern Room, carrying the scent of brine and rotting kelp. Elias pulled his woolen collar up to his chin, his fingers numb and clumsy inside thick leather gloves.\n\nHe was the last man on the island. The automated systems had failed three winters ago, a glitch in the solar-powered backup that no engineer had been able to fix before the supply boat stopped coming. Now, it was just him, the light, and the sea.\n\nElias climbed the spiral stairs, his boots heavy with salt spray. The iron steps groaned under his weight, a sound that seemed to mock the storm outside. He reached the top, the glass walls vibrating with the force of the gale. Outside, the world was a blur of black and white. The ocean was invisible; it was only a presence, a weight pressing against the stone foundation of the tower.\n\nHe"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":173,
"tok_s":96.8,
"text":"```python\nimport csv\nfrom typing import List, Dict\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, str]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False"
"text":"The wind didn\u2019t just howl; it screamed against the glass of Lantern House 42, demanding entry into every crack and crevice that had been sealed with tar in previous years but now wept like old wounds at this one\u2019s age: forty-three winters since its construction. My name was Elias Thorne\u2014no middle initial\u2014and I kept no logbook for visitors because there were none; only ships passed by, their hulls cutting through the black water as if they too feared what lay beneath it more than above or beside them in fog so thick that even my own reflection vanished from sight when looked upon directly.\n\nI climbed each morning not to greet dawn but check whether light still lived within me\u2014the bulb\u2019s heart had failed twice before, once during a storm named *Harrow*,"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":29,
"drafts_offered":110,
"tok_s":52.2,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in my teeth, searching for something to hold onto until I died.\u201d said Elias Thorne as he adjusted his collar against November\u2019s bite.** He was not merely complaining of cold weather but articulating a poetic metaphor: that even after death (metaphorically), there would be no relief from this relentless gale.\n\nElias had been the last human keeper on any significant stretch coast since automation became standard practice, yet here he stood , stubbornly refusing to let technology replace his vigilance . His cottage perched atop a crumbling granite foundation was more than just shelter; it housed centuries of handwritten logs detailing every ship\u2019s name and cargo load that passed within sight.\n\nOn most nights Elias tended the light mechanically\u2014"
"text":"The phrase \"the History Of The Sea\" is inherently ambiguous, as it could refer to several distinct historical narratives: geological formation (how oceans came into existence), maritime navigation and exploration histories (\"history at/over/across/on/in/of/to/from/toward/off by means around above below through under upon alongside against among between within without throughout despite during after before since until while when where who whom whose which what how why whether\"), or perhaps even metaphorical interpretations involving human interaction with water bodies over time periods like ages eras epochs generations centuries decades years months days hours minutes seconds moments instants times occasions instances events episodes incidents occurrences situations circumstances conditions states phases stages steps processes procedures sequences series chains cycles loops repetitions iterations continuations progressions developments evolutions transformations changes alterations modifications adjustments corrections revisions amendments updates upgrades improvements enhancements refinements optimizations perfecting completing finishing concluding ending terminating ceasing stopping halting pausing resuming continuing proceeding advancing progressing moving shifting translating rotating reflecting mirroring flipping turning twisting bending curving arches spirals helix coils twists turns rotations reflections mirrors flips inversions reversings reversion backtracking retracing reversing undo redo do again repeat repeats repeated repeatedly repetition repetitive repetitiveness repete repeater repeatingly recurrent recurrence recurring recurs recur recurred recursion recursive recursively recurse recessed recessionary regress regression regressed retrograde reverse reversed reversal revert reverted reverberate resonated echo echoed echoing echoes echolocation sonar radar radio telecommunication communication communications communicate communicated communicating communicative comms comm chatter chat chats chatting chatted talk talks talked"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and shadow.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined his life for thirty years.\n\nElias was not a man of many words. He was a man of many tasks. His hands, mapped with scars from broken glass and rusted iron, moved with a rhythm that predated his memory. He checked the clockwork mechanism of the Fresnel lens, ensuring the rotation was smooth, silent, and precise. *Tick. Tick. Tick.* The sound was the heartbeat of the coast."
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":184,
"tok_s":85.2,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots smelled faintly brine and old coffee as he climbed spiral stairs worn smooth by decades feet his hands were calloused from gripping iron railings that had been polished to a dull shine\n\nElias was not merely keeping light alive; He kept order intact. Every night, at precisely 18:07 hours local time\u2014adjusted for leap seconds and magnetic drift\u2014he wound the clockwork mechanism inside brass housing behind Fresnel lens This wasn\u2019t vanity It\u2019s necessity The rotation speed must remain constant or ships lose their bearing in fog where compasses fail\n\nHe spoke softly to himself while working, murmuring numbers aloud because silence made him"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":67,
"repeats":0,
"drafts_accepted":14,
"drafts_offered":68,
"tok_s":41.2,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in his teeth, like he had no mouth to hold back its fury anymore that night.\u201d Elara pulled her wool coat tighter around herself as she climbed spiral staircase of **Blackrock Lighthouse**. Her boots were slick with salt spray from previous days.<|im_end|>"
"text":"The ocean is not merely water; it has been, and remains to this day one giant storybook that weaves together geology with biology as wellas human culture into an epic narrative spanning billions years from its earliest beginnings in primordial times through modern-day ecological crises The vast expanse covers more than seventy percent Earth\u2019s surface area making up nearly all life support systems on our planet yet despite being so large humans have only explored less five thousand square kilometers out total 361 million km\u00b2 which means over ninety-nine point nine eight six per cent unexplored territory This makes oceans arguably larger unknown territories compared space exploration where NASA estimates they\u2019ve mapped roughly ten billion stars versus current marine surveys covering just two hundred fifty-six species documented globally against estimated millions undiscovered organisms including deep-sea creatures like anglerfish or vampire squid whose existence was first recorded centuries ago but still remain poorly understood due extreme pressures depths exceeding eleven miles below Pacific Trenches Mariana located near Guam island chain reaching maximum depth known today at approximately thirty-seven feet under pressure equivalent atmospheric levels causing structural collapse if unprotected by specialized vessels capable handling such conditions without failure mechanisms designed specifically for submersible operations operating within these zones require titanium hulls rated withstand forces greater gravity acting upon objects falling vertically downward acceleration rates calculated using Newtonian physics principles applied universally across gravitational fields regardless medium density variations present throughout different layers stratosphere troposphere mesospheres thermoclines etceterally speaking however when discussing aquatic environments instead air masses replace gases dissolved oxygen nitrogen arg"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy-three years old, and he had been the keeper of the Blackwood Point Light for forty of those years. The other thirty-three had been spent in a small, drafty cottage that smelled perpetually of kerosene and damp wood.\n\nFor decades, Elias had tended the great Fresnel lens, polishing the glass prisms until they gleamed like diamonds, winding the clockwork mechanism that turned the beam, and trimming the wick of the oil lamp. It was a job of patience, of precision, of silence. The light was his charge. The sea was his neighbor.\n\nBut this morning, the sea was not his neighbor. It was his enemy.\n\nThe storm had been forecasted for three days, but it had arrived in one. The barometric pressure had dropped so sharply that Elias\u2019s ears popped."
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":178,
"tok_s":63.7,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots smelled faintly brine and old coffee as he climbed spiral stairs worn smooth by decades feet his hands were calloused from gripping iron railings that had been polished to a dull shine.\n\nElias wasn\u2019t just keeping light; was guarding silence the beam swept across black water every seven seconds cutting through fog so thick it swallowed sound entirely for three nights straight no ships answered its signal not even fishing trawlers seeking shelter they\u2019d long abandoned this stretch of coast where currents shifted unpredictably pulling debris from sunken wrecks into tangled heaps that snagged anchors and tore hulls open like wounds.\n\nHe remembered the last time someone had"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":59,
"repeats":0,
"drafts_accepted":11,
"drafts_offered":46,
"tok_s":33.7,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in his teeth, trying to pry them apart like loose planks of weathered oak outside your own mouth again and then you are going away from home is fine but for now let me write a proper story.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 11",
"tokens":60,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":48,
"tok_s":43.2,
"text":"Aardvark \nAntelope allonax (Oryx) \u2013 *Note: Using standard names for clarity* \n\nLet me provide a clean list of exactly **1** animal name on each single physical text file/terminal output stream with no other characters except the newline (\\n).<|im_end|>"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element\u2014it became civilization itself through its capacity as both barrier (to land) but also conduit between lands\u2014and thus gave rise directly from being part nature\u2019s own evolutionary engine into becoming humanity's greatest historical actor across all time scales since first emergence until now present-day future projections still ongoing today within our current era context window limits allowed here by system constraints set above threshold levels required before output generation begins below line break marker inserted immediately after final punctuation mark ending entire preceding paragraph block structure defined earlier during initial drafting phase prior execution stage completed successfully without errors detected internally via internal validation routines executed automatically upon completion cycle termination signal received end-of-file condition triggered correctly according standard protocol specifications followed strictly adhered-to throughout process flow control logic implemented using structured programming constructs compatible with target runtime environment configuration parameters specified herein explicitly declared beforehand at top-level scope declaration section located near beginning source code file header comments included inline documentation style guidelines observed consistently applied uniformly everywhere else similarly formatted sections appearing later down document body text area scrolling vertically downward directionally aligned left margin justified right edge wrapped around fixed-width column boundaries determined dynamically based available screen real estate measured pixel-per-unit scaling factor adjusted responsively depending viewport dimensions reported back client-side rendering pipeline components responsible"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the fabric stiff with salt and ice. Outside the thick glass of the lantern room, the Atlantic was a churning cauldron of black and white, the waves crashing against the granite base with a violence that shook the very foundations of the tower. It was November, 1924, and the storm had been raging for three days.\n\nElias was not a man who feared the dark. He was a man who feared the silence.\n\nHe checked the clock on the wall. 11:42 PM. The rotation of the great Fresnel lens was steady, a rhythmic *whirrr-click, whirrr-click* that seemed to hold the chaos of the ocean at bay. The beam swept out, a blade of pure, concentrated light cutting through the rain and sleet, illuminating the jagged teeth of the reef below.\n\n"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":179,
"tok_s":81.3,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots were heavy with salt and exhaustion as he climbed one hundred twelve steps to reach his post at dawn every morning for thirty years. His name was written on a brass plaque by door: **Elias Thorne**.\n\nHe wasn\u2019t just keeping light; He kept time, weathered hands gripping the cold iron railing until knuckles turned white against frostbite and windburn that never healed fully because they were always there waiting to be felt again when he stepped outside into storm or calm alike. The lens above him was a Fresnel giant\u2019s eye made of hand-ground crystal prisms each one polished daily with chamois cloth soaked in whale"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":65,
"repeats":0,
"drafts_accepted":11,
"drafts_offered":58,
"tok_s":42.6,
"text":"The wind didn\u2019t just blow at Cape Hollow; it screamed through the gaps in my cottage\u2019s shutters, rattling them like loose teeth until they slammed against their hinges every night for eleven months of winter and summer alike.\u201d I had never seen calm water here\u2014only gray churning seas that swallowed ships smaller than themselves<|im_end|>"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by hundreds millions years yet continues shaping every aspect from climate regulation through evolutionary biology down even into modern industrial logistics today's global supply chains depend upon them directly indirectly via atmospheric pressure systems weather patterns migration routes food webs carbon cycles oxygen generation thermal buffering tidal forces salinity gradients depth stratification light attenuation currents circulation gyres upwelling zones thermoclines halothermics bathymetric features continental shelves abyssal plains trenching ridges volcanic arcs island formations atoll structures seamounts submarine canons canyon walls cliff faces overhang ledge platforms reef flats fore-reef back-lagoon inner-shelf outer-slope drop-offs pinnacle pinnacles spires columns arches tunnels caves grottos swim-through passages chimney vents blowholes fissures cracks crevices joints faults fractures breaks gaps holes openings void spaces hollow chambers cavernous interiors subterranean networks underground rivers lakes pools springs wells shaft pits sinks drains sewers pipes conduits tubes channels gutters ditches furrows grooves rills runnels streams brooks rivulets creeks tributaries branches forks splits divergences con"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the fabric stiff with salt and ice. Outside the thick glass of the lantern room, the Atlantic was a churning cauldron of black and white, the waves crashing against the granite base with a violence that shook the very foundations of the tower. It was November, 1924, and the storm had been raging for three days.\n\nElias was not a man who feared the dark. He was a man who feared the silence.\n\nHe checked the clock on the wall. 11:42 PM. The rotation of the great Fresnel lens was steady, a rhythmic *whirrr-click, whirrr-click* that seemed to hold the chaos of the ocean at bay. The beam swept out, a blade of pure, concentrated light cutting through the rain and sleet, illuminating the jagged teeth of the reef below.\n\n"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":179,
"tok_s":76.2,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots were heavy with salt and exhaustion as he climbed one hundred twelve steps to reach his post at dawn every morning for thirty years. His name was written on a brass plaque by door: **Elias Thorne**.\n\nHe wasn\u2019t just keeping light; He kept time, weathered hands gripping the cold iron railing until knuckles turned white against frostbite and windburn that never healed fully because they were always there waiting to be felt again when he stepped outside into storm or calm alike. The lens above him was a Fresnel giant\u2019s eye made of hand-ground crystal prisms each one polished daily with chamois cloth soaked in whale"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":65,
"repeats":0,
"drafts_accepted":11,
"drafts_offered":58,
"tok_s":42.3,
"text":"The wind didn\u2019t just blow at Cape Hollow; it screamed through the gaps in my cottage\u2019s shutters, rattling them like loose teeth until they slammed against their hinges every night for eleven months of winter and summer alike.\u201d I had never seen calm water here\u2014only gray churning seas that swallowed ships smaller than themselves<|im_end|>"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by hundreds millions years yet continues shaping every aspect from climate regulation through evolutionary biology down even into modern industrial logistics today's global supply chains depend upon them directly indirectly via atmospheric pressure systems weather patterns migration routes food webs carbon cycles oxygen generation thermal buffering tidal forces salinity gradients depth stratification light attenuation currents circulation gyres upwelling zones thermoclines halothermics bathymetric features continental shelves abyssal plains trenching ridges volcanic arcs island formations atoll structures seamounts submarine canons canyon walls cliff faces overhang ledge platforms reef flats fore-reef back-lagoon inner-shelf outer-slope drop-offs pinnacle pinnacles spires columns arches tunnels caves grottos swim-through passages chimney vents blowholes fissures cracks crevices joints faults fractures breaks gaps holes openings void spaces hollow chambers cavernous interiors subterranean networks underground rivers lakes pools springs wells shaft pits sinks drains sewers pipes conduits tubes channels gutters ditches furrows grooves rills runnels streams brooks rivulets creeks tributaries branches forks splits divergences con"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was the last of the line, the only man left on the jagged spine of Widow\u2019s Rock. For forty years, he had tended the light. For forty years, he had listened to the ocean\u2019s endless, hungry whisper.\n\nTonight was different. The storm wasn\u2019t just a storm; it was a *tempest*. The waves didn\u2019t crash; they *shattered*. They threw themselves against the granite base of the lighthouse with the force of a battering ram, sending tremors up through the soles of Elias\u2019s boots.\n\nHe climbed the spiral stairs, one hundred and twelve steps of iron and rust. His hand, calloused and scarred from decades of polishing the brass railing, found the familiar groove in the metal. He didn\u2019t look up. He didn\u2019t look down. He looked *forward*."
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":176,
"tok_s":102.5,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty jail cell below sea level three hundred feet down to where Elias kept his boots dry and ready for emergencies when he had no time even think about changing them.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":140,
"repeats":0,
"drafts_accepted":22,
"drafts_offered":90,
"tok_s":50.1,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in his teeth, demanding entry into every crack and crevice of Stone Harbor Lighthous eKeepers have always been solitary figures against isolation\u2019s greatest adversary: time itself Elias Thorne knew this intimately. For thirty years he had kept watch over one hundred thousand ships each night scanning for lights that might signal disaster or hope\n\nHis job was simple but deadly serious He checked weather logs every day without fail The foghorn echoed through the tower three times before dawn Each morning at 4:02 AM Elias would climb eighty-two steps to reach his office where coffee cooled beside a single candle flickering in drafts from below<|im_end|>"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth\u2019s surface.\u201d \u2014 *This quote by Rachel Carson serves as an apt introduction.* The story we tell today begins with that first drop falling into primordial pools billions years ago\u2014but what follows next? What happens when those drops become oceans themselves\u2014when they begin moving across continents carrying sediment from mountains down rivers until finally reaching their final resting place at last\u2014the deep abyssal plains where sunlight cannot reach nor pressure crushes everything else alive there below depths greater than six kilometers beneath waves crashing against shores above them all around world forever changing shape over time due plate tectonics shifting landmass slowly apart together again repeatedly throughout geological epochs spanning hundreds millions or even thousands million+ yrs since formation early Hadean era through Archeozoic eons onward present Holocene epoch still ongoing now right here under current conditions observed via satellites monitoring global climate patterns affecting currents temperatures salinity levels biodiversity distribution human impacts altering ecosystems directly indirectly causing extinction events locally globally recently historically prehistorically anciently mythologically culturally spiritually philosoph scientifically mathematically physically chem biol geo astro cosmological sociopolitical economic environmental ecological evolutionary genetic linguistic literary artistic musical cinematic theatrical photographic journalistic digital analog virtual real tangible intangible metaphysical ontosemantic semiotic symbolic allegorical narrative descriptive analytical interpretive evaluative critical reflective introspective contemplatory ponderous thoughtful considerate attentive observant perceptive insightful intelligent knowledgeable informed educated literati learned skilled capable"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and green.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined his life for thirty years.\n\nElias was not a man of many words. He was a man of many tasks. His hands, mapped with scars from rope burns and broken glass, moved with a rhythm that predated his memory. He checked the clockwork mechanism of the Fresnel lens, listening for the faint, rhythmic *tick-tick-tick* that signaled the rotation was true. If the rhythm faltered, ships"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":178,
"tok_s":104.1,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty jail cell below sea level to above cloud height at once\u2014no matter how high you climbed or deep into its foundations down through rock and saltwater. It was November again when Elias first noticed that his logbook had begun recording entries he hadn\u2019t written yet: *Storm surge expected Tuesday, but the sky is clear.*\n\nHe stopped writing for a moment to listen\u2014not with ears alone\u2014but by pressing both palms flat against cold iron walls where condensation formed in slow spirals like fingerprints left behind after touching something invisible. The metal hummed faintly beneath his skin as if remembering old storms long past their dates on paper calendars torn from newspapers dated three years ahead of today"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":63,
"tok_s":52.6,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in his teeth, trying to get inside where he had spent thirty years refusing entry except for family and friends like him who were old enough too know better than young men ever could be again about what safety meant. It was November 2014 when Elias Thorne first stepped onto that spiral staircase with only one boot on because his other foot hurt after walking from the harbor office all morning long before anyone else thought he needed shoes off there for work even if nobody saw them do it but everyone knew why so they did not ask him anymore once this happened already twice last winter near same season now here again today too soon then tomorrow maybe never will come back unless someone sends message tonight via radio signal code Morse dot dash pattern"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\u201d \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \u201cthe Sea\u201d or \"Ocean\" (terms often used interchangeably but distinct geographically) was never just an element within our planet\u2019s geography\u2014it became its engine itself\u2014a living system whose evolution shaped everything else around us from climate patterns down through evolutionary biology right up into modern civilization's economic structures cultural identities political boundaries ecological systems scientific understanding technological capabilities philosophical frameworks psychological experiences artistic expressions literary metaphors cinematic narratives musical compositions architectural designs urban planning strategies agricultural practices industrial processes medical discoveries legal precedents educational curricula religious traditions spiritual beliefs mythological stories folkloric tales poetic verses dramatic scripts visual arts photographic images digital media social networks behavioral norms emotional responses cognitive models decision-making algorithms predictive analytics statistical probabilities probabilistic reasoning causal inference counterfactual thinking hypothetical scenarios normative ethics deontal logic modal semantics temporal dynamics spatial relations topographical features hydrological cycles meteor phenomena atmospheric chemistry geological formations tectonic shifts volcanic eruptions seismic events tsunamis hurricanes cyclones monsoons typhoons droughts floods storms winds currents waves swells breakers surf foam spray mist fog rain snow ice glaciers berg floes drift pack-ice brine salinity pH alkalines acidity hardness softness temperature pressure depth light darkness color hue saturation brightness contrast luminance chroma tone value shade tint reflection"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, mapped with veins like the roots of an ancient oak, gripped the cold iron railing.\n\nFor forty years, Elias had tended the light at the end of the world. Not the end of the map, but the end of the land. Here, the Atlantic threw itself against the granite cliffs with such violence that the spray reached the gallery, thirty feet above the rocks.\n\nHe checked the clock. 11:42 PM. The fog was thickening, a gray soup that swallowed the horizon. In the old days, he would have climbed the stairs to the lantern room, polished the Fresnel lens, and wound the clockwork mechanism that turned the great beam. But the automation had changed"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":179,
"tok_s":80.7,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to check what was happening with that storm and why they were calling me out there when all we had left for tonight\u2019s shift is one last pass through our logs before dawn breaks.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":83,
"tok_s":43.9,
"text":"The wind didn\u2019t just blow at Cape Harrow; it screamed through the gaps in its teeth, hungry and relentless for three centuries of stone salt-spray history that had long since forgotten to be polite or gentle with itself against anything smaller than God Himself was willing acknowledge as divine protection from drowning souls who were already lost before they ever touched water.\n\nElias knew this rhythm better he himself could hear it sometimes; his boots worn smooth by pacing the spiral stair\u2019s center-post groove where thousands upon countless steps had rubbed away texture until all that remained beneath sole leather now felt like polished bone carved out slowly but surely over time into something more hollow than wood ever really should be allowed become when used so often enough within such confined circular spaces meant only for vertical ascent down always descending again back upward never truly leaving"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by billions years yet continues shaping every aspect from climate regulation through evolutionary biology all way down modern industrial supply chains today's global shipping lanes carry cargo across vast distances because they were once crossed only slowly via coastal hopping routes used since antiquity until steamships revolutionized travel during late nineteenth century when diesel engines replaced wind power entirely allowing container ships carrying thousands TEUs (Twenty-foot Equivalent Units) per voyage crossing entire Pacific basins without stopping at intermediate ports anymore unlike earlier sailing vessels needing favorable winds currents seasonal patterns dictating departure windows based upon monsoon calendars established centuries ago before technological advances changed those constraints fundamentally altering logistics networks worldwide thereby transforming economic structures globally while simultaneously creating ecological disruptions due invasive species transported unintentionally alongside legitimate goods causing cascading ecosystem collapses locally where non-native organisms outcompete native fauna disrupting food webs leading population declines affecting biodiversity indices measured scientifically using standardized metrics like Shannon-Wiener diversity index calculations applied consistently over decades revealing trends showing significant decreases post-introduction events correl strongly with introduction dates matching historical records documenting first sightings confirmed taxonomically classified formally according International Code Nomenclature rules"
"text":"The wind didn\u2019t just howl; it screamed. It tore through the gaps in the iron railing of the Lantern Room, carrying the scent of brine and rotting kelp. Elias pulled his woolen collar up to his chin, his fingers numb and clumsy inside thick leather gloves.\n\nHe was the last man on the island. The automated systems had failed three winters ago, a glitch in the solar panel array that no engineer had been able to fix before the supply boat stopped running. Now, it was just him, the light, and the sea.\n\nElias climbed the spiral stairs, his boots heavy with salt spray. The iron steps groaned under his weight, a sound that seemed to mock the storm outside. He reached the top, the glass walls vibrating with the force of the gale.\n\nInside the lantern room, the great Fresnel lens sat like a dormant eye. It was cold, dark, and waiting.\n\nElias moved to the control panel. His"
},
{
"name":"code",
"tok_s":103.6,
"reps":[
103.6
],
"hit_rate":0.957731667226356,
"text":"```python\nimport csv\nfrom typing import List, Dict\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, str]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False"
}
],
"stage_timing":[],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"strata generate: layer split: CUDA1 holds its weights, session and head; 18.13 GiB free",
"strata generate: layer split auto: CUDA0 84 SMs at 2.81 GHz -> 0.31 ms per layer, 8.30 GiB for experts",
"strata generate: layer split auto: CUDA1 82 SMs at 1.86 GHz -> 0.48 ms per layer, 14.14 GiB for experts",
"strata generate: layer split auto: K=22 - predicted 19.7 ms per decode window; the caches hold 11298 of 12288 profiled pairs (~99.7% of the routed mass)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the fabric stiff with salt and ice. Outside the thick glass of the lantern room, the Atlantic was a churning cauldron of black and white, the waves crashing against the granite base with a violence that shook the very foundations of the tower. It was November, 1924, and the storm had been raging for three days.\n\nElias was not a man who feared the dark. He was a man who feared the silence.\n\nHe checked the clock on the wall. 11:42 PM. The rotation of the great Fresnel lens was steady, a rhythmic *whirrr-click, whirrr-click* that seemed to sync with his own heartbeat. He had been doing this for forty years. His hands were mapped with scars from rope burns and broken glass, his eyes permanently squinted against the glare of the bulb."
},
{
"name":"code",
"tok_s":74.0,
"reps":[
74.0
],
"hit_rate":0.5873783774988882,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and shadow.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined Elias\u2019s life for thirty years. He stood before the great Fresnel lens, a cathedral of glass and brass that caught the light of the single, powerful bulb and fractured it into a thousand beams.\n\nHe checked the clock. 11:42 PM. The rotation was smooth, silent, a hypnotic sweep that painted the darkness with a slow, rhythmic pulse. *Flash. Dark. Flash. Dark.*\n\nE"
},
{
"name":"code",
"tok_s":97.0,
"reps":[
97.0
],
"hit_rate":0.9562207324057541,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"strata generate: layer split: CUDA1 holds its weights, session and head; 17.96 GiB free",
"strata generate: layer split auto: CUDA0 84 SMs at 2.81 GHz -> 0.31 ms per layer, 8.13 GiB for experts",
"strata generate: layer split auto: CUDA1 82 SMs at 1.86 GHz -> 0.48 ms per layer, 13.98 GiB for experts",
"strata generate: layer split auto: K=22 - predicted 19.7 ms per decode window; the caches hold 11199 of 12288 profiled pairs (~99.7% of the routed mass)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, rough as sandpaper, gripped the iron railing as he ascended the final flight of steps to the lantern room.\n\nBelow him, the Atlantic churned in a frenzy, whitecaps exploding against the black rocks like fireworks. Above, the sky was a bruised purple, heavy with rain that hadn\u2019t yet fallen.\n\nElias checked the clock. 11:42 PM. The light was due.\n\nHe didn\u2019t flip a switch. He didn\u2019t press a button. He reached for the heavy brass lever that had worn smooth by his palm over the years. With a grunt that matched the strain of his aging muscles, he pulled it down.\n\n*Clunk.*\n\nThe"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":181,
"tok_s":105.5,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to that date and time for context because you asked me about a lighthouse keeper but gave no specifics beyond \u201cshort story,\u201d so let\u2019s set it: isolated coastal station off the Oregon coast during winter storm season, 1987.\n\nMy name is Elias Vane (I made this fictional; real keepers often kept journals signed with initials only), and I\u2019m writing from memory because my logbook got waterlogged when a rogue wave breached lower storage room at dawn\u2014so here\u2019s what happened:<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":109,
"repeats":0,
"drafts_accepted":13,
"drafts_offered":56,
"tok_s":54.1,
"text":"The wind didn\u2019t just blow at Cape Hollow; it screamed through the gaps in my cottage\u2019s shutters, rattling them like loose teeth until they settled into silence with every storm break out of three hundred days here being storms that broke something else again but mostly kept me warm because I burned coal and wood whenever we had any left over from last winter when even then if you\u2019re lucky enough to get through it all without freezing yourself so hard your own thoughts become ice too\n\nBut Elias knew better than anyone: fear was useful only until dawn<|im_end|>"
"text":"The ocean is not merely water; it has been, and continues to be one with humanity\u2019s most profound narrative arc in human civilization: **the story we tell ourselves through our relationship** between landmasses separated by vast expansiveness that defines both isolation from other worlds (and thus their own unique evolutionary paths) as well than connection across them via currents wind patterns tides depth pressure temperature salinity biodiversity geology climate change economic trade routes ecological systems cultural exchange political boundaries scientific discovery technological advancement environmental degradation conservation efforts maritime archaeography historical cartographic evolution legal frameworks international law enforcement mechanisms insurance liability regimes regulatory compliance standards safety protocols navigation aids communication networks emergency response strategies search-and-rescue operations salvage rights ownership disputes heritage protection zones sustainable harvesting practices aquaculture development infrastructure engineering projects coastal management policies urban planning regulations zoning ordinances building codes architectural design guidelines aesthetic considerations functional requirements accessibility provisions energy efficiency measures thermal insulation techniques moisture control methods ventilation optimization lighting solutions acoustic soundproofing structural integrity assessments seismic resistance evaluations flood mitigation plans storm surge barriers levee construction seawall reinforcement breakwater installation harbor dredging channel maintenance buoy placement lighthouse automation radar monitoring sonar mapping bathymetric surveys hydrographic charts nautical almanacs tide tables current predictions weather forecasts marine meteorological models satellite imagery remote sensing data assimilation numerical simulations computational fluid dynamics finite element analysis partial differential equations tensor calculus linear algebra probability theory statistics stochastic processes Markov chains Bayesian inference machine learning artificial neural network deep convolutional recurrent transformer attention heads generative adversarial diffusion transformers large language model tokenization"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, rough as sandpaper, gripped the iron railing as he ascended the final flight of steps to the lantern room.\n\nBelow him, the Atlantic churned in a frenzy, whitecaps exploding against the black rocks like fireworks. Above, the sky was a bruised purple, heavy with rain that hadn\u2019t yet fallen.\n\nElias checked the clock. 11:42 PM. The light was due.\n\nHe didn\u2019t flip a switch. He didn\u2019t press a button. He reached for the heavy brass lever that had worn smooth by his palm over the years. With a grunt that matched the strain of his aging muscles, he pulled it down.\n\n*Clunk.*\n\nThe"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":181,
"tok_s":99.4,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to that date and time for context because you asked me about a lighthouse keeper but gave no specifics beyond \u201cshort story,\u201d so let\u2019s set it: isolated coastal station off the Oregon coast during winter storm season, 1987.\n\nMy name is Elias Vane (I made this fictional; real keepers often kept journals signed with initials only), and I\u2019m writing from memory because my logbook got waterlogged when a rogue wave breached lower storage room at dawn\u2014so here\u2019s what happened:<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":109,
"repeats":0,
"drafts_accepted":13,
"drafts_offered":56,
"tok_s":42.1,
"text":"The wind didn\u2019t just blow at Cape Hollow; it screamed through the gaps in my cottage\u2019s shutters, rattling them like loose teeth until they settled into silence with every storm break out of three hundred days here being storms that broke something else again but mostly kept me warm because I burned coal and wood whenever we had any left over from last winter when even then if you\u2019re lucky enough to get through it all without freezing yourself so hard your own thoughts become ice too\n\nBut Elias knew better than anyone: fear was useful only until dawn<|im_end|>"
"text":"The ocean is not merely water; it has been, and continues to be one with humanity\u2019s most profound narrative arc in human civilization: **the story we tell ourselves through our relationship** between landmasses separated by vast expansiveness that defines both isolation from other worlds (and thus their own unique evolutionary paths) as well than connection across them via currents wind patterns tides depth pressure temperature salinity biodiversity geology climate change economic trade routes ecological systems cultural exchange political boundaries scientific discovery technological advancement environmental degradation conservation efforts maritime archaeography historical cartographic evolution legal frameworks international law enforcement mechanisms insurance liability regimes regulatory compliance standards safety protocols navigation aids communication networks emergency response strategies search-and-rescue operations salvage rights ownership disputes heritage protection zones sustainable harvesting practices aquaculture development infrastructure engineering projects coastal management policies urban planning regulations zoning ordinances building codes architectural design guidelines aesthetic considerations functional requirements accessibility provisions energy efficiency measures thermal insulation techniques moisture control methods ventilation optimization lighting solutions acoustic soundproofing structural integrity assessments seismic resistance evaluations flood mitigation plans storm surge barriers levee construction seawall reinforcement breakwater installation harbor dredging channel maintenance buoy placement lighthouse automation radar monitoring sonar mapping bathymetric surveys hydrographic charts nautical almanacs tide tables current predictions weather forecasts marine meteorological models satellite imagery remote sensing data assimilation numerical simulations computational fluid dynamics finite element analysis partial differential equations tensor calculus linear algebra probability theory statistics stochastic processes Markov chains Bayesian inference machine learning artificial neural network deep convolutional recurrent transformer attention heads generative adversarial diffusion transformers large language model tokenization"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, rough as sandpaper, knew every bolt, every lever, and every creak in the iron bones of the tower.\n\nFor forty years, Elias had been the guardian of the Point. Not just of the light, but of the silence between the waves.\n\nIt was November. The sky was a bruised purple, heavy with rain that hadn\u2019t yet fallen. The sea was a churning cauldron of grey and white, throwing itself against the granite base with a violence that shook the floorboards beneath Elias\u2019s boots.\n\nHe climbed the stairs. One hundred and twelve steps. He counted them by feel, his left hand trailing along the cold iron railing. At the top, the air was"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":137,
"drafts_offered":178,
"tok_s":103.4,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots were heavy with salt and exhaustion as he climbed one hundred twelve steps to reach top room where his world began again.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":58,
"repeats":0,
"drafts_accepted":14,
"drafts_offered":54,
"tok_s":52.9,
"text":"The wind didn\u2019t just blow at Cape Harrow; it screamed through the gaps in stone and steel, carrying with them spray from three miles out into churning black water that no longer resembled an ocean so much as abyssal chaos waiting for its prey to surface beneath waves of salt.<|im_end|>"
"text":"The ocean is not merely water; it has been, and continues to be one with humanity\u2019s most profound narrative arc in human civilization: **the story told by waves**. It serves as both witness (through sedimentary layers)and participant( through currents that carry messages across continents). Its \"history\" spans geological epochs where life first emerged from primordial pools into complex ecosystems today\u2014and cultural histories shaped entirely around its roleas connector between distant lands or barrier separating them forever depending on perspective taken at different times throughout millennia since ancient civilizations began building boats outof logs carved using stone tools until modern container ships carrying billions tons cargo daily cross Atlantic routes once navigated onlyby stars alone under moonlight during nights when sailors feared monsters lurking beneath hulls made fragile against storms they could barely predict even now despite advanced meteorology systems developed over past century after industrial revolution transformed maritime trade patterns globally due economic pressures driving demand for faster logistics chains linking factories producing electronics manufactured inside cities located far inland near rivers feeding estuaries opening directly onto open seas connected via canals engineered centuries ago like Suez Canal built originallyfor military purposes later adapted commercial use post-World War II era reshaping global supply networks dramatically because geopolitical shifts altered which ports remained accessible politically versus physically blocked temporarily then reopened permanently following treaty agreements signed among nations agreeing upon shared responsibilities regarding environmental stewardship including coral reef restoration efforts funded jointly international organizations coordinating scientific research conducted aboard vessels specifically designed deep-sea exploration equipped robotic submersibles capable descending trenches deeper than any previous technology allowed"
"text":"The wind didn\u2019t just blow; it screamed. It tore through the iron gallery of the Blackwood Point Lighthouse, rattling the glass panes like a prisoner shaking its bars.\n\nElias Thorne was seventy years old, and he had been the keeper for forty of them. He knew every creak in the spiral staircase, every draft that slipped under the door, and every shift in the weather that the barometer couldn\u2019t quite predict. He knew that when the fog rolled in thick enough to swallow the horizon, the world ceased to exist. There was only the light, the stone, and him.\n\nIt was November, the month of storms. The sea was a churning cauldron of grey and white, throwing itself against the granite base with a violence that made the floorboards tremble. Elias stood by the lantern room, his hands wrapped around a mug of coffee that had gone cold hours ago. He wasn\u2019t drinking it. He was holding it for warmth."
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":137,
"drafts_offered":163,
"tok_s":70.7,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv(file_path: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys being the header values\n and values being the corresponding cell values.\n \n Args:\n file_path: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(file_path, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_basic():\n \"\"\"Test basic CSV parsing with simple data.\"\"\"\n import tempfile\n import os\n \n # Create a temporary CSV"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots were heavy with salt and exhaustion as he climbed one hundred twelve steps to reach topmost gallery where his world began.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":30,
"drafts_offered":111,
"tok_s":34.7,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in his stone cottage like an angry ghost trying to get inside, yet finding no door left unlocked because Elias never let one stay that way for more than three seconds after entering or leaving during storm season. He had been doing this since 1987\u2014since before he even knew why\u2014and now it was muscle memory so deep into his bones and fingers\u2019 calloused skin texture patterns worn thin by years of gripping the same iron railing at midnight watch that if you asked him what time exactly, down to quarter-hour increments when fog rolled in thick enough where light beams bent around itself like old rope coiling tight on a winch drum spooling out slowly from below deck floor level storage locker number seven under stairs leading"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\u201d \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \u201cthe Sea\u201d or \"Ocean\" (terms often used interchangeably but distinct geographically) was never just an element within our planet\u2019s geography\u2014it became its engine itself\u2014a living system whose existence predates humanity by billions years yet continues shaping us today through climate regulation oxygen production food supply trade routes cultural identity ecological resilience economic development political boundaries scientific discovery technological innovation environmental crisis conservation efforts historical memory philosophical reflection poetic metaphor musical composition cinematic storytelling literary narrative visual art architectural inspiration culinary tradition spiritual symbolism psychological influence sociological transformation educational foundation medical necessity agricultural dependency industrial revolution urban planning infrastructural engineering legal framework regulatory policy international cooperation global governance planetary stewardship existential risk mitigation sustainable practice ethical consideration moral imperative social justice democratic participation civic engagement public health community welfare individual responsibility personal accountability collective action shared experience common understanding mutual respect reciprocal trust collaborative effort cooperative endeavor joint venture unified mission integrated strategy coordinated plan synchronized schedule aligned timeline concurrent execution simultaneous operation parallel processing sequential progression iterative refinement recursive improvement continuous adaptation dynamic adjustment flexible response adaptive behavior responsive conduct proactive initiative anticipatory foresight predictive modeling probabilistic assessment statistical analysis empirical observation experimental verification theoretical hypothesis conceptual abstraction logical deduction rational inference critical evaluation analytical reasoning systematic investigation methodical inquiry structured exploration comprehensive examination detailed scrutiny meticulous inspection thorough review exhaustive audit complete survey full census total enumeration absolute"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, mapped with veins like the roots of an ancient oak, gripped the cold iron railing.\n\nFor forty years, Elias had tended the light at the end of the world. Not the end of the map, but the end of the land. Here, the Atlantic threw itself against the granite cliffs with such violence that the spray reached the gallery, thirty feet above the rocks.\n\nHe checked the clock. 11:42 PM. The fog was thickening, a gray soup that swallowed the horizon. In the old days, he would have climbed the stairs to the lantern room, polished the Fresnel lens, and wound the clockwork mechanism that turned the great beam. But the automation had changed"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":179,
"tok_s":81.6,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to check what was happening with that storm and why they were calling me out there when all we had left for tonight\u2019s shift is one last pass through our logs before dawn breaks.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":83,
"tok_s":44.1,
"text":"The wind didn\u2019t just blow at Cape Harrow; it screamed through the gaps in its teeth, hungry and relentless for three centuries of stone salt-spray history that had long since forgotten to be polite or gentle with itself against anything smaller than God Himself was willing acknowledge as divine protection from drowning souls who were already lost before they ever touched water.\n\nElias knew this rhythm better he himself could hear it sometimes; his boots worn smooth by pacing the spiral stair\u2019s center-post groove where thousands upon countless steps had rubbed away texture until all that remained beneath sole leather now felt like polished bone carved out slowly but surely over time into something more hollow than wood ever really should be allowed become when used so often enough within such confined circular spaces meant only for vertical ascent down always descending again back upward never truly leaving"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by billions years yet continues shaping every aspect from climate regulation through evolutionary biology all way down modern industrial supply chains today's global shipping lanes carry cargo across vast distances because they were once crossed only slowly via coastal hopping routes used since antiquity until steamships revolutionized travel during late nineteenth century when diesel engines replaced wind power entirely allowing container ships carrying thousands TEUs (Twenty-foot Equivalent Units) per voyage crossing entire Pacific basins without stopping at intermediate ports anymore unlike earlier sailing vessels needing favorable winds currents seasonal patterns dictating departure windows based upon monsoon calendars established centuries ago before technological advances changed those constraints fundamentally altering logistics networks worldwide thereby transforming economic structures globally while simultaneously creating ecological disruptions due invasive species transported unintentionally alongside legitimate goods causing cascading ecosystem collapses locally where non-native organisms outcompete native fauna disrupting food webs leading population declines affecting biodiversity indices measured scientifically using standardized metrics like Shannon-Wiener diversity index calculations applied consistently over decades revealing trends showing significant decreases post-introduction events correl strongly with introduction dates matching historical records documenting first sightings confirmed taxonomically classified formally according International Code Nomenclature rules"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, mapped with veins like the roots of an ancient oak, gripped the cold iron railing.\n\nFor forty years, Elias had tended the light at the end of the world. Not the end of the map, but the end of the land. Here, the Atlantic threw itself against the granite cliffs with such violence that the spray reached the gallery, thirty feet above the rocks.\n\nHe checked the clock. 11:42 PM. The fog was thickening, a gray soup that swallowed the horizon. In the old days, he would have climbed the stairs to the lantern room, polished the Fresnel lens, and wound the clockwork mechanism that turned the great beam. But the automation had changed"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":179,
"tok_s":80.6,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to check what was happening with that storm and why they were calling me out there when all we had left for tonight\u2019s shift is one last pass through our logs before dawn breaks.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":83,
"tok_s":43.3,
"text":"The wind didn\u2019t just blow at Cape Harrow; it screamed through the gaps in its teeth, hungry and relentless for three centuries of stone salt-spray history that had long since forgotten to be polite or gentle with itself against anything smaller than God Himself was willing acknowledge as divine protection from drowning souls who were already lost before they ever touched water.\n\nElias knew this rhythm better he himself could hear it sometimes; his boots worn smooth by pacing the spiral stair\u2019s center-post groove where thousands upon countless steps had rubbed away texture until all that remained beneath sole leather now felt like polished bone carved out slowly but surely over time into something more hollow than wood ever really should be allowed become when used so often enough within such confined circular spaces meant only for vertical ascent down always descending again back upward never truly leaving"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by billions years yet continues shaping every aspect from climate regulation through evolutionary biology all way down modern industrial supply chains today's global shipping lanes carry cargo across vast distances because they were once crossed only slowly via coastal hopping routes used since antiquity until steamships revolutionized travel during late nineteenth century when diesel engines replaced wind power entirely allowing container ships carrying thousands TEUs (Twenty-foot Equivalent Units) per voyage crossing entire Pacific basins without stopping at intermediate ports anymore unlike earlier sailing vessels needing favorable winds currents seasonal patterns dictating departure windows based upon monsoon calendars established centuries ago before technological advances changed those constraints fundamentally altering logistics networks worldwide thereby transforming economic structures globally while simultaneously creating ecological disruptions due invasive species transported unintentionally alongside legitimate goods causing cascading ecosystem collapses locally where non-native organisms outcompete native fauna disrupting food webs leading population declines affecting biodiversity indices measured scientifically using standardized metrics like Shannon-Wiener diversity index calculations applied consistently over decades revealing trends showing significant decreases post-introduction events correl strongly with introduction dates matching historical records documenting first sightings confirmed taxonomically classified formally according International Code Nomenclature rules"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, mapped with veins like the roots of an ancient oak, gripped the cold iron railing.\n\nFor forty years, Elias had tended the light at the end of the world. Not the end of the map, but the end of the land. Here, the Atlantic threw itself against the granite cliffs with such violence that the spray reached the gallery, thirty feet above the rocks.\n\nHe checked the clock. 11:42 PM. The fog was thickening, a gray soup that swallowed the horizon. In the old days, he would have climbed the stairs to the lantern room, polished the Fresnel lens, and wound the clockwork mechanism that turned the great beam. But the automation had changed"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":179,
"tok_s":80.5,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to check what was happening with that storm and why they were calling me out there when all we had left for tonight\u2019s shift is one last pass through our logs before dawn breaks.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":83,
"tok_s":44.3,
"text":"The wind didn\u2019t just blow at Cape Harrow; it screamed through the gaps in its teeth, hungry and relentless for three centuries of stone salt-spray history that had long since forgotten to be polite or gentle with itself against anything smaller than God Himself was willing acknowledge as divine protection from drowning souls who were already lost before they ever touched water.\n\nElias knew this rhythm better he himself could hear it sometimes; his boots worn smooth by pacing the spiral stair\u2019s center-post groove where thousands upon countless steps had rubbed away texture until all that remained beneath sole leather now felt like polished bone carved out slowly but surely over time into something more hollow than wood ever really should be allowed become when used so often enough within such confined circular spaces meant only for vertical ascent down always descending again back upward never truly leaving"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by billions years yet continues shaping every aspect from climate regulation through evolutionary biology all way down modern industrial supply chains today's global shipping lanes carry cargo across vast distances because they were once crossed only slowly via coastal hopping routes used since antiquity until steamships revolutionized travel during late nineteenth century when diesel engines replaced wind power entirely allowing container ships carrying thousands TEUs (Twenty-foot Equivalent Units) per voyage crossing entire Pacific basins without stopping at intermediate ports anymore unlike earlier sailing vessels needing favorable winds currents seasonal patterns dictating departure windows based upon monsoon calendars established centuries ago before technological advances changed those constraints fundamentally altering logistics networks worldwide thereby transforming economic structures globally while simultaneously creating ecological disruptions due invasive species transported unintentionally alongside legitimate goods causing cascading ecosystem collapses locally where non-native organisms outcompete native fauna disrupting food webs leading population declines affecting biodiversity indices measured scientifically using standardized metrics like Shannon-Wiener diversity index calculations applied consistently over decades revealing trends showing significant decreases post-introduction events correl strongly with introduction dates matching historical records documenting first sightings confirmed taxonomically classified formally according International Code Nomenclature rules"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning black void, punctuated by whitecaps that looked like teeth.\n\nHe climbed the spiral stairs, his knees protesting with every step. The iron rungs were cold, slick with condensation that had leaked through the hairline cracks in the masonry. At the top, the lantern room hummed with the low, steady thrum of the rotating lens. It was a sound Elias had learned to sleep with, a heartbeat of the house.\n\nHe checked the clock. 11:42 PM. The light swept across the water, a slow, hypnotic arc. *Flash. Darkness. Flash. Darkness.*\n\nFor a moment, the beam caught something that wasn\u2019t water. A shape"
},
{
"name":"code",
"tok_s":92.9,
"hit_rate":0.8636411400016346,
"text":"```python\nimport csv\nfrom typing import List, Dict\n\n\ndef parse_csv(filepath: str) -> List[Dict[str, str]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header row\n and values from the corresponding row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(filepath, 'r', newline='') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n```<|im_end|>"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning cauldron of black ink and white foam, throwing itself against the granite base with the force of a battering ram.\n\nInside the lantern room, the air was thick with the smell of kerosene and old brass. Elias wiped his hands on his woolen trousers, leaving faint, oily smears on the fabric. He checked the clock. 11:42 PM. The rotation was due.\n\nHe reached for the crank handle, a heavy iron wheel worn smooth by decades of his own palms. With a grunt that matched the strain of his shoulders, he began to turn it. *Click. Click. Click.* The gears engaged, and the massive Fresnel lens began its slow"
},
{
"name":"code",
"tok_s":88.0,
"hit_rate":0.7011385968111186,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, the glass was vibrating like a plucked cello string. The sea, usually a rhythmic pulse against the granite base, was now a chaotic, frothing beast, throwing itself against the rocks with the force of a battering ram.\n\nElias climbed the spiral stairs, his hand gripping the cold iron railing. His knees ached, not from age, but from the constant vibration of the floor beneath his feet. He reached the lantern room, the heart of the lighthouse, and checked the clockwork mechanism. The great lens, a prism of Fresnel glass, rotated with a steady, hypnotic *whirrr-click, whirrr-click*. It was the only sound that mattered. The light swept out into the black void, cutting through"
},
{
"name":"code",
"tok_s":103.8,
"reps":[
97.6,
106.4,
107.3
],
"hit_rate":0.9984863945578232,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 524 windows; per window: wait for the GPU 6.456 ms, pool + plan 0.181 ms, host staging 0.581 ms, commit 0.206 ms",
"strata serve: stage 1: 524 windows; per window: wait for the GPU 14.872 ms, pool + plan 0.501 ms, host staging 0.000 ms, commit 0.387 ms",
"strata serve: stage 0: 593 windows; per window: wait for the GPU 6.479 ms, pool + plan 0.185 ms, host staging 0.515 ms, commit 0.207 ms",
"strata serve: stage 1: 593 windows; per window: wait for the GPU 15.053 ms, pool + plan 0.496 ms, host staging 0.000 ms, commit 0.390 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 18)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, the glass was vibrating like a plucked cello string. The sea, usually a rhythmic pulse against the granite base, was now a chaotic, frothing beast, throwing itself against the rocks with the force of a battering ram.\n\nElias stood at the top of the spiral staircase, his hand resting on the cold iron railing. He was not afraid. Fear was for the tourists who came in summer, wide-eyed and clutching their cameras. Elias was a keeper. He was a guardian.\n\nHe checked the logbook. The entry for the night was simple: *Wind, 60 knots. Visibility, zero. Light, operational.*\n\nHe walked down the stairs, his boots heavy on the metal grating. The smell of salt"
},
{
"name":"code",
"tok_s":107.4,
"reps":[
98.9,
111.3,
112.1
],
"hit_rate":0.9986286196529353,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header row\n and values from the corresponding row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with"
}
],
"stage_timing":[
"strata serve: stage 0: 533 windows; per window: wait for the GPU 6.226 ms, pool + plan 0.294 ms, host staging 0.561 ms, commit 0.225 ms",
"strata serve: stage 1: 533 windows; per window: wait for the GPU 14.156 ms, pool + plan 0.373 ms, host staging 0.000 ms, commit 0.371 ms",
"strata serve: stage 0: 599 windows; per window: wait for the GPU 6.314 ms, pool + plan 0.292 ms, host staging 0.501 ms, commit 0.225 ms",
"strata serve: stage 1: 599 windows; per window: wait for the GPU 14.366 ms, pool + plan 0.375 ms, host staging 0.000 ms, commit 0.374 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 19)",
"text":"The wind didn\u2019t just blow; it screamed. It clawed at the granite base of the tower, trying to pry it from the bedrock, and it howled through the narrow slits of the lantern room glass.\n\nElias Thorne was seventy years old, but he moved with the grace of a man who had spent fifty years listening to the sea. He knew the rhythm of the waves against the rocks below\u2014the *thud, thud, crash*\u2014and he knew when that rhythm changed. When the sea grew too quiet, or too violent, he knew a storm was coming, or a ship was in trouble.\n\nTonight, the storm was a Category 4. The rain lashed against the windows like gravel thrown by a giant. Elias stood in the center of the lantern room, his hand resting on the cold brass railing. He wasn\u2019t afraid. He was a guardian.\n\nHe checked the clockwork mechanism. It was old, a relic from 1"
},
{
"name":"code",
"tok_s":105.9,
"reps":[
103.2,
106.8,
107.7
],
"hit_rate":0.9981849807154201,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 537 windows; per window: wait for the GPU 6.685 ms, pool + plan 0.403 ms, host staging 0.593 ms, commit 0.215 ms",
"strata serve: stage 1: 537 windows; per window: wait for the GPU 13.928 ms, pool + plan 0.282 ms, host staging 0.000 ms, commit 0.359 ms",
"strata serve: stage 0: 606 windows; per window: wait for the GPU 6.764 ms, pool + plan 0.399 ms, host staging 0.528 ms, commit 0.216 ms",
"strata serve: stage 1: 606 windows; per window: wait for the GPU 14.108 ms, pool + plan 0.288 ms, host staging 0.000 ms, commit 0.362 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 20)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and shadow.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined his life for thirty years.\n\nElias was not a man of many words. He was a man of many tasks. His hands, mapped with scars from rope burns and broken glass, moved with a rhythm that predated his memory. He checked the clockwork mechanism of the Fresnel lens, ensuring the rotation was smooth, silent, and steady. *Tick. Tick. Tick.* The sound was the heartbeat of the coast.\n\n"
},
{
"name":"code",
"tok_s":106.7,
"reps":[
100.6,
110.5,
109.0
],
"hit_rate":0.998218711927068,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 527 windows; per window: wait for the GPU 6.934 ms, pool + plan 0.620 ms, host staging 0.530 ms, commit 0.245 ms",
"strata serve: stage 1: 527 windows; per window: wait for the GPU 13.912 ms, pool + plan 0.365 ms, host staging 0.000 ms, commit 0.359 ms",
"strata serve: stage 0: 595 windows; per window: wait for the GPU 7.031 ms, pool + plan 0.595 ms, host staging 0.472 ms, commit 0.245 ms",
"strata serve: stage 1: 595 windows; per window: wait for the GPU 14.051 ms, pool + plan 0.361 ms, host staging 0.000 ms, commit 0.361 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 21)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy-three years old, and he had been the keeper of the Blackwood Point Light for forty of those years. The other thirty-three had been spent in a small, drafty cottage on the mainland, waiting for the day when the sea would finally claim him.\n\nTonight, the storm was a Category 4. The waves weren\u2019t just crashing; they were climbing. They reached the gallery railing, ten feet above the base of the tower, spraying a fine mist of brine that coated Elias\u2019s glasses.\n\nHe climbed the spiral stairs, his knees protesting with every step. The iron steps were cold, slick with condensation. At the top, in the lantern room, the air was thin and sharp. The great Fresnel lens, a cathedral of glass and brass, stood silent. It was time to wake it.\n\n"
},
{
"name":"code",
"tok_s":106.4,
"reps":[
100.7,
110.3,
108.1
],
"hit_rate":0.9971798370756316,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 529 windows; per window: wait for the GPU 7.229 ms, pool + plan 0.561 ms, host staging 0.529 ms, commit 0.233 ms",
"strata serve: stage 1: 529 windows; per window: wait for the GPU 13.058 ms, pool + plan 0.253 ms, host staging 0.000 ms, commit 0.334 ms",
"strata serve: stage 0: 598 windows; per window: wait for the GPU 7.325 ms, pool + plan 0.552 ms, host staging 0.469 ms, commit 0.235 ms",
"strata serve: stage 1: 598 windows; per window: wait for the GPU 13.238 ms, pool + plan 0.260 ms, host staging 0.000 ms, commit 0.337 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 22)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning black void, swallowing the horizon, throwing white foam against the granite base like a tantrum.\n\nInside the lantern room, the air was thick with the smell of kerosene and old brass. Elias wiped his hands on his wool trousers, the fabric stiff with salt and grease. He checked the clockwork mechanism that turned the lens. It was a beast of a thing, a Fresnel lens of eight feet in diameter, a cathedral of glass and crystal that focused the beam of the lamp into a single, piercing needle of light.\n\nHe wound the weights. *Click, click, click.* The gears engaged. The massive lens began its slow, hypnotic rotation.\n\nFor a moment, the"
},
{
"name":"code",
"tok_s":108.4,
"reps":[
105.1,
109.0,
111.1
],
"hit_rate":0.9961698810290479,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 537 windows; per window: wait for the GPU 7.423 ms, pool + plan 0.953 ms, host staging 0.542 ms, commit 0.237 ms",
"strata serve: stage 1: 537 windows; per window: wait for the GPU 12.223 ms, pool + plan 0.236 ms, host staging 0.000 ms, commit 0.324 ms",
"strata serve: stage 0: 606 windows; per window: wait for the GPU 7.519 ms, pool + plan 0.904 ms, host staging 0.483 ms, commit 0.239 ms",
"strata serve: stage 1: 606 windows; per window: wait for the GPU 12.395 ms, pool + plan 0.242 ms, host staging 0.000 ms, commit 0.326 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 23)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and shadow.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined Elias\u2019s life for thirty years. He was not merely a keeper; he was a guardian of silence.\n\nHe checked the clock. 11:42 PM. The rotation of the great Fresnel lens was due. With a practiced hand, he engaged the brake on the clockwork mechanism. The heavy brass gears groaned, then began their slow, rhythmic turn. *Tick. Tick. Tick.* It was a heartbeat"
},
{
"name":"code",
"tok_s":109.5,
"reps":[
104.9,
111.0,
112.6
],
"hit_rate":0.9953796671318199,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 519 windows; per window: wait for the GPU 7.835 ms, pool + plan 0.767 ms, host staging 0.557 ms, commit 0.244 ms",
"strata serve: stage 1: 519 windows; per window: wait for the GPU 11.922 ms, pool + plan 0.232 ms, host staging 0.000 ms, commit 0.325 ms",
"strata serve: stage 0: 587 windows; per window: wait for the GPU 7.943 ms, pool + plan 0.744 ms, host staging 0.497 ms, commit 0.246 ms",
"strata serve: stage 1: 587 windows; per window: wait for the GPU 12.073 ms, pool + plan 0.238 ms, host staging 0.000 ms, commit 0.326 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 24)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and green.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined Elias\u2019s life for thirty years. He was a keeper of the light, but more importantly, he was a keeper of time.\n\nHe checked his pocket watch. 11:42 PM. The rotation of the great Fresnel lens was due. With a practiced, rhythmic *clunk-clunk-clunk*, he wound the clockwork mechanism. The massive lens began its slow, hypnotic turn, sweeping its beam across"
},
{
"name":"code",
"tok_s":109.7,
"reps":[
104.4,
112.7,
111.9
],
"hit_rate":0.9938670271352759,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 519 windows; per window: wait for the GPU 8.376 ms, pool + plan 1.030 ms, host staging 0.566 ms, commit 0.273 ms",
"strata serve: stage 1: 519 windows; per window: wait for the GPU 10.886 ms, pool + plan 0.214 ms, host staging 0.000 ms, commit 0.297 ms",
"strata serve: stage 0: 588 windows; per window: wait for the GPU 8.506 ms, pool + plan 0.985 ms, host staging 0.505 ms, commit 0.274 ms",
"strata serve: stage 1: 588 windows; per window: wait for the GPU 11.036 ms, pool + plan 0.219 ms, host staging 0.000 ms, commit 0.298 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 26)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, the glass was vibrating like a plucked cello string. The sea, usually a rhythmic pulse against the granite base, was now a chaotic, frothing beast, throwing itself against the rocks with the force of a battering ram.\n\nElias climbed the spiral stairs, his hand gripping the cold iron railing. His knees ached, not from age, but from the constant vibration of the floor beneath his feet. He reached the lantern room, the heart of the lighthouse, and checked the clockwork mechanism. The great lens, a prism of Fresnel glass, rotated with a steady, hypnotic *whirrr-click, whirrr-click*. It was the only sound that mattered. The light swept out into the black void, cutting through"
},
{
"name":"code",
"tok_s":112.5,
"reps":[
101.4,
112.1,
124.1
],
"hit_rate":0.9896636587366694,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv(file_path: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys being the header values\n and values being the corresponding cell values.\n \n Args:\n file_path: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(file_path, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n```<|im_end|>"
}
],
"stage_timing":[
"strata serve: stage 0: 527 windows; per window: wait for the GPU 8.765 ms, pool + plan 1.401 ms, host staging 0.593 ms, commit 0.281 ms",
"strata serve: stage 1: 527 windows; per window: wait for the GPU 9.893 ms, pool + plan 0.195 ms, host staging 0.000 ms, commit 0.276 ms",
"strata serve: stage 0: 578 windows; per window: wait for the GPU 8.908 ms, pool + plan 1.359 ms, host staging 0.559 ms, commit 0.284 ms",
"strata serve: stage 1: 578 windows; per window: wait for the GPU 10.025 ms, pool + plan 0.200 ms, host staging 0.000 ms, commit 0.278 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 28)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, the glass was vibrating like a plucked cello string. The sea, usually a rhythmic pulse against the granite base, was now a chaotic, frothing beast, throwing itself against the rocks with the force of a battering ram.\n\nElias stood at the top of the spiral staircase, his hand resting on the cold iron railing. He was not afraid. Fear was for the tourists who came in summer, wide-eyed and clutching their cameras. Elias was a keeper. He was a guardian.\n\nHe checked the logbook. The entry for the night was simple: *Wind, 60 knots. Visibility, zero. Light, operational.*\n\nHe walked down the stairs, his boots heavy on the metal grating. The smell of salt"
},
{
"name":"code",
"tok_s":109.6,
"reps":[
100.7,
114.1,
114.1
],
"hit_rate":0.9825370343977666,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 540 windows; per window: wait for the GPU 9.206 ms, pool + plan 1.794 ms, host staging 0.525 ms, commit 0.298 ms",
"strata serve: stage 1: 540 windows; per window: wait for the GPU 8.767 ms, pool + plan 0.173 ms, host staging 0.000 ms, commit 0.242 ms",
"strata serve: stage 0: 608 windows; per window: wait for the GPU 9.390 ms, pool + plan 1.733 ms, host staging 0.471 ms, commit 0.301 ms",
"strata serve: stage 1: 608 windows; per window: wait for the GPU 8.910 ms, pool + plan 0.178 ms, host staging 0.000 ms, commit 0.244 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 30)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning cauldron of black ink and white foam, throwing itself against the granite base with the force of a battering ram.\n\nInside the lantern room, the air was thick with the smell of kerosene and old brass. Elias wiped his hands on his woolen trousers, leaving faint, oily smears on the fabric. He checked the clock. 11:42 PM. The rotation was due.\n\nHe reached for the crank handle, a heavy iron wheel worn smooth by decades of his own palms. With a grunt that matched the strain of his shoulders, he began to turn it. *Click. Click. Click.* The gears engaged, and the massive Fresnel lens began its slow"
},
{
"name":"code",
"tok_s":108.7,
"reps":[
98.8,
112.2,
115.1
],
"hit_rate":0.9777186846317358,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 533 windows; per window: wait for the GPU 9.839 ms, pool + plan 2.192 ms, host staging 0.585 ms, commit 0.318 ms",
"strata serve: stage 1: 533 windows; per window: wait for the GPU 8.014 ms, pool + plan 0.156 ms, host staging 0.000 ms, commit 0.231 ms",
"strata serve: stage 0: 602 windows; per window: wait for the GPU 10.014 ms, pool + plan 2.111 ms, host staging 0.520 ms, commit 0.321 ms",
"strata serve: stage 1: 602 windows; per window: wait for the GPU 8.094 ms, pool + plan 0.160 ms, host staging 0.000 ms, commit 0.232 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 32)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy-three years old, and his hands were a map of the coastline he had guarded for fifty years. The knuckles were swollen, the skin cracked like dry earth, but they moved with a precision that no machine could replicate.\n\nHe stood on the gallery of the *Harrow Point* lighthouse, the iron railing cold against his palms. Below, the Atlantic churned in a frenzy, whitecaps exploding against the black rocks like fireworks. Above, the sky was a bruised purple, heavy with rain that hadn\u2019t yet fallen.\n\nElias checked the clock. 11:42 PM. The light was due.\n\nHe didn\u2019t look at the gauge. He didn\u2019t check the fuel levels. He knew. He always knew. The rhythm of the ocean, the pressure of the air, the subtle vibration"
},
{
"name":"code",
"tok_s":105.2,
"reps":[
87.8,
113.5,
114.2
],
"hit_rate":0.9109811738648947,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 382 windows; per window: wait for the GPU 15.155 ms, pool + plan 4.331 ms, host staging 0.656 ms, commit 0.458 ms",
"strata serve: stage 0: 455 windows; per window: wait for the GPU 15.790 ms, pool + plan 5.552 ms, host staging 0.688 ms, commit 0.467 ms",
"strata serve: stage 0: 524 windows; per window: wait for the GPU 16.064 ms, pool + plan 5.529 ms, host staging 0.606 ms, commit 0.472 ms",
"strata serve: stage 0: 593 windows; per window: wait for the GPU 16.305 ms, pool + plan 5.462 ms, host staging 0.540 ms, commit 0.475 ms"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning cauldron of black ink and white foam, throwing itself against the granite base with the force of a battering ram.\n\nInside the lantern room, the air was thick with the smell of kerosene and old brass. Elias wiped his hands on his woolen trousers, leaving faint, oily smears on the fabric. He checked the clock. 11:42 PM. The rotation was due.\n\nHe reached for the crank handle, a heavy iron wheel worn smooth by decades of his own palms. With a grunt that matched the strain of his shoulders, he began to turn it. *Click. Click. Click.* The gears engaged, and the massive Fresnel lens began its slow"
},
{
"name":"code",
"tok_s":88.2,
"hit_rate":0.7011385968111186,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning black void, punctuated by whitecaps that looked like teeth.\n\nHe climbed the spiral stairs, his boots heavy with salt and exhaustion. The iron steps groaned under his weight, a sound that had become so familiar to him that he could sleep to it. But tonight, he wasn\u2019t sleeping. He was working.\n\nThe lens was the heart of the lighthouse. A massive, rotating Fresnel lens, a cathedral of glass and brass that turned with a mechanical hum. Elias checked the clockwork mechanism. It was smooth, rhythmic, a heartbeat in the dark. But the light itself was failing.\n\nHe climbed higher, into the lantern room. The wind battered the glass panes,"
},
{
"name":"code",
"tok_s":87.5,
"reps":[
87.5
],
"hit_rate":0.7075461363025227,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 39 windows; per window: wait for the GPU 17.252 ms, pool + plan 8.473 ms, host staging 0.611 ms, commit 0.525 ms",
"strata serve: stage 0: 47 windows; per window: wait for the GPU 18.018 ms, pool + plan 8.616 ms, host staging 0.607 ms, commit 0.534 ms",
"strata serve: stage 0: 169 windows; per window: wait for the GPU 15.223 ms, pool + plan 5.717 ms, host staging 0.688 ms, commit 0.467 ms",
"strata serve: stage 0: 243 windows; per window: wait for the GPU 16.500 ms, pool + plan 7.316 ms, host staging 0.742 ms, commit 0.481 ms"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and shadow.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined his life for thirty years.\n\nElias was not a man of many words. He was a man of many tasks. His hands, mapped with scars from rope burns and broken glass, moved with a rhythm that predated his memory. He checked the clockwork mechanism of the Fresnel lens, listening for the faint, rhythmic *tick-tick-tick* that signaled the rotation was true. If the rhythm faltered, ships"
},
{
"name":"code",
"tok_s":108.9,
"reps":[
103.3,
113.7,
109.5
],
"hit_rate":0.9971795386830036,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 542 windows; per window: wait for the GPU 7.167 ms, pool + plan 0.572 ms, host staging 0.550 ms, commit 0.232 ms",
"strata serve: stage 1: 542 windows; per window: wait for the GPU 12.855 ms, pool + plan 0.250 ms, host staging 0.000 ms, commit 0.334 ms",
"strata serve: stage 0: 610 windows; per window: wait for the GPU 7.276 ms, pool + plan 0.559 ms, host staging 0.493 ms, commit 0.234 ms",
"strata serve: stage 1: 610 windows; per window: wait for the GPU 13.048 ms, pool + plan 0.258 ms, host staging 0.000 ms, commit 0.337 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"strata generate: layer split: CUDA1 holds its weights, session and head; 18.13 GiB free",
"strata generate: layer split auto: CUDA0 84 SMs at 2.81 GHz -> 0.31 ms per layer, 8.30 GiB for experts",
"strata generate: layer split auto: CUDA1 82 SMs at 1.86 GHz -> 0.48 ms per layer, 14.14 GiB for experts",
"strata generate: layer split auto: K=22 - predicted 19.7 ms per decode window; the caches hold 11298 of 12288 profiled pairs (~99.7% of the routed mass)",
"text":"The wind didn\u2019t just blow; it screamed. It tore through the iron gallery of the Blackwood Point Lighthouse, rattling the glass panes like a prisoner shaking its bars.\n\nElias Thorne was seventy years old, and he had been the keeper for forty of them. He knew every creak of the spiral staircase, every draft that slipped under the door, and every shift in the barometric pressure that preceded a storm. He was not merely a worker; he was the heartbeat of the coast.\n\nIt was November, the month of grey seas and shorter days. The fog had rolled in thick, a wall of white so dense that even the beam of the Fresnel lens seemed to struggle against it. Elias stood in the lantern room, his hands roughened by salt and rope, adjusting the rotation mechanism. The great lens, a cathedral of glass and brass, began its slow, rhythmic sweep. *Flash. Dark. Flash. Dark.*\n\nHe checked the log"
},
{
"name":"code",
"tok_s":105.1,
"reps":[
98.4,
109.9,
107.0
],
"hit_rate":0.9949900793650793,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 492 windows; per window: wait for the GPU 15.016 ms, pool + plan 0.297 ms, host staging 0.517 ms, commit 0.404 ms",
"strata serve: stage 1: 492 windows; per window: wait for the GPU 5.775 ms, pool + plan 0.854 ms, host staging 0.001 ms, commit 0.210 ms",
"strata serve: stage 0: 561 windows; per window: wait for the GPU 15.257 ms, pool + plan 0.306 ms, host staging 0.455 ms, commit 0.408 ms",
"strata serve: stage 1: 561 windows; per window: wait for the GPU 5.852 ms, pool + plan 0.820 ms, host staging 0.001 ms, commit 0.210 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"strata generate: layer split: CUDA1 holds its weights, session and head; 9.98 GiB free",
"strata generate: layer split auto: CUDA0 82 SMs at 1.86 GHz -> 0.48 ms per layer, 16.45 GiB for experts",
"strata generate: layer split auto: CUDA1 84 SMs at 2.81 GHz -> 0.31 ms per layer, 5.99 GiB for experts",
"strata generate: layer split auto: K=30 - predicted 20.8 ms per decode window; the caches hold 10684 of 12288 profiled pairs (~99.5% of the routed mass)",
"text":"The wind didn\u2019t just blow; it screamed. It tore through the iron gallery of the Blackwood Point Lighthouse, rattling the glass panes like a prisoner shaking its bars.\n\nElias Thorne was seventy years old, and he had been the keeper for forty of them. He knew every creak in the spiral staircase, every draft that slipped under the door, and every shift in the weather that the barometer couldn\u2019t quite predict. He knew that when the fog rolled in thick enough to swallow the horizon, the world ceased to exist. There was only the light, the stone, and him.\n\nIt was November, the month of storms. The sea was a churning cauldron of grey and white, throwing itself against the granite base with the force of a battering ram. Elias stood in the lantern room, his hands rough as sandpaper, adjusting the massive Fresnel lens. The rotation mechanism groaned, a sound like a ship\u2019s anchor chain being h"
},
{
"name":"code",
"tok_s":89.6,
"reps":[
81.0,
104.8,
83.0
],
"hit_rate":0.9975041387598007,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 2: 522 windows; per window: wait for the GPU 12.405 ms, pool + plan 0.242 ms, host staging 0.006 ms, commit 0.375 ms",
"strata serve: stage 0: 590 windows; per window: wait for the GPU 7.387 ms, pool + plan 0.549 ms, host staging 0.517 ms, commit 0.248 ms",
"strata serve: stage 1: 590 windows; per window: wait for the GPU 3.071 ms, pool + plan 0.011 ms, host staging 0.000 ms, commit 0.606 ms",
"strata serve: stage 2: 590 windows; per window: wait for the GPU 12.592 ms, pool + plan 0.250 ms, host staging 0.006 ms, commit 0.377 ms"
],
"engine_lines":[
"strata generate: layer split across 3 GPUs: CUDA0, then CUDA1,2 (split auto)",
"strata generate: layer split: CUDA2 holds its weights, session and head; 18.13 GiB free",
"strata generate: layer split auto: CUDA0 84 SMs at 2.81 GHz -> 0.31 ms per layer, 8.30 GiB for experts",
"strata generate: layer split auto: CUDA1 68 SMs at 1.54 GHz -> 0.69 ms per layer, 13.50 GiB for experts",
"strata generate: layer split auto: CUDA2 82 SMs at 1.86 GHz -> 0.48 ms per layer, 14.14 GiB for experts",
"strata generate: layer split auto: K=22,23 - predicted 19.9 ms per decode window; the caches hold 11298 of 12288 profiled pairs (~99.7% of the routed mass)",
"text":"The wind didn\u2019t just blow; it screamed. It tore through the iron gallery of the Blackwood Point Lighthouse, rattling the glass panes like a prisoner shaking its bars.\n\nElias Thorne was seventy years old, and he had been the keeper for forty of them. He knew every creak in the spiral staircase, every draft that slipped under the door, and every shift in the weather that the barometer couldn\u2019t quite predict. He knew that when the fog rolled in thick enough to swallow the horizon, the world ceased to exist. There was only the light, the stone, and him.\n\nIt was November, the month of storms. The sea was a churning cauldron of grey and white, throwing itself against the granite base with a violence that made the floorboards tremble. Elias stood by the lantern room, his hands wrapped around a mug of coffee that had gone cold hours ago. He wasn\u2019t drinking it. He was holding it for warmth."
},
{
"name":"code",
"tok_s":111.1,
"hit_rate":1.0,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv(file_path: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys being the header values\n and values being the corresponding cell values.\n \n Args:\n file_path: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(file_path, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_basic():\n \"\"\"Test basic CSV parsing with simple data.\"\"\"\n import tempfile\n import os\n \n # Create a temporary CSV"
}
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning cauldron of black ink and white foam, throwing itself against the granite base with the force of a battering ram.\n\nInside the lantern room, the air was thick with the smell of kerosene and old brass. Elias wiped his hands on his woolen trousers, leaving faint, oily smears on the fabric. He checked the clock. 11:42 PM. The rotation was due.\n\nHe reached for the crank handle, a heavy iron wheel worn smooth by decades of his own palms. With a grunt that matched the strain of his shoulders, he began to turn it. The massive Fresnel lens, a cathedral of glass and crystal, began its slow, hypnotic sweep"
},
{
"name":"code",
"tok_s":101.8,
"hit_rate":0.991767530342175,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"text":"The wind didn\u2019t just blow; it screamed. It clawed at the granite base of the tower, trying to pry the lighthouse from its bedrock. Inside, however, the air was still, smelling faintly of kerosene and old paper.\n\nElias had been the keeper of the Point for thirty-two years. He knew every creak in the floorboards, every draft that slipped through the window seals, and exactly how much oil the great Fresnel lens needed to turn its slow, hypnotic rotation.\n\nIt was 2:00 AM. The storm was at its peak. The sea was a chaotic black mirror, whipped into whitecaps that looked like jagged teeth.\n\nElias climbed the spiral stairs, his hand gripping the cold iron railing. His boots, heavy with salt and wear, thumped a steady rhythm against the stone. *Thump. Thump. Thump.*\n\nAt the top, the lantern room hummed with the vibration of"
},
{
"name":"code",
"tok_s":86.4,
"hit_rate":0.9798276439089693,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 16)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy-three years old, and he had been the keeper of the Blackwood Point Light for forty of those years. The other thirty-three had been spent in a hospital bed, waiting for the end.\n\nEvery night, Elias climbed the one hundred and twelve steps of the spiral staircase. His knees, worn smooth by the friction of his own boots over decades, protested with a sharp ache. He counted them. *One, two, three\u2026* The rhythm was his heartbeat. If he lost the count, he lost the light.\n\nAt the top, the lantern room hummed with the low, electric thrum of the Fresnel lens. It was a cathedral of glass and brass, a machine that had never failed him. Not once. Not in four decades.\n\nHe polished the brass railing with a rag that smelled of lemon and"
},
{
"name":"code",
"tok_s":104.3,
"hit_rate":0.9842543225870746,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 22)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the fabric stiff with salt and ice. Outside the thick glass of the lantern room, the Atlantic was a churning cauldron of black and white, the waves crashing against the granite base with a violence that shook the very foundations of the tower. It was November, 1924, and the storm had been raging for three days.\n\nElias was not a man who feared the dark. He was a man who feared the silence.\n\nHe checked the clock on the wall. 11:42 PM. The rotation of the great Fresnel lens was steady, a rhythmic *whirrr-click, whirrr-click* that seemed to sync with his own heartbeat. He had been doing this for forty years. His hands were mapped with scars from rope burns and broken glass, his eyes permanently squinted against the glare of the bulb."
},
{
"name":"code",
"tok_s":93.4,
"hit_rate":0.990286044973545,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, rough as sandpaper, knew every bolt, every lever, and every creak in the iron bones of the tower.\n\nFor forty years, Elias had been the guardian of the Point. Not just of the light, but of the silence between the waves.\n\nIt was November. The sky was a bruised purple, heavy with rain that hadn\u2019t yet fallen. The sea was a churning cauldron of grey and white, throwing itself against the granite base with a violence that shook the floorboards beneath Elias\u2019s boots.\n\nHe climbed the stairs. One hundred and twelve steps. He counted them by feel, his left hand trailing along the cold iron railing. At the top, the air was"
},
{
"name":"code",
"tok_s":58.0,
"reps":[
58.0
],
"hit_rate":0.9700542735219787,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 168 windows; per window: wait for the GPU 9.683 ms, pool + plan 0.509 ms, host staging 0.714 ms, commit 0.320 ms",
"strata serve: stage 1: 168 windows; per window: wait for the GPU 25.253 ms, pool + plan 0.881 ms, host staging 0.000 ms, commit 0.559 ms",
"strata serve: stage 0: 242 windows; per window: wait for the GPU 9.860 ms, pool + plan 0.550 ms, host staging 0.750 ms, commit 0.328 ms",
"strata serve: stage 1: 242 windows; per window: wait for the GPU 26.029 ms, pool + plan 0.980 ms, host staging 0.000 ms, commit 0.571 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"text":"The wind didn\u2019t just blow; it screamed. It tore through the iron gallery of the Blackwood Point Lighthouse, rattling the glass panes like a prisoner shaking its bars.\n\nElias Thorne was seventy years old, and he had been the keeper for forty of them. He knew every creak in the spiral staircase, every draft that slipped under the door, and every shift in the weather that the barometer couldn\u2019t quite predict. He knew that when the fog rolled in thick enough to swallow the horizon, the world ceased to exist. There was only the light, the stone, and him.\n\nIt was November, the month of storms. The sea was a churning cauldron of grey and white, throwing itself against the granite base with a violence that made the floorboards tremble. Elias stood by the lantern room, his hands wrapped around a mug of coffee that had gone cold hours ago. He wasn\u2019t drinking it. He was holding it for warmth."
},
{
"name":"code",
"tok_s":69.7,
"reps":[
69.7
],
"hit_rate":1.0,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv(file_path: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys being the header values\n and values being the corresponding cell values.\n \n Args:\n file_path: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(file_path, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_basic():\n \"\"\"Test basic CSV parsing with simple data.\"\"\"\n import tempfile\n import os\n \n # Create a temporary CSV"
}
],
"stage_timing":[
"strata serve: stage 2: 165 windows; per window: wait for the GPU 13.452 ms, pool + plan 0.162 ms, host staging 0.000 ms, commit 0.328 ms",
"strata serve: stage 0: 234 windows; per window: wait for the GPU 6.235 ms, pool + plan 0.145 ms, host staging 0.704 ms, commit 0.214 ms",
"strata serve: stage 1: 234 windows; per window: wait for the GPU 15.592 ms, pool + plan 0.246 ms, host staging 0.000 ms, commit 0.657 ms",
"strata serve: stage 2: 234 windows; per window: wait for the GPU 13.916 ms, pool + plan 0.177 ms, host staging 0.000 ms, commit 0.335 ms"
],
"engine_lines":[
"strata generate: layer split across 3 GPUs: CUDA0, then CUDA1,2 (split auto)",
Status: preliminary; prompt set and harness guidance requested upstream.
## Hardware
| Component | Guest-visible details |
|---|---|
| CPU | AMD EPYC 7453, 20 vCPU, AVX2; AVX-512 not exposed |
| RAM | 165 GiB visible to Ubuntu 24.04.5 KVM guest; physical DIMM layout and bandwidth not verified |
| GPUs | 2x RTX 3090 24 GB; PCIe Gen4 x16 maximum reported per card; driver 595.91.07 |
| Power limit | 280 W configured on each GPU; defaults are 350 W / 420 W, max-adjustable limits 366 W / 450 W |
| Storage | 541 GiB ext4 guest data disk; host storage layout not verified |
The power values above are configured caps, not measured workload draw. Idle draw during the hardware capture was approximately 35 W / 29 W.
## Method
Native Linux setup, local build from v0.1.26, `--prefill auto`, shipped expert profile, automatic expert cache, int8 KV, MTP on, images off, experimental speed projection off. The GPU power limit was 280 W per card. Native runs used a systemd service with `LimitMEMLOCK=infinity`; other llama-swap model containers were stopped.
The local API harness used the public Strata source tree as its prompt corpus, with generated tasks, `temperature: 0`, `reasoning_effort: none`, and `max_tokens: 256`. It is not the upstream code-agent prompt set or harness. Runs use unique request IDs and no conversation reuse in speed mode. Three runs per speed cell are summarized by medians; actual `usage.prompt_tokens` are recorded because nominal context labels can drift. The first decode run is also shown. Several requests ended early on EOS, so actual generated counts are in the raw JSON.
No calibration comparison, upstream prompt comparison, `STRATA_SPLIT_TIMING` stage timing, or under-load power-draw capture has been done. Treat all prompt/decode comparisons with upstream as indicative only until the requested upstream method is available.
## Results
See [`matrix.md`](matrix.md) for summary tables and [`matrix.json`](matrix.json) for all successful run records. Per-run source JSON, logs, the local harness, and scrubbed hardware facts are in [`data/`](data/).
Highlights:
- IQ3_XXS on one 3090: 1K–262K cells, three runs each; 262K prompts were 232,183 / 259,589 / 261,068 tokens and fit within the 262,144 context.
- Coder IQ1_M at 32K: single-GPU median decode 66.8 tok/s; dual-GPU auto (K=19) median 98.6 tok/s.
- Fixed Coder K sweep: K=22 was the highest observed three-run median (103.3 tok/s); independently generated prompts mean this is not a paired ranking.
- IQ3_XXS and IQ3_S dual-GPU auto both selected K=24 at 32K.
- IQ3_XXS and IQ3_S needles passed 3/3 at 8K / 16K / 28K on the tested single/dual configurations; two-turn checkpoint reuse passed for both.
## Limits and follow-up
- Upstream prompts/harness are not yet available; all current speed data uses a local source-tree corpus.
- The guest exposes one NUMA node and 165 GiB RAM, but physical DIMM topology and memory bandwidth are unknown.
- PCIe x16 is the reported maximum link width; the engine probes were about 23–26 GB/s. Verify PCIe generation/width under sustained load before claiming a direct x16-vs-x4 comparison.
- Coder dual-GPU needle/two-turn checks and dual-GPU timing breakdowns are not run yet.
- K=18 has one successful run at 83.1 tok/s; a repeated startup failed while uploading a dense weight with an out-of-memory error, so it is not a three-run median.
- Calibration/default comparisons and IQ3_S single-GPU results remain outstanding.
"answer_head":"Looking at the last file above, `docs/AMD_HIP_PERFORMANCE.md`, it documents the performance evidence and correctness limits for the experimental AMD HIP backend targeting gfx1100 (RX 7900 XTX). It records:\n\n- The exact build/runtime configuration used for the benchmark (env vars, CLI flags, model pa",
Loaded 100 of 720 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Niko1221