Reporting only (class E): no engine speedup, no change to what the engine computes.
serve/server.py
- A prompt read that gives way (BYIELD) and resumes keeps the request's external prefix boundary, and the prompt
time and tokens of every leg are summed exactly once (prefill_leg_stats; PR #5318).
- The admission's DONE times one token and BDONE counts that token but times only the slot: the slot time now
includes the admission time, and the output token is not counted twice. A figure the engine never reported
(a close at the admission token, in the slot, a carry back to the solo path) stays unknown (None) instead of
turning into 0 or into the API's output count (PR #5307).
- The two patches overlap in the request finalization; ended_on_yield is taken before the unknown-figures step so
the identity check still sees the yield's own record.
PfTimer::mark(): a mark in the phase already running is not recorded (PR #4006). The totals are the same, a
fraction of the event records; under WDDM every record is a submission of its own and the launch gaps they made
were charged to "dequant". The rule is a pure function in prefill/timer_marks.hpp with a host test.
Tests: serve/test_yield_accounting.py (new: two yields, solo and admission yields, cancel after one and two
yields), serve/test_parallel.py BatchLegAccounting (EOS and length finishes, zero slot time, close at the
admission token and in the slot, slot to solo carry), serve/test_request_accounting.py (unknown stays unknown,
explicit zero), tests/prefill/timer_marks_test.cpp (same totals with 18 of 23 records skipped).
Co-authored-by: YusenZhang0601 <[email protected]>
Co-authored-by: shoucandanghehe <[email protected]>
Co-authored-by: Vop3p <[email protected]>
Conflict in serve/server.py main(): LIFECYCLE wraps _main in the shared activity lease (#4463), OWN wraps it in the decoupled console (#4479 / #2481). Kept both: the lease is taken first, and _main runs inside decoupled_console() unless STRATA_BLOCKING_CONSOLE=1; the lease is closed in the same finally.
Conflict in strata_cli/bench_facts.py (the redaction helper): kept both sides' intent. The host name comes from sysinfo.node() (LIFE) and the key is redacted with usable_key(key), word=True, exact=True (SECURE).
DRAIN_TIMEOUT_S (300 s) and QUARANTINE_S (600 s) apply only after a request has left and the server has sent STOP / BSTOP:
_drain_control (the finally of generate_batched), _quarantine_control and the BDONE wait of _release_slot_when_done. They
never cover a generation a client is still reading. But they were counted from the start of the drain, so an engine that
was told to stop and kept producing lines (an older engine runs on to max_new; a long prompt read ends at its next chunk)
could be ended after 900 s, with every other request in it. Now every line the engine writes starts the limit again: only
an engine that says nothing at all for 300 s + 600 s after the stop is ended.
Tests (fake engine, limits scaled to 0.3 s + 0.4 s): an engine that keeps talking for ~1.8 s after the stop is not ended and
the request in another slot is untouched (fails with the previous commit's absolute limits); a silent one is ended; the
drain-timeout leak test now really closes the stream mid-read (the old version was ended by a BYIELD before the drain);
long replies, far over the limits, complete. Also: a client that is idle while the engine thinks is not cut by the 60 s
write limit (0.3 s between tokens, 0.1 s limit, the whole reply arrives).
On a batch engine the control lines carry one exchange at a time and every request takes `ctl` before it sends or reads
there. VRAM and SAVE / RESTORE read the same lines for their answer without it (POST /v1/vram took it itself, the image
encoder's cache resize, the load-time reserve and the session handler did not): run beside a request they took its
T / PP / DONE lines, and the request lost its reply (or session_file ended the engine as out of step).
StrataEngine.vram / session_file now run under _exclusive_control(): ctl on a batch engine (a thread that already holds it,
as Service.vram does, is marked and does not wait for itself), nothing to take for one request at a time. Waiting longer
than CONTROL_WAIT_S raises the existing "a request is still running; try again when it has finished". No output changes.
Test: the fake engine answers VRAM / SAVE / RESTORE; a request streams while the call is made. On the base the request
ends with 212 of 421 tokens; with the fix it is complete and the call is answered.
print() in the server blocks when the console stops taking output (a Windows console in selection mode, a paused
terminal, a launcher that stopped reading the pipe), in whatever thread called it - including those that own the
engine's turn, a cancel or an admission. While serve main() runs, stdout and stderr (when they are the process's own)
are replaced by serve/nonblocking_out.NonBlockingOut: write() queues and returns, one daemon thread writes to the real
stream, the oldest text is dropped (and counted) when the queue is over 1 MiB / 4000 chunks. Restored and drained at the
end of main(). Protocol output (engine pipe, HTTP) is not touched. STRATA_BLOCKING_CONSOLE=1 turns it off.
No visible text is added; a drop is counted, not announced.
Tests: a plain print to an unread pipe blocks (control); the wrapper never blocks, keeps the newest lines in order, and
drains. Approach credit: the proposal in #4479 (one process-owned output consumer).
The thread that writes the SSE stream to a client also holds the engine's turn (the one-request path). A client that
stops reading without closing (a hung process, a dropped network) filled the socket buffers and the write blocked until
the OS gave the connection up: the engine idle, every other request waiting. A write that makes no progress for
STREAM_WRITE_TIMEOUT_S (60 s; STRATA_STREAM_WRITE_TIMEOUT_S, 0 = off) now ends as a disconnect: the engine is
stopped and the turn given back. No text changes.
Test: a client that never reads a 400k-token stream; on the base the turn is still held after 20 s, with the fix it is
given back at once. Part 2 of #4271 (arrival order of the one-request path: the turn is a plain Lock) is not done here.
_take_control read its turn, gave up the lock and then waited: a notification sent in between was lost and the waiter
slept up to 0.5 s although it could go. It now reads the turn (and the short-first arbitration) again under the lock
before waiting, and a waiter that leaves (cancelled, closed) notifies the others.
Approach credit: PR #4738. Tests are new (a hooked Condition forces the schedule; on the base the waiter waits).
- GpuBusy before any segment and with zero tokens: the history row says "error" with no prompt tokens and takes no part
in the totals, latencies or last timings (#3040); a continuation after native work keeps the old accounting;
- the engine's and the image encoder's stdout are decoded with errors="replace": on Windows a byte such as 0x8e ended
the reader thread (UnicodeDecodeError) and the server lost the engine's output. Protocol lines stay ASCII.
Not taken from PR #5044: the q36 engine OOM messages (engine C++, and the wording goes to the coordinator first).
Approach credit: PR #5044 (Python parts). Tests: test_parallel (refused admission accounting), test_serve_robustness
(a 0x8e byte before READY and in a reply).
StrataEngine.__init__ cleaned up a start that ended before READY through self.proc; restart()'s close() on another
thread had set it to None (AttributeError: NoneType has no attribute stdin, a recoverable engine abort became a 400).
The cleanup now uses the process object of THIS start, captured once right after it was spawned.
Approach credit: PR #4792 (same idea; the test is new: the closing thread is forced inside proc.wait()).
Polls come from several request and monitor threads. A second poll moved `pos` between the first one's size check and
its read: lines were read twice (parks counted double) and a read length could go negative (ValueError). Found by
HWSTALL while triaging #3443.
serve/test_serve_robustness.py: the overlap is forced with a hooked getsize; on the base every parked line is counted
twice (8 != 4).
A --batch request read its slot's lines from `self.slot_q[slot]`, the control lines from `self.lines`, and sent its
BSTOP / BYIELD / STOP with the slot number, all looked up when used. After the engine was restarted the same names meant
the NEW process: a request of the old one read (and could stop) the new one's requests. And a request that left before
the engine had ended its exchange (a drain that timed out after 300 s) gave the slot and the control lines back, so the
engine's late T / PP / DONE / BADM / BT frames reached the next request as its own.
- the request notes the queues of the engine process it was admitted on and reads only those; a send meant for that
process (BSTOP, BYIELD, STOP) is not sent to a later one (EngineDied instead); slot state is only given back on the
same process;
- an exchange that is not drained in DRAIN_TIMEOUT_S is quarantined: the control lines and the admission's slot stay with
it, in the background, until the engine gives DONE / BADM (then a continued admission is BSTOPped and freed at its
BDONE); if the engine never does within QUARANTINE_S it is ended, as a silent engine is (#481). A slot's BDONE wait that
times out ends the engine too, instead of freeing a slot whose lines may still come;
- the pump drops BT / BDONE frames for a slot no request has claimed (counted in stale_frames);
- _ctl_mode, _ctl_result, _yielded, _last_done, _ctl_erred are per request thread on a batch engine;
- serve/test_stream_ownership.py (BatchOwnership): interleaved clients with canaries, a cancel in the middle of the prompt
read followed by reuse, a drain that times out (DEAFSTOP), the engine ending during a drain and restarting. On the base
the restart case fails: requests of the old process stay attached to the new process's slots.
The other cases of the register (mid-prefill cancel, drain timeout, slot not handed on) pass on the base with the fake
engine; they are kept as regression tests. The real-engine check follows.
ReplicaEngine.generate released the per-replica run lock in its finally without closing the child generator. The
child's own STOP and drain-to-DONE ran later, when the wrapper frame was collected, after the next request had sent
its GEN: the two read each other's tokens and DONE (an empty reply, or another request's text), and a cleanup error
was lost as an ignored generator exception. The unbatched engine's shared `last` was also read by Service after the
lock was free, so a reply could carry the next request's figures.
- close the child before the lock is given back; a close that fails ends the replica (heal() starts it again) and the
error still reaches Service (EngineSilent handling is unchanged);
- copy the new DONE figures into a per-caller object while the replica is owned; with no new DONE the caller's earlier
object stays, so an old result is never recorded as new;
- serve/test_stream_ownership.py: canary replies from a fake engine, a hooked run lock that forces the legal schedule
(the successor takes the replica right after release), 30 forced hand-offs and 60 unforced back-to-back requests.
On the base: empty reply for the successor, the predecessor's canary in its text, 30 of 30 forced hand-offs crossed,
figures of the other request.
Approach credit: PR #2886 (scheme A: explicit child close, caller-local DONE snapshot). Written in this tree.
pick_slot also recognises an earlier message boundary of a slot's held stream (the last question replaced), so the request no longer overwrites another chat's idle slot. A boundary every held conversation shares (one client's system prompt) names no owner; busy slots are never picked. Tests: branch affinity, busy owner, shared root, interleaved branches (retained prefix 96000 -> 115200 of 115200 tokens).
Approach from PR #5040: _close_verdict holds while </tool_call> is still arriving, and the streaming scan of a body whose end is known runs as final. #4701 (final call) and #4702 (longer code fence) are already fixed on the base; the corpus gets the four 4701 specimens (PR #4762 and #5040 fixtures) and runs them whole, in pieces and cut at every byte, with streamed arguments on and off.
Approach from PR #4204. sd-cli at the pinned stable-diffusion.cpp commit reads --prompt-file as raw bytes (checked in examples/common/common.cpp at 2988060). The job folder is the 0700 mkdtemp one; the file is 0600, created exclusively and removed with the job's other files. The stand-in sd-cli reads the file; a canary test checks the recorded argument list.
requirements.txt and the four hash-locked terminal locks (windows, linux x86_64, linux aarch64, macOS arm64) move to 13.0.0; Python 3.10 stays on
12.3.0. Image.open() in the vision path, the EXIF probe and the DeepSeek picture reader now only opens JPEG, PNG, GIF, BMP, WebP, TIFF and AVIF, so a
JPEG 2000 file never reaches OpenJPEG on any Pillow version. Approach from PR #4220 (version split by Python range).
- drafting.blocked_by(named=True): only --pipeline-windows still blocks; Metal keeps the old rule (no per-request drafters there)
- apply/launcher/plan pass the choice; the all-cards split engine runs --draft dflash2 --draft-model PACK only for DFlash2 by name AND --split-engine on
- the split engine's own DFlash2 start line counts as "DFlash2 on"; a split run config with --draft dflash2 is left as it is
- app Drafting hint and Terminal --drafting help say it (owner-approved wording, English only)
- tests: terminal drafting/single/split-parallel, split engine, serve parallel, node parallel-default
App: `strata drafting plan/fetch` gain --no-split, and every drafting call the host makes (control-runtime.mjs draftingPlan, draftingFetch) carries it, so Auto in the app on
2 NVIDIA / 4 R9700 cards is the layer split with DFlash2 alone, no draft layer is planned or fetched for the split, and the Models screen label and fit say DFlash2 again
(capabilities.dflash2Split stays false). The Terminal's plan is unchanged (split + MTP). The gguf_dir fix of installMtpLayer and the explicit-MTP path stay.
Lanes: split_config cuts the lanes to 1..8 (the engine refuses "--batch must be 1..8"): the AMD four-card default PARALLEL 32, a layer config's own --batch (IQ3_S on 4 R9700
gave 32 and fell back to the layer split) and a profile parallel above 8; serve/server.py parallel_args' split branch cuts the same.
Tests: tests/terminal/test_split_parallel.py (AppDoesNotSplit, LaneClamp), apps/host/tests/drafting.test.mjs.
The split runs the model's draft layer (--mtp) and exits at start without it ("sfnx: --native and --mtp are required"). The plan and the fetch for Auto
resolved to DFlash2 (drafter 1.2 GB, no draft layer), so on 2 x sm_86/sm_120 or 4 x gfx1201 the split was chosen at start, failed, and the server fell back
to the layer split after "Using the all-cards split engine on 2 cards".
(B) strata_control/split_engine.expected(): the rule apply_to_config uses, asked before anything is on disk (model assumed when its file is not there yet).
Runner.split_expected / resolved_drafting and drafting_inputs: Auto where the split will take the start is the MTP, so download_plan and the fetch include the
draft layer (5.2 GB) and not DFlash2's drafter, setup gets --mtp on without --drafter dflash2, and the plan says "Drafting: MTP (the all-cards split engine uses
the MTP draft layer, 5.2 GB)". The app's `strata drafting plan/fetch` (drafting_cmd._inputs) follows the same rule. A person's own DFlash2 / off, --split-engine
off, one card, other cards, three cards keep what they had.
(A) Runner.apply_split_engine: when the --mtp folder is missing or empty at start the split is not picked (Auto: silently, the layer split with DFlash2 starts as
the plan said; "on": one line saying to add --drafting mtp). _auto_gates() is the Auto-only part of the rule, shared by apply_to_config and expected().
Tests: tests/terminal/test_split_parallel.py (Terminal and app plan paths, AMD 4 x gfx1201 and 2, one card, sm_89, own DFlash2, off, missing draft layer).
The split's run config always carries --batch N. ensure_profile still wrote the launch profile's parallel=<card fit> (4 on 32 GB or more) beside
it, and strata serve --profile (cli._effective) refuses a parallel override next to --batch: every 2-card NVIDIA start with the split on Auto
stopped with "parallel override conflicts with --batch/--slots in run config args" before the engine ran. Same for the AMD four-card default.
- runner.ensure_profile: no parallel override when the run config already names its slots (run_config_owns_slots)
- runner.apply_split_engine: the person's --parallel N is the split's lane count (cfg["parallel"] for the rule; taken out again on a refusal)
- cli._effective: a parallel override on the split's run config (split_engine_note) is not a conflict (the app's Concurrent requests); other configs keep the check
- server.parallel_args: "parallel" on the split's run config sets its --batch
- tests/terminal/test_split_parallel.py: Terminal path, app path, AMD four-card, lanes from the flag, the check kept for other configs
A card whose expert cache stays below the 128 slots a prompt-path loan keeps behind used to be squeezed (reserve lowered to 100 MiB free, then the first long
prompt aborted in an mmq allocation) or gave the prompt path 600 MiB of its own buffers and stopped. The loan now leaves 16 slots, the auto lend cap is lifted
for such a card, and every 128-slot floor in the lend, request-time lend and VRAM-shrink paths follows it (g_lend_keep). When the draft layer, the head and a
working cache with room left do not fit in the free VRAM before the layer loads, the start runs without the layer and says so (STRATA_MTP_KEEP=1 keeps it).
An out-of-memory in the layer or head load with --batch names --batch and --max-context. Measured on an 8 GB mimic at 128K with --kv-resident 20480: DFlash2
alone 391 MiB free, 69-73 tok/s, a 32K-token prompt in 19.8 s; MTP asked: layer dropped, 557 MiB free, 62-70 tok/s, 32K prompt in 13.4 s. 12 GB unchanged.
Niko1221