web app: attach text files - drop them on the chat or use the attach button (issue #30 )

Text files (code, notes, logs, data; up to 512 KB) become chips next to the message and are sent as "File: name" plus a fenced block (a longer fence when the file contains one). Binary files are refused with a toast; pictures work as before (when images are on). The attach button is always there now. Kept small on purpose: the web app is for quick tests and settings.
engine: fix a race in the CPU expert pool, and never hang silently again (issue #29 , engine 0.1.12)
2026-11-07 14:21:06 -10:00 · 2026-09-27 20:53:38 +02:00 · 2026-09-27 20:53:38 +02:00 · 2026-09-27 17:50:38 +02:00 · 2026-09-27 17:45:18 +02:00 · 2026-09-27 17:48:41 +02:00
18 changed files with 883 additions and 112 deletions
@@ -550,6 +550,10 @@ if(STRATA_BUILD_TESTS)
  add_executable(pool_test src/kernels/cpu/pool_test.cpp)
  target_link_libraries(pool_test PRIVATE strata_kernels_cpu)
  add_test(NAME pool_test COMMAND pool_test --selftest WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
+  # issue #29: tiny batches with sleeping workers, as on a big-VRAM card; a hang fails it
+  add_executable(pool_stress src/kernels/cpu/pool_stress.cpp)
+  target_link_libraries(pool_stress PRIVATE strata_kernels_cpu)
+  add_test(NAME pool_stress COMMAND pool_stress 20)
 endif()

 if(STRATA_BUILD_TESTS)
@@ -3,54 +3,22 @@
 <p align="center"><b>Run a 125-billion-parameter AI model on a normal gaming PC</b><br>
 one NVIDIA card (12-24 GB) + 64 GB of RAM · Windows or Linux · one click to install</p>

+<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="A voxel pagoda garden that Strata's model wrote, running in the browser"></a><br>
+<sub>A voxel pagoda garden, 1 shot prompt running on an RTX 5070 with Strata (IQ3_S, 128K context) ·
+<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">full video (49 s)</a></sub></p>
+
 Strata runs **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** - a large, smart AI model that
 normally needs a server - on your own PC. It writes its answers at **60-95 tokens per second** (a token is about ¾
 of a word): faster than you can read.

- **Private:** everything runs on your PC. Nothing is sent anywhere.
- **Works with your apps:** chat apps, coding agents and scripts that speak the OpenAI or Anthropic API just work.
- **Sees pictures** too, if you want (screenshots, photos, scanned pages).
 - **Free and open source.**

-> **Jump to:** [Is my PC enough?](#is-my-pc-enough) · [Install](#install-3-steps) · [How fast?](#how-fast-is-it) ·
-> [Which model?](#which-model-should-i-pick) · [Using it](#using-it) · [Problems?](#something-went-wrong) ·
-> [How it works](#how-does-it-work) · [All the details](docs/DETAILS.md)
+> **Jump to:** [How fast?](#how-fast-is-it) · [Which model?](#which-model-should-i-pick) · [Install](#install) ·
+> [Using it](#using-it) · [Problems?](#something-went-wrong) · [How it works](#how-does-it-work) ·
+> [All the details](docs/DETAILS.md)

 ---

-## Is my PC enough?
-
-| You need | |
-| --- | --- |
-| **Graphics card** | NVIDIA RTX 30, 40 or 50 series with **12 GB of VRAM or more** |
-| **Memory (RAM)** | **64 GB** |
-| **Free disk space** | ~80 GB (an SSD makes the first start much faster) |
-| **System** | Windows 10/11, or Linux |
-
-That's it. The only thing you install yourself is a current **NVIDIA driver**
-([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA App). Everything else - Python, the engine, the
-model - is set up for you.
-
-## Install (3 steps)
-
-**Windows**
-
-1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it).
-2. Double-click **`START-HERE.bat`**.
-3. Answer 4 questions - or just press Enter each time for the recommended choice:
-   - **Which model?** The original, or Swift 1.5 (a version that thinks shorter and answers sooner)
-   - **Which size?** Q2_0, IQ2_XS, IQ3_XXS or IQ3_S - see [which model](#which-model-should-i-pick)
-   - **How much context?** How much text it can keep in mind at once (it suggests one for your card)
-   - **Images?** Whether it should also read pictures
-
-Then it downloads everything (the model is ~70 GB, so the first time takes a while - you can stop and it picks up
-where it left off) and **starts the model**. Your browser opens the Strata app at `http://127.0.0.1:8080`.
-
-**Next time**, just double-click `START-HERE.bat` again: it starts right away, nothing is downloaded twice. Close its
-window to stop the model.
-
-**Linux:** run `./setup.sh` - same questions, same result.
-
 ## How fast is it?

 Measured on an RTX 5070 (12 GB), a Ryzen 5 7600 and 64 GB of RAM:
@@ -73,24 +41,58 @@ A card with more VRAM is faster, because more of the model fits on the GPU: an R

 **The size** (the same model, compressed more or less):

-| Size | Download | Speed | Quality | Pick it if... |
-| --- | ---: | --- | --- | --- |
-| **Q2_0** | 66 GB | fastest | good | you want speed |
-| **IQ2_XS** | 68 GB | fast | better | you want a good all-rounder (**recommended**) |
-| **IQ3_XXS** | 76 GB | slower | great | you want better answers (uses 43 GB of your 64 GB RAM) |
-| **IQ3_S** | 84 GB | slowest | best: matches the full model on the published tests | you want the very best answers (original model only; uses 50 GB of your 64 GB RAM, so close other big programs) |
+| Model | RAM+VRAM Requirements | Speed | Quality |
+| --- | ---: | --- | --- |
+| **Q2_0** | 37.6 GB | fastest | good |
+| **IQ2_XS** | 39.2 GB | fast | better (**recommended**) |
+| **IQ3_XXS** | 47.0 GB | slower | great |
+| **IQ3_S** | 54.8 GB | slowest | best: matches the full model on the published tests (original model only) |
+
+**Will it fit?** Shard 1 is the part of the model that gets loaded when it starts: its experts go into your **RAM**,
+the rest onto your graphics card (the second shard, a 29 GB lookup table, stays on the SSD). So it fits when your
+**RAM is at least shard 1 + about 10 GB** for Windows and your other programs. With 64 GB of RAM every size fits
+(IQ3_S with little else open); with 48 GB, Q2_0 and IQ2_XS. A bigger graphics card makes it faster, but it doesn't
+lower the RAM needed.

 **The version:**

 - **Qwen3.8-Flash-Next** - the original.
 - **[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** - a fine-tune by UkisAI
  that thinks much shorter before answering, so you get the answer sooner, with about the same quality. Same speed per
-  token. Its own license applies (see its page).
+  token, and about the same RAM as the same size of the original (no IQ3_S). Its own license applies (see its page).

 Not sure? Take **IQ2_XS**. You can add another one later with `START-HERE.bat --setup`.

+## Install
+
+**You need:** an NVIDIA RTX 30, 40 or 50 card with 12 GB of VRAM or more, enough RAM for the size you pick (above),
+~80 GB of free disk space (an SSD makes the first start much faster), and Windows 10/11 or Linux. The only thing you
+install yourself is a current **NVIDIA driver** ([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA
+App). Everything else - Python, the engine, the model - is set up for you.
+
+**Windows**
+
+1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it).
+2. Double-click **`START-HERE.bat`**.
+3. Answer a few questions - or just press Enter each time for the recommended choice:
+   - **Which model and size?** The original or Swift 1.5, and Q2_0, IQ2_XS, IQ3_XXS or IQ3_S - see [above](#which-model-should-i-pick)
+   - **How much context?** How much text it can keep in mind at once (it suggests one for your card)
+   - **Images?** Whether it should also read pictures
+   - **Experimental speed projection?** Off unless you say yes - [read what it does](docs/DETAILS.md#experimental-speed-projection-experimental-off-by-default) first
+
+Then it downloads everything (the model is ~70 GB, so the first time takes a while - you can stop and it picks up
+where it left off) and **starts the model**. Your browser opens the Strata app at `http://127.0.0.1:8080`.
+
+**Next time**, just double-click `START-HERE.bat` again: it starts right away, nothing is downloaded twice. Close its
+window to stop the model.
+
+**Linux:** run `./setup.sh` - same questions, same result.
+
 ## Using it

+<p align="center"><img src="docs/media/runpagoda.png" width="900" alt="The Strata app's Monitor tab next to a coding agent"><br>
+<sub>The Strata app's <b>Monitor</b> (left) while a coding agent writes the pagoda garden from the video (right)</sub></p>
+
 - **In the browser:** `http://127.0.0.1:8080` - the Strata app (it opens by itself when the model starts): **Chat**, a
  live **Monitor** of the model and your GPU/CPU/RAM, and **About** with the settings and addresses.
 - **Chat in the terminal:** `.venv\Scripts\python chat.py`
@@ -109,17 +111,35 @@ Not sure? Take **IQ2_XS**. You can add another one later with `START-HERE.bat --

 ## Something went wrong?

-| What you see | What to do |
-| --- | --- |
-| `the NVIDIA driver is too old` | Update the driver (NVIDIA App or nvidia.com/drivers), restart the PC, run `START-HERE.bat` again. |
-| It stopped during download or setup | Run `START-HERE.bat` again - it continues where it stopped. |
-| `port 8080 is already in use` | Strata is already running - look for its window. |
-| The first start takes minutes | Normal: it loads 35-43 GB into RAM. The next start is faster. |
-| Slow, and the disk light is busy | Not enough free RAM: close other programs (browsers use a lot), or pick Q2_0 / IQ2_XS. |
-| "prompt exceeds the context" | The conversation is longer than the context you chose: run `START-HERE.bat --setup` and pick more. |
+**My PC froze, or got very slow, the first time Strata started.**
+That's normal the first time. Strata loads 35-55 GB into your RAM, locks part of it for the graphics card, and works
+out how much of the model fits on your GPU. The mouse can freeze for a few minutes. **Wait, and don't close the
+window.** The next starts are much faster. Still frozen after 10 minutes? Restart the PC, close other programs
+(browsers use a lot of RAM) and try again. If it keeps happening, pick a smaller size (Q2_0 or IQ2_XS).

-More in the [full troubleshooting table](docs/DETAILS.md#troubleshooting). Still stuck? Open an issue and attach
-`strata-<model>.log` from this folder.
+**It stopped while downloading or installing.**
+Run `START-HERE.bat` again. It continues where it stopped.
+
+**It says the NVIDIA driver is too old.**
+Update it (NVIDIA App or [nvidia.com/drivers](https://www.nvidia.com/drivers)), restart the PC, and run
+`START-HERE.bat` again.
+
+**It says port 8080 is already in use.**
+Strata is already running. Look for its window.
+
+**It's very slow and the disk light keeps blinking.**
+Your PC is out of free RAM. Close other programs, or pick a smaller size (Q2_0 or IQ2_XS).
+
+**An answer stopped with "the engine stopped unexpectedly".**
+Usually not enough RAM (on Linux the system then stops the engine). Just send your message again: Strata starts the
+engine by itself. If it keeps happening, close other programs or pick a smaller size.
+
+**It says the prompt exceeds the context.**
+The conversation is longer than the context you chose. Start a new chat, or run `START-HERE.bat --setup` and pick more
+context.
+
+**Still stuck?** Look in the [full troubleshooting table](docs/DETAILS.md#troubleshooting), or open an issue and
+attach `strata-<model>.log` from the Strata folder.

 ## How does it work?

@@ -371,14 +371,17 @@ the document, +0.4% on the chat. Details: `bench/results/2026-09-27-esp/`.
 | Python or the build tools could not be installed | Install what it names (links are printed), then run it again. Everything already done is kept. |
 | `port 8080 is already in use` | Strata is already running (look for its window), or another program uses the port: `START-HERE.bat --port 8081`. |
 | `cudaHostRegister ... out of memory` in the log | Normal on Windows: the engine pins the experts in per-layer slices instead. Only a problem if the load then fails. |
-| The first start takes minutes | It is reading 34-43 GB into RAM; the second start is faster while the files are in the OS cache. |
+| The first start takes minutes | It is reading 34-55 GB into RAM; the second start is faster while the files are in the OS cache. |
+| The PC freezes for a few minutes at the first start | Normal the first time: the engine loads the experts into RAM, pins part of it for the GPU and sizes the expert cache. Wait; don't close the window. Still frozen after 10 minutes: restart the PC, close other programs, try again, or pick a smaller size. |
+| `the engine stopped unexpectedly (exit code ...)` | The engine process ended mid-answer - usually out of RAM (Linux ends the biggest program: `sudo dmesg \| grep -i -E 'killed process\|out of memory'`). The next request starts it again by itself. If it repeats: close other programs or pick a smaller size. The server also warns at start when the model's experts leave less than ~6 GB of RAM for everything else. |
 | Slow output, disk light busy | Not enough free RAM: close other programs, or choose Q2_0 / IQ2_XS. |
 | `prompt ... exceeds the context` | The request is longer than the context you chose: run setup again with a bigger `--context`. |
 | Slower than the tables | The monitor plugged into the GPU and other GPU programs take VRAM from the expert cache; RAM running below its rated speed (enable EXPO/XMP in the BIOS) slows the CPU half. |
 | `this server was started without the vision encoder` | The model was set up for text only: run setup again with `--vision gpu`. |
 | A picture is refused or `cannot read the image` | The file is not a picture Pillow can open (JPEG, PNG, WebP, GIF, BMP, TIFF, AVIF work). |
 | Pictures are slow (10-30 s) | The encoder runs on the CPU: run setup again with `--vision gpu` (needs ~1.4 GB of VRAM). |
-| A request never finishes (older engines, mostly with pictures) | Run `START-HERE.bat` once to get engine 0.1.2 or newer. It keeps a real margin of VRAM free: the log says `... MiB of VRAM free with everything loaded` and warns when it is close to 0. |
+| A request never finishes: "reading the prompt", GPU "100%" at low power | The GPU ran out of VRAM (engines before 0.1.9 could end with ~30 MiB free at large contexts). Run `START-HERE.bat` once to get engine 0.1.9 or newer; the log then says `... MiB of VRAM free with everything loaded` (a few hundred) and names the `--vram-reserve-mib` to add if it is low. |
+| Generation stops mid-answer, GPU "100%", one CPU core busy | Fixed in engine 0.1.12 (issue #29, a race in the CPU expert pool on big-VRAM cards). Since then a request that stops moving for 2 minutes ends with an error instead of hanging: the log says `no progress for 120 s ... (issue #29)` with where it stopped, and the next request starts the engine again. If you see that line, please open an issue with it. (`STRATA_WATCHDOG_S` sets the 120 s; 0 turns it off.) |
 | Anything else | The engine log is `strata-<model>.log` in this folder. |

 ---
@@ -0,0 +1,35 @@
+// include/strata/core/progress.hpp - issue #29: whether a request is still moving, and where it is.
+//
+// The engine's host loop and the GPU wait on each other through flags; a protocol bug there does not crash, it
+// spins forever (the GPU "100%", one CPU core busy, no tokens).  `--serve` runs a watchdog thread over this: a
+// request whose heartbeat stops for `STRATA_WATCHDOG_S` seconds (default 120; 0 = off) ends the engine with the
+// stage it was stuck in, and the server starts it again instead of hanging.  Tokens, prompt chunks and verify
+// windows beat; the stage is two relaxed stores per layer, which the token path does not notice.
+#pragma once
+
+#include <atomic>
+#include <cstdint>
+
+namespace strata::core {
+
+struct Progress {
+    std::atomic<uint64_t> beats{0};
+    std::atomic<bool> busy{false};
+    std::atomic<const char*> where{"idle"};
+    std::atomic<int64_t> detail{-1};
+};
+
+inline Progress& progress() {
+    static Progress p;
+    return p;
+}
+
+inline void progress_at(const char* where, int64_t detail = -1) {
+    Progress& p = progress();
+    p.where.store(where, std::memory_order_relaxed);
+    p.detail.store(detail, std::memory_order_relaxed);
+}
+
+inline void progress_beat() { progress().beats.fetch_add(1, std::memory_order_relaxed); }
+
+}  // namespace strata::core
@@ -14,9 +14,17 @@
 // THE COMPLETION PROTOCOL, because this is where a pool usually goes wrong.  `run()` waits for `done == n`
 // AND for every worker to PARK.  Waiting only for `done` is not enough: a worker can still be inside the
 // drain loop after its last `done` increment, and the host resetting `head` underneath it would let that
-// worker claim a job from the NEXT batch before the next batch has been published.  The
-// `done`-then-`parked` pair makes the handover unambiguous, and the second wait costs a few hundred cycles
-// against a layer that takes milliseconds.
+// worker claim a job from the NEXT batch before the next batch has been published.
+//
+// **AND `parked` IS NOT ENOUGH EITHER (issue #29).**  A worker that went to sleep (after `kSpinBeforeSleep`) is
+// still counted as parked when it wakes, so for a moment after it has seen a new epoch the host believes it is
+// idle.  On a card with most experts in VRAM the workers sleep in the middle of a request, tiny batches finish
+// before a sleeper is awake, and that moment comes round constantly: a late worker could claim from the NEXT
+// batch while the host was still writing it, run a job twice, or add to `done` after the host had reset it -
+// and `done != n` then never ended, with the GPU waiting on the pool forever.  So every claim carries its
+// batch: `head` is one 64-bit word `epoch | njobs | index`, a claim is a CAS that only succeeds for the epoch
+// the worker woke for, and a late worker's claim simply fails.  A claim that succeeds belongs to the current
+// batch, whose description the host cannot change until that job's `done` has landed.
 #pragma once

 #include "strata/kernels/cpu/expert.hpp"
@@ -142,14 +150,26 @@ public:
    /// **A PARKED WORKER SPINS FOR THIS LONG, THEN SLEEPS.**  The park is a `_mm_pause` spin because a layer's
    /// batches are microseconds apart and a wake-up from the OS costs more than that.  But a spin that never ends
    /// keeps every worker's core at 100% while the engine waits for a request - issue #4, "CPU 50% even when
-    /// doing nothing" (7 of the 5700X's 16 threads).  Inside a request the gaps between batches are far below
-    /// this, so the token path still never sleeps; between requests the workers block on `sleep_cv_`.
+    /// doing nothing" (7 of the 5700X's 16 threads).  Between requests the workers block on `sleep_cv_`.  NOT only
+    /// between requests: with most experts in VRAM (a 24 GB card) many layers have no CPU work, so the workers
+    /// also sleep mid-request - which the claim protocol above must survive (issue #29).
+    /// `STRATA_POOL_SPIN_US` overrides it (a test knob: a short spin makes the workers sleep constantly).
    static constexpr std::chrono::milliseconds kSpinBeforeSleep{20};
+    /// A pool wait that sees no completion for this long is a bug; the engine stops with a message instead of
+    /// spinning forever, and the server starts it again (issue #29).
+    static constexpr std::chrono::seconds kStall{60};

 private:
    void worker(int id);
-    void drain(int id, ExpertScratch& scratch);
+    void drain(int id, ExpertScratch& scratch, uint32_t epoch);
    void run_phase(int mode, int n_tasks);
+    /// Claim the next job of batch `epoch`, or -1 (that batch is exhausted, or it is not the current one).
+    int claim(uint32_t epoch);
+    /// Publish the batch whose description the caller has just written: reset `done`, then `head`, then the epoch.
+    uint32_t begin_batch(int n);
+    /// The host's waits, bounded by `kStall`.
+    void wait_parked(const char* what);
+    void wait_done(int n);
    /// Bump `epoch_`, and wake the workers that went to sleep.  Every publish goes through here.
    void publish();

@@ -176,7 +196,7 @@ private:
    //
    // The counter was diagnostic only - `pauses()` was read in one place, to print a number nothing branched on
    // - so it is deleted rather than amortised.  `alignas(64)` then stops the remaining four sharing.
-    alignas(64) std::atomic<uint32_t> head_{0};
+    alignas(64) std::atomic<uint64_t> head_{0};   // epoch << 32 | njobs << 16 | next index (issue #29)
    alignas(64) std::atomic<uint32_t> done_{0};
    alignas(64) std::atomic<uint32_t> parked_{0};
    alignas(64) std::atomic<uint32_t> epoch_{0};
@@ -186,6 +206,7 @@ private:
    alignas(64) std::atomic<uint32_t> sleepers_{0};
    std::mutex sleep_mu_;
    std::condition_variable sleep_cv_;
+    std::chrono::microseconds spin_before_sleep_{kSpinBeforeSleep};
    std::vector<std::thread> threads_;
    std::vector<ExpertScratch> scratch_;   // one per worker: no allocation, no false sharing of the hot data
    // run_split state: mode 0 = whole experts, 1 = gate/up row parts, 2 = down row parts
@@ -72,6 +72,10 @@ class MockEngine:
            yield t


+class EngineDied(RuntimeError):
+    """The engine process ended in the middle of a request (issue #27: on Linux, the out-of-memory killer)."""
+
+
 class StrataEngine:
    """The resident engine: `strata --serve` reads `GEN <max_new> <ids>` lines and streams `T <id>` lines, then
    `DONE ...`.  Requests are serialized by the service's FIFO, so one pipe is enough.
@@ -83,6 +87,8 @@ class StrataEngine:

    def __init__(self, exe: str, args: list[str], cwd: str | None = None, log: str | None = None,
                 env: dict | None = None):
+        self.spawn = (exe, list(args), cwd, log, env)   # to start it again after it died (issue #27)
+        self.log_path = log
        self.log = open(log, "a", encoding="utf-8") if log else subprocess.DEVNULL
        self.proc = subprocess.Popen([exe, "--serve", *args], cwd=cwd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                                     stderr=self.log, text=True, encoding="utf-8", bufsize=1, env=env)
@@ -114,8 +120,47 @@ class StrataEngine:
    def _pump(self):
        for line in self.proc.stdout:
            self.lines.put(line)
+        self.ended = True                               # its output closed: it is gone, even before the OS says so
        self.lines.put(None)

+    def death_note(self) -> str:
+        """Why the engine most likely ended, from the end of its log: its own watchdog (issue #29), else RAM."""
+        tail = ""
+        try:
+            with open(self.log_path, "rb") as f:
+                f.seek(0, 2)
+                f.seek(max(0, f.tell() - 4096))
+                tail = f.read().decode("utf-8", "replace")
+        except (OSError, TypeError):
+            pass
+        for line in reversed(tail.splitlines()):
+            if "issue #29" in line:
+                return ("The engine stopped itself because it had stopped making progress - a hang it caught. Its log "
+                        "line: " + line.strip() + " - please report it at github.com/Niko1221/Strata/issues.")
+        return ("The usual cause is running out of RAM: Linux then ends the biggest program (check: sudo dmesg | "
+                "grep -i -E 'killed process|out of memory'); Windows slows down instead. Close other programs or use a "
+                "smaller model (Q2_0 / IQ2_XS).")
+
+    def alive(self) -> bool:
+        return not getattr(self, "ended", False) and self.proc.poll() is None
+
+    def exit_code(self):
+        try:
+            return self.proc.wait(timeout=5)
+        except subprocess.TimeoutExpired:
+            return None
+
+    def restart(self):
+        """Start the engine again (the same command) after it died; the new process has its own line queue."""
+        try:
+            self.proc.kill()
+        except OSError:
+            pass
+        info = dict(self.info)
+        self.ended = False
+        self.__init__(*self.spawn)
+        self.info = {**info, **self.info}
+
    def _parse_done(self, line):
        f = line.split()
        self.last = {"generated": int(f[1]), "prompt_tokens": int(f[2]), "prompt_ms": float(f[3]),
@@ -176,8 +221,11 @@ class StrataEngine:
        self.progress = None
        head = f"GENI {int(max_new)}{self.projection_key(sampling or {})} {embeddings}" if embeddings else \
            f"GEN {int(max_new)}{self.sampling_keys(sampling or {}) if not embeddings else ''}"
-        self.proc.stdin.write(f"{head} {','.join(str(int(t)) for t in ids)}\n")
-        self.proc.stdin.flush()
+        try:
+            self.proc.stdin.write(f"{head} {','.join(str(int(t)) for t in ids)}\n")
+            self.proc.stdin.flush()
+        except OSError:                                  # the pipe is gone: the engine died (not the client)
+            raise EngineDied(f"the engine stopped unexpectedly (exit code {self.exit_code()})") from None
        done = False
        try:
            while True:
@@ -190,7 +238,7 @@ class StrataEngine:
                    continue
                if line is None:
                    done = True
-                    raise RuntimeError("the engine process ended")
+                    raise EngineDied(f"the engine stopped unexpectedly (exit code {self.exit_code()})")
                if line.startswith("T "):
                    if cancel.is_set():
                        return
@@ -387,6 +435,8 @@ class Service:
        self.engine, self.tok, self.template, self.model, self.vision = engine, tokenizer, template, model_name, vision
        self.fit_max_tokens = fit_max_tokens          # --fit-max-tokens: clamp the output cap instead of 400
        self.sampling_defaults = dict(sampling_defaults or {})   # the run config's `sampling` block
+        self.shared = {}                              # the web app's Chat settings for every client (POST /settings)
+        self.shared_path = None                       # where they are kept between starts (next to the config)
        self.fifo = threading.Lock()
        self.embeddings = threading.local()           # the current request's image embeddings file (GENI)
        self.api_key = ""                              # when set, /v1/* needs it (Bearer or x-api-key)
@@ -396,6 +446,41 @@ class Service:
        self.stop_ids = set(tokenizer.encode(IM_END, parse_special=True) +
                            tokenizer.encode("<|endoftext|>", parse_special=True))

+    def set_shared(self, defaults) -> dict:
+        """The Chat settings every client gets for what it leaves out; {} / None = clients use their own again."""
+        self.shared = clean_shared_defaults(defaults)
+        if self.shared_path:
+            try:
+                if self.shared:
+                    Path(self.shared_path).write_text(json.dumps(self.shared, indent=1), encoding="utf-8")
+                else:
+                    Path(self.shared_path).unlink(missing_ok=True)
+            except OSError as e:
+                print(f"[strata] could not save the shared settings: {e}", flush=True)
+        return self.shared
+
+    def with_shared(self, req: dict, api: str) -> dict:
+        """The request with the shared thinking level and max tokens filled in where it has none of its own."""
+        s = self.shared
+        if not s:
+            return req
+        req = dict(req)
+        if "max_tokens" in s and not req.get("max_tokens") and not req.get("max_completion_tokens"):
+            req["max_tokens"] = s["max_tokens"]
+        effort = s.get("reasoning_effort")
+        if effort:
+            if api == "openai":
+                ctk = req.get("chat_template_kwargs") if isinstance(req.get("chat_template_kwargs"), dict) else {}
+                if not req.get("reasoning_effort") and not req.get("reasoning") and \
+                        "enable_thinking" not in ctk and "reasoning_effort" not in ctk:
+                    req["reasoning_effort"] = effort
+            elif not req.get("thinking") and not req.get("output_config"):
+                if effort == "none":
+                    req["thinking"] = {"type": "disabled"}
+                else:
+                    req["output_config"] = {"effort": effort}
+        return req
+
    def start_telemetry(self):
        """The hardware sampler behind GET /metrics (serve/telemetry.py), recording this server's tok/s too."""
        if getattr(self, "telemetry", None) is None:
@@ -518,9 +603,10 @@ class Service:

    def run(self, ids, thinking, tools, max_new, sampling, cancel) -> Iterator[tuple[str, object]]:
        """Yields ("event", Event) as text arrives, then ("done", {"finish": .., "completion_tokens": ..})."""
-        if self.sampling_defaults:     # config defaults; the request's own fields win (explicit 0 stays greedy)
+        defaults = {**self.sampling_defaults, **self.shared}   # the config's, then the Chat settings shared with apps
+        if defaults:                   # the request's own fields win (explicit 0 stays greedy)
            req_values = {k: v for k, v in (sampling or {}).items() if v is not None}
-            sampling = {**self.sampling_defaults, **req_values}
+            sampling = {**defaults, **req_values}
        parser = OutputParser(thinking=thinking, tools=tools, stream_tools=True)
        detok, n, finish = Detokenizer(self.tok), 0, "length"
        raw_ids = []                                    # every generated id (STRATA_DEBUG: dump raw model text)
@@ -531,6 +617,14 @@ class Service:
            with self.fifo:
                with self.status_lock:
                    self.status["queued"] -= 1
+                if hasattr(self.engine, "alive") and not self.engine.alive():
+                    # issue #27: it died in an earlier request - start it again instead of failing every request
+                    code = self.engine.exit_code() if hasattr(self.engine, "exit_code") else None
+                    print(f"[strata] the engine had stopped (exit code {code}); starting it again "
+                          "(a minute or two) ...", flush=True)
+                    self.engine.restart()
+                    print("[strata] the engine is running again", flush=True)
+                with self.status_lock:
                    self.status.update(busy=True, phase="reading the prompt", prompt_tokens=len(ids), generated=0,
                                       started=time.time(), first_token=None, tool=None, tail="", max_tokens=max_new)
                last_print = time.time()
@@ -555,6 +649,13 @@ class Service:
                            yield "event", ev
                    if cancel.is_set():
                        finish = "cancel"
+                except EngineDied as e:
+                    finish = "error"
+                    note = self.engine.death_note() if hasattr(self.engine, "death_note") else ""
+                    print(f"[strata] {e}. {note} The next request starts the engine again."
+                          f"{' Its log: ' + self.engine.log_path if getattr(self.engine, 'log_path', None) else ''}",
+                          flush=True)
+                    raise
                finally:
                    gen.close()                         # STOP+drain to THIS request's DONE while still holding the
                    #                                     fifo, so a stop-token break can't leave the shared engine
@@ -831,6 +932,10 @@ def make_handler(svc: Service):
                if self._authorized():
                    self._json(200, svc.metrics())
                return
+            if path == "/settings":
+                if self._authorized():
+                    self._json(200, {"shared": bool(svc.shared), "defaults": svc.shared})
+                return
            if path == "":
                body = (ROOT / "serve" / "web" / "index.html").read_bytes()
                self.send_response(200)
@@ -861,6 +966,9 @@ def make_handler(svc: Service):
        def do_POST(self):
            if not self._authorized():
                return
+            if self.path.rstrip("/") == "/settings":
+                self._settings()
+                return
            try:
                req = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0))) or b"{}")
                if self.path.rstrip("/") == "/v1/chat/completions":
@@ -871,6 +979,30 @@ def make_handler(svc: Service):
                    self._json(404, {"error": {"message": "not found"}})
            except ValueError as e:
                self._json(400, {"error": {"type": "invalid_request_error", "message": str(e)}})
+            except EngineDied as e:                          # before the answer started (not streamed)
+                self._json(503, {"error": {"type": "server_error", "message": f"{e}; the next request restarts it"}})
+
+        def _settings(self):
+            # They change what every client gets, so only the app's own page may set them: JSON only (a form or a
+            # "simple" cross-site request can't send it without a CORS preflight, which this server never grants),
+            # and no foreign Origin
+            body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
+            if not self.headers.get("Content-Type", "").startswith("application/json"):
+                self._json(415, {"error": {"message": "send application/json"}})
+                return
+            origin = self.headers.get("Origin")
+            if origin and origin.split("://", 1)[-1] != self.headers.get("Host", ""):
+                self._json(403, {"error": {"message": "settings can only be changed from Strata's own page"}})
+                return
+            try:
+                req = json.loads(body or b"{}")
+                shared = svc.set_shared(req.get("defaults") if isinstance(req, dict) else None)
+            except ValueError as e:
+                self._json(400, {"error": {"type": "invalid_request_error", "message": str(e)}})
+                return
+            print("[strata] other apps now use the Chat settings: " + ", ".join(f"{k}={v}" for k, v in shared.items())
+                  if shared else "[strata] other apps use their own settings again", flush=True)
+            self._json(200, {"shared": bool(shared), "defaults": shared})

        def _sse(self):
            self.send_response(200)
@@ -879,6 +1011,7 @@ def make_handler(svc: Service):
            self.end_headers()

        def _openai(self, req):
+            req = svc.with_shared(req, "openai")
            messages, tools, kw = openai_to_messages(req)
            max_new = int(req.get("max_completion_tokens") or req.get("max_tokens") or 0)   # 0/-1: the rest of the context
            ids, thinking, max_new = svc.prepare(messages, tools, kw, max_new)
@@ -899,8 +1032,12 @@ def make_handler(svc: Service):
            except OSError:
                cancel.set()                                 # client went away: stop the engine
                chunks.close()
+            except EngineDied as e:                          # mid-stream: say so, then end the stream properly
+                err = {"error": {"type": "server_error", "message": f"{e}; the next request restarts it"}}
+                self.wfile.write(b"data: " + json.dumps(err).encode() + b"\n\ndata: [DONE]\n\n")

        def _anthropic(self, req):
+            req = svc.with_shared(req, "anthropic")
            messages, tools, kw = anthropic_to_messages(req)
            max_new = int(req.get("max_tokens") or 0)                  # 0/-1: the rest of the context
            ids, thinking, max_new = svc.prepare(messages, tools, kw, max_new)
@@ -922,6 +1059,9 @@ def make_handler(svc: Service):
            except OSError:
                cancel.set()
                events.close()
+            except EngineDied as e:                          # mid-stream: Anthropic's error event
+                err = {"type": "error", "error": {"type": "api_error", "message": f"{e}; the next request restarts it"}}
+                self.wfile.write(b"event: error\ndata: " + json.dumps(err).encode() + b"\n\n")

    return Handler

@@ -932,6 +1072,26 @@ class Server(ThreadingHTTPServer):
    allow_reuse_address = os.name != "nt"


+def warn_tight_ram(arena_mib) -> None:
+    """The model's experts live in RAM (INFO arena_mib, engine 0.1.10+).  With less than ~6 GB left beside them for the
+    system, the engine and this server, Linux ends the engine mid-answer when memory runs out (issue #27) and Windows
+    pages to disk; say so at start instead of after a lost answer."""
+    if not isinstance(arena_mib, int) or arena_mib <= 0:
+        return
+    try:
+        import psutil
+        total = psutil.virtual_memory().total
+    except Exception:  # noqa: BLE001 - psutil is optional here
+        return
+    left = total / 2**30 - arena_mib / 1024
+    if left < 6:
+        print(f"[strata] WARNING: RAM is tight - the model's experts take {arena_mib / 1024:.1f} GB of this PC's "
+              f"{total / 2**30:.0f} GB, leaving {left:.1f} GB for everything else. "
+              + ("Linux may stop the engine in the middle of an answer. " if os.name != "nt" else
+                 "Windows will slow down (paging to disk). ")
+              + "Close other programs, or run START-HERE --setup and pick a smaller size (Q2_0 / IQ2_XS).", flush=True)
+
+
 def lan_addresses() -> list[str]:
    """This PC's IPv4 addresses on its networks (what another device types in), without loopback/link-local."""
    import socket
@@ -958,6 +1118,46 @@ def serve(svc: Service, host="127.0.0.1", port=8095) -> ThreadingHTTPServer:
    return httpd


+SHARED_KEYS = ("reasoning_effort", "temperature", "top_p", "top_k", "seed", "max_tokens", "experimental_speed_projection")
+
+
+def clean_shared_defaults(d) -> dict:
+    """The Chat settings other apps get (POST /settings): only known keys, each checked; ValueError names a bad one."""
+    if d is None:
+        return {}
+    if not isinstance(d, dict):
+        raise ValueError("defaults must be an object")
+    out = {}
+    for key, value in d.items():
+        if value is None or value == "":
+            continue
+        number = isinstance(value, (int, float)) and not isinstance(value, bool)
+        if key == "reasoning_effort":
+            if value not in ("none", "low", "medium", "high"):
+                raise ValueError("reasoning_effort: none, low, medium or high")
+        elif key == "temperature":
+            if not number or not 0 <= value <= 2:
+                raise ValueError("temperature: 0..2")
+        elif key == "top_p":
+            if not number or not 0 < value <= 1:
+                raise ValueError("top_p: 0 < top_p <= 1")
+        elif key == "top_k":
+            if not number or value != int(value) or not 1 <= value <= 64:
+                raise ValueError("top_k: an integer 1..64")
+            value = int(value)
+        elif key in ("seed", "max_tokens"):
+            if not number or value != int(value) or value <= 0:
+                raise ValueError(f"{key}: a positive integer")
+            value = int(value)
+        elif key == "experimental_speed_projection":
+            if not isinstance(value, bool):
+                raise ValueError("experimental_speed_projection: true or false")
+        else:
+            raise ValueError(f"unknown setting {key!r}")
+        out[key] = float(value) if key in ("temperature", "top_p") else value
+    return out
+
+
 def sampling_defaults_from_config(cfg: dict) -> dict:
    """The run config's optional `sampling` block: defaults for the sampling fields a request leaves out, so
    a plain client gets configured sampling instead of greedy.  Supported: temperature, top_p, top_k, min_p,
@@ -1072,6 +1272,7 @@ def main() -> int:
                            env=env)
        print("loading the model (the first start takes a minute or two) ...", flush=True)
        engine = StrataEngine(cfg["exe"], cfg["args"], cwd=cfg.get("cwd"), log=cfg.get("log"), env=env)
+        warn_tight_ram(engine.info.get("arena_mib"))
    else:
        engine, vision, sampling_defaults = MockEngine(tok, a.script), None, {}
    # the model's own chat template (exported with its tokenizer), else the original model's
@@ -1081,6 +1282,15 @@ def main() -> int:
                  sampling_defaults=sampling_defaults,
                  fit_max_tokens=a.fit_max_tokens or cfg.get("fit_max_tokens") is True)
    svc.api_key = a.api_key or cfg.get("api_key", "")
+    if a.config:                                        # the Chat settings shared with other apps, from last time
+        svc.shared_path = str(Path(a.config).with_suffix("")) + ".shared-settings.json"
+        try:
+            svc.shared = clean_shared_defaults(json.loads(Path(svc.shared_path).read_text(encoding="utf-8")))
+            if svc.shared:
+                print("[strata] other apps use the Chat settings: " +
+                      ", ".join(f"{k}={v}" for k, v in svc.shared.items()), flush=True)
+        except (OSError, ValueError):
+            svc.shared = {}
    httpd = serve(svc, host=a.host, port=a.port)
    here = "127.0.0.1" if a.host in ("0.0.0.0", "", "::") else a.host
    print(f"ready: http://{here}:{a.port}/v1  (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, "
@@ -14,7 +14,7 @@ from pathlib import Path

 sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
 from serve.frontend import ChatTemplate  # noqa: E402
-from serve.server import CTX_SLACK, ByteTokenizer, MockEngine, Service, serve  # noqa: E402
+from serve.server import CTX_SLACK, ByteTokenizer, EngineDied, MockEngine, Service, serve  # noqa: E402

 ROOT = Path(__file__).resolve().parents[1]
 CTX = 4096
@@ -165,6 +165,157 @@ class FitMaxTokens(unittest.TestCase):
        self.assertIn("no room to answer", b["error"]["message"])


+class DyingEngine(MockEngine):
+    """Issue #27: an engine that dies after a few tokens of its first answer, and comes back when restarted."""
+
+    def __init__(self, tok, script, max_context):
+        super().__init__(tok, script, max_context=max_context)
+        self.dead, self.restarts, self.die_after = False, 0, 5
+
+    def alive(self):
+        return not self.dead
+
+    def restart(self):
+        self.dead, self.die_after = False, None
+        self.restarts += 1
+
+    def generate(self, ids, max_new, sampling, cancel, embeddings=None):
+        for i, t in enumerate(super().generate(ids, max_new, sampling, cancel, embeddings)):
+            if self.die_after is not None and i == self.die_after:
+                self.dead = True
+                raise EngineDied("the engine stopped unexpectedly (exit code -9)")
+            yield t
+
+
+class EngineDeath(unittest.TestCase):
+    """Issue #27: a dead engine is an error (not "length"), and the next request starts it again."""
+
+    def test_error_then_restart(self):
+        tok = ByteTokenizer()
+        eng = DyingEngine(tok, "</think>\n\n" + ANSWER, max_context=CTX)
+        svc = Service(eng, tok, ChatTemplate(ROOT / "serve/chat_template.jinja"))
+        httpd = serve(svc, port=0)
+        base = f"http://127.0.0.1:{httpd.server_address[1]}"
+        try:
+            def post(body):
+                req = urllib.request.Request(base + "/v1/chat/completions", data=json.dumps(body).encode(),
+                                             headers={"Content-Type": "application/json"})
+                try:
+                    with urllib.request.urlopen(req, timeout=30) as r:
+                        return r.status, r.read().decode()
+                except urllib.error.HTTPError as e:
+                    with e:
+                        return e.code, e.read().decode()
+            msgs = [{"role": "user", "content": "hi"}]
+            code, text = post({"model": "m", "messages": msgs, "max_tokens": 50, "stream": True})
+            self.assertEqual(code, 200)
+            self.assertIn('"error"', text)
+            self.assertIn("stopped unexpectedly", text)
+            self.assertTrue(text.rstrip().endswith("data: [DONE]"))
+            self.assertEqual(svc.metrics()["requests"][0]["finish"], "error")
+            code, text = post({"model": "m", "messages": msgs, "max_tokens": 50})
+            self.assertEqual(code, 200, text)
+            self.assertEqual(eng.restarts, 1)
+            self.assertEqual(json.loads(text)["usage"]["completion_tokens"], 50)
+        finally:
+            httpd.shutdown()
+            httpd.server_close()
+
+
+class SharedSettings(unittest.TestCase):
+    """The web app's "Use for other apps too": POST /settings makes its Chat settings every client's defaults."""
+
+    @classmethod
+    def setUpClass(cls):
+        import tempfile
+
+        class Sampled(RecordingEngine):
+            def generate(self, ids, max_new, sampling, cancel, embeddings=None):
+                self.last_sampling = dict(sampling or {})
+                yield from super().generate(ids, max_new, sampling, cancel, embeddings)
+
+        tok = ByteTokenizer()
+        cls.engine = Sampled(tok, "</think>\n\nhello", max_context=CTX)
+        cls.svc = Service(cls.engine, tok, ChatTemplate(ROOT / "serve/chat_template.jinja"))
+        cls.tmp = tempfile.TemporaryDirectory()
+        cls.svc.shared_path = os.path.join(cls.tmp.name, "strata-x.shared-settings.json")
+        cls.httpd = serve(cls.svc, port=0)
+        cls.base = f"http://127.0.0.1:{cls.httpd.server_address[1]}"
+
+    @classmethod
+    def tearDownClass(cls):
+        cls.httpd.shutdown()
+        cls.httpd.server_close()
+        cls.tmp.cleanup()
+
+    def req(self, path, body, headers=None, raw=None):
+        h = {"Content-Type": "application/json", **(headers or {})}
+        r = urllib.request.Request(self.base + path, data=raw if raw is not None else json.dumps(body).encode(), headers=h)
+        try:
+            with urllib.request.urlopen(r, timeout=30) as resp:
+                return resp.status, json.loads(resp.read())
+        except urllib.error.HTTPError as e:
+            with e:
+                return e.code, json.loads(e.read())
+
+    def chat(self, **extra):
+        return self.req("/v1/chat/completions", {"model": "m", "messages": [{"role": "user", "content": "hi"}], **extra})
+
+    def tearDown(self):
+        self.svc.set_shared(None)
+
+    def test_other_apps_get_the_chat_settings(self):
+        d = {"temperature": 0.3, "top_p": 0.9, "top_k": 10, "seed": 7, "max_tokens": 77,
+             "reasoning_effort": "low", "experimental_speed_projection": False}
+        code, b = self.req("/settings", {"defaults": d})
+        self.assertEqual(code, 200, b)
+        self.assertTrue(b["shared"])
+        self.assertTrue(os.path.exists(self.svc.shared_path))
+        code, _ = self.chat()                                        # a client that sets nothing
+        self.assertEqual(code, 200)
+        got = self.engine.last_sampling
+        for k in ("temperature", "top_p", "top_k", "seed", "experimental_speed_projection"):
+            self.assertEqual(got[k], d[k], k)
+        self.assertEqual(self.engine.last_max_new, 77)
+        code, _ = self.chat(temperature=0.9, max_tokens=5)          # its own values win
+        self.assertEqual(self.engine.last_sampling["temperature"], 0.9)
+        self.assertEqual(self.engine.last_max_new, 5)
+        r = self.svc.with_shared({"messages": []}, "openai")
+        self.assertEqual(r["reasoning_effort"], "low")
+        self.assertEqual(self.svc.with_shared({"reasoning_effort": "high"}, "openai")["reasoning_effort"], "high")
+        self.assertEqual(self.svc.with_shared({}, "anthropic")["output_config"], {"effort": "low"})
+
+    def test_off_again(self):
+        self.req("/settings", {"defaults": {"temperature": 0.3}})
+        code, b = self.req("/settings", {"defaults": None})
+        self.assertEqual((code, b["shared"]), (200, False))
+        self.assertFalse(os.path.exists(self.svc.shared_path))
+        self.chat()
+        self.assertNotIn("temperature", self.engine.last_sampling)
+
+    def test_only_strata_s_own_page_may_set_them(self):
+        code, _ = self.req("/settings", None, {"Content-Type": "text/plain"}, raw=b'{"defaults": {"temperature": 1}}')
+        self.assertEqual(code, 415)
+        code, _ = self.req("/settings", {"defaults": {"temperature": 1}}, {"Origin": "http://evil.example"})
+        self.assertEqual(code, 403)
+        code, b = self.req("/settings", {"defaults": {"temperature": 9}})
+        self.assertEqual(code, 400)
+        self.assertIn("temperature", b["error"]["message"])
+        self.assertEqual(self.svc.shared, {})
+        host = self.base.split("://", 1)[1]
+        code, _ = self.req("/settings", {"defaults": {"temperature": 1}}, {"Origin": "http://" + host})
+        self.assertEqual(code, 200)
+
+    def test_they_need_the_key_when_one_is_set(self):
+        self.svc.api_key = "secret"
+        try:
+            self.assertEqual(self.req("/settings", {"defaults": {"temperature": 1}})[0], 401)
+            self.assertEqual(self.req("/settings", {"defaults": {"temperature": 1}},
+                                      {"Authorization": "Bearer secret"})[0], 200)
+        finally:
+            self.svc.api_key = ""
+
+
 class WebApp(unittest.TestCase):
    """The web app (PR #22's dashboard idea, rebuilt): its page and files, and GET /metrics."""

@@ -58,6 +58,7 @@ main { flex: 1; min-height: 0; display: flex; }
 .st-btn--send { width: 42px; padding: 0; }
 .composer-hint { font-size: var(--st-fs-xs); color: var(--st-ink-muted); margin-right: var(--st-s-2); font-variant-numeric: tabular-nums; }
 .attachments { display: flex; flex-wrap: wrap; gap: var(--st-s-2); padding: 2px 4px 0; }
+.st-composer.dragging { outline: 2px dashed var(--st-accent); outline-offset: 3px; }
 .chip { display: inline-flex; align-items: center; gap: 6px; height: 30px; padding: 0 6px 0 10px; border-radius: var(--st-r-pill);
        background: var(--st-surface-2); border: 1px solid var(--st-line); font-size: var(--st-fs-xs); color: var(--st-ink-soft); }
 .chip .st-btn--icon { width: 22px; height: 22px; border-radius: 50%; }
@@ -107,6 +108,7 @@ main { flex: 1; min-height: 0; display: flex; }
 .facts .st-btn--icon .st-icon { width: 16px; height: 16px; }
 .toggle-row { display: flex; align-items: center; justify-content: space-between; gap: var(--st-s-4); font-size: var(--st-fs-sm);
              font-weight: var(--st-fw-medium); color: var(--st-ink-soft); }
+.toggle-row .st-toggle { flex: none; }   /* a long label must not squeeze the switch */

 /* ---------------------------------------------------------------- drawer */
 .st-drawer { z-index: 40; }
@@ -98,7 +98,8 @@ let health = {model: "strata", images: false, max_context: 0};
 async function loadHealth() {
  try {
    health = await (await fetch("/health")).json();
-    $("attach-btn").hidden = !health.images;
+    $("attach-btn").title = health.images ? "Attach a text file or a picture (or drop it here)"
+                                          : "Attach a text file (or drop it here)";
    $("chat-empty-sub").textContent = `${health.model} runs on this PC. Nothing leaves it.`;
  } catch (e) {
    setTimeout(loadHealth, 2000);
@@ -394,7 +395,8 @@ let attachments = [];                 // {name, url}
 let busy = null;                      // {controller, msg}

 function saveChat() {
-  store.set("chat", messages.map((m) => ({...m, images: (m.images || []).map((i) => ({name: i.name}))})));
+  store.set("chat", messages.map((m) => ({...m, images: (m.images || []).map((i) => ({name: i.name})),
+                                           files: (m.files || []).map((f) => ({name: f.name}))})));
 }
 function timeStr(t) { return new Date(t).toLocaleTimeString([], {hour: "2-digit", minute: "2-digit"}); }

@@ -403,6 +405,18 @@ function msgEl(m, i) {
  el.className = `st-msg st-msg--${m.role}`;
  el.dataset.i = i;
  if (m.role === "user") {
+    if (m.files && m.files.length) {
+      const wrap = document.createElement("div");
+      wrap.className = "msg-images";
+      for (const f of m.files) {
+        const c = document.createElement("span");
+        c.className = "chip";
+        c.innerHTML = icon("attach", "st-icon st-icon--sm");
+        c.append(f.name);
+        wrap.appendChild(c);
+      }
+      el.appendChild(wrap);
+    }
    if (m.images && m.images.length) {
      const wrap = document.createElement("div");
      wrap.className = "msg-images";
@@ -486,8 +500,9 @@ function apiMessages() {
  for (const m of messages) {
    if (m.role === "user") {
      const imgs = (m.images || []).filter((i) => i.url);
-      out.push({role: "user", content: imgs.length ? [{type: "text", text: m.text},
-        ...imgs.map((i) => ({type: "image_url", image_url: {url: i.url}}))] : m.text});
+      const text = userText(m);
+      out.push({role: "user", content: imgs.length ? [{type: "text", text},
+        ...imgs.map((i) => ({type: "image_url", image_url: {url: i.url}}))] : text});
    } else if (m.text && !m.error) {
      out.push({role: "assistant", content: m.text});
    }
@@ -504,7 +519,8 @@ function setBusy(on) {
 async function send() {
  const text = $("input").value.trim();
  if ((!text && !attachments.length) || busy) return;
-  messages.push({role: "user", text, images: attachments, time: Date.now()});
+  messages.push({role: "user", text, images: attachments.filter((a) => a.kind !== "file"),
+                 files: attachments.filter((a) => a.kind === "file"), time: Date.now()});
  attachments = [];
  renderAttachments();
  $("input").value = "";
@@ -615,16 +631,45 @@ $("export-btn").onclick = () => {
  setTimeout(() => URL.revokeObjectURL(a.href), 5000);
 };

-// pictures: the attach button, or paste into the message box
+// pictures and text files: the attach button, dropping them on the chat, or pasting a picture (issue #30)
+const TEXT_EXT = /\.(txt|md|markdown|rst|tex|py|pyi|ipynb|js|mjs|cjs|ts|tsx|jsx|vue|svelte|json|jsonl|csv|tsv|log|ya?ml|toml|ini|cfg|conf|env|xml|html?|css|scss|less|c|cc|cpp|cxx|h|hh|hpp|cu|cuh|rs|go|java|kt|kts|swift|rb|php|pl|lua|r|jl|scala|sql|sh|bash|zsh|fish|ps1|psm1|bat|cmd|diff|patch|gradle|cmake|mk|dockerfile|gitignore|proto|graphql)$/i;
+const MAX_TEXT_FILE = 512 * 1024;
+function isTextFile(f) {
+  return f.type.startsWith("text/") || /json|xml|javascript|yaml|toml|x-sh|x-python/.test(f.type) ||
+         TEXT_EXT.test(f.name) || /(^|[\\/])(makefile|dockerfile|readme|license)$/i.test(f.name);
+}
 function addFiles(files) {
  for (const f of files) {
-    if (!f.type.startsWith("image/")) continue;
-    if (f.size > 20e6) { toast("warn", "Picture too large", `${f.name} is over 20 MB.`); continue; }
+    if (f.type.startsWith("image/")) {
+      if (!health.images) { toast("warn", "Pictures are off", "This model was set up for text only."); continue; }
+      if (f.size > 20e6) { toast("warn", "Picture too large", `${f.name} is over 20 MB.`); continue; }
+      const r = new FileReader();
+      r.onload = () => { attachments.push({kind: "image", name: f.name || "pasted image", url: r.result}); renderAttachments(); };
+      r.readAsDataURL(f);
+      continue;
+    }
+    if (!isTextFile(f)) { toast("warn", "Not a text file", `${f.name}: attach text files (code, notes, logs, data)${health.images ? " or pictures" : ""}.`); continue; }
+    if (f.size > MAX_TEXT_FILE) { toast("warn", "File too large", `${f.name} is over 512 KB.`); continue; }
    const r = new FileReader();
-    r.onload = () => { attachments.push({name: f.name || "pasted image", url: r.result}); renderAttachments(); };
-    r.readAsDataURL(f);
+    r.onload = () => {
+      const text = String(r.result);
+      if (text.includes("\u0000")) { toast("warn", "Not a text file", `${f.name} looks like a binary file.`); return; }
+      attachments.push({kind: "file", name: f.name, text});
+      renderAttachments();
+    };
+    r.readAsText(f);
  }
 }
+// a file's text in the message, fenced with more backticks than it contains itself
+function fileBlock(f) {
+  const longest = Math.max(2, ...(f.text.match(/`+/g) || []).map((s) => s.length));
+  const fence = "`".repeat(longest + 1);
+  return `File: ${f.name}\n${fence}\n${f.text}\n${fence}`;
+}
+function userText(m) {
+  const files = (m.files || []).filter((f) => f.text != null);
+  return [m.text, ...files.map(fileBlock)].filter((s) => s).join("\n\n");
+}
 function renderAttachments() {
  const box = $("attachments");
  box.hidden = !attachments.length;
@@ -632,7 +677,7 @@ function renderAttachments() {
  attachments.forEach((a, i) => {
    const c = document.createElement("span");
    c.className = "chip";
-    c.innerHTML = icon("image", "st-icon st-icon--sm");
+    c.innerHTML = icon(a.kind === "file" ? "attach" : "image", "st-icon st-icon--sm");
    c.append(a.name);
    const x = document.createElement("button");
    x.type = "button"; x.className = "st-btn st-btn--icon"; x.setAttribute("aria-label", "Remove");
@@ -643,6 +688,23 @@ function renderAttachments() {
  });
 }
 $("attach-btn").onclick = () => $("file").click();
+// drop files on the chat or the message box
+for (const id of ["chat", "composer"]) {
+  const el = $(id);
+  el.addEventListener("dragover", (e) => {
+    if (![...(e.dataTransfer || {}).types || []].includes("Files")) return;
+    e.preventDefault();
+    $("composer").classList.add("dragging");
+  });
+  el.addEventListener("dragleave", () => $("composer").classList.remove("dragging"));
+  el.addEventListener("drop", (e) => {
+    $("composer").classList.remove("dragging");
+    if (!e.dataTransfer || !e.dataTransfer.files.length) return;
+    e.preventDefault();
+    addFiles(e.dataTransfer.files);
+    $("input").focus();
+  });
+}
 $("file").onchange = () => { addFiles($("file").files); $("file").value = ""; };
 $("input").addEventListener("paste", (e) => {
  if (!health.images) return;
@@ -655,7 +717,7 @@ function openDrawer(open) {
  $("drawer").dataset.open = String(open);
  $("drawer").setAttribute("aria-hidden", String(!open));
  $("scrim").hidden = !open;
-  if (open) loadDrawer();
+  if (open) { loadDrawer(); loadShared(); }
 }
 function loadDrawer(s = settings) {
  for (const b of $("s-thinking").children) b.setAttribute("aria-checked", String(b.dataset.v === s.thinking));
@@ -664,8 +726,36 @@ function loadDrawer(s = settings) {
  $("s-show").setAttribute("aria-checked", String(!!s.show));
  $("s-esp").setAttribute("aria-checked", String(s.esp !== false));
  $("esp-row").hidden = !projectionLoaded();
+  $("s-share").setAttribute("aria-checked", String(sharedOn));
  outputs();
 }
+// "Use for other apps too": the server keeps these settings as every client's defaults (GET/POST /settings)
+let sharedOn = false;
+async function loadShared() {
+  try {
+    const r = await fetch("/settings", {headers: headers()});
+    if (r.ok) sharedOn = !!(await r.json()).shared;
+  } catch (e) { /* an older server: the switch just stays off */ }
+  $("s-share").setAttribute("aria-checked", String(sharedOn));
+}
+function sharedDefaults(s) {
+  const d = {reasoning_effort: s.thinking, temperature: +s.temperature};
+  if (+s.temperature > 0) Object.assign(d, {top_p: +s.top_p, top_k: +s.top_k});
+  if (s.seed) d.seed = +s.seed;
+  if (s.max) d.max_tokens = +s.max;
+  if (projectionLoaded()) d.experimental_speed_projection = s.esp !== false;
+  return d;
+}
+async function saveShared(on, s) {
+  const r = await fetch("/settings", {method: "POST", headers: headers(true),
+                                      body: JSON.stringify({defaults: on ? sharedDefaults(s) : null})});
+  if (!r.ok) {
+    let msg = `HTTP ${r.status}`;
+    try { msg = (await r.json()).error.message || msg; } catch (e) { /* not json */ }
+    throw new Error(msg);
+  }
+  sharedOn = !!(await r.json()).shared;
+}
 // the engine was started with the experimental-speed-projection control vector (INFO cvec=...)
 function projectionLoaded() {
  const c = lastMetrics && lastMetrics.engine ? lastMetrics.engine.cvec : 0;
@@ -684,15 +774,27 @@ for (const b of $("s-thinking").children) b.onclick = () => { for (const x of $(
 for (const id of ["s-temp", "s-topp", "s-topk"]) $(id).oninput = outputs;
 $("s-show").onclick = () => $("s-show").setAttribute("aria-checked", String($("s-show").getAttribute("aria-checked") !== "true"));
 $("s-esp").onclick = () => $("s-esp").setAttribute("aria-checked", String($("s-esp").getAttribute("aria-checked") !== "true"));
+$("s-share").onclick = () => $("s-share").setAttribute("aria-checked", String($("s-share").getAttribute("aria-checked") !== "true"));
 $("s-reset").onclick = () => loadDrawer(DEFAULTS);
-$("s-apply").onclick = () => {
+$("s-apply").onclick = async () => {
  const sel = [...$("s-thinking").children].find((b) => b.getAttribute("aria-checked") === "true");
  settings = {thinking: sel ? sel.dataset.v : "high", temperature: +$("s-temp").value, top_p: +$("s-topp").value,
              top_k: +$("s-topk").value, max: $("s-max").value.trim(), seed: $("s-seed").value.trim(),
              show: $("s-show").getAttribute("aria-checked") === "true",
              esp: $("s-esp").getAttribute("aria-checked") === "true"};
  store.set("sampling", settings);
+  const share = $("s-share").getAttribute("aria-checked") === "true";
  openDrawer(false);
+  if (share || sharedOn) {
+    try {
+      await saveShared(share, settings);
+      toast("success", "Sampling saved", share ? "Other apps (omp, API clients) use these settings from their next request."
+                                               : "Other apps use their own settings again.");
+    } catch (e) {
+      toast("error", "Saved here, but not for other apps", e.message, 6000);
+    }
+    return;
+  }
  toast("success", "Sampling saved", settings.temperature === 0 ? "Greedy: the same question gives the same answer." : "");
 };
 $("sampling-btn").onclick = () => openDrawer(true);
@@ -50,9 +50,9 @@
        <div class="attachments" id="attachments" hidden></div>
        <textarea id="input" rows="2" placeholder="Ask anything&hellip;" aria-label="Message"></textarea>
        <div class="st-composer__bar">
-          <button type="button" class="st-btn st-btn--icon" id="attach-btn" title="Attach a picture" aria-label="Attach a picture" hidden>
+          <button type="button" class="st-btn st-btn--icon" id="attach-btn" title="Attach a text file (or drop it here)" aria-label="Attach a file">
            <svg class="st-icon"><use href="/web/sprite.svg#i-attach"/></svg></button>
-          <input type="file" id="file" accept="image/*" multiple hidden>
+          <input type="file" id="file" multiple hidden>
          <button type="button" class="st-btn st-btn--icon" id="new-btn" title="New chat" aria-label="New chat">
            <svg class="st-icon"><use href="/web/sprite.svg#i-new-chat"/></svg></button>
          <button type="button" class="st-btn st-btn--icon" id="export-btn" title="Save this chat as Markdown" aria-label="Save this chat">
@@ -195,6 +195,9 @@
    <div class="toggle-row" id="esp-row" hidden><span>Experimental speed projection<br><span class="muted small">the
      engine's control vector; off = the stock model. Switching reads the chat again once</span></span>
      <button class="st-toggle" id="s-esp" role="switch" aria-checked="true" aria-label="Experimental speed projection"></button></div>
+    <div class="toggle-row"><span>Use for other apps too<br><span class="muted small">omp and other API clients get
+      these settings for anything they don't set themselves</span></span>
+      <button class="st-toggle" id="s-share" role="switch" aria-checked="false" aria-label="Use for other apps too"></button></div>
    <div class="drawer-actions">
      <button class="st-btn st-btn--secondary" id="s-reset" type="button">Reset</button>
      <button class="st-btn st-btn--primary" id="s-apply" type="button">Apply</button>
@@ -56,7 +56,7 @@ PREBUILT_ASSET = "strata-windows-x64.zip" if WIN else "strata-linux-x64.zip"
 # the CUDA libraries the ready-made engine loads (the same CUDA 13.0 it is built with), from NVIDIA's pip packages
 CUDA_WHEELS = ["nvidia-cublas==13.0.2.14", "nvidia-cuda-runtime==13.0.96"]
 MIN_DRIVER = 580                       # CUDA 13.0
-MIN_ENGINE = (0, 1, 9)                 # the head loaded before the expert cache is sized (no 128K stalls), v0.1.9
+MIN_ENGINE = (0, 1, 12)                # the expert-pool race and the serve watchdog (issue #29), v0.1.12
 PY_PACKAGES = ["numpy", "jinja2", "regex", "pyyaml", "tqdm", "requests", "cmake", "ninja", "pillow", "psutil"]

 MODELS = {
@@ -1,5 +1,6 @@
 // src/core/session.cpp - one token through all 48 layers.  See the header for why the graphs are per-layer.
 #include "strata/core/session.hpp"
+#include "strata/core/progress.hpp"

 #include "strata/kernels/qsa.hpp"
 #include "strata/kernels/elementwise.hpp"
@@ -888,6 +889,7 @@ bool session_run_token(const ModelGeometry& g, int64_t pos, int32_t pos_base, Se
        const auto t0 = Clock::now();
        auto last_flush = t0;
        uint32_t spins = 0;
+        progress_at("token: waiting for the GPU to reach layer", l);
        while (*seq < want) {
            STRATA_SPIN_PAUSE();
            if ((++spins & 1023u) != 0) continue;
@@ -909,6 +911,7 @@ bool session_run_token(const ModelGeometry& g, int64_t pos, int32_t pos_base, Se
            }
        }
        const auto t1 = Clock::now();
+        progress_at("token: the CPU experts of layer", l);
        if (pool != nullptr) pool(user, s.db->h_x_f, s.db->h_ids, s.db->h_weights, g.n_embd, s.k, y_miss_host);
        std::atomic_thread_fence(std::memory_order_seq_cst);
        _mm_sfence();
@@ -917,8 +920,11 @@ bool session_run_token(const ModelGeometry& g, int64_t pos, int32_t pos_base, Se
        tg.ms_wait += std::chrono::duration<double, std::milli>(t1 - t0).count();
        tg.ms_pool += std::chrono::duration<double, std::milli>(t2 - t1).count();
    }
+    progress_at("token: waiting for the GPU to finish the token");
    const cudaError_t se = cudaStreamSynchronize(cs);
    if (se != cudaSuccess) { err = std::string("session_run_token: ") + cudaGetErrorString(se); return false; }
+    progress_at("decode");
+    progress_beat();
    return true;
 }

@@ -28,6 +28,7 @@
 #include "strata/kernels/rope.hpp"
 #include "strata/kernels/s2_expert_grouped.hpp"
 #include "strata/kernels/sampler.hpp"
+#include "strata/core/progress.hpp"
 #include "strata/kernels/shared_expert.hpp"
 #include "strata/kernels/verify_kernels.hpp"

@@ -772,6 +773,7 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool,
        const Clock::time_point a = Clock::now();
        auto last_flush = a;
        uint32_t spins = 0;
+        progress_at("verify window: waiting for the GPU to reach layer", l);
        while (*seq < want) {
            _mm_pause();
            if ((++spins & 1023u) != 0) continue;
@@ -792,6 +794,7 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool,
        cur_layer_ = want - 1;
        set_plan_slot(grp);
        const int tb = gtb[grp], n = gte[grp] - gtb[grp];
+        progress_at("verify window: the CPU experts of layer", l);
        if (pool != nullptr)
            pool(user, h_x_ + (size_t) tb * g.n_embd, h_ids_ + (size_t) tb * ss.k, n, ss.k,
                 h_ymiss_ + (size_t) tb * ss.k * g.n_embd, l);
@@ -812,8 +815,10 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool,
        ms_wait += std::chrono::duration<double, std::milli>(b - a).count();
        ms_pool += ms_since(b);
    }
+    progress_at("verify window: waiting for the GPU to finish the window (flags A/B/M raised)", (int64_t) T);
    const cudaError_t se = cudaStreamSynchronize(cs_);
    if (se != cudaSuccess) { err = std::string("verify: ") + cudaGetErrorString(se); return false; }
+    progress_at("verify window: waiting for the expert copies", (int64_t) T);
    cudaStreamSynchronize(copy_);   // no host function of this window may raise flag B in the next one
    // ---- a sampled or penalized request: the head's sampling again, host-side so its parameters are this call's
    // own (a captured kernel would replay the same draws forever).  Row t's draw is Philox(seed, pos0 + t): tied to
@@ -832,6 +837,8 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool,
    for (int t = 0; t < T; ++t) out[t] = ((volatile int32_t*) h_out_)[t];
    VDBG("window done\n");
    ++windows;
+    progress_at("decode");
+    progress_beat();
    return true;
 }

@@ -8,6 +8,7 @@
 #include <immintrin.h>

 #include <cstdio>
+#include <cstdlib>

 #if defined(_WIN32)
 #define WIN32_LEAN_AND_MEAN
@@ -19,6 +20,12 @@

 namespace strata::kernels::cpu {

+namespace {
+constexpr uint64_t pack_head(uint32_t epoch, uint32_t n, uint32_t i) {
+    return ((uint64_t) epoch << 32) | ((uint64_t) n << 16) | (uint64_t) i;
+}
+}  // namespace
+
 std::vector<int> physical_cores(bool skip_first) {
    std::vector<int> cores;
 #if defined(_WIN32)
@@ -109,6 +116,8 @@ void restore_thread_affinity(long long previous) {
 }

 ExpertPool::ExpertPool(int n_workers, bool pin, bool host_works) : host_works_(host_works) {
+    if (const char* e = std::getenv("STRATA_POOL_SPIN_US"))   // a test knob; see kSpinBeforeSleep
+        spin_before_sleep_ = std::chrono::microseconds((std::max)(0, std::atoi(e)));
    const std::vector<int> cores = physical_cores(true);
    n_ = n_workers > 0 ? n_workers : (int) cores.size();
    if (n_ < 1) n_ = 1;
@@ -170,7 +179,7 @@ void ExpertPool::worker(int id) {
            if (stop_.load(std::memory_order_relaxed)) return;
            _mm_pause();
            if ((++spins & 1023u) != 0) continue;
-            if (std::chrono::steady_clock::now() - parked_at < kSpinBeforeSleep) continue;
+            if (std::chrono::steady_clock::now() - parked_at < spin_before_sleep_) continue;
            std::unique_lock<std::mutex> lk(sleep_mu_);
            sleepers_.fetch_add(1, std::memory_order_seq_cst);
            sleep_cv_.wait(lk, [&] {
@@ -179,21 +188,87 @@ void ExpertPool::worker(int id) {
            sleepers_.fetch_sub(1, std::memory_order_relaxed);
        }
        if (stop_.load(std::memory_order_acquire)) return;
-        seen = epoch_.load(std::memory_order_relaxed);
+        // acquire: the batch this epoch published (`head`, and the description before it) is visible from here
+        seen = epoch_.load(std::memory_order_acquire);
        parked_.fetch_sub(1, std::memory_order_acq_rel);   // leaving the park

        // Drain: one claim per iteration, so a slow worker takes fewer experts and a fast one takes more.
-        // Every job is the same size (all experts are 1,382,400 bytes), so there is nothing to schedule.
-        drain(id, scratch_[(size_t) id]);
+        // Every job is the same size (all experts are 1,382,400 bytes), so there is nothing to schedule.  Only
+        // this epoch's jobs: if the host has already moved on, the claims fail and the worker parks again.
+        drain(id, scratch_[(size_t) id], seen);
        parked_.fetch_add(1, std::memory_order_acq_rel);   // back at the park
    }
 }

-void ExpertPool::drain(int id, ExpertScratch& scratch) {
+int ExpertPool::claim(uint32_t epoch) {
+    uint64_t h = head_.load(std::memory_order_acquire);
+    for (;;) {
+        if ((uint32_t) (h >> 32) != epoch) return -1;               // not the batch this thread woke for
+        const uint32_t n = (uint32_t) (h >> 16) & 0xffffu, i = (uint32_t) h & 0xffffu;
+        if (i >= n) return -1;                                       // exhausted
+        if (head_.compare_exchange_weak(h, h + 1, std::memory_order_acq_rel, std::memory_order_acquire))
+            return (int) i;
+    }
+}
+
+uint32_t ExpertPool::begin_batch(int n) {
+    if (n < 0 || n > 0xffff) {
+        std::fprintf(stderr, "strata: expert pool batch of %d jobs is out of range\n", n);
+        std::abort();
+    }
+    // Every job of the previous batch has completed (`wait_done`), and a claim of it can no longer succeed, so
+    // nothing adds to `done` until this batch's first claim - which the release below orders after the reset.
+    done_.store(0, std::memory_order_relaxed);
+    const uint32_t e = epoch_.load(std::memory_order_relaxed) + 1;   // only the host bumps the epoch
+    head_.store(pack_head(e, (uint32_t) n, 0), std::memory_order_release);
+    publish();
+    return e;
+}
+
+void ExpertPool::wait_parked(const char* what) {
+    uint32_t spins = 0;
+    std::chrono::steady_clock::time_point t0{};
+    while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) {
+        _mm_pause();
+        if ((++spins & 1023u) != 0) continue;
+        const auto now = std::chrono::steady_clock::now();
+        if (spins == 1024u) t0 = now;
+        else if (now - t0 > kStall) {
+            std::fprintf(stderr, "strata: the CPU expert pool stalled %s (%u of %d workers parked) - stopping the engine "
+                                 "so the server can start it again (issue #29)\n",
+                         what, parked_.load(), n_);
+            std::fflush(stderr);
+            std::abort();
+        }
+    }
+}
+
+void ExpertPool::wait_done(int n) {
+    uint32_t spins = 0, seen = 0;
+    std::chrono::steady_clock::time_point t0{};
+    for (;;) {
+        const uint32_t d = done_.load(std::memory_order_acquire);
+        if (d >= (uint32_t) n) return;                                // `>=`: never a wait that an overshoot outlives
+        _mm_pause();
+        if ((++spins & 1023u) != 0) continue;
+        const auto now = std::chrono::steady_clock::now();
+        if (spins == 1024u || d != seen) { t0 = now; seen = d; }     // progress restarts the clock
+        else if (now - t0 > kStall) {
+            std::fprintf(stderr, "strata: the CPU expert pool stalled: %u of %d jobs done, %u of %d workers parked - "
+                                 "stopping the engine so the server can start it again (issue #29)\n",
+                         d, n, parked_.load(), n_);
+            std::fflush(stderr);
+            std::abort();
+        }
+    }
+}
+
+void ExpertPool::drain(int id, ExpertScratch& scratch, uint32_t epoch) {
    (void) id;
    for (;;) {
-        const uint32_t i = head_.fetch_add(1, std::memory_order_relaxed);
-        if (i >= (uint32_t) njobs_) break;
+        const int ci = claim(epoch);
+        if (ci < 0) break;
+        const uint32_t i = (uint32_t) ci;
        if (mode_ == 0) {
            const ExpertJob& j = jobs_[i];
            s2_expert_vnni_q(j.blob, *j.act, j.out, scratch);
@@ -267,15 +342,13 @@ void ExpertPool::drain(int id, ExpertScratch& scratch) {
 }

 void ExpertPool::run_phase(int mode, int n_tasks) {
-    while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause();
+    wait_parked("before a phase");
    mode_ = mode;
    njobs_ = n_tasks;
-    head_.store(0, std::memory_order_relaxed);
-    done_.store(0, std::memory_order_relaxed);
-    publish();
-    if (host_works_) drain(-1, host_scratch_);
-    while (done_.load(std::memory_order_acquire) != (uint32_t) n_tasks) _mm_pause();
-    while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause();
+    const uint32_t e = begin_batch(n_tasks);
+    if (host_works_) drain(-1, host_scratch_, e);
+    wait_done(n_tasks);
+    wait_parked("after a phase");
 }

 void ExpertPool::run_split(ExpertJob* jobs, int n) {
@@ -376,14 +449,12 @@ void ExpertPool::run(ExpertJob* jobs, int n) {
    // THE THREE PHASES ARE TIMED SEPARATELY.  They were one number, which cannot distinguish a pool that is
    // slow at the WORK from one that is slow at the SYNCHRONISATION - and those need opposite fixes.
    const auto t_a = std::chrono::steady_clock::now();
-    while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause();
+    wait_parked("before a batch");
    const auto t_b = std::chrono::steady_clock::now();
    jobs_ = jobs;
    njobs_ = n;
    mode_ = 0;
-    head_.store(0, std::memory_order_relaxed);
-    done_.store(0, std::memory_order_relaxed);
-    publish();   // a release (seq_cst): jobs_/njobs_ are visible before the bump
+    const uint32_t e = begin_batch(n);   // done, then head (release), then the epoch: the batch is described first

    // ---- **THE HOST DRAINS TOO (R2.2), INSTEAD OF SPINNING ON `done_`.**
    //
@@ -401,19 +472,19 @@ void ExpertPool::run(ExpertJob* jobs, int n) {
    // device after `run()` returns, so the write must be published, not merely performed.
    if (host_works_) {
        for (;;) {
-            const uint32_t i = head_.fetch_add(1, std::memory_order_relaxed);
-            if (i >= (uint32_t) n) break;
-            const ExpertJob& j = jobs_[i];
+            const int ci = claim(e);
+            if (ci < 0) break;
+            const ExpertJob& j = jobs_[ci];
            s2_expert_vnni_q(j.blob, *j.act, j.out, host_scratch_);
            done_.fetch_add(1, std::memory_order_release);
        }
    }

-    while (done_.load(std::memory_order_acquire) != (uint32_t) n) _mm_pause();
+    wait_done(n);
    // And park again, so the next `run` starts from a known state.  See the header for why `done` alone is
    // not enough.
    const auto t_c = std::chrono::steady_clock::now();
-    while (parked_.load(std::memory_order_acquire) != (uint32_t) n_) _mm_pause();
+    wait_parked("after a batch");
    const auto t_d = std::chrono::steady_clock::now();

    ms_wait_park_ += std::chrono::duration<double, std::milli>(t_b - t_a).count();
@@ -0,0 +1,99 @@
+// src/kernels/cpu/pool_stress.cpp - issue #29: the expert pool under the load a big-VRAM card gives it.
+//
+// With most experts resident on the GPU, many layers hand the pool nothing, the workers go to sleep in the middle
+// of a request, and the next tiny batch (one or two experts) is often finished by the host and the awake workers
+// before a sleeper has even woken.  That late wake-up is what the park/claim protocol has to survive.  This drives
+// hundreds of thousands of such batches - 0-4 jobs, run() and run_split(), with pauses long enough for the workers
+// to sleep - and checks that every batch returns, with every job's output written.  A watchdog thread turns a
+// hang into a failure instead of a frozen test.
+//
+//   pool_stress [seconds]        (default 20)
+#include "strata/kernels/cpu/pool.hpp"
+#include "strata/kernels/cpu/expert.hpp"
+
+#include <atomic>
+#include <chrono>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <random>
+#include <thread>
+#include <vector>
+
+namespace cpu = strata::kernels::cpu;
+
+int main(int argc, char** argv) {
+    const double seconds = argc > 1 ? std::atof(argv[1]) : 20.0;
+    const cpu::CpuFeatures feat = cpu::cpu_features();
+    if (!feat.usable()) {
+        std::printf("CPU lacks %s: pool_stress SKIPPED\n", feat.reason());
+        return 0;
+    }
+    constexpr int NJ = 4;
+    std::vector<std::vector<uint8_t>> blobs(NJ, std::vector<uint8_t>(cpu::BLOB, 0));
+    std::vector<float> x(cpu::H, 0.25f);
+    cpu::ActQ act;
+    cpu::act_quant_q8_1(x.data(), cpu::H, act);
+    std::vector<float> out((size_t) NJ * cpu::H);
+    std::vector<cpu::ExpertJob> jobs(NJ);
+    for (int e = 0; e < NJ; ++e) {
+        jobs[(size_t) e].blob = blobs[(size_t) e].data();
+        jobs[(size_t) e].act = &act;
+        jobs[(size_t) e].out = out.data() + (size_t) e * cpu::H;
+        jobs[(size_t) e].slot = e;
+    }
+
+    cpu::ExpertPool pool;
+    std::printf("pool_stress: %d workers + host, %.0f s\n", pool.workers(), seconds);
+
+    // the watchdog: a batch that does not return within 10 s is a hang
+    std::atomic<long long> beat{0};
+    std::atomic<bool> finished{false};
+    std::thread dog([&] {
+        long long last = -1;
+        auto since = std::chrono::steady_clock::now();
+        while (!finished.load()) {
+            std::this_thread::sleep_for(std::chrono::milliseconds(200));
+            const long long b = beat.load();
+            const auto now = std::chrono::steady_clock::now();
+            if (b != last) { last = b; since = now; continue; }
+            if (now - since > std::chrono::seconds(10)) {
+                std::printf("\npool_stress: HANG - batch %lld has not returned for 10 s\n", b);
+                std::fflush(stdout);
+                std::_Exit(3);
+            }
+        }
+    });
+
+    std::mt19937 rng(29);
+    const float SENTINEL = -1.2345e33f;
+    long long batches = 0, sleeps = 0, missed = 0;
+    const auto t0 = std::chrono::steady_clock::now();
+    while (std::chrono::steady_clock::now() - t0 < std::chrono::duration<double>(seconds)) {
+        // now and then a gap long enough for every worker to go to sleep (the pool sleeps after 20 ms)
+        if (rng() % 64 == 0) {
+            std::this_thread::sleep_for(std::chrono::milliseconds(21 + (int) (rng() % 10)));
+            ++sleeps;
+        } else if (rng() % 4 == 0) {
+            std::this_thread::sleep_for(std::chrono::microseconds(rng() % 300));
+        }
+        const int n = (int) (rng() % (NJ + 1));
+        for (int e = 0; e < n; ++e) std::fill(jobs[(size_t) e].out, jobs[(size_t) e].out + cpu::H, SENTINEL);
+        if (rng() % 2) pool.run(jobs.data(), n);
+        else pool.run_split(jobs.data(), n);
+        for (int e = 0; e < n; ++e)
+            if (jobs[(size_t) e].out[0] == SENTINEL || jobs[(size_t) e].out[cpu::H - 1] == SENTINEL) ++missed;
+        beat.fetch_add(1);
+        ++batches;
+    }
+    finished.store(true);
+    dog.join();
+    std::printf("pool_stress: %lld batches (%lld after the workers slept), %lld jobs not run\n", batches, sleeps,
+                missed);
+    if (missed) {
+        std::printf("pool_stress: FAILED\n");
+        return 1;
+    }
+    std::printf("pool_stress OK\n");
+    return 0;
+}
@@ -45,6 +45,7 @@
 #include "strata/spec/draft_policy.hpp"
 #include "strata/spec/suffix_drafter.hpp"
 #include "strata/kernels/cvec.hpp"
+#include "strata/core/progress.hpp"
 #ifndef NOMINMAX
 #define NOMINMAX   // gguf_reader.hpp includes windows.h
 #endif
@@ -2200,6 +2201,8 @@ int main(int argc, char** argv) {
            const double ms = std::chrono::duration<double, std::milli>(Clock::now() - pp_t0).count();
            std::printf("PP %lld %lld %.0f %.1f\n", (long long) done, (long long) pp_total, ms,
                        ms > 0.0 ? 1000.0 * (double) (done - pp_from) / ms : 0.0);
+            strata::core::progress_at("reading the prompt (batched), done up to token", done);
+            strata::core::progress_beat();
            std::fflush(stdout);
            if (o.prompt_cache_every > 0 && done >= pp_next_check) {
                if (!checkpoint_at(done)) { e = "saving a conversation checkpoint failed"; return false; }
@@ -2327,12 +2330,38 @@ int main(int argc, char** argv) {
            size_t free_b = 0, total_b = 0;
            cudaMemGetInfo(&free_b, &total_b);
            std::printf("INFO context=%lld kv=%s kv_resident=%lld expert_slots=%lld expert_cache_mib=%lld spec=%d "
-                        "mtp_max=%d lookup=%d vram_free_mib=%lld cvec=%s\n",
+                        "mtp_max=%d lookup=%d vram_free_mib=%lld cvec=%s arena_mib=%lld\n",
                        (long long) o.max_context, o.kv.c_str(),
                        (long long) (g.n_qsa_layers() > 0 && ss.qsa_states[0].kv_mode == 1
                                         ? ss.qsa_states[0].n_slots * 4 : 0),
                        (long long) xcache.slots(), (long long) (xcache.bytes() >> 20), o.spec, o.mtp_max_t,
-                        o.suffix_draft, (long long) (free_b >> 20), cvec_summary.c_str());
+                        o.suffix_draft, (long long) (free_b >> 20), cvec_summary.c_str(),
+                        (long long) (strata::kernels::cpu::expert_layout().total >> 20));
+        }
+        // issue #29: a request whose heartbeat (tokens, prompt chunks, verify windows) stops for this long is stuck on
+        // a flag nobody will raise - end the engine with where it was, so the server starts it again instead of the
+        // GPU spinning forever.  STRATA_WATCHDOG_S=0 turns it off.
+        {
+            const char* ws = std::getenv("STRATA_WATCHDOG_S");
+            const int limit = ws ? std::atoi(ws) : 120;
+            if (limit > 0)
+                std::thread([limit] {
+                    strata::core::Progress& p = strata::core::progress();
+                    uint64_t last = p.beats.load();
+                    auto since = std::chrono::steady_clock::now();
+                    for (;;) {
+                        std::this_thread::sleep_for(std::chrono::seconds(1));
+                        const auto now = std::chrono::steady_clock::now();
+                        const uint64_t b = p.beats.load();
+                        if (!p.busy.load() || b != last) { last = b; since = now; continue; }
+                        if (now - since < std::chrono::seconds(limit)) continue;
+                        std::fprintf(stderr, "strata serve: no progress for %d s during a request (%s %lld) - stopping "
+                                             "the engine so the server starts it again (issue #29)\n",
+                                     limit, p.where.load(), (long long) p.detail.load());
+                        std::fflush(stderr);
+                        std::abort();
+                    }
+                }).detach();
        }
        std::printf("READY %lld stop\n", (long long) o.max_context);   // "stop": this engine honours STOP
        std::fflush(stdout);
@@ -2353,6 +2382,11 @@ int main(int argc, char** argv) {
        std::vector<const float*> row_ptr;
        while (next_line(line)) {
            if (line == "QUIT") break;
+            // the watchdog watches a request from here until this iteration ends, whichever way it ends
+            struct BusyScope {
+                BusyScope() { strata::core::progress().busy.store(true); strata::core::progress_at("request"); }
+                ~BusyScope() { strata::core::progress().busy.store(false); strata::core::progress_at("idle"); }
+            } busy_scope;
            stop_req.store(false);   // a STOP that arrived between requests is stale
            const bool geni = line.rfind("GENI ", 0) == 0;
            if (!geni && line.rfind("GEN ", 0) != 0) {
@@ -2572,6 +2606,7 @@ int main(int argc, char** argv) {
            pp_t0 = r0;
            pp_next_check = reread_to > 0 ? INT64_MAX : resume + o.prompt_cache_every;
            std::printf("RESUME %lld\n", (long long) resume);   // before reading: this many prompt tokens are reused
+            strata::core::progress_at("reading the prompt, from token", read_from);
            std::fflush(stdout);
            // A SHORT PART OF THE PROMPT - the new message of a chat that continues from a checkpoint, the assistant
            // header - goes through the verify windows, S tokens at a time, as decode reads them.  The batched path
@@ -2611,6 +2646,7 @@ int main(int argc, char** argv) {
                const double ms = std::chrono::duration<double, std::milli>(Clock::now() - pp_t0).count();
                std::printf("PP %lld %lld %.0f %.1f\n", (long long) b, (long long) pp_total, ms,
                            ms > 0.0 ? 1000.0 * (double) (b - pp_from) / ms : 0.0);
+                strata::core::progress_beat();
                std::fflush(stdout);
                return true;
            };
@@ -2794,6 +2830,7 @@ int main(int argc, char** argv) {
                bool eos = false;
                for (int i = 0; i <= a && produced_n < max_new && !eos; ++i) {
                    std::printf("T %d\n", (int) outv[(size_t) i]);
+                    strata::core::progress_beat();
                    ++produced_n;
                    if (o.suffix_draft > 0) sfx.append(outv[(size_t) i]);
                    eos = std::find(o.eos_ids.begin(), o.eos_ids.end(), (int64_t) outv[(size_t) i]) != o.eos_ids.end();
Author	SHA1	Message	Date
Niko1221	6255c8fd14	web app: attach text files - drop them on the chat or use the attach button (issue #30 ) Text files (code, notes, logs, data; up to 512 KB) become chips next to the message and are sent as "File: name" plus a fenced block (a longer fence when the file contains one). Binary files are refused with a toast; pictures work as before (when images are on). The attach button is always there now. Kept small on purpose: the web app is for quick tests and settings.	2026-09-27 20:53:38 +02:00
Niko1221	9ffd6c9461	engine: fix a race in the CPU expert pool, and never hang silently again (issue #29 , engine 0.1.12) On a card with most experts in VRAM (RTX 4090) generation could stop for good after thousands of tokens: the GPU "100%" at low power, one CPU core busy, no tokens, no error. - The pool's workers sleep after 20 ms without work (issue #4). With most experts resident many layers have no CPU work, so they also slept mid-request, and a worker that woke late was still counted as parked: it could claim from the next batch while the host was writing it, run a job twice, or add to `done` after the reset - and `done != n` then never ended while the GPU waited on the pool. Every claim now carries its batch: `head` is one 64-bit word (epoch \| njobs \| index), a claim is a CAS that only succeeds for the epoch the worker woke for, and the host waits with `<` instead of `!=`. Same speed (pool_test 47.2 GB/s). - Every pool wait is bounded (60 s: a message and the engine stops), and --serve runs a watchdog: a request with no progress (tokens, prompt chunks, verify windows) for STRATA_WATCHDOG_S (120) seconds ends the engine with where it was stuck (strata/core/progress.hpp). The server (issue #27) answers with an error, names the cause from the log, and starts the engine again on the next request. - pool_stress: tiny batches with sleeping workers; STRATA_POOL_SPIN_US shortens the spin to make them sleep constantly. Tested: pool_test, pool_stress; 25,000-token generation at 262K with KV streaming and constantly sleeping workers (73.9 tok/s, unchanged); the watchdog deliberately tripped (2 s): error, log line, restart, next answer.	2026-09-27 20:53:38 +02:00
Niko1221	1e3bded2f7	web app: "Use for other apps too" - the Chat settings as every client's defaults A switch in the Sampling drawer. With it on, Apply stores the drawer's settings on the server (POST /settings): thinking level, temperature, top-p, top-k, seed, max tokens and the experimental speed projection. omp and other API clients then get them for anything they don't send themselves, from their next request (no restart); what a client sends still wins. Off again, clients use their own settings. - Kept between starts in <config>.shared-settings.json next to the run config; the server window says what other apps use. - Only Strata's own page can change them: JSON only (a cross-site form or simple request can't send it without a CORS preflight, which this server never grants), no foreign Origin, and the API key when one is set. - Values are checked (bad ones are a 400 naming the setting). - Tests: defaults reach the engine, a client's own values win, off clears them and the file, 415/403/400/401.	2026-09-27 17:50:38 +02:00
Niko1221	ae3dc218ea	web app: a long switch label no longer squeezes the switch The Sampling drawer's "Experimental speed projection" row has a longer label than "Show thinking", and the flex row shrank its switch, clipping the knob at the drawer's edge. Switches in toggle rows keep their size.	2026-09-27 17:45:18 +02:00
Niko	3cba59ea31	Update README.md	2026-09-27 17:48:41 +02:00
Niko	ecba7dfc10	Update README.md	2026-09-27 17:46:12 +02:00
Niko1221	1314ebfc7f	README: a video up top, speed and model choice first, simpler troubleshooting (engine 0.1.10) - An animated preview of the pagoda garden demo under the title (the full video is a release download: GitHub does not play videos from a repository), and a screenshot of the app in "Using it". - "Is my PC enough?" folded into Install; "How fast is it?" and "Which model should I pick?" come first. The model table shows shard 1 (what gets loaded) instead of the download, with a plain fit rule: RAM >= shard 1 + ~10 GB, since the experts live in RAM. - "Something went wrong?" as short questions and answers, including the first-start freeze and "the engine stopped unexpectedly". - DETAILS troubleshooting: the same two rows, and the VRAM stall row now points to engine 0.1.9. - setup requires engine 0.1.10 (INFO arena_mib, the start-up RAM warning).	2026-09-27 17:30:09 +02:00
Niko1221	7664149081	serve: an engine that dies mid-answer is an error, and the next request restarts it (issue #27 ) On Linux with too little RAM for the model's experts (IQ3_XXS on 47 GB), the out-of-memory killer ended the engine mid-answer. The server then recorded the reply as "length" ("Max tokens" in the Monitor, no tok/s), the stream was cut with a traceback, and every later request failed on the dead pipe. - The engine's death is EngineDied: the reply ends with a clear error event (OpenAI and Anthropic streams; 503 before the answer), the Monitor says Error, and the server window names the likely cause (RAM; dmesg). - The next request starts the engine again (same command) instead of failing forever; the engine counts as gone as soon as its output closes, and a failed write to it is not mistaken for the client leaving. - The engine's INFO line reports its expert arena (arena_mib); at start the server warns when less than ~6 GB of RAM is left beside it. - Test: a fake engine dies mid-stream -> error event + [DONE], Monitor "error", the next request restarts it and answers. End to end with Q2_0: strata.exe killed mid-answer -> error (exit code 1); next request restarts it (31 s) and answers.	2026-09-27 17:23:55 +02:00