After a fatal GPU error the engine ends itself and the server starts it again, instead of staying up and failing every later request (#4350)

DeepSeek: an illegal memory access, launch failure or hardware exception in a prompt read or a decode round used to turn into an ERR/cancel while the engine kept running on a dead GPU context, so every later request failed with HTTP 400 until someone restarted it. strata-dsv41 now prints 'ERR fatal GPU error: ...' and exits (code 3). Qwen: a failed decode window, commit or final commit wait that leaves a sticky fatal runtime error does the same (the prompt-read path already did, #224), by the runtime's error text so CUDA, HIP and SYCL builds agree.
Server: that ERR line fails the request in flight as an engine death (503, retryable) and marks the engine ended, so the next request starts it again at once, with one plain log line and a plain death note. Restart-loop guard: a third fatal death within ten minutes stops the restarts and answers 503 with the reason until the window has passed; one death counts once.
Test hooks (off unless set): DS4_INJECT_FATAL=N / STRATA_INJECT_FATAL=N fail the Nth request / decode window with a real fatal error's text; *_ONCE=<file> fires it once across restarts.
Tested: serve/test_fatal_gpu_restart.py (7 tests); through the real server, request 2 injected, DeepSeek on boston (IQ1_M pack, 2x A6000): request 2 got a 503, the restart took 44 s, requests 1, 3, 4, 5, 6 answered 42; Qwen on a P100 (IQ2_XS), fault injected mid-decode in request 4: 503, restart, requests 1-3, 5, 6 answered 42. The guard counted '1 of 3' there and '2 of 3' in an earlier run where the injection fired on every restart. NOT tested: a real GPU fault (none reproduces), the Windows and Metal builds, the third-death refusal through a running server (unit-tested only), conversations coming back from saved slots after the restart (no check made).

serve/server.py

@@ -1279,6 +1279,10 @@ class StrataEngine:
             if "issue #29" in line:
                 return ("The engine stopped itself because it had stopped making progress - a hang it caught. Its log "
                         "line: " + line.strip() + " - please report it at github.com/Niko1221/Strata/issues.")
+        if (self.last_err or "").startswith("fatal GPU error"):    # #4350: the engine ended itself; the next request starts it again
+            return ("The graphics card reported a fatal error (" + self.last_err[len("fatal GPU error:"):].strip()[:160] + "), so the engine "
+                    "ended. The next request starts it again. If this repeats, please report it at github.com/Niko1221/Strata/issues "
+                    "with the log.")
         proc = self.proc                                 # None while a restart has not started the next one yet
         rc = proc.poll() if proc is not None else None
         if rc is not None and rc >= 0 and self.last_err:  # its own last words on stdout: they say why
@@ -1599,6 +1603,7 @@ class StrataEngine:
                     self._yielded = (int(f[1]), int(f[2]))
             elif line.startswith("ERR"):
                 self._ctl_erred = True                   # #1059: the engine refused it and is idle: no DONE follows
+                self._raise_if_fatal_gpu(line)
                 raise ValueError(line[4:].strip())
             if line.startswith("DONE") and self._ctl_mode == "solo":
                 self._ctl_result = ("done", None)
@@ -1637,6 +1642,17 @@ class StrataEngine:
             self.last = {**self.last, "finish": f[3], "decode_ms": float(f[4]),
                          "generated": int(f[2]) if f[2].isdigit() else self.last.get("generated")}
 
+    def _raise_if_fatal_gpu(self, line: str) -> None:
+        """#4350: the engine's ERR line "fatal GPU error: ..." means it is ending (its GPU context is gone).  The request in flight fails
+        as an engine death (a 503 the client can send again), not as a bad request (400), and the engine counts as ended, so the next
+        request starts it again at once instead of finding a closed pipe."""
+        text = line[4:].strip()
+        if not text.startswith("fatal GPU error"):
+            return
+        self.last_err = text
+        self.ended = True
+        raise EngineDied("the GPU reported a fatal error and the engine ended: " + text[len("fatal GPU error:"):].strip()[:200])
+
     def _bdone_after_eos(self, slot: int, wait_s: float = 2.0):
         """#1869: a request in a slot whose consumer left at the stop token has its BDONE right behind it (the engine prints it
         in the same window).  Read it here, so the request's figures are the slot leg's (the background wait would eat it,
@@ -2441,6 +2457,7 @@ class StrataEngine:
                     return
                 elif line.startswith("ERR"):
                     done = True
+                    self._raise_if_fatal_gpu(line)
                     raise ValueError(line[4:].strip())
         finally:
             if not done:                                  # the consumer stopped early: stop the engine, drain to DONE
@@ -4618,6 +4635,34 @@ class Service:
         except Exception:
             return None
 
+    FATAL_GPU_MAX = 3            # fatal GPU errors of one engine within FATAL_GPU_WINDOW_S: after that it is not started again
+    FATAL_GPU_WINDOW_S = 600.0
+
+    def _note_fatal_gpu_exit(self, code) -> None:
+        """#4350: the engine ends itself after a fatal GPU error (the device context cannot be used again) and says so in its last
+        ERR line.  The next request starts it again, like any engine death, with one plain line here; the request that was in
+        flight already failed with a 503.  Three such deaths within ten minutes mean a restart will not help (a bad card, driver or
+        model state): then the server stops starting it and says so (503), until the ten minutes have passed."""
+        why = getattr(self.engine, "last_err", None) or ""
+        if "fatal GPU error" not in why:
+            return
+        pid = getattr(getattr(self.engine, "proc", None), "pid", None)
+        seen = self.__dict__.setdefault("_fatal_gpu_pids", set())
+        times = self.__dict__.setdefault("_fatal_gpu_times", [])
+        now = time.time()
+        if pid not in seen:                              # one death counts once, however many requests find it dead
+            seen.add(pid)
+            times.append(now)
+        times[:] = [t for t in times if now - t < self.FATAL_GPU_WINDOW_S]
+        print(f"[strata] the GPU reported a fatal error ({why[len('fatal GPU error:'):].strip()[:160]}); the engine ended "
+              f"(exit code {code}) and is started again; the request that was running failed and can be sent again "
+              f"({len(times)} of {self.FATAL_GPU_MAX} allowed in {self.FATAL_GPU_WINDOW_S / 60:g} minutes)", flush=True)
+        if len(times) >= self.FATAL_GPU_MAX:
+            raise EngineDied(f"the GPU reported a fatal error {len(times)} times in {self.FATAL_GPU_WINDOW_S / 60:g} minutes, so "
+                             "the engine is not started again (the last: " + why[:160] + "). Check the graphics driver, the "
+                             "card's temperature and power, and the log; a request after "
+                             f"{self.FATAL_GPU_WINDOW_S / 60:g} minutes tries again")
+
     def ensure_loaded(self):
         """Start the engine if it is not running (unloaded, or it died - issue #27), after the before_load hook and
         the free-VRAM check.  The caller holds self.fifo."""
@@ -4686,6 +4731,7 @@ class Service:
             print("[strata] loading the model again (it was unloaded) ...", flush=True)
         else:
             code = self.engine.exit_code() if hasattr(self.engine, "exit_code") else None
+            self._note_fatal_gpu_exit(code)
             print(f"[strata] the engine had stopped (exit code {code}); starting it again "
                   "(a minute or two) ...", flush=True)
         try:

serve/test_fatal_gpu_restart.py

@@ -0,0 +1,82 @@
+"""serve/test_fatal_gpu_restart.py: #4350.  After a fatal GPU error the DeepSeek engine ends itself (ERR fatal GPU error: ...) and the
+server starts it again on the next request; three such deaths within ten minutes stop the restarts with a plain 503 text.
+CPU only, no engine: the server's bookkeeping is called on a stand-in."""
+import sys
+import time
+import types
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+from serve import server as S  # noqa: E402
+
+
+def stand_in(pid, last_err):
+    return types.SimpleNamespace(engine=types.SimpleNamespace(last_err=last_err, proc=types.SimpleNamespace(pid=pid)))
+
+
+class FatalGpuRestart(unittest.TestCase):
+    def note(self, svc, code=3):
+        S.Service._note_fatal_gpu_exit.__get__(svc)(code)
+
+    def make(self, pid, err="fatal GPU error: cudaStreamSynchronize(s): an illegal memory access was encountered"):
+        svc = stand_in(pid, err)
+        svc.FATAL_GPU_MAX = S.Service.FATAL_GPU_MAX
+        svc.FATAL_GPU_WINDOW_S = S.Service.FATAL_GPU_WINDOW_S
+        return svc
+
+    def test_one_fatal_death_is_restarted_with_one_plain_line(self):
+        svc = self.make(100)
+        self.note(svc)                      # no exception: the caller goes on to restart the engine
+
+    def test_an_ordinary_death_is_not_counted(self):
+        svc = self.make(100, err="the engine ran out of memory")
+        for pid in range(10):
+            svc.engine.proc.pid = pid
+            self.note(svc)
+        self.assertFalse(svc.__dict__.get("_fatal_gpu_times"))
+
+    def test_the_same_death_seen_by_many_requests_counts_once(self):
+        svc = self.make(100)
+        for _ in range(8):
+            self.note(svc)
+        self.assertEqual(len(svc._fatal_gpu_times), 1)
+
+    def test_three_fatal_deaths_in_ten_minutes_stop_the_restarts_and_say_so(self):
+        svc = self.make(1)
+        self.note(svc)
+        svc.engine.proc.pid = 2
+        self.note(svc)
+        svc.engine.proc.pid = 3
+        with self.assertRaises(S.EngineDied) as cm:
+            self.note(svc)
+        self.assertIn("3 times in 10 minutes", str(cm.exception))
+        self.assertIn("not started again", str(cm.exception))
+        with self.assertRaises(S.EngineDied):          # the same dead engine asked again: still refused
+            self.note(svc)
+
+    def test_after_the_window_it_tries_again(self):
+        svc = self.make(1)
+        self.note(svc)
+        svc.engine.proc.pid = 2
+        self.note(svc)
+        svc._fatal_gpu_times[:] = [t - S.Service.FATAL_GPU_WINDOW_S - 1 for t in svc._fatal_gpu_times]
+        svc.engine.proc.pid = 3
+        self.note(svc)                      # the two old ones aged out: only this one counts
+
+    def test_the_request_in_flight_fails_as_an_engine_death_not_a_bad_request(self):
+        eng = types.SimpleNamespace(last_err=None, ended=False)
+        with self.assertRaises(S.EngineDied) as cm:
+            S.StrataEngine._raise_if_fatal_gpu(eng, "ERR fatal GPU error: cudaStreamSynchronize(s): an illegal memory access was encountered")
+        self.assertIn("illegal memory access", str(cm.exception))
+        self.assertTrue(eng.ended)                                  # the next request starts the engine at once
+        self.assertTrue(eng.last_err.startswith("fatal GPU error"))
+
+    def test_an_ordinary_engine_error_stays_a_bad_request(self):
+        eng = types.SimpleNamespace(last_err=None, ended=False)
+        S.StrataEngine._raise_if_fatal_gpu(eng, "ERR the prompt (9 tokens) and max_new 5 are more than the context 8")   # no exception
+        self.assertFalse(eng.ended)
+
+
+if __name__ == "__main__":
+    unittest.main()

src/dsv41/ds4_main.cpp

@@ -301,6 +301,41 @@ struct Pic {
 
 // GENI's file: int32 'DSI1', int32 count, then per picture int64 start, int32 width, int32 height and its
 // height x width x 3 RGB bytes, in prompt order
+namespace {
+// After a fatal GPU error (an illegal memory access, a launch failure, a hardware exception) the device context is gone: every later
+// CUDA / HIP call of this process fails, so an engine that keeps serving only turns each request into an error until someone restarts
+// it (#4350, an unattended agent loop is dead).  The engine says so on stdout and exits; the server starts it again on the next
+// request, like any engine death, and fails only the request in flight with a 503 (retryable).
+bool gpu_fatal_text(const std::string& e) {
+    static const char* const k[] = {"illegal memory access", "unspecified launch failure", "hardware exception", "illegal instruction",
+                                    "misaligned address", "device-side assert", "uncorrectable ECC", "launch timed out",
+                                    "launch failure", "device lost", "ERROR_DEVICE_LOST"};
+    for (const char* w : k)
+        if (e.find(w) != std::string::npos) return true;
+    return false;
+}
+[[noreturn]] void die_gpu_fatal(const std::string& e) {
+    std::fprintf(stderr, "ds4: fatal GPU error (%s): the GPU context cannot be used again, so the engine ends and the server starts it again\n",
+                 e.c_str());
+    std::fflush(stderr);
+    std::printf("ERR fatal GPU error: %s\n", e.c_str());
+    std::fflush(stdout);
+    std::_Exit(3);   // (not a clean unwind: freeing device memory on a dead context can hang, as the Qwen engine's #224 exit knows)
+}
+// DS4_INJECT_FATAL=N (a test hook, off when unset): the Nth GEN request fails with the text of a real fatal error, taking the same path
+bool inject_fatal_now() {
+    static const long n = [] { const char* v = std::getenv("DS4_INJECT_FATAL"); return v != nullptr ? std::atol(v) : 0L; }();
+    static long seen = 0;
+    if (!(n > 0 && ++seen == n)) return false;
+    // DS4_INJECT_FATAL_ONCE=<file>: fires once across restarts (the file is made when it fires; while it exists nothing is injected)
+    if (const char* f = std::getenv("DS4_INJECT_FATAL_ONCE"); f != nullptr && *f) {
+        if (std::FILE* t = std::fopen(f, "rb")) { std::fclose(t); return false; }
+        if (std::FILE* t = std::fopen(f, "wb")) std::fclose(t);
+    }
+    return true;
+}
+}  // namespace
+
 bool read_pics(const std::string& path, std::vector<Pic>& v, std::string& err) {
     std::FILE* f = std::fopen(path.c_str(), "rb");
     if (f == nullptr) {
@@ -903,6 +938,8 @@ int serve(strata::ds4::Model& m, const std::string& eos_list, bool spec, float c
         }
         m.set_cancel(nullptr);
         m.set_images({});
+        if (inject_fatal_now()) die_gpu_fatal("injected (DS4_INJECT_FATAL): an illegal memory access was encountered");
+        if (!ok && gpu_fatal_text(err)) die_gpu_fatal(err);
         if (stopped || (!ok && m.cancelled())) {
             const bool back = from > 0 && m.restore((int64_t) from) && (size_t) m.pos() == from;
             if (!back) {
@@ -992,6 +1029,7 @@ int serve(strata::ds4::Model& m, const std::string& eos_list, bool spec, float c
                 if (!round(m, tok, room, try_spec, r.conf >= 0 ? r.conf : conf_min, r.temperature, r.top_p, r.top_k,
                            r.min_p, r.seed, counter, got,
                            off, acc, err, r.cum)) {
+                    if (gpu_fatal_text(err)) die_gpu_fatal(err);
                     std::fprintf(stderr, "ds4: %s\n", err.c_str());
                     finish = "cancel";
                     break;

src/program/generate.cpp

@@ -2894,6 +2894,45 @@ static bool prepare_gated_window(strata::core::gated::Packet& packet,
     return true;
 }
 
+// #4350: after a fatal GPU error (an illegal memory access, a launch failure) the device context cannot be used again, and unwinding the
+// destructors on it can hang until the stall watchdog (#224).  A failed decode window / commit therefore ends the engine at once when the
+// runtime holds a sticky error, with "ERR fatal GPU error: ..." as its last stdout line: the server fails the request in flight with a 503
+// and starts the engine again on the next request, like any engine death.
+// STRATA_INJECT_FATAL=N (a test hook, off when unset): the Nth decode window fails with the text of a real fatal error, the same way.
+[[noreturn]] static void exit_gpu_fatal(const std::string& what) {
+    std::fprintf(stderr, "strata serve: fatal GPU error (%s): the engine ends so that the server can start it again\n", what.c_str());
+    std::fflush(stderr);
+    std::printf("ERR fatal GPU error: %s\n", what.c_str());
+    std::fflush(stdout);
+    std::_Exit(3);
+}
+static void exit_if_gpu_fatal(const std::string& err) {
+    static const long inject = [] { const char* v = std::getenv("STRATA_INJECT_FATAL"); return v != nullptr ? std::atol(v) : 0L; }();
+    static long windows = 0;
+    if (inject > 0 && ++windows == inject) {
+        // STRATA_INJECT_FATAL_ONCE=<file>: fires once across restarts (the file is made when it fires; while it exists nothing is injected)
+        const char* once = std::getenv("STRATA_INJECT_FATAL_ONCE");
+        bool fire = true;
+        if (once != nullptr && *once) {
+            if (std::FILE* t = std::fopen(once, "rb")) { std::fclose(t); fire = false; }
+            else if (std::FILE* t2 = std::fopen(once, "wb")) std::fclose(t2);
+        }
+        if (fire) exit_gpu_fatal("injected (STRATA_INJECT_FATAL): an illegal memory access was encountered");
+    }
+    const cudaError_t sticky = cudaPeekAtLastError();
+    if (sticky != cudaSuccess) {
+        // (by the runtime's own text, so that CUDA, HIP and SYCL builds read it the same way)
+        const std::string text = cudaGetErrorString(sticky);
+        static const char* const gone[] = {"illegal memory access", "illegal address", "unspecified launch failure", "launch failure",
+                                           "hardware exception", "illegal instruction", "misaligned address", "uncorrectable", "launch timed out",
+                                           "device-side assert"};
+        bool context_gone = false;
+        for (const char* w : gone)
+            if (text.find(w) != std::string::npos) context_gone = true;
+        if (context_gone) exit_gpu_fatal(err.empty() ? std::string(cudaGetErrorString(sticky)) : err);
+    }
+}
+
 #if defined(STRATA_USE_HIP) && defined(_WIN32)
 static int engine_main(int argc, char** argv) {   // main() is at the end of this file (#3431)
 #else
@@ -16990,7 +17029,9 @@ int main(int argc, char** argv) {
                 if (gated_proposal) ver.set_spec_proposal(proposal.n_q ? proposal.mu.data() : nullptr, proposal.n_q);
                 else if (use_mtp && mtp.prob() && !from_sfx && !first_window && T_mtp > 1) ver.set_spec_q(mtp.spec_q(), T_mtp - 1);
                 const auto decode_pool_fn = use_mtp && mtp.prob() ? invariant_win_pool_fn : win_pool_fn;
+                exit_if_gpu_fatal("");   // (the test hook counts windows here)
                 if (!ver.run(T, window.data(), p, decode_pool_fn, win_pool_user, outv.data(), err) || drive.d.failed) {
+                    exit_if_gpu_fatal(drive.d.failed && drive.d.fail ? drive.d.fail : err);
                     std::printf("ERR %s\n", drive.d.failed && drive.d.fail ? drive.d.fail : err.c_str());
                     return 1;
                 }
@@ -17026,6 +17067,7 @@ int main(int argc, char** argv) {
                 if (!drive.d.usage.empty() && !ajob && ((rounds + 1) % o.adapt_iv(rounds)) == 0)
                     adapt_thr = std::thread([&] { strata::platform::name_this_thread("strata-adapt"); strata::aux_cpus::pin_current_thread(); adapt_ok = adapt(); });
                 if (!ver.commit(a + 1, err)) {
+                    exit_if_gpu_fatal(err);
                     if (adapt_thr.joinable()) adapt_thr.join();
                     std::printf("ERR %s\n", err.c_str());
                     return 1;
@@ -17141,6 +17183,7 @@ int main(int argc, char** argv) {
             const double decode_ms = std::chrono::duration<double, std::milli>(Clock::now() - d0).count();
             // the last commit (set_commit_async): the session is complete before anything reads or copies it
             if (!ver.wait_commit(err)) {
+                exit_if_gpu_fatal(err);
                 std::printf("ERR %s\n", err.c_str());
                 return 1;
             }