#465 setup: recommend "parallel" only where the experts mostly fit in VRAM

On a card whose experts mostly run on the CPU the slots cost speed per request (RTX 5070, Q2_0: 11-24 %), so setup
now recommends slots only when the expert cache (every card of a split counted) still holds half of the model's
experts beside them (Q2_0: 24 GB and up, or a split); elsewhere it stays at one at a time and says "parallel N
reduces waiting for several users but costs about 10-25% speed per request on this card". --parallel N is
honoured as asked, as before. BATCHING.md: the table with the change against one at a time.

docs/BATCHING.md

@@ -12,7 +12,8 @@ It is opt-in and changes nothing when the options are absent (#465; the engine p
 ## Turning it on
 
 One GPU: add `"parallel": 2` to the model's config (`strata-<model>.json`) and restart, or run setup with
-`--parallel 2`. Setup says what it recommends for the card (below); a larger number is kept as asked, with a note.
+`--parallel 2`. Setup recommends it only where it does not cost speed (below); any number you ask for is kept as
+asked, with a note when it is more than setup would recommend.
 
 ```
 "parallel": 2
@@ -42,9 +43,14 @@ KV, more with a longer context unless the KV cache streams (`--kv-resident`: the
 stays in VRAM, and each slot's whole KV cache takes pinned RAM - 1.6 GB at 128K). On a card whose experts mostly run
 on the CPU, a batch also reads about as many distinct experts as the requests one by one (different conversations
 route to different experts), so the gain is in **latency** (nobody waits for a whole answer), and a request alone
-runs a little slower (the smaller expert cache) - see the measurements below. Setup recommends as many slots as take
-at most a fifth of the expert cache (~VRAM less 5 GB), up to 4: 2 on a 12 GB card, 3 on 16 GB, 4 from 24 GB; none
-below 12 GB.
+runs slower (the smaller expert cache): 11-24 % on a 12 GB card, see the measurements below.
+
+So setup recommends `"parallel"` **only where the experts mostly fit in VRAM**: the expert cache (each card's VRAM
+less ~5 GB, every card of a layer split counted) must still hold at least half of the model's experts beside the
+slots, and the slots may take at most a fifth of it, up to 4 slots. With Q2_0 at 32K that is 3 slots on a 24 GB
+card, 4 from 32 GB or on a split such as 2 x 16 GB; IQ3_S needs 32 GB or a split. Everywhere else (any 12 or 16 GB
+card alone) it stays at one at a time and setup says: "parallel N reduces waiting for several users but costs
+about 10-25% speed per request on this card". `--parallel N` is honoured as asked either way.
 
 ## How the server uses the slots
 
@@ -121,24 +127,25 @@ counter-based draw (Philox(seed, position)).
 One RTX 5070 (12 GB), Ryzen 5 7600, 64 GB DDR5, Q2_0, 32K context, through the HTTP server: C different requests
 sent at once (an 800-word essay each, 256 tokens per answer, greedy, thinking off), median of 3 rounds:
 
-| Concurrent | Setting | Total tok/s | Per request tok/s | First token: median / last of the round |
-| ---: | --- | ---: | ---: | ---: |
-| 1 | one at a time (default) | 74.3 | 83.6 | 0.4 s / 0.4 s |
-| 1 | `"parallel": 2` | 67.3 | 74.8 | 0.4 s / 0.4 s |
-| 1 | `"parallel": 4` | 57.9 | 63.8 | 0.4 s / 0.4 s |
-| 2 | one at a time | 71.6 | 77.3 | 2.1 s / 4.1 s |
-| 2 | `"parallel": 2` | 61.0 | 32.6 | 0.5 s / 0.9 s |
-| 2 | `"parallel": 4` | 54.6 | 28.7 | 0.6 s / 0.7 s |
-| 4 | one at a time | 70.7 | 79.6 | 6.0 s / 11.2 s |
-| 4 | `"parallel": 2` | 61.2 | 32.2 | 4.5 s / 9.3 s |
-| 4 | `"parallel": 4` | 63.1 | 16.9 | 1.0 s / 1.8 s |
+| Concurrent | Setting | Total tok/s | vs one at a time | Per request tok/s | First token: median / last of the round |
+| ---: | --- | ---: | ---: | ---: | ---: |
+| 1 | one at a time (default; what setup recommends on this card) | 74.3 | - | 83.6 | 0.4 s / 0.4 s |
+| 1 | `"parallel": 2` | 67.3 | -9 % | 74.8 | 0.4 s / 0.4 s |
+| 1 | `"parallel": 4` | 57.9 | -22 % | 63.8 | 0.4 s / 0.4 s |
+| 2 | one at a time | 71.6 | - | 77.3 | 2.1 s / 4.1 s |
+| 2 | `"parallel": 2` | 61.0 | -15 % | 32.6 | 0.5 s / 0.9 s |
+| 2 | `"parallel": 4` | 54.6 | -24 % | 28.7 | 0.6 s / 0.7 s |
+| 4 | one at a time | 70.7 | - | 79.6 | 6.0 s / 11.2 s |
+| 4 | `"parallel": 2` | 61.2 | -13 % | 32.2 | 4.5 s / 9.3 s |
+| 4 | `"parallel": 4` | 63.1 | -11 % | 16.9 | 1.0 s / 1.8 s |
 
 On this card the slots buy **waiting time, not speed**: the fourth of four requests starts after 1.8 s instead of
 11.2 s, but together they decode 11-24 % slower than one after the other, and a request alone loses 11 % (2 slots)
 or 24 % (4 slots), because the slots' sessions (0.56 GiB each) come out of the expert cache and most experts run
 on the CPU: a batch window over 4 conversations reads 24 CPU experts per layer against ~8 for one, so it costs about
 what the 4 tokens cost one after the other (`strata batch:` in the engine log: 54 ms per 4-row window, ~20 ms per
-1-row window). Cards that hold most experts in VRAM, and a layer split, are where the slots also add speed (below).
+1-row window). This is why setup leaves a 12 GB card at one at a time. Cards that hold most experts in VRAM, and a
+layer split, are where the slots also add speed (below).
 
 A 4-GPU layer split (4 x 16 GB, PCIe Gen3), IQ3_S, `--batch 8 --batch-groups 4 --trim-stage-weights`, through
 the HTTP server, 400 tokens per answer, temperature 0.7 (PR #559):

setup.py

@@ -3078,6 +3078,9 @@ def small_card_note(ctx: int, draft_vocab: str | None) -> list[str]:
 
 PARALLEL_MAX = 8               # #465: the engine's batch window holds at most 8 requests
 PARALLEL_SHARE = 0.2           # #465: the slots' sessions may take this share of the VRAM the expert cache would hold
+PARALLEL_HELD = 0.5            # #465: ... and only where the cache still holds this share of the experts beside them
+PARALLEL_COST_NOTE = ("parallel N reduces waiting for several users but costs about 10-25% speed per request on this "
+                      "card")
 
 
 def parallel_slot_gb(ctx: int, kv: str, streaming: bool) -> float:
@@ -3087,33 +3090,43 @@ def parallel_slot_gb(ctx: int, kv: str, streaming: bool) -> float:
     return (min(ctx, 32768) if streaming else ctx) * kv_tok / 1e9 + 0.17
 
 
-def parallel_recommend(vram_gb: float, ctx: int, kv: str, streaming: bool) -> int:
-    """#465: how many requests at once ("parallel") this card can take without the slots' sessions eating more than
-    PARALLEL_SHARE of its expert cache (~VRAM less 5 GB): 0 = none (one at a time).  Every slot's VRAM is expert cache
-    lost, and on a card whose experts mostly run on the CPU the batch reads ~as many experts as the requests one by one:
-    measured on a 12 GB RTX 5070 (Q2_0, 32K), 2 requests at once 66 tok/s together (one alone ~65), 4 at once 45."""
-    cache_gb = max(0.0, vram_gb - 5)
-    n = int(PARALLEL_SHARE * cache_gb / parallel_slot_gb(ctx, kv, streaming))
-    return min(4, n) if n >= 2 else 0
+def parallel_recommend(vram_gbs, arena_gb: float, ctx: int, kv: str, streaming: bool) -> int:
+    """#465: how many requests at once ("parallel") to recommend: 0 = none (one at a time).  Only where the experts
+    mostly fit in VRAM - the expert cache (each card's VRAM less ~5 GB, every card of a layer split) still holds
+    PARALLEL_HELD of the model's experts beside the slots' sessions, which take at most PARALLEL_SHARE of it, up to 4.
+    Where the experts mostly run on the CPU a batch reads about as many experts as the requests one by one and every
+    slot's VRAM is expert cache lost: measured on a 12 GB RTX 5070 (Q2_0, 32K), a request alone 11-24% slower with 2-4
+    slots, 4 requests together 63 tok/s against 71 one after the other (docs/BATCHING.md)."""
+    if isinstance(vram_gbs, (int, float)):
+        vram_gbs = [vram_gbs]
+    cache_gb = sum(max(0.0, v - 5) for v in vram_gbs)
+    slot = parallel_slot_gb(ctx, kv, streaming)
+    best = 0
+    for n in (2, 3, 4):
+        if n * slot <= PARALLEL_SHARE * cache_gb and (cache_gb - n * slot) >= PARALLEL_HELD * arena_gb:
+            best = n
+    return best
 
 
-def parallel_note(asked: int | None, vram_gb: float, ctx: int, kv: str, streaming: bool) -> list[str]:
-    """#465: what setup says about "parallel": the recommendation, or how the asked count compares with it (kept as
-    asked: recommend, never force)."""
-    rec = parallel_recommend(vram_gb, ctx, kv, streaming)
+def parallel_note(asked: int | None, vram_gbs, arena_gb: float, ctx: int, kv: str, streaming: bool) -> list[str]:
+    """#465: what setup says about "parallel": the recommendation (or, where it would cost speed, why it is left at
+    one), or how the asked count compares with it (kept as asked: recommend, never force)."""
+    rec = parallel_recommend(vram_gbs, arena_gb, ctx, kv, streaming)
     slot = parallel_slot_gb(ctx, kv, streaming)
     if asked is None or asked <= 1:
         if not rec:
-            return []
+            return [f"Several requests at once: left at one at a time - {PARALLEL_COST_NOTE} (docs/BATCHING.md)."]
         return [f"Several requests at once (opt-in): --parallel {rec} decodes up to {rec} together instead of one "
                 f"after the other (each takes ~{slot:.1f} GB of VRAM from the expert cache; docs/BATCHING.md)."]
     lines = [f"parallel requests: {asked} at once (each takes ~{slot:.1f} GB of VRAM from the expert cache, "
              f"{asked * slot:.1f} GB in all)"]
     if asked > PARALLEL_MAX:
         lines.append(f"the engine runs at most {PARALLEL_MAX} at once; it will use {PARALLEL_MAX}")
-    if asked > max(rec, 1):
-        lines.append(f"recommended for this card: {rec or 'one at a time'} - more slots leave fewer experts in VRAM, "
-                     "which can make every request slower; kept as you chose")
+    if not rec:
+        lines.append(f"recommended for this card: one at a time - {PARALLEL_COST_NOTE}; kept as you chose")
+    elif asked > rec:
+        lines.append(f"recommended for this card: {rec} - more slots leave fewer experts in VRAM, which can make every "
+                     "request slower; kept as you chose")
     return lines
 
 
@@ -4023,7 +4036,8 @@ def main() -> int:
     if a.parallel is not None:
         if a.parallel >= 2:
             cfg["parallel"] = a.parallel
-            for i, line in enumerate(parallel_note(a.parallel, gpu.get("vram_gb", 0.0), ctx, kv, streaming)):
+            for i, line in enumerate(parallel_note(a.parallel, [g.get("vram_gb", 0.0) for g in chosen],
+                                                   MODELS[model]["arena_gb"], ctx, kv, streaming)):
                 (ok if i == 0 else warn)(line)
         else:
             cfg["parallel"] = 1
@@ -4067,8 +4081,9 @@ def main() -> int:
     say(f"  Next time:        just run {'START-HERE.bat' if WIN else './setup.sh'} (or {script.name}) - it starts right away")
     if vision != "none":
         say("  Images:           send them in the chat page, in chat.py (/image <path>) or over the API")
-    if a.parallel is None and not multi:               # #465: the opt-in, said once (nothing changes)
-        for line in parallel_note(None, gpu.get("vram_gb", 0.0), ctx, kv, "--kv-resident" in cfg["args"]):
+    if a.parallel is None:                             # #465: the opt-in, said once (nothing changes)
+        for line in parallel_note(None, [g.get("vram_gb", 0.0) for g in chosen], MODELS[model]["arena_gb"], ctx, kv,
+                                  "--kv-resident" in cfg["args"]):
             say("  " + line)
     if tuned is False:                                 # #447: a failed tuning is repeated here, not only above
         say("  Tuning:           FAILED (the reason is above): the default settings stay - "

tools/test_setup_parallel.py

@@ -16,22 +16,37 @@ class Parallel(unittest.TestCase):
         self.assertEqual(S.parallel_slot_gb(262144, "int8", True), S.parallel_slot_gb(32768, "int8", False))
         self.assertLess(S.parallel_slot_gb(32768, "q4_0", False), S.parallel_slot_gb(32768, "int8", False))
 
-    def test_recommendation_by_card(self):
-        self.assertEqual(S.parallel_recommend(8, 32768, "int8", False), 0)      # too small: one at a time
-        self.assertEqual(S.parallel_recommend(12, 32768, "int8", False), 2)     # the measured 5070
-        self.assertEqual(S.parallel_recommend(24, 32768, "int8", False), 4)
-        self.assertEqual(S.parallel_recommend(48, 32768, "int8", False), 4)     # capped at 4
-        self.assertEqual(S.parallel_recommend(12, 262144, "int8", False), 0)    # a 262K KV cache per slot: no
-        self.assertEqual(S.parallel_recommend(12, 262144, "int8", True), 2)     # ... unless it streams
+    def test_recommendation_only_where_the_experts_mostly_fit(self):
+        q2, iq3s = S.MODELS["Q2_0"]["arena_gb"], S.MODELS["IQ3_S"]["arena_gb"]
+        # most experts on the CPU: slots cost speed per request (measured on the 12 GB 5070) - none recommended
+        self.assertEqual(S.parallel_recommend(12, q2, 32768, "int8", False), 0)
+        self.assertEqual(S.parallel_recommend(16, q2, 32768, "int8", False), 0)
+        self.assertEqual(S.parallel_recommend(24, iq3s, 32768, "int8", False), 0)
+        # the cache holds most of the experts beside the slots: recommended
+        self.assertEqual(S.parallel_recommend(24, q2, 32768, "int8", False), 3)
+        self.assertEqual(S.parallel_recommend(48, q2, 32768, "int8", False), 4)       # capped at 4
+        self.assertEqual(S.parallel_recommend([24, 24], iq3s, 32768, "int8", False), 4)   # a layer split: every card
+        self.assertEqual(S.parallel_recommend([16, 16], q2, 32768, "int8", False), 4)
+        self.assertEqual(S.parallel_recommend(32, q2, 262144, "int8", False), 0)      # a 262K KV per slot: no
+        self.assertEqual(S.parallel_recommend(32, q2, 262144, "int8", True), 4)       # ... unless it streams
 
     def test_notes_recommend_never_force(self):
-        self.assertEqual(S.parallel_note(None, 8, 32768, "int8", False), [])
-        self.assertIn("--parallel 2", S.parallel_note(None, 12, 32768, "int8", False)[0])
-        lines = S.parallel_note(4, 12, 32768, "int8", False)
+        q2 = S.MODELS["Q2_0"]["arena_gb"]
+        # 12 GB: left at one, with the reason
+        note = S.parallel_note(None, 12, q2, 32768, "int8", False)
+        self.assertEqual(len(note), 1)
+        self.assertIn("one at a time", note[0])
+        self.assertIn("costs about 10-25% speed per request on this card", note[0])
+        self.assertIn("--parallel 3", S.parallel_note(None, 24, q2, 32768, "int8", False)[0])
+        # asked on 12 GB: kept, with the cost said
+        lines = S.parallel_note(4, 12, q2, 32768, "int8", False)
         self.assertIn("4 at once", lines[0])
-        self.assertTrue(any("recommended for this card: 2" in x and "kept as you chose" in x for x in lines))
-        self.assertTrue(any("at most 8" in x for x in S.parallel_note(12, 48, 32768, "int8", False)))
-        self.assertEqual(len(S.parallel_note(2, 12, 32768, "int8", False)), 1)   # as recommended: no warning
+        self.assertTrue(any("one at a time" in x and "kept as you chose" in x and "10-25%" in x for x in lines))
+        # asked above the recommendation where it pays
+        lines = S.parallel_note(4, 24, q2, 32768, "int8", False)
+        self.assertTrue(any("recommended for this card: 3" in x for x in lines))
+        self.assertTrue(any("at most 8" in x for x in S.parallel_note(12, 48, q2, 32768, "int8", False)))
+        self.assertEqual(len(S.parallel_note(3, 24, q2, 32768, "int8", False)), 1)   # as recommended: no warning
 
 
 if __name__ == "__main__":