Layer split + --batch: the default pipeline wins over --batch-mtp (STRATA_BATCH_MTP_SERIAL=1 or --batch-groups 1 keeps the serial MTP windows) (#2038)

0.1.42 let --batch-mtp run on a layer split and made the --batch-groups auto default give way to it: one serial group. On
2x RTX A4000 (IQ3_XXS, 3 pairs) four streams aggregated 62.2 tok/s against 92.7 on 0.1.41's pipelined groups (-33%, 0/3 pairs
faster), two streams 55.7 against 71.4 (-22%). With --batch-groups 2 (which turns --batch-mtp off) 88.3 and 71.1. The start-up
line says why --batch-mtp is off and how to keep it. --adapt-async 1 still gives one group. Solo decode and the answers are
untouched.

docs/BATCHING.md

@@ -35,7 +35,7 @@ drafter's does). A window carries two rows per active slot (its token and its pr
 four slots are in one window; more slots (`--batch` up to what fits) rotate through. The windows run through the stages
 one after the other (`--batch-groups 1`): on 0.1.41 a layer split with `--batch` 2 or more pipelines the slots in groups
 by default, and the pipelined path does not run the slots' MTP drafts, so with `--batch-mtp` and no `--batch-groups` the
-engine runs one group and says so; `--batch-groups G` (G above 1) or `--batch-groups auto` given on the command line
+engine (0.1.43.1) pipelines and leaves `--batch-mtp` off, and says so (0.1.42/0.1.43 ran one serial group with it: 4 streams on 2x A4000 aggregated 63.9 tok/s against 91.8 pipelined, 85.8 pipelined without `--batch-mtp`; #2038). `--batch-groups 1` or `STRATA_BATCH_MTP_SERIAL=1` keeps `--batch-mtp` with serial windows, which is the better choice for about two streams as measured below; `--batch-groups G` (G above 1) or `--batch-groups auto` given on the command line
 keeps the pipeline and turns `--batch-mtp` off. Only a layer split of two stages on two GPUs has been run.
 
 Measured on a layer split of 2x RTX 3080 20 GB (220 W cap) with a Xeon E5-2696 v4, UD-Q4_K_XL, `"layer_split": "23"`,
@@ -86,7 +86,7 @@ With a layer split, the engine options go into the config's `args`:
 | --- | --- |
 | `"parallel": N` / `--batch N` / `--slots N` (2..8 normally) | up to N conversations have batch slots; more requests wait for a free slot. Each slot gets its own state (a session carved like the stage's own: GDN recurrence, QSA K/V and indexer, PLE history) on every GPU of the split. With grouped MTP, more than 8 slots can rotate through eight-row windows if memory permits. |
 | `--batch-groups G` | with a layer split (default: auto, below): the N slots in G groups that flow through the GPUs as a pipeline (GPU k runs one group while GPU k+1 runs another). G must divide N. 1 = all slots in one window, GPU after GPU. |
-| `--batch-groups auto` | the default on a layer split since 0.1.41 (give no `--batch-groups`): the engine pipelines one group per GPU stage (the most that divide the slots; 8 slots on 4 GPUs = 4 groups of 2) and says so (`INFO batch_groups=G`). `--batch-groups 1` turns it off (all slots in one window, GPU after GPU). Measured, 8 clients, total tok/s against one group: 4 x R9700 166 against 86, 2 GPUs 109 against 78, 3 GPUs 110 against 78. With `--batch-mtp` or `--adapt-async 1` and no `--batch-groups` the default is one group (the slots' MTP drafts and the adaptive tier between batch windows do not run in pipelined groups) and the log says so; `--batch-groups G` or `--batch-groups auto` given with `--batch-mtp` pipelines the groups and turns `--batch-mtp` off, and given with `--adapt-async 1` pipelines the groups and turns `--adapt-async` off (the pipelined path does not adapt). |
+| `--batch-groups auto` | the default on a layer split since 0.1.41 (give no `--batch-groups`): the engine pipelines one group per GPU stage (the most that divide the slots; 8 slots on 4 GPUs = 4 groups of 2) and says so (`INFO batch_groups=G`). `--batch-groups 1` turns it off (all slots in one window, GPU after GPU). Measured, 8 clients, total tok/s against one group: 4 x R9700 166 against 86, 2 GPUs 109 against 78, 3 GPUs 110 against 78. With `--adapt-async 1` and no `--batch-groups` the default is one group (the adaptive tier between batch windows does not run in pipelined groups) and the log says so; with `--batch-mtp` and no `--batch-groups` the pipeline wins and `--batch-mtp` is left off (`--batch-groups 1` or `STRATA_BATCH_MTP_SERIAL=1` keeps it); `--batch-groups G` or `--batch-groups auto` given with `--batch-mtp` pipelines the groups and turns `--batch-mtp` off, and given with `--adapt-async 1` pipelines the groups and turns `--adapt-async` off (the pipelined path does not adapt). |
 | `--trim-stage-weights` | with an **explicit** `--layer-split` (e.g. `12,24,36`, not `auto`): every GPU loads only the dense weights of its own layers instead of the whole model's (the same as `STRATA_STAGE_TRIM=1`, PR #639). The VRAM this frees goes to the expert cache. Useful without `--batch` too. |
 
 The engine never refuses a count it cannot run: it says so in its log and runs what it can - at most 8 slots by

include/strata/program/batch_groups.hpp

@@ -3,9 +3,12 @@
 // `--batch-groups N` pipelines N groups of slots through the stages of a layer split; `--batch-groups auto` picks the
 // most groups (at most one per stage) that divide the slots.  Since 0.1.41 a layer split with `--batch` >= 2 and no
 // `--batch-groups` at all means `auto`.  The pipelined path does not run `--batch-mtp`'s drafts or the batch
-// adaptive tier (`--adapt-async 1` between batch windows), so when one of those is asked for and the number of groups
-// was NOT given, `auto` resolves to one group (the serial windows they run in) and says why.  A number or `auto`
-// given on the command line is honoured as before.
+// adaptive tier (`--adapt-async 1` between batch windows).  `--adapt-async 1` with the number of groups NOT given still
+// makes `auto` resolve to one group (and says why).  `--batch-mtp` does not any more (0.1.43.1): on 2x A4000 and 2x RTX
+// 3090 four concurrent streams lost 25-30% with the serial MTP windows against the pipelined default, so the default
+// pipelines and `--batch-mtp` is left off there (said at start); STRATA_BATCH_MTP_SERIAL=1 or `--batch-groups 1` keeps
+// the serial windows with the slots' drafts (the 0.1.42/0.1.43 behaviour).  A number or `auto` given on the command line
+// is honoured as before.
 //
 // Pure (no GPU, no environment read): generate.cpp passes in what it knows.
 #pragma once
@@ -23,6 +26,7 @@ struct Input {
     int later_stages = 0;      ///< the layer split's stages after CUDA0 (0: no layer split)
     bool batch_mtp = false;    ///< --batch-mtp is on (it passed the gates that do not concern the groups)
     bool adapt_async = false;  ///< --adapt-async 1 was asked for
+    bool mtp_serial = false;   ///< STRATA_BATCH_MTP_SERIAL=1: --batch-mtp with no --batch-groups runs one serial group (0.1.43)
 };
 
 struct Result {
@@ -30,6 +34,7 @@ struct Result {
     bool from_auto = false;            ///< the number came from `auto` (given or the 0.1.41 default)
     Serial serial = Serial::None;      ///< `auto` would have pipelined, but this feature needs the serial windows
     int would_be = 1;                  ///< what `auto` would have picked without it (== groups unless `serial`)
+    bool mtp_dropped = false;          ///< --batch-mtp met the default pipeline: the pipeline wins, --batch-mtp is off
 };
 
 /// The most groups, at most one per stage of the layer split (`later_stages` + 1 GPUs), that divide `batch`.
@@ -51,8 +56,9 @@ inline Result resolve(const Input& in) {
     r.would_be = auto_groups(in.later_stages, in.batch);
     r.groups = r.would_be;
     if (!in.set && r.would_be > 1) {   // only the default gives way; `--batch-groups auto` asked for the pipeline
-        if (in.batch_mtp) r.serial = Serial::BatchMtp;
+        if (in.batch_mtp && in.mtp_serial) r.serial = Serial::BatchMtp;
         else if (in.adapt_async) r.serial = Serial::AdaptAsync;
+        r.mtp_dropped = in.batch_mtp && r.serial == Serial::None;
         if (r.serial != Serial::None) r.groups = 1;
     }
     return r;

src/program/batch_groups_test.cpp

@@ -49,9 +49,13 @@ int main() {
     {   // the production shape: parallel 2 on a 2-GPU split with --batch-mtp
         bg::Input i = in(2, 1);
         i.batch_mtp = true;
-        const auto r = bg::resolve(i);
-        check(r.groups == 1 && r.from_auto && r.serial == bg::Serial::BatchMtp && r.would_be == 2,
-              "--batch-mtp, nothing given: one group, because of --batch-mtp");
+        auto r = bg::resolve(i);
+        check(r.groups == 2 && r.from_auto && r.serial == bg::Serial::None && r.mtp_dropped,
+              "--batch-mtp, nothing given: the pipeline wins, --batch-mtp is dropped");
+        i.mtp_serial = true;
+        r = bg::resolve(i);
+        check(r.groups == 1 && r.from_auto && r.serial == bg::Serial::BatchMtp && r.would_be == 2 && !r.mtp_dropped,
+              "STRATA_BATCH_MTP_SERIAL: one group, because of --batch-mtp");
         check(bg::serial_what(r.serial)[0] == '-', "the reason names the feature");
     }
     {
@@ -64,7 +68,9 @@ int main() {
         bg::Input i = in(2, 1);
         i.batch_mtp = true;
         i.adapt_async = true;
-        check(bg::resolve(i).serial == bg::Serial::BatchMtp, "both: --batch-mtp is named");
+        check(bg::resolve(i).serial == bg::Serial::AdaptAsync, "both: --adapt-async 1 still gives one group");
+        i.mtp_serial = true;
+        check(bg::resolve(i).serial == bg::Serial::BatchMtp, "both with STRATA_BATCH_MTP_SERIAL: --batch-mtp is named");
     }
     {   // nothing to give way: the slots do not divide, so auto is one group anyway and there is nothing to say
         bg::Input i = in(3, 1);

src/program/generate.cpp

@@ -4360,6 +4360,7 @@ int main(int argc, char** argv) {
         bin.later_stages = (int) stages.size();
         bin.batch_mtp = mtp_on;
         bin.adapt_async = o.adapt_async != 0;
+        { const char* ms = std::getenv("STRATA_BATCH_MTP_SERIAL"); bin.mtp_serial = ms != nullptr && ms[0] != '\0' && ms[0] != '0'; }
         return strata::program::batch_groups::resolve(bin);
     };
     const char* batch_mtp_env = std::getenv("STRATA_BATCH_MTP");
@@ -4372,7 +4373,12 @@ int main(int argc, char** argv) {
                         : !o.serve ? "it needs --serve" : nullptr;
         // the groups it would run with (the default gives way to it: one group; a number or auto given pipelines)
         if (why == nullptr && !stages.empty() && batch_groups_for(true).groups > 1)
-            why = "it does not combine with --batch-groups (pipelined slot groups) yet";
+            why = batch_groups_for(true).mtp_dropped
+                ? "a layer split with --batch pipelines the slots in groups by default and the pipelined groups run no slot "
+                  "drafts (4 streams on 2 GPUs: 25-30% faster than the serial MTP windows); --batch-groups 1 or "
+                  "STRATA_BATCH_MTP_SERIAL=1 keeps --batch-mtp with serial windows (fewer streams, or when prompts arrive "
+                  "rarely, it can be the faster choice)"
+                : "it does not combine with --batch-groups (pipelined slot groups) yet";
         // (a layer split with helper-GPU expert caches or --remote-expert-opt has not been run with slot drafters)
         if (why == nullptr && !stages.empty() && (remote_caches || o.remote_expert_opt))
             why = "it is not run with helper expert caches on a layer split";