149: Linux defaults - zero-copy page-cached experts, io prefetch and its decode gate; cgroup RAM check counts the clean page cache

- STRATA_COLD_OVERLAP defaults to 4 on Linux (0 elsewhere), STRATA_IO_PREFETCH to 1 on Linux for the
  file tier (mapped GGUF in place or experts.bin), STRATA_IO_PF_GATE to 1 with io prefetch. Off switches:
  STRATA_COLD_OVERLAP=0, STRATA_IO_PREFETCH=0, STRATA_IO_PF_GATE=0. One startup line names them.
- cgroup_available_bytes: every clean file page of the group (memory.stat file, active + inactive; else
  inactive_file) is reclaimable, less min(half of it, limit / 64), as MemAvailable counts page cache on a
  bare host. A container whose own reads left active cache was judged short of RAM and its file tier
  switched to unbuffered reads (an 18 GiB RAM tier in a 32 GiB group: 1.2 GiB available instead of ~11).
  Unit tests with mock memory.stat / memory.max values.
- The per-request cold-reads line only with STRATA_COLD_OVERLAP or STRATA_IO_STATS set.
- docs/DETAILS.md: the 0.2.0 Linux file-tier defaults.

docs/DETAILS.md

@@ -261,7 +261,28 @@ from the GGUF gave identical tokens and logits. An expert read from the GGUF is
 engine fetches a layer's missing experts on 8 threads (`STRATA_FETCH_THREADS`) with one batched page request
 (Windows `PrefetchVirtualMemory`). With an `experts.bin` in the pack, nothing changes. Setup does not use this yet.
 
-**Linux file-tier I/O path (opt in, `STRATA_IO_PREFETCH=1`; 0.1.40.2):** the mapped file tier reads experts through
+**Linux file tier, 0.2.0 defaults (zero-copy page-cached experts, io prefetch, the decode gate):** when experts are read
+from the files (a GGUF read in place, `--mmap-experts`, or an `experts.bin` file tier), Linux now runs three things by default.
+(1) **Zero-copy** (`STRATA_COLD_OVERLAP=4`): an expert of a GGUF read in place whose three tensors' pages are all in the page
+cache (mincore) is computed straight from the mapping; before, every such expert was first assembled into a staging buffer
+(a copy of ~1.5 MB per Q2_0 expert, on the decode's critical path). Same bytes, same rows, same output (identity gates
+IDENTICAL). (2) **io prefetch** (`STRATA_IO_PREFETCH=1`, the path described below): whole-blob `pread`s and the router-
+predicted experts of the next layers read ahead. (3) **the decode gate** (`STRATA_IO_PF_GATE=1`): every 16 decode windows the
+decode's read-ahead is judged - it runs only while the page cache has room for the experts (the memory available, the
+tightest cgroup limit included, over the experts' bytes: off below 0.55, on above 0.65, `STRATA_IO_PF_GATE_LO` / `_HI`) and
+while it pays (at least 128 MB read ahead in 16 windows and 10% of it used, `STRATA_IO_PF_GATE_USEFUL`; a probe every 256
+windows when it closed for that). Closed, the decode reads as it did without io prefetch (the look-ahead predicts one layer
+and warms with an madvise hint); the prompt path keeps its preads either way. Measured on an RTX PRO 5000 emulating a 12 GB
+card (`--expert-cache 4448`, Q2_0 GGUF in place, page cache dropped before each engine, RAM limited by a cgroup), 5 pairs,
+story / code / doc8k decode, 4K / 20K prompts, against the 0.1.4x defaults: 32 GB of RAM 1.8 / 2.8 / 3.1x decode, 1.5 / 2.0x
+prompts; 16 GB 1.2 / 1.4 / 1.4x decode, 1.6 / 1.8x prompts (the gate stays closed there); an 18 GiB RAM tier B32_RESULT. On a real RTX 3060 12 GB (32 GB cgroup, 3 pairs) zero-copy alone gave 1.14 / 1.44 / 1.47x decode, 1.06 / 1.02x
+prompts. With it the container RAM check counts a cgroup's clean page cache (active and inactive) as available, as
+MemAvailable does on a bare host (before, a 32 GiB container whose own reads left 26 GiB of cache was judged short of RAM
+and its file tier switched to unbuffered reads). Off switches: `STRATA_COLD_OVERLAP=0`, `STRATA_IO_PREFETCH=0`,
+`STRATA_IO_PF_GATE=0`; the startup log names what is on. Windows and macOS are unchanged. Measured detail:
+phaseA-tests/notes-149-l620.md.
+
+**Linux file-tier I/O path (`STRATA_IO_PREFETCH=1`; 0.1.40.2, on by default on Linux since 0.2.0 - see above):** the mapped file tier reads experts through
 page faults. With this on, the layer's uncached experts and the router-predicted ones of the next layer (`STRATA_LOOKAHEAD_K`,
 `STRATA_IO_PREFETCH_DEPTH`, default 2 when on) are read by I/O threads (`STRATA_IO_PF_THREADS`, default 8) with whole-blob
 `pread`s into the page cache, and the CPU kernels read the mapping as before. The bytes are the same, so the output is the

include/strata/core/expert_source.hpp

@@ -65,9 +65,14 @@ struct CgroupMemoryStat {
     uint64_t file_dirty = 0;
     uint64_t file_writeback = 0;
     bool valid = false;
+    uint64_t file = 0;   ///< 149: memory.stat "file" (active + inactive page cache); 0 when not reported
 };
 
-/// Calculate additional bytes under a finite cgroup limit after reclaiming only clean inactive file cache.
+/// Calculate additional bytes under a finite cgroup limit after reclaiming the group's clean page cache, as
+/// MemAvailable counts it on a bare host: every clean file page (active and inactive: memory.stat "file", else
+/// inactive_file), less a low-watermark share the kernel keeps (min(half the clean cache, limit / 64)).  A process
+/// whose own reads (a GPU cache fill, a read-ahead) left gigabytes of active cache in its group was judged short of
+/// RAM before (149: 1.2 GiB "available" of a 32 GiB group, the file tier then switched to unbuffered reads).
 /// Returns false when the required memory.stat counters were unavailable.
 bool cgroup_available_bytes(uint64_t limit, const CgroupMemoryStat& stat, uint64_t& bytes);
 

src/core/expert_source.cpp

@@ -75,10 +75,12 @@ bool cgroup_available_bytes(uint64_t limit, const CgroupMemoryStat& stat, uint64
     bytes = 0;
     if (!stat.valid) return false;
 
-    // memory.stat's inactive_file can race memory.current, so bound it to charged usage first.
-    uint64_t reclaimable = std::min(stat.inactive_file, stat.current);
+    // memory.stat's counters can race memory.current, so bound them to charged usage first.  The whole clean page
+    // cache is reclaimable (149: MemAvailable counts active file pages too); the kernel keeps a low-watermark share.
+    uint64_t reclaimable = std::min(std::max(stat.file, stat.inactive_file), stat.current);
     reclaimable = stat.file_dirty >= reclaimable ? 0 : reclaimable - stat.file_dirty;
     reclaimable = stat.file_writeback >= reclaimable ? 0 : reclaimable - stat.file_writeback;
+    reclaimable -= std::min(reclaimable / 2, limit / 64);
 
     // Reclaiming clean file pages reduces usage; saturating subtraction also handles a transient over-limit read.
     const uint64_t usage_after_reclaim = stat.current - reclaimable;
@@ -291,7 +293,7 @@ bool read_cgroup_memory_stat(const std::filesystem::path& path, uint64_t current
     std::ifstream input(path / "memory.stat");
     if (!input) return false;
 
-    bool inactive_file = false, file_dirty = false, file_writeback = false;
+    bool inactive_file = false, file_dirty = false, file_writeback = false, file = false;
     std::string line;
     while (std::getline(input, line)) {
         std::istringstream fields(line);
@@ -313,6 +315,10 @@ bool read_cgroup_memory_stat(const std::filesystem::path& path, uint64_t current
             if (file_writeback) return false;
             file_writeback = true;
             stat.file_writeback = value;
+        } else if (key == "file") {   // 149: optional (every v2 kernel has it); 0 = inactive_file alone
+            if (file) return false;
+            file = true;
+            stat.file = value;
         }
     }
     if (!input.eof() || !inactive_file || !file_dirty || !file_writeback) return false;
@@ -771,7 +777,8 @@ void FileExpertSource::set_io_prefetch(bool on, int threads) {
         if (const char* m = std::getenv("STRATA_IO_PF_BUDGET_MB")) io_budget_fixed_ = (uint64_t) std::max(0.0, std::atof(m) * 1048576.0);
         if (const char* m = std::getenv("STRATA_IO_PF_BUDGET_FRAC")) io_budget_frac_ = std::clamp(std::atof(m), 0.01, 4.0);
         if (const char* m = std::getenv("STRATA_IO_PF_TARGET")) io_target_ = std::clamp(std::atof(m), 0.0, 1.0);
-        if (const char* m = std::getenv("STRATA_IO_PF_GATE")) gate_on_ = std::atoi(m) != 0;
+        gate_on_ = true;   // 0.2.0: the decode gate comes with io prefetch; STRATA_IO_PF_GATE=0 turns it off
+        if (const char* m = std::getenv("STRATA_IO_PF_GATE"); m != nullptr && m[0] != '\0') gate_on_ = std::atoi(m) != 0;
         if (const char* m = std::getenv("STRATA_IO_PF_GATE_LO")) gate_lo_ = std::clamp(std::atof(m), 0.0, 1.0);
         if (const char* m = std::getenv("STRATA_IO_PF_GATE_HI")) gate_hi_ = std::clamp(std::atof(m), gate_lo_, 1.0);
         if (const char* m = std::getenv("STRATA_IO_PF_GATE_USEFUL")) gate_min_useful_ = std::clamp(std::atof(m), 0.0, 1.0);
@@ -1815,7 +1822,14 @@ int64_t FileExpertSource::prefetch_begin(int64_t layer, const int64_t* experts,
     prefetch_end();
     async_todo_.clear();
     async_fill_us_ = 0;
-    static const int mode = [] { const char* v = std::getenv("STRATA_COLD_OVERLAP"); return v != nullptr ? std::atoi(v) : 0; }();
+    static const int mode = [] {   // as the dispatch reads it (0.2.0: 4 by default on Linux)
+        const char* v = std::getenv("STRATA_COLD_OVERLAP");
+#if defined(__linux__)
+        return v == nullptr || v[0] == '\0' ? 4 : std::atoi(v);
+#else
+        return v == nullptr || v[0] == '\0' ? 0 : std::atoi(v);
+#endif
+    }();
     claim_todo(layer, experts, n, async_todo_, pending, mode >= 3);
     if (mode == 4) {   // zero-copy only: the blobs that must be read are read here, as `prefetch` reads them
         const auto t0 = std::chrono::steady_clock::now();
@@ -4370,9 +4384,14 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
     const auto c0 = std::chrono::steady_clock::now();
     const FileExpertSource::DecodeScope decode_scope;   // 149 STRATA_IO_PF_GATE: this thread reads for the decode
     // 149-cold STRATA_COLD_OVERLAP: 0 off (default), 1 overlap, 2 measure only, 3 overlap + zero-copy, 4 zero-copy only
+    // 0.2.0: 4 by default on Linux (zero-copy needs the page cache's residency query); elsewhere 0
     static const int cold_env = [] {
         const char* v = std::getenv("STRATA_COLD_OVERLAP");
-        return v == nullptr ? 0 : std::clamp(std::atoi(v), 0, 4);
+#if defined(__linux__)
+        return v == nullptr || v[0] == '\0' ? 4 : std::clamp(std::atoi(v), 0, 4);
+#else
+        return v == nullptr || v[0] == '\0' ? 0 : std::clamp(std::atoi(v), 0, 4);
+#endif
     }();
     d.cold_mode = cold_env;
     pt("begin");

src/program/generate.cpp

@@ -5445,19 +5445,35 @@ int main(int argc, char** argv) {
                                      "(%s)\n", why.c_str());
         }
 #endif
-        // Linux I/O path (opt in): whole-blob preads instead of page faults, and the predicted experts of the next
-        // layers read ahead on I/O threads while the current layer computes (see FileExpertSource::set_io_prefetch)
-        if (const char* v = std::getenv("STRATA_IO_PREFETCH"); v != nullptr && std::atoi(v) != 0) {
+        // Linux I/O path: whole-blob preads instead of page faults, and the predicted experts of the next layers read
+        // ahead on I/O threads while the current layer computes (see FileExpertSource::set_io_prefetch).  0.2.0: on by
+        // default on Linux, with the decode gate (STRATA_IO_PF_GATE) and zero-copy page-cached experts
+        // (STRATA_COLD_OVERLAP=4); STRATA_IO_PREFETCH=0 / STRATA_IO_PF_GATE=0 / STRATA_COLD_OVERLAP=0 turn them off.
+        const char* io_pf_env = std::getenv("STRATA_IO_PREFETCH");
+#if defined(__linux__)
+        const bool io_pf_default = io_pf_env == nullptr || io_pf_env[0] == '\0';
+#else
+        const bool io_pf_default = false;
+#endif
+        if (io_pf_default || (io_pf_env != nullptr && std::atoi(io_pf_env) != 0)) {
             const char* th = std::getenv("STRATA_IO_PF_THREADS");
             src.set_io_prefetch(true, th != nullptr ? std::atoi(th) : 0);
-            if (src.io_prefetch())
+            if (src.io_prefetch() && io_pf_default) {
+                const char* cv = std::getenv("STRATA_COLD_OVERLAP");
+                std::fprintf(stderr, "strata generate: file tier (Linux defaults): page-cached experts computed from the "
+                                     "mapping (STRATA_COLD_OVERLAP=%s), io prefetch with whole-blob preads on %s I/O threads, "
+                                     "the decode's read-ahead %s; off: STRATA_COLD_OVERLAP=0, STRATA_IO_PREFETCH=0, "
+                                     "STRATA_IO_PF_GATE=0\n", cv != nullptr && cv[0] != '\0' ? cv : "4", th != nullptr ? th : "8",
+                             src.gate_on() ? "only while the page cache has room for it and it pays (STRATA_IO_PF_GATE=1)"
+                                           : "always (STRATA_IO_PF_GATE=0)");
+            } else if (src.io_prefetch())
                 std::fprintf(stderr, "strata generate: file tier io prefetch ON (STRATA_IO_PREFETCH=1): whole-blob preads, "
-                                     "the layer's uncached experts and the predicted ones read by %s I/O threads (STRATA_IO_PF_THREADS, default 8)\n",
-                             th != nullptr ? th : "8");
+                                     "the layer's uncached experts and the predicted ones read by %s I/O threads (STRATA_IO_PF_THREADS, default 8)%s\n",
+                             th != nullptr ? th : "8", src.gate_on() ? "; the decode's read-ahead gated (STRATA_IO_PF_GATE=0: always)" : "");
             if (src.io_prefetch() && src.io_sched())
                 std::fprintf(stderr, "strata generate: io prefetch scheduling ON (STRATA_IO_PF_SCHED=1): a deadline per read, "
                                      "confidence admission (STRATA_IO_PF_MINCONF), a per-window byte budget (STRATA_IO_PF_BUDGET_MB / _FRAC)\n");
-            if (!src.io_prefetch())
+            if (!src.io_prefetch() && !io_pf_default)
                 std::fprintf(stderr, "strata generate: STRATA_IO_PREFETCH=1 is not available here (Linux mapped file tier "
                                      "only, not with unbuffered reads); the file tier reads as before\n");
             if (src.io_prefetch()) {   // recommend, never force: the engine runs as asked
@@ -14954,7 +14970,8 @@ int main(int argc, char** argv) {
                     fb_prev = fb;
                 }
             }
-            if (drive.d.cold_mode != 0) {   // 149-cold STRATA_COLD_OVERLAP: this request's layers that had a blob to read
+            static const bool cold_log = std::getenv("STRATA_COLD_OVERLAP") != nullptr || std::getenv("STRATA_IO_STATS") != nullptr;
+            if (drive.d.cold_mode != 0 && cold_log) {   // 149-cold STRATA_COLD_OVERLAP: this request's layers that had a blob to read
                 static strata::core::ExpertDispatch cp;
                 const strata::core::ExpertDispatch& c = drive.d;
                 const double nl = (double) std::max<int64_t>(1, c.cold_layers - cp.cold_layers);

tests/core/file_expert_source_test.cpp

@@ -433,18 +433,41 @@ void test_cgroup_memory_budget() {
     constexpr uint64_t GiB = 1ull << 30;
     uint64_t bytes = 0;
 
+    // the kernel keeps min(half the clean cache, limit / 64) of it: 56 GiB / 64 = 0.875 GiB here
+    const uint64_t keep56 = 56 * GiB / 64;
     CgroupMemoryStat clean_cache{40 * GiB, 32 * GiB, 0, 0, true};
     require(cgroup_available_bytes(56 * GiB, clean_cache, bytes), "valid cgroup memory.stat was rejected");
-    require(bytes == 48 * GiB, "clean inactive file cache was not credited against the cgroup cap");
+    require(bytes == 48 * GiB - keep56, "clean inactive file cache was not credited against the cgroup cap");
 
     CgroupMemoryStat dirty_cache{40 * GiB, 32 * GiB, 4 * GiB, 2 * GiB, true};
     require(cgroup_available_bytes(56 * GiB, dirty_cache, bytes), "valid dirty-cache counters were rejected");
-    require(bytes == 42 * GiB, "dirty/writeback pages were incorrectly counted as reclaimable");
+    require(bytes == 42 * GiB - keep56, "dirty/writeback pages were incorrectly counted as reclaimable");
 
     CgroupMemoryStat oversized_inactive{10 * GiB, 20 * GiB, 0, 0, true};
     require(cgroup_available_bytes(12 * GiB, oversized_inactive, bytes),
             "valid oversized inactive-file counter was rejected");
-    require(bytes == 12 * GiB, "inactive-file accounting exceeded charged current usage");
+    require(bytes == 12 * GiB - 12 * GiB / 64, "inactive-file accounting exceeded charged current usage");
+
+    // 149: the ACTIVE page cache is reclaimable too (memory.stat "file" = active + inactive), as MemAvailable counts
+    // it: a 32 GiB group whose own reads left 26 GiB of cache, 25 GiB of it active, and 4.5 GiB of anonymous memory
+    CgroupMemoryStat active_cache{30 * GiB + GiB / 2, 1 * GiB, 0, 0, true, 26 * GiB};
+    require(cgroup_available_bytes(32 * GiB, active_cache, bytes), "a memory.stat with file was rejected");
+    require(bytes == 32 * GiB - (4 * GiB + GiB / 2) - 32 * GiB / 64, "the active page cache was not credited");
+    // without "file" (an old kernel's stat): inactive_file alone, as before
+    CgroupMemoryStat no_file{30 * GiB + GiB / 2, 1 * GiB, 0, 0, true};
+    require(cgroup_available_bytes(32 * GiB, no_file, bytes) && bytes == 32 * GiB - (29 * GiB + GiB / 2) - GiB / 2,
+            "a stat without file did not fall back to inactive_file");
+    // dirty and writeback pages of the active cache stay charged; a small cache keeps half of itself
+    CgroupMemoryStat dirty_active{20 * GiB, 1 * GiB, 2 * GiB, 1 * GiB, true, 10 * GiB};
+    require(cgroup_available_bytes(32 * GiB, dirty_active, bytes) && bytes == 32 * GiB - (20 * GiB - 7 * GiB) - GiB / 2,
+            "dirty / writeback pages of the active cache were counted as reclaimable");
+    CgroupMemoryStat tiny_cache{8 * GiB, 0, 0, 0, true, 256ull << 20};
+    require(cgroup_available_bytes(32 * GiB, tiny_cache, bytes) && bytes == 32 * GiB - 8 * GiB + (128ull << 20),
+            "a small clean cache was not kept at half");
+    // "file" larger than what is charged (a racy read) is bounded by memory.current
+    CgroupMemoryStat racy{4 * GiB, 1 * GiB, 0, 0, true, 9 * GiB};
+    require(cgroup_available_bytes(32 * GiB, racy, bytes) && bytes == 32 * GiB - GiB / 2,
+            "a file counter above memory.current escaped the bound");
 
     const uint64_t max = std::numeric_limits<uint64_t>::max();
     CgroupMemoryStat large_counters{max, max, max - 1, max, true};
@@ -562,7 +585,13 @@ void test_host_memory() {
     put("fs/box/memory.max", std::to_string(48 * GiB) + "\n");
     put("fs/box/memory.current", std::to_string(16 * GiB) + "\n");
     put("fs/box/memory.stat", "inactive_file " + std::to_string(4 * GiB) + "\nfile_dirty 0\nfile_writeback 0\n");
-    require(probe("0::/box\n") && m.available == 36 * GiB && m.cgroup_limit == 48 * GiB, "v2 limit");
+    require(probe("0::/box\n") && m.available == 36 * GiB - 48 * GiB / 64 && m.cgroup_limit == 48 * GiB, "v2 limit");
+    // 149: with "file" (active + inactive cache) the active part counts too: 12 GiB of cache, 4 GiB anonymous
+    put("fs/box/memory.stat", "file " + std::to_string(12 * GiB) + "\ninactive_file " + std::to_string(4 * GiB) +
+                              "\nfile_dirty 0\nfile_writeback 0\n");
+    require(probe("0::/box\n") && m.available == 44 * GiB - 48 * GiB / 64 && m.cgroup_limit == 48 * GiB,
+            "v2 limit, the active page cache credited");
+    put("fs/box/memory.stat", "inactive_file " + std::to_string(4 * GiB) + "\nfile_dirty 0\nfile_writeback 0\n");
     put("fs/box/memory.max", "max\n");
     require(probe("0::/box\n") && m.available == 100 * GiB && m.cgroup_limit == ~0ull, "v2 max");
     put("fs/box/memory.max", "48G\n");