build(lock): two machine build slots instead of one

Founder decision, 2026-10-25: at most two native builds per machine.

On this 36 GB M4 Max (14 cores), the single exclusive build lock queued every
agent behind one tui compile. That compile is a ~965k-line crate whose front
end keeps most cores idle; its unit-test rustc peaks near 8 GB
(BUILD_PERFORMANCE.md). Today, sessions waited 15 to 30 minutes behind
each other's tui builds while the machine was mostly idle.

scripts/build-lock.py now holds one of CODEWHALE_BUILD_SLOTS (default 2)
exclusive flocks:
- `build.lock` is slot 0, so an older single-slot copy of the script, or a
  CI runner pointed at the same file, still excludes the build in that slot.
- `build.lock.1` and up are the extra slots.
- A build beyond the slot count prints every holder and polls once a second.
- CODEWHALE_BUILD_SLOTS=1 restores the old behaviour, and a non-integer
  value exits 2 with an error.

docs/BUILD_PERFORMANCE.md's "One build per machine" section is updated to
match. The workspace AGENTS.md (operator file, outside any repo) and
codewhale-app/AGENTS.md now say "at most two".

Verified with a scratch lock file and three concurrent 3-second commands:
- Jobs 1 and 2 ran together and finished at 3s and 4s.
- Job 3 printed both holders, acquired a slot after 3s, and finished at 7s.
- CODEWHALE_BUILD_SLOTS=1 ran.
- CODEWHALE_BUILD_SLOTS=x exited 2.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs/BUILD_PERFORMANCE.md

@@ -392,15 +392,20 @@ TUI-DOG-017) — left as they are.
 6. Then `fleet/`, `tools/`, `core/engine` — each behind the crate boundary
    its tests already respect, measured with the A0 table.
 
-## One build per machine
+## Two builds per machine
 
-`scripts/dev-cargo.sh` and `scripts/dev-test.sh` hold an exclusive machine-wide
-build lock (`<cache root>/build.lock`, via `scripts/build-lock.py`) for the
-whole Cargo invocation. Cargo's own lock is per target directory, so two
-agents building into different target dirs still ran concurrently and exhausted
-memory. A second build waits and prints who holds the lock. Set
-`CODEWHALE_BUILD_LOCK=0` to skip it, or `CODEWHALE_BUILD_LOCK_FILE` to name the
-lock file. A self-hosted CI runner on the same machine joins the lock when its
+`scripts/dev-cargo.sh` and `scripts/dev-test.sh` hold one of
+`CODEWHALE_BUILD_SLOTS` (default 2) machine-wide build slots
+(`<cache root>/build.lock`, then `build.lock.1`, ..., via
+`scripts/build-lock.py`) for the whole Cargo invocation. Cargo's own lock is
+per target directory, so agents building into different target dirs once ran
+unbounded and exhausted memory. Two slots is the founder's 2026-10-25 setting
+for a 36 GB M4 Max: the tui unit-test rustc peaks near 8 GB, and one huge
+crate keeps most cores idle during its front end, so a second build roughly
+doubles throughput. A build beyond the slot count waits and prints every
+holder. Set `CODEWHALE_BUILD_SLOTS=1` on a smaller machine,
+`CODEWHALE_BUILD_LOCK=0` to skip the lock, or `CODEWHALE_BUILD_LOCK_FILE` to
+name the slot-0 lock file. A self-hosted CI runner on the same machine joins the lock when its
 `.env` sets `CODEWHALE_BUILD_LOCK_FILE` to the same path: the macOS Test job
 then holds it from the first test build to the end of the job. The lock is advisory: Cargo started
 directly, outside these scripts, does not take it, and on platforms without

scripts/build-lock.py

@@ -1,13 +1,15 @@
 #!/usr/bin/env python3
-"""Run one command while holding this machine's Codewhale build lock.
+"""Run one command while holding one of this machine's Codewhale build slots.
 
     scripts/build-lock.py <lock-file> -- <command> [args...]
 
-Several agents share one checkout and one memory-constrained machine. Cargo's
-own lock is per target directory, so builds into different target dirs still
-run concurrently and exhaust memory. This holds an exclusive flock on a lock
-file under the persistent cache root for the lifetime of the command, and
-says who holds it while waiting.
+Several agents share one checkout and one machine. Cargo's own lock is per
+target directory, so builds into different target dirs still run concurrently
+and can exhaust memory. This holds an exclusive flock on one of
+CODEWHALE_BUILD_SLOTS lock files (default 2: `<lock-file>`, `<lock-file>.1`,
+...) under the persistent cache root for the lifetime of the command, and says
+who holds every slot while waiting. Slot 0 is `<lock-file>` itself, so an older
+single-slot copy of this script still excludes the build in that slot.
 
 The command runs as a child, never via exec: a daemon it spawns (an sccache
 server, say) must not inherit the descriptor and pin the lock after the build
@@ -42,25 +44,50 @@ def main() -> int:
         print("build-lock: fcntl unavailable; running without the machine build lock", file=sys.stderr)
         return subprocess.call(command, env=env)
 
-    os.makedirs(os.path.dirname(lock_path) or ".", exist_ok=True)
-    fd = os.open(lock_path, os.O_RDWR | os.O_CREAT, 0o600)
     try:
-        fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
-    except BlockingIOError:
-        holder = os.pread(fd, 512, 0).decode(errors="replace").strip() or "holder unknown"
+        slots = max(1, int(os.environ.get("CODEWHALE_BUILD_SLOTS", "2")))
+    except ValueError:
+        print("build-lock: CODEWHALE_BUILD_SLOTS must be a positive integer", file=sys.stderr)
+        return 2
+    os.makedirs(os.path.dirname(lock_path) or ".", exist_ok=True)
+    fds = [
+        os.open(lock_path if slot == 0 else f"{lock_path}.{slot}", os.O_RDWR | os.O_CREAT, 0o600)
+        for slot in range(slots)
+    ]
+
+    def try_slots():
+        for slot_fd in fds:
+            try:
+                fcntl.flock(slot_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+                return slot_fd
+            except BlockingIOError:
+                continue
+        return None
+
+    fd = try_slots()
+    if fd is None:
+        holders = "; ".join(
+            os.pread(slot_fd, 512, 0).decode(errors="replace").strip() or "holder unknown"
+            for slot_fd in fds
+        )
         print(
-            f"build-lock: waiting for another Cargo build on this machine ({holder}); "
+            f"build-lock: all {slots} build slots are busy ({holders}); "
             "set CODEWHALE_BUILD_LOCK=0 to skip",
             file=sys.stderr,
             flush=True,
         )
         started = time.monotonic()
-        fcntl.flock(fd, fcntl.LOCK_EX)
+        while fd is None:
+            time.sleep(1)
+            fd = try_slots()
         print(
             f"build-lock: acquired after {time.monotonic() - started:.0f}s",
             file=sys.stderr,
             flush=True,
         )
+    for slot_fd in fds:
+        if slot_fd != fd:
+            os.close(slot_fd)
     os.ftruncate(fd, 0)
     summary = " ".join(command[:5])
     os.pwrite(fd, f"pid {os.getpid()} in {os.getcwd()}: {summary}\n".encode(), 0)