Commit Graph
2 Commits
Author SHA1 Message Date
KRATOS fbbfdbf3f2 Harden background daemon: stdout integrity, lifecycle, and single-instance lock (#274)
* harden background daemon: stdout integrity, lifecycle, single-instance lock

Deep review of internal/daemon (worker pool, launcher, control server,
session fan-out, single-instance lock) found and fixes a stdout-integrity
bug plus several lifecycle/locking hazards. Each fix has a regression test;
the daemon suite passes under -race.

Worker stdout integrity
- Replace the capped bufio.Scanner (64 KiB) on a worker's stdout with an
  uncapped bufio.Reader: a legitimately large stream-json line from our own
  trusted worker is no longer silently dropped (the 1 MiB cap belongs to
  untrusted network frames, not this local pipe).
- Surface a stdout read error as a run failure instead of swallowing it as a
  clean success, and kill the worker first so it can't block writing to the
  now-unread pipe and hang Wait.

Control-server lifecycle
- Track open connections and close them on Shutdown so a single idle/hostile
  client can no longer wedge drain (and SIGTERM cleanup) forever.
- Guard the listener with a mutex and re-check the shutdown signal after bind
  so a shutdown requested in the bind window is not lost.
- Honor shutdown during buffered-history replay too.

Session fan-out
- A slow subscriber that drops live lines now receives a namespaced gap
  notice when its channel drains, instead of silently losing output.
- Hold the session lock across the (non-blocking) broadcast sends to close a
  latent send-on-closed-channel panic when a subscriber cancels mid-send.

Single-instance lock
- Reclaim a stale lock with an atomic rename-with-verify instead of a blind
  remove, so two instances starting at once cannot both reclaim and end up
  both holding the lock; a holder that reacquires in the gap is restored.

Client + pool
- Bound the control handshake with an I/O deadline so a peer that never
  replies can't wedge Dial/NewClientConn.
- Key the active-worker map by a monotonic id rather than PID (the OS reuses
  PIDs the instant a worker exits).
- Group-terminate on context cancel so a stuck worker's children are not
  orphaned.

* daemon: add -race regression test for the Session.Line lock-hold invariant

Session.Line holds s.mu across its non-blocking subscriber sends so cancel()/
finish() cannot close a subscriber channel mid-send (send-on-closed-channel
panic). That invariant had no concurrency test. Add one that drives Line
concurrently with Subscribe/cancel and finish; it panics with a data race if
the lock-hold is reverted and passes with it (verified both ways under -race).
2026-06-20 22:21:28 +05:30
KRATOS 135e3c5657 feat(daemon): zero daemon — worker pool + session supervisor over a local control socket (#203)
* feat(daemon): add `zero daemon` — worker pool + session supervisor over a local control socket

Adds internal/daemon + `zero daemon start|stop|status|run|attach`: a long-running
local service that supervises a pool of headless `zero exec` workers and routes
multiple agent sessions to them over an owner-only Unix-domain control socket.
Additive — interactive and one-shot exec are unchanged.

Reuses zero's building blocks:
- internal/background: ConfigureChildProcessGroup (process-group setup) +
  TerminateProcess (cross-platform terminate) for worker kill/drain. The pool
  does its own os/exec and uses background only for those helpers.
- internal/streamjson: the worker speaks stream-json on stdout; the daemon
  forwards those lines to clients.
- internal/cli exec: a worker is `zero exec --output-format stream-json`.

Pieces (reimplemented in Go from a reference daemon):
- protocol.go: framed control codec — 4-byte BE length + 1 kind byte, 1 MiB cap,
  JSON control messages, version negotiation.
- pool.go + backoff.go: bounded worker pool — at-least-once dispatch with
  exponential backoff+jitter retry capped by MaxAttempts, EXIT_TEMPFAIL(75)/
  EXIT_PERMANENT(76) handling, graceful drain + kill-timeout. zero's exec is
  one-shot, so workers are on-demand (one session per worker = the lease), not
  warm long-lived slots (a persistent worker mode is a follow-up).
- session.go: SessionManager — sessionID->worker routing, lease (one active
  request per worker), queue when the pool is full, per-session output buffer for
  attach + metrics.
- server.go + lock.go + socket.go + paths.go: single-instance PID-file lock with
  stale-lock (dead-PID) recovery, status file, owner-only socket, accept loop;
  cleans up lock/socket/status on exit.
- launcher.go: production worker launcher. client.go: control-socket client.

Security model:
- Control socket is LOCAL-ONLY (AF_UNIX loopback file, never a TCP port) and
  owner-only: socket 0600 + parent dir 0700 on POSIX (Windows relies on the
  per-user profile ACL; a dedicated pipe ACL is a follow-up).
- Workers stay sandboxed: the launcher SCRUBS the sandbox re-entrancy markers
  (ZERO_SANDBOXED / ZERO_SANDBOX_BACKEND) from each worker's env so the worker
  re-establishes its OWN sandbox instead of inheriting a pass-through plan, while
  propagating the rest of the policy config/env. The daemon never bypasses the
  sandbox.
- Control frames over the 1 MiB cap are rejected before allocation.

Robustness: a crashed worker never takes down the daemon (backoff+jitter retry,
MaxAttempts cap, EXIT_PERMANENT stops retry); drain gives a grace window then
force-kills stragglers; a stale lock from an unclean exit is reclaimed.

Out of scope (follow-up): remote session-linking; warm pre-spawned persistent
workers; Windows named-pipe ACL.

Tests: protocol round-trip + oversize/unknown-kind rejection; backoff bounds;
pool spawn/retry-backoff/permanent/cap/queue/drain; session lease/route/queue/
attach; lock single-instance + stale recovery; launcher env-scrub (incl. an
end-to-end check that the spawned worker process receives scrubbed env); server
e2e start->run->stream-json->status->attach->stop with a fake worker; CLI
usage/validation/not-running paths. Live-verified start/status/stop (0600 socket,
0700 dir, file cleanup). Platform files behind explicit //go:build tags.

* daemon: address review — help listing, extra-arg rejection, lock write error, bounded sessions

- cli/app.go: list `daemon` in `zero --help` so the command is discoverable.
- cli/daemon: `stop`/`status` reject extra arguments; `attach` rejects more than
  one session id (fail fast on typos instead of silently ignoring tokens).
- daemon/lock: check the PID fmt.Fprintf error — a partial/failed write would
  leave a malformed lock file that another process reads as stale and wrongly
  reclaims, breaking the single-instance guarantee; on write failure remove the
  file and return the error.
- daemon/session: bound the registry with MaxSessions (default 256) — evict the
  oldest FINISHED sessions once past the cap so a long-running daemon doesn't
  accumulate sessions without limit; running/queued sessions are never evicted, so
  attach-after-finish for recent sessions is preserved.

Tests: CLI extra-arg rejection; finished-over-cap eviction + running-never-evicted.
Gates: gofmt, vet, build, test (incl -race), staticcheck (no new), govulncheck (0),
deadcode (no new), GOOS=linux+windows cross-compile.
2026-06-14 22:16:37 +05:30