Files
zero/internal/swarm/coordinator.go
KRATOS 7fb0092de7 fix: agent runtime resilience — stream stalls, weak-model tool args, and sub-agent reliability (#349)
* fix(local): defeat the macOS stale-pooled-connection hang (transport + stall-retry)

Root cause (you confirmed it: both chatgpt AND ollama hung, only on macOS): providers
fell back to http.DefaultClient, whose pooled keep-alive connections get reused on a
later turn after the server/NAT silently dropped them. The model call is a POST
(non-idempotent), so Go won't auto-retry it on a fresh conn — it blocks forever.
macOS keeps dead pooled conns around longer than Linux, so it reproduced on macOS only.

1. providerio.HTTPClient now returns a shared, tuned transport (ResponseHeaderTimeout
   60s bounds a hung POST that gets no response; IdleConnTimeout 30s stops dead conns
   from lingering to be reused). Routed openai/codex through it too (anthropic/gemini
   already used HTTPClient). Slow first tokens / long reasoning are unaffected — the
   header timeout only bounds time-to-first-response-header, not the stream body.
2. Agent loop now retries a stream idle/stall timeout that produced NO output yet, on
   a fresh connection (capped + backoff). A partial-output turn is NOT retried (would
   duplicate). Recovers any stall that slips past the transport timeout.

Local only — not pushed.

* fix(local): tolerate concatenated tool-arg JSON + widen response-header timeout

1) Weak models (minimax-m3, glm) pack MULTIPLE concatenated top-level JSON objects
   into one tool-call arguments slot ({A}{B}), which json.Unmarshal rejects with
   'invalid character {' after top-level value' (~9 failed tool calls/session).
   New recoverableToolArguments returns the first JSON value ONLY when the payload is
   one value optionally followed by more WHOLE JSON values; it rejects a bad/truncated
   first value OR trailing NON-JSON garbage ({"x":1}xyz) so genuine corruption still
   errors (adversarial-review fix). decodeToolArguments uses it at the central dispatch
   (executeToolCall) so {A}{B} runs the first object; historySafeToolCalls uses the
   SAME helper so the replayed transcript records the recovered first object (not {}),
   keeping dispatch and history consistent. Truly-malformed JSON still hits the
   existing sanitize-to-{} + retry path. Trailing objects are dropped (the model
   re-issues on a later turn).

2) Raise the shared transport ResponseHeaderTimeout 60s -> 120s so a slow cloud proxy
   (ollama *:cloud) that withholds its 200 header until first token isn't wrongly
   aborted; still bounds a truly dead reused connection.

Local only — not pushed.

* feat(local): add /turns command + raise default maxTurns 30->50

Agentic multi-step runs (build + delegate + swarm + MCP + self-verify) routinely hit
the 30-turn ceiling mid-task and never reached the later steps; removing /mode earlier
also left NO interactive way to raise the budget.

- /turns [n]: per-session tool-turn budget setter (commandGroupSession), mirroring
  /selfcorrect. Empty/status shows the current budget; a positive int sets
  agentOptions.MaxTurns (read by the run at model.go); clamped to maxTurnsCeiling=500
  to guard against typos; invalid/non-positive shows usage without changing it.
- defaultMaxTurns 30 -> 50 (matches the old 'deep' preset). Raise per-run with /turns
  for long tasks (e.g. /turns 150 for a full acceptance run).

Local only — not pushed.

* fix(local): swarm reports failed members as failed + sub-agents inherit /turns budget

Issue B — a swarm member whose child exited non-zero (e.g. exit 4 / max-turns) was
recorded [done] (swarm_status '0 failed'), so the orchestrator — which can't see the
AGENTS panel — trusted incomplete work. The specialist launcher ignored
res.Result.Status; it now surfaces a StatusError result as a member FAILURE
(errors.New(report)) so the swarm marks it [failed]. Added Coordinator.FailWithSession
and switched the member watch path to it so a failed member keeps its session id
(still drillable) and its report rides along as the failure message.

Issue A — a member hit its OWN turn limit because /turns raised only the parent's
budget; children re-resolved config.json's default. New config.SetMaxTurnsEnv exports
ZERO_MAX_TURNS, read in applyEnv -> cfg.MaxTurns; /turns now exports it so spawned
sub-agents / swarm members (which inherit the env) run with the same budget. A
non-numeric/zero env value never clobbers the configured budget.

Tests: launcher non-zero-exit -> failed (+ session kept); SetMaxTurnsEnv round-trip
+ garbage-env guard; /turns exports + clamps the env. Full gate green (73 pkg).

Local only — not pushed.

* fix(local): corrective error for unknown specialist name + clamp ZERO_MAX_TURNS

1) A model that invents a specialist name (validator-runner, file-creator,
   file-writer — none registered) got an opaque 'specialist not found' and kept
   spawning doomed sub-agents. loadManifest now lists the available specialists and
   notes that 'worker' has shell/edit while explorer/code-review are read-only, so the
   model self-corrects to a capable specialist. New availableSpecialistList helper.

2) Defense-in-depth (review nit): applyEnv now clamps an inherited ZERO_MAX_TURNS to
   MaxTurnsCeiling, so a raw shell 'export ZERO_MAX_TURNS=999999' can't bypass the
   /turns ceiling. Ceiling promoted to config.MaxTurnsCeiling (single source of truth,
   referenced by the TUI /turns clamp).

Tests: unknown-name error lists worker/explorer/code-review; applyEnv clamps an
over-ceiling env value. Full gate green (73 pkg).

Local only — not pushed.

* fix(local): surface sub-agent kill signal instead of opaque exit -1

When a sub-agent child is terminated by a signal, command.Wait returns an ExitError
whose ExitCode() is -1, losing the cause. Capture ProcessState.String() (portable,
e.g. "signal: killed") into ChildRunResult.Signal and have BuildFinalResult render
it — "Subagent terminated by a signal (signal: killed) — killed before it finished.
Likely causes: the OS out-of-memory killer ... a timeout, or cancellation." — instead
of an opaque "Subagent failed (exit -1)". The message lists causes rather than
asserting OOM, since timeout/cancellation also SIGKILL the child. A captured signal
always surfaces even if a late run_end reported a clean exit. The background task
path (which can't carry the signal name) emits the same hint for a negative exit code.

* address CodeRabbit review on #349

- loop.go: gate stall-retry on a turn-level forwardedAnything flag (wrapped
  callbacks), so a retry never re-streams on top of output already forwarded earlier
  in the turn (e.g. across the reactive-compaction retry). New test covers a
  reasoning-then-stall (collected.Text empty but output forwarded).
- loop.go: route a reissued stream's non-stall error through the SAME recovery as
  the initial stream (image-rejection wrapping / context-limit compaction) via a
  shared recoverStreamError closure, instead of returning it raw.
- exec.go: drop the hardcoded worker/explorer/code-review guidance from the
  unknown-specialist error; rely on the dynamically-rendered available list (a
  custom registry may not have those names).
- streamer.go: kill-signal message lists OOM/timeout/cancellation evenhandedly
  instead of asserting OOM (the branch also covers timeouts/cancellations).
- launcher_specialist.go + exec.go: preserve the child SessionID on a post-start
  failure too, so FailWithSession keeps those failed members drillable.
- model.go: block /turns while a run is active (mid-run change would make the
  inherited ZERO_MAX_TURNS budget inconsistent for later-spawned sub-agents).
2026-07-01 09:15:08 +05:30

343 lines
9.8 KiB
Go

package swarm
import (
"context"
"errors"
"fmt"
"sort"
"sync"
"time"
)
// TaskStatus is the lifecycle state of a swarm task.
type TaskStatus string
const (
// StatusPending: registered but the member has not started.
StatusPending TaskStatus = "pending"
// StatusRunning: the member is executing.
StatusRunning TaskStatus = "running"
// StatusDone: the member finished successfully.
StatusDone TaskStatus = "done"
// StatusFailed: the member exited with an error.
StatusFailed TaskStatus = "failed"
// StatusHandedOff: ownership was transferred to another agent.
StatusHandedOff TaskStatus = "handed-off"
)
// terminal reports whether a status is final (no further transitions expected).
func (s TaskStatus) terminal() bool {
return s == StatusDone || s == StatusFailed || s == StatusHandedOff
}
// valid reports whether a status is one of the known lifecycle states. Unknown
// statuses are rejected (fail closed) so a task can never enter an undefined
// state that Summarize and the lifecycle logic cannot reason about.
func (s TaskStatus) valid() bool {
switch s {
case StatusPending, StatusRunning, StatusDone, StatusFailed, StatusHandedOff:
return true
default:
return false
}
}
// Task is one unit of work tracked by the coordinator. It is returned by value
// from snapshots so callers cannot mutate coordinator state without the lock.
type Task struct {
ID string
AgentID string
Team string
Description string
Status TaskStatus
Result string
Err string
// SessionID is the member's durable child session id, recorded on completion
// so the TUI can drill into the member's conversation. Empty until done.
SessionID string
CreatedAt time.Time
UpdatedAt time.Time
}
// agentColors palette for stable per-agent coloring in the TUI/status. Provider
// -neutral ANSI-ish names; the renderer maps them to styles.
var agentColors = []string{"cyan", "magenta", "green", "yellow", "blue", "red"}
// Coordinator is the in-memory task registry + team color assigner shared by an
// orchestrator and its members. It is safe for concurrent use.
type Coordinator struct {
mu sync.RWMutex
tasks map[string]*Task
colors map[string]string // agentID -> color
colorIndex int
now func() time.Time // injectable clock for tests
// changed is closed (and replaced) on every task state change so WaitSettled
// can block for a transition without polling. Always non-nil after construction.
changed chan struct{}
}
// NewCoordinator returns an empty coordinator using the wall clock.
func NewCoordinator() *Coordinator {
return &Coordinator{
tasks: map[string]*Task{},
colors: map[string]string{},
now: time.Now,
changed: make(chan struct{}),
}
}
// notifyChangeLocked wakes every WaitSettled caller by closing the current change
// channel and installing a fresh one. Must be called while holding c.mu for
// writing; lazily initialises changed so a zero-value coordinator is still safe.
func (c *Coordinator) notifyChangeLocked() {
if c.changed != nil {
close(c.changed)
}
c.changed = make(chan struct{})
}
// ErrTaskExists is returned when registering a duplicate task ID.
var ErrTaskExists = errors.New("swarm: task already registered")
// ErrUnknownTask is returned when updating/removing a missing task ID.
var ErrUnknownTask = errors.New("swarm: unknown task")
// Register adds a new pending task. The ID must be unique and non-empty.
func (c *Coordinator) Register(id, agentID, team, description string) (Task, error) {
if id == "" {
return Task{}, errors.New("swarm: task id is required")
}
c.mu.Lock()
defer c.mu.Unlock()
if _, ok := c.tasks[id]; ok {
return Task{}, fmt.Errorf("%w: %s", ErrTaskExists, id)
}
now := c.now()
t := &Task{
ID: id,
AgentID: agentID,
Team: team,
Description: description,
Status: StatusPending,
CreatedAt: now,
UpdatedAt: now,
}
c.tasks[id] = t
c.assignColorLocked(agentID)
c.notifyChangeLocked()
return *t, nil
}
// SetStatus transitions a task. Transitions out of a terminal state are
// rejected (fail closed) so a late member update cannot resurrect a finished
// task.
func (c *Coordinator) SetStatus(id string, status TaskStatus) error {
if !status.valid() {
return fmt.Errorf("swarm: invalid task status %q", status)
}
c.mu.Lock()
defer c.mu.Unlock()
t, ok := c.tasks[id]
if !ok {
return fmt.Errorf("%w: %s", ErrUnknownTask, id)
}
if t.Status.terminal() && t.Status != status {
return fmt.Errorf("swarm: task %s is %s (terminal); cannot move to %s", id, t.Status, status)
}
t.Status = status
t.UpdatedAt = c.now()
c.notifyChangeLocked()
return nil
}
// Complete marks a task done with its result.
func (c *Coordinator) Complete(id, result string) error {
return c.finish(id, StatusDone, result, "", "")
}
// CompleteWithSession marks a task done with its result and the member's durable
// child session id (so the TUI can drill into the member's conversation).
func (c *Coordinator) CompleteWithSession(id, result, sessionID string) error {
return c.finish(id, StatusDone, result, "", sessionID)
}
// Fail marks a task failed with an error message.
func (c *Coordinator) Fail(id, errMsg string) error {
return c.finish(id, StatusFailed, "", errMsg, "")
}
// FailWithSession marks a task failed but keeps its child session id, so a member
// that ran and failed (e.g. exited non-zero / hit max-turns) is still drillable.
func (c *Coordinator) FailWithSession(id, errMsg, sessionID string) error {
return c.finish(id, StatusFailed, "", errMsg, sessionID)
}
func (c *Coordinator) finish(id string, status TaskStatus, result, errMsg, sessionID string) error {
c.mu.Lock()
defer c.mu.Unlock()
t, ok := c.tasks[id]
if !ok {
return fmt.Errorf("%w: %s", ErrUnknownTask, id)
}
if t.Status.terminal() {
return fmt.Errorf("swarm: task %s already %s", id, t.Status)
}
t.Status = status
t.Result = result
t.Err = errMsg
if sessionID != "" {
t.SessionID = sessionID
}
t.UpdatedAt = c.now()
c.notifyChangeLocked()
return nil
}
// Reassign transfers a task to a new agent (handoff/orphan-adoption). The task
// returns to pending under the new owner unless it is already terminal.
func (c *Coordinator) Reassign(id, newAgentID string) error {
c.mu.Lock()
defer c.mu.Unlock()
t, ok := c.tasks[id]
if !ok {
return fmt.Errorf("%w: %s", ErrUnknownTask, id)
}
if t.Status.terminal() {
return fmt.Errorf("swarm: task %s already %s; cannot reassign", id, t.Status)
}
t.AgentID = newAgentID
t.Status = StatusPending
t.UpdatedAt = c.now()
c.assignColorLocked(newAgentID)
c.notifyChangeLocked()
return nil
}
// Get returns a snapshot of one task.
func (c *Coordinator) Get(id string) (Task, bool) {
c.mu.RLock()
defer c.mu.RUnlock()
t, ok := c.tasks[id]
if !ok {
return Task{}, false
}
return *t, true
}
// List returns snapshots of all tasks, ordered by creation time then ID for
// stable output.
func (c *Coordinator) List() []Task {
c.mu.RLock()
out := make([]Task, 0, len(c.tasks))
for _, t := range c.tasks {
out = append(out, *t)
}
c.mu.RUnlock()
sortTasksByCreation(out)
return out
}
// sortTasksByCreation orders tasks by creation time then ID for stable output.
func sortTasksByCreation(tasks []Task) {
sort.Slice(tasks, func(i, j int) bool {
if tasks[i].CreatedAt.Equal(tasks[j].CreatedAt) {
return tasks[i].ID < tasks[j].ID
}
return tasks[i].CreatedAt.Before(tasks[j].CreatedAt)
})
}
// WaitSettled blocks until every task in the scope (one team, or all teams when
// team=="") has reached a terminal status, then returns a snapshot of the scope.
// It also returns — with the latest partial state — as soon as ctx is done, so a
// stuck or long-running member can't hang the caller forever (the caller bounds
// it with a timeout / user-cancel context). A scope with no tasks is settled
// immediately, so an empty or unknown team never blocks. This is the primitive
// that lets swarm_collect deliver final results in one call instead of forcing
// the orchestrator to poll swarm_status in a loop.
//
// The settled decision and the returned snapshot are taken in the SAME locked
// pass, so a concurrent Register/Reassign slipping in between cannot make the
// call return a non-terminal task it never actually waited on.
func (c *Coordinator) WaitSettled(ctx context.Context, team string) []Task {
for {
c.mu.RLock()
settled := true
snapshot := make([]Task, 0, len(c.tasks))
for _, t := range c.tasks {
if team != "" && t.Team != team {
continue
}
snapshot = append(snapshot, *t)
if !t.Status.terminal() {
settled = false
}
}
ch := c.changed
c.mu.RUnlock()
if settled || ch == nil {
sortTasksByCreation(snapshot)
return snapshot
}
select {
case <-ctx.Done():
sortTasksByCreation(snapshot)
return snapshot
case <-ch:
}
}
}
// Color returns the stable color assigned to an agent (assigning one on first
// use), so status output colors each member consistently.
func (c *Coordinator) Color(agentID string) string {
c.mu.Lock()
defer c.mu.Unlock()
return c.assignColorLocked(agentID)
}
func (c *Coordinator) assignColorLocked(agentID string) string {
if agentID == "" {
return ""
}
if color, ok := c.colors[agentID]; ok {
return color
}
color := agentColors[c.colorIndex%len(agentColors)]
c.colors[agentID] = color
c.colorIndex++
return color
}
// Summary is an aggregate count of tasks by status for the status tool.
type Summary struct {
Total int
Pending int
Running int
Done int
Failed int
HandedOff int
}
// Summarize aggregates current task counts.
func (c *Coordinator) Summarize() Summary {
c.mu.RLock()
defer c.mu.RUnlock()
s := Summary{Total: len(c.tasks)}
for _, t := range c.tasks {
switch t.Status {
case StatusPending:
s.Pending++
case StatusRunning:
s.Running++
case StatusDone:
s.Done++
case StatusFailed:
s.Failed++
case StatusHandedOff:
s.HandedOff++
}
}
return s
}