mirror of
https://github.com/Twigpine/zero.git
synced 2026-11-11 23:40:28 -10:00
* fix(local): defeat the macOS stale-pooled-connection hang (transport + stall-retry)
Root cause (you confirmed it: both chatgpt AND ollama hung, only on macOS): providers
fell back to http.DefaultClient, whose pooled keep-alive connections get reused on a
later turn after the server/NAT silently dropped them. The model call is a POST
(non-idempotent), so Go won't auto-retry it on a fresh conn — it blocks forever.
macOS keeps dead pooled conns around longer than Linux, so it reproduced on macOS only.
1. providerio.HTTPClient now returns a shared, tuned transport (ResponseHeaderTimeout
60s bounds a hung POST that gets no response; IdleConnTimeout 30s stops dead conns
from lingering to be reused). Routed openai/codex through it too (anthropic/gemini
already used HTTPClient). Slow first tokens / long reasoning are unaffected — the
header timeout only bounds time-to-first-response-header, not the stream body.
2. Agent loop now retries a stream idle/stall timeout that produced NO output yet, on
a fresh connection (capped + backoff). A partial-output turn is NOT retried (would
duplicate). Recovers any stall that slips past the transport timeout.
Local only — not pushed.
* fix(local): tolerate concatenated tool-arg JSON + widen response-header timeout
1) Weak models (minimax-m3, glm) pack MULTIPLE concatenated top-level JSON objects
into one tool-call arguments slot ({A}{B}), which json.Unmarshal rejects with
'invalid character {' after top-level value' (~9 failed tool calls/session).
New recoverableToolArguments returns the first JSON value ONLY when the payload is
one value optionally followed by more WHOLE JSON values; it rejects a bad/truncated
first value OR trailing NON-JSON garbage ({"x":1}xyz) so genuine corruption still
errors (adversarial-review fix). decodeToolArguments uses it at the central dispatch
(executeToolCall) so {A}{B} runs the first object; historySafeToolCalls uses the
SAME helper so the replayed transcript records the recovered first object (not {}),
keeping dispatch and history consistent. Truly-malformed JSON still hits the
existing sanitize-to-{} + retry path. Trailing objects are dropped (the model
re-issues on a later turn).
2) Raise the shared transport ResponseHeaderTimeout 60s -> 120s so a slow cloud proxy
(ollama *:cloud) that withholds its 200 header until first token isn't wrongly
aborted; still bounds a truly dead reused connection.
Local only — not pushed.
* feat(local): add /turns command + raise default maxTurns 30->50
Agentic multi-step runs (build + delegate + swarm + MCP + self-verify) routinely hit
the 30-turn ceiling mid-task and never reached the later steps; removing /mode earlier
also left NO interactive way to raise the budget.
- /turns [n]: per-session tool-turn budget setter (commandGroupSession), mirroring
/selfcorrect. Empty/status shows the current budget; a positive int sets
agentOptions.MaxTurns (read by the run at model.go); clamped to maxTurnsCeiling=500
to guard against typos; invalid/non-positive shows usage without changing it.
- defaultMaxTurns 30 -> 50 (matches the old 'deep' preset). Raise per-run with /turns
for long tasks (e.g. /turns 150 for a full acceptance run).
Local only — not pushed.
* fix(local): swarm reports failed members as failed + sub-agents inherit /turns budget
Issue B — a swarm member whose child exited non-zero (e.g. exit 4 / max-turns) was
recorded [done] (swarm_status '0 failed'), so the orchestrator — which can't see the
AGENTS panel — trusted incomplete work. The specialist launcher ignored
res.Result.Status; it now surfaces a StatusError result as a member FAILURE
(errors.New(report)) so the swarm marks it [failed]. Added Coordinator.FailWithSession
and switched the member watch path to it so a failed member keeps its session id
(still drillable) and its report rides along as the failure message.
Issue A — a member hit its OWN turn limit because /turns raised only the parent's
budget; children re-resolved config.json's default. New config.SetMaxTurnsEnv exports
ZERO_MAX_TURNS, read in applyEnv -> cfg.MaxTurns; /turns now exports it so spawned
sub-agents / swarm members (which inherit the env) run with the same budget. A
non-numeric/zero env value never clobbers the configured budget.
Tests: launcher non-zero-exit -> failed (+ session kept); SetMaxTurnsEnv round-trip
+ garbage-env guard; /turns exports + clamps the env. Full gate green (73 pkg).
Local only — not pushed.
* fix(local): corrective error for unknown specialist name + clamp ZERO_MAX_TURNS
1) A model that invents a specialist name (validator-runner, file-creator,
file-writer — none registered) got an opaque 'specialist not found' and kept
spawning doomed sub-agents. loadManifest now lists the available specialists and
notes that 'worker' has shell/edit while explorer/code-review are read-only, so the
model self-corrects to a capable specialist. New availableSpecialistList helper.
2) Defense-in-depth (review nit): applyEnv now clamps an inherited ZERO_MAX_TURNS to
MaxTurnsCeiling, so a raw shell 'export ZERO_MAX_TURNS=999999' can't bypass the
/turns ceiling. Ceiling promoted to config.MaxTurnsCeiling (single source of truth,
referenced by the TUI /turns clamp).
Tests: unknown-name error lists worker/explorer/code-review; applyEnv clamps an
over-ceiling env value. Full gate green (73 pkg).
Local only — not pushed.
* fix(local): surface sub-agent kill signal instead of opaque exit -1
When a sub-agent child is terminated by a signal, command.Wait returns an ExitError
whose ExitCode() is -1, losing the cause. Capture ProcessState.String() (portable,
e.g. "signal: killed") into ChildRunResult.Signal and have BuildFinalResult render
it — "Subagent terminated by a signal (signal: killed) — killed before it finished.
Likely causes: the OS out-of-memory killer ... a timeout, or cancellation." — instead
of an opaque "Subagent failed (exit -1)". The message lists causes rather than
asserting OOM, since timeout/cancellation also SIGKILL the child. A captured signal
always surfaces even if a late run_end reported a clean exit. The background task
path (which can't carry the signal name) emits the same hint for a negative exit code.
* address CodeRabbit review on #349
- loop.go: gate stall-retry on a turn-level forwardedAnything flag (wrapped
callbacks), so a retry never re-streams on top of output already forwarded earlier
in the turn (e.g. across the reactive-compaction retry). New test covers a
reasoning-then-stall (collected.Text empty but output forwarded).
- loop.go: route a reissued stream's non-stall error through the SAME recovery as
the initial stream (image-rejection wrapping / context-limit compaction) via a
shared recoverStreamError closure, instead of returning it raw.
- exec.go: drop the hardcoded worker/explorer/code-review guidance from the
unknown-specialist error; rely on the dynamically-rendered available list (a
custom registry may not have those names).
- streamer.go: kill-signal message lists OOM/timeout/cancellation evenhandedly
instead of asserting OOM (the branch also covers timeouts/cancellations).
- launcher_specialist.go + exec.go: preserve the child SessionID on a post-start
failure too, so FailWithSession keeps those failed members drillable.
- model.go: block /turns while a run is active (mid-run change would make the
inherited ZERO_MAX_TURNS budget inconsistent for later-spawned sub-agents).
307 lines
8.3 KiB
Go
307 lines
8.3 KiB
Go
package specialist
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"os"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/Gitlawb/zero/internal/background"
|
|
"github.com/Gitlawb/zero/internal/sessions"
|
|
"github.com/Gitlawb/zero/internal/streamjson"
|
|
"github.com/Gitlawb/zero/internal/tools"
|
|
)
|
|
|
|
const (
|
|
defaultTaskOutputPollInterval = 500 * time.Millisecond
|
|
defaultTaskOutputTimeout = 30 * time.Second
|
|
maxTaskOutputTimeout = 20 * time.Minute
|
|
)
|
|
|
|
type OutputTool struct {
|
|
manager *background.Manager
|
|
managerFunc BackgroundManagerFunc
|
|
SessionStore *sessions.Store
|
|
PollInterval time.Duration
|
|
DefaultTimeout time.Duration
|
|
MaxTimeout time.Duration
|
|
}
|
|
|
|
type outputParameters struct {
|
|
TaskID string
|
|
Block bool
|
|
TimeoutMS int
|
|
}
|
|
|
|
func NewOutputTool(manager *background.Manager) *OutputTool {
|
|
return &OutputTool{manager: manager}
|
|
}
|
|
|
|
func newOutputToolWithManagerFunc(managerFunc BackgroundManagerFunc, sessionStore *sessions.Store) *OutputTool {
|
|
return &OutputTool{managerFunc: managerFunc, SessionStore: sessionStore}
|
|
}
|
|
|
|
func (tool *OutputTool) Name() string {
|
|
return "TaskOutput"
|
|
}
|
|
|
|
func (tool *OutputTool) Description() string {
|
|
return "Poll the output of a background Zero specialist task."
|
|
}
|
|
|
|
func (tool *OutputTool) Parameters() tools.Schema {
|
|
return tools.Schema{
|
|
Type: "object",
|
|
Properties: map[string]tools.PropertySchema{
|
|
"task_id": {
|
|
Type: "string",
|
|
Description: "Background task id returned by Task.",
|
|
},
|
|
"block": {
|
|
Type: "boolean",
|
|
Description: "Wait for the task to finish before returning, up to timeout.",
|
|
Default: false,
|
|
},
|
|
"timeout": {
|
|
Type: "integer",
|
|
Description: "Maximum wait time in milliseconds when block is true.",
|
|
Default: 30000,
|
|
},
|
|
},
|
|
Required: []string{"task_id"},
|
|
AdditionalProperties: false,
|
|
}
|
|
}
|
|
|
|
func (tool *OutputTool) Safety() tools.Safety {
|
|
return tools.Safety{
|
|
SideEffect: tools.SideEffectRead,
|
|
Permission: tools.PermissionAllow,
|
|
Reason: "Reads a background specialist task output file.",
|
|
}
|
|
}
|
|
|
|
func (tool *OutputTool) Run(ctx context.Context, args map[string]any) tools.Result {
|
|
params, err := parseOutputParameters(args)
|
|
if err != nil {
|
|
return taskError(err)
|
|
}
|
|
manager, err := tool.backgroundManager()
|
|
if err != nil {
|
|
return taskError(err)
|
|
}
|
|
task, ok := manager.Get(params.TaskID)
|
|
if !ok {
|
|
return taskError(fmt.Errorf("background task not found: %s", params.TaskID))
|
|
}
|
|
if !params.Block {
|
|
return tool.readOutput(task)
|
|
}
|
|
return tool.blockOnOutput(ctx, manager, task.ID, params.TimeoutMS)
|
|
}
|
|
|
|
func (tool *OutputTool) blockOnOutput(ctx context.Context, manager *background.Manager, taskID string, timeoutMS int) tools.Result {
|
|
if ctx == nil {
|
|
ctx = context.Background()
|
|
}
|
|
timeout := tool.timeout(timeoutMS)
|
|
timer := time.NewTimer(timeout)
|
|
defer timer.Stop()
|
|
interval := tool.pollInterval()
|
|
ticker := time.NewTicker(interval)
|
|
defer ticker.Stop()
|
|
|
|
for {
|
|
task, ok := manager.Get(taskID)
|
|
if !ok {
|
|
return taskError(fmt.Errorf("background task not found: %s", taskID))
|
|
}
|
|
if task.Status != background.StatusRunning {
|
|
return tool.readOutput(task)
|
|
}
|
|
select {
|
|
case <-ctx.Done():
|
|
return tool.readLatestOutput(manager, taskID)
|
|
case <-timer.C:
|
|
return tool.readLatestOutput(manager, taskID)
|
|
case <-ticker.C:
|
|
}
|
|
}
|
|
}
|
|
|
|
func (tool *OutputTool) readLatestOutput(manager *background.Manager, taskID string) tools.Result {
|
|
task, ok := manager.Get(taskID)
|
|
if !ok {
|
|
return taskError(fmt.Errorf("background task not found: %s", taskID))
|
|
}
|
|
return tool.readOutput(task)
|
|
}
|
|
|
|
func (tool *OutputTool) readOutput(task background.Task) tools.Result {
|
|
data, err := os.ReadFile(task.OutputFile)
|
|
if err != nil {
|
|
return taskError(err)
|
|
}
|
|
dataString := string(data)
|
|
summary, rawLines := summarizeTaskData(dataString, task.ExitCode)
|
|
if task.Status != background.StatusRunning {
|
|
Executor{SessionStore: tool.SessionStore}.recordBackgroundTaskAccounting(task, summary)
|
|
}
|
|
return tools.Result{
|
|
Status: tools.StatusOK,
|
|
Output: formatTaskOutputSummary(task, summary, rawLines),
|
|
Meta: map[string]string{
|
|
"task_id": string(task.ID),
|
|
"status": string(task.Status),
|
|
},
|
|
}
|
|
}
|
|
|
|
func (tool *OutputTool) backgroundManager() (*background.Manager, error) {
|
|
if tool.manager != nil {
|
|
return tool.manager, nil
|
|
}
|
|
if tool.managerFunc != nil {
|
|
return tool.managerFunc()
|
|
}
|
|
return nil, fmt.Errorf("background manager is not configured")
|
|
}
|
|
|
|
func (tool *OutputTool) pollInterval() time.Duration {
|
|
if tool.PollInterval > 0 {
|
|
return tool.PollInterval
|
|
}
|
|
return defaultTaskOutputPollInterval
|
|
}
|
|
|
|
func (tool *OutputTool) timeout(timeoutMS int) time.Duration {
|
|
timeout := time.Duration(timeoutMS) * time.Millisecond
|
|
if timeout <= 0 {
|
|
timeout = tool.DefaultTimeout
|
|
}
|
|
if timeout <= 0 {
|
|
timeout = defaultTaskOutputTimeout
|
|
}
|
|
maxTimeout := tool.MaxTimeout
|
|
if maxTimeout <= 0 {
|
|
maxTimeout = maxTaskOutputTimeout
|
|
}
|
|
if timeout > maxTimeout {
|
|
return maxTimeout
|
|
}
|
|
return timeout
|
|
}
|
|
|
|
func parseOutputParameters(args map[string]any) (outputParameters, error) {
|
|
taskID, err := optionalTaskString(args, "task_id")
|
|
if err != nil {
|
|
return outputParameters{}, err
|
|
}
|
|
block, err := optionalTaskBool(args, "block")
|
|
if err != nil {
|
|
return outputParameters{}, err
|
|
}
|
|
timeout, err := optionalTaskInt(args, "timeout")
|
|
if err != nil {
|
|
return outputParameters{}, err
|
|
}
|
|
params := outputParameters{
|
|
TaskID: strings.TrimSpace(taskID),
|
|
Block: block,
|
|
TimeoutMS: timeout,
|
|
}
|
|
if params.TaskID == "" {
|
|
return outputParameters{}, fmt.Errorf("task output requires task_id")
|
|
}
|
|
return params, nil
|
|
}
|
|
|
|
func optionalTaskInt(args map[string]any, key string) (int, error) {
|
|
if args == nil {
|
|
return 0, nil
|
|
}
|
|
value, ok := args[key]
|
|
if !ok || value == nil {
|
|
return 0, nil
|
|
}
|
|
switch number := value.(type) {
|
|
case int:
|
|
return number, nil
|
|
case int64:
|
|
return int(number), nil
|
|
case float64:
|
|
if number != float64(int(number)) {
|
|
return 0, fmt.Errorf("task %s must be an integer", key)
|
|
}
|
|
return int(number), nil
|
|
default:
|
|
return 0, fmt.Errorf("task %s must be an integer", key)
|
|
}
|
|
}
|
|
|
|
func formatTaskOutputSummary(task background.Task, summary StreamResult, rawLines []string) string {
|
|
var builder strings.Builder
|
|
fmt.Fprintf(&builder, "task_id: %s\n", task.ID)
|
|
fmt.Fprintf(&builder, "status: %s\n", task.Status)
|
|
if task.SpecialistName != "" {
|
|
fmt.Fprintf(&builder, "specialist: %s\n", task.SpecialistName)
|
|
}
|
|
if task.Description != "" {
|
|
fmt.Fprintf(&builder, "description: %s\n", task.Description)
|
|
}
|
|
if task.PID > 0 {
|
|
fmt.Fprintf(&builder, "pid: %d\n", task.PID)
|
|
}
|
|
if !task.StartedAt.IsZero() {
|
|
fmt.Fprintf(&builder, "started_at: %s\n", task.StartedAt.Format(time.RFC3339))
|
|
}
|
|
if !task.CompletedAt.IsZero() {
|
|
fmt.Fprintf(&builder, "completed_at: %s\n", task.CompletedAt.Format(time.RFC3339))
|
|
fmt.Fprintf(&builder, "exit_code: %d\n", task.ExitCode)
|
|
if task.ExitCode < 0 {
|
|
// A negative exit code means the background child was terminated by a
|
|
// signal rather than exiting (the int-only launcher can't carry the
|
|
// signal name). Give the same actionable hint the foreground path does.
|
|
builder.WriteString("note: terminated by a signal — killed before it finished. Likely causes: the OS out-of-memory killer (common when many sub-agents run in parallel — try fewer), a timeout, or cancellation.\n")
|
|
}
|
|
}
|
|
|
|
if summary.Text != "" {
|
|
fmt.Fprintf(&builder, "\noutput:\n%s\n", summary.Text)
|
|
}
|
|
if len(summary.Tools) > 0 {
|
|
fmt.Fprintf(&builder, "\ntools: %s\n", strings.Join(summary.Tools, ", "))
|
|
}
|
|
if len(summary.Errors) > 0 {
|
|
fmt.Fprintf(&builder, "\nerrors: %s\n", strings.Join(summary.Errors, "; "))
|
|
}
|
|
if len(rawLines) > 0 {
|
|
fmt.Fprintf(&builder, "\nraw:\n%s\n", strings.Join(rawLines, "\n"))
|
|
}
|
|
if summary.Text == "" && len(summary.Tools) == 0 && len(summary.Errors) == 0 && len(rawLines) == 0 {
|
|
builder.WriteString("\noutput: <empty>\n")
|
|
}
|
|
return strings.TrimSpace(builder.String())
|
|
}
|
|
|
|
func summarizeTaskData(data string, exitCode int) (StreamResult, []string) {
|
|
events := []streamjson.Event{}
|
|
rawLines := []string{}
|
|
for _, line := range strings.Split(data, "\n") {
|
|
rawLine := strings.TrimSuffix(line, "\r")
|
|
trimmed := strings.TrimSpace(rawLine)
|
|
if trimmed == "" {
|
|
continue
|
|
}
|
|
var event streamjson.Event
|
|
if err := json.Unmarshal([]byte(trimmed), &event); err != nil || event.Type == "" {
|
|
rawLines = append(rawLines, rawLine)
|
|
continue
|
|
}
|
|
events = append(events, event)
|
|
}
|
|
return SummarizeStream(events, exitCode), rawLines
|
|
}
|