Files
zero/internal/tools/spill_test.go
Kevin Codex 95ccd5bc32 perf: universal tool-output ceiling with spill + async post-edit diagnostics (#518)
* feat(tools): universal token-based output ceiling with spill-to-disk

Cap every tool result at the registry boundary (16k estimated tokens,
ZERO_TOOL_OUTPUT_CEILING_TOKENS-tunable, 0 disables) unless the tool
manages its own deliberate budget. Head+tail truncation, full output
spilled to the existing redacted spill dir and re-readable via
grep/read_file, so nothing is lost — only deferred.

This closes the three unbounded paths a single call could flood the
context window through (and then re-bill every following turn):
- web_fetch: up to 256KiB default / 2MiB max of non-HTML body
- skill: full SKILL.md returned uncapped
- MCP-served tools: no budget at all

Self-budgeting tools opt out via an unexported marker interface (an MCP
server can never exempt itself): bash, exec_command (model-raisable),
read_file, read_minified_file, grep, glob, list_directory. Browser
snapshots deliberately stay under the net.

bash itself is tightened to match the unified token model: emit budget
96KiB -> 32KiB per stream (~24k -> ~8k tokens worst case per stream),
while capture retention stays at 96KiB head+tail so the new spill file
covers 6x more of the output than the transcript shows.

* perf(agent): collect post-edit LSP diagnostics off the tool-call path

Inline post-edit diagnostics blocked every edit_file/write_file result on
the language server: a >=300ms quiet-period debounce that reset on each
publish, seconds on the first edit per language, capped at 10s — the
largest recurring latency tax in an edit-heavy session.

Diagnostics are now collected in the background: a file is enqueued the
moment a mutating tool finishes, a single worker checks it while the rest
of the batch executes, and the loop drains completed results just before
building the NEXT provider request, appending errors as a user nudge.
The model always issues another request after an edit turn (to read its
tool results), so it still sees introduced errors at the same decision
point — no tool call ever stalls on the language server.

A file re-edited before its check runs is checked once in its final
state (the checker re-reads the file); re-edited after, it is re-queued
and the newer result wins. A drain that finds the worker still busy
gives up after 3s and delivers on a later turn — results stay accurate
because any further edit re-enqueues the file.

The tools-side inline mechanism (tools.RunOptions.Diagnostics) is kept
for direct API callers; the loop just no longer wires it.

* fix: address review — max-turns diagnostics drain, capture-gap marker, test hardening

- Drain background diagnostics before the max-turns final-answer request
  so an error introduced by the LAST turn's edit is reported in the
  summary instead of silently dropped (+ regression test).
- spillBashStreams marks the head/tail junction with a capture-gap
  marker when boundedBuffer dropped the middle of a stream, so a spilled
  log never reads as contiguous when it is not; junction position is
  asserted in the test.
- Tests override TMP/TEMP alongside TMPDIR (os.TempDir reads TMP/TEMP on
  Windows) via a shared setTestTempDir helper.
- TestBudgetBashCaptureReportsTrueTotal now asserts spill_path meta and
  spill file content; exemption-list test gains read_minified_file.

* fix(test): use a real workspace root in async diagnostics tests

filepath.IsAbs("/ws") is false on Windows (no drive letter), so the
collector test's workspace-absolute assertion failed on the windows
smoke job. Root the fake workspace at t.TempDir() — absolute on every
platform — and assert against it.

* fix: pattern-scrub spill files and gate finalization on pending diagnostics

Review follow-ups (Vasanthdev2004):

- Spill files now pass the pattern-based secret scanner (AWS keys,
  GitHub/Slack/OpenAI tokens, PEM blocks, JWTs) in addition to the
  configured-key scrub. A bash spill runs before formatBashOutput's
  scan, so without this a spilled file held in cleartext exactly the
  credentials the transcript redacts. Applied centrally in
  spillTruncatedOutput, covering bash, exec_command, and ceiling spills.

- Finalization diagnostics gate: a no-tool-call final answer now drains
  pending post-edit diagnostics with the full inline-era budget (10s,
  vs the 3s per-turn wait) before the run is accepted; introduced
  errors append the nudge and give the model one more turn instead of
  being silently dropped when no later turn exists. The max-turns
  summary path uses the same finalization budget. Free for runs that
  never edited — an idle collector returns immediately.
2026-07-05 23:50:31 +08:00

110 lines
3.5 KiB
Go

package tools
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestTruncateExecOutputSpillWritesFullOutput(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
long := strings.Repeat("line of build output\n", 5000) // ~100KB > 40KB budget
truncated, wasTruncated := truncateExecOutputSpill(long, defaultMaxOutputTokens, "bash")
if !wasTruncated {
t.Fatal("output over budget must truncate")
}
if !strings.Contains(truncated, "full output saved to ") {
t.Fatalf("truncation notice missing spill path: %q", truncated[:200])
}
// Extract the path and verify the file holds the complete output.
start := strings.Index(truncated, "full output saved to ") + len("full output saved to ")
end := strings.Index(truncated[start:], " (grep")
spillPath := truncated[start : start+end]
content, err := os.ReadFile(spillPath)
if err != nil {
t.Fatalf("spill file unreadable: %v", err)
}
if string(content) != long {
t.Fatalf("spill file must hold the full output: got %d bytes, want %d", len(content), len(long))
}
}
func TestTruncateExecOutputUnderBudgetUnchanged(t *testing.T) {
output, wasTruncated := truncateExecOutputSpill("short output", defaultMaxOutputTokens, "bash")
if wasTruncated || output != "short output" {
t.Fatalf("under-budget output must pass through untouched: %q", output)
}
}
func TestSweepSpillDirRemovesOnlyOldFiles(t *testing.T) {
dir := t.TempDir()
oldFile := filepath.Join(dir, "bash-old.txt")
newFile := filepath.Join(dir, "bash-new.txt")
for _, path := range []string{oldFile, newFile} {
if err := os.WriteFile(path, []byte("x"), 0o600); err != nil {
t.Fatal(err)
}
}
stale := time.Now().Add(-spillRetention - time.Hour)
if err := os.Chtimes(oldFile, stale, stale); err != nil {
t.Fatal(err)
}
sweepSpillDir(dir)
if _, err := os.Stat(oldFile); !os.IsNotExist(err) {
t.Fatal("stale spill file must be removed")
}
if _, err := os.Stat(newFile); err != nil {
t.Fatal("fresh spill file must survive the sweep")
}
}
// A spill happens BEFORE bash's model-facing formatter runs its pattern-based
// secret scan, so the spill itself must scrub pattern-matched credentials —
// otherwise the file would hold in cleartext exactly what the transcript hides.
func TestSpillTruncatedOutputScrubsPatternSecrets(t *testing.T) {
setTestTempDir(t)
githubToken := "ghp_" + strings.Repeat("a", 36)
body := "before\nAKIAIOSFODNN7EXAMPLE\n" + githubToken + "\nafter"
path := spillTruncatedOutput("bash", body)
if path == "" {
t.Fatal("spill must return a file path")
}
content, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(content), "AKIAIOSFODNN7EXAMPLE") {
t.Fatal("AWS access key reached the spill file in cleartext")
}
if strings.Contains(string(content), githubToken) {
t.Fatal("GitHub token reached the spill file in cleartext")
}
if !strings.Contains(string(content), "before") || !strings.Contains(string(content), "after") {
t.Fatalf("non-secret content must survive the scrub: %q", content)
}
}
func TestSpillTruncatedOutputWritesFile(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
path := spillTruncatedOutput("exec_command", "some output body")
if path == "" {
t.Fatal("spill must return a file path")
}
if base := filepath.Base(path); !strings.HasPrefix(base, "exec_command-") {
t.Fatalf("spill file name must carry the tool prefix: %s", base)
}
content, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
if string(content) != "some output body" {
t.Fatalf("unexpected spill content: %q", content)
}
}