Compare commits

..
Author SHA1 Message Date
anandh8x a3b09218aa Add deterministic completion policy
Extract completion decisions from the agent loop into a feature-gated typed policy with complete, incomplete, and uncertain outcomes. Preserve bounded plan-stall nudges and allow at most one semantic acceptance check for self-correcting runs.

Tested: make build

Tested: make test

Tested: go fmt ./...

Tested: go vet ./...

Tested: make lint

Tested: govulncheck ./...

Note: repository-wide pinned golangci-lint reports 36 pre-existing unrelated findings; internal/agent/... reports 0 issues.
2026-07-18 00:21:22 +05:30
KRATOSandKRATOS e5670c427f perf(output): add token-aware semantic output budgeting (PR11) (#717)
* perf(output): add token-aware budget contract

* perf(output): add semantic retention policies

* perf(output): budget results after redaction

* perf(output): propagate tool truncation metadata

* perf(output): trace semantic budget decisions

* perf(output): classify core tool output

* docs(output): describe semantic output budgeting

* test(output): make spill fallback portable on windows

* fix(output): address semantic budgeting review findings

* test(output): make hook budgeting tests portable

* fix(output): retain oversized single-line file content

* fix(output): apply semantic budgets to captured output

* fix(output): preserve hook budgets and search paths

* fix(output): preserve alias and truncation metadata

* test(trace): validate output budget metadata

---------

Co-authored-by: KRATOS <[email protected]>
2026-07-18 00:02:43 +05:30
KRATOSandKRATOS 31d45d5f14 perf(agent): concurrent read-only tool batches via capability gate (#715)
Upgrade the parallel tool planner from SideEffectRead probes to the PR5
CapabilitiesOf contract: EffectReadOnly + ThreadSafe + auto-allowed, with
resource-key conflict boundaries so same-path reads stay sequential.

Mark audited pure reads ThreadSafe (read_file, read_minified_file,
list_directory, glob, grep, skill, web_fetch). Glob/grep use scopedScan
directory keys (nil for workspace-wide "." so scans do not false-conflict).

Tests cover the capability gate, key conflicts, extendParallelRun windows,
and catalog ThreadSafe audit for concurrent-safe reads.

Co-authored-by: KRATOS <[email protected]>
2026-07-17 17:52:11 +05:30
Vasanth T 727ad4d321 perf(turn-bench): Phase 0 — strengthen oracles so pass rate can't be misread as correctness (#712)
* perf(turn-bench): Phase 0 — strengthen oracles so pass rate can't be misread as correctness

The baseline manifest's pass/fail contract had three holes that let a weak
or no-op agent read as a pass:

  - edit grep oracles were substring checks: a reworded line (e.g.
    fmt.Println("debug: starting") -> fmt.Println("starting up")) passed
    `! grep 'debug: starting'`.
  - refactor lived in the build tier with a non-positive `go build ./...`,
    so a no-op refactor passed.
  - the 24 read-only tasks (nav/longproc/longctx/parallel) had no oracle;
    a zero-exit only proved the turn ran.

This makes the tiers honest:

  - edit/refactor now carry a stamped oracle_test.go compiled by
    `go test ./...` — the Go compiler is the structural verifier, so a
    no-op refactor, a missing field, or a reworded-but-not-removed line
    fails to compile/run. The test is stamped into the fixture copy AFTER
    the agent run, so it can't interfere with the agent's own build or be
    pre-seen.
  - nav gains an answer-oracle: the harness captures the agent's final
    answer (.zero-answer.txt) and greps it for determinable facts. Count
    tasks (nav-04/05/08) use a structured `count: N` token so the oracle is
    tight rather than a loose prose-contains on "0".
  - refactor graduates from build to correctness (its oracle is now
    positive); nav graduates from latency-only to correctness. The build
    tier is empty (buildOnlyClasses: []).
  - longproc/longctx/parallel stay permanently latency-only — there is no
    deterministic oracle for "report whether the build succeeds" or
    "summarize a large file"; documented in MANIFEST.md.

Tier counts: correctness 34 (edit 10 + fix 8 + nav 10 + refactor 6),
build 0, latency 14. Total 48 unchanged.

TurnSchemaVersion 2 -> 3: the result shape is unchanged (only class-list
membership shifts) but correctnessPassRate now means something strictly
stronger, so the bump prevents a v2->v2 cross-version comparison from
misreading the jump as a model improvement.

Prerequisite fix: the mutating fixtures had no go.mod, so `go test ./...`
in the temp copy failed with "cannot find main module". Added go.mod per
fixture (edit/fix/refactor/nav) and moved copyFixture's temp dir one level
below the system temp root — Go ignores go.mod in direct children of the
temp dir (a hijack guard), which would have broken every compiler-backed
oracle. This also fixes the pre-existing fix/refactor oracles, which were
silently broken because the test suite used fake runners that never ran
go test.

Tests: stub-binary gating tests prove each oracle fails the wrong thing
(no-op refactor, missing field, wrong nav answer, reworded debug print) and
passes the right thing (real refactor, added field, correct answer); an
edit-01 test proves the scoped negative grep doesn't false-fail a correct
rename that leaves the doc comment. Aggregation, schema, and unit tests
cover the v3 tier model and the answer-capture helper.

* Address CodeRabbit review on PR #712: tighten oracles + isolation

- nav count oracles (nav-01/04/05/08): anchored line-exact
  `^count: N$` with the fixture's exact expected count, so a substring
  like "the count: 0" or a wrong count like "count: 3" fails instead
  of rubber-stamping. nav-01 added to the structured-count contract.
- stampOracleAndAnswer now errors when a task has an oracle or
  verification but no workspace fixture, so verification can't run in
  the caller's cwd with the oracle never compiled (false-pass closed).
- copyFixture uses a unique 0700 os.MkdirTemp parent under the temp
  root and nests the copy beneath it (grandchild of temp root), so the
  Go go.mod hijack guard is satisfied and cleanup removes the whole
  parent. Avoids a predictable shared world-readable dir.
- Rejection tests require a non-empty VerifyErr via a shared
  assertVerifyFailed helper, proving the failure came from
  verification rather than a harness error or a silent non-pass.
- Unify streamJSONExitCode + streamJSONFinalText behind one
  parseStreamJSON parser (behavior-preserving; scanner limits and
  malformed-event handling now live in one place).

* Address gnanam1990 review on PR #712: close count=0 gameability + stronger oracles

- nav count tasks (nav-04/05/08) had a count=0 ground truth, so an agent
  that never opened the workspace and always emitted "count: 0" passed
  three correctness tasks -- the exact pass-rate-misread-as-correctness
  failure this PR closes. nav-04 now has a real TestGreet (count 1) and
  nav-05 a real TODO (count 1); nav-01's file count moves to 5. nav-08
  stays at 0 (a real third-party import would need a network fetch and
  break the offline suite) but its oracle now requires the answer to name
  both stdlib imports (fmt from main.go, testing from main_test.go), so a
  blind count:0 or a main.go-only read fails. nav-08's prompt now asks the
  agent to enumerate and classify every import, so naming the stdlib
  imports is part of a correct answer, not a false-fail.
- refactor-05: stamp a behavioral oracle (TestGreetWrapped) so deleting
  both Wrapper and greetWrapped, or stubbing the caller to return "",
  fails to compile/run. The negative grep alone passed that wrong edit.
- refactor-04: replace the brittle exact-text shell grep with a stamped
  file-reading oracle using a spacing-tolerant regex, so a bare
  map[string]int decl under any spacing/:= variant can't dodge it, and
  the Stats type must be referenced in main.go.
- edit-05: also assert the "debug: starting" message text is absent, so
  a respell onto fmt.Printf / backticks / log.Println / println that
  still prints the message fails (the fmt.Println(" check missed those).
- v3 doc comments on ClassSummary / TurnBenchResult /
  FormatTurnBenchSummary now describe the v3 tiers (correctness =
  edit/fix/nav/refactor, build empty, latency = longproc/longctx/
  parallel) instead of the stale v2 model.
- stampOracleAndAnswer failure now mutates the existing outcome instead
  of returning a fresh one, so the already-parsed trace and trace issue
  survive a stamp error (attribution is preserved, pass/fail unchanged).
- Schema comment provenance: "PR #701" -> neutral "Phase 0 oracle
  hardening" (this PR is #712).
- New gating + satisfiable stub tests for refactor-04/05 and edit-05, and
  count-oracle zero-guess / inspection-proof tests for nav-04/05/08.

* Fix macOS CI: copyFixture isolation test was slash-sensitive

TestCopyFixtureIsolatesSourceFromMutation compared filepath.Dir(parent)
against os.TempDir() verbatim. On macOS $TMPDIR ends in a trailing slash
("/var/folders/.../T/") while filepath.Dir strips it, so the equality
false-failed even though the parent is genuinely a direct child of the
temp root. Linux (/tmp, no slash) and Windows (no slash in this Go
build) matched by accident.

The production copyFixture nesting is correct on all platforms (the
grandchild copy sits one level below the temp root, satisfying Go's
go.mod hijack guard); only the assertion was slash-sensitive. Normalize
both sides with filepath.Clean.

* Close nav-04/05 blind-count-one gameability (anandh8x review)

nav-04 and nav-05 had a count-only oracle (anchored '^count:1$'). Once the
ground truth became 1, an agent that never inspects the fixture and always
emits 'count: 1' passed both — the blind-count-one surface the prior
non-zero-ground-truth fix opened. Pair the count with a workspace-derived
named fact that proves inspection, mirroring nav-08's fmt+testing:

- nav-04 now also requires 'TestGreet' (the one test function in
  main_test.go) — a specific, non-generic name a blind agent can't produce
  without reading main.go to learn a greet function exists. Fully closes
  the blind-count-one surface.
- nav-05 now also requires 'main.go' (the file holding the one TODO).
  main.go is the conventional entrypoint and the most guessable filename, so
  this reduces rather than fully closes the surface — a blind 'count: 1, in
  main.go' can still pass. Accepted for Phase 0: the blind floor drops below
  100% (a bare 'count: 1' fails), so a model still moves the pass rate by
  inspecting. MANIFEST.md documents this residual honestly.

Two gating tests per task: RejectsBlindCountOne (bare 'count: 1' now fails)
and, for nav-05, RejectsSubstring (main.go present but unanchored count
still fails — parity with nav-04). Existing accept tests still pass (their
answers already name TestGreet / main.go).

Doc nits also fixed: MANIFEST.md #701 -> #712; refactor-05 removed from the
'no oracleTest' inventory (it now has TestGreetWrapped); the nav-01 named-
fact prose corrected to the three files the oracle actually anchors on;
test comment #701 -> #712.
2026-07-17 17:32:45 +05:30
50 changed files with 4198 additions and 372 deletions

No files matched your search

+16 -3
View File
@@ -272,7 +272,8 @@ sequenceDiagram
Agent->>Tool: execute(ctx, args)
Tool-->>Agent: output, error, metadata
Agent->>Hooks: afterTool hooks
Agent->>Agent: redact, truncate, classify success/failure
Agent->>Agent: redact secrets, apply semantic output budget, enforce byte ceiling
Agent->>Agent: classify success/failure
Agent-->>Surface: OnToolResult callback
Agent->>Transcript: append tool result message
end
@@ -604,10 +605,22 @@ flowchart TD
Prompt -- Yes --> Decision[Permission callback]
Prompt -- No --> Run[Registry.RunWithOptions]
Decision --> Run
Run --> Redact[Redact secrets + enforce output ceiling]
Redact --> Message[Return tool result to model]
Run --> Redact[Redact secrets]
Redact --> Budget[Token-aware semantic output budget]
Budget --> Ceiling[Existing hard byte ceiling]
Ceiling --> Message[Return one tool result to model]
```
Oversized results use deterministic, provider-neutral estimated-token budgets.
Policies retain useful structure for files, search matches, tests, process logs,
diffs, and worker conclusions; tools without a declared category use a UTF-8-safe
head/tail fallback. The estimate is intentionally conservative for non-ASCII
text and is not exact provider tokenization. Existing byte ceilings remain the
authoritative safety limit. When the existing spill mechanism can persist the
complete redacted text received by the budgeting layer, the result includes its
safe spill reference. This does not imply capture of subprocess bytes already
discarded by a tool's established internal buffer.
Core tool groups include:
- **Read-only tools**: file reads, directory listing, glob, grep, LSP navigation,
+6 -1
View File
@@ -141,9 +141,11 @@ func TestRunAskUserCancellationAbortsRun(t *testing.T) {
registry := registryWithAskUser()
args := `{"questions":[{"question":"Which framework?"}]}`
provider := providerCallingAskUserThenAnswer(args, "done")
var toolResults []ToolResult
result, err := Run(context.Background(), "clarify", provider, Options{
Registry: registry,
Registry: registry,
OnToolResult: func(result ToolResult) { toolResults = append(toolResults, result) },
OnAskUser: func(_ context.Context, _ AskUserRequest) (AskUserResponse, error) {
return AskUserResponse{}, context.Canceled
},
@@ -158,6 +160,9 @@ func TestRunAskUserCancellationAbortsRun(t *testing.T) {
if len(provider.requests) != 1 {
t.Fatalf("expected the run to stop after the canceled ask_user (1 turn), got %d", len(provider.requests))
}
if len(toolResults) != 1 || toolResults[0].ToolCallID == "" {
t.Fatalf("cancellation must emit exactly one result for the call, got %#v", toolResults)
}
// The recorded tool result must reflect cancellation, not a synthetic answer.
var toolMsg string
for _, m := range result.Messages {
+86
View File
@@ -0,0 +1,86 @@
package agent
// CompletionDecision is the deterministic outcome of evaluating a text-only
// assistant turn. Uncertain means the loop needs one bounded follow-up action
// before it can finalize the run.
type CompletionDecision string
const (
CompletionUncertain CompletionDecision = "uncertain"
CompletionComplete CompletionDecision = "complete"
CompletionIncomplete CompletionDecision = "incomplete"
)
type completionAction string
const (
completionActionNone completionAction = ""
completionActionContinue completionAction = "continue"
completionActionSemanticCheck completionAction = "semantic_check"
)
type completionEvaluation struct {
Decision CompletionDecision
Action completionAction
Reason string
}
// completionPolicy owns the small amount of state needed to keep uncertain
// decisions bounded. requireSemanticCheck is set only for run profiles that
// already opted into self-correction; ordinary completion remains entirely
// local and adds no model call.
type completionPolicy struct {
continueNudges int
requireSemanticCheck bool
semanticCheckRequested bool
}
func newCompletionPolicy(requireSemanticCheck bool) *completionPolicy {
return &completionPolicy{requireSemanticCheck: requireSemanticCheck}
}
func (policy *completionPolicy) evaluate(text string, planPending bool) completionEvaluation {
// A direct admission is strong local evidence and takes precedence over all
// weaker signals, avoiding both continuation nudges and semantic checks.
if reason := selfReportedIncompletion(text); reason != "" {
return completionEvaluation{Decision: CompletionIncomplete, Reason: reason}
}
// A continuation cue is strong unfinished evidence; a pending plan is only
// weak evidence because bookkeeping can be stale. Both get a bounded chance
// to continue, but only a persisted cue is ultimately incomplete.
cue := endsWithContinuationCue(text)
if cue || planPending {
if policy.continueNudges < maxContinueNudges {
policy.continueNudges++
reason := "your message ended mid-step"
if !cue {
reason = "pending plan items remain — finish them, or mark them complete with update_plan if you are done"
}
return completionEvaluation{
Decision: CompletionUncertain,
Action: completionActionContinue,
Reason: reason,
}
}
if cue {
return completionEvaluation{
Decision: CompletionIncomplete,
Reason: "your message ended mid-step",
}
}
}
// Profiles with self-correction enabled require one task-grounded semantic
// check. It is requested at most once per run; the next locally complete turn
// is accepted without another model call.
if policy.requireSemanticCheck && !policy.semanticCheckRequested {
policy.semanticCheckRequested = true
return completionEvaluation{
Decision: CompletionUncertain,
Action: completionActionSemanticCheck,
}
}
return completionEvaluation{Decision: CompletionComplete, Action: completionActionNone}
}
+64
View File
@@ -0,0 +1,64 @@
package agent
import "testing"
func TestCompletionPolicyLocalEvidenceDecidesWithoutSemanticCheck(t *testing.T) {
policy := newCompletionPolicy(false)
complete := policy.evaluate("Done. All required checks pass.", false)
if complete.Decision != CompletionComplete {
t.Fatalf("confident completion decision = %q, want %q", complete.Decision, CompletionComplete)
}
incomplete := policy.evaluate("I couldn't verify the result, so this is my best guess.", false)
if incomplete.Decision != CompletionIncomplete {
t.Fatalf("admitted failure decision = %q, want %q", incomplete.Decision, CompletionIncomplete)
}
}
func TestCompletionPolicyPreservesBoundedPlanStallProtection(t *testing.T) {
policy := newCompletionPolicy(false)
for attempt := 0; attempt < maxContinueNudges; attempt++ {
got := policy.evaluate("Let me inspect the remaining configuration:", true)
if got.Decision != CompletionUncertain || got.Action != completionActionContinue {
t.Fatalf("attempt %d = (%q, %q), want uncertain continue", attempt+1, got.Decision, got.Action)
}
}
got := policy.evaluate("Let me inspect the remaining configuration:", true)
if got.Decision != CompletionIncomplete {
t.Fatalf("decision after nudge budget = %q, want %q", got.Decision, CompletionIncomplete)
}
}
func TestCompletionPolicyTreatsPendingPlanAsWeakEvidence(t *testing.T) {
policy := newCompletionPolicy(false)
for attempt := 0; attempt < maxContinueNudges; attempt++ {
got := policy.evaluate("All set.", true)
if got.Decision != CompletionUncertain || got.Action != completionActionContinue {
t.Fatalf("attempt %d = (%q, %q), want uncertain continue", attempt+1, got.Decision, got.Action)
}
}
got := policy.evaluate("All set.", true)
if got.Decision != CompletionComplete {
t.Fatalf("stale plan decision after nudge budget = %q, want %q", got.Decision, CompletionComplete)
}
}
func TestCompletionPolicyAllowsExactlyOneRequiredSemanticCheck(t *testing.T) {
policy := newCompletionPolicy(true)
first := policy.evaluate("Implemented and tested.", false)
if first.Decision != CompletionUncertain || first.Action != completionActionSemanticCheck {
t.Fatalf("first decision = (%q, %q), want uncertain semantic check", first.Decision, first.Action)
}
second := policy.evaluate("PASS. The result meets the task criterion.", false)
if second.Decision != CompletionComplete {
t.Fatalf("post-check decision = %q, want %q", second.Decision, CompletionComplete)
}
if second.Action == completionActionSemanticCheck {
t.Fatal("semantic check was requested more than once")
}
}
+51 -63
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"io"
"sort"
"strconv"
"strings"
"sync"
@@ -169,13 +170,10 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// loaded tool's full schema; it lives only for the run (v1 within-run scope).
loaded := map[string]bool{}
// continueNudges counts how many times the headless completion gate
// (Options.RequireCompletionSignal) has re-prompted a no-tool-call turn that
// stopped with work still unfinished. Bounded by maxContinueNudges.
continueNudges := 0
// acceptanceRequested records that the one-time task-grounded acceptance check
// has already been demanded this run, so it fires at most once.
acceptanceRequested := false
// The feature-gated completion policy keeps local completion decisions and its
// bounded follow-up state out of the central loop. Self-correcting profiles
// opt into the one permitted task-grounded semantic check.
completionPolicy := newCompletionPolicy(options.SelfCorrect != nil)
// toolDefCache memoizes each tool's rendered JSON-schema definition across
// turns (a tool's advertised schema is stable for the run), so partitionTools
@@ -520,68 +518,32 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// model's final answer ONLY when the work is actually done. Default off
// (RequireCompletionSignal), so interactive runs stay byte-identical.
if options.RequireCompletionSignal {
// (1) Self-report downgrade (strongest, unambiguous): the model's own
// final message admits it guessed / could not meet the objective. Checked
// FIRST so an admitted-impossible task is downgraded immediately (no wasted
// continue-nudges) and reports the accurate reason.
if reason := selfReportedIncompletion(collected.Text); reason != "" {
evaluation := completionPolicy.evaluate(collected.Text, guards.pendingPlanItems())
switch evaluation.Decision {
case CompletionIncomplete:
result.Incomplete = true
result.IncompleteReason = reason
result.IncompleteReason = evaluation.Reason
result.FinalAnswer = collected.Text
result.Messages = copyMessages(messages)
return result, nil
}
// (2) The model stopped without a tool call while work may be unfinished:
// - a continuation cue ("…Let me check the config:") is an unambiguous
// mid-step stop;
// - pending update_plan items are a WEAK, ambiguous signal (the model may
// have finished without re-marking the last step).
// Nudge to continue (bounded). After the budget: a persisted continuation
// cue finalizes INCOMPLETE; pending-plan WITHOUT a cue does NOT (that would
// false-fail a completed run with stale bookkeeping) — fall through to the
// acceptance check / success.
cue := endsWithContinuationCue(collected.Text)
planPending := guards.pendingPlanItems()
if cue || planPending {
if continueNudges < maxContinueNudges {
continueNudges++
case CompletionUncertain:
switch evaluation.Action {
case completionActionContinue:
options.Trace.Counter(trace.CounterCompletionNudges, 1)
reason := "your message ended mid-step"
if !cue {
reason = "pending plan items remain — finish them, or mark them complete with update_plan if you are done"
}
messages = append(messages, zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: continueNudge(reason),
Content: continueNudge(evaluation.Reason),
})
case completionActionSemanticCheck:
options.Trace.Counter(trace.CounterAcceptanceChecks, 1)
messages = append(messages, zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: acceptanceVerificationNudge(),
})
continue
}
if cue {
result.Incomplete = true
result.IncompleteReason = "your message ended mid-step"
result.FinalAnswer = collected.Text
result.Messages = copyMessages(messages)
return result, nil
}
// pending-plan only, budget spent: trust the model's completion claim
// over stale plan bookkeeping; fall through.
}
// (3) Task-grounded acceptance: before accepting a "done" turn as success,
// require ONE acceptance check grounded in the task's stated criterion
// (only when self-correct is on). Rejects "well-formed == correct",
// "existing-tests-pass == objective met", and "result == baseline" false
// successes. Bounded to a single pass; a genuine post-check completion
// (no admission, no cue) then finalizes as success on the next turn.
if options.SelfCorrect != nil && !acceptanceRequested {
acceptanceRequested = true
options.Trace.Counter(trace.CounterAcceptanceChecks, 1)
messages = append(messages, zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: acceptanceVerificationNudge(),
})
continue
case CompletionComplete:
// Local evidence is sufficient; proceed to final diagnostics.
}
}
// Finalization diagnostics gate: edits from this run may still have
@@ -632,10 +594,9 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// The scan happens lazily at the run's first index — never ahead of a
// pending mutating call — so read-after-write ordering is preserved.
if index >= precomputedEnd {
runEnd := index
for runEnd < len(collected.ToolCalls) && parallelSafeToolCall(registry, collected.ToolCalls[runEnd], options) {
runEnd++
}
// Capability-safe consecutive run (ReadOnly+ThreadSafe, no
// resource-key conflicts). See extendParallelRun.
runEnd := extendParallelRun(registry, collected.ToolCalls, index, options)
if runEnd-index >= 2 {
batchSpan := options.Trace.Span(trace.SpanToolExecution)
precomputed = executeParallelReadBatch(ctx, registry, collected.ToolCalls, index, runEnd, permissionMode, options)
@@ -657,6 +618,7 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
toolSpan.End()
}
options.Trace.Counter(trace.CounterToolCalls, 1)
recordOutputBudgetTrace(options.Trace, toolResult)
if options.OnToolResult != nil {
options.OnToolResult(toolResult)
}
@@ -842,6 +804,28 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
return result, nil
}
func recordOutputBudgetTrace(recorder *trace.Recorder, result ToolResult) {
if recorder == nil || result.Meta["output_budget_category"] == "" {
return
}
parseInt := func(key string) int {
value, _ := strconv.Atoi(result.Meta[key])
return value
}
spillCreated, _ := strconv.ParseBool(result.Meta["output_budget_spill_created"])
recorder.EmitOutputBudget(trace.OutputBudgetEvent{
Tool: result.Name,
Category: result.Meta["output_budget_category"],
OriginalBytes: parseInt("output_budget_original_bytes"),
RetainedBytes: parseInt("output_budget_retained_bytes"),
EstimatedOriginalTokens: parseInt("output_budget_estimated_original_tokens"),
EstimatedRetainedTokens: parseInt("output_budget_estimated_retained_tokens"),
Truncated: result.Truncated,
Reason: result.Meta["output_budget_reason"],
SpillCreated: spillCreated,
})
}
func finalAnswerAfterMaxTurns(ctx context.Context, provider Provider, messages []zeroruntime.Message, toolDefs []zeroruntime.ToolDefinition, options Options) (string, []zeroruntime.Message, string) {
finalMessages := copyMessages(messages)
finalMessages = append(finalMessages, zeroruntime.Message{
@@ -1296,6 +1280,7 @@ func executeToolCall(ctx context.Context, registry *tools.Registry, call ToolCal
if didRedact {
result.Redacted = true
}
result = registry.RebudgetAfterHook(call.Name, args, result)
}
}
// Secret scrubbing happens at the registry boundary (the single point both
@@ -1306,6 +1291,7 @@ func executeToolCall(ctx context.Context, registry *tools.Registry, call ToolCal
Name: call.Name,
Status: result.Status,
Output: result.Output,
Truncated: result.Truncated,
Meta: result.Meta,
Redacted: result.Redacted,
ChangedFiles: result.ChangedFiles,
@@ -1616,6 +1602,7 @@ func toolResultFromPrePermissionReject(call ToolCall, result tools.Result) ToolR
Name: call.Name,
Status: result.Status,
Output: output,
Truncated: result.Truncated,
Meta: meta,
Redacted: result.Redacted || outputRedacted || summaryRedacted || metaRedacted,
ChangedFiles: result.ChangedFiles,
@@ -1885,6 +1872,7 @@ func askUserFallbackResult(ctx context.Context, registry *tools.Registry, call T
Name: call.Name,
Status: result.Status,
Output: result.Output,
Truncated: result.Truncated,
Meta: result.Meta,
Redacted: result.Redacted,
ChangedFiles: result.ChangedFiles,
@@ -0,0 +1,167 @@
package agent
import (
"context"
"os"
"strconv"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/hooks"
"github.com/Gitlawb/zero/internal/tools"
"github.com/Gitlawb/zero/internal/trace"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
type propagationOutputTool struct {
output string
}
func TestOutputBudgetHookHelperProcess(t *testing.T) {
for index, arg := range os.Args {
if arg != "--zero-output-budget-hook" || index+1 >= len(os.Args) {
continue
}
if _, err := os.Stdout.WriteString(os.Args[index+1]); err != nil {
os.Exit(2)
}
os.Exit(0)
}
}
func largeOutputBudgetHookDispatcher() *hooks.Dispatcher {
feedback := strings.Repeat("hook feedback ", 200)
return hooks.NewDispatcher(hooks.DispatcherOptions{Config: hooks.Config{
Enabled: true,
Hooks: []hooks.Definition{{
ID: "large-feedback",
Event: hooks.EventAfterTool,
Matcher: "propagation_output",
Command: os.Args[0],
Args: []string{
"-test.run=TestOutputBudgetHookHelperProcess",
"--",
"--zero-output-budget-hook",
feedback,
},
Enabled: true,
}},
}})
}
func (tool propagationOutputTool) Name() string { return "propagation_output" }
func (tool propagationOutputTool) Description() string { return "returns output for propagation tests" }
func (tool propagationOutputTool) Parameters() tools.Schema {
return tools.Schema{Type: "object", AdditionalProperties: false}
}
func (tool propagationOutputTool) Safety() tools.Safety {
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow, Reason: "test read"}
}
func (tool propagationOutputTool) Run(context.Context, map[string]any) tools.Result {
return tools.Result{Status: tools.StatusOK, Output: tool.output}
}
func TestExecuteToolCallPropagatesOutputTruncation(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
t.Setenv("ZERO_TOOL_OUTPUT_CEILING_TOKENS", "80")
registry := tools.NewRegistry()
registry.Register(propagationOutputTool{output: strings.Repeat("large output\n", 1000)})
result, abortErr := executeToolCall(context.Background(), registry, ToolCall{
ID: "call-budget",
Name: "propagation_output",
Arguments: `{}`,
}, PermissionModeAuto, Options{Cwd: t.TempDir()})
if abortErr != nil {
t.Fatalf("executeToolCall abort error: %v", abortErr)
}
if !result.Truncated {
t.Fatalf("agent ToolResult lost tools.Result.Truncated: %#v", result)
}
if result.Meta["spill_path"] == "" {
t.Fatalf("agent ToolResult lost spill metadata: %#v", result.Meta)
}
}
func TestRecordOutputBudgetTraceUsesOnlyCompactMetadata(t *testing.T) {
recorder := trace.NewRecorder("session", "run", "")
recorder.Start()
recordOutputBudgetTrace(recorder, ToolResult{
Name: "grep",
Truncated: true,
Output: "SECRET OUTPUT MUST NOT ENTER TRACE",
Meta: map[string]string{
"output_budget_category": "search",
"output_budget_original_bytes": "1000",
"output_budget_retained_bytes": "100",
"output_budget_estimated_original_tokens": "250",
"output_budget_estimated_retained_tokens": "25",
"output_budget_reason": "semantic_search_budget",
"output_budget_spill_created": "true",
"spill_path": "/secret/path/not-for-trace",
},
})
events := recorder.Finish().OutputBudgets
if len(events) != 1 {
t.Fatalf("events = %#v", events)
}
event := events[0]
if event.Tool != "grep" || event.Category != "search" || event.OriginalBytes != 1000 || event.RetainedBytes != 100 || !event.SpillCreated {
t.Fatalf("unexpected trace event: %#v", event)
}
}
func TestExecuteToolCallRebudgetsOversizedAfterToolFeedback(t *testing.T) {
t.Setenv("ZERO_TOOL_OUTPUT_CEILING_TOKENS", "80")
registry := tools.NewRegistry()
registry.Register(propagationOutputTool{output: "tool output"})
dispatcher := largeOutputBudgetHookDispatcher()
result, abortErr := executeToolCall(context.Background(), registry, ToolCall{
ID: "call-hook-budget",
Name: "propagation_output",
Arguments: `{}`,
}, PermissionModeAuto, Options{Hooks: dispatcher})
if abortErr != nil {
t.Fatalf("executeToolCall abort error: %v", abortErr)
}
if !result.Truncated || len(result.Output) > 80*4 {
t.Fatalf("afterTool feedback bypassed output budget: truncated=%t bytes=%d meta=%#v", result.Truncated, len(result.Output), result.Meta)
}
if result.Meta["output_budget_category"] == "" || result.Meta["output_budget_retained_bytes"] != strconv.Itoa(len(result.Output)) {
t.Fatalf("post-hook budget metadata does not describe final output: %#v", result.Meta)
}
}
func TestRunTraceReflectsPostHookBudget(t *testing.T) {
t.Setenv("ZERO_TOOL_OUTPUT_CEILING_TOKENS", "80")
registry := tools.NewRegistry()
registry.Register(propagationOutputTool{output: "tool output"})
dispatcher := largeOutputBudgetHookDispatcher()
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
{
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call-hook-trace", ToolName: "propagation_output"},
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call-hook-trace", ArgumentsFragment: `{}`},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call-hook-trace"},
{Type: zeroruntime.StreamEventDone},
},
{{Type: zeroruntime.StreamEventText, Content: "done"}, {Type: zeroruntime.StreamEventDone}},
}}
recorder := trace.NewRecorder("session", "run", "")
var toolResults []ToolResult
if _, err := Run(context.Background(), "budget hook", provider, Options{
Registry: registry,
Hooks: dispatcher,
Trace: recorder,
OnToolResult: func(result ToolResult) { toolResults = append(toolResults, result) },
}); err != nil {
t.Fatalf("Run: %v", err)
}
if len(toolResults) != 1 || !toolResults[0].Truncated {
t.Fatalf("tool result = %#v, want one truncated post-hook result", toolResults)
}
events := recorder.Finish().OutputBudgets
if len(events) != 1 || !events[0].Truncated || events[0].RetainedBytes != len(toolResults[0].Output) {
t.Fatalf("trace does not describe final post-hook output: events=%#v result=%#v", events, toolResults[0])
}
}
+111 -11
View File
@@ -10,12 +10,23 @@ import (
// Parallel read-ahead for tool batches. When a turn requests several
// independent lookups (read_file + grep + glob is the common shape), executing
// them one after another serializes pure I/O waits. A consecutive run of
// auto-allowed read-only calls is executed concurrently instead; results are
// capability-safe read-only calls is executed concurrently instead; results are
// then consumed in the original call order, so guard counters, message
// ordering, abort semantics, and the surface's call/result event pairing are
// byte-identical to sequential execution. Runs never span a mutating call: a
// read that follows a write must observe the write, so eligibility is decided
// per consecutive run, not per batch.
//
// Eligibility uses the PR5 tool-effect contract (tools.CapabilitiesOf):
//
// Effect == ReadOnly
// AND ThreadSafe == true
// AND auto-allowed (no interactive permission prompt on the hot path)
// AND no resource-key conflict with an earlier call in the same concurrent window
//
// Unknown, mutators, interactive tools, and non-thread-safe reads stay
// sequential. Empty resource keys do not conflict (ThreadSafe is the safety
// gate); shared non-empty keys force a batch boundary.
// maxParallelReadTools bounds concurrent read-only tool executions in a turn.
const maxParallelReadTools = 8
@@ -28,28 +39,117 @@ type precomputedToolResult struct {
}
// parallelSafeToolCall reports whether call may run concurrently with its
// neighbors: the tool must exist, be side-effect-free (SideEffectRead), and be
// auto-allowed for these args, so no interactive prompt or workspace mutation
// is on the hot path. Loop-intercepted tools (ask_user, request_permissions)
// and tool_search (mutates the deferred-tool set) stay sequential.
// neighbors under the PR5 capability contract. Loop-intercepted tools
// (ask_user, request_permissions) and tool_search (mutates the deferred-tool
// set) always stay sequential.
func parallelSafeToolCall(registry *tools.Registry, call ToolCall, options Options) bool {
switch call.Name {
case "ask_user", tools.RequestPermissionsToolName, tools.ToolSearchToolName:
return false
}
tool, found := registry.Get(call.Name)
if !found || tool.Safety().SideEffect != tools.SideEffectRead {
if !found {
return false
}
args := map[string]any{}
if call.Arguments != "" {
if err := decodeToolArguments(call.Arguments, &args); err != nil {
return false
}
caps := tools.CapabilitiesOf(tool)
// Fail-closed: only audited concurrent-safe pure reads.
if caps.Effect != tools.EffectReadOnly || !caps.ThreadSafe {
return false
}
args, ok := decodeCallArgs(call)
if !ok {
return false
}
return effectivePermission(tool, args) == tools.PermissionAllow
}
// decodeCallArgs decodes tool call JSON arguments. Returns false on malformed
// input so the call stays sequential (never panics into the parallel path).
func decodeCallArgs(call ToolCall) (map[string]any, bool) {
args := map[string]any{}
if call.Arguments == "" {
return args, true
}
if err := decodeToolArguments(call.Arguments, &args); err != nil {
return nil, false
}
return args, true
}
// resourceKeysForCall returns the conflict keys for a call, or nil when the
// tool has no ResourceKeys function / no keys for these args.
func resourceKeysForCall(registry *tools.Registry, call ToolCall) []string {
tool, found := registry.Get(call.Name)
if !found {
return nil
}
caps := tools.CapabilitiesOf(tool)
if caps.ResourceKeys == nil {
return nil
}
args, ok := decodeCallArgs(call)
if !ok {
return nil
}
return caps.ResourceKeys(args)
}
// resourceKeysConflict reports whether two key sets share any non-empty key.
// Empty key sets never conflict: ThreadSafe is the eligibility gate; keys only
// refine conflict detection among tools that declare specific resources.
func resourceKeysConflict(a, b []string) bool {
if len(a) == 0 || len(b) == 0 {
return false
}
seen := make(map[string]struct{}, len(a))
for _, key := range a {
if key == "" {
continue
}
seen[key] = struct{}{}
}
for _, key := range b {
if key == "" {
continue
}
if _, ok := seen[key]; ok {
return true
}
}
return false
}
// extendParallelRun returns the exclusive end index of a consecutive parallel-
// safe run starting at start. The run stops before the first call that is not
// parallel-safe or that conflicts on resource keys with any earlier call in
// the same window (so two read_file calls on the same path stay sequential).
func extendParallelRun(registry *tools.Registry, calls []ToolCall, start int, options Options) int {
if start >= len(calls) || !parallelSafeToolCall(registry, calls[start], options) {
return start
}
end := start + 1
keysWindow := [][]string{resourceKeysForCall(registry, calls[start])}
for end < len(calls) {
if !parallelSafeToolCall(registry, calls[end], options) {
break
}
nextKeys := resourceKeysForCall(registry, calls[end])
conflict := false
for _, prev := range keysWindow {
if resourceKeysConflict(prev, nextKeys) {
conflict = true
break
}
}
if conflict {
break
}
keysWindow = append(keysWindow, nextKeys)
end++
}
return end
}
// executeParallelReadBatch runs calls[start:end] concurrently (bounded by
// maxParallelReadTools) and returns results indexed relative to start. All
// execution-side callbacks that can fire inside executeToolCall are serialized
+104
View File
@@ -57,6 +57,20 @@ func (tool *probeTool) Parameters() tools.Schema {
func (tool *probeTool) Safety() tools.Safety {
return tools.Safety{SideEffect: tool.sideEffect, Permission: tools.PermissionAllow, Reason: "test"}
}
// Capabilities maps the probe's SideEffect into the PR5 capability contract so
// parallelSafeToolCall (CapabilitiesOf) can classify reads vs mutators. Without
// this, probes default to EffectUnknown and never enter the concurrent path.
func (tool *probeTool) Capabilities() tools.ToolCapabilities {
switch tool.sideEffect {
case tools.SideEffectRead:
return tools.ToolCapabilities{Effect: tools.EffectReadOnly, ThreadSafe: true}
case tools.SideEffectWrite:
return tools.ToolCapabilities{Effect: tools.EffectWorkspaceWrite, ThreadSafe: false}
default:
return tools.UnknownCapabilities()
}
}
func (tool *probeTool) Run(_ context.Context, args map[string]any) tools.Result {
id, _ := args["id"].(string)
tool.mu.Lock()
@@ -92,6 +106,11 @@ func TestParallelSafeToolCall(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(&probeTool{name: "probe_read", sideEffect: tools.SideEffectRead})
registry.Register(&probeTool{name: "probe_write", sideEffect: tools.SideEffectWrite})
// Read-only but not ThreadSafe: must stay sequential under the PR5 gate.
registry.Register(&keyedProbeTool{
probeTool: probeTool{name: "probe_read_serial", sideEffect: tools.SideEffectRead},
threadSafe: false,
})
call := func(name, args string) ToolCall { return ToolCall{ID: "c", Name: name, Arguments: args} }
if !parallelSafeToolCall(registry, call("probe_read", `{"id":"a"}`), Options{}) {
@@ -109,6 +128,91 @@ func TestParallelSafeToolCall(t *testing.T) {
if parallelSafeToolCall(registry, call("ask_user", `{}`), Options{}) {
t.Fatal("loop-intercepted tools must stay sequential")
}
if parallelSafeToolCall(registry, call("probe_read_serial", `{"id":"a"}`), Options{}) {
t.Fatal("ReadOnly without ThreadSafe must not be parallel-safe")
}
}
func TestResourceKeysConflict(t *testing.T) {
if resourceKeysConflict(nil, []string{"file:a"}) {
t.Fatal("empty vs non-empty must not conflict")
}
if resourceKeysConflict([]string{"file:a"}, nil) {
t.Fatal("non-empty vs empty must not conflict")
}
if resourceKeysConflict([]string{"file:a"}, []string{"file:b"}) {
t.Fatal("distinct keys must not conflict")
}
if !resourceKeysConflict([]string{"file:a", "file:b"}, []string{"file:b"}) {
t.Fatal("shared key must conflict")
}
if resourceKeysConflict([]string{"", "file:a"}, []string{""}) {
t.Fatal("empty-string keys must be ignored")
}
}
func TestExtendParallelRunResourceKeyBoundary(t *testing.T) {
// Two keyed reads on the same path must not share a concurrent window;
// distinct paths may batch together.
registry := tools.NewRegistry()
registry.Register(&keyedProbeTool{
probeTool: probeTool{name: "keyed_read", sideEffect: tools.SideEffectRead},
threadSafe: true,
keys: func(args map[string]any) []string {
id, _ := args["id"].(string)
if id == "" {
return nil
}
return []string{"file:" + id}
},
})
calls := []ToolCall{
{ID: "1", Name: "keyed_read", Arguments: `{"id":"a"}`},
{ID: "2", Name: "keyed_read", Arguments: `{"id":"b"}`},
{ID: "3", Name: "keyed_read", Arguments: `{"id":"a"}`}, // conflicts with call 0
}
// start=0: a and b share no key → run covers [0,2)
if end := extendParallelRun(registry, calls, 0, Options{}); end != 2 {
t.Fatalf("extend from 0 = %d, want 2 (stop before second file:a)", end)
}
// start=2: single remaining call
if end := extendParallelRun(registry, calls, 2, Options{}); end != 3 {
t.Fatalf("extend from 2 = %d, want 3", end)
}
// Empty keys never conflict: three keyless safe reads form one window.
registry.Register(&probeTool{name: "keyless_read", sideEffect: tools.SideEffectRead})
keyless := []ToolCall{
{ID: "1", Name: "keyless_read", Arguments: `{"id":"x"}`},
{ID: "2", Name: "keyless_read", Arguments: `{"id":"y"}`},
{ID: "3", Name: "keyless_read", Arguments: `{"id":"z"}`},
}
if end := extendParallelRun(registry, keyless, 0, Options{}); end != 3 {
t.Fatalf("keyless extend = %d, want 3", end)
}
}
// keyedProbeTool is a probe with explicit ResourceKeys / ThreadSafe for
// planner unit tests.
type keyedProbeTool struct {
probeTool
threadSafe bool
keys func(args map[string]any) []string
}
func (tool *keyedProbeTool) Capabilities() tools.ToolCapabilities {
caps := tools.ToolCapabilities{
Effect: tools.EffectReadOnly,
ThreadSafe: tool.threadSafe,
}
if tool.keys != nil {
caps.ResourceKeys = tool.keys
}
// Honor mutator side effects from the embedded probe when used as write.
if tool.sideEffect == tools.SideEffectWrite {
caps.Effect = tools.EffectWorkspaceWrite
caps.ThreadSafe = false
}
return caps
}
func TestRunExecutesConsecutiveReadsConcurrently(t *testing.T) {
+12 -6
View File
@@ -61,10 +61,13 @@ const (
)
type ToolResult struct {
ToolCallID string
Name string
Status tools.Status
Output string
ToolCallID string
Name string
Status tools.Status
Output string
// Truncated reports that the tool's model-visible output omitted content.
// The full result may be recoverable through Meta["spill_path"].
Truncated bool
Meta map[string]string
Redacted bool
ChangedFiles []string
@@ -312,8 +315,11 @@ type Options struct {
// continuation cue ("…Let me check the config:"). The loop then nudges the
// model to continue instead, bounded by maxContinueNudges (and still by
// MaxTurns and the run deadline); if the model keeps stalling, the run
// finalizes as INCOMPLETE (Result.Incomplete) rather than success. Default
// false leaves the loop byte-identical, so the interactive TUI is unaffected.
// finalizes as INCOMPLETE (Result.Incomplete) rather than success. When the
// run's profile also enables SelfCorrect, an otherwise-complete turn gets one
// task-grounded semantic check before success; profiles without SelfCorrect
// add no model call. Default false leaves the loop byte-identical, so the
// interactive TUI is unaffected.
RequireCompletionSignal bool
runPermissions *permissionRunState
+3
View File
@@ -636,6 +636,9 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
if len(result.Meta) > 0 {
payload["meta"] = result.Meta
}
if result.Truncated {
payload["truncated"] = true
}
if result.Redacted {
payload["redacted"] = true
}
+5 -1
View File
@@ -153,6 +153,9 @@ func (writer *execEventWriter) toolResult(result agent.ToolResult) {
if len(result.Meta) > 0 {
payload["meta"] = result.Meta
}
if result.Truncated {
payload["truncated"] = true
}
if result.Redacted {
payload["redacted"] = true
}
@@ -166,7 +169,8 @@ func (writer *execEventWriter) toolResult(result agent.ToolResult) {
return
}
if writer.format == execOutputStreamJSON {
output, truncated := truncateForStreamJSONOutput(result.Output)
output, surfaceTruncated := truncateForStreamJSONOutput(result.Output)
truncated := result.Truncated || surfaceTruncated
event := streamjson.Event{
Type: streamjson.EventToolResult,
RunID: writer.runID,
+36
View File
@@ -1,8 +1,12 @@
package cli
import (
"bytes"
"encoding/json"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/agent"
"github.com/Gitlawb/zero/internal/tools"
)
@@ -16,3 +20,35 @@ func TestStreamJSONSideEffectReportsNoneForControlTool(t *testing.T) {
t.Fatalf("streamJSONSideEffect(escalate_model) = %q, want none", got)
}
}
func TestExecWriterPropagatesToolResultTruncation(t *testing.T) {
for _, format := range []execOutputFormat{execOutputJSON, execOutputStreamJSON} {
t.Run(string(format), func(t *testing.T) {
var stdout, stderr bytes.Buffer
writer := execEventWriter{
stdout: &stdout,
stderr: &stderr,
format: format,
runID: "run_budget",
streamedText: &strings.Builder{},
}
writer.toolResult(agent.ToolResult{
ToolCallID: "call_budget",
Name: "read_file",
Status: tools.StatusOK,
Output: "bounded output",
Truncated: true,
})
if writer.err != nil {
t.Fatalf("toolResult: %v", writer.err)
}
var payload map[string]any
if err := json.Unmarshal(bytes.TrimSpace(stdout.Bytes()), &payload); err != nil {
t.Fatalf("decode output %q: %v", stdout.String(), err)
}
if payload["truncated"] != true {
t.Fatalf("truncated = %#v, want true; payload=%#v", payload["truncated"], payload)
}
})
}
}
+120 -35
View File
@@ -10,16 +10,19 @@ matters for months, so the contract is written down here.
48 tasks across seven classes:
| class | count | oracle | tier |
|-----------|-------|-----------------------------|-------------|
| nav | 10 | none | latency |
| edit | 10 | substring grep | correctness |
| fix | 8 | scoped `go test -run <name>` | correctness |
| refactor | 6 | `go build ./...` | build |
| longproc | 4 | none | latency |
| longctx | 4 | none | latency |
| parallel | 6 | none | latency |
| **total** | **48**| | |
| class | count | oracle | tier |
|-----------|-------|-----------------------------------------|-------------|
| nav | 10 | grep on captured final answer | correctness |
| edit | 10 | stamped `oracle_test.go` + `go test` | correctness |
| fix | 8 | scoped `go test -run <name>` | correctness |
| refactor | 6 | stamped `oracle_test.go` + `go test` | correctness |
| longproc | 4 | none | latency |
| longctx | 4 | none | latency |
| parallel | 6 | none | latency |
| **total** | **48**| | |
Tier counts: **correctness 34** (10 edit + 8 fix + 10 nav + 6 refactor),
**build 0**, **latency 14** (4 longproc + 4 longctx + 6 parallel).
## Oracle tiers
@@ -27,16 +30,21 @@ Pass/fail is reported per tier so the report cannot be misread as a blanket
correctness verdict. The tier is decided per task from oracle presence and the
manifest's `buildOnlyClasses` list:
- **Correctness** (18 tasks: 10 edit + 8 fix) — a positive oracle (substring
grep that the requested change landed, or a scoped `go test -run` that the
bug fix passes). This is the only pass rate that can move with model quality:
`tasksVerified` / `tasksPassed` / `correctnessPassRate`.
- **Build-only** (6 tasks: refactor) — `go build ./...` proves the edit
compiles, not that the refactor achieved its goal (a no-op refactor passes).
The manifest declares `buildOnlyClasses: ["refactor"]`. Reported as
`buildCheckedTasks` / `buildPassedTasks` / `buildPassRate`, never in
`correctnessPassRate`.
- **Latency-only** (24 tasks: nav, longproc, longctx, parallel) — no
- **Correctness** (34 tasks: edit, fix, nav, refactor) — a positive oracle. For
edit/refactor the harness stamps an `oracle_test.go` (from the task's
`oracleTest` field) into the fixture copy **after** the agent run, then runs
`go test ./...`: the Go compiler is the structural verifier, so a no-op
refactor, a missing field, or a reworded-but-not-removed line fails to
compile/run. fix uses a scoped `go test -run <name>`. nav greps the agent's
captured final answer (`.zero-answer.txt`) for determinable facts. This is
the only pass rate that can move with model quality: `tasksVerified` /
`tasksPassed` / `correctnessPassRate`.
- **Build-only** (0 tasks) — the tier is empty. refactor used to live here with
a non-positive `go build ./...` (a no-op refactor passed); PR #712 gave it a
structural oracle, graduating it to correctness. `buildOnlyClasses` is `[]`.
`buildCheckedTasks` / `buildPassedTasks` / `buildPassRate` are reported as 0
and never in `correctnessPassRate`.
- **Latency-only** (14 tasks: longproc, longctx, parallel) — no
`verificationCommand`. An exit 0 only proves the turn ran, not that the answer
was right. They contribute to latency and span attribution only and are
counted in `latencyOnlyTasks`, never in any pass rate.
@@ -45,22 +53,89 @@ A task's tier is driven by oracle **presence** first: a task with no
`verificationCommand` is latency-only even if its class is listed in
`buildOnlyClasses`, so a missing oracle can never silently pass on exit 0.
## Known limitations (deferred)
### How the stamped oracle and captured answer work
These are accepted for the Phase 0 baseline and tracked in a follow-up issue;
they do not block the baseline because the tier split keeps the report honest:
The production runner (`NewTurnExecRunner` in `turn_bench.go`) copies the fixture
into an isolated temp dir, runs `zero exec --trace`, then — before running the
`verificationCommand` — stamps two files into the copy:
- The edit grep oracles are substring checks. A rename oracle asserts the new
name is present AND the old name is gone (compound `bash -c`), but an
add-field oracle (`grep -R Label .`) only proves the string appears, not that
it landed on the right struct. Strengthening these to `go vet`/`go build` +
structural assertions is follow-up work.
- The refactor build oracle is non-positive (a no-op refactor passes). A
structural verifier that proves the refactor happened is follow-up work;
until then refactor is `buildPassRate`, not `correctnessPassRate`.
- The 24 read-only tasks have no oracle. Adding deterministic oracles for nav
(diff against an expected answer) and documenting longproc/longctx/parallel
as permanently latency-only is follow-up work.
- `oracle_test.go` from `task.OracleTest` (edit/refactor). It is stamped *after*
the agent run so it can't interfere with the agent's own `go build`/`go test`
during the task (refactor-03's `package zeroapp` test would break a pre-rename
build) and can't be pre-seen or tampered with. The `verificationCommand`
(`go test ./...`) then compiles and runs it.
- `.zero-answer.txt` from the `{"type":"final","text":...}` event in the
stream-json output (nav). The `verificationCommand` is a compound `bash -c`
grep requiring the determinable facts (e.g. nav-09 requires the answer to
mention `port`, `name`, and `retries`). An empty file (no `final` event) makes
the grep fail, which is the correct outcome for a run that produced no answer.
The `oracleTest` source uses compile-time references where the assertion is
structural (`var _ = Config{}.Label` for edit-03, `var _ = formatGreeting` for
refactor-01) and behavioral `Test…` functions where the assertion is a value
(edit-04 `Version != "1.1.0"`, edit-10 `greet("x") != "hello, world"`, edit-09
`load("nonexistent")` wrapping). Tasks whose oracle is a plain grep/build with
no stamped test (edit-02, refactor-02) carry no `oracleTest`.
## Known limitations
These are accepted for the Phase 0 baseline; the tier split keeps the report
honest where they apply:
- **longproc / longctx / parallel are permanently latency-only.** There is no
deterministic oracle for "report whether the build succeeds", "summarize a
large generated file", or "report the first line of each of six files" — the
answer is free-form prose and a contains-check would rubber-stamp. They stay
in `latencyOnlyTasks` and never enter a pass rate.
- **nav count tasks (nav-01, nav-04, nav-05, nav-08) require a structured,
line-anchored `count: N` token with the fixture's exact expected count AND a
workspace-derived fact that proves the agent actually inspected the fixture —
not the count alone.** Their prompts ask the agent to state the count as
`count: N` on its own line and to name the thing(s) counted, and the oracle
greps for an anchored `^count: N$` with the exact expected N plus a named fact:
nav-01 requires `count: 5` and names `main.go`, `config.json`, and `README.md`
(the three the oracle anchors on — a correct answer lists all five files, but
the verification contract only requires those three be named); nav-04 names
the one test function (`TestGreet` from `main_test.go`); nav-05 names the file
holding the one TODO (`main.go`); nav-08 names both stdlib imports (`fmt` from
`main.go`, `testing` from `main_test.go`). A substring like "the count: 1" or a
wrong count like "count: 3" fails the anchor; a bare "count: 1" with no named
fact fails the inspection check. Non-zero ground truth alone was not enough —
it closes the always-guess-zero game but opens an always-guess-one game (an
agent that never opens the workspace and emits "count: 1" would pass a
count-only oracle). The named-fact requirement closes that surface: a blind
count is no longer sufficient, the agent must commit to a fact only readable
from the fixture. nav-04 and nav-05 were given a real `TestGreet` and a real
TODO so the named fact exists to require; nav-08 stays at count 0 (a real
third-party import would need a network fetch and break the offline suite), so
its named fact is the stdlib imports the agent must enumerate across every file
to conclude 0 third-party. The named facts are not all equally tight. nav-04's
`TestGreet` is a specific, non-generic identifier a blind agent cannot produce
without reading `main.go` to learn a greet function exists, so it fully closes
the blind-count-one surface. nav-05's `main.go` is the conventional Go
entrypoint and thus the most guessable filename, so it *reduces* rather than
fully closes that surface — a blind "count: 1, in main.go" can still pass.
Accepted for a Phase 0 baseline: the blind floor on nav-05 is below 100% (a
bare "count: 1" fails), so a model still moves the pass rate by actually
inspecting; strengthening it further (e.g. requiring the TODO's surrounding
symbol) would trade output freedom for determinism and risks false-failing a
terse-but-correct answer. nav-08's `testing` half is the tighter of its two:
`fmt` alone is the most common Go stdlib import and could be named blind, but
`testing` only appears in the test file, so naming it proves the agent read more
than `main.go`. This trades a little output freedom for determinism (a loose
prose-contains on "0"/"1" would match almost any answer, and an unanchored
`count: N` would match inside a longer line).
- **nav-03 / nav-06 / nav-07 are lenient contains-oracles on real facts.** A
wrong-but-plausible answer that happens to mention the real symbols (e.g.
"Config" and "MaxRetries" for nav-06) could in principle pass. Accepted for a
latency-first baseline; the oracle still catches an agent that didn't look at
the code at all.
- **Rename oracles use a declaration-site negative grep** (`! grep -RIn
'MaxRetries =' .`) rather than a broad `! grep -R MaxRetries .`, so a correct
rename that leaves a doc-comment mention of the old name is not false-failed.
The negative grep is still required: without it, a no-op that *adds* the new
name and *keeps* the old would pass the compile-time `var _ = RetryLimit` and
`go test`.
## Fixtures
@@ -68,4 +143,14 @@ Each task's `workspaceFixture` points at a small self-contained workspace under
`testdata/` so the suite runs offline and repeatably. Mutating tasks (edit,
fix, refactor) run against a per-invocation **copy** of their fixture, so the
checked-in fixtures stay clean and one task's edits can't bleed into the next
iteration or a later task.
iteration or a later task.
Each mutating fixture (and nav) carries a `go.mod` (`module <pkg>; go 1.22`) so
`go test ./...` / `go build ./...` oracles work in the copy: `copyFixture`
creates a unique, 0700 parent under the system temp root
(`os.MkdirTemp(os.TempDir(), "zero-turn-bench-*")`) and nests the copy beneath
it, so the copy is a *grandchild* of the temp root. Go ignores `go.mod` in
direct children of the system temp dir (a hijack guard), so the copy must sit
one level below that root or every compiler-backed oracle would fail with
"cannot find main module". The unique parent also avoids a predictable
shared directory and is removed wholesale on cleanup.
+540 -57
View File
@@ -1,62 +1,545 @@
{
"id": "zero-baseline-turn",
"name": "Zero per-turn baseline (Phase 0)",
"description": "Latency-first baseline: measures where a turn spends wall time, with pass/fail reported per oracle tier so it cannot be misread as a blanket correctness verdict. Read-only classes (nav, longproc, longctx, parallel) carry no verification oracle — a zero-exit only proves the turn ran, so they count as latency-only and never in any pass rate. Correctness classes (edit, fix) carry a positive oracle (substring grep or a scoped `go test`) and are the only tasks in correctnessPassRate. Build-only classes (refactor) carry `go build ./...`, which proves the edit compiles but not that the refactor achieved its goal, so they are reported as buildPassRate and excluded from correctnessPassRate.",
"buildOnlyClasses": ["refactor"],
"description": "Latency-first baseline: measures where a turn spends wall time, with pass/fail reported per oracle tier so it cannot be misread as a blanket correctness verdict. Correctness classes (edit, fix, nav, refactor) carry a positive oracle: edit/refactor stamp an oracle_test.go compiled by `go test ./...` (the Go compiler is the structural verifier \u2014 a no-op refactor or a missing field fails to compile), fix runs a scoped `go test -run`, and nav greps the agent's captured final answer (.zero-answer.txt) for determinable facts. tasksVerified/tasksPassed/correctnessPassRate cover exactly these. Latency-only classes (longproc, longctx, parallel) carry no verification oracle \u2014 a zero-exit only proves the turn ran, so they count as latency-only and never in any pass rate. The build tier is empty: refactor graduated to correctness once its oracle became structural-positive. See MANIFEST.md for the per-task oracle specs and known limitations.",
"buildOnlyClasses": [],
"tasks": [
{"id": "nav-01", "name": "list repo files", "class": "nav", "prompt": "List the files in this repository and report their count. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-02", "name": "find a function", "class": "nav", "prompt": "Find the function named 'greet' and report the file and line it is defined on. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-03", "name": "summarize module", "class": "nav", "prompt": "Read main.go and summarize what this program does in two sentences. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-04", "name": "count tests", "class": "nav", "prompt": "Count the test functions (functions whose names start with Test) in this repository and report the number. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-05", "name": "locate a string", "class": "nav", "prompt": "Find every occurrence of the string 'TODO' in the repository and report the files. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-06", "name": "describe package", "class": "nav", "prompt": "Describe the exported API of this package by reading the source. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-07", "name": "trace a call", "class": "nav", "prompt": "Trace which function calls which other function starting from main. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-08", "name": "find imports", "class": "nav", "prompt": "List every third-party import used by this repository. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-09", "name": "report config", "class": "nav", "prompt": "Read config.json and report the keys it contains. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-10", "name": "doc lookup", "class": "nav", "prompt": "Find the doc comment on the greet function and quote it. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "edit-01", "name": "rename a constant", "class": "edit", "prompt": "Rename the constant MaxRetries to RetryLimit everywhere it appears.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R RetryLimit . && ! grep -R MaxRetries ."]},
{"id": "edit-02", "name": "fix a typo", "class": "edit", "prompt": "Fix the typo in the comment above the greet function ('receieves' should be 'receives').", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "! grep -R receieves ."]},
{"id": "edit-03", "name": "add a field", "class": "edit", "prompt": "Add a string field named Label to the Config struct in main.go.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "Label", "."]},
{"id": "edit-04", "name": "bump a version", "class": "edit", "prompt": "Change the version string in version.go from 1.0.0 to 1.1.0.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R 1.1.0 . && ! grep -R 1.0.0 ."]},
{"id": "edit-05", "name": "remove a log line", "class": "edit", "prompt": "Remove the line that prints 'debug: starting'.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "! grep -R 'debug: starting' ."]},
{"id": "edit-06", "name": "add a license header", "class": "edit", "prompt": "Add a comment line '// SPDX-License-Identifier: MIT' at the top of main.go.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "SPDX-License-Identifier", "."]},
{"id": "edit-07", "name": "change a default", "class": "edit", "prompt": "Change the default value of the Port constant from 8080 to 9090.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R 9090 . && ! grep -R 8080 ."]},
{"id": "edit-08", "name": "add a getter", "class": "edit", "prompt": "Add a method func (c *Config) GetLabel() string that returns c.Label.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R 'func (c *Config) GetLabel' ."]},
{"id": "edit-09", "name": "wrap an error", "class": "edit", "prompt": "In the load function, wrap the returned error with fmt.Errorf so it includes 'load failed'.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "load failed", "."]},
{"id": "edit-10", "name": "update a message", "class": "edit", "prompt": "Change the printed greeting from 'hello' to 'hello, world'.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "hello, world", "."]},
{"id": "fix-01", "name": "off-by-one", "class": "fix", "prompt": "In bugs.go the Sum function has an off-by-one error that skips the last element; fix the loop so it sums all elements.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestSumIncludesLastElement", "./..."]},
{"id": "fix-02", "name": "nil deref", "class": "fix", "prompt": "In bugs.go DefaultConfig returns nil and Port() dereferences it; make DefaultConfig return a valid Config with Port 8080.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestPortReturnsDefault", "./..."]},
{"id": "fix-03", "name": "wrong operator", "class": "fix", "prompt": "In bugs.go Max uses the wrong inequality and returns the smaller value; fix it to return the larger.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestMaxReturnsLarger", "./..."]},
{"id": "fix-04", "name": "truncated read", "class": "fix", "prompt": "In bugs.go ReadFirst returns only a single character instead of the full content; fix it to return the whole string.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestReadFirstReturnsFullContent", "./..."]},
{"id": "fix-05", "name": "swapped args", "class": "fix", "prompt": "In bugs.go Divide swaps its operands; fix the order so Divide(10,2) returns 5.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestDivideOrder", "./..."]},
{"id": "fix-06", "name": "missing return", "class": "fix", "prompt": "In bugs.go Classify is missing a return for the high case; add it so Classify(500) returns \"high\".", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestClassifyHigh", "./..."]},
{"id": "fix-07", "name": "wrong label prefix", "class": "fix", "prompt": "In bugs.go Label returns 'admin-<name>' but the test expects 'user-<name>'; change the prefix from 'admin-' to 'user-'.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestLabelUsesString", "./..."]},
{"id": "fix-08", "name": "race on counter", "class": "fix", "prompt": "In bugs.go the global counter is incremented without synchronization; guard it with a mutex.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-race", "-run", "TestCounterRace", "./..."]},
{"id": "refactor-01", "name": "extract helper", "class": "refactor", "prompt": "Extract the duplicated greeting logic into a single helper function and call it from both sites.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "refactor-02", "name": "split a file", "class": "refactor", "prompt": "Split main.go into two files: one for Config, one for the rest, keeping the package building.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "refactor-03", "name": "rename package", "class": "refactor", "prompt": "Rename the package from app to zeroapp everywhere and keep it building.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "refactor-04", "name": "introduce a type", "class": "refactor", "prompt": "Introduce a named type for the map[string]int used in main.go and update its users.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "refactor-05", "name": "inline a wrapper", "class": "refactor", "prompt": "Inline the thin wrapper function into its single caller and remove it.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "refactor-06", "name": "consolidate errors", "class": "refactor", "prompt": "Consolidate the repeated error-construction blocks into a single helper.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "longproc-01", "name": "long build", "class": "longproc", "prompt": "Run the build command and report whether it succeeds. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
{"id": "longproc-02", "name": "long test", "class": "longproc", "prompt": "Run go test ./... and report the result. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
{"id": "longproc-03", "name": "bench run", "class": "longproc", "prompt": "Run go test -bench=. -benchtime=1x and report the benchmark numbers. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
{"id": "longproc-04", "name": "vet run", "class": "longproc", "prompt": "Run go vet ./... and report the output. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
{"id": "longctx-01", "name": "summarize large file", "class": "longctx", "prompt": "Read the large generated file big.go and summarize its structure. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
{"id": "longctx-02", "name": "find in large file", "class": "longctx", "prompt": "In big.go, find every function that returns an error and list them. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
{"id": "longctx-03", "name": "cross-file question", "class": "longctx", "prompt": "Read all .go files and report which two functions share the same name across files. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
{"id": "longctx-04", "name": "trace through large file", "class": "longctx", "prompt": "Starting from Handle, trace the call path through big.go and report the chain. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
{"id": "parallel-01", "name": "read six files", "class": "parallel", "prompt": "Read a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt, then report the first line of each. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
{"id": "parallel-02", "name": "grep six patterns", "class": "parallel", "prompt": "Search for 'alpha', 'beta', 'gamma', 'delta', 'epsilon', and 'zeta' across the repo and report which files contain each. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
{"id": "parallel-03", "name": "list six dirs", "class": "parallel", "prompt": "List the contents of dir1, dir2, dir3, dir4, dir5, and dir6. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
{"id": "parallel-04", "name": "stat six files", "class": "parallel", "prompt": "Report the line count of a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
{"id": "parallel-05", "name": "read configs", "class": "parallel", "prompt": "Read config1.json, config2.json, config3.json, config4.json, config5.json, and config6.json and report each top-level key. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
{"id": "parallel-06", "name": "summarize six", "class": "parallel", "prompt": "Read the six .txt files and give a one-line summary of each. Do not modify anything.", "workspaceFixture": "../testdata/parallel"}
{
"id": "nav-01",
"name": "list repo files",
"class": "nav",
"prompt": "List the files in this repository and report their count. State the count as 'count: N' on its own line. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'main.go' .zero-answer.txt && grep -iq 'config.json' .zero-answer.txt && grep -iq 'readme' .zero-answer.txt && grep -iqE '^count:[[:space:]]*5$' .zero-answer.txt"
]
},
{
"id": "nav-02",
"name": "find a function",
"class": "nav",
"prompt": "Find the function named 'greet' and report the file and line it is defined on. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'main.go' .zero-answer.txt && grep -iqE '\\b1[5-7]\\b' .zero-answer.txt"
]
},
{
"id": "nav-03",
"name": "summarize module",
"class": "nav",
"prompt": "Read main.go and summarize what this program does in two sentences. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iqE 'greet|hello|Config' .zero-answer.txt"
]
},
{
"id": "nav-04",
"name": "count tests",
"class": "nav",
"prompt": "Count the test functions (functions whose names start with Test) in this repository. Name each one you find and report the number. State the count as 'count: N' on its own line. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'TestGreet' .zero-answer.txt && grep -iqE '^count:[[:space:]]*1$' .zero-answer.txt"
]
},
{
"id": "nav-05",
"name": "locate a string",
"class": "nav",
"prompt": "Find every occurrence of the string 'TODO' in the repository and report the file(s) they appear in. State the count as 'count: N' on its own line. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'main.go' .zero-answer.txt && grep -iqE '^count:[[:space:]]*1$' .zero-answer.txt"
]
},
{
"id": "nav-06",
"name": "describe package",
"class": "nav",
"prompt": "Describe the exported API of this package by reading the source. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq config .zero-answer.txt && grep -iq maxretries .zero-answer.txt"
]
},
{
"id": "nav-07",
"name": "trace a call",
"class": "nav",
"prompt": "Trace which function calls which other function starting from main. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq main .zero-answer.txt && grep -iq greet .zero-answer.txt"
]
},
{
"id": "nav-08",
"name": "find imports",
"class": "nav",
"prompt": "List every import used by this repository and classify each as standard library or third-party. State the number of third-party (non-stdlib) imports as 'count: N' on its own line. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'fmt' .zero-answer.txt && grep -iq 'testing' .zero-answer.txt && grep -iqE '^count:[[:space:]]*0$' .zero-answer.txt"
]
},
{
"id": "nav-09",
"name": "report config",
"class": "nav",
"prompt": "Read config.json and report the keys it contains. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq port .zero-answer.txt && grep -iq name .zero-answer.txt && grep -iq retries .zero-answer.txt"
]
},
{
"id": "nav-10",
"name": "doc lookup",
"class": "nav",
"prompt": "Find the doc comment on the greet function and quote it. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'greet returns a greeting' .zero-answer.txt"
]
},
{
"id": "edit-01",
"name": "rename a constant",
"class": "edit",
"prompt": "Rename the constant MaxRetries to RetryLimit everywhere it appears.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep -RIn 'MaxRetries =' ."
],
"oracleTest": "package edit\n\nvar _ = RetryLimit\n"
},
{
"id": "edit-02",
"name": "fix a typo",
"class": "edit",
"prompt": "Fix the typo in the comment above the greet function ('receieves' should be 'receives').",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep -R receieves . && grep -R receives ."
]
},
{
"id": "edit-03",
"name": "add a field",
"class": "edit",
"prompt": "Add a string field named Label to the Config struct in main.go.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nvar _ string = Config{}.Label\n"
},
{
"id": "edit-04",
"name": "bump a version",
"class": "edit",
"prompt": "Change the version string in version.go from 1.0.0 to 1.1.0.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestVersion(t *testing.T) {\n\tif Version != \"1.1.0\" {\n\t\tt.Fatalf(\"Version=%q\", Version)\n\t}\n}\n"
},
{
"id": "edit-05",
"name": "remove a log line",
"class": "edit",
"prompt": "Remove the line that prints 'debug: starting'.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport (\n\t\"os\"\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestNoDebugPrintln(t *testing.T) {\n\tb, err := os.ReadFile(\"main.go\")\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\t// The greet print is fmt.Println(greet(...)) with no string literal, so a\n\t// remaining fmt.Println(\"...\") call means the debug line was not removed\n\t// (or was only reworded, not deleted).\n\tif strings.Contains(string(b), `fmt.Println(\"`) {\n\t\tt.Fatal(\"main.go still has a fmt.Println string-literal call; the debug line was not removed\")\n\t}\n\t// The task is to remove the line that prints \"debug: starting\", so the\n\t// message text itself must be gone -- this catches a reword onto a different\n\t// call form (fmt.Printf, backticks, log.Println, println) that the\n\t// string-literal check above would miss.\n\tif strings.Contains(string(b), \"debug: starting\") {\n\t\tt.Fatal(\"main.go still prints 'debug: starting'; the line was not removed (only reworded or respelled)\")\n\t}\n}\n"
},
{
"id": "edit-06",
"name": "add a license header",
"class": "edit",
"prompt": "Add a comment line '// SPDX-License-Identifier: MIT' at the top of main.go.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport (\n\t\"os\"\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestSPDXHeader(t *testing.T) {\n\tb, err := os.ReadFile(\"main.go\")\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\tif !strings.HasPrefix(string(b), \"// SPDX-License-Identifier: MIT\") {\n\t\tt.Fatal(\"main.go must start with the SPDX-License-Identifier: MIT header\")\n\t}\n}\n"
},
{
"id": "edit-07",
"name": "change a default",
"class": "edit",
"prompt": "Change the default value of the Port constant from 8080 to 9090.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestPort(t *testing.T) {\n\tif Port != 9090 {\n\t\tt.Fatalf(\"Port=%d, want 9090\", Port)\n\t}\n}\n"
},
{
"id": "edit-08",
"name": "add a getter",
"class": "edit",
"prompt": "Add a method func (c *Config) GetName() string that returns c.Name.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestGetName(t *testing.T) {\n\tc := Config{Name: \"x\"}\n\tif got := c.GetName(); got != \"x\" {\n\t\tt.Fatalf(\"GetName()=%q, want %q\", got, \"x\")\n\t}\n}\n"
},
{
"id": "edit-09",
"name": "wrap an error",
"class": "edit",
"prompt": "In the load function, wrap the returned error with fmt.Errorf so it includes 'load failed'.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport (\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestLoadWrapsError(t *testing.T) {\n\t_, err := load(\"nonexistent-file-xyz\")\n\tif err == nil {\n\t\tt.Fatal(\"want a non-nil error for a missing file\")\n\t}\n\tif !strings.Contains(err.Error(), \"load failed\") {\n\t\tt.Fatalf(\"err=%v, want it to contain 'load failed'\", err)\n\t}\n}\n"
},
{
"id": "edit-10",
"name": "update a message",
"class": "edit",
"prompt": "Change the printed greeting from 'hello' to 'hello, world'.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestGreet(t *testing.T) {\n\tif got := greet(\"x\"); got != \"hello, world\" {\n\t\tt.Fatalf(\"greet()=%q, want %q\", got, \"hello, world\")\n\t}\n}\n"
},
{
"id": "fix-01",
"name": "off-by-one",
"class": "fix",
"prompt": "In bugs.go the Sum function has an off-by-one error that skips the last element; fix the loop so it sums all elements.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestSumIncludesLastElement",
"./..."
]
},
{
"id": "fix-02",
"name": "nil deref",
"class": "fix",
"prompt": "In bugs.go DefaultConfig returns nil and Port() dereferences it; make DefaultConfig return a valid Config with Port 8080.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestPortReturnsDefault",
"./..."
]
},
{
"id": "fix-03",
"name": "wrong operator",
"class": "fix",
"prompt": "In bugs.go Max uses the wrong inequality and returns the smaller value; fix it to return the larger.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestMaxReturnsLarger",
"./..."
]
},
{
"id": "fix-04",
"name": "truncated read",
"class": "fix",
"prompt": "In bugs.go ReadFirst returns only a single character instead of the full content; fix it to return the whole string.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestReadFirstReturnsFullContent",
"./..."
]
},
{
"id": "fix-05",
"name": "swapped args",
"class": "fix",
"prompt": "In bugs.go Divide swaps its operands; fix the order so Divide(10,2) returns 5.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestDivideOrder",
"./..."
]
},
{
"id": "fix-06",
"name": "missing return",
"class": "fix",
"prompt": "In bugs.go Classify is missing a return for the high case; add it so Classify(500) returns \"high\".",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestClassifyHigh",
"./..."
]
},
{
"id": "fix-07",
"name": "wrong label prefix",
"class": "fix",
"prompt": "In bugs.go Label returns 'admin-<name>' but the test expects 'user-<name>'; change the prefix from 'admin-' to 'user-'.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestLabelUsesString",
"./..."
]
},
{
"id": "fix-08",
"name": "race on counter",
"class": "fix",
"prompt": "In bugs.go the global counter is incremented without synchronization; guard it with a mutex.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-race",
"-run",
"TestCounterRace",
"./..."
]
},
{
"id": "refactor-01",
"name": "extract helper",
"class": "refactor",
"prompt": "Extract the duplicated fmt.Sprintf(\"hello, %s\", ...) greeting logic into a single helper function named formatGreeting and call it from both GreetFromConfig and GreetByName.",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"bash",
"-c",
"go test ./... && [ $(grep -c 'hello, %s' main.go) -eq 1 ]"
],
"oracleTest": "package refactor\n\nvar _ = formatGreeting\n"
},
{
"id": "refactor-02",
"name": "split a file",
"class": "refactor",
"prompt": "Move the Config type into a new file named config.go, keeping the package building.",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"bash",
"-c",
"go test ./... && grep -q 'type Config' config.go && ! grep -q 'type Config' main.go"
]
},
{
"id": "refactor-03",
"name": "rename package",
"class": "refactor",
"prompt": "Rename the package from refactor to zeroapp everywhere and keep it building.",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep -R 'package refactor' . && grep -R 'package zeroapp' ."
],
"oracleTest": "package zeroapp\n\nvar _ = Config{}\n"
},
{
"id": "refactor-04",
"name": "introduce a type",
"class": "refactor",
"prompt": "Introduce a named type `type Stats map[string]int` for the bare map used by stats, and update its users (Record, Lookup).",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package refactor\n\nimport (\n\t\"os\"\n\t\"regexp\"\n\t\"strings\"\n\t\"testing\"\n)\n\n// TestStatsNamedType proves the bare map was replaced by the named Stats type:\n// the old `var stats = map[string]int` (or `stats := map[string]int{}`) form is\n// gone -- matched with a spacing-tolerant regex so a `var stats=map[string]int`\n// or `stats :=` variant can't dodge it -- and the Stats type is actually\n// referenced in main.go, so a no-op that only adds an unused `type Stats` while\n// keeping the bare map fails. `go test ./...` compiling the fixture proves the\n// type declaration is valid.\nfunc TestStatsNamedType(t *testing.T) {\n\tb, err := os.ReadFile(\"main.go\")\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\tsrc := string(b)\n\tif regexp.MustCompile(`var\\s+stats\\s*=\\s*map\\[string\\]int`).MatchString(src) ||\n\t\tregexp.MustCompile(`stats\\s*:=\\s*map\\[string\\]int`).MatchString(src) {\n\t\tt.Fatal(\"main.go still declares stats as a bare map[string]int; the named Stats type was not adopted\")\n\t}\n\tif !strings.Contains(src, \"Stats\") {\n\t\tt.Fatal(\"main.go does not reference the Stats type; refactor-04 was a no-op\")\n\t}\n}\n"
},
{
"id": "refactor-05",
"name": "inline a wrapper",
"class": "refactor",
"prompt": "Inline the Wrapper function into greetWrapped (its single caller) and remove the Wrapper function.",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep -R 'func Wrapper' ."
],
"oracleTest": "package refactor\n\nimport \"testing\"\n\n// TestGreetWrapped proves greetWrapped still exists and behaves as the inlined\n// Wrapper did: deleting both Wrapper and greetWrapped (or stubbing greetWrapped\n// to return \"\") fails to compile or run, so the oracle can't be passed by\n// removing the target along with its caller. Paired with the ! grep 'func\n// Wrapper' verificationCommand that proves Wrapper itself is gone.\nfunc TestGreetWrapped(t *testing.T) {\n\tif got := greetWrapped(\"x\"); got != \"hello, x\" {\n\t\tt.Fatalf(\"greetWrapped(x) = %q, want %q\", got, \"hello, x\")\n\t}\n}\n"
},
{
"id": "refactor-06",
"name": "consolidate errors",
"class": "refactor",
"prompt": "Consolidate the three error-construction functions (firstError, secondError, buildError) into a single helper func makeLoadError(ctx string) error that returns fmt.Errorf(\"load failed: %s\", ctx), and call it from all three.",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"bash",
"-c",
"go test ./... && [ $(grep -c 'fmt.Errorf(\"load failed' main.go) -eq 1 ]"
],
"oracleTest": "package refactor\n\nvar _ = makeLoadError\n"
},
{
"id": "longproc-01",
"name": "long build",
"class": "longproc",
"prompt": "Run the build command and report whether it succeeds. Do not modify anything.",
"workspaceFixture": "../testdata/longproc"
},
{
"id": "longproc-02",
"name": "long test",
"class": "longproc",
"prompt": "Run go test ./... and report the result. Do not modify anything.",
"workspaceFixture": "../testdata/longproc"
},
{
"id": "longproc-03",
"name": "bench run",
"class": "longproc",
"prompt": "Run go test -bench=. -benchtime=1x and report the benchmark numbers. Do not modify anything.",
"workspaceFixture": "../testdata/longproc"
},
{
"id": "longproc-04",
"name": "vet run",
"class": "longproc",
"prompt": "Run go vet ./... and report the output. Do not modify anything.",
"workspaceFixture": "../testdata/longproc"
},
{
"id": "longctx-01",
"name": "summarize large file",
"class": "longctx",
"prompt": "Read the large generated file big.go and summarize its structure. Do not modify anything.",
"workspaceFixture": "../testdata/longctx"
},
{
"id": "longctx-02",
"name": "find in large file",
"class": "longctx",
"prompt": "In big.go, find every function that returns an error and list them. Do not modify anything.",
"workspaceFixture": "../testdata/longctx"
},
{
"id": "longctx-03",
"name": "cross-file question",
"class": "longctx",
"prompt": "Read all .go files and report which two functions share the same name across files. Do not modify anything.",
"workspaceFixture": "../testdata/longctx"
},
{
"id": "longctx-04",
"name": "trace through large file",
"class": "longctx",
"prompt": "Starting from Handle, trace the call path through big.go and report the chain. Do not modify anything.",
"workspaceFixture": "../testdata/longctx"
},
{
"id": "parallel-01",
"name": "read six files",
"class": "parallel",
"prompt": "Read a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt, then report the first line of each. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
},
{
"id": "parallel-02",
"name": "grep six patterns",
"class": "parallel",
"prompt": "Search for 'alpha', 'beta', 'gamma', 'delta', 'epsilon', and 'zeta' across the repo and report which files contain each. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
},
{
"id": "parallel-03",
"name": "list six dirs",
"class": "parallel",
"prompt": "List the contents of dir1, dir2, dir3, dir4, dir5, and dir6. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
},
{
"id": "parallel-04",
"name": "stat six files",
"class": "parallel",
"prompt": "Report the line count of a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
},
{
"id": "parallel-05",
"name": "read configs",
"class": "parallel",
"prompt": "Read config1.json, config2.json, config3.json, config4.json, config5.json, and config6.json and report each top-level key. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
},
{
"id": "parallel-06",
"name": "summarize six",
"class": "parallel",
"prompt": "Read the six .txt files and give a one-line summary of each. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
}
]
}
}
+53 -9
View File
@@ -51,6 +51,14 @@ type BenchTask struct {
Prompt string `json:"prompt"`
WorkspaceFixture string `json:"workspaceFixture,omitempty"`
VerificationCommand []string `json:"verificationCommand,omitempty"`
// OracleTest is Go source stamped into <fixtureCopy>/oracle_test.go AFTER
// the agent run (so it can't interfere with the agent's own go build/test
// during the task and can't be pre-seen). The verificationCommand then runs
// `go test ./...`, making the Go compiler the structural verifier: a
// compile-time reference (e.g. `var _ = Config{}.Label`) or a behavioral call
// fails to compile/run when the task wasn't actually done. Empty for tasks
// whose verificationCommand is a plain grep/build with no stamped test.
OracleTest string `json:"oracleTest,omitempty"`
}
// TaskConfig is the recorded configuration of a benchmark run. The honest
@@ -389,14 +397,26 @@ func runVerification(ctx context.Context, task BenchTask) TaskOutcome {
return TaskOutcome{Passed: true}
}
// streamJSONExitCode scans stream-json output for the terminal run_end event and
// returns its exit code. ZERO emits one JSON object per line; the last run_end
// (or final) event carries the exit code. Lines that don't parse are skipped.
func streamJSONExitCode(output []byte) (int, bool) {
// streamJSONResult holds the terminal events parsed from one pass over a
// stream-json output: the last run_end exit code (if any) and the last final
// event's text (the agent's answer, if any). streamJSONExitCode and
// streamJSONFinalText both consume this so the scanner limits and
// malformed-event handling live in exactly one place.
type streamJSONResult struct {
exitCode int
haveExit bool
finalText string
}
// parseStreamJSON scans stream-json output once and returns the terminal
// run_end exit code and the final event's text. ZERO emits one JSON object per
// line; the last run_end carries the exit code and the last final carries the
// answer. Lines that don't parse are skipped. The scanner buffer is raised so a
// large single-line event (e.g. a big final answer) doesn't error out.
func parseStreamJSON(output []byte) streamJSONResult {
scanner := bufio.NewScanner(bytes.NewReader(output))
scanner.Buffer(make([]byte, 0, 64*1024), 4*1024*1024)
exitCode := 0
found := false
var r streamJSONResult
for scanner.Scan() {
line := bytes.TrimSpace(scanner.Bytes())
if len(line) == 0 || line[0] != '{' {
@@ -405,16 +425,40 @@ func streamJSONExitCode(output []byte) (int, bool) {
var event struct {
Type string `json:"type"`
ExitCode *int `json:"exitCode"`
Text string `json:"text"`
}
if err := json.Unmarshal(line, &event); err != nil {
continue
}
if event.Type == "run_end" && event.ExitCode != nil {
exitCode = *event.ExitCode
found = true
r.exitCode = *event.ExitCode
r.haveExit = true
}
if event.Type == "final" {
r.finalText = event.Text
}
}
return exitCode, found
return r
}
// streamJSONExitCode scans stream-json output for the terminal run_end event and
// returns its exit code. The last run_end event carries the exit code. Lines
// that don't parse are skipped.
func streamJSONExitCode(output []byte) (int, bool) {
r := parseStreamJSON(output)
return r.exitCode, r.haveExit
}
// streamJSONFinalText scans stream-json output for the terminal "final" event
// and returns its text (the agent's final answer). ZERO emits one final event
// per run on both the success and incomplete paths (exec.go writer.final), so
// the last one wins. The turn benchmark writes this to .zero-answer.txt in the
// fixture copy so nav answer-oracles can grep the captured answer rather than
// the raw stream. Returns "" when no final event was emitted (e.g. a provider
// error path) — the caller writes an empty file and a nav grep then fails,
// which is the correct outcome for a run that produced no answer.
func streamJSONFinalText(output []byte) string {
return parseStreamJSON(output).finalText
}
func firstLine(text string) string {
+3
View File
@@ -0,0 +1,3 @@
module edit
go 1.22
+3
View File
@@ -0,0 +1,3 @@
module fix
go 1.22
+3
View File
@@ -0,0 +1,3 @@
module nav
go 1.22
+4
View File
@@ -22,3 +22,7 @@ func greet(name string) string {
func main() {
fmt.Println(greet("world"))
}
// TODO: replace the demo greet with the real client call before shipping.
// (Present so nav-05's find-the-markers task has a non-zero, inspection-
// required answer; an agent that always emits "count: 0" fails.)
+13
View File
@@ -0,0 +1,13 @@
package nav
import "testing"
// TestGreet is a real test so nav-04's "count the test functions" has a
// non-zero, inspection-required answer: an agent that never opens the workspace
// and always emits "count: 0" fails, which is the whole point of graduating nav
// into the correctness tier. It runs green offline (no third-party deps).
func TestGreet(t *testing.T) {
if got := greet("x"); got != "hello, x" {
t.Fatalf("greet(x) = %q, want %q", got, "hello, x")
}
}
+3
View File
@@ -0,0 +1,3 @@
module refactor
go 1.22
+6
View File
@@ -28,6 +28,12 @@ func Wrapper(name string) string {
return GreetByName(name)
}
// greetWrapped is Wrapper's single caller — refactor-05 inlines Wrapper here
// and removes the Wrapper function.
func greetWrapped(name string) string {
return Wrapper(name)
}
// stats is a bare map used across the file — refactor-04 introduces a named type.
var stats = map[string]int{}
+119 -30
View File
@@ -27,7 +27,20 @@ import (
// buildPassedTasks/buildPassRate cover the non-positive build-check classes
// (refactor); latencyOnlyTasks covers the no-oracle classes (nav/longproc/
// longctx/parallel). tasksAttempted is still every task run.
const TurnSchemaVersion = 2
//
// v3 (Phase 0 oracle hardening) strengthens the oracles so the tiers are honest: edit/refactor
// now carry a stamped oracle_test.go compiled by `go test ./...` (the Go
// compiler is the structural verifier — a no-op refactor or a missing field
// fails to compile), and nav carries an answer-oracle grepping the agent's
// captured final answer. refactor graduates from build to correctness (its
// oracle is now positive), nav graduates from latency-only to correctness, and
// the build tier is empty (nothing is only-build-checked anymore). The result
// SHAPE is unchanged — only class-list membership shifts — but
// correctnessPassRate now means something strictly stronger, so the bump
// prevents a v2→v2 cross-version comparison from misreading the jump as a model
// improvement. New counts: correctness 34 (edit 10 + fix 8 + nav 10 + refactor
// 6), build 0, latency 14 (longproc 4 + longctx 4 + parallel 6).
const TurnSchemaVersion = 3
// TurnRunner runs one benchmark task and reports its outcome plus the captured
// per-turn trace. A non-nil Err means the run failed to execute (process crash);
@@ -89,10 +102,11 @@ type LatencySource struct {
// ClassSummary is the per-class (task group) roll-up.
//
// Passed is the count that passed THIS class's oracle: a correctness pass for
// edit/fix, a build pass for refactor, and always 0 for the no-oracle classes
// (nav/longproc/longctx/parallel). Verified is how many tasks in the class
// carry an oracle (correctness or build); LatencyOnly is how many carry none.
// A latency-only class therefore reports Passed=0, Verified=0, LatencyOnly=Tasks.
// the positive-oracle classes (edit/fix/nav/refactor), and always 0 for the
// no-oracle classes (longproc/longctx/parallel) — the build tier is empty in v3
// (refactor graduated to correctness). Verified is how many tasks in the class
// carry an oracle; LatencyOnly is how many carry none. A latency-only class
// therefore reports Passed=0, Verified=0, LatencyOnly=Tasks.
type ClassSummary struct {
Tasks int `json:"tasks"`
Verified int `json:"verified"`
@@ -107,13 +121,14 @@ type ClassSummary struct {
// Pass/fail is split into three oracle tiers so it cannot be misread as a
// blanket correctness verdict:
// - Correctness (tasksVerified / tasksPassed / correctnessPassRate): tasks
// with a positive oracle (edit/fix). This is the only pass rate that can
// move with model quality.
// - Build-only (buildCheckedTasks / buildPassedTasks / buildPassRate): tasks
// whose oracle is a non-positive build check (refactor `go build`). A pass
// means it compiles, not that the refactor is correct.
// - Latency-only (latencyOnlyTasks): tasks with no oracle (nav/longproc/
// longctx/parallel). They ran for latency and span attribution only and are
// with a positive oracle (edit/fix/nav/refactor). This is the only pass
// rate that can move with model quality.
// - Build-only (buildCheckedTasks / buildPassedTasks / buildPassRate): the
// non-positive build-check tier. Empty in v3 — refactor graduated to
// correctness (its stamped oracle is positive) — so buildCheckedTasks is
// always 0 and buildPassRate is 0. The fields remain for schema stability.
// - Latency-only (latencyOnlyTasks): tasks with no oracle (longproc/longctx/
// parallel). They ran for latency and span attribution only and are
// excluded from every pass rate.
//
// tasksAttempted is still the total number of tasks run across all three tiers.
@@ -432,8 +447,9 @@ func FormatTurnBenchSummary(result TurnBenchResult) string {
"model: " + displayOrUnknown(result.Model),
// The headline separates the three oracle tiers so an exit-0 read-only
// task can never inflate a "pass rate" that reads as correctness:
// correctness (positive oracle, edit/fix), build (non-positive `go build`,
// refactor), and latency-only (no oracle, nav/longproc/longctx/parallel).
// correctness (positive oracle: edit/fix/nav/refactor), build (empty in
// v3 — refactor graduated to correctness), and latency-only (no oracle:
// longproc/longctx/parallel).
fmt.Sprintf("tasks: %d total | correctness %d/%d (%.0f%%) | build %d/%d (%.0f%%) | latency-only %d | %d iter",
result.TasksAttempted,
result.TasksPassed, result.TasksVerified, result.CorrectnessPassRate*100,
@@ -509,11 +525,13 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
// fixture or bleed into a later iteration of the same task. When no
// fixture is configured the agent runs in the caller's cwd as before.
if fixture := strings.TrimSpace(task.WorkspaceFixture); fixture != "" {
copyDir, cerr := copyFixture(fixture)
copyDir, parent, cerr := copyFixture(fixture)
if cerr != nil {
return TurnTaskOutcome{Err: fmt.Errorf("isolate fixture: %w", cerr)}
}
defer os.RemoveAll(copyDir)
// Clean the whole unique parent (which owns copyDir) so the
// per-invocation scratch dir never leaks.
defer os.RemoveAll(parent)
task.WorkspaceFixture = copyDir
}
@@ -577,6 +595,22 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
if outcome.VerifyErr != "" {
return outcome
}
// Stamp the compiler-backed oracle and capture the agent's final answer
// BEFORE running the verification command. Both write into the fixture
// copy (task.WorkspaceFixture, set to copyDir above) so runVerification —
// which uses that same dir as cmd.Dir — sees them. Stamping happens only
// after the agent run so the oracle test can't interfere with the agent's
// own go build/test during the task (e.g. refactor-03's package-zeroapp
// test would break a pre-rename build) and can't be pre-seen or tampered
// with.
if err := stampOracleAndAnswer(task, outBuf.Bytes()); err != nil {
// Preserve the already-parsed trace, trace issue, and wall time: a
// stamp failure is a harness error, but the run still produced a
// trace worth attributing, so mutate the existing outcome rather than
// returning a fresh one that drops Trace/TraceIssue.
outcome.Err = fmt.Errorf("stamp oracle: %w", err)
return outcome
}
if len(task.VerificationCommand) > 0 {
if vOutcome := runVerification(ctx, task); !vOutcome.Passed {
outcome.VerifyErr = strings.TrimSpace(vOutcome.Detail)
@@ -584,7 +618,7 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
}
// A positive oracle (grep/test/build) passed: this is the only path
// that sets Passed. A task with no verificationCommand is latency-only
// (read-only nav/longproc/longctx/parallel): its exit 0 proves the turn
// (read-only longproc/longctx/parallel): its exit 0 proves the turn
// ran, not that the answer was right, so it never reports Passed and the
// harness counts it in latencyOnlyTasks rather than any pass rate.
outcome.Passed = true
@@ -594,22 +628,41 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
}
// copyFixture copies the fixture directory at src into a fresh temp dir and
// returns the copy's path. Used to give each benchmark invocation an isolated
// workspace so mutating tasks (edit/fix/refactor) can't dirty the checked-in
// fixture or a later iteration.
func copyFixture(src string) (string, error) {
// returns the copy's path (dst) plus the unique parent dir that owns it (parent,
// which the caller must RemoveAll to clean up dst and its sibling scratch space
// together). Used to give each benchmark invocation an isolated workspace so
// mutating tasks (edit/fix/refactor) can't dirty the checked-in fixture or a
// later iteration.
//
// The copy is placed two levels BELOW the system temp root: a unique 0700
// parent (created fresh per invocation via os.MkdirTemp, so concurrent runs and
// different users can't share or traverse a predictable scratch dir) and the
// fixture copy beneath it. Go ignores go.mod files in DIRECT children of the
// system temp dir (a hijack guard), which would make the fixture's go.mod —
// and thus every `go test ./...`/`go build ./...` oracle — fail with "cannot
// find main module". Keeping the copy a grandchild of the temp root makes Go
// respect the copied go.mod so the compiler-backed oracles actually run.
func copyFixture(src string) (dst, parent string, err error) {
info, err := os.Stat(src)
if err != nil {
return "", err
return "", "", err
}
if !info.IsDir() {
return "", fmt.Errorf("fixture %q is not a directory", src)
return "", "", fmt.Errorf("fixture %q is not a directory", src)
}
dst, err := os.MkdirTemp("", "zero-turn-fixture-*")
// Unique, 0700-per-invocation parent directly under the temp root. The
// fixture copy is created BENEATH it, so the copy is a grandchild of the
// temp root and its go.mod is respected (see the doc comment above).
parent, err = os.MkdirTemp(os.TempDir(), "zero-turn-bench-*")
if err != nil {
return "", err
return "", "", err
}
err = filepath.WalkDir(src, func(path string, d fs.DirEntry, werr error) error {
dst, err = os.MkdirTemp(parent, "zero-turn-fixture-*")
if err != nil {
os.RemoveAll(parent)
return "", "", err
}
werr := filepath.WalkDir(src, func(path string, d fs.DirEntry, werr error) error {
if werr != nil {
return werr
}
@@ -627,11 +680,47 @@ func copyFixture(src string) (string, error) {
}
return os.WriteFile(target, data, 0o644)
})
if err != nil {
os.RemoveAll(dst)
return "", err
if werr != nil {
os.RemoveAll(parent) // removes dst too
return "", "", werr
}
return dst, nil
return dst, parent, nil
}
// stampOracleAndAnswer writes the compiler-backed oracle test (when configured)
// and the agent's captured final answer into the fixture copy, so the
// verification command can compile the oracle and grep the answer. Both files
// land in task.WorkspaceFixture, which the runner has already set to the
// fixture copy and which runVerification uses as cmd.Dir. The answer file is
// always written (empty when no final event was captured) so a nav answer-oracle
// fails cleanly on a run that produced no answer rather than reading a stale or
// absent file.
//
// A task that needs an oracle — a stamped oracle_test.go OR a verification
// command that reads .zero-answer.txt — but has no workspace fixture is
// rejected: there is nowhere safe to stamp, so runVerification would otherwise
// run in the caller's cwd with the oracle never compiled (or the answer never
// captured), which can read as a false pass. A fixtureless task with no oracle
// and no verification (a pure latency-only run in the caller's cwd) is left
// alone.
func stampOracleAndAnswer(task BenchTask, outBuf []byte) error {
dir := strings.TrimSpace(task.WorkspaceFixture)
if dir == "" {
if task.OracleTest != "" || len(task.VerificationCommand) > 0 {
return fmt.Errorf("stamp oracle: task %q has an oracle or verification but no workspace fixture to stamp into", task.ID)
}
return nil
}
if ot := task.OracleTest; ot != "" {
if err := os.WriteFile(filepath.Join(dir, "oracle_test.go"), []byte(ot), 0o644); err != nil {
return fmt.Errorf("write oracle_test.go: %w", err)
}
}
answer := streamJSONFinalText(outBuf)
if err := os.WriteFile(filepath.Join(dir, ".zero-answer.txt"), []byte(answer), 0o644); err != nil {
return fmt.Errorf("write .zero-answer.txt: %w", err)
}
return nil
}
func buildTurnExecArgs(task BenchTask, rc RunContext, tracePath string, extraArgs []string) []string {
+645 -31
View File
@@ -51,21 +51,26 @@ func cannedTrace(genMs, toolMs int, tokens int64) *trace.TurnTrace {
}
func TestRunTurnBenchAggregation(t *testing.T) {
// v3 tier model: nav and refactor both carry a verificationCommand and are
// NOT in buildOnlyClasses, so they count as correctness; longproc has no
// oracle and is latency-only. The build tier is empty (buildOnlyClasses is
// nil), matching the v3 baseline manifest where refactor graduated from
// build to correctness.
set := TaskSet{
ID: "fake-suite",
Tasks: []BenchTask{
{ID: "t1", Class: "nav", Prompt: "p1"}, // latency-only
{ID: "t2", Class: "nav", Prompt: "p2"}, // latency-only
{ID: "t3", Class: "edit", Prompt: "p3", VerificationCommand: []string{"true"}}, // correctness
{ID: "t4", Class: "refactor", Prompt: "p4", VerificationCommand: []string{"true"}}, // build-only
{ID: "t1", Class: "nav", Prompt: "p1", VerificationCommand: []string{"true"}}, // correctness
{ID: "t2", Class: "edit", Prompt: "p2", VerificationCommand: []string{"true"}}, // correctness
{ID: "t3", Class: "refactor", Prompt: "p3", VerificationCommand: []string{"true"}}, // correctness
{ID: "t4", Class: "longproc", Prompt: "p4"}, // latency-only
},
BuildOnlyClasses: []string{"refactor"},
BuildOnlyClasses: nil,
}
canned := map[string]*trace.TurnTrace{
"t1": cannedTrace(100, 10, 1000),
"t2": cannedTrace(300, 10, 1000),
"t3": cannedTrace(200, 50, 2000),
"t4": cannedTrace(150, 20, 1500),
"t2": cannedTrace(200, 50, 2000),
"t3": cannedTrace(150, 20, 1500),
"t4": cannedTrace(300, 10, 1000),
}
cfg := TurnBenchConfig{
Model: "fake-model",
@@ -77,33 +82,36 @@ func TestRunTurnBenchAggregation(t *testing.T) {
if err != nil {
t.Fatalf("RunTurnBench: %v", err)
}
// 4 tasks attempted; the tier split is 2 latency-only, 1 correctness, 1 build.
// 4 tasks attempted; the v3 tier split is 3 correctness (nav, edit, refactor),
// 1 latency-only (longproc), and 0 build (the build tier is empty).
if result.TasksAttempted != 4 {
t.Fatalf("attempted=%d, want 4", result.TasksAttempted)
}
if result.TasksVerified != 1 || result.TasksPassed != 1 {
t.Fatalf("correctness verified=%d passed=%d, want 1/1", result.TasksVerified, result.TasksPassed)
if result.TasksVerified != 3 || result.TasksPassed != 3 {
t.Fatalf("correctness verified=%d passed=%d, want 3/3", result.TasksVerified, result.TasksPassed)
}
if result.LatencyOnlyTasks != 2 {
t.Fatalf("latencyOnly=%d, want 2", result.LatencyOnlyTasks)
if result.LatencyOnlyTasks != 1 {
t.Fatalf("latencyOnly=%d, want 1", result.LatencyOnlyTasks)
}
if result.BuildCheckedTasks != 1 || result.BuildPassedTasks != 1 {
t.Fatalf("build checked=%d passed=%d, want 1/1", result.BuildCheckedTasks, result.BuildPassedTasks)
if result.BuildCheckedTasks != 0 || result.BuildPassedTasks != 0 {
t.Fatalf("build checked=%d passed=%d, want 0/0 (empty build tier)", result.BuildCheckedTasks, result.BuildPassedTasks)
}
if result.CorrectnessPassRate != 1.0 {
t.Fatalf("correctnessPassRate=%v, want 1.0", result.CorrectnessPassRate)
}
if result.BuildPassRate != 1.0 {
t.Fatalf("buildPassRate=%v, want 1.0", result.BuildPassRate)
// passRate returns 0 for a 0/0 denominator, so an empty build tier reports 0
// (not NaN) and cannot be misread as a perfect build score.
if result.BuildPassRate != 0 {
t.Fatalf("buildPassRate=%v, want 0 (empty build tier is 0/0 -> 0)", result.BuildPassRate)
}
if len(result.CorrectnessClasses) != 1 || result.CorrectnessClasses[0] != "edit" {
t.Fatalf("correctnessClasses=%v, want [edit]", result.CorrectnessClasses)
if len(result.CorrectnessClasses) != 3 || result.CorrectnessClasses[0] != "edit" || result.CorrectnessClasses[1] != "nav" || result.CorrectnessClasses[2] != "refactor" {
t.Fatalf("correctnessClasses=%v, want [edit nav refactor]", result.CorrectnessClasses)
}
if len(result.BuildOnlyClasses) != 1 || result.BuildOnlyClasses[0] != "refactor" {
t.Fatalf("buildOnlyClasses=%v, want [refactor]", result.BuildOnlyClasses)
if len(result.BuildOnlyClasses) != 0 {
t.Fatalf("buildOnlyClasses=%v, want empty (v3: build tier empty)", result.BuildOnlyClasses)
}
if len(result.LatencyOnlyClasses) != 1 || result.LatencyOnlyClasses[0] != "nav" {
t.Fatalf("latencyOnlyClasses=%v, want [nav]", result.LatencyOnlyClasses)
if len(result.LatencyOnlyClasses) != 1 || result.LatencyOnlyClasses[0] != "longproc" {
t.Fatalf("latencyOnlyClasses=%v, want [longproc]", result.LatencyOnlyClasses)
}
if result.SchemaVersion != TurnSchemaVersion {
t.Fatalf("schemaVersion = %d, want %d", result.SchemaVersion, TurnSchemaVersion)
@@ -112,8 +120,8 @@ func TestRunTurnBenchAggregation(t *testing.T) {
t.Fatalf("date = %q", result.Date)
}
// Per-span: generation appears in all four (100+300+200+150=750ms), tool in
// all four (10+10+50+20=90ms). Count must equal the number of tasks * iterations.
// Per-span: generation appears in all four (100+200+150+300=750ms), tool in
// all four (10+50+20+10=90ms). Count must equal the number of tasks * iterations.
gen := result.PerSpan[trace.SpanGeneration]
if gen.Count != 4 {
t.Fatalf("generation count = %d, want 4", gen.Count)
@@ -135,7 +143,7 @@ func TestRunTurnBenchAggregation(t *testing.T) {
t.Fatalf("top latency not ranked by share: %+v", result.TopLatency)
}
// Totals: 4 model requests, 4 tool calls, input tokens 1000+1000+2000+1500=5500.
// Totals: 4 model requests, 4 tool calls, input tokens 1000+2000+1500+1000=5500.
if result.Totals.ModelRequests != 4 {
t.Fatalf("modelRequests = %d, want 4", result.Totals.ModelRequests)
}
@@ -149,10 +157,10 @@ func TestRunTurnBenchAggregation(t *testing.T) {
t.Fatalf("outputTokens = %d, want 2750", result.Totals.OutputTokens)
}
// Per-class tier roll-up: nav is latency-only (0 verified, 2 latency-only),
// edit is correctness (1/1 verified passed), refactor is build (1/1 passed).
// Per-class tier roll-up: nav/edit/refactor are correctness (1/1 verified
// passed, 0 latency-only); longproc is latency-only (0 verified, 1 latency).
nav := result.PerClass["nav"]
if nav.Tasks != 2 || nav.Verified != 0 || nav.Passed != 0 || nav.LatencyOnly != 2 {
if nav.Tasks != 1 || nav.Verified != 1 || nav.Passed != 1 || nav.LatencyOnly != 0 {
t.Fatalf("nav class = %+v", nav)
}
edit := result.PerClass["edit"]
@@ -163,6 +171,10 @@ func TestRunTurnBenchAggregation(t *testing.T) {
if refactor.Tasks != 1 || refactor.Verified != 1 || refactor.Passed != 1 || refactor.LatencyOnly != 0 {
t.Fatalf("refactor class = %+v", refactor)
}
longproc := result.PerClass["longproc"]
if longproc.Tasks != 1 || longproc.Verified != 0 || longproc.Passed != 0 || longproc.LatencyOnly != 1 {
t.Fatalf("longproc class = %+v", longproc)
}
}
// TestRunTurnBenchLatencyOnlyNeverPassed asserts the honesty gate: a task with
@@ -424,17 +436,31 @@ func TestCopyFixtureIsolatesSourceFromMutation(t *testing.T) {
t.Fatalf("read source main.go: %v", err)
}
copyDir, err := copyFixture(src)
copyDir, parent, err := copyFixture(src)
if err != nil {
t.Fatalf("copyFixture: %v", err)
}
defer os.RemoveAll(copyDir)
defer os.RemoveAll(parent) // cleans copyDir too, since copyDir lives under parent
// The copy must be a real copy, not a symlink/alias of the source, and
// mutating it must not touch the source.
if copyDir == src {
t.Fatalf("copyFixture returned the source dir, not a copy: %s", copyDir)
}
// The copy must be a grandchild of the system temp root (parent is a direct
// child of Temp, copyDir is a direct child of parent). Go ignores go.mod in
// direct children of Temp (a hijack guard), so this nesting is what lets the
// compiler-backed oracles actually run in the copy. Compare via filepath.Clean
// because os.TempDir() may carry a trailing slash (macOS $TMPDIR ends in
// "/T/") while filepath.Dir strips it; the raw != would false-fail on macOS
// even though the parent is genuinely a direct child.
tempRoot := filepath.Clean(os.TempDir())
if filepath.Dir(parent) != tempRoot {
t.Fatalf("parent %q is not a direct child of temp root %q", parent, tempRoot)
}
if filepath.Dir(copyDir) != parent {
t.Fatalf("copyDir %q is not nested under parent %q", copyDir, parent)
}
if err := os.WriteFile(filepath.Join(copyDir, "main.go"), []byte("package main\n\n// mutated\nfunc main() {}\n"), 0o644); err != nil {
t.Fatalf("mutate copy: %v", err)
}
@@ -453,3 +479,591 @@ func TestCopyFixtureIsolatesSourceFromMutation(t *testing.T) {
t.Fatalf("file added to copy appeared in source: %v", err)
}
}
// TestRunTurnBenchBuildOnlyMechanism exercises the build-only tier code path in
// isolation. The v3 baseline manifest leaves this tier empty (refactor graduated
// to correctness), but the mechanism is still supported for future use, so a
// focused test keeps it covered: a class listed in buildOnlyClasses with a
// verificationCommand counts in buildCheckedTasks/buildPassedTasks — never in
// tasksVerified/correctnessPassRate — and a build-pass cannot leak into the
// correctness number.
func TestRunTurnBenchBuildOnlyMechanism(t *testing.T) {
set := TaskSet{
ID: "build-suite",
Tasks: []BenchTask{
{ID: "b1", Class: "buildcheck", Prompt: "p", VerificationCommand: []string{"true"}},
},
BuildOnlyClasses: []string{"buildcheck"},
}
canned := map[string]*trace.TurnTrace{"b1": cannedTrace(120, 30, 900)}
result, err := RunTurnBench(context.Background(), set, TurnBenchConfig{
Model: "fake-model",
Runner: fakeTurnRunner(canned),
Now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
})
if err != nil {
t.Fatalf("RunTurnBench: %v", err)
}
if result.BuildCheckedTasks != 1 || result.BuildPassedTasks != 1 {
t.Fatalf("build checked=%d passed=%d, want 1/1", result.BuildCheckedTasks, result.BuildPassedTasks)
}
if result.BuildPassRate != 1.0 {
t.Fatalf("buildPassRate=%v, want 1.0", result.BuildPassRate)
}
// A build-only task must NOT count toward correctness.
if result.TasksVerified != 0 || result.TasksPassed != 0 {
t.Fatalf("build-only leaked into correctness: verified=%d passed=%d, want 0/0",
result.TasksVerified, result.TasksPassed)
}
if result.CorrectnessPassRate != 0 {
t.Fatalf("correctnessPassRate=%v, want 0 with only build-only tasks", result.CorrectnessPassRate)
}
if len(result.BuildOnlyClasses) != 1 || result.BuildOnlyClasses[0] != "buildcheck" {
t.Fatalf("buildOnlyClasses=%v, want [buildcheck]", result.BuildOnlyClasses)
}
if len(result.CorrectnessClasses) != 0 {
t.Fatalf("correctnessClasses=%v, want empty", result.CorrectnessClasses)
}
bc := result.PerClass["buildcheck"]
if bc.Tasks != 1 || bc.Verified != 1 || bc.Passed != 1 || bc.LatencyOnly != 0 {
t.Fatalf("buildcheck class = %+v, want 1/1 verified passed", bc)
}
}
// TestTurnSchemaVersion3 pins the schema bump. v3 records the tier
// reclassification (refactor structural-positive and nav answer-oracles moved
// into correctnessPassRate), so a v2->v3 cross-version comparison cannot
// silently misread the jump as a model improvement — exactly the misread the
// tier system exists to prevent.
func TestTurnSchemaVersion3(t *testing.T) {
if TurnSchemaVersion != 3 {
t.Fatalf("TurnSchemaVersion = %d, want 3", TurnSchemaVersion)
}
}
// TestStreamJSONFinalTextExtractsAnswer, TestStreamJSONFinalTextEmptyWhenNoFinal,
// and TestStreamJSONFinalTextLastWins cover the pure capture helper that the
// nav answer-oracle depends on: it scans stream-json for the terminal "final"
// event and returns its text, "" when none was emitted, and the last when
// multiple appear (the success path emits exactly one, but the incomplete path
// can also emit one, so last-wins matches streamJSONExitCode's tie-break).
func TestStreamJSONFinalTextExtractsAnswer(t *testing.T) {
out := []byte(`{"type":"text","text":"thinking..."}
{"type":"final","text":"the keys are port, name, and retries"}
{"type":"run_end","exitCode":0}
`)
if got := streamJSONFinalText(out); got != "the keys are port, name, and retries" {
t.Fatalf("streamJSONFinalText = %q, want the final text", got)
}
}
func TestStreamJSONFinalTextEmptyWhenNoFinal(t *testing.T) {
out := []byte(`{"type":"text","text":"hi"}
{"type":"run_end","exitCode":0}
`)
if got := streamJSONFinalText(out); got != "" {
t.Fatalf("streamJSONFinalText = %q, want empty when no final event", got)
}
}
func TestStreamJSONFinalTextLastWins(t *testing.T) {
out := []byte(`{"type":"final","text":"first"}
{"type":"final","text":"second"}
`)
if got := streamJSONFinalText(out); got != "second" {
t.Fatalf("streamJSONFinalText = %q, want second (last final wins)", got)
}
}
// loadBaselineTask returns the named task from the checked-in baseline manifest.
// The stub-binary oracle tests below run the REAL manifest oracles against the
// REAL fixtures through the REAL NewTurnExecRunner, so a manifest edit that
// breaks an oracle is caught here rather than only in production.
func loadBaselineTask(t *testing.T, id string) BenchTask {
t.Helper()
set, err := LoadTaskSet(filepath.Join("manifests", "baseline.json"))
if err != nil {
t.Fatalf("LoadTaskSet: %v", err)
}
for _, task := range set.Tasks {
if task.ID == id {
return task
}
}
t.Fatalf("manifest has no task %q", id)
return BenchTask{}
}
// runTurnStub runs one manifest task through the production NewTurnExecRunner
// with a stub "zero" binary whose body is a POSIX sh script (writeExecStub
// skips on Windows). The stub is invoked with cmd.Dir set to the fixture copy,
// so it can both emit canned stream-json AND mutate the copy (apply or omit the
// fix) before the runner stamps the oracle and runs the verification command.
func runTurnStub(t *testing.T, task BenchTask, stubBody string) TurnTaskOutcome {
t.Helper()
stub := writeExecStub(t, stubBody)
return NewTurnExecRunner(stub)(context.Background(), task, RunContext{Model: "fake-model"})
}
// assertVerifyFailed asserts an outcome failed specifically because the oracle
// rejected the work — Passed is false, there is no harness error (Err nil), and
// VerifyErr carries the surfaced failure detail. This is stronger than merely
// checking Passed=false: it distinguishes a genuine oracle rejection from a
// harness error (Err set) or a silent non-pass (Passed=false with no VerifyErr,
// which would mean the failure never went through verification at all). The
// gating stubs all emit a clean run_end (exitCode 0), so a non-empty VerifyErr
// here can only come from the verification command failing.
func assertVerifyFailed(t *testing.T, label string, outcome TurnTaskOutcome) {
t.Helper()
if outcome.Err != nil {
t.Fatalf("%s: want a verify fail, got harness error: %v", label, outcome.Err)
}
if outcome.Passed {
t.Fatalf("%s: want verify fail, got Passed=true", label)
}
if strings.TrimSpace(outcome.VerifyErr) == "" {
t.Fatalf("%s: want verify fail, but VerifyErr is empty (failure did not come from verification): %+v", label, outcome)
}
}
// --- Gating tests: the oracle FAILS the wrong thing (no-op / wrong answer) ---
// TestStampedOracleRejectsNoOpRefactor is the core #712 fix: refactor used to
// live in the build tier where a no-op `go build ./...` passed. Now a no-op
// agent (the stub emits a clean run_end but touches nothing) leaves the fixture
// with no formatGreeting helper, so the stamped `var _ = formatGreeting` fails
// to compile and `go test ./...` fails — the task is failed, not passed.
func TestStampedOracleRejectsNoOpRefactor(t *testing.T) {
task := loadBaselineTask(t, "refactor-01")
outcome := runTurnStub(t, task, `echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "no-op refactor", outcome)
}
// TestStampedOracleRejectsMissingField proves edit-03's oracle is structural: a
// no-op leaves Config with no Label field, so `var _ string = Config{}.Label`
// fails to compile and the task fails.
func TestStampedOracleRejectsMissingField(t *testing.T) {
task := loadBaselineTask(t, "edit-03")
outcome := runTurnStub(t, task, `echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "missing-field edit-03", outcome)
}
// TestNavAnswerOracleRejectsWrongAnswer proves the nav oracle greps the CAPTURED
// answer, not the raw stream: the stub emits a final answer missing the
// required keys, the harness writes it to .zero-answer.txt, and the compound
// grep fails — so a plausible-but-wrong answer cannot pass nav-09.
func TestNavAnswerOracleRejectsWrongAnswer(t *testing.T) {
task := loadBaselineTask(t, "nav-09")
outcome := runTurnStub(t, task, `echo '{"type":"final","text":"the keys are foo and bar"}'
echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "wrong nav-09 answer", outcome)
}
// TestNavNoFinalTextFails proves a run that produced no answer fails nav: with
// no "final" event, .zero-answer.txt is empty and the grep finds nothing.
func TestNavNoFinalTextFails(t *testing.T) {
task := loadBaselineTask(t, "nav-09")
outcome := runTurnStub(t, task, `echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-09 no final", outcome)
}
// TestEdit05RejectsRewordedDebugPrint proves edit-05's oracle catches a reword,
// not just a deletion: changing `fmt.Println("debug: starting")` to
// `fmt.Println("starting up")` leaves a fmt.Println string-literal call, so the
// stamped oracle's `strings.Contains(main.go, fmt.Println(")` check fails. A
// plain `! grep 'debug: starting'` would have rubber-stamped this reword.
func TestEdit05RejectsRewordedDebugPrint(t *testing.T) {
task := loadBaselineTask(t, "edit-05")
outcome := runTurnStub(t, task, `sed 's/debug: starting/starting up/' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "reworded edit-05 debug print", outcome)
}
// TestEdit05RejectsRespelledDebugPrint proves the second edit-05 check (the
// message text "debug: starting" must be absent) catches a respell onto a
// different call form that the fmt.Println(" string-literal check misses: the
// agent rewrites the line to fmt.Printf("debug: starting\n"), which still prints
// the message but is no longer a fmt.Println string-literal. The
// strings.Contains("debug: starting") assertion fails the task.
func TestEdit05RejectsRespelledDebugPrint(t *testing.T) {
task := loadBaselineTask(t, "edit-05")
outcome := runTurnStub(t, task, `sed 's/fmt.Println("debug: starting")/fmt.Printf("debug: starting\\n")/' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "respelled edit-05 debug print", outcome)
}
// TestRefactor05RejectsStubbedCaller proves refactor-05's behavioral oracle
// catches a wrong edit the negative grep alone would pass: the agent deletes
// Wrapper (so ! grep 'func Wrapper' succeeds) but stubs greetWrapped to return
// "" instead of inlining. The stamped TestGreetWrapped runs greetWrapped("x"),
// gets "" not "hello, x", and fails — so deleting the target along with its
// caller (or stubbing the caller) can no longer pass the oracle.
func TestRefactor05RejectsStubbedCaller(t *testing.T) {
task := loadBaselineTask(t, "refactor-05")
outcome := runTurnStub(t, task, `sed -e 's/return Wrapper(name)/return ""/' -e '/^func Wrapper(name string) string {/,/^}/d' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "stubbed refactor-05 caller", outcome)
}
// TestRefactor04RejectsBareMapKept proves refactor-04's file-reading oracle
// catches a no-op that the old exact-text shell grep would miss: the agent adds
// an unused `type Stats map[string]int` (so the compile-time `var _ Stats`
// passes) but keeps the bare `var stats = map[string]int{}` under the original
// spacing. The spacing-tolerant regex in TestStatsNamedType still matches the
// bare-map declaration, so the task fails — Record/Lookup never moved onto the
// named type.
func TestRefactor04RejectsBareMapKept(t *testing.T) {
task := loadBaselineTask(t, "refactor-04")
outcome := runTurnStub(t, task, `sed 's/type Config struct {/type Stats map[string]int\n\ntype Config struct {/' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "refactor-04 bare-map kept", outcome)
}
// TestEdit01IgnoresDocComment proves edit-01's scoped negative grep does NOT
// false-fail a correct rename that leaves the doc comment mentioning the old
// name. The stub renames the declaration (`MaxRetries =` -> `RetryLimit =`) but
// leaves `// MaxRetries is the maximum...`; the scoped `! grep -RIn 'MaxRetries ='`
// matches declaration sites only, so the comment is ignored, `var _ = RetryLimit`
// compiles, and the task passes.
func TestEdit01IgnoresDocComment(t *testing.T) {
task := loadBaselineTask(t, "edit-01")
outcome := runTurnStub(t, task, `sed 's/const MaxRetries = 3/const RetryLimit = 3/' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct rename with kept doc comment should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("edit-01 oracle must not false-fail a correct rename that leaves the doc comment: %+v", outcome)
}
}
// --- Satisfiable tests: the oracle PASSES the right thing (real fix applied) ---
// TestStampedOraclePassesWhenRefactorHappened proves the refactor-01 oracle is
// not an always-fail gate: when the stub applies the real refactor (extract
// formatGreeting, call it from both callers), `var _ = formatGreeting` compiles
// and the `hello, %s` grep-count is 1 (only the helper holds the literal), so
// the task passes.
func TestStampedOraclePassesWhenRefactorHappened(t *testing.T) {
task := loadBaselineTask(t, "refactor-01")
outcome := runTurnStub(t, task, `sed -e 's/return fmt.Sprintf("hello, %s", c.Name)/return formatGreeting(c.Name)/' -e 's/return fmt.Sprintf("hello, %s", name)/return formatGreeting(name)/' main.go > .zero-tmp
cat >> .zero-tmp <<'EOF'
func formatGreeting(name string) string {
return fmt.Sprintf("hello, %s", name)
}
EOF
mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("real refactor should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("refactor-01 oracle must pass when formatGreeting is extracted: %+v", outcome)
}
}
// TestEdit03PassesWhenFieldAdded proves edit-03's oracle passes when the field
// is actually added: Config gets a Label string field, so `var _ string =
// Config{}.Label` compiles and the task passes.
func TestEdit03PassesWhenFieldAdded(t *testing.T) {
task := loadBaselineTask(t, "edit-03")
outcome := runTurnStub(t, task, `awk '/Name string/ && !d {print; print "Label string"; d=1; next} {print}' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("real field-add should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("edit-03 oracle must pass when Label is added: %+v", outcome)
}
}
// TestNav09PassesWithCorrectAnswer proves the nav-09 oracle passes when the
// captured answer names all three config keys.
func TestNav09PassesWithCorrectAnswer(t *testing.T) {
task := loadBaselineTask(t, "nav-09")
outcome := runTurnStub(t, task, `echo '{"type":"final","text":"config.json has three keys: port, name, and retries."}'
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct nav answer should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("nav-09 oracle must pass when the answer names port/name/retries: %+v", outcome)
}
}
// TestRefactor05PassesWhenInlined proves refactor-05's oracle is not an
// always-fail gate: when the stub applies the real inline (greetWrapped calls
// GreetByName directly, Wrapper removed), TestGreetWrapped gets "hello, x" and
// the ! grep 'func Wrapper' check passes, so the task passes.
func TestRefactor05PassesWhenInlined(t *testing.T) {
task := loadBaselineTask(t, "refactor-05")
outcome := runTurnStub(t, task, `sed -e 's/return Wrapper(name)/return GreetByName(name)/' -e '/^func Wrapper(name string) string {/,/^}/d' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("real inline should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("refactor-05 oracle must pass when Wrapper is inlined into greetWrapped: %+v", outcome)
}
}
// TestRefactor04PassesWhenTyped proves refactor-04's oracle passes when the
// bare map is actually replaced by the named type: the stub introduces
// `type Stats map[string]int` and retypes the var to `var stats = Stats{}`,
// so the bare-map regex no longer matches, Stats is referenced, `var _ Stats`
// compiles, and go test passes.
func TestRefactor04PassesWhenTyped(t *testing.T) {
task := loadBaselineTask(t, "refactor-04")
outcome := runTurnStub(t, task, `sed -e 's/type Config struct {/type Stats map[string]int\n\ntype Config struct {/' -e 's/var stats = map\[string\]int{}/var stats = Stats{}/' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("real typed refactor should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("refactor-04 oracle must pass when stats is typed as Stats: %+v", outcome)
}
}
// TestStampOracleRejectsFixturelessOracleTask covers the fail-closed path in
// stampOracleAndAnswer: a task that carries an oracle (a stamped OracleTest or a
// verification command) but has no workspace fixture has nowhere safe to stamp,
// so it must error rather than let runVerification run in the caller's cwd with
// the oracle never compiled (or the answer never captured). A fixtureless task
// with no oracle and no verification (a pure latency-only run in the caller's
// cwd) is left alone. This is a direct unit test (no stub), so it runs on all
// platforms.
func TestStampOracleRejectsFixturelessOracleTask(t *testing.T) {
// OracleTest set, no verificationCommand: isolates the `OracleTest != ""`
// reject branch so a regression that only checked VerificationCommand would
// fail here (the both-set case below would still pass it).
withOracleTest := BenchTask{
ID: "t",
OracleTest: "package x\n\nvar _ = Missing\n",
}
if err := stampOracleAndAnswer(withOracleTest, nil); err == nil {
t.Fatal("fixtureless task with an OracleTest must be rejected, got nil error")
}
// verificationCommand set, no OracleTest: isolates the other reject branch.
withVerify := BenchTask{
ID: "t",
VerificationCommand: []string{"bash", "-c", "grep x .zero-answer.txt"},
}
if err := stampOracleAndAnswer(withVerify, nil); err == nil {
t.Fatal("fixtureless task with a verificationCommand must be rejected, got nil error")
}
// Both set: the `||` rejects, and this guards against a regression that
// inverted the logic to `&&`.
withBoth := BenchTask{
ID: "t",
OracleTest: "package x\n\nvar _ = Missing\n",
VerificationCommand: []string{"go", "test", "./..."},
}
if err := stampOracleAndAnswer(withBoth, nil); err == nil {
t.Fatal("fixtureless task with both an OracleTest and a verificationCommand must be rejected, got nil error")
}
latencyOnly := BenchTask{ID: "t"}
if err := stampOracleAndAnswer(latencyOnly, nil); err != nil {
t.Fatalf("fixtureless latency-only task must not be rejected, got: %v", err)
}
}
// --- Count-oracle tests: anchored `^count: N$` with the exact expected count ---
// TestNav01CountOracleAcceptsExactCount proves nav-01's anchored count oracle
// passes when the answer names the files AND states the exact count (5 — the
// fixture has README.md, config.json, go.mod, main.go, main_test.go) on its own
// line. printf emits the JSON with a literal \n escape so the captured answer
// has "count: 5" on its own line.
func TestNav01CountOracleAcceptsExactCount(t *testing.T) {
task := loadBaselineTask(t, "nav-01")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"main.go, config.json, go.mod, README.md, main_test.go\ncount: 5"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct nav-01 answer should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("nav-01 oracle must pass when the answer names the files and states count: 5: %+v", outcome)
}
}
// TestNav01CountOracleRejectsWrongCount proves the anchored count oracle rejects
// a wrong count: the file names are all present, but "count: 3" does not match
// `^count: 5$`, so the task fails at verification.
func TestNav01CountOracleRejectsWrongCount(t *testing.T) {
task := loadBaselineTask(t, "nav-01")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"main.go, config.json, go.mod, README.md, main_test.go\ncount: 3"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-01 wrong count", outcome)
}
// TestNav04CountOracleRejectsSubstring proves the line anchor rejects a
// substring: "the count: 1 is the number" contains "count: 1" (the right
// value) but the line does not start with "count:", so `^count:[[:space:]]*1$`
// does not match. A pre-anchor oracle (`count:\s*1`) would have rubber-stamped
// this.
func TestNav04CountOracleRejectsSubstring(t *testing.T) {
task := loadBaselineTask(t, "nav-04")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"the count: 1 is the number of test functions"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-04 substring count", outcome)
}
// TestNav04CountOracleAcceptsExactCount proves the nav-04 oracle passes for a
// real answer: the count is anchored on its own line AND the answer names the
// one Test* function (TestGreet, from main_test.go). The fixture's ground truth
// is exactly one test function, so count: 1 is right, and naming TestGreet is
// the inspection-proof that the agent actually read main_test.go.
func TestNav04CountOracleAcceptsExactCount(t *testing.T) {
task := loadBaselineTask(t, "nav-04")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"There is one test function: TestGreet in main_test.go.\ncount: 1"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct nav-04 answer should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("nav-04 oracle must pass when the answer names TestGreet and states count: 1 on its own line: %+v", outcome)
}
}
// TestNav04CountOracleRejectsZeroGuess is the count=0 gameability gate: an agent
// that never opens the workspace and always emits "count: 0" used to pass nav-04
// when the ground truth was 0. The fixture now has one real Test* function, so
// the ground truth is 1 and a clean "count: 0" fails.
func TestNav04CountOracleRejectsZeroGuess(t *testing.T) {
task := loadBaselineTask(t, "nav-04")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 0"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-04 zero-guess", outcome)
}
// TestNav04CountOracleRejectsBlindCountOne is the count=1 gameability gate: once
// the ground truth became 1, an agent that never opens the workspace and always
// emits "count: 1" would pass a count-only oracle. nav-04's oracle also requires
// the answer to name the test function (TestGreet), so a bare "count: 1" with no
// named fact fails — proving the blind-count-one surface is closed.
func TestNav04CountOracleRejectsBlindCountOne(t *testing.T) {
task := loadBaselineTask(t, "nav-04")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 1"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-04 blind-count-one", outcome)
}
// TestNav05CountOracleAcceptsExactCount proves nav-05's oracle passes for a real
// answer: the count is anchored on its own line AND the answer names the file
// holding the one TODO (main.go). Naming main.go is the inspection-proof that the
// agent actually located the TODO rather than guessing the count.
func TestNav05CountOracleAcceptsExactCount(t *testing.T) {
task := loadBaselineTask(t, "nav-05")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"One TODO in main.go.\ncount: 1"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct nav-05 answer should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("nav-05 oracle must pass when the answer names main.go and states count: 1: %+v", outcome)
}
}
// TestNav05CountOracleRejectsZeroGuess is nav-05's count=0 gameability gate: the
// fixture now has one real TODO, so a clean "count: 0" (the always-guess-zero
// answer) fails.
func TestNav05CountOracleRejectsZeroGuess(t *testing.T) {
task := loadBaselineTask(t, "nav-05")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 0"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-05 zero-guess", outcome)
}
// TestNav05CountOracleRejectsBlindCountOne is nav-05's count=1 gameability gate:
// once the ground truth became 1, an agent that never opens the workspace and
// always emits "count: 1" would pass a count-only oracle. nav-05's oracle also
// requires the answer to name the file (main.go), so a bare "count: 1" with no
// named file fails — proving the blind-count-one surface is closed.
func TestNav05CountOracleRejectsBlindCountOne(t *testing.T) {
task := loadBaselineTask(t, "nav-05")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 1"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-05 blind-count-one", outcome)
}
// TestNav05CountOracleRejectsSubstring proves the line anchor still bites even
// when the named fact is present: "the count: 1 is the number of TODOs in
// main.go" names main.go and contains "count: 1" (the right value), but the
// count does not stand on its own line, so `^count:[[:space:]]*1$` does not
// match and the oracle fails. A pre-anchor oracle (`count:\s*1`) would have
// rubber-stamped this. Mirrors TestNav04CountOracleRejectsSubstring.
func TestNav05CountOracleRejectsSubstring(t *testing.T) {
task := loadBaselineTask(t, "nav-05")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"the count: 1 is the number of TODOs in main.go"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-05 substring count", outcome)
}
// TestNav08CountOracleAcceptsStdlibAnswer proves nav-08's oracle passes when the
// agent enumerated the imports across BOTH .go files and named the stdlib ones
// (fmt from main.go, testing from main_test.go): the answer states the only
// imports are stdlib so there are 0 third-party, with "count: 0" on its own
// line. The fmt+testing requirement is the inspection-proof — it forces the
// agent to have read all the files rather than blindly emitting "count: 0".
func TestNav08CountOracleAcceptsStdlibAnswer(t *testing.T) {
task := loadBaselineTask(t, "nav-08")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"The only imports are fmt (main.go) and testing (main_test.go), both standard library, so there are no third-party imports.\ncount: 0"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct nav-08 answer should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("nav-08 oracle must pass when the answer names fmt+testing and states count: 0: %+v", outcome)
}
}
// TestNav08CountOracleRejectsAnswerWithoutFmt proves the inspection-proof bites:
// an answer that states the right count (0) but never names the stdlib imports it
// examined ("No third-party imports. count: 0") fails, so an agent that didn't
// actually look at the imports can't pass by guessing the count alone.
func TestNav08CountOracleRejectsAnswerWithoutFmt(t *testing.T) {
task := loadBaselineTask(t, "nav-08")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"No third-party imports.\ncount: 0"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-08 no-fmt", outcome)
}
// TestNav08CountOracleRejectsFmtOnlyNoTesting proves the tighter inspection-proof:
// an answer that names fmt (from main.go) but misses testing (from main_test.go)
// fails, so an agent that only read main.go — or a guesser that names the one
// most-common stdlib import blind — can't pass. The agent must enumerate every
// file's imports, which is what the prompt asks for.
func TestNav08CountOracleRejectsFmtOnlyNoTesting(t *testing.T) {
task := loadBaselineTask(t, "nav-08")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"The only import is fmt, which is standard library. count: 0"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-08 fmt-only-no-testing", outcome)
}
+41 -6
View File
@@ -23,6 +23,15 @@ type bashTool struct {
scope PathScope
}
func (bashTool) outputCategory(args map[string]any) outputCategory {
command, _ := bashCommandArg(args)
return shellOutputCategory(command)
}
func bashCommandArg(args map[string]any) (string, error) {
return aliasedStringArg(args, []string{"command", "cmd", "script", "shell"}, "", true, false)
}
func NewBashTool(workspaceRoot string) Tool {
return NewScopedBashTool(workspaceRoot, nil)
}
@@ -63,15 +72,19 @@ func NewScopedBashTool(workspaceRoot string, scope PathScope) Tool {
}
func (tool bashTool) Run(ctx context.Context, args map[string]any) Result {
return tool.run(ctx, args, nil)
return tool.run(ctx, args, nil, true)
}
func (tool bashTool) RunWithSandbox(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine) Result {
return tool.run(ctx, args, engine)
return tool.run(ctx, args, engine, true)
}
func (tool bashTool) run(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine) Result {
commandText, err := aliasedStringArg(args, []string{"command", "cmd", "script", "shell"}, "", true, false)
func (tool bashTool) RunWithOptions(ctx context.Context, args map[string]any, options RunOptions) Result {
return tool.run(ctx, args, options.Sandbox, false)
}
func (tool bashTool) run(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine, directBudget bool) Result {
commandText, err := bashCommandArg(args)
if err != nil {
return errorResult("Error: Invalid arguments for bash: " + err.Error())
}
@@ -172,7 +185,7 @@ func (tool bashTool) run(ctx context.Context, args map[string]any, engine *zeroS
}
}
markLikelySandboxDenial(meta, plan, exitCode, stdoutText, stderrText)
outText, errText, truncated := budgetBashCapture(stdoutText, stdout.total, stderrText, stderrTotal, meta)
outText, errText, truncated := prepareBashOutput(stdoutText, stdout.total, stderrText, stderrTotal, meta, directBudget)
return Result{
Status: StatusError,
Output: formatBashOutputWithShellHint(outText, errText, exitCode, meta),
@@ -182,7 +195,7 @@ func (tool bashTool) run(ctx context.Context, args map[string]any, engine *zeroS
}
markLikelySandboxDenial(meta, plan, exitCode, stdoutText, stderrText)
outText, errText, truncated := budgetBashCapture(stdoutText, stdout.total, stderrText, stderrTotal, meta)
outText, errText, truncated := prepareBashOutput(stdoutText, stdout.total, stderrText, stderrTotal, meta, directBudget)
if meta[SandboxLikelyDeniedMeta] == "true" {
return Result{
Status: StatusError,
@@ -415,6 +428,28 @@ func budgetBashOutput(stdout string, stderr string, meta map[string]string) (str
return budgetBashCapture(stdout, len(stdout), stderr, len(stderr), meta)
}
// prepareBashOutput keeps direct callers on the established positional budget.
// Registry calls retain the existing bounded capture but leave final semantic
// reduction and spill creation to the post-redaction registry boundary.
func prepareBashOutput(out string, outTotal int, errStr string, errTotal int, meta map[string]string, directBudget bool) (string, string, bool) {
if directBudget {
return budgetBashCapture(out, outTotal, errStr, errTotal, meta)
}
outText := sectionWithCaptureGap(out, outTotal)
errText := sectionWithCaptureGap(errStr, errTotal)
truncated := outTotal > len(out) || errTotal > len(errStr)
if meta != nil {
meta["raw_bytes"] = strconv.Itoa(outTotal + errTotal)
meta["emitted_bytes"] = strconv.Itoa(len(outText) + len(errText))
meta["estimated_tokens"] = strconv.Itoa(estimatedTokensFromBytes(len(outText) + len(errText)))
if truncated {
meta["truncated"] = "true"
meta["truncation_reason"] = "capture_budget"
}
}
return outText, errText, truncated
}
// budgetBashCapture is budgetBashOutput for the streaming-capture path: outTotal
// and errTotal are the true byte counts (from boundedBuffer.total), which may
// exceed the retained strings when the middle was dropped during capture. Meta's
+12
View File
@@ -37,6 +37,18 @@ func TestBudgetBashOutputSmallPassesThrough(t *testing.T) {
}
}
func TestPrepareBashOutputDefersRegistrySemanticBudget(t *testing.T) {
output := "START\n" + strings.Repeat("progress\n", 8_000) + "ERROR: retained\nEND"
registryOutput, _, registryTruncated := prepareBashOutput(output, len(output), "", 0, map[string]string{}, false)
if registryTruncated || registryOutput != output {
t.Fatalf("registry preparation must preserve capture-bounded output for semantic reduction: truncated=%t bytes=%d", registryTruncated, len(registryOutput))
}
directOutput, _, directTruncated := prepareBashOutput(output, len(output), "", 0, map[string]string{}, true)
if !directTruncated || len(directOutput) > bashOutputBudgetBytes+512 {
t.Fatalf("direct path must preserve legacy positional budget: truncated=%t bytes=%d", directTruncated, len(directOutput))
}
}
// Oversized stdout is truncated head+tail: both the first and last lines survive,
// the middle is dropped behind a marker, meta is flagged, and the captured text
// is spilled to a re-readable file.
+40 -4
View File
@@ -300,6 +300,18 @@ func TestInteractiveClassifications(t *testing.T) {
func TestReadOnlyClassifications(t *testing.T) {
root := t.TempDir()
// PR6 audited concurrent-safe pure reads (mutex-guarded FileTracker,
// independent scan I/O, independent HTTP). Others stay sequential until
// their shared state is proven safe under concurrent Run.
threadSafeReads := map[string]bool{
"read_file": true,
"read_minified_file": true,
"list_directory": true,
"glob": true,
"grep": true,
"skill": true,
"web_fetch": true,
}
for _, name := range []string{"read_file", "read_minified_file", "list_directory", "glob", "grep", "lsp_navigate", "skill", "web_fetch", "web_search", "tool_search"} {
var found Tool
for _, tool := range BuiltinCatalog(root) {
@@ -315,14 +327,38 @@ func TestReadOnlyClassifications(t *testing.T) {
if caps.Effect != EffectReadOnly {
t.Errorf("%s effect = %v, want ReadOnly", name, caps.Effect)
}
// Current catalog intentionally keeps read tools non-thread-safe
// until each is audited for shared mutable state (FileTracker, etc.).
if caps.ThreadSafe {
t.Errorf("%s ThreadSafe=true without explicit audit", name)
wantSafe := threadSafeReads[name]
if caps.ThreadSafe != wantSafe {
t.Errorf("%s ThreadSafe=%v, want %v", name, caps.ThreadSafe, wantSafe)
}
}
}
func TestScopedScanResourceKeys(t *testing.T) {
// Workspace-wide scan → no keys (do not false-conflict every concurrent scan).
if keys := scopedScanResourceKeys(nil); keys != nil {
t.Fatalf("nil args = %v, want nil", keys)
}
if keys := scopedScanResourceKeys(map[string]any{}); keys != nil {
t.Fatalf("empty args = %v, want nil", keys)
}
if keys := scopedScanResourceKeys(map[string]any{"cwd": "."}); keys != nil {
t.Fatalf("cwd=. = %v, want nil", keys)
}
if keys := scopedScanResourceKeys(map[string]any{"path": "."}); keys != nil {
t.Fatalf("path=. = %v, want nil", keys)
}
// Scoped path → directory: key (glob uses cwd; grep uses path).
keys := scopedScanResourceKeys(map[string]any{"cwd": "internal/tools"})
if len(keys) != 1 || keys[0] != ResourceKeyDirectory+NormalizeResourcePath("internal/tools") {
t.Fatalf("cwd scoped = %v", keys)
}
keys = scopedScanResourceKeys(map[string]any{"path": "./pkg/../pkg/api"})
if len(keys) != 1 || keys[0] != ResourceKeyDirectory+NormalizeResourcePath("./pkg/../pkg/api") {
t.Fatalf("path scoped = %v", keys)
}
}
func TestValidateBuiltinCatalogRejectsUnknownRaw(t *testing.T) {
// Prove the gate is not a rubber stamp: raw Unknown+ThreadSafe is rejected.
problems := ValidateCapabilities("ghost", ToolCapabilities{Effect: EffectUnknown, ThreadSafe: true})
+32 -7
View File
@@ -427,6 +427,15 @@ type execCommandTool struct {
manager *execSessionManager
}
func (execCommandTool) outputCategory(args map[string]any) outputCategory {
command, _ := execCommandArg(args)
return shellOutputCategory(command)
}
func execCommandArg(args map[string]any) (string, error) {
return aliasedStringArg(args, []string{"cmd", "command", "script", "shell"}, "", true, false)
}
func NewExecCommandTool(workspaceRoot string, manager *execSessionManager) Tool {
return NewScopedExecCommandTool(workspaceRoot, nil, manager)
}
@@ -475,11 +484,15 @@ func NewScopedExecCommandTool(workspaceRoot string, scope PathScope, manager *ex
}
func (tool execCommandTool) Run(ctx context.Context, args map[string]any) Result {
return tool.run(ctx, args, nil)
return tool.run(ctx, args, nil, true)
}
func (tool execCommandTool) RunWithSandbox(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine) Result {
return tool.run(ctx, args, engine)
return tool.run(ctx, args, engine, true)
}
func (tool execCommandTool) RunWithOptions(ctx context.Context, args map[string]any, options RunOptions) Result {
return tool.run(ctx, args, options.Sandbox, false)
}
func (tool execCommandTool) ExecSessions() []ExecSessionSnapshot {
@@ -494,8 +507,8 @@ func (tool execCommandTool) StopAllExecSessions() []int {
return tool.manager.stopAll()
}
func (tool execCommandTool) run(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine) Result {
commandText, err := aliasedStringArg(args, []string{"cmd", "command", "script", "shell"}, "", true, false)
func (tool execCommandTool) run(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine, directBudget bool) Result {
commandText, err := execCommandArg(args)
if err != nil {
return errorResult("Error: Invalid arguments for exec_command: " + err.Error())
}
@@ -550,7 +563,7 @@ func (tool execCommandTool) run(ctx context.Context, args map[string]any, engine
if exited {
tool.manager.remove(session.id)
}
return execToolResult(execToolResultInput{
return execToolResultWithBudget(execToolResultInput{
commandText: commandText,
output: output,
outputBufferTruncated: outputTruncated,
@@ -561,7 +574,7 @@ func (tool execCommandTool) run(ctx context.Context, args map[string]any, engine
tty: session.tty,
plan: session.plan,
maxOutputTokens: maxOutputTokens,
})
}, directBudget)
}
func (tool execCommandTool) startSession(commandText string, absoluteCwd string, relativeCwd string, ttyRequested bool, engine *zeroSandbox.Engine, sandboxPermissions SandboxPermissionOverride) (*execSession, error) {
@@ -694,6 +707,10 @@ type writeStdinTool struct {
manager *execSessionManager
}
func (writeStdinTool) outputCategory(map[string]any) outputCategory {
return outputCategoryProcess
}
func NewWriteStdinTool(manager *execSessionManager) Tool {
if manager == nil {
manager = defaultExecSessionManager
@@ -843,7 +860,15 @@ type execToolResultInput struct {
}
func execToolResult(input execToolResultInput) Result {
output, truncated := truncateExecOutput(input.output, input.maxOutputTokens)
return execToolResultWithBudget(input, true)
}
func execToolResultWithBudget(input execToolResultInput, directBudget bool) Result {
output := input.output
truncated := false
if directBudget {
output, truncated = truncateExecOutput(input.output, input.maxOutputTokens)
}
meta := map[string]string{
"cwd": input.relativeCwd,
"tty": strconv.FormatBool(input.tty),
+24 -15
View File
@@ -18,6 +18,8 @@ type globTool struct {
scope PathScope
}
func (globTool) outputCategory(map[string]any) outputCategory { return outputCategorySearch }
func NewGlobTool(workspaceRoot string) Tool {
return NewScopedGlobTool(workspaceRoot, nil)
}
@@ -39,7 +41,7 @@ func NewScopedGlobTool(workspaceRoot string, scope PathScope) Tool {
AdditionalProperties: false,
},
safety: readOnlySafety("Finds matching paths without reading contents or modifying files."),
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: workspaceResourceKeys},
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: scopedScanResourceKeys},
},
workspaceRoot: normalizeWorkspaceRoot(workspaceRoot),
scope: scope,
@@ -47,17 +49,25 @@ func NewScopedGlobTool(workspaceRoot string, scope PathScope) Tool {
}
func (tool globTool) Run(ctx context.Context, args map[string]any) Result {
return tool.runWith(ctx, args, readExcluder{})
return tool.runWith(ctx, args, readExcluder{}, true)
}
func (tool globTool) RunWithOptions(ctx context.Context, args map[string]any, options RunOptions) Result {
exclude := readExcluder{}
if options.Sandbox != nil {
exclude = sandboxReadExcluder(options.Sandbox)
}
return tool.runWith(ctx, args, exclude, false)
}
// RunWithSandbox runs glob while skipping subtrees the sandbox policy denies
// reads to (DenyRead). With no DenyRead configured the excluder is a no-op and
// behavior is unchanged.
func (tool globTool) RunWithSandbox(ctx context.Context, args map[string]any, engine *sandbox.Engine) Result {
return tool.runWith(ctx, args, sandboxReadExcluder(engine))
return tool.runWith(ctx, args, sandboxReadExcluder(engine), true)
}
func (tool globTool) runWith(ctx context.Context, args map[string]any, exclude readExcluder) Result {
func (tool globTool) runWith(ctx context.Context, args map[string]any, exclude readExcluder, directBudget bool) Result {
pattern, err := aliasedStringArg(args, []string{"pattern", "glob", "match", "query", "expression"}, "", true, false)
if err != nil {
return errorResult("Error: Invalid arguments for glob: " + err.Error())
@@ -110,24 +120,23 @@ func (tool globTool) runWith(ctx context.Context, args map[string]any, exclude r
if truncated {
output += fmt.Sprintf("\n\n[truncated: showing first %d of %d matches; increase limit or narrow cwd/pattern]", len(matches), totalMatches)
}
budgeted := applyOutputBudget(output, searchOutputBudgetBytes, "increase limit or narrow cwd/pattern")
meta := outputBudgetMeta(budgeted)
meta := map[string]string{}
meta["pattern"] = pattern
if truncated || budgeted.Truncated {
if truncated {
meta["truncated"] = "true"
if budgeted.Truncated {
meta["truncation_reason"] = "byte_budget"
} else {
meta["truncation_reason"] = "limit"
}
meta["truncation_reason"] = "limit"
}
return Result{
result := Result{
Status: StatusOK,
Output: budgeted.Output,
Truncated: truncated || budgeted.Truncated,
Output: output,
Truncated: truncated,
Meta: meta,
}
if directBudget {
return applyLegacyByteBudgetToResult(result, searchOutputBudgetBytes, "increase limit or narrow cwd/pattern")
}
return result
}
func scanGlob(ctx context.Context, root string, displayRoot string, matcher *regexp.Regexp, includeDirs bool, exclude readExcluder) ([]string, error) {
+30 -24
View File
@@ -21,6 +21,8 @@ type grepTool struct {
scope PathScope
}
func (grepTool) outputCategory(map[string]any) outputCategory { return outputCategorySearch }
type grepMatch struct {
file string
line int
@@ -54,7 +56,7 @@ func NewScopedGrepTool(workspaceRoot string, scope PathScope) Tool {
AdditionalProperties: false,
},
safety: readOnlySafety("Searches file paths and matching lines without modifying files."),
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: workspaceResourceKeys},
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: scopedScanResourceKeys},
},
workspaceRoot: normalizeWorkspaceRoot(workspaceRoot),
scope: scope,
@@ -62,7 +64,15 @@ func NewScopedGrepTool(workspaceRoot string, scope PathScope) Tool {
}
func (tool grepTool) Run(ctx context.Context, args map[string]any) Result {
return tool.runWith(ctx, args, readExcluder{})
return tool.runWith(ctx, args, readExcluder{}, true)
}
func (tool grepTool) RunWithOptions(ctx context.Context, args map[string]any, options RunOptions) Result {
exclude := readExcluder{}
if options.Sandbox != nil {
exclude = sandboxReadExcluder(options.Sandbox)
}
return tool.runWith(ctx, args, exclude, false)
}
// RunWithSandbox runs the search while skipping subtrees the sandbox policy
@@ -70,10 +80,10 @@ func (tool grepTool) Run(ctx context.Context, args map[string]any) Result {
// path. With no DenyRead configured the excluder is a no-op and behavior is
// unchanged.
func (tool grepTool) RunWithSandbox(ctx context.Context, args map[string]any, engine *sandbox.Engine) Result {
return tool.runWith(ctx, args, sandboxReadExcluder(engine))
return tool.runWith(ctx, args, sandboxReadExcluder(engine), true)
}
func (tool grepTool) runWith(ctx context.Context, args map[string]any, exclude readExcluder) Result {
func (tool grepTool) runWith(ctx context.Context, args map[string]any, exclude readExcluder, directBudget bool) Result {
pattern, err := aliasedStringArg(args, []string{"pattern", "query", "regex", "search", "expression"}, "", true, false)
if err != nil {
return errorResult("Error: Invalid arguments for grep: " + err.Error())
@@ -154,7 +164,7 @@ func (tool grepTool) runWith(ctx context.Context, args map[string]any, exclude r
}
return errorResult("Error running grep: " + err.Error())
}
return collector.result()
return applyDirectSearchBudget(collector.result(), directBudget, "narrow path/glob/pattern to continue")
case "files_with_matches":
collector := &grepFileListCollector{}
if err := scanGrepMatches(ctx, resolvedRoot, target, globMatcher, exclude, absolutePaths, presenceGrepLineMatcher(compiled), collector.collect); err != nil {
@@ -163,7 +173,7 @@ func (tool grepTool) runWith(ctx context.Context, args map[string]any, exclude r
}
return errorResult("Error running grep: " + err.Error())
}
return collector.result()
return applyDirectSearchBudget(collector.result(), directBudget, "narrow path/glob/pattern to continue")
default:
collector := &grepContentCollector{headLimit: headLimit}
if err := scanGrepMatches(ctx, resolvedRoot, target, globMatcher, exclude, absolutePaths, presenceGrepLineMatcher(compiled), collector.collect); err != nil {
@@ -172,10 +182,17 @@ func (tool grepTool) runWith(ctx context.Context, args map[string]any, exclude r
}
return errorResult("Error running grep: " + err.Error())
}
return collector.result()
return applyDirectSearchBudget(collector.result(), directBudget, "narrow path/glob/pattern or increase head_limit")
}
}
func applyDirectSearchBudget(result Result, directBudget bool, hint string) Result {
if !directBudget {
return result
}
return applyLegacyByteBudgetToResult(result, searchOutputBudgetBytes, hint)
}
// resolveGrepRoot picks the scope root whose EvalSymlinks-resolved path contains
// the already-resolved target, so that confineGrepFile computes correct
// workspace-relative paths even when the target lives in an extra root.
@@ -455,13 +472,7 @@ func (collector *grepFileListCollector) result() Result {
return okResult("No matches found.")
}
sort.Strings(collector.files)
budgeted := applyOutputBudget(strings.Join(collector.files, "\n"), searchOutputBudgetBytes, "narrow path/glob/pattern to continue")
meta := outputBudgetMeta(budgeted)
if budgeted.Truncated {
meta["truncated"] = "true"
meta["truncation_reason"] = "byte_budget"
}
return Result{Status: StatusOK, Output: budgeted.Output, Truncated: budgeted.Truncated, Meta: meta}
return Result{Status: StatusOK, Output: strings.Join(collector.files, "\n")}
}
type grepContentCollector struct {
@@ -494,20 +505,15 @@ func (collector *grepContentCollector) result() Result {
if truncated {
output += fmt.Sprintf("\n\n[truncated: showing first %d matches; narrow path/glob/pattern or increase head_limit]", len(lines))
}
budgeted := applyOutputBudget(output, searchOutputBudgetBytes, "narrow path/glob/pattern or increase head_limit")
meta := outputBudgetMeta(budgeted)
if truncated || budgeted.Truncated {
meta := map[string]string{}
if truncated {
meta["truncated"] = "true"
if budgeted.Truncated {
meta["truncation_reason"] = "byte_budget"
} else {
meta["truncation_reason"] = "head_limit"
}
meta["truncation_reason"] = "head_limit"
}
return Result{
Status: StatusOK,
Output: budgeted.Output,
Truncated: truncated || budgeted.Truncated,
Output: output,
Truncated: truncated,
Meta: meta,
}
}
+17 -7
View File
@@ -15,6 +15,10 @@ type listDirectoryTool struct {
scope PathScope
}
func (listDirectoryTool) outputCategory(map[string]any) outputCategory {
return outputCategorySearch
}
func NewListDirectoryTool(workspaceRoot string) Tool {
return NewScopedListDirectoryTool(workspaceRoot, nil)
}
@@ -34,7 +38,7 @@ func NewScopedListDirectoryTool(workspaceRoot string, scope PathScope) Tool {
AdditionalProperties: false,
},
safety: readOnlySafety("Lists directory entries without modifying files."),
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: directoryResourceKeys},
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: directoryResourceKeys},
},
workspaceRoot: normalizeWorkspaceRoot(workspaceRoot),
scope: scope,
@@ -42,6 +46,14 @@ func NewScopedListDirectoryTool(workspaceRoot string, scope PathScope) Tool {
}
func (tool listDirectoryTool) Run(_ context.Context, args map[string]any) Result {
return tool.run(args, true)
}
func (tool listDirectoryTool) RunWithOptions(_ context.Context, args map[string]any, _ RunOptions) Result {
return tool.run(args, false)
}
func (tool listDirectoryTool) run(args map[string]any, directBudget bool) Result {
// Optional with a "." default: treat an explicit empty path (a common
// weak-model quirk) the same as the key being absent rather than erroring.
requestedPath, err := aliasedStringArg(args, []string{"path", "directory", "dir"}, ".", false, true)
@@ -76,13 +88,11 @@ func (tool listDirectoryTool) Run(_ context.Context, args map[string]any) Result
return okResult("Directory is empty: " + relativePath)
}
output := "Contents of " + relativePath + ":\n\n" + strings.Join(entries, "\n")
budgeted := applyOutputBudget(output, searchOutputBudgetBytes, "use path, recursive=false, or a smaller max_depth to narrow the listing")
meta := outputBudgetMeta(budgeted)
if budgeted.Truncated {
meta["truncated"] = "true"
meta["truncation_reason"] = "byte_budget"
result := Result{Status: StatusOK, Output: output}
if directBudget {
return applyLegacyByteBudgetToResult(result, searchOutputBudgetBytes, "use path, recursive=false, or a smaller max_depth to narrow the listing")
}
return Result{Status: StatusOK, Output: budgeted.Output, Truncated: budgeted.Truncated, Meta: meta}
return result
}
func listDirectoryEntries(path string, depth int, maxDepth int) ([]string, error) {
+254
View File
@@ -0,0 +1,254 @@
package tools
import (
"strconv"
"strings"
)
const (
outputBudgetCategoryMeta = "output_budget_category"
outputBudgetOriginalBytesMeta = "output_budget_original_bytes"
outputBudgetRetainedBytesMeta = "output_budget_retained_bytes"
outputBudgetEstimatedOriginalTokensMeta = "output_budget_estimated_original_tokens"
outputBudgetEstimatedRetainedTokensMeta = "output_budget_estimated_retained_tokens"
outputBudgetReasonMeta = "output_budget_reason"
outputBudgetSpillCreatedMeta = "output_budget_spill_created"
)
// applyRegistryOutputBudget is the common post-redaction semantic budgeting
// boundary for tools that do not already own a deliberate output budget.
func applyRegistryOutputBudget(tool Tool, toolName string, args map[string]any, result Result) Result {
budget := registryOutputBudget(toolName)
if budget.maxEstimatedTokens <= 0 && budget.hardMaxBytes <= 0 {
return result // preserve ZERO_TOOL_OUTPUT_CEILING_TOKENS=0 semantics
}
category := resolveOutputCategory(tool, toolName, args)
budgeted := budgetSemanticOutput(result.Output, category, budget)
if budgeted.truncated {
budgeted = attachExistingSpill(toolName, result.Output, budget, budgeted)
}
if result.Truncated {
budgeted.truncated = true
if budgeted.reason == "" {
budgeted.reason = result.Meta["truncation_reason"]
if budgeted.reason == "" {
budgeted.reason = "upstream_tool_budget"
}
}
if budgeted.spillPath == "" {
budgeted.spillPath = result.Meta["spill_path"]
}
}
result.Output = budgeted.text
result.Truncated = result.Truncated || budgeted.truncated
result.Meta = addOutputBudgetMetadata(result.Meta, budgeted)
return result
}
// applySelfManagedOutputBudget applies semantic retention to tools that retain
// output with their own capture-aware limits. Their explicit budgets remain
// authoritative, rather than being replaced by the registry default ceiling.
func applySelfManagedOutputBudget(tool Tool, toolName string, args map[string]any, result Result) Result {
budget := selfManagedOutputBudget(toolName, args)
if budget.maxEstimatedTokens <= 0 && budget.hardMaxBytes <= 0 {
return annotateSelfBudgetedOutput(tool, toolName, args, result)
}
category := resolveOutputCategory(tool, toolName, args)
budgeted := budgetSemanticOutput(result.Output, category, budget)
if result.Truncated && !budgeted.truncated {
budgeted.truncated = true
budgeted.reason = result.Meta["truncation_reason"]
if budgeted.reason == "" {
budgeted.reason = "upstream_tool_budget"
}
}
if budgeted.truncated {
budgeted = attachExistingSpill(toolName, result.Output, budget, budgeted)
}
result.Output = budgeted.text
result.Truncated = result.Truncated || budgeted.truncated
result.Meta = addOutputBudgetMetadata(result.Meta, budgeted)
return result
}
func selfManagedOutputBudget(toolName string, args map[string]any) outputBudget {
switch toolName {
case "bash":
// bash previously exposed at most 32 KiB from each stream. Keep that
// combined 64 KiB ceiling while selecting its contents semantically.
return outputBudget{maxEstimatedTokens: (bashOutputBudgetBytes * 2) / 4, hardMaxBytes: bashOutputBudgetBytes * 2}
case ExecCommandToolName:
maxOutputTokens := defaultMaxOutputTokens
if parsed, err := intArg(args, "max_output_tokens", defaultMaxOutputTokens, 1, maxExecOutputTokenRequest); err == nil {
maxOutputTokens = parsed
}
return outputBudget{maxEstimatedTokens: maxOutputTokens, hardMaxBytes: maxOutputTokens * 4}
default:
return outputBudget{}
}
}
// RebudgetAfterHook reapplies the registry's redaction and output limits after
// an afterTool hook appends model-visible feedback to a completed result. The
// initial registry pass has already run; this second pass is limited to the
// newly combined result so hooks cannot bypass the established safety ceiling.
func (registry *Registry) RebudgetAfterHook(toolName string, args map[string]any, result Result) Result {
result = scrubResultSecrets(result)
tool, _ := registry.Get(toolName)
if _, ok := tool.(selfBudgeting); ok {
// Match the primary registry boundary: self-managed tools keep their
// call-specific capture/output budget instead of being tightened or
// loosened to the generic registry ceiling after hook feedback.
return applySelfManagedOutputBudget(tool, toolName, args, result)
}
result = applyRegistryOutputBudget(tool, toolName, args, result)
return enforceOutputCeiling(toolName, result)
}
func registryOutputBudget(toolName string) outputBudget {
switch toolName {
case "read_file", "read_minified_file":
return outputBudget{maxEstimatedTokens: readOutputBudgetBytes / 4, hardMaxBytes: readOutputBudgetBytes}
case "grep", "glob", "list_directory":
return outputBudget{maxEstimatedTokens: searchOutputBudgetBytes / 4, hardMaxBytes: searchOutputBudgetBytes}
default:
ceilingTokens := resolveOutputCeilingTokens()
if ceilingTokens <= 0 {
return outputBudget{}
}
return outputBudget{maxEstimatedTokens: ceilingTokens, hardMaxBytes: ceilingTokens * 4}
}
}
func resolveOutputCategory(tool Tool, toolName string, args map[string]any) outputCategory {
if provider, ok := tool.(outputPolicyProvider); ok {
if category := provider.outputCategory(args); category != "" {
return category
}
}
switch toolName {
case "Task", "swarm_collect":
return outputCategoryWorker
case "apply_patch":
return outputCategoryDiff
case "write_stdin":
return outputCategoryProcess
default:
return outputCategoryDefault
}
}
// annotateSelfBudgetedOutput records the same compact metadata for tools whose
// existing capture/budget implementation remains authoritative in PR11. It
// does not re-budget their text or claim that raw_bytes represents every byte a
// subprocess produced beyond its established capture limits.
func annotateSelfBudgetedOutput(tool Tool, toolName string, args map[string]any, result Result) Result {
retainedBytes := len(result.Output)
originalBytes := retainedBytes
if parsed, err := strconv.Atoi(result.Meta["raw_bytes"]); err == nil && parsed > originalBytes {
originalBytes = parsed
}
retainedTokens := estimateOutputTokens(result.Output)
originalTokens := retainedTokens
if originalBytes > retainedBytes {
originalTokens = max(originalTokens, estimatedTokensFromBytes(originalBytes))
}
reason := result.Meta["truncation_reason"]
if result.Truncated && reason == "" {
reason = "upstream_tool_budget"
}
observed := budgetedOutput{
text: result.Output,
originalBytes: originalBytes,
retainedBytes: retainedBytes,
estimatedOriginalTokens: originalTokens,
estimatedRetainedTokens: retainedTokens,
truncated: result.Truncated,
category: resolveOutputCategory(tool, toolName, args),
reason: reason,
spillPath: result.Meta["spill_path"],
}
result.Meta = addOutputBudgetMetadata(result.Meta, observed)
return result
}
func shellOutputCategory(command string) outputCategory {
normalized := strings.ToLower(strings.TrimSpace(command))
if containsAny(normalized,
"go test", "pytest", "python -m pytest", "cargo test", "npm test", "npm run test",
"pnpm test", "yarn test", "bun test", "dotnet test", "mvn test", "gradle test", "phpunit") {
return outputCategoryTest
}
if containsAny(normalized, "git diff", "git show", "git format-patch", " diff -u", "diff --git") || strings.HasPrefix(normalized, "diff ") {
return outputCategoryDiff
}
return outputCategoryProcess
}
// attachExistingSpill reuses Zero's hardened per-user spill directory. output
// is the already-redacted text received by this layer; it may itself be a
// capture-bounded view produced by a subprocess tool.
func attachExistingSpill(toolName, output string, budget outputBudget, current budgetedOutput) budgetedOutput {
path := spillTruncatedOutput(toolName, output)
if path == "" {
return current
}
notice := "[zero] full output received by budgeting layer saved to " + path + " (grep or read_file it instead of re-running)"
reduced := outputBudget{
maxEstimatedTokens: max(1, budget.maxEstimatedTokens-estimateOutputTokens("\n"+notice)),
hardMaxBytes: max(1, budget.hardMaxBytes-len("\n"+notice)),
}
base := budgetSemanticOutput(output, current.category, reduced)
text := strings.TrimRight(base.text, "\n") + "\n" + notice
if !fitsOutputBudget(text, budget) {
// An unusually long temp path or tiny configured ceiling can leave no
// room for the full notice. Keep the safe bounded result; the spill still
// exists but is intentionally not advertised with a chopped reference.
current.spillPath = path
return current
}
base.text = text
base.retainedBytes = len(text)
base.estimatedRetainedTokens = estimateOutputTokens(text)
base.truncated = true
if base.reason == "" {
base.reason = current.reason
}
base.category = current.category
base.spillPath = path
return base
}
func addOutputBudgetMetadata(meta map[string]string, output budgetedOutput) map[string]string {
if meta == nil {
meta = map[string]string{}
}
meta[outputBudgetCategoryMeta] = string(output.category)
meta[outputBudgetOriginalBytesMeta] = strconv.Itoa(output.originalBytes)
meta[outputBudgetRetainedBytesMeta] = strconv.Itoa(output.retainedBytes)
meta[outputBudgetEstimatedOriginalTokensMeta] = strconv.Itoa(output.estimatedOriginalTokens)
meta[outputBudgetEstimatedRetainedTokensMeta] = strconv.Itoa(output.estimatedRetainedTokens)
meta[outputBudgetSpillCreatedMeta] = strconv.FormatBool(output.spillPath != "")
// These established fields describe the final model-visible output. Refresh
// them on every pass because redaction and after-tool hooks may change the
// text even when this pass does not newly truncate it.
meta["emitted_bytes"] = strconv.Itoa(output.retainedBytes)
meta["estimated_tokens"] = strconv.Itoa(output.estimatedRetainedTokens)
if output.reason != "" {
meta[outputBudgetReasonMeta] = output.reason
}
if output.truncated {
// Preserve the existing metadata vocabulary used by callers and tests.
if _, exists := meta["raw_bytes"]; !exists {
meta["raw_bytes"] = strconv.Itoa(output.originalBytes)
}
meta["truncated"] = "true"
meta["truncation_reason"] = output.reason
}
if output.spillPath != "" {
meta["spill_path"] = output.spillPath
}
return meta
}
+381
View File
@@ -0,0 +1,381 @@
package tools
import (
"context"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"unicode/utf8"
)
func TestRegistryBudgetRunsAfterRedactionAndSpillsRedactedOutput(t *testing.T) {
setTestTempDir(t)
secret := "ghp_" + strings.Repeat("a", 36)
big := "HEAD\n" + secret + "\n" + strings.Repeat("🙂 noisy output\n", 20_000) + "TAIL"
registry := NewRegistry()
registry.Register(newCeilingFakeTool("redacted_big", big))
result := registry.Run(context.Background(), "redacted_big", map[string]any{})
if !result.Truncated || result.Meta[outputBudgetSpillCreatedMeta] != "true" {
t.Fatalf("unexpected budget result: truncated=%t meta=%#v", result.Truncated, result.Meta)
}
if strings.Contains(result.Output, secret) || !utf8.ValidString(result.Output) {
t.Fatalf("exposed output leaked secret or invalid UTF-8: %q", result.Output)
}
spillPath := result.Meta["spill_path"]
content, err := os.ReadFile(spillPath)
if err != nil {
t.Fatalf("read spill: %v", err)
}
if strings.Contains(string(content), secret) {
t.Fatal("spill contains unredacted secret")
}
if !strings.Contains(string(content), "HEAD") || !strings.Contains(string(content), "TAIL") {
t.Fatal("spill does not contain the complete redacted output received by the budget layer")
}
}
func TestRegistryBudgetSpillFailureFallsBackToBoundedOutput(t *testing.T) {
temp := t.TempDir()
blockedTemp := filepath.Join(temp, "not-a-directory")
if err := os.WriteFile(blockedTemp, []byte("x"), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("TMPDIR", blockedTemp)
t.Setenv("TMP", blockedTemp)
t.Setenv("TEMP", blockedTemp)
t.Setenv(outputCeilingEnv, "100")
registry := NewRegistry()
registry.Register(newCeilingFakeTool("spill_failure", strings.Repeat("large output\n", 1000)))
result := registry.Run(context.Background(), "spill_failure", map[string]any{})
if !result.Truncated || len(result.Output) > 400 {
t.Fatalf("fallback is not bounded: truncated=%t bytes=%d", result.Truncated, len(result.Output))
}
if result.Meta[outputBudgetSpillCreatedMeta] != "false" || result.Meta["spill_path"] != "" {
t.Fatalf("spill failure incorrectly advertised: %#v", result.Meta)
}
}
func TestRegistryMissingPolicyUsesDefaultAndExactHardCeiling(t *testing.T) {
setTestTempDir(t)
t.Setenv(outputCeilingEnv, "80")
input := "HEAD\n" + strings.Repeat("x", 3000) + "\nTAIL"
registry := NewRegistry()
registry.Register(newCeilingFakeTool("no_policy", input))
result := registry.Run(context.Background(), "no_policy", map[string]any{})
if got := result.Meta[outputBudgetCategoryMeta]; got != string(outputCategoryDefault) {
t.Fatalf("category = %q, want default", got)
}
if len(result.Output) > 80*4 {
t.Fatalf("output = %d bytes, hard ceiling %d", len(result.Output), 80*4)
}
if result.Meta[outputBudgetOriginalBytesMeta] != strconv.Itoa(len(input)) {
t.Fatalf("original size metadata = %q", result.Meta[outputBudgetOriginalBytesMeta])
}
}
func TestRegistrySmallOutputRemainsByteIdentical(t *testing.T) {
registry := NewRegistry()
registry.Register(newCeilingFakeTool("small_identity", "hello\nworld\n"))
result := registry.Run(context.Background(), "small_identity", map[string]any{})
if result.Output != "hello\nworld\n" || result.Truncated {
t.Fatalf("small output changed: %#v", result)
}
}
func TestDirectReadAndSearchToolsKeepLegacyByteBudgets(t *testing.T) {
root := t.TempDir()
if err := os.WriteFile(filepath.Join(root, "large-read.txt"), []byte(strings.Repeat("source line with enough content\n", 7000)), 0o600); err != nil {
t.Fatal(err)
}
if result := NewReadFileTool(root).Run(context.Background(), map[string]any{"path": "large-read.txt"}); !result.Truncated || len(result.Output) > readOutputBudgetBytes {
t.Fatalf("direct read_file output is not bounded: %#v", result)
}
for index := 0; index < 800; index++ {
name := fmt.Sprintf("%04d-%s.txt", index, strings.Repeat("n", 80))
if err := os.WriteFile(filepath.Join(root, name), []byte("match\n"), 0o600); err != nil {
t.Fatal(err)
}
}
if err := os.WriteFile(filepath.Join(root, "large-content.txt"), []byte(strings.Repeat("match "+strings.Repeat("x", 2000)+"\n", 50)), 0o600); err != nil {
t.Fatal(err)
}
for name, result := range map[string]Result{
"list_directory": NewListDirectoryTool(root).Run(context.Background(), map[string]any{"recursive": false}),
"glob": NewGlobTool(root).Run(context.Background(), map[string]any{"pattern": "*.txt", "limit": 1000}),
"grep files": NewGrepTool(root).Run(context.Background(), map[string]any{"pattern": "match", "output_mode": "files_with_matches"}),
"grep content": NewGrepTool(root).Run(context.Background(), map[string]any{"pattern": "match", "path": "large-content.txt", "head_limit": 50}),
} {
if !result.Truncated || len(result.Output) > searchOutputBudgetBytes {
t.Fatalf("direct %s output is not bounded: truncated=%t bytes=%d meta=%#v", name, result.Truncated, len(result.Output), result.Meta)
}
}
}
func TestRegistryLargeFileUsesSemanticFilePolicy(t *testing.T) {
setTestTempDir(t)
root := t.TempDir()
var content strings.Builder
content.WriteString("HEAD_MARK\n")
for line := 0; line < 7000; line++ {
content.WriteString("ordinary source line with enough text to fill the output budget\n")
}
content.WriteString("TAIL_MARK\n")
if err := os.WriteFile(filepath.Join(root, "large.txt"), []byte(content.String()), 0o600); err != nil {
t.Fatal(err)
}
registry := NewRegistry()
registry.Register(NewReadFileTool(root))
result := registry.Run(context.Background(), "read_file", map[string]any{"path": "large.txt"})
if !result.Truncated || result.Meta[outputBudgetCategoryMeta] != string(outputCategoryFile) {
t.Fatalf("large file was not semantically budgeted: truncated=%t meta=%#v", result.Truncated, result.Meta)
}
if len(result.Output) > readOutputBudgetBytes {
t.Fatalf("large file output = %d bytes, ceiling %d", len(result.Output), readOutputBudgetBytes)
}
for _, want := range []string{"File: large.txt", "HEAD_MARK", "TAIL_MARK"} {
if !strings.Contains(result.Output, want) {
t.Fatalf("large file output missing %q", want)
}
}
}
func TestRegistryLargeSingleLineFileRetainsHeadAndTail(t *testing.T) {
setTestTempDir(t)
root := t.TempDir()
content := "HEAD_SINGLE_LINE_MARK" + strings.Repeat("x", readOutputBudgetBytes*2) + "TAIL_SINGLE_LINE_MARK"
if err := os.WriteFile(filepath.Join(root, "large.min.js"), []byte(content), 0o600); err != nil {
t.Fatal(err)
}
registry := NewRegistry()
registry.Register(NewReadFileTool(root))
result := registry.Run(context.Background(), "read_file", map[string]any{"path": "large.min.js"})
if !result.Truncated || result.Meta[outputBudgetCategoryMeta] != string(outputCategoryFile) {
t.Fatalf("single-line file was not semantically budgeted: truncated=%t meta=%#v", result.Truncated, result.Meta)
}
if len(result.Output) > readOutputBudgetBytes || !utf8.ValidString(result.Output) {
t.Fatalf("single-line file output is not safely bounded: bytes=%d valid=%t", len(result.Output), utf8.ValidString(result.Output))
}
if rawBytes, err := strconv.Atoi(result.Meta["raw_bytes"]); err != nil || rawBytes <= len(result.Output) {
t.Fatalf("single-line file capture was not bounded head/tail output: raw=%q emitted=%d", result.Meta["raw_bytes"], len(result.Output))
}
for _, want := range []string{"File: large.min.js", "HEAD_SINGLE_LINE_MARK", "TAIL_SINGLE_LINE_MARK"} {
if !strings.Contains(result.Output, want) {
t.Fatalf("single-line file output missing %q", want)
}
}
}
func TestRegistryMinifiedFileUpdatesEmittedBytesAfterBudgeting(t *testing.T) {
setTestTempDir(t)
root := t.TempDir()
content := "HEAD_MINIFIED_MARK" + strings.Repeat("x", readOutputBudgetBytes*2) + "TAIL_MINIFIED_MARK"
if err := os.WriteFile(filepath.Join(root, "large.txt"), []byte(content), 0o600); err != nil {
t.Fatal(err)
}
registry := NewRegistry()
registry.Register(NewReadMinifiedFileTool(root))
result := registry.Run(context.Background(), "read_minified_file", map[string]any{"path": "large.txt"})
if !result.Truncated || result.Meta["emitted_bytes"] != strconv.Itoa(len(result.Output)) {
t.Fatalf("minified metadata does not describe final output: truncated=%t emitted=%q bytes=%d", result.Truncated, result.Meta["emitted_bytes"], len(result.Output))
}
for _, want := range []string{"HEAD_MINIFIED_MARK", "TAIL_MINIFIED_MARK"} {
if !strings.Contains(result.Output, want) {
t.Fatalf("minified single-line file output missing %q", want)
}
}
}
func TestRegistryGrepUsesSemanticMultiFileCoverage(t *testing.T) {
setTestTempDir(t)
root := t.TempDir()
for _, name := range []string{"a.txt", "b.txt", "c.txt", "d.txt"} {
body := strings.Repeat("needle "+strings.Repeat(name, 15)+"\n", 350)
if err := os.WriteFile(filepath.Join(root, name), []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
registry := NewRegistry()
registry.Register(NewGrepTool(root))
result := registry.Run(context.Background(), "grep", map[string]any{
"pattern": "needle",
"path": ".",
"head_limit": 1400,
})
if !result.Truncated || result.Meta[outputBudgetCategoryMeta] != string(outputCategorySearch) {
t.Fatalf("grep was not semantically budgeted: truncated=%t meta=%#v", result.Truncated, result.Meta)
}
for _, name := range []string{"a.txt:", "b.txt:", "c.txt:", "d.txt:"} {
if !strings.Contains(result.Output, name) {
t.Fatalf("search output lost %s coverage", name)
}
}
}
func TestShellOutputCategoryClassification(t *testing.T) {
tests := []struct {
command string
want outputCategory
}{
{"go test ./...", outputCategoryTest},
{"python -m pytest -q", outputCategoryTest},
{"git diff --stat main...HEAD", outputCategoryDiff},
{"git show HEAD", outputCategoryDiff},
{"make build", outputCategoryProcess},
}
for _, test := range tests {
if got := shellOutputCategory(test.command); got != test.want {
t.Errorf("shellOutputCategory(%q) = %q, want %q", test.command, got, test.want)
}
}
}
func TestShellToolOutputCategoryUsesAllCommandAliases(t *testing.T) {
providers := []struct {
name string
provider outputPolicyProvider
}{
{name: "bash", provider: NewBashTool(t.TempDir()).(outputPolicyProvider)},
{name: "exec_command", provider: NewExecCommandTool(t.TempDir(), newExecSessionManager()).(outputPolicyProvider)},
}
for _, provider := range providers {
for _, alias := range []string{"command", "cmd", "script", "shell"} {
t.Run(provider.name+"/"+alias, func(t *testing.T) {
if got := provider.provider.outputCategory(map[string]any{alias: "go test ./..."}); got != outputCategoryTest {
t.Fatalf("category = %q, want test", got)
}
})
}
}
if got := providers[0].provider.outputCategory(map[string]any{"command": "make build", "cmd": "go test ./..."}); got != outputCategoryProcess {
t.Fatalf("bash alias precedence category = %q, want process", got)
}
if got := providers[1].provider.outputCategory(map[string]any{"cmd": "go test ./...", "command": "make build"}); got != outputCategoryTest {
t.Fatalf("exec_command alias precedence category = %q, want test", got)
}
}
func TestRegistryBudgetPreservesExistingTruncationAndRefreshesMetadata(t *testing.T) {
output := "post-hook output"
result := applyRegistryOutputBudget(newCeilingFakeTool("existing_truncation", output), "existing_truncation", map[string]any{}, Result{
Status: StatusOK,
Output: output,
Truncated: true,
Meta: map[string]string{
"emitted_bytes": "999",
"estimated_tokens": "999",
"truncation_reason": "capture_budget",
},
})
if !result.Truncated || result.Meta["truncated"] != "true" || result.Meta[outputBudgetReasonMeta] != "capture_budget" {
t.Fatalf("prior truncation state was lost: %#v", result)
}
if result.Meta["emitted_bytes"] != strconv.Itoa(len(output)) || result.Meta["estimated_tokens"] != strconv.Itoa(estimateOutputTokens(output)) {
t.Fatalf("final output metadata is stale: %#v", result.Meta)
}
if result.Meta[outputBudgetSpillCreatedMeta] != "false" {
t.Fatalf("existing truncation unexpectedly created a new spill: %#v", result.Meta)
}
}
func TestSelfManagedOutputBudgetUsesSemanticTestPolicy(t *testing.T) {
setTestTempDir(t)
input := "stdout:\n" + strings.Repeat("PASS progress\n", 8_000) +
"--- FAIL: TestImportant (0.01s)\nthing_test.go:42: expected 7, got 9\nFAIL\nexit_code: 1"
tests := []struct {
name string
tool Tool
args map[string]any
}{
{name: "bash", tool: NewBashTool(t.TempDir()), args: map[string]any{"command": "go test ./..."}},
{name: "exec", tool: NewExecCommandTool(t.TempDir(), newExecSessionManager()), args: map[string]any{"cmd": "go test ./...", "max_output_tokens": 16_000}},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got := applySelfManagedOutputBudget(test.tool, test.tool.Name(), test.args, Result{Status: StatusError, Output: input, Meta: map[string]string{}})
if !got.Truncated || got.Meta[outputBudgetCategoryMeta] != string(outputCategoryTest) {
t.Fatalf("semantic budget was not applied: %#v", got)
}
for _, want := range []string{"TestImportant", "expected 7", "FAIL", "exit_code: 1"} {
if !strings.Contains(got.Output, want) {
t.Fatalf("semantic test output missing %q:\n%s", want, got.Output)
}
}
})
}
}
type semanticBashFakeTool struct{ ceilingFakeTool }
func (semanticBashFakeTool) managesOutputBudget() {}
func (semanticBashFakeTool) outputCategory(args map[string]any) outputCategory {
command, _ := args["command"].(string)
return shellOutputCategory(command)
}
func TestRegistryAppliesSemanticBudgetToSelfManagedShellOutput(t *testing.T) {
setTestTempDir(t)
input := "startup\n" + strings.Repeat("PASS progress\n", 8_000) +
"--- FAIL: TestRegistryImportant (0.01s)\nexpected 7, got 9\nFAIL\nexit_code: 1"
registry := NewRegistry()
registry.Register(semanticBashFakeTool{newCeilingFakeTool("bash", input)})
result := registry.Run(context.Background(), "bash", map[string]any{"command": "go test ./..."})
if !result.Truncated || result.Meta[outputBudgetReasonMeta] != "semantic_test_budget" {
t.Fatalf("registry did not apply semantic test budget: %#v", result.Meta)
}
for _, want := range []string{"TestRegistryImportant", "expected 7", "FAIL", "exit_code: 1"} {
if !strings.Contains(result.Output, want) {
t.Fatalf("registry semantic test output missing %q:\n%s", want, result.Output)
}
}
}
func TestRegistryAppliesSemanticBudgetToSelfManagedProcessOutput(t *testing.T) {
setTestTempDir(t)
input := "starting build\n" + strings.Repeat("progress\n", 8_000) +
"WARNING: cache is cold\nERROR: compilation failed\nexit_code: 1"
registry := NewRegistry()
registry.Register(semanticBashFakeTool{newCeilingFakeTool("bash", input)})
result := registry.Run(context.Background(), "bash", map[string]any{"command": "make build"})
if !result.Truncated || result.Meta[outputBudgetReasonMeta] != "semantic_process_budget" {
t.Fatalf("registry did not apply semantic process budget: %#v", result.Meta)
}
for _, want := range []string{"starting build", "WARNING", "ERROR", "exit_code: 1"} {
if !strings.Contains(result.Output, want) {
t.Fatalf("registry semantic process output missing %q:\n%s", want, result.Output)
}
}
}
func TestRebudgetAfterHookPreservesSelfManagedExecBudget(t *testing.T) {
setTestTempDir(t)
registry := NewRegistry()
registry.Register(NewExecCommandTool(t.TempDir(), newExecSessionManager()))
t.Run("tight call budget remains authoritative", func(t *testing.T) {
args := map[string]any{"cmd": "make build", "max_output_tokens": 10}
result := registry.RebudgetAfterHook(ExecCommandToolName, args, Result{Status: StatusOK, Output: strings.Repeat("x", 1_000)})
if !result.Truncated || len(result.Output) > 40 {
t.Fatalf("post-hook output escaped call budget: truncated=%t bytes=%d meta=%#v", result.Truncated, len(result.Output), result.Meta)
}
})
t.Run("raised call budget is not replaced by generic ceiling", func(t *testing.T) {
args := map[string]any{"cmd": "make build", "max_output_tokens": 20_000}
output := strings.Repeat("x", 70_000)
result := registry.RebudgetAfterHook(ExecCommandToolName, args, Result{Status: StatusOK, Output: output})
if result.Truncated || result.Output != output {
t.Fatalf("post-hook output was incorrectly limited by generic ceiling: truncated=%t bytes=%d", result.Truncated, len(result.Output))
}
})
}
+213 -2
View File
@@ -4,6 +4,7 @@ import (
"fmt"
"strconv"
"strings"
"unicode/utf8"
)
const (
@@ -11,6 +12,167 @@ const (
searchOutputBudgetBytes = 64 * 1024
)
// outputCategory selects the deterministic retention policy used for a tool
// result. It is deliberately internal: categories are an implementation detail
// of the tools boundary, not part of Zero's public tool protocol.
type outputCategory string
const (
outputCategoryDefault outputCategory = "default"
outputCategoryFile outputCategory = "file"
outputCategorySearch outputCategory = "search"
outputCategoryTest outputCategory = "test"
outputCategoryProcess outputCategory = "process"
outputCategoryDiff outputCategory = "diff"
outputCategoryWorker outputCategory = "worker"
)
// outputBudget combines a provider-neutral estimated-token target with the
// existing byte ceiling. The byte ceiling is always authoritative: semantic
// retention may use fewer bytes, but can never use more.
type outputBudget struct {
maxEstimatedTokens int
hardMaxBytes int
}
// budgetedOutput describes the text retained by a semantic output policy. The
// original size is the output received by this layer; it does not claim to be
// every byte an underlying subprocess may have produced before its own capture
// limits were applied.
type budgetedOutput struct {
text string
originalBytes int
retainedBytes int
estimatedOriginalTokens int
estimatedRetainedTokens int
truncated bool
category outputCategory
reason string
spillPath string
}
// outputPolicyProvider optionally assigns a semantic category to a tool call.
// Tools that do not implement it use outputCategoryDefault.
type outputPolicyProvider interface {
outputCategory(args map[string]any) outputCategory
}
// estimateOutputTokens is a deterministic, provider-neutral estimate used only
// for output budgeting. It is not exact provider tokenization. ASCII non-space
// text uses the repository's established four-bytes-per-token approximation;
// each byte of non-ASCII UTF-8 is counted as a token, intentionally
// overestimating multilingual text and emoji rather than letting them bypass a
// budget. Existing hard byte ceilings remain the final safety limit.
func estimateOutputTokens(value string) int {
asciiNonSpace, nonASCIIBytes := outputTokenComponents(value)
return (asciiNonSpace+3)/4 + nonASCIIBytes
}
func outputTokenComponents(value string) (asciiNonSpace int, nonASCIIBytes int) {
for index := 0; index < len(value); {
r, size := utf8.DecodeRuneInString(value[index:])
if r == utf8.RuneError && size == 1 {
// Invalid input is charged conservatively one token per byte. Budget
// slicing itself remains rune-safe for valid UTF-8 tool output.
nonASCIIBytes++
index++
continue
}
if r < utf8.RuneSelf {
switch r {
case ' ', '\t', '\n', '\r', '\f', '\v':
default:
asciiNonSpace++
}
} else {
nonASCIIBytes += size
}
index += size
}
return asciiNonSpace, nonASCIIBytes
}
const defaultSemanticTruncationNotice = "\n[zero] output truncated\n"
// budgetDefaultOutput applies the safe fallback policy: retain a rune-safe head
// and tail around a stable truncation notice. Small output is returned
// byte-identically. Spill creation is intentionally handled by the shared
// boundary integration so every semantic policy uses the existing spill path.
func budgetDefaultOutput(output string, budget outputBudget) budgetedOutput {
result := budgetedOutput{
text: output,
originalBytes: len(output),
retainedBytes: len(output),
estimatedOriginalTokens: estimateOutputTokens(output),
estimatedRetainedTokens: estimateOutputTokens(output),
category: outputCategoryDefault,
}
if fitsOutputBudget(output, budget) {
return result
}
reason := outputBudgetReason(result.originalBytes, result.estimatedOriginalTokens, budget)
maxContentBytes := len(output)
if budget.hardMaxBytes > 0 {
maxContentBytes = min(maxContentBytes, max(0, budget.hardMaxBytes-len(defaultSemanticTruncationNotice)))
}
// Find the largest deterministic head+tail window that satisfies both the
// estimate and the hard byte ceiling. fitsOutputBudget is monotonic as this
// window grows, so binary search avoids repeatedly trimming one rune at a time.
low, high := 0, maxContentBytes
best := ""
if fitsOutputBudget(defaultSemanticTruncationNotice, budget) {
best = defaultSemanticTruncationNotice
}
for low <= high {
window := low + (high-low)/2
candidate := defaultHeadTail(output, window)
if fitsOutputBudget(candidate, budget) {
best = candidate
low = window + 1
} else {
high = window - 1
}
}
result.text = best
result.retainedBytes = len(best)
result.estimatedRetainedTokens = estimateOutputTokens(best)
result.truncated = true
result.reason = reason
return result
}
func defaultHeadTail(output string, contentBytes int) string {
if contentBytes <= 0 {
return defaultSemanticTruncationNotice
}
headBytes := contentBytes / 2
tailBytes := contentBytes - headBytes
return utf8Prefix(output, headBytes) + defaultSemanticTruncationNotice + utf8Suffix(output, tailBytes)
}
func fitsOutputBudget(output string, budget outputBudget) bool {
if budget.hardMaxBytes > 0 && len(output) > budget.hardMaxBytes {
return false
}
return budget.maxEstimatedTokens <= 0 || estimateOutputTokens(output) <= budget.maxEstimatedTokens
}
func outputBudgetReason(originalBytes int, estimatedTokens int, budget outputBudget) string {
overTokens := budget.maxEstimatedTokens > 0 && estimatedTokens > budget.maxEstimatedTokens
overBytes := budget.hardMaxBytes > 0 && originalBytes > budget.hardMaxBytes
switch {
case overTokens && overBytes:
return "token_and_byte_budget"
case overTokens:
return "estimated_token_budget"
default:
return "hard_byte_ceiling"
}
}
type outputBudgetResult struct {
Output string
Truncated bool
@@ -56,6 +218,7 @@ func estimatedTokensFromBytes(bytes int) int {
type outputBudgetBuilder struct {
builder strings.Builder
capture *boundedBuffer
rawBytes int
maxBytes int
hint string
@@ -65,17 +228,60 @@ func newOutputBudgetBuilder(maxBytes int, hint string) *outputBudgetBuilder {
return &outputBudgetBuilder{maxBytes: maxBytes, hint: hint}
}
// newHeadTailOutputBudgetBuilder bounds capture while retaining both ends of
// an input for a later semantic policy. It is used for registry-driven file
// reads; direct Tool.Run calls keep the legacy prefix-only builder above.
func newHeadTailOutputBudgetBuilder(maxBytes int, hint string) *outputBudgetBuilder {
headBytes := maxBytes / 2
return &outputBudgetBuilder{
capture: newBoundedBuffer(headBytes, maxBytes-headBytes),
maxBytes: maxBytes,
hint: hint,
}
}
func (builder *outputBudgetBuilder) WriteString(value string) {
builder.rawBytes += len(value)
if builder.maxBytes <= 0 || builder.builder.Len() >= builder.maxBytes {
if builder.capture != nil {
_, _ = builder.capture.Write([]byte(value))
return
}
if builder.maxBytes <= 0 {
builder.builder.WriteString(value)
return
}
if builder.builder.Len() >= builder.maxBytes {
return
}
remaining := builder.maxBytes - builder.builder.Len()
builder.builder.WriteString(utf8Prefix(value, remaining))
}
// applyLegacyByteBudgetToResult preserves direct Tool.Run behavior for callers
// that intentionally bypass Registry.RunWithOptions. Agent/MCP execution uses
// the registry's post-redaction semantic boundary instead.
func applyLegacyByteBudgetToResult(result Result, maxBytes int, hint string) Result {
budgeted := applyOutputBudget(result.Output, maxBytes, hint)
result.Output = budgeted.Output
result.Truncated = result.Truncated || budgeted.Truncated
if result.Meta == nil {
result.Meta = map[string]string{}
}
for key, value := range outputBudgetMeta(budgeted) {
result.Meta[key] = value
}
if budgeted.Truncated {
result.Meta["truncated"] = "true"
result.Meta["truncation_reason"] = "byte_budget"
}
return result
}
func (builder *outputBudgetBuilder) Result() outputBudgetResult {
output := builder.builder.String()
if builder.capture != nil {
output = builder.capture.retained()
}
result := outputBudgetResult{
Output: output,
RawBytes: builder.rawBytes,
@@ -90,7 +296,12 @@ func (builder *outputBudgetBuilder) Result() outputBudgetResult {
if budget < 0 {
budget = 0
}
result.Output = utf8Prefix(output, budget) + marker
if builder.capture != nil {
headBytes := budget / 2
result.Output = utf8Prefix(output, headBytes) + marker + utf8Suffix(output, budget-headBytes)
} else {
result.Output = utf8Prefix(output, budget) + marker
}
result.Truncated = true
result.EmittedBytes = len(result.Output)
return result
+90
View File
@@ -0,0 +1,90 @@
package tools
import (
"strings"
"testing"
"unicode/utf8"
)
func TestEstimateOutputTokensASCIIAndUnicode(t *testing.T) {
if got := estimateOutputTokens("abcd efgh"); got != 2 {
t.Fatalf("ASCII estimate = %d, want 2", got)
}
unicodeText := "🙂界"
if got := estimateOutputTokens(unicodeText); got < len([]byte(unicodeText)) {
t.Fatalf("Unicode estimate = %d, want conservative estimate >= %d", got, len([]byte(unicodeText)))
}
if first, second := estimateOutputTokens(unicodeText), estimateOutputTokens(unicodeText); first != second {
t.Fatalf("estimator is not deterministic: %d != %d", first, second)
}
if input := strings.Repeat("\u0080", 4); estimateOutputTokens(input) < len([]byte(input)) {
t.Fatalf("U+0080 estimate = %d, want conservative estimate >= %d", estimateOutputTokens(input), len([]byte(input)))
}
}
func TestBudgetDefaultOutputLeavesSmallOutputByteIdentical(t *testing.T) {
input := "small output\nwith spacing\n"
got := budgetDefaultOutput(input, outputBudget{maxEstimatedTokens: 100, hardMaxBytes: 1024})
if got.text != input {
t.Fatalf("small output changed: got %q want %q", got.text, input)
}
if got.truncated || got.originalBytes != len(input) || got.retainedBytes != len(input) {
t.Fatalf("unexpected small-output metadata: %#v", got)
}
}
func TestBudgetDefaultOutputKeepsUTF8HeadTailWithinHardCeiling(t *testing.T) {
input := "HEAD🙂\n" + strings.Repeat("界", 200) + "\nTAIL🙂"
const hardMax = 120
got := budgetDefaultOutput(input, outputBudget{maxEstimatedTokens: 10_000, hardMaxBytes: hardMax})
if !got.truncated {
t.Fatal("large output was not truncated")
}
if len(got.text) > hardMax {
t.Fatalf("retained %d bytes, hard ceiling %d", len(got.text), hardMax)
}
if !utf8.ValidString(got.text) {
t.Fatalf("budgeted output is invalid UTF-8: %q", got.text)
}
for _, want := range []string{"HEAD", "TAIL", "output truncated"} {
if !strings.Contains(got.text, want) {
t.Fatalf("budgeted output missing %q: %q", want, got.text)
}
}
}
func TestBudgetDefaultOutputHonorsEstimatedTokenBudget(t *testing.T) {
input := strings.Repeat("🙂", 100)
got := budgetDefaultOutput(input, outputBudget{maxEstimatedTokens: 60, hardMaxBytes: 1024})
if !got.truncated || got.reason != "estimated_token_budget" {
t.Fatalf("unexpected token-budget result: %#v", got)
}
if got.estimatedRetainedTokens > 60 {
t.Fatalf("retained estimate = %d, budget 60", got.estimatedRetainedTokens)
}
}
func TestBudgetDefaultOutputDeterministic(t *testing.T) {
input := "head\n" + strings.Repeat("same line\n", 1000) + "tail\n"
budget := outputBudget{maxEstimatedTokens: 80, hardMaxBytes: 512}
want := budgetDefaultOutput(input, budget)
for iteration := 0; iteration < 20; iteration++ {
got := budgetDefaultOutput(input, budget)
if got != want {
t.Fatalf("iteration %d differs:\n got %#v\nwant %#v", iteration, got, want)
}
}
}
func TestBudgetDefaultOutputTinyBudgetsNeverExceedTheirCeilings(t *testing.T) {
input := strings.Repeat("oversized output ", 100)
for _, budget := range []outputBudget{
{hardMaxBytes: len(defaultSemanticTruncationNotice) - 1},
{maxEstimatedTokens: estimateOutputTokens(defaultSemanticTruncationNotice) - 1},
} {
got := budgetDefaultOutput(input, budget)
if !got.truncated || !fitsOutputBudget(got.text, budget) {
t.Fatalf("tiny budget result does not fit: budget=%#v result=%#v", budget, got)
}
}
}
+17 -15
View File
@@ -25,23 +25,19 @@ const defaultOutputCeilingTokens = 16_000
// disables the ceiling entirely; unset or unparsable keeps the default.
const outputCeilingEnv = "ZERO_TOOL_OUTPUT_CEILING_TOKENS"
// selfBudgeting marks a tool that enforces its own deliberate output budget —
// possibly model-raisable (exec_command) — which the registry ceiling must not
// second-guess. The method is unexported on purpose: only tools in this
// package can opt out, so an MCP-served tool can never exempt itself.
// selfBudgeting marks a tool with a deliberate capture-aware output budget —
// possibly model-raisable (exec_command). The registry applies semantic
// retention within that explicit budget rather than replacing it with the
// default ceiling. The method is unexported so an MCP-served tool cannot opt
// into this path.
type selfBudgeting interface{ managesOutputBudget() }
// The exemption list, kept in one place. Each of these applies its own budget
// before returning: bash (bashOutputBudgetBytes per stream + spill),
// exec_command (model-raisable token budget + spill), read tools (128 KiB),
// search tools (64 KiB).
func (bashTool) managesOutputBudget() {}
func (execCommandTool) managesOutputBudget() {}
func (readFileTool) managesOutputBudget() {}
func (readMinifiedFileTool) managesOutputBudget() {}
func (grepTool) managesOutputBudget() {}
func (globTool) managesOutputBudget() {}
func (listDirectoryTool) managesOutputBudget() {}
// The list is kept in one place. Shell/process tools retain their established
// capture-aware budgets: bash (per-stream bounded capture) and exec_command
// (model-raisable bounded session output). File/search tools use the standard
// shared post-redaction boundary.
func (bashTool) managesOutputBudget() {}
func (execCommandTool) managesOutputBudget() {}
func resolveOutputCeilingTokens() int {
raw := strings.TrimSpace(os.Getenv(outputCeilingEnv))
@@ -61,6 +57,12 @@ func resolveOutputCeilingTokens() int {
// and the spill file agree on what was hidden.
func enforceOutputCeiling(toolName string, result Result) Result {
ceiling := resolveOutputCeilingTokens()
switch toolName {
case "read_file", "read_minified_file":
ceiling = readOutputBudgetBytes / 4
case "grep", "glob", "list_directory":
ceiling = searchOutputBudgetBytes / 4
}
if ceiling <= 0 {
return result
}
+7 -7
View File
@@ -55,10 +55,10 @@ func TestOutputCeilingCapsUnbudgetedTool(t *testing.T) {
if result.Meta["raw_bytes"] != strconv.Itoa(len(big)) {
t.Fatalf("raw_bytes = %s, want %d", result.Meta["raw_bytes"], len(big))
}
if !strings.Contains(result.Output, "full output saved to ") {
if !strings.Contains(result.Output, "full output received by budgeting layer saved to ") {
t.Fatalf("ceiling truncation must include a spill hint: %q", result.Output[:200])
}
start := strings.Index(result.Output, "full output saved to ") + len("full output saved to ")
start := strings.Index(result.Output, "full output received by budgeting layer saved to ") + len("full output received by budgeting layer saved to ")
end := strings.Index(result.Output[start:], " (grep")
content, err := os.ReadFile(result.Output[start : start+end])
if err != nil {
@@ -123,11 +123,6 @@ func TestSelfBudgetingExemptionList(t *testing.T) {
exempt := []Tool{
NewBashTool(dir),
NewExecCommandTool(dir, newExecSessionManager()),
NewReadFileTool(dir),
NewReadMinifiedFileTool(dir),
NewGrepTool(dir),
NewGlobTool(dir),
NewListDirectoryTool(dir),
}
for _, tool := range exempt {
if _, ok := tool.(selfBudgeting); !ok {
@@ -137,4 +132,9 @@ func TestSelfBudgetingExemptionList(t *testing.T) {
if _, ok := NewWebFetchTool().(selfBudgeting); ok {
t.Error("web_fetch must NOT be exempt — the ceiling is its only budget")
}
for _, tool := range []Tool{NewReadFileTool(dir), NewReadMinifiedFileTool(dir), NewGrepTool(dir), NewGlobTool(dir), NewListDirectoryTool(dir)} {
if _, ok := tool.(selfBudgeting); ok {
t.Errorf("%s must use the shared post-redaction budget boundary", tool.Name())
}
}
}
+447
View File
@@ -0,0 +1,447 @@
package tools
import (
"fmt"
"sort"
"strconv"
"strings"
)
// budgetSemanticOutput applies the deterministic policy for category. The
// caller supplies output after secret redaction. Small results pass through
// byte-identically; only oversized results are semantically reduced.
func budgetSemanticOutput(output string, category outputCategory, budget outputBudget) budgetedOutput {
if category == "" {
category = outputCategoryDefault
}
if fitsOutputBudget(output, budget) {
return unchangedBudgetedOutput(output, category)
}
var retained string
switch category {
case outputCategoryFile:
retained = budgetFileLines(output, budget)
case outputCategorySearch:
retained = budgetSearchLines(output, budget)
case outputCategoryTest:
retained = budgetTestLines(output, budget)
case outputCategoryProcess:
retained = budgetProcessLines(output, budget)
case outputCategoryDiff:
retained = budgetDiffUnits(output, budget)
case outputCategoryWorker:
retained = budgetWorkerLines(output, budget)
default:
return budgetDefaultOutput(output, budget)
}
if retained == "" || !fitsOutputBudget(retained, budget) {
fallback := budgetDefaultOutput(output, budget)
fallback.category = category
return fallback
}
return truncatedBudgetedOutput(output, retained, category, "semantic_"+string(category)+"_budget")
}
func unchangedBudgetedOutput(output string, category outputCategory) budgetedOutput {
estimated := estimateOutputTokens(output)
return budgetedOutput{
text: output,
originalBytes: len(output),
retainedBytes: len(output),
estimatedOriginalTokens: estimated,
estimatedRetainedTokens: estimated,
category: category,
}
}
func truncatedBudgetedOutput(original, retained string, category outputCategory, reason string) budgetedOutput {
return budgetedOutput{
text: retained,
originalBytes: len(original),
retainedBytes: len(retained),
estimatedOriginalTokens: estimateOutputTokens(original),
estimatedRetainedTokens: estimateOutputTokens(retained),
truncated: true,
category: category,
reason: reason,
}
}
func budgetFileLines(output string, budget outputBudget) string {
lines := outputLines(output)
priorities := make([]int, 0, len(lines))
// Keep the existing file/range header first, then alternate from the start
// and end so both requested-location boundaries survive.
if len(lines) > 0 {
priorities = append(priorities, 0)
}
for left, right := 1, len(lines)-1; left <= right; left, right = left+1, right-1 {
priorities = append(priorities, left)
if right != left {
priorities = append(priorities, right)
}
}
units := lineDiffUnits(lines)
selected := selectPrioritizedUnits(units, priorities, budget)
for index := range selected {
// Index zero is the file/range header. If no non-blank requested
// content line fits as a complete line, let the common caller fall
// back to its UTF-8-safe head/tail policy rather than returning only
// a header and omission marker for a minified single-line file.
if index > 0 && strings.TrimSpace(lines[index]) != "" {
return renderSelectedUnits(units, selected)
}
}
return ""
}
func budgetSearchLines(output string, budget outputBudget) string {
lines := collapseConsecutiveDuplicateLines(outputLines(output))
priorities := make([]int, 0, len(lines))
for index, line := range lines {
lower := strings.ToLower(line)
// A match body naturally contains the search term (often literally
// "match"); treat only non-result lines as summaries so one busy file
// cannot consume the budget before cross-file representatives are chosen.
if searchResultFile(line) == "" && (strings.Contains(lower, "match") || strings.Contains(lower, "truncated") || strings.Contains(lower, "result")) {
priorities = append(priorities, index)
}
}
if len(lines) > 0 {
priorities = append(priorities, 0, len(lines)-1)
}
seenFiles := map[string]bool{}
for index, line := range lines {
if file := searchResultFile(line); file != "" && !seenFiles[file] {
seenFiles[file] = true
priorities = append(priorities, index)
}
}
priorities = append(priorities, sequence(len(lines))...)
return retainPrioritizedLines(lines, priorities, budget)
}
func budgetTestLines(output string, budget outputBudget) string {
lines := collapseConsecutiveDuplicateLines(outputLines(output))
priorities := make([]int, 0, len(lines))
for index, line := range lines {
if isTestFailureLine(line) {
for contextIndex := max(0, index-2); contextIndex <= min(len(lines)-1, index+3); contextIndex++ {
priorities = append(priorities, contextIndex)
}
}
}
// Final summaries and process status usually live at the tail.
for index := max(0, len(lines)-16); index < len(lines); index++ {
priorities = append(priorities, index)
}
for index := 0; index < min(8, len(lines)); index++ {
priorities = append(priorities, index)
}
priorities = append(priorities, sequence(len(lines))...)
return retainPrioritizedLines(lines, priorities, budget)
}
func budgetProcessLines(output string, budget outputBudget) string {
lines := collapseConsecutiveDuplicateLines(outputLines(output))
priorities := make([]int, 0, len(lines))
for index := 0; index < min(10, len(lines)); index++ {
priorities = append(priorities, index)
}
seenDiagnostic := map[string]bool{}
for index, line := range lines {
lower := strings.ToLower(line)
if containsAny(lower, "error", "warning", "warn:", "fatal", "panic", "failed", "denied") && !seenDiagnostic[line] {
seenDiagnostic[line] = true
priorities = append(priorities, index)
}
}
for index := max(0, len(lines)-16); index < len(lines); index++ {
priorities = append(priorities, index)
}
priorities = append(priorities, sequence(len(lines))...)
return retainPrioritizedLines(lines, priorities, budget)
}
func budgetWorkerLines(output string, budget outputBudget) string {
lines := collapseConsecutiveDuplicateLines(outputLines(output))
priorities := make([]int, 0, len(lines))
for index, line := range lines {
lower := strings.ToLower(line)
if containsAny(lower, "status", "error", "failed", "failure", "session_id", "changed", "files", "tools executed", "conclusion", "result") {
priorities = append(priorities, index)
}
}
// A specialist's final conclusion is conventionally at the end.
for index := max(0, len(lines)-20); index < len(lines); index++ {
priorities = append(priorities, index)
}
for index := 0; index < min(6, len(lines)); index++ {
priorities = append(priorities, index)
}
priorities = append(priorities, sequence(len(lines))...)
return retainPrioritizedLines(lines, priorities, budget)
}
type diffUnit struct {
order int
text string
key bool
}
func budgetDiffUnits(output string, budget outputBudget) string {
units := splitDiffUnits(output)
if len(units) == 0 {
return ""
}
priority := make([]int, 0, len(units))
for index, unit := range units {
if unit.key {
priority = append(priority, index)
}
}
priority = append(priority, sequence(len(units))...)
return retainPrioritizedUnits(units, priority, budget)
}
// splitDiffUnits keeps each hunk indivisible. File headers/stat text are key
// units so broad file coverage is selected before secondary hunks.
func splitDiffUnits(output string) []diffUnit {
lines := outputLines(output)
units := make([]diffUnit, 0)
var current []string
currentKey := false
flush := func() {
if len(current) == 0 {
return
}
units = append(units, diffUnit{order: len(units), text: strings.Join(current, "\n"), key: currentKey})
current = nil
currentKey = false
}
for _, line := range lines {
if strings.HasPrefix(line, "diff --git ") {
flush()
currentKey = true
current = append(current, line)
continue
}
if strings.HasPrefix(line, "@@ ") {
flush()
current = append(current, line)
continue
}
if len(current) == 0 {
currentKey = true // diff stat/summary or standalone file headers
}
current = append(current, line)
}
flush()
return units
}
func retainPrioritizedLines(lines []string, priorities []int, budget outputBudget) string {
return retainPrioritizedUnits(lineDiffUnits(lines), priorities, budget)
}
func lineDiffUnits(lines []string) []diffUnit {
units := make([]diffUnit, 0, len(lines))
for index, line := range lines {
units = append(units, diffUnit{order: index, text: line})
}
return units
}
func retainPrioritizedUnits(units []diffUnit, priorities []int, budget outputBudget) string {
return renderSelectedUnits(units, selectPrioritizedUnits(units, priorities, budget))
}
func selectPrioritizedUnits(units []diffUnit, priorities []int, budget outputBudget) map[int]bool {
selected := map[int]bool{}
indexes := make([]int, 0, len(priorities))
cost := retainedUnitCost{}
for _, index := range priorities {
if index < 0 || index >= len(units) || selected[index] {
continue
}
position := sort.SearchInts(indexes, index)
previous, next := -1, len(units)
if position > 0 {
previous = indexes[position-1]
}
if position < len(indexes) {
next = indexes[position]
}
candidate := cost
candidate.add(units[index].text)
if len(indexes) == 0 {
candidate.addOmission(index)
candidate.addOmission(len(units) - index - 1)
} else {
candidate.removeOmission(next - previous - 1)
candidate.addOmission(index - previous - 1)
candidate.addOmission(next - index - 1)
}
if !candidate.fits(budget) {
continue
}
selected[index] = true
indexes = append(indexes, 0)
copy(indexes[position+1:], indexes[position:])
indexes[position] = index
cost = candidate
}
return selected
}
// retainedUnitCost tracks the exact rendered size of selected units without
// repeatedly rebuilding the full candidate text. Newlines between rendered
// parts add bytes but no estimated tokens, so the token components remain
// additive even as priorities select units out of source order.
type retainedUnitCost struct {
textBytes int
asciiNonSpace int
nonASCIIBytes int
parts int
}
func (cost *retainedUnitCost) add(text string) {
ascii, nonASCII := outputTokenComponents(text)
cost.textBytes += len(text)
cost.asciiNonSpace += ascii
cost.nonASCIIBytes += nonASCII
cost.parts++
}
func (cost *retainedUnitCost) remove(text string) {
ascii, nonASCII := outputTokenComponents(text)
cost.textBytes -= len(text)
cost.asciiNonSpace -= ascii
cost.nonASCIIBytes -= nonASCII
cost.parts--
}
func (cost *retainedUnitCost) addOmission(count int) {
if count > 0 {
cost.add(omittedSectionsMarker(count))
}
}
func (cost *retainedUnitCost) removeOmission(count int) {
if count > 0 {
cost.remove(omittedSectionsMarker(count))
}
}
func (cost retainedUnitCost) fits(budget outputBudget) bool {
bytes := cost.textBytes + max(0, cost.parts-1)
tokens := (cost.asciiNonSpace+3)/4 + cost.nonASCIIBytes
return (budget.hardMaxBytes <= 0 || bytes <= budget.hardMaxBytes) &&
(budget.maxEstimatedTokens <= 0 || tokens <= budget.maxEstimatedTokens)
}
func omittedSectionsMarker(count int) string {
return fmt.Sprintf("[zero] ... %d section(s) omitted ...", count)
}
func renderSelectedUnits(units []diffUnit, selected map[int]bool) string {
indexes := make([]int, 0, len(selected))
for index := range selected {
indexes = append(indexes, index)
}
sort.Ints(indexes)
if len(indexes) == 0 {
return ""
}
parts := make([]string, 0, len(indexes)*2)
previous := -1
for _, index := range indexes {
if previous >= 0 && index != previous+1 {
omitted := index - previous - 1
parts = append(parts, omittedSectionsMarker(omitted))
} else if previous < 0 && index > 0 {
parts = append(parts, omittedSectionsMarker(index))
}
parts = append(parts, units[index].text)
previous = index
}
if previous < len(units)-1 {
parts = append(parts, omittedSectionsMarker(len(units)-previous-1))
}
return strings.Join(parts, "\n")
}
func outputLines(output string) []string {
return strings.Split(strings.TrimRight(output, "\r\n"), "\n")
}
func collapseConsecutiveDuplicateLines(lines []string) []string {
if len(lines) < 2 {
return lines
}
result := make([]string, 0, len(lines))
for index := 0; index < len(lines); {
end := index + 1
for end < len(lines) && lines[end] == lines[index] {
end++
}
result = append(result, lines[index])
if count := end - index; count > 1 {
result = append(result, fmt.Sprintf("[zero] previous line repeated %d more time(s)", count-1))
}
index = end
}
return result
}
func searchResultFile(line string) string {
// Search output is path:line: text. Locate the first numeric field from
// the left after an optional Windows drive prefix. Match text may itself
// contain numeric colon-delimited fields, which are not record structure.
searchFrom := 0
if len(line) >= 2 && line[1] == ':' && ((line[0] >= 'A' && line[0] <= 'Z') || (line[0] >= 'a' && line[0] <= 'z')) {
searchFrom = 2
}
for searchFrom < len(line) {
lineStartOffset := strings.IndexByte(line[searchFrom:], ':')
if lineStartOffset < 0 {
return ""
}
lineStart := searchFrom + lineStartOffset
lineEndOffset := strings.IndexByte(line[lineStart+1:], ':')
if lineEndOffset < 0 {
return ""
}
lineEnd := lineStart + 1 + lineEndOffset
if _, err := strconv.Atoi(line[lineStart+1 : lineEnd]); err == nil {
return strings.TrimSpace(line[:lineStart])
}
searchFrom = lineStart + 1
}
return ""
}
func isTestFailureLine(line string) bool {
lower := strings.ToLower(line)
return containsAny(lower, "--- fail:", "failed", "failure", "panic", "assert", "error", "fatal", "expected", "actual:")
}
func containsAny(value string, needles ...string) bool {
for _, needle := range needles {
if strings.Contains(value, needle) {
return true
}
}
return false
}
func sequence(length int) []int {
result := make([]int, length)
for index := range result {
result[index] = index
}
return result
}
+144
View File
@@ -0,0 +1,144 @@
package tools
import (
"fmt"
"strings"
"testing"
"unicode/utf8"
)
func semanticTestBudget() outputBudget {
return outputBudget{maxEstimatedTokens: 140, hardMaxBytes: 900}
}
func TestFileOutputPolicyKeepsCompleteHeadAndTailLines(t *testing.T) {
lines := []string{"File: huge.go (500 lines)", "1 | package main"}
for index := 2; index < 500; index++ {
lines = append(lines, fmt.Sprintf("%d | value := %d", index, index))
}
lines = append(lines, "500 | // END_MARK")
got := budgetSemanticOutput(strings.Join(lines, "\n"), outputCategoryFile, semanticTestBudget())
if !got.truncated || !strings.Contains(got.text, "File: huge.go") || !strings.Contains(got.text, "END_MARK") {
t.Fatalf("file policy lost boundaries: %#v", got)
}
if !utf8.ValidString(got.text) || strings.Contains(got.text, "value := 49\ufffd") {
t.Fatalf("file policy produced invalid or split text: %q", got.text)
}
}
func TestSearchOutputPolicyPreservesMultiFileCoverage(t *testing.T) {
var lines []string
for _, file := range []string{"a.go", "b.go", "c.go", "d.go"} {
for line := 1; line <= 30; line++ {
lines = append(lines, fmt.Sprintf("%s:%d: match value %d", file, line, line))
}
}
lines = append(lines, "120 matches found")
got := budgetSemanticOutput(strings.Join(lines, "\n"), outputCategorySearch, semanticTestBudget())
for _, want := range []string{"a.go:", "b.go:", "c.go:", "d.go:", "120 matches"} {
if !strings.Contains(got.text, want) {
t.Fatalf("search policy missing %q:\n%s", want, got.text)
}
}
}
func TestSearchOutputPolicyPreservesWindowsPathCoverage(t *testing.T) {
var lines []string
for _, file := range []string{"a.go", "b.go", "c.go", "d.go"} {
for line := 1; line <= 30; line++ {
lines = append(lines, fmt.Sprintf(`C:\workspace\%s:%d: match value %d`, file, line, line))
}
}
lines = append(lines, "120 matches found")
got := budgetSemanticOutput(strings.Join(lines, "\n"), outputCategorySearch, semanticTestBudget())
for _, want := range []string{`C:\workspace\a.go:`, `C:\workspace\b.go:`, `C:\workspace\c.go:`, `C:\workspace\d.go:`, "120 matches"} {
if !strings.Contains(got.text, want) {
t.Fatalf("search policy missing %q:\n%s", want, got.text)
}
}
}
func TestSearchResultFileIgnoresNumericMatchContent(t *testing.T) {
tests := []struct {
line string
want string
}{
{line: `C:\workspace\a.go:12: error code:123: failed`, want: `C:\workspace\a.go`},
{line: `/workspace/a.go:12: error code:123: failed`, want: `/workspace/a.go`},
{line: `/workspace/name:with-colon.go:12: match`, want: `/workspace/name:with-colon.go`},
}
for _, test := range tests {
if got := searchResultFile(test.line); got != test.want {
t.Errorf("searchResultFile(%q) = %q, want %q", test.line, got, test.want)
}
}
}
func TestProcessOutputPolicyCollapsesRepetitiveLogsAndKeepsDiagnostics(t *testing.T) {
input := "starting server\n" + strings.Repeat("polling...\n", 300) + "WARNING: queue slow\nERROR: request failed\nshutdown complete"
got := budgetSemanticOutput(input, outputCategoryProcess, semanticTestBudget())
for _, want := range []string{"starting server", "repeated", "WARNING", "ERROR", "shutdown complete"} {
if !strings.Contains(got.text, want) {
t.Fatalf("process policy missing %q:\n%s", want, got.text)
}
}
if strings.Count(got.text, "polling...") > 1 {
t.Fatalf("repetitive line was not collapsed:\n%s", got.text)
}
}
func TestTestOutputPolicyKeepsFailureAndFinalSummary(t *testing.T) {
input := "=== RUN TestSuite\n" + strings.Repeat("ok progress\n", 300) +
"--- FAIL: TestImportant (0.01s)\n thing_test.go:42: expected 7, got 9\nFAIL\nexit status 1"
got := budgetSemanticOutput(input, outputCategoryTest, semanticTestBudget())
for _, want := range []string{"TestImportant", "expected 7", "FAIL", "exit status 1"} {
if !strings.Contains(got.text, want) {
t.Fatalf("test policy missing %q:\n%s", want, got.text)
}
}
}
func TestDiffOutputPolicyKeepsFilesAndCompleteHunks(t *testing.T) {
var diff strings.Builder
for file := 1; file <= 4; file++ {
fmt.Fprintf(&diff, "diff --git a/f%d.go b/f%d.go\n--- a/f%d.go\n+++ b/f%d.go\n", file, file, file, file)
for hunk := 1; hunk <= 8; hunk++ {
fmt.Fprintf(&diff, "@@ -%d,2 +%d,2 @@\n-old%d_%d\n+new%d_%d\n", hunk, hunk, file, hunk, file, hunk)
}
}
got := budgetSemanticOutput(diff.String(), outputCategoryDiff, outputBudget{maxEstimatedTokens: 220, hardMaxBytes: 1400})
for file := 1; file <= 4; file++ {
if !strings.Contains(got.text, fmt.Sprintf("diff --git a/f%d.go", file)) {
t.Fatalf("diff policy lost file %d header:\n%s", file, got.text)
}
}
for _, line := range strings.Split(got.text, "\n") {
if strings.HasPrefix(line, "@@ ") {
// Each retained hunk is an indivisible three-line unit.
if !strings.Contains(got.text, line+"\n-old") {
t.Fatalf("hunk header was sliced from its body: %q", line)
}
}
}
}
func TestWorkerOutputPolicyKeepsStatusErrorsAndConclusion(t *testing.T) {
input := "session_id: child_1\nstatus: error\n" + strings.Repeat("worker progress\n", 200) +
"tools executed: grep, read_file\nchanged files: a.go, b.go\nerror: verification failed\nConclusion: fix the nil check before merging."
got := budgetSemanticOutput(input, outputCategoryWorker, semanticTestBudget())
for _, want := range []string{"session_id", "status: error", "tools executed", "changed files", "verification failed", "Conclusion"} {
if !strings.Contains(got.text, want) {
t.Fatalf("worker policy missing %q:\n%s", want, got.text)
}
}
}
func TestSemanticPoliciesAreDeterministic(t *testing.T) {
input := "start\n" + strings.Repeat("tick\n", 500) + "ERROR: boom\nend"
want := budgetSemanticOutput(input, outputCategoryProcess, semanticTestBudget())
for iteration := 0; iteration < 20; iteration++ {
if got := budgetSemanticOutput(input, outputCategoryProcess, semanticTestBudget()); got != want {
t.Fatalf("iteration %d differs:\n got %#v\nwant %#v", iteration, got, want)
}
}
}
+39 -9
View File
@@ -18,6 +18,8 @@ type readFileTool struct {
scope PathScope
}
func (readFileTool) outputCategory(map[string]any) outputCategory { return outputCategoryFile }
func NewReadFileTool(workspaceRoot string) Tool {
return NewScopedReadFileTool(workspaceRoot, nil)
}
@@ -39,8 +41,10 @@ func NewScopedReadFileTool(workspaceRoot string, scope PathScope) Tool {
AdditionalProperties: false,
},
safety: readOnlySafety("Reads file contents without modifying files."),
// ThreadSafe=false: may update FileTracker session state on read.
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: fileResourceKeys},
// ThreadSafe: FileTracker is mutex-guarded; concurrent reads of
// distinct paths are safe. Same-path calls still serialize via
// resource-key conflict detection in the agent parallel planner.
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: fileResourceKeys},
},
workspaceRoot: normalizeWorkspaceRoot(workspaceRoot),
scope: scope,
@@ -48,10 +52,14 @@ func NewScopedReadFileTool(workspaceRoot string, scope PathScope) Tool {
}
func (tool readFileTool) Run(ctx context.Context, args map[string]any) Result {
return tool.RunWithOptions(ctx, args, RunOptions{})
return tool.run(args, RunOptions{}, true)
}
func (tool readFileTool) RunWithOptions(_ context.Context, args map[string]any, options RunOptions) Result {
return tool.run(args, options, false)
}
func (tool readFileTool) run(args map[string]any, options RunOptions, directBudget bool) Result {
requestedPath, err := aliasedStringArg(args, []string{"path", "file", "file_path", "filepath", "filename"}, "", true, false)
if err != nil {
return errorResult("Error: Invalid arguments for read_file: " + err.Error())
@@ -84,10 +92,14 @@ func (tool readFileTool) RunWithOptions(_ context.Context, args map[string]any,
// not the authoritative content hash.
options.FileTracker.RecordHash(absolutePath, stats.hash, stats.info)
return renderReadFileRange(absolutePath, relativePath, stats.lines, startLine, endLine, maxLines)
maxBytes := 0
if directBudget {
maxBytes = readOutputBudgetBytes
}
return renderReadFileRange(absolutePath, relativePath, stats.lines, startLine, endLine, maxLines, maxBytes)
}
func renderReadFileRange(absolutePath string, relativePath string, total int, startLine int, endLine int, maxLines int) Result {
func renderReadFileRange(absolutePath string, relativePath string, total int, startLine int, endLine int, maxLines int, maxBytes int) Result {
if startLine > total {
return okResult(fmt.Sprintf("File: %s\n(start_line %d is past the end of the file, which has %d lines)", relativePath, startLine, total))
}
@@ -112,7 +124,16 @@ func renderReadFileRange(absolutePath string, relativePath string, total int, st
header = fmt.Sprintf("File: %s (lines %d-%d of %d)", relativePath, startLine, lastLine, total)
}
budgetedOutput := newOutputBudgetBuilder(readOutputBudgetBytes, "use start_line/end_line or max_lines to continue with a smaller range")
// The shared registry boundary applies the file policy after redaction. Its
// input still needs a bounded capture: retain the requested range's head and
// tail instead of materializing an arbitrarily large file in memory. Direct
// Tool.Run calls preserve the legacy prefix-only byte budget.
var budgetedOutput *outputBudgetBuilder
if maxBytes > 0 {
budgetedOutput = newOutputBudgetBuilder(maxBytes, "use start_line/end_line or max_lines to continue with a smaller range")
} else {
budgetedOutput = newHeadTailOutputBudgetBuilder(readOutputBudgetBytes, "use start_line/end_line or max_lines to continue with a smaller range")
}
budgetedOutput.WriteString(header)
budgetedOutput.WriteString("\n\n")
if err := appendReadFileRange(budgetedOutput, absolutePath, startLine, selectedLines, width); err != nil {
@@ -126,10 +147,19 @@ func renderReadFileRange(absolutePath string, relativePath string, total int, st
}
budgeted := budgetedOutput.Result()
meta := outputBudgetMeta(budgeted)
if budgeted.Truncated {
meta := map[string]string{}
if maxBytes > 0 || budgeted.Truncated {
for key, value := range outputBudgetMeta(budgeted) {
meta[key] = value
}
}
if truncated || budgeted.Truncated {
meta["truncated"] = "true"
meta["truncation_reason"] = "byte_budget"
if budgeted.Truncated {
meta["truncation_reason"] = "byte_budget"
} else {
meta["truncation_reason"] = "max_lines"
}
}
return Result{
Status: StatusOK,
+16 -10
View File
@@ -16,6 +16,10 @@ type readMinifiedFileTool struct {
scope PathScope
}
func (readMinifiedFileTool) outputCategory(map[string]any) outputCategory {
return outputCategoryFile
}
func NewReadMinifiedFileTool(workspaceRoot string) Tool {
return NewScopedReadMinifiedFileTool(workspaceRoot, nil)
}
@@ -34,7 +38,7 @@ func NewScopedReadMinifiedFileTool(workspaceRoot string, scope PathScope) Tool {
AdditionalProperties: false,
},
safety: readOnlySafety("Reads a minified view of file contents without modifying files."),
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: fileResourceKeys},
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: fileResourceKeys},
},
workspaceRoot: normalizeWorkspaceRoot(workspaceRoot),
scope: scope,
@@ -42,10 +46,14 @@ func NewScopedReadMinifiedFileTool(workspaceRoot string, scope PathScope) Tool {
}
func (tool readMinifiedFileTool) Run(ctx context.Context, args map[string]any) Result {
return tool.RunWithOptions(ctx, args, RunOptions{})
return tool.run(args, RunOptions{}, true)
}
func (tool readMinifiedFileTool) RunWithOptions(_ context.Context, args map[string]any, options RunOptions) Result {
return tool.run(args, options, false)
}
func (tool readMinifiedFileTool) run(args map[string]any, options RunOptions, directBudget bool) Result {
requestedPath, err := aliasedStringArg(args, []string{"path", "file", "file_path", "filepath", "filename"}, "", true, false)
if err != nil {
return errorResult("Error: Invalid arguments for read_minified_file: " + err.Error())
@@ -92,23 +100,21 @@ func (tool readMinifiedFileTool) RunWithOptions(_ context.Context, args map[stri
savedTokens = estimatedTokensFromBytes(savedBytes)
}
output := header + "\n\n" + result.Content
budgeted := applyOutputBudget(output, readOutputBudgetBytes, "use read_file with start_line/end_line or max_lines for a smaller exact range")
meta := outputBudgetMeta(budgeted)
meta := map[string]string{}
meta["path"] = relativePath
meta["mode"] = result.Language
meta["compacted"] = strconv.FormatBool(result.Applied)
meta["raw_bytes"] = strconv.Itoa(rawBytes)
meta["compact_bytes"] = strconv.Itoa(compactBytes)
meta["emitted_bytes"] = strconv.Itoa(budgeted.EmittedBytes)
meta["emitted_bytes"] = strconv.Itoa(len(output))
meta["raw_lines"] = strconv.Itoa(rawLines)
meta["emitted_lines"] = strconv.Itoa(minLines)
meta["estimated_tokens_saved"] = strconv.Itoa(savedTokens)
if budgeted.Truncated {
meta["truncated"] = "true"
meta["truncation_reason"] = "byte_budget"
toolResult := Result{Status: StatusOK, Output: output, Meta: meta}
if directBudget {
return applyLegacyByteBudgetToResult(toolResult, readOutputBudgetBytes, "use read_file with start_line/end_line or max_lines for a smaller exact range")
}
return Result{Status: StatusOK, Output: budgeted.Output, Truncated: budgeted.Truncated, Meta: meta}
return toolResult
}
// lineCount reports the number of newline-separated lines in s (an empty string
+9 -4
View File
@@ -126,20 +126,25 @@ func (registry *Registry) RunWithOptions(ctx context.Context, name string, args
// args/paths) are redacted at the boundary just like tool output. The output
// ceiling runs after the scrub so the transcript and the spill file agree on
// what was hidden.
ceilingExempt := false
selfManagedOutput := false
var tool Tool
var ok bool
defer func() {
result = scrubResultSecrets(result)
if !ceilingExempt {
if selfManagedOutput {
result = applySelfManagedOutputBudget(tool, name, args, result)
} else {
result = applyRegistryOutputBudget(tool, name, args, result)
result = enforceOutputCeiling(name, result)
}
}()
tool, ok := registry.Get(name)
tool, ok = registry.Get(name)
if !ok {
return errorResult(`Error: Unknown tool "` + name + `".`)
}
if _, ok := tool.(selfBudgeting); ok {
ceilingExempt = true
selfManagedOutput = true
}
if rejecter, ok := tool.(PrePermissionRejecter); ok {
if res, rejected := rejecter.RejectBeforePermission(args); rejected {
+14
View File
@@ -127,6 +127,20 @@ func workspaceResourceKeys(_ map[string]any) []string {
return []string{ResourceKeyWorkspace + "root"}
}
// scopedScanResourceKeys returns conflict keys for directory-scoped scan tools
// (glob, grep). Workspace-wide scans (missing / empty / ".") return nil so
// they do not force a false conflict with every other concurrent scan —
// ThreadSafe is the concurrency gate; keys refine same-directory collisions.
// A non-default path/cwd yields a single directory: key.
func scopedScanResourceKeys(args map[string]any) []string {
path := firstStringArg(args, "path", "cwd", "dir", "directory")
normalized := NormalizeResourcePath(path)
if normalized == "" || normalized == "." {
return nil
}
return []string{ResourceKeyDirectory + normalized}
}
// multiFileResourceKeys collects path and paths[] arguments into file: keys.
func multiFileResourceKeys(args map[string]any) []string {
var keys []string
+1 -1
View File
@@ -49,7 +49,7 @@ func NewSkillTool(dir string) *skillTool {
AdditionalProperties: false,
},
safety: readOnlySafety("Reads a local skill file; gathers reusable instructions only."),
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false},
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true},
},
}
}
+5 -2
View File
@@ -141,8 +141,11 @@ func newWebFetchToolWithClientAndResolver(client *http.Client, resolver webFetch
Reason: "Fetches remote URL content over the network.",
AdvertiseInAuto: true,
},
// Network read; ThreadSafe=false until client proven concurrent-safe.
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: endpointResourceKeys},
// Concurrent-safe: net/http.Client is documented as safe for
// concurrent use; each call issues independent request I/O.
// parallelSafeToolCall still requires PermissionAllow (prompted
// network fetches stay sequential until auto-allowed).
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: endpointResourceKeys},
},
client: client,
resolver: resolver,
+28
View File
@@ -117,6 +117,25 @@ func WriteNDJSON(w io.Writer, t *TurnTrace) error {
return err
}
}
// Output budget events stay in tool-result emission order. Sorting them by
// tool/category would destroy correlation with concurrent calls whose results
// are deliberately consumed in original call order.
for _, event := range t.OutputBudgets {
if err := enc.Encode(map[string]any{
"type": "output_budget",
"tool": event.Tool,
"category": event.Category,
"original_bytes": event.OriginalBytes,
"retained_bytes": event.RetainedBytes,
"estimated_original_tokens": event.EstimatedOriginalTokens,
"estimated_retained_tokens": event.EstimatedRetainedTokens,
"truncated": event.Truncated,
"reason": event.Reason,
"spill_created": event.SpillCreated,
}); err != nil {
return err
}
}
return nil
}
@@ -177,6 +196,15 @@ func WriteText(w io.Writer, t *TurnTrace) error {
for _, c := range counters {
write(" %-22s %d\n", c.Name, c.Value)
}
if len(t.OutputBudgets) > 0 {
write("output budgets:\n")
for _, event := range t.OutputBudgets {
write(" tool=%s category=%s bytes=%d/%d tokens=%d/%d truncated=%t reason=%s spill=%t\n",
event.Tool, event.Category, event.RetainedBytes, event.OriginalBytes,
event.EstimatedRetainedTokens, event.EstimatedOriginalTokens,
event.Truncated, event.Reason, event.SpillCreated)
}
}
return firstErr
}
+100
View File
@@ -0,0 +1,100 @@
package trace
import (
"bytes"
"encoding/json"
"io"
"strings"
"testing"
)
func TestOutputBudgetTraceRoundTripContainsNoOutput(t *testing.T) {
recorder := NewRecorder("session", "run", "")
recorder.Start()
event := OutputBudgetEvent{
Tool: "grep",
Category: "search",
OriginalBytes: 10000,
RetainedBytes: 1000,
EstimatedOriginalTokens: 2500,
EstimatedRetainedTokens: 250,
Truncated: true,
Reason: "semantic_search_budget",
SpillCreated: true,
}
recorder.EmitOutputBudget(event)
var encoded bytes.Buffer
if err := WriteNDJSON(&encoded, recorder.Finish()); err != nil {
t.Fatalf("WriteNDJSON: %v", err)
}
const secretOutput = "secret output body"
if strings.Contains(encoded.String(), secretOutput) {
t.Fatal("output_budget trace contains raw secret output")
}
allowedKeys := map[string]bool{
"type": true, "tool": true, "category": true, "original_bytes": true, "retained_bytes": true,
"estimated_original_tokens": true, "estimated_retained_tokens": true, "truncated": true,
"reason": true, "spill_created": true,
}
findUndocumentedKey := func(record map[string]any) string {
for key := range record {
if !allowedKeys[key] {
return key
}
}
return ""
}
decoder := json.NewDecoder(strings.NewReader(encoded.String()))
var outputBudgetRecord map[string]any
for {
var record map[string]any
if err := decoder.Decode(&record); err != nil {
if err == io.EOF {
break
}
t.Fatalf("decode trace record: %v", err)
}
if record["type"] == "output_budget" {
outputBudgetRecord = record
break
}
}
if outputBudgetRecord == nil {
t.Fatal("missing output_budget trace record")
}
if key := findUndocumentedKey(outputBudgetRecord); key != "" {
t.Fatalf("output_budget trace contains undocumented key %q", key)
}
for key := range allowedKeys {
if _, exists := outputBudgetRecord[key]; !exists {
t.Fatalf("output_budget trace missing documented key %q", key)
}
}
recordJSON, err := json.Marshal(outputBudgetRecord)
if err != nil {
t.Fatalf("encode output_budget trace record: %v", err)
}
var serializedEvent OutputBudgetEvent
if err := json.Unmarshal(recordJSON, &serializedEvent); err != nil {
t.Fatalf("decode output_budget trace event: %v", err)
}
if serializedEvent != event {
t.Fatalf("serialized output_budget event = %#v, want %#v", serializedEvent, event)
}
contaminated := make(map[string]any, len(outputBudgetRecord)+1)
for key, value := range outputBudgetRecord {
contaminated[key] = value
}
contaminated["output"] = secretOutput
if key := findUndocumentedKey(contaminated); key == "" {
t.Fatal("trace key validation would allow raw secret output")
}
parsed, err := ReadNDJSON(strings.NewReader(encoded.String()))
if err != nil {
t.Fatalf("ReadNDJSON: %v", err)
}
if len(parsed.OutputBudgets) != 1 || parsed.OutputBudgets[0] != event {
t.Fatalf("unexpected round trip: %#v", parsed.OutputBudgets)
}
}
+21 -1
View File
@@ -116,6 +116,21 @@ func ReadNDJSON(r io.Reader) (*TurnTrace, error) {
SchemaHash: stringField(obj, "schema"),
CompletePrefixHash: stringField(obj, "complete_prefix"),
})
case "output_budget":
if !sawTraceHeader {
return nil, errors.New("parse trace: not a valid NDJSON trace (no type:trace header)")
}
t.OutputBudgets = append(t.OutputBudgets, OutputBudgetEvent{
Tool: stringField(obj, "tool"),
Category: stringField(obj, "category"),
OriginalBytes: int(parseInt64(obj["original_bytes"])),
RetainedBytes: int(parseInt64(obj["retained_bytes"])),
EstimatedOriginalTokens: int(parseInt64(obj["estimated_original_tokens"])),
EstimatedRetainedTokens: int(parseInt64(obj["estimated_retained_tokens"])),
Truncated: boolField(obj, "truncated"),
Reason: stringField(obj, "reason"),
SpillCreated: boolField(obj, "spill_created"),
})
default:
// Unknown event type: tolerate (forward-compat) but only after a
// header has been seen.
@@ -136,12 +151,17 @@ func ReadNDJSON(r io.Reader) (*TurnTrace, error) {
if !sawTraceHeader {
return nil, errors.New("parse trace: non-empty input had no type:trace header")
}
if len(t.Spans) == 0 && len(t.Counters) == 0 && len(t.PrefixHashes) == 0 {
if len(t.Spans) == 0 && len(t.Counters) == 0 && len(t.PrefixHashes) == 0 && len(t.OutputBudgets) == 0 {
return nil, errors.New("parse trace: header present but no spans, counters, or prefix hashes recovered (corrupt or truncated)")
}
return t, nil
}
func boolField(obj map[string]any, key string) bool {
value, _ := obj[key].(bool)
return value
}
// stringField returns obj[key] as a string, or "" if the key is missing or
// the value is not a string. JSON-marshaled trace events always emit
// string fields as JSON strings, so the type assertion is the right
+16
View File
@@ -188,6 +188,21 @@ func (r *Recorder) EmitPrefixHash(p PrefixHash) {
r.tr.PrefixHashes = append(r.tr.PrefixHashes, p)
}
// EmitOutputBudget records one content-free output budgeting decision. Calls
// are retained in transcript emission order, including when the underlying
// tools executed concurrently.
func (r *Recorder) EmitOutputBudget(event OutputBudgetEvent) {
if r == nil {
return
}
r.mu.Lock()
defer r.mu.Unlock()
if r.finished {
return
}
r.tr.OutputBudgets = append(r.tr.OutputBudgets, event)
}
// Finish stamps CompletedAt, derives each span's parent (by interval
// containment) and exclusive time, and returns a snapshot of the trace. Calling
// Finish more than once returns the same snapshot.
@@ -207,6 +222,7 @@ func (r *Recorder) Finish() *TurnTrace {
snap.Spans = append([]Span(nil), r.tr.Spans...)
snap.Counters = append([]Counter(nil), r.tr.Counters...)
snap.PrefixHashes = append([]PrefixHash(nil), r.tr.PrefixHashes...)
snap.OutputBudgets = append([]OutputBudgetEvent(nil), r.tr.OutputBudgets...)
return &snap
}
+27 -11
View File
@@ -79,20 +79,36 @@ type Counter struct {
Value int64 `json:"value"`
}
// OutputBudgetEvent is compact, content-free metadata for one tool result's
// output budgeting decision. It deliberately carries no output text, paths, or
// arguments so tracing cannot become a secret-bearing side channel.
type OutputBudgetEvent struct {
Tool string `json:"tool"`
Category string `json:"category"`
OriginalBytes int `json:"original_bytes"`
RetainedBytes int `json:"retained_bytes"`
EstimatedOriginalTokens int `json:"estimated_original_tokens"`
EstimatedRetainedTokens int `json:"estimated_retained_tokens"`
Truncated bool `json:"truncated"`
Reason string `json:"reason,omitempty"`
SpillCreated bool `json:"spill_created"`
}
// TurnTrace is the finished record for one agent.Run. It is the value
// emitters serialize; it is not mutated after Finish returns a snapshot.
type TurnTrace struct {
SessionID string `json:"session_id"`
RunID string `json:"run_id"`
Profile string `json:"profile,omitempty"`
StartedAt time.Time `json:"started_at"`
FirstVisibleEventAt time.Time `json:"first_visible_event_at,omitempty"`
FirstUsefulActionAt time.Time `json:"first_useful_action_at,omitempty"`
FirstTokenAt time.Time `json:"first_token_at,omitempty"`
CompletedAt time.Time `json:"completed_at"`
Spans []Span `json:"spans"`
Counters []Counter `json:"counters"`
PrefixHashes []PrefixHash `json:"prefix_hashes,omitempty"`
SessionID string `json:"session_id"`
RunID string `json:"run_id"`
Profile string `json:"profile,omitempty"`
StartedAt time.Time `json:"started_at"`
FirstVisibleEventAt time.Time `json:"first_visible_event_at,omitempty"`
FirstUsefulActionAt time.Time `json:"first_useful_action_at,omitempty"`
FirstTokenAt time.Time `json:"first_token_at,omitempty"`
CompletedAt time.Time `json:"completed_at"`
Spans []Span `json:"spans"`
Counters []Counter `json:"counters"`
PrefixHashes []PrefixHash `json:"prefix_hashes,omitempty"`
OutputBudgets []OutputBudgetEvent `json:"output_budgets,omitempty"`
}
// PrefixHash is one fingerprint of the prompt prefix emitted by an agent run.