Compare commits

..
Author SHA1 Message Date
anandh8x c0a1cbee87 Fix Elvish completion syntax
Close generated Elvish conditional branches, use value equality, and validate generated script structure plus native Bash and Zsh syntax.

Refs #499
2026-07-20 10:03:16 +05:30
anandh8x 7ce144a785 Complete shell target suggestions
Expose every supported shell as a completions subcommand candidate and document Elvish activation.

Refs #499
2026-07-20 09:42:59 +05:30
anandh8x 056373f6f3 Add shell completion generation
Add zero completions for bash, zsh, fish, PowerShell, and Elvish using one shared command tree that covers aliases, nested commands, root flags, and common exec flags.

Validate missing and unsupported shell arguments as usage errors and document safe installation examples in command help.

Refs #499
2026-07-20 09:12:39 +05:30
Anandan 9242b9e037 Add transcript-grounded structured task state (#761)
* Add transcript-grounded structured task state

Project existing plan, tool-result, changed-file, verification, and completion events into a deterministic per-run snapshot.

Emit content-free aggregate snapshots to traces. Carry bounded objective context through compaction and ground completion checks only when the projection matches the transcript; mismatches retain transcript-derived behavior and drop stale carried state.

Tested: make fmt-check

Tested: go vet ./...

Tested: go test ./...

Tested: go test -race ./internal/agent ./internal/trace

Tested: go run ./cmd/zero-release build

Tested: go run ./cmd/zero-release smoke

Tested: govulncheck ./... (no vulnerabilities found)

Advisory: pinned golangci-lint reports only unrelated pre-existing findings

Tested: git diff HEAD --check

* Preserve task objective across repeated compaction

Keep the immutable run objective when plan calls have been elided, while admitting mutable task fields only when plan parity still matches the transcript.

Make plan parsing match update_plan rejection semantics, coalesce and cap task-state trace events, rename parity to its plan-only scope, use explicit nilable dependencies, and remove confirmable objective fingerprints from traces.

Tested: make fmt-check

Tested: go vet ./...

Tested: go test ./...

Tested: go test -race ./internal/agent ./internal/trace

Tested: go run ./cmd/zero-release build

Tested: go run ./cmd/zero-release smoke

Tested: govulncheck ./... (no vulnerabilities found)

Advisory: pinned golangci-lint reports only unrelated pre-existing findings

Tested: git diff HEAD --check
2026-07-19 21:38:58 +05:30
Anandan 2e267bdbee feat(tui): add isolated /btw conversations (#748)
* feat(tui): add isolated btw conversations

Add /btw with inline-question support, isolated non-resumable session forks, background main-run routing, and safe return behavior.

Refs #637

* test(tui): cover btw return controls

Preserve drafts when Ctrl+C is pressed during an active BTW run and cover returning to the parent through the documented /btw toggle.

* Address BTW isolation review feedback

* Reject explicit resumes of side sessions

* Harden BTW isolation lifecycle
2026-07-19 20:37:38 +05:30
Anandan 739a47e3ea perf(agent): preserve prompt cache prefixes (#760)
* perf(agent): preserve prompt cache prefixes

Build the system prompt once per run, fingerprint the exact emitted prompt and ordered tool definitions, and expose the full system-prompt hash in traces.

Add normalized and serialized two-turn regressions covering append-only messages, stable tools, and a stable session cache key.

* fix(tools): stabilize registry iteration order

Return registered tools in name order so provider-visible tool lists and prompt-prefix fingerprints cannot drift with Go map iteration. Add a regression covering the registry ordering contract.
2026-07-19 20:29:58 +05:30
KRATOSandKRATOS ce4a996ffa fix(oauth): validate discovered endpoints before merge/use (#511) (#739)
Discovery metadata was merged without the https/loopback endpoint rule that
configured endpoints already pass, letting a malicious issuer downgrade the
authorization/token/device/registration endpoint after config validation.

- Add exported ValidateEndpointURL as the single endpoint-safety rule;
  validateTokenEndpoint now delegates to it.
- Backstop in BuildAuthorizationURL (the choke point both the provider and MCP
  flows build their browser URL through).
- Validate each discovered endpoint before merge in the provider resolveEndpoints
  and validate resolved MCP metadata in both branches; fail closed on insecure.

Co-authored-by: KRATOS <[email protected]>
2026-07-19 09:56:32 +05:30
Vasanth T 378d538e24 perf(agent): execution profiles with one-shot posture escalation (PR10b+PR10c) (#740)
* perf(agent): execution profiles, --exec-profile, bench passthrough, TUI /profile (PR10b+10c)

* fix(execprofile): harden profiles against review findings

- fast escalates on critical-risk mutations, not high: the sandbox marks
  every shell command high before command analysis, so a high trigger
  spent the one-shot on the first go test of any task
- escalation can now restore the provider-default effort a profile
  displaced (PostureEscalation.RestoreDefaultEffort; a plain empty
  target means leave untouched)
- TUI /turns while a profile is active pins the budget: disarms the
  escalation turn target (mirroring exec's explicit --max-turns rule)
  and profile reverts use touched bits instead of value equality, so
  explicit choices survive even when they coincide with profile values
- spec-draft runs get the same escalation policy as the main run
  instead of the displaced budget with no safety net
- bench validates and canonicalizes --exec-profile before spawning:
  an unknown name previously produced an exit-0 report full of
  millisecond usage-error walls
- turn summary prints exec-profile alongside mode; zero exec rejects
  an inline-empty --exec-profile=; balanced trace label documented and
  pinned as deliberate attribution

* fix(execprofile): explicit effort choices disarm the escalation effort restore

An explicit effort set AFTER profile selection could be cleared by a
mid-run escalation (the effort analog of the /turns pin, introduced by
RestoreDefaultEffort): TUI /effort and Ctrl+T now disarm the restore on
a policy clone, and the spec-draft path only arms it when no explicit
--spec-reasoning-effort governs the draft.

* fix(execprofile): address review findings on the profile surfaces

- --use-spec projects away a profile's self-correct knob instead of
  silently arming an option the draft runner ignores (the explicit
  --self-correct --use-spec combination is already a parse-time error)
- RunTurnBench canonicalizes and validates ExecProfile at the library
  boundary, so a direct caller's casing/whitespace cannot fork report
  labels and an unknown name fails before anything runs
- reverting a profile whose displaced budget was zero clears
  ZERO_MAX_TURNS (SetMaxTurnsEnv ignores zero, so sub-agents would have
  kept the removed profile's budget)
- the TUI effort-disarm test fails instead of skipping when the catalog
  stops filling low effort

* fix(tui): reconcile an active profile's effort fill on model switch

The fill is per-model, not a session preference: fast selected on a
model without low fills nothing, and switching to a model that supports
it should behave like selecting the profile there; the reverse switch
must not keep sending a level the destination does not support. Both
switch paths (the /model command and the cross-provider picker) now
re-derive the fill, leaving explicitly touched efforts alone and
keeping the escalation's effort restore in step. Also: the revert-env
test asserts key removal via LookupEnv, and applyProfileTurnBudget's
pinned-budget parameter is named and documented as covering mode
presets, not just the flag.

* fix(tui): void profile effort bookkeeping when the model-switch drop erases it

The /model path's pre-existing unsupported-effort drop ran before the
profile reconciliation and left contradictory state behind: a dropped
touched override kept blocking the profile from governing, and a
dropped profile fill left a stale applied value with the effort restore
still armed. The drop itself is load-bearing (the TUI run path forwards
the effort to the provider unfiltered), so instead of preserving a
known-unsupported level, the voided choice now resets the profile's
effort bookkeeping and the reconcile re-derives from scratch. The whole
sequence lives in one ring-parameterized helper driven by the switch
target's authoritative ring, unit-tested against synthetic rings for
both the void-then-refill arc and the supported-override survival.

* fix(tui): only drop an explicit effort when the target ring is authoritative

A nil ring meant both a catalog model with no effort controls and a
live-discovered custom model the catalog cannot vouch for. The drop now
requires an authoritative ring (a resolved catalog entry, whose ring may
legitimately be empty); an unknown ring preserves an explicit preference,
matching the cross-provider picker's behavior, while the profile's own
fill still applies only where support is known.
2026-07-19 00:53:17 +05:30
Vasanth T 015452c1c9 fix(perfbench): keep the stamped answer file out of negative oracle greps (#737)
* fix(perfbench): keep the stamped answer file out of negative oracle greps

* fix(perfbench): exclude stamped answer from positive oracle greps too
2026-07-18 22:54:02 +05:30
Vasanth T af875df587 perf(agent): posture-escalation signals and controller (PR10a) (#736)
* trace(perf): add posture escalation counter

* feat(agent): typed posture-escalation policy and tool-result risk field

* feat(agent): posture controller wiring and executed-risk stamping

* test(agent): cover posture controller, escalation act path, and risk stamping

* test(agent): pin zero risk on not-executed results

* fix(agent): wire uncertain-path escalation and validate risk thresholds
2026-07-18 22:32:02 +05:30
Vasanth T dbd9443014 fix(perfbench): absolutize the bench binary and make errored tasks first-class (#730)
* fix(perfbench): absolutize the bench binary and make errored tasks first-class

* fix(perfbench): state that errored tasks count as pass-rate failures

* fix(perfbench): align errored-task wording with the actual tier accounting

* chore(perfbench): never commit generated baseline reports
2026-07-18 21:39:52 +05:30
Anandan 0fae754460 docs: refresh repository agent guidelines (#734)
* docs: refresh repository agent guidelines

* docs: make validation guidance non-mutating
2026-07-18 21:37:36 +05:30
Vasanth T 2cc6f43bf7 docs(community): wire Discussions as the front door for questions and ideas (#732)
* docs(community): wire Discussions as the front door for questions and ideas

* docs(community): restore LF endings and make feature routing coherent
2026-07-18 21:34:58 +05:30
uma-prasadandCursor 77960229b8 fix(doctor): detect missing native binary during runtime checks (#450)
* fix(doctor): report missing npm native binary in wrapper

Co-authored-by: Cursor <[email protected]>

* fix(doctor): preserve JSON output for missing native binary

* fix(doctor): preserve doctor CLI behavior in wrapper fallback

Co-authored-by: Cursor <[email protected]>

* style: gofmt npmwrapper doctor fallback tests

Co-authored-by: Cursor <[email protected]>

* fix(doctor): reuse shared Bun recovery helpers in wrapper fallback

Extract bunRecoveryParagraph and related helpers so the doctor missing-binary text path and the generic wrapper bail share the same copy. Add regression tests with ZERO_WRAPPER_SIMULATE_BUN to guard against future drift.

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: Cursor <[email protected]>
2026-07-18 22:49:19 +08:00
Vasanth T 60dc84e7a3 perf(openai): optimized turn session — background prewarm and prefix telemetry (PR8) (#723)
* trace(perf): add provider prewarm span and prefix-stability counters

* feat(openai): optimized turn session with prewarm and prefix telemetry

* feat(providers): gate optimized OpenAI turn sessions behind env flag

* feat(cli): wire optimized turn sessions into headless exec

* docs(readme): document the optimized OpenAI turn-session toggle

* fix(openai): include tool descriptions in prefix digest, harden gate and test bounds

* fix(openai): isolate prewarm tracing, skip non-reusable transports, fingerprint wire order and cache key

* test(openai): pin keep-alive transports in probe-expecting tests

* feat(providers): preserve resolved capabilities on switched-model fallback sessions
2026-07-18 14:24:50 +05:30
Vasanth T 30e2c3f7ff perf(providers): provider capabilities and default turn-session adapter (PR7) (#720)
* perf(providers): add provider capabilities and default turn-session adapter (PR7)

* fix(providers): project effective reasoning efforts into capabilities

* feat(agent): carry optimized turn sessions across mid-run model switches

* test(agent): cover swap-time session open failure
2026-07-18 09:33:07 +05:30
Anandan 18cce358a2 Add deterministic completion policy (#719)
Extract completion decisions from the agent loop into a feature-gated typed policy with complete, incomplete, and uncertain outcomes. Preserve bounded plan-stall nudges and allow at most one semantic acceptance check for self-correcting runs.

Tested: make build

Tested: make test

Tested: go fmt ./...

Tested: go vet ./...

Tested: make lint

Tested: govulncheck ./...

Note: repository-wide pinned golangci-lint reports 36 pre-existing unrelated findings; internal/agent/... reports 0 issues.
2026-07-18 00:29:58 +05:30
KRATOSandKRATOS e5670c427f perf(output): add token-aware semantic output budgeting (PR11) (#717)
* perf(output): add token-aware budget contract

* perf(output): add semantic retention policies

* perf(output): budget results after redaction

* perf(output): propagate tool truncation metadata

* perf(output): trace semantic budget decisions

* perf(output): classify core tool output

* docs(output): describe semantic output budgeting

* test(output): make spill fallback portable on windows

* fix(output): address semantic budgeting review findings

* test(output): make hook budgeting tests portable

* fix(output): retain oversized single-line file content

* fix(output): apply semantic budgets to captured output

* fix(output): preserve hook budgets and search paths

* fix(output): preserve alias and truncation metadata

* test(trace): validate output budget metadata

---------

Co-authored-by: KRATOS <[email protected]>
2026-07-18 00:02:43 +05:30
KRATOSandKRATOS 31d45d5f14 perf(agent): concurrent read-only tool batches via capability gate (#715)
Upgrade the parallel tool planner from SideEffectRead probes to the PR5
CapabilitiesOf contract: EffectReadOnly + ThreadSafe + auto-allowed, with
resource-key conflict boundaries so same-path reads stay sequential.

Mark audited pure reads ThreadSafe (read_file, read_minified_file,
list_directory, glob, grep, skill, web_fetch). Glob/grep use scopedScan
directory keys (nil for workspace-wide "." so scans do not false-conflict).

Tests cover the capability gate, key conflicts, extendParallelRun windows,
and catalog ThreadSafe audit for concurrent-safe reads.

Co-authored-by: KRATOS <[email protected]>
2026-07-17 17:52:11 +05:30
Vasanth T 727ad4d321 perf(turn-bench): Phase 0 — strengthen oracles so pass rate can't be misread as correctness (#712)
* perf(turn-bench): Phase 0 — strengthen oracles so pass rate can't be misread as correctness

The baseline manifest's pass/fail contract had three holes that let a weak
or no-op agent read as a pass:

  - edit grep oracles were substring checks: a reworded line (e.g.
    fmt.Println("debug: starting") -> fmt.Println("starting up")) passed
    `! grep 'debug: starting'`.
  - refactor lived in the build tier with a non-positive `go build ./...`,
    so a no-op refactor passed.
  - the 24 read-only tasks (nav/longproc/longctx/parallel) had no oracle;
    a zero-exit only proved the turn ran.

This makes the tiers honest:

  - edit/refactor now carry a stamped oracle_test.go compiled by
    `go test ./...` — the Go compiler is the structural verifier, so a
    no-op refactor, a missing field, or a reworded-but-not-removed line
    fails to compile/run. The test is stamped into the fixture copy AFTER
    the agent run, so it can't interfere with the agent's own build or be
    pre-seen.
  - nav gains an answer-oracle: the harness captures the agent's final
    answer (.zero-answer.txt) and greps it for determinable facts. Count
    tasks (nav-04/05/08) use a structured `count: N` token so the oracle is
    tight rather than a loose prose-contains on "0".
  - refactor graduates from build to correctness (its oracle is now
    positive); nav graduates from latency-only to correctness. The build
    tier is empty (buildOnlyClasses: []).
  - longproc/longctx/parallel stay permanently latency-only — there is no
    deterministic oracle for "report whether the build succeeds" or
    "summarize a large file"; documented in MANIFEST.md.

Tier counts: correctness 34 (edit 10 + fix 8 + nav 10 + refactor 6),
build 0, latency 14. Total 48 unchanged.

TurnSchemaVersion 2 -> 3: the result shape is unchanged (only class-list
membership shifts) but correctnessPassRate now means something strictly
stronger, so the bump prevents a v2->v2 cross-version comparison from
misreading the jump as a model improvement.

Prerequisite fix: the mutating fixtures had no go.mod, so `go test ./...`
in the temp copy failed with "cannot find main module". Added go.mod per
fixture (edit/fix/refactor/nav) and moved copyFixture's temp dir one level
below the system temp root — Go ignores go.mod in direct children of the
temp dir (a hijack guard), which would have broken every compiler-backed
oracle. This also fixes the pre-existing fix/refactor oracles, which were
silently broken because the test suite used fake runners that never ran
go test.

Tests: stub-binary gating tests prove each oracle fails the wrong thing
(no-op refactor, missing field, wrong nav answer, reworded debug print) and
passes the right thing (real refactor, added field, correct answer); an
edit-01 test proves the scoped negative grep doesn't false-fail a correct
rename that leaves the doc comment. Aggregation, schema, and unit tests
cover the v3 tier model and the answer-capture helper.

* Address CodeRabbit review on PR #712: tighten oracles + isolation

- nav count oracles (nav-01/04/05/08): anchored line-exact
  `^count: N$` with the fixture's exact expected count, so a substring
  like "the count: 0" or a wrong count like "count: 3" fails instead
  of rubber-stamping. nav-01 added to the structured-count contract.
- stampOracleAndAnswer now errors when a task has an oracle or
  verification but no workspace fixture, so verification can't run in
  the caller's cwd with the oracle never compiled (false-pass closed).
- copyFixture uses a unique 0700 os.MkdirTemp parent under the temp
  root and nests the copy beneath it (grandchild of temp root), so the
  Go go.mod hijack guard is satisfied and cleanup removes the whole
  parent. Avoids a predictable shared world-readable dir.
- Rejection tests require a non-empty VerifyErr via a shared
  assertVerifyFailed helper, proving the failure came from
  verification rather than a harness error or a silent non-pass.
- Unify streamJSONExitCode + streamJSONFinalText behind one
  parseStreamJSON parser (behavior-preserving; scanner limits and
  malformed-event handling now live in one place).

* Address gnanam1990 review on PR #712: close count=0 gameability + stronger oracles

- nav count tasks (nav-04/05/08) had a count=0 ground truth, so an agent
  that never opened the workspace and always emitted "count: 0" passed
  three correctness tasks -- the exact pass-rate-misread-as-correctness
  failure this PR closes. nav-04 now has a real TestGreet (count 1) and
  nav-05 a real TODO (count 1); nav-01's file count moves to 5. nav-08
  stays at 0 (a real third-party import would need a network fetch and
  break the offline suite) but its oracle now requires the answer to name
  both stdlib imports (fmt from main.go, testing from main_test.go), so a
  blind count:0 or a main.go-only read fails. nav-08's prompt now asks the
  agent to enumerate and classify every import, so naming the stdlib
  imports is part of a correct answer, not a false-fail.
- refactor-05: stamp a behavioral oracle (TestGreetWrapped) so deleting
  both Wrapper and greetWrapped, or stubbing the caller to return "",
  fails to compile/run. The negative grep alone passed that wrong edit.
- refactor-04: replace the brittle exact-text shell grep with a stamped
  file-reading oracle using a spacing-tolerant regex, so a bare
  map[string]int decl under any spacing/:= variant can't dodge it, and
  the Stats type must be referenced in main.go.
- edit-05: also assert the "debug: starting" message text is absent, so
  a respell onto fmt.Printf / backticks / log.Println / println that
  still prints the message fails (the fmt.Println(" check missed those).
- v3 doc comments on ClassSummary / TurnBenchResult /
  FormatTurnBenchSummary now describe the v3 tiers (correctness =
  edit/fix/nav/refactor, build empty, latency = longproc/longctx/
  parallel) instead of the stale v2 model.
- stampOracleAndAnswer failure now mutates the existing outcome instead
  of returning a fresh one, so the already-parsed trace and trace issue
  survive a stamp error (attribution is preserved, pass/fail unchanged).
- Schema comment provenance: "PR #701" -> neutral "Phase 0 oracle
  hardening" (this PR is #712).
- New gating + satisfiable stub tests for refactor-04/05 and edit-05, and
  count-oracle zero-guess / inspection-proof tests for nav-04/05/08.

* Fix macOS CI: copyFixture isolation test was slash-sensitive

TestCopyFixtureIsolatesSourceFromMutation compared filepath.Dir(parent)
against os.TempDir() verbatim. On macOS $TMPDIR ends in a trailing slash
("/var/folders/.../T/") while filepath.Dir strips it, so the equality
false-failed even though the parent is genuinely a direct child of the
temp root. Linux (/tmp, no slash) and Windows (no slash in this Go
build) matched by accident.

The production copyFixture nesting is correct on all platforms (the
grandchild copy sits one level below the temp root, satisfying Go's
go.mod hijack guard); only the assertion was slash-sensitive. Normalize
both sides with filepath.Clean.

* Close nav-04/05 blind-count-one gameability (anandh8x review)

nav-04 and nav-05 had a count-only oracle (anchored '^count:1$'). Once the
ground truth became 1, an agent that never inspects the fixture and always
emits 'count: 1' passed both — the blind-count-one surface the prior
non-zero-ground-truth fix opened. Pair the count with a workspace-derived
named fact that proves inspection, mirroring nav-08's fmt+testing:

- nav-04 now also requires 'TestGreet' (the one test function in
  main_test.go) — a specific, non-generic name a blind agent can't produce
  without reading main.go to learn a greet function exists. Fully closes
  the blind-count-one surface.
- nav-05 now also requires 'main.go' (the file holding the one TODO).
  main.go is the conventional entrypoint and the most guessable filename, so
  this reduces rather than fully closes the surface — a blind 'count: 1, in
  main.go' can still pass. Accepted for Phase 0: the blind floor drops below
  100% (a bare 'count: 1' fails), so a model still moves the pass rate by
  inspecting. MANIFEST.md documents this residual honestly.

Two gating tests per task: RejectsBlindCountOne (bare 'count: 1' now fails)
and, for nav-05, RejectsSubstring (main.go present but unanchored count
still fails — parity with nav-04). Existing accept tests still pass (their
answers already name TestGreet / main.go).

Doc nits also fixed: MANIFEST.md #701 -> #712; refactor-05 removed from the
'no oracleTest' inventory (it now has TestGreetWrapped); the nav-01 named-
fact prose corrected to the three files the oracle actually anchors on;
test comment #701 -> #712.
2026-07-17 17:32:45 +05:30
github-actions[bot] e666395d47 chore(main): release 0.4.0 (#625)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-07-17 13:25:31 +08:00
PierrunoYTandClaude Sonnet 5 9043baedcf fix(sandbox): scrub dynamic credential env vars (#682)
* fix(sandbox): scrub dynamic credential env vars

* fix(sandbox): tolerate env assignments in configured sensitive keys

A config value mistakenly given as a full assignment (e.g.
"COMPANY_LLM_SECRET=...") would never match the real env key during
scrubbing, silently re-exposing the credential to sandboxed commands.
normalizeSensitiveEnvKeys now keeps only the name part before '=' and
re-trims it, per PR review.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(sandbox): scrub credentials on direct/degraded command plans

directCommandPlan (used when the native sandbox backend is unavailable,
disabled, or not required, including the EnforcementDegraded fallback)
left Env nil whenever spec.Env was nil, so exec.Cmd inherited the full
caller environment unscrubbed. Configured apiKeyEnv credentials and
dynamically named ZERO_OAUTH_<NAME>_CLIENT_SECRET variables leaked
through that path even though the wrapped sandbox plans already
scrubbed them.

Route direct plans through scrubSensitiveEnv the same way wrapped
plans do, snapshotting os.Environ() when spec.Env is nil so scrubbing
still applies. Adds a regression test covering the degraded-fallback
path with both a configured and a dynamically named secret.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

---------

Co-authored-by: Claude Fable 5 <[email protected]>
2026-07-17 13:04:12 +08:00
Ashwinhegde19 4d8c31cf16 fix(config): enforce MCP trust boundary so project config cannot override user disable (#609)
A user-level MCP server disable is now sticky. The project config
(a cloned repo's ./.zero/config.json) cannot re-enable a server the
user disabled, and cannot disable a server the user explicitly enabled.

The previous merge just assigned base.Disabled = next.Disabled, so a repo
that set disabled:true could silently defeat a user's explicit mcp enable
<server>. The guard is now symmetric via a mergeMCPServer canReenable
flag: once a higher-trust scope has explicitly set the disabled flag, a
lower-trust layer may not override that decision in either direction. Only
the CLI override scope (canReenable=true) may lift a sticky user decision.

The prior disabledSet-before-check ordering bug is also fixed: the
higher-trust scope's prior disabledSet/Disabled is captured before folding
in next's flag, so a project layer can still disable a default-enabled
server (e.g. firecrawl) rather than being silently ignored.

Ports the fix onto main's refactored MCP merge (internal/config/mcp_merge.go
and mergeProjectMCPConfig), which splits project-scope merging from the
general mergeMCPConfig path.

Fixes #512
2026-07-17 13:03:32 +08:00
PierrunoYTandClaude Sonnet 5 824ecdbcf9 fix(hooks): run sessionEnd hooks after Esc/Ctrl+C interrupts (#606)
* fix(hooks): run sessionEnd hooks after Esc/Ctrl+C interrupts

On interrupt, Run returns with ctx already canceled. The deferred
dispatchSessionEnd call passed that same ctx into Hooks.Dispatch, whose
context.WithTimeout derives an already-canceled child context, so the
hook process never actually launches (recorded as timed-out/error in
the audit log). Since most sessions end via interrupt, sessionEnd
hooks were effectively unreliable.

Detach the sessionEnd dispatch from the run context's cancellation
with context.WithoutCancel so the dispatcher's own timeout still
applies without being short-circuited by the run's cancellation.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* test(agent): fall back to GOROOT for hook regression

* test(agent): justify GOROOT fallback

---------

Co-authored-by: Claude Sonnet 5 <[email protected]>
2026-07-17 13:03:14 +08:00
6b9c2f0c08 feat(aimlapi): AI/ML API provider with guided onboarding (top-up + key issuance) (#655)
* feat(aimlapi): AI/ML API guided onboarding (top-up + key issuance)

Add the AI/ML API provider integration to the zero CLI: a guided TUI
onboarding sub-flow that takes an existing key or an email top-up and
writes the issued key into the provider profile, attributed to the
Gitlawb rebate partner.

- internal/aimlapi: OpenAI-compatible client, partner-checkout top-up,
  key validation, streaming, and endpoint config (Paths A/B/C).
- providercatalog / providermodelcatalog: register the aimlapi provider
  with default headers and models.
- tui: aimlapi onboarding screen wired into the provider wizard and
  onboarding flow.

Default endpoints point at production (auth/app/api/pay/aimlapi.com);
each is overridable via AIMLAPI_*_URL env vars.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(aimlapi): address review findings in onboarding flow

- topup: use errors.As for the 5xx retry check so a wrapped APIError
  still retries instead of aborting a paid checkout.
- onboarding: a Path-A top-up on an email with no account now lands on a
  single coherent "key saved, balance not topped up" screen instead of a
  green "ready" plus a red error.
- onboarding: an empty amount field is rejected instead of silently
  falling through to the $25 default.
- collapse the two identical top-up prompt constants into MsgTopUpPrompt
  and drop the dead amountPrompt() branch.
- reuse stdlib maps.Clone instead of a third bespoke header-copy helper.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(aimlapi): retry transient poll errors and sharpen amount validation

* fix(aimlapi): retry transient poll errors and surface real amount errors

* docs(aimlapi): document exported client API; drop out-of-scope gitignore entry

Addresses the two CodeRabbit PR checks on #655:

- Add doc comments to the exported aimlapi client API (Client, APIError,
  the session/checkout/onboarding types and their methods) and
  ParseAmountUSD, to satisfy the docstring-coverage check.
- Remove the unrelated /zero.exe~ .gitignore entry that an earlier
  review-fix commit added; it is out of scope for the provider work, so
  the PR no longer touches .gitignore.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix(aimlapi): render aimlapi row like other recommended providers

* fix(aimlapi): resolve onboarding review findings (Path A funding, base URL, request cancellation)

* fix(aimlapi): validate pasted API keys

* test(aimlapi): clarify pasted key validation

* test(aimlapi): cover key-balance low-balance transition

* fix(aimlapi): reject non-finite top-up amounts

* fix(cli): scope and canonicalize provider headers

* fix(tui): distinguish invalid verification codes

* feat(aimlapi): resume account-bound top-ups safely

* feat(tui): offer top-up for existing AIMLAPI keys

* feat(tui): reuse AIMLAPI env key on first run

* feat(tui): reuse configured AIMLAPI profiles

* test(aimlapi): trim redundant flow coverage

* fix(tui): preserve existing AIMLAPI profile metadata

* fix(tui): clone AIMLAPI headers in env fallback

* fix(aimlapi): address remaining review findings

* fix(aimlapi): harden onboarding lifecycle

* fix(aimlapi): preserve validated checkout context

* fix(aimlapi): harden checkout retry state

* fix(tui): clarify AIMLAPI configured choices

* fix(aimlapi): make checkout retries idempotent

* fix(aimlapi): validate checkout responses

* fix(providers): reconcile aimlapi catalog after rebase

---------

Co-authored-by: Lookoff123 <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
2026-07-17 12:49:38 +08:00
KRATOSandKRATOS 8ef8576df7 perf(tools): add explicit effect metadata for safe concurrency (#705)
* perf(tools): add explicit tool effect metadata

Introduce fail-closed EffectClass and ToolCapabilities on every built-in
tool so a later concurrent read-only batch PR can decide safety from
explicit metadata rather than tool names. Plugin and MCP tools stay
EffectUnknown. No concurrent execution is enabled.

* fix(tools): address review feedback on effect metadata

Harden capability normalization and resource keys before PR6 consumes them:

- Force ThreadSafe=false for every non-ReadOnly effect at read time
- Reject duplicate builtin names in ValidateBuiltinCatalog
- Coerce numeric session ids and accept "session" for process keys
- Strip path/query before userinfo in endpoint host parsing
- Wire apply_patch keys from unified-diff paths + cwd
- Reclassify bash as WorkspaceWrite (one-shot shell mutator)
- Expand regression tests for keys, normalization, and gates

* fix(tools): keep multiFileResourceKeys used by apply_patch keys

Satisfies unused lint after switching apply_patch to applyPatchResourceKeys.

* fix(tools): resolve CodeRabbit residual capability/key issues

- Join apply_patch diff paths under cwd for conflict keys
- Validate constructor-declared capabilities before normalization
- Unify session/process ID key extraction helper

---------

Co-authored-by: KRATOS <[email protected]>
2026-07-17 12:48:36 +08:00
Anandanandanandh8x 1c5c6e78a8 feat(perf): emit prompt-prefix hash fingerprint per turn (#704)
Compute a seven-field SHA-256 fingerprint of the cacheable prompt
prefix (base instructions, confirmation policy, project context, skills,
tools, tool schemas, complete) and emit one prefix_hash event per turn
through the trace recorder. The complete hash is the SHA-256 of the
canonical join of the other six, so any sub-component drift is
observable both individually and in aggregate.

The decomposition exists so a downstream observer can detect which
sub-component of the prefix broke cache stability across turns: a
stable complete_prefix hash means the cacheable prefix is byte-
identical, and the sub-hash that changed names the culprit.

No behavior change for untraced runs. For OpenAI, the existing
prompt_cache_key (session ID) is unchanged; the fingerprint is purely
additive observability. The Anthropic cache_control breakpoint work
already in main is not touched.

Co-authored-by: anandh8x <[email protected]>
2026-07-17 12:47:53 +08:00
Leonardo Faoro 7f669f4550 feat(tui): Ctrl+X leader chords and emacs menu navigation (#699)
* feat(tui): Ctrl+X leader chords and emacs menu navigation

Add Ctrl+X leader shortcuts for common slash commands (with Ctrl+X ?
chord map), and Ctrl+P/N previous/next selection in modals and pickers.

* fix(tui): drop Ctrl+X e /edit leader chord

/edit replaces the composer and would discard an in-progress draft; leave
it unbound so leader shortcuts stay draft-preserving.

* fix(tui): consume idle Ctrl+N as a no-op

Reserve Ctrl+N for emacs menu navigation so it never falls through to
remapped global bindings when no selection surface is open.

* docs: drop manual CHANGELOG entries for TUI shortcuts
2026-07-17 12:44:00 +08:00
euxaristia 2b42cd567b fix(tui): stop the composer cursor blinking while typing or unfocused (#672)
* fix(tui): stop the composer cursor blinking while typing or unfocused

The composer's cursor blinked on a fixed timer unconditionally, including
mid-keystroke, which made typing feel janky since the cursor's job is to mark
the current position. Gate the existing blink tick on two states instead:
force solid while a keystroke landed within the last 500ms (reusing the
existing lastCharTime tracker), and hide it entirely while the terminal
window is unfocused (new terminalFocused field, driven by tea.FocusMsg/
BlurMsg). Idle and focused keeps the prior blink behavior unchanged.

Also decouples view.ReportFocus from the notifier (it was previously gated on
m.notifier != nil, which is effectively always true in production but left
cursor-focus behavior fragile if that ever changes) so focus reporting is
requested unconditionally.

* fix(tui): apply caret visibility synchronously and in the argument-hint line

- A keypress, focus, or blur now updates composerCursorVisible in its own
  message handler instead of waiting for the next 530ms composerBlinkMsg
  tick, which could render a typed character caret-less, leave a visible
  caret in an unfocused terminal, or keep it hidden after a refocus.
- commandArgumentHintComposerLine takes the caret visibility state instead
  of painting its cursor cell unconditionally, so the required-argument
  hint path follows the same focus and blink contract as the main
  composer renderer.
- New tests assert the state immediately after each message, with no blink
  tick in between, and cover the argument-hint render in both states.

* fix(tui): apply the caret state on composer paste

A paste is the same immediate-input transition as a keypress: routePaste now
refreshes lastCharTime and sets the caret visible for both the bracketed
tea.PasteMsg and right-click clipboardReadMsg paths, so a paste right after
the blink phase hid the caret renders solid immediately instead of waiting
for a tick that would toggle off a stale idle state.
2026-07-17 12:43:30 +08:00
euxaristia 665bd4b3d0 WIP: feat(tui): TUI theme presets (Dune and Neon) (#634)
* draft(tui): add theme interface and Claude/Codex presets proposal

Introduces the Theme interface alongside the initial specifications and styling models for the Claude (card-based, warm) and Codex (high-density, split-screen cyberpunk) layout presets. Detailed design plans are documented in docs/THEMES.md.

* fix(tui): implement Claude/Codex themes on the existing palette system

theme.go had replaced the tuiTheme/buildTheme/palette architecture every
renderer depends on with a standalone Theme interface using the wrong
lipgloss import, breaking the build and orphaning the existing theme
registry, /theme command, and picker.

Revert theme.go to the working implementation and add claude and codex
as real palette entries in theme_palettes.go, following the file's own
extension pattern. Both pass the WCAG AA contrast and gray-ramp
invariants asserted across the whole registry, and are picked up
automatically by --theme, ZERO_THEME, /theme, and the picker.

Rewrite docs/THEMES.md to describe the actual palette-registry
architecture instead of the removed interface.

* fix(tui): rename Claude/Codex to neutral Dune/Neon, ensure WCAG AA contrast compliance, and add tests

* style(tui): gofmt theme_select_test.go (drop trailing whitespace)

The ubuntu smoke run fails on gofmt -l; this clears the only unformatted
file in the draft so CI goes green.

* test(tui): cover the CLI/env resolution path for the dune and neon presets

TestNewThemePresetsWired only checked lookupTheme and the IsDark
grouping. Add coverage for the actual --theme/ZERO_THEME precedence
path (resolveThemeMode) and confirm applyTheme swaps zeroTheme to each
preset's own palette rather than a built-in fallback.

* fix(tui): raise Dune/Neon palette contrast and fix theme docs

Darken dune's accent, blue, and red tokens so the selected-row caret,
local-model dot, and diff deletion text clear WCAG AA against selBg and
delBg, including after ANSI-256 downsampling. Also correct THEMES.md's
description of bare /theme (it opens a live-preview picker, not a
list), and add the dune/neon names to the ZERO_THEME/--theme tables in
README.md and README_ZH.md.

* fix(tui): keep Dune and Neon readable after ANSI-256 downsampling

- Dune's accent becomes #724028: still AA on selBg and panel in truecolor
  (5.46:1 and 7.03:1), and its xterm-256 quantization (#444444) now clears
  AA on quantized selBg at 6.47:1 where the previous #7c4712 quantized to
  #875f00 at 3.81:1. Blue already survived quantization at 4.67:1.
- Neon's diff bands move to hexes that quantize to distinct green and red
  xterm entries (addBg #083c10 -> #005f00, delBg #3c0810 -> #5f0000,
  addBgWord #147828 -> #008700, delBgWord #74202e -> #870000) instead of
  collapsing to the same grays, while keeping every truecolor invariant.
- New TestExtendedThemeANSI256Contrast quantizes the palette tokens the way
  a 256-color terminal does and asserts the Dune selected-row pairs stay AA
  and the Neon bands keep their green/red identity and mutual separation.
- docs/THEMES.md is now linked from the Documentation sections of both
  READMEs, and the Unreleased changelog inventory says twelve themes and
  names dune and neon.

* fix(tui): keep Neon diff content AA after quantization, fix error border

- faintest becomes #74c468: line numbers now quantize to #87d75f, which
  clears AA on the xterm-green addBg (#005f00) where the old #58af69
  quantized to 2.95:1, while staying dimmer than faint so the gray ramp
  stays monotonic.
- addInk becomes #ecffdc, quantizing to #ffffd7 for 4.60:1 on the word
  band's #008700 (was 4.29:1). Both rendered diff-content pairs are now
  asserted in the ANSI-256 test instead of documented as omissions.
- cardErr becomes #9a4042: 3.01:1 against the panel (2.43:1 before) and
  3.65:1 quantized, meeting the 3:1 non-text component threshold, with the
  border surface asserted in both renderings.
- docs/THEMES.md no longer claims the extended invariants run against the
  whole registry: it now says new palettes must be added to the
  per-palette contrast tests or given equivalent coverage.
2026-07-17 12:38:51 +08:00
173 changed files with 19254 additions and 838 deletions

No files matched your search

+6
View File
@@ -1,5 +1,11 @@
blank_issues_enabled: false
contact_links:
- name: Questions and setup help
url: https://github.com/Gitlawb/zero/discussions/categories/q-a
about: Ask in Discussions Q&A. Include zero --version, your OS, provider/model, and zero doctor output for setup problems.
- name: Feature ideas
url: https://github.com/Gitlawb/zero/discussions/categories/ideas
about: Propose and discuss ideas in Discussions before opening a PR (Zero is in controlled development, see CONTRIBUTING.md).
- name: Security vulnerability
url: https://github.com/Gitlawb/zero/security/advisories/new
about: Report security issues privately via GitHub Security Advisories (see SECURITY.md). Do not file a public issue.
+13 -3
View File
@@ -1,12 +1,22 @@
name: Feature request
description: Propose a new capability or change
name: Feature request (previously discussed)
description: File only after your idea gained traction in Discussions Ideas or a maintainer asked for a tracking issue
labels: ["enhancement"]
body:
- type: markdown
attributes:
value: |
Please do not begin implementation until the issue has been reviewed and marked with the
New feature ideas start in [Discussions Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas),
not here. This form is for the step after that: an idea that gained traction in Discussions,
or one a maintainer asked to have tracked as an issue. Please link the discussion below.
Do not begin implementation until the issue has been reviewed and marked with the
`issue-approved` label (see CONTRIBUTING.md).
- type: input
id: discussion
attributes:
label: Ideas discussion
description: Link to the Discussions Ideas thread (or note which maintainer asked for this issue).
validations:
required: true
- type: textarea
id: problem
attributes:
+3
View File
@@ -55,3 +55,6 @@ docs/superpowers/
zero-termux-arm64
zero-linux-sandbox
zero-seccomp
# Generated benchmark reports are evidence for one configuration, never tree state
internal/perfbench/reports/*.json
+1 -1
View File
@@ -1 +1 @@
{".":"0.3.0"}
{".":"0.4.0"}
+63 -19
View File
@@ -1,27 +1,71 @@
---
description: Project guidelines for the open-source CLI coding agent (zero) repository.
globs: "*.go, *.js, *.md, *.json, *.toml, *.yaml, *.yml"
alwaysApply: false
---
# Repository Guidelines for Zero
# Repository Conventions for Zero
These instructions apply to all work in this repository. For the user-facing
guide to extending Zero with specialists, hooks, plugins, MCP, and skills, see
[docs/EXTENDING.md](docs/EXTENDING.md).
This file outlines the project conventions and repository guidelines for coding agents when working on the `zero` repository. For the general guide on how to extend Zero (write specialist sub-agents, hooks, plugins, MCP, skills), see [docs/EXTENDING.md](docs/EXTENDING.md).
## 1. Contribution and Pull Request Rules
## 1. Project Conventions
- Before opening any pull request, **all contributors**—including maintainers,
community contributors, and coding agents—must read and follow
[CONTRIBUTING.md](CONTRIBUTING.md).
- Community pull requests require an existing parent issue with the
`issue-approved` label. Team members may open pull requests through the
internal development process described in `CONTRIBUTING.md`.
- Keep each change focused on the approved or assigned scope. Do not include
unrelated fixes, refactors, formatting churn, generated output, or existing
local changes in the same commit or pull request.
- Discuss new implementation languages, runtimes, major dependency changes,
and broad architectural rewrites with maintainers before implementation.
- Pull request descriptions must explain what changed and why, link the parent
issue when required, and list the tests or verification performed. Include
screenshots or a short recording for user-visible UI changes when practical.
- Build with `make`, not `go build` directly.
- Tests live next to the source file (`foo_test.go` next to `foo.go`).
- Run `make lint` before opening a PR.
- Never edit files under `third_party/` — those are vendored.
- Unify functions/methods where possible to prevent codebase inflation. Prefer a single cross-platform function with conditional checks over duplicating helpers per platform.
## 2. Repository and Implementation Conventions
## 2. Guidelines for Coding Agents
- Use the Go version declared in `go.mod`. Do not hardcode a different local
toolchain version in scripts or documentation.
- Use the repository build and release commands (`make` and
`go run ./cmd/zero-release ...`) instead of inventing parallel build flows.
- Keep tests beside their source files (`foo_test.go` next to `foo.go`). Add a
regression test for behavior changes and run affected concurrent code under
the race detector.
- Never edit files under `third_party/`; they are vendored.
- Prefer one cross-platform function with small conditional checks over
duplicated platform-specific helpers when the behavior can remain unified.
- Do not commit generated benchmark reports from
`internal/perfbench/reports/*.json`; reports are configuration-specific
evidence, not repository state.
- Preserve the user's working tree. Do not overwrite, delete, stage, or commit
unrelated tracked or untracked files.
If you are an AI coding agent executing tasks in this repository, you **MUST** run all Go code quality and security checks before committing code or completing your task:
## 3. Required Validation
1. **Format & Vet**: Run `go fmt ./...` and `go vet ./...`.
2. **Lint**: Run `go run github.com/golangci/golangci-lint/v2/cmd/[email protected] run --enable-only unused,ineffassign,staticcheck ./...`.
3. **Security**: Run `go run golang.org/x/vuln/cmd/[email protected] ./...`.
Run validation from the repository root before committing, opening a pull
request, or completing an implementation task:
If any check fails or cannot be run, do not ignore it. Prompt the user for instructions or setup assistance before proceeding.
1. **Formatting check**: `make fmt-check`. If it fails, format with
`go fmt ./...` (or `make fmt`) and run the check again.
2. **Vet**: `go vet ./...` (or `make vet`).
3. **Tests**: `go test ./...`. Use `make test` for the full race-enabled suite,
or run focused tests with `-race`, when concurrency is affected.
4. **Build**: `go run ./cmd/zero-release build`.
5. **Smoke test**: `go run ./cmd/zero-release smoke`.
6. **Advisory lint**:
`go run github.com/golangci/golangci-lint/v2/cmd/[email protected] run --enable-only unused,ineffassign,staticcheck ./...`.
7. **Security**:
`go run golang.org/x/vuln/cmd/[email protected] ./...`.
8. **Diff hygiene**: `git diff HEAD --check` (covers staged and unstaged
tracked changes).
`make lint` currently runs the formatting check and `go vet`; it does **not**
run golangci-lint. The pinned golangci-lint job is advisory in CI while the
existing repository-wide backlog is cleaned up. Fix findings introduced by or
related to the current change. Unrelated pre-existing advisory findings do not
justify expanding a focused pull request; report them separately.
Formatting, vet, tests, build, smoke, diff hygiene, and govulncheck are hard
requirements. If a related check fails, fix it. If a required check cannot run
because of the environment or fails for an unrelated external reason, report
the exact failure and obtain maintainer direction rather than silently ignoring
it.
+60 -4
View File
@@ -6,6 +6,60 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html) once the first release is
tagged. Until then, source builds report the version `dev`.
## [0.4.0](https://github.com/Gitlawb/zero/compare/v0.3.0...v0.4.0) (2026-07-17)
### Features
* **aimlapi:** AI/ML API provider with guided onboarding (top-up + key issuance) ([#655](https://github.com/Gitlawb/zero/issues/655)) ([6b9c2f0](https://github.com/Gitlawb/zero/commit/6b9c2f0c083e02bcd3aad68fb7af0501a9e7cd61))
* **cli:** show ZERO wordmark on --version ([#673](https://github.com/Gitlawb/zero/issues/673)) ([9acb411](https://github.com/Gitlawb/zero/commit/9acb4113cc3337b3f361a16278e9cf11ca105e34))
* **cli:** wire MCP serve WorkspaceRoot and --add-dir scope ([#694](https://github.com/Gitlawb/zero/issues/694)) ([75a78e7](https://github.com/Gitlawb/zero/commit/75a78e715bf23154f69c8c79cb58eb4b535b2a2a))
* **npm:** ship the native binary as platform optionalDependencies ([#626](https://github.com/Gitlawb/zero/issues/626)) ([5e1405d](https://github.com/Gitlawb/zero/commit/5e1405d0b7abff5b3ccb3cfdb66d64d6d3322922))
* **perf:** emit prompt-prefix hash fingerprint per turn ([#704](https://github.com/Gitlawb/zero/issues/704)) ([1c5c6e7](https://github.com/Gitlawb/zero/commit/1c5c6e78a8a0e228bdf53d7be90934fbce9d98c3))
* **providers:** add AI/ML API preset (rebased onto main) ([#621](https://github.com/Gitlawb/zero/issues/621)) ([d66a9dd](https://github.com/Gitlawb/zero/commit/d66a9dda69c32aa59f4bea903cfefe00d4b7adef))
* **providers:** refresh MiniMax model coverage ([#665](https://github.com/Gitlawb/zero/issues/665)) ([fa3052a](https://github.com/Gitlawb/zero/commit/fa3052a1422a4ad30a3a6295829564f42dee31a8))
* **skills:** discover shared ~/.agents/skills with multi-root skill loading ([#696](https://github.com/Gitlawb/zero/issues/696)) ([7d57999](https://github.com/Gitlawb/zero/commit/7d579996b43741a2e57e3e38fcdd8484fcfc34e9))
* **tui:** Ctrl+X leader chords and emacs menu navigation ([#699](https://github.com/Gitlawb/zero/issues/699)) ([7f669f4](https://github.com/Gitlawb/zero/commit/7f669f455021be51319ee3b8298cd48a17f745c7))
* **tui:** press up to edit queued messages ([#656](https://github.com/Gitlawb/zero/issues/656)) ([4c986d3](https://github.com/Gitlawb/zero/commit/4c986d327b5ed26338d295c23bea5839681db8d0))
### Bug Fixes
* **acp:** make truncateHint rune-safe ([#614](https://github.com/Gitlawb/zero/issues/614)) ([ddc4927](https://github.com/Gitlawb/zero/commit/ddc4927aac5544bf4dd2c46615aeda5a81c96576))
* **agent,tui:** resolve git branch detection when starting Zero in subdirectories ([#613](https://github.com/Gitlawb/zero/issues/613)) ([0184581](https://github.com/Gitlawb/zero/commit/0184581ec234ea414e0a47bc33e8b0f4ddfb497b))
* **agent:** raise default and deep-mode turn budgets ([#650](https://github.com/Gitlawb/zero/issues/650)) ([635c93a](https://github.com/Gitlawb/zero/commit/635c93af51ebc20f3e0917917e55a79edfe27c35))
* **cli:** prevent consuming positional arguments as flag values ([#619](https://github.com/Gitlawb/zero/issues/619)) ([5b4f48d](https://github.com/Gitlawb/zero/commit/5b4f48d2dcb66402c13bde0c3cfe9c9371da19fb))
* **config:** enforce MCP trust boundary so project config cannot override user disable ([#609](https://github.com/Gitlawb/zero/issues/609)) ([4d8c31c](https://github.com/Gitlawb/zero/commit/4d8c31cf16a3080344e3be7039408fcae71c075d)), closes [#512](https://github.com/Gitlawb/zero/issues/512)
* **config:** surface unknown/typo'd config fields instead of silently dropping them ([#645](https://github.com/Gitlawb/zero/issues/645)) ([893b7b4](https://github.com/Gitlawb/zero/commit/893b7b424cc203a2fcf92327a4e25c84286a90e0))
* **cron:** prevent cron job Mutate from clobbering concurrent updates ([#630](https://github.com/Gitlawb/zero/issues/630)) ([e4bd703](https://github.com/Gitlawb/zero/commit/e4bd703cfb28dab2dfa3c2ddba46237e1bb2e164))
* **daemon:** handle os.ErrPermission as collision during O_EXCL lock creation ([#616](https://github.com/Gitlawb/zero/issues/616)) ([8ea5384](https://github.com/Gitlawb/zero/commit/8ea53841a5d54c37a153779ae86bea010659433c))
* **exec:** stop false INCOMPLETE downgrades on conversational final messages ([#608](https://github.com/Gitlawb/zero/issues/608)) ([b6117af](https://github.com/Gitlawb/zero/commit/b6117af86d6bc87a4ee66910e99d76cb16b03fed))
* harden MCP credential boundaries ([#597](https://github.com/Gitlawb/zero/issues/597)) ([fdddb05](https://github.com/Gitlawb/zero/commit/fdddb05ba84b1600ae6c3a20028bf83afe474c44))
* **hooks:** fail closed on launch failures for beforeTool hooks ([#629](https://github.com/Gitlawb/zero/issues/629)) ([dc06fe7](https://github.com/Gitlawb/zero/commit/dc06fe72caf45f72d2cba1e8a835c0f5b405c1e8))
* **hooks:** run sessionEnd hooks after Esc/Ctrl+C interrupts ([#606](https://github.com/Gitlawb/zero/issues/606)) ([824ecdb](https://github.com/Gitlawb/zero/commit/824ecdbcf9c467c35ef4e2666770fdadcb5bf402))
* **keyring:** pass generic password via stdin on macOS ([#574](https://github.com/Gitlawb/zero/issues/574)) ([91ea6de](https://github.com/Gitlawb/zero/commit/91ea6ded7503538834a84d090f78670a363c62d3))
* **lock:** prevent POSIX lock file overwrite and leak on Windows/Unix ([#628](https://github.com/Gitlawb/zero/issues/628)) ([da41c3a](https://github.com/Gitlawb/zero/commit/da41c3a75b782d6e0836fe13346321e40a90fbb4))
* **openai:** omit prompt_cache_key for openai-compatible providers ([#636](https://github.com/Gitlawb/zero/issues/636)) ([1af5882](https://github.com/Gitlawb/zero/commit/1af58828eb3c22567599c000736c913a290959d2)), closes [#624](https://github.com/Gitlawb/zero/issues/624)
* **plugins:** resolve relative executable paths against plugin root ([#627](https://github.com/Gitlawb/zero/issues/627)) ([2efe6d5](https://github.com/Gitlawb/zero/commit/2efe6d539e29374b3ef39c2290bdea81f33a228b))
* **sandbox:** remove windowsWriteRestricted flag to fix DenyRead bypass ([#612](https://github.com/Gitlawb/zero/issues/612)) ([3d96ac7](https://github.com/Gitlawb/zero/commit/3d96ac7e55c760a97f28c0e6ceaf1ec3b4ab717a))
* **sandbox:** scrub dynamic credential env vars ([#682](https://github.com/Gitlawb/zero/issues/682)) ([9043bae](https://github.com/Gitlawb/zero/commit/9043baedcff7776c7373645b57563cac06b31847))
* **sandbox:** scrub sensitive credentials from sandbox environment ([#660](https://github.com/Gitlawb/zero/issues/660)) ([6fc1220](https://github.com/Gitlawb/zero/commit/6fc1220f6ac66fb3ae67b637cbbed7068d2213c0))
* **sandbox:** unblock git fetch/commit/add under the write-restricted sandbox ([#654](https://github.com/Gitlawb/zero/issues/654)) ([5c4815a](https://github.com/Gitlawb/zero/commit/5c4815a66ed07d9cf90b825adfd936d3ac07639d))
* **sandbox:** use WRITE_RESTRICTED token when no DenyRead paths are configured ([#658](https://github.com/Gitlawb/zero/issues/658)) ([a5d2e32](https://github.com/Gitlawb/zero/commit/a5d2e327c8681671aa8a9e5378801215b747edcf))
* **securefile,credstore:** call Sync on temp file before close and rename ([#631](https://github.com/Gitlawb/zero/issues/631)) ([212734a](https://github.com/Gitlawb/zero/commit/212734adf3b5f982e22385161205aa1afe4634fe))
* **securefile:** reclaim stale lock files to prevent permanent DOS ([#615](https://github.com/Gitlawb/zero/issues/615)) ([8536cc8](https://github.com/Gitlawb/zero/commit/8536cc87f7a885f9e436d6ef28f5f325201623dc))
* **swarm:** wait for job.Runs directly in scheduler skip test ([#667](https://github.com/Gitlawb/zero/issues/667)) ([1bb6b57](https://github.com/Gitlawb/zero/commit/1bb6b5745af90321d1e657a12a1976cded5dd1bd))
* **tools:** classify silent wrapped Windows command failures as sandbox denials ([#659](https://github.com/Gitlawb/zero/issues/659)) ([8bd9742](https://github.com/Gitlawb/zero/commit/8bd9742fa95c41b93ea4e718628aed0ff3ae9dd0))
* **tools:** preserve SysProcAttr during PTY fallback ([#618](https://github.com/Gitlawb/zero/issues/618)) ([f78b36c](https://github.com/Gitlawb/zero/commit/f78b36c770daa4577a9f99265b18a354454e36eb))
* **tui:** resolve pending askUser callbacks to prevent runner hangs ([#620](https://github.com/Gitlawb/zero/issues/620)) ([aa73a76](https://github.com/Gitlawb/zero/commit/aa73a76f1bd1b6fe97bac2fbff2d61b7474139f2))
* **tui:** stop the composer cursor blinking while typing or unfocused ([#672](https://github.com/Gitlawb/zero/issues/672)) ([2b42cd5](https://github.com/Gitlawb/zero/commit/2b42cd567b96d6f7e0818594e53ff31cce1e42e9))
* **windows:** resolve absolute path for taskkill to prevent hijacking ([#617](https://github.com/Gitlawb/zero/issues/617)) ([2db00ee](https://github.com/Gitlawb/zero/commit/2db00ee3d57db97e0fbe23cb8628e8bcb47f6f09))
### Performance Improvements
* **tools:** add explicit effect metadata for safe concurrency ([#705](https://github.com/Gitlawb/zero/issues/705)) ([8ef8576](https://github.com/Gitlawb/zero/commit/8ef8576df7d0775a5b815bc0776a794cadb75c34))
## [0.3.0](https://github.com/Gitlawb/zero/compare/v0.2.0...v0.3.0) (2026-07-09)
@@ -134,10 +188,12 @@ tagged. Until then, source builds report the version `dev`.
GitHub issue/PR templates.
- Interactive `/theme` picker: bare `/theme` opens a popup that live-previews each palette as you move
and applies on select (Esc reverts).
- Ten built-in color themes alongside the `dark`/`light` built-ins — `dracula`, `nord`, `gruvbox`,
`tokyo-night`, `catppuccin`, `one-dark`, `solarized-dark`, `rose-pine`, `everforest`, and
`solarized-light` — selectable via `/theme <name>`, `--theme <name>`, or `ZERO_THEME`. Every palette
is contrast-audited to WCAG AA. The built-in light theme was reworked for legibility.
- Twelve built-in color themes alongside the `dark`/`light` built-ins — `dracula`, `nord`, `gruvbox`,
`tokyo-night`, `catppuccin`, `one-dark`, `solarized-dark`, `rose-pine`, `everforest`,
`solarized-light`, `dune`, and `neon` — selectable via `/theme <name>`, `--theme <name>`, or
`ZERO_THEME`. Every palette is contrast-audited to WCAG AA, and the new presets are additionally
audited after xterm-256 downsampling; see [docs/THEMES.md](docs/THEMES.md). The built-in light
theme was reworked for legibility.
- `--theme <name>` flag for the TUI, accepting `auto` or any registered theme (previously only the
`ZERO_THEME` env var existed).
- "Accessibility / Appearance" section in the README documenting `NO_COLOR`, `ZERO_THEME`, `/theme`,
+26 -5
View File
@@ -8,6 +8,13 @@ stabilizes. Community participation is still welcome, especially through bug
reports, feature requests, reproduction details, logs, and discussion on existing
issues.
Questions and setup help belong in
[GitHub Discussions](https://github.com/Gitlawb/zero/discussions) (see
[SUPPORT.md](SUPPORT.md)), not in the issue tracker. Feature ideas start in the
[Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas) category;
an idea that gains traction there is the path to an approved issue and,
eventually, an accepted PR.
Please read this policy before opening a pull request. Pull requests that do not
follow it may be closed without review.
@@ -33,7 +40,11 @@ unless the issue has been explicitly approved for outside contribution.
There are three primary ways to contribute:
1. Open a bug report with clear reproduction steps.
2. Open a feature request describing the problem, use case, and expected behavior.
2. Propose a feature in
[Discussions Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas),
describing the problem, use case, and expected behavior. A feature-request
issue is filed only after the idea gains traction there or a maintainer asks
for one.
3. Add useful context to an existing issue, such as logs, screenshots, examples,
affected versions, or reproduction cases.
@@ -46,7 +57,8 @@ All community pull requests require an approved parent issue.
Before starting implementation work:
1. Open an issue using the appropriate issue template.
1. For a bug, open an issue using the bug-report template. For a feature, start
in Discussions Ideas; the feature-request issue comes after the discussion.
2. Describe the bug, feature request, or functional change clearly.
3. Wait for the core team to review the issue.
4. Do not open a pull request unless the issue has the `issue-approved` label.
@@ -133,7 +145,12 @@ Clear bug reports are the most useful way to help the project during this phase.
## Submitting a Feature Request
When filing a feature request, please include:
Feature requests start as a thread in
[Discussions Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas),
not as an issue. The feature-request issue form exists for the step AFTER that:
when an idea has gained traction in Discussions or a maintainer has asked for an
issue to track it. Whether in the discussion or the eventual issue, please
include:
- The problem you are trying to solve.
- The behavior or functionality you are requesting.
@@ -156,5 +173,9 @@ contribution phase.
## Getting Help
If you are unsure whether a change is appropriate, open an issue first and ask.
Please keep discussion focused, specific, and respectful of maintainer time.
General questions, setup help, and "is this a bug or my config" uncertainty all
go to
[Discussions Q&A](https://github.com/Gitlawb/zero/discussions/categories/q-a).
If you are unsure whether a proposed code change is appropriate, raise it in
Discussions Ideas or on the relevant issue first and ask. Please keep discussion
focused, specific, and respectful of maintainer time.
+30 -2
View File
@@ -191,9 +191,11 @@ Useful controls:
|---|---|
| `Enter` | send the prompt |
| `/` | open slash-command suggestions |
| `Ctrl+X` then letter | common slash commands (e.g. `m` → `/model`; `Ctrl+X` `?` for full list) |
| `Ctrl+P` / `Ctrl+N` | previous / next item in menus (arrows still work) |
| `Shift+Tab` | cycle permission mode |
| `Ctrl+B` | show/hide the sidebar |
| `Ctrl+C` | cancel or exit |
| `Ctrl+C` | cancel, exit, or return from a `/btw` conversation |
Common slash commands:
@@ -203,6 +205,7 @@ Common slash commands:
| `/spec`, `/plan` | draft and review a plan before building |
| `/image` | attach an image for vision-capable models |
| `/resume`, `/rewind` | continue or roll back local sessions |
| `/btw [question]` | ask in an isolated fork without adding the side conversation to the main session |
| `/loop` | repeat a prompt or custom `/command` on an interval (`/loop 5m /babysit-prs`) or self-paced |
| `/compact`, `/context` | manage context usage |
| `/permissions`, `/tools` | inspect available tools and policy |
@@ -339,7 +342,7 @@ manifest.
| Control | Effect |
|---|---|
| `NO_COLOR=<anything>` | disables color output |
| `ZERO_THEME=<name>` | selects the startup theme (`auto`, `dark`, `light`, or a color theme like `dracula`, `nord`, `gruvbox`, `tokyo-night`, `catppuccin`, `one-dark`, `solarized-dark`, `rose-pine`, `everforest`, `solarized-light`) |
| `ZERO_THEME=<name>` | selects the startup theme (`auto`, `dark`, `light`, or a color theme like `dracula`, `nord`, `gruvbox`, `tokyo-night`, `catppuccin`, `one-dark`, `solarized-dark`, `rose-pine`, `everforest`, `neon`, `solarized-light`, `dune`) |
| `--theme <name>` | selects the TUI theme from the CLI (same names) |
| `/theme` | opens the theme picker inside the TUI (live preview; `/theme <name>` switches directly) |
| `ZERO_NO_FADE=1` | disables streaming fade animation |
@@ -356,6 +359,12 @@ go run ./cmd/zero-release smoke
go run ./cmd/zero-perf-bench
```
Experimental: `ZERO_OPENAI_TURN_SESSION=1` enables the optimized OpenAI turn
session (background connection prewarm + request-prefix telemetry) for headless
`zero exec` runs against official OpenAI profiles. Off by default; `0`/`false`
disable. A/B-benchmark it by running the same `zero-perf-bench` suite with the
variable unset and set.
### Code Quality and Security Checks
Before committing any changes, ensure all Go code quality and security checks pass. Pinned `go run` commands matching CI constraints can be used directly without prior installation:
@@ -386,6 +395,7 @@ go run ./cmd/zero-release build --goos windows --goarch amd64 --output dist/zero
- [Install](docs/INSTALL.md)
- [Update flow](docs/UPDATE.md)
- [Themes](docs/THEMES.md)
- [Stream-JSON protocol](docs/STREAM_JSON_PROTOCOL.md)
- [Specialists](docs/SPECIALISTS.md)
- [GitHub Action](docs/GITHUB_ACTION.md)
@@ -393,6 +403,24 @@ go run ./cmd/zero-release build --goos windows --goarch amd64 --output dist/zero
- [Performance](docs/PERFORMANCE.md)
- [Agent evals](docs/AGENT_EVALS.md)
## Community
Questions, setup help, ideas, and sharing all live in
[GitHub Discussions](https://github.com/Gitlawb/zero/discussions):
| Category | Use it for |
|---|---|
| [Q&A](https://github.com/Gitlawb/zero/discussions/categories/q-a) | Setup help, provider/model configuration, "how do I" questions |
| [Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas) | Feature proposals and design discussion before any PR |
| [Show and tell](https://github.com/Gitlawb/zero/discussions/categories/show-and-tell) | Your skills, plugins, MCP setups, themes, and workflows |
| [Announcements](https://github.com/Gitlawb/zero/discussions/categories/announcements) | Releases and project news from the maintainers |
For a good Q&A answer fast, include `zero --version`, your OS and install
method, the provider/model in use, and `zero doctor` output. See
[SUPPORT.md](SUPPORT.md). Bugs belong in
[issues](https://github.com/Gitlawb/zero/issues/new/choose); security reports
follow [SECURITY.md](SECURITY.md), never a public thread.
## Contributing
Contributions are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md), run the
+2 -1
View File
@@ -234,7 +234,7 @@ zero update 检查更新版本
| 控制 | 效果 |
|---|---|
| `NO_COLOR=<任意值>` | 禁用颜色输出 |
| `ZERO_THEME=<名称>` | 选择启动主题(`auto`、`dark`、`light`,或颜色主题如 `dracula`、`nord`、`gruvbox`、`tokyo-night`、`catppuccin`、`one-dark`、`solarized-dark`、`rose-pine`、`everforest`、`solarized-light`) |
| `ZERO_THEME=<名称>` | 选择启动主题(`auto`、`dark`、`light`,或颜色主题如 `dracula`、`nord`、`gruvbox`、`tokyo-night`、`catppuccin`、`one-dark`、`solarized-dark`、`rose-pine`、`everforest`、`neon`、`solarized-light`、`dune`) |
| `--theme <名称>` | 从 CLI 选择 TUI 主题(相同名称) |
| `/theme` | 在 TUI 中打开主题选择器(实时预览;`/theme <名称>` 直接切换) |
| `ZERO_NO_FADE=1` | 禁用流式淡入动画 |
@@ -261,6 +261,7 @@ go run ./cmd/zero-release build --goos windows --goarch amd64 --output dist/zero
- [安装](docs/INSTALL.md)
- [更新流程](docs/UPDATE.md)
- [主题](docs/THEMES.md)
- [Stream-JSON 协议](docs/STREAM_JSON_PROTOCOL.md)
- [专家](docs/SPECIALISTS.md)
- [GitHub Action](docs/GITHUB_ACTION.md)
+36
View File
@@ -0,0 +1,36 @@
# Getting help with Zero
## Questions, setup help, configuration
Ask in [GitHub Discussions](https://github.com/Gitlawb/zero/discussions). Use
[Q&A](https://github.com/Gitlawb/zero/discussions/categories/q-a) so answers can
be marked and found by the next person.
The fastest way to a useful answer is to include:
- `zero --version` output
- your OS (Windows, macOS, Linux) and how you installed Zero (npm, install
script, source)
- the provider and model you are using (`zero providers list`)
- `zero doctor` output for setup problems
- the exact command you ran and the exact error text
## Bugs
If Zero does something wrong (crashes, wrong output, broken behavior), file a
[bug report](https://github.com/Gitlawb/zero/issues/new/choose). If you are not
sure whether it is a bug or a setup problem, start in Discussions and we will
promote it to an issue if it is one.
## Feature ideas
Start a thread in
[Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas). Zero is
in controlled development (see CONTRIBUTING.md), so discussing an idea first is
also the path to getting a PR accepted.
## Security issues
Never file a public issue or discussion. Report privately via
[GitHub Security Advisories](https://github.com/Gitlawb/zero/security/advisories/new)
(see SECURITY.md).
+159 -15
View File
@@ -55,6 +55,133 @@ function localControlHelperManifest(packageRoot) {
return JSON.stringify({ version: 1, helpers });
}
// Mirrors internal/cli/observability.go parseDoctorArgs + writeDoctorHelp so the
// wrapper's missing-binary doctor fallback matches the Go doctor CLI surface.
const DOCTOR_HELP = `Usage:
zero doctor [flags]
Runs Go backend health checks for config and provider setup.
Flags:
--json Print JSON report
--connectivity Include provider endpoint connectivity probe when available
-h, --help Show this help
`;
const EXIT_USAGE = 2;
function installedByBun() {
if (process.env.ZERO_WRAPPER_SIMULATE_BUN === '1') {
return true;
}
return process.execPath.includes('bun') || !!process.versions?.bun;
}
function bunRecoveryParagraph() {
return (
'You installed with Bun, which does not run dependency lifecycle scripts\n' +
'by default. Trust the package to run the blocked postinstall:\n' +
' bun pm trust @gitlawb/zero (project install)\n' +
' bun pm -g trust @gitlawb/zero (global install)\n' +
'On Bun versions without `bun pm trust`, add\n' +
' "trustedDependencies": ["@gitlawb/zero"]\n' +
'to your project package.json and reinstall.\n' +
'\n'
);
}
function buildFromSourceParagraph() {
return (
'If this platform has no prebuilt binary, build from source:\n' +
'https://github.com/Gitlawb/zero (go run ./cmd/zero, requires Go 1.26+).'
);
}
function missingBinaryContextParagraph() {
return (
'Normally npm installs it as an optional dependency of @gitlawb/zero\n' +
`(@gitlawb/zero-${process.platform}-${process.arch}), and the wrapper can\n` +
'also download it from the GitHub Release when it is missing.'
);
}
function missingNativeRecoveryParagraphs(postinstallScript) {
const ranByBun = installedByBun();
return (
missingBinaryContextParagraph() +
'\n' +
'\n' +
'Things to try:\n' +
' - reinstall without omitting optional dependencies:\n' +
' npm install -g @gitlawb/zero\n' +
' - run the downloader manually (needs write access to the package dir):\n' +
` node "${postinstallScript}"\n` +
'\n' +
(ranByBun ? bunRecoveryParagraph() : '') +
buildFromSourceParagraph()
);
}
function formatGenericMissingNativeBinaryMessage(postinstallScript) {
return (
'[zero] No native binary is available for this install.\n' +
missingNativeRecoveryParagraphs(postinstallScript)
);
}
function parseDoctorArgs(args) {
let json = false;
for (const arg of args) {
switch (arg) {
case '-h':
case '--help':
case 'help':
return { kind: 'help' };
case '--json':
json = true;
break;
case '--connectivity':
break;
default:
return { kind: 'error', message: `unknown doctor flag ${JSON.stringify(arg)}` };
}
}
return { kind: 'run', json };
}
function missingNativeDoctorJSONReport(postinstallScript) {
return {
generatedAt: new Date().toISOString().replace(/\.\d{3}Z$/, 'Z'),
ok: false,
checks: [
{
id: 'runtime.go',
label: 'Go runtime',
status: 'fail',
message: 'Native zero binary is missing next to the npm wrapper.',
details: {
remedy: `node "${postinstallScript}"`,
},
},
],
};
}
function missingNativeDoctorTextReport(postinstallScript) {
return (
'Zero doctor report (' +
new Date().toISOString() +
')\n' +
'Overall: fail\n' +
'[fail] runtime.go - Native zero binary is missing next to the npm wrapper.\n' +
' remedy: node "' +
postinstallScript +
'"\n' +
'\n' +
missingNativeRecoveryParagraphs(postinstallScript)
);
}
// The platform payload is a version of @gitlawb/zero itself, installed under
// an npm: alias (see docs/NPM_PACKAGING.md). The alias name is derived from
// process.platform/process.arch, so an unsupported platform simply fails to
@@ -114,21 +241,38 @@ function resolveNativeBinary() {
const nativePath = resolveNativeBinary();
if (!nativePath) {
console.error(
'[zero] No native binary is available for this install.\n' +
'Normally npm installs it as an optional dependency of @gitlawb/zero\n' +
`(@gitlawb/zero-${process.platform}-${process.arch}), and the wrapper can\n` +
'also download it from the GitHub Release when it is missing.\n' +
'\n' +
'Things to try:\n' +
' - reinstall without omitting optional dependencies:\n' +
' npm install -g @gitlawb/zero\n' +
' - run the downloader manually (needs write access to the package dir):\n' +
` node "${join(packageRoot, 'scripts', 'postinstall.mjs')}"\n` +
'\n' +
'If this platform has no prebuilt binary, build from source:\n' +
'https://github.com/Gitlawb/zero (go run ./cmd/zero, requires Go 1.26+).',
);
const postinstallScript = join(packageRoot, 'scripts', 'postinstall.mjs');
// `zero doctor` is the diagnostic command: when the native binary is missing
// it's the one invocation that must NOT bail with the generic wrapper error,
// because that's exactly the blind alley issue #405 reports. Instead, surface
// a doctor-shaped FAIL line for the runtime so the user's diagnostic finds the
// real cause. We branch on a literal 'doctor' subcommand only (matching `zero
// doctor` and `zero doctor --connectivity`), preserving the existing bail for
// every other invocation (exec, providers list, TUI, --version, etc.).
const argv = process.argv.slice(2);
const isDoctor = argv.length > 0 && argv[0] === 'doctor';
if (isDoctor) {
const parsed = parseDoctorArgs(argv.slice(1));
if (parsed.kind === 'help') {
process.stdout.write(DOCTOR_HELP);
process.exit(0);
}
if (parsed.kind === 'error') {
process.stderr.write(`[zero] ${parsed.message}\n`);
process.exit(EXIT_USAGE);
}
if (parsed.json) {
process.stdout.write(JSON.stringify(missingNativeDoctorJSONReport(postinstallScript), null, 2) + '\n');
process.exit(1);
}
process.stdout.write(missingNativeDoctorTextReport(postinstallScript) + '\n');
process.exit(1);
}
console.error(formatGenericMissingNativeBinaryMessage(postinstallScript));
process.exit(1);
}
+17
View File
@@ -4,6 +4,8 @@ import (
"bytes"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/perfbench"
)
func TestParseArgsUsesCliAndEnvOverrides(t *testing.T) {
@@ -92,3 +94,18 @@ func TestHelpText(t *testing.T) {
func emptyEnv(string) string {
return ""
}
func TestTurnExitCodeFailsWhenEveryTaskErrored(t *testing.T) {
var stderr bytes.Buffer
allErrored := perfbench.TurnBenchResult{TasksAttempted: 3, TasksErrored: 3}
if code := turnExitCode(allErrored, &stderr); code != 1 {
t.Fatalf("exit=%d, want 1 when every task errored", code)
}
if !strings.Contains(stderr.String(), "no accepted benchmark sample") {
t.Fatalf("stderr missing explanation: %q", stderr.String())
}
partial := perfbench.TurnBenchResult{TasksAttempted: 3, TasksErrored: 2}
if code := turnExitCode(partial, &stderr); code != 0 {
t.Fatalf("exit=%d, want 0 for partial errors (summary surfaces them)", code)
}
}
+42 -1
View File
@@ -9,6 +9,7 @@ import (
"path/filepath"
"strings"
"github.com/Gitlawb/zero/internal/execprofile"
"github.com/Gitlawb/zero/internal/perfbench"
)
@@ -20,6 +21,7 @@ type turnOptions struct {
SuitePath string
Model string
Mode string
ExecProfile string
SelfCorrect bool
Binary string
Iterations int
@@ -64,6 +66,7 @@ func runTurnCommand(args []string, getenv func(string) string, stdout io.Writer,
result, err := perfbench.RunTurnBench(context.Background(), set, perfbench.TurnBenchConfig{
Model: options.Model,
Mode: options.Mode,
ExecProfile: options.ExecProfile,
SelfCorrect: options.SelfCorrect,
Version: options.Version,
Commit: options.Commit,
@@ -86,9 +89,23 @@ func runTurnCommand(args []string, getenv func(string) string, stdout io.Writer,
_, _ = fmt.Fprintln(stderr, "[zero] Turn benchmark failed: "+err.Error())
return 1
}
return 0
return turnExitCode(result, stderr)
}
_, _ = fmt.Fprintln(stdout, perfbench.FormatTurnBenchSummary(result))
return turnExitCode(result, stderr)
}
// turnExitCode fails the command when every attempted task errored (no
// iteration produced an accepted benchmark sample): such a report measures
// nothing, and exiting 0 would let a broken configuration (missing binary, bad
// path, failing harness step) pass as a clean baseline. Partial errors keep
// exit 0 — the summary surfaces them loudly and the surviving samples are
// still valid measurements.
func turnExitCode(result perfbench.TurnBenchResult, stderr io.Writer) int {
if result.TasksAttempted > 0 && result.TasksErrored == result.TasksAttempted {
_, _ = fmt.Fprintln(stderr, "[zero] Turn benchmark failed: every task errored with no accepted benchmark sample (see warnings); the report contains no valid measurements")
return 1
}
return 0
}
@@ -123,6 +140,13 @@ func parseTurnArgs(args []string, getenv func(string) string) (turnOptions, erro
}
options.Mode = value
index = next
case "--exec-profile":
value, next, err := readOptionValue(args, inlineValue, index, flag)
if err != nil {
return options, err
}
options.ExecProfile = value
index = next
case "--binary":
value, next, err := readOptionValue(args, inlineValue, index, flag)
if err != nil {
@@ -195,6 +219,20 @@ func parseTurnArgs(args []string, getenv func(string) string) (turnOptions, erro
if strings.TrimSpace(options.Model) == "" && !options.DryRun {
return options, fmt.Errorf("--model is required (or pass --dry-run)")
}
// Validate the profile here, before anything spawns: an unknown name would
// otherwise make every child exit with a usage error in milliseconds, and
// those near-zero walls would be recorded as valid latency samples in an
// exit-0 report — exactly the misread-as-improvement trap the harness
// closed for spawn failures. Normalizing to the catalog name also keeps
// the stamped execProfile canonical (Lookup is case-insensitive), so two
// captures of the same posture always compare equal.
if raw := strings.TrimSpace(options.ExecProfile); raw != "" {
profile, ok := execprofile.Lookup(raw)
if !ok {
return options, fmt.Errorf("unknown execution profile %q for --exec-profile. Valid profiles: %s", raw, strings.Join(execprofile.Names(), ", "))
}
options.ExecProfile = profile.Name
}
return options, nil
}
@@ -225,6 +263,9 @@ func turnHelpText() string {
" --suite <path> Task set JSON file (required)",
" --model <model> Model to run (required unless --dry-run)",
" --mode <name> Exec mode preset to apply",
" --exec-profile <name>",
" Execution profile for every task (fast|balanced|thorough);",
" forwarded to zero exec and stamped into the result",
" --self-correct Enable the post-edit verify-and-correct loop",
" --binary <path> Path to the `zero` binary (default: zero on PATH / repo root)",
" --iterations <n> Times to run each task (default: 1)",
+65
View File
@@ -0,0 +1,65 @@
package main
import (
"strings"
"testing"
)
func TestParseTurnArgsExecProfile(t *testing.T) {
getenv := func(string) string { return "" }
options, err := parseTurnArgs([]string{
"--suite", "suite.json",
"--model", "m",
"--exec-profile", "fast",
}, getenv)
if err != nil {
t.Fatalf("parseTurnArgs: %v", err)
}
if options.ExecProfile != "fast" {
t.Fatalf("ExecProfile = %q, want fast", options.ExecProfile)
}
// Inline form.
options, err = parseTurnArgs([]string{
"--suite=suite.json",
"--model=m",
"--exec-profile=thorough",
}, getenv)
if err != nil {
t.Fatalf("parseTurnArgs inline: %v", err)
}
if options.ExecProfile != "thorough" {
t.Fatalf("ExecProfile = %q, want thorough", options.ExecProfile)
}
}
// The bench validates the profile BEFORE spawning anything: an unknown name
// would otherwise make every child exit with a usage error in milliseconds
// and record those near-zero walls as valid latency samples in an exit-0
// report. Lookup normalization also keeps the stamped name canonical so two
// captures of the same posture compare equal.
func TestParseTurnArgsExecProfileValidatesAndNormalizes(t *testing.T) {
getenv := func(string) string { return "" }
options, err := parseTurnArgs([]string{
"--suite", "suite.json", "--model", "m", "--exec-profile", "FAST",
}, getenv)
if err != nil {
t.Fatalf("parseTurnArgs: %v", err)
}
if options.ExecProfile != "fast" {
t.Fatalf("ExecProfile = %q, want the canonical fast", options.ExecProfile)
}
_, err = parseTurnArgs([]string{
"--suite", "suite.json", "--model", "m", "--exec-profile", "blanced",
}, getenv)
if err == nil {
t.Fatal("an unknown profile must fail parse, before anything spawns")
}
for _, name := range []string{"balanced", "fast", "thorough"} {
if !strings.Contains(err.Error(), name) {
t.Fatalf("error must list %q, got %q", name, err.Error())
}
}
}
+16 -3
View File
@@ -272,7 +272,8 @@ sequenceDiagram
Agent->>Tool: execute(ctx, args)
Tool-->>Agent: output, error, metadata
Agent->>Hooks: afterTool hooks
Agent->>Agent: redact, truncate, classify success/failure
Agent->>Agent: redact secrets, apply semantic output budget, enforce byte ceiling
Agent->>Agent: classify success/failure
Agent-->>Surface: OnToolResult callback
Agent->>Transcript: append tool result message
end
@@ -604,10 +605,22 @@ flowchart TD
Prompt -- Yes --> Decision[Permission callback]
Prompt -- No --> Run[Registry.RunWithOptions]
Decision --> Run
Run --> Redact[Redact secrets + enforce output ceiling]
Redact --> Message[Return tool result to model]
Run --> Redact[Redact secrets]
Redact --> Budget[Token-aware semantic output budget]
Budget --> Ceiling[Existing hard byte ceiling]
Ceiling --> Message[Return one tool result to model]
```
Oversized results use deterministic, provider-neutral estimated-token budgets.
Policies retain useful structure for files, search matches, tests, process logs,
diffs, and worker conclusions; tools without a declared category use a UTF-8-safe
head/tail fallback. The estimate is intentionally conservative for non-ASCII
text and is not exact provider tokenization. Existing byte ceilings remain the
authoritative safety limit. When the existing spill mechanism can persist the
complete redacted text received by the budgeting layer, the result includes its
safe spill reference. This does not imply capture of subprocess bytes already
discarded by a tool's established internal buffer.
Core tool groups include:
- **Read-only tools**: file reads, directory listing, glob, grep, LSP navigation,
+39
View File
@@ -0,0 +1,39 @@
# TUI Themes
Zero's TUI ships a set of built-in color themes. Pick one with `--theme <name>`,
the `ZERO_THEME` environment variable, or the `/theme <name>` command while
running. `auto` (the default) follows the terminal's detected background.
Run `/theme` with no argument to open a picker: move through the list to
live-preview each theme, press Enter to apply the highlighted one, or Esc to
cancel and restore the previously active theme. Run `/theme list` to print
the active theme and the registered names without opening the picker.
## Dune (`dune`)
A warm sand-and-cream palette: sand/cream surface, charcoal ink, and a soft
amber accent.
## Neon (`neon`)
A neon-on-black palette: pitch-black surface, bright green ink, and a cyan accent.
## Adding a theme
Every theme is a `palette` (a table of color hex tokens) plus one entry in
`themeRegistry`, both in `internal/tui/theme_palettes.go`. `buildTheme` in
`internal/tui/theme.go` turns a `palette` into the resolved `lipgloss.Style`
set every renderer reads from the active `zeroTheme`. Adding a theme means
adding a new `palette{...}` literal, a `themeRegistry` entry, and test
coverage for the new palette (see below).
Registry-wide tests in `internal/tui/theme_select_test.go` assert the basic
WCAG AA text tokens, the gray-ramp order, the diff word-span pairs, and the
selected-row band for every entry. The rendered-surface invariants beyond
those (permission surfaces, selected-row secondary text, diff gutters, and
the xterm-256 downsampling checks) are asserted per palette, not against the
whole registry: `TestExtendedThemeContrastInvariants` and
`TestExtendedThemeANSI256Contrast` enumerate the palettes they cover. A new
theme must be added to those tests (or given equivalent palette-specific
assertions), or CI can stay green while its permission, selected-row, and
diff surfaces ship unreadable.
+6 -1
View File
@@ -141,9 +141,11 @@ func TestRunAskUserCancellationAbortsRun(t *testing.T) {
registry := registryWithAskUser()
args := `{"questions":[{"question":"Which framework?"}]}`
provider := providerCallingAskUserThenAnswer(args, "done")
var toolResults []ToolResult
result, err := Run(context.Background(), "clarify", provider, Options{
Registry: registry,
Registry: registry,
OnToolResult: func(result ToolResult) { toolResults = append(toolResults, result) },
OnAskUser: func(_ context.Context, _ AskUserRequest) (AskUserResponse, error) {
return AskUserResponse{}, context.Canceled
},
@@ -158,6 +160,9 @@ func TestRunAskUserCancellationAbortsRun(t *testing.T) {
if len(provider.requests) != 1 {
t.Fatalf("expected the run to stop after the canceled ask_user (1 turn), got %d", len(provider.requests))
}
if len(toolResults) != 1 || toolResults[0].ToolCallID == "" {
t.Fatalf("cancellation must emit exactly one result for the call, got %#v", toolResults)
}
// The recorded tool result must reflect cancellation, not a synthetic answer.
var toolMsg string
for _, m := range result.Messages {
+12 -3
View File
@@ -56,6 +56,10 @@ type CompactionOptions struct {
// injected so Compact stays pure and testable; the agent loop wires it to a
// real provider call.
Summarize func(toSummarize []zeroruntime.Message) (string, error)
// taskState is a snapshot supplied by the running agent. Its immutable
// objective is always preserved; mutable fields are admitted only when its
// plan projection still matches the transcript.
taskState *taskStateSnapshot
}
// CompactionResult is the metadata-bearing result returned by CompactMessages.
@@ -220,7 +224,7 @@ func CompactMessages(messages []zeroruntime.Message, opts CompactionOptions) (Co
// Preserve structured state (active plan + loaded skills) from the elided
// middle verbatim, so it is not lost or paraphrased away by the prose summary.
content := appendPreservedState(summaryLabel+"\n"+summary, middle)
content := appendPreservedState(summaryLabel+"\n"+summary, middle, opts.taskState)
compacted := make([]zeroruntime.Message, 0, systemEnd+1+(len(messages)-boundary))
compacted = append(compacted, messages[:systemEnd]...)
@@ -326,6 +330,7 @@ type compactionState struct {
// OnText is deliberately NOT forwarded (compaction stays invisible to the user),
// but its token COST must still be counted so usage reports and budgets include it.
onUsage func(Usage)
task *taskState
// calibrationRatio scales the raw byte/4 token estimate toward the provider's
// real prompt-token count. ApproxTextTokens over-counts code-heavy content by
@@ -365,13 +370,15 @@ func (state *compactionState) calibratedTokens(raw int) int {
return int(float64(raw) * state.calibrationRatio)
}
func newCompactionState(options Options) *compactionState {
return &compactionState{
func newCompactionState(options Options, task *taskState) *compactionState {
state := &compactionState{
enabled: options.ContextWindow > 0,
threshold: compactionThreshold(options.ContextWindow),
preserveLast: options.CompactionPreserveLast,
onUsage: options.OnUsage,
task: task,
}
return state
}
// maybeCompact runs proactive compaction at the top of a turn. It returns the
@@ -418,6 +425,7 @@ func (state *compactionState) maybeCompact(
compacted, err := Compact(messages, CompactionOptions{
PreserveLast: state.preserveLast,
Summarize: summarizeClosure(ctx, provider, state.onUsage),
taskState: state.task.snapshotForCompaction(messages),
})
if err != nil {
// Summarizer failed: keep the original history. The reactive path (or a
@@ -468,6 +476,7 @@ func (state *compactionState) recover(
result, compactErr := Compact(messages, CompactionOptions{
PreserveLast: state.preserveLast,
Summarize: summarizeClosure(ctx, provider, state.onUsage),
taskState: state.task.snapshotForCompaction(messages),
})
if compactErr != nil {
// A genuine compaction attempt was made (and failed): the budget is spent
+60 -29
View File
@@ -30,6 +30,10 @@ const (
const (
maxRecentEdits = 20
maxEditNoteBytes = 160
// maxTaskObjectiveBytes keeps the original objective recognizable after
// compaction without allowing an unusually large prompt to recreate the
// context pressure the compaction just removed.
maxTaskObjectiveBytes = 512
)
const (
@@ -76,32 +80,14 @@ func extractLatestPlan(messages []zeroruntime.Message) string {
// ({"plan":[{content,status,...}]}) as terse status-tagged bullet lines. Returns
// "" on malformed arguments or an empty plan.
func formatPlanArguments(arguments string) string {
var parsed struct {
Plan []struct {
Content string `json:"content"`
Step string `json:"step"`
Status string `json:"status"`
Notes string `json:"notes"`
} `json:"plan"`
}
if err := json.Unmarshal([]byte(strings.TrimSpace(arguments)), &parsed); err != nil {
plan, ok := parseTaskPlan(arguments)
if !ok {
return ""
}
lines := make([]string, 0, len(parsed.Plan))
for _, item := range parsed.Plan {
content := strings.TrimSpace(item.Content)
if content == "" {
content = strings.TrimSpace(item.Step)
}
if content == "" {
continue
}
status := strings.TrimSpace(item.Status)
if status == "" {
status = "pending"
}
line := "- [" + status + "] " + content
if notes := strings.TrimSpace(item.Notes); notes != "" {
lines := make([]string, 0, len(plan))
for _, item := range plan {
line := "- [" + item.Status + "] " + item.Content
if notes := item.Notes; notes != "" {
line += "\n Notes: " + notes
}
lines = append(lines, line)
@@ -431,12 +417,25 @@ func capBody(body string) string {
// preservedState is the JSON shape of the carried-across-compaction block.
type preservedState struct {
Plan string `json:"plan,omitempty"`
Task *preservedTaskState `json:"task,omitempty"`
RecentEdits []preservedEdit `json:"recent_edits,omitempty"`
Tools []preservedTool `json:"tools,omitempty"`
Skills []preservedSkill `json:"skills,omitempty"`
ProjectInstructions []preservedInstruction `json:"project_instructions,omitempty"`
}
type preservedTaskState struct {
Objective string `json:"objective"`
Status taskStatus `json:"status,omitempty"`
Pending int `json:"pending,omitempty"`
InProgress int `json:"in_progress,omitempty"`
Completed int `json:"completed,omitempty"`
Failed int `json:"failed,omitempty"`
VerificationPassed int `json:"verification_passed,omitempty"`
VerificationFailed int `json:"verification_failed,omitempty"`
VerificationOutcome Outcome `json:"verification_outcome,omitempty"`
}
type preservedEdit struct {
Path string `json:"path"`
Note string `json:"note,omitempty"`
@@ -464,8 +463,27 @@ type preservedInstruction struct {
// may live only inside the injected summary message, which on a later compaction
// lands in middle with no real tool calls left to extract. Fresh tool calls and
// instruction blocks override the carried-forward copy by name/source.
func appendPreservedState(summary string, middle []zeroruntime.Message) string {
func appendPreservedState(summary string, middle []zeroruntime.Message, taskSnapshot *taskStateSnapshot) string {
priorState := parsePreservedStateBlock(latestSummaryContent(middle))
task := priorState.Task
if taskSnapshot != nil {
task = &preservedTaskState{
Objective: capTaskObjective(taskSnapshot.Objective),
}
// Plan parity corroborates only the mutable task projection. The objective
// comes directly from the run prompt and is immutable, so it must survive
// even after compaction removes the plan tool call needed for comparison.
if taskSnapshot.PlanParity == taskPlanParityMatch {
task.Status = taskSnapshot.Status
task.Pending = taskSnapshot.Plan.Pending
task.InProgress = taskSnapshot.Plan.InProgress
task.Completed = taskSnapshot.Plan.Completed
task.Failed = taskSnapshot.Plan.Failed
task.VerificationPassed = taskSnapshot.Verification.Passed
task.VerificationFailed = taskSnapshot.Verification.Failed
task.VerificationOutcome = taskSnapshot.Verification.LastOutcome
}
}
// Plan: a fresh update_plan in middle is authoritative; otherwise carry the
// plan preserved by an earlier compaction.
@@ -493,12 +511,25 @@ func appendPreservedState(summary string, middle []zeroruntime.Message) string {
projectInstructionEntries(middle),
)
if block := formatPreservedState(plan, edits, tools, skills, instructions); block != "" {
if block := formatPreservedState(plan, task, edits, tools, skills, instructions); block != "" {
summary += "\n\n" + block
}
return summary
}
func capTaskObjective(objective string) string {
objective = strings.TrimSpace(objective)
if len(objective) <= maxTaskObjectiveBytes {
return objective
}
const suffix = "…"
limit := maxTaskObjectiveBytes - len(suffix)
for limit > 0 && !utf8.RuneStart(objective[limit]) {
limit--
}
return strings.TrimSpace(objective[:limit]) + suffix
}
// mergeRecentEdits overlays fresh edits onto edits preserved by an earlier
// compaction. Unlike mergeSkillEntries (which keeps refreshed entries in their
// original slot), a path touched again by a fresh edit MOVES to the newest
@@ -555,11 +586,11 @@ func preservedEditsToEntries(edits []preservedEdit) []skillEntry {
// formatPreservedState renders state as the labelled, single-line
// JSON block. Returns "" when there is nothing to preserve.
func formatPreservedState(plan string, edits, tools, skills, instructions []skillEntry) string {
if plan == "" && len(edits) == 0 && len(tools) == 0 && len(skills) == 0 && len(instructions) == 0 {
func formatPreservedState(plan string, task *preservedTaskState, edits, tools, skills, instructions []skillEntry) string {
if plan == "" && task == nil && len(edits) == 0 && len(tools) == 0 && len(skills) == 0 && len(instructions) == 0 {
return ""
}
state := preservedState{Plan: plan}
state := preservedState{Plan: plan, Task: task}
for _, e := range edits {
state.RecentEdits = append(state.RecentEdits, preservedEdit{Path: e.name, Note: e.body})
}
+101 -5
View File
@@ -1,6 +1,7 @@
package agent
import (
"encoding/json"
"strings"
"testing"
"unicode/utf8"
@@ -59,6 +60,103 @@ func TestCompactPreservesActivePlan(t *testing.T) {
}
}
func TestCompactPreservesBoundedTaskContext(t *testing.T) {
objective := strings.Repeat("世", maxTaskObjectiveBytes)
task := newTaskState(objective, nil)
task.observe(taskStateEvent{kind: taskStateEventPlan, arguments: `{"plan":[{"content":"write code","status":"in_progress"},{"content":"add tests","status":"pending"}]}`})
messages := stateConversation()
compacted, err := Compact(messages, CompactionOptions{
PreserveLast: 2,
Summarize: func([]zeroruntime.Message) (string, error) { return "SUMMARY", nil },
taskState: task.snapshotForCompaction(messages),
})
if err != nil {
t.Fatalf("Compact: %v", err)
}
state := parsePreservedStateBlock(compacted[1].Content)
if state.Task == nil || state.Task.Status != taskStatusActive || state.Task.InProgress != 1 || state.Task.Pending != 1 {
t.Fatalf("unexpected compact task state: %#v", state.Task)
}
if len(state.Task.Objective) > maxTaskObjectiveBytes || !utf8.ValidString(state.Task.Objective) {
t.Fatalf("objective was not safely bounded: %d bytes %q", len(state.Task.Objective), state.Task.Objective)
}
}
func TestCompactPreservesObjectiveAfterPlanParityMismatch(t *testing.T) {
prior := preservedState{Task: &preservedTaskState{Objective: "stale objective", Status: taskStatusActive, Pending: 1}}
encoded, err := json.Marshal(prior)
if err != nil {
t.Fatal(err)
}
messages := []zeroruntime.Message{
{Role: zeroruntime.MessageRoleSystem, Content: "system"},
{Role: zeroruntime.MessageRoleUser, Content: summaryLabel + "\nold\n\n" + preservedStateLabel + "\n" + string(encoded)},
{Role: zeroruntime.MessageRoleAssistant, Content: "continuing"},
{Role: zeroruntime.MessageRoleUser, Content: "more"},
{Role: zeroruntime.MessageRoleAssistant, Content: "working"},
{Role: zeroruntime.MessageRoleUser, Content: "again"},
{Role: zeroruntime.MessageRoleAssistant, Content: "done"},
}
compacted, err := Compact(messages, CompactionOptions{
PreserveLast: 2,
Summarize: func([]zeroruntime.Message) (string, error) { return "SUMMARY", nil },
taskState: &taskStateSnapshot{
Objective: "current objective",
PlanParity: taskPlanParityMismatch,
},
})
if err != nil {
t.Fatalf("Compact: %v", err)
}
state := parsePreservedStateBlock(compacted[1].Content)
if state.Task == nil || state.Task.Objective != "current objective" {
t.Fatalf("immutable objective was lost on plan mismatch: %#v", state.Task)
}
if state.Task.Status != "" || state.Task.Pending != 0 {
t.Fatalf("uncorroborated mutable task fields survived plan mismatch: %#v", state.Task)
}
}
func TestTaskObjectiveSurvivesRepeatedCompactionWithoutPlanRefresh(t *testing.T) {
task := newTaskState("keep this objective", nil)
task.observe(taskStateEvent{kind: taskStateEventPlan, arguments: `{"plan":[{"content":"write code","status":"in_progress"},{"content":"add tests","status":"pending"}]}`})
messages := stateConversation()
first, err := Compact(messages, CompactionOptions{
PreserveLast: 2,
Summarize: func([]zeroruntime.Message) (string, error) { return "FIRST", nil },
taskState: task.snapshotForCompaction(messages),
})
if err != nil {
t.Fatalf("first Compact: %v", err)
}
secondInput := append(append([]zeroruntime.Message{}, first...),
zeroruntime.Message{Role: zeroruntime.MessageRoleUser, Content: "more"},
zeroruntime.Message{Role: zeroruntime.MessageRoleAssistant, Content: "working"},
zeroruntime.Message{Role: zeroruntime.MessageRoleUser, Content: "again"},
zeroruntime.Message{Role: zeroruntime.MessageRoleAssistant, Content: "done"},
)
snapshot := task.snapshotForCompaction(secondInput)
if snapshot.PlanParity != taskPlanParityMismatch {
t.Fatalf("plan call should be absent after first compaction, parity=%q", snapshot.PlanParity)
}
second, err := Compact(secondInput, CompactionOptions{
PreserveLast: 2,
Summarize: func([]zeroruntime.Message) (string, error) { return "SECOND", nil },
taskState: snapshot,
})
if err != nil {
t.Fatalf("second Compact: %v", err)
}
state := parsePreservedStateBlock(second[1].Content)
if state.Task == nil || state.Task.Objective != "keep this objective" {
t.Fatalf("objective lost after second compaction: %#v", state.Task)
}
if state.Task.Status != "" || state.Task.InProgress != 0 {
t.Fatalf("uncorroborated mutable fields survived second compaction: %#v", state.Task)
}
}
func TestCompactPreservesLoadedSkills(t *testing.T) {
summary := compactStateConversation(t, stateConversation())
if !strings.Contains(summary, preservedStateLabel) {
@@ -283,12 +381,10 @@ func TestExtractLatestPlanReturnsMostRecent(t *testing.T) {
}
}
func TestFormatPlanArgumentsAcceptsStepAlias(t *testing.T) {
func TestFormatPlanArgumentsRejectsUnsupportedStepAlias(t *testing.T) {
got := formatPlanArguments(`{"plan":[{"step":"write failing test","status":"in_progress"},{"content":"keep existing shape","status":"pending"}]}`)
for _, want := range []string{"- [in_progress] write failing test", "- [pending] keep existing shape"} {
if !strings.Contains(got, want) {
t.Fatalf("expected %q in formatted plan, got %q", want, got)
}
if got != "" {
t.Fatalf("unsupported alias should reject the whole plan like update_plan, got %q", got)
}
}
+2 -2
View File
@@ -581,7 +581,7 @@ func TestCompactNeverProducesConsecutiveUserMessages(t *testing.T) {
}
func TestRecoverNoopDoesNotConsumeReactiveBudget(t *testing.T) {
st := newCompactionState(Options{ContextWindow: 1000, CompactionPreserveLast: 2})
st := newCompactionState(Options{ContextWindow: 1000, CompactionPreserveLast: 2}, nil)
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{{
{Type: zeroruntime.StreamEventText, Content: "SUMMARY"}, {Type: zeroruntime.StreamEventDone},
}}}
@@ -628,7 +628,7 @@ func TestRecoverNoopDoesNotConsumeReactiveBudget(t *testing.T) {
}
func TestRecoverDisabledIsNoop(t *testing.T) {
st := newCompactionState(Options{ContextWindow: 0})
st := newCompactionState(Options{ContextWindow: 0}, nil)
msgs := []zeroruntime.Message{{Role: zeroruntime.MessageRoleUser, Content: "x"}}
called := false
// recover must not invoke the provider/summarize when disabled, even on a
+86
View File
@@ -0,0 +1,86 @@
package agent
// CompletionDecision is the deterministic outcome of evaluating a text-only
// assistant turn. Uncertain means the loop needs one bounded follow-up action
// before it can finalize the run.
type CompletionDecision string
const (
CompletionUncertain CompletionDecision = "uncertain"
CompletionComplete CompletionDecision = "complete"
CompletionIncomplete CompletionDecision = "incomplete"
)
type completionAction string
const (
completionActionNone completionAction = ""
completionActionContinue completionAction = "continue"
completionActionSemanticCheck completionAction = "semantic_check"
)
type completionEvaluation struct {
Decision CompletionDecision
Action completionAction
Reason string
}
// completionPolicy owns the small amount of state needed to keep uncertain
// decisions bounded. requireSemanticCheck is set only for run profiles that
// already opted into self-correction; ordinary completion remains entirely
// local and adds no model call.
type completionPolicy struct {
continueNudges int
requireSemanticCheck bool
semanticCheckRequested bool
}
func newCompletionPolicy(requireSemanticCheck bool) *completionPolicy {
return &completionPolicy{requireSemanticCheck: requireSemanticCheck}
}
func (policy *completionPolicy) evaluate(text string, context completionContext) completionEvaluation {
// A direct admission is strong local evidence and takes precedence over all
// weaker signals, avoiding both continuation nudges and semantic checks.
if reason := selfReportedIncompletion(text); reason != "" {
return completionEvaluation{Decision: CompletionIncomplete, Reason: reason}
}
// A continuation cue is strong unfinished evidence; a pending plan is only
// weak evidence because bookkeeping can be stale. Both get a bounded chance
// to continue, but only a persisted cue is ultimately incomplete.
cue := endsWithContinuationCue(text)
if cue || context.PlanPending {
if policy.continueNudges < maxContinueNudges {
policy.continueNudges++
reason := "your message ended mid-step"
if !cue {
reason = "pending plan items remain — finish them, or mark them complete with update_plan if you are done"
}
return completionEvaluation{
Decision: CompletionUncertain,
Action: completionActionContinue,
Reason: reason,
}
}
if cue {
return completionEvaluation{
Decision: CompletionIncomplete,
Reason: "your message ended mid-step",
}
}
}
// Profiles with self-correction enabled require one task-grounded semantic
// check. It is requested at most once per run; the next locally complete turn
// is accepted without another model call.
if policy.requireSemanticCheck && !policy.semanticCheckRequested {
policy.semanticCheckRequested = true
return completionEvaluation{
Decision: CompletionUncertain,
Action: completionActionSemanticCheck,
}
}
return completionEvaluation{Decision: CompletionComplete, Action: completionActionNone}
}
+75
View File
@@ -0,0 +1,75 @@
package agent
import (
"strings"
"testing"
)
func TestCompletionPolicyLocalEvidenceDecidesWithoutSemanticCheck(t *testing.T) {
policy := newCompletionPolicy(false)
complete := policy.evaluate("Done. All required checks pass.", completionContext{})
if complete.Decision != CompletionComplete {
t.Fatalf("confident completion decision = %q, want %q", complete.Decision, CompletionComplete)
}
incomplete := policy.evaluate("I couldn't verify the result, so this is my best guess.", completionContext{})
if incomplete.Decision != CompletionIncomplete {
t.Fatalf("admitted failure decision = %q, want %q", incomplete.Decision, CompletionIncomplete)
}
}
func TestCompletionPolicyPreservesBoundedPlanStallProtection(t *testing.T) {
policy := newCompletionPolicy(false)
for attempt := 0; attempt < maxContinueNudges; attempt++ {
got := policy.evaluate("Let me inspect the remaining configuration:", completionContext{PlanPending: true})
if got.Decision != CompletionUncertain || got.Action != completionActionContinue {
t.Fatalf("attempt %d = (%q, %q), want uncertain continue", attempt+1, got.Decision, got.Action)
}
}
got := policy.evaluate("Let me inspect the remaining configuration:", completionContext{PlanPending: true})
if got.Decision != CompletionIncomplete {
t.Fatalf("decision after nudge budget = %q, want %q", got.Decision, CompletionIncomplete)
}
}
func TestCompletionPolicyTreatsPendingPlanAsWeakEvidence(t *testing.T) {
policy := newCompletionPolicy(false)
for attempt := 0; attempt < maxContinueNudges; attempt++ {
got := policy.evaluate("All set.", completionContext{PlanPending: true})
if got.Decision != CompletionUncertain || got.Action != completionActionContinue {
t.Fatalf("attempt %d = (%q, %q), want uncertain continue", attempt+1, got.Decision, got.Action)
}
}
got := policy.evaluate("All set.", completionContext{PlanPending: true})
if got.Decision != CompletionComplete {
t.Fatalf("stale plan decision after nudge budget = %q, want %q", got.Decision, CompletionComplete)
}
}
func TestCompletionPolicyAllowsExactlyOneRequiredSemanticCheck(t *testing.T) {
policy := newCompletionPolicy(true)
first := policy.evaluate("Implemented and tested.", completionContext{})
if first.Decision != CompletionUncertain || first.Action != completionActionSemanticCheck {
t.Fatalf("first decision = (%q, %q), want uncertain semantic check", first.Decision, first.Action)
}
second := policy.evaluate("PASS. The result meets the task criterion.", completionContext{})
if second.Decision != CompletionComplete {
t.Fatalf("post-check decision = %q, want %q", second.Decision, CompletionComplete)
}
if second.Action == completionActionSemanticCheck {
t.Fatal("semantic check was requested more than once")
}
}
func TestAcceptanceVerificationNudgeIncludesBoundedObjective(t *testing.T) {
objective := "make the requested behavior observable"
nudge := acceptanceVerificationNudge(objective)
if !strings.Contains(nudge, "Task objective: "+objective) {
t.Fatalf("semantic check was not grounded in the objective: %q", nudge)
}
}
+11 -13
View File
@@ -1,7 +1,6 @@
package agent
import (
"encoding/json"
"strconv"
"strings"
@@ -127,9 +126,8 @@ func endsWithContinuationCue(text string) bool {
// which the update_plan tool coerces to "pending") counts as remaining, matching
// that tool's own status normalization.
func planStatusRemaining(status string) bool {
switch strings.ToLower(strings.TrimSpace(status)) {
case "completed", "complete", "done", "finished", "resolved", "✓", "x", "[x]",
"failed", "fail", "error", "errored", "blocked", "cancelled", "canceled", "abandoned", "skipped":
switch normalizeTaskPlanStatus(status) {
case "completed", "failed":
return false
default:
return true
@@ -307,8 +305,8 @@ func hasAnyPrefix(s string, prefixes []string) bool {
// the bug-hunt surfaced: well-formed output treated as correct; pre-existing tests
// passing treated as the objective being met; and a result that merely matches a
// baseline the task asked to beat or improve. General — no task-specific content.
func acceptanceVerificationNudge() string {
return "Before this task can be marked complete, " + acceptanceNudgeMarker + " — " +
func acceptanceVerificationNudge(objective string) string {
nudge := "Before this task can be marked complete, " + acceptanceNudgeMarker + " — " +
"NOT the shape or format of your output, NOT that pre-existing tests pass, and NOT that your " +
"result merely matches a baseline you were asked to beat or improve. " +
"Re-read the original task, then run a concrete check that exercises the actual requirement: " +
@@ -316,6 +314,10 @@ func acceptanceVerificationNudge() string {
"thing the task asked you to produce, recover, fix, or optimize. " +
"If that check passes, reply PASS and cite the evidence. " +
"If it does not pass — or you cannot run such a check — say so plainly and keep working; do not claim success."
if objective = capTaskObjective(objective); objective != "" {
nudge += "\n\nTask objective: " + objective
}
return nudge
}
// toolFailureHintMarker is a stable substring for tests.
@@ -553,16 +555,12 @@ func (state *guardState) pendingPlanItems() bool {
// many items are still remaining. Malformed arguments leave the prior count
// unchanged (best-effort — the plan panel itself tolerates the same).
func (state *guardState) observePlanUpdate(arguments string) {
var parsed struct {
Plan []struct {
Status string `json:"status"`
} `json:"plan"`
}
if json.Unmarshal([]byte(arguments), &parsed) != nil {
plan, ok := parseTaskPlan(arguments)
if !ok {
return
}
pending := 0
for _, item := range parsed.Plan {
for _, item := range plan {
if planStatusRemaining(item.Status) {
pending++
}
+223 -81
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"io"
"sort"
"strconv"
"strings"
"sync"
@@ -135,6 +136,33 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
}
}
// Establish the run's turn session — the seam an optimized provider session
// (connection reuse, prewarm, native compaction) plugs into without touching
// this loop. Default: wrap the passed provider so the session's Stream IS
// provider.StreamCompletion, byte-identical to calling it directly. Opened
// before dispatchSessionStart so a failed open is a clean run-start error
// with no session hooks fired and nothing to unwind.
turnSessions := options.TurnSessionProvider
if turnSessions == nil {
turnSessions = zeroruntime.NewProviderTurnSessionProvider(provider, zeroruntime.ProviderCapabilities{})
}
session, sessionErr := turnSessions.OpenTurnSession(ctx)
if sessionErr != nil {
return Result{}, fmt.Errorf("agent: open turn session: %w", sessionErr)
}
// Closed via closure (not `defer session.Close()`) so after a mid-run model
// swap the CURRENT session is the one closed at teardown — the swap closes
// the old one inline. Close errors are advisory and never fail the run.
defer func() { _ = session.Close() }()
// Best-effort prewarm: the default adapter no-ops; a real session may prime
// a connection. Failure is ignored by contract (never fatal).
_ = session.Prewarm(ctx)
// Route ALL provider I/O — per-turn streams, mid-stream retries, the
// compaction summarizer, and the final-answer request — through the session
// by reassigning the loop's provider local. Every existing call site reads
// this local, so no signatures change anywhere downstream.
provider = sessionProvider{session: session}
maxTurns := options.MaxTurns
if maxTurns <= 0 {
maxTurns = 12
@@ -153,10 +181,46 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
options.runPermissions = runPermissions
defer runPermissions.cleanup()
messages := zeroruntime.SeedMessagesWithImages(buildSystemPrompt(options), prompt, options.Images)
promptParts := buildSystemPromptParts(options)
messages := zeroruntime.SeedMessagesWithImages(promptParts.prompt, prompt, options.Images)
guards := newGuardState()
compactor := newCompactionState(options)
task := newTaskState(prompt, options.Trace)
compactor := newCompactionState(options, task)
defer func() {
// A final transcript comparison is observational. It records drift but
// never rewrites the result or changes execution after the fact.
task.observePlanParity(result.Messages)
// Tool-result observations are intentionally coalesced; this final event
// guarantees their aggregate counts are present even on an early return.
task.emit()
}()
// The execution-profile posture controller. nil Options.Profile (every
// existing caller) makes every observe/decide call a no-op, keeping the
// loop byte-identical for unprofiled runs.
posture := newProfileController(options.Profile)
// applyPosture applies at most one posture escalation when an armed trigger
// has fired: raise the hoisted turn ceiling, restore effort and the
// completion gate, stamp the counter. Shared by the end-of-turn tail and
// the completion-gate uncertain path (which continues the loop before the
// tail runs). No-op forever after the first escalation and for unprofiled
// runs.
applyPosture := func() {
if target, fired := posture.maybeEscalate(); fired {
if target.MaxTurns > maxTurns {
maxTurns = target.MaxTurns
}
if target.ReasoningEffort != "" {
options.ReasoningEffort = target.ReasoningEffort
} else if target.RestoreDefaultEffort {
options.ReasoningEffort = ""
}
if target.RestoreCompletionGate {
options.RequireCompletionSignal = true
}
options.Trace.Counter(trace.CounterPostureEscalations, 1)
}
}
// Background post-edit diagnostics: files changed by mutating tools are
// checked off the tool-call critical path and any errors are appended as a
@@ -169,13 +233,10 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// loaded tool's full schema; it lives only for the run (v1 within-run scope).
loaded := map[string]bool{}
// continueNudges counts how many times the headless completion gate
// (Options.RequireCompletionSignal) has re-prompted a no-tool-call turn that
// stopped with work still unfinished. Bounded by maxContinueNudges.
continueNudges := 0
// acceptanceRequested records that the one-time task-grounded acceptance check
// has already been demanded this run, so it fires at most once.
acceptanceRequested := false
// The feature-gated completion policy keeps local completion decisions and its
// bounded follow-up state out of the central loop. Self-correcting profiles
// opt into the one permitted task-grounded semantic check.
completionPolicy := newCompletionPolicy(options.SelfCorrect != nil)
// toolDefCache memoizes each tool's rendered JSON-schema definition across
// turns (a tool's advertised schema is stable for the run), so partitionTools
@@ -185,7 +246,11 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
result = Result{Messages: copyMessages(messages)}
dispatchSessionStart(ctx, options)
defer func() {
dispatchSessionEnd(ctx, options, result, err)
// Use a context detached from ctx's cancellation: on Esc/Ctrl+C the run
// context is already canceled by the time this defer runs, and a canceled
// parent would make Hooks.Dispatch's context.WithTimeout derive an
// already-expired context, so the sessionEnd hook would never actually run.
dispatchSessionEnd(context.WithoutCancel(ctx), options, result, err)
}()
for turn := 0; turn < maxTurns; turn++ {
result.Turns = turn + 1
@@ -213,6 +278,24 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
exposed, _ := partitionToolsCached(registry, permissionMode, options, loaded, toolDefCache)
toolPartitionSpan.End()
// Fingerprint the exact system prompt built at run start plus this turn's
// provider-visible tool definitions in their emitted order. Reusing the
// retained prompt parts avoids rebuilding workspace context while ensuring
// the trace describes the same bytes carried by the request.
if options.Trace != nil {
fp := computePrefixFingerprint(buildPromptSubstringsFromParts(promptParts, exposed))
options.Trace.EmitPrefixHash(trace.PrefixHash{
SystemPromptHash: fp.SystemPromptHash,
BaseInstructionsHash: fp.BaseInstructionsHash,
ConfirmationPolicyHash: fp.ConfirmationPolicyHash,
ProjectContextHash: fp.ProjectContextHash,
SkillsHash: fp.SkillsHash,
ToolsHash: fp.ToolsHash,
SchemaHash: fp.SchemaHash,
CompletePrefixHash: fp.CompletePrefixHash,
})
}
// PROACTIVE compaction: if the history is approaching the model's
// context window, summarize the oldest middle before building the
// request. A no-op when ContextWindow == 0 (compaction disabled).
@@ -496,69 +579,42 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// model's final answer ONLY when the work is actually done. Default off
// (RequireCompletionSignal), so interactive runs stay byte-identical.
if options.RequireCompletionSignal {
// (1) Self-report downgrade (strongest, unambiguous): the model's own
// final message admits it guessed / could not meet the objective. Checked
// FIRST so an admitted-impossible task is downgraded immediately (no wasted
// continue-nudges) and reports the accurate reason.
if reason := selfReportedIncompletion(collected.Text); reason != "" {
completionContext := task.completionContext(messages, guards.pendingPlanItems())
evaluation := completionPolicy.evaluate(collected.Text, completionContext)
task.observe(taskStateEvent{kind: taskStateEventCompletion, completion: evaluation})
switch evaluation.Decision {
case CompletionIncomplete:
result.Incomplete = true
result.IncompleteReason = reason
result.IncompleteReason = evaluation.Reason
result.FinalAnswer = collected.Text
result.Messages = copyMessages(messages)
return result, nil
}
// (2) The model stopped without a tool call while work may be unfinished:
// - a continuation cue ("…Let me check the config:") is an unambiguous
// mid-step stop;
// - pending update_plan items are a WEAK, ambiguous signal (the model may
// have finished without re-marking the last step).
// Nudge to continue (bounded). After the budget: a persisted continuation
// cue finalizes INCOMPLETE; pending-plan WITHOUT a cue does NOT (that would
// false-fail a completed run with stale bookkeeping) — fall through to the
// acceptance check / success.
cue := endsWithContinuationCue(collected.Text)
planPending := guards.pendingPlanItems()
if cue || planPending {
if continueNudges < maxContinueNudges {
continueNudges++
case CompletionUncertain:
// Observe the uncertainty signal and apply any escalation NOW:
// this branch continues the turn loop before the end-of-turn
// act point, so deferring would skip escalation entirely.
posture.observeUncertain()
applyPosture()
switch evaluation.Action {
case completionActionContinue:
options.Trace.Counter(trace.CounterCompletionNudges, 1)
reason := "your message ended mid-step"
if !cue {
reason = "pending plan items remain — finish them, or mark them complete with update_plan if you are done"
}
messages = append(messages, zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: continueNudge(reason),
Content: continueNudge(evaluation.Reason),
})
case completionActionSemanticCheck:
options.Trace.Counter(trace.CounterAcceptanceChecks, 1)
messages = append(messages, zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: acceptanceVerificationNudge(completionContext.Objective),
})
continue
}
if cue {
result.Incomplete = true
result.IncompleteReason = "your message ended mid-step"
result.FinalAnswer = collected.Text
result.Messages = copyMessages(messages)
return result, nil
}
// pending-plan only, budget spent: trust the model's completion claim
// over stale plan bookkeeping; fall through.
}
// (3) Task-grounded acceptance: before accepting a "done" turn as success,
// require ONE acceptance check grounded in the task's stated criterion
// (only when self-correct is on). Rejects "well-formed == correct",
// "existing-tests-pass == objective met", and "result == baseline" false
// successes. Bounded to a single pass; a genuine post-check completion
// (no admission, no cue) then finalizes as success on the next turn.
if options.SelfCorrect != nil && !acceptanceRequested {
acceptanceRequested = true
options.Trace.Counter(trace.CounterAcceptanceChecks, 1)
messages = append(messages, zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: acceptanceVerificationNudge(),
})
continue
case CompletionComplete:
// Local evidence is sufficient; proceed to final diagnostics.
}
} else {
task.observe(taskStateEvent{kind: taskStateEventCompletion, completion: completionEvaluation{Decision: CompletionComplete}})
}
// Finalization diagnostics gate: edits from this run may still have
// checks in flight — the per-turn drain waits only briefly and defers
@@ -582,6 +638,11 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// executing so the empty-turn counter resets and plan-tracking signals
// stay current.
guards.observeTurn(collected)
for _, call := range collected.ToolCalls {
if call.Name == planToolName {
task.observe(taskStateEvent{kind: taskStateEventPlan, arguments: call.Arguments})
}
}
failureHint := ""
// turnRequestedModel records the FIRST mid-run escalation target requested
@@ -608,10 +669,9 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// The scan happens lazily at the run's first index — never ahead of a
// pending mutating call — so read-after-write ordering is preserved.
if index >= precomputedEnd {
runEnd := index
for runEnd < len(collected.ToolCalls) && parallelSafeToolCall(registry, collected.ToolCalls[runEnd], options) {
runEnd++
}
// Capability-safe consecutive run (ReadOnly+ThreadSafe, no
// resource-key conflicts). See extendParallelRun.
runEnd := extendParallelRun(registry, collected.ToolCalls, index, options)
if runEnd-index >= 2 {
batchSpan := options.Trace.Span(trace.SpanToolExecution)
precomputed = executeParallelReadBatch(ctx, registry, collected.ToolCalls, index, runEnd, permissionMode, options)
@@ -633,6 +693,8 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
toolSpan.End()
}
options.Trace.Counter(trace.CounterToolCalls, 1)
recordOutputBudgetTrace(options.Trace, toolResult)
task.observe(taskStateEvent{kind: taskStateEventToolResult, toolResult: toolResult})
if options.OnToolResult != nil {
options.OnToolResult(toolResult)
}
@@ -679,6 +741,7 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// would misdirect the model toward JSON shape or blocked behavior.
retriableFailure := isRetriableToolError(toolResult)
outcome := guards.observeToolResult(call.Name, retriableFailure, toolResult.Output)
posture.observeToolOutcome(outcome, toolResult)
if outcome.Stop {
// The assistant message advertised EVERY collected tool call, but
// the guard halts mid-turn so the calls after this one never run.
@@ -711,7 +774,10 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// the assistant's tool_results stay contiguous (a user message between
// tool_results breaks strict provider replay). nil SelfCorrect is a no-op.
if options.SelfCorrect != nil && len(changedFilesThisBatch) > 0 {
if feedback, _ := options.SelfCorrect.AfterEdit(ctx, dedupeStrings(changedFilesThisBatch)); feedback != "" {
feedback, selfCorrectOutcome := options.SelfCorrect.AfterEdit(ctx, dedupeStrings(changedFilesThisBatch))
posture.observeSelfCorrect(selfCorrectOutcome)
task.observe(taskStateEvent{kind: taskStateEventVerification, verification: selfCorrectOutcome})
if feedback != "" {
messages = append(messages, zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: feedback,
@@ -724,25 +790,53 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// of the run. At most one switch per turn (first request wins). A switcher
// error is NON-FATAL — record a brief note and continue on the current
// model. nil switcher ⇒ requests are ignored entirely (escalation off).
if turnRequestedModel != "" && options.ModelSwitcher != nil {
newProvider, switchErr := options.ModelSwitcher(ctx, turnRequestedModel)
if turnRequestedModel != "" && (options.ModelSessionSwitcher != nil || options.ModelSwitcher != nil) {
// Resolve the escalated model's session source. The target-aware
// ModelSessionSwitcher is preferred: its TurnSessionProvider carries an
// optimized session (and capabilities) across the swap. The legacy
// ModelSwitcher fallback returns a bare Provider, wrapped in the
// default no-op session — identical to pre-seam behavior.
var newSessions zeroruntime.TurnSessionProvider
var switchErr error
if options.ModelSessionSwitcher != nil {
newSessions, switchErr = options.ModelSessionSwitcher(ctx, turnRequestedModel)
} else {
var newProvider Provider
newProvider, switchErr = options.ModelSwitcher(ctx, turnRequestedModel)
if switchErr == nil && newProvider != nil {
newSessions = zeroruntime.NewProviderTurnSessionProvider(newProvider, zeroruntime.ProviderCapabilities{})
}
}
if switchErr != nil {
messages = append(messages, zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: escalationFailedNoticePrefix + " (" + turnRequestedModel + "): " + switchErr.Error() + ". Continuing on " + options.Model + ".",
})
} else if newProvider != nil {
// Reassign the local provider so the next turn's StreamCompletion and
// compaction use it; update options.Model so subsequent RunOptions.Model,
// context-window sizing, and usage attribution follow the new model.
provider = newProvider
options.Model = turnRequestedModel
options.Trace.Counter(trace.CounterModelSwitches, 1)
// KNOWN LIMITATION (deferred): the compactor's context-window budget
// is fixed at run start from options.ContextWindow and is NOT updated
// here, so a switch to a model with a different window keeps compacting
// against the original budget. Fixing it needs a ModelSwitcher contract
// change (return the new window) — out of scope for this change.
} else if newSessions != nil {
// Open the new session and close the old one, so the next turn's
// streams and compaction use the new model. For the default adapter
// the open never errors and Close is a no-op, so this stays
// byte-identical to a direct provider reassignment; the guarded open
// and prewarm matter once a session holds real state.
newSession, openErr := newSessions.OpenTurnSession(ctx)
if openErr != nil {
messages = append(messages, zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: escalationFailedNoticePrefix + " (" + turnRequestedModel + "): " + openErr.Error() + ". Continuing on " + options.Model + ".",
})
} else {
_ = session.Close()
session = newSession
_ = session.Prewarm(ctx)
provider = sessionProvider{session: session}
options.Model = turnRequestedModel
options.Trace.Counter(trace.CounterModelSwitches, 1)
// KNOWN LIMITATION (deferred): the compactor's context-window budget
// is fixed at run start from options.ContextWindow and is NOT updated
// here, so a switch to a model with a different window keeps compacting
// against the original budget. Fixing it needs the switcher to also
// report the new window — out of scope for this change.
}
}
}
@@ -776,6 +870,13 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
Content: reminder,
})
}
// One-shot posture escalation: when an armed profile trigger fired this
// turn, restore the stricter knob values the profile displaced. Mutates
// only per-turn-read policy (the hoisted turn ceiling, reasoning effort,
// completion gate); never messages, model, or session. No-op forever
// after the first escalation and for every unprofiled run.
applyPosture()
}
if ctx.Err() != nil {
@@ -818,6 +919,28 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
return result, nil
}
func recordOutputBudgetTrace(recorder *trace.Recorder, result ToolResult) {
if recorder == nil || result.Meta["output_budget_category"] == "" {
return
}
parseInt := func(key string) int {
value, _ := strconv.Atoi(result.Meta[key])
return value
}
spillCreated, _ := strconv.ParseBool(result.Meta["output_budget_spill_created"])
recorder.EmitOutputBudget(trace.OutputBudgetEvent{
Tool: result.Name,
Category: result.Meta["output_budget_category"],
OriginalBytes: parseInt("output_budget_original_bytes"),
RetainedBytes: parseInt("output_budget_retained_bytes"),
EstimatedOriginalTokens: parseInt("output_budget_estimated_original_tokens"),
EstimatedRetainedTokens: parseInt("output_budget_estimated_retained_tokens"),
Truncated: result.Truncated,
Reason: result.Meta["output_budget_reason"],
SpillCreated: spillCreated,
})
}
func finalAnswerAfterMaxTurns(ctx context.Context, provider Provider, messages []zeroruntime.Message, toolDefs []zeroruntime.ToolDefinition, options Options) (string, []zeroruntime.Message, string) {
finalMessages := copyMessages(messages)
finalMessages = append(finalMessages, zeroruntime.Message{
@@ -1272,16 +1395,33 @@ func executeToolCall(ctx context.Context, registry *tools.Registry, call ToolCal
if didRedact {
result.Redacted = true
}
result = registry.RebudgetAfterHook(call.Name, args, result)
}
}
// Stamp the sandbox risk classification for this EXECUTED call so run-policy
// observers can see the risk of an allowed mutation. Prefer the preflight
// decision's classification when the sandbox evaluated the call; otherwise
// run the same pure classifier the permission path uses. Denied/canceled
// results returned earlier keep the zero value.
executedRisk := sandbox.Risk{}
if preflightDecision != nil {
executedRisk = preflightDecision.Risk
} else if toolFound {
// Unknown-tool results fall through here with a nil tool; they carry a
// denial and keep the zero risk value.
executedRisk = sandbox.Classify(sandboxRequest(call.Name, tool, args, permissionGranted, permissionMode, options))
}
// Secret scrubbing happens at the registry boundary (the single point both
// the agent loop and the MCP server pass through), so result.Output is
// already redacted here and result.Redacted reflects whether it changed.
return ToolResult{
Risk: executedRisk,
ToolCallID: call.ID,
Name: call.Name,
Status: result.Status,
Output: result.Output,
Truncated: result.Truncated,
Meta: result.Meta,
Redacted: result.Redacted,
ChangedFiles: result.ChangedFiles,
@@ -1592,6 +1732,7 @@ func toolResultFromPrePermissionReject(call ToolCall, result tools.Result) ToolR
Name: call.Name,
Status: result.Status,
Output: output,
Truncated: result.Truncated,
Meta: meta,
Redacted: result.Redacted || outputRedacted || summaryRedacted || metaRedacted,
ChangedFiles: result.ChangedFiles,
@@ -1861,6 +2002,7 @@ func askUserFallbackResult(ctx context.Context, registry *tools.Registry, call T
Name: call.Name,
Status: result.Status,
Output: result.Output,
Truncated: result.Truncated,
Meta: result.Meta,
Redacted: result.Redacted,
ChangedFiles: result.ChangedFiles,
+123
View File
@@ -6,7 +6,9 @@ import (
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"reflect"
"runtime"
"strings"
"testing"
@@ -93,6 +95,87 @@ func TestRunDispatchesSessionLifecycleHooks(t *testing.T) {
}
}
// cancelingProvider simulates an Esc/Ctrl+C abort: it cancels the run's own
// context mid-stream (as the TUI's interrupt handler would) and returns no
// events, so Run observes ctx.Err() and returns early with an already-canceled
// context in scope for the deferred sessionEnd dispatch.
type cancelingProvider struct {
cancel context.CancelFunc
}
func (provider *cancelingProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
provider.cancel()
ch := make(chan zeroruntime.StreamEvent)
close(ch)
return ch, nil
}
// TestRunDispatchesSessionEndHookAfterContextCancellation is a regression test
// for a bug where an interrupted run (Esc/Ctrl+C) canceled ctx before the
// deferred sessionEnd dispatch ran, so Hooks.Dispatch's context.WithTimeout(ctx, ...)
// derived an already-canceled context and the hook command never actually launched.
func TestRunDispatchesSessionEndHookAfterContextCancellation(t *testing.T) {
goBinary, err := exec.LookPath("go")
if err != nil {
// The test binary runs on the build machine, so its build-time GOROOT
// still identifies the toolchain that produced it.
goRoot := runtime.GOROOT() //nolint:staticcheck // Safe for this non-portable test binary.
goBinary = filepath.Join(goRoot, "bin", "go")
if runtime.GOOS == "windows" {
goBinary += ".exe"
}
if _, statErr := os.Stat(goBinary); statErr != nil {
t.Skipf("go binary unavailable on PATH or in GOROOT: %v", statErr)
}
}
audit, err := hooks.NewAuditStore(hooks.AuditStoreOptions{AuditPath: filepath.Join(t.TempDir(), "audit.jsonl")})
if err != nil {
t.Fatalf("NewAuditStore: %v", err)
}
dispatcher := hooks.NewDispatcher(hooks.DispatcherOptions{
Config: hooks.Config{
Enabled: true,
Hooks: []hooks.Definition{
{ID: "zero.session-end", Event: hooks.EventSessionEnd, Command: goBinary, Args: []string{"version"}, Enabled: true},
},
},
Audit: audit,
})
ctx, cancel := context.WithCancel(context.Background())
provider := &cancelingProvider{cancel: cancel}
_, err = Run(ctx, "hi", provider, Options{
SessionID: "session-cancel",
Cwd: t.TempDir(),
ProviderName: "test-provider",
Model: "test-model",
Hooks: dispatcher,
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("Run err = %v, want context.Canceled", err)
}
events, err := audit.ReadEvents()
if err != nil {
t.Fatalf("ReadEvents: %v", err)
}
var completedStatus hooks.AuditStatus
found := false
for _, event := range events {
if event.Type == "hook_execution_completed" && event.Event == hooks.EventSessionEnd {
completedStatus = event.Status
found = true
}
}
if !found {
t.Fatalf("no sessionEnd hook_execution_completed audit event, events=%#v", events)
}
if completedStatus != hooks.AuditCompleted {
t.Fatalf("sessionEnd hook status = %q, want %q (hook must actually run despite ctx cancellation)", completedStatus, hooks.AuditCompleted)
}
}
type recordingWebSearchTool struct {
calls []map[string]any
}
@@ -1176,6 +1259,46 @@ func TestRunExecutesToolCallThroughRegistry(t *testing.T) {
}
}
func TestRunPreservesRequestPrefixAcrossTurns(t *testing.T) {
root := t.TempDir()
writeAgentTestFile(t, filepath.Join(root, "notes.txt"), "alpha\n")
registry := tools.NewRegistry()
registry.Register(tools.NewReadFileTool(root))
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
{
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call-1", ToolName: "read_file"},
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call-1", ArgumentsFragment: `{"path":"notes.txt"}`},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call-1"},
{Type: zeroruntime.StreamEventDone},
},
{
{Type: zeroruntime.StreamEventText, Content: "done"},
{Type: zeroruntime.StreamEventDone},
},
}}
if _, err := Run(context.Background(), "read notes", provider, Options{
Cwd: root,
Registry: registry,
SessionID: "session-stable-prefix",
}); err != nil {
t.Fatal(err)
}
if len(provider.requests) != 2 {
t.Fatalf("provider requests = %d, want 2", len(provider.requests))
}
first, second := provider.requests[0], provider.requests[1]
if len(second.Messages) < len(first.Messages) || !reflect.DeepEqual(second.Messages[:len(first.Messages)], first.Messages) {
t.Fatalf("first request messages must be an exact prefix of the second:\nfirst=%#v\nsecond=%#v", first.Messages, second.Messages)
}
if !reflect.DeepEqual(first.Tools, second.Tools) {
t.Fatalf("tool definitions drifted between turns:\nfirst=%#v\nsecond=%#v", first.Tools, second.Tools)
}
if first.PromptCacheKey != "session-stable-prefix" || second.PromptCacheKey != first.PromptCacheKey {
t.Fatalf("prompt cache key must remain stable: first=%q second=%q", first.PromptCacheKey, second.PromptCacheKey)
}
}
func TestRunSanitizesMalformedToolCallArgumentsBeforeRetry(t *testing.T) {
root := t.TempDir()
registry := tools.NewRegistry()
@@ -0,0 +1,167 @@
package agent
import (
"context"
"os"
"strconv"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/hooks"
"github.com/Gitlawb/zero/internal/tools"
"github.com/Gitlawb/zero/internal/trace"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
type propagationOutputTool struct {
output string
}
func TestOutputBudgetHookHelperProcess(t *testing.T) {
for index, arg := range os.Args {
if arg != "--zero-output-budget-hook" || index+1 >= len(os.Args) {
continue
}
if _, err := os.Stdout.WriteString(os.Args[index+1]); err != nil {
os.Exit(2)
}
os.Exit(0)
}
}
func largeOutputBudgetHookDispatcher() *hooks.Dispatcher {
feedback := strings.Repeat("hook feedback ", 200)
return hooks.NewDispatcher(hooks.DispatcherOptions{Config: hooks.Config{
Enabled: true,
Hooks: []hooks.Definition{{
ID: "large-feedback",
Event: hooks.EventAfterTool,
Matcher: "propagation_output",
Command: os.Args[0],
Args: []string{
"-test.run=TestOutputBudgetHookHelperProcess",
"--",
"--zero-output-budget-hook",
feedback,
},
Enabled: true,
}},
}})
}
func (tool propagationOutputTool) Name() string { return "propagation_output" }
func (tool propagationOutputTool) Description() string { return "returns output for propagation tests" }
func (tool propagationOutputTool) Parameters() tools.Schema {
return tools.Schema{Type: "object", AdditionalProperties: false}
}
func (tool propagationOutputTool) Safety() tools.Safety {
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow, Reason: "test read"}
}
func (tool propagationOutputTool) Run(context.Context, map[string]any) tools.Result {
return tools.Result{Status: tools.StatusOK, Output: tool.output}
}
func TestExecuteToolCallPropagatesOutputTruncation(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
t.Setenv("ZERO_TOOL_OUTPUT_CEILING_TOKENS", "80")
registry := tools.NewRegistry()
registry.Register(propagationOutputTool{output: strings.Repeat("large output\n", 1000)})
result, abortErr := executeToolCall(context.Background(), registry, ToolCall{
ID: "call-budget",
Name: "propagation_output",
Arguments: `{}`,
}, PermissionModeAuto, Options{Cwd: t.TempDir()})
if abortErr != nil {
t.Fatalf("executeToolCall abort error: %v", abortErr)
}
if !result.Truncated {
t.Fatalf("agent ToolResult lost tools.Result.Truncated: %#v", result)
}
if result.Meta["spill_path"] == "" {
t.Fatalf("agent ToolResult lost spill metadata: %#v", result.Meta)
}
}
func TestRecordOutputBudgetTraceUsesOnlyCompactMetadata(t *testing.T) {
recorder := trace.NewRecorder("session", "run", "")
recorder.Start()
recordOutputBudgetTrace(recorder, ToolResult{
Name: "grep",
Truncated: true,
Output: "SECRET OUTPUT MUST NOT ENTER TRACE",
Meta: map[string]string{
"output_budget_category": "search",
"output_budget_original_bytes": "1000",
"output_budget_retained_bytes": "100",
"output_budget_estimated_original_tokens": "250",
"output_budget_estimated_retained_tokens": "25",
"output_budget_reason": "semantic_search_budget",
"output_budget_spill_created": "true",
"spill_path": "/secret/path/not-for-trace",
},
})
events := recorder.Finish().OutputBudgets
if len(events) != 1 {
t.Fatalf("events = %#v", events)
}
event := events[0]
if event.Tool != "grep" || event.Category != "search" || event.OriginalBytes != 1000 || event.RetainedBytes != 100 || !event.SpillCreated {
t.Fatalf("unexpected trace event: %#v", event)
}
}
func TestExecuteToolCallRebudgetsOversizedAfterToolFeedback(t *testing.T) {
t.Setenv("ZERO_TOOL_OUTPUT_CEILING_TOKENS", "80")
registry := tools.NewRegistry()
registry.Register(propagationOutputTool{output: "tool output"})
dispatcher := largeOutputBudgetHookDispatcher()
result, abortErr := executeToolCall(context.Background(), registry, ToolCall{
ID: "call-hook-budget",
Name: "propagation_output",
Arguments: `{}`,
}, PermissionModeAuto, Options{Hooks: dispatcher})
if abortErr != nil {
t.Fatalf("executeToolCall abort error: %v", abortErr)
}
if !result.Truncated || len(result.Output) > 80*4 {
t.Fatalf("afterTool feedback bypassed output budget: truncated=%t bytes=%d meta=%#v", result.Truncated, len(result.Output), result.Meta)
}
if result.Meta["output_budget_category"] == "" || result.Meta["output_budget_retained_bytes"] != strconv.Itoa(len(result.Output)) {
t.Fatalf("post-hook budget metadata does not describe final output: %#v", result.Meta)
}
}
func TestRunTraceReflectsPostHookBudget(t *testing.T) {
t.Setenv("ZERO_TOOL_OUTPUT_CEILING_TOKENS", "80")
registry := tools.NewRegistry()
registry.Register(propagationOutputTool{output: "tool output"})
dispatcher := largeOutputBudgetHookDispatcher()
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
{
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call-hook-trace", ToolName: "propagation_output"},
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call-hook-trace", ArgumentsFragment: `{}`},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call-hook-trace"},
{Type: zeroruntime.StreamEventDone},
},
{{Type: zeroruntime.StreamEventText, Content: "done"}, {Type: zeroruntime.StreamEventDone}},
}}
recorder := trace.NewRecorder("session", "run", "")
var toolResults []ToolResult
if _, err := Run(context.Background(), "budget hook", provider, Options{
Registry: registry,
Hooks: dispatcher,
Trace: recorder,
OnToolResult: func(result ToolResult) { toolResults = append(toolResults, result) },
}); err != nil {
t.Fatalf("Run: %v", err)
}
if len(toolResults) != 1 || !toolResults[0].Truncated {
t.Fatalf("tool result = %#v, want one truncated post-hook result", toolResults)
}
events := recorder.Finish().OutputBudgets
if len(events) != 1 || !events[0].Truncated || events[0].RetainedBytes != len(toolResults[0].Output) {
t.Fatalf("trace does not describe final post-hook output: events=%#v result=%#v", events, toolResults[0])
}
}
+111 -11
View File
@@ -10,12 +10,23 @@ import (
// Parallel read-ahead for tool batches. When a turn requests several
// independent lookups (read_file + grep + glob is the common shape), executing
// them one after another serializes pure I/O waits. A consecutive run of
// auto-allowed read-only calls is executed concurrently instead; results are
// capability-safe read-only calls is executed concurrently instead; results are
// then consumed in the original call order, so guard counters, message
// ordering, abort semantics, and the surface's call/result event pairing are
// byte-identical to sequential execution. Runs never span a mutating call: a
// read that follows a write must observe the write, so eligibility is decided
// per consecutive run, not per batch.
//
// Eligibility uses the PR5 tool-effect contract (tools.CapabilitiesOf):
//
// Effect == ReadOnly
// AND ThreadSafe == true
// AND auto-allowed (no interactive permission prompt on the hot path)
// AND no resource-key conflict with an earlier call in the same concurrent window
//
// Unknown, mutators, interactive tools, and non-thread-safe reads stay
// sequential. Empty resource keys do not conflict (ThreadSafe is the safety
// gate); shared non-empty keys force a batch boundary.
// maxParallelReadTools bounds concurrent read-only tool executions in a turn.
const maxParallelReadTools = 8
@@ -28,28 +39,117 @@ type precomputedToolResult struct {
}
// parallelSafeToolCall reports whether call may run concurrently with its
// neighbors: the tool must exist, be side-effect-free (SideEffectRead), and be
// auto-allowed for these args, so no interactive prompt or workspace mutation
// is on the hot path. Loop-intercepted tools (ask_user, request_permissions)
// and tool_search (mutates the deferred-tool set) stay sequential.
// neighbors under the PR5 capability contract. Loop-intercepted tools
// (ask_user, request_permissions) and tool_search (mutates the deferred-tool
// set) always stay sequential.
func parallelSafeToolCall(registry *tools.Registry, call ToolCall, options Options) bool {
switch call.Name {
case "ask_user", tools.RequestPermissionsToolName, tools.ToolSearchToolName:
return false
}
tool, found := registry.Get(call.Name)
if !found || tool.Safety().SideEffect != tools.SideEffectRead {
if !found {
return false
}
args := map[string]any{}
if call.Arguments != "" {
if err := decodeToolArguments(call.Arguments, &args); err != nil {
return false
}
caps := tools.CapabilitiesOf(tool)
// Fail-closed: only audited concurrent-safe pure reads.
if caps.Effect != tools.EffectReadOnly || !caps.ThreadSafe {
return false
}
args, ok := decodeCallArgs(call)
if !ok {
return false
}
return effectivePermission(tool, args) == tools.PermissionAllow
}
// decodeCallArgs decodes tool call JSON arguments. Returns false on malformed
// input so the call stays sequential (never panics into the parallel path).
func decodeCallArgs(call ToolCall) (map[string]any, bool) {
args := map[string]any{}
if call.Arguments == "" {
return args, true
}
if err := decodeToolArguments(call.Arguments, &args); err != nil {
return nil, false
}
return args, true
}
// resourceKeysForCall returns the conflict keys for a call, or nil when the
// tool has no ResourceKeys function / no keys for these args.
func resourceKeysForCall(registry *tools.Registry, call ToolCall) []string {
tool, found := registry.Get(call.Name)
if !found {
return nil
}
caps := tools.CapabilitiesOf(tool)
if caps.ResourceKeys == nil {
return nil
}
args, ok := decodeCallArgs(call)
if !ok {
return nil
}
return caps.ResourceKeys(args)
}
// resourceKeysConflict reports whether two key sets share any non-empty key.
// Empty key sets never conflict: ThreadSafe is the eligibility gate; keys only
// refine conflict detection among tools that declare specific resources.
func resourceKeysConflict(a, b []string) bool {
if len(a) == 0 || len(b) == 0 {
return false
}
seen := make(map[string]struct{}, len(a))
for _, key := range a {
if key == "" {
continue
}
seen[key] = struct{}{}
}
for _, key := range b {
if key == "" {
continue
}
if _, ok := seen[key]; ok {
return true
}
}
return false
}
// extendParallelRun returns the exclusive end index of a consecutive parallel-
// safe run starting at start. The run stops before the first call that is not
// parallel-safe or that conflicts on resource keys with any earlier call in
// the same window (so two read_file calls on the same path stay sequential).
func extendParallelRun(registry *tools.Registry, calls []ToolCall, start int, options Options) int {
if start >= len(calls) || !parallelSafeToolCall(registry, calls[start], options) {
return start
}
end := start + 1
keysWindow := [][]string{resourceKeysForCall(registry, calls[start])}
for end < len(calls) {
if !parallelSafeToolCall(registry, calls[end], options) {
break
}
nextKeys := resourceKeysForCall(registry, calls[end])
conflict := false
for _, prev := range keysWindow {
if resourceKeysConflict(prev, nextKeys) {
conflict = true
break
}
}
if conflict {
break
}
keysWindow = append(keysWindow, nextKeys)
end++
}
return end
}
// executeParallelReadBatch runs calls[start:end] concurrently (bounded by
// maxParallelReadTools) and returns results indexed relative to start. All
// execution-side callbacks that can fire inside executeToolCall are serialized
+104
View File
@@ -57,6 +57,20 @@ func (tool *probeTool) Parameters() tools.Schema {
func (tool *probeTool) Safety() tools.Safety {
return tools.Safety{SideEffect: tool.sideEffect, Permission: tools.PermissionAllow, Reason: "test"}
}
// Capabilities maps the probe's SideEffect into the PR5 capability contract so
// parallelSafeToolCall (CapabilitiesOf) can classify reads vs mutators. Without
// this, probes default to EffectUnknown and never enter the concurrent path.
func (tool *probeTool) Capabilities() tools.ToolCapabilities {
switch tool.sideEffect {
case tools.SideEffectRead:
return tools.ToolCapabilities{Effect: tools.EffectReadOnly, ThreadSafe: true}
case tools.SideEffectWrite:
return tools.ToolCapabilities{Effect: tools.EffectWorkspaceWrite, ThreadSafe: false}
default:
return tools.UnknownCapabilities()
}
}
func (tool *probeTool) Run(_ context.Context, args map[string]any) tools.Result {
id, _ := args["id"].(string)
tool.mu.Lock()
@@ -92,6 +106,11 @@ func TestParallelSafeToolCall(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(&probeTool{name: "probe_read", sideEffect: tools.SideEffectRead})
registry.Register(&probeTool{name: "probe_write", sideEffect: tools.SideEffectWrite})
// Read-only but not ThreadSafe: must stay sequential under the PR5 gate.
registry.Register(&keyedProbeTool{
probeTool: probeTool{name: "probe_read_serial", sideEffect: tools.SideEffectRead},
threadSafe: false,
})
call := func(name, args string) ToolCall { return ToolCall{ID: "c", Name: name, Arguments: args} }
if !parallelSafeToolCall(registry, call("probe_read", `{"id":"a"}`), Options{}) {
@@ -109,6 +128,91 @@ func TestParallelSafeToolCall(t *testing.T) {
if parallelSafeToolCall(registry, call("ask_user", `{}`), Options{}) {
t.Fatal("loop-intercepted tools must stay sequential")
}
if parallelSafeToolCall(registry, call("probe_read_serial", `{"id":"a"}`), Options{}) {
t.Fatal("ReadOnly without ThreadSafe must not be parallel-safe")
}
}
func TestResourceKeysConflict(t *testing.T) {
if resourceKeysConflict(nil, []string{"file:a"}) {
t.Fatal("empty vs non-empty must not conflict")
}
if resourceKeysConflict([]string{"file:a"}, nil) {
t.Fatal("non-empty vs empty must not conflict")
}
if resourceKeysConflict([]string{"file:a"}, []string{"file:b"}) {
t.Fatal("distinct keys must not conflict")
}
if !resourceKeysConflict([]string{"file:a", "file:b"}, []string{"file:b"}) {
t.Fatal("shared key must conflict")
}
if resourceKeysConflict([]string{"", "file:a"}, []string{""}) {
t.Fatal("empty-string keys must be ignored")
}
}
func TestExtendParallelRunResourceKeyBoundary(t *testing.T) {
// Two keyed reads on the same path must not share a concurrent window;
// distinct paths may batch together.
registry := tools.NewRegistry()
registry.Register(&keyedProbeTool{
probeTool: probeTool{name: "keyed_read", sideEffect: tools.SideEffectRead},
threadSafe: true,
keys: func(args map[string]any) []string {
id, _ := args["id"].(string)
if id == "" {
return nil
}
return []string{"file:" + id}
},
})
calls := []ToolCall{
{ID: "1", Name: "keyed_read", Arguments: `{"id":"a"}`},
{ID: "2", Name: "keyed_read", Arguments: `{"id":"b"}`},
{ID: "3", Name: "keyed_read", Arguments: `{"id":"a"}`}, // conflicts with call 0
}
// start=0: a and b share no key → run covers [0,2)
if end := extendParallelRun(registry, calls, 0, Options{}); end != 2 {
t.Fatalf("extend from 0 = %d, want 2 (stop before second file:a)", end)
}
// start=2: single remaining call
if end := extendParallelRun(registry, calls, 2, Options{}); end != 3 {
t.Fatalf("extend from 2 = %d, want 3", end)
}
// Empty keys never conflict: three keyless safe reads form one window.
registry.Register(&probeTool{name: "keyless_read", sideEffect: tools.SideEffectRead})
keyless := []ToolCall{
{ID: "1", Name: "keyless_read", Arguments: `{"id":"x"}`},
{ID: "2", Name: "keyless_read", Arguments: `{"id":"y"}`},
{ID: "3", Name: "keyless_read", Arguments: `{"id":"z"}`},
}
if end := extendParallelRun(registry, keyless, 0, Options{}); end != 3 {
t.Fatalf("keyless extend = %d, want 3", end)
}
}
// keyedProbeTool is a probe with explicit ResourceKeys / ThreadSafe for
// planner unit tests.
type keyedProbeTool struct {
probeTool
threadSafe bool
keys func(args map[string]any) []string
}
func (tool *keyedProbeTool) Capabilities() tools.ToolCapabilities {
caps := tools.ToolCapabilities{
Effect: tools.EffectReadOnly,
ThreadSafe: tool.threadSafe,
}
if tool.keys != nil {
caps.ResourceKeys = tool.keys
}
// Honor mutator side effects from the embedded probe when used as write.
if tool.sideEffect == tools.SideEffectWrite {
caps.Effect = tools.EffectWorkspaceWrite
caps.ThreadSafe = false
}
return caps
}
func TestRunExecutesConsecutiveReadsConcurrently(t *testing.T) {
+118
View File
@@ -0,0 +1,118 @@
package agent
import (
"github.com/Gitlawb/zero/internal/sandbox"
)
// profileController observes per-turn run signals and decides at most one
// posture escalation per run for the armed execution profile. A nil policy (or
// a policy without Escalate) makes every method a no-op, so runs without a
// profile are byte-identical to before — the same opt-in convention as Trace
// and SelfCorrect.
//
// The controller only ever CHANGES loop policy knobs (turn ceiling, reasoning
// effort, completion gate) through the values the loop applies at its act
// point; it never injects messages, never touches the model or session, and
// never interacts with the model-initiated escalate_model path.
type profileController struct {
policy *PostureEscalation
escalated bool
uncertainSeen int
failureTripped bool
riskTripped bool
scTripped bool
uncertainTrip bool
}
// newProfileController is nil-safe: a nil ProfilePolicy (the default for every
// existing caller) or a nil Escalate yields a controller that observes and
// decides nothing.
func newProfileController(policy *ProfilePolicy) *profileController {
if policy == nil {
return &profileController{}
}
return &profileController{policy: policy.Escalate}
}
// observeToolOutcome watches two signals from an executed tool call: the
// repeated-failure guard's streak (reusing the guard's own counting, not a
// second failure model) and the sandbox risk classification of an executed
// result.
func (c *profileController) observeToolOutcome(outcome toolFailureOutcome, result ToolResult) {
if c.policy == nil || c.escalated {
return
}
if c.policy.OnToolFailureStreak > 0 && outcome.Count >= c.policy.OnToolFailureStreak {
c.failureTripped = true
}
if threshold := riskRank(c.policy.OnRiskyMutation); threshold > 0 &&
result.DenialReason == DenialNone &&
riskRank(result.Risk.Level) >= threshold {
// The call EXECUTED (was not denied): partial failures count too, since
// the mutation ran. An unrecognized threshold ranks 0 and disables the
// signal instead of matching everything.
c.riskTripped = true
}
}
// observeUncertain counts uncertain completion evaluations (continue nudges and
// semantic checks). Headless-only by construction: the completion gate never
// runs interactively.
func (c *profileController) observeUncertain() {
if c.policy == nil || c.escalated || c.policy.OnCompletionUncertain <= 0 {
return
}
c.uncertainSeen++
if c.uncertainSeen >= c.policy.OnCompletionUncertain {
c.uncertainTrip = true
}
}
// observeSelfCorrect watches the post-edit verification outcome. Any failing
// outcome (correcting, reported, aborted) is a signal; passed and disabled are
// not.
func (c *profileController) observeSelfCorrect(outcome Outcome) {
if c.policy == nil || c.escalated || !c.policy.OnSelfCorrectFailure {
return
}
switch outcome {
case OutcomeCorrecting, OutcomeReported, OutcomeAborted:
c.scTripped = true
}
}
// maybeEscalate reports the escalation target exactly once, the first time any
// armed trigger has fired. Subsequent calls return false for the rest of the
// run: escalation is one-shot and never de-escalates, so no cooldown state is
// needed.
func (c *profileController) maybeEscalate() (PostureEscalation, bool) {
if c.policy == nil || c.escalated {
return PostureEscalation{}, false
}
if !(c.failureTripped || c.riskTripped || c.scTripped || c.uncertainTrip) {
return PostureEscalation{}, false
}
c.escalated = true
return *c.policy, true
}
// riskRank orders sandbox risk levels for threshold comparison. Unknown levels
// rank 0: an unrecognized RESULT level can never meet a valid threshold, and an
// unrecognized THRESHOLD is rejected before comparison (rank 0 disables the
// signal) so a typo in a profile can never make every result match.
func riskRank(level sandbox.RiskLevel) int {
switch level {
case sandbox.RiskLow:
return 1
case sandbox.RiskMedium:
return 2
case sandbox.RiskHigh:
return 3
case sandbox.RiskCritical:
return 4
default:
return 0
}
}
+345
View File
@@ -0,0 +1,345 @@
package agent
import (
"context"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/sandbox"
"github.com/Gitlawb/zero/internal/tools"
"github.com/Gitlawb/zero/internal/trace"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
func TestProfileControllerNilPolicyIsNoOp(t *testing.T) {
for _, controller := range []*profileController{
newProfileController(nil),
newProfileController(&ProfilePolicy{Name: "balanced"}), // Escalate nil
} {
controller.observeToolOutcome(toolFailureOutcome{Count: 99}, ToolResult{Status: tools.StatusOK, Risk: sandbox.Risk{Level: sandbox.RiskCritical}})
controller.observeUncertain()
controller.observeSelfCorrect(OutcomeAborted)
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("nil policy must never escalate")
}
}
}
func TestProfileControllerEscalatesOnFailureStreak(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnToolFailureStreak: 2, MaxTurns: 80}})
controller.observeToolOutcome(toolFailureOutcome{Count: 1}, ToolResult{Status: tools.StatusError})
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("streak below threshold must not escalate")
}
controller.observeToolOutcome(toolFailureOutcome{Count: 2}, ToolResult{Status: tools.StatusError})
target, fired := controller.maybeEscalate()
if !fired || target.MaxTurns != 80 {
t.Fatalf("expected escalation with the policy target, got fired=%t target=%+v", fired, target)
}
}
func TestProfileControllerEscalatesOnRiskyMutation(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnRiskyMutation: sandbox.RiskHigh}})
// Medium risk on an executed result: below threshold.
controller.observeToolOutcome(toolFailureOutcome{}, ToolResult{Status: tools.StatusOK, Risk: sandbox.Risk{Level: sandbox.RiskMedium}})
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("medium risk must not trip a high threshold")
}
// Critical risk on a DENIED result: never counts (the mutation never ran).
controller.observeToolOutcome(toolFailureOutcome{}, ToolResult{Status: tools.StatusError, DenialReason: DenialSandboxBlock, Risk: sandbox.Risk{Level: sandbox.RiskCritical}})
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("a denied result must not trip the risky-mutation signal")
}
// Critical risk on an executed PARTIAL FAILURE (error status, no denial):
// the mutation ran, so it counts.
controller.observeToolOutcome(toolFailureOutcome{}, ToolResult{Status: tools.StatusError, Risk: sandbox.Risk{Level: sandbox.RiskCritical}})
if _, fired := controller.maybeEscalate(); !fired {
t.Fatal("an executed partial failure at critical risk must trip a high threshold")
}
}
func TestProfileControllerIgnoresInvalidRiskThreshold(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnRiskyMutation: sandbox.RiskLevel("hgih")}})
controller.observeToolOutcome(toolFailureOutcome{}, ToolResult{Status: tools.StatusOK, Risk: sandbox.Risk{Level: sandbox.RiskCritical}})
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("an unrecognized threshold must disable the signal, not match every result")
}
}
func TestProfileControllerEscalatesOnSelfCorrectFailure(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnSelfCorrectFailure: true}})
controller.observeSelfCorrect(OutcomePassed)
controller.observeSelfCorrect(OutcomeDisabled)
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("passing/disabled outcomes must not escalate")
}
controller.observeSelfCorrect(OutcomeReported)
if _, fired := controller.maybeEscalate(); !fired {
t.Fatal("a failing verification outcome must escalate")
}
}
func TestProfileControllerEscalatesOnUncertainCompletion(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnCompletionUncertain: 2}})
controller.observeUncertain()
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("first uncertain evaluation must not trip a threshold of 2")
}
controller.observeUncertain()
if _, fired := controller.maybeEscalate(); !fired {
t.Fatal("second uncertain evaluation must escalate")
}
}
func TestProfileControllerEscalatesAtMostOnce(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnToolFailureStreak: 1}})
controller.observeToolOutcome(toolFailureOutcome{Count: 1}, ToolResult{Status: tools.StatusError})
if _, fired := controller.maybeEscalate(); !fired {
t.Fatal("expected first escalation")
}
controller.observeToolOutcome(toolFailureOutcome{Count: 5}, ToolResult{Status: tools.StatusOK, Risk: sandbox.Risk{Level: sandbox.RiskCritical}})
controller.observeUncertain()
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("escalation must be one-shot per run")
}
}
// TestPostureEscalationOnUncertainCompletion proves the uncertain-path act
// point: the completion gate's continue branch escapes the end-of-turn tail,
// so escalation must apply before the continue. A cue turn under the gate
// counts as uncertain, escalates, and the very next request carries the target
// effort.
func TestPostureEscalationOnUncertainCompletion(t *testing.T) {
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
{
{Type: zeroruntime.StreamEventText, Content: "Let me read the file:"},
{Type: zeroruntime.StreamEventDone},
},
{
{Type: zeroruntime.StreamEventText, Content: "Done. All set."},
{Type: zeroruntime.StreamEventDone},
},
}}
recorder := trace.NewRecorder("posture-session", "run-2", "fast")
result, err := Run(context.Background(), "go", provider, Options{
Registry: tools.NewRegistry(),
MaxTurns: 10,
RequireCompletionSignal: true,
Trace: recorder,
Profile: &ProfilePolicy{
Name: "fast",
Escalate: &PostureEscalation{
ReasoningEffort: "high",
OnCompletionUncertain: 1,
},
},
})
if err != nil {
t.Fatal(err)
}
if result.FinalAnswer != "Done. All set." {
t.Fatalf("expected the completed answer, got %q", result.FinalAnswer)
}
if len(provider.requests) != 2 {
t.Fatalf("expected 2 requests, got %d", len(provider.requests))
}
if provider.requests[1].ReasoningEffort != "high" {
t.Fatalf("expected the post-escalation request to carry the target effort, got %q", provider.requests[1].ReasoningEffort)
}
tr := recorder.Finish()
if got := tr.Counter(trace.CounterPostureEscalations); got != 1 {
t.Fatalf("posture_escalations = %d, want 1", got)
}
}
// A profile that filled the run's effort displaced the provider default ("");
// RestoreDefaultEffort lets escalation restore that default, which a plain ""
// ReasoningEffort target cannot express (it means "leave untouched").
func TestPostureEscalationRestoresDefaultEffort(t *testing.T) {
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
{
{Type: zeroruntime.StreamEventText, Content: "Let me read the file:"},
{Type: zeroruntime.StreamEventDone},
},
{
{Type: zeroruntime.StreamEventText, Content: "Done. All set."},
{Type: zeroruntime.StreamEventDone},
},
}}
result, err := Run(context.Background(), "go", provider, Options{
Registry: tools.NewRegistry(),
MaxTurns: 10,
ReasoningEffort: "low",
RequireCompletionSignal: true,
Profile: &ProfilePolicy{
Name: "fast",
Escalate: &PostureEscalation{
RestoreDefaultEffort: true,
OnCompletionUncertain: 1,
},
},
})
if err != nil {
t.Fatal(err)
}
if result.FinalAnswer != "Done. All set." {
t.Fatalf("expected the completed answer, got %q", result.FinalAnswer)
}
if len(provider.requests) != 2 {
t.Fatalf("expected 2 requests, got %d", len(provider.requests))
}
if provider.requests[0].ReasoningEffort != "low" {
t.Fatalf("pre-escalation request effort = %q, want the profile's low", provider.requests[0].ReasoningEffort)
}
if provider.requests[1].ReasoningEffort != "" {
t.Fatalf("post-escalation request effort = %q, want the restored provider default (empty)", provider.requests[1].ReasoningEffort)
}
}
// failingProfileTool always errors with the same signature so the repeated-
// failure guard builds a streak.
type failingProfileTool struct{}
func (failingProfileTool) Name() string { return "flaky_probe" }
func (failingProfileTool) Description() string { return "always fails identically (test)" }
func (failingProfileTool) Parameters() tools.Schema {
return tools.Schema{Type: "object", AdditionalProperties: false}
}
func (failingProfileTool) Safety() tools.Safety {
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow}
}
func (failingProfileTool) Run(context.Context, map[string]any) tools.Result {
return tools.Result{Status: tools.StatusError, Output: "probe exploded: same signature"}
}
// TestPostureEscalationRaisesTurnCeilingMidRun proves the end-to-end act path:
// a Fast-style run with a 2-turn ceiling hits a failure streak, escalates, and
// finishes on a turn the original ceiling would have cut off, with the effort
// target applied to the post-escalation request and the counter emitted.
func TestPostureEscalationRaisesTurnCeilingMidRun(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(failingProfileTool{})
toolTurn := []zeroruntime.StreamEvent{
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "c1", ToolName: "flaky_probe"},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "c1"},
{Type: zeroruntime.StreamEventDone},
}
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
toolTurn,
toolTurn,
{
{Type: zeroruntime.StreamEventText, Content: "recovered after escalation"},
{Type: zeroruntime.StreamEventDone},
},
}}
recorder := trace.NewRecorder("posture-session", "run-1", "fast")
result, err := Run(context.Background(), "go", provider, Options{
Registry: registry,
MaxTurns: 2,
Trace: recorder,
Profile: &ProfilePolicy{
Name: "fast",
Escalate: &PostureEscalation{
MaxTurns: 4,
ReasoningEffort: "high",
OnToolFailureStreak: 2,
},
},
})
if err != nil {
t.Fatal(err)
}
if result.FinalAnswer != "recovered after escalation" {
t.Fatalf("expected the run to continue past the original 2-turn ceiling, got %q (turns=%d)", result.FinalAnswer, result.Turns)
}
if len(provider.requests) != 3 {
t.Fatalf("expected 3 requests (ceiling raised from 2 to 4), got %d", len(provider.requests))
}
if provider.requests[2].ReasoningEffort != "high" {
t.Fatalf("expected the post-escalation request to carry the target effort, got %q", provider.requests[2].ReasoningEffort)
}
tr := recorder.Finish()
if got := tr.Counter(trace.CounterPostureEscalations); got != 1 {
t.Fatalf("posture_escalations = %d, want 1", got)
}
}
// TestPostureEscalationAbsentWithoutProfile pins the no-regression invariant at
// the loop level: the identical failing script without a profile ends at the
// original ceiling with the max-turns answer.
func TestPostureEscalationAbsentWithoutProfile(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(failingProfileTool{})
toolTurn := []zeroruntime.StreamEvent{
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "c1", ToolName: "flaky_probe"},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "c1"},
{Type: zeroruntime.StreamEventDone},
}
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{toolTurn, toolTurn}}
result, err := Run(context.Background(), "go", provider, Options{
Registry: registry,
MaxTurns: 2,
})
if err != nil {
t.Fatal(err)
}
if len(provider.requests) > 3 {
t.Fatalf("unprofiled run must not extend the ceiling, got %d requests", len(provider.requests))
}
if !strings.Contains(result.FinalAnswer, "maximum number of turns") {
t.Fatalf("unprofiled run must end with the max-turns answer, got %q", result.FinalAnswer)
}
}
// TestExecuteToolCallClassifiesRiskWithoutSandbox verifies the executed-risk
// stamp on the unsandboxed path: a read-only tool classifies low.
func TestExecuteToolCallClassifiesRiskWithoutSandbox(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(failingProfileTool{})
result, abortErr := executeToolCall(
context.Background(),
registry,
ToolCall{ID: "c1", Name: "flaky_probe"},
PermissionModeAuto,
Options{},
)
if abortErr != nil {
t.Fatal(abortErr)
}
if result.Risk.Level == "" {
t.Fatal("executed result must carry a risk classification")
}
if riskRank(result.Risk.Level) > riskRank(sandbox.RiskMedium) {
t.Fatalf("read-only tool classified %q, want at most medium", result.Risk.Level)
}
}
// TestUnknownToolResultCarriesZeroRisk verifies a call that never executed a
// tool (unknown name, nil tool) keeps the zero risk value instead of panicking
// or inventing a classification.
func TestUnknownToolResultCarriesZeroRisk(t *testing.T) {
registry := tools.NewRegistry()
result, abortErr := executeToolCall(
context.Background(),
registry,
ToolCall{ID: "c1", Name: "no_such_tool"},
PermissionModeAuto,
Options{},
)
if abortErr != nil {
t.Fatal(abortErr)
}
if result.Status == tools.StatusOK {
t.Fatal("expected an error result for an unknown tool")
}
if result.Risk.Level != "" {
t.Fatalf("not-executed result must keep the zero risk value, got %q", result.Risk.Level)
}
}
+244
View File
@@ -0,0 +1,244 @@
package agent
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"sort"
"strings"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
// promptSubstrings are the cacheable components of the prompt a run sends to a
// model. systemPrompt is the exact joined system-message content. The remaining
// fields retain useful diagnostic boundaries within that prompt and tool list.
type promptSubstrings struct {
systemPrompt string
baseInstructions string
confirmationPolicy string
projectContext string
skills string
tools string
schema string
}
// buildPromptSubstrings builds the prompt and retains both its exact joined
// form and narrower diagnostic components. Callers that already built the
// prompt should use buildPromptSubstringsFromParts to avoid repeating workspace
// reads.
func buildPromptSubstrings(options Options, exposed []zeroruntime.ToolDefinition) promptSubstrings {
return buildPromptSubstringsFromParts(buildSystemPromptParts(options), exposed)
}
func buildPromptSubstringsFromParts(parts systemPromptParts, exposed []zeroruntime.ToolDefinition) promptSubstrings {
toolsSubstr, schemaSubstr := toolSubstrings(exposed)
return promptSubstrings{
systemPrompt: parts.prompt,
baseInstructions: parts.baseInstructions,
confirmationPolicy: parts.confirmationPolicy,
projectContext: parts.projectContext,
skills: parts.skills,
tools: toolsSubstr,
schema: schemaSubstr,
}
}
// toolSubstrings preserves the emitted tool order. Names and descriptions are
// fingerprinted separately from schemas so either kind of drift is identifiable.
// Order is part of request identity: silently sorting here would hide a provider-
// visible reordering and could report a false cache hit.
func toolSubstrings(exposed []zeroruntime.ToolDefinition) (toolsSubstr, schemaSubstr string) {
if len(exposed) == 0 {
return "", ""
}
var toolsSB, schemaSB strings.Builder
for _, def := range exposed {
writeLengthPrefixed(&toolsSB, def.Name)
writeLengthPrefixed(&toolsSB, def.Description)
// Parameters is rendered to a canonical JSON string per tool. The
// schema-render cache in internal/agent (used by partitionToolsCached)
// guarantees the same render is byte-identical across turns for the
// same tool, so this substring is a stable hash input.
writeLengthPrefixed(&schemaSB, def.Name)
writeLengthPrefixed(&schemaSB, canonicalSchemaString(def.Parameters))
}
return toolsSB.String(), schemaSB.String()
}
func writeLengthPrefixed(sb *strings.Builder, value string) {
fmt.Fprintf(sb, "%d:", len(value))
sb.WriteString(value)
}
// canonicalSchemaString renders a tool's parameter schema to a stable string.
// tools.ToolDef.Parameters is a map[string]any in the common case; Go's
// fmt.Sprintf("%v", m) iterates a map in random order, which would produce
// a different hash for the same Parameters value across calls and defeat
// the fingerprint. encoding/json marshals maps with keys sorted
// alphabetically, so json.Marshal is the primary stable render. The
// fallback for the rare non-JSON-compatible value (functions, channels,
// NaN/Inf floats, cyclic references) is a stable key-sorted stringifier
// that walks the value with sorted keys for maps and a leading
// "__non_json:" prefix so a future schema change is visible in the trace
// (a SchemaHash collision from the same value) rather than a silent hash
// drift.
//
// Note: SchemaHash is a stability signal, not a wire-identity signal. The
// bytes json.Marshal produces for a Go map[string]any may differ from
// the bytes the provider actually sends on the wire (provider encoders
// use their own JSON conventions; an Anthropic schema may serialize with
// different whitespace, key order, or number formatting). A consumer
// must NOT assume SchemaHash matches the provider's wire schema — only
// that two turns with the same Go-side Parameters produce the same
// SchemaHash. This is sufficient for the trace's purpose (drift
// detection) but not for a content-based equality check.
func canonicalSchemaString(params map[string]any) string {
if len(params) == 0 {
return ""
}
data, err := json.Marshal(params)
if err != nil {
return "__non_json:" + stableStringify(params)
}
return string(data)
}
// stableStringify renders v to a deterministic string. Maps have their keys
// sorted alphabetically; slices are walked in order; primitives use their
// natural Go format. The function is recursive but bounded by the size of
// the input, so a cyclic reference is not reachable here (the json.Marshal
// fallback is hit when json.Marshal itself returns an error, which for the
// common case is a non-JSON-compatible value, not a cycle — cycles are
// rare in tool schemas and acceptable to mis-render in the fallback path
// since the trace's contract is "the hash is stable for the same input,"
// not "the fallback is lossless"). Used only when json.Marshal fails.
func stableStringify(v any) string {
var sb strings.Builder
writeStable(&sb, v)
return sb.String()
}
func writeStable(sb *strings.Builder, v any) {
switch x := v.(type) {
case nil:
sb.WriteString("null")
case bool:
if x {
sb.WriteString("true")
} else {
sb.WriteString("false")
}
case string:
sb.WriteByte('"')
sb.WriteString(x)
sb.WriteByte('"')
case float64:
// Use %g for compact, deterministic float rendering. NaN and Inf are
// not JSON-encodable (which is why we are in the fallback path) and
// render as "NaN" / "+Inf" / "-Inf" — distinct, stable strings.
fmt.Fprintf(sb, "%g", x)
case float32:
fmt.Fprintf(sb, "%g", float64(x))
case int:
fmt.Fprintf(sb, "%d", x)
case int64:
fmt.Fprintf(sb, "%d", x)
case int32:
fmt.Fprintf(sb, "%d", x)
case uint:
fmt.Fprintf(sb, "%d", x)
case uint64:
fmt.Fprintf(sb, "%d", x)
case uint32:
fmt.Fprintf(sb, "%d", x)
case map[string]any:
sb.WriteByte('{')
keys := make([]string, 0, len(x))
for k := range x {
keys = append(keys, k)
}
sort.Strings(keys)
for i, k := range keys {
if i > 0 {
sb.WriteByte(',')
}
sb.WriteByte('"')
sb.WriteString(k)
sb.WriteString(`":`)
writeStable(sb, x[k])
}
sb.WriteByte('}')
case []any:
sb.WriteByte('[')
for i, item := range x {
if i > 0 {
sb.WriteByte(',')
}
writeStable(sb, item)
}
sb.WriteByte(']')
default:
// Last-resort: include the type so distinct values produce distinct
// strings even if their default format collides. Without the type
// tag, fmt.Sprintf("%v", x) for two different types could produce
// the same bytes (rare, but the type prefix makes it impossible).
fmt.Fprintf(sb, "<%T:%v>", x, x)
}
}
// ComputePrefixFingerprint builds and fingerprints a prompt for callers that do
// not already have one. The agent loop uses computePrefixFingerprint with the
// prompt parts it built once at run start, ensuring tracing observes the exact
// request content without repeating workspace I/O.
func ComputePrefixFingerprint(options Options, exposed []zeroruntime.ToolDefinition) prefixFingerprint {
return computePrefixFingerprint(buildPromptSubstrings(options, exposed))
}
func computePrefixFingerprint(subs promptSubstrings) prefixFingerprint {
systemPrompt := sha256hex(subs.systemPrompt)
base := sha256hex(subs.baseInstructions)
policy := sha256hex(subs.confirmationPolicy)
project := sha256hex(subs.projectContext)
skills := sha256hex(subs.skills)
toolsH := sha256hex(subs.tools)
schema := sha256hex(subs.schema)
complete := sha256hex(strings.Join([]string{
systemPrompt, toolsH, schema,
}, "|"))
return prefixFingerprint{
SystemPromptHash: systemPrompt,
BaseInstructionsHash: base,
ConfirmationPolicyHash: policy,
ProjectContextHash: project,
SkillsHash: skills,
ToolsHash: toolsH,
SchemaHash: schema,
CompletePrefixHash: complete,
}
}
// prefixFingerprint is the agent-side shape of a prompt-prefix fingerprint. It
// is converted to a trace.PrefixHash at the loop boundary (see EmitPrefixHash
// in loop.go) so the trace package does not need to import this one. The field
// names match the trace.PrefixHash JSON tags 1:1.
type prefixFingerprint struct {
SystemPromptHash string
BaseInstructionsHash string
ConfirmationPolicyHash string
ProjectContextHash string
SkillsHash string
ToolsHash string
SchemaHash string
CompletePrefixHash string
}
// sha256hex returns the hex-encoded SHA-256 of s. Empty input produces the
// hash of the empty string, which is a constant; callers that want to
// distinguish "absent" from "empty" should check s before calling.
func sha256hex(s string) string {
sum := sha256.Sum256([]byte(s))
return hex.EncodeToString(sum[:])
}
+249
View File
@@ -0,0 +1,249 @@
package agent
import (
"strings"
"testing"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
// TestComputePrefixFingerprintStableAcrossCalls asserts the headline property
// the trace is meant to expose: two ComputePrefixFingerprint calls over the
// same Options and the same exposed tool list produce byte-identical hashes.
// This is the regression catch for "did anyone introduce non-determinism into
// the prompt prefix."
func TestComputePrefixFingerprintStableAcrossCalls(t *testing.T) {
opts := Options{Cwd: t.TempDir(), SystemPrompt: "core"}
exposed := []zeroruntime.ToolDefinition{
{Name: "read_file", Parameters: map[string]any{"type": "object"}},
{Name: "grep", Parameters: map[string]any{"type": "object"}},
}
first := ComputePrefixFingerprint(opts, exposed)
second := ComputePrefixFingerprint(opts, exposed)
if first != second {
t.Fatalf("fingerprint must be stable across calls with the same input:\n first=%+v\n second=%+v", first, second)
}
if first.CompletePrefixHash == "" {
t.Fatalf("CompletePrefixHash must be non-empty for a non-trivial prompt")
}
}
// Tool order and descriptions are provider-visible and therefore part of the
// cacheable prefix identity.
func TestComputePrefixFingerprintToolsHashMatchesEmittedOrderAndDescriptions(t *testing.T) {
opts := Options{Cwd: t.TempDir(), SystemPrompt: "core"}
a := []zeroruntime.ToolDefinition{
{Name: "read_file", Description: "read", Parameters: map[string]any{"schema": "schema-read"}},
{Name: "grep", Description: "search", Parameters: map[string]any{"schema": "schema-grep"}},
}
b := []zeroruntime.ToolDefinition{
{Name: "grep", Description: "search", Parameters: map[string]any{"schema": "schema-grep"}},
{Name: "read_file", Description: "read", Parameters: map[string]any{"schema": "schema-read"}},
}
fpA := ComputePrefixFingerprint(opts, a)
fpB := ComputePrefixFingerprint(opts, b)
if fpA.ToolsHash == fpB.ToolsHash {
t.Fatalf("ToolsHash must change when emitted order changes: a=%s b=%s", fpA.ToolsHash, fpB.ToolsHash)
}
c := append([]zeroruntime.ToolDefinition(nil), a...)
c[0].Description = "read a file"
fpC := ComputePrefixFingerprint(opts, c)
if fpA.ToolsHash == fpC.ToolsHash {
t.Fatalf("ToolsHash must change when a description changes: a=%s c=%s", fpA.ToolsHash, fpC.ToolsHash)
}
}
// TestComputePrefixFingerprintSchemaChangeChangesSchemaHash asserts that a
// change to a tool's Parameters schema is visible in the SchemaHash. A schema
// edit that should be cache-stable (e.g. field reorder, doc tweak) will move
// the hash and surface in the trace; a schema edit that legitimately
// invalidates the cache also moves the hash, which is the right behavior.
func TestComputePrefixFingerprintSchemaChangeChangesSchemaHash(t *testing.T) {
opts := Options{Cwd: t.TempDir(), SystemPrompt: "core"}
a := []zeroruntime.ToolDefinition{
{Name: "read_file", Parameters: map[string]any{"version": "v1"}},
}
b := []zeroruntime.ToolDefinition{
{Name: "read_file", Parameters: map[string]any{"version": "v2"}},
}
fpA := ComputePrefixFingerprint(opts, a)
fpB := ComputePrefixFingerprint(opts, b)
if fpA.SchemaHash == fpB.SchemaHash {
t.Fatalf("SchemaHash must change when tool Parameters change: a=%s b=%s", fpA.SchemaHash, fpB.SchemaHash)
}
}
// TestComputePrefixFingerprintSystemPromptChangeChangesBaseHash asserts the
// most common drift path: a system-prompt edit (e.g. an agent.md update, a
// model addendum change) must move the base-instructions hash, which moves
// the complete-prefix hash, which is what the trace is for.
func TestComputePrefixFingerprintSystemPromptChangeChangesBaseHash(t *testing.T) {
optsA := Options{Cwd: t.TempDir(), SystemPrompt: "core v1"}
optsB := Options{Cwd: t.TempDir(), SystemPrompt: "core v2"}
exposed := []zeroruntime.ToolDefinition{{Name: "noop", Parameters: map[string]any{}}}
fpA := ComputePrefixFingerprint(optsA, exposed)
fpB := ComputePrefixFingerprint(optsB, exposed)
if fpA.BaseInstructionsHash == fpB.BaseInstructionsHash {
t.Fatalf("BaseInstructionsHash must change when SystemPrompt changes: a=%s b=%s", fpA.BaseInstructionsHash, fpB.BaseInstructionsHash)
}
if fpA.CompletePrefixHash == fpB.CompletePrefixHash {
t.Fatalf("CompletePrefixHash must change when any sub-hash changes: a=%s b=%s", fpA.CompletePrefixHash, fpB.CompletePrefixHash)
}
}
// CompletePrefixHash represents the exact joined system prompt and the ordered
// tool definitions. Narrower system-section hashes remain diagnostic only.
func TestComputePrefixFingerprintCompleteIsAggregateOfSubHashes(t *testing.T) {
opts := Options{Cwd: t.TempDir(), SystemPrompt: "core"}
exposed := []zeroruntime.ToolDefinition{{Name: "read_file", Parameters: map[string]any{"k": "v"}}}
fp := ComputePrefixFingerprint(opts, exposed)
// Manually compute what the canonical join should be.
expected := sha256hex(strings.Join([]string{
fp.SystemPromptHash,
fp.ToolsHash,
fp.SchemaHash,
}, "|"))
if fp.CompletePrefixHash != expected {
t.Fatalf("CompletePrefixHash must equal sha256hex of the exact system, tools, and schema hashes:\n got: %s\n expected: %s", fp.CompletePrefixHash, expected)
}
}
func TestComputePrefixFingerprintCoversEverySystemPromptSection(t *testing.T) {
base := Options{Cwd: t.TempDir(), SystemPrompt: "core", Model: "gpt-5"}
changed := base
changed.ResponseStyle = "concise"
fpBase := ComputePrefixFingerprint(base, nil)
fpChanged := ComputePrefixFingerprint(changed, nil)
if fpBase.SystemPromptHash == fpChanged.SystemPromptHash {
t.Fatal("SystemPromptHash must change when any rendered system section changes")
}
if fpBase.CompletePrefixHash == fpChanged.CompletePrefixHash {
t.Fatal("CompletePrefixHash must include the exact full system prompt")
}
}
// TestBuildPromptSubstringsDefaultOptions asserts the invariants of the
// substrings helper for default Options: which substrings are non-empty,
// which are empty, and that the substring-to-hash round-trip is lossless.
func TestBuildPromptSubstringsDefaultOptions(t *testing.T) {
opts := Options{
Cwd: t.TempDir(),
SystemPrompt: "test core",
}
subs := buildPromptSubstrings(opts, nil)
// Invariants the trace depends on (default Options):
// 1. baseInstructions substring equals the core system prompt bytes.
// 2. confirmationPolicy substring is non-empty (the embedded policy
// is always present, post TrimSpace).
// 3. skills, tools, and schema substrings are empty (no skills or
// tools configured for default Options).
if subs.baseInstructions != "test core" {
t.Fatalf("baseInstructions substring must equal the core system prompt: got %q want %q", subs.baseInstructions, "test core")
}
if subs.systemPrompt != buildSystemPrompt(opts) {
t.Fatal("systemPrompt substring must equal the exact joined system prompt")
}
if subs.confirmationPolicy == "" {
t.Fatalf("confirmationPolicy substring must be non-empty (the embedded policy is always present)")
}
if subs.skills != "" {
t.Fatalf("skills substring must be empty for default Options: got %q", subs.skills)
}
// Round-trip: the corresponding fingerprint hashes must equal
// sha256hex of the substrings, with no truncation or reformatting
// between the two layers.
fp := ComputePrefixFingerprint(opts, nil)
if fp.SystemPromptHash != sha256hex(subs.systemPrompt) {
t.Fatalf("SystemPromptHash in fingerprint must equal sha256hex of the joined prompt")
}
if fp.BaseInstructionsHash != sha256hex(subs.baseInstructions) {
t.Fatalf("BaseInstructionsHash in fingerprint must equal sha256hex of the substring")
}
if fp.ConfirmationPolicyHash != sha256hex(subs.confirmationPolicy) {
t.Fatalf("ConfirmationPolicyHash in fingerprint must equal sha256hex of the substring")
}
}
// TestCanonicalSchemaStringStableForMapParameters asserts the headline
// property the trace needs: the same map[string]any produces the same
// canonical string across calls, even though Go's fmt.Sprintf("%v", m)
// iterates a map in random order. json.Marshal sorts map keys
// alphabetically, which is the fix.
func TestCanonicalSchemaStringStableForMapParameters(t *testing.T) {
params := map[string]any{
"description": "read a file",
"properties": map[string]any{
"path": map[string]any{"type": "string"},
},
"required": []any{"path"},
"type": "object",
}
first := canonicalSchemaString(params)
second := canonicalSchemaString(params)
if first != second {
t.Fatalf("canonicalSchemaString must be stable across calls for the same Parameters:\n first=%s\n second=%s", first, second)
}
if first == "" {
t.Fatal("canonicalSchemaString must produce a non-empty string for a non-empty map")
}
// Two maps with the same key/value pairs (different declaration
// order in source) must produce identical canonical strings. This
// is the property that defeats Go's map iteration randomization.
reordered := map[string]any{
"type": "object",
"required": []any{"path"},
"properties": map[string]any{
"path": map[string]any{"type": "string"},
},
"description": "read a file",
}
if canonicalSchemaString(params) != canonicalSchemaString(reordered) {
t.Fatal("canonicalSchemaString must produce identical output for maps with the same key/value pairs in different declaration orders")
}
}
// TestCanonicalSchemaStringFallbackStableForNonJSONValue exercises the
// non-JSON fallback in canonicalSchemaString: when Parameters contains a
// value json.Marshal cannot encode (here, a channel), the function must
// fall back to a key-sorted stringification that is stable across calls
// and across map-iteration order. The previous fallback used
// fmt.Sprintf("%v", params), which iterates the map in random order and
// produced a different hash for the same value across calls — defeating
// the fingerprint. The current fallback is stable by construction.
func TestCanonicalSchemaStringFallbackStableForNonJSONValue(t *testing.T) {
ch := make(chan int) // channels are not JSON-encodable
params := map[string]any{
"description": "a tool with a non-JSON value",
"channel": ch,
"z_first": 1,
"a_second": 2,
}
first := canonicalSchemaString(params)
second := canonicalSchemaString(params)
if first == "" {
t.Fatal("canonicalSchemaString must produce a non-empty string for the fallback path")
}
if first != second {
t.Fatalf("canonicalSchemaString fallback must be stable across calls:\n first=%s\n second=%s", first, second)
}
// A re-ordered map (same key/value pairs, different declaration order)
// must produce the same fallback string. This is the property the
// key-sorted stringifier provides that fmt.Sprintf("%v", m) does not.
reordered := map[string]any{
"a_second": 2,
"z_first": 1,
"channel": ch,
"description": "a tool with a non-JSON value",
}
if canonicalSchemaString(params) != canonicalSchemaString(reordered) {
t.Fatalf("canonicalSchemaString fallback must produce identical output for maps with the same key/value pairs in different declaration orders:\n first=%s\n second=%s", first, canonicalSchemaString(reordered))
}
// The fallback must include the "__non_json:" prefix so a consumer
// can tell the bytes are not a JSON-marshaled schema (a hash collision
// between a json.Marshal result and a stableStringify result is
// possible in theory; the prefix makes the source observable).
if !strings.HasPrefix(first, "__non_json:") {
t.Fatalf("canonicalSchemaString fallback must start with the __non_json: prefix, got: %s", first)
}
}
+30 -5
View File
@@ -70,6 +70,22 @@ const (
// guidelines), and the safety confirmation policy. It is built once per run so
// every turn shares one (cacheable) system turn.
func buildSystemPrompt(options Options) string {
return buildSystemPromptParts(options).prompt
}
// systemPromptParts retains both the exact joined prompt and the diagnostic
// sections used by prompt-prefix tracing. Building these together prevents the
// tracing path from re-reading workspace state and guarantees that the hash is
// computed from the same prompt bytes placed in the request.
type systemPromptParts struct {
prompt string
baseInstructions string
confirmationPolicy string
projectContext string
skills string
}
func buildSystemPromptParts(options Options) systemPromptParts {
core := strings.TrimSpace(options.SystemPrompt)
if core == "" {
core = strings.TrimSpace(coreSystemPrompt)
@@ -97,22 +113,31 @@ func buildSystemPrompt(options Options) string {
if user := userGuidelines(); user != "" {
sections = append(sections, user)
}
if ws := workspaceContext(options.Cwd); ws != "" {
sections = append(sections, ws)
project := workspaceContext(options.Cwd)
if project != "" {
sections = append(sections, project)
}
if delegation := specialistDelegationContext(options); delegation != "" {
sections = append(sections, delegation)
}
if skillsBlock := skillsContext(options); skillsBlock != "" {
skillsBlock := skillsContext(options)
if skillsBlock != "" {
sections = append(sections, skillsBlock)
}
if style := responseStyleContext(options); style != "" {
sections = append(sections, style)
}
if policy := strings.TrimSpace(confirmationPolicy); policy != "" {
policy := strings.TrimSpace(confirmationPolicy)
if policy != "" {
sections = append(sections, policy)
}
return strings.Join(sections, "\n\n")
return systemPromptParts{
prompt: strings.Join(sections, "\n\n"),
baseInstructions: core,
confirmationPolicy: policy,
projectContext: project,
skills: skillsBlock,
}
}
// responseStyleContext renders the operator-selected reply style (TUI /style) as
+359
View File
@@ -0,0 +1,359 @@
package agent
import (
"encoding/json"
"reflect"
"sort"
"strings"
"github.com/Gitlawb/zero/internal/tools"
"github.com/Gitlawb/zero/internal/trace"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
type taskStatus string
const (
taskStatusActive taskStatus = "active"
taskStatusComplete taskStatus = "complete"
taskStatusIncomplete taskStatus = "incomplete"
)
type taskPlanParity string
const (
taskPlanParityUnknown taskPlanParity = "unknown"
taskPlanParityMatch taskPlanParity = "match"
taskPlanParityMismatch taskPlanParity = "mismatch"
)
type taskPlanItem struct {
Content string `json:"content"`
Status string `json:"status"`
Notes string `json:"notes,omitempty"`
}
type taskPlanState struct {
Items []taskPlanItem `json:"items,omitempty"`
Pending int `json:"pending"`
InProgress int `json:"in_progress"`
Completed int `json:"completed"`
Failed int `json:"failed"`
}
type taskToolState struct {
Succeeded int `json:"succeeded"`
Failed int `json:"failed"`
}
type taskVerificationState struct {
Passed int `json:"passed"`
Failed int `json:"failed"`
LastOutcome Outcome `json:"last_outcome,omitempty"`
}
type taskStateSnapshot struct {
Revision int `json:"revision"`
Objective string `json:"objective"`
Status taskStatus `json:"status"`
Plan taskPlanState `json:"plan"`
Tools taskToolState `json:"tools"`
Verification taskVerificationState `json:"verification"`
ChangedFiles []string `json:"changed_files,omitempty"`
CompletionDecision CompletionDecision `json:"completion_decision,omitempty"`
CompletionReason string `json:"completion_reason,omitempty"`
PlanParity taskPlanParity `json:"plan_parity"`
}
type taskStateEventKind int
const (
taskStateEventPlan taskStateEventKind = iota + 1
taskStateEventToolResult
taskStateEventVerification
taskStateEventCompletion
)
type taskStateEvent struct {
kind taskStateEventKind
arguments string
toolResult ToolResult
verification Outcome
completion completionEvaluation
}
// taskState is a deterministic projection of facts the loop already observes.
// It does not initiate work or replace the transcript; consumers must check
// transcript parity before using its compact representation.
type taskState struct {
snapshotValue taskStateSnapshot
changedFiles map[string]struct{}
recorder *trace.Recorder
planObserved bool
}
func newTaskState(objective string, recorder *trace.Recorder) *taskState {
state := &taskState{
snapshotValue: taskStateSnapshot{
Objective: strings.TrimSpace(objective),
Status: taskStatusActive,
PlanParity: taskPlanParityUnknown,
},
changedFiles: map[string]struct{}{},
}
state.recorder = recorder
return state
}
func (state *taskState) observe(event taskStateEvent) bool {
if state == nil {
return false
}
changed := false
switch event.kind {
case taskStateEventPlan:
plan, ok := parseTaskPlan(event.arguments)
if !ok {
return false
}
state.snapshotValue.Plan = summarizeTaskPlan(plan)
state.planObserved = true
state.markActive()
changed = true
case taskStateEventToolResult:
state.markActive()
if event.toolResult.Status == tools.StatusOK {
state.snapshotValue.Tools.Succeeded++
} else {
state.snapshotValue.Tools.Failed++
}
for _, path := range event.toolResult.ChangedFiles {
path = strings.TrimSpace(path)
if path != "" {
state.changedFiles[path] = struct{}{}
}
}
state.snapshotValue.ChangedFiles = sortedKeys(state.changedFiles)
changed = true
case taskStateEventVerification:
if event.verification == OutcomeDisabled {
return false
}
state.markActive()
state.snapshotValue.Verification.LastOutcome = event.verification
switch event.verification {
case OutcomePassed:
state.snapshotValue.Verification.Passed++
case OutcomeCorrecting, OutcomeReported, OutcomeAborted:
state.snapshotValue.Verification.Failed++
}
changed = true
case taskStateEventCompletion:
state.snapshotValue.CompletionDecision = event.completion.Decision
state.snapshotValue.CompletionReason = event.completion.Reason
switch event.completion.Decision {
case CompletionComplete:
state.snapshotValue.Status = taskStatusComplete
case CompletionIncomplete:
state.snapshotValue.Status = taskStatusIncomplete
default:
state.snapshotValue.Status = taskStatusActive
}
changed = true
}
if changed {
state.snapshotValue.Revision++
if event.kind != taskStateEventToolResult {
state.emit()
}
}
return changed
}
func (state *taskState) markActive() {
state.snapshotValue.Status = taskStatusActive
state.snapshotValue.CompletionDecision = ""
state.snapshotValue.CompletionReason = ""
}
func (state *taskState) snapshot() taskStateSnapshot {
if state == nil {
return taskStateSnapshot{}
}
snapshot := state.snapshotValue
snapshot.Plan.Items = append([]taskPlanItem(nil), state.snapshotValue.Plan.Items...)
snapshot.ChangedFiles = append([]string(nil), state.snapshotValue.ChangedFiles...)
return snapshot
}
// observePlanParity compares only the latest plan projection with the plan tool
// calls still present in messages. It mutates the snapshot and emits when the
// parity value changes; objective, tool, and verification fields are not part of
// this comparison.
func (state *taskState) observePlanParity(messages []zeroruntime.Message) taskPlanParity {
if state == nil {
return taskPlanParityUnknown
}
transcriptPlan, found, valid := latestTaskPlan(messages)
tracked := state.snapshotValue.Plan.Items
parity := taskPlanParityMatch
if !valid || found != state.planObserved || (found && !reflect.DeepEqual(transcriptPlan, tracked)) {
parity = taskPlanParityMismatch
}
if state.snapshotValue.PlanParity != parity {
state.snapshotValue.PlanParity = parity
state.snapshotValue.Revision++
state.emit()
}
return parity
}
type completionContext struct {
Objective string
PlanPending bool
PlanMatchesTranscript bool
}
func (state *taskState) completionContext(messages []zeroruntime.Message, transcriptPlanPending bool) completionContext {
context := completionContext{PlanPending: transcriptPlanPending}
if state == nil {
return context
}
context.Objective = state.snapshotValue.Objective
context.PlanMatchesTranscript = state.observePlanParity(messages) == taskPlanParityMatch
if context.PlanMatchesTranscript {
context.PlanPending = state.snapshotValue.Plan.Pending+state.snapshotValue.Plan.InProgress > 0
}
return context
}
func (state *taskState) snapshotForCompaction(messages []zeroruntime.Message) *taskStateSnapshot {
if state == nil {
return nil
}
state.observePlanParity(messages)
snapshot := state.snapshot()
return &snapshot
}
func (state *taskState) emit() {
if state == nil || state.recorder == nil {
return
}
snapshot := state.snapshotValue
state.recorder.EmitTaskState(trace.TaskStateEvent{
Revision: snapshot.Revision,
Status: string(snapshot.Status),
PlanPending: snapshot.Plan.Pending,
PlanInProgress: snapshot.Plan.InProgress,
PlanCompleted: snapshot.Plan.Completed,
PlanFailed: snapshot.Plan.Failed,
ToolsSucceeded: snapshot.Tools.Succeeded,
ToolsFailed: snapshot.Tools.Failed,
VerificationPassed: snapshot.Verification.Passed,
VerificationFailed: snapshot.Verification.Failed,
VerificationOutcome: string(snapshot.Verification.LastOutcome),
ChangedFileCount: len(snapshot.ChangedFiles),
CompletionDecision: string(snapshot.CompletionDecision),
PlanParity: string(snapshot.PlanParity),
})
}
func parseTaskPlan(arguments string) ([]taskPlanItem, bool) {
var object map[string]json.RawMessage
if json.Unmarshal([]byte(strings.TrimSpace(arguments)), &object) != nil {
return nil, false
}
rawPlan, ok := object["plan"]
if !ok || string(rawPlan) == "null" {
return nil, false
}
var parsed []struct {
ID *string `json:"id"`
Content string `json:"content"`
Status string `json:"status"`
Notes string `json:"notes"`
}
if json.Unmarshal(rawPlan, &parsed) != nil {
return nil, false
}
plan := make([]taskPlanItem, 0, len(parsed))
for _, raw := range parsed {
if raw.Content == "" {
return nil, false
}
plan = append(plan, taskPlanItem{
Content: raw.Content,
Status: normalizeTaskPlanStatus(raw.Status),
Notes: raw.Notes,
})
}
lastInProgress := -1
for index := range plan {
if plan[index].Status == "in_progress" {
if lastInProgress >= 0 {
plan[lastInProgress].Status = "completed"
}
lastInProgress = index
}
}
if len(plan) == 0 {
return nil, true
}
return plan, true
}
func normalizeTaskPlanStatus(status string) string {
switch strings.ToLower(strings.TrimSpace(status)) {
case "completed", "complete", "done", "finished", "resolved", "✓", "x", "[x]":
return "completed"
case "in_progress", "in-progress", "inprogress", "in progress", "active", "doing", "started", "current", "wip", "ongoing", "running":
return "in_progress"
case "failed", "fail", "error", "errored", "blocked", "cancelled", "canceled", "abandoned", "skipped":
return "failed"
default:
return "pending"
}
}
func summarizeTaskPlan(items []taskPlanItem) taskPlanState {
plan := taskPlanState{Items: append([]taskPlanItem(nil), items...)}
for _, item := range items {
switch item.Status {
case "completed":
plan.Completed++
case "in_progress":
plan.InProgress++
case "failed":
plan.Failed++
default:
plan.Pending++
}
}
return plan
}
func latestTaskPlan(messages []zeroruntime.Message) ([]taskPlanItem, bool, bool) {
for i := len(messages) - 1; i >= 0; i-- {
for j := len(messages[i].ToolCalls) - 1; j >= 0; j-- {
call := messages[i].ToolCalls[j]
if call.Name != planToolName {
continue
}
plan, ok := parseTaskPlan(call.Arguments)
return plan, true, ok
}
}
return nil, false, true
}
func sortedKeys(values map[string]struct{}) []string {
keys := make([]string, 0, len(values))
for value := range values {
keys = append(keys, value)
}
// Paths are evidence, not chronology. Stable ordering makes replay snapshots
// deterministic even if events originate from parallel read batches later.
sort.Strings(keys)
return keys
}
+204
View File
@@ -0,0 +1,204 @@
package agent
import (
"context"
"reflect"
"testing"
"github.com/Gitlawb/zero/internal/tools"
"github.com/Gitlawb/zero/internal/trace"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
func TestTaskStateReplayIsDeterministic(t *testing.T) {
events := []taskStateEvent{
{kind: taskStateEventPlan, arguments: `{"plan":[{"content":"inspect","status":"done"},{"content":"implement","status":"in progress"},{"content":"verify"}]}`},
{kind: taskStateEventToolResult, toolResult: ToolResult{Name: "apply_patch", Status: tools.StatusOK, ChangedFiles: []string{"a.go", "a.go", "b.go"}}},
{kind: taskStateEventToolResult, toolResult: ToolResult{Name: "go test", Status: tools.StatusError}},
{kind: taskStateEventVerification, verification: OutcomePassed},
{kind: taskStateEventCompletion, completion: completionEvaluation{Decision: CompletionUncertain, Reason: "pending work"}},
}
first := newTaskState("ship the change", nil)
second := newTaskState("ship the change", nil)
for _, event := range events {
first.observe(event)
second.observe(event)
}
if !reflect.DeepEqual(first.snapshot(), second.snapshot()) {
t.Fatalf("replaying the same events produced different snapshots:\nfirst: %#v\nsecond: %#v", first.snapshot(), second.snapshot())
}
got := first.snapshot()
if got.Status != taskStatusActive || got.Plan.Pending != 1 || got.Plan.InProgress != 1 || got.Plan.Completed != 1 {
t.Fatalf("unexpected plan snapshot: %#v", got)
}
if got.Tools.Succeeded != 1 || got.Tools.Failed != 1 || len(got.ChangedFiles) != 2 {
t.Fatalf("unexpected evidence snapshot: %#v", got)
}
if got.Verification.Passed != 1 || got.Verification.Failed != 0 || got.Verification.LastOutcome != OutcomePassed {
t.Fatalf("unexpected verification snapshot: %#v", got.Verification)
}
}
func TestTaskStateCoalescesToolResultsIntoNextTraceEvent(t *testing.T) {
recorder := trace.NewRecorder("session", "run", "")
state := newTaskState("objective", recorder)
for range 20 {
state.observe(taskStateEvent{kind: taskStateEventToolResult, toolResult: ToolResult{Status: tools.StatusOK}})
}
if events := recorder.Finish().TaskStates; len(events) != 0 {
t.Fatalf("tool results should be coalesced, got %d trace events", len(events))
}
recorder = trace.NewRecorder("session", "run-2", "")
state = newTaskState("objective", recorder)
for range 20 {
state.observe(taskStateEvent{kind: taskStateEventToolResult, toolResult: ToolResult{Status: tools.StatusOK}})
}
state.observe(taskStateEvent{kind: taskStateEventCompletion, completion: completionEvaluation{Decision: CompletionComplete}})
events := recorder.Finish().TaskStates
if len(events) != 1 || events[0].ToolsSucceeded != 20 {
t.Fatalf("coalesced tool total missing from completion snapshot: %#v", events)
}
}
func TestRunEmitsTaskStateFromExistingLoopEvents(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(tools.NewUpdatePlanTool())
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
{
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "plan-1", ToolName: planToolName},
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "plan-1", ArgumentsFragment: `{"plan":[{"content":"implement","status":"completed"}]}`},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "plan-1"},
{Type: zeroruntime.StreamEventDone},
},
{
{Type: zeroruntime.StreamEventText, Content: "done"},
{Type: zeroruntime.StreamEventDone},
},
}}
recorder := trace.NewRecorder("session", "run", "")
result, err := Run(context.Background(), "implement the change", provider, Options{
Registry: registry,
Trace: recorder,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
if result.FinalAnswer != "done" {
t.Fatalf("final answer = %q, want done", result.FinalAnswer)
}
events := recorder.Finish().TaskStates
if len(events) < 3 {
t.Fatalf("expected plan, tool, completion, and parity snapshots, got %#v", events)
}
last := events[len(events)-1]
if last.Status != string(taskStatusComplete) || last.PlanCompleted != 1 || last.ToolsSucceeded != 1 || last.PlanParity != string(taskPlanParityMatch) {
t.Fatalf("unexpected final task snapshot: %#v", last)
}
}
func TestTaskStateSnapshotIsImmutable(t *testing.T) {
state := newTaskState("objective", nil)
state.observe(taskStateEvent{kind: taskStateEventPlan, arguments: `{"plan":[{"content":"one","status":"pending"}]}`})
state.observe(taskStateEvent{kind: taskStateEventToolResult, toolResult: ToolResult{Status: tools.StatusOK, ChangedFiles: []string{"one.go"}}})
snapshot := state.snapshot()
snapshot.Plan.Items[0].Content = "mutated"
snapshot.ChangedFiles[0] = "mutated.go"
fresh := state.snapshot()
if fresh.Plan.Items[0].Content != "one" || fresh.ChangedFiles[0] != "one.go" {
t.Fatalf("snapshot mutation leaked into task state: %#v", fresh)
}
}
func TestTaskStatePlanParityUsesLatestPlan(t *testing.T) {
state := newTaskState("objective", nil)
state.observe(taskStateEvent{kind: taskStateEventPlan, arguments: `{"plan":[{"content":"old","status":"completed"}]}`})
state.observe(taskStateEvent{kind: taskStateEventPlan, arguments: `{"plan":[{"content":"new","status":"in_progress"}]}`})
messages := []zeroruntime.Message{{Role: zeroruntime.MessageRoleAssistant, ToolCalls: []zeroruntime.ToolCall{
{Name: planToolName, Arguments: `{"plan":[{"content":"new","status":"in_progress"}]}`},
}}}
if parity := state.observePlanParity(messages); parity != taskPlanParityMatch {
t.Fatalf("parity = %q, want match", parity)
}
messages[0].ToolCalls[0].Arguments = `{"plan":[{"content":"different","status":"pending"}]}`
if parity := state.observePlanParity(messages); parity != taskPlanParityMismatch {
t.Fatalf("parity = %q, want mismatch", parity)
}
}
func TestTaskStateMatchesPlanToolNormalization(t *testing.T) {
state := newTaskState("objective", nil)
arguments := `{"plan":[{"content":"first","status":"in_progress"},{"content":"second","status":"in_progress"}]}`
state.observe(taskStateEvent{kind: taskStateEventPlan, arguments: arguments})
snapshot := state.snapshot()
if snapshot.Plan.Completed != 1 || snapshot.Plan.InProgress != 1 {
t.Fatalf("multiple active items were not normalized like the plan tool: %#v", snapshot.Plan)
}
messages := []zeroruntime.Message{{Role: zeroruntime.MessageRoleAssistant, ToolCalls: []zeroruntime.ToolCall{{Name: planToolName, Arguments: arguments}}}}
if parity := state.observePlanParity(messages); parity != taskPlanParityMatch {
t.Fatalf("normalized state should still match its transcript event, got %q", parity)
}
empty := newTaskState("objective", nil)
empty.observe(taskStateEvent{kind: taskStateEventPlan, arguments: `{"plan":[]}`})
emptyMessages := []zeroruntime.Message{{Role: zeroruntime.MessageRoleAssistant, ToolCalls: []zeroruntime.ToolCall{{Name: planToolName, Arguments: `{"plan":[]}`}}}}
if parity := empty.observePlanParity(emptyMessages); parity != taskPlanParityMatch {
t.Fatalf("explicit empty plan should match transcript, got %q", parity)
}
}
func TestTaskStateContextFallsBackWhenTranscriptDiffers(t *testing.T) {
state := newTaskState("objective", nil)
state.observe(taskStateEvent{kind: taskStateEventPlan, arguments: `{"plan":[{"content":"tracked","status":"completed"}]}`})
messages := []zeroruntime.Message{{Role: zeroruntime.MessageRoleAssistant, ToolCalls: []zeroruntime.ToolCall{
{Name: planToolName, Arguments: `{"plan":[{"content":"transcript","status":"pending"}]}`},
}}}
context := state.completionContext(messages, true)
if !context.PlanPending || context.PlanMatchesTranscript {
t.Fatalf("completion context must retain transcript truth on mismatch: %#v", context)
}
if context.Objective != "objective" {
t.Fatalf("objective lost from completion context: %#v", context)
}
}
func TestTaskStateCompactionSnapshotRetainsObjectiveOnPlanMismatch(t *testing.T) {
state := newTaskState("objective", nil)
state.observe(taskStateEvent{kind: taskStateEventPlan, arguments: `{"plan":[{"content":"verify","status":"pending"}]}`})
matching := []zeroruntime.Message{{Role: zeroruntime.MessageRoleAssistant, ToolCalls: []zeroruntime.ToolCall{
{Name: planToolName, Arguments: `{"plan":[{"content":"verify","status":"pending"}]}`},
}}}
if snapshot := state.snapshotForCompaction(matching); snapshot == nil || snapshot.Objective != "objective" {
t.Fatalf("matching transcript should produce compact state, got %#v", snapshot)
}
mismatching := append([]zeroruntime.Message(nil), matching...)
mismatching[0].ToolCalls = []zeroruntime.ToolCall{{Name: planToolName, Arguments: `{"plan":[{"content":"other","status":"pending"}]}`}}
if snapshot := state.snapshotForCompaction(mismatching); snapshot == nil || snapshot.Objective != "objective" || snapshot.PlanParity != taskPlanParityMismatch {
t.Fatalf("plan mismatch must retain immutable objective and mark mutable state uncorroborated, got %#v", snapshot)
}
}
func TestParseTaskPlanRejectsArgumentsTheToolWouldReject(t *testing.T) {
for _, arguments := range []string{
`{}`,
`{"plan":null}`,
`{"plan":[{"step":"alias is not accepted"}]}`,
`{"plan":[{"content":""}]}`,
`{"plan":[{"content":"valid"},{"content":""}]}`,
`{"plan":[{"id":4,"content":"valid"}]}`,
} {
if plan, ok := parseTaskPlan(arguments); ok {
t.Fatalf("parseTaskPlan(%s) = %#v, true; want rejected", arguments, plan)
}
}
}
+21
View File
@@ -0,0 +1,21 @@
package agent
import (
"context"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
// sessionProvider adapts a TurnSession back to the Provider interface so the
// loop's existing Provider-shaped call sites (streamWithReconnect, the
// compaction summarizer, finalAnswerAfterMaxTurns) route their stream I/O
// through the session without any signature change. For the default adapter
// this reduces to the wrapped provider's StreamCompletion, so behavior is
// byte-identical; an optimized session (PR8) takes effect here transparently.
type sessionProvider struct {
session zeroruntime.TurnSession
}
func (s sessionProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
return s.session.Stream(ctx, request)
}
+367
View File
@@ -0,0 +1,367 @@
package agent
import (
"context"
"errors"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/tools"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
// fakeTurnSession wraps an inner provider for streaming while counting the
// session lifecycle, so tests can observe exactly how the loop drives it.
type fakeTurnSession struct {
inner Provider
prewarms int
streams int
closes int
closeErr error
}
func (s *fakeTurnSession) Prewarm(context.Context) error { s.prewarms++; return nil }
func (s *fakeTurnSession) Stream(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
s.streams++
return s.inner.StreamCompletion(ctx, request)
}
func (s *fakeTurnSession) Compact(context.Context, zeroruntime.CompletionRequest) ([]zeroruntime.Message, error) {
return nil, zeroruntime.ErrCompactionUnsupported
}
func (s *fakeTurnSession) Close() error { s.closes++; return s.closeErr }
// fakeTurnSessionProvider opens a scripted session (or fails to).
type fakeTurnSessionProvider struct {
session *fakeTurnSession
openErr error
opens int
}
func (p *fakeTurnSessionProvider) OpenTurnSession(context.Context) (zeroruntime.TurnSession, error) {
p.opens++
if p.openErr != nil {
return nil, p.openErr
}
return p.session, nil
}
func (p *fakeTurnSessionProvider) Capabilities() zeroruntime.ProviderCapabilities {
return zeroruntime.ProviderCapabilities{}
}
func singleAnswerTurns(answer string) [][]zeroruntime.StreamEvent {
return [][]zeroruntime.StreamEvent{{
{Type: zeroruntime.StreamEventText, Content: answer},
{Type: zeroruntime.StreamEventDone},
}}
}
// TestRunOpenTurnSessionFailureSurfaces verifies a failed session open is a
// clean run-start error: no panic, no stream ever issued, nothing to close.
func TestRunOpenTurnSessionFailureSurfaces(t *testing.T) {
positional := &mockProvider{turns: singleAnswerTurns("never reached")}
tsp := &fakeTurnSessionProvider{openErr: errors.New("handshake refused")}
_, err := Run(context.Background(), "go", positional, Options{
MaxTurns: 2,
TurnSessionProvider: tsp,
})
if err == nil {
t.Fatal("expected an error from a failed session open")
}
if !strings.Contains(err.Error(), "open turn session") || !strings.Contains(err.Error(), "handshake refused") {
t.Fatalf("expected a wrapped open-turn-session error, got %v", err)
}
if len(positional.requests) != 0 {
t.Fatalf("expected no provider request after a failed open, got %d", len(positional.requests))
}
if tsp.opens != 1 {
t.Fatalf("expected exactly one open attempt, got %d", tsp.opens)
}
}
// TestRunCloseFailureIsSafe verifies a Close error at teardown is swallowed:
// the run still returns its normal result.
func TestRunCloseFailureIsSafe(t *testing.T) {
inner := &mockProvider{turns: singleAnswerTurns("done")}
session := &fakeTurnSession{inner: inner, closeErr: errors.New("close blew up")}
tsp := &fakeTurnSessionProvider{session: session}
result, err := Run(context.Background(), "go", inner, Options{
MaxTurns: 2,
TurnSessionProvider: tsp,
})
if err != nil {
t.Fatalf("expected Close failure to be swallowed, got %v", err)
}
if result.FinalAnswer != "done" {
t.Fatalf("expected normal result despite Close error, got %q", result.FinalAnswer)
}
if session.closes != 1 {
t.Fatalf("expected exactly one Close at teardown, got %d", session.closes)
}
}
// TestRunStreamsThroughTurnSession verifies that when a TurnSessionProvider is
// wired, every model request of the run flows through the session (and the
// session is prewarmed once and closed once).
func TestRunStreamsThroughTurnSession(t *testing.T) {
inner := &mockProvider{turns: singleAnswerTurns("via session")}
session := &fakeTurnSession{inner: inner}
tsp := &fakeTurnSessionProvider{session: session}
// The positional provider is a DIFFERENT mock: it must stay untouched,
// proving the session (not the positional provider) carried the run.
positional := &mockProvider{turns: singleAnswerTurns("wrong path")}
result, err := Run(context.Background(), "go", positional, Options{
MaxTurns: 2,
TurnSessionProvider: tsp,
})
if err != nil {
t.Fatal(err)
}
if result.FinalAnswer != "via session" {
t.Fatalf("expected the session-backed answer, got %q", result.FinalAnswer)
}
if len(positional.requests) != 0 {
t.Fatalf("expected the positional provider to be bypassed, got %d requests", len(positional.requests))
}
if session.streams == 0 || session.streams != len(inner.requests) {
t.Fatalf("expected every request to flow through the session: streams=%d inner=%d", session.streams, len(inner.requests))
}
if session.prewarms != 1 {
t.Fatalf("expected exactly one prewarm, got %d", session.prewarms)
}
if session.closes != 1 {
t.Fatalf("expected exactly one close, got %d", session.closes)
}
}
// TestRunDefaultTurnSessionMatchesRawProvider verifies the default (nil
// TurnSessionProvider) path is behavior-identical to passing an explicit
// default adapter over the same provider: same result, same request count.
func TestRunDefaultTurnSessionMatchesRawProvider(t *testing.T) {
run := func(explicit bool) (Result, *mockProvider) {
provider := &mockProvider{turns: escalateThenAnswerTurns("identical")}
registry := tools.NewRegistry()
registry.Register(escalatingTool{target: ""})
options := Options{Registry: registry, MaxTurns: 4}
if explicit {
options.TurnSessionProvider = zeroruntime.NewProviderTurnSessionProvider(provider, zeroruntime.ProviderCapabilities{})
}
result, err := Run(context.Background(), "go", provider, options)
if err != nil {
t.Fatal(err)
}
return result, provider
}
defaultResult, defaultProvider := run(false)
explicitResult, explicitProvider := run(true)
if defaultResult.FinalAnswer != explicitResult.FinalAnswer {
t.Fatalf("final answers diverged: default=%q explicit=%q", defaultResult.FinalAnswer, explicitResult.FinalAnswer)
}
if defaultResult.Turns != explicitResult.Turns {
t.Fatalf("turn counts diverged: default=%d explicit=%d", defaultResult.Turns, explicitResult.Turns)
}
if len(defaultProvider.requests) != len(explicitProvider.requests) {
t.Fatalf("request counts diverged: default=%d explicit=%d", len(defaultProvider.requests), len(explicitProvider.requests))
}
}
// TestRunModelSwitchClosesSessionAndContinues verifies a mid-run model switch
// closes the active session exactly once and streams the rest of the run on
// the new provider.
func TestRunModelSwitchClosesSessionAndContinues(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(escalatingTool{target: "claude-opus-4.1"})
firstInner := &mockProvider{turns: escalateThenAnswerTurns("unused")}
firstSession := &fakeTurnSession{inner: firstInner}
tsp := &fakeTurnSessionProvider{session: firstSession}
secondProvider := &mockProvider{turns: singleAnswerTurns("answered after switch")}
result, err := Run(context.Background(), "go", firstInner, Options{
Registry: registry,
Model: "claude-sonnet-4.5",
MaxTurns: 4,
TurnSessionProvider: tsp,
ModelSwitcher: func(_ context.Context, _ string) (Provider, error) {
return secondProvider, nil
},
})
if err != nil {
t.Fatal(err)
}
if result.FinalAnswer != "answered after switch" {
t.Fatalf("expected the post-switch answer, got %q", result.FinalAnswer)
}
// The original session handled exactly the escalation turn, then was closed
// once by the swap (the teardown defer closes the NEW session, not this one).
if firstSession.streams != 1 {
t.Fatalf("expected the original session to stream exactly the escalation turn, got %d", firstSession.streams)
}
if firstSession.closes != 1 {
t.Fatalf("expected the swap to close the original session exactly once, got %d", firstSession.closes)
}
if len(secondProvider.requests) != 1 {
t.Fatalf("expected the post-switch turn on the new provider, got %d requests", len(secondProvider.requests))
}
}
// TestRunModelSessionSwitcherCarriesOptimizedSession verifies the target-aware
// switcher: the switched-to TurnSessionProvider's DISTINCT session receives the
// post-switch streams, is prewarmed exactly once, and is closed exactly once at
// teardown — and the legacy ModelSwitcher is never consulted when both are set.
func TestRunModelSessionSwitcherCarriesOptimizedSession(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(escalatingTool{target: "claude-opus-4.1"})
firstInner := &mockProvider{turns: escalateThenAnswerTurns("unused")}
firstSession := &fakeTurnSession{inner: firstInner}
initial := &fakeTurnSessionProvider{session: firstSession}
switchedInner := &mockProvider{turns: singleAnswerTurns("answered on the switched session")}
switchedSession := &fakeTurnSession{inner: switchedInner}
switched := &fakeTurnSessionProvider{session: switchedSession}
var switchedTo string
legacyCalled := false
result, err := Run(context.Background(), "go", firstInner, Options{
Registry: registry,
Model: "claude-sonnet-4.5",
MaxTurns: 4,
TurnSessionProvider: initial,
ModelSessionSwitcher: func(_ context.Context, modelID string) (zeroruntime.TurnSessionProvider, error) {
switchedTo = modelID
return switched, nil
},
ModelSwitcher: func(_ context.Context, _ string) (Provider, error) {
legacyCalled = true
return nil, errors.New("legacy switcher must not be consulted")
},
})
if err != nil {
t.Fatal(err)
}
if result.FinalAnswer != "answered on the switched session" {
t.Fatalf("expected the switched session's answer, got %q", result.FinalAnswer)
}
if switchedTo != "claude-opus-4.1" {
t.Fatalf("expected switch to claude-opus-4.1, got %q", switchedTo)
}
if legacyCalled {
t.Fatal("legacy ModelSwitcher was consulted despite ModelSessionSwitcher being set")
}
if switched.opens != 1 {
t.Fatalf("expected exactly one open of the switched provider, got %d", switched.opens)
}
// The switched session carries the rest of the run: it streams the answer
// turn, is prewarmed exactly once at the swap, and is closed exactly once by
// the teardown defer.
if switchedSession.streams != 1 || len(switchedInner.requests) != 1 {
t.Fatalf("expected the switched session to stream the post-switch turn, streams=%d inner=%d", switchedSession.streams, len(switchedInner.requests))
}
if switchedSession.prewarms != 1 {
t.Fatalf("expected the switched session to be prewarmed exactly once, got %d", switchedSession.prewarms)
}
if switchedSession.closes != 1 {
t.Fatalf("expected the switched session to be closed exactly once at teardown, got %d", switchedSession.closes)
}
// The original session ends with the swap: one stream, one close.
if firstSession.streams != 1 || firstSession.closes != 1 {
t.Fatalf("expected the original session to end at the swap (streams=%d closes=%d)", firstSession.streams, firstSession.closes)
}
}
// TestRunModelSessionSwitcherOpenFailureIsNonFatal verifies the swap-time open
// failure: the switcher returns a TurnSessionProvider whose OpenTurnSession
// fails. The run stays non-fatal on the ORIGINAL session (closed only at
// teardown), and the switch-failure note reaches the next request.
func TestRunModelSessionSwitcherOpenFailureIsNonFatal(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(escalatingTool{target: "claude-opus-4.1"})
inner := &mockProvider{turns: escalateThenAnswerTurns("recovered after failed open")}
session := &fakeTurnSession{inner: inner}
failingOpen := &fakeTurnSessionProvider{openErr: errors.New("switched handshake refused")}
result, err := Run(context.Background(), "go", inner, Options{
Registry: registry,
Model: "claude-sonnet-4.5",
MaxTurns: 4,
TurnSessionProvider: &fakeTurnSessionProvider{session: session},
ModelSessionSwitcher: func(_ context.Context, _ string) (zeroruntime.TurnSessionProvider, error) {
return failingOpen, nil
},
})
if err != nil {
t.Fatalf("expected a failed swap open to be non-fatal, got %v", err)
}
if result.FinalAnswer != "recovered after failed open" {
t.Fatalf("expected the run to recover on the original session, got %q", result.FinalAnswer)
}
if failingOpen.opens != 1 {
t.Fatalf("expected exactly one open attempt on the switched provider, got %d", failingOpen.opens)
}
// The original session serves both turns and is closed only at teardown —
// never by the failed swap.
if session.streams != 2 || session.closes != 1 {
t.Fatalf("expected the original session to serve both turns and close once (streams=%d closes=%d)", session.streams, session.closes)
}
var sawNote bool
for _, message := range inner.requests[1].Messages {
if message.Role == zeroruntime.MessageRoleUser && strings.Contains(strings.ToLower(message.Content), "could not switch") {
sawNote = true
}
}
if !sawNote {
t.Fatalf("expected a switch-failure note on the next turn, messages: %+v", inner.requests[1].Messages)
}
}
// TestRunModelSessionSwitcherErrorIsNonFatal verifies a ModelSessionSwitcher
// error keeps the run on the current session with a transcript note — the same
// non-fatal contract as ModelSwitcher.
func TestRunModelSessionSwitcherErrorIsNonFatal(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(escalatingTool{target: "claude-opus-4.1"})
inner := &mockProvider{turns: escalateThenAnswerTurns("recovered on original session")}
session := &fakeTurnSession{inner: inner}
result, err := Run(context.Background(), "go", inner, Options{
Registry: registry,
Model: "claude-sonnet-4.5",
MaxTurns: 4,
TurnSessionProvider: &fakeTurnSessionProvider{session: session},
ModelSessionSwitcher: func(_ context.Context, _ string) (zeroruntime.TurnSessionProvider, error) {
return nil, errors.New("session build blew up")
},
})
if err != nil {
t.Fatalf("expected session-switcher error to be non-fatal, got %v", err)
}
if result.FinalAnswer != "recovered on original session" {
t.Fatalf("expected the run to recover on the original session, got %q", result.FinalAnswer)
}
// Both turns stream through the original session; it closes once at teardown.
if session.streams != 2 || session.closes != 1 {
t.Fatalf("expected the original session to serve both turns and close once (streams=%d closes=%d)", session.streams, session.closes)
}
var sawNote bool
for _, request := range inner.requests[1].Messages {
if request.Role == zeroruntime.MessageRoleUser && strings.Contains(strings.ToLower(request.Content), "could not switch") {
sawNote = true
}
}
if !sawNote {
t.Fatalf("expected a non-fatal switch-failure note on the next turn, messages: %+v", inner.requests[1].Messages)
}
}
+87 -6
View File
@@ -61,10 +61,13 @@ const (
)
type ToolResult struct {
ToolCallID string
Name string
Status tools.Status
Output string
ToolCallID string
Name string
Status tools.Status
Output string
// Truncated reports that the tool's model-visible output omitted content.
// The full result may be recoverable through Meta["spill_path"].
Truncated bool
Meta map[string]string
Redacted bool
ChangedFiles []string
@@ -72,6 +75,12 @@ type ToolResult struct {
// DenialReason categorizes why a tool call was blocked (empty when it ran).
// It lets a surface distinguish the cause precisely instead of parsing Output.
DenialReason DenialCategory
// Risk is the sandbox risk classification of this call, stamped for EXECUTED
// results so run-policy observers (the execution-profile controller) can see
// the risk level of an allowed mutation. It mirrors the classification the
// permission path already computes; denied or canceled results keep the zero
// value. Pure observation: nothing about permissions or sandboxing changes.
Risk sandbox.Risk
// LoadedTools carries the deferred-tool names a tool_search call asked the
// loop to expose next turn (lifted from Meta["load_tools"]). nil for every
// ordinary tool result; only tool_search populates it.
@@ -95,6 +104,54 @@ const (
DenialHookBlocked DenialCategory = "hook_blocked" // vetoed by a beforeTool hook
)
// ProfilePolicy is the loop-facing slice of a selected execution profile.
// The surface (exec flag, TUI command) resolves a named profile into this
// policy; the loop only ever sees the policy, never the catalog.
type ProfilePolicy struct {
// Name labels trace counters and diagnostics (e.g. "fast"). The trace
// recorder's profile label is set by the caller at recorder construction.
Name string
// Escalate, when non-nil, arms one-shot in-run posture escalation.
Escalate *PostureEscalation
}
// PostureEscalation describes a one-shot escalation to stricter knob values,
// applied mid-run when an armed trigger fires. Targets are the values the
// selected profile DISPLACED at run start (i.e. "restore the balanced
// posture"), so escalation can never introduce a value that was not already
// valid for this run and model. Zero-valued targets leave that knob untouched.
type PostureEscalation struct {
// MaxTurns raises the turn ceiling to this value when greater than the
// ceiling in effect. 0 leaves the ceiling untouched.
MaxTurns int
// ReasoningEffort replaces the run's effort when non-empty.
ReasoningEffort string
// RestoreDefaultEffort clears the run's effort override (back to the
// provider/model default) when true. It exists because the displaced value
// of a profile-filled effort is "" — which as a ReasoningEffort target
// means "leave untouched" — so restoring the default needs its own signal.
// Ignored when ReasoningEffort is non-empty.
RestoreDefaultEffort bool
// RestoreCompletionGate re-enables RequireCompletionSignal (headless
// completion semantics) when true.
RestoreCompletionGate bool
// Triggers. A zero value disables that signal entirely.
// OnToolFailureStreak fires when the repeated-failure guard observes a
// same-tool retriable-failure streak of at least this length.
OnToolFailureStreak int
// OnCompletionUncertain fires on the Nth uncertain completion evaluation
// (continue nudge or semantic check). Headless only: the completion gate
// never runs interactively.
OnCompletionUncertain int
// OnSelfCorrectFailure fires when a post-edit verification cycle reports a
// failing outcome (correcting, reported, or aborted).
OnSelfCorrectFailure bool
// OnRiskyMutation fires when an EXECUTED tool result carries a sandbox risk
// level at or above this threshold. Empty disables the signal.
OnRiskyMutation sandbox.RiskLevel
}
type PermissionRequest struct {
ToolCallID string `json:"toolCallId"`
ToolName string `json:"name"`
@@ -280,6 +337,27 @@ type Options struct {
// request), so every existing caller is unaffected. A returned error is
// non-fatal: the run continues on the current model.
ModelSwitcher func(ctx context.Context, modelID string) (Provider, error)
// TurnSessionProvider, when set, supplies the turn session the run streams
// through — the seam an optimized provider session (connection reuse,
// prewarm, native compaction) plugs into without touching the loop. nil
// keeps the default: the loop wraps the passed provider in a no-op session
// whose Stream IS provider.StreamCompletion, so behavior is byte-identical
// and every existing caller is unaffected.
TurnSessionProvider zeroruntime.TurnSessionProvider
// ModelSessionSwitcher, when set, is the target-aware escalation hook: the
// loop prefers it over ModelSwitcher, and its TurnSessionProvider keeps an
// optimized session (and its capabilities) across a mid-run model switch.
// nil falls back to ModelSwitcher, whose bare Provider is wrapped in the
// default no-op session — today's behavior, unchanged. Same non-fatal error
// contract as ModelSwitcher: a returned error records a note and the run
// continues on the current model and session.
ModelSessionSwitcher func(ctx context.Context, modelID string) (zeroruntime.TurnSessionProvider, error)
// Profile, when set, arms the execution-profile posture controller for this
// run (auto-escalation to stricter knob values on failure/uncertainty/risky
// mutation signals). nil — the default everywhere today — leaves the loop
// byte-identical: no observation, no escalation, no counters. Same opt-in
// convention as Trace and SelfCorrect.
Profile *ProfilePolicy
// Trace, when set, records per-turn timing for the run: the loop stamps
// spans (prompt build, generation, tool execution, permission wait,
// compaction, provider connect) and counters (model requests, tool calls,
@@ -312,8 +390,11 @@ type Options struct {
// continuation cue ("…Let me check the config:"). The loop then nudges the
// model to continue instead, bounded by maxContinueNudges (and still by
// MaxTurns and the run deadline); if the model keeps stalling, the run
// finalizes as INCOMPLETE (Result.Incomplete) rather than success. Default
// false leaves the loop byte-identical, so the interactive TUI is unaffected.
// finalizes as INCOMPLETE (Result.Incomplete) rather than success. When the
// run's profile also enables SelfCorrect, an otherwise-complete turn gets one
// task-grounded semantic check before success; profiles without SelfCorrect
// add no model call. Default false leaves the loop byte-identical, so the
// interactive TUI is unaffected.
RequireCompletionSignal bool
runPermissions *permissionRunState
+259
View File
@@ -0,0 +1,259 @@
package aimlapi
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"time"
)
// PaymentMethod selects how a partner-checkout top-up is paid for.
type PaymentMethod string
const (
PaymentMethodCard PaymentMethod = "card"
PaymentMethodCrypto PaymentMethod = "crypto"
)
// SessionStatus is the lifecycle state of a partner-checkout session, as reported
// by the aimlapi.com backend while polling for payment.
type SessionStatus string
const (
SessionStatusPendingAuth SessionStatus = "pending_auth"
SessionStatusPendingPayment SessionStatus = "pending_payment"
SessionStatusPaid SessionStatus = "paid"
SessionStatusExchanging SessionStatus = "exchanging"
SessionStatusExchanged SessionStatus = "exchanged"
SessionStatusCancelled SessionStatus = "cancelled"
SessionStatusExpired SessionStatus = "expired"
SessionStatusFailed SessionStatus = "failed"
)
// AuthResult is the bearer token minted by a passwordless sign-in / sign-up.
type AuthResult struct {
Token string `json:"token"`
Exp int64 `json:"exp"`
}
// PartnerCheckoutSession is a partner-attributed top-up session; its SessionToken
// addresses it on later pay/exchange/poll calls.
type PartnerCheckoutSession struct {
ID string `json:"id"`
SessionToken string `json:"sessionToken"`
PartnerID string `json:"partnerId"`
PartnerName *string `json:"partnerName"`
UserID *int64 `json:"userId"`
AmountUSDMinor *int64 `json:"amountUsdMinor"`
Status SessionStatus `json:"status"`
IssuedKeyID *string `json:"issuedKeyId"`
ReturnURL *string `json:"returnUrl"`
}
// PaymentSession is the hosted payment-provider checkout; PayURL is the page the
// user opens to pay.
type PaymentSession struct {
ProviderSessionID string `json:"providerSessionId"`
PayURL string `json:"payUrl"`
}
// PayResult pairs the hosted checkout with the updated partner-checkout session
// returned when a top-up payment is initiated.
type PayResult struct {
Checkout PaymentSession `json:"checkout"`
PartnerCheckout PartnerCheckoutSession `json:"partnerCheckout"`
}
// ExchangeResult holds the API key minted when a paid session is exchanged.
type ExchangeResult struct {
APIKey string `json:"apiKey"`
APIKeyID string `json:"apiKeyId"`
}
// TopUpByKeyRequest funds a partner-checkout session with the API key that owns
// the account (Path A / env key), instead of an email session.
type TopUpByKeyRequest struct {
SessionToken string
AmountUSDMinor int
PaymentSessionID string // idempotency: same id → same checkout, never a second charge
AutoTopUp bool // enroll the account in automatic top-up at checkout (card, saved off-session)
SuccessURL string
CancelURL string
}
// TopUpByKeyResult is the hosted checkout returned by TopUpByKey.
type TopUpByKeyResult struct {
Checkout PaymentSession `json:"checkout"`
}
// APIError is a non-2xx HTTP response from the aimlapi.com API. Status carries the
// code so callers can branch (e.g. 401 = bad key, 5xx = retryable).
type APIError struct {
Message string
Status int
Body string
}
// Error renders the method, endpoint, status, and (when present) the response body.
func (e APIError) Error() string {
if strings.TrimSpace(e.Body) != "" {
return fmt.Sprintf("%s: HTTP %d: %s", e.Message, e.Status, strings.TrimSpace(e.Body))
}
return fmt.Sprintf("%s: HTTP %d", e.Message, e.Status)
}
// Client talks to the aimlapi.com auth, app, and inference APIs for the onboarding
// and partner-checkout top-up flows.
type Client struct {
endpoints Endpoints
httpClient *http.Client
}
func NewClient(endpoints Endpoints, httpClient *http.Client) *Client {
if httpClient == nil {
httpClient = &http.Client{Timeout: 60 * time.Second}
}
return &Client{endpoints: endpoints, httpClient: httpClient}
}
// CreateSession opens a partner-checkout session attributed to partnerID; the
// returned SessionToken addresses it on the pay/exchange/poll calls.
func (c *Client) CreateSession(ctx context.Context, partnerID string, partnerName string, returnURL string) (PartnerCheckoutSession, error) {
body := map[string]any{"partnerId": partnerID}
if strings.TrimSpace(partnerName) != "" {
body["partnerName"] = strings.TrimSpace(partnerName)
}
if strings.TrimSpace(returnURL) != "" {
body["returnUrl"] = strings.TrimSpace(returnURL)
}
var result PartnerCheckoutSession
err := c.request(ctx, http.MethodPost, strings.TrimRight(c.endpoints.AppBaseURL, "/")+"/v3/partner-checkout/sessions", "", body, &result)
return result, err
}
// GetSession fetches the current state of a partner-checkout session; used to poll
// for payment completion.
func (c *Client) GetSession(ctx context.Context, sessionToken string) (PartnerCheckoutSession, error) {
var result PartnerCheckoutSession
err := c.request(ctx, http.MethodGet, strings.TrimRight(c.endpoints.AppBaseURL, "/")+"/v3/partner-checkout/sessions/"+urlPathEscape(sessionToken), "", nil, &result)
return result, err
}
// Pay initiates payment for a session and returns the hosted checkout URL.
// autoTopUp enrolls the account in automatic top-up at checkout time.
func (c *Client) Pay(ctx context.Context, bearer string, sessionToken string, amountUSDMinor int, paymentSessionID string, method PaymentMethod, successURL string, cancelURL string, autoTopUp bool) (PayResult, error) {
body := map[string]any{
"amountUsdMinor": amountUSDMinor,
"paymentSessionId": paymentSessionID,
"method": method,
}
if strings.TrimSpace(successURL) != "" {
body["successUrl"] = strings.TrimSpace(successURL)
}
if strings.TrimSpace(cancelURL) != "" {
body["cancelUrl"] = strings.TrimSpace(cancelURL)
}
// Only sent when enabled: enrolls the account in automatic top-up. The backend
// honours this field at checkout time.
if autoTopUp {
body["autoTopUp"] = true
}
var result PayResult
err := c.request(ctx, http.MethodPost, strings.TrimRight(c.endpoints.AppBaseURL, "/")+"/v3/partner-checkout/sessions/"+urlPathEscape(sessionToken)+"/pay", bearer, body, &result)
return result, err
}
// Exchange converts a paid session into a freshly minted API key (new-account flow).
func (c *Client) Exchange(ctx context.Context, bearer string, sessionToken string) (ExchangeResult, error) {
var result ExchangeResult
err := c.request(ctx, http.MethodPost, strings.TrimRight(c.endpoints.AppBaseURL, "/")+"/v3/partner-checkout/sessions/"+urlPathEscape(sessionToken)+"/exchange", bearer, nil, &result)
return result, err
}
// TopUpByKey funds a partner-checkout session using the raw API key that owns the
// account (Path A / env key), returning the hosted checkout. It binds the session
// to the key's account server-side, so no email session is needed and no key is
// exchanged. req.PaymentSessionID makes a retry idempotent (same id → same
// checkout, never a second charge).
func (c *Client) TopUpByKey(ctx context.Context, apiKey string, req TopUpByKeyRequest) (TopUpByKeyResult, error) {
body := map[string]any{
"sessionToken": req.SessionToken,
"amountUsdMinor": req.AmountUSDMinor,
"paymentSessionId": req.PaymentSessionID,
}
if req.AutoTopUp {
body["autoTopUp"] = true
}
if strings.TrimSpace(req.SuccessURL) != "" {
body["successUrl"] = strings.TrimSpace(req.SuccessURL)
}
if strings.TrimSpace(req.CancelURL) != "" {
body["cancelUrl"] = strings.TrimSpace(req.CancelURL)
}
var result TopUpByKeyResult
err := c.request(ctx, http.MethodPost, c.billingV2URL("/billing/topup"), apiKey, body, &result)
return result, err
}
// billingV2URL builds a v2 billing API URL from the inference base. The top-up
// endpoint lives next to the balance endpoint on the API gateway, one version up
// (".../v1" → ".../v2/billing/..."), so it re-anchors the version while keeping any
// host/prefix an override supplied.
func (c *Client) billingV2URL(path string) string {
base := strings.TrimRight(strings.TrimSpace(c.endpoints.InferenceBaseURL), "/")
base = strings.TrimSuffix(base, "/v1")
return strings.TrimRight(base, "/") + "/v2" + path
}
func (c *Client) request(ctx context.Context, method string, endpoint string, bearer string, body any, out any) error {
var reader io.Reader
if body != nil {
data, err := json.Marshal(body)
if err != nil {
return err
}
reader = bytes.NewReader(data)
}
request, err := http.NewRequestWithContext(ctx, method, endpoint, reader)
if err != nil {
return err
}
request.Header.Set("Accept", "application/json")
if body != nil {
request.Header.Set("Content-Type", "application/json")
}
if strings.TrimSpace(bearer) != "" {
request.Header.Set("Authorization", "Bearer "+strings.TrimSpace(bearer))
}
response, err := c.httpClient.Do(request)
if err != nil {
return fmt.Errorf("network request to %s failed: %w", endpoint, err)
}
defer response.Body.Close()
text, readErr := io.ReadAll(io.LimitReader(response.Body, 1<<20))
if readErr != nil {
return readErr
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
return APIError{Message: method + " " + endpoint, Status: response.StatusCode, Body: string(text)}
}
if out == nil {
return nil
}
if len(bytes.TrimSpace(text)) == 0 {
return APIError{Message: method + " " + endpoint + " returned empty body", Status: response.StatusCode}
}
if err := json.Unmarshal(text, out); err != nil {
return APIError{Message: method + " " + endpoint + " returned non-JSON body", Status: response.StatusCode, Body: string(text)}
}
return nil
}
func urlPathEscape(value string) string {
return url.PathEscape(value)
}
+106
View File
@@ -0,0 +1,106 @@
package aimlapi
import (
"context"
"fmt"
"net/http"
"strings"
)
// Additional client methods backing the interactive TUI onboarding flow. They
// target the endpoints shipped in the aimlapi backend:
// - GET {inference}/billing/balance (Path A: validate key + balance)
// - PATCH{auth}/v1/auth/account (Path B: does the account exist?)
// - POST {auth}/v1/auth/sign-in/code(/verify) (Path B existing: passwordless login)
// - POST {auth}/v1/auth/account/passwordless (Path B new: email-only signup)
// - POST {app}/v1/keys (mint a key once we hold a session)
// BalanceResult is the wallet balance returned by GetBalance, with the backend's
// low-balance flag and threshold.
type BalanceResult struct {
Balance float64 `json:"balance"`
LowBalance bool `json:"lowBalance"`
LowBalanceThreshold float64 `json:"lowBalanceThreshold"`
}
// CheckResult reports whether an email already has an account, driving the
// sign-in vs sign-up branch of the onboarding flow.
type CheckResult struct {
// "sign-in" (account exists) or "sign-up" (no account yet).
Action string `json:"action"`
Provider *string `json:"provider"`
}
// CreatedKey is an API key minted by CreateKey.
type CreatedKey struct {
Key string `json:"key"`
ID string `json:"id"`
}
func (c *Client) authURL(path string) string {
return strings.TrimRight(c.endpoints.AuthBaseURL, "/") + path
}
func (c *Client) appURL(path string) string {
return strings.TrimRight(c.endpoints.AppBaseURL, "/") + path
}
func (c *Client) inferenceURL(path string) string {
return strings.TrimRight(c.endpoints.InferenceBaseURL, "/") + path
}
// GetBalance doubles as key validation: a 401 means the key is invalid, a 2xx
// returns the wallet balance. Auth is the raw API key itself.
func (c *Client) GetBalance(ctx context.Context, apiKey string) (BalanceResult, error) {
var result BalanceResult
err := c.request(ctx, http.MethodGet, c.inferenceURL("/billing/balance"), apiKey, nil, &result)
return result, err
}
// CheckAccount reports whether an email already has an aimlapi.com account
// (Action "sign-in") or not ("sign-up").
func (c *Client) CheckAccount(ctx context.Context, email string) (CheckResult, error) {
var result CheckResult
err := c.request(ctx, http.MethodPatch, c.authURL("/v1/auth/account"), "", map[string]any{"email": email}, &result)
return result, err
}
// SendSignInCode emails a 6-digit sign-in code to an existing account.
func (c *Client) SendSignInCode(ctx context.Context, email string) error {
return c.request(ctx, http.MethodPost, c.authURL("/v1/auth/sign-in/code"), "", map[string]any{"email": email}, nil)
}
// VerifySignInCode exchanges an emailed code for a session bearer token.
func (c *Client) VerifySignInCode(ctx context.Context, email string, code string) (AuthResult, error) {
var result AuthResult
err := c.request(ctx, http.MethodPost, c.authURL("/v1/auth/sign-in/code/verify"), "", map[string]any{"email": email, "code": code}, &result)
if err == nil && strings.TrimSpace(result.Token) == "" {
err = fmt.Errorf("aimlapi.com did not return an auth token")
}
return result, err
}
// CreatePasswordlessAccount registers a new email-only account and returns its
// session bearer token.
func (c *Client) CreatePasswordlessAccount(ctx context.Context, email string) (AuthResult, error) {
var result AuthResult
err := c.request(ctx, http.MethodPost, c.authURL("/v1/auth/account/passwordless"), "", map[string]any{"email": email}, &result)
if err == nil && strings.TrimSpace(result.Token) == "" {
err = fmt.Errorf("aimlapi.com did not return an auth token")
}
return result, err
}
// CreateKey mints a fresh API key for a session-holding user.
func (c *Client) CreateKey(ctx context.Context, bearer string, name string) (CreatedKey, error) {
body := map[string]any{}
if strings.TrimSpace(name) != "" {
body["name"] = strings.TrimSpace(name)
}
var result CreatedKey
err := c.request(ctx, http.MethodPost, c.appURL("/v1/keys"), bearer, body, &result)
if err == nil && strings.TrimSpace(result.Key) == "" {
err = fmt.Errorf("aimlapi.com did not return an API key")
}
return result, err
}
+159
View File
@@ -0,0 +1,159 @@
package aimlapi
import (
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"testing"
)
func TestCreateKeySendsAuthenticatedRequestAndDecodesKey(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost || r.URL.Path != "/v1/keys" {
t.Fatalf("request = %s %s, want POST /v1/keys", r.Method, r.URL.Path)
}
if got := r.Header.Get("Authorization"); got != "Bearer session-token" {
t.Fatalf("Authorization = %q", got)
}
var body map[string]string
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Fatal(err)
}
if body["name"] != "Zero CLI" {
t.Fatalf("name = %q", body["name"])
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"key":"sk-issued","id":"key-id"}`))
}))
defer server.Close()
client := NewClient(Endpoints{AppBaseURL: server.URL}, server.Client())
key, err := client.CreateKey(context.Background(), " session-token ", " Zero CLI ")
if err != nil {
t.Fatal(err)
}
if key.Key != "sk-issued" || key.ID != "key-id" {
t.Fatalf("key = %#v", key)
}
}
func TestCreateKeyRejectsEmptyKeyResponse(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"id":"key-id"}`))
}))
defer server.Close()
client := NewClient(Endpoints{AppBaseURL: server.URL}, server.Client())
if _, err := client.CreateKey(context.Background(), "session-token", "Zero"); err == nil {
t.Fatal("CreateKey accepted a response without an API key")
}
}
func TestAuthMethodsRejectEmptyTokenResponses(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"token":""}`))
}))
defer server.Close()
client := NewClient(Endpoints{AuthBaseURL: server.URL}, server.Client())
tests := []struct {
name string
call func() error
}{
{
name: "verify sign-in code",
call: func() error {
_, err := client.VerifySignInCode(context.Background(), "[email protected]", "123456")
return err
},
},
{
name: "create passwordless account",
call: func() error {
_, err := client.CreatePasswordlessAccount(context.Background(), "[email protected]")
return err
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if err := test.call(); err == nil {
t.Fatal("accepted a response without an auth token")
}
})
}
}
func TestClientReturnsTypedAPIError(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
http.Error(w, "invalid key", http.StatusUnauthorized)
}))
defer server.Close()
client := NewClient(Endpoints{InferenceBaseURL: server.URL}, server.Client())
_, err := client.GetBalance(context.Background(), "bad-key")
var apiErr APIError
if !errors.As(err, &apiErr) {
t.Fatalf("error = %T %v, want APIError", err, err)
}
if apiErr.Status != http.StatusUnauthorized || apiErr.Body != "invalid key\n" {
t.Fatalf("APIError = %#v", apiErr)
}
}
func TestTypedResponseRejectsEmptySuccessBody(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
client := NewClient(Endpoints{InferenceBaseURL: server.URL}, server.Client())
if _, err := client.GetBalance(context.Background(), "key"); err == nil {
t.Fatal("GetBalance accepted an empty successful response")
}
}
func TestBodylessOperationAcceptsEmptySuccessBody(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
client := NewClient(Endpoints{AuthBaseURL: server.URL}, server.Client())
if err := client.SendSignInCode(context.Background(), "[email protected]"); err != nil {
t.Fatalf("SendSignInCode() error = %v", err)
}
}
func TestParseAmountUSD(t *testing.T) {
tests := []struct {
name string
value string
want int
wantErr bool
}{
{name: "default", want: DefaultAmountUSDMinor},
{name: "minimum", value: "20", want: 2000},
{name: "cents", value: "25.49", want: 2549},
{name: "below minimum", value: "19.99", wantErr: true},
{name: "above maximum", value: "10001", wantErr: true},
{name: "not a number", value: "twenty", wantErr: true},
{name: "nan", value: "NaN", wantErr: true},
{name: "infinity", value: "Inf", wantErr: true},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got, err := ParseAmountUSD(test.value)
if (err != nil) != test.wantErr {
t.Fatalf("ParseAmountUSD(%q) error = %v", test.value, err)
}
if got != test.want {
t.Fatalf("ParseAmountUSD(%q) = %d, want %d", test.value, got, test.want)
}
})
}
}
+113
View File
@@ -0,0 +1,113 @@
package aimlapi
import (
"net/url"
"os"
"strings"
)
type Endpoints struct {
AuthBaseURL string
AppBaseURL string
InferenceBaseURL string
PayBaseURL string
// VerificationBaseURL is the aimlapi web-app frontend base. The browser is sent
// back here after the partner checkout completes (see BuildPartnerReturnURL), so
// it must point at the same environment's front as the one the CLI is talking to
// to land the user there. Empty falls back to DefaultReturnURL.
VerificationBaseURL string
}
const (
DefaultPartnerID = "part_62yQoGYDq4Yqnrj2R1iGrDNJ"
DefaultPartnerName = "Gitlawb"
PartnerHeaderName = "X-AIMLAPI-Partner-ID"
// DefaultReturnURL is the https fallback the browser is sent to after the
// checkout / OAuth consent finishes when no frontend base is known. It must be
// a real web page the browser can open — NOT a custom scheme like
// zero://aimlapi/complete, which fails with "the scheme does not have a
// registered handler" because the CLI registers no OS protocol handler. The
// CLI learns of success by polling; this URL is purely the browser's landing.
DefaultReturnURL = "https://aimlapi.com/app"
DefaultModel = "anthropic/claude-sonnet-5"
MinAmountUSDMinor = 2000
MaxAmountUSDMinor = 1000000
DefaultAmountUSDMinor = 2500
)
func ResolveEndpoints() Endpoints {
return Endpoints{
AuthBaseURL: envOrDefault("AIMLAPI_AUTH_URL", "https://auth.aimlapi.com"),
AppBaseURL: envOrDefault("AIMLAPI_APP_URL", "https://app.aimlapi.com"),
InferenceBaseURL: envOrDefault("AIMLAPI_INFERENCE_URL", "https://api.aimlapi.com/v1"),
PayBaseURL: envOrDefault("AIMLAPI_PAY_URL", "https://pay.aimlapi.com"),
VerificationBaseURL: envOrDefault("AIMLAPI_VERIFICATION_BASE_URL", "https://aimlapi.com/app"),
}
}
// ResolvePartnerID applies the same explicit-option, environment, and default
// precedence everywhere partner attribution is used. Keeping checkout creation
// and saved inference profiles on one resolver prevents them from attributing
// the same user to different partners.
func ResolvePartnerID(explicit string) string {
if value := strings.TrimSpace(explicit); value != "" {
return value
}
return envOrDefault("AIMLAPI_PARTNER_ID", DefaultPartnerID)
}
// WithResolvedPartnerHeader returns a copy with the effective partner ID set.
// Header matching is case-insensitive so an existing spelling is replaced rather
// than duplicated. Callers remain responsible for applying catalog attribution
// only to the canonical AIMLAPI endpoint.
func WithResolvedPartnerHeader(headers map[string]string) map[string]string {
resolved := make(map[string]string, len(headers)+1)
for key, value := range headers {
if strings.EqualFold(strings.TrimSpace(key), PartnerHeaderName) {
continue
}
resolved[key] = value
}
resolved[PartnerHeaderName] = ResolvePartnerID("")
return resolved
}
func BuildPartnerCheckoutReturnURLs(appBaseURL string, sessionToken string) (successURL string, cancelURL string) {
base := strings.TrimRight(strings.TrimSpace(appBaseURL), "/")
if base == "" {
// With no base there is nothing to anchor an absolute URL to. Return empty
// so the caller omits the return URLs entirely (client.Pay skips empty ones,
// leaving the backend's generic checkout) instead of emitting a broken
// relative "/checkout?...".
return "", ""
}
token := url.QueryEscape(sessionToken)
return base + "/checkout?checkout=success&partnerCheckout=1&sessionToken=" + token,
base + "/checkout?checkout=cancel&partnerCheckout=1&sessionToken=" + token
}
// BuildPartnerReturnURL returns the https page the browser is sent back to once
// the partner checkout or OAuth consent completes — the aimlapi web app the user
// opened the flow from. frontendBaseURL is normally endpoints.VerificationBaseURL
// (the front that hosts the login/consent page), so the return follows the same
// environment. AIMLAPI_RETURN_URL overrides it outright; an empty base falls back
// to DefaultReturnURL. It is deliberately NOT a custom scheme (zero://…): no
// browser can hand a custom scheme off without an OS-registered handler, which a
// CLI does not install.
func BuildPartnerReturnURL(frontendBaseURL string) string {
if override := strings.TrimSpace(os.Getenv("AIMLAPI_RETURN_URL")); override != "" {
return override
}
if base := strings.TrimRight(strings.TrimSpace(frontendBaseURL), "/"); base != "" {
return base
}
return DefaultReturnURL
}
func envOrDefault(name string, fallback string) string {
if value := strings.TrimSpace(os.Getenv(name)); value != "" {
return value
}
return fallback
}
+55
View File
@@ -0,0 +1,55 @@
package aimlapi
import "testing"
func TestResolveEndpointsDefaults(t *testing.T) {
t.Setenv("AIMLAPI_AUTH_URL", "")
t.Setenv("AIMLAPI_APP_URL", "")
t.Setenv("AIMLAPI_INFERENCE_URL", "")
t.Setenv("AIMLAPI_PAY_URL", "")
endpoints := ResolveEndpoints()
if endpoints.AuthBaseURL != "https://auth.aimlapi.com" ||
endpoints.AppBaseURL != "https://app.aimlapi.com" ||
endpoints.InferenceBaseURL != "https://api.aimlapi.com/v1" ||
endpoints.PayBaseURL != "https://pay.aimlapi.com" {
t.Fatalf("unexpected endpoints: %#v", endpoints)
}
}
func TestResolvedPartnerHeaderUsesEnvironmentOverrideWithoutDuplicates(t *testing.T) {
t.Setenv("AIMLAPI_PARTNER_ID", "part_override")
headers := WithResolvedPartnerHeader(map[string]string{
"x-aimlapi-partner-id": "part_catalog",
"X-AIMLAPI-Integration-Repo": "Gitlawb/zero",
})
if got := headers[PartnerHeaderName]; got != "part_override" {
t.Fatalf("partner header = %q, want part_override", got)
}
if _, ok := headers["x-aimlapi-partner-id"]; ok {
t.Fatalf("case-insensitive partner header was duplicated: %#v", headers)
}
if got := headers["X-AIMLAPI-Integration-Repo"]; got != "Gitlawb/zero" {
t.Fatalf("unrelated header changed: %#v", headers)
}
}
func TestBuildPartnerReturnURL(t *testing.T) {
t.Setenv("AIMLAPI_RETURN_URL", "")
// The frontend base (the web app the flow was opened from) is returned as-is,
// trailing slash trimmed — an https page the browser can actually land on.
if got := BuildPartnerReturnURL("https://aimlapi.com/app/"); got != "https://aimlapi.com/app" {
t.Fatalf("return URL = %q, want the frontend base", got)
}
// An empty base falls back to the https default, never a custom scheme.
if got := BuildPartnerReturnURL(""); got != DefaultReturnURL {
t.Fatalf("empty base return URL = %q, want %q", got, DefaultReturnURL)
}
t.Setenv("AIMLAPI_RETURN_URL", "https://example.test/done")
if got := BuildPartnerReturnURL("https://aimlapi.com/app"); got != "https://example.test/done" {
t.Fatalf("env override return URL = %q", got)
}
}
+42
View File
@@ -0,0 +1,42 @@
package aimlapi
// UI strings shared by the first-run and provider-wizard onboarding surfaces.
const (
// Path A — existing API key. Pick-path prompt + option labels.
MsgAPIKeyInputPrompt = "Enter your aimlapi.com key."
MsgAPIKeyInvalid = "API key is invalid. Please make sure you enter a valid aimlapi.com key."
MsgPickPathPrompt = "Do you have aimlapi.com key?"
MsgPickPathHaveKey = "I already have aimlapi.com key"
MsgPickPathHaveHint = "Proceed to paste the key"
MsgPickPathNewUser = "I am a new user"
MsgPickPathNewHint = "One click set up"
// Path B — email.
MsgEnterEmail = "Enter your email.\nTo access aimlapi.com dashboard."
MsgEmailInvalid = "Email format is incorrect."
MsgAccountActionInvalid = "aimlapi.com returned an unsupported account action. Please try again."
// %s = email
MsgCodeSent = "We sent a 6-digit code to %s.\nEnter it below to continue."
MsgCodeIncorrect = "Code you've entered is incorrect."
// Balance check.
MsgLowBalance = "Your aimlapi.com balance is running low.\nIt is recommended to top up your balance."
MsgEverythingRuns = "Everything is ready."
// Low-balance choices (spec wording).
MsgLowBalanceTopUp = "Sure, let's do that"
MsgLowBalanceSkip = "I'll skip topping up the balance for now"
// Top-up workflow.
MsgTopUpPrompt = "Add credits.\nEnter an amount (min $20)."
MsgAmountRequired = "Please enter a top-up amount."
// %s = checkout URL
MsgTopUpBrowserFallback = "If the browser did not open automatically please use this link to top up your account: %s"
MsgTopUpFailed = "Top up failed. Please try again."
MsgTopUpSuccess = "Top-up successful."
// Success — key delivered.
// %s = amount in dollars (e.g. "25")
MsgTopUpAddedFmt = "$%s has been added to your balance."
// %s = email
MsgSuccessMagicLink = "We've emailed you a magic link to %s. Use it to access your aimlapi.com account and review your usage and balance."
)
+26
View File
@@ -0,0 +1,26 @@
package aimlapi
import "testing"
func TestValidEmailRejectsIncompleteDomains(t *testing.T) {
tests := []struct {
value string
valid bool
}{
{value: "[email protected]", valid: true},
{value: "[email protected]", valid: true},
{value: "123@123.", valid: false},
{value: "123@123", valid: false},
{value: "[email protected]", valid: false},
{value: "[email protected]", valid: false},
{value: "[email protected]", valid: false},
{value: "[email protected]", valid: false},
}
for _, test := range tests {
t.Run(test.value, func(t *testing.T) {
if got := ValidEmail(test.value); got != test.valid {
t.Fatalf("ValidEmail(%q) = %v, want %v", test.value, got, test.valid)
}
})
}
}
+335
View File
@@ -0,0 +1,335 @@
package aimlapi
import (
"context"
"fmt"
"net/http"
"net/url"
"strings"
)
// StreamTopUpOptions drives a session-based partner-checkout top-up for the
// interactive (TUI) onboarding. The caller already holds a user session —
// obtained from an email-code sign-in (Path B existing) or a
// passwordless new account (Path B sign-up) — so this skips authentication and
// runs create-session → pay → wait-for-payment, optionally exchanging the paid
// session into a fresh key (new accounts) versus keeping the caller's key.
type StreamTopUpOptions struct {
SessionToken string // user session bearer (required)
AmountUSD string // dollars; parsed via ParseAmountUSD (min $20)
// InferenceBaseURL pins key-bound billing and the returned provider profile to
// the endpoint that validated the credential. Empty uses ResolveEndpoints.
InferenceBaseURL string
Method PaymentMethod
AutoTopUp bool // enroll the account in automatic top-up at checkout time
Exchange bool // exchange the paid session into a new key (new accounts)
Model string
PartnerID string
PartnerName string
NoOpen bool
// ResumeSessionToken, when set, resumes that partner-checkout session instead
// of opening a new one, so a retry after an ambiguous failure (a lost Pay/
// Exchange response) can never double-charge or strand an already-paid session.
ResumeSessionToken string
// PaymentSessionID is generated once per amount/auto-top-up intent and reused
// on retry so an ambiguous Pay response cannot create a second checkout.
PaymentSessionID string
HTTPClient *http.Client
OpenBrowser func(string) error
OnStatus func(Status, string)
// OnSession reports the partner-checkout session token the moment it exists (a
// fresh CreateSession, or the resumed one), so the caller can retain it and
// resume on a later failure. It fires with "" when the session is dead and the
// retained token should be dropped (a fresh run must start over).
OnSession func(sessionToken string)
}
// topupPhase is where StreamTopUp enters its create-checkout → poll → exchange
// pipeline. A fresh run starts at phasePay; a resume skips ahead to whatever the
// existing session's status makes safe.
type topupPhase int
const (
phasePay topupPhase = iota // (re-)open the Stripe checkout, then poll + exchange
phasePoll // checkout already open; poll for payment, never re-pay
phaseExchange // already paid; exchange only (recovers a lost key)
phaseWaitExchange // an exchange claim is in flight; poll, never call Exchange twice
)
// StreamTopUp performs the browser-checkout top-up and reports progress via
// OnStatus.
func StreamTopUp(ctx context.Context, options StreamTopUpOptions) (ProvisionedKey, error) {
endpoints := ResolveEndpoints()
if baseURL := strings.TrimSpace(options.InferenceBaseURL); baseURL != "" {
endpoints.InferenceBaseURL = baseURL
}
client := NewClient(endpoints, options.HTTPClient)
partnerID := ResolvePartnerID(options.PartnerID)
partnerName := firstNonEmpty(options.PartnerName, DefaultPartnerName)
method := options.Method
if method != PaymentMethodCrypto {
method = PaymentMethodCard
}
amount, err := ParseAmountUSD(options.AmountUSD)
if err != nil {
return ProvisionedKey{}, err
}
if strings.TrimSpace(options.SessionToken) == "" {
return ProvisionedKey{}, fmt.Errorf("a session is required to top up")
}
if strings.TrimSpace(options.PaymentSessionID) == "" {
return ProvisionedKey{}, fmt.Errorf("a payment session id is required to top up")
}
status(options.OnStatus, StatusCreatingSession, "")
sessionToken, phase, err := resolveTopupSession(ctx, client, options, partnerID, partnerName, endpoints)
if err != nil {
return ProvisionedKey{}, err
}
// Retain the live token so an ambiguous failure below can resume this session
// rather than open a second checkout on the next attempt.
reportSession(options.OnSession, sessionToken)
if phase <= phasePay {
successURL, cancelURL := BuildPartnerCheckoutReturnURLs(endpoints.PayBaseURL, sessionToken)
status(options.OnStatus, StatusOpeningCheckout, "")
pay, err := client.Pay(ctx, options.SessionToken, sessionToken, amount, options.PaymentSessionID, method, successURL, cancelURL, options.AutoTopUp)
if err != nil {
return ProvisionedKey{}, err
}
checkoutURL, err := validatedCheckoutURL(pay.Checkout.PayURL)
if err != nil {
return ProvisionedKey{}, err
}
announceCheckout(options.OnStatus, options.OpenBrowser, options.NoOpen, checkoutURL)
}
paidToken := sessionToken
if phase <= phasePoll {
status(options.OnStatus, StatusWaitingPayment, "")
paid, err := pollUntilPaid(ctx, client, sessionToken, options.OnSession)
if err != nil {
return ProvisionedKey{}, err
}
paidToken = paid.SessionToken
if paid.Status == SessionStatusExchanging {
// Another/lost request claimed exchange while we were polling payment.
// Carry that fact into the provisioning branch so it follows the claim
// instead of issuing a second single-flight Exchange request.
phase = phaseWaitExchange
}
}
result := ProvisionedKey{
BaseURL: endpoints.InferenceBaseURL,
Model: firstNonEmpty(options.Model, DefaultModel),
}
if options.Exchange {
status(options.OnStatus, StatusProvisioningKey, "")
if phase == phaseWaitExchange {
return ProvisionedKey{}, pollUntilExchangeSettled(ctx, client, sessionToken, options.OnSession)
}
exchange, err := client.Exchange(ctx, options.SessionToken, paidToken)
if err != nil {
return ProvisionedKey{}, err
}
if strings.TrimSpace(exchange.APIKey) == "" {
return ProvisionedKey{}, fmt.Errorf("aimlapi.com did not return an API key")
}
result.APIKey = exchange.APIKey
result.APIKeyID = exchange.APIKeyID
}
return result, nil
}
// resolveTopupSession returns the partner-checkout session token to drive and the
// phase to enter. A fresh run opens a new session at phasePay. A resume inspects
// the existing session and enters the earliest phase that cannot re-charge:
// - pending_auth → phasePay: payment never started, so (re-)Pay is safe.
// - pending_payment → phasePoll: a checkout is already open; poll, never re-Pay.
// - paid → phaseExchange: exchange the paid session once.
// - exchanging → phaseWaitExchange: another/lost request owns the atomic
// exchange claim; poll it, never issue a second Exchange request.
// - exchanged → terminal: the one-shot raw key is gone; point at dashboard
// recovery and keep the token so a re-run repeats that guidance, not a charge.
// - cancelled/expired/failed → terminal: drop the dead token so a re-run is fresh.
func resolveTopupSession(ctx context.Context, client *Client, options StreamTopUpOptions, partnerID, partnerName string, endpoints Endpoints) (string, topupPhase, error) {
resume := strings.TrimSpace(options.ResumeSessionToken)
if resume == "" {
session, err := client.CreateSession(ctx, partnerID, partnerName, BuildPartnerReturnURL(endpoints.VerificationBaseURL))
if err != nil {
return "", phasePay, err
}
return session.SessionToken, phasePay, nil
}
session, err := client.GetSession(ctx, resume)
if err != nil {
if isTerminalSessionAPIError(err) {
reportSession(options.OnSession, "")
}
return "", phasePay, err
}
switch session.Status {
case SessionStatusPendingAuth:
return resume, phasePay, nil
case SessionStatusPendingPayment:
return resume, phasePoll, nil
case SessionStatusPaid:
return resume, phaseExchange, nil
case SessionStatusExchanging:
return resume, phaseWaitExchange, nil
case SessionStatusExchanged:
return "", phasePay, fmt.Errorf("session was already exchanged; rotate the key from the aimlapi.com dashboard")
default:
reportSession(options.OnSession, "") // drop the dead token → next run starts fresh
return "", phasePay, fmt.Errorf("payment %s; re-run the top-up to try again", session.Status)
}
}
func reportSession(callback func(string), sessionToken string) {
if callback != nil {
callback(sessionToken)
}
}
// announceCheckout opens the hosted checkout in the browser (unless suppressed)
// and reports the URL either way, so the TUI can always show the direct link.
func announceCheckout(onStatus func(Status, string), open func(string) error, noOpen bool, checkoutURL string) {
if noOpen || open == nil {
status(onStatus, StatusOpeningCheckout, checkoutURL)
return
}
if err := open(checkoutURL); err != nil {
status(onStatus, StatusOpeningCheckout, "Open manually: "+checkoutURL)
return
}
status(onStatus, StatusOpeningCheckout, checkoutURL)
}
func validatedCheckoutURL(value string) (string, error) {
checkoutURL := strings.TrimSpace(value)
parsed, err := url.Parse(checkoutURL)
if err != nil || !parsed.IsAbs() || !strings.EqualFold(parsed.Scheme, "https") || parsed.Hostname() == "" || parsed.User != nil {
return "", fmt.Errorf("payment provider did not return a valid HTTPS checkout URL")
}
return checkoutURL, nil
}
// StreamTopUpByKeyOptions drives a top-up funded with the API key that already
// owns the account (Path A pasted key, or AIMLAPI_API_KEY). Unlike StreamTopUp it
// needs no email session and never exchanges a key — the caller already holds one,
// so it only funds the wallet: create-session → pay-by-key → wait-for-payment.
type StreamTopUpByKeyOptions struct {
APIKey string // raw key that owns the account (required)
AmountUSD string // dollars; parsed via ParseAmountUSD (min $20)
InferenceBaseURL string // validated key endpoint; empty uses ResolveEndpoints
AutoTopUp bool // enroll the account in automatic top-up at checkout time
PartnerID string
PartnerName string
NoOpen bool
// ResumeSessionToken resumes an existing partner-checkout session instead of
// opening a new one. PaymentSessionID is the idempotency handle carried across
// retries so a re-issued pay returns the same checkout, never a second charge.
ResumeSessionToken string
PaymentSessionID string // required; generate once via NewPaymentSessionID, reuse on retry
HTTPClient *http.Client
OpenBrowser func(string) error
OnStatus func(Status, string)
OnSession func(sessionToken string)
}
// StreamTopUpByKey funds the account behind APIKey via the hosted checkout and
// reports progress through OnStatus. It returns an empty ProvisionedKey: the
// pasted/env key is unchanged, only the balance grows.
func StreamTopUpByKey(ctx context.Context, options StreamTopUpByKeyOptions) (ProvisionedKey, error) {
endpoints := ResolveEndpoints()
if baseURL := strings.TrimSpace(options.InferenceBaseURL); baseURL != "" {
endpoints.InferenceBaseURL = baseURL
}
client := NewClient(endpoints, options.HTTPClient)
partnerID := ResolvePartnerID(options.PartnerID)
partnerName := firstNonEmpty(options.PartnerName, DefaultPartnerName)
amount, err := ParseAmountUSD(options.AmountUSD)
if err != nil {
return ProvisionedKey{}, err
}
if strings.TrimSpace(options.APIKey) == "" {
return ProvisionedKey{}, fmt.Errorf("an API key is required to top up")
}
if strings.TrimSpace(options.PaymentSessionID) == "" {
return ProvisionedKey{}, fmt.Errorf("a payment session id is required to top up")
}
status(options.OnStatus, StatusCreatingSession, "")
sessionToken, phase, err := resolveByKeySession(ctx, client, options, partnerID, partnerName, endpoints)
if err != nil {
return ProvisionedKey{}, err
}
reportSession(options.OnSession, sessionToken)
if phase <= phasePay {
successURL, cancelURL := BuildPartnerCheckoutReturnURLs(endpoints.PayBaseURL, sessionToken)
status(options.OnStatus, StatusOpeningCheckout, "")
pay, err := client.TopUpByKey(ctx, options.APIKey, TopUpByKeyRequest{
SessionToken: sessionToken,
AmountUSDMinor: amount,
PaymentSessionID: options.PaymentSessionID,
AutoTopUp: options.AutoTopUp,
SuccessURL: successURL,
CancelURL: cancelURL,
})
if err != nil {
return ProvisionedKey{}, err
}
checkoutURL, err := validatedCheckoutURL(pay.Checkout.PayURL)
if err != nil {
return ProvisionedKey{}, err
}
announceCheckout(options.OnStatus, options.OpenBrowser, options.NoOpen, checkoutURL)
}
if phase <= phasePoll {
status(options.OnStatus, StatusWaitingPayment, "")
if _, err := pollUntilPaid(ctx, client, sessionToken, options.OnSession); err != nil {
return ProvisionedKey{}, err
}
}
// No exchange: the account is funded and the caller keeps its existing key.
return ProvisionedKey{}, nil
}
// resolveByKeySession mirrors resolveTopupSession for the by-key flow. Because
// by-key never exchanges, a paid/exchanging/exchanged session all mean "already
// funded, done" (phaseExchange skips straight past pay + poll to the empty result).
func resolveByKeySession(ctx context.Context, client *Client, options StreamTopUpByKeyOptions, partnerID, partnerName string, endpoints Endpoints) (string, topupPhase, error) {
resume := strings.TrimSpace(options.ResumeSessionToken)
if resume == "" {
session, err := client.CreateSession(ctx, partnerID, partnerName, BuildPartnerReturnURL(endpoints.VerificationBaseURL))
if err != nil {
return "", phasePay, err
}
return session.SessionToken, phasePay, nil
}
session, err := client.GetSession(ctx, resume)
if err != nil {
if isTerminalSessionAPIError(err) {
reportSession(options.OnSession, "")
}
return "", phasePay, err
}
switch session.Status {
case SessionStatusPendingAuth:
return resume, phasePay, nil
case SessionStatusPendingPayment:
return resume, phasePoll, nil
case SessionStatusPaid, SessionStatusExchanging, SessionStatusExchanged:
return resume, phaseExchange, nil
default:
reportSession(options.OnSession, "") // drop the dead token → next run starts fresh
return "", phasePay, fmt.Errorf("payment %s; re-run the top-up to try again", session.Status)
}
}
+457
View File
@@ -0,0 +1,457 @@
package aimlapi
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestStreamTopUpByKeyFundsWithoutExchange(t *testing.T) {
var topupCalls, exchangeCalls int
var topupBody map[string]any
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/partner-checkout/sessions"):
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_tok", Status: SessionStatusPendingAuth})
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/v2/billing/topup"):
topupCalls++
_ = json.NewDecoder(r.Body).Decode(&topupBody)
_ = json.NewEncoder(w).Encode(TopUpByKeyResult{Checkout: PaymentSession{PayURL: "https://pay/checkout"}})
case r.Method == http.MethodGet && strings.Contains(r.URL.Path, "/partner-checkout/sessions/"):
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_tok", Status: SessionStatusPaid})
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/exchange"):
exchangeCalls++
t.Errorf("by-key must never exchange a key")
default:
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
}
}))
defer server.Close()
t.Setenv("AIMLAPI_APP_URL", server.URL)
t.Setenv("AIMLAPI_INFERENCE_URL", server.URL+"/v1")
var opened string
result, err := StreamTopUpByKey(context.Background(), StreamTopUpByKeyOptions{
APIKey: "sk-user",
AmountUSD: "20",
AutoTopUp: true,
PaymentSessionID: "pay-1",
OpenBrowser: func(u string) error { opened = u; return nil },
})
if err != nil {
t.Fatalf("StreamTopUpByKey = %v", err)
}
if result.APIKey != "" {
t.Fatalf("by-key must not mint a key, got %q", result.APIKey)
}
if topupCalls != 1 || exchangeCalls != 0 {
t.Fatalf("topup=%d exchange=%d, want 1/0", topupCalls, exchangeCalls)
}
if opened != "https://pay/checkout" {
t.Fatalf("opened checkout = %q", opened)
}
if topupBody["autoTopUp"] != true || topupBody["paymentSessionId"] != "pay-1" {
t.Fatalf("topup body missing autoTopUp/paymentSessionId: %#v", topupBody)
}
}
func TestValidatedCheckoutURLRequiresAbsoluteHTTPS(t *testing.T) {
tests := []struct {
name string
value string
want string
wantErr bool
}{
{name: "https", value: " https://pay.example.test/checkout?id=1 ", want: "https://pay.example.test/checkout?id=1"},
{name: "empty", wantErr: true},
{name: "relative", value: "/checkout", wantErr: true},
{name: "http", value: "http://pay.example.test/checkout", wantErr: true},
{name: "file", value: "file:///tmp/checkout", wantErr: true},
{name: "custom protocol", value: "wallet://checkout/123", wantErr: true},
{name: "userinfo", value: "https://[email protected]/checkout", wantErr: true},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got, err := validatedCheckoutURL(test.value)
if (err != nil) != test.wantErr {
t.Fatalf("validatedCheckoutURL(%q) error = %v", test.value, err)
}
if got != test.want {
t.Fatalf("validatedCheckoutURL(%q) = %q, want %q", test.value, got, test.want)
}
})
}
}
func TestStreamTopUpByKeyPinsValidatedInferenceEndpoint(t *testing.T) {
validatedCalls := 0
validated := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/partner-checkout/sessions"):
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_tok", Status: SessionStatusPendingAuth})
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/v2/billing/topup"):
validatedCalls++
_ = json.NewEncoder(w).Encode(TopUpByKeyResult{Checkout: PaymentSession{PayURL: "https://pay/checkout"}})
case r.Method == http.MethodGet && strings.Contains(r.URL.Path, "/partner-checkout/sessions/"):
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_tok", Status: SessionStatusPaid})
default:
t.Errorf("unexpected validated-endpoint request: %s %s", r.Method, r.URL.Path)
}
}))
defer validated.Close()
override := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Errorf("validated key leaked to process override: %s %s", r.Method, r.URL.Path)
}))
defer override.Close()
t.Setenv("AIMLAPI_APP_URL", validated.URL)
t.Setenv("AIMLAPI_INFERENCE_URL", override.URL)
_, err := StreamTopUpByKey(context.Background(), StreamTopUpByKeyOptions{
APIKey: "production-key",
AmountUSD: "20",
InferenceBaseURL: validated.URL,
PaymentSessionID: "payment-1",
OpenBrowser: func(string) error { return nil },
})
if err != nil {
t.Fatalf("StreamTopUpByKey() error = %v", err)
}
if validatedCalls != 1 {
t.Fatalf("validated endpoint top-up calls = %d, want 1", validatedCalls)
}
}
func TestStreamTopUpEmailRetrySendsStablePaymentSessionID(t *testing.T) {
getCalls := 0
var paymentSessionID string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && strings.Contains(r.URL.Path, "/partner-checkout/sessions/"):
status := SessionStatusPendingAuth
if getCalls > 0 {
status = SessionStatusPaid
}
getCalls++
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_retry", Status: status})
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/pay"):
var body map[string]any
_ = json.NewDecoder(r.Body).Decode(&body)
paymentSessionID, _ = body["paymentSessionId"].(string)
_ = json.NewEncoder(w).Encode(PayResult{Checkout: PaymentSession{PayURL: "https://pay/checkout"}})
default:
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
}
}))
defer server.Close()
t.Setenv("AIMLAPI_APP_URL", server.URL)
_, err := StreamTopUp(context.Background(), StreamTopUpOptions{
SessionToken: "bearer",
ResumeSessionToken: "pcs_retry",
PaymentSessionID: "payment-stable",
AmountUSD: "20",
NoOpen: true,
})
if err != nil {
t.Fatalf("StreamTopUp() error = %v", err)
}
if paymentSessionID != "payment-stable" {
t.Fatalf("paymentSessionId = %q, want stable retry id", paymentSessionID)
}
}
// resumeServer is a partner-checkout backend fake that fails the test if the
// re-charging endpoints (create-session / pay) are ever hit, and serves a
// scripted GetSession status sequence so a resume can be driven deterministically.
type resumeServer struct {
t *testing.T
statuses []string // GetSession status per call, last value repeats
getCalls int
payCalls int
createCall int
exchanged int
}
func (rs *resumeServer) handler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/partner-checkout/sessions"):
rs.createCall++
rs.t.Errorf("resume must not create a new session")
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/pay"):
rs.payCalls++
rs.t.Errorf("resume must not re-pay (double charge)")
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/exchange"):
rs.exchanged++
_ = json.NewEncoder(w).Encode(ExchangeResult{APIKey: "sk-resumed", APIKeyID: "key_1"})
case r.Method == http.MethodGet:
status := rs.statuses[len(rs.statuses)-1]
if rs.getCalls < len(rs.statuses) {
status = rs.statuses[rs.getCalls]
}
rs.getCalls++
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pc_tok", Status: SessionStatus(status)})
default:
rs.t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
}
})
}
func TestStreamTopUpResumePaidExchangesWithoutRecharge(t *testing.T) {
rs := &resumeServer{t: t, statuses: []string{"paid"}}
server := httptest.NewServer(rs.handler())
defer server.Close()
t.Setenv("AIMLAPI_APP_URL", server.URL)
var seen string
result, err := StreamTopUp(context.Background(), StreamTopUpOptions{
SessionToken: "bearer",
ResumeSessionToken: "pc_tok",
PaymentSessionID: "payment-1",
AmountUSD: "20",
Exchange: true,
NoOpen: true,
OnSession: func(token string) { seen = token },
})
if err != nil {
t.Fatalf("StreamTopUp resume = %v", err)
}
if result.APIKey != "sk-resumed" {
t.Fatalf("APIKey = %q, want recovered key", result.APIKey)
}
if rs.exchanged != 1 {
t.Fatalf("exchange calls = %d, want 1", rs.exchanged)
}
if seen != "pc_tok" {
t.Fatalf("OnSession token = %q, want retained pc_tok", seen)
}
}
func TestStreamTopUpResumePendingPaymentPollsNeverRepays(t *testing.T) {
// First GetSession (resume resolution) sees the still-open checkout; the next
// (poll) sees it settle. Pay must never be called — the fake fails the test if
// it is.
rs := &resumeServer{t: t, statuses: []string{"pending_payment", "paid"}}
server := httptest.NewServer(rs.handler())
defer server.Close()
t.Setenv("AIMLAPI_APP_URL", server.URL)
result, err := StreamTopUp(context.Background(), StreamTopUpOptions{
SessionToken: "bearer",
ResumeSessionToken: "pc_tok",
PaymentSessionID: "payment-1",
AmountUSD: "20",
Exchange: true,
NoOpen: true,
})
if err != nil {
t.Fatalf("StreamTopUp resume = %v", err)
}
if result.APIKey != "sk-resumed" {
t.Fatalf("APIKey = %q, want recovered key", result.APIKey)
}
if rs.payCalls != 0 {
t.Fatalf("pay calls = %d, want 0", rs.payCalls)
}
}
func TestStreamTopUpPollingDeadSessionClearsRetainedToken(t *testing.T) {
rs := &resumeServer{t: t, statuses: []string{"pending_payment", "expired"}}
server := httptest.NewServer(rs.handler())
defer server.Close()
t.Setenv("AIMLAPI_APP_URL", server.URL)
seen := []string{}
_, err := StreamTopUp(context.Background(), StreamTopUpOptions{
SessionToken: "bearer",
ResumeSessionToken: "pc_tok",
PaymentSessionID: "payment-1",
AmountUSD: "20",
Exchange: true,
NoOpen: true,
OnSession: func(token string) { seen = append(seen, token) },
})
if err == nil || !strings.Contains(err.Error(), "payment expired") {
t.Fatalf("err = %v, want expired payment error", err)
}
if len(seen) == 0 || seen[len(seen)-1] != "" {
t.Fatalf("OnSession events = %#v, want terminal clear", seen)
}
}
func TestPollUntilPaidTerminal4xxClearsRetainedToken(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "session rejected", http.StatusNotFound)
}))
defer server.Close()
seen := "retained"
_, err := pollUntilPaid(context.Background(), NewClient(Endpoints{AppBaseURL: server.URL}, server.Client()), "pc_missing", func(token string) {
seen = token
})
if err == nil {
t.Fatal("pollUntilPaid() error = nil, want terminal 4xx")
}
if seen != "" {
t.Fatalf("retained session = %q, want cleared", seen)
}
}
func TestPollUntilExchangeSettledClearsDeadSession(t *testing.T) {
for _, test := range []struct {
name string
statusCode int
status SessionStatus
}{
{name: "terminal API rejection", statusCode: http.StatusNotFound},
{name: "expired exchange claim", status: SessionStatusExpired},
} {
t.Run(test.name, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
if test.statusCode != 0 {
http.Error(w, "session rejected", test.statusCode)
return
}
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_dead", Status: test.status})
}))
defer server.Close()
seen := "retained"
err := pollUntilExchangeSettled(context.Background(), NewClient(Endpoints{AppBaseURL: server.URL}, server.Client()), "pcs_dead", func(token string) {
seen = token
})
if err == nil {
t.Fatal("pollUntilExchangeSettled() error = nil")
}
if seen != "" {
t.Fatalf("retained session = %q, want cleared", seen)
}
})
}
}
func TestResumeHelpersClearOnlyRejectedSessions(t *testing.T) {
tests := []struct {
name string
status int
wantClear bool
}{
{name: "terminal 4xx", status: http.StatusNotFound, wantClear: true},
{name: "retryable 5xx", status: http.StatusServiceUnavailable, wantClear: false},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
http.Error(w, "session lookup failed", test.status)
}))
defer server.Close()
client := NewClient(Endpoints{AppBaseURL: server.URL}, server.Client())
for _, flow := range []struct {
name string
resolve func(func(string)) error
}{
{
name: "session bearer",
resolve: func(onSession func(string)) error {
_, _, err := resolveTopupSession(context.Background(), client, StreamTopUpOptions{
ResumeSessionToken: "pc_rejected", OnSession: onSession,
}, DefaultPartnerID, DefaultPartnerName, Endpoints{})
return err
},
},
{
name: "api key",
resolve: func(onSession func(string)) error {
_, _, err := resolveByKeySession(context.Background(), client, StreamTopUpByKeyOptions{
ResumeSessionToken: "pc_rejected", OnSession: onSession,
}, DefaultPartnerID, DefaultPartnerName, Endpoints{})
return err
},
},
} {
t.Run(flow.name, func(t *testing.T) {
seen := "retained"
if err := flow.resolve(func(token string) { seen = token }); err == nil {
t.Fatal("resolve error = nil")
}
if cleared := seen == ""; cleared != test.wantClear {
t.Fatalf("session cleared = %v, want %v", cleared, test.wantClear)
}
})
}
})
}
}
func TestStreamTopUpResumeExchangedReturnsRecoveryError(t *testing.T) {
rs := &resumeServer{t: t, statuses: []string{"exchanged"}}
server := httptest.NewServer(rs.handler())
defer server.Close()
t.Setenv("AIMLAPI_APP_URL", server.URL)
_, err := StreamTopUp(context.Background(), StreamTopUpOptions{
SessionToken: "bearer",
ResumeSessionToken: "pc_tok",
PaymentSessionID: "payment-1",
AmountUSD: "20",
Exchange: true,
NoOpen: true,
})
if err == nil || !strings.Contains(err.Error(), "already exchanged") {
t.Fatalf("err = %v, want an already-exchanged recovery error", err)
}
if rs.exchanged != 0 {
t.Fatalf("exchange calls = %d, want 0 (no second key)", rs.exchanged)
}
}
func TestStreamTopUpResumeExchangingPollsWithoutSecondExchange(t *testing.T) {
rs := &resumeServer{t: t, statuses: []string{"exchanging", "exchanged"}}
server := httptest.NewServer(rs.handler())
defer server.Close()
t.Setenv("AIMLAPI_APP_URL", server.URL)
_, err := StreamTopUp(context.Background(), StreamTopUpOptions{
SessionToken: "bearer",
ResumeSessionToken: "pc_tok",
PaymentSessionID: "payment-1",
AmountUSD: "20",
Exchange: true,
NoOpen: true,
})
if err == nil || !strings.Contains(err.Error(), "already exchanged") {
t.Fatalf("err = %v, want dashboard recovery guidance", err)
}
if rs.exchanged != 0 {
t.Fatalf("exchange calls = %d, want 0 while an exchange claim is in flight", rs.exchanged)
}
if rs.createCall != 0 || rs.payCalls != 0 {
t.Fatalf("resume created/paid a new checkout: create=%d pay=%d", rs.createCall, rs.payCalls)
}
}
func TestStreamTopUpPollObservesExchangingWithoutSecondExchange(t *testing.T) {
rs := &resumeServer{t: t, statuses: []string{"pending_payment", "exchanging", "exchanged"}}
server := httptest.NewServer(rs.handler())
defer server.Close()
t.Setenv("AIMLAPI_APP_URL", server.URL)
_, err := StreamTopUp(context.Background(), StreamTopUpOptions{
SessionToken: "bearer",
ResumeSessionToken: "pc_tok",
PaymentSessionID: "payment-1",
AmountUSD: "20",
Exchange: true,
NoOpen: true,
})
if err == nil || !strings.Contains(err.Error(), "already exchanged") {
t.Fatalf("err = %v, want dashboard recovery guidance", err)
}
if rs.exchanged != 0 {
t.Fatalf("exchange calls = %d, want 0 after poll observed an existing claim", rs.exchanged)
}
}
+186
View File
@@ -0,0 +1,186 @@
package aimlapi
import (
"context"
"crypto/rand"
"errors"
"fmt"
"math"
"net/http"
"strconv"
"strings"
"time"
)
// Status is a progress phase reported by StreamTopUp through its OnStatus callback.
type Status string
const (
StatusCreatingSession Status = "creating-session"
StatusOpeningCheckout Status = "opening-checkout"
StatusWaitingPayment Status = "waiting-payment"
StatusProvisioningKey Status = "provisioning-key"
)
// ProvisionedKey is the outcome of a top-up: the (optionally minted) API key plus
// the inference base URL and model to write into the provider profile.
type ProvisionedKey struct {
APIKey string
APIKeyID string
BaseURL string
Model string
}
const (
pollInterval = 3 * time.Second
pollTimeout = 20 * time.Minute
)
// ParseAmountUSD parses a dollar string into USD minor units (cents), enforcing the
// min/max top-up bounds. An empty value yields the default amount.
func ParseAmountUSD(value string) (int, error) {
value = strings.TrimSpace(value)
if value == "" {
return DefaultAmountUSDMinor, nil
}
dollars, err := strconv.ParseFloat(value, 64)
// NaN/±Inf parse without error and slip past the min/max comparisons below
// (every ordered comparison with NaN is false), so reject them explicitly
// before the cents conversion turns them into a garbage minor-unit amount.
if err != nil || math.IsNaN(dollars) || math.IsInf(dollars, 0) || dollars <= 0 {
return 0, fmt.Errorf("invalid amount %q; pass a positive USD amount", value)
}
minor := int(dollars*100 + 0.5)
if minor < MinAmountUSDMinor {
return 0, fmt.Errorf("minimum top-up is $%d", MinAmountUSDMinor/100)
}
if minor > MaxAmountUSDMinor {
return 0, fmt.Errorf("maximum top-up is $%d", MaxAmountUSDMinor/100)
}
return minor, nil
}
// NewPaymentSessionID returns a random UUIDv4-shaped idempotency id for a top-up.
// The caller generates it once per top-up intent and reuses it on retry so the
// backend (and Stripe) coalesce retries onto the same checkout instead of a
// second charge.
func NewPaymentSessionID() (string, error) {
var b [16]byte
if _, err := rand.Read(b[:]); err != nil {
return "", err
}
b[6] = (b[6] & 0x0f) | 0x40 // version 4
b[8] = (b[8] & 0x3f) | 0x80 // variant 10
return fmt.Sprintf("%x-%x-%x-%x-%x", b[0:4], b[4:6], b[6:8], b[8:10], b[10:16]), nil
}
func pollUntilPaid(ctx context.Context, client *Client, sessionToken string, onSession func(string)) (PartnerCheckoutSession, error) {
deadline := time.Now().Add(pollTimeout)
for time.Now().Before(deadline) {
session, err := client.GetSession(ctx, sessionToken)
if err != nil {
// Context cancellation/deadline is terminal (also covers an in-flight
// request cancelled mid-poll, whose transport error wraps ctx.Err()).
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return PartnerCheckoutSession{}, err
}
// A 4xx API response is terminal. Everything else — a 5xx or a transient
// transport error (non-APIError, e.g. a dropped connection during the
// wait) — is retried until the poll deadline.
if isTerminalSessionAPIError(err) {
reportSession(onSession, "")
return PartnerCheckoutSession{}, err
}
if err := sleepContext(ctx, pollInterval); err != nil {
return PartnerCheckoutSession{}, err
}
continue
}
switch session.Status {
case SessionStatusPaid, SessionStatusExchanging:
return session, nil
case SessionStatusExchanged:
return PartnerCheckoutSession{}, fmt.Errorf("session was already exchanged; rotate the key from the aimlapi.com dashboard")
case SessionStatusCancelled, SessionStatusExpired, SessionStatusFailed:
// The checkout cannot recover from these states. Drop the retained token
// (and the TUI's paired by-key idempotency ID) now so the very next retry
// creates a fresh payment intent.
reportSession(onSession, "")
return PartnerCheckoutSession{}, fmt.Errorf("payment %s; re-run the top-up to try again", session.Status)
default:
if err := sleepContext(ctx, pollInterval); err != nil {
return PartnerCheckoutSession{}, err
}
}
}
return PartnerCheckoutSession{}, fmt.Errorf("timed out waiting for payment; re-run once the payment clears")
}
func isTerminalSessionAPIError(err error) bool {
var apiErr APIError
return errors.As(err, &apiErr) && apiErr.Status >= http.StatusBadRequest && apiErr.Status < http.StatusInternalServerError
}
// pollUntilExchangeSettled follows an exchange claim that was already in flight
// when a retry resumed the session. Exchange is single-flight and the raw key is
// returned only to the winning request, so issuing Exchange again is never safe.
// Once the claim completes, direct the user to the dashboard recovery path.
func pollUntilExchangeSettled(ctx context.Context, client *Client, sessionToken string, onSession func(string)) error {
deadline := time.Now().Add(pollTimeout)
for time.Now().Before(deadline) {
session, err := client.GetSession(ctx, sessionToken)
if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return err
}
if isTerminalSessionAPIError(err) {
reportSession(onSession, "")
return err
}
if err := sleepContext(ctx, pollInterval); err != nil {
return err
}
continue
}
switch session.Status {
case SessionStatusExchanged:
return fmt.Errorf("session was already exchanged; rotate the key from the aimlapi.com dashboard")
case SessionStatusExchanging:
if err := sleepContext(ctx, pollInterval); err != nil {
return err
}
case SessionStatusCancelled, SessionStatusExpired, SessionStatusFailed:
reportSession(onSession, "")
return fmt.Errorf("key provisioning %s; rotate the key from the aimlapi.com dashboard", session.Status)
default:
return fmt.Errorf("key provisioning returned to %s; re-run the top-up", session.Status)
}
}
return fmt.Errorf("timed out waiting for key provisioning; retry to check the same session")
}
func sleepContext(ctx context.Context, delay time.Duration) error {
timer := time.NewTimer(delay)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
}
func status(callback func(Status, string), value Status, detail string) {
if callback != nil {
callback(value, detail)
}
}
func firstNonEmpty(values ...string) string {
for _, value := range values {
if value = strings.TrimSpace(value); value != "" {
return value
}
}
return ""
}
+38
View File
@@ -0,0 +1,38 @@
package aimlapi
import (
"net/mail"
"strings"
)
// ValidEmail performs the lightweight validation used before starting the
// passwordless onboarding request.
func ValidEmail(value string) bool {
value = strings.TrimSpace(value)
address, err := mail.ParseAddress(value)
if err != nil || address.Address != value {
return false
}
at := strings.LastIndexByte(value, '@')
if at <= 0 || at == len(value)-1 {
return false
}
domain := value[at+1:]
if strings.HasPrefix(domain, ".") || strings.HasSuffix(domain, ".") || strings.Contains(domain, "..") {
return false
}
dot := strings.LastIndexByte(domain, '.')
if dot <= 0 || dot == len(domain)-1 {
return false
}
tld := domain[dot+1:]
if len(tld) < 2 {
return false
}
for _, char := range tld {
if char < 'A' || char > 'Z' && char < 'a' || char > 'z' {
return false
}
}
return true
}
+12 -5
View File
@@ -368,6 +368,8 @@ func runWithDeps(args []string, stdout io.Writer, stderr io.Writer, deps appDeps
case "exec":
// Forward leading --add-dir occurrences so exec's own parser collects them.
return runExec(append(addDirFlagArgs(addDirs), args[1:]...), stdout, stderr, deps)
case "completions":
return runCompletions(args[1:], stdout, stderr)
case "daemon":
return runDaemon(args[1:], stdout, stderr, deps)
case "config":
@@ -766,11 +768,12 @@ func runInteractiveTUIWithSetup(stderr io.Writer, deps appDeps, permissionMode a
}
sandboxBackend := deps.selectSandboxBackend(sandbox.BackendOptions{})
sandboxEngine := sandbox.NewEngine(sandbox.EngineOptions{
WorkspaceRoot: workspaceRoot,
Policy: applyConfiguredSandboxPolicy(sandbox.DefaultPolicy(), resolved.Sandbox),
Store: sandboxStore,
Backend: sandboxBackend,
Scope: scope,
WorkspaceRoot: workspaceRoot,
Policy: applyConfiguredSandboxPolicy(sandbox.DefaultPolicy(), resolved.Sandbox),
Store: sandboxStore,
Backend: sandboxBackend,
Scope: scope,
SensitiveEnvKeys: providerSensitiveEnvKeys(resolved),
})
lastKnownMCPConfig := mcpConfig
fileTracker := tools.NewFileTracker()
@@ -1156,6 +1159,7 @@ Usage:
Commands:
exec Run a one-shot prompt through the Go agent runtime
completions Generate shell completion scripts
daemon Manage the local background worker daemon (start/stop/status/run/attach)
setup Guide first-run provider setup
config Inspect resolved Go configuration without leaking secrets
@@ -1340,6 +1344,9 @@ Flags:
--spec-reasoning-effort <effort>
Override draft reasoning effort when --use-spec is set
--max-turns <number> Override the maximum agent loop turns
--exec-profile <name> Apply an execution profile (balanced, fast, thorough): loop
posture only (turn budget, effort, self-correction, escalation);
composes with --mode (which picks the model) and explicit flags win
--auto <low|medium|high> Set exec autonomy; high enables unsafe tools
--enabled-tools <tools> Only expose these comma or space separated tools
--disabled-tools <tools> Hide these comma or space separated tools
+77
View File
@@ -348,6 +348,83 @@ func TestRunProvidersAddWritesCatalogProfile(t *testing.T) {
}
}
func TestRunProvidersAddAimlapiWritesDefaultHeaders(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
exitCode := runWithDeps([]string{"providers", "add", "aimlapi"}, &stdout, &stderr, providerSetupDeps(configPath))
if exitCode != exitSuccess {
t.Fatalf("expected exit code %d, got %d: %s", exitSuccess, exitCode, stderr.String())
}
cfg := readFileConfig(t, configPath)
if len(cfg.Providers) != 1 {
t.Fatalf("providers = %#v, want one provider", cfg.Providers)
}
profile := cfg.Providers[0]
if profile.Name != "aimlapi" ||
profile.CatalogID != "aimlapi" ||
profile.BaseURL != "https://api.aimlapi.com/v1" ||
profile.Model != "anthropic/claude-sonnet-5" ||
profile.APIKeyEnv != "AIMLAPI_API_KEY" {
t.Fatalf("unexpected provider profile: %#v", profile)
}
if profile.CustomHeaders["X-AIMLAPI-Partner-ID"] != "part_62yQoGYDq4Yqnrj2R1iGrDNJ" ||
profile.CustomHeaders["X-AIMLAPI-Integration-Repo"] != "Gitlawb/zero" {
t.Fatalf("missing aimlapi.com default headers: %#v", profile.CustomHeaders)
}
}
func TestRunProvidersAddAimlapiMixedCaseHeaderOverride(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
// A differently-cased override must replace the catalog header in place, not
// leave both spellings behind to race when request construction canonicalizes.
exitCode := runWithDeps([]string{
"providers", "add", "aimlapi",
"--header", "x-aimlapi-partner-id=part_override",
}, &stdout, &stderr, providerSetupDeps(configPath))
if exitCode != exitSuccess {
t.Fatalf("expected exit code %d, got %d: %s", exitSuccess, exitCode, stderr.String())
}
profile := readFileConfig(t, configPath).Providers[0]
if _, ok := profile.CustomHeaders["x-aimlapi-partner-id"]; ok {
t.Fatalf("lowercase override left a duplicate key: %#v", profile.CustomHeaders)
}
if profile.CustomHeaders["X-AIMLAPI-Partner-ID"] != "part_override" {
t.Fatalf("override did not win on the canonical key: %#v", profile.CustomHeaders)
}
}
func TestRunProvidersAddAimlapiOverrideDropsCatalogHeaders(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
exitCode := runWithDeps([]string{
"providers", "add", "aimlapi",
"--base-url", "https://staging.example/v1",
"--header", "X-Environment=staging",
}, &stdout, &stderr, providerSetupDeps(configPath))
if exitCode != exitSuccess {
t.Fatalf("expected exit code %d, got %d: %s", exitSuccess, exitCode, stderr.String())
}
profile := readFileConfig(t, configPath).Providers[0]
if _, ok := profile.CustomHeaders["X-AIMLAPI-Partner-ID"]; ok {
t.Fatalf("partner attribution leaked to overridden endpoint: %#v", profile.CustomHeaders)
}
if _, ok := profile.CustomHeaders["X-AIMLAPI-Integration-Repo"]; ok {
t.Fatalf("integration attribution leaked to overridden endpoint: %#v", profile.CustomHeaders)
}
if profile.CustomHeaders["X-Environment"] != "staging" {
t.Fatalf("explicit user header was dropped: %#v", profile.CustomHeaders)
}
}
func TestRunProvidersAddWritesCustomHeaders(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
+404
View File
@@ -0,0 +1,404 @@
package cli
import (
"fmt"
"io"
"sort"
"strings"
)
type completionNode struct {
names []string
flags []string
children []completionNode
}
type completionContext struct {
path string
candidates []string
transitions map[string]string
}
var completionRoot = completionNode{
flags: []string{"-h", "--help", "-v", "--version", "-p", "--prompt", "--add-dir", "--theme", "--skip-permissions-unsafe"},
children: []completionNode{
{names: []string{"exec"}, flags: []string{
"-h", "--help", "-f", "--file", "--image", "--add-dir", "--mode", "-m", "--model",
"--use-spec", "--spec-model", "--spec-reasoning-effort", "--max-turns", "--exec-profile",
"--auto", "--enabled-tools", "--disabled-tools", "--list-tools", "--profile", "-r",
"--reasoning-effort", "-C", "--cwd", "-w", "--worktree", "--worktree-dir", "-i",
"--input-format", "-o", "--output-format", "--prompt", "--resume", "--fork",
"--calling-session-id", "--calling-tool-use-id", "--tag", "--depth", "--session-title",
"--init-session-id", "--skip-permissions-unsafe", "--allow-escalation", "--self-correct",
"--notify", "--no-notify", "--no-completion-gate",
}},
{
names: []string{"completions"},
flags: []string{"-h", "--help"},
children: leafNodes("bash", "zsh", "fish", "powershell", "elvish"),
},
{names: []string{"daemon"}, children: leafNodes("start", "stop", "status", "run", "attach", "serve-remote", "link")},
{names: []string{"setup"}},
{names: []string{"config"}},
{names: []string{"models"}, children: []completionNode{{names: []string{"list", "ls"}}}},
{names: []string{"providers"}, children: []completionNode{
{names: []string{"current"}}, {names: []string{"list"}}, {names: []string{"catalog"}},
{names: []string{"add"}}, {names: []string{"check"}}, {names: []string{"use"}},
{names: []string{"remove", "rm"}}, {names: []string{"rename"}}, {names: []string{"setup"}},
{names: []string{"detect"}}, {names: []string{"models"}},
}},
{names: []string{"doctor"}},
{names: []string{"context"}},
{names: []string{"repo-map", "repomap"}},
{names: []string{"search", "find"}},
{names: []string{"sessions", "session"}, children: leafNodes("list", "children", "lineage", "tree", "rewind-plan", "rewind", "compact-plan")},
{names: []string{"spec"}, children: leafNodes("show", "approve", "reject")},
{names: []string{"init"}},
{names: []string{"specialists", "specialist"}, children: []completionNode{
{names: []string{"list"}}, {names: []string{"show"}}, {names: []string{"create"}},
{names: []string{"delete", "rm"}}, {names: []string{"edit"}}, {names: []string{"path"}},
}},
{names: []string{"plugins", "plugin"}, children: []completionNode{
{names: []string{"list"}}, {names: []string{"add"}}, {names: []string{"remove", "rm"}},
}},
{names: []string{"backends", "backend"}, children: leafNodes("doctor")},
{names: []string{"skills", "skill"}, children: []completionNode{
{names: []string{"list"}}, {names: []string{"add"}}, {names: []string{"info"}}, {names: []string{"remove", "rm"}},
}},
{names: []string{"tools", "tool"}, children: leafNodes("make", "list")},
{names: []string{"hooks"}, children: []completionNode{
{names: []string{"list"}}, {names: []string{"add"}}, {names: []string{"remove", "rm"}},
{names: []string{"enable"}}, {names: []string{"disable"}},
}},
{names: []string{"mcp"}, children: []completionNode{
{names: []string{"add"}}, {names: []string{"remove", "rm"}}, {names: []string{"enable"}},
{names: []string{"disable"}}, {names: []string{"check"}}, {names: []string{"list"}},
{names: []string{"permissions"}, children: leafNodes("list", "revoke", "clear")},
{names: []string{"tools"}, children: leafNodes("list")},
{names: []string{"oauth"}, children: leafNodes("login", "logout", "status")},
}},
{names: []string{"auth"}, children: leafNodes("openrouter", "chatgpt", "login", "logout", "status", "refresh")},
{names: []string{"sandbox"}, children: []completionNode{
{names: []string{"policy"}}, {names: []string{"setup"}}, {names: []string{"check"}},
{names: []string{"grants"}, children: leafNodes("list", "allow", "deny", "revoke", "clear")},
}},
{names: []string{"update"}},
{names: []string{"upgrade"}},
{names: []string{"worktrees", "worktree"}, children: leafNodes("prepare")},
{names: []string{"verify"}},
{names: []string{"trust"}, children: leafNodes("list", "remove")},
{names: []string{"eval"}, children: leafNodes("validate", "run", "bench")},
{names: []string{"changes", "change"}, children: []completionNode{
{names: []string{"inspect", "status"}}, {names: []string{"commit"}}, {names: []string{"push"}},
{names: []string{"pr", "pull-request"}},
}},
{names: []string{"usage"}, children: leafNodes("report")},
{names: []string{"cron"}, children: leafNodes("add", "list", "rm", "pause", "resume", "run")},
{names: []string{"repo-info", "repoinfo"}},
{names: []string{"serve"}},
{names: []string{"acp"}},
{names: []string{"help"}},
{names: []string{"version"}},
},
}
func leafNodes(names ...string) []completionNode {
nodes := make([]completionNode, 0, len(names))
for _, name := range names {
nodes = append(nodes, completionNode{names: []string{name}})
}
return nodes
}
func runCompletions(args []string, stdout io.Writer, stderr io.Writer) int {
if len(args) == 1 && (args[0] == "-h" || args[0] == "--help" || args[0] == "help") {
if err := writeCompletionsHelp(stdout); err != nil {
return exitCrash
}
return exitSuccess
}
if len(args) == 0 {
return writeExecUsageError(stderr, "shell required. Use `zero completions <bash|zsh|fish|powershell|elvish>`.")
}
if len(args) != 1 {
return writeExecUsageError(stderr, fmt.Sprintf("unexpected completions argument %q", args[1]))
}
contexts := completionContexts(completionRoot)
var err error
switch strings.ToLower(strings.TrimSpace(args[0])) {
case "bash":
err = writeBashCompletions(stdout, contexts)
case "zsh":
err = writeZshCompletions(stdout, contexts)
case "fish":
err = writeFishCompletions(stdout, contexts)
case "powershell":
err = writePowerShellCompletions(stdout, contexts)
case "elvish":
err = writeElvishCompletions(stdout, contexts)
default:
return writeExecUsageError(stderr, fmt.Sprintf("unsupported shell %q; expected bash, zsh, fish, powershell, or elvish", args[0]))
}
if err != nil {
return exitCrash
}
return exitSuccess
}
func writeCompletionsHelp(w io.Writer) error {
_, err := fmt.Fprint(w, `Generate shell completion scripts.
Usage:
zero completions <shell>
Arguments:
<shell> Target shell: bash, zsh, fish, powershell, or elvish
Examples:
source <(zero completions bash)
source <(zero completions zsh)
zero completions fish > ~/.config/fish/completions/zero.fish
zero completions powershell >> $PROFILE
eval (zero completions elvish | slurp)
Flags:
-h, --help Show this help
`)
return err
}
func completionContexts(root completionNode) []completionContext {
contexts := []completionContext{}
var visit func(completionNode, []string)
visit = func(node completionNode, paths []string) {
candidates := append([]string{}, node.flags...)
transitions := map[string]string{}
for _, child := range node.children {
candidates = append(candidates, child.names...)
for _, path := range paths {
for _, name := range child.names {
childPath := strings.TrimSpace(path + " " + name)
transitions[path+"|"+name] = childPath
}
}
}
candidates = uniqueStrings(candidates)
for _, path := range paths {
contexts = append(contexts, completionContext{path: path, candidates: candidates, transitions: transitions})
}
for _, child := range node.children {
childPaths := make([]string, 0, len(paths)*len(child.names))
for _, path := range paths {
for _, name := range child.names {
childPaths = append(childPaths, strings.TrimSpace(path+" "+name))
}
}
visit(child, childPaths)
}
}
visit(root, []string{""})
sort.SliceStable(contexts, func(i, j int) bool { return contexts[i].path < contexts[j].path })
return contexts
}
func uniqueStrings(values []string) []string {
seen := map[string]bool{}
result := make([]string, 0, len(values))
for _, value := range values {
if value == "" || seen[value] {
continue
}
seen[value] = true
result = append(result, value)
}
return result
}
func allTransitions(contexts []completionContext) map[string]string {
transitions := map[string]string{}
for _, context := range contexts {
for key, value := range context.transitions {
transitions[key] = value
}
}
return transitions
}
func sortedTransitionKeys(transitions map[string]string) []string {
keys := make([]string, 0, len(transitions))
for key := range transitions {
keys = append(keys, key)
}
sort.Strings(keys)
return keys
}
func shellWords(values []string) string {
return strings.Join(values, " ")
}
func writeBashCompletions(w io.Writer, contexts []completionContext) error {
if _, err := fmt.Fprintln(w, "# bash completion for zero\n_zero() {\n local cur context token i candidates\n cur=\"${COMP_WORDS[COMP_CWORD]}\"\n context=\"\""); err != nil {
return err
}
if err := writePOSIXContextLoop(w, contexts, " for ((i = 1; i < COMP_CWORD; i++)); do", " token=\"${COMP_WORDS[i]}\"", " done"); err != nil {
return err
}
if err := writePOSIXCandidateCase(w, contexts, " "); err != nil {
return err
}
_, err := fmt.Fprintln(w, " COMPREPLY=( $(compgen -W \"$candidates\" -- \"$cur\") )\n}\ncomplete -F _zero zero")
return err
}
func writeZshCompletions(w io.Writer, contexts []completionContext) error {
if _, err := fmt.Fprintln(w, "#compdef zero\n_zero() {\n local context token i\n local -a candidates\n context=\"\""); err != nil {
return err
}
if err := writePOSIXContextLoop(w, contexts, " for ((i = 2; i < CURRENT; i++)); do", " token=\"${words[i]}\"", " done"); err != nil {
return err
}
if _, err := fmt.Fprintln(w, " case \"$context\" in"); err != nil {
return err
}
for _, context := range contexts {
if _, err := fmt.Fprintf(w, " %q) candidates=(%s) ;;\n", context.path, quotedWords(context.candidates)); err != nil {
return err
}
}
_, err := fmt.Fprintln(w, " esac\n compadd -- \"${candidates[@]}\"\n}\ncompdef _zero zero")
return err
}
func writePOSIXContextLoop(w io.Writer, contexts []completionContext, start string, token string, end string) error {
if _, err := fmt.Fprintln(w, start); err != nil {
return err
}
if _, err := fmt.Fprintln(w, token); err != nil {
return err
}
if _, err := fmt.Fprintln(w, " case \"$context|$token\" in"); err != nil {
return err
}
transitions := allTransitions(contexts)
for _, key := range sortedTransitionKeys(transitions) {
if _, err := fmt.Fprintf(w, " %q) context=%q ;;\n", key, transitions[key]); err != nil {
return err
}
}
if _, err := fmt.Fprintln(w, " esac"); err != nil {
return err
}
_, err := fmt.Fprintln(w, end)
return err
}
func writePOSIXCandidateCase(w io.Writer, contexts []completionContext, indent string) error {
if _, err := fmt.Fprintln(w, indent+"case \"$context\" in"); err != nil {
return err
}
for _, context := range contexts {
if _, err := fmt.Fprintf(w, "%s %q) candidates=%q ;;\n", indent, context.path, shellWords(context.candidates)); err != nil {
return err
}
}
_, err := fmt.Fprintln(w, indent+"esac")
return err
}
func quotedWords(values []string) string {
quoted := make([]string, 0, len(values))
for _, value := range values {
quoted = append(quoted, fmt.Sprintf("%q", value))
}
return strings.Join(quoted, " ")
}
func writeFishCompletions(w io.Writer, contexts []completionContext) error {
if _, err := fmt.Fprintln(w, "# fish completion for zero\nfunction __zero_completion_context\n set -l context ''\n set -l tokens (commandline -opc)"); err != nil {
return err
}
if _, err := fmt.Fprintln(w, " for token in $tokens[2..-1]\n switch \"$context|$token\""); err != nil {
return err
}
transitions := allTransitions(contexts)
for _, key := range sortedTransitionKeys(transitions) {
if _, err := fmt.Fprintf(w, " case %q\n set context %q\n", key, transitions[key]); err != nil {
return err
}
}
if _, err := fmt.Fprintln(w, " end\n end\n if test -z \"$context\"\n echo __root__\n else\n echo \"$context\"\n end\nend"); err != nil {
return err
}
for _, context := range contexts {
path := context.path
if path == "" {
path = "__root__"
}
if _, err := fmt.Fprintf(w, "complete -c zero -f -n 'test (__zero_completion_context) = %q' -a %q\n", path, shellWords(context.candidates)); err != nil {
return err
}
}
return nil
}
func writePowerShellCompletions(w io.Writer, contexts []completionContext) error {
if _, err := fmt.Fprintln(w, "# PowerShell completion for zero\nRegister-ArgumentCompleter -Native -CommandName zero -ScriptBlock {\n param($wordToComplete, $commandAst, $cursorPosition)\n $context = ''\n $elements = @($commandAst.CommandElements)\n $limit = $elements.Count\n if ($wordToComplete -ne '' -and $limit -gt 1) { $limit-- }\n for ($i = 1; $i -lt $limit; $i++) {\n $token = $elements[$i].Extent.Text\n switch (\"$context|$token\") {"); err != nil {
return err
}
transitions := allTransitions(contexts)
for _, key := range sortedTransitionKeys(transitions) {
if _, err := fmt.Fprintf(w, " '%s' { $context = '%s'; break }\n", key, transitions[key]); err != nil {
return err
}
}
if _, err := fmt.Fprintln(w, " }\n }\n $candidates = switch ($context) {"); err != nil {
return err
}
for _, context := range contexts {
if _, err := fmt.Fprintf(w, " '%s' { @(%s) }\n", context.path, powershellWords(context.candidates)); err != nil {
return err
}
}
_, err := fmt.Fprintln(w, " }\n $candidates | Where-Object { $_ -like \"$wordToComplete*\" } | ForEach-Object {\n [System.Management.Automation.CompletionResult]::new($_, $_, 'ParameterValue', $_)\n }\n}")
return err
}
func powershellWords(values []string) string {
quoted := make([]string, 0, len(values))
for _, value := range values {
quoted = append(quoted, "'"+strings.ReplaceAll(value, "'", "''")+"'")
}
return strings.Join(quoted, ", ")
}
func writeElvishCompletions(w io.Writer, contexts []completionContext) error {
if _, err := fmt.Fprintln(w, "# Elvish completion for zero\nset edit:completion:arg-completer[zero] = {|@args|\n var context = ''\n for token $args[1..-1] {"); err != nil {
return err
}
transitions := allTransitions(contexts)
for index, key := range sortedTransitionKeys(transitions) {
keyword := "if"
if index > 0 {
keyword = "} elif"
}
if _, err := fmt.Fprintf(w, " %s (eq $context'|'$token %q) {\n set context = %q\n", keyword, key, transitions[key]); err != nil {
return err
}
}
if _, err := fmt.Fprintln(w, " }\n }"); err != nil {
return err
}
for index, context := range contexts {
keyword := "if"
if index > 0 {
keyword = "} elif"
}
if _, err := fmt.Fprintf(w, " %s (eq $context %q) {\n put %s\n", keyword, context.path, shellWords(context.candidates)); err != nil {
return err
}
}
_, err := fmt.Fprintln(w, " }\n}")
return err
}
+175
View File
@@ -0,0 +1,175 @@
package cli
import (
"bytes"
"os/exec"
"strings"
"testing"
)
func TestCompletionsHelpAndRootHelp(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
if code := runWithDeps([]string{"completions", "--help"}, &stdout, &stderr, appDeps{}); code != exitSuccess {
t.Fatalf("completion help exit code = %d, want %d: %s", code, exitSuccess, stderr.String())
}
for _, want := range []string{
"zero completions <shell>",
"bash, zsh, fish, powershell, or elvish",
"source <(zero completions bash)",
"~/.config/fish/completions/zero.fish",
"eval (zero completions elvish | slurp)",
} {
if !strings.Contains(stdout.String(), want) {
t.Errorf("completion help missing %q:\n%s", want, stdout.String())
}
}
stdout.Reset()
stderr.Reset()
if code := runWithDeps([]string{"--help"}, &stdout, &stderr, appDeps{}); code != exitSuccess {
t.Fatalf("root help exit code = %d, want %d: %s", code, exitSuccess, stderr.String())
}
if !strings.Contains(stdout.String(), "completions Generate shell completion scripts") {
t.Fatalf("root help does not list completions command:\n%s", stdout.String())
}
}
func TestCompletionsRejectsMissingUnknownAndExtraShellArguments(t *testing.T) {
tests := []struct {
name string
args []string
want string
}{
{name: "missing", args: []string{"completions"}, want: "shell required"},
{name: "unknown", args: []string{"completions", "nu"}, want: "unsupported shell"},
{name: "extra", args: []string{"completions", "bash", "extra"}, want: "unexpected completions argument"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
if code := runWithDeps(test.args, &stdout, &stderr, appDeps{}); code != exitUsage {
t.Fatalf("exit code = %d, want %d: %s", code, exitUsage, stderr.String())
}
if stdout.Len() != 0 {
t.Fatalf("stdout = %q, want empty", stdout.String())
}
if !strings.Contains(stderr.String(), test.want) {
t.Fatalf("stderr = %q, want substring %q", stderr.String(), test.want)
}
})
}
}
func TestCompletionsGeneratesEverySupportedShell(t *testing.T) {
tests := []struct {
shell string
marker string
syntaxShell string
}{
{shell: "bash", marker: "complete -F _zero zero", syntaxShell: "bash"},
{shell: "zsh", marker: "#compdef zero", syntaxShell: "zsh"},
{shell: "fish", marker: "complete -c zero"},
{shell: "powershell", marker: "Register-ArgumentCompleter -Native -CommandName zero"},
{shell: "elvish", marker: "edit:completion:arg-completer[zero]"},
}
for _, test := range tests {
t.Run(test.shell, func(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
if code := runWithDeps([]string{"completions", test.shell}, &stdout, &stderr, appDeps{}); code != exitSuccess {
t.Fatalf("exit code = %d, want %d: %s", code, exitSuccess, stderr.String())
}
if !strings.Contains(stdout.String(), test.marker) {
t.Fatalf("%s completion missing marker %q:\n%s", test.shell, test.marker, stdout.String())
}
for _, want := range []string{"daemon", "mcp oauth", "sandbox grants", "--output-format"} {
if !strings.Contains(stdout.String(), want) {
t.Errorf("%s completion missing %q", test.shell, want)
}
}
script := stdout.String()
switch test.shell {
case "fish":
assertBalancedFishBlocks(t, script)
case "powershell", "elvish":
assertBalancedBraces(t, script)
}
if test.syntaxShell != "" {
assertNativeShellSyntax(t, test.syntaxShell, script)
}
})
}
}
func assertBalancedBraces(t *testing.T, script string) {
t.Helper()
if opens, closes := strings.Count(script, "{"), strings.Count(script, "}"); opens != closes {
t.Errorf("unbalanced braces: %d opening, %d closing", opens, closes)
}
}
func assertBalancedFishBlocks(t *testing.T, script string) {
t.Helper()
depth := 0
for _, line := range strings.Split(script, "\n") {
fields := strings.Fields(line)
if len(fields) == 0 {
continue
}
switch fields[0] {
case "function", "for", "switch", "if":
depth++
case "end":
depth--
if depth < 0 {
t.Fatal("fish completion closes a block before one is opened")
}
}
}
if depth != 0 {
t.Errorf("fish completion has %d unclosed blocks", depth)
}
}
func assertNativeShellSyntax(t *testing.T, shell, script string) {
t.Helper()
path, err := exec.LookPath(shell)
if err != nil {
t.Skipf("%s is not installed", shell)
}
command := exec.Command(path, "-n")
command.Stdin = strings.NewReader(script)
if output, err := command.CombinedOutput(); err != nil {
t.Fatalf("%s syntax check failed: %v\n%s", shell, err, output)
}
}
func TestCompletionTreeCoversAliasesNestingAndCommonFlags(t *testing.T) {
contexts := completionContexts(completionRoot)
byPath := make(map[string][]string, len(contexts))
for _, context := range contexts {
byPath[context.path] = context.candidates
}
assertCandidates(t, byPath[""], "sessions", "session", "plugins", "plugin", "worktrees", "worktree", "--add-dir", "--theme", "-p", "--prompt")
assertCandidates(t, byPath["exec"], "--model", "--cwd", "--worktree", "--output-format", "--resume", "--skip-permissions-unsafe")
assertCandidates(t, byPath["daemon"], "start", "stop", "status", "run", "attach")
assertCandidates(t, byPath["mcp oauth"], "login", "logout", "status")
assertCandidates(t, byPath["sandbox grants"], "list", "allow", "deny", "revoke", "clear")
assertCandidates(t, byPath["completions"], "bash", "zsh", "fish", "powershell", "elvish")
}
func assertCandidates(t *testing.T, got []string, wants ...string) {
t.Helper()
set := make(map[string]bool, len(got))
for _, candidate := range got {
set[candidate] = true
}
for _, want := range wants {
if !set[want] {
t.Errorf("candidates %v do not contain %q", got, want)
}
}
}
+166 -30
View File
@@ -13,6 +13,7 @@ import (
"github.com/Gitlawb/zero/internal/agent"
"github.com/Gitlawb/zero/internal/config"
"github.com/Gitlawb/zero/internal/errhint"
"github.com/Gitlawb/zero/internal/execprofile"
"github.com/Gitlawb/zero/internal/imageinput"
"github.com/Gitlawb/zero/internal/lsp"
"github.com/Gitlawb/zero/internal/modelregistry"
@@ -69,7 +70,14 @@ type execOptions struct {
// intentionally inert: nothing consumes it. Model selection is driven by
// --model / --mode instead. See writeExecHelp ("Accept legacy model profile
// selection") and TestRunExecAcceptsLegacyModelProfileFlags.
modelProfile string
modelProfile string
// execProfile selects a named execution profile (balanced, fast, thorough):
// a loop-posture bundle (turn budget, reasoning effort, self-correction,
// escalation triggers) applied AFTER --mode with the same
// fill-only-if-unset rule, so precedence is explicit flag > mode > profile.
// Distinct from the mode preset also named "fast" (which picks a model) and
// from the legacy inert --profile above. See internal/execprofile.
execProfile string
reasoningEffort string
useSpec bool
specModel string
@@ -167,6 +175,14 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
if err := applyExecMode(&options); err != nil {
return writeExecFormatUsageError(stdout, stderr, options.outputFormat, err.Error())
}
// A profile tunes loop posture the way a mode tunes model selection, and
// runs after it so the mode's fills count as "set" and win. MaxTurns is
// deferred to config resolution below, where the displaced resolved budget
// is known and becomes the escalation restore target.
execProfile, execProfileFilledEffort, err := applyExecProfile(&options)
if err != nil {
return writeExecFormatUsageError(stdout, stderr, options.outputFormat, err.Error())
}
workspaceRoot, err := resolveWorkspaceRoot(options.cwd, deps)
if err != nil {
@@ -272,6 +288,8 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
}
return writeExecProviderError(stdout, stderr, options.outputFormat, "provider_error", err.Error())
}
var displacedMaxTurns int
resolved.MaxTurns, displacedMaxTurns = applyProfileTurnBudget(execProfile, options.maxTurns, resolved.MaxTurns)
registerLocalControlTools(registry, workspaceRoot, resolved.LocalControl)
if err := validateExecToolFilters(options, registry); err != nil {
return writeExecFormatUsageError(stdout, stderr, options.outputFormat, err.Error())
@@ -412,6 +430,36 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
}
}
// Optimized OpenAI turn sessions (ZERO_OPENAI_TURN_SESSION, default off).
// nil when gated off or the profile is ineligible: agent.Run then wraps the
// provider in its default adapter — the exact code path of today. The
// session switcher is installed only when the run START is optimized, so
// the legacy ModelSwitcher path above stays untouched otherwise.
turnSessions, _ := providers.OptimizedTurnSessions(resolved.Provider, provider, providers.Options{})
var modelSessionSwitcher func(context.Context, string) (zeroruntime.TurnSessionProvider, error)
if options.allowEscalation && turnSessions != nil {
modelSessionSwitcher = func(_ context.Context, modelID string) (zeroruntime.TurnSessionProvider, error) {
switchedProfile := resolved.Provider
switchedProfile.Model = modelID
switchedProvider, err := deps.newProvider(switchedProfile)
if err != nil {
return nil, err
}
if switchedProvider == nil {
// The loop treats a nil session source as "no swap" — mirror the
// legacy closure's (nil, nil) contract.
return nil, nil
}
currentModel = modelID
if optimized, ok := providers.OptimizedTurnSessions(switchedProfile, switchedProvider, providers.Options{}); ok {
return optimized, nil
}
// Ineligible switch target: default adapter, but with the switched
// model's resolved capability projection preserved.
return providers.DefaultTurnSessions(switchedProfile, switchedProvider, providers.Options{}), nil
}
}
runMetadata, err := resolveExecRunMetadata(resolved.Provider)
if err != nil {
return writeExecProviderError(stdout, stderr, options.outputFormat, "provider_error", err.Error())
@@ -456,6 +504,13 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
reasoningEffort: forwardEffort,
specPermissionMode: permissionMode,
notifier: notifier,
// The profile displaced resolved.MaxTurns above, so the spec-draft
// run arms the same escalation policy as the main run (in practice
// only the failure-streak and risky-mutation triggers can fire
// here: the draft runs without the completion gate or a
// self-corrector).
profilePolicy: execProfile.Policy(displacedMaxTurns,
specProfileEffortFilled(execProfileFilledEffort, options.specReasoningEffort)),
})
}
@@ -498,7 +553,7 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
var traceRecorder *trace.Recorder
var traceSnapshot *trace.TurnTrace
if tracePath != "" {
traceRecorder = trace.NewRecorder(preparedSession.Session.SessionID, runID, "")
traceRecorder = trace.NewRecorder(preparedSession.Session.SessionID, runID, execProfile.Name)
defer func() {
if err := writeTraceSnapshot(traceSnapshot, tracePath, stderr); err != nil {
fmt.Fprintf(stderr, "[zero] failed to write trace: %s\n", err)
@@ -572,29 +627,32 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
hookDispatcher, hookSkip := newHookDispatcherWithExtra(workspaceRoot, pluginActivation.hooks, trustRoot)
emitTrustNotice(stderr, hookSkip, pluginActivation.trustSkip, mcpSkip)
result, err := agent.Run(runCtx, agentPrompt, provider, agent.Options{
MaxTurns: resolved.MaxTurns,
ContextWindow: resolveAgentContextWindow(runCtx, modelRegistry, resolved.Provider),
DeferThreshold: effectiveDeferThreshold,
Specialists: specialistRuntime.specialistInfos(),
Skills: pluginActivation.skillInfos(deps.skillsDir()),
SessionID: preparedSession.Session.SessionID,
CallingSessionID: options.callingSessionID,
CallingToolUseID: options.callingToolUseID,
Tag: options.tag,
Depth: options.depth,
SessionTitle: sessionTitle,
ProviderName: resolved.Provider.Name,
Model: resolved.Provider.Model,
ModelSwitcher: modelSwitcher,
ReasoningEffort: forwardEffort,
Trace: traceRecorder,
Cwd: workspaceRoot,
Images: images,
Registry: registry,
PermissionMode: permissionMode,
Autonomy: options.autonomy,
SelfCorrect: selfCorrector,
FileDiagnostics: fileDiagnostics,
MaxTurns: resolved.MaxTurns,
ContextWindow: resolveAgentContextWindow(runCtx, modelRegistry, resolved.Provider),
DeferThreshold: effectiveDeferThreshold,
Specialists: specialistRuntime.specialistInfos(),
Skills: pluginActivation.skillInfos(deps.skillsDir()),
SessionID: preparedSession.Session.SessionID,
CallingSessionID: options.callingSessionID,
CallingToolUseID: options.callingToolUseID,
Tag: options.tag,
Depth: options.depth,
SessionTitle: sessionTitle,
ProviderName: resolved.Provider.Name,
Model: resolved.Provider.Model,
ModelSwitcher: modelSwitcher,
TurnSessionProvider: turnSessions,
ModelSessionSwitcher: modelSessionSwitcher,
ReasoningEffort: forwardEffort,
Trace: traceRecorder,
Cwd: workspaceRoot,
Images: images,
Registry: registry,
PermissionMode: permissionMode,
Autonomy: options.autonomy,
SelfCorrect: selfCorrector,
FileDiagnostics: fileDiagnostics,
Profile: execProfile.Policy(displacedMaxTurns, execProfileFilledEffort),
// Headless exec: don't accept a no-tool-call turn as "done" while work
// clearly remains (pending plan items / a mid-step continuation cue) —
// nudge to continue, and finalize as INCOMPLETE rather than false success
@@ -636,6 +694,9 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
if len(result.Meta) > 0 {
payload["meta"] = result.Meta
}
if result.Truncated {
payload["truncated"] = true
}
if result.Redacted {
payload["redacted"] = true
}
@@ -853,14 +914,28 @@ func buildExecSandboxEngine(workspaceRoot string, resolved config.ResolvedConfig
policy := applyConfiguredSandboxPolicy(sandbox.DefaultPolicy(), resolved.Sandbox)
backend := deps.selectSandboxBackend(sandbox.BackendOptions{})
return sandbox.NewEngine(sandbox.EngineOptions{
WorkspaceRoot: workspaceRoot,
Policy: policy,
Store: store,
Backend: backend,
Scope: scope,
WorkspaceRoot: workspaceRoot,
Policy: policy,
Store: store,
Backend: backend,
Scope: scope,
SensitiveEnvKeys: providerSensitiveEnvKeys(resolved),
}), nil
}
func providerSensitiveEnvKeys(resolved config.ResolvedConfig) []string {
keys := make([]string, 0, len(resolved.Providers)+1)
for _, profile := range resolved.Providers {
if key := strings.TrimSpace(profile.APIKeyEnv); key != "" {
keys = append(keys, key)
}
}
if key := strings.TrimSpace(resolved.Provider.APIKeyEnv); key != "" {
keys = append(keys, key)
}
return keys
}
// applyConfiguredSandboxPolicy overlays every config-sourced sandbox knob onto
// the default policy.
func applyConfiguredSandboxPolicy(policy sandbox.Policy, cfg config.SandboxConfig) sandbox.Policy {
@@ -1084,6 +1159,67 @@ func applyExecMode(options *execOptions) error {
return nil
}
// applyExecProfile expands an --exec-profile selection onto the exec options.
// Like applyExecMode it only fills fields the caller left unset, and it runs
// AFTER the mode so a mode-filled field counts as set: precedence is explicit
// flag > --mode > --exec-profile. Only the options-level knobs are applied here
// (reasoning effort still flows through the model-supported gating downstream;
// self-correct is presence-only so a profile can only turn it on). MaxTurns is
// intentionally NOT applied here — the profile displaces the resolved budget
// after config resolution, where the displaced value is known and becomes the
// escalation restore target. An unknown profile is a usage error listing the
// valid names. Not to be confused with the legacy inert --profile flag.
// The second return reports whether the profile actually filled the reasoning
// effort (false when the user set one explicitly), so the escalation policy
// knows the displaced effort was the provider default and can restore it.
func applyExecProfile(options *execOptions) (execprofile.Profile, bool, error) {
name := strings.TrimSpace(options.execProfile)
if name == "" {
return execprofile.Profile{}, false, nil
}
profile, ok := execprofile.Lookup(name)
if !ok {
return execprofile.Profile{}, false, execUsageError{fmt.Sprintf("unknown execution profile %q. Valid profiles: %s.", options.execProfile, strings.Join(execprofile.Names(), ", "))}
}
effortFilled := false
if options.reasoningEffort == "" && profile.ReasoningEffort != "" {
options.reasoningEffort = profile.ReasoningEffort
effortFilled = true
}
// Spec-safe projection: the spec-draft path wires no self-corrector (an
// explicit --self-correct --use-spec is rejected at parse time, before
// profiles apply), so a profile's self-correct knob is meaningless there.
// Project it away rather than erroring: the user asked for a thorough
// DRAFT, and the profile's other knobs (budget, effort) apply to it fine.
if profile.SelfCorrect && !options.useSpec {
options.selfCorrect = true
}
return profile, effortFilled, nil
}
// specProfileEffortFilled reports whether the profile's effort fill actually
// governs the spec-draft run. An explicit --spec-reasoning-effort replaces the
// filled effort for the draft, so the escalation's effort restore must not arm
// there: escalation must never clear an effort the user pinned by hand.
func specProfileEffortFilled(effortFilled bool, specReasoningEffort string) bool {
return effortFilled && strings.TrimSpace(specReasoningEffort) == ""
}
// applyProfileTurnBudget decides the run's turn budget once config is resolved.
// The profile displaces the RESOLVED value, not the flag: a pinned budget
// always wins and the profile backs off entirely, while an env/config budget
// is the "balanced posture" a mid-run escalation can restore. pinnedMaxTurns
// is any caller-pinned budget — an explicit --max-turns flag OR a mode
// preset's fill (both land in options.maxTurns, and a mode outranks a profile
// just like the flag does). displaced is 0 when nothing was displaced, so an
// escalation leaves the ceiling untouched.
func applyProfileTurnBudget(profile execprofile.Profile, pinnedMaxTurns int, resolvedMaxTurns int) (effective int, displaced int) {
if profile.MaxTurns > 0 && pinnedMaxTurns == 0 {
return profile.MaxTurns, resolvedMaxTurns
}
return resolvedMaxTurns, 0
}
// resolveSelectedModel routes a user-supplied --model value through the model
// registry so that fuzzy aliases (e.g. "sonnet 4.5") resolve to canonical ids
// and deprecated models auto-redirect to their fallback. It returns the model id
+13
View File
@@ -147,6 +147,19 @@ func parseExecArgs(args []string) (execOptions, bool, error) {
index = next
case strings.HasPrefix(arg, "--profile="):
options.modelProfile = strings.TrimSpace(strings.TrimPrefix(arg, "--profile="))
case arg == "--exec-profile":
value, next, err := nextFlagValue(args, index, arg)
if err != nil {
return options, false, err
}
options.execProfile = strings.TrimSpace(value)
index = next
case strings.HasPrefix(arg, "--exec-profile="):
value, err := requiredInlineFlagValue(arg, "--exec-profile")
if err != nil {
return options, false, err
}
options.execProfile = value
case arg == "-r" || arg == "--reasoning-effort":
value, next, err := nextFlagValue(args, index, arg)
if err != nil {
+224
View File
@@ -0,0 +1,224 @@
package cli
import (
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/execprofile"
)
func TestApplyExecProfileFillsOnlyUnset(t *testing.T) {
// Unset effort: the profile fills it and reports the fill, so the
// escalation policy knows the displaced effort was the provider default.
options := execOptions{execProfile: "fast"}
profile, effortFilled, err := applyExecProfile(&options)
if err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if profile.Name != "fast" {
t.Fatalf("profile.Name = %q, want fast", profile.Name)
}
if options.reasoningEffort != "low" {
t.Fatalf("reasoningEffort = %q, want the profile's low", options.reasoningEffort)
}
if !effortFilled {
t.Fatal("the profile filled the effort, so effortFilled must report it")
}
// Set effort (explicit flag or a mode's fill): the profile backs off.
options = execOptions{execProfile: "fast", reasoningEffort: "high"}
_, effortFilled, err = applyExecProfile(&options)
if err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if options.reasoningEffort != "high" {
t.Fatalf("reasoningEffort = %q, an explicit value must win over the profile", options.reasoningEffort)
}
if effortFilled {
t.Fatal("the profile backed off, so effortFilled must be false")
}
}
func TestApplyExecProfileThoroughArmsSelfCorrect(t *testing.T) {
options := execOptions{execProfile: "thorough"}
if _, _, err := applyExecProfile(&options); err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if !options.selfCorrect {
t.Fatal("thorough must arm the post-edit self-correct loop")
}
if options.reasoningEffort != "high" {
t.Fatalf("reasoningEffort = %q, want high", options.reasoningEffort)
}
}
// The spec-draft path wires no self-corrector (explicit --self-correct
// --use-spec is rejected at parse time, before profiles apply), so a profile's
// self-correct knob is projected away under --use-spec instead of silently
// arming a knob the draft runner ignores. The other knobs still apply.
func TestApplyExecProfileSpecProjectsAwaySelfCorrect(t *testing.T) {
options := execOptions{execProfile: "thorough", useSpec: true}
if _, _, err := applyExecProfile(&options); err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if options.selfCorrect {
t.Fatal("thorough under --use-spec must not arm self-correct (the draft runner has none)")
}
if options.reasoningEffort != "high" {
t.Fatalf("reasoningEffort = %q, the profile's other knobs must still apply", options.reasoningEffort)
}
}
func TestApplyExecProfileUnknownIsUsageError(t *testing.T) {
options := execOptions{execProfile: "turbo"}
_, _, err := applyExecProfile(&options)
if err == nil {
t.Fatal("expected a usage error for an unknown profile")
}
if _, ok := err.(execUsageError); !ok {
t.Fatalf("expected execUsageError, got %T: %v", err, err)
}
for _, name := range execprofile.Names() {
if !strings.Contains(err.Error(), name) {
t.Fatalf("usage error must list %q, got %q", name, err.Error())
}
}
}
// Precedence: explicit flag > --mode > --exec-profile. The deep mode fills
// effort=high and max-turns=160; the fast profile must not override either,
// because a mode-filled field counts as set by the time the profile runs.
func TestExecProfilePrecedenceFlagOverModeOverProfile(t *testing.T) {
options := execOptions{mode: "deep", execProfile: "fast"}
if err := applyExecMode(&options); err != nil {
t.Fatalf("applyExecMode: %v", err)
}
profile, _, err := applyExecProfile(&options)
if err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if options.reasoningEffort != "high" {
t.Fatalf("reasoningEffort = %q, the mode's high must win over the profile's low", options.reasoningEffort)
}
// The mode filled options.maxTurns, so the turn budget must not be displaced.
effective, displaced := applyProfileTurnBudget(profile, options.maxTurns, options.maxTurns)
if effective != 160 || displaced != 0 {
t.Fatalf("turn budget = (%d, displaced %d), the mode's 160 must win with nothing displaced", effective, displaced)
}
}
func TestApplyProfileTurnBudget(t *testing.T) {
fast, _ := execprofile.Lookup("fast")
balanced, _ := execprofile.Lookup("balanced")
// No explicit flag: the profile displaces the resolved budget.
if effective, displaced := applyProfileTurnBudget(fast, 0, 80); effective != 30 || displaced != 80 {
t.Fatalf("fast over resolved 80 = (%d, %d), want (30, 80)", effective, displaced)
}
// Explicit --max-turns pins the budget; the profile backs off entirely.
if effective, displaced := applyProfileTurnBudget(fast, 50, 50); effective != 50 || displaced != 0 {
t.Fatalf("fast with explicit 50 = (%d, %d), want (50, 0)", effective, displaced)
}
// Balanced never displaces anything.
if effective, displaced := applyProfileTurnBudget(balanced, 0, 80); effective != 80 || displaced != 0 {
t.Fatalf("balanced over resolved 80 = (%d, %d), want (80, 0)", effective, displaced)
}
}
// The no-regression invariant at the options level: selecting balanced leaves
// the options byte-identical to not selecting a profile at all, and produces
// no loop policy.
func TestExecProfileBalancedLeavesOptionsUntouched(t *testing.T) {
options := execOptions{execProfile: "balanced"}
reference := options
profile, _, err := applyExecProfile(&options)
if err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if !reflect.DeepEqual(options, reference) {
t.Fatalf("balanced changed the options: %+v vs %+v", options, reference)
}
if policy := profile.Policy(80, false); policy != nil {
t.Fatalf("balanced must produce a nil policy, got %+v", policy)
}
}
// --exec-profile= with an inline empty value must error like its two-token
// form and the sibling --mode=, not silently run the default posture (the
// classic shell hazard: an unset variable expanding into the inline form).
func TestParseExecProfileInlineEmptyIsError(t *testing.T) {
_, _, err := parseExecArgs([]string{"--exec-profile=", "hello"})
if err == nil || !strings.Contains(err.Error(), "--exec-profile requires a value") {
t.Fatalf("expected a required-value error, got %v", err)
}
}
// Full echo-provider run: the selected profile must be stamped into the
// per-turn trace so benchmark attribution can group runs by posture.
func TestRunExecTraceRecordsSelectedProfile(t *testing.T) {
tracePath := filepath.Join(t.TempDir(), "trace.ndjson")
exitCode, _, stderr := runExecWithEcho(t, []string{
"exec", "--exec-profile", "fast", "--trace", tracePath, "hello",
})
if exitCode != exitSuccess {
t.Fatalf("expected exit %d, got %d: %s", exitSuccess, exitCode, stderr)
}
raw, err := os.ReadFile(tracePath)
if err != nil {
t.Fatalf("read trace: %v", err)
}
if !strings.Contains(string(raw), `"profile":"fast"`) {
t.Fatalf("trace must record the selected profile, got %s", raw)
}
}
// An explicit --spec-reasoning-effort governs the spec draft's effort, so the
// escalation effort-restore must not arm there even when the profile filled
// the (unused) main effort.
func TestSpecProfileEffortFilled(t *testing.T) {
if !specProfileEffortFilled(true, "") {
t.Fatal("no explicit spec effort: the profile's fill governs, restore must arm")
}
if specProfileEffortFilled(true, "high") {
t.Fatal("an explicit spec effort must disarm the effort restore")
}
if specProfileEffortFilled(false, "") {
t.Fatal("nothing filled, nothing to restore")
}
}
// The trace label deliberately records the SELECTED profile, balanced
// included, so captures are self-describing. Run behavior stays identical
// (balanced fills nothing and arms no policy); the label is the one intended
// difference in the opt-in trace artifact.
func TestRunExecTraceRecordsBalancedSelection(t *testing.T) {
tracePath := filepath.Join(t.TempDir(), "trace.ndjson")
exitCode, _, stderr := runExecWithEcho(t, []string{
"exec", "--exec-profile", "balanced", "--trace", tracePath, "hello",
})
if exitCode != exitSuccess {
t.Fatalf("expected exit %d, got %d: %s", exitSuccess, exitCode, stderr)
}
raw, err := os.ReadFile(tracePath)
if err != nil {
t.Fatalf("read trace: %v", err)
}
if !strings.Contains(string(raw), `"profile":"balanced"`) {
t.Fatalf("trace must record the balanced selection, got %s", raw)
}
}
func TestRunExecUnknownExecProfileIsUsageError(t *testing.T) {
exitCode, _, stderr := runExecWithEcho(t, []string{
"exec", "--exec-profile", "turbo", "hello",
})
if exitCode != exitUsage {
t.Fatalf("expected usage exit %d, got %d", exitUsage, exitCode)
}
if !strings.Contains(stderr, "unknown execution profile") {
t.Fatalf("expected an unknown-profile usage error, got %q", stderr)
}
}
+4 -1
View File
@@ -72,8 +72,11 @@ func preflightExecSession(options execOptions) error {
if session == nil {
return execUsageError{"Zero session not found: " + options.resume}
}
if !sessions.IsResumableKind(session.SessionKind) {
return execUsageError{"Zero session is not resumable: " + options.resume}
}
case options.resumeLatest:
latest, err := store.Latest()
latest, err := store.LatestResumable()
if err != nil {
return err
}
+26
View File
@@ -0,0 +1,26 @@
package cli
import (
"strings"
"testing"
"github.com/Gitlawb/zero/internal/sessions"
)
func TestPreflightRejectsSideSessions(t *testing.T) {
t.Setenv("XDG_DATA_HOME", t.TempDir())
store := sessions.NewStore(sessions.StoreOptions{})
if _, err := store.Create(sessions.CreateInput{SessionID: "side", SessionKind: sessions.SessionKindSide}); err != nil {
t.Fatalf("create side session: %v", err)
}
err := preflightExecSession(execOptions{resumeLatest: true})
if err == nil || !strings.Contains(err.Error(), "No Zero sessions available to resume") {
t.Fatalf("preflightExecSession error = %v, want no resumable sessions", err)
}
err = preflightExecSession(execOptions{resume: "side"})
if err == nil || !strings.Contains(err.Error(), "Zero session is not resumable: side") {
t.Fatalf("explicit side-session preflight error = %v, want non-resumable rejection", err)
}
}
+6
View File
@@ -46,6 +46,11 @@ type execSpecDraftRun struct {
reasoningEffort string
specPermissionMode agent.PermissionMode
notifier *notify.Notifier
// profilePolicy carries the selected execution profile's escalation policy
// (nil when none): the profile displaces resolved.MaxTurns before this
// path branches off, so the spec-draft run needs the same safety net as
// the main run.
profilePolicy *agent.ProfilePolicy
}
type execSpecDraftInfo struct {
@@ -119,6 +124,7 @@ func runExecSpecDraft(run execSpecDraftRun) int {
ProviderName: run.resolved.Provider.Name,
Model: run.resolved.Provider.Model,
ReasoningEffort: run.reasoningEffort,
Profile: run.profilePolicy,
Cwd: run.workspaceRoot,
SystemPrompt: specmode.DraftSystemPrompt,
Images: run.images,
+4
View File
@@ -549,10 +549,14 @@ func TestExecMemberAutoToolListIncludesMutators(t *testing.T) {
}
func TestRunExecAcceptsLegacyModelProfileFlags(t *testing.T) {
// --profile (legacy, inert model profile) and --exec-profile (execution
// profile) are distinct flags and must coexist on one invocation.
exitCode, stdout, stderr := runExecWithEcho(t, []string{
"exec",
"--profile",
"fast",
"--exec-profile",
"balanced",
"--reasoning-effort",
"low",
"hello",
+125
View File
@@ -0,0 +1,125 @@
package cli
import (
"bytes"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/Gitlawb/zero/internal/config"
"github.com/Gitlawb/zero/internal/providers/openai"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
// TestRunExecOptimizedSessionUnderGate proves the end-to-end wiring: with
// ZERO_OPENAI_TURN_SESSION on and a real official-OpenAI provider, a headless
// run streams through the optimized session — the server sees the prewarm HEAD
// probe in addition to the turn's POST.
func TestRunExecOptimizedSessionUnderGate(t *testing.T) {
t.Setenv("XDG_DATA_HOME", t.TempDir())
t.Setenv("ZERO_OPENAI_TURN_SESSION", "1")
cwd := t.TempDir()
var heads, posts atomic.Int64
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodHead:
heads.Add(1)
w.WriteHeader(http.StatusMethodNotAllowed)
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/chat/completions"):
posts.Add(1)
w.Header().Set("Content-Type", "text/event-stream")
_, _ = w.Write([]byte("data: {\"choices\":[{\"delta\":{\"content\":\"done\"}}]}\n\n"))
_, _ = w.Write([]byte("data: [DONE]\n\n"))
default:
w.WriteHeader(http.StatusNotFound)
}
}))
defer server.Close()
var stdout, stderr bytes.Buffer
exitCode := runWithDeps([]string{"exec", "say done"}, &stdout, &stderr, appDeps{
getwd: func() (string, error) { return cwd, nil },
resolveConfig: func(string, config.Overrides) (config.ResolvedConfig, error) {
return config.ResolvedConfig{
ActiveProvider: "official-openai",
Provider: config.ProviderProfile{
Name: "official-openai",
ProviderKind: config.ProviderKindOpenAI,
BaseURL: server.URL,
APIKey: "sk-exec-test",
Model: "pr8-exec-model",
},
}, nil
},
newProvider: func(profile config.ProviderProfile) (zeroruntime.Provider, error) {
// Pin a keep-alive transport so the probe fires on every platform:
// the shared default transport disables keep-alives on macOS, where
// the session correctly skips the prewarm.
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.DisableKeepAlives = false
return openai.New(openai.Options{
APIKey: profile.APIKey,
BaseURL: profile.BaseURL,
Model: profile.Model,
HTTPClient: &http.Client{Transport: transport},
})
},
})
if exitCode != exitSuccess {
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
}
if got := posts.Load(); got < 1 {
t.Fatalf("server saw %d chat-completions POSTs, want >= 1", got)
}
// The prewarm probe is asynchronous; it fires well before the model turn
// finishes, but poll rather than assuming scheduling order. The deadline
// exceeds the production prewarmTimeout (3s) so a slow probe cannot flake
// this assertion.
deadline := time.Now().Add(5 * time.Second)
for heads.Load() == 0 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if got := heads.Load(); got != 1 {
t.Fatalf("server saw %d prewarm HEAD probes, want exactly 1", got)
}
}
// TestRunExecGateOnFallsBackForFakeProvider proves fallback safety: with the
// gate on but a provider that is not the concrete *openai.Provider, the run
// proceeds on the default path exactly as today.
func TestRunExecGateOnFallsBackForFakeProvider(t *testing.T) {
t.Setenv("XDG_DATA_HOME", t.TempDir())
t.Setenv("ZERO_OPENAI_TURN_SESSION", "1")
cwd := t.TempDir()
var builds int
var stdout, stderr bytes.Buffer
exitCode := runWithDeps([]string{"exec", "--model", "claude-haiku-4.5", "hello"}, &stdout, &stderr, appDeps{
getwd: func() (string, error) { return cwd, nil },
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
model := "claude-haiku-4.5"
if overrides.Provider.Model != "" {
model = overrides.Provider.Model
}
cfg := execResolvedConfig()
cfg.Provider.ProviderKind = config.ProviderKindAnthropic
cfg.Provider.Model = model
cfg.MaxTurns = 3
return cfg, nil
},
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
builds++
return &escalatingExecProvider{}, nil
},
})
if exitCode != exitSuccess {
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
}
if builds != 1 {
t.Fatalf("newProvider called %d times, want 1 (default path, no optimized session)", builds)
}
}
+5 -1
View File
@@ -153,6 +153,9 @@ func (writer *execEventWriter) toolResult(result agent.ToolResult) {
if len(result.Meta) > 0 {
payload["meta"] = result.Meta
}
if result.Truncated {
payload["truncated"] = true
}
if result.Redacted {
payload["redacted"] = true
}
@@ -166,7 +169,8 @@ func (writer *execEventWriter) toolResult(result agent.ToolResult) {
return
}
if writer.format == execOutputStreamJSON {
output, truncated := truncateForStreamJSONOutput(result.Output)
output, surfaceTruncated := truncateForStreamJSONOutput(result.Output)
truncated := result.Truncated || surfaceTruncated
event := streamjson.Event{
Type: streamjson.EventToolResult,
RunID: writer.runID,
+36
View File
@@ -1,8 +1,12 @@
package cli
import (
"bytes"
"encoding/json"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/agent"
"github.com/Gitlawb/zero/internal/tools"
)
@@ -16,3 +20,35 @@ func TestStreamJSONSideEffectReportsNoneForControlTool(t *testing.T) {
t.Fatalf("streamJSONSideEffect(escalate_model) = %q, want none", got)
}
}
func TestExecWriterPropagatesToolResultTruncation(t *testing.T) {
for _, format := range []execOutputFormat{execOutputJSON, execOutputStreamJSON} {
t.Run(string(format), func(t *testing.T) {
var stdout, stderr bytes.Buffer
writer := execEventWriter{
stdout: &stdout,
stderr: &stderr,
format: format,
runID: "run_budget",
streamedText: &strings.Builder{},
}
writer.toolResult(agent.ToolResult{
ToolCallID: "call_budget",
Name: "read_file",
Status: tools.StatusOK,
Output: "bounded output",
Truncated: true,
})
if writer.err != nil {
t.Fatalf("toolResult: %v", writer.err)
}
var payload map[string]any
if err := json.Unmarshal(bytes.TrimSpace(stdout.Bytes()), &payload); err != nil {
t.Fatalf("decode output %q: %v", stdout.String(), err)
}
if payload["truncated"] != true {
t.Fatalf("truncated = %#v, want true; payload=%#v", payload["truncated"], payload)
}
})
}
}
@@ -17,7 +17,6 @@ func TestProviderCatalogRuntimeProvidersShowOnboardingSetupHints(t *testing.T) {
defaultModel string
}{
{id: "openai", name: "OpenAI", transport: "openai", defaultModel: "gpt-4.1"},
{id: "aimlapi", name: "AI/ML API", transport: "openai-compatible", defaultModel: "openai/gpt-5-chat"},
{id: "groq", name: "Groq", transport: "openai-compatible", defaultModel: "llama-3.3-70b-versatile"},
{id: "longcat", name: "LongCat", transport: "openai-compatible", defaultModel: "LongCat-2.0"},
{id: "ollama-cloud", name: "Ollama Cloud", transport: "openai-compatible", defaultModel: "qwen3-coder:480b"},
@@ -48,21 +47,6 @@ func TestProviderCatalogRuntimeProvidersShowOnboardingSetupHints(t *testing.T) {
}
}
func TestProviderCatalogAIMLAPIRequiresAPIKey(t *testing.T) {
output := runProviderCatalogOnboarding(t)
block := providerCatalogOnboardingBlock(t, output, "aimlapi")
for _, want := range []string{
"requiresAuth=true",
"authEnvVars=AIMLAPI_API_KEY",
"setup: zero providers setup aimlapi --set-active",
} {
if !strings.Contains(block, want) {
t.Fatalf("expected AIMLAPI catalog block to contain %q, got:\n%s", want, block)
}
}
}
func TestProviderCatalogLongCatRequiresAPIKey(t *testing.T) {
output := runProviderCatalogOnboarding(t)
block := providerCatalogOnboardingBlock(t, output, "longcat")
+62 -3
View File
@@ -3,8 +3,10 @@ package cli
import (
"fmt"
"io"
"net/http"
"strings"
"github.com/Gitlawb/zero/internal/aimlapi"
"github.com/Gitlawb/zero/internal/config"
"github.com/Gitlawb/zero/internal/providercatalog"
"github.com/Gitlawb/zero/internal/providerhealth"
@@ -424,22 +426,39 @@ func providerProfileForAdd(options providerAddOptions) (config.ProviderProfile,
if apiKeyEnv == "" && len(descriptor.AuthEnvVars) > 0 {
apiKeyEnv = descriptor.AuthEnvVars[0]
}
baseURL := firstNonEmptyCLI(options.baseURL, descriptor.DefaultBaseURL)
var catalogHeaders map[string]string
if sameProviderSetupBaseURL(baseURL, descriptor.DefaultBaseURL) {
catalogHeaders = descriptor.CustomHeaders
if strings.EqualFold(strings.TrimSpace(descriptor.ID), "aimlapi") {
catalogHeaders = aimlapi.WithResolvedPartnerHeader(catalogHeaders)
}
}
profile := config.ProviderProfile{
Name: name,
ProviderKind: providerKindForDescriptor(descriptor),
CatalogID: descriptor.ID,
BaseURL: firstNonEmptyCLI(options.baseURL, descriptor.DefaultBaseURL),
BaseURL: baseURL,
APIKeyEnv: apiKeyEnv,
APIFormat: firstAPIFormat(descriptor),
AuthHeader: strings.TrimSpace(options.authHeader),
AuthScheme: normalizeAuthScheme(options.authScheme),
AuthHeaderValue: strings.TrimSpace(options.authHeaderValue),
CustomHeaders: copyProviderHeaders(options.customHeaders),
Model: firstNonEmptyCLI(options.model, descriptor.DefaultModel),
// Catalog attribution belongs only to the catalog endpoint. Explicit
// user headers still apply to an override/staging/proxy endpoint.
CustomHeaders: mergeProviderHeaders(catalogHeaders, options.customHeaders),
Model: firstNonEmptyCLI(options.model, descriptor.DefaultModel),
}
return profile, nil
}
func sameProviderSetupBaseURL(left string, right string) bool {
return strings.EqualFold(
strings.TrimRight(strings.TrimSpace(left), "/"),
strings.TrimRight(strings.TrimSpace(right), "/"),
)
}
func selectProviderForCheck(resolved config.ResolvedConfig, name string) (config.ProviderProfile, error) {
name = strings.TrimSpace(name)
if name == "" {
@@ -544,3 +563,43 @@ func copyProviderHeaders(headers map[string]string) map[string]string {
}
return copied
}
func mergeProviderHeaders(base map[string]string, overrides map[string]string) map[string]string {
merged := copyProviderHeaders(base)
if len(overrides) == 0 {
return merged
}
if merged == nil {
merged = map[string]string{}
}
for key, value := range overrides {
// HTTP header names are case-insensitive and request construction
// canonicalizes them (http.Header.Set), so an override that differs from a
// base key only in case must replace that key's value in place — otherwise
// both survive as distinct map entries and race non-deterministically at
// build time. Keep the existing spelling (the catalog's, which the resolver
// re-merges verbatim); only a genuinely new header is added as typed.
if existing, ok := headerKeyFold(merged, key); ok {
merged[existing] = value
continue
}
merged[key] = value
}
return merged
}
// headerKeyFold returns the key in headers whose canonical HTTP form matches key,
// so a case-insensitive override lands on the existing entry instead of adding a
// colliding duplicate.
func headerKeyFold(headers map[string]string, key string) (string, bool) {
if _, ok := headers[key]; ok {
return key, true
}
canonical := http.CanonicalHeaderKey(key)
for existing := range headers {
if http.CanonicalHeaderKey(existing) == canonical {
return existing, true
}
}
return "", false
}
@@ -0,0 +1,23 @@
package cli
import (
"reflect"
"testing"
"github.com/Gitlawb/zero/internal/config"
)
func TestProviderSensitiveEnvKeys(t *testing.T) {
resolved := config.ResolvedConfig{
Providers: []config.ProviderProfile{
{Name: "custom", APIKeyEnv: " COMPANY_LLM_SECRET "},
{Name: "catalog", APIKeyEnv: "OPENAI_API_KEY"},
{Name: "inline"},
},
Provider: config.ProviderProfile{Name: "active", APIKeyEnv: "ACTIVE_PROVIDER_SECRET"},
}
want := []string{"COMPANY_LLM_SECRET", "OPENAI_API_KEY", "ACTIVE_PROVIDER_SECRET"}
if got := providerSensitiveEnvKeys(resolved); !reflect.DeepEqual(got, want) {
t.Fatalf("providerSensitiveEnvKeys() = %v, want %v", got, want)
}
}
+3 -3
View File
@@ -216,10 +216,10 @@ func parseNonEmptySessionsFlag(flag string, value string) (string, error) {
func parseSessionKindFlag(value string) (sessions.SessionKind, error) {
kind := sessions.SessionKind(strings.ToLower(strings.TrimSpace(value)))
switch kind {
case sessions.SessionKindFork, sessions.SessionKindChild, sessions.SessionKindSpecDraft, sessions.SessionKindSpecImpl:
case sessions.SessionKindFork, sessions.SessionKindChild, sessions.SessionKindSide, sessions.SessionKindSpecDraft, sessions.SessionKindSpecImpl:
return kind, nil
default:
return "", execUsageError{fmt.Sprintf("invalid --kind %q. Expected fork, child, spec-draft, or spec-impl.", value)}
return "", execUsageError{fmt.Sprintf("invalid --kind %q. Expected fork, child, side, spec-draft, or spec-impl.", value)}
}
}
@@ -535,7 +535,7 @@ Commands:
Flags:
--json Print JSON output
--kind <kind> Filter list by fork, child, spec-draft, or spec-impl
--kind <kind> Filter list by fork, child, side, spec-draft, or spec-impl
--sequence <n> Rewind target sequence (rewind-plan, rewind)
--event <id> Rewind target event id (rewind-plan, rewind)
--exclude-target Drop the target event (rewind-plan, rewind)
+14
View File
@@ -233,6 +233,9 @@ func TestRunSessionsListFiltersByKind(t *testing.T) {
}); err != nil {
t.Fatal(err)
}
if _, err := store.Create(sessions.CreateInput{SessionID: "side", SessionKind: sessions.SessionKindSide, Title: "BTW side"}); err != nil {
t.Fatal(err)
}
var stdout bytes.Buffer
var stderr bytes.Buffer
@@ -249,6 +252,17 @@ func TestRunSessionsListFiltersByKind(t *testing.T) {
t.Fatalf("filtered sessions output = %q", output)
}
stdout.Reset()
stderr.Reset()
exitCode = runWithDeps([]string{"sessions", "list", "--kind", "side"}, &stdout, &stderr, appDeps{
newSessionStore: func() *sessions.Store {
return store
},
})
if exitCode != exitSuccess || !strings.Contains(stdout.String(), "side") || strings.Contains(stdout.String(), "regular") {
t.Fatalf("sessions list --kind side exit=%d stdout=%q stderr=%q", exitCode, stdout.String(), stderr.String())
}
stdout.Reset()
stderr.Reset()
exitCode = runWithDeps([]string{"sessions", "tree", "draft", "--kind", "spec-draft"}, &stdout, &stderr, appDeps{
+48
View File
@@ -200,6 +200,54 @@ func TestSaveSetupProviderStoresPastedAPIKey(t *testing.T) {
}
}
func TestSaveSetupProviderAimlapiEnvReferenceAndOverrideHeaders(t *testing.T) {
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
result, err := saveSetupProvider(appDeps{
userConfigPath: func() (string, error) { return configPath, nil },
}, tui.SetupSelection{
CatalogID: "aimlapi",
BaseURL: "https://staging.example/v1",
Model: "anthropic/claude-sonnet-5",
}, setupSaveOptions{})
if err != nil {
t.Fatalf("saveSetupProvider() error = %v", err)
}
if result.Provider.APIKey != "" || result.Provider.APIKeyEnv != "AIMLAPI_API_KEY" {
t.Fatalf("runtime profile did not retain AIMLAPI_API_KEY reference: %+v", result.Provider)
}
if len(result.Provider.CustomHeaders) != 0 {
t.Fatalf("catalog headers leaked to first-run override: %#v", result.Provider.CustomHeaders)
}
persisted := readFileConfig(t, configPath).Providers[0]
if persisted.APIKey != "" || persisted.APIKeyEnv != "AIMLAPI_API_KEY" || persisted.APIKeyStored {
t.Fatalf("persisted credential source changed: %+v", persisted)
}
if len(persisted.CustomHeaders) != 0 {
t.Fatalf("persisted override contains catalog headers: %#v", persisted.CustomHeaders)
}
}
func TestSaveSetupProviderAimlapiUsesResolvedPartnerOverride(t *testing.T) {
t.Setenv("AIMLAPI_PARTNER_ID", "part_override")
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
result, err := saveSetupProvider(appDeps{
userConfigPath: func() (string, error) { return configPath, nil },
}, tui.SetupSelection{
CatalogID: "aimlapi",
BaseURL: "https://api.aimlapi.com/v1",
Model: "anthropic/claude-sonnet-5",
}, setupSaveOptions{})
if err != nil {
t.Fatalf("saveSetupProvider() error = %v", err)
}
if got := result.Provider.CustomHeaders["X-AIMLAPI-Partner-ID"]; got != "part_override" {
t.Fatalf("runtime partner header = %q, want part_override", got)
}
if got := readFileConfig(t, configPath).Providers[0].CustomHeaders["X-AIMLAPI-Partner-ID"]; got != "part_override" {
t.Fatalf("persisted partner header = %q, want part_override", got)
}
}
func TestSaveSetupProviderStoresCustomEndpointSelection(t *testing.T) {
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
+32 -6
View File
@@ -6,7 +6,7 @@ import (
"strings"
)
func mergeMCPConfig(dst *MCPConfig, src MCPConfig) {
func mergeMCPConfig(dst *MCPConfig, src MCPConfig, canReenable bool) {
if len(src.Servers) == 0 {
return
}
@@ -14,10 +14,14 @@ func mergeMCPConfig(dst *MCPConfig, src MCPConfig) {
dst.Servers = map[string]MCPServerConfig{}
}
for name, server := range src.Servers {
dst.Servers[name] = mergeMCPServer(dst.Servers[name], server)
dst.Servers[name] = mergeMCPServer(dst.Servers[name], server, canReenable)
}
}
// mergeProjectMCPConfig merges a project-scoped (lower-trust) MCP config into
// dst. It never re-enables a server the user explicitly disabled or disabled a
// server the user explicitly enabled: it merges with canReenable=false so the
// trust-boundary guard in mergeMCPServer keeps the user's decision authoritative.
func mergeProjectMCPConfig(dst *MCPConfig, src MCPConfig) error {
if len(src.Servers) == 0 {
return nil
@@ -27,7 +31,7 @@ func mergeProjectMCPConfig(dst *MCPConfig, src MCPConfig) error {
}
for name, server := range src.Servers {
base := dst.Servers[name]
candidate := mergeMCPServer(base, server)
candidate := mergeMCPServer(base, server, false)
if projectMCPServerTargetChanges(base, server) && hasInheritedMCPCredentialMaterial(server, candidate) {
return fmt.Errorf("project MCP server %q cannot override target while inheriting user credentials; set headers/env/oauth explicitly or use a new server name", name)
}
@@ -37,7 +41,11 @@ func mergeProjectMCPConfig(dst *MCPConfig, src MCPConfig) error {
return nil
}
func mergeMCPServer(base MCPServerConfig, next MCPServerConfig) MCPServerConfig {
// mergeMCPServer merges a later config layer's MCP server into the base. The
// canReenable flag marks the CLI-override scope, the only layer that is allowed
// to lift a sticky user-level disable/enable decision (see the disabled-handling
// block).
func mergeMCPServer(base MCPServerConfig, next MCPServerConfig, canReenable bool) MCPServerConfig {
if strings.TrimSpace(next.Type) != "" {
base.Type = next.Type
}
@@ -62,8 +70,26 @@ func mergeMCPServer(base MCPServerConfig, next MCPServerConfig) MCPServerConfig
if next.OAuth != nil {
base.OAuth = next.OAuth
}
// Capture the higher-trust scope's prior decision before folding in
// next.disabledSet. The trust boundary must be evaluated against the state
// the higher-trust layer actually left behind, not against next's own flag.
baseDisabledSet := base.disabledSet
baseDisabled := base.Disabled
if next.disabledSet {
base.disabledSet = true
}
if next.disabledSet || next.Disabled {
base.Disabled = next.Disabled
// Trust boundary: once a higher-trust scope has explicitly set the
// disabled flag, a lower-trust layer (project config) cannot override
// that decision in either direction — the user's choice (whether to
// enable or disable a server) wins over the repo. This blocks a repo
// from re-enabling a server the user disabled, and from disabling a
// server the user explicitly enabled. The only layer allowed to
// override an explicit higher-scope decision is the CLI override scope
// (canReenable=true).
if canReenable || !(baseDisabledSet && baseDisabled != next.Disabled) {
base.Disabled = next.Disabled
}
}
if next.configured {
base.configured = true
@@ -75,7 +101,7 @@ func mergeMCPServer(base MCPServerConfig, next MCPServerConfig) MCPServerConfig
}
func projectMCPServerTargetChanges(base MCPServerConfig, next MCPServerConfig) bool {
if strings.TrimSpace(next.Type) != "" && mcpServerTransportKind(base) != mcpServerTransportKind(mergeMCPServer(base, next)) {
if strings.TrimSpace(next.Type) != "" && mcpServerTransportKind(base) != mcpServerTransportKind(mergeMCPServer(base, next, true)) {
return true
}
if strings.TrimSpace(next.URL) != "" && strings.TrimSpace(next.URL) != strings.TrimSpace(base.URL) {
+47 -4
View File
@@ -9,6 +9,7 @@ import (
"strconv"
"strings"
"github.com/Gitlawb/zero/internal/aimlapi"
"github.com/Gitlawb/zero/internal/modelregistry"
"github.com/Gitlawb/zero/internal/notify"
"github.com/Gitlawb/zero/internal/providercatalog"
@@ -170,7 +171,10 @@ func ResolveMCP(options ResolveOptions) (MCPConfig, error) {
if err != nil {
return MCPConfig{}, err
}
mergeMCPConfig(&cfg.MCP, fileConfig.MCP)
// User config is higher-trust than project config: it may re-enable a
// server the user disabled (a user-level disable is sticky, but the user
// scope itself may lift it).
mergeMCPConfig(&cfg.MCP, fileConfig.MCP, true)
}
// Drop the project layer when the workspace is untrusted, so a cloned repo's
// ./.zero/config.json cannot register (and spawn) MCP servers. Fail-closed:
@@ -184,7 +188,7 @@ func ResolveMCP(options ResolveOptions) (MCPConfig, error) {
return MCPConfig{}, err
}
}
mergeMCPConfig(&cfg.MCP, options.Overrides.MCP)
mergeMCPConfig(&cfg.MCP, options.Overrides.MCP, true)
return cfg.MCP, nil
}
@@ -211,7 +215,7 @@ func mergeConfig(dst *FileConfig, src FileConfig) {
for _, provider := range src.Providers {
mergeProvider(dst, provider)
}
mergeMCPConfig(&dst.MCP, src.MCP)
mergeMCPConfig(&dst.MCP, src.MCP, true)
if network := strings.TrimSpace(src.Sandbox.Network); network != "" {
dst.Sandbox.Network = network
}
@@ -709,7 +713,7 @@ func applyOverrides(cfg *FileConfig, overrides Overrides) {
if hasProviderFields(overrides.Provider) {
mergeProvider(cfg, overrides.Provider)
}
mergeMCPConfig(&cfg.MCP, overrides.MCP)
mergeMCPConfig(&cfg.MCP, overrides.MCP, true)
}
func mergeLocalControlConfig(dst *LocalControlConfig, src LocalControlConfig) {
@@ -1039,6 +1043,45 @@ func applyCatalogDescriptor(profile *ProviderProfile, descriptor providercatalog
if profile.APIKeyEnv == "" && len(descriptor.AuthEnvVars) > 0 && (!explicitBaseURL || sameBaseURL(profile.BaseURL, descriptor.DefaultBaseURL)) {
profile.APIKeyEnv = descriptor.AuthEnvVars[0]
}
canonicalCatalogEndpoint := !explicitBaseURL || sameBaseURL(profile.BaseURL, descriptor.DefaultBaseURL)
if len(descriptor.CustomHeaders) > 0 && canonicalCatalogEndpoint {
catalogHeaders := descriptor.CustomHeaders
if strings.EqualFold(strings.TrimSpace(descriptor.ID), "aimlapi") {
catalogHeaders = aimlapi.WithResolvedPartnerHeader(catalogHeaders)
}
merged := copyStringMap(catalogHeaders)
for key, value := range profile.CustomHeaders {
// Header names are case-insensitive. Preserve the catalog spelling while
// replacing its value so request construction cannot see two colliding
// map entries whose eventual winner depends on iteration order.
existingKey := ""
for candidate := range merged {
if strings.EqualFold(candidate, key) {
existingKey = candidate
break
}
}
if existingKey != "" {
merged[existingKey] = value
continue
}
merged[key] = value
}
profile.CustomHeaders = merged
} else if strings.EqualFold(strings.TrimSpace(descriptor.ID), "aimlapi") && !canonicalCatalogEndpoint {
// AIMLAPI attribution is owned by the catalog endpoint. A profile can retain
// those generated headers after its base URL is edited; strip their names
// before sending requests to an arbitrary staging/proxy host while preserving
// unrelated headers explicitly supplied by the user.
for profileKey := range profile.CustomHeaders {
for catalogKey := range descriptor.CustomHeaders {
if strings.EqualFold(profileKey, catalogKey) {
delete(profile.CustomHeaders, profileKey)
break
}
}
}
}
}
func sameBaseURL(left string, right string) bool {
+190 -3
View File
@@ -9,6 +9,7 @@ import (
"testing"
"github.com/Gitlawb/zero/internal/modelregistry"
"github.com/Gitlawb/zero/internal/providercatalog"
)
func TestResolveAppliesLayerPrecedence(t *testing.T) {
@@ -554,7 +555,7 @@ func TestResolveReplacesMCPServerOverlayCollections(t *testing.T) {
}
}
func TestResolveMCPServerLayersCanClearAndReenable(t *testing.T) {
func TestResolveMCPServerLayersCannotReenableUserDisabled(t *testing.T) {
userPath := writeConfig(t, `{
"mcp": {
"servers": {
@@ -588,9 +589,11 @@ func TestResolveMCPServerLayersCanClearAndReenable(t *testing.T) {
}
docs := resolved.MCP.Servers["docs"]
if docs.Disabled {
t.Fatalf("docs.Disabled = true, want project layer to re-enable")
// A user-level disable is sticky: project config must not re-enable it.
if !docs.Disabled {
t.Fatal("docs.Disabled = false, want user-level disable to remain sticky")
}
// Non-disable fields from the project layer still merge normally.
if len(docs.Args) != 0 {
t.Fatalf("docs.Args = %#v, want project layer to clear inherited args", docs.Args)
}
@@ -599,6 +602,125 @@ func TestResolveMCPServerLayersCanClearAndReenable(t *testing.T) {
}
}
func TestMergeMCPStickyDisableReenable(t *testing.T) {
disabled := MCPServerConfig{Disabled: true, disabledSet: true}
// A lower-trust (project) layer must not lift a sticky user disable.
project := MCPServerConfig{Disabled: false, disabledSet: true}
if got := mergeMCPServer(disabled, project, false); !got.Disabled {
t.Fatal("project layer re-enabled a sticky user disable")
}
// The user scope (explicit mcp enable command / CLI override) may re-enable.
override := MCPServerConfig{Disabled: false, disabledSet: true}
if got := mergeMCPServer(disabled, override, true); got.Disabled {
t.Fatal("user-scope override must be able to re-enable")
}
// A user-scope disable stacked on a project attempt still sticks.
if got := mergeMCPServer(disabled, project, false); !got.Disabled {
t.Fatal("sticky disable must survive a project re-enable attempt")
}
// Baseline: a project layer may disable a default-enabled server when no
// higher-trust scope has explicitly set disabled (the trust boundary must
// not block legitimate project disables). Regression guard for the
// disabledSet-before-check ordering bug.
defaultEnabled := MCPServerConfig{Disabled: false, disabledSet: false}
projectDisable := MCPServerConfig{Disabled: true, disabledSet: true}
if got := mergeMCPServer(defaultEnabled, projectDisable, false); !got.Disabled {
t.Fatal("project layer should be able to disable a default-enabled server")
}
// And an empty project layer must not flip an unconfigured server to disabled.
if got := mergeMCPServer(MCPServerConfig{}, MCPServerConfig{}, false); got.Disabled {
t.Fatal("an empty project layer must not disable an unconfigured server")
}
}
// TestResolveMCPCannotReenableUserDisabled exercises the trust boundary at the
// actual ResolveMCP entry point: a project config (project scope) must not lift
// a disable the user set in their higher-trust user config.
func TestResolveMCPCannotReenableUserDisabled(t *testing.T) {
userPath := writeConfig(t, `{
"mcp": {
"servers": {
"docs": {
"type": "stdio",
"command": "docs-mcp",
"disabled": true
}
}
}
}`)
// Project config tries to re-enable it.
projectPath := writeConfig(t, `{
"mcpServers": {
"docs": {
"disabled": false
}
}
}`)
resolved, err := ResolveMCP(ResolveOptions{
UserConfigPath: userPath,
ProjectConfigPath: projectPath,
})
if err != nil {
t.Fatalf("ResolveMCP() error = %v", err)
}
docs, ok := resolved.Servers["docs"]
if !ok {
t.Fatal("docs server missing from resolved config")
}
if !docs.Disabled {
t.Fatalf("docs.Disabled = false, want user-level disable to remain sticky across project re-enable")
}
}
// TestResolveMCPUserLiftsProjectDisabled asserts the reverse direction is open:
// a user config (higher-trust scope) re-enabling a server the project config
// disabled must be permitted, since user > project in the trust hierarchy.
func TestResolveMCPUserLiftsProjectDisabled(t *testing.T) {
// Project config disables the server.
projectPath := writeConfig(t, `{
"mcpServers": {
"docs": {
"type": "stdio",
"command": "docs-mcp",
"disabled": true
}
}
}`)
// User config re-enables it.
userPath := writeConfig(t, `{
"mcp": {
"servers": {
"docs": {
"disabled": false
}
}
}
}`)
resolved, err := ResolveMCP(ResolveOptions{
UserConfigPath: userPath,
ProjectConfigPath: projectPath,
})
if err != nil {
t.Fatalf("ResolveMCP() error = %v", err)
}
docs, ok := resolved.Servers["docs"]
if !ok {
t.Fatal("docs server missing from resolved config")
}
if docs.Disabled {
t.Fatalf("docs.Disabled = true, want user scope to lift the project-level disable")
}
}
func TestResolveRejectsDuplicateMCPRootAliases(t *testing.T) {
path := writeConfig(t, `{
"mcpServers": {
@@ -1374,6 +1496,71 @@ func TestResolveProviderProfileExtendedJSONAliases(t *testing.T) {
}
}
func TestApplyCatalogDescriptorFoldsCustomHeadersCaseInsensitively(t *testing.T) {
t.Setenv("AIMLAPI_PARTNER_ID", "part_env")
descriptor, err := providercatalog.Require("aimlapi")
if err != nil {
t.Fatal(err)
}
profile := ProviderProfile{
BaseURL: descriptor.DefaultBaseURL,
CustomHeaders: map[string]string{
"x-aimlapi-partner-id": "part_override",
},
}
applyCatalogDescriptor(&profile, descriptor, true)
if got := profile.CustomHeaders["X-AIMLAPI-Partner-ID"]; got != "part_override" {
t.Fatalf("canonical header value = %q, want override", got)
}
if _, duplicate := profile.CustomHeaders["x-aimlapi-partner-id"]; duplicate {
t.Fatalf("case-colliding header survived merge: %#v", profile.CustomHeaders)
}
}
func TestApplyCatalogDescriptorUsesAimlapiPartnerEnvironmentOverride(t *testing.T) {
t.Setenv("AIMLAPI_PARTNER_ID", "part_env")
descriptor, err := providercatalog.Require("aimlapi")
if err != nil {
t.Fatal(err)
}
profile := ProviderProfile{BaseURL: descriptor.DefaultBaseURL}
applyCatalogDescriptor(&profile, descriptor, true)
if got := profile.CustomHeaders["X-AIMLAPI-Partner-ID"]; got != "part_env" {
t.Fatalf("partner header = %q, want part_env", got)
}
}
func TestApplyCatalogDescriptorStripsAimlapiAttributionFromRetargetedProfile(t *testing.T) {
descriptor, err := providercatalog.Require("aimlapi")
if err != nil {
t.Fatal(err)
}
profile := ProviderProfile{
BaseURL: "https://proxy.example.test/v1",
CustomHeaders: map[string]string{
"x-aimlapi-partner-id": "persisted-partner",
"X-AIMLAPI-Integration-Repo": "Gitlawb/zero",
"X-AIMLAPI-Integration-Version": "zero",
"X-Environment": "staging",
},
}
applyCatalogDescriptor(&profile, descriptor, true)
for key := range profile.CustomHeaders {
if strings.HasPrefix(strings.ToLower(key), "x-aimlapi-") {
t.Fatalf("catalog attribution survived retargeting: %#v", profile.CustomHeaders)
}
}
if profile.CustomHeaders["X-Environment"] != "staging" {
t.Fatalf("user header was removed: %#v", profile.CustomHeaders)
}
}
func TestResolveProviderProfileParseThinkTagsFalseAlias(t *testing.T) {
path := writeConfig(t, `{
"activeProvider": "custom",
+153
View File
@@ -0,0 +1,153 @@
// Package execprofile defines named execution profiles: bundles of loop-posture
// knobs (turn budget, reasoning effort, self-correction, escalation triggers)
// that tune how hard the agent loop tries, independent of which model runs.
// Profiles COMPOSE with --mode: a mode answers "what runs" (model, tools),
// a profile answers "how hard the loop tries". Selection precedence is
// explicit flag > --mode > profile, enforced by the callers applying profiles
// last with the same fill-only-if-unset rule modes use.
//
// The catalog is deliberately tiny and value-stable: balanced is the empty
// profile (selecting it leaves the run's options, budget, and loop behavior
// identical to an unflagged run — asserted by test; only the opt-in trace and
// bench artifacts record the selected profile name, by design, so captures are
// self-describing), fast trades turn budget and effort for latency with a
// one-shot escalation back to the displaced posture as the safety net, and
// thorough raises the budget and arms full self-correction.
package execprofile
import (
"sort"
"strings"
"github.com/Gitlawb/zero/internal/agent"
"github.com/Gitlawb/zero/internal/sandbox"
)
// Profile is a named execution posture. Zero-valued fields inherit the run's
// existing value (flag, mode, config, or built-in default) — a profile only
// ever fills knobs the caller left unset, except MaxTurns which REPLACES the
// resolved budget (that displacement is what escalation restores).
type Profile struct {
// Name identifies the profile in traces, help text, and diagnostics.
Name string
// MaxTurns replaces the resolved per-run tool-turn budget when > 0 and the
// user did not pass an explicit --max-turns. The displaced resolved value
// becomes the escalation target (see Policy).
MaxTurns int
// ReasoningEffort fills the run's effort when the user left it unset. It
// still flows through the model-supported gating at the call site, so an
// unsupported level degrades exactly like an explicit flag would.
ReasoningEffort string
// SelfCorrect arms the post-edit verify-and-correct loop. Profiles can only
// turn it ON (the flag is presence-only, so false is indistinguishable from
// unset and must never override an explicit opt-in).
SelfCorrect bool
// Escalation triggers. All zero means the profile never escalates and
// Policy returns nil (the loop stays byte-identical to a nil-profile run).
// Targets are NOT part of the catalog: escalation restores the values the
// profile displaced at selection time, never invented ones, so targets are
// stamped by Policy from what the caller measured.
EscalateOnToolFailureStreak int
EscalateOnCompletionUncertain int
EscalateOnSelfCorrectFailure bool
EscalateOnRiskyMutation sandbox.RiskLevel
}
// The catalog. Values marked provisional are floors/ceilings derived from the
// Phase 0 baseline's read-class evidence (successful nav runs used single-digit
// turns; the mutating classes produced no successful samples to tune from) and
// are expected to be re-tuned from the post-oracle-fix re-capture. The
// escalation safety net is what makes shipping provisional Fast values safe:
// a run that hits trouble gets its displaced budget back mid-run.
var (
// Balanced is the default posture: empty on purpose. Selecting it changes
// nothing — the no-regression invariant of the whole feature.
Balanced = Profile{Name: "balanced"}
// Fast starts cheap (30 turns, low effort) and arms every escalation
// trigger so a struggling run restores its displaced posture: two
// same-tool retriable failures in a row, a second uncertain completion
// evaluation, a failing self-correction cycle, or a critical-risk
// mutation. The risk threshold is deliberately Critical, not High: the
// sandbox classifies EVERY shell command as at least high risk before any
// command analysis, so a High trigger would fire on the first `go test`
// of virtually any coding task and spend the one-shot on turn one.
// Critical marks the genuinely scary categories (destructive commands,
// piped installers, out-of-workspace writes, network mutations).
Fast = Profile{
Name: "fast",
MaxTurns: 30,
ReasoningEffort: "low",
EscalateOnToolFailureStreak: 2,
EscalateOnCompletionUncertain: 2,
EscalateOnSelfCorrectFailure: true,
EscalateOnRiskyMutation: sandbox.RiskCritical,
}
// Thorough doubles the default budget, asks for high effort, and arms the
// full post-edit verify-and-correct loop. Already the maximum posture, so
// it has nothing to escalate to.
Thorough = Profile{
Name: "thorough",
MaxTurns: 160,
ReasoningEffort: "high",
SelfCorrect: true,
}
)
var catalog = map[string]Profile{
Balanced.Name: Balanced,
Fast.Name: Fast,
Thorough.Name: Thorough,
}
// Lookup resolves a profile by name, case-insensitively and ignoring
// surrounding whitespace.
func Lookup(name string) (Profile, bool) {
profile, ok := catalog[strings.ToLower(strings.TrimSpace(name))]
return profile, ok
}
// Names returns the catalog's profile names, sorted, for usage errors and help.
func Names() []string {
names := make([]string, 0, len(catalog))
for name := range catalog {
names = append(names, name)
}
sort.Strings(names)
return names
}
// Policy projects the profile into the loop-facing ProfilePolicy.
//
// displacedMaxTurns is the turn budget this profile displaced at selection time
// (what the run would have used without it) and becomes the escalation's
// restore target; pass 0 when the profile did not displace the budget (e.g.
// the user pinned --max-turns explicitly) so escalation leaves the ceiling
// untouched. effortFilled reports whether the profile actually filled the
// run's reasoning effort (it backs off when the user set one explicitly); the
// displaced effort is then "" (the provider default) by construction, which
// the escalation restores via RestoreDefaultEffort — a plain "" target would
// mean "leave untouched".
//
// Profiles with no armed triggers return nil: the loop treats a nil policy as
// "no profile" (no observation, no counters), which keeps balanced and
// thorough runs byte-identical to the same knob values set by hand.
func (p Profile) Policy(displacedMaxTurns int, effortFilled bool) *agent.ProfilePolicy {
if p.EscalateOnToolFailureStreak == 0 && p.EscalateOnCompletionUncertain == 0 &&
!p.EscalateOnSelfCorrectFailure && p.EscalateOnRiskyMutation == "" {
return nil
}
return &agent.ProfilePolicy{
Name: p.Name,
Escalate: &agent.PostureEscalation{
MaxTurns: displacedMaxTurns,
RestoreDefaultEffort: effortFilled,
OnToolFailureStreak: p.EscalateOnToolFailureStreak,
OnCompletionUncertain: p.EscalateOnCompletionUncertain,
OnSelfCorrectFailure: p.EscalateOnSelfCorrectFailure,
OnRiskyMutation: p.EscalateOnRiskyMutation,
},
}
}
+131
View File
@@ -0,0 +1,131 @@
package execprofile
import (
"reflect"
"testing"
"github.com/Gitlawb/zero/internal/sandbox"
)
// The no-regression invariant of the whole feature: balanced must be the empty
// profile (nothing but a name), so selecting it cannot change a run.
func TestBalancedProfileIsEmpty(t *testing.T) {
profile, ok := Lookup("balanced")
if !ok {
t.Fatal("balanced must exist in the catalog")
}
if profile.Name != "balanced" {
t.Fatalf("Name = %q, want %q", profile.Name, "balanced")
}
profile.Name = ""
if !reflect.DeepEqual(profile, Profile{}) {
t.Fatalf("balanced must be zero-valued apart from its name, got %+v", profile)
}
if policy := Balanced.Policy(80, false); policy != nil {
t.Fatalf("balanced Policy must be nil (byte-identical loop), got %+v", policy)
}
}
func TestLookupIsCaseAndSpaceInsensitive(t *testing.T) {
for _, name := range []string{"fast", "Fast", " FAST "} {
if _, ok := Lookup(name); !ok {
t.Fatalf("Lookup(%q) should resolve", name)
}
}
if _, ok := Lookup("turbo"); ok {
t.Fatal("Lookup(turbo) should not resolve")
}
}
func TestNamesAreSorted(t *testing.T) {
want := []string{"balanced", "fast", "thorough"}
if got := Names(); !reflect.DeepEqual(got, want) {
t.Fatalf("Names() = %v, want %v", got, want)
}
}
// Escalation targets must be the displaced values only — the catalog carries
// triggers, the call site supplies what was displaced, and nothing else may
// appear in the escalation.
func TestFastPolicyTargetsDisplacedValuesOnly(t *testing.T) {
policy := Fast.Policy(80, true)
if policy == nil {
t.Fatal("fast must arm an escalation policy")
}
if policy.Name != "fast" {
t.Fatalf("policy.Name = %q, want fast", policy.Name)
}
esc := policy.Escalate
if esc == nil {
t.Fatal("fast policy must carry an escalation")
}
if esc.MaxTurns != 80 {
t.Fatalf("Escalate.MaxTurns = %d, want the displaced 80", esc.MaxTurns)
}
if esc.ReasoningEffort != "" {
t.Fatalf("Escalate.ReasoningEffort = %q, want empty (displaced effort is the provider default by construction)", esc.ReasoningEffort)
}
if !esc.RestoreDefaultEffort {
t.Fatal("effort was profile-filled, so escalation must restore the provider default")
}
if esc.RestoreCompletionGate {
t.Fatal("fast does not touch the completion gate, so escalation must not either")
}
if esc.OnToolFailureStreak != Fast.EscalateOnToolFailureStreak ||
esc.OnCompletionUncertain != Fast.EscalateOnCompletionUncertain ||
esc.OnSelfCorrectFailure != Fast.EscalateOnSelfCorrectFailure ||
esc.OnRiskyMutation != Fast.EscalateOnRiskyMutation {
t.Fatalf("escalation triggers must mirror the catalog, got %+v", esc)
}
}
// The effort restore arms only when the profile actually filled the effort:
// with an explicit user effort the profile backed off, so escalation must not
// clear the user's choice.
func TestFastPolicyExplicitEffortIsNeverRestored(t *testing.T) {
policy := Fast.Policy(80, false)
if policy == nil || policy.Escalate == nil {
t.Fatal("fast must still arm its triggers")
}
if policy.Escalate.RestoreDefaultEffort {
t.Fatal("the profile did not fill the effort, so escalation must leave it alone")
}
}
// A profile that did not displace the budget (explicit --max-turns pinned it)
// must not let escalation move the ceiling at all.
func TestFastPolicyZeroDisplacedLeavesCeilingUntouched(t *testing.T) {
policy := Fast.Policy(0, false)
if policy == nil || policy.Escalate == nil {
t.Fatal("fast must still arm its triggers with a zero displaced budget")
}
if policy.Escalate.MaxTurns != 0 {
t.Fatalf("Escalate.MaxTurns = %d, want 0 (no displaced value to restore)", policy.Escalate.MaxTurns)
}
}
// Thorough is already the maximum posture: no triggers, so no policy — the
// loop must stay byte-identical to the same knobs set by hand.
func TestThoroughPolicyIsNil(t *testing.T) {
if policy := Thorough.Policy(80, true); policy != nil {
t.Fatalf("thorough Policy must be nil, got %+v", policy)
}
}
// Pin the catalog's provisional values so a retune is a deliberate,
// test-visible diff (these floors came from the Phase 0 baseline's read-class
// evidence and are expected to move after the post-oracle-fix re-capture).
func TestCatalogProvisionalValues(t *testing.T) {
if Fast.MaxTurns != 30 || Fast.ReasoningEffort != "low" || Fast.SelfCorrect {
t.Fatalf("fast knobs changed: %+v", Fast)
}
// Critical, not High: the sandbox classifies every shell command as at
// least high risk, so a High trigger would spend the one-shot escalation
// on the first benign `go test` of any coding task.
if Fast.EscalateOnRiskyMutation != sandbox.RiskCritical {
t.Fatalf("fast risky-mutation trigger = %q, want %q", Fast.EscalateOnRiskyMutation, sandbox.RiskCritical)
}
if Thorough.MaxTurns != 160 || Thorough.ReasoningEffort != "high" || !Thorough.SelfCorrect {
t.Fatalf("thorough knobs changed: %+v", Thorough)
}
}
+26
View File
@@ -133,6 +133,26 @@ func discoverAuthorizationServer(ctx context.Context, client *http.Client, baseU
}, nil
}
// validateResolvedEndpoints applies the shared https/loopback endpoint rule to
// every non-empty endpoint in the resolved metadata — configured or discovered
// alike — so discovery metadata can never downgrade an MCP login to an insecure
// or attacker-controlled authorization, token, or registration endpoint.
func validateResolvedEndpoints(metadata authServerMetadata) error {
for _, ep := range []struct{ kind, url string }{
{"authorization", metadata.AuthorizationEndpoint},
{"token", metadata.TokenEndpoint},
{"registration", metadata.RegistrationEndpoint},
} {
if strings.TrimSpace(ep.url) == "" {
continue
}
if err := oauth.ValidateEndpointURL(ep.url); err != nil {
return fmt.Errorf("mcp oauth: %s endpoint: %w", ep.kind, err)
}
}
return nil
}
// resolveAuthorizationServer discovers metadata and applies explicit config
// overrides. Configured endpoints take precedence over discovered ones, and act
// as a fallback when discovery fails or omits a value.
@@ -148,6 +168,9 @@ func resolveAuthorizationServer(ctx context.Context, client *http.Client, baseUR
TokenEndpoint: strings.TrimSpace(cfg.TokenEndpoint),
RegistrationEndpoint: strings.TrimSpace(cfg.RegistrationEndpoint),
}
if err := validateResolvedEndpoints(metadata); err != nil {
return authServerMetadata{}, err
}
return metadata, nil
}
@@ -179,6 +202,9 @@ func resolveAuthorizationServer(ctx context.Context, client *http.Client, baseUR
if strings.TrimSpace(metadata.TokenEndpoint) == "" {
return authServerMetadata{}, errors.New("no token endpoint discovered or configured")
}
if err := validateResolvedEndpoints(metadata); err != nil {
return authServerMetadata{}, err
}
return metadata, nil
}
+16
View File
@@ -73,6 +73,22 @@ func TestResolveEndpointsFallsBackToConfig(t *testing.T) {
}
}
// Discovered MCP metadata that serves an insecure endpoint must be rejected
// before use, so discovery cannot downgrade the MCP login to an
// attacker-controlled origin (issue #511).
func TestResolveAuthorizationServerRejectsInsecureDiscovered(t *testing.T) {
mux := http.NewServeMux()
server := httptest.NewServer(mux)
defer server.Close()
mux.HandleFunc("/.well-known/oauth-authorization-server", func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(`{"issuer":"` + server.URL + `","token_endpoint":"` + server.URL + `/token","authorization_endpoint":"http://evil.example/authorize"}`))
})
_, err := resolveAuthorizationServer(context.Background(), server.Client(), server.URL, OAuthConfig{})
if err == nil || !strings.Contains(err.Error(), "authorization endpoint") {
t.Fatalf("err = %v, want insecure authorization endpoint rejection", err)
}
}
// roundTripperFunc adapts a function to http.RoundTripper so a test can
// observe every outbound request.
type roundTripperFunc func(*http.Request) (*http.Response, error)
+263
View File
@@ -12,6 +12,54 @@ import (
"time"
)
// Mirror internal/cli/exec.go exit codes so wrapper doctor fallback tests assert
// the same CLI contract as the Go doctor command.
const (
wrapperExitSuccess = 0
wrapperExitUsage = 2
wrapperExitDoctor = 1
)
// Canonical Bun recovery copy from bin/zero.js bunRecoveryParagraph() — shared
// by the generic missing-binary path and the doctor text fallback.
const (
bunRecoveryLead = "You installed with Bun, which does not run dependency lifecycle scripts"
bunPmTrustProject = "bun pm trust @gitlawb/zero"
bunPmTrustGlobal = "bun pm -g trust @gitlawb/zero"
bunRecoveryTrustedDeps = `"trustedDependencies": ["@gitlawb/zero"]`
buildFromSourceLead = "If this platform has no prebuilt binary, build from source:"
)
func runWrapperFixture(t *testing.T, wrapperPath string, args ...string) (stdout string, stderr string, exitCode int) {
return runWrapperFixtureWithEnv(t, wrapperPath, nil, args...)
}
func runWrapperFixtureWithEnv(t *testing.T, wrapperPath string, extraEnv []string, args ...string) (stdout string, stderr string, exitCode int) {
t.Helper()
node := requireNode(t)
ctx, cancel := context.WithTimeout(context.Background(), nodeWrapperTimeout())
defer cancel()
command := nodeWrapperCommand(ctx, node, wrapperPath, args...)
command.Env = append(withoutEnvKey(command.Env, "ZERO_LOCAL_CONTROL_HELPERS"), "ZERO_LOCAL_CONTROL_HELPERS=")
command.Env = append(withoutEnvKey(command.Env, "ZERO_WRAPPER_SIMULATE_BUN"), extraEnv...)
var stdoutBuf, stderrBuf strings.Builder
command.Stdout = &stdoutBuf
command.Stderr = &stderrBuf
err := command.Run()
if ctx.Err() != nil {
t.Fatalf("wrapper timed out: %v; stdout: %s stderr: %s", ctx.Err(), stdoutBuf.String(), stderrBuf.String())
}
exitCode = wrapperExitSuccess
if err != nil {
exitErr, ok := err.(*exec.ExitError)
if !ok {
t.Fatalf("wrapper err = %v, want *exec.ExitError; stdout=%s stderr=%s", err, stdoutBuf.String(), stderrBuf.String())
}
exitCode = exitErr.ExitCode()
}
return stdoutBuf.String(), stderrBuf.String(), exitCode
}
func TestPackageBinPointsToNodeWrapper(t *testing.T) {
root := repoRoot(t)
bytes, err := os.ReadFile(filepath.Join(root, "package.json"))
@@ -393,6 +441,221 @@ func nodeArchName() string {
}
}
// Issue #405: `zero doctor` is the diagnostic command, so when the native
// binary is the thing that's broken it must NOT bail with the generic wrapper
// error; that's exactly the blind alley the bug report calls out. Instead it
// emits a doctor-shaped FAIL line for the runtime so the user's own diagnostic
// surfaces the real cause. We assert on the doctor-report shape (so the doctor
// UX matches what the Go-side doctor.Format produces) and on exit 1 (a missing
// binary is still a hard failure, not a pass).
func TestNodeWrapperDoctorReportsMissingNativeBinaryAsDoctorFail(t *testing.T) {
wrapperPath := copyWrapperFixture(t)
stdout, stderr, exitCode := runWrapperFixture(t, wrapperPath, "doctor")
if exitCode != wrapperExitDoctor {
t.Fatalf("doctor exit = %d, want %d; stdout=%s stderr=%s", exitCode, wrapperExitDoctor, stdout, stderr)
}
if strings.TrimSpace(stderr) != "" {
t.Fatalf("doctor text report must go to stdout only, got stderr=%q", stderr)
}
// Doctor-shaped report, not the generic wrapper bail. Matches the shape the
// Go-side doctor.Format emits: a header, "Overall: <pass/fail>", then
// "[<status>] <id> - <message>" lines.
if !strings.Contains(stdout, "Zero doctor report (") {
t.Fatalf("doctor output should start with a doctor report header, got stdout=%q", stdout)
}
if !strings.Contains(stdout, "Overall: fail") {
t.Fatalf("doctor overall must be fail when the native binary is missing, got stdout=%q", stdout)
}
if !strings.Contains(stdout, "[fail] runtime.go") {
t.Fatalf("doctor must report a failing runtime.go check, got stdout=%q", stdout)
}
if !strings.Contains(stdout, "Native zero binary is missing next to the npm wrapper") {
t.Fatalf("doctor must name the actual cause (missing native binary), got stdout=%q", stdout)
}
// The actionable remedy must point at the postinstall script that would fix
// the install, not just "build from source".
if !strings.Contains(stdout, "postinstall.mjs") {
t.Fatalf("doctor remedy should name the postinstall script, got stdout=%q", stdout)
}
// Regression guard for the original blind-alley bug: the doctor path must
// NOT emit the generic wrapper bail (that's what sent users debugging the
// wrong thing).
if strings.Contains(stdout, "[zero] No native binary is available for this install") {
t.Fatalf("doctor must not emit the generic wrapper bail, got stdout=%q", stdout)
}
}
func TestNodeWrapperDoctorJSONReportsMissingNativeBinaryAsJSONFail(t *testing.T) {
wrapperPath := copyWrapperFixture(t)
stdout, stderr, exitCode := runWrapperFixture(t, wrapperPath, "doctor", "--json")
if exitCode != wrapperExitDoctor {
t.Fatalf("doctor --json exit = %d, want %d; stdout=%s stderr=%s", exitCode, wrapperExitDoctor, stdout, stderr)
}
if strings.TrimSpace(stderr) != "" {
t.Fatalf("doctor --json should write machine-readable output to stdout only, got stderr=%q", stderr)
}
var report struct {
GeneratedAt string `json:"generatedAt"`
OK bool `json:"ok"`
Checks []struct {
ID string `json:"id"`
Label string `json:"label"`
Status string `json:"status"`
Message string `json:"message"`
Details map[string]any `json:"details"`
} `json:"checks"`
}
if err := json.Unmarshal([]byte(stdout), &report); err != nil {
t.Fatalf("doctor --json stdout should be valid JSON, got %q: %v", stdout, err)
}
if report.GeneratedAt == "" {
t.Fatalf("doctor --json report should include generatedAt: %#v", report)
}
if report.OK {
t.Fatalf("doctor --json ok = true, want false: %#v", report)
}
if len(report.Checks) != 1 {
t.Fatalf("doctor --json checks length = %d, want 1: %#v", len(report.Checks), report.Checks)
}
check := report.Checks[0]
if check.ID != "runtime.go" || check.Label != "Go runtime" || check.Status != "fail" {
t.Fatalf("doctor --json runtime check = %#v, want failing runtime.go check", check)
}
if !strings.Contains(check.Message, "Native zero binary is missing next to the npm wrapper") {
t.Fatalf("doctor --json must name the actual cause, got %#v", check)
}
remedy, _ := check.Details["remedy"].(string)
if !strings.Contains(remedy, "postinstall.mjs") {
t.Fatalf("doctor --json remedy should name the postinstall script, got %#v", check.Details)
}
}
func TestNodeWrapperDoctorHelpShowsUsage(t *testing.T) {
wrapperPath := copyWrapperFixture(t)
for _, args := range [][]string{{"doctor", "--help"}, {"doctor", "help"}, {"doctor", "-h"}} {
stdout, stderr, exitCode := runWrapperFixture(t, wrapperPath, args...)
if exitCode != wrapperExitSuccess {
t.Fatalf("%v exit = %d, want %d; stdout=%s stderr=%s", args, exitCode, wrapperExitSuccess, stdout, stderr)
}
if strings.TrimSpace(stderr) != "" {
t.Fatalf("%v help must write to stdout only, got stderr=%q", args, stderr)
}
if !strings.Contains(stdout, "Usage:") || !strings.Contains(stdout, "zero doctor [flags]") {
t.Fatalf("%v help output = %q, want doctor usage text", args, stdout)
}
}
}
func TestNodeWrapperDoctorRejectsUnknownFlag(t *testing.T) {
wrapperPath := copyWrapperFixture(t)
stdout, stderr, exitCode := runWrapperFixture(t, wrapperPath, "doctor", "--bogus")
if exitCode != wrapperExitUsage {
t.Fatalf("doctor --bogus exit = %d, want %d; stdout=%s stderr=%s", exitCode, wrapperExitUsage, stdout, stderr)
}
if strings.TrimSpace(stdout) != "" {
t.Fatalf("invalid doctor flag should not write stdout, got %q", stdout)
}
if stderr != "[zero] unknown doctor flag \"--bogus\"\n" {
t.Fatalf("invalid doctor flag stderr = %q", stderr)
}
}
// `doctor --connectivity` (a valid doctor invocation with a trailing flag) must
// take the same doctor-shaped path: parseDoctorArgs accepts --connectivity and
// the missing-binary fallback still reports the runtime failure.
func TestNodeWrapperDoctorWithFlagsStillReportsDoctorFail(t *testing.T) {
wrapperPath := copyWrapperFixture(t)
stdout, stderr, exitCode := runWrapperFixture(t, wrapperPath, "doctor", "--connectivity")
if exitCode != wrapperExitDoctor {
t.Fatalf("doctor --connectivity exit = %d, want %d; stdout=%s stderr=%s", exitCode, wrapperExitDoctor, stdout, stderr)
}
if strings.TrimSpace(stderr) != "" {
t.Fatalf("doctor --connectivity must write to stdout only, got stderr=%q", stderr)
}
if !strings.Contains(stdout, "[fail] runtime.go") {
t.Fatalf("doctor --connectivity must still emit the failing runtime.go line, got stdout=%q", stdout)
}
if strings.Contains(stdout, "[zero] No native binary is available for this install") {
t.Fatalf("doctor --connectivity must not fall back to the generic bail, got stdout=%q", stdout)
}
}
func TestNodeWrapperGenericReportsBunRecoveryWhenInstalledByBun(t *testing.T) {
wrapperPath := copyWrapperFixture(t)
_, stderr, exitCode := runWrapperFixtureWithEnv(t, wrapperPath, []string{"ZERO_WRAPPER_SIMULATE_BUN=1"}, "--version")
if exitCode != wrapperExitDoctor {
t.Fatalf("generic missing-binary exit = %d, want %d; stderr=%s", exitCode, wrapperExitDoctor, stderr)
}
assertBunRecoveryMessage(t, stderr)
if !strings.Contains(stderr, buildFromSourceLead) {
t.Fatalf("generic bun stderr should include build-from-source guidance, got %q", stderr)
}
}
func TestNodeWrapperDoctorReportsBunRecoveryWhenInstalledByBun(t *testing.T) {
wrapperPath := copyWrapperFixture(t)
stdout, stderr, exitCode := runWrapperFixtureWithEnv(t, wrapperPath, []string{"ZERO_WRAPPER_SIMULATE_BUN=1"}, "doctor")
if exitCode != wrapperExitDoctor {
t.Fatalf("doctor bun exit = %d, want %d; stdout=%s stderr=%s", exitCode, wrapperExitDoctor, stdout, stderr)
}
if strings.TrimSpace(stderr) != "" {
t.Fatalf("doctor bun report must go to stdout only, got stderr=%q", stderr)
}
if !strings.Contains(stdout, "[fail] runtime.go") {
t.Fatalf("doctor bun report must still fail runtime.go, got stdout=%q", stdout)
}
assertBunRecoveryMessage(t, stdout)
if !strings.Contains(stdout, buildFromSourceLead) {
t.Fatalf("doctor bun report should reuse generic build-from-source guidance, got stdout=%q", stdout)
}
if strings.Contains(stdout, "If reinstall fails") {
t.Fatalf("doctor bun report must not use a separate build-from-source string, got stdout=%q", stdout)
}
}
func TestNodeWrapperDoctorAndGenericShareBunRecoveryCopy(t *testing.T) {
wrapperPath := copyWrapperFixture(t)
simulateBun := []string{"ZERO_WRAPPER_SIMULATE_BUN=1"}
_, genericStderr, _ := runWrapperFixtureWithEnv(t, wrapperPath, simulateBun, "--version")
doctorStdout, _, _ := runWrapperFixtureWithEnv(t, wrapperPath, simulateBun, "doctor")
genericBun := extractBunRecoveryBlock(genericStderr)
doctorBun := extractBunRecoveryBlock(doctorStdout)
if genericBun != doctorBun {
t.Fatalf("bun recovery copy drift:\ngeneric=%q\ndoctor=%q", genericBun, doctorBun)
}
}
func assertBunRecoveryMessage(t *testing.T, output string) {
t.Helper()
if !strings.Contains(output, bunRecoveryLead) {
t.Fatalf("output should include bun recovery lead, got %q", output)
}
if !strings.Contains(output, bunPmTrustProject) {
t.Fatalf("output should include project bun pm trust guidance, got %q", output)
}
if !strings.Contains(output, bunPmTrustGlobal) {
t.Fatalf("output should include global bun pm trust guidance, got %q", output)
}
if !strings.Contains(output, bunRecoveryTrustedDeps) {
t.Fatalf("output should include trustedDependencies fallback guidance, got %q", output)
}
if !strings.Contains(output, "to your project package.json and reinstall.") {
t.Fatalf("output should mention package.json reinstall fallback, got %q", output)
}
}
func extractBunRecoveryBlock(output string) string {
start := strings.Index(output, bunRecoveryLead)
if start < 0 {
return ""
}
end := strings.Index(output[start:], buildFromSourceLead)
if end < 0 {
return strings.TrimSpace(output[start:])
}
return strings.TrimSpace(output[start : start+end])
}
func TestNodeWrapperLaunchesNativeBinary(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("mock executable fixture uses a POSIX shell script")
+21 -5
View File
@@ -39,6 +39,13 @@ func BuildAuthorizationURL(cfg Config, pkce PKCE, state, redirectURI string, ext
if endpoint == "" {
return "", errors.New("oauth: no authorization endpoint configured")
}
// Backstop: validate the authorization endpoint at the shared choke point
// both the provider and MCP flows build their browser URL through, so a
// discovery-downgraded endpoint can never open in the browser even if a
// merge site missed it.
if err := ValidateEndpointURL(endpoint); err != nil {
return "", err
}
parsed, err := url.Parse(endpoint)
if err != nil {
return "", fmt.Errorf("oauth: parse authorization endpoint: %w", err)
@@ -83,13 +90,15 @@ func isReservedAuthParam(key string) bool {
}
}
// validateTokenEndpoint refuses to send a credential to a token endpoint that is
// not https, unless it is a loopback host (mirrors oauth2-client.js
// validateTokenEndpointURL). This prevents leaking codes/tokens over cleartext.
func validateTokenEndpoint(endpoint string) error {
// ValidateEndpointURL refuses an OAuth endpoint that is not https, unless it is
// a loopback host (mirrors oauth2-client.js validateTokenEndpointURL). Every
// credential-bearing endpoint — configured OR learned from discovery — must
// pass this single rule, so discovery metadata can never downgrade a login to
// cleartext or redirect it to an attacker-controlled http origin.
func ValidateEndpointURL(endpoint string) error {
parsed, err := url.Parse(trimmed(endpoint))
if err != nil || parsed.Host == "" {
return fmt.Errorf("oauth: invalid token endpoint %q", endpoint)
return fmt.Errorf("oauth: invalid endpoint %q", endpoint)
}
if parsed.Scheme == "https" {
return nil
@@ -100,6 +109,13 @@ func validateTokenEndpoint(endpoint string) error {
return fmt.Errorf("%w: %s", ErrInsecureTokenEndpoint, parsed.Scheme+"://"+parsed.Host)
}
// validateTokenEndpoint checks a token endpoint against the shared endpoint rule
// (kept as a named helper for the token-exchange and device call sites). This
// prevents leaking codes/tokens over cleartext.
func validateTokenEndpoint(endpoint string) error {
return ValidateEndpointURL(endpoint)
}
func isLoopbackHost(host string) bool {
if host == "localhost" {
return true
+14
View File
@@ -112,6 +112,20 @@ func TestBuildAuthorizationURLRejectsPlain(t *testing.T) {
}
}
// The shared choke point both flows build their browser URL through must refuse
// an insecure authorization endpoint, so a discovery-downgraded endpoint can
// never open in the browser even if a merge site missed it (issue #511).
func TestBuildAuthorizationURLRejectsInsecureEndpoint(t *testing.T) {
_, err := BuildAuthorizationURL(
Config{AuthorizationEndpoint: "http://evil.example/authorize", ClientID: "c"},
PKCE{Method: MethodS256, Challenge: "chal"},
"state", "http://127.0.0.1/cb", nil,
)
if !errors.Is(err, ErrInsecureTokenEndpoint) {
t.Fatalf("err = %v, want ErrInsecureTokenEndpoint", err)
}
}
func TestValidateTokenEndpoint(t *testing.T) {
cases := map[string]bool{
"https://auth.example.com/token": true,
+16 -3
View File
@@ -218,13 +218,26 @@ func (m *Manager) resolveEndpoints(ctx context.Context, cfg Config) (Config, err
// may already be sufficient for the chosen flow. Only merge on success (and
// never return from the error branch, which would trip the nilerr linter).
if meta, err := m.discover(ctx, cfg.IssuerURL); err == nil {
if cfg.AuthorizationEndpoint == "" {
// A discovered endpoint must pass the same https/loopback rule as a
// configured one before it is merged, so discovery metadata can never
// silently downgrade the login to an insecure or attacker-controlled
// endpoint. Fail closed on a bad endpoint rather than merging it.
if cfg.AuthorizationEndpoint == "" && meta.AuthorizationEndpoint != "" {
if err := ValidateEndpointURL(meta.AuthorizationEndpoint); err != nil {
return cfg, fmt.Errorf("oauth: discovered authorization endpoint: %w", err)
}
cfg.AuthorizationEndpoint = meta.AuthorizationEndpoint
}
if cfg.TokenEndpoint == "" {
if cfg.TokenEndpoint == "" && meta.TokenEndpoint != "" {
if err := ValidateEndpointURL(meta.TokenEndpoint); err != nil {
return cfg, fmt.Errorf("oauth: discovered token endpoint: %w", err)
}
cfg.TokenEndpoint = meta.TokenEndpoint
}
if cfg.DeviceAuthorizationEndpoint == "" {
if cfg.DeviceAuthorizationEndpoint == "" && meta.DeviceAuthorizationEndpoint != "" {
if err := ValidateEndpointURL(meta.DeviceAuthorizationEndpoint); err != nil {
return cfg, fmt.Errorf("oauth: discovered device authorization endpoint: %w", err)
}
cfg.DeviceAuthorizationEndpoint = meta.DeviceAuthorizationEndpoint
}
}
+19
View File
@@ -2,6 +2,7 @@ package oauth
import (
"context"
"errors"
"io"
"net/http"
"net/http/httptest"
@@ -200,6 +201,24 @@ func TestManagerGetFreshDiscoversIssuerEndpoints(t *testing.T) {
}
}
// A discovery document that serves an insecure (non-https, non-loopback)
// endpoint must be rejected before it is merged, so discovery metadata cannot
// silently downgrade the login to an attacker-controlled origin (issue #511).
func TestResolveEndpointsRejectsInsecureDiscoveredEndpoint(t *testing.T) {
mux := http.NewServeMux()
server := httptest.NewServer(mux)
defer server.Close()
mux.HandleFunc("/.well-known/oauth-authorization-server", func(w http.ResponseWriter, _ *http.Request) {
// Loopback issuer, but an attacker-controlled http authorization endpoint.
_, _ = io.WriteString(w, `{"issuer":"`+server.URL+`","token_endpoint":"`+server.URL+`/token","authorization_endpoint":"http://evil.example/authorize"}`)
})
m := managerFor(t, map[string]string{}, nil)
_, err := m.resolveEndpoints(context.Background(), Config{IssuerURL: server.URL})
if !errors.Is(err, ErrInsecureTokenEndpoint) {
t.Fatalf("resolveEndpoints err = %v, want ErrInsecureTokenEndpoint", err)
}
}
func TestManagerGetFreshRefreshesExpired(t *testing.T) {
fp := newFakeProvider(t, `{"access_token":"fresh-at","expires_in":3600}`)
env := map[string]string{
+120 -35
View File
@@ -10,16 +10,19 @@ matters for months, so the contract is written down here.
48 tasks across seven classes:
| class | count | oracle | tier |
|-----------|-------|-----------------------------|-------------|
| nav | 10 | none | latency |
| edit | 10 | substring grep | correctness |
| fix | 8 | scoped `go test -run <name>` | correctness |
| refactor | 6 | `go build ./...` | build |
| longproc | 4 | none | latency |
| longctx | 4 | none | latency |
| parallel | 6 | none | latency |
| **total** | **48**| | |
| class | count | oracle | tier |
|-----------|-------|-----------------------------------------|-------------|
| nav | 10 | grep on captured final answer | correctness |
| edit | 10 | stamped `oracle_test.go` + `go test` | correctness |
| fix | 8 | scoped `go test -run <name>` | correctness |
| refactor | 6 | stamped `oracle_test.go` + `go test` | correctness |
| longproc | 4 | none | latency |
| longctx | 4 | none | latency |
| parallel | 6 | none | latency |
| **total** | **48**| | |
Tier counts: **correctness 34** (10 edit + 8 fix + 10 nav + 6 refactor),
**build 0**, **latency 14** (4 longproc + 4 longctx + 6 parallel).
## Oracle tiers
@@ -27,16 +30,21 @@ Pass/fail is reported per tier so the report cannot be misread as a blanket
correctness verdict. The tier is decided per task from oracle presence and the
manifest's `buildOnlyClasses` list:
- **Correctness** (18 tasks: 10 edit + 8 fix) — a positive oracle (substring
grep that the requested change landed, or a scoped `go test -run` that the
bug fix passes). This is the only pass rate that can move with model quality:
`tasksVerified` / `tasksPassed` / `correctnessPassRate`.
- **Build-only** (6 tasks: refactor) — `go build ./...` proves the edit
compiles, not that the refactor achieved its goal (a no-op refactor passes).
The manifest declares `buildOnlyClasses: ["refactor"]`. Reported as
`buildCheckedTasks` / `buildPassedTasks` / `buildPassRate`, never in
`correctnessPassRate`.
- **Latency-only** (24 tasks: nav, longproc, longctx, parallel) — no
- **Correctness** (34 tasks: edit, fix, nav, refactor) — a positive oracle. For
edit/refactor the harness stamps an `oracle_test.go` (from the task's
`oracleTest` field) into the fixture copy **after** the agent run, then runs
`go test ./...`: the Go compiler is the structural verifier, so a no-op
refactor, a missing field, or a reworded-but-not-removed line fails to
compile/run. fix uses a scoped `go test -run <name>`. nav greps the agent's
captured final answer (`.zero-answer.txt`) for determinable facts. This is
the only pass rate that can move with model quality: `tasksVerified` /
`tasksPassed` / `correctnessPassRate`.
- **Build-only** (0 tasks) — the tier is empty. refactor used to live here with
a non-positive `go build ./...` (a no-op refactor passed); PR #712 gave it a
structural oracle, graduating it to correctness. `buildOnlyClasses` is `[]`.
`buildCheckedTasks` / `buildPassedTasks` / `buildPassRate` are reported as 0
and never in `correctnessPassRate`.
- **Latency-only** (14 tasks: longproc, longctx, parallel) — no
`verificationCommand`. An exit 0 only proves the turn ran, not that the answer
was right. They contribute to latency and span attribution only and are
counted in `latencyOnlyTasks`, never in any pass rate.
@@ -45,22 +53,89 @@ A task's tier is driven by oracle **presence** first: a task with no
`verificationCommand` is latency-only even if its class is listed in
`buildOnlyClasses`, so a missing oracle can never silently pass on exit 0.
## Known limitations (deferred)
### How the stamped oracle and captured answer work
These are accepted for the Phase 0 baseline and tracked in a follow-up issue;
they do not block the baseline because the tier split keeps the report honest:
The production runner (`NewTurnExecRunner` in `turn_bench.go`) copies the fixture
into an isolated temp dir, runs `zero exec --trace`, then — before running the
`verificationCommand` — stamps two files into the copy:
- The edit grep oracles are substring checks. A rename oracle asserts the new
name is present AND the old name is gone (compound `bash -c`), but an
add-field oracle (`grep -R Label .`) only proves the string appears, not that
it landed on the right struct. Strengthening these to `go vet`/`go build` +
structural assertions is follow-up work.
- The refactor build oracle is non-positive (a no-op refactor passes). A
structural verifier that proves the refactor happened is follow-up work;
until then refactor is `buildPassRate`, not `correctnessPassRate`.
- The 24 read-only tasks have no oracle. Adding deterministic oracles for nav
(diff against an expected answer) and documenting longproc/longctx/parallel
as permanently latency-only is follow-up work.
- `oracle_test.go` from `task.OracleTest` (edit/refactor). It is stamped *after*
the agent run so it can't interfere with the agent's own `go build`/`go test`
during the task (refactor-03's `package zeroapp` test would break a pre-rename
build) and can't be pre-seen or tampered with. The `verificationCommand`
(`go test ./...`) then compiles and runs it.
- `.zero-answer.txt` from the `{"type":"final","text":...}` event in the
stream-json output (nav). The `verificationCommand` is a compound `bash -c`
grep requiring the determinable facts (e.g. nav-09 requires the answer to
mention `port`, `name`, and `retries`). An empty file (no `final` event) makes
the grep fail, which is the correct outcome for a run that produced no answer.
The `oracleTest` source uses compile-time references where the assertion is
structural (`var _ = Config{}.Label` for edit-03, `var _ = formatGreeting` for
refactor-01) and behavioral `Test…` functions where the assertion is a value
(edit-04 `Version != "1.1.0"`, edit-10 `greet("x") != "hello, world"`, edit-09
`load("nonexistent")` wrapping). Tasks whose oracle is a plain grep/build with
no stamped test (edit-02, refactor-02) carry no `oracleTest`.
## Known limitations
These are accepted for the Phase 0 baseline; the tier split keeps the report
honest where they apply:
- **longproc / longctx / parallel are permanently latency-only.** There is no
deterministic oracle for "report whether the build succeeds", "summarize a
large generated file", or "report the first line of each of six files" — the
answer is free-form prose and a contains-check would rubber-stamp. They stay
in `latencyOnlyTasks` and never enter a pass rate.
- **nav count tasks (nav-01, nav-04, nav-05, nav-08) require a structured,
line-anchored `count: N` token with the fixture's exact expected count AND a
workspace-derived fact that proves the agent actually inspected the fixture —
not the count alone.** Their prompts ask the agent to state the count as
`count: N` on its own line and to name the thing(s) counted, and the oracle
greps for an anchored `^count: N$` with the exact expected N plus a named fact:
nav-01 requires `count: 5` and names `main.go`, `config.json`, and `README.md`
(the three the oracle anchors on — a correct answer lists all five files, but
the verification contract only requires those three be named); nav-04 names
the one test function (`TestGreet` from `main_test.go`); nav-05 names the file
holding the one TODO (`main.go`); nav-08 names both stdlib imports (`fmt` from
`main.go`, `testing` from `main_test.go`). A substring like "the count: 1" or a
wrong count like "count: 3" fails the anchor; a bare "count: 1" with no named
fact fails the inspection check. Non-zero ground truth alone was not enough —
it closes the always-guess-zero game but opens an always-guess-one game (an
agent that never opens the workspace and emits "count: 1" would pass a
count-only oracle). The named-fact requirement closes that surface: a blind
count is no longer sufficient, the agent must commit to a fact only readable
from the fixture. nav-04 and nav-05 were given a real `TestGreet` and a real
TODO so the named fact exists to require; nav-08 stays at count 0 (a real
third-party import would need a network fetch and break the offline suite), so
its named fact is the stdlib imports the agent must enumerate across every file
to conclude 0 third-party. The named facts are not all equally tight. nav-04's
`TestGreet` is a specific, non-generic identifier a blind agent cannot produce
without reading `main.go` to learn a greet function exists, so it fully closes
the blind-count-one surface. nav-05's `main.go` is the conventional Go
entrypoint and thus the most guessable filename, so it *reduces* rather than
fully closes that surface — a blind "count: 1, in main.go" can still pass.
Accepted for a Phase 0 baseline: the blind floor on nav-05 is below 100% (a
bare "count: 1" fails), so a model still moves the pass rate by actually
inspecting; strengthening it further (e.g. requiring the TODO's surrounding
symbol) would trade output freedom for determinism and risks false-failing a
terse-but-correct answer. nav-08's `testing` half is the tighter of its two:
`fmt` alone is the most common Go stdlib import and could be named blind, but
`testing` only appears in the test file, so naming it proves the agent read more
than `main.go`. This trades a little output freedom for determinism (a loose
prose-contains on "0"/"1" would match almost any answer, and an unanchored
`count: N` would match inside a longer line).
- **nav-03 / nav-06 / nav-07 are lenient contains-oracles on real facts.** A
wrong-but-plausible answer that happens to mention the real symbols (e.g.
"Config" and "MaxRetries" for nav-06) could in principle pass. Accepted for a
latency-first baseline; the oracle still catches an agent that didn't look at
the code at all.
- **Rename oracles use a declaration-site negative grep** (`! grep -RIn
'MaxRetries =' .`) rather than a broad `! grep -R MaxRetries .`, so a correct
rename that leaves a doc-comment mention of the old name is not false-failed.
The negative grep is still required: without it, a no-op that *adds* the new
name and *keeps* the old would pass the compile-time `var _ = RetryLimit` and
`go test`.
## Fixtures
@@ -68,4 +143,14 @@ Each task's `workspaceFixture` points at a small self-contained workspace under
`testdata/` so the suite runs offline and repeatably. Mutating tasks (edit,
fix, refactor) run against a per-invocation **copy** of their fixture, so the
checked-in fixtures stay clean and one task's edits can't bleed into the next
iteration or a later task.
iteration or a later task.
Each mutating fixture (and nav) carries a `go.mod` (`module <pkg>; go 1.22`) so
`go test ./...` / `go build ./...` oracles work in the copy: `copyFixture`
creates a unique, 0700 parent under the system temp root
(`os.MkdirTemp(os.TempDir(), "zero-turn-bench-*")`) and nests the copy beneath
it, so the copy is a *grandchild* of the temp root. Go ignores `go.mod` in
direct children of the system temp dir (a hijack guard), so the copy must sit
one level below that root or every compiler-backed oracle would fail with
"cannot find main module". The unique parent also avoids a predictable
shared directory and is removed wholesale on cleanup.
+540 -57
View File
@@ -1,62 +1,545 @@
{
"id": "zero-baseline-turn",
"name": "Zero per-turn baseline (Phase 0)",
"description": "Latency-first baseline: measures where a turn spends wall time, with pass/fail reported per oracle tier so it cannot be misread as a blanket correctness verdict. Read-only classes (nav, longproc, longctx, parallel) carry no verification oracle — a zero-exit only proves the turn ran, so they count as latency-only and never in any pass rate. Correctness classes (edit, fix) carry a positive oracle (substring grep or a scoped `go test`) and are the only tasks in correctnessPassRate. Build-only classes (refactor) carry `go build ./...`, which proves the edit compiles but not that the refactor achieved its goal, so they are reported as buildPassRate and excluded from correctnessPassRate.",
"buildOnlyClasses": ["refactor"],
"description": "Latency-first baseline: measures where a turn spends wall time, with pass/fail reported per oracle tier so it cannot be misread as a blanket correctness verdict. Correctness classes (edit, fix, nav, refactor) carry a positive oracle: edit/refactor stamp an oracle_test.go compiled by `go test ./...` (the Go compiler is the structural verifier \u2014 a no-op refactor or a missing field fails to compile), fix runs a scoped `go test -run`, and nav greps the agent's captured final answer (.zero-answer.txt) for determinable facts. tasksVerified/tasksPassed/correctnessPassRate cover exactly these. Latency-only classes (longproc, longctx, parallel) carry no verification oracle \u2014 a zero-exit only proves the turn ran, so they count as latency-only and never in any pass rate. The build tier is empty: refactor graduated to correctness once its oracle became structural-positive. See MANIFEST.md for the per-task oracle specs and known limitations.",
"buildOnlyClasses": [],
"tasks": [
{"id": "nav-01", "name": "list repo files", "class": "nav", "prompt": "List the files in this repository and report their count. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-02", "name": "find a function", "class": "nav", "prompt": "Find the function named 'greet' and report the file and line it is defined on. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-03", "name": "summarize module", "class": "nav", "prompt": "Read main.go and summarize what this program does in two sentences. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-04", "name": "count tests", "class": "nav", "prompt": "Count the test functions (functions whose names start with Test) in this repository and report the number. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-05", "name": "locate a string", "class": "nav", "prompt": "Find every occurrence of the string 'TODO' in the repository and report the files. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-06", "name": "describe package", "class": "nav", "prompt": "Describe the exported API of this package by reading the source. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-07", "name": "trace a call", "class": "nav", "prompt": "Trace which function calls which other function starting from main. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-08", "name": "find imports", "class": "nav", "prompt": "List every third-party import used by this repository. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-09", "name": "report config", "class": "nav", "prompt": "Read config.json and report the keys it contains. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "nav-10", "name": "doc lookup", "class": "nav", "prompt": "Find the doc comment on the greet function and quote it. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
{"id": "edit-01", "name": "rename a constant", "class": "edit", "prompt": "Rename the constant MaxRetries to RetryLimit everywhere it appears.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R RetryLimit . && ! grep -R MaxRetries ."]},
{"id": "edit-02", "name": "fix a typo", "class": "edit", "prompt": "Fix the typo in the comment above the greet function ('receieves' should be 'receives').", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "! grep -R receieves ."]},
{"id": "edit-03", "name": "add a field", "class": "edit", "prompt": "Add a string field named Label to the Config struct in main.go.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "Label", "."]},
{"id": "edit-04", "name": "bump a version", "class": "edit", "prompt": "Change the version string in version.go from 1.0.0 to 1.1.0.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R 1.1.0 . && ! grep -R 1.0.0 ."]},
{"id": "edit-05", "name": "remove a log line", "class": "edit", "prompt": "Remove the line that prints 'debug: starting'.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "! grep -R 'debug: starting' ."]},
{"id": "edit-06", "name": "add a license header", "class": "edit", "prompt": "Add a comment line '// SPDX-License-Identifier: MIT' at the top of main.go.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "SPDX-License-Identifier", "."]},
{"id": "edit-07", "name": "change a default", "class": "edit", "prompt": "Change the default value of the Port constant from 8080 to 9090.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R 9090 . && ! grep -R 8080 ."]},
{"id": "edit-08", "name": "add a getter", "class": "edit", "prompt": "Add a method func (c *Config) GetLabel() string that returns c.Label.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R 'func (c *Config) GetLabel' ."]},
{"id": "edit-09", "name": "wrap an error", "class": "edit", "prompt": "In the load function, wrap the returned error with fmt.Errorf so it includes 'load failed'.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "load failed", "."]},
{"id": "edit-10", "name": "update a message", "class": "edit", "prompt": "Change the printed greeting from 'hello' to 'hello, world'.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "hello, world", "."]},
{"id": "fix-01", "name": "off-by-one", "class": "fix", "prompt": "In bugs.go the Sum function has an off-by-one error that skips the last element; fix the loop so it sums all elements.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestSumIncludesLastElement", "./..."]},
{"id": "fix-02", "name": "nil deref", "class": "fix", "prompt": "In bugs.go DefaultConfig returns nil and Port() dereferences it; make DefaultConfig return a valid Config with Port 8080.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestPortReturnsDefault", "./..."]},
{"id": "fix-03", "name": "wrong operator", "class": "fix", "prompt": "In bugs.go Max uses the wrong inequality and returns the smaller value; fix it to return the larger.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestMaxReturnsLarger", "./..."]},
{"id": "fix-04", "name": "truncated read", "class": "fix", "prompt": "In bugs.go ReadFirst returns only a single character instead of the full content; fix it to return the whole string.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestReadFirstReturnsFullContent", "./..."]},
{"id": "fix-05", "name": "swapped args", "class": "fix", "prompt": "In bugs.go Divide swaps its operands; fix the order so Divide(10,2) returns 5.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestDivideOrder", "./..."]},
{"id": "fix-06", "name": "missing return", "class": "fix", "prompt": "In bugs.go Classify is missing a return for the high case; add it so Classify(500) returns \"high\".", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestClassifyHigh", "./..."]},
{"id": "fix-07", "name": "wrong label prefix", "class": "fix", "prompt": "In bugs.go Label returns 'admin-<name>' but the test expects 'user-<name>'; change the prefix from 'admin-' to 'user-'.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestLabelUsesString", "./..."]},
{"id": "fix-08", "name": "race on counter", "class": "fix", "prompt": "In bugs.go the global counter is incremented without synchronization; guard it with a mutex.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-race", "-run", "TestCounterRace", "./..."]},
{"id": "refactor-01", "name": "extract helper", "class": "refactor", "prompt": "Extract the duplicated greeting logic into a single helper function and call it from both sites.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "refactor-02", "name": "split a file", "class": "refactor", "prompt": "Split main.go into two files: one for Config, one for the rest, keeping the package building.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "refactor-03", "name": "rename package", "class": "refactor", "prompt": "Rename the package from app to zeroapp everywhere and keep it building.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "refactor-04", "name": "introduce a type", "class": "refactor", "prompt": "Introduce a named type for the map[string]int used in main.go and update its users.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "refactor-05", "name": "inline a wrapper", "class": "refactor", "prompt": "Inline the thin wrapper function into its single caller and remove it.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "refactor-06", "name": "consolidate errors", "class": "refactor", "prompt": "Consolidate the repeated error-construction blocks into a single helper.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
{"id": "longproc-01", "name": "long build", "class": "longproc", "prompt": "Run the build command and report whether it succeeds. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
{"id": "longproc-02", "name": "long test", "class": "longproc", "prompt": "Run go test ./... and report the result. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
{"id": "longproc-03", "name": "bench run", "class": "longproc", "prompt": "Run go test -bench=. -benchtime=1x and report the benchmark numbers. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
{"id": "longproc-04", "name": "vet run", "class": "longproc", "prompt": "Run go vet ./... and report the output. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
{"id": "longctx-01", "name": "summarize large file", "class": "longctx", "prompt": "Read the large generated file big.go and summarize its structure. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
{"id": "longctx-02", "name": "find in large file", "class": "longctx", "prompt": "In big.go, find every function that returns an error and list them. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
{"id": "longctx-03", "name": "cross-file question", "class": "longctx", "prompt": "Read all .go files and report which two functions share the same name across files. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
{"id": "longctx-04", "name": "trace through large file", "class": "longctx", "prompt": "Starting from Handle, trace the call path through big.go and report the chain. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
{"id": "parallel-01", "name": "read six files", "class": "parallel", "prompt": "Read a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt, then report the first line of each. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
{"id": "parallel-02", "name": "grep six patterns", "class": "parallel", "prompt": "Search for 'alpha', 'beta', 'gamma', 'delta', 'epsilon', and 'zeta' across the repo and report which files contain each. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
{"id": "parallel-03", "name": "list six dirs", "class": "parallel", "prompt": "List the contents of dir1, dir2, dir3, dir4, dir5, and dir6. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
{"id": "parallel-04", "name": "stat six files", "class": "parallel", "prompt": "Report the line count of a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
{"id": "parallel-05", "name": "read configs", "class": "parallel", "prompt": "Read config1.json, config2.json, config3.json, config4.json, config5.json, and config6.json and report each top-level key. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
{"id": "parallel-06", "name": "summarize six", "class": "parallel", "prompt": "Read the six .txt files and give a one-line summary of each. Do not modify anything.", "workspaceFixture": "../testdata/parallel"}
{
"id": "nav-01",
"name": "list repo files",
"class": "nav",
"prompt": "List the files in this repository and report their count. State the count as 'count: N' on its own line. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'main.go' .zero-answer.txt && grep -iq 'config.json' .zero-answer.txt && grep -iq 'readme' .zero-answer.txt && grep -iqE '^count:[[:space:]]*5$' .zero-answer.txt"
]
},
{
"id": "nav-02",
"name": "find a function",
"class": "nav",
"prompt": "Find the function named 'greet' and report the file and line it is defined on. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'main.go' .zero-answer.txt && grep -iqE '\\b1[5-7]\\b' .zero-answer.txt"
]
},
{
"id": "nav-03",
"name": "summarize module",
"class": "nav",
"prompt": "Read main.go and summarize what this program does in two sentences. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iqE 'greet|hello|Config' .zero-answer.txt"
]
},
{
"id": "nav-04",
"name": "count tests",
"class": "nav",
"prompt": "Count the test functions (functions whose names start with Test) in this repository. Name each one you find and report the number. State the count as 'count: N' on its own line. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'TestGreet' .zero-answer.txt && grep -iqE '^count:[[:space:]]*1$' .zero-answer.txt"
]
},
{
"id": "nav-05",
"name": "locate a string",
"class": "nav",
"prompt": "Find every occurrence of the string 'TODO' in the repository and report the file(s) they appear in. State the count as 'count: N' on its own line. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'main.go' .zero-answer.txt && grep -iqE '^count:[[:space:]]*1$' .zero-answer.txt"
]
},
{
"id": "nav-06",
"name": "describe package",
"class": "nav",
"prompt": "Describe the exported API of this package by reading the source. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq config .zero-answer.txt && grep -iq maxretries .zero-answer.txt"
]
},
{
"id": "nav-07",
"name": "trace a call",
"class": "nav",
"prompt": "Trace which function calls which other function starting from main. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq main .zero-answer.txt && grep -iq greet .zero-answer.txt"
]
},
{
"id": "nav-08",
"name": "find imports",
"class": "nav",
"prompt": "List every import used by this repository and classify each as standard library or third-party. State the number of third-party (non-stdlib) imports as 'count: N' on its own line. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'fmt' .zero-answer.txt && grep -iq 'testing' .zero-answer.txt && grep -iqE '^count:[[:space:]]*0$' .zero-answer.txt"
]
},
{
"id": "nav-09",
"name": "report config",
"class": "nav",
"prompt": "Read config.json and report the keys it contains. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq port .zero-answer.txt && grep -iq name .zero-answer.txt && grep -iq retries .zero-answer.txt"
]
},
{
"id": "nav-10",
"name": "doc lookup",
"class": "nav",
"prompt": "Find the doc comment on the greet function and quote it. Do not modify anything.",
"workspaceFixture": "../testdata/nav",
"verificationCommand": [
"bash",
"-c",
"grep -iq 'greet returns a greeting' .zero-answer.txt"
]
},
{
"id": "edit-01",
"name": "rename a constant",
"class": "edit",
"prompt": "Rename the constant MaxRetries to RetryLimit everywhere it appears.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep --exclude=.zero-answer.txt -RIn 'MaxRetries =' ."
],
"oracleTest": "package edit\n\nvar _ = RetryLimit\n"
},
{
"id": "edit-02",
"name": "fix a typo",
"class": "edit",
"prompt": "Fix the typo in the comment above the greet function ('receieves' should be 'receives').",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep --exclude=.zero-answer.txt -R receieves . && grep --exclude=.zero-answer.txt -R receives ."
]
},
{
"id": "edit-03",
"name": "add a field",
"class": "edit",
"prompt": "Add a string field named Label to the Config struct in main.go.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nvar _ string = Config{}.Label\n"
},
{
"id": "edit-04",
"name": "bump a version",
"class": "edit",
"prompt": "Change the version string in version.go from 1.0.0 to 1.1.0.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestVersion(t *testing.T) {\n\tif Version != \"1.1.0\" {\n\t\tt.Fatalf(\"Version=%q\", Version)\n\t}\n}\n"
},
{
"id": "edit-05",
"name": "remove a log line",
"class": "edit",
"prompt": "Remove the line that prints 'debug: starting'.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport (\n\t\"os\"\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestNoDebugPrintln(t *testing.T) {\n\tb, err := os.ReadFile(\"main.go\")\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\t// The greet print is fmt.Println(greet(...)) with no string literal, so a\n\t// remaining fmt.Println(\"...\") call means the debug line was not removed\n\t// (or was only reworded, not deleted).\n\tif strings.Contains(string(b), `fmt.Println(\"`) {\n\t\tt.Fatal(\"main.go still has a fmt.Println string-literal call; the debug line was not removed\")\n\t}\n\t// The task is to remove the line that prints \"debug: starting\", so the\n\t// message text itself must be gone -- this catches a reword onto a different\n\t// call form (fmt.Printf, backticks, log.Println, println) that the\n\t// string-literal check above would miss.\n\tif strings.Contains(string(b), \"debug: starting\") {\n\t\tt.Fatal(\"main.go still prints 'debug: starting'; the line was not removed (only reworded or respelled)\")\n\t}\n}\n"
},
{
"id": "edit-06",
"name": "add a license header",
"class": "edit",
"prompt": "Add a comment line '// SPDX-License-Identifier: MIT' at the top of main.go.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport (\n\t\"os\"\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestSPDXHeader(t *testing.T) {\n\tb, err := os.ReadFile(\"main.go\")\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\tif !strings.HasPrefix(string(b), \"// SPDX-License-Identifier: MIT\") {\n\t\tt.Fatal(\"main.go must start with the SPDX-License-Identifier: MIT header\")\n\t}\n}\n"
},
{
"id": "edit-07",
"name": "change a default",
"class": "edit",
"prompt": "Change the default value of the Port constant from 8080 to 9090.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestPort(t *testing.T) {\n\tif Port != 9090 {\n\t\tt.Fatalf(\"Port=%d, want 9090\", Port)\n\t}\n}\n"
},
{
"id": "edit-08",
"name": "add a getter",
"class": "edit",
"prompt": "Add a method func (c *Config) GetName() string that returns c.Name.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestGetName(t *testing.T) {\n\tc := Config{Name: \"x\"}\n\tif got := c.GetName(); got != \"x\" {\n\t\tt.Fatalf(\"GetName()=%q, want %q\", got, \"x\")\n\t}\n}\n"
},
{
"id": "edit-09",
"name": "wrap an error",
"class": "edit",
"prompt": "In the load function, wrap the returned error with fmt.Errorf so it includes 'load failed'.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport (\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestLoadWrapsError(t *testing.T) {\n\t_, err := load(\"nonexistent-file-xyz\")\n\tif err == nil {\n\t\tt.Fatal(\"want a non-nil error for a missing file\")\n\t}\n\tif !strings.Contains(err.Error(), \"load failed\") {\n\t\tt.Fatalf(\"err=%v, want it to contain 'load failed'\", err)\n\t}\n}\n"
},
{
"id": "edit-10",
"name": "update a message",
"class": "edit",
"prompt": "Change the printed greeting from 'hello' to 'hello, world'.",
"workspaceFixture": "../testdata/edit",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestGreet(t *testing.T) {\n\tif got := greet(\"x\"); got != \"hello, world\" {\n\t\tt.Fatalf(\"greet()=%q, want %q\", got, \"hello, world\")\n\t}\n}\n"
},
{
"id": "fix-01",
"name": "off-by-one",
"class": "fix",
"prompt": "In bugs.go the Sum function has an off-by-one error that skips the last element; fix the loop so it sums all elements.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestSumIncludesLastElement",
"./..."
]
},
{
"id": "fix-02",
"name": "nil deref",
"class": "fix",
"prompt": "In bugs.go DefaultConfig returns nil and Port() dereferences it; make DefaultConfig return a valid Config with Port 8080.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestPortReturnsDefault",
"./..."
]
},
{
"id": "fix-03",
"name": "wrong operator",
"class": "fix",
"prompt": "In bugs.go Max uses the wrong inequality and returns the smaller value; fix it to return the larger.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestMaxReturnsLarger",
"./..."
]
},
{
"id": "fix-04",
"name": "truncated read",
"class": "fix",
"prompt": "In bugs.go ReadFirst returns only a single character instead of the full content; fix it to return the whole string.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestReadFirstReturnsFullContent",
"./..."
]
},
{
"id": "fix-05",
"name": "swapped args",
"class": "fix",
"prompt": "In bugs.go Divide swaps its operands; fix the order so Divide(10,2) returns 5.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestDivideOrder",
"./..."
]
},
{
"id": "fix-06",
"name": "missing return",
"class": "fix",
"prompt": "In bugs.go Classify is missing a return for the high case; add it so Classify(500) returns \"high\".",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestClassifyHigh",
"./..."
]
},
{
"id": "fix-07",
"name": "wrong label prefix",
"class": "fix",
"prompt": "In bugs.go Label returns 'admin-<name>' but the test expects 'user-<name>'; change the prefix from 'admin-' to 'user-'.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-run",
"TestLabelUsesString",
"./..."
]
},
{
"id": "fix-08",
"name": "race on counter",
"class": "fix",
"prompt": "In bugs.go the global counter is incremented without synchronization; guard it with a mutex.",
"workspaceFixture": "../testdata/fix",
"verificationCommand": [
"go",
"test",
"-race",
"-run",
"TestCounterRace",
"./..."
]
},
{
"id": "refactor-01",
"name": "extract helper",
"class": "refactor",
"prompt": "Extract the duplicated fmt.Sprintf(\"hello, %s\", ...) greeting logic into a single helper function named formatGreeting and call it from both GreetFromConfig and GreetByName.",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"bash",
"-c",
"go test ./... && [ $(grep -c 'hello, %s' main.go) -eq 1 ]"
],
"oracleTest": "package refactor\n\nvar _ = formatGreeting\n"
},
{
"id": "refactor-02",
"name": "split a file",
"class": "refactor",
"prompt": "Move the Config type into a new file named config.go, keeping the package building.",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"bash",
"-c",
"go test ./... && grep -q 'type Config' config.go && ! grep -q 'type Config' main.go"
]
},
{
"id": "refactor-03",
"name": "rename package",
"class": "refactor",
"prompt": "Rename the package from refactor to zeroapp everywhere and keep it building.",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep --exclude=.zero-answer.txt -R 'package refactor' . && grep --exclude=.zero-answer.txt -R 'package zeroapp' ."
],
"oracleTest": "package zeroapp\n\nvar _ = Config{}\n"
},
{
"id": "refactor-04",
"name": "introduce a type",
"class": "refactor",
"prompt": "Introduce a named type `type Stats map[string]int` for the bare map used by stats, and update its users (Record, Lookup).",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"go",
"test",
"./..."
],
"oracleTest": "package refactor\n\nimport (\n\t\"os\"\n\t\"regexp\"\n\t\"strings\"\n\t\"testing\"\n)\n\n// TestStatsNamedType proves the bare map was replaced by the named Stats type:\n// the old `var stats = map[string]int` (or `stats := map[string]int{}`) form is\n// gone -- matched with a spacing-tolerant regex so a `var stats=map[string]int`\n// or `stats :=` variant can't dodge it -- and the Stats type is actually\n// referenced in main.go, so a no-op that only adds an unused `type Stats` while\n// keeping the bare map fails. `go test ./...` compiling the fixture proves the\n// type declaration is valid.\nfunc TestStatsNamedType(t *testing.T) {\n\tb, err := os.ReadFile(\"main.go\")\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\tsrc := string(b)\n\tif regexp.MustCompile(`var\\s+stats\\s*=\\s*map\\[string\\]int`).MatchString(src) ||\n\t\tregexp.MustCompile(`stats\\s*:=\\s*map\\[string\\]int`).MatchString(src) {\n\t\tt.Fatal(\"main.go still declares stats as a bare map[string]int; the named Stats type was not adopted\")\n\t}\n\tif !strings.Contains(src, \"Stats\") {\n\t\tt.Fatal(\"main.go does not reference the Stats type; refactor-04 was a no-op\")\n\t}\n}\n"
},
{
"id": "refactor-05",
"name": "inline a wrapper",
"class": "refactor",
"prompt": "Inline the Wrapper function into greetWrapped (its single caller) and remove the Wrapper function.",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep --exclude=.zero-answer.txt -R 'func Wrapper' ."
],
"oracleTest": "package refactor\n\nimport \"testing\"\n\n// TestGreetWrapped proves greetWrapped still exists and behaves as the inlined\n// Wrapper did: deleting both Wrapper and greetWrapped (or stubbing greetWrapped\n// to return \"\") fails to compile or run, so the oracle can't be passed by\n// removing the target along with its caller. Paired with the ! grep 'func\n// Wrapper' verificationCommand that proves Wrapper itself is gone.\nfunc TestGreetWrapped(t *testing.T) {\n\tif got := greetWrapped(\"x\"); got != \"hello, x\" {\n\t\tt.Fatalf(\"greetWrapped(x) = %q, want %q\", got, \"hello, x\")\n\t}\n}\n"
},
{
"id": "refactor-06",
"name": "consolidate errors",
"class": "refactor",
"prompt": "Consolidate the three error-construction functions (firstError, secondError, buildError) into a single helper func makeLoadError(ctx string) error that returns fmt.Errorf(\"load failed: %s\", ctx), and call it from all three.",
"workspaceFixture": "../testdata/refactor",
"verificationCommand": [
"bash",
"-c",
"go test ./... && [ $(grep -c 'fmt.Errorf(\"load failed' main.go) -eq 1 ]"
],
"oracleTest": "package refactor\n\nvar _ = makeLoadError\n"
},
{
"id": "longproc-01",
"name": "long build",
"class": "longproc",
"prompt": "Run the build command and report whether it succeeds. Do not modify anything.",
"workspaceFixture": "../testdata/longproc"
},
{
"id": "longproc-02",
"name": "long test",
"class": "longproc",
"prompt": "Run go test ./... and report the result. Do not modify anything.",
"workspaceFixture": "../testdata/longproc"
},
{
"id": "longproc-03",
"name": "bench run",
"class": "longproc",
"prompt": "Run go test -bench=. -benchtime=1x and report the benchmark numbers. Do not modify anything.",
"workspaceFixture": "../testdata/longproc"
},
{
"id": "longproc-04",
"name": "vet run",
"class": "longproc",
"prompt": "Run go vet ./... and report the output. Do not modify anything.",
"workspaceFixture": "../testdata/longproc"
},
{
"id": "longctx-01",
"name": "summarize large file",
"class": "longctx",
"prompt": "Read the large generated file big.go and summarize its structure. Do not modify anything.",
"workspaceFixture": "../testdata/longctx"
},
{
"id": "longctx-02",
"name": "find in large file",
"class": "longctx",
"prompt": "In big.go, find every function that returns an error and list them. Do not modify anything.",
"workspaceFixture": "../testdata/longctx"
},
{
"id": "longctx-03",
"name": "cross-file question",
"class": "longctx",
"prompt": "Read all .go files and report which two functions share the same name across files. Do not modify anything.",
"workspaceFixture": "../testdata/longctx"
},
{
"id": "longctx-04",
"name": "trace through large file",
"class": "longctx",
"prompt": "Starting from Handle, trace the call path through big.go and report the chain. Do not modify anything.",
"workspaceFixture": "../testdata/longctx"
},
{
"id": "parallel-01",
"name": "read six files",
"class": "parallel",
"prompt": "Read a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt, then report the first line of each. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
},
{
"id": "parallel-02",
"name": "grep six patterns",
"class": "parallel",
"prompt": "Search for 'alpha', 'beta', 'gamma', 'delta', 'epsilon', and 'zeta' across the repo and report which files contain each. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
},
{
"id": "parallel-03",
"name": "list six dirs",
"class": "parallel",
"prompt": "List the contents of dir1, dir2, dir3, dir4, dir5, and dir6. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
},
{
"id": "parallel-04",
"name": "stat six files",
"class": "parallel",
"prompt": "Report the line count of a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
},
{
"id": "parallel-05",
"name": "read configs",
"class": "parallel",
"prompt": "Read config1.json, config2.json, config3.json, config4.json, config5.json, and config6.json and report each top-level key. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
},
{
"id": "parallel-06",
"name": "summarize six",
"class": "parallel",
"prompt": "Read the six .txt files and give a one-line summary of each. Do not modify anything.",
"workspaceFixture": "../testdata/parallel"
}
]
}
}
+57 -9
View File
@@ -51,6 +51,14 @@ type BenchTask struct {
Prompt string `json:"prompt"`
WorkspaceFixture string `json:"workspaceFixture,omitempty"`
VerificationCommand []string `json:"verificationCommand,omitempty"`
// OracleTest is Go source stamped into <fixtureCopy>/oracle_test.go AFTER
// the agent run (so it can't interfere with the agent's own go build/test
// during the task and can't be pre-seen). The verificationCommand then runs
// `go test ./...`, making the Go compiler the structural verifier: a
// compile-time reference (e.g. `var _ = Config{}.Label`) or a behavioral call
// fails to compile/run when the task wasn't actually done. Empty for tasks
// whose verificationCommand is a plain grep/build with no stamped test.
OracleTest string `json:"oracleTest,omitempty"`
}
// TaskConfig is the recorded configuration of a benchmark run. The honest
@@ -77,6 +85,10 @@ type RunContext struct {
Model string
Mode string
SelfCorrect bool
// ExecProfile, when non-empty, is forwarded to the agent invocation as
// --exec-profile (turn benchmark runner; the task benchmark leaves it
// empty).
ExecProfile string
}
// TaskRunner runs a single benchmark task and returns its outcome.
@@ -389,14 +401,26 @@ func runVerification(ctx context.Context, task BenchTask) TaskOutcome {
return TaskOutcome{Passed: true}
}
// streamJSONExitCode scans stream-json output for the terminal run_end event and
// returns its exit code. ZERO emits one JSON object per line; the last run_end
// (or final) event carries the exit code. Lines that don't parse are skipped.
func streamJSONExitCode(output []byte) (int, bool) {
// streamJSONResult holds the terminal events parsed from one pass over a
// stream-json output: the last run_end exit code (if any) and the last final
// event's text (the agent's answer, if any). streamJSONExitCode and
// streamJSONFinalText both consume this so the scanner limits and
// malformed-event handling live in exactly one place.
type streamJSONResult struct {
exitCode int
haveExit bool
finalText string
}
// parseStreamJSON scans stream-json output once and returns the terminal
// run_end exit code and the final event's text. ZERO emits one JSON object per
// line; the last run_end carries the exit code and the last final carries the
// answer. Lines that don't parse are skipped. The scanner buffer is raised so a
// large single-line event (e.g. a big final answer) doesn't error out.
func parseStreamJSON(output []byte) streamJSONResult {
scanner := bufio.NewScanner(bytes.NewReader(output))
scanner.Buffer(make([]byte, 0, 64*1024), 4*1024*1024)
exitCode := 0
found := false
var r streamJSONResult
for scanner.Scan() {
line := bytes.TrimSpace(scanner.Bytes())
if len(line) == 0 || line[0] != '{' {
@@ -405,16 +429,40 @@ func streamJSONExitCode(output []byte) (int, bool) {
var event struct {
Type string `json:"type"`
ExitCode *int `json:"exitCode"`
Text string `json:"text"`
}
if err := json.Unmarshal(line, &event); err != nil {
continue
}
if event.Type == "run_end" && event.ExitCode != nil {
exitCode = *event.ExitCode
found = true
r.exitCode = *event.ExitCode
r.haveExit = true
}
if event.Type == "final" {
r.finalText = event.Text
}
}
return exitCode, found
return r
}
// streamJSONExitCode scans stream-json output for the terminal run_end event and
// returns its exit code. The last run_end event carries the exit code. Lines
// that don't parse are skipped.
func streamJSONExitCode(output []byte) (int, bool) {
r := parseStreamJSON(output)
return r.exitCode, r.haveExit
}
// streamJSONFinalText scans stream-json output for the terminal "final" event
// and returns its text (the agent's final answer). ZERO emits one final event
// per run on both the success and incomplete paths (exec.go writer.final), so
// the last one wins. The turn benchmark writes this to .zero-answer.txt in the
// fixture copy so nav answer-oracles can grep the captured answer rather than
// the raw stream. Returns "" when no final event was emitted (e.g. a provider
// error path) — the caller writes an empty file and a nav grep then fails,
// which is the correct outcome for a run that produced no answer.
func streamJSONFinalText(output []byte) string {
return parseStreamJSON(output).finalText
}
func firstLine(text string) string {
+3
View File
@@ -0,0 +1,3 @@
module edit
go 1.22
+3
View File
@@ -0,0 +1,3 @@
module fix
go 1.22
+3
View File
@@ -0,0 +1,3 @@
module nav
go 1.22
+4
View File
@@ -22,3 +22,7 @@ func greet(name string) string {
func main() {
fmt.Println(greet("world"))
}
// TODO: replace the demo greet with the real client call before shipping.
// (Present so nav-05's find-the-markers task has a non-zero, inspection-
// required answer; an agent that always emits "count: 0" fails.)
+13
View File
@@ -0,0 +1,13 @@
package nav
import "testing"
// TestGreet is a real test so nav-04's "count the test functions" has a
// non-zero, inspection-required answer: an agent that never opens the workspace
// and always emits "count: 0" fails, which is the whole point of graduating nav
// into the correctness tier. It runs green offline (no third-party deps).
func TestGreet(t *testing.T) {
if got := greet("x"); got != "hello, x" {
t.Fatalf("greet(x) = %q, want %q", got, "hello, x")
}
}
+3
View File
@@ -0,0 +1,3 @@
module refactor
go 1.22
+6
View File
@@ -28,6 +28,12 @@ func Wrapper(name string) string {
return GreetByName(name)
}
// greetWrapped is Wrapper's single caller — refactor-05 inlines Wrapper here
// and removes the Wrapper function.
func greetWrapped(name string) string {
return Wrapper(name)
}
// stats is a bare map used across the file — refactor-04 introduces a named type.
var stats = map[string]int{}
+198 -32
View File
@@ -15,6 +15,7 @@ import (
"strings"
"time"
"github.com/Gitlawb/zero/internal/execprofile"
"github.com/Gitlawb/zero/internal/trace"
)
@@ -27,7 +28,38 @@ import (
// buildPassedTasks/buildPassRate cover the non-positive build-check classes
// (refactor); latencyOnlyTasks covers the no-oracle classes (nav/longproc/
// longctx/parallel). tasksAttempted is still every task run.
const TurnSchemaVersion = 2
//
// v3 (Phase 0 oracle hardening) strengthens the oracles so the tiers are honest: edit/refactor
// now carry a stamped oracle_test.go compiled by `go test ./...` (the Go
// compiler is the structural verifier — a no-op refactor or a missing field
// fails to compile), and nav carries an answer-oracle grepping the agent's
// captured final answer. refactor graduates from build to correctness (its
// oracle is now positive), nav graduates from latency-only to correctness, and
// the build tier is empty (nothing is only-build-checked anymore). The result
// SHAPE is unchanged — only class-list membership shifts — but
// correctnessPassRate now means something strictly stronger, so the bump
// prevents a v2→v2 cross-version comparison from misreading the jump as a model
// improvement. New counts: correctness 34 (edit 10 + fix 8 + nav 10 + refactor
// 6), build 0, latency 14 (longproc 4 + longctx 4 + parallel 6).
//
// v4 adds tasksErrored: the number of tasks whose every iteration produced no
// accepted benchmark sample. That covers pre-run failures (missing binary,
// spawn error, process crash) and harness errors after a successful agent run
// (e.g. oracle stamping failed) alike — in every case the iteration yields no
// usable measurement. Errored tasks were previously visible only in warnings,
// so a run where nothing was actually measured still printed a clean-looking
// 0% pass summary and exited 0. tasksErrored makes that state first-class:
// the summary surfaces it, and the turn command exits nonzero when every
// attempted task errored. Accounting: an errored oracle task stays in its
// tier denominator as a failure; an errored latency-only task counts in
// latencyOnlyTasks and, as always, in no pass rate.
//
// v5 is ADDITIVE ONLY: execProfile records the execution profile the run was
// benchmarked under (omitted when none was selected), so profile A/B captures
// are self-describing and two reports can't be compared without noticing they
// ran different postures. No existing field changed shape or meaning; a v4
// consumer reading a v5 report misses only the new field.
const TurnSchemaVersion = 5
// TurnRunner runs one benchmark task and reports its outcome plus the captured
// per-turn trace. A non-nil Err means the run failed to execute (process crash);
@@ -54,6 +86,10 @@ type TurnBenchConfig struct {
Model string
Mode string
SelfCorrect bool
// ExecProfile, when non-empty, runs every task under the named execution
// profile (zero exec --exec-profile) and stamps it into the result so the
// report is self-describing for profile A/B comparisons.
ExecProfile string
Version string
Commit string
// Iterations is how many times each task is run. The per-process `zero exec`
@@ -89,10 +125,11 @@ type LatencySource struct {
// ClassSummary is the per-class (task group) roll-up.
//
// Passed is the count that passed THIS class's oracle: a correctness pass for
// edit/fix, a build pass for refactor, and always 0 for the no-oracle classes
// (nav/longproc/longctx/parallel). Verified is how many tasks in the class
// carry an oracle (correctness or build); LatencyOnly is how many carry none.
// A latency-only class therefore reports Passed=0, Verified=0, LatencyOnly=Tasks.
// the positive-oracle classes (edit/fix/nav/refactor), and always 0 for the
// no-oracle classes (longproc/longctx/parallel) — the build tier is empty in v3
// (refactor graduated to correctness). Verified is how many tasks in the class
// carry an oracle; LatencyOnly is how many carry none. A latency-only class
// therefore reports Passed=0, Verified=0, LatencyOnly=Tasks.
type ClassSummary struct {
Tasks int `json:"tasks"`
Verified int `json:"verified"`
@@ -107,13 +144,14 @@ type ClassSummary struct {
// Pass/fail is split into three oracle tiers so it cannot be misread as a
// blanket correctness verdict:
// - Correctness (tasksVerified / tasksPassed / correctnessPassRate): tasks
// with a positive oracle (edit/fix). This is the only pass rate that can
// move with model quality.
// - Build-only (buildCheckedTasks / buildPassedTasks / buildPassRate): tasks
// whose oracle is a non-positive build check (refactor `go build`). A pass
// means it compiles, not that the refactor is correct.
// - Latency-only (latencyOnlyTasks): tasks with no oracle (nav/longproc/
// longctx/parallel). They ran for latency and span attribution only and are
// with a positive oracle (edit/fix/nav/refactor). This is the only pass
// rate that can move with model quality.
// - Build-only (buildCheckedTasks / buildPassedTasks / buildPassRate): the
// non-positive build-check tier. Empty in v3 — refactor graduated to
// correctness (its stamped oracle is positive) — so buildCheckedTasks is
// always 0 and buildPassRate is 0. The fields remain for schema stability.
// - Latency-only (latencyOnlyTasks): tasks with no oracle (longproc/longctx/
// parallel). They ran for latency and span attribution only and are
// excluded from every pass rate.
//
// tasksAttempted is still the total number of tasks run across all three tiers.
@@ -124,11 +162,13 @@ type TurnBenchResult struct {
Suite string `json:"suite"`
Model string `json:"model"`
Mode string `json:"mode,omitempty"`
ExecProfile string `json:"execProfile,omitempty"`
SelfCorrect bool `json:"selfCorrect"`
Version string `json:"version,omitempty"`
Commit string `json:"commit,omitempty"`
Date string `json:"date"`
TasksAttempted int `json:"tasksAttempted"`
TasksErrored int `json:"tasksErrored"`
TasksVerified int `json:"tasksVerified"`
TasksPassed int `json:"tasksPassed"`
LatencyOnlyTasks int `json:"latencyOnlyTasks"`
@@ -174,6 +214,19 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
if cfg.Runner == nil {
return TurnBenchResult{}, errors.New("turn benchmark requires a runner")
}
// Canonicalize the profile once at the boundary so every consumer — the
// child exec args, the stamped result, the summary — carries the same
// catalog name regardless of the caller's casing/whitespace, and a direct
// library caller cannot slip an unknown name into a report the way the CLI
// (which validates at parse time) cannot.
benchProfile := strings.TrimSpace(cfg.ExecProfile)
if benchProfile != "" {
profile, ok := execprofile.Lookup(benchProfile)
if !ok {
return TurnBenchResult{}, fmt.Errorf("unknown execution profile %q (valid: %s)", cfg.ExecProfile, strings.Join(execprofile.Names(), ", "))
}
benchProfile = profile.Name
}
iterations := cfg.Iterations
if iterations < 1 {
iterations = 1
@@ -182,7 +235,7 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
if now == nil {
now = time.Now
}
rc := RunContext{Model: cfg.Model, Mode: cfg.Mode, SelfCorrect: cfg.SelfCorrect}
rc := RunContext{Model: cfg.Model, Mode: cfg.Mode, SelfCorrect: cfg.SelfCorrect, ExecProfile: benchProfile}
perSpanSamples := map[string][]float64{}
classWalls := map[string][]float64{}
@@ -212,6 +265,7 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
Suite: strings.TrimSpace(set.ID),
Model: strings.TrimSpace(cfg.Model),
Mode: strings.TrimSpace(cfg.Mode),
ExecProfile: benchProfile,
SelfCorrect: cfg.SelfCorrect,
Version: strings.TrimSpace(cfg.Version),
Commit: strings.TrimSpace(cfg.Commit),
@@ -234,6 +288,7 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
// runner is cold-start, so a flaky pass on one iteration and a fail on
// another is a real regression signal, not noise to average away.
passedForTask := true
erroredIterations := 0
for iter := 0; iter < iterations; iter++ {
outcome := cfg.Runner(ctx, task, rc)
if outcome.TraceIssue != "" {
@@ -251,6 +306,7 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
Message: fmt.Sprintf("task %s: %v", task.ID, outcome.Err),
})
passedForTask = false
erroredIterations++
continue
}
if !outcome.Passed {
@@ -291,6 +347,14 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
// counters. A latency-only task (no verificationCommand) is never counted
// in any pass rate even when the runner reports Passed — an exit-0
// read-only run proves the turn ran, not that the answer was right.
if erroredIterations == iterations {
// Every iteration produced no accepted sample (spawn failure or a
// harness error after the run). The task still counts in its tier
// below — an oracle task as a failure, a latency-only task in no
// pass rate as always — but the errored state is first-class so a
// broken run can never print a clean-looking summary.
result.TasksErrored++
}
hasOracle := len(task.VerificationCommand) > 0
switch {
case !hasOracle:
@@ -432,17 +496,35 @@ func FormatTurnBenchSummary(result TurnBenchResult) string {
"model: " + displayOrUnknown(result.Model),
// The headline separates the three oracle tiers so an exit-0 read-only
// task can never inflate a "pass rate" that reads as correctness:
// correctness (positive oracle, edit/fix), build (non-positive `go build`,
// refactor), and latency-only (no oracle, nav/longproc/longctx/parallel).
// correctness (positive oracle: edit/fix/nav/refactor), build (empty in
// v3 — refactor graduated to correctness), and latency-only (no oracle:
// longproc/longctx/parallel).
fmt.Sprintf("tasks: %d total | correctness %d/%d (%.0f%%) | build %d/%d (%.0f%%) | latency-only %d | %d iter",
result.TasksAttempted,
result.TasksPassed, result.TasksVerified, result.CorrectnessPassRate*100,
result.BuildPassedTasks, result.BuildCheckedTasks, result.BuildPassRate*100,
result.LatencyOnlyTasks, result.Iterations),
}
if result.TasksErrored > 0 {
lines = append(lines, fmt.Sprintf("ERRORED: %d task(s) produced no accepted benchmark sample (spawn/crash or harness error) — errored oracle tasks count as failures in their tier's pass rate; latency-only tasks stay out of pass rates as always", result.TasksErrored))
shown := 0
for _, warning := range result.Warnings {
if warning.Metric != "run" {
continue
}
lines = append(lines, " "+warning.Message)
shown++
if shown == 3 {
break
}
}
}
if result.Mode != "" {
lines = append(lines, "mode: "+result.Mode)
}
if result.ExecProfile != "" {
lines = append(lines, "exec-profile: "+result.ExecProfile)
}
if len(result.TopLatency) > 0 {
lines = append(lines, "top latency sources:")
for _, src := range result.TopLatency {
@@ -509,11 +591,13 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
// fixture or bleed into a later iteration of the same task. When no
// fixture is configured the agent runs in the caller's cwd as before.
if fixture := strings.TrimSpace(task.WorkspaceFixture); fixture != "" {
copyDir, cerr := copyFixture(fixture)
copyDir, parent, cerr := copyFixture(fixture)
if cerr != nil {
return TurnTaskOutcome{Err: fmt.Errorf("isolate fixture: %w", cerr)}
}
defer os.RemoveAll(copyDir)
// Clean the whole unique parent (which owns copyDir) so the
// per-invocation scratch dir never leaks.
defer os.RemoveAll(parent)
task.WorkspaceFixture = copyDir
}
@@ -577,6 +661,22 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
if outcome.VerifyErr != "" {
return outcome
}
// Stamp the compiler-backed oracle and capture the agent's final answer
// BEFORE running the verification command. Both write into the fixture
// copy (task.WorkspaceFixture, set to copyDir above) so runVerification —
// which uses that same dir as cmd.Dir — sees them. Stamping happens only
// after the agent run so the oracle test can't interfere with the agent's
// own go build/test during the task (e.g. refactor-03's package-zeroapp
// test would break a pre-rename build) and can't be pre-seen or tampered
// with.
if err := stampOracleAndAnswer(task, outBuf.Bytes()); err != nil {
// Preserve the already-parsed trace, trace issue, and wall time: a
// stamp failure is a harness error, but the run still produced a
// trace worth attributing, so mutate the existing outcome rather than
// returning a fresh one that drops Trace/TraceIssue.
outcome.Err = fmt.Errorf("stamp oracle: %w", err)
return outcome
}
if len(task.VerificationCommand) > 0 {
if vOutcome := runVerification(ctx, task); !vOutcome.Passed {
outcome.VerifyErr = strings.TrimSpace(vOutcome.Detail)
@@ -584,7 +684,7 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
}
// A positive oracle (grep/test/build) passed: this is the only path
// that sets Passed. A task with no verificationCommand is latency-only
// (read-only nav/longproc/longctx/parallel): its exit 0 proves the turn
// (read-only longproc/longctx/parallel): its exit 0 proves the turn
// ran, not that the answer was right, so it never reports Passed and the
// harness counts it in latencyOnlyTasks rather than any pass rate.
outcome.Passed = true
@@ -594,22 +694,41 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
}
// copyFixture copies the fixture directory at src into a fresh temp dir and
// returns the copy's path. Used to give each benchmark invocation an isolated
// workspace so mutating tasks (edit/fix/refactor) can't dirty the checked-in
// fixture or a later iteration.
func copyFixture(src string) (string, error) {
// returns the copy's path (dst) plus the unique parent dir that owns it (parent,
// which the caller must RemoveAll to clean up dst and its sibling scratch space
// together). Used to give each benchmark invocation an isolated workspace so
// mutating tasks (edit/fix/refactor) can't dirty the checked-in fixture or a
// later iteration.
//
// The copy is placed two levels BELOW the system temp root: a unique 0700
// parent (created fresh per invocation via os.MkdirTemp, so concurrent runs and
// different users can't share or traverse a predictable scratch dir) and the
// fixture copy beneath it. Go ignores go.mod files in DIRECT children of the
// system temp dir (a hijack guard), which would make the fixture's go.mod —
// and thus every `go test ./...`/`go build ./...` oracle — fail with "cannot
// find main module". Keeping the copy a grandchild of the temp root makes Go
// respect the copied go.mod so the compiler-backed oracles actually run.
func copyFixture(src string) (dst, parent string, err error) {
info, err := os.Stat(src)
if err != nil {
return "", err
return "", "", err
}
if !info.IsDir() {
return "", fmt.Errorf("fixture %q is not a directory", src)
return "", "", fmt.Errorf("fixture %q is not a directory", src)
}
dst, err := os.MkdirTemp("", "zero-turn-fixture-*")
// Unique, 0700-per-invocation parent directly under the temp root. The
// fixture copy is created BENEATH it, so the copy is a grandchild of the
// temp root and its go.mod is respected (see the doc comment above).
parent, err = os.MkdirTemp(os.TempDir(), "zero-turn-bench-*")
if err != nil {
return "", err
return "", "", err
}
err = filepath.WalkDir(src, func(path string, d fs.DirEntry, werr error) error {
dst, err = os.MkdirTemp(parent, "zero-turn-fixture-*")
if err != nil {
os.RemoveAll(parent)
return "", "", err
}
werr := filepath.WalkDir(src, func(path string, d fs.DirEntry, werr error) error {
if werr != nil {
return werr
}
@@ -627,11 +746,47 @@ func copyFixture(src string) (string, error) {
}
return os.WriteFile(target, data, 0o644)
})
if err != nil {
os.RemoveAll(dst)
return "", err
if werr != nil {
os.RemoveAll(parent) // removes dst too
return "", "", werr
}
return dst, nil
return dst, parent, nil
}
// stampOracleAndAnswer writes the compiler-backed oracle test (when configured)
// and the agent's captured final answer into the fixture copy, so the
// verification command can compile the oracle and grep the answer. Both files
// land in task.WorkspaceFixture, which the runner has already set to the
// fixture copy and which runVerification uses as cmd.Dir. The answer file is
// always written (empty when no final event was captured) so a nav answer-oracle
// fails cleanly on a run that produced no answer rather than reading a stale or
// absent file.
//
// A task that needs an oracle — a stamped oracle_test.go OR a verification
// command that reads .zero-answer.txt — but has no workspace fixture is
// rejected: there is nowhere safe to stamp, so runVerification would otherwise
// run in the caller's cwd with the oracle never compiled (or the answer never
// captured), which can read as a false pass. A fixtureless task with no oracle
// and no verification (a pure latency-only run in the caller's cwd) is left
// alone.
func stampOracleAndAnswer(task BenchTask, outBuf []byte) error {
dir := strings.TrimSpace(task.WorkspaceFixture)
if dir == "" {
if task.OracleTest != "" || len(task.VerificationCommand) > 0 {
return fmt.Errorf("stamp oracle: task %q has an oracle or verification but no workspace fixture to stamp into", task.ID)
}
return nil
}
if ot := task.OracleTest; ot != "" {
if err := os.WriteFile(filepath.Join(dir, "oracle_test.go"), []byte(ot), 0o644); err != nil {
return fmt.Errorf("write oracle_test.go: %w", err)
}
}
answer := streamJSONFinalText(outBuf)
if err := os.WriteFile(filepath.Join(dir, ".zero-answer.txt"), []byte(answer), 0o644); err != nil {
return fmt.Errorf("write .zero-answer.txt: %w", err)
}
return nil
}
func buildTurnExecArgs(task BenchTask, rc RunContext, tracePath string, extraArgs []string) []string {
@@ -645,6 +800,9 @@ func buildTurnExecArgs(task BenchTask, rc RunContext, tracePath string, extraArg
if rc.SelfCorrect {
args = append(args, "--self-correct")
}
if profile := strings.TrimSpace(rc.ExecProfile); profile != "" {
args = append(args, "--exec-profile", profile)
}
args = append(args, extraArgs...)
args = append(args, task.Prompt)
return args
@@ -658,7 +816,15 @@ func ResolveBinary(explicit string) (string, error) {
if _, err := os.Stat(v); err != nil {
return "", fmt.Errorf("trace binary not found: %w", err)
}
return v, nil
// Pin the explicit path to an absolute one: the turn runner sets each
// child's cmd.Dir to a per-task fixture copy, so a relative path
// (./zero, the Makefile default) that stats fine here would fail to
// spawn from inside every fixture dir, silently erroring all tasks.
absolute, err := filepath.Abs(v)
if err != nil {
return "", fmt.Errorf("resolve trace binary path: %w", err)
}
return absolute, nil
}
if path, err := exec.LookPath("zero"); err == nil {
return path, nil
+833 -31
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
@@ -51,21 +52,26 @@ func cannedTrace(genMs, toolMs int, tokens int64) *trace.TurnTrace {
}
func TestRunTurnBenchAggregation(t *testing.T) {
// v3 tier model: nav and refactor both carry a verificationCommand and are
// NOT in buildOnlyClasses, so they count as correctness; longproc has no
// oracle and is latency-only. The build tier is empty (buildOnlyClasses is
// nil), matching the v3 baseline manifest where refactor graduated from
// build to correctness.
set := TaskSet{
ID: "fake-suite",
Tasks: []BenchTask{
{ID: "t1", Class: "nav", Prompt: "p1"}, // latency-only
{ID: "t2", Class: "nav", Prompt: "p2"}, // latency-only
{ID: "t3", Class: "edit", Prompt: "p3", VerificationCommand: []string{"true"}}, // correctness
{ID: "t4", Class: "refactor", Prompt: "p4", VerificationCommand: []string{"true"}}, // build-only
{ID: "t1", Class: "nav", Prompt: "p1", VerificationCommand: []string{"true"}}, // correctness
{ID: "t2", Class: "edit", Prompt: "p2", VerificationCommand: []string{"true"}}, // correctness
{ID: "t3", Class: "refactor", Prompt: "p3", VerificationCommand: []string{"true"}}, // correctness
{ID: "t4", Class: "longproc", Prompt: "p4"}, // latency-only
},
BuildOnlyClasses: []string{"refactor"},
BuildOnlyClasses: nil,
}
canned := map[string]*trace.TurnTrace{
"t1": cannedTrace(100, 10, 1000),
"t2": cannedTrace(300, 10, 1000),
"t3": cannedTrace(200, 50, 2000),
"t4": cannedTrace(150, 20, 1500),
"t2": cannedTrace(200, 50, 2000),
"t3": cannedTrace(150, 20, 1500),
"t4": cannedTrace(300, 10, 1000),
}
cfg := TurnBenchConfig{
Model: "fake-model",
@@ -77,33 +83,36 @@ func TestRunTurnBenchAggregation(t *testing.T) {
if err != nil {
t.Fatalf("RunTurnBench: %v", err)
}
// 4 tasks attempted; the tier split is 2 latency-only, 1 correctness, 1 build.
// 4 tasks attempted; the v3 tier split is 3 correctness (nav, edit, refactor),
// 1 latency-only (longproc), and 0 build (the build tier is empty).
if result.TasksAttempted != 4 {
t.Fatalf("attempted=%d, want 4", result.TasksAttempted)
}
if result.TasksVerified != 1 || result.TasksPassed != 1 {
t.Fatalf("correctness verified=%d passed=%d, want 1/1", result.TasksVerified, result.TasksPassed)
if result.TasksVerified != 3 || result.TasksPassed != 3 {
t.Fatalf("correctness verified=%d passed=%d, want 3/3", result.TasksVerified, result.TasksPassed)
}
if result.LatencyOnlyTasks != 2 {
t.Fatalf("latencyOnly=%d, want 2", result.LatencyOnlyTasks)
if result.LatencyOnlyTasks != 1 {
t.Fatalf("latencyOnly=%d, want 1", result.LatencyOnlyTasks)
}
if result.BuildCheckedTasks != 1 || result.BuildPassedTasks != 1 {
t.Fatalf("build checked=%d passed=%d, want 1/1", result.BuildCheckedTasks, result.BuildPassedTasks)
if result.BuildCheckedTasks != 0 || result.BuildPassedTasks != 0 {
t.Fatalf("build checked=%d passed=%d, want 0/0 (empty build tier)", result.BuildCheckedTasks, result.BuildPassedTasks)
}
if result.CorrectnessPassRate != 1.0 {
t.Fatalf("correctnessPassRate=%v, want 1.0", result.CorrectnessPassRate)
}
if result.BuildPassRate != 1.0 {
t.Fatalf("buildPassRate=%v, want 1.0", result.BuildPassRate)
// passRate returns 0 for a 0/0 denominator, so an empty build tier reports 0
// (not NaN) and cannot be misread as a perfect build score.
if result.BuildPassRate != 0 {
t.Fatalf("buildPassRate=%v, want 0 (empty build tier is 0/0 -> 0)", result.BuildPassRate)
}
if len(result.CorrectnessClasses) != 1 || result.CorrectnessClasses[0] != "edit" {
t.Fatalf("correctnessClasses=%v, want [edit]", result.CorrectnessClasses)
if len(result.CorrectnessClasses) != 3 || result.CorrectnessClasses[0] != "edit" || result.CorrectnessClasses[1] != "nav" || result.CorrectnessClasses[2] != "refactor" {
t.Fatalf("correctnessClasses=%v, want [edit nav refactor]", result.CorrectnessClasses)
}
if len(result.BuildOnlyClasses) != 1 || result.BuildOnlyClasses[0] != "refactor" {
t.Fatalf("buildOnlyClasses=%v, want [refactor]", result.BuildOnlyClasses)
if len(result.BuildOnlyClasses) != 0 {
t.Fatalf("buildOnlyClasses=%v, want empty (v3: build tier empty)", result.BuildOnlyClasses)
}
if len(result.LatencyOnlyClasses) != 1 || result.LatencyOnlyClasses[0] != "nav" {
t.Fatalf("latencyOnlyClasses=%v, want [nav]", result.LatencyOnlyClasses)
if len(result.LatencyOnlyClasses) != 1 || result.LatencyOnlyClasses[0] != "longproc" {
t.Fatalf("latencyOnlyClasses=%v, want [longproc]", result.LatencyOnlyClasses)
}
if result.SchemaVersion != TurnSchemaVersion {
t.Fatalf("schemaVersion = %d, want %d", result.SchemaVersion, TurnSchemaVersion)
@@ -112,8 +121,8 @@ func TestRunTurnBenchAggregation(t *testing.T) {
t.Fatalf("date = %q", result.Date)
}
// Per-span: generation appears in all four (100+300+200+150=750ms), tool in
// all four (10+10+50+20=90ms). Count must equal the number of tasks * iterations.
// Per-span: generation appears in all four (100+200+150+300=750ms), tool in
// all four (10+50+20+10=90ms). Count must equal the number of tasks * iterations.
gen := result.PerSpan[trace.SpanGeneration]
if gen.Count != 4 {
t.Fatalf("generation count = %d, want 4", gen.Count)
@@ -135,7 +144,7 @@ func TestRunTurnBenchAggregation(t *testing.T) {
t.Fatalf("top latency not ranked by share: %+v", result.TopLatency)
}
// Totals: 4 model requests, 4 tool calls, input tokens 1000+1000+2000+1500=5500.
// Totals: 4 model requests, 4 tool calls, input tokens 1000+2000+1500+1000=5500.
if result.Totals.ModelRequests != 4 {
t.Fatalf("modelRequests = %d, want 4", result.Totals.ModelRequests)
}
@@ -149,10 +158,10 @@ func TestRunTurnBenchAggregation(t *testing.T) {
t.Fatalf("outputTokens = %d, want 2750", result.Totals.OutputTokens)
}
// Per-class tier roll-up: nav is latency-only (0 verified, 2 latency-only),
// edit is correctness (1/1 verified passed), refactor is build (1/1 passed).
// Per-class tier roll-up: nav/edit/refactor are correctness (1/1 verified
// passed, 0 latency-only); longproc is latency-only (0 verified, 1 latency).
nav := result.PerClass["nav"]
if nav.Tasks != 2 || nav.Verified != 0 || nav.Passed != 0 || nav.LatencyOnly != 2 {
if nav.Tasks != 1 || nav.Verified != 1 || nav.Passed != 1 || nav.LatencyOnly != 0 {
t.Fatalf("nav class = %+v", nav)
}
edit := result.PerClass["edit"]
@@ -163,6 +172,10 @@ func TestRunTurnBenchAggregation(t *testing.T) {
if refactor.Tasks != 1 || refactor.Verified != 1 || refactor.Passed != 1 || refactor.LatencyOnly != 0 {
t.Fatalf("refactor class = %+v", refactor)
}
longproc := result.PerClass["longproc"]
if longproc.Tasks != 1 || longproc.Verified != 0 || longproc.Passed != 0 || longproc.LatencyOnly != 1 {
t.Fatalf("longproc class = %+v", longproc)
}
}
// TestRunTurnBenchLatencyOnlyNeverPassed asserts the honesty gate: a task with
@@ -424,17 +437,31 @@ func TestCopyFixtureIsolatesSourceFromMutation(t *testing.T) {
t.Fatalf("read source main.go: %v", err)
}
copyDir, err := copyFixture(src)
copyDir, parent, err := copyFixture(src)
if err != nil {
t.Fatalf("copyFixture: %v", err)
}
defer os.RemoveAll(copyDir)
defer os.RemoveAll(parent) // cleans copyDir too, since copyDir lives under parent
// The copy must be a real copy, not a symlink/alias of the source, and
// mutating it must not touch the source.
if copyDir == src {
t.Fatalf("copyFixture returned the source dir, not a copy: %s", copyDir)
}
// The copy must be a grandchild of the system temp root (parent is a direct
// child of Temp, copyDir is a direct child of parent). Go ignores go.mod in
// direct children of Temp (a hijack guard), so this nesting is what lets the
// compiler-backed oracles actually run in the copy. Compare via filepath.Clean
// because os.TempDir() may carry a trailing slash (macOS $TMPDIR ends in
// "/T/") while filepath.Dir strips it; the raw != would false-fail on macOS
// even though the parent is genuinely a direct child.
tempRoot := filepath.Clean(os.TempDir())
if filepath.Dir(parent) != tempRoot {
t.Fatalf("parent %q is not a direct child of temp root %q", parent, tempRoot)
}
if filepath.Dir(copyDir) != parent {
t.Fatalf("copyDir %q is not nested under parent %q", copyDir, parent)
}
if err := os.WriteFile(filepath.Join(copyDir, "main.go"), []byte("package main\n\n// mutated\nfunc main() {}\n"), 0o644); err != nil {
t.Fatalf("mutate copy: %v", err)
}
@@ -453,3 +480,778 @@ func TestCopyFixtureIsolatesSourceFromMutation(t *testing.T) {
t.Fatalf("file added to copy appeared in source: %v", err)
}
}
// TestRunTurnBenchBuildOnlyMechanism exercises the build-only tier code path in
// isolation. The v3 baseline manifest leaves this tier empty (refactor graduated
// to correctness), but the mechanism is still supported for future use, so a
// focused test keeps it covered: a class listed in buildOnlyClasses with a
// verificationCommand counts in buildCheckedTasks/buildPassedTasks — never in
// tasksVerified/correctnessPassRate — and a build-pass cannot leak into the
// correctness number.
func TestRunTurnBenchBuildOnlyMechanism(t *testing.T) {
set := TaskSet{
ID: "build-suite",
Tasks: []BenchTask{
{ID: "b1", Class: "buildcheck", Prompt: "p", VerificationCommand: []string{"true"}},
},
BuildOnlyClasses: []string{"buildcheck"},
}
canned := map[string]*trace.TurnTrace{"b1": cannedTrace(120, 30, 900)}
result, err := RunTurnBench(context.Background(), set, TurnBenchConfig{
Model: "fake-model",
Runner: fakeTurnRunner(canned),
Now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
})
if err != nil {
t.Fatalf("RunTurnBench: %v", err)
}
if result.BuildCheckedTasks != 1 || result.BuildPassedTasks != 1 {
t.Fatalf("build checked=%d passed=%d, want 1/1", result.BuildCheckedTasks, result.BuildPassedTasks)
}
if result.BuildPassRate != 1.0 {
t.Fatalf("buildPassRate=%v, want 1.0", result.BuildPassRate)
}
// A build-only task must NOT count toward correctness.
if result.TasksVerified != 0 || result.TasksPassed != 0 {
t.Fatalf("build-only leaked into correctness: verified=%d passed=%d, want 0/0",
result.TasksVerified, result.TasksPassed)
}
if result.CorrectnessPassRate != 0 {
t.Fatalf("correctnessPassRate=%v, want 0 with only build-only tasks", result.CorrectnessPassRate)
}
if len(result.BuildOnlyClasses) != 1 || result.BuildOnlyClasses[0] != "buildcheck" {
t.Fatalf("buildOnlyClasses=%v, want [buildcheck]", result.BuildOnlyClasses)
}
if len(result.CorrectnessClasses) != 0 {
t.Fatalf("correctnessClasses=%v, want empty", result.CorrectnessClasses)
}
bc := result.PerClass["buildcheck"]
if bc.Tasks != 1 || bc.Verified != 1 || bc.Passed != 1 || bc.LatencyOnly != 0 {
t.Fatalf("buildcheck class = %+v, want 1/1 verified passed", bc)
}
}
// TestTurnSchemaVersion pins the schema bump so a shape change is a conscious
// decision. v3 recorded the tier reclassification (refactor structural-positive
// and nav answer-oracles moved into correctnessPassRate). v4 adds tasksErrored:
// tasks whose every iteration died before the agent produced a run are now
// first-class in the report instead of visible only in warnings, so a
// spawn-broken run cannot print a clean-looking summary or exit 0. v5 is
// additive only: execProfile stamps the execution profile the run was
// benchmarked under so profile A/B reports are self-describing.
func TestTurnSchemaVersion(t *testing.T) {
if TurnSchemaVersion != 5 {
t.Fatalf("TurnSchemaVersion = %d, want 5", TurnSchemaVersion)
}
}
// TestStreamJSONFinalTextExtractsAnswer, TestStreamJSONFinalTextEmptyWhenNoFinal,
// and TestStreamJSONFinalTextLastWins cover the pure capture helper that the
// nav answer-oracle depends on: it scans stream-json for the terminal "final"
// event and returns its text, "" when none was emitted, and the last when
// multiple appear (the success path emits exactly one, but the incomplete path
// can also emit one, so last-wins matches streamJSONExitCode's tie-break).
func TestStreamJSONFinalTextExtractsAnswer(t *testing.T) {
out := []byte(`{"type":"text","text":"thinking..."}
{"type":"final","text":"the keys are port, name, and retries"}
{"type":"run_end","exitCode":0}
`)
if got := streamJSONFinalText(out); got != "the keys are port, name, and retries" {
t.Fatalf("streamJSONFinalText = %q, want the final text", got)
}
}
func TestStreamJSONFinalTextEmptyWhenNoFinal(t *testing.T) {
out := []byte(`{"type":"text","text":"hi"}
{"type":"run_end","exitCode":0}
`)
if got := streamJSONFinalText(out); got != "" {
t.Fatalf("streamJSONFinalText = %q, want empty when no final event", got)
}
}
func TestStreamJSONFinalTextLastWins(t *testing.T) {
out := []byte(`{"type":"final","text":"first"}
{"type":"final","text":"second"}
`)
if got := streamJSONFinalText(out); got != "second" {
t.Fatalf("streamJSONFinalText = %q, want second (last final wins)", got)
}
}
// loadBaselineTask returns the named task from the checked-in baseline manifest.
// The stub-binary oracle tests below run the REAL manifest oracles against the
// REAL fixtures through the REAL NewTurnExecRunner, so a manifest edit that
// breaks an oracle is caught here rather than only in production.
func loadBaselineTask(t *testing.T, id string) BenchTask {
t.Helper()
set, err := LoadTaskSet(filepath.Join("manifests", "baseline.json"))
if err != nil {
t.Fatalf("LoadTaskSet: %v", err)
}
for _, task := range set.Tasks {
if task.ID == id {
return task
}
}
t.Fatalf("manifest has no task %q", id)
return BenchTask{}
}
// runTurnStub runs one manifest task through the production NewTurnExecRunner
// with a stub "zero" binary whose body is a POSIX sh script (writeExecStub
// skips on Windows). The stub is invoked with cmd.Dir set to the fixture copy,
// so it can both emit canned stream-json AND mutate the copy (apply or omit the
// fix) before the runner stamps the oracle and runs the verification command.
func runTurnStub(t *testing.T, task BenchTask, stubBody string) TurnTaskOutcome {
t.Helper()
stub := writeExecStub(t, stubBody)
return NewTurnExecRunner(stub)(context.Background(), task, RunContext{Model: "fake-model"})
}
// assertVerifyFailed asserts an outcome failed specifically because the oracle
// rejected the work — Passed is false, there is no harness error (Err nil), and
// VerifyErr carries the surfaced failure detail. This is stronger than merely
// checking Passed=false: it distinguishes a genuine oracle rejection from a
// harness error (Err set) or a silent non-pass (Passed=false with no VerifyErr,
// which would mean the failure never went through verification at all). The
// gating stubs all emit a clean run_end (exitCode 0), so a non-empty VerifyErr
// here can only come from the verification command failing.
func assertVerifyFailed(t *testing.T, label string, outcome TurnTaskOutcome) {
t.Helper()
if outcome.Err != nil {
t.Fatalf("%s: want a verify fail, got harness error: %v", label, outcome.Err)
}
if outcome.Passed {
t.Fatalf("%s: want verify fail, got Passed=true", label)
}
if strings.TrimSpace(outcome.VerifyErr) == "" {
t.Fatalf("%s: want verify fail, but VerifyErr is empty (failure did not come from verification): %+v", label, outcome)
}
}
// --- Gating tests: the oracle FAILS the wrong thing (no-op / wrong answer) ---
// TestStampedOracleRejectsNoOpRefactor is the core #712 fix: refactor used to
// live in the build tier where a no-op `go build ./...` passed. Now a no-op
// agent (the stub emits a clean run_end but touches nothing) leaves the fixture
// with no formatGreeting helper, so the stamped `var _ = formatGreeting` fails
// to compile and `go test ./...` fails — the task is failed, not passed.
func TestStampedOracleRejectsNoOpRefactor(t *testing.T) {
task := loadBaselineTask(t, "refactor-01")
outcome := runTurnStub(t, task, `echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "no-op refactor", outcome)
}
// TestStampedOracleRejectsMissingField proves edit-03's oracle is structural: a
// no-op leaves Config with no Label field, so `var _ string = Config{}.Label`
// fails to compile and the task fails.
func TestStampedOracleRejectsMissingField(t *testing.T) {
task := loadBaselineTask(t, "edit-03")
outcome := runTurnStub(t, task, `echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "missing-field edit-03", outcome)
}
// TestNavAnswerOracleRejectsWrongAnswer proves the nav oracle greps the CAPTURED
// answer, not the raw stream: the stub emits a final answer missing the
// required keys, the harness writes it to .zero-answer.txt, and the compound
// grep fails — so a plausible-but-wrong answer cannot pass nav-09.
func TestNavAnswerOracleRejectsWrongAnswer(t *testing.T) {
task := loadBaselineTask(t, "nav-09")
outcome := runTurnStub(t, task, `echo '{"type":"final","text":"the keys are foo and bar"}'
echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "wrong nav-09 answer", outcome)
}
// TestNavNoFinalTextFails proves a run that produced no answer fails nav: with
// no "final" event, .zero-answer.txt is empty and the grep finds nothing.
func TestNavNoFinalTextFails(t *testing.T) {
task := loadBaselineTask(t, "nav-09")
outcome := runTurnStub(t, task, `echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-09 no final", outcome)
}
// TestEdit05RejectsRewordedDebugPrint proves edit-05's oracle catches a reword,
// not just a deletion: changing `fmt.Println("debug: starting")` to
// `fmt.Println("starting up")` leaves a fmt.Println string-literal call, so the
// stamped oracle's `strings.Contains(main.go, fmt.Println(")` check fails. A
// plain `! grep 'debug: starting'` would have rubber-stamped this reword.
func TestEdit05RejectsRewordedDebugPrint(t *testing.T) {
task := loadBaselineTask(t, "edit-05")
outcome := runTurnStub(t, task, `sed 's/debug: starting/starting up/' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "reworded edit-05 debug print", outcome)
}
// TestEdit05RejectsRespelledDebugPrint proves the second edit-05 check (the
// message text "debug: starting" must be absent) catches a respell onto a
// different call form that the fmt.Println(" string-literal check misses: the
// agent rewrites the line to fmt.Printf("debug: starting\n"), which still prints
// the message but is no longer a fmt.Println string-literal. The
// strings.Contains("debug: starting") assertion fails the task.
func TestEdit05RejectsRespelledDebugPrint(t *testing.T) {
task := loadBaselineTask(t, "edit-05")
outcome := runTurnStub(t, task, `sed 's/fmt.Println("debug: starting")/fmt.Printf("debug: starting\\n")/' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "respelled edit-05 debug print", outcome)
}
// TestRefactor05RejectsStubbedCaller proves refactor-05's behavioral oracle
// catches a wrong edit the negative grep alone would pass: the agent deletes
// Wrapper (so ! grep 'func Wrapper' succeeds) but stubs greetWrapped to return
// "" instead of inlining. The stamped TestGreetWrapped runs greetWrapped("x"),
// gets "" not "hello, x", and fails — so deleting the target along with its
// caller (or stubbing the caller) can no longer pass the oracle.
func TestRefactor05RejectsStubbedCaller(t *testing.T) {
task := loadBaselineTask(t, "refactor-05")
outcome := runTurnStub(t, task, `sed -e 's/return Wrapper(name)/return ""/' -e '/^func Wrapper(name string) string {/,/^}/d' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "stubbed refactor-05 caller", outcome)
}
// TestRefactor04RejectsBareMapKept proves refactor-04's file-reading oracle
// catches a no-op that the old exact-text shell grep would miss: the agent adds
// an unused `type Stats map[string]int` (so the compile-time `var _ Stats`
// passes) but keeps the bare `var stats = map[string]int{}` under the original
// spacing. The spacing-tolerant regex in TestStatsNamedType still matches the
// bare-map declaration, so the task fails — Record/Lookup never moved onto the
// named type.
func TestRefactor04RejectsBareMapKept(t *testing.T) {
task := loadBaselineTask(t, "refactor-04")
outcome := runTurnStub(t, task, `sed 's/type Config struct {/type Stats map[string]int\n\ntype Config struct {/' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "refactor-04 bare-map kept", outcome)
}
// TestEdit01IgnoresDocComment proves edit-01's scoped negative grep does NOT
// false-fail a correct rename that leaves the doc comment mentioning the old
// name. The stub renames the declaration (`MaxRetries =` -> `RetryLimit =`) but
// leaves `// MaxRetries is the maximum...`; the scoped `! grep -RIn 'MaxRetries ='`
// matches declaration sites only, so the comment is ignored, `var _ = RetryLimit`
// compiles, and the task passes.
func TestEdit01IgnoresDocComment(t *testing.T) {
task := loadBaselineTask(t, "edit-01")
outcome := runTurnStub(t, task, `sed 's/const MaxRetries = 3/const RetryLimit = 3/' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct rename with kept doc comment should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("edit-01 oracle must not false-fail a correct rename that leaves the doc comment: %+v", outcome)
}
}
// --- Satisfiable tests: the oracle PASSES the right thing (real fix applied) ---
// TestStampedOraclePassesWhenRefactorHappened proves the refactor-01 oracle is
// not an always-fail gate: when the stub applies the real refactor (extract
// formatGreeting, call it from both callers), `var _ = formatGreeting` compiles
// and the `hello, %s` grep-count is 1 (only the helper holds the literal), so
// the task passes.
func TestStampedOraclePassesWhenRefactorHappened(t *testing.T) {
task := loadBaselineTask(t, "refactor-01")
outcome := runTurnStub(t, task, `sed -e 's/return fmt.Sprintf("hello, %s", c.Name)/return formatGreeting(c.Name)/' -e 's/return fmt.Sprintf("hello, %s", name)/return formatGreeting(name)/' main.go > .zero-tmp
cat >> .zero-tmp <<'EOF'
func formatGreeting(name string) string {
return fmt.Sprintf("hello, %s", name)
}
EOF
mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("real refactor should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("refactor-01 oracle must pass when formatGreeting is extracted: %+v", outcome)
}
}
// TestEdit03PassesWhenFieldAdded proves edit-03's oracle passes when the field
// is actually added: Config gets a Label string field, so `var _ string =
// Config{}.Label` compiles and the task passes.
func TestEdit03PassesWhenFieldAdded(t *testing.T) {
task := loadBaselineTask(t, "edit-03")
outcome := runTurnStub(t, task, `awk '/Name string/ && !d {print; print "Label string"; d=1; next} {print}' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("real field-add should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("edit-03 oracle must pass when Label is added: %+v", outcome)
}
}
// TestNav09PassesWithCorrectAnswer proves the nav-09 oracle passes when the
// captured answer names all three config keys.
func TestNav09PassesWithCorrectAnswer(t *testing.T) {
task := loadBaselineTask(t, "nav-09")
outcome := runTurnStub(t, task, `echo '{"type":"final","text":"config.json has three keys: port, name, and retries."}'
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct nav answer should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("nav-09 oracle must pass when the answer names port/name/retries: %+v", outcome)
}
}
// TestRefactor05PassesWhenInlined proves refactor-05's oracle is not an
// always-fail gate: when the stub applies the real inline (greetWrapped calls
// GreetByName directly, Wrapper removed), TestGreetWrapped gets "hello, x" and
// the ! grep 'func Wrapper' check passes, so the task passes.
func TestRefactor05PassesWhenInlined(t *testing.T) {
task := loadBaselineTask(t, "refactor-05")
outcome := runTurnStub(t, task, `sed -e 's/return Wrapper(name)/return GreetByName(name)/' -e '/^func Wrapper(name string) string {/,/^}/d' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("real inline should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("refactor-05 oracle must pass when Wrapper is inlined into greetWrapped: %+v", outcome)
}
}
// TestRefactor04PassesWhenTyped proves refactor-04's oracle passes when the
// bare map is actually replaced by the named type: the stub introduces
// `type Stats map[string]int` and retypes the var to `var stats = Stats{}`,
// so the bare-map regex no longer matches, Stats is referenced, `var _ Stats`
// compiles, and go test passes.
func TestRefactor04PassesWhenTyped(t *testing.T) {
task := loadBaselineTask(t, "refactor-04")
outcome := runTurnStub(t, task, `sed -e 's/type Config struct {/type Stats map[string]int\n\ntype Config struct {/' -e 's/var stats = map\[string\]int{}/var stats = Stats{}/' main.go > .zero-tmp && mv .zero-tmp main.go
echo '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("real typed refactor should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("refactor-04 oracle must pass when stats is typed as Stats: %+v", outcome)
}
}
// TestStampOracleRejectsFixturelessOracleTask covers the fail-closed path in
// stampOracleAndAnswer: a task that carries an oracle (a stamped OracleTest or a
// verification command) but has no workspace fixture has nowhere safe to stamp,
// so it must error rather than let runVerification run in the caller's cwd with
// the oracle never compiled (or the answer never captured). A fixtureless task
// with no oracle and no verification (a pure latency-only run in the caller's
// cwd) is left alone. This is a direct unit test (no stub), so it runs on all
// platforms.
func TestStampOracleRejectsFixturelessOracleTask(t *testing.T) {
// OracleTest set, no verificationCommand: isolates the `OracleTest != ""`
// reject branch so a regression that only checked VerificationCommand would
// fail here (the both-set case below would still pass it).
withOracleTest := BenchTask{
ID: "t",
OracleTest: "package x\n\nvar _ = Missing\n",
}
if err := stampOracleAndAnswer(withOracleTest, nil); err == nil {
t.Fatal("fixtureless task with an OracleTest must be rejected, got nil error")
}
// verificationCommand set, no OracleTest: isolates the other reject branch.
withVerify := BenchTask{
ID: "t",
VerificationCommand: []string{"bash", "-c", "grep x .zero-answer.txt"},
}
if err := stampOracleAndAnswer(withVerify, nil); err == nil {
t.Fatal("fixtureless task with a verificationCommand must be rejected, got nil error")
}
// Both set: the `||` rejects, and this guards against a regression that
// inverted the logic to `&&`.
withBoth := BenchTask{
ID: "t",
OracleTest: "package x\n\nvar _ = Missing\n",
VerificationCommand: []string{"go", "test", "./..."},
}
if err := stampOracleAndAnswer(withBoth, nil); err == nil {
t.Fatal("fixtureless task with both an OracleTest and a verificationCommand must be rejected, got nil error")
}
latencyOnly := BenchTask{ID: "t"}
if err := stampOracleAndAnswer(latencyOnly, nil); err != nil {
t.Fatalf("fixtureless latency-only task must not be rejected, got: %v", err)
}
}
// --- Count-oracle tests: anchored `^count: N$` with the exact expected count ---
// TestNav01CountOracleAcceptsExactCount proves nav-01's anchored count oracle
// passes when the answer names the files AND states the exact count (5 — the
// fixture has README.md, config.json, go.mod, main.go, main_test.go) on its own
// line. printf emits the JSON with a literal \n escape so the captured answer
// has "count: 5" on its own line.
func TestNav01CountOracleAcceptsExactCount(t *testing.T) {
task := loadBaselineTask(t, "nav-01")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"main.go, config.json, go.mod, README.md, main_test.go\ncount: 5"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct nav-01 answer should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("nav-01 oracle must pass when the answer names the files and states count: 5: %+v", outcome)
}
}
// TestNav01CountOracleRejectsWrongCount proves the anchored count oracle rejects
// a wrong count: the file names are all present, but "count: 3" does not match
// `^count: 5$`, so the task fails at verification.
func TestNav01CountOracleRejectsWrongCount(t *testing.T) {
task := loadBaselineTask(t, "nav-01")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"main.go, config.json, go.mod, README.md, main_test.go\ncount: 3"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-01 wrong count", outcome)
}
// TestNav04CountOracleRejectsSubstring proves the line anchor rejects a
// substring: "the count: 1 is the number" contains "count: 1" (the right
// value) but the line does not start with "count:", so `^count:[[:space:]]*1$`
// does not match. A pre-anchor oracle (`count:\s*1`) would have rubber-stamped
// this.
func TestNav04CountOracleRejectsSubstring(t *testing.T) {
task := loadBaselineTask(t, "nav-04")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"the count: 1 is the number of test functions"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-04 substring count", outcome)
}
// TestNav04CountOracleAcceptsExactCount proves the nav-04 oracle passes for a
// real answer: the count is anchored on its own line AND the answer names the
// one Test* function (TestGreet, from main_test.go). The fixture's ground truth
// is exactly one test function, so count: 1 is right, and naming TestGreet is
// the inspection-proof that the agent actually read main_test.go.
func TestNav04CountOracleAcceptsExactCount(t *testing.T) {
task := loadBaselineTask(t, "nav-04")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"There is one test function: TestGreet in main_test.go.\ncount: 1"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct nav-04 answer should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("nav-04 oracle must pass when the answer names TestGreet and states count: 1 on its own line: %+v", outcome)
}
}
// TestNav04CountOracleRejectsZeroGuess is the count=0 gameability gate: an agent
// that never opens the workspace and always emits "count: 0" used to pass nav-04
// when the ground truth was 0. The fixture now has one real Test* function, so
// the ground truth is 1 and a clean "count: 0" fails.
func TestNav04CountOracleRejectsZeroGuess(t *testing.T) {
task := loadBaselineTask(t, "nav-04")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 0"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-04 zero-guess", outcome)
}
// TestNav04CountOracleRejectsBlindCountOne is the count=1 gameability gate: once
// the ground truth became 1, an agent that never opens the workspace and always
// emits "count: 1" would pass a count-only oracle. nav-04's oracle also requires
// the answer to name the test function (TestGreet), so a bare "count: 1" with no
// named fact fails — proving the blind-count-one surface is closed.
func TestNav04CountOracleRejectsBlindCountOne(t *testing.T) {
task := loadBaselineTask(t, "nav-04")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 1"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-04 blind-count-one", outcome)
}
// TestNav05CountOracleAcceptsExactCount proves nav-05's oracle passes for a real
// answer: the count is anchored on its own line AND the answer names the file
// holding the one TODO (main.go). Naming main.go is the inspection-proof that the
// agent actually located the TODO rather than guessing the count.
func TestNav05CountOracleAcceptsExactCount(t *testing.T) {
task := loadBaselineTask(t, "nav-05")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"One TODO in main.go.\ncount: 1"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct nav-05 answer should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("nav-05 oracle must pass when the answer names main.go and states count: 1: %+v", outcome)
}
}
// TestNav05CountOracleRejectsZeroGuess is nav-05's count=0 gameability gate: the
// fixture now has one real TODO, so a clean "count: 0" (the always-guess-zero
// answer) fails.
func TestNav05CountOracleRejectsZeroGuess(t *testing.T) {
task := loadBaselineTask(t, "nav-05")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 0"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-05 zero-guess", outcome)
}
// TestNav05CountOracleRejectsBlindCountOne is nav-05's count=1 gameability gate:
// once the ground truth became 1, an agent that never opens the workspace and
// always emits "count: 1" would pass a count-only oracle. nav-05's oracle also
// requires the answer to name the file (main.go), so a bare "count: 1" with no
// named file fails — proving the blind-count-one surface is closed.
func TestNav05CountOracleRejectsBlindCountOne(t *testing.T) {
task := loadBaselineTask(t, "nav-05")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 1"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-05 blind-count-one", outcome)
}
// TestNav05CountOracleRejectsSubstring proves the line anchor still bites even
// when the named fact is present: "the count: 1 is the number of TODOs in
// main.go" names main.go and contains "count: 1" (the right value), but the
// count does not stand on its own line, so `^count:[[:space:]]*1$` does not
// match and the oracle fails. A pre-anchor oracle (`count:\s*1`) would have
// rubber-stamped this. Mirrors TestNav04CountOracleRejectsSubstring.
func TestNav05CountOracleRejectsSubstring(t *testing.T) {
task := loadBaselineTask(t, "nav-05")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"the count: 1 is the number of TODOs in main.go"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-05 substring count", outcome)
}
// TestNav08CountOracleAcceptsStdlibAnswer proves nav-08's oracle passes when the
// agent enumerated the imports across BOTH .go files and named the stdlib ones
// (fmt from main.go, testing from main_test.go): the answer states the only
// imports are stdlib so there are 0 third-party, with "count: 0" on its own
// line. The fmt+testing requirement is the inspection-proof — it forces the
// agent to have read all the files rather than blindly emitting "count: 0".
func TestNav08CountOracleAcceptsStdlibAnswer(t *testing.T) {
task := loadBaselineTask(t, "nav-08")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"The only imports are fmt (main.go) and testing (main_test.go), both standard library, so there are no third-party imports.\ncount: 0"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
if outcome.Err != nil {
t.Fatalf("correct nav-08 answer should pass, got harness error: %v", outcome.Err)
}
if !outcome.Passed {
t.Fatalf("nav-08 oracle must pass when the answer names fmt+testing and states count: 0: %+v", outcome)
}
}
// TestNav08CountOracleRejectsAnswerWithoutFmt proves the inspection-proof bites:
// an answer that states the right count (0) but never names the stdlib imports it
// examined ("No third-party imports. count: 0") fails, so an agent that didn't
// actually look at the imports can't pass by guessing the count alone.
func TestNav08CountOracleRejectsAnswerWithoutFmt(t *testing.T) {
task := loadBaselineTask(t, "nav-08")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"No third-party imports.\ncount: 0"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-08 no-fmt", outcome)
}
// TestNav08CountOracleRejectsFmtOnlyNoTesting proves the tighter inspection-proof:
// an answer that names fmt (from main.go) but misses testing (from main_test.go)
// fails, so an agent that only read main.go — or a guesser that names the one
// most-common stdlib import blind — can't pass. The agent must enumerate every
// file's imports, which is what the prompt asks for.
func TestNav08CountOracleRejectsFmtOnlyNoTesting(t *testing.T) {
task := loadBaselineTask(t, "nav-08")
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"The only import is fmt, which is standard library. count: 0"}'
printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-08 fmt-only-no-testing", outcome)
}
func TestResolveBinaryAbsolutizesExplicitPath(t *testing.T) {
dir := t.TempDir()
name := "zero-probe.exe"
if err := os.WriteFile(filepath.Join(dir, name), []byte("x"), 0o755); err != nil {
t.Fatal(err)
}
cwd, err := os.Getwd()
if err != nil {
t.Fatal(err)
}
if err := os.Chdir(dir); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = os.Chdir(cwd) })
resolved, err := ResolveBinary("./" + name)
if err != nil {
t.Fatalf("ResolveBinary: %v", err)
}
if !filepath.IsAbs(resolved) {
t.Fatalf("ResolveBinary returned relative path %q; the turn runner sets cmd.Dir per task, so a relative binary fails to spawn from every fixture copy", resolved)
}
}
func TestRunTurnBenchCountsErroredTasks(t *testing.T) {
set := TaskSet{
ID: "errored-suite",
Tasks: []BenchTask{
{ID: "t1", Class: "nav", Prompt: "p1", VerificationCommand: []string{"true"}},
{ID: "t2", Class: "longproc", Prompt: "p2"},
},
}
cfg := TurnBenchConfig{
Model: "fake-model",
Iterations: 1,
Runner: func(context.Context, BenchTask, RunContext) TurnTaskOutcome {
return TurnTaskOutcome{Err: errors.New("fork/exec ./zero: file does not exist")}
},
Now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
}
result, err := RunTurnBench(context.Background(), set, cfg)
if err != nil {
t.Fatalf("RunTurnBench: %v", err)
}
if result.TasksErrored != 2 {
t.Fatalf("TasksErrored=%d, want 2 (every iteration of both tasks errored)", result.TasksErrored)
}
if len(result.Warnings) != 2 {
t.Fatalf("warnings=%d, want 2 run warnings", len(result.Warnings))
}
summary := FormatTurnBenchSummary(result)
if !strings.Contains(summary, "ERRORED: 2 task(s)") {
t.Fatalf("summary does not surface the errored tasks:\n%s", summary)
}
if !strings.Contains(summary, "fork/exec ./zero") {
t.Fatalf("summary does not echo the underlying run error:\n%s", summary)
}
}
func TestRunTurnBenchPartialErrorStillCounts(t *testing.T) {
set := TaskSet{
ID: "partial-suite",
Tasks: []BenchTask{
{ID: "ok", Class: "nav", Prompt: "p", VerificationCommand: []string{"true"}},
{ID: "dead", Class: "nav", Prompt: "p", VerificationCommand: []string{"true"}},
},
}
canned := map[string]*trace.TurnTrace{"ok": cannedTrace(100, 10, 1000)}
inner := fakeTurnRunner(canned)
cfg := TurnBenchConfig{
Model: "fake-model",
Iterations: 1,
Runner: func(ctx context.Context, task BenchTask, rc RunContext) TurnTaskOutcome {
if task.ID == "dead" {
return TurnTaskOutcome{Err: errors.New("spawn failed")}
}
return inner(ctx, task, rc)
},
Now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
}
result, err := RunTurnBench(context.Background(), set, cfg)
if err != nil {
t.Fatalf("RunTurnBench: %v", err)
}
if result.TasksErrored != 1 {
t.Fatalf("TasksErrored=%d, want 1", result.TasksErrored)
}
if result.TasksAttempted != 2 {
t.Fatalf("TasksAttempted=%d, want 2", result.TasksAttempted)
}
}
// The exec-profile passthrough: the profile must reach every task's zero exec
// invocation as --exec-profile (with the prompt staying last) and stay out of
// the args entirely when unset.
func TestBuildTurnExecArgsIncludesExecProfile(t *testing.T) {
task := BenchTask{ID: "t", Prompt: "do the thing"}
args := buildTurnExecArgs(task, RunContext{Model: "m", ExecProfile: "fast"}, "trace.ndjson", nil)
found := false
for i := 0; i < len(args)-1; i++ {
if args[i] == "--exec-profile" && args[i+1] == "fast" {
found = true
break
}
}
if !found {
t.Fatalf("args must carry --exec-profile fast, got %v", args)
}
if args[len(args)-1] != "do the thing" {
t.Fatalf("prompt must stay the last argument, got %v", args)
}
args = buildTurnExecArgs(task, RunContext{Model: "m"}, "trace.ndjson", nil)
for _, arg := range args {
if arg == "--exec-profile" {
t.Fatalf("no profile configured, but args carry --exec-profile: %v", args)
}
}
}
// The configured profile must reach the runner's RunContext and be stamped
// into the result, so a profile A/B report is self-describing. The boundary
// canonicalizes (case/whitespace) so equivalent postures always carry the same
// label, and rejects unknown names so a direct library caller cannot slip an
// unvalidated profile into a report the way the CLI (parse-time check) cannot.
func TestRunTurnBenchStampsExecProfile(t *testing.T) {
set := TaskSet{
ID: "profile-suite",
Tasks: []BenchTask{{ID: "t1", Class: "longproc", Prompt: "p"}},
}
var gotProfile string
cfg := TurnBenchConfig{
Model: "fake-model",
ExecProfile: " FAST ",
Iterations: 1,
Runner: func(_ context.Context, _ BenchTask, rc RunContext) TurnTaskOutcome {
gotProfile = rc.ExecProfile
return TurnTaskOutcome{Passed: true, WallMs: 10, Trace: cannedTrace(10, 1, 100)}
},
Now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
}
result, err := RunTurnBench(context.Background(), set, cfg)
if err != nil {
t.Fatalf("RunTurnBench: %v", err)
}
if gotProfile != "fast" {
t.Fatalf("runner RunContext.ExecProfile = %q, want the canonical fast", gotProfile)
}
if result.ExecProfile != "fast" {
t.Fatalf("result.ExecProfile = %q, want the canonical fast", result.ExecProfile)
}
raw, err := json.Marshal(result)
if err != nil {
t.Fatalf("marshal result: %v", err)
}
if !strings.Contains(string(raw), `"execProfile":"fast"`) {
t.Fatal("published JSON must carry execProfile")
}
}
func TestRunTurnBenchRejectsUnknownExecProfile(t *testing.T) {
set := TaskSet{
ID: "profile-suite",
Tasks: []BenchTask{{ID: "t1", Class: "longproc", Prompt: "p"}},
}
ran := false
cfg := TurnBenchConfig{
Model: "fake-model",
ExecProfile: "blanced",
Iterations: 1,
Runner: func(context.Context, BenchTask, RunContext) TurnTaskOutcome {
ran = true
return TurnTaskOutcome{Passed: true, WallMs: 10, Trace: cannedTrace(10, 1, 100)}
},
}
_, err := RunTurnBench(context.Background(), set, cfg)
if err == nil || !strings.Contains(err.Error(), "unknown execution profile") {
t.Fatalf("err = %v, want an unknown-profile rejection", err)
}
if ran {
t.Fatal("nothing may run under an unknown profile")
}
}
+39 -7
View File
@@ -34,7 +34,7 @@ const (
var ErrUnknownProvider = errors.New("unknown provider")
// AIMLAPIID is the provider catalog identifier for the AI/ML API preset.
// AIMLAPIID is the provider catalog identifier for the aimlapi.com preset.
const AIMLAPIID = "aimlapi"
type Descriptor struct {
@@ -44,6 +44,7 @@ type Descriptor struct {
DefaultBaseURL string
DefaultModel string
AuthEnvVars []string
CustomHeaders map[string]string
RequiresAuth bool
UsesAmbientAuth bool
Public bool
@@ -70,9 +71,9 @@ type Descriptor struct {
// headless / SSH use) in addition to the browser flow.
OAuthDeviceFlow bool
// Recommended marks the provider that should be surfaced first and badged
// (★ … (recommended)) in every catalog-ordered list and picker. At most one
// descriptor should set this.
// Recommended marks a provider surfaced at the top and badged in
// catalog-ordered lists and pickers. The recommended descriptors are the first
// entries in the catalog, in order.
Recommended bool
}
@@ -103,6 +104,11 @@ var descriptors = []Descriptor{
// minimax, qwen, google, nvidia, tencent, z-ai). Flat /v1/chat/completions
// with a Bearer ogw_live_… key; listed first and badged in every picker.
recommended(openAICompat("gitlawb-opengateway", "GitLawb OpenGateway", "https://opengateway.gitlawb.com/v1", "mimo-v2.5-pro", []string{"GITLAWB_OPENGATEWAY_API_KEY"}, "gitlawb opengateway", "opengateway")),
// aimlapi.com — OpenAI-compatible aggregating gateway, also badged as
// recommended (second, right after the OpenGateway default). The built-in TUI
// onboarding can register/top up an account and save the issued key; the partner
// headers attribute usage for the rebate.
recommended(aimlapi()),
openAI("openai", "OpenAI", "https://api.openai.com/v1", "gpt-4.1", []string{"OPENAI_API_KEY"}),
anthropic("anthropic", "Anthropic", "https://api.anthropic.com", "claude-sonnet-4.5", []string{"ANTHROPIC_API_KEY"}),
google("google", "Google", "https://generativelanguage.googleapis.com", "gemini-2.5-pro", []string{"GEMINI_API_KEY", "GOOGLE_API_KEY"}, "gemini"),
@@ -110,7 +116,6 @@ var descriptors = []Descriptor{
localOpenAI("ollama", "Ollama Local", "http://localhost:11434/v1", "llama3.1", "ollama local"),
localOpenAI("lmstudio", "LM Studio", "http://localhost:1234/v1", "local-model", "lm-studio", "lm studio"),
oauthProvider(openAICompat("openrouter", "OpenRouter", "https://openrouter.ai/api/v1", "openai/gpt-4.1", []string{"OPENROUTER_API_KEY"}), true, false),
openAICompat(AIMLAPIID, "AI/ML API", "https://api.aimlapi.com/v1", "openai/gpt-5-chat", []string{"AIMLAPI_API_KEY"}, "aiml api", "ai/ml api"),
// Hugging Face Inference Providers — OpenAI-compatible router at
// https://router.huggingface.co/v1 exposes hundreds of OSS models. OAuth
// requires a one-time app registration at huggingface.co/settings/applications/new
@@ -302,8 +307,21 @@ func google(id string, name string, baseURL string, model string, env []string,
}
}
// recommended marks a descriptor as the recommended default so list/picker
// surfaces sort it first and render the ★ … (recommended) badge.
func aimlapi() Descriptor {
descriptor := openAICompat(AIMLAPIID, "aimlapi.com", "https://api.aimlapi.com/v1", "anthropic/claude-sonnet-5", []string{"AIMLAPI_API_KEY"}, "aimlapi", "aiml api", "ai/ml api", "ai ml api")
descriptor.CustomHeaders = map[string]string{
"X-AIMLAPI-Partner-ID": "part_62yQoGYDq4Yqnrj2R1iGrDNJ",
"X-AIMLAPI-Integration-Repo": "Gitlawb/zero",
"X-AIMLAPI-Integration-Version": "zero",
}
// Onboarding is key-based only: the TUI sub-flow takes an existing key or an
// email top-up (internal/aimlapi Paths A/B). Not flagged
// OAuth-capable, so it does not appear in the wizard's "Sign in with OAuth" list.
return descriptor
}
// recommended marks a descriptor as a recommended default so list/picker
// surfaces sort it to the top and render a provider-specific badge.
func recommended(descriptor Descriptor) Descriptor {
descriptor.Recommended = true
return descriptor
@@ -384,5 +402,19 @@ func cloneDescriptor(descriptor Descriptor) Descriptor {
descriptor.AuthEnvVars = append([]string{}, descriptor.AuthEnvVars...)
descriptor.SupportedAPIFormats = append([]APIFormat{}, descriptor.SupportedAPIFormats...)
descriptor.Aliases = append([]string{}, descriptor.Aliases...)
if descriptor.CustomHeaders != nil {
descriptor.CustomHeaders = copyStringMap(descriptor.CustomHeaders)
}
return descriptor
}
func copyStringMap(values map[string]string) map[string]string {
if values == nil {
return nil
}
copied := make(map[string]string, len(values))
for key, value := range values {
copied[key] = value
}
return copied
}
+23 -17
View File
@@ -9,6 +9,7 @@ import (
var expectedCatalogIDs = []string{
"gitlawb-opengateway",
"aimlapi",
"openai",
"anthropic",
"google",
@@ -16,7 +17,6 @@ var expectedCatalogIDs = []string{
"ollama",
"lmstudio",
"openrouter",
"aimlapi",
"huggingface",
"chatgpt",
"groq",
@@ -69,25 +69,31 @@ func TestAllHasStableUniqueIDs(t *testing.T) {
}
}
func TestRecommendedProviderIsFirstAndUnique(t *testing.T) {
func TestRecommendedProvidersAreTopOfCatalog(t *testing.T) {
descriptors := All()
if len(descriptors) == 0 {
t.Fatal("All() returned no descriptors")
// The recommended providers are badged and pinned to the top of the catalog,
// in this order (OpenGateway remains the default; aimlapi.com is also badged).
wantTop := []string{"gitlawb-opengateway", "aimlapi"}
if len(descriptors) < len(wantTop) {
t.Fatalf("All() returned %d descriptors, want at least %d", len(descriptors), len(wantTop))
}
if !descriptors[0].Recommended {
t.Fatalf("All()[0] = %q, want it to be the recommended provider", descriptors[0].ID)
}
if descriptors[0].ID != "gitlawb-opengateway" {
t.Fatalf("recommended provider = %q, want %q", descriptors[0].ID, "gitlawb-opengateway")
for index, id := range wantTop {
if descriptors[index].ID != id {
t.Fatalf("descriptors[%d] = %q, want %q", index, descriptors[index].ID, id)
}
if !descriptors[index].Recommended {
t.Fatalf("descriptors[%d] (%q) should be recommended", index, id)
}
}
// Exactly those are recommended, and they are contiguous at the top.
recommended := 0
for _, descriptor := range descriptors {
if descriptor.Recommended {
recommended++
}
}
if recommended != 1 {
t.Fatalf("recommended descriptor count = %d, want exactly 1", recommended)
if recommended != len(wantTop) {
t.Fatalf("recommended descriptor count = %d, want %d", recommended, len(wantTop))
}
}
@@ -109,14 +115,14 @@ func TestAIMLAPIDescriptor(t *testing.T) {
if err != nil {
t.Fatalf("Require(aimlapi) error = %v", err)
}
if descriptor.Name != "AI/ML API" {
t.Fatalf("Name = %q, want AI/ML API", descriptor.Name)
if descriptor.Name != "aimlapi.com" {
t.Fatalf("Name = %q, want aimlapi.com", descriptor.Name)
}
if descriptor.DefaultBaseURL != "https://api.aimlapi.com/v1" {
t.Fatalf("DefaultBaseURL = %q, want AI/ML API endpoint", descriptor.DefaultBaseURL)
t.Fatalf("DefaultBaseURL = %q, want aimlapi.com endpoint", descriptor.DefaultBaseURL)
}
if descriptor.DefaultModel != "openai/gpt-5-chat" {
t.Fatalf("DefaultModel = %q, want openai/gpt-5-chat", descriptor.DefaultModel)
if descriptor.DefaultModel != "anthropic/claude-sonnet-5" {
t.Fatalf("DefaultModel = %q, want anthropic/claude-sonnet-5", descriptor.DefaultModel)
}
if descriptor.Transport != TransportOpenAICompatible {
t.Fatalf("Transport = %q, want %q", descriptor.Transport, TransportOpenAICompatible)
@@ -316,7 +322,7 @@ func TestListByTransportPreservesCatalogOrder(t *testing.T) {
TransportBedrock: {"bedrock"},
TransportVertex: {"vertex"},
TransportAnthropicCompat: {"minimax", "minimaxi-cn", "opencode-go-anthropic-compatible", "custom-anthropic-compatible"},
TransportOpenAICompat: {"gitlawb-opengateway", "ollama-cloud", "ollama", "lmstudio", "openrouter", "aimlapi", "huggingface", "chatgpt", "groq", "deepseek", "together", "dashscope", "moonshot", "longcat", "nvidia-nim", "mistral", "github", "xai", "venice", "xiaomi-mimo", "bankr", "zai", "zai-cn", "kilocode", "opencode", "opencode-go", "atomic-chat", "chatgpt-proxy", "custom-openai-compatible"},
TransportOpenAICompat: {"gitlawb-opengateway", "aimlapi", "ollama-cloud", "ollama", "lmstudio", "openrouter", "huggingface", "chatgpt", "groq", "deepseek", "together", "dashscope", "moonshot", "longcat", "nvidia-nim", "mistral", "github", "xai", "venice", "xiaomi-mimo", "bankr", "zai", "zai-cn", "kilocode", "opencode", "opencode-go", "atomic-chat", "chatgpt-proxy", "custom-openai-compatible"},
}
for transport, wantIDs := range cases {
+11
View File
@@ -61,6 +61,17 @@ var zaiCuratedModels = []Model{
}
var curatedModels = map[string][]Model{
// aimlapi.com aggregates every upstream provider behind one OpenAI-compatible
// endpoint, so the curated defaults are a cross-provider "best of" flagship
// list. The live /v1/models discovery (when a key is present) replaces this;
// these are the offline/pre-key fallback and the picker's default order.
"aimlapi": {
{ID: "anthropic/claude-sonnet-5", Description: "catalog default (Anthropic Claude Sonnet 5)"},
{ID: "google/gemini-3.5-flash", Description: "Google Gemini Flash 3.5"},
{ID: "openai/gpt-5.5-2026-04-23", Description: "OpenAI GPT-5.5"},
{ID: "qwen/qwen-3.7-max", Description: "Qwen 3.7 Max"},
{ID: "deepseek/deepseek-v4-pro", Description: "DeepSeek V4"},
},
"ollama-cloud": {
{ID: "qwen3-coder:480b", Description: "catalog default"},
{ID: "gpt-oss:120b", Description: "agentic coding model"},
@@ -33,6 +33,11 @@ func TestModelsAreProviderScoped(t *testing.T) {
want: []string{"gpt-5.5", "gpt-5.4", "gpt-5.4-mini", "gpt-5.3-codex-spark"},
notWant: []string{"gpt-5", "gpt-4.1", "openai/gpt-4.1"},
},
{
provider: "aimlapi",
want: []string{"anthropic/claude-sonnet-5", "google/gemini-3.5-flash", "openai/gpt-5.5-2026-04-23", "qwen/qwen-3.7-max", "deepseek/deepseek-v4-pro"},
notWant: []string{"gpt-4o", "openai/gpt-4o", "anthropic/claude-opus-4.8", "x-ai/grok-4-5"},
},
{
provider: "mistral",
want: []string{"mistral-large-latest", "codestral-latest"},
+51
View File
@@ -108,6 +108,57 @@ func New(profile config.ProviderProfile, options Options) (zeroruntime.Provider,
}
}
// NewTurnSessionProvider builds the default TurnSessionProvider for a resolved
// profile: it constructs the provider exactly as New does, then wraps it with a
// ProviderCapabilities projection computed from the same model-registry entry
// New already resolves. The default session's Stream is the provider's
// StreamCompletion, so runtime behavior is identical to using New directly.
func NewTurnSessionProvider(profile config.ProviderProfile, options Options) (zeroruntime.TurnSessionProvider, error) {
provider, err := New(profile, options)
if err != nil {
return nil, err
}
caps, err := resolveCapabilities(profile, options)
if err != nil {
return nil, err
}
return zeroruntime.NewProviderTurnSessionProvider(provider, caps), nil
}
// resolveCapabilities projects the resolved profile's model-registry entry into
// the flat zeroruntime.ProviderCapabilities (zeroruntime cannot reference
// modelregistry types — modelregistry imports zeroruntime, so a typed field
// would form an import cycle). A model absent from the registry yields only the
// resolved API model id and max-output tokens; the rest stays zero (unknown).
func resolveCapabilities(profile config.ProviderProfile, options Options) (zeroruntime.ProviderCapabilities, error) {
resolved, err := resolveProfile(profile, options)
if err != nil {
return zeroruntime.ProviderCapabilities{}, err
}
caps := zeroruntime.ProviderCapabilities{
Model: resolved.apiModel,
MaxOutputTokens: resolved.maxOutputTokens,
}
registry, err := defaultRegistry(options.ModelRegistry)
if err != nil {
return zeroruntime.ProviderCapabilities{}, err
}
if entry, ok := registry.Get(strings.TrimSpace(profile.Model)); ok {
caps.ContextWindow = entry.ContextLimits.ContextWindow
caps.SupportsVision = entry.Supports(modelregistry.ModelCapabilityVision)
caps.SupportsReasoning = entry.Supports(modelregistry.ModelCapabilityReasoning)
caps.SupportsPromptCache = entry.Supports(modelregistry.ModelCapabilityPromptCache)
// Read efforts through Registry.ReasoningEfforts (not the raw entry) so
// the projection matches what the /effort picker and the run-time
// resolver advertise — including the name-based fallback for catalog
// entries that enumerate no efforts of their own.
for _, effort := range registry.ReasoningEfforts(entry.ID) {
caps.ReasoningEfforts = append(caps.ReasoningEfforts, string(effort))
}
}
return caps, nil
}
func parseThinkTagsForProfile(profile config.ProviderProfile, resolved resolvedProfile) bool {
if profile.ParseThinkTags != nil {
return *profile.ParseThinkTags
@@ -0,0 +1,194 @@
package providers
import (
"context"
"errors"
"testing"
"github.com/Gitlawb/zero/internal/config"
"github.com/Gitlawb/zero/internal/modelregistry"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
// TestNewTurnSessionProviderForEveryKind asserts every current provider kind
// wraps into the default TurnSessionProvider: construction succeeds, a session
// opens, and Compact reports unsupported (the default adapter contract).
func TestNewTurnSessionProviderForEveryKind(t *testing.T) {
kinds := []config.ProviderKind{
config.ProviderKindOpenAI,
config.ProviderKindOpenAICompatible,
config.ProviderKindAnthropic,
config.ProviderKindAnthropicCompat,
config.ProviderKindGoogle,
}
for _, kind := range kinds {
t.Run(string(kind), func(t *testing.T) {
tsp, err := NewTurnSessionProvider(config.ProviderProfile{
Name: "pr7-" + string(kind),
ProviderKind: kind,
BaseURL: "https://provider.example/v1",
APIKey: "sk-turn-session-test",
// A model absent from the registry: resolution falls back to the
// raw id, so this test stays hermetic across catalog changes.
Model: "pr7-unregistered-model",
}, Options{UserAgent: "zero-turn-session-test"})
if err != nil {
t.Fatalf("NewTurnSessionProvider(%s): %v", kind, err)
}
session, err := tsp.OpenTurnSession(context.Background())
if err != nil {
t.Fatalf("OpenTurnSession(%s): %v", kind, err)
}
defer func() {
if closeErr := session.Close(); closeErr != nil {
t.Fatalf("Close(%s): %v", kind, closeErr)
}
}()
if err := session.Prewarm(context.Background()); err != nil {
t.Fatalf("Prewarm(%s): %v", kind, err)
}
if _, compactErr := session.Compact(context.Background(), zeroruntime.CompletionRequest{}); !errors.Is(compactErr, zeroruntime.ErrCompactionUnsupported) {
t.Fatalf("Compact(%s) = %v, want ErrCompactionUnsupported", kind, compactErr)
}
caps := tsp.Capabilities()
if caps.Model != "pr7-unregistered-model" {
t.Fatalf("Capabilities(%s).Model = %q, want the resolved api model", kind, caps.Model)
}
if caps.NativeCompaction {
t.Fatalf("Capabilities(%s).NativeCompaction = true, want false for the default adapter", kind)
}
})
}
}
// TestNewTurnSessionProviderProjectsRegistryCapabilities asserts the factory
// projects the model-registry entry (context limits, capability flags,
// reasoning efforts) into the flat ProviderCapabilities.
func TestNewTurnSessionProviderProjectsRegistryCapabilities(t *testing.T) {
registry, err := modelregistry.NewRegistry([]modelregistry.ModelEntry{{
ID: "pr7-caps-model",
DisplayName: "PR7 Capability Probe",
APIModel: "pr7-caps-api-model",
Provider: modelregistry.ProviderOpenAI,
ContextLimits: modelregistry.ContextLimits{
ContextWindow: 200_000,
MaxOutputTokens: 64_000,
},
ReasoningEfforts: []modelregistry.ReasoningEffort{
modelregistry.ReasoningEffortLow,
modelregistry.ReasoningEffortMedium,
modelregistry.ReasoningEffortHigh,
},
Capabilities: modelregistry.ModelCapabilities{
modelregistry.ModelCapabilityChat,
modelregistry.ModelCapabilityVision,
modelregistry.ModelCapabilityReasoning,
modelregistry.ModelCapabilityPromptCache,
},
Cost: modelregistry.ModelCost{
Currency: "USD",
Unit: "per_1m_tokens",
InputPerMillion: 1,
OutputPerMillion: 2,
Source: "https://provider.example/pricing (test fixture)",
SourceLastVerified: "2026-07-18",
},
Status: modelregistry.ModelStatusActive,
Aliases: []string{"pr7-caps"},
}})
if err != nil {
t.Fatalf("NewRegistry: %v", err)
}
tsp, err := NewTurnSessionProvider(config.ProviderProfile{
Name: "pr7-caps",
ProviderKind: config.ProviderKindOpenAI,
BaseURL: "https://provider.example/v1",
APIKey: "sk-turn-session-test",
Model: "pr7-caps-model",
}, Options{
UserAgent: "zero-turn-session-test",
ModelRegistry: &registry,
})
if err != nil {
t.Fatalf("NewTurnSessionProvider: %v", err)
}
caps := tsp.Capabilities()
if caps.Model != "pr7-caps-api-model" {
t.Fatalf("Model = %q, want the registry APIModel", caps.Model)
}
if caps.ContextWindow != 200_000 || caps.MaxOutputTokens != 64_000 {
t.Fatalf("limits = (%d, %d), want (200000, 64000)", caps.ContextWindow, caps.MaxOutputTokens)
}
if !caps.SupportsVision || !caps.SupportsReasoning || !caps.SupportsPromptCache {
t.Fatalf("capability flags = %+v, want vision/reasoning/prompt-cache all true", caps)
}
if len(caps.ReasoningEfforts) != 3 || caps.ReasoningEfforts[0] != "low" || caps.ReasoningEfforts[2] != "high" {
t.Fatalf("ReasoningEfforts = %v, want [low medium high]", caps.ReasoningEfforts)
}
if caps.NativeCompaction {
t.Fatal("NativeCompaction = true, want false for the default adapter")
}
}
// TestNewTurnSessionProviderUsesEffectiveReasoningEfforts asserts the
// projection reads efforts through Registry.ReasoningEfforts, so a catalog
// entry that enumerates no efforts of its own still reports the name-inferred
// effective tiers the /effort picker and run-time resolver advertise.
func TestNewTurnSessionProviderUsesEffectiveReasoningEfforts(t *testing.T) {
registry, err := modelregistry.NewRegistry([]modelregistry.ModelEntry{{
// A gpt-5-family id with NO ReasoningEfforts listed: the effective
// efforts come from name inference, differing from the raw entry.
ID: "gpt-5-pr7-probe",
DisplayName: "PR7 Effective Efforts Probe",
APIModel: "gpt-5-pr7-probe-api",
Provider: modelregistry.ProviderOpenAI,
ContextLimits: modelregistry.ContextLimits{
ContextWindow: 400_000,
MaxOutputTokens: 128_000,
},
Capabilities: modelregistry.ModelCapabilities{
modelregistry.ModelCapabilityChat,
modelregistry.ModelCapabilityReasoning,
},
Cost: modelregistry.ModelCost{
Currency: "USD",
Unit: "per_1m_tokens",
InputPerMillion: 1,
OutputPerMillion: 2,
Source: "https://provider.example/pricing (test fixture)",
SourceLastVerified: "2026-07-18",
},
Status: modelregistry.ModelStatusActive,
Aliases: []string{"pr7-effective-efforts"},
}})
if err != nil {
t.Fatalf("NewRegistry: %v", err)
}
tsp, err := NewTurnSessionProvider(config.ProviderProfile{
Name: "pr7-effective",
ProviderKind: config.ProviderKindOpenAI,
BaseURL: "https://provider.example/v1",
APIKey: "sk-turn-session-test",
Model: "gpt-5-pr7-probe",
}, Options{
UserAgent: "zero-turn-session-test",
ModelRegistry: &registry,
})
if err != nil {
t.Fatalf("NewTurnSessionProvider: %v", err)
}
got := tsp.Capabilities().ReasoningEfforts
want := []string{"minimal", "low", "medium", "high"}
if len(got) != len(want) {
t.Fatalf("ReasoningEfforts = %v, want %v (name-inferred fallback)", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("ReasoningEfforts[%d] = %q, want %q (full: %v)", i, got[i], want[i], got)
}
}
}
Loaded 100 of 173 files, more files were not shown because too many files have changed in this diff. Show more