Compare commits

..
Author SHA1 Message Date
anandh8x 5fb01dae77 docs: make validation guidance non-mutating 2026-07-18 21:24:07 +05:30
anandh8x f90fd8567e docs: refresh repository agent guidelines 2026-07-18 21:16:00 +05:30
41 changed files with 65 additions and 2703 deletions

No files matched your search

-6
View File
@@ -1,11 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: Questions and setup help
url: https://github.com/Gitlawb/zero/discussions/categories/q-a
about: Ask in Discussions Q&A. Include zero --version, your OS, provider/model, and zero doctor output for setup problems.
- name: Feature ideas
url: https://github.com/Gitlawb/zero/discussions/categories/ideas
about: Propose and discuss ideas in Discussions before opening a PR (Zero is in controlled development, see CONTRIBUTING.md).
- name: Security vulnerability
url: https://github.com/Gitlawb/zero/security/advisories/new
about: Report security issues privately via GitHub Security Advisories (see SECURITY.md). Do not file a public issue.
+3 -13
View File
@@ -1,22 +1,12 @@
name: Feature request (previously discussed)
description: File only after your idea gained traction in Discussions Ideas or a maintainer asked for a tracking issue
name: Feature request
description: Propose a new capability or change
labels: ["enhancement"]
body:
- type: markdown
attributes:
value: |
New feature ideas start in [Discussions Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas),
not here. This form is for the step after that: an idea that gained traction in Discussions,
or one a maintainer asked to have tracked as an issue. Please link the discussion below.
Do not begin implementation until the issue has been reviewed and marked with the
Please do not begin implementation until the issue has been reviewed and marked with the
`issue-approved` label (see CONTRIBUTING.md).
- type: input
id: discussion
attributes:
label: Ideas discussion
description: Link to the Discussions Ideas thread (or note which maintainer asked for this issue).
validations:
required: true
- type: textarea
id: problem
attributes:
-3
View File
@@ -55,6 +55,3 @@ docs/superpowers/
zero-termux-arm64
zero-linux-sandbox
zero-seccomp
# Generated benchmark reports are evidence for one configuration, never tree state
internal/perfbench/reports/*.json
+5 -26
View File
@@ -8,13 +8,6 @@ stabilizes. Community participation is still welcome, especially through bug
reports, feature requests, reproduction details, logs, and discussion on existing
issues.
Questions and setup help belong in
[GitHub Discussions](https://github.com/Gitlawb/zero/discussions) (see
[SUPPORT.md](SUPPORT.md)), not in the issue tracker. Feature ideas start in the
[Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas) category;
an idea that gains traction there is the path to an approved issue and,
eventually, an accepted PR.
Please read this policy before opening a pull request. Pull requests that do not
follow it may be closed without review.
@@ -40,11 +33,7 @@ unless the issue has been explicitly approved for outside contribution.
There are three primary ways to contribute:
1. Open a bug report with clear reproduction steps.
2. Propose a feature in
[Discussions Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas),
describing the problem, use case, and expected behavior. A feature-request
issue is filed only after the idea gains traction there or a maintainer asks
for one.
2. Open a feature request describing the problem, use case, and expected behavior.
3. Add useful context to an existing issue, such as logs, screenshots, examples,
affected versions, or reproduction cases.
@@ -57,8 +46,7 @@ All community pull requests require an approved parent issue.
Before starting implementation work:
1. For a bug, open an issue using the bug-report template. For a feature, start
in Discussions Ideas; the feature-request issue comes after the discussion.
1. Open an issue using the appropriate issue template.
2. Describe the bug, feature request, or functional change clearly.
3. Wait for the core team to review the issue.
4. Do not open a pull request unless the issue has the `issue-approved` label.
@@ -145,12 +133,7 @@ Clear bug reports are the most useful way to help the project during this phase.
## Submitting a Feature Request
Feature requests start as a thread in
[Discussions Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas),
not as an issue. The feature-request issue form exists for the step AFTER that:
when an idea has gained traction in Discussions or a maintainer has asked for an
issue to track it. Whether in the discussion or the eventual issue, please
include:
When filing a feature request, please include:
- The problem you are trying to solve.
- The behavior or functionality you are requesting.
@@ -173,9 +156,5 @@ contribution phase.
## Getting Help
General questions, setup help, and "is this a bug or my config" uncertainty all
go to
[Discussions Q&A](https://github.com/Gitlawb/zero/discussions/categories/q-a).
If you are unsure whether a proposed code change is appropriate, raise it in
Discussions Ideas or on the relevant issue first and ask. Please keep discussion
focused, specific, and respectful of maintainer time.
If you are unsure whether a change is appropriate, open an issue first and ask.
Please keep discussion focused, specific, and respectful of maintainer time.
-18
View File
@@ -402,24 +402,6 @@ go run ./cmd/zero-release build --goos windows --goarch amd64 --output dist/zero
- [Performance](docs/PERFORMANCE.md)
- [Agent evals](docs/AGENT_EVALS.md)
## Community
Questions, setup help, ideas, and sharing all live in
[GitHub Discussions](https://github.com/Gitlawb/zero/discussions):
| Category | Use it for |
|---|---|
| [Q&A](https://github.com/Gitlawb/zero/discussions/categories/q-a) | Setup help, provider/model configuration, "how do I" questions |
| [Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas) | Feature proposals and design discussion before any PR |
| [Show and tell](https://github.com/Gitlawb/zero/discussions/categories/show-and-tell) | Your skills, plugins, MCP setups, themes, and workflows |
| [Announcements](https://github.com/Gitlawb/zero/discussions/categories/announcements) | Releases and project news from the maintainers |
For a good Q&A answer fast, include `zero --version`, your OS and install
method, the provider/model in use, and `zero doctor` output. See
[SUPPORT.md](SUPPORT.md). Bugs belong in
[issues](https://github.com/Gitlawb/zero/issues/new/choose); security reports
follow [SECURITY.md](SECURITY.md), never a public thread.
## Contributing
Contributions are welcome. Read [CONTRIBUTING.md](CONTRIBUTING.md), run the
-36
View File
@@ -1,36 +0,0 @@
# Getting help with Zero
## Questions, setup help, configuration
Ask in [GitHub Discussions](https://github.com/Gitlawb/zero/discussions). Use
[Q&A](https://github.com/Gitlawb/zero/discussions/categories/q-a) so answers can
be marked and found by the next person.
The fastest way to a useful answer is to include:
- `zero --version` output
- your OS (Windows, macOS, Linux) and how you installed Zero (npm, install
script, source)
- the provider and model you are using (`zero providers list`)
- `zero doctor` output for setup problems
- the exact command you ran and the exact error text
## Bugs
If Zero does something wrong (crashes, wrong output, broken behavior), file a
[bug report](https://github.com/Gitlawb/zero/issues/new/choose). If you are not
sure whether it is a bug or a setup problem, start in Discussions and we will
promote it to an issue if it is one.
## Feature ideas
Start a thread in
[Ideas](https://github.com/Gitlawb/zero/discussions/categories/ideas). Zero is
in controlled development (see CONTRIBUTING.md), so discussing an idea first is
also the path to getting a PR accepted.
## Security issues
Never file a public issue or discussion. Report privately via
[GitHub Security Advisories](https://github.com/Gitlawb/zero/security/advisories/new)
(see SECURITY.md).
-17
View File
@@ -4,8 +4,6 @@ import (
"bytes"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/perfbench"
)
func TestParseArgsUsesCliAndEnvOverrides(t *testing.T) {
@@ -94,18 +92,3 @@ func TestHelpText(t *testing.T) {
func emptyEnv(string) string {
return ""
}
func TestTurnExitCodeFailsWhenEveryTaskErrored(t *testing.T) {
var stderr bytes.Buffer
allErrored := perfbench.TurnBenchResult{TasksAttempted: 3, TasksErrored: 3}
if code := turnExitCode(allErrored, &stderr); code != 1 {
t.Fatalf("exit=%d, want 1 when every task errored", code)
}
if !strings.Contains(stderr.String(), "no accepted benchmark sample") {
t.Fatalf("stderr missing explanation: %q", stderr.String())
}
partial := perfbench.TurnBenchResult{TasksAttempted: 3, TasksErrored: 2}
if code := turnExitCode(partial, &stderr); code != 0 {
t.Fatalf("exit=%d, want 0 for partial errors (summary surfaces them)", code)
}
}
+1 -42
View File
@@ -9,7 +9,6 @@ import (
"path/filepath"
"strings"
"github.com/Gitlawb/zero/internal/execprofile"
"github.com/Gitlawb/zero/internal/perfbench"
)
@@ -21,7 +20,6 @@ type turnOptions struct {
SuitePath string
Model string
Mode string
ExecProfile string
SelfCorrect bool
Binary string
Iterations int
@@ -66,7 +64,6 @@ func runTurnCommand(args []string, getenv func(string) string, stdout io.Writer,
result, err := perfbench.RunTurnBench(context.Background(), set, perfbench.TurnBenchConfig{
Model: options.Model,
Mode: options.Mode,
ExecProfile: options.ExecProfile,
SelfCorrect: options.SelfCorrect,
Version: options.Version,
Commit: options.Commit,
@@ -89,23 +86,9 @@ func runTurnCommand(args []string, getenv func(string) string, stdout io.Writer,
_, _ = fmt.Fprintln(stderr, "[zero] Turn benchmark failed: "+err.Error())
return 1
}
return turnExitCode(result, stderr)
return 0
}
_, _ = fmt.Fprintln(stdout, perfbench.FormatTurnBenchSummary(result))
return turnExitCode(result, stderr)
}
// turnExitCode fails the command when every attempted task errored (no
// iteration produced an accepted benchmark sample): such a report measures
// nothing, and exiting 0 would let a broken configuration (missing binary, bad
// path, failing harness step) pass as a clean baseline. Partial errors keep
// exit 0 — the summary surfaces them loudly and the surviving samples are
// still valid measurements.
func turnExitCode(result perfbench.TurnBenchResult, stderr io.Writer) int {
if result.TasksAttempted > 0 && result.TasksErrored == result.TasksAttempted {
_, _ = fmt.Fprintln(stderr, "[zero] Turn benchmark failed: every task errored with no accepted benchmark sample (see warnings); the report contains no valid measurements")
return 1
}
return 0
}
@@ -140,13 +123,6 @@ func parseTurnArgs(args []string, getenv func(string) string) (turnOptions, erro
}
options.Mode = value
index = next
case "--exec-profile":
value, next, err := readOptionValue(args, inlineValue, index, flag)
if err != nil {
return options, err
}
options.ExecProfile = value
index = next
case "--binary":
value, next, err := readOptionValue(args, inlineValue, index, flag)
if err != nil {
@@ -219,20 +195,6 @@ func parseTurnArgs(args []string, getenv func(string) string) (turnOptions, erro
if strings.TrimSpace(options.Model) == "" && !options.DryRun {
return options, fmt.Errorf("--model is required (or pass --dry-run)")
}
// Validate the profile here, before anything spawns: an unknown name would
// otherwise make every child exit with a usage error in milliseconds, and
// those near-zero walls would be recorded as valid latency samples in an
// exit-0 report — exactly the misread-as-improvement trap the harness
// closed for spawn failures. Normalizing to the catalog name also keeps
// the stamped execProfile canonical (Lookup is case-insensitive), so two
// captures of the same posture always compare equal.
if raw := strings.TrimSpace(options.ExecProfile); raw != "" {
profile, ok := execprofile.Lookup(raw)
if !ok {
return options, fmt.Errorf("unknown execution profile %q for --exec-profile. Valid profiles: %s", raw, strings.Join(execprofile.Names(), ", "))
}
options.ExecProfile = profile.Name
}
return options, nil
}
@@ -263,9 +225,6 @@ func turnHelpText() string {
" --suite <path> Task set JSON file (required)",
" --model <model> Model to run (required unless --dry-run)",
" --mode <name> Exec mode preset to apply",
" --exec-profile <name>",
" Execution profile for every task (fast|balanced|thorough);",
" forwarded to zero exec and stamped into the result",
" --self-correct Enable the post-edit verify-and-correct loop",
" --binary <path> Path to the `zero` binary (default: zero on PATH / repo root)",
" --iterations <n> Times to run each task (default: 1)",
-65
View File
@@ -1,65 +0,0 @@
package main
import (
"strings"
"testing"
)
func TestParseTurnArgsExecProfile(t *testing.T) {
getenv := func(string) string { return "" }
options, err := parseTurnArgs([]string{
"--suite", "suite.json",
"--model", "m",
"--exec-profile", "fast",
}, getenv)
if err != nil {
t.Fatalf("parseTurnArgs: %v", err)
}
if options.ExecProfile != "fast" {
t.Fatalf("ExecProfile = %q, want fast", options.ExecProfile)
}
// Inline form.
options, err = parseTurnArgs([]string{
"--suite=suite.json",
"--model=m",
"--exec-profile=thorough",
}, getenv)
if err != nil {
t.Fatalf("parseTurnArgs inline: %v", err)
}
if options.ExecProfile != "thorough" {
t.Fatalf("ExecProfile = %q, want thorough", options.ExecProfile)
}
}
// The bench validates the profile BEFORE spawning anything: an unknown name
// would otherwise make every child exit with a usage error in milliseconds
// and record those near-zero walls as valid latency samples in an exit-0
// report. Lookup normalization also keeps the stamped name canonical so two
// captures of the same posture compare equal.
func TestParseTurnArgsExecProfileValidatesAndNormalizes(t *testing.T) {
getenv := func(string) string { return "" }
options, err := parseTurnArgs([]string{
"--suite", "suite.json", "--model", "m", "--exec-profile", "FAST",
}, getenv)
if err != nil {
t.Fatalf("parseTurnArgs: %v", err)
}
if options.ExecProfile != "fast" {
t.Fatalf("ExecProfile = %q, want the canonical fast", options.ExecProfile)
}
_, err = parseTurnArgs([]string{
"--suite", "suite.json", "--model", "m", "--exec-profile", "blanced",
}, getenv)
if err == nil {
t.Fatal("an unknown profile must fail parse, before anything spawns")
}
for _, name := range []string{"balanced", "fast", "thorough"} {
if !strings.Contains(err.Error(), name) {
t.Fatalf("error must list %q, got %q", name, err.Error())
}
}
}
+1 -57
View File
@@ -185,32 +185,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
guards := newGuardState()
compactor := newCompactionState(options)
// The execution-profile posture controller. nil Options.Profile (every
// existing caller) makes every observe/decide call a no-op, keeping the
// loop byte-identical for unprofiled runs.
posture := newProfileController(options.Profile)
// applyPosture applies at most one posture escalation when an armed trigger
// has fired: raise the hoisted turn ceiling, restore effort and the
// completion gate, stamp the counter. Shared by the end-of-turn tail and
// the completion-gate uncertain path (which continues the loop before the
// tail runs). No-op forever after the first escalation and for unprofiled
// runs.
applyPosture := func() {
if target, fired := posture.maybeEscalate(); fired {
if target.MaxTurns > maxTurns {
maxTurns = target.MaxTurns
}
if target.ReasoningEffort != "" {
options.ReasoningEffort = target.ReasoningEffort
} else if target.RestoreDefaultEffort {
options.ReasoningEffort = ""
}
if target.RestoreCompletionGate {
options.RequireCompletionSignal = true
}
options.Trace.Counter(trace.CounterPostureEscalations, 1)
}
}
// Background post-edit diagnostics: files changed by mutating tools are
// checked off the tool-call critical path and any errors are appended as a
@@ -580,11 +554,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
result.Messages = copyMessages(messages)
return result, nil
case CompletionUncertain:
// Observe the uncertainty signal and apply any escalation NOW:
// this branch continues the turn loop before the end-of-turn
// act point, so deferring would skip escalation entirely.
posture.observeUncertain()
applyPosture()
switch evaluation.Action {
case completionActionContinue:
options.Trace.Counter(trace.CounterCompletionNudges, 1)
@@ -723,7 +692,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// would misdirect the model toward JSON shape or blocked behavior.
retriableFailure := isRetriableToolError(toolResult)
outcome := guards.observeToolResult(call.Name, retriableFailure, toolResult.Output)
posture.observeToolOutcome(outcome, toolResult)
if outcome.Stop {
// The assistant message advertised EVERY collected tool call, but
// the guard halts mid-turn so the calls after this one never run.
@@ -756,9 +724,7 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// the assistant's tool_results stay contiguous (a user message between
// tool_results breaks strict provider replay). nil SelfCorrect is a no-op.
if options.SelfCorrect != nil && len(changedFilesThisBatch) > 0 {
feedback, selfCorrectOutcome := options.SelfCorrect.AfterEdit(ctx, dedupeStrings(changedFilesThisBatch))
posture.observeSelfCorrect(selfCorrectOutcome)
if feedback != "" {
if feedback, _ := options.SelfCorrect.AfterEdit(ctx, dedupeStrings(changedFilesThisBatch)); feedback != "" {
messages = append(messages, zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: feedback,
@@ -851,13 +817,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
Content: reminder,
})
}
// One-shot posture escalation: when an armed profile trigger fired this
// turn, restore the stricter knob values the profile displaced. Mutates
// only per-turn-read policy (the hoisted turn ceiling, reasoning effort,
// completion gate); never messages, model, or session. No-op forever
// after the first escalation and for every unprofiled run.
applyPosture()
}
if ctx.Err() != nil {
@@ -1379,25 +1338,10 @@ func executeToolCall(ctx context.Context, registry *tools.Registry, call ToolCal
result = registry.RebudgetAfterHook(call.Name, args, result)
}
}
// Stamp the sandbox risk classification for this EXECUTED call so run-policy
// observers can see the risk of an allowed mutation. Prefer the preflight
// decision's classification when the sandbox evaluated the call; otherwise
// run the same pure classifier the permission path uses. Denied/canceled
// results returned earlier keep the zero value.
executedRisk := sandbox.Risk{}
if preflightDecision != nil {
executedRisk = preflightDecision.Risk
} else if toolFound {
// Unknown-tool results fall through here with a nil tool; they carry a
// denial and keep the zero risk value.
executedRisk = sandbox.Classify(sandboxRequest(call.Name, tool, args, permissionGranted, permissionMode, options))
}
// Secret scrubbing happens at the registry boundary (the single point both
// the agent loop and the MCP server pass through), so result.Output is
// already redacted here and result.Redacted reflects whether it changed.
return ToolResult{
Risk: executedRisk,
ToolCallID: call.ID,
Name: call.Name,
Status: result.Status,
-118
View File
@@ -1,118 +0,0 @@
package agent
import (
"github.com/Gitlawb/zero/internal/sandbox"
)
// profileController observes per-turn run signals and decides at most one
// posture escalation per run for the armed execution profile. A nil policy (or
// a policy without Escalate) makes every method a no-op, so runs without a
// profile are byte-identical to before — the same opt-in convention as Trace
// and SelfCorrect.
//
// The controller only ever CHANGES loop policy knobs (turn ceiling, reasoning
// effort, completion gate) through the values the loop applies at its act
// point; it never injects messages, never touches the model or session, and
// never interacts with the model-initiated escalate_model path.
type profileController struct {
policy *PostureEscalation
escalated bool
uncertainSeen int
failureTripped bool
riskTripped bool
scTripped bool
uncertainTrip bool
}
// newProfileController is nil-safe: a nil ProfilePolicy (the default for every
// existing caller) or a nil Escalate yields a controller that observes and
// decides nothing.
func newProfileController(policy *ProfilePolicy) *profileController {
if policy == nil {
return &profileController{}
}
return &profileController{policy: policy.Escalate}
}
// observeToolOutcome watches two signals from an executed tool call: the
// repeated-failure guard's streak (reusing the guard's own counting, not a
// second failure model) and the sandbox risk classification of an executed
// result.
func (c *profileController) observeToolOutcome(outcome toolFailureOutcome, result ToolResult) {
if c.policy == nil || c.escalated {
return
}
if c.policy.OnToolFailureStreak > 0 && outcome.Count >= c.policy.OnToolFailureStreak {
c.failureTripped = true
}
if threshold := riskRank(c.policy.OnRiskyMutation); threshold > 0 &&
result.DenialReason == DenialNone &&
riskRank(result.Risk.Level) >= threshold {
// The call EXECUTED (was not denied): partial failures count too, since
// the mutation ran. An unrecognized threshold ranks 0 and disables the
// signal instead of matching everything.
c.riskTripped = true
}
}
// observeUncertain counts uncertain completion evaluations (continue nudges and
// semantic checks). Headless-only by construction: the completion gate never
// runs interactively.
func (c *profileController) observeUncertain() {
if c.policy == nil || c.escalated || c.policy.OnCompletionUncertain <= 0 {
return
}
c.uncertainSeen++
if c.uncertainSeen >= c.policy.OnCompletionUncertain {
c.uncertainTrip = true
}
}
// observeSelfCorrect watches the post-edit verification outcome. Any failing
// outcome (correcting, reported, aborted) is a signal; passed and disabled are
// not.
func (c *profileController) observeSelfCorrect(outcome Outcome) {
if c.policy == nil || c.escalated || !c.policy.OnSelfCorrectFailure {
return
}
switch outcome {
case OutcomeCorrecting, OutcomeReported, OutcomeAborted:
c.scTripped = true
}
}
// maybeEscalate reports the escalation target exactly once, the first time any
// armed trigger has fired. Subsequent calls return false for the rest of the
// run: escalation is one-shot and never de-escalates, so no cooldown state is
// needed.
func (c *profileController) maybeEscalate() (PostureEscalation, bool) {
if c.policy == nil || c.escalated {
return PostureEscalation{}, false
}
if !(c.failureTripped || c.riskTripped || c.scTripped || c.uncertainTrip) {
return PostureEscalation{}, false
}
c.escalated = true
return *c.policy, true
}
// riskRank orders sandbox risk levels for threshold comparison. Unknown levels
// rank 0: an unrecognized RESULT level can never meet a valid threshold, and an
// unrecognized THRESHOLD is rejected before comparison (rank 0 disables the
// signal) so a typo in a profile can never make every result match.
func riskRank(level sandbox.RiskLevel) int {
switch level {
case sandbox.RiskLow:
return 1
case sandbox.RiskMedium:
return 2
case sandbox.RiskHigh:
return 3
case sandbox.RiskCritical:
return 4
default:
return 0
}
}
-345
View File
@@ -1,345 +0,0 @@
package agent
import (
"context"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/sandbox"
"github.com/Gitlawb/zero/internal/tools"
"github.com/Gitlawb/zero/internal/trace"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
func TestProfileControllerNilPolicyIsNoOp(t *testing.T) {
for _, controller := range []*profileController{
newProfileController(nil),
newProfileController(&ProfilePolicy{Name: "balanced"}), // Escalate nil
} {
controller.observeToolOutcome(toolFailureOutcome{Count: 99}, ToolResult{Status: tools.StatusOK, Risk: sandbox.Risk{Level: sandbox.RiskCritical}})
controller.observeUncertain()
controller.observeSelfCorrect(OutcomeAborted)
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("nil policy must never escalate")
}
}
}
func TestProfileControllerEscalatesOnFailureStreak(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnToolFailureStreak: 2, MaxTurns: 80}})
controller.observeToolOutcome(toolFailureOutcome{Count: 1}, ToolResult{Status: tools.StatusError})
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("streak below threshold must not escalate")
}
controller.observeToolOutcome(toolFailureOutcome{Count: 2}, ToolResult{Status: tools.StatusError})
target, fired := controller.maybeEscalate()
if !fired || target.MaxTurns != 80 {
t.Fatalf("expected escalation with the policy target, got fired=%t target=%+v", fired, target)
}
}
func TestProfileControllerEscalatesOnRiskyMutation(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnRiskyMutation: sandbox.RiskHigh}})
// Medium risk on an executed result: below threshold.
controller.observeToolOutcome(toolFailureOutcome{}, ToolResult{Status: tools.StatusOK, Risk: sandbox.Risk{Level: sandbox.RiskMedium}})
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("medium risk must not trip a high threshold")
}
// Critical risk on a DENIED result: never counts (the mutation never ran).
controller.observeToolOutcome(toolFailureOutcome{}, ToolResult{Status: tools.StatusError, DenialReason: DenialSandboxBlock, Risk: sandbox.Risk{Level: sandbox.RiskCritical}})
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("a denied result must not trip the risky-mutation signal")
}
// Critical risk on an executed PARTIAL FAILURE (error status, no denial):
// the mutation ran, so it counts.
controller.observeToolOutcome(toolFailureOutcome{}, ToolResult{Status: tools.StatusError, Risk: sandbox.Risk{Level: sandbox.RiskCritical}})
if _, fired := controller.maybeEscalate(); !fired {
t.Fatal("an executed partial failure at critical risk must trip a high threshold")
}
}
func TestProfileControllerIgnoresInvalidRiskThreshold(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnRiskyMutation: sandbox.RiskLevel("hgih")}})
controller.observeToolOutcome(toolFailureOutcome{}, ToolResult{Status: tools.StatusOK, Risk: sandbox.Risk{Level: sandbox.RiskCritical}})
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("an unrecognized threshold must disable the signal, not match every result")
}
}
func TestProfileControllerEscalatesOnSelfCorrectFailure(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnSelfCorrectFailure: true}})
controller.observeSelfCorrect(OutcomePassed)
controller.observeSelfCorrect(OutcomeDisabled)
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("passing/disabled outcomes must not escalate")
}
controller.observeSelfCorrect(OutcomeReported)
if _, fired := controller.maybeEscalate(); !fired {
t.Fatal("a failing verification outcome must escalate")
}
}
func TestProfileControllerEscalatesOnUncertainCompletion(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnCompletionUncertain: 2}})
controller.observeUncertain()
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("first uncertain evaluation must not trip a threshold of 2")
}
controller.observeUncertain()
if _, fired := controller.maybeEscalate(); !fired {
t.Fatal("second uncertain evaluation must escalate")
}
}
func TestProfileControllerEscalatesAtMostOnce(t *testing.T) {
controller := newProfileController(&ProfilePolicy{Escalate: &PostureEscalation{OnToolFailureStreak: 1}})
controller.observeToolOutcome(toolFailureOutcome{Count: 1}, ToolResult{Status: tools.StatusError})
if _, fired := controller.maybeEscalate(); !fired {
t.Fatal("expected first escalation")
}
controller.observeToolOutcome(toolFailureOutcome{Count: 5}, ToolResult{Status: tools.StatusOK, Risk: sandbox.Risk{Level: sandbox.RiskCritical}})
controller.observeUncertain()
if _, fired := controller.maybeEscalate(); fired {
t.Fatal("escalation must be one-shot per run")
}
}
// TestPostureEscalationOnUncertainCompletion proves the uncertain-path act
// point: the completion gate's continue branch escapes the end-of-turn tail,
// so escalation must apply before the continue. A cue turn under the gate
// counts as uncertain, escalates, and the very next request carries the target
// effort.
func TestPostureEscalationOnUncertainCompletion(t *testing.T) {
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
{
{Type: zeroruntime.StreamEventText, Content: "Let me read the file:"},
{Type: zeroruntime.StreamEventDone},
},
{
{Type: zeroruntime.StreamEventText, Content: "Done. All set."},
{Type: zeroruntime.StreamEventDone},
},
}}
recorder := trace.NewRecorder("posture-session", "run-2", "fast")
result, err := Run(context.Background(), "go", provider, Options{
Registry: tools.NewRegistry(),
MaxTurns: 10,
RequireCompletionSignal: true,
Trace: recorder,
Profile: &ProfilePolicy{
Name: "fast",
Escalate: &PostureEscalation{
ReasoningEffort: "high",
OnCompletionUncertain: 1,
},
},
})
if err != nil {
t.Fatal(err)
}
if result.FinalAnswer != "Done. All set." {
t.Fatalf("expected the completed answer, got %q", result.FinalAnswer)
}
if len(provider.requests) != 2 {
t.Fatalf("expected 2 requests, got %d", len(provider.requests))
}
if provider.requests[1].ReasoningEffort != "high" {
t.Fatalf("expected the post-escalation request to carry the target effort, got %q", provider.requests[1].ReasoningEffort)
}
tr := recorder.Finish()
if got := tr.Counter(trace.CounterPostureEscalations); got != 1 {
t.Fatalf("posture_escalations = %d, want 1", got)
}
}
// A profile that filled the run's effort displaced the provider default ("");
// RestoreDefaultEffort lets escalation restore that default, which a plain ""
// ReasoningEffort target cannot express (it means "leave untouched").
func TestPostureEscalationRestoresDefaultEffort(t *testing.T) {
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
{
{Type: zeroruntime.StreamEventText, Content: "Let me read the file:"},
{Type: zeroruntime.StreamEventDone},
},
{
{Type: zeroruntime.StreamEventText, Content: "Done. All set."},
{Type: zeroruntime.StreamEventDone},
},
}}
result, err := Run(context.Background(), "go", provider, Options{
Registry: tools.NewRegistry(),
MaxTurns: 10,
ReasoningEffort: "low",
RequireCompletionSignal: true,
Profile: &ProfilePolicy{
Name: "fast",
Escalate: &PostureEscalation{
RestoreDefaultEffort: true,
OnCompletionUncertain: 1,
},
},
})
if err != nil {
t.Fatal(err)
}
if result.FinalAnswer != "Done. All set." {
t.Fatalf("expected the completed answer, got %q", result.FinalAnswer)
}
if len(provider.requests) != 2 {
t.Fatalf("expected 2 requests, got %d", len(provider.requests))
}
if provider.requests[0].ReasoningEffort != "low" {
t.Fatalf("pre-escalation request effort = %q, want the profile's low", provider.requests[0].ReasoningEffort)
}
if provider.requests[1].ReasoningEffort != "" {
t.Fatalf("post-escalation request effort = %q, want the restored provider default (empty)", provider.requests[1].ReasoningEffort)
}
}
// failingProfileTool always errors with the same signature so the repeated-
// failure guard builds a streak.
type failingProfileTool struct{}
func (failingProfileTool) Name() string { return "flaky_probe" }
func (failingProfileTool) Description() string { return "always fails identically (test)" }
func (failingProfileTool) Parameters() tools.Schema {
return tools.Schema{Type: "object", AdditionalProperties: false}
}
func (failingProfileTool) Safety() tools.Safety {
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow}
}
func (failingProfileTool) Run(context.Context, map[string]any) tools.Result {
return tools.Result{Status: tools.StatusError, Output: "probe exploded: same signature"}
}
// TestPostureEscalationRaisesTurnCeilingMidRun proves the end-to-end act path:
// a Fast-style run with a 2-turn ceiling hits a failure streak, escalates, and
// finishes on a turn the original ceiling would have cut off, with the effort
// target applied to the post-escalation request and the counter emitted.
func TestPostureEscalationRaisesTurnCeilingMidRun(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(failingProfileTool{})
toolTurn := []zeroruntime.StreamEvent{
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "c1", ToolName: "flaky_probe"},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "c1"},
{Type: zeroruntime.StreamEventDone},
}
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
toolTurn,
toolTurn,
{
{Type: zeroruntime.StreamEventText, Content: "recovered after escalation"},
{Type: zeroruntime.StreamEventDone},
},
}}
recorder := trace.NewRecorder("posture-session", "run-1", "fast")
result, err := Run(context.Background(), "go", provider, Options{
Registry: registry,
MaxTurns: 2,
Trace: recorder,
Profile: &ProfilePolicy{
Name: "fast",
Escalate: &PostureEscalation{
MaxTurns: 4,
ReasoningEffort: "high",
OnToolFailureStreak: 2,
},
},
})
if err != nil {
t.Fatal(err)
}
if result.FinalAnswer != "recovered after escalation" {
t.Fatalf("expected the run to continue past the original 2-turn ceiling, got %q (turns=%d)", result.FinalAnswer, result.Turns)
}
if len(provider.requests) != 3 {
t.Fatalf("expected 3 requests (ceiling raised from 2 to 4), got %d", len(provider.requests))
}
if provider.requests[2].ReasoningEffort != "high" {
t.Fatalf("expected the post-escalation request to carry the target effort, got %q", provider.requests[2].ReasoningEffort)
}
tr := recorder.Finish()
if got := tr.Counter(trace.CounterPostureEscalations); got != 1 {
t.Fatalf("posture_escalations = %d, want 1", got)
}
}
// TestPostureEscalationAbsentWithoutProfile pins the no-regression invariant at
// the loop level: the identical failing script without a profile ends at the
// original ceiling with the max-turns answer.
func TestPostureEscalationAbsentWithoutProfile(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(failingProfileTool{})
toolTurn := []zeroruntime.StreamEvent{
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "c1", ToolName: "flaky_probe"},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "c1"},
{Type: zeroruntime.StreamEventDone},
}
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{toolTurn, toolTurn}}
result, err := Run(context.Background(), "go", provider, Options{
Registry: registry,
MaxTurns: 2,
})
if err != nil {
t.Fatal(err)
}
if len(provider.requests) > 3 {
t.Fatalf("unprofiled run must not extend the ceiling, got %d requests", len(provider.requests))
}
if !strings.Contains(result.FinalAnswer, "maximum number of turns") {
t.Fatalf("unprofiled run must end with the max-turns answer, got %q", result.FinalAnswer)
}
}
// TestExecuteToolCallClassifiesRiskWithoutSandbox verifies the executed-risk
// stamp on the unsandboxed path: a read-only tool classifies low.
func TestExecuteToolCallClassifiesRiskWithoutSandbox(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(failingProfileTool{})
result, abortErr := executeToolCall(
context.Background(),
registry,
ToolCall{ID: "c1", Name: "flaky_probe"},
PermissionModeAuto,
Options{},
)
if abortErr != nil {
t.Fatal(abortErr)
}
if result.Risk.Level == "" {
t.Fatal("executed result must carry a risk classification")
}
if riskRank(result.Risk.Level) > riskRank(sandbox.RiskMedium) {
t.Fatalf("read-only tool classified %q, want at most medium", result.Risk.Level)
}
}
// TestUnknownToolResultCarriesZeroRisk verifies a call that never executed a
// tool (unknown name, nil tool) keeps the zero risk value instead of panicking
// or inventing a classification.
func TestUnknownToolResultCarriesZeroRisk(t *testing.T) {
registry := tools.NewRegistry()
result, abortErr := executeToolCall(
context.Background(),
registry,
ToolCall{ID: "c1", Name: "no_such_tool"},
PermissionModeAuto,
Options{},
)
if abortErr != nil {
t.Fatal(abortErr)
}
if result.Status == tools.StatusOK {
t.Fatal("expected an error result for an unknown tool")
}
if result.Risk.Level != "" {
t.Fatalf("not-executed result must keep the zero risk value, got %q", result.Risk.Level)
}
}
-60
View File
@@ -75,12 +75,6 @@ type ToolResult struct {
// DenialReason categorizes why a tool call was blocked (empty when it ran).
// It lets a surface distinguish the cause precisely instead of parsing Output.
DenialReason DenialCategory
// Risk is the sandbox risk classification of this call, stamped for EXECUTED
// results so run-policy observers (the execution-profile controller) can see
// the risk level of an allowed mutation. It mirrors the classification the
// permission path already computes; denied or canceled results keep the zero
// value. Pure observation: nothing about permissions or sandboxing changes.
Risk sandbox.Risk
// LoadedTools carries the deferred-tool names a tool_search call asked the
// loop to expose next turn (lifted from Meta["load_tools"]). nil for every
// ordinary tool result; only tool_search populates it.
@@ -104,54 +98,6 @@ const (
DenialHookBlocked DenialCategory = "hook_blocked" // vetoed by a beforeTool hook
)
// ProfilePolicy is the loop-facing slice of a selected execution profile.
// The surface (exec flag, TUI command) resolves a named profile into this
// policy; the loop only ever sees the policy, never the catalog.
type ProfilePolicy struct {
// Name labels trace counters and diagnostics (e.g. "fast"). The trace
// recorder's profile label is set by the caller at recorder construction.
Name string
// Escalate, when non-nil, arms one-shot in-run posture escalation.
Escalate *PostureEscalation
}
// PostureEscalation describes a one-shot escalation to stricter knob values,
// applied mid-run when an armed trigger fires. Targets are the values the
// selected profile DISPLACED at run start (i.e. "restore the balanced
// posture"), so escalation can never introduce a value that was not already
// valid for this run and model. Zero-valued targets leave that knob untouched.
type PostureEscalation struct {
// MaxTurns raises the turn ceiling to this value when greater than the
// ceiling in effect. 0 leaves the ceiling untouched.
MaxTurns int
// ReasoningEffort replaces the run's effort when non-empty.
ReasoningEffort string
// RestoreDefaultEffort clears the run's effort override (back to the
// provider/model default) when true. It exists because the displaced value
// of a profile-filled effort is "" — which as a ReasoningEffort target
// means "leave untouched" — so restoring the default needs its own signal.
// Ignored when ReasoningEffort is non-empty.
RestoreDefaultEffort bool
// RestoreCompletionGate re-enables RequireCompletionSignal (headless
// completion semantics) when true.
RestoreCompletionGate bool
// Triggers. A zero value disables that signal entirely.
// OnToolFailureStreak fires when the repeated-failure guard observes a
// same-tool retriable-failure streak of at least this length.
OnToolFailureStreak int
// OnCompletionUncertain fires on the Nth uncertain completion evaluation
// (continue nudge or semantic check). Headless only: the completion gate
// never runs interactively.
OnCompletionUncertain int
// OnSelfCorrectFailure fires when a post-edit verification cycle reports a
// failing outcome (correcting, reported, or aborted).
OnSelfCorrectFailure bool
// OnRiskyMutation fires when an EXECUTED tool result carries a sandbox risk
// level at or above this threshold. Empty disables the signal.
OnRiskyMutation sandbox.RiskLevel
}
type PermissionRequest struct {
ToolCallID string `json:"toolCallId"`
ToolName string `json:"name"`
@@ -352,12 +298,6 @@ type Options struct {
// contract as ModelSwitcher: a returned error records a note and the run
// continues on the current model and session.
ModelSessionSwitcher func(ctx context.Context, modelID string) (zeroruntime.TurnSessionProvider, error)
// Profile, when set, arms the execution-profile posture controller for this
// run (auto-escalation to stricter knob values on failure/uncertainty/risky
// mutation signals). nil — the default everywhere today — leaves the loop
// byte-identical: no observation, no escalation, no counters. Same opt-in
// convention as Trace and SelfCorrect.
Profile *ProfilePolicy
// Trace, when set, records per-turn timing for the run: the loop stamps
// spans (prompt build, generation, tool execution, permission wait,
// compaction, provider connect) and counters (model requests, tool calls,
-3
View File
@@ -1341,9 +1341,6 @@ Flags:
--spec-reasoning-effort <effort>
Override draft reasoning effort when --use-spec is set
--max-turns <number> Override the maximum agent loop turns
--exec-profile <name> Apply an execution profile (balanced, fast, thorough): loop
posture only (turn budget, effort, self-correction, escalation);
composes with --mode (which picks the model) and explicit flags win
--auto <low|medium|high> Set exec autonomy; high enables unsafe tools
--enabled-tools <tools> Only expose these comma or space separated tools
--disabled-tools <tools> Hide these comma or space separated tools
+2 -89
View File
@@ -13,7 +13,6 @@ import (
"github.com/Gitlawb/zero/internal/agent"
"github.com/Gitlawb/zero/internal/config"
"github.com/Gitlawb/zero/internal/errhint"
"github.com/Gitlawb/zero/internal/execprofile"
"github.com/Gitlawb/zero/internal/imageinput"
"github.com/Gitlawb/zero/internal/lsp"
"github.com/Gitlawb/zero/internal/modelregistry"
@@ -70,14 +69,7 @@ type execOptions struct {
// intentionally inert: nothing consumes it. Model selection is driven by
// --model / --mode instead. See writeExecHelp ("Accept legacy model profile
// selection") and TestRunExecAcceptsLegacyModelProfileFlags.
modelProfile string
// execProfile selects a named execution profile (balanced, fast, thorough):
// a loop-posture bundle (turn budget, reasoning effort, self-correction,
// escalation triggers) applied AFTER --mode with the same
// fill-only-if-unset rule, so precedence is explicit flag > mode > profile.
// Distinct from the mode preset also named "fast" (which picks a model) and
// from the legacy inert --profile above. See internal/execprofile.
execProfile string
modelProfile string
reasoningEffort string
useSpec bool
specModel string
@@ -175,14 +167,6 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
if err := applyExecMode(&options); err != nil {
return writeExecFormatUsageError(stdout, stderr, options.outputFormat, err.Error())
}
// A profile tunes loop posture the way a mode tunes model selection, and
// runs after it so the mode's fills count as "set" and win. MaxTurns is
// deferred to config resolution below, where the displaced resolved budget
// is known and becomes the escalation restore target.
execProfile, execProfileFilledEffort, err := applyExecProfile(&options)
if err != nil {
return writeExecFormatUsageError(stdout, stderr, options.outputFormat, err.Error())
}
workspaceRoot, err := resolveWorkspaceRoot(options.cwd, deps)
if err != nil {
@@ -288,8 +272,6 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
}
return writeExecProviderError(stdout, stderr, options.outputFormat, "provider_error", err.Error())
}
var displacedMaxTurns int
resolved.MaxTurns, displacedMaxTurns = applyProfileTurnBudget(execProfile, options.maxTurns, resolved.MaxTurns)
registerLocalControlTools(registry, workspaceRoot, resolved.LocalControl)
if err := validateExecToolFilters(options, registry); err != nil {
return writeExecFormatUsageError(stdout, stderr, options.outputFormat, err.Error())
@@ -504,13 +486,6 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
reasoningEffort: forwardEffort,
specPermissionMode: permissionMode,
notifier: notifier,
// The profile displaced resolved.MaxTurns above, so the spec-draft
// run arms the same escalation policy as the main run (in practice
// only the failure-streak and risky-mutation triggers can fire
// here: the draft runs without the completion gate or a
// self-corrector).
profilePolicy: execProfile.Policy(displacedMaxTurns,
specProfileEffortFilled(execProfileFilledEffort, options.specReasoningEffort)),
})
}
@@ -553,7 +528,7 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
var traceRecorder *trace.Recorder
var traceSnapshot *trace.TurnTrace
if tracePath != "" {
traceRecorder = trace.NewRecorder(preparedSession.Session.SessionID, runID, execProfile.Name)
traceRecorder = trace.NewRecorder(preparedSession.Session.SessionID, runID, "")
defer func() {
if err := writeTraceSnapshot(traceSnapshot, tracePath, stderr); err != nil {
fmt.Fprintf(stderr, "[zero] failed to write trace: %s\n", err)
@@ -652,7 +627,6 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
Autonomy: options.autonomy,
SelfCorrect: selfCorrector,
FileDiagnostics: fileDiagnostics,
Profile: execProfile.Policy(displacedMaxTurns, execProfileFilledEffort),
// Headless exec: don't accept a no-tool-call turn as "done" while work
// clearly remains (pending plan items / a mid-step continuation cue) —
// nudge to continue, and finalize as INCOMPLETE rather than false success
@@ -1159,67 +1133,6 @@ func applyExecMode(options *execOptions) error {
return nil
}
// applyExecProfile expands an --exec-profile selection onto the exec options.
// Like applyExecMode it only fills fields the caller left unset, and it runs
// AFTER the mode so a mode-filled field counts as set: precedence is explicit
// flag > --mode > --exec-profile. Only the options-level knobs are applied here
// (reasoning effort still flows through the model-supported gating downstream;
// self-correct is presence-only so a profile can only turn it on). MaxTurns is
// intentionally NOT applied here — the profile displaces the resolved budget
// after config resolution, where the displaced value is known and becomes the
// escalation restore target. An unknown profile is a usage error listing the
// valid names. Not to be confused with the legacy inert --profile flag.
// The second return reports whether the profile actually filled the reasoning
// effort (false when the user set one explicitly), so the escalation policy
// knows the displaced effort was the provider default and can restore it.
func applyExecProfile(options *execOptions) (execprofile.Profile, bool, error) {
name := strings.TrimSpace(options.execProfile)
if name == "" {
return execprofile.Profile{}, false, nil
}
profile, ok := execprofile.Lookup(name)
if !ok {
return execprofile.Profile{}, false, execUsageError{fmt.Sprintf("unknown execution profile %q. Valid profiles: %s.", options.execProfile, strings.Join(execprofile.Names(), ", "))}
}
effortFilled := false
if options.reasoningEffort == "" && profile.ReasoningEffort != "" {
options.reasoningEffort = profile.ReasoningEffort
effortFilled = true
}
// Spec-safe projection: the spec-draft path wires no self-corrector (an
// explicit --self-correct --use-spec is rejected at parse time, before
// profiles apply), so a profile's self-correct knob is meaningless there.
// Project it away rather than erroring: the user asked for a thorough
// DRAFT, and the profile's other knobs (budget, effort) apply to it fine.
if profile.SelfCorrect && !options.useSpec {
options.selfCorrect = true
}
return profile, effortFilled, nil
}
// specProfileEffortFilled reports whether the profile's effort fill actually
// governs the spec-draft run. An explicit --spec-reasoning-effort replaces the
// filled effort for the draft, so the escalation's effort restore must not arm
// there: escalation must never clear an effort the user pinned by hand.
func specProfileEffortFilled(effortFilled bool, specReasoningEffort string) bool {
return effortFilled && strings.TrimSpace(specReasoningEffort) == ""
}
// applyProfileTurnBudget decides the run's turn budget once config is resolved.
// The profile displaces the RESOLVED value, not the flag: a pinned budget
// always wins and the profile backs off entirely, while an env/config budget
// is the "balanced posture" a mid-run escalation can restore. pinnedMaxTurns
// is any caller-pinned budget — an explicit --max-turns flag OR a mode
// preset's fill (both land in options.maxTurns, and a mode outranks a profile
// just like the flag does). displaced is 0 when nothing was displaced, so an
// escalation leaves the ceiling untouched.
func applyProfileTurnBudget(profile execprofile.Profile, pinnedMaxTurns int, resolvedMaxTurns int) (effective int, displaced int) {
if profile.MaxTurns > 0 && pinnedMaxTurns == 0 {
return profile.MaxTurns, resolvedMaxTurns
}
return resolvedMaxTurns, 0
}
// resolveSelectedModel routes a user-supplied --model value through the model
// registry so that fuzzy aliases (e.g. "sonnet 4.5") resolve to canonical ids
// and deprecated models auto-redirect to their fallback. It returns the model id
-13
View File
@@ -147,19 +147,6 @@ func parseExecArgs(args []string) (execOptions, bool, error) {
index = next
case strings.HasPrefix(arg, "--profile="):
options.modelProfile = strings.TrimSpace(strings.TrimPrefix(arg, "--profile="))
case arg == "--exec-profile":
value, next, err := nextFlagValue(args, index, arg)
if err != nil {
return options, false, err
}
options.execProfile = strings.TrimSpace(value)
index = next
case strings.HasPrefix(arg, "--exec-profile="):
value, err := requiredInlineFlagValue(arg, "--exec-profile")
if err != nil {
return options, false, err
}
options.execProfile = value
case arg == "-r" || arg == "--reasoning-effort":
value, next, err := nextFlagValue(args, index, arg)
if err != nil {
-224
View File
@@ -1,224 +0,0 @@
package cli
import (
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/execprofile"
)
func TestApplyExecProfileFillsOnlyUnset(t *testing.T) {
// Unset effort: the profile fills it and reports the fill, so the
// escalation policy knows the displaced effort was the provider default.
options := execOptions{execProfile: "fast"}
profile, effortFilled, err := applyExecProfile(&options)
if err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if profile.Name != "fast" {
t.Fatalf("profile.Name = %q, want fast", profile.Name)
}
if options.reasoningEffort != "low" {
t.Fatalf("reasoningEffort = %q, want the profile's low", options.reasoningEffort)
}
if !effortFilled {
t.Fatal("the profile filled the effort, so effortFilled must report it")
}
// Set effort (explicit flag or a mode's fill): the profile backs off.
options = execOptions{execProfile: "fast", reasoningEffort: "high"}
_, effortFilled, err = applyExecProfile(&options)
if err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if options.reasoningEffort != "high" {
t.Fatalf("reasoningEffort = %q, an explicit value must win over the profile", options.reasoningEffort)
}
if effortFilled {
t.Fatal("the profile backed off, so effortFilled must be false")
}
}
func TestApplyExecProfileThoroughArmsSelfCorrect(t *testing.T) {
options := execOptions{execProfile: "thorough"}
if _, _, err := applyExecProfile(&options); err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if !options.selfCorrect {
t.Fatal("thorough must arm the post-edit self-correct loop")
}
if options.reasoningEffort != "high" {
t.Fatalf("reasoningEffort = %q, want high", options.reasoningEffort)
}
}
// The spec-draft path wires no self-corrector (explicit --self-correct
// --use-spec is rejected at parse time, before profiles apply), so a profile's
// self-correct knob is projected away under --use-spec instead of silently
// arming a knob the draft runner ignores. The other knobs still apply.
func TestApplyExecProfileSpecProjectsAwaySelfCorrect(t *testing.T) {
options := execOptions{execProfile: "thorough", useSpec: true}
if _, _, err := applyExecProfile(&options); err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if options.selfCorrect {
t.Fatal("thorough under --use-spec must not arm self-correct (the draft runner has none)")
}
if options.reasoningEffort != "high" {
t.Fatalf("reasoningEffort = %q, the profile's other knobs must still apply", options.reasoningEffort)
}
}
func TestApplyExecProfileUnknownIsUsageError(t *testing.T) {
options := execOptions{execProfile: "turbo"}
_, _, err := applyExecProfile(&options)
if err == nil {
t.Fatal("expected a usage error for an unknown profile")
}
if _, ok := err.(execUsageError); !ok {
t.Fatalf("expected execUsageError, got %T: %v", err, err)
}
for _, name := range execprofile.Names() {
if !strings.Contains(err.Error(), name) {
t.Fatalf("usage error must list %q, got %q", name, err.Error())
}
}
}
// Precedence: explicit flag > --mode > --exec-profile. The deep mode fills
// effort=high and max-turns=160; the fast profile must not override either,
// because a mode-filled field counts as set by the time the profile runs.
func TestExecProfilePrecedenceFlagOverModeOverProfile(t *testing.T) {
options := execOptions{mode: "deep", execProfile: "fast"}
if err := applyExecMode(&options); err != nil {
t.Fatalf("applyExecMode: %v", err)
}
profile, _, err := applyExecProfile(&options)
if err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if options.reasoningEffort != "high" {
t.Fatalf("reasoningEffort = %q, the mode's high must win over the profile's low", options.reasoningEffort)
}
// The mode filled options.maxTurns, so the turn budget must not be displaced.
effective, displaced := applyProfileTurnBudget(profile, options.maxTurns, options.maxTurns)
if effective != 160 || displaced != 0 {
t.Fatalf("turn budget = (%d, displaced %d), the mode's 160 must win with nothing displaced", effective, displaced)
}
}
func TestApplyProfileTurnBudget(t *testing.T) {
fast, _ := execprofile.Lookup("fast")
balanced, _ := execprofile.Lookup("balanced")
// No explicit flag: the profile displaces the resolved budget.
if effective, displaced := applyProfileTurnBudget(fast, 0, 80); effective != 30 || displaced != 80 {
t.Fatalf("fast over resolved 80 = (%d, %d), want (30, 80)", effective, displaced)
}
// Explicit --max-turns pins the budget; the profile backs off entirely.
if effective, displaced := applyProfileTurnBudget(fast, 50, 50); effective != 50 || displaced != 0 {
t.Fatalf("fast with explicit 50 = (%d, %d), want (50, 0)", effective, displaced)
}
// Balanced never displaces anything.
if effective, displaced := applyProfileTurnBudget(balanced, 0, 80); effective != 80 || displaced != 0 {
t.Fatalf("balanced over resolved 80 = (%d, %d), want (80, 0)", effective, displaced)
}
}
// The no-regression invariant at the options level: selecting balanced leaves
// the options byte-identical to not selecting a profile at all, and produces
// no loop policy.
func TestExecProfileBalancedLeavesOptionsUntouched(t *testing.T) {
options := execOptions{execProfile: "balanced"}
reference := options
profile, _, err := applyExecProfile(&options)
if err != nil {
t.Fatalf("applyExecProfile: %v", err)
}
if !reflect.DeepEqual(options, reference) {
t.Fatalf("balanced changed the options: %+v vs %+v", options, reference)
}
if policy := profile.Policy(80, false); policy != nil {
t.Fatalf("balanced must produce a nil policy, got %+v", policy)
}
}
// --exec-profile= with an inline empty value must error like its two-token
// form and the sibling --mode=, not silently run the default posture (the
// classic shell hazard: an unset variable expanding into the inline form).
func TestParseExecProfileInlineEmptyIsError(t *testing.T) {
_, _, err := parseExecArgs([]string{"--exec-profile=", "hello"})
if err == nil || !strings.Contains(err.Error(), "--exec-profile requires a value") {
t.Fatalf("expected a required-value error, got %v", err)
}
}
// Full echo-provider run: the selected profile must be stamped into the
// per-turn trace so benchmark attribution can group runs by posture.
func TestRunExecTraceRecordsSelectedProfile(t *testing.T) {
tracePath := filepath.Join(t.TempDir(), "trace.ndjson")
exitCode, _, stderr := runExecWithEcho(t, []string{
"exec", "--exec-profile", "fast", "--trace", tracePath, "hello",
})
if exitCode != exitSuccess {
t.Fatalf("expected exit %d, got %d: %s", exitSuccess, exitCode, stderr)
}
raw, err := os.ReadFile(tracePath)
if err != nil {
t.Fatalf("read trace: %v", err)
}
if !strings.Contains(string(raw), `"profile":"fast"`) {
t.Fatalf("trace must record the selected profile, got %s", raw)
}
}
// An explicit --spec-reasoning-effort governs the spec draft's effort, so the
// escalation effort-restore must not arm there even when the profile filled
// the (unused) main effort.
func TestSpecProfileEffortFilled(t *testing.T) {
if !specProfileEffortFilled(true, "") {
t.Fatal("no explicit spec effort: the profile's fill governs, restore must arm")
}
if specProfileEffortFilled(true, "high") {
t.Fatal("an explicit spec effort must disarm the effort restore")
}
if specProfileEffortFilled(false, "") {
t.Fatal("nothing filled, nothing to restore")
}
}
// The trace label deliberately records the SELECTED profile, balanced
// included, so captures are self-describing. Run behavior stays identical
// (balanced fills nothing and arms no policy); the label is the one intended
// difference in the opt-in trace artifact.
func TestRunExecTraceRecordsBalancedSelection(t *testing.T) {
tracePath := filepath.Join(t.TempDir(), "trace.ndjson")
exitCode, _, stderr := runExecWithEcho(t, []string{
"exec", "--exec-profile", "balanced", "--trace", tracePath, "hello",
})
if exitCode != exitSuccess {
t.Fatalf("expected exit %d, got %d: %s", exitSuccess, exitCode, stderr)
}
raw, err := os.ReadFile(tracePath)
if err != nil {
t.Fatalf("read trace: %v", err)
}
if !strings.Contains(string(raw), `"profile":"balanced"`) {
t.Fatalf("trace must record the balanced selection, got %s", raw)
}
}
func TestRunExecUnknownExecProfileIsUsageError(t *testing.T) {
exitCode, _, stderr := runExecWithEcho(t, []string{
"exec", "--exec-profile", "turbo", "hello",
})
if exitCode != exitUsage {
t.Fatalf("expected usage exit %d, got %d", exitUsage, exitCode)
}
if !strings.Contains(stderr, "unknown execution profile") {
t.Fatalf("expected an unknown-profile usage error, got %q", stderr)
}
}
-6
View File
@@ -46,11 +46,6 @@ type execSpecDraftRun struct {
reasoningEffort string
specPermissionMode agent.PermissionMode
notifier *notify.Notifier
// profilePolicy carries the selected execution profile's escalation policy
// (nil when none): the profile displaces resolved.MaxTurns before this
// path branches off, so the spec-draft run needs the same safety net as
// the main run.
profilePolicy *agent.ProfilePolicy
}
type execSpecDraftInfo struct {
@@ -124,7 +119,6 @@ func runExecSpecDraft(run execSpecDraftRun) int {
ProviderName: run.resolved.Provider.Name,
Model: run.resolved.Provider.Model,
ReasoningEffort: run.reasoningEffort,
Profile: run.profilePolicy,
Cwd: run.workspaceRoot,
SystemPrompt: specmode.DraftSystemPrompt,
Images: run.images,
-4
View File
@@ -549,14 +549,10 @@ func TestExecMemberAutoToolListIncludesMutators(t *testing.T) {
}
func TestRunExecAcceptsLegacyModelProfileFlags(t *testing.T) {
// --profile (legacy, inert model profile) and --exec-profile (execution
// profile) are distinct flags and must coexist on one invocation.
exitCode, stdout, stderr := runExecWithEcho(t, []string{
"exec",
"--profile",
"fast",
"--exec-profile",
"balanced",
"--reasoning-effort",
"low",
"hello",
-153
View File
@@ -1,153 +0,0 @@
// Package execprofile defines named execution profiles: bundles of loop-posture
// knobs (turn budget, reasoning effort, self-correction, escalation triggers)
// that tune how hard the agent loop tries, independent of which model runs.
// Profiles COMPOSE with --mode: a mode answers "what runs" (model, tools),
// a profile answers "how hard the loop tries". Selection precedence is
// explicit flag > --mode > profile, enforced by the callers applying profiles
// last with the same fill-only-if-unset rule modes use.
//
// The catalog is deliberately tiny and value-stable: balanced is the empty
// profile (selecting it leaves the run's options, budget, and loop behavior
// identical to an unflagged run — asserted by test; only the opt-in trace and
// bench artifacts record the selected profile name, by design, so captures are
// self-describing), fast trades turn budget and effort for latency with a
// one-shot escalation back to the displaced posture as the safety net, and
// thorough raises the budget and arms full self-correction.
package execprofile
import (
"sort"
"strings"
"github.com/Gitlawb/zero/internal/agent"
"github.com/Gitlawb/zero/internal/sandbox"
)
// Profile is a named execution posture. Zero-valued fields inherit the run's
// existing value (flag, mode, config, or built-in default) — a profile only
// ever fills knobs the caller left unset, except MaxTurns which REPLACES the
// resolved budget (that displacement is what escalation restores).
type Profile struct {
// Name identifies the profile in traces, help text, and diagnostics.
Name string
// MaxTurns replaces the resolved per-run tool-turn budget when > 0 and the
// user did not pass an explicit --max-turns. The displaced resolved value
// becomes the escalation target (see Policy).
MaxTurns int
// ReasoningEffort fills the run's effort when the user left it unset. It
// still flows through the model-supported gating at the call site, so an
// unsupported level degrades exactly like an explicit flag would.
ReasoningEffort string
// SelfCorrect arms the post-edit verify-and-correct loop. Profiles can only
// turn it ON (the flag is presence-only, so false is indistinguishable from
// unset and must never override an explicit opt-in).
SelfCorrect bool
// Escalation triggers. All zero means the profile never escalates and
// Policy returns nil (the loop stays byte-identical to a nil-profile run).
// Targets are NOT part of the catalog: escalation restores the values the
// profile displaced at selection time, never invented ones, so targets are
// stamped by Policy from what the caller measured.
EscalateOnToolFailureStreak int
EscalateOnCompletionUncertain int
EscalateOnSelfCorrectFailure bool
EscalateOnRiskyMutation sandbox.RiskLevel
}
// The catalog. Values marked provisional are floors/ceilings derived from the
// Phase 0 baseline's read-class evidence (successful nav runs used single-digit
// turns; the mutating classes produced no successful samples to tune from) and
// are expected to be re-tuned from the post-oracle-fix re-capture. The
// escalation safety net is what makes shipping provisional Fast values safe:
// a run that hits trouble gets its displaced budget back mid-run.
var (
// Balanced is the default posture: empty on purpose. Selecting it changes
// nothing — the no-regression invariant of the whole feature.
Balanced = Profile{Name: "balanced"}
// Fast starts cheap (30 turns, low effort) and arms every escalation
// trigger so a struggling run restores its displaced posture: two
// same-tool retriable failures in a row, a second uncertain completion
// evaluation, a failing self-correction cycle, or a critical-risk
// mutation. The risk threshold is deliberately Critical, not High: the
// sandbox classifies EVERY shell command as at least high risk before any
// command analysis, so a High trigger would fire on the first `go test`
// of virtually any coding task and spend the one-shot on turn one.
// Critical marks the genuinely scary categories (destructive commands,
// piped installers, out-of-workspace writes, network mutations).
Fast = Profile{
Name: "fast",
MaxTurns: 30,
ReasoningEffort: "low",
EscalateOnToolFailureStreak: 2,
EscalateOnCompletionUncertain: 2,
EscalateOnSelfCorrectFailure: true,
EscalateOnRiskyMutation: sandbox.RiskCritical,
}
// Thorough doubles the default budget, asks for high effort, and arms the
// full post-edit verify-and-correct loop. Already the maximum posture, so
// it has nothing to escalate to.
Thorough = Profile{
Name: "thorough",
MaxTurns: 160,
ReasoningEffort: "high",
SelfCorrect: true,
}
)
var catalog = map[string]Profile{
Balanced.Name: Balanced,
Fast.Name: Fast,
Thorough.Name: Thorough,
}
// Lookup resolves a profile by name, case-insensitively and ignoring
// surrounding whitespace.
func Lookup(name string) (Profile, bool) {
profile, ok := catalog[strings.ToLower(strings.TrimSpace(name))]
return profile, ok
}
// Names returns the catalog's profile names, sorted, for usage errors and help.
func Names() []string {
names := make([]string, 0, len(catalog))
for name := range catalog {
names = append(names, name)
}
sort.Strings(names)
return names
}
// Policy projects the profile into the loop-facing ProfilePolicy.
//
// displacedMaxTurns is the turn budget this profile displaced at selection time
// (what the run would have used without it) and becomes the escalation's
// restore target; pass 0 when the profile did not displace the budget (e.g.
// the user pinned --max-turns explicitly) so escalation leaves the ceiling
// untouched. effortFilled reports whether the profile actually filled the
// run's reasoning effort (it backs off when the user set one explicitly); the
// displaced effort is then "" (the provider default) by construction, which
// the escalation restores via RestoreDefaultEffort — a plain "" target would
// mean "leave untouched".
//
// Profiles with no armed triggers return nil: the loop treats a nil policy as
// "no profile" (no observation, no counters), which keeps balanced and
// thorough runs byte-identical to the same knob values set by hand.
func (p Profile) Policy(displacedMaxTurns int, effortFilled bool) *agent.ProfilePolicy {
if p.EscalateOnToolFailureStreak == 0 && p.EscalateOnCompletionUncertain == 0 &&
!p.EscalateOnSelfCorrectFailure && p.EscalateOnRiskyMutation == "" {
return nil
}
return &agent.ProfilePolicy{
Name: p.Name,
Escalate: &agent.PostureEscalation{
MaxTurns: displacedMaxTurns,
RestoreDefaultEffort: effortFilled,
OnToolFailureStreak: p.EscalateOnToolFailureStreak,
OnCompletionUncertain: p.EscalateOnCompletionUncertain,
OnSelfCorrectFailure: p.EscalateOnSelfCorrectFailure,
OnRiskyMutation: p.EscalateOnRiskyMutation,
},
}
}
-131
View File
@@ -1,131 +0,0 @@
package execprofile
import (
"reflect"
"testing"
"github.com/Gitlawb/zero/internal/sandbox"
)
// The no-regression invariant of the whole feature: balanced must be the empty
// profile (nothing but a name), so selecting it cannot change a run.
func TestBalancedProfileIsEmpty(t *testing.T) {
profile, ok := Lookup("balanced")
if !ok {
t.Fatal("balanced must exist in the catalog")
}
if profile.Name != "balanced" {
t.Fatalf("Name = %q, want %q", profile.Name, "balanced")
}
profile.Name = ""
if !reflect.DeepEqual(profile, Profile{}) {
t.Fatalf("balanced must be zero-valued apart from its name, got %+v", profile)
}
if policy := Balanced.Policy(80, false); policy != nil {
t.Fatalf("balanced Policy must be nil (byte-identical loop), got %+v", policy)
}
}
func TestLookupIsCaseAndSpaceInsensitive(t *testing.T) {
for _, name := range []string{"fast", "Fast", " FAST "} {
if _, ok := Lookup(name); !ok {
t.Fatalf("Lookup(%q) should resolve", name)
}
}
if _, ok := Lookup("turbo"); ok {
t.Fatal("Lookup(turbo) should not resolve")
}
}
func TestNamesAreSorted(t *testing.T) {
want := []string{"balanced", "fast", "thorough"}
if got := Names(); !reflect.DeepEqual(got, want) {
t.Fatalf("Names() = %v, want %v", got, want)
}
}
// Escalation targets must be the displaced values only — the catalog carries
// triggers, the call site supplies what was displaced, and nothing else may
// appear in the escalation.
func TestFastPolicyTargetsDisplacedValuesOnly(t *testing.T) {
policy := Fast.Policy(80, true)
if policy == nil {
t.Fatal("fast must arm an escalation policy")
}
if policy.Name != "fast" {
t.Fatalf("policy.Name = %q, want fast", policy.Name)
}
esc := policy.Escalate
if esc == nil {
t.Fatal("fast policy must carry an escalation")
}
if esc.MaxTurns != 80 {
t.Fatalf("Escalate.MaxTurns = %d, want the displaced 80", esc.MaxTurns)
}
if esc.ReasoningEffort != "" {
t.Fatalf("Escalate.ReasoningEffort = %q, want empty (displaced effort is the provider default by construction)", esc.ReasoningEffort)
}
if !esc.RestoreDefaultEffort {
t.Fatal("effort was profile-filled, so escalation must restore the provider default")
}
if esc.RestoreCompletionGate {
t.Fatal("fast does not touch the completion gate, so escalation must not either")
}
if esc.OnToolFailureStreak != Fast.EscalateOnToolFailureStreak ||
esc.OnCompletionUncertain != Fast.EscalateOnCompletionUncertain ||
esc.OnSelfCorrectFailure != Fast.EscalateOnSelfCorrectFailure ||
esc.OnRiskyMutation != Fast.EscalateOnRiskyMutation {
t.Fatalf("escalation triggers must mirror the catalog, got %+v", esc)
}
}
// The effort restore arms only when the profile actually filled the effort:
// with an explicit user effort the profile backed off, so escalation must not
// clear the user's choice.
func TestFastPolicyExplicitEffortIsNeverRestored(t *testing.T) {
policy := Fast.Policy(80, false)
if policy == nil || policy.Escalate == nil {
t.Fatal("fast must still arm its triggers")
}
if policy.Escalate.RestoreDefaultEffort {
t.Fatal("the profile did not fill the effort, so escalation must leave it alone")
}
}
// A profile that did not displace the budget (explicit --max-turns pinned it)
// must not let escalation move the ceiling at all.
func TestFastPolicyZeroDisplacedLeavesCeilingUntouched(t *testing.T) {
policy := Fast.Policy(0, false)
if policy == nil || policy.Escalate == nil {
t.Fatal("fast must still arm its triggers with a zero displaced budget")
}
if policy.Escalate.MaxTurns != 0 {
t.Fatalf("Escalate.MaxTurns = %d, want 0 (no displaced value to restore)", policy.Escalate.MaxTurns)
}
}
// Thorough is already the maximum posture: no triggers, so no policy — the
// loop must stay byte-identical to the same knobs set by hand.
func TestThoroughPolicyIsNil(t *testing.T) {
if policy := Thorough.Policy(80, true); policy != nil {
t.Fatalf("thorough Policy must be nil, got %+v", policy)
}
}
// Pin the catalog's provisional values so a retune is a deliberate,
// test-visible diff (these floors came from the Phase 0 baseline's read-class
// evidence and are expected to move after the post-oracle-fix re-capture).
func TestCatalogProvisionalValues(t *testing.T) {
if Fast.MaxTurns != 30 || Fast.ReasoningEffort != "low" || Fast.SelfCorrect {
t.Fatalf("fast knobs changed: %+v", Fast)
}
// Critical, not High: the sandbox classifies every shell command as at
// least high risk, so a High trigger would spend the one-shot escalation
// on the first benign `go test` of any coding task.
if Fast.EscalateOnRiskyMutation != sandbox.RiskCritical {
t.Fatalf("fast risky-mutation trigger = %q, want %q", Fast.EscalateOnRiskyMutation, sandbox.RiskCritical)
}
if Thorough.MaxTurns != 160 || Thorough.ReasoningEffort != "high" || !Thorough.SelfCorrect {
t.Fatalf("thorough knobs changed: %+v", Thorough)
}
}
-26
View File
@@ -133,26 +133,6 @@ func discoverAuthorizationServer(ctx context.Context, client *http.Client, baseU
}, nil
}
// validateResolvedEndpoints applies the shared https/loopback endpoint rule to
// every non-empty endpoint in the resolved metadata — configured or discovered
// alike — so discovery metadata can never downgrade an MCP login to an insecure
// or attacker-controlled authorization, token, or registration endpoint.
func validateResolvedEndpoints(metadata authServerMetadata) error {
for _, ep := range []struct{ kind, url string }{
{"authorization", metadata.AuthorizationEndpoint},
{"token", metadata.TokenEndpoint},
{"registration", metadata.RegistrationEndpoint},
} {
if strings.TrimSpace(ep.url) == "" {
continue
}
if err := oauth.ValidateEndpointURL(ep.url); err != nil {
return fmt.Errorf("mcp oauth: %s endpoint: %w", ep.kind, err)
}
}
return nil
}
// resolveAuthorizationServer discovers metadata and applies explicit config
// overrides. Configured endpoints take precedence over discovered ones, and act
// as a fallback when discovery fails or omits a value.
@@ -168,9 +148,6 @@ func resolveAuthorizationServer(ctx context.Context, client *http.Client, baseUR
TokenEndpoint: strings.TrimSpace(cfg.TokenEndpoint),
RegistrationEndpoint: strings.TrimSpace(cfg.RegistrationEndpoint),
}
if err := validateResolvedEndpoints(metadata); err != nil {
return authServerMetadata{}, err
}
return metadata, nil
}
@@ -202,9 +179,6 @@ func resolveAuthorizationServer(ctx context.Context, client *http.Client, baseUR
if strings.TrimSpace(metadata.TokenEndpoint) == "" {
return authServerMetadata{}, errors.New("no token endpoint discovered or configured")
}
if err := validateResolvedEndpoints(metadata); err != nil {
return authServerMetadata{}, err
}
return metadata, nil
}
-16
View File
@@ -73,22 +73,6 @@ func TestResolveEndpointsFallsBackToConfig(t *testing.T) {
}
}
// Discovered MCP metadata that serves an insecure endpoint must be rejected
// before use, so discovery cannot downgrade the MCP login to an
// attacker-controlled origin (issue #511).
func TestResolveAuthorizationServerRejectsInsecureDiscovered(t *testing.T) {
mux := http.NewServeMux()
server := httptest.NewServer(mux)
defer server.Close()
mux.HandleFunc("/.well-known/oauth-authorization-server", func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(`{"issuer":"` + server.URL + `","token_endpoint":"` + server.URL + `/token","authorization_endpoint":"http://evil.example/authorize"}`))
})
_, err := resolveAuthorizationServer(context.Background(), server.Client(), server.URL, OAuthConfig{})
if err == nil || !strings.Contains(err.Error(), "authorization endpoint") {
t.Fatalf("err = %v, want insecure authorization endpoint rejection", err)
}
}
// roundTripperFunc adapts a function to http.RoundTripper so a test can
// observe every outbound request.
type roundTripperFunc func(*http.Request) (*http.Response, error)
+6 -6
View File
@@ -49,9 +49,9 @@ func RequestDeviceCode(ctx context.Context, client *http.Client, cfg Config, now
if err := validateTokenEndpoint(endpoint); err != nil {
return DeviceAuth{}, err
}
// Refuse redirects so a device-authorization redirect can't replay the
// client_id/client_secret POST body to an unvalidated origin.
client = withoutRedirects(client)
if client == nil {
client = http.DefaultClient
}
if now == nil {
now = time.Now
}
@@ -157,9 +157,9 @@ func pollDeviceOnce(ctx context.Context, client *http.Client, cfg Config, device
if err := validateTokenEndpoint(cfg.TokenEndpoint); err != nil {
return Token{}, err
}
// Refuse redirects so a poll redirect can't replay the device_code/secret
// POST body to an unvalidated origin.
client = withoutRedirects(client)
if client == nil {
client = http.DefaultClient
}
form := url.Values{}
form.Set("grant_type", deviceGrantType)
form.Set("device_code", deviceCode)
-25
View File
@@ -2,7 +2,6 @@ package oauth
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strings"
@@ -29,30 +28,6 @@ func TestRequestDeviceCodeSendsClientSecret(t *testing.T) {
}
}
func TestRequestDeviceCodeRefusesRedirect(t *testing.T) {
endpoint, hits := redirectTrap(t, http.StatusTemporaryRedirect) // 307
_, err := RequestDeviceCode(context.Background(), http.DefaultClient,
Config{ClientID: "c", ClientSecret: "shh", DeviceAuthorizationEndpoint: endpoint}, nil)
if !errors.Is(err, ErrUnsafeRedirect) {
t.Fatalf("RequestDeviceCode err = %v, want ErrUnsafeRedirect", err)
}
if n := hits(); n != 0 {
t.Fatalf("attacker received %d request(s) — client_secret replayed", n)
}
}
func TestPollDeviceOnceRefusesRedirect(t *testing.T) {
endpoint, hits := redirectTrap(t, http.StatusPermanentRedirect) // 308
_, err := pollDeviceOnce(context.Background(), http.DefaultClient,
Config{ClientID: "c", ClientSecret: "shh", TokenEndpoint: endpoint}, "device-code", nil)
if !errors.Is(err, ErrUnsafeRedirect) {
t.Fatalf("pollDeviceOnce err = %v, want ErrUnsafeRedirect", err)
}
if n := hits(); n != 0 {
t.Fatalf("attacker received %d request(s) — device_code/secret replayed", n)
}
}
func TestRequestDeviceCode(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(`{"device_code":"dc","user_code":"WXYZ-1234","verification_uri":"https://x/dev","verification_uri_complete":"https://x/dev?c=WXYZ","expires_in":600,"interval":3}`))
+8 -41
View File
@@ -39,13 +39,6 @@ func BuildAuthorizationURL(cfg Config, pkce PKCE, state, redirectURI string, ext
if endpoint == "" {
return "", errors.New("oauth: no authorization endpoint configured")
}
// Backstop: validate the authorization endpoint at the shared choke point
// both the provider and MCP flows build their browser URL through, so a
// discovery-downgraded endpoint can never open in the browser even if a
// merge site missed it.
if err := ValidateEndpointURL(endpoint); err != nil {
return "", err
}
parsed, err := url.Parse(endpoint)
if err != nil {
return "", fmt.Errorf("oauth: parse authorization endpoint: %w", err)
@@ -90,15 +83,13 @@ func isReservedAuthParam(key string) bool {
}
}
// ValidateEndpointURL refuses an OAuth endpoint that is not https, unless it is
// a loopback host (mirrors oauth2-client.js validateTokenEndpointURL). Every
// credential-bearing endpoint — configured OR learned from discovery — must
// pass this single rule, so discovery metadata can never downgrade a login to
// cleartext or redirect it to an attacker-controlled http origin.
func ValidateEndpointURL(endpoint string) error {
// validateTokenEndpoint refuses to send a credential to a token endpoint that is
// not https, unless it is a loopback host (mirrors oauth2-client.js
// validateTokenEndpointURL). This prevents leaking codes/tokens over cleartext.
func validateTokenEndpoint(endpoint string) error {
parsed, err := url.Parse(trimmed(endpoint))
if err != nil || parsed.Host == "" {
return fmt.Errorf("oauth: invalid endpoint %q", endpoint)
return fmt.Errorf("oauth: invalid token endpoint %q", endpoint)
}
if parsed.Scheme == "https" {
return nil
@@ -109,30 +100,6 @@ func ValidateEndpointURL(endpoint string) error {
return fmt.Errorf("%w: %s", ErrInsecureTokenEndpoint, parsed.Scheme+"://"+parsed.Host)
}
// validateTokenEndpoint checks a token endpoint against the shared endpoint rule
// (kept as a named helper for the token-exchange and device call sites). This
// prevents leaking codes/tokens over cleartext.
func validateTokenEndpoint(endpoint string) error {
return ValidateEndpointURL(endpoint)
}
// withoutRedirects returns a shallow copy of client whose redirect policy
// refuses to follow ANY redirect. A credential-bearing OAuth POST (token
// exchange/refresh, device authorization, device-token poll) must never let a
// 307/308 replay its form body — codes, PKCE verifiers, refresh tokens, client
// secrets — to a target that was never endpoint-validated. The caller's client
// is left unmodified.
func withoutRedirects(client *http.Client) *http.Client {
if client == nil {
client = http.DefaultClient
}
copied := *client
copied.CheckRedirect = func(*http.Request, []*http.Request) error {
return ErrUnsafeRedirect
}
return &copied
}
func isLoopbackHost(host string) bool {
if host == "localhost" {
return true
@@ -193,9 +160,9 @@ func PostToken(ctx context.Context, client *http.Client, tokenEndpoint string, f
if err := validateTokenEndpoint(tokenEndpoint); err != nil {
return Token{}, err
}
// Refuse redirects so a 307/308 from the token endpoint can't replay this
// credential-bearing POST body to another (unvalidated) origin.
client = withoutRedirects(client)
if client == nil {
client = http.DefaultClient
}
if now == nil {
now = time.Now
}
-74
View File
@@ -5,12 +5,10 @@ import (
"crypto/sha256"
"encoding/base64"
"errors"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync/atomic"
"testing"
"time"
)
@@ -114,78 +112,6 @@ func TestBuildAuthorizationURLRejectsPlain(t *testing.T) {
}
}
// The shared choke point both flows build their browser URL through must refuse
// an insecure authorization endpoint, so a discovery-downgraded endpoint can
// never open in the browser even if a merge site missed it (issue #511).
func TestBuildAuthorizationURLRejectsInsecureEndpoint(t *testing.T) {
_, err := BuildAuthorizationURL(
Config{AuthorizationEndpoint: "http://evil.example/authorize", ClientID: "c"},
PKCE{Method: MethodS256, Challenge: "chal"},
"state", "http://127.0.0.1/cb", nil,
)
if !errors.Is(err, ErrInsecureTokenEndpoint) {
t.Fatalf("err = %v, want ErrInsecureTokenEndpoint", err)
}
}
// redirectTrap starts an "attacker" server that counts any hit, and a credential
// endpoint that redirects to it with the given status. It returns the endpoint
// URL and a func reporting how many times the attacker was contacted — a
// non-zero count means the credential-bearing POST body was replayed (#729).
func redirectTrap(t *testing.T, status int) (endpoint string, attackerHits func() int32) {
t.Helper()
var hits int32
attacker := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
atomic.AddInt32(&hits, 1)
_, _ = io.WriteString(w, `{"access_token":"leaked","refresh_token":"leaked"}`)
}))
t.Cleanup(attacker.Close)
endp := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, attacker.URL, status)
}))
t.Cleanup(endp.Close)
return endp.URL, func() int32 { return atomic.LoadInt32(&hits) }
}
func TestPostTokenRefusesRedirect(t *testing.T) {
endpoint, hits := redirectTrap(t, http.StatusTemporaryRedirect) // 307
form := url.Values{}
form.Set("grant_type", "authorization_code")
form.Set("code", "the-code")
form.Set("client_secret", "shh")
_, err := PostToken(context.Background(), http.DefaultClient, endpoint, form, Token{}, nil)
if !errors.Is(err, ErrUnsafeRedirect) {
t.Fatalf("PostToken err = %v, want ErrUnsafeRedirect", err)
}
if n := hits(); n != 0 {
t.Fatalf("attacker received %d request(s) — credential body was replayed", n)
}
}
func TestExchangeCodeRefusesRedirect(t *testing.T) {
endpoint, hits := redirectTrap(t, http.StatusPermanentRedirect) // 308
_, err := ExchangeCode(context.Background(), http.DefaultClient,
Config{ClientID: "c", TokenEndpoint: endpoint}, "the-code", "verifier", "http://127.0.0.1/cb", nil)
if !errors.Is(err, ErrUnsafeRedirect) {
t.Fatalf("ExchangeCode err = %v, want ErrUnsafeRedirect", err)
}
if n := hits(); n != 0 {
t.Fatalf("attacker received %d request(s) — code/verifier replayed", n)
}
}
func TestRefreshRefusesRedirect(t *testing.T) {
endpoint, hits := redirectTrap(t, http.StatusTemporaryRedirect) // 307
_, err := Refresh(context.Background(), http.DefaultClient,
Config{ClientID: "c", TokenEndpoint: endpoint}, Token{RefreshToken: "rt"}, nil)
if !errors.Is(err, ErrUnsafeRedirect) {
t.Fatalf("Refresh err = %v, want ErrUnsafeRedirect", err)
}
if n := hits(); n != 0 {
t.Fatalf("attacker received %d request(s) — refresh_token replayed", n)
}
}
func TestValidateTokenEndpoint(t *testing.T) {
cases := map[string]bool{
"https://auth.example.com/token": true,
+3 -16
View File
@@ -218,26 +218,13 @@ func (m *Manager) resolveEndpoints(ctx context.Context, cfg Config) (Config, err
// may already be sufficient for the chosen flow. Only merge on success (and
// never return from the error branch, which would trip the nilerr linter).
if meta, err := m.discover(ctx, cfg.IssuerURL); err == nil {
// A discovered endpoint must pass the same https/loopback rule as a
// configured one before it is merged, so discovery metadata can never
// silently downgrade the login to an insecure or attacker-controlled
// endpoint. Fail closed on a bad endpoint rather than merging it.
if cfg.AuthorizationEndpoint == "" && meta.AuthorizationEndpoint != "" {
if err := ValidateEndpointURL(meta.AuthorizationEndpoint); err != nil {
return cfg, fmt.Errorf("oauth: discovered authorization endpoint: %w", err)
}
if cfg.AuthorizationEndpoint == "" {
cfg.AuthorizationEndpoint = meta.AuthorizationEndpoint
}
if cfg.TokenEndpoint == "" && meta.TokenEndpoint != "" {
if err := ValidateEndpointURL(meta.TokenEndpoint); err != nil {
return cfg, fmt.Errorf("oauth: discovered token endpoint: %w", err)
}
if cfg.TokenEndpoint == "" {
cfg.TokenEndpoint = meta.TokenEndpoint
}
if cfg.DeviceAuthorizationEndpoint == "" && meta.DeviceAuthorizationEndpoint != "" {
if err := ValidateEndpointURL(meta.DeviceAuthorizationEndpoint); err != nil {
return cfg, fmt.Errorf("oauth: discovered device authorization endpoint: %w", err)
}
if cfg.DeviceAuthorizationEndpoint == "" {
cfg.DeviceAuthorizationEndpoint = meta.DeviceAuthorizationEndpoint
}
}
-19
View File
@@ -2,7 +2,6 @@ package oauth
import (
"context"
"errors"
"io"
"net/http"
"net/http/httptest"
@@ -201,24 +200,6 @@ func TestManagerGetFreshDiscoversIssuerEndpoints(t *testing.T) {
}
}
// A discovery document that serves an insecure (non-https, non-loopback)
// endpoint must be rejected before it is merged, so discovery metadata cannot
// silently downgrade the login to an attacker-controlled origin (issue #511).
func TestResolveEndpointsRejectsInsecureDiscoveredEndpoint(t *testing.T) {
mux := http.NewServeMux()
server := httptest.NewServer(mux)
defer server.Close()
mux.HandleFunc("/.well-known/oauth-authorization-server", func(w http.ResponseWriter, _ *http.Request) {
// Loopback issuer, but an attacker-controlled http authorization endpoint.
_, _ = io.WriteString(w, `{"issuer":"`+server.URL+`","token_endpoint":"`+server.URL+`/token","authorization_endpoint":"http://evil.example/authorize"}`)
})
m := managerFor(t, map[string]string{}, nil)
_, err := m.resolveEndpoints(context.Background(), Config{IssuerURL: server.URL})
if !errors.Is(err, ErrInsecureTokenEndpoint) {
t.Fatalf("resolveEndpoints err = %v, want ErrInsecureTokenEndpoint", err)
}
}
func TestManagerGetFreshRefreshesExpired(t *testing.T) {
fp := newFakeProvider(t, `{"access_token":"fresh-at","expires_in":3600}`)
env := map[string]string{
-5
View File
@@ -90,11 +90,6 @@ var (
// ErrInsecureTokenEndpoint is returned when a credential would be sent over a
// non-https, non-loopback endpoint.
ErrInsecureTokenEndpoint = errors.New("oauth: refusing to send credential to a non-https token endpoint")
// ErrUnsafeRedirect is returned when a credential-bearing OAuth POST (token
// exchange/refresh, device authorization, or device-token poll) is redirected.
// Following it would replay the form body — codes, PKCE verifiers, refresh
// tokens, client secrets — to a target that never passed endpoint validation.
ErrUnsafeRedirect = errors.New("oauth: refusing to follow a redirect from a credential endpoint")
// ErrNoRefreshToken is returned when a refresh is attempted without one.
ErrNoRefreshToken = errors.New("oauth: no refresh token available")
// ErrAuthorizationPending is the RFC 8628 "keep polling" signal.
+4 -4
View File
@@ -133,7 +133,7 @@
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep --exclude=.zero-answer.txt -RIn 'MaxRetries =' ."
"go test ./... && ! grep -RIn 'MaxRetries =' ."
],
"oracleTest": "package edit\n\nvar _ = RetryLimit\n"
},
@@ -146,7 +146,7 @@
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep --exclude=.zero-answer.txt -R receieves . && grep --exclude=.zero-answer.txt -R receives ."
"go test ./... && ! grep -R receieves . && grep -R receives ."
]
},
{
@@ -400,7 +400,7 @@
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep --exclude=.zero-answer.txt -R 'package refactor' . && grep --exclude=.zero-answer.txt -R 'package zeroapp' ."
"go test ./... && ! grep -R 'package refactor' . && grep -R 'package zeroapp' ."
],
"oracleTest": "package zeroapp\n\nvar _ = Config{}\n"
},
@@ -426,7 +426,7 @@
"verificationCommand": [
"bash",
"-c",
"go test ./... && ! grep --exclude=.zero-answer.txt -R 'func Wrapper' ."
"go test ./... && ! grep -R 'func Wrapper' ."
],
"oracleTest": "package refactor\n\nimport \"testing\"\n\n// TestGreetWrapped proves greetWrapped still exists and behaves as the inlined\n// Wrapper did: deleting both Wrapper and greetWrapped (or stubbing greetWrapped\n// to return \"\") fails to compile or run, so the oracle can't be passed by\n// removing the target along with its caller. Paired with the ! grep 'func\n// Wrapper' verificationCommand that proves Wrapper itself is gone.\nfunc TestGreetWrapped(t *testing.T) {\n\tif got := greetWrapped(\"x\"); got != \"hello, x\" {\n\t\tt.Fatalf(\"greetWrapped(x) = %q, want %q\", got, \"hello, x\")\n\t}\n}\n"
},
-4
View File
@@ -85,10 +85,6 @@ type RunContext struct {
Model string
Mode string
SelfCorrect bool
// ExecProfile, when non-empty, is forwarded to the agent invocation as
// --exec-profile (turn benchmark runner; the task benchmark leaves it
// empty).
ExecProfile string
}
// TaskRunner runs a single benchmark task and returns its outcome.
+3 -80
View File
@@ -15,7 +15,6 @@ import (
"strings"
"time"
"github.com/Gitlawb/zero/internal/execprofile"
"github.com/Gitlawb/zero/internal/trace"
)
@@ -41,25 +40,7 @@ import (
// prevents a v2→v2 cross-version comparison from misreading the jump as a model
// improvement. New counts: correctness 34 (edit 10 + fix 8 + nav 10 + refactor
// 6), build 0, latency 14 (longproc 4 + longctx 4 + parallel 6).
//
// v4 adds tasksErrored: the number of tasks whose every iteration produced no
// accepted benchmark sample. That covers pre-run failures (missing binary,
// spawn error, process crash) and harness errors after a successful agent run
// (e.g. oracle stamping failed) alike — in every case the iteration yields no
// usable measurement. Errored tasks were previously visible only in warnings,
// so a run where nothing was actually measured still printed a clean-looking
// 0% pass summary and exited 0. tasksErrored makes that state first-class:
// the summary surfaces it, and the turn command exits nonzero when every
// attempted task errored. Accounting: an errored oracle task stays in its
// tier denominator as a failure; an errored latency-only task counts in
// latencyOnlyTasks and, as always, in no pass rate.
//
// v5 is ADDITIVE ONLY: execProfile records the execution profile the run was
// benchmarked under (omitted when none was selected), so profile A/B captures
// are self-describing and two reports can't be compared without noticing they
// ran different postures. No existing field changed shape or meaning; a v4
// consumer reading a v5 report misses only the new field.
const TurnSchemaVersion = 5
const TurnSchemaVersion = 3
// TurnRunner runs one benchmark task and reports its outcome plus the captured
// per-turn trace. A non-nil Err means the run failed to execute (process crash);
@@ -86,10 +67,6 @@ type TurnBenchConfig struct {
Model string
Mode string
SelfCorrect bool
// ExecProfile, when non-empty, runs every task under the named execution
// profile (zero exec --exec-profile) and stamps it into the result so the
// report is self-describing for profile A/B comparisons.
ExecProfile string
Version string
Commit string
// Iterations is how many times each task is run. The per-process `zero exec`
@@ -162,13 +139,11 @@ type TurnBenchResult struct {
Suite string `json:"suite"`
Model string `json:"model"`
Mode string `json:"mode,omitempty"`
ExecProfile string `json:"execProfile,omitempty"`
SelfCorrect bool `json:"selfCorrect"`
Version string `json:"version,omitempty"`
Commit string `json:"commit,omitempty"`
Date string `json:"date"`
TasksAttempted int `json:"tasksAttempted"`
TasksErrored int `json:"tasksErrored"`
TasksVerified int `json:"tasksVerified"`
TasksPassed int `json:"tasksPassed"`
LatencyOnlyTasks int `json:"latencyOnlyTasks"`
@@ -214,19 +189,6 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
if cfg.Runner == nil {
return TurnBenchResult{}, errors.New("turn benchmark requires a runner")
}
// Canonicalize the profile once at the boundary so every consumer — the
// child exec args, the stamped result, the summary — carries the same
// catalog name regardless of the caller's casing/whitespace, and a direct
// library caller cannot slip an unknown name into a report the way the CLI
// (which validates at parse time) cannot.
benchProfile := strings.TrimSpace(cfg.ExecProfile)
if benchProfile != "" {
profile, ok := execprofile.Lookup(benchProfile)
if !ok {
return TurnBenchResult{}, fmt.Errorf("unknown execution profile %q (valid: %s)", cfg.ExecProfile, strings.Join(execprofile.Names(), ", "))
}
benchProfile = profile.Name
}
iterations := cfg.Iterations
if iterations < 1 {
iterations = 1
@@ -235,7 +197,7 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
if now == nil {
now = time.Now
}
rc := RunContext{Model: cfg.Model, Mode: cfg.Mode, SelfCorrect: cfg.SelfCorrect, ExecProfile: benchProfile}
rc := RunContext{Model: cfg.Model, Mode: cfg.Mode, SelfCorrect: cfg.SelfCorrect}
perSpanSamples := map[string][]float64{}
classWalls := map[string][]float64{}
@@ -265,7 +227,6 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
Suite: strings.TrimSpace(set.ID),
Model: strings.TrimSpace(cfg.Model),
Mode: strings.TrimSpace(cfg.Mode),
ExecProfile: benchProfile,
SelfCorrect: cfg.SelfCorrect,
Version: strings.TrimSpace(cfg.Version),
Commit: strings.TrimSpace(cfg.Commit),
@@ -288,7 +249,6 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
// runner is cold-start, so a flaky pass on one iteration and a fail on
// another is a real regression signal, not noise to average away.
passedForTask := true
erroredIterations := 0
for iter := 0; iter < iterations; iter++ {
outcome := cfg.Runner(ctx, task, rc)
if outcome.TraceIssue != "" {
@@ -306,7 +266,6 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
Message: fmt.Sprintf("task %s: %v", task.ID, outcome.Err),
})
passedForTask = false
erroredIterations++
continue
}
if !outcome.Passed {
@@ -347,14 +306,6 @@ func RunTurnBench(ctx context.Context, set TaskSet, cfg TurnBenchConfig) (TurnBe
// counters. A latency-only task (no verificationCommand) is never counted
// in any pass rate even when the runner reports Passed — an exit-0
// read-only run proves the turn ran, not that the answer was right.
if erroredIterations == iterations {
// Every iteration produced no accepted sample (spawn failure or a
// harness error after the run). The task still counts in its tier
// below — an oracle task as a failure, a latency-only task in no
// pass rate as always — but the errored state is first-class so a
// broken run can never print a clean-looking summary.
result.TasksErrored++
}
hasOracle := len(task.VerificationCommand) > 0
switch {
case !hasOracle:
@@ -505,26 +456,9 @@ func FormatTurnBenchSummary(result TurnBenchResult) string {
result.BuildPassedTasks, result.BuildCheckedTasks, result.BuildPassRate*100,
result.LatencyOnlyTasks, result.Iterations),
}
if result.TasksErrored > 0 {
lines = append(lines, fmt.Sprintf("ERRORED: %d task(s) produced no accepted benchmark sample (spawn/crash or harness error) — errored oracle tasks count as failures in their tier's pass rate; latency-only tasks stay out of pass rates as always", result.TasksErrored))
shown := 0
for _, warning := range result.Warnings {
if warning.Metric != "run" {
continue
}
lines = append(lines, " "+warning.Message)
shown++
if shown == 3 {
break
}
}
}
if result.Mode != "" {
lines = append(lines, "mode: "+result.Mode)
}
if result.ExecProfile != "" {
lines = append(lines, "exec-profile: "+result.ExecProfile)
}
if len(result.TopLatency) > 0 {
lines = append(lines, "top latency sources:")
for _, src := range result.TopLatency {
@@ -800,9 +734,6 @@ func buildTurnExecArgs(task BenchTask, rc RunContext, tracePath string, extraArg
if rc.SelfCorrect {
args = append(args, "--self-correct")
}
if profile := strings.TrimSpace(rc.ExecProfile); profile != "" {
args = append(args, "--exec-profile", profile)
}
args = append(args, extraArgs...)
args = append(args, task.Prompt)
return args
@@ -816,15 +747,7 @@ func ResolveBinary(explicit string) (string, error) {
if _, err := os.Stat(v); err != nil {
return "", fmt.Errorf("trace binary not found: %w", err)
}
// Pin the explicit path to an absolute one: the turn runner sets each
// child's cmd.Dir to a per-task fixture copy, so a relative path
// (./zero, the Makefile default) that stats fine here would fail to
// spawn from inside every fixture dir, silently erroring all tasks.
absolute, err := filepath.Abs(v)
if err != nil {
return "", fmt.Errorf("resolve trace binary path: %w", err)
}
return absolute, nil
return v, nil
}
if path, err := exec.LookPath("zero"); err == nil {
return path, nil
+8 -196
View File
@@ -4,7 +4,6 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
@@ -531,17 +530,14 @@ func TestRunTurnBenchBuildOnlyMechanism(t *testing.T) {
}
}
// TestTurnSchemaVersion pins the schema bump so a shape change is a conscious
// decision. v3 recorded the tier reclassification (refactor structural-positive
// and nav answer-oracles moved into correctnessPassRate). v4 adds tasksErrored:
// tasks whose every iteration died before the agent produced a run are now
// first-class in the report instead of visible only in warnings, so a
// spawn-broken run cannot print a clean-looking summary or exit 0. v5 is
// additive only: execProfile stamps the execution profile the run was
// benchmarked under so profile A/B reports are self-describing.
func TestTurnSchemaVersion(t *testing.T) {
if TurnSchemaVersion != 5 {
t.Fatalf("TurnSchemaVersion = %d, want 5", TurnSchemaVersion)
// TestTurnSchemaVersion3 pins the schema bump. v3 records the tier
// reclassification (refactor structural-positive and nav answer-oracles moved
// into correctnessPassRate), so a v2->v3 cross-version comparison cannot
// silently misread the jump as a model improvement — exactly the misread the
// tier system exists to prevent.
func TestTurnSchemaVersion3(t *testing.T) {
if TurnSchemaVersion != 3 {
t.Fatalf("TurnSchemaVersion = %d, want 3", TurnSchemaVersion)
}
}
@@ -1071,187 +1067,3 @@ printf '%s\n' '{"type":"run_end","exitCode":0}'
`)
assertVerifyFailed(t, "nav-08 fmt-only-no-testing", outcome)
}
func TestResolveBinaryAbsolutizesExplicitPath(t *testing.T) {
dir := t.TempDir()
name := "zero-probe.exe"
if err := os.WriteFile(filepath.Join(dir, name), []byte("x"), 0o755); err != nil {
t.Fatal(err)
}
cwd, err := os.Getwd()
if err != nil {
t.Fatal(err)
}
if err := os.Chdir(dir); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = os.Chdir(cwd) })
resolved, err := ResolveBinary("./" + name)
if err != nil {
t.Fatalf("ResolveBinary: %v", err)
}
if !filepath.IsAbs(resolved) {
t.Fatalf("ResolveBinary returned relative path %q; the turn runner sets cmd.Dir per task, so a relative binary fails to spawn from every fixture copy", resolved)
}
}
func TestRunTurnBenchCountsErroredTasks(t *testing.T) {
set := TaskSet{
ID: "errored-suite",
Tasks: []BenchTask{
{ID: "t1", Class: "nav", Prompt: "p1", VerificationCommand: []string{"true"}},
{ID: "t2", Class: "longproc", Prompt: "p2"},
},
}
cfg := TurnBenchConfig{
Model: "fake-model",
Iterations: 1,
Runner: func(context.Context, BenchTask, RunContext) TurnTaskOutcome {
return TurnTaskOutcome{Err: errors.New("fork/exec ./zero: file does not exist")}
},
Now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
}
result, err := RunTurnBench(context.Background(), set, cfg)
if err != nil {
t.Fatalf("RunTurnBench: %v", err)
}
if result.TasksErrored != 2 {
t.Fatalf("TasksErrored=%d, want 2 (every iteration of both tasks errored)", result.TasksErrored)
}
if len(result.Warnings) != 2 {
t.Fatalf("warnings=%d, want 2 run warnings", len(result.Warnings))
}
summary := FormatTurnBenchSummary(result)
if !strings.Contains(summary, "ERRORED: 2 task(s)") {
t.Fatalf("summary does not surface the errored tasks:\n%s", summary)
}
if !strings.Contains(summary, "fork/exec ./zero") {
t.Fatalf("summary does not echo the underlying run error:\n%s", summary)
}
}
func TestRunTurnBenchPartialErrorStillCounts(t *testing.T) {
set := TaskSet{
ID: "partial-suite",
Tasks: []BenchTask{
{ID: "ok", Class: "nav", Prompt: "p", VerificationCommand: []string{"true"}},
{ID: "dead", Class: "nav", Prompt: "p", VerificationCommand: []string{"true"}},
},
}
canned := map[string]*trace.TurnTrace{"ok": cannedTrace(100, 10, 1000)}
inner := fakeTurnRunner(canned)
cfg := TurnBenchConfig{
Model: "fake-model",
Iterations: 1,
Runner: func(ctx context.Context, task BenchTask, rc RunContext) TurnTaskOutcome {
if task.ID == "dead" {
return TurnTaskOutcome{Err: errors.New("spawn failed")}
}
return inner(ctx, task, rc)
},
Now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
}
result, err := RunTurnBench(context.Background(), set, cfg)
if err != nil {
t.Fatalf("RunTurnBench: %v", err)
}
if result.TasksErrored != 1 {
t.Fatalf("TasksErrored=%d, want 1", result.TasksErrored)
}
if result.TasksAttempted != 2 {
t.Fatalf("TasksAttempted=%d, want 2", result.TasksAttempted)
}
}
// The exec-profile passthrough: the profile must reach every task's zero exec
// invocation as --exec-profile (with the prompt staying last) and stay out of
// the args entirely when unset.
func TestBuildTurnExecArgsIncludesExecProfile(t *testing.T) {
task := BenchTask{ID: "t", Prompt: "do the thing"}
args := buildTurnExecArgs(task, RunContext{Model: "m", ExecProfile: "fast"}, "trace.ndjson", nil)
found := false
for i := 0; i < len(args)-1; i++ {
if args[i] == "--exec-profile" && args[i+1] == "fast" {
found = true
break
}
}
if !found {
t.Fatalf("args must carry --exec-profile fast, got %v", args)
}
if args[len(args)-1] != "do the thing" {
t.Fatalf("prompt must stay the last argument, got %v", args)
}
args = buildTurnExecArgs(task, RunContext{Model: "m"}, "trace.ndjson", nil)
for _, arg := range args {
if arg == "--exec-profile" {
t.Fatalf("no profile configured, but args carry --exec-profile: %v", args)
}
}
}
// The configured profile must reach the runner's RunContext and be stamped
// into the result, so a profile A/B report is self-describing. The boundary
// canonicalizes (case/whitespace) so equivalent postures always carry the same
// label, and rejects unknown names so a direct library caller cannot slip an
// unvalidated profile into a report the way the CLI (parse-time check) cannot.
func TestRunTurnBenchStampsExecProfile(t *testing.T) {
set := TaskSet{
ID: "profile-suite",
Tasks: []BenchTask{{ID: "t1", Class: "longproc", Prompt: "p"}},
}
var gotProfile string
cfg := TurnBenchConfig{
Model: "fake-model",
ExecProfile: " FAST ",
Iterations: 1,
Runner: func(_ context.Context, _ BenchTask, rc RunContext) TurnTaskOutcome {
gotProfile = rc.ExecProfile
return TurnTaskOutcome{Passed: true, WallMs: 10, Trace: cannedTrace(10, 1, 100)}
},
Now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
}
result, err := RunTurnBench(context.Background(), set, cfg)
if err != nil {
t.Fatalf("RunTurnBench: %v", err)
}
if gotProfile != "fast" {
t.Fatalf("runner RunContext.ExecProfile = %q, want the canonical fast", gotProfile)
}
if result.ExecProfile != "fast" {
t.Fatalf("result.ExecProfile = %q, want the canonical fast", result.ExecProfile)
}
raw, err := json.Marshal(result)
if err != nil {
t.Fatalf("marshal result: %v", err)
}
if !strings.Contains(string(raw), `"execProfile":"fast"`) {
t.Fatal("published JSON must carry execProfile")
}
}
func TestRunTurnBenchRejectsUnknownExecProfile(t *testing.T) {
set := TaskSet{
ID: "profile-suite",
Tasks: []BenchTask{{ID: "t1", Class: "longproc", Prompt: "p"}},
}
ran := false
cfg := TurnBenchConfig{
Model: "fake-model",
ExecProfile: "blanced",
Iterations: 1,
Runner: func(context.Context, BenchTask, RunContext) TurnTaskOutcome {
ran = true
return TurnTaskOutcome{Passed: true, WallMs: 10, Trace: cannedTrace(10, 1, 100)}
},
}
_, err := RunTurnBench(context.Background(), set, cfg)
if err == nil || !strings.Contains(err.Error(), "unknown execution profile") {
t.Fatalf("err = %v, want an unknown-profile rejection", err)
}
if ran {
t.Fatal("nothing may run under an unknown profile")
}
}
-6
View File
@@ -57,11 +57,6 @@ const (
CounterPrewarmAttempts = "prewarm_attempts"
CounterPrefixStable = "prefix_stable"
CounterPrefixDrift = "prefix_drift"
// CounterPostureEscalations counts one-shot in-run posture escalations by
// the execution-profile controller. Deliberately distinct from
// model_switches: a posture escalation changes loop policy knobs, never the
// model or session.
CounterPostureEscalations = "posture_escalations"
)
// Span is one named wall interval attributed to part of a run. Each stamp is
@@ -294,7 +289,6 @@ func OptionalEventKeys() []string {
"counter:" + CounterPrewarmAttempts,
"counter:" + CounterPrefixStable,
"counter:" + CounterPrefixDrift,
"counter:" + CounterPostureEscalations,
"event:prefix_hash",
}
}
-9
View File
@@ -577,12 +577,3 @@ func TestCounterPrecisionRoundTrip(t *testing.T) {
t.Fatalf("counter round-trip = %d, want %d (float64 precision loss)", got, big)
}
}
func TestOptionalEventKeysIncludePostureEscalations(t *testing.T) {
for _, key := range OptionalEventKeys() {
if key == "counter:"+CounterPostureEscalations {
return
}
}
t.Fatalf("posture_escalations missing from OptionalEventKeys: %v", OptionalEventKeys())
}
+7 -16
View File
@@ -464,15 +464,13 @@ func (m model) handleModelCommand(args string) (model, string) {
// dedupe against the entry just persisted here, showing the same switch twice.
config.RecentModelEntry{Provider: m.providerName, Model: target.modelID},
)
// Drop a known-unsupported preference, void any profile bookkeeping the
// drop erased, and re-derive an active profile's per-model effort fill.
// The ring is authoritative only for catalog-resolved targets
// (target.entry non-nil); live-discovered/custom targets carry no support
// knowledge, so an explicit preference survives onto them. resetEffort
// reports a dropped preference nothing refilled (shown as a reset to
// auto).
var resetEffort bool
m, resetEffort = m.reconcileEffortForModelSwitch(target.reasoningEfforts, target.entry != nil)
resetEffort := false
if m.reasoningEffort != "" && !reasoningEffortAllowed(target.reasoningEfforts, m.reasoningEffort) {
// Drop an unsupported carry-over preference and fall back to the
// model's effective default for the new model.
m.reasoningEffort = ""
resetEffort = true
}
effortLine := "effort: " + m.effortDisplay()
if resetEffort {
// Preference was dropped: show "auto" (model default applies), not a
@@ -553,13 +551,6 @@ func (m model) switchProviderModel(providerName, modelID string) (model, string,
m.providerProfile = target
m.providerName = target.Name
m.modelName = target.Model
// An active profile's effort fill is per-model: re-derive it for the
// destination, exactly like handleModelCommand does. No generic
// unsupported-drop here: cross-provider targets are often custom models
// the catalog cannot vouch for either way, so an explicit preference is
// carried (pre-existing behavior) while the profile's own fill stays
// conservative — it only ever applies where support is known.
m = m.reconcileProfileAfterModelSwitch(m.availableReasoningEfforts())
// Record the outgoing pair too — see the matching comment in
// handleModelCommand for why (keeps the session's starting model from
// silently dropping out of "Recent" on the first switch away from it).
-8
View File
@@ -41,7 +41,6 @@ const (
commandAddDir
commandSelfCorrect
commandTurns
commandProfile
commandRetry
commandEdit
commandCopy
@@ -287,13 +286,6 @@ var commandDefinitions = []commandDefinition{
description: "Show or set the per-run tool-turn budget for this session (raise it for long multi-step tasks).",
kind: commandTurns,
},
{
name: "/profile",
usage: "/profile [status|balanced|fast|thorough]",
group: commandGroupSession,
description: "Show or switch the execution profile for the next run (loop posture: turn budget, effort, self-correction, escalation; model selection is unchanged).",
kind: commandProfile,
},
{
name: "/retry",
usage: "/retry",
+14 -42
View File
@@ -128,36 +128,20 @@ type model struct {
permissionMode agent.PermissionMode
selfCorrectTests bool
reasoningEffort modelregistry.ReasoningEffort
// Active execution profile (set by /profile; applies to the NEXT run).
// The displaced/applied pairs let a switch or /profile balanced restore
// exactly what the profile replaced while leaving later manual overrides
// (/turns, Ctrl+T, /selfcorrect) alone: each knob is only reverted when it
// still holds the value the profile applied.
execProfileName string
execProfileDisplacedMaxTurns int
execProfileAppliedMaxTurns int
execProfileAppliedEffort modelregistry.ReasoningEffort
execProfileArmedSelfCorrect bool
// The touched bits record explicit user choices made while a profile is
// active (/turns, /effort, Ctrl+T, /selfcorrect); a touched knob is never
// reverted, even when its value coincides with what the profile applied.
execProfileTurnsTouched bool
execProfileEffortTouched bool
execProfileSelfCorrectTouched bool
responseStyle string
keyBindings keyBindings
themeMode themeMode // palette preference: auto (default), dark, light
hasDarkBg bool // last terminal background-detection result (auto mode)
userAgent string
compactRequests int
compactInFlight bool
compactFrame int
lastCompactResult *CompactResult
lastCompactError string
unpricedRequests int
unpricedTokens int
lastUsage usage.Normalized
lastUsageSeen bool
responseStyle string
keyBindings keyBindings
themeMode themeMode // palette preference: auto (default), dark, light
hasDarkBg bool // last terminal background-detection result (auto mode)
userAgent string
compactRequests int
compactInFlight bool
compactFrame int
lastCompactResult *CompactResult
lastCompactError string
unpricedRequests int
unpricedTokens int
lastUsage usage.Normalized
lastUsageSeen bool
// turnLatencySum / turnLatencyCount accumulate completed-run wall time so
// /context can show a rolling average turn latency (the "is it slow?" signal).
// Reset by /new.
@@ -4362,18 +4346,6 @@ func (m model) dispatchCommand(command parsedCommand) (tea.Model, tea.Cmd) {
m, text = m.handleTurnsCommand(command.text)
m.transcript = reduceTranscript(m.transcript, transcriptAction{kind: actionAppendSystem, text: text})
return m, nil
case commandProfile:
// Same idle-session rule as /turns: switching the profile mutates the
// turn budget (and its ZERO_MAX_TURNS propagation), so a change needs
// an idle session; bare /profile (status) is always allowed.
if m.pending && strings.TrimSpace(command.text) != "" && !strings.EqualFold(strings.TrimSpace(command.text), "status") {
m.transcript = reduceTranscript(m.transcript, transcriptAction{kind: actionAppendSystem, text: "Profile\nFinish or stop the current run before switching the execution profile."})
return m, nil
}
text := ""
m, text = m.handleProfileCommand(command.text)
m.transcript = reduceTranscript(m.transcript, transcriptAction{kind: actionAppendSystem, text: text})
return m, nil
case commandTheme:
// Bare `/theme` opens the popup picker (live preview on move, apply on
// Enter), matching /model and /effort. An explicit `/theme auto|dark|light`
-432
View File
@@ -1,432 +0,0 @@
package tui
import (
"context"
"os"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/config"
"github.com/Gitlawb/zero/internal/modelregistry"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
// profileSwitchModel builds a session on a model whose catalog entry supports
// the fast profile's "low" effort, with a second saved provider whose custom
// model has no catalog entry (so no effort ring) — the two directions the
// model-switch reconciliation must handle.
func profileSwitchModel(t *testing.T) model {
t.Helper()
return newModel(context.Background(), Options{
ProviderName: "anthropic",
ModelName: "claude-sonnet-4.5",
Provider: &fakeProvider{},
ProviderProfile: config.ProviderProfile{Name: "anthropic", CatalogID: "anthropic", Model: "claude-sonnet-4.5", APIKey: "k"},
SavedProviders: []config.ProviderProfile{
{Name: "anthropic", CatalogID: "anthropic", Model: "claude-sonnet-4.5", APIKey: "k"},
{Name: "ollama", CatalogID: "ollama", ProviderKind: config.ProviderKindOpenAICompatible, BaseURL: "http://localhost:11434/v1", Model: "kimi-k2.7-code:cloud"},
},
NewProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
return &fakeProvider{}, nil
},
})
}
// The profile's effort fill is per-model and must be re-derived on a model
// switch: dropped when the destination does not support the level, refilled
// when a later destination does, with the escalation's effort restore tracking
// whether the profile currently governs the effort.
func TestProfileEffortReconciledOnModelSwitch(t *testing.T) {
m := profileSwitchModel(t)
m, _ = m.handleProfileCommand("fast")
if m.reasoningEffort != "low" || m.agentOptions.Profile == nil || !m.agentOptions.Profile.Escalate.RestoreDefaultEffort {
t.Fatalf("fast on a low-supporting model must fill low and arm the restore, got effort %q", m.reasoningEffort)
}
// Supported -> unsupported: the profile-applied level must not survive
// onto a model with no effort ring.
m, text, ok, _ := m.switchProviderModel("ollama", "kimi-k2.7-code:cloud")
if !ok {
t.Fatalf("switch to ollama failed: %q", text)
}
if m.reasoningEffort != "" || m.execProfileAppliedEffort != "" {
t.Fatalf("effort = %q applied = %q, want both cleared on an unsupported destination", m.reasoningEffort, m.execProfileAppliedEffort)
}
if m.agentOptions.Profile.Escalate.RestoreDefaultEffort {
t.Fatal("the profile no longer governs the effort, so the restore must disarm")
}
if m.execProfileName != "fast" || m.agentOptions.MaxTurns != 30 {
t.Fatalf("profile must stay active across the switch: name %q turns %d", m.execProfileName, m.agentOptions.MaxTurns)
}
// Unsupported -> supported: switching (back) to a supporting model must
// behave like selecting the profile there.
m, text, ok, _ = m.switchProviderModel("anthropic", "claude-sonnet-4.5")
if !ok {
t.Fatalf("switch back to anthropic failed: %q", text)
}
if m.reasoningEffort != "low" || m.execProfileAppliedEffort != "low" {
t.Fatalf("effort = %q applied = %q, want the profile's low refilled", m.reasoningEffort, m.execProfileAppliedEffort)
}
if !m.agentOptions.Profile.Escalate.RestoreDefaultEffort {
t.Fatal("the profile governs the effort again, so the restore must re-arm")
}
}
// The direct /model path (reconcileEffortForModelSwitch, driven by the
// target's authoritative ring) drops a KNOWN-unsupported effort — the TUI run
// path forwards the effort unfiltered, so it must not survive. When that drop
// voids a choice made under an active profile, the profile's effort
// bookkeeping resets and the profile resumes governing: a later switch to a
// supporting ring fills the profile's level again.
func TestProfileEffortVoidedByModelSwitchDrop(t *testing.T) {
m := profileSwitchModel(t)
m, _ = m.handleProfileCommand("fast")
m, _ = m.handleEffortCommand("high")
if m.reasoningEffort != "high" || !m.execProfileEffortTouched {
t.Fatalf("setup: effort %q touched %v, want an explicit high", m.reasoningEffort, m.execProfileEffortTouched)
}
// Destination with an AUTHORITATIVE empty ring (a catalog model without
// effort controls): "high" is known-unsupported, the drop fires, and the
// voided choice resets the profile's bookkeeping.
m, reset := m.reconcileEffortForModelSwitch(nil, true)
if !reset {
t.Fatal("a dropped preference with nothing refilled must report the reset")
}
if m.reasoningEffort != "" {
t.Fatalf("effort = %q, a known-unsupported level must not survive", m.reasoningEffort)
}
if m.execProfileEffortTouched || m.execProfileAppliedEffort != "" {
t.Fatalf("voided choice must reset the bookkeeping: touched %v applied %q", m.execProfileEffortTouched, m.execProfileAppliedEffort)
}
if m.agentOptions.Profile.Escalate.RestoreDefaultEffort {
t.Fatal("no profile-governed effort on this model, so the restore must be disarmed")
}
if m.execProfileName != "fast" || m.agentOptions.MaxTurns != 30 {
t.Fatalf("profile must stay active: name %q turns %d", m.execProfileName, m.agentOptions.MaxTurns)
}
// With the void cleared, a destination that supports the profile's level
// behaves like selecting the profile there: the fill returns.
ring := []modelregistry.ReasoningEffort{"low", "medium", "high"}
m, reset = m.reconcileEffortForModelSwitch(ring, true)
if reset {
t.Fatal("a refill is not a reset to auto")
}
if m.reasoningEffort != "low" || m.execProfileAppliedEffort != "low" {
t.Fatalf("effort = %q applied = %q, want the profile's low refilled", m.reasoningEffort, m.execProfileAppliedEffort)
}
if !m.agentOptions.Profile.Escalate.RestoreDefaultEffort {
t.Fatal("the profile governs the effort again, so the restore must re-arm")
}
}
// A touched effort the destination DOES support survives the direct /model
// path untouched — the drop only fires for known-unsupported levels.
func TestProfileEffortTouchedSurvivesSupportedModelSwitch(t *testing.T) {
m := profileSwitchModel(t)
m, _ = m.handleProfileCommand("fast")
m, _ = m.handleEffortCommand("high")
m, reset := m.reconcileEffortForModelSwitch([]modelregistry.ReasoningEffort{"low", "medium", "high"}, true)
if reset {
t.Fatal("nothing was dropped, so no reset")
}
if m.reasoningEffort != "high" || !m.execProfileEffortTouched {
t.Fatalf("effort = %q touched %v, an explicit choice the destination supports must survive", m.reasoningEffort, m.execProfileEffortTouched)
}
if m.agentOptions.Profile.Escalate.RestoreDefaultEffort {
t.Fatal("an explicit choice keeps the restore disarmed")
}
}
// An UNKNOWN ring (live-discovered/custom target with no catalog entry) is
// not evidence of unsupported: an explicit preference survives, exactly as it
// does on the cross-provider picker path, while the profile's own fill still
// retreats to models where support is known.
func TestProfileEffortUnknownRingPreservesExplicitChoice(t *testing.T) {
m := profileSwitchModel(t)
m, _ = m.handleProfileCommand("fast")
m, _ = m.handleEffortCommand("high")
m, reset := m.reconcileEffortForModelSwitch(nil, false)
if reset {
t.Fatal("an unknown ring must not report a reset")
}
if m.reasoningEffort != "high" || !m.execProfileEffortTouched {
t.Fatalf("effort = %q touched %v, an explicit choice must survive an unknown ring", m.reasoningEffort, m.execProfileEffortTouched)
}
// Untouched profile fill on an unknown ring: the fill retreats (the
// profile only governs where support is known), with clean bookkeeping.
m2 := profileSwitchModel(t)
m2, _ = m2.handleProfileCommand("fast")
m2, reset = m2.reconcileEffortForModelSwitch(nil, false)
if reset {
t.Fatal("a profile-fill retreat is not an unsupported-preference reset")
}
if m2.reasoningEffort != "" || m2.execProfileAppliedEffort != "" {
t.Fatalf("effort = %q applied = %q, the profile fill must retreat on an unknown ring", m2.reasoningEffort, m2.execProfileAppliedEffort)
}
if m2.agentOptions.Profile.Escalate.RestoreDefaultEffort {
t.Fatal("no profile-governed effort, so the restore must be disarmed")
}
}
// An explicitly touched effort is the user's choice: the reconciliation must
// leave it alone entirely across model switches.
func TestProfileEffortTouchedSurvivesModelSwitch(t *testing.T) {
m := profileSwitchModel(t)
m, _ = m.handleProfileCommand("fast")
m, _ = m.handleEffortCommand("high")
m, text, ok, _ := m.switchProviderModel("ollama", "kimi-k2.7-code:cloud")
if !ok {
t.Fatalf("switch to ollama failed: %q", text)
}
if m.reasoningEffort != "high" {
t.Fatalf("effort = %q, an explicit choice must survive the switch untouched", m.reasoningEffort)
}
if m.agentOptions.Profile.Escalate.RestoreDefaultEffort {
t.Fatal("an explicit choice keeps the restore disarmed")
}
}
func TestProfileCommandStatus(t *testing.T) {
_, out := model{}.handleProfileCommand("")
if !strings.Contains(out, "balanced (default)") {
t.Fatalf("bare /profile must report the balanced default, got %q", out)
}
_, out = model{}.handleProfileCommand("status")
if !strings.Contains(out, "balanced (default)") {
t.Fatalf("/profile status must report the balanced default, got %q", out)
}
}
func TestProfileCommandUnknownValue(t *testing.T) {
got, out := model{}.handleProfileCommand("turbo")
if !strings.Contains(out, "Usage:") {
t.Fatalf("unknown profile must print usage, got %q", out)
}
if got.execProfileName != "" || got.agentOptions.Profile != nil {
t.Fatal("unknown profile must not change any state")
}
}
// Fast applies to the next run: the turn budget drops, the escalation policy
// is armed with the DISPLACED budget as its restore target, and effort stays
// untouched when the session has no model-supported effort ring (model{}).
func TestProfileCommandSetsFastPosture(t *testing.T) {
m := model{}
m.agentOptions.MaxTurns = 80
got, out := m.handleProfileCommand("fast")
if got.agentOptions.MaxTurns != 30 {
t.Fatalf("MaxTurns = %d, want the fast profile's 30", got.agentOptions.MaxTurns)
}
if got.execProfileName != "fast" {
t.Fatalf("execProfileName = %q, want fast", got.execProfileName)
}
policy := got.agentOptions.Profile
if policy == nil || policy.Escalate == nil {
t.Fatal("fast must arm an escalation policy on agentOptions")
}
if policy.Escalate.MaxTurns != 80 {
t.Fatalf("Escalate.MaxTurns = %d, want the displaced 80", policy.Escalate.MaxTurns)
}
if got.reasoningEffort != "" {
t.Fatalf("effort = %q, must stay auto when the model exposes no effort ring", got.reasoningEffort)
}
if !strings.Contains(out, "fast") || !strings.Contains(out, "headless-only") {
t.Fatalf("status must name the profile and the TUI signal asymmetry, got %q", out)
}
}
func TestProfileCommandThoroughArmsSelfCorrect(t *testing.T) {
m := model{}
m.agentOptions.MaxTurns = 80
got, _ := m.handleProfileCommand("thorough")
if got.agentOptions.MaxTurns != 160 {
t.Fatalf("MaxTurns = %d, want thorough's 160", got.agentOptions.MaxTurns)
}
if !got.selfCorrectTests {
t.Fatal("thorough must arm full self-correction")
}
if got.agentOptions.Profile != nil {
t.Fatal("thorough has no escalation triggers, so the loop policy must stay nil")
}
}
func TestProfileCommandBalancedRestoresDisplaced(t *testing.T) {
m := model{}
m.agentOptions.MaxTurns = 80
m, _ = m.handleProfileCommand("fast")
got, _ := m.handleProfileCommand("balanced")
if got.agentOptions.MaxTurns != 80 {
t.Fatalf("MaxTurns = %d, want the displaced 80 restored", got.agentOptions.MaxTurns)
}
if got.execProfileName != "" || got.agentOptions.Profile != nil {
t.Fatal("balanced must clear the profile state and loop policy")
}
}
// Profiles must not stack: switching fast -> thorough restores fast's
// displacement first, so thorough displaces the ORIGINAL budget and a later
// balanced lands back on it.
func TestProfileCommandSwitchDoesNotStack(t *testing.T) {
m := model{}
m.agentOptions.MaxTurns = 80
m, _ = m.handleProfileCommand("fast")
m, _ = m.handleProfileCommand("thorough")
if m.agentOptions.MaxTurns != 160 {
t.Fatalf("MaxTurns = %d, want thorough's 160", m.agentOptions.MaxTurns)
}
if m.execProfileDisplacedMaxTurns != 80 {
t.Fatalf("displaced = %d, want the original 80 (not fast's 30)", m.execProfileDisplacedMaxTurns)
}
m, _ = m.handleProfileCommand("balanced")
if m.agentOptions.MaxTurns != 80 {
t.Fatalf("MaxTurns = %d, want the original 80 restored", m.agentOptions.MaxTurns)
}
}
// An explicit /turns while a profile is active is a pinned budget: it must
// disarm the escalation's turn target (mirroring headless exec, where an
// explicit --max-turns leaves nothing displaced) while the other triggers
// stay armed, and it must survive a later revert.
func TestProfileCommandTurnsPinDisarmsEscalationTarget(t *testing.T) {
m := model{}
m.agentOptions.MaxTurns = 80
m, _ = m.handleProfileCommand("fast")
m, _ = m.handleTurnsCommand("20")
if m.agentOptions.MaxTurns != 20 {
t.Fatalf("MaxTurns = %d, want the pinned 20", m.agentOptions.MaxTurns)
}
policy := m.agentOptions.Profile
if policy == nil || policy.Escalate == nil {
t.Fatal("the profile's other escalation triggers must stay armed")
}
if policy.Escalate.MaxTurns != 0 {
t.Fatalf("Escalate.MaxTurns = %d, want 0 (a pinned budget must never be raised by escalation)", policy.Escalate.MaxTurns)
}
if policy.Escalate.OnToolFailureStreak == 0 {
t.Fatal("disarming the turn target must not disarm the other triggers")
}
m, _ = m.handleProfileCommand("balanced")
if m.agentOptions.MaxTurns != 20 {
t.Fatalf("MaxTurns = %d, the pinned 20 must survive the revert", m.agentOptions.MaxTurns)
}
}
// A manual override that COINCIDES with the profile's own value must still
// survive the revert: touched beats value equality.
func TestProfileCommandRevertLeavesCoincidingOverride(t *testing.T) {
m := model{}
m.agentOptions.MaxTurns = 80
m, _ = m.handleProfileCommand("fast")
m, _ = m.handleTurnsCommand("30") // explicit, happens to equal fast's 30
m, _ = m.handleProfileCommand("balanced")
if m.agentOptions.MaxTurns != 30 {
t.Fatalf("MaxTurns = %d, the explicit /turns 30 must survive even though it equals fast's value", m.agentOptions.MaxTurns)
}
// Same for self-correct: /selfcorrect off then on under thorough is an
// explicit opt-in, not the profile's arming.
m2 := model{}
m2.agentOptions.MaxTurns = 80
m2, _ = m2.handleProfileCommand("thorough")
m2, _ = m2.handleSelfCorrectCommand("off")
m2, _ = m2.handleSelfCorrectCommand("on")
m2, _ = m2.handleProfileCommand("balanced")
if !m2.selfCorrectTests {
t.Fatal("an explicit /selfcorrect on must survive the revert even though thorough also armed it")
}
}
// Reverting a profile whose displaced budget was zero (nothing was set before
// it) must clear ZERO_MAX_TURNS: SetMaxTurnsEnv ignores zero, so without an
// explicit unset, spawned sub-agents would keep the removed profile's budget.
func TestProfileCommandRevertFromZeroClearsMaxTurnsEnv(t *testing.T) {
t.Setenv(config.MaxTurnsEnv, "")
m := model{} // MaxTurns 0: the profile displaces nothing
m, _ = m.handleProfileCommand("fast")
if got := os.Getenv(config.MaxTurnsEnv); got != "30" {
t.Fatalf("env after apply = %q, want the profile's 30", got)
}
m, _ = m.handleProfileCommand("balanced")
if m.agentOptions.MaxTurns != 0 {
t.Fatalf("MaxTurns = %d, want the displaced 0 restored", m.agentOptions.MaxTurns)
}
if got, present := os.LookupEnv(config.MaxTurnsEnv); present {
t.Fatalf("env after revert = %q (present), want the key removed", got)
}
}
// An explicit effort choice after /profile must disarm the escalation's
// effort restore (the effort analog of the /turns pin): a mid-run escalation
// must never clear an effort the user pinned by hand. Needs a catalog model
// with an effort ring so the profile's fill actually applies.
func TestProfileCommandExplicitEffortDisarmsRestore(t *testing.T) {
m := model{modelName: "claude-sonnet-4.5"}
m.agentOptions.MaxTurns = 80
m, _ = m.handleProfileCommand("fast")
if m.reasoningEffort != "low" {
t.Fatalf("profile did not fill low effort for the test model: got %q", m.reasoningEffort)
}
policy := m.agentOptions.Profile
if policy == nil || policy.Escalate == nil || !policy.Escalate.RestoreDefaultEffort {
t.Fatal("profile-filled effort must arm the escalation effort restore")
}
m, _ = m.handleEffortCommand("high")
if m.reasoningEffort != "high" {
t.Fatalf("effort = %q, want the explicit high", m.reasoningEffort)
}
policy = m.agentOptions.Profile
if policy == nil || policy.Escalate == nil {
t.Fatal("the escalation policy must stay armed")
}
if policy.Escalate.RestoreDefaultEffort {
t.Fatal("an explicit /effort must disarm the effort restore")
}
if policy.Escalate.MaxTurns != 80 {
t.Fatalf("Escalate.MaxTurns = %d, disarming effort must not touch the turn target", policy.Escalate.MaxTurns)
}
// And the explicit choice survives a later revert (touched bit).
m, _ = m.handleProfileCommand("balanced")
if m.reasoningEffort != "high" {
t.Fatalf("effort = %q, the explicit high must survive the revert", m.reasoningEffort)
}
}
// A manual override made after selecting a profile survives the revert: the
// profile only restores knobs that still hold the value it applied.
func TestProfileCommandRevertLeavesManualOverride(t *testing.T) {
m := model{}
m.agentOptions.MaxTurns = 80
m, _ = m.handleProfileCommand("fast")
m, _ = m.handleTurnsCommand("100")
got, _ := m.handleProfileCommand("balanced")
if got.agentOptions.MaxTurns != 100 {
t.Fatalf("MaxTurns = %d, the user's explicit /turns 100 must survive the revert", got.agentOptions.MaxTurns)
}
// An explicit /selfcorrect choice armed BEFORE thorough must survive too.
m2 := model{selfCorrectTests: true}
m2.agentOptions.MaxTurns = 80
m2, _ = m2.handleProfileCommand("thorough")
m2, _ = m2.handleProfileCommand("balanced")
if !m2.selfCorrectTests {
t.Fatal("an explicit self-correct opt-in must survive profile revert")
}
}
-249
View File
@@ -3,7 +3,6 @@ package tui
import (
"context"
"fmt"
"os"
"strconv"
"strings"
@@ -11,7 +10,6 @@ import (
"github.com/Gitlawb/zero/internal/agent"
"github.com/Gitlawb/zero/internal/config"
"github.com/Gitlawb/zero/internal/execprofile"
"github.com/Gitlawb/zero/internal/modelregistry"
"github.com/Gitlawb/zero/internal/sessions"
"github.com/Gitlawb/zero/internal/usage"
@@ -63,7 +61,6 @@ func (m model) handleEffortCommand(args string) (model, string) {
}
if args == "auto" {
m.reasoningEffort = ""
m = m.markProfileEffortTouched()
return m, m.effortStatusCard("auto", "Reasoning effort selection will follow the active model/provider defaults.")
}
@@ -81,7 +78,6 @@ func (m model) handleEffortCommand(args string) (model, string) {
}
m.reasoningEffort = requested
m = m.markProfileEffortTouched()
return m, m.effortStatusCard(string(requested), "Reasoning effort preference is stored for this TUI session.")
}
@@ -135,104 +131,6 @@ func (m model) effortText() string {
})
}
// markProfileEffortTouched records an explicit effort choice made while a
// profile is active: the touched bit protects the choice from a later profile
// revert, and disarming the policy's RestoreDefaultEffort stops a mid-run
// escalation from clearing an effort the user pinned by hand.
func (m model) markProfileEffortTouched() model {
if m.execProfileName == "" {
return m
}
m.execProfileEffortTouched = true
return m.setProfileEffortRestore(false)
}
// setProfileEffortRestore rewrites the armed policy's RestoreDefaultEffort on
// a clone (an in-flight run may hold the current pointer), like the /turns
// turn-target disarm. No-op when no escalation policy is armed or the flag
// already matches.
func (m model) setProfileEffortRestore(restore bool) model {
if m.agentOptions.Profile == nil || m.agentOptions.Profile.Escalate == nil ||
m.agentOptions.Profile.Escalate.RestoreDefaultEffort == restore {
return m
}
policy := *m.agentOptions.Profile
escalate := *policy.Escalate
escalate.RestoreDefaultEffort = restore
policy.Escalate = &escalate
m.agentOptions.Profile = &policy
return m
}
// reconcileEffortForModelSwitch applies the effort rules for a model switch.
// efforts is the destination's supported ring and ringKnown whether that ring
// is authoritative (a catalog entry, whose ring may be legitimately empty) or
// missing knowledge (a live-discovered/custom model the catalog cannot vouch
// for either way).
//
// First the pre-existing generic rule: a KNOWN-unsupported preference is
// dropped — the TUI run path forwards the effort to the provider unfiltered,
// so it must not survive the switch, for a profile-touched effort exactly as
// for a plain one. An UNKNOWN ring never drops an explicit preference: the
// model may well support it, matching the cross-provider picker's behavior
// for custom models. When a drop voids a choice living under an active
// profile (the profile's own fill or a user override of it), the profile's
// effort bookkeeping resets so a choice that no longer exists stops binding
// and no stale applied/restore state remains.
//
// Then the profile rule: an active, untouched profile re-derives its fill for
// the destination (conservative in both directions — the fill only ever
// applies where support is known). The second return reports whether an
// unsupported preference was dropped and nothing refilled (the caller's
// "(unsupported preference reset)" display case).
func (m model) reconcileEffortForModelSwitch(efforts []modelregistry.ReasoningEffort, ringKnown bool) (model, bool) {
dropped := false
if ringKnown && m.reasoningEffort != "" && !reasoningEffortAllowed(efforts, m.reasoningEffort) {
m.reasoningEffort = ""
dropped = true
if m.execProfileName != "" {
m.execProfileEffortTouched = false
m.execProfileAppliedEffort = ""
m = m.setProfileEffortRestore(false)
}
}
m = m.reconcileProfileAfterModelSwitch(efforts)
return m, dropped && m.reasoningEffort == ""
}
// reconcileProfileAfterModelSwitch re-derives an active profile's effort fill
// for the model the session just switched to, whose supported ring is passed
// in. The fill is per-model, not a session preference: fast selected on a
// model without "low" fills nothing, and switching to a model that supports
// it should behave exactly like selecting the profile there (and the reverse
// switch must not keep sending a level the destination does not support). An
// explicitly touched effort is the user's choice and is never reconciled; the
// escalation's RestoreDefaultEffort tracks whether the profile currently
// governs the effort.
func (m model) reconcileProfileAfterModelSwitch(efforts []modelregistry.ReasoningEffort) model {
if m.execProfileName == "" || m.execProfileEffortTouched {
return m
}
profile, ok := execprofile.Lookup(m.execProfileName)
if !ok || profile.ReasoningEffort == "" {
return m
}
want := modelregistry.ReasoningEffort(profile.ReasoningEffort)
supported := reasoningEffortAllowed(efforts, want)
filled := m.execProfileAppliedEffort != ""
switch {
case supported && !filled && m.reasoningEffort == "":
m.reasoningEffort = want
m.execProfileAppliedEffort = want
m = m.setProfileEffortRestore(true)
case !supported && filled && m.reasoningEffort == m.execProfileAppliedEffort:
m.reasoningEffort = ""
m.execProfileAppliedEffort = ""
m = m.setProfileEffortRestore(false)
}
return m
}
func (m model) availableReasoningEfforts() []modelregistry.ReasoningEffort {
if strings.TrimSpace(m.modelName) == "" {
return nil
@@ -263,10 +161,6 @@ func (m model) cycleReasoningEffort() (model, tea.Cmd) {
if len(efforts) == 0 {
return m, nil
}
// Every branch below changes the effort; a cycle while a profile is active
// is an explicit user choice that must survive both a later profile revert
// and any mid-run escalation.
m = m.markProfileEffortTouched()
if m.reasoningEffort == "" {
m.reasoningEffort = efforts[0]
return m, nil
@@ -360,12 +254,6 @@ func (m model) handleSelfCorrectCommand(args string) (model, string) {
default:
return m, "Self-correct\nUsage: /selfcorrect [status|on|off|tests|full|lsp]"
}
// An explicit choice while a profile is active must survive a later
// profile revert, even when it lands on the value the profile applied
// (e.g. /selfcorrect off then on under thorough).
if m.execProfileName != "" {
m.execProfileSelfCorrectTouched = true
}
return m, m.selfCorrectText()
}
@@ -390,146 +278,9 @@ func (m model) handleTurnsCommand(args string) (model, string) {
// Propagate the budget to spawned sub-agents / swarm members (which inherit the
// environment) so a delegated task gets the same budget, not config.json's default.
config.SetMaxTurnsEnv(n)
// An explicit /turns while a profile is active is a pinned budget: mirror
// headless exec's rule (an explicit --max-turns leaves nothing displaced)
// by disarming the escalation's turn target, so a mid-run escalation can
// never raise the budget the user just pinned. The other escalation
// triggers stay armed. Mark the knob touched so a later profile revert
// leaves this choice alone even if n coincides with the profile's value.
if m.execProfileName != "" {
m.execProfileTurnsTouched = true
if m.agentOptions.Profile != nil && m.agentOptions.Profile.Escalate != nil {
policy := *m.agentOptions.Profile
escalate := *policy.Escalate
escalate.MaxTurns = 0
policy.Escalate = &escalate
m.agentOptions.Profile = &policy
}
}
return m, m.turnsText()
}
// handleProfileCommand shows or switches the session's execution profile: a
// loop-posture bundle (turn budget, reasoning effort, self-correction,
// escalation triggers) applied to the NEXT run, composing with the selected
// model rather than replacing it. Switching first restores whatever the
// previous profile displaced (profiles never stack), and balanced IS that
// restore: the empty profile, so selecting it just clears the posture.
func (m model) handleProfileCommand(args string) (model, string) {
args = strings.TrimSpace(args)
if args == "" || strings.EqualFold(args, "status") {
return m, m.profileText()
}
profile, ok := execprofile.Lookup(args)
if !ok {
return m, "Profile\nUsage: /profile [status|" + strings.Join(execprofile.Names(), "|") + "] — switch the next run's loop posture."
}
m = m.revertExecProfile()
if profile.Name == execprofile.Balanced.Name {
return m, m.profileText()
}
// Turn budget: displace the session's current budget. The displaced value
// becomes the escalation restore target, exactly like headless exec, and
// the new budget propagates to spawned sub-agents the same way /turns does.
displacedMaxTurns := 0
if profile.MaxTurns > 0 {
displacedMaxTurns = m.agentOptions.MaxTurns
m.agentOptions.MaxTurns = profile.MaxTurns
m.execProfileAppliedMaxTurns = profile.MaxTurns
m.execProfileDisplacedMaxTurns = displacedMaxTurns
config.SetMaxTurnsEnv(profile.MaxTurns)
}
// Reasoning effort: fill only when the session is on auto AND the active
// model supports the profile's level, mirroring exec's supported-effort
// gating. An explicit user choice always wins over the profile.
if want := modelregistry.ReasoningEffort(profile.ReasoningEffort); want != "" && m.reasoningEffort == "" {
if reasoningEffortAllowed(m.availableReasoningEfforts(), want) {
m.reasoningEffort = want
m.execProfileAppliedEffort = want
}
}
// Self-correction is presence-only: a profile can arm it but never disarm
// an explicit /selfcorrect choice.
if profile.SelfCorrect && !m.selfCorrectTests {
m.selfCorrectTests = true
m.execProfileArmedSelfCorrect = true
}
m.agentOptions.Profile = profile.Policy(displacedMaxTurns, m.execProfileAppliedEffort != "")
m.execProfileName = profile.Name
return m, m.profileText()
}
// revertExecProfile restores what the active profile displaced. Each knob is
// restored only when the user has not explicitly touched it since the profile
// was applied (the touched bits, set by /turns, /effort, Ctrl+T, and
// /selfcorrect) AND it still holds the value the profile applied — so manual
// overrides survive the revert even when they coincide with the profile's own
// value (e.g. /turns 30 under fast, or /selfcorrect off-then-on under
// thorough).
func (m model) revertExecProfile() model {
if m.execProfileName == "" {
return m
}
if !m.execProfileTurnsTouched && m.execProfileAppliedMaxTurns > 0 && m.agentOptions.MaxTurns == m.execProfileAppliedMaxTurns {
m.agentOptions.MaxTurns = m.execProfileDisplacedMaxTurns
if m.execProfileDisplacedMaxTurns > 0 {
config.SetMaxTurnsEnv(m.execProfileDisplacedMaxTurns)
} else {
// The profile's apply exported its budget to ZERO_MAX_TURNS (for
// sub-agents), and SetMaxTurnsEnv ignores zero — so restoring a
// zero displaced budget must clear the env explicitly or spawned
// children would keep the removed profile's budget.
_ = os.Unsetenv(config.MaxTurnsEnv)
}
}
if !m.execProfileEffortTouched && m.execProfileAppliedEffort != "" && m.reasoningEffort == m.execProfileAppliedEffort {
m.reasoningEffort = ""
}
if !m.execProfileSelfCorrectTouched && m.execProfileArmedSelfCorrect && m.selfCorrectTests {
m.selfCorrectTests = false
}
m.agentOptions.Profile = nil
m.execProfileName = ""
m.execProfileDisplacedMaxTurns = 0
m.execProfileAppliedMaxTurns = 0
m.execProfileAppliedEffort = ""
m.execProfileArmedSelfCorrect = false
m.execProfileTurnsTouched = false
m.execProfileEffortTouched = false
m.execProfileSelfCorrectTouched = false
return m
}
func (m model) profileText() string {
name := m.execProfileName
if name == "" {
name = execprofile.Balanced.Name + " (default)"
}
lines := []string{
"execution profile: " + name,
fmt.Sprintf("max tool-turns per run: %d", m.agentOptions.MaxTurns),
"reasoning effort: " + m.effortDisplay(),
}
if m.agentOptions.Profile != nil && m.agentOptions.Profile.Escalate != nil {
turnTarget := "keeps the pinned turn budget"
if target := m.agentOptions.Profile.Escalate.MaxTurns; target > 0 {
turnTarget = fmt.Sprintf("restores the turn budget to %d", target)
}
lines = append(lines,
"escalation: armed — one-shot on a tool-failure streak, a failing self-correct cycle, or a critical-risk mutation; "+turnTarget,
"note: the uncertain-completion signal is headless-only (the completion gate never runs interactively), so it cannot fire in the TUI")
}
return renderCommandOutput(commandOutput{
Title: "Profile",
Status: commandStatusOK,
Sections: []commandSection{{
Title: "State",
Lines: lines,
}},
Hints: []string{"/profile " + strings.Join(execprofile.Names(), "|") + " switches the next run's loop posture (turn budget, effort, self-correction, escalation); pick the model separately with /model"},
})
}
func (m model) turnsText() string {
return renderCommandOutput(commandOutput{
Title: "Turns",