mirror of
https://github.com/Twigpine/zero.git
synced 2026-11-10 05:01:28 -10:00
KRATOS
474f494b93
* audit batch 1: redact nested secrets fully; count image tokens in compaction estimate
- secrets.Redact: replace longest matches first so a containing PRIVATE KEY
block is redacted before a nested shorter match corrupts its exact string
(audit medium internal/secrets/scanner.go:81) + regression test.
- agent compaction estimateTokens: add a flat per-image token cost so an
image-heavy context trends toward compaction instead of reading as ~0
(audit medium internal/agent/compaction.go:58) + test.
- ledger: re-verify against current main; mark scanner.go:35 and
redaction.go:362 already-fixed-on-main, record this batch's fixes.
* audit batch 2: worktrees defaultRunGit separates stdout/stderr
Capture git stdout and stderr into separate buffers instead of CombinedOutput,
so parsed rev-parse output is not polluted by git warnings and CommandResult.Stderr
carries the real error text (audit medium internal/worktrees/worktrees.go:247) + test.
* audit: re-verify all 87 pending findings against current main
Independent agent per file re-checked each open/partial finding against the
actual current code (the 06-11 ledger was verified on a since-diverged branch),
with an adversarial second pass on every already_fixed verdict. Authoritative
result: 78 genuinely pending (66 still_open + 12 partial); 9 rows were already
fixed on main and are reclassified. Full table in 2026-06-13-reverification.md.
* audit batch 3 (agent): cancellation identity, always-allow w/o sandbox, compaction usage, dedup
- loop.go: check ctx.Err() before collected.Error so a cancelled run returns
context.Canceled (errors.Is holds) instead of a plain stringified error.
- loop.go: 'Always allow' with no sandbox engine now allows THIS call instead of
denying it (persistence is skipped when there's nowhere to persist).
- loop.go: remove a duplicate schema["items"] assignment in propertyToRuntimeMap.
- compaction.go: forward OnUsage (token accounting) through the summarizer while
still omitting OnText, so compaction's token cost is counted in usage/budgets.
- tests for all three behavioral fixes (each fails pre-fix).
* audit batch 4 (cli): list-tools -o json, interrupted -o json terminal, surface session-record failures
- exec --list-tools -o json now emits a JSON tool listing instead of falling
through to the human-readable text (audit medium exec.go:182) + test.
- interrupted run with -o json now emits a terminal error+done on stdout instead
of only printing to stderr, so a JSON consumer sees a clean end (medium exec.go:500) + test.
- execSessionRecorder: surface a latched session-append failure to stderr once at
run end (deferred) instead of silently dropping it (medium exec_sessions.go:114/115) + test.
- left app.go trailing-arg ignore as-is: documented-intentional, TUI has no argv prompt.
* audit batch 5 (zerogit): status -z porcelain (renames + non-ASCII paths); document env-drop on plain Runner
- parseStatus: switch `git status` to `--porcelain -z` and parse the
NUL-delimited form (audit medium zerogit.go:212). Previously the default
--short output was split on newlines, so a rename `R old -> new` was kept as
one bogus path and a non-ASCII/whitespace filename arrived C-quoted+escaped
(e.g. `"caf\303\251.txt"`). The -z form never quotes paths and emits a
rename/copy as `XY <dest>\0<src>`, so we now record the destination verbatim
and consume the source field. + TestParseStatusZHandlesRenamesAndSpecialPaths.
- resolveRunners: document that the plain-Runner EnvRunner adapter intentionally
drops env (audit low zerogit.go:417). The branch is reached only when a caller
supplies a custom Runner without a RunGitEnv — in practice fake-Runner tests.
Production leaves both nil and gets defaultRunGit/defaultRunGitEnv, which honor
GIT_INDEX_FILE, so snapshot-diff isolation holds. Routing env calls to real git
instead would break the fake-Runner test pattern and fix nothing in production.
- update Inspect/Commit mock tests to the -z status format and the new command.
* audit batch 6 (config): reject negative maxTurns; drop dead contract-gap helpers
- FileConfig.UnmarshalJSON now rejects a negative maxTurns instead of letting the
`MaxTurns > 0` merge gates silently drop it and fall back to the default,
hiding the misconfiguration (audit medium resolver.go:136). 0 is left as the
"unset" sentinel (omitempty) and still falls back to the default; the CLI
--max-turns flag continues to reject 0 too since there it is distinguishable
from absent. + TestResolveRejectsNegativeMaxTurns / ...ZeroFallsBackToDefault.
- remove dead config/contracts.go (ContractGap, DefaultContractGaps,
FindContractGapsByMilestone, ContractOwnerRuntime) and its test — referenced
only by that test, never by production (audit low contracts.go:12).
- NOT changed: ResolveMCP intentionally does not run the provider command
(TestResolveMCPDoesNotRunProviderCommand guards this), so the "ZERO_PROVIDER_COMMAND
MCP servers dropped" row is by-design, not a defect.
* audit batch 7 (mcp): bound captured MCP server stderr
connectStdio attached a plain bytes.Buffer as cmd.Stderr that is only ever read
on initialize failure, yet it kept growing for the entire process lifetime as a
long-lived server logged to stderr (audit medium client.go:98). Replace it with
a concurrency-safe boundedBuffer capped at 64 KiB that keeps the head (enough for
init-error diagnostics) and discards the rest, reporting full write lengths so
os/exec never sees a short write. + TestBoundedBufferCapsRetainedBytes.
Left as-is (not defects worth the risk in the MCP client):
- client.go writer.write under client.mu: already designed to release the lock
before the unbounded response wait; the residual concern is a server that stops
draining stdin, an involved async rewrite for a theoretical deadlock.
- readLoop not answering server-initiated requests (ping): responding safely
needs re-locking around the shared writer; no configured server relies on it.
* audit batch 8 (tools/providers): read_file truncation marker; drop dead provider branch
- read_file: when a max_lines cut occurs, append an explicit "[truncated: N more
line(s)... set start_line=X to continue]" marker to the output. Previously only
the Result.Truncated flag was set, which is invisible in the rendered text, so
the model could not distinguish a cut read from a complete one (audit low
read_file.go:89). + TestReadFileToolMarksTruncation.
- providers/factory: simplify `if !explicitProvider || isImplicitOpenAI(...)` to
`if !explicitProvider` and delete the now-unused isImplicitOpenAI. The clause
was dead: explicitProvider==true implies ProviderKind or Provider is set, but
isImplicitOpenAI required both empty, so it could never contribute a case
(audit low factory.go:118). Behavior-preserving; providers tests still green.
* audit batch 9 (sessions): Tree builds from a single List() snapshot (no N+1 disk scans)
Store.Tree recursed via ListChildren per node, and each ListChildren ran a full
store.List() that re-read every metadata.json from disk — O(nodes * total-sessions)
reads for one tree (audit medium lineage.go:140). Snapshot all sessions once, index
children by parent in memory, and recurse over that. Child ordering is factored into
sortChildSessions so ListChildren and Tree stay identical; path-scoped cycle
detection (delete(seen,...)) is preserved. Existing tree/lineage tests still pass.
Left as-is: store.go timestamp() second precision (RFC3339) — switching to
RFC3339Nano would break lexical UpdatedAt ordering against existing second-precision
records ("...00.5Z" sorts before "...00Z"), so it is not a safe one-line change.
* audit batch 10 (specialist): serialize accounting dedup-then-append
recordSpecialistStop and appendSpecialistUsageRollup checked for an existing event
(specialistEventExists → ReadEvents) and then appended under SEPARATE store locks,
so two concurrent finishers — a foreground onExit racing a TaskOutput poll, or a
background reaper — could both pass the dedup check and write duplicate stop/usage
events (audit medium accounting.go:72). Guard each check-then-append pair with a
package-level accountingMu (Executor is used by value, so a struct field would not
be shared). + TestRecordSpecialistStopDedupesUnderConcurrency (passes under -race).
* audit batch 11 (zeroruntime/gemini): O(1) text accumulation; gemini cached-token counts
- zeroruntime CollectStreamWithOptions accumulated assistant text with
`collected.Text += event.Content`, reallocating the whole string on every chunk
(O(n^2) over a long streamed response). Accumulate in a strings.Builder on the
collector and materialize it once in flush, the single finalization point hit
before every return (audit low helpers.go:104). Existing collect tests still pass.
- gemini decoded usageMetadata without cachedContentTokenCount, so a cached-prompt
response reported zero cached input tokens while the other providers report them
(audit medium gemini/types.go:88). Decode the field and thread it through
streamState into NormalizeUsage as CachedInputTokens. + extended
TestStreamCompletionEmitsTextUsageAndReasoningTokens.
* audit batch 12 (specialist): parse child stream without a line-length cap; drop dead helper
- ParseStream used a bufio.Scanner capped at 1 MiB, so one large stream-json line
from a child (a big tool result or final answer) hit bufio.ErrTooLong and aborted
the entire specialist run (audit medium streamer.go:43). Switch to a bufio.Reader
with ReadString, which has no per-line limit; the child is our own trusted
subprocess, so its output length is the legitimate bound. Line numbering and
blank-line/EOF handling are preserved. + TestParseStreamHandlesLineLargerThan1MiB.
- remove dead formatTaskOutput (output_tool.go:244): zero callers and zero tests;
it only forwarded to formatTaskOutputSummary, which is the one actually used.
* audit batch 13 (background/specialist): kill background tasks as a process group
Background children were launched in the parent's process group and terminated by
signalling only the single PID, so any process the task forked was orphaned and
outlived a cancel; the signal also risked hitting a recycled PID after the child
was reaped (audit medium process_posix.go:38/50).
- ConfigureChildProcessGroup (Setpgid on POSIX, no-op on Windows where taskkill /T
already tree-kills) is applied at launch so the child leads its own group.
- terminateProcess now SIGTERM/SIGKILLs the negative PID (the whole group), with a
pid>1 guard so a bogus 0/1 can never expand to "our own group" or "everything",
and treats ESRCH as already-gone. Liveness polls the group. Targeting a group
(rarely recycled vs. a bare PID) also shrinks the reaped-then-recycled window.
- specialist launchBackgroundProcess opts every child into its own group; on a
SetPID failure it now kills the child + group, marks the task errored, and records
the stop (deduped) instead of leaking an untracked orphan.
- tests launch as group leaders; new TestTerminateProcessKillsForkedChildren proves
a forked grandchild dies with the group.
* audit batch 14 (sandbox): quote-aware shell segment splitting
splitShellSegments used a quote-unaware strings.Replacer, so a shell operator
inside quotes was treated as a separator: `git commit -m "use top | less"` split
into a `less` segment and `echo "a; vim b"` into a `vim` segment, both falsely
flagged as interactive (audit medium safe_command.go:138).
Replace it with a quote-aware scanner that mirrors the shell: single quotes make
everything literal; double quotes keep $(...)/`...` substitution active but treat
|, ;, && as literal; unquoted text splits on the same operator set as before
(;, |, ||, &&) plus substitution boundaries. Real unquoted operators still split,
so an interactive program behind a separator — or inside a live `"$(...)"`
substitution — is still caught (no new false negatives, the security-relevant
direction). + TestDetectInteractiveQuoteAwareSeparators covering both directions.
* audit batch 15 (cron): collapse DST fall-back repeated hour to one fire
On a fall-back day the clock repeats an hour (e.g. 01:30 EDT then 01:30 EST), so
Next("30 1 * * *", <01:30 EDT>) returned the same-day 01:30 EST — firing a daily
job twice (audit medium schedule.go:171, which previously guarded only the
spring-forward gap).
Next now skips a match whose local wall-clock minute equals `after`'s but whose
absolute time is later (the repeated-hour duplicate) and keeps searching. Normal
forward search always advances the wall-clock, so this only triggers on the
fall-back repeat; when the scheduler passes the last fire time as `after`, the
repeated hour collapses to a single fire. + TestNextDSTFallBackDoesNotRepeatHour
(spring-forward termination test still passes).
Note: Next is stateless, so it dedupes relative to `after`; a caller that passes
an arbitrary mid-hour "now" instead of the last fire time is out of scope here.
* audit batch 16 (mcp): skip SSE notifications before the response; larger SSE buffer
- decodeSSERPCMessage returned the first non-empty `message` event unconditionally,
so a server-initiated notification/request arriving before the response made the
caller see an id mismatch and fail (audit medium network_client.go:621). It now
skips any message carrying a method (notifications/requests have one; a response
does not) and returns the first real response; the caller's rpcIDMatches still
validates the id. + TestDecodeSSERPCMessageSkipsNotifications.
- raise the per-event SSE scanner cap from 1 MiB to a bounded 8 MiB so a large but
legitimate MCP message no longer hits bufio.ErrTooLong and fails the request
(audit medium network_client.go:637); still bounded against an unbounded server.
Out of scope (kept): full reconnect of a persistent SSE stream after a fatal read
error — a larger change than this remediation.
* audit batch 17 (sessions): fsync session metadata before the atomic rename
writeMetadata wrote the temp file with os.WriteFile (no sync) and renamed it into
place, so a crash right after the rename could expose a metadata file whose bytes
were never flushed — a torn or empty file that corrupts the whole session (audit
medium store.go:632). Write+fsync the temp via a new writeFileSync helper before
renaming, closing that window. + TestWriteFileSyncRoundTrips.
Deliberately scoped:
- Event-log appends (appendEventLocked) are NOT fsynced per append: that would add
a disk flush to every event (every tool call), and a hard-crash loss of the log's
unsynced tail is acceptable, whereas torn METADATA is not. Metadata is the
integrity-critical part and is what this syncs.
- Directory fsync is omitted for cross-platform simplicity (it errors on Windows);
the file fsync already prevents the torn-contents failure, the serious case.
Left for follow-up (in this bucket but not safe/clear wins here):
- hooks append sequence still re-reads the file per append; the in-memory cache is
cross-process-unsafe and the safe last-line read is fiddly for a low-volume log.
- Fork event re-append/de-dup is nuanced usage-accounting and needs its own change.
* audit batch 18 (agent): count tool-defs in compaction estimate; don't deny an always-allow on persist failure
- maybeCompact's estimate omitted the tool definitions, which ride on every
request, so the real input could exceed the model limit while the message-only
estimate stayed under threshold and compaction never fired (audit medium
compaction.go:58). partitionTools is now computed before maybeCompact (it does
not depend on the messages) and the exposed tools' estimated tokens are added to
both the threshold check and the post-compaction shrink check. + estimateToolDefTokens
and TestEstimateToolDefTokensCountsDefinitions.
- PermissionDecisionAlwaysAllow denied the call when persistPermissionGrant failed,
overriding what the user explicitly allowed (audit medium loop.go:468). Persisting
is now best-effort: a failure (or no sandbox engine) just means the grant is not
remembered and the user is re-prompted next time, not denied now. This also drops
the dead requestEvent.GrantMatched/Grant writes (audit low loop.go:473) — the
emitted event derives them from the sandbox decision via buildPermissionEvent.
Left intentional (documented, not defects): the reactive-retry path omits OnText to
avoid duplicating mid-stream text (loop.go:183); OnContext/MeasureContext is a
complete, tested context-breakdown scaffold awaiting a consumer (types.go:183).
* audit batch 19 (usage): price escalated usage from the event's own model
exec persists payload["model"] on --allow-escalation runs (the model can change
mid-run), but BuildReport priced every event from the session's Metadata.ModelID
and never read the payload, so usage from an escalated model was mis-priced at the
base model (audit medium report.go:17). usageEventPayload now carries Model and
BuildReport prefers it, falling back to the session model when absent (older
events stay byte-identical and behave as before). + TestBuildReportPricesFromEventModelWhenPresent.
* audit batch 20 (sessions): don't copy usage events into a fork (no double-count)
Fork re-appended every parent event, including provider_usage, into the child, so
a usage report aggregating the parent and the fork double-counted the parent's
usage (audit medium store.go:387). Skip EventUsage when copying — it already
counted against the parent and is not part of the conversation history a fork
replays — and record the actual copied count in the fork marker. + TestStoreForkSkipsUsageEvents.
* audit batch 21 (cli/skills): warn on duplicate skill names instead of silently dropping
skills.Load deduplicates same-named skills (first directory wins) and records the
dropped collisions via skills.Duplicates, but nothing surfaced them, so a shadowed
skill just vanished with no signal — and Duplicates had no production caller (audit
low skills.go:91). `skills list` now reports each collision to stderr (keeping
stdout/--json clean), using Duplicates. + TestRunSkillsListWarnsOnDuplicateNames.
Note: skills.Get is kept — it is a by-name accessor used throughout the package's
tests, not dead code.
* audit batch 22 (cli/cron): capture the run error from stream-json stdout
Cron jobs run exec with --output-format stream-json, so a failure is reported as an
`error` event on STDOUT, but the run record's Error was set only from stderr — so a
failed job often recorded an empty/uninformative error while the real message sat
unread in outBuf (audit low cron_run.go:150). On a non-zero exit, scan the stdout
stream-json for the error event's message and prefer it, falling back to stderr.
+ extractStreamJSONError and TestExtractStreamJSONError.
* audit batch 23 (sandbox): richer macOS seatbelt profile (signal + mach-lookup)
The generated sandbox-exec profile was default-deny with only process/sysctl/file
allowances, so two real gaps remained on macOS (sandbox quality report H1):
- A sandboxed script could not signal the children it spawned — `kill` returned
"Operation not permitted" under seatbelt, breaking common `cmd & ... kill`,
test-runner, and timeout patterns. Add `(allow signal (target self) (target pgrp))`,
which stays scoped to the command's own process group.
- XPC to system daemons was denied, so keychain (securityd/trustd), user/group
lookup (opendirectoryd), preferences (cfprefsd), network config
(SystemConfiguration), launch services, and the pasteboard all failed. Add a
curated `(allow mach-lookup ...)` allowlist (`sandboxMachServices`).
Neither touches the file-write or network rules, so the workspace boundary is
unchanged — verified on a real macOS host: `security`/`scutil`/`pbpaste`/self-kill
now succeed while writes to $HOME and /etc are still denied. iokit-open was tried
and dropped as unnecessary. + unit assertions (TestSandboxExecProfileGrantsSignalAndMachLookup)
and extended the darwin integration test with a self-kill case and an
outside-workspace deny check.
* audit batch 24 (sandbox/tools): remove dead OnSandboxDecision hook
RunOptions.OnSandboxDecision had no setter anywhere, yet registry.RunWithOptions
spawned a goroutine (with panic recovery) for it on every sandboxed tool call —
pure per-call overhead for an unreachable callback (sandbox report M1). The sandbox
decision is already surfaced to observers via the permission event
(buildPermissionEvent reads the decision), so this hook is redundant; delete the
field and its call site.
* audit batch 25 (sandbox/cli): exact-path grants — create and revoke by path
The engine already matched ScopeFile grants and DeriveScope already produced them
from path/file tool args, but the CLI could neither create one explicitly nor
revoke a single one (sandbox report L3 / prompt 7). Add:
- GrantStore.RevokePath(tool, path): revokes only the grant whose scope matches the
given file/dir, leaving tool-wide and other-path grants intact (path canonicalized
to absolute like stored scopes). + TestGrantStoreRevokePathRemovesOnlyMatchingScope.
- `zero sandbox grants allow <tool> --path <p>`: persists an exact-file (ScopeFile)
grant, path resolved to absolute so it matches lookups and revokes.
- `zero sandbox grants revoke <tool> --path <p>`: revokes just that grant (default
still revokes all grants for the tool). + TestRunSandboxGrantsCreateAndRevokeByPath.
The TUI permission-card "Allow always (exact path)" option is intentionally left for
the TUI feature branches (it overlaps #187/#188); the engine + CLI now cover the
full create/match/revoke round-trip for file-scoped grants.
* audit batch 26 (sandbox): Engine.Precheck — report violations before execution
Add Engine.Precheck(ctx, request) []Violation, which reports the sandbox
violations that would block a tool request before it runs (sandbox report / prompt 6).
It reuses Evaluate so policy is not duplicated: an allowed or merely-prompted
request yields no violations; a denied request yields its violation (via the shared
violationsFromDecision helper, with a ViolationPolicyDenied fallback for a deny that
carries no structured violation). A nil engine yields nothing.
+ TestEnginePrecheckReportsViolationsBeforeExecution.
Wiring Precheck into the batch-confirmation prompt is intentionally left to the TUI
branch: it touches prompt rendering, and short-circuiting the agent loop's deny path
would skip the before/after-tool hooks that currently fire around a policy-denied
call. The engine API is in place for that consumer.
* audit batch 27 (sandbox): scoped-egress domain prompt with timeout + caching
The scoped-egress proxy failed closed on any host outside the allowlist. Add an
opt-in DomainPrompt callback (prompt 8): an UNKNOWN host (neither allowed nor
explicitly denied) is authorized at request time via the callback instead of being
refused outright. An explicitly denied host is still refused without prompting; the
callback is bounded by PromptTimeout (default 60s, timeout/error => deny); decisions
are cached for the proxy's lifetime so a host is prompted at most once. With no
callback set the strict fail-closed behavior is unchanged. Wired through both the
CONNECT and HTTP handlers via authorizeTarget.
+ TestEgressProxyAuthorizeDomainPrompt / ...TimeoutDenies / ...NilPromptFailsClosed.
The user-facing prompt that supplies the callback is left to the permission UI (TUI)
branch; the egress mechanism (callback + timeout + cache) is in place and tested.
* audit batch 28 (sandbox): AST-based shell analysis (AnalyzeCommand)
Add AnalyzeCommand(script) AnalysisResult, a static shell analyzer built on the
mvdan.cc/sh/v3 parser (prompt 12). Walking the parsed command tree makes it a more
precise second opinion than the regex detector: a program name only counts when it
is an actual command, so `echo "vim ..."`, `printf 'open with vim'`, and
`git commit -m "rm -rf /"` are clean, while real commands are flagged —
interactive (editors/pagers + REPLs, with the same -e/script suppression as
nonInteractiveREPLFlags), destructive (rm -rf/-fr, dd, mkfs, shred, ...), and
network (curl/wget/ssh/nc/...). An unparseable script yields TooComplex so callers
can treat it as higher-risk. + TestAnalyzeCommand (16 cases) / ...EmptyIsClean.
Dependency: mvdan.cc/sh/v3 pinned to v3.11.0 — compatible with the existing go 1.24
directive (the latest v3.13.1 would have bumped the module to go 1.25). It is kept
as a standalone analyzer for callers to consult; feeding it into Engine.Evaluate is
deliberately not done here, since that would change allow/deny classification.
* audit batch 29 (sandbox): unify proxy-env generation (ProxyEnv)
Export ProxyEnv as the single source of truth for proxy-env injection (prompt 9
req 1), replacing the unexported scopedProxyEnv used by the sandboxed-shell path.
+ TestProxyEnv.
Investigation that bounds the rest of prompt 9 (documented in the ProxyEnv doc):
- web_fetch already honors HTTP_PROXY/HTTPS_PROXY — its transport clones
http.DefaultTransport, whose Proxy is http.ProxyFromEnvironment.
- MCP children already inherit the parent env (mergeProcessEnv appends os.Environ),
so they pick up any proxy vars the agent has.
So the only missing link to scope FetchUrl/MCP is a SESSION-level egress proxy whose
address reaches the agent process. That is deliberately NOT wired here: doing it
safely requires allowlisting the active LLM provider's own domain first, or the
agent's provider calls would route through the scoped proxy and be denied — a
correctness pitfall that also can't be verified end-to-end on this host. Flagged
rather than shipped blind.
* audit batch 30 (sandbox): Linux seccomp Unix-socket block — UNVERIFIED, needs Linux CI
Adds a pure-Go (no cgo, no vendored binary) seccomp BPF filter that denies
socket(AF_UNIX) with EPERM on x86-64/arm64, closing the Unix-socket gap bubblewrap's
filesystem/network isolation leaves open (prompt 11):
- internal/sandbox/seccomp.go: the classic-BPF program as a platform-neutral
[]sockFilter, built and UNIT-TESTED (structure + every jump offset in range — the
classic way a hand-written BPF filter goes silently wrong).
- seccomp_linux.go: ApplyUnixSocketBlock installs it via prctl(NO_NEW_PRIVS) +
prctl(SET_SECCOMP, MODE_FILTER). seccomp_other.go is a no-op everywhere else.
- cmd/zero-seccomp: a tiny linux-only exec wrapper that applies the filter then
execs the real command (the standard way to install seccomp in a child with
os/exec); degrades gracefully (warns + runs) if the kernel lacks seccomp.
⚠️ HONEST CAVEAT: I developed this on macOS and CANNOT run the actual AF_UNIX block
here. darwin build, `GOOS=linux` cross-compile, vet, and the structure test all pass,
but the unit test verifies the program's SHAPE, not that the kernel actually blocks
AF_UNIX — that MUST be verified on Linux CI (create a Unix socket under the wrapper
and assert EPERM). Safe to ship as-is: it is linux-only, no-op on every other
platform, and not wired into the bwrap command yet (opt-in prefixing the sandboxed
command with zero-seccomp is the remaining integration), so it cannot regress any
current path.
* audit batch 31 (sandbox): macOS denial log monitor (opt-in, default off)
Re-adds the macOS seatbelt-denial monitor as an opt-in feature (prompt 10). When
Policy.MonitorDenials is set, the sandbox-exec profile tags its default-deny with a
marker and bash tails `log stream` for that tag during the command, appending any
captured denials as a <sandbox_violations> block to the command's stderr so the
model can see what was blocked.
- DenialMonitor (darwin) tails/parses tagged denials with a capped, deduped ring
buffer and noise filtering; no-op stub on other platforms. parseSandboxDenyLine is
unit-tested. + profile-tag test + appendSandboxViolations test.
- Default OFF: with MonitorTag empty, StartDenialMonitor is a no-op and the command
path is byte-for-byte unchanged, so there is no regression or per-command overhead
unless enabled.
CAVEAT: on at least one macOS host (verified live) seatbelt denials are NOT delivered
to the `log stream`-queryable unified log, so the monitor captures nothing there. It
is the standard mechanism and works on macOS versions that do log denials; shipping
it default-off means it never affects whether a command runs.
* audit batch 32 (sandbox): wire seccomp Unix-socket block into bubblewrap (opt-in)
Completes prompt 11: the zero-seccomp helper (which installs the AF_UNIX seccomp
filter) is now actually invoked by the sandbox instead of only existing as a
standalone binary.
- bubblewrapCommandPlan, when Policy.BlockUnixSockets is set, ro-binds the
discovered zero-seccomp helper into the sandbox at its host path and prefixes the
command with it so the filter is installed before the real argv runs.
- findSeccompHelper discovers the helper next to the running executable first, then
on PATH; it returns "" when absent, so the sandbox degrades gracefully (runs
without the extra filter — bubblewrap's fs/net isolation still applies) rather
than failing the command. Exposed as a package var so the wiring is unit-tested
on any OS.
- Reachable from config: SandboxConfig.BlockUnixSockets and .MonitorDenials now map
onto the policy in applyConfiguredSandboxPolicy (opt-in only — config can turn a
hardening flag on, never off). Both default OFF.
- Tests: helper-prefix present when enabled / absent by default; config flags apply
independently and stay off when omitted.
Default OFF and Linux-only effect; the runtime seccomp enforcement still needs a
Linux host/CI to exercise end-to-end (the wiring and BPF program are unit-tested).
* audit batch 33 (sandbox): enforce scoped/deny network policy in web_fetch & web_search
Completes prompt 9: the built-in network tools now honor the SAME allow/deny/scoped
policy the sandboxed shell egress proxy enforces, instead of bypassing it. Before
this, web_fetch/web_search would reach any public host even when the policy was
NetworkDeny or NetworkScoped — the egress proxy only constrained shell commands.
- New shared gate Engine.NetworkHostAllowed(host) reuses the egress domain matcher
(domainAllowed/normalizeDomains/effectiveNetwork) as the single source of truth:
allow → all hosts; deny → none; scoped → only AllowedDomains minus DeniedDomains
(empty allowlist collapses to deny). Disabled policy / nil engine impose nothing.
- web_fetch is now sandbox-aware: the host is checked before the request AND on every
redirect, so a denied/unlisted host fails closed before any dial. Plumbed via a
networkPolicyGate so the no-sandbox path is byte-for-byte unchanged.
- web_search is sandbox-aware: the configured backend's endpoint host is checked
before the query leaves the machine; blocked under deny, and under scoped unless the
backend host is allowlisted.
- The registry already routes sandbox-aware tools through RunWithSandbox when an
engine is present, so this activates automatically wherever the sandbox is wired.
- Tests: engine gate matrix (deny/allow/scoped/denylist/empty/disabled/nil + host:port);
web_fetch & web_search block denied/unlisted hosts without dialing and allow listed
ones; endpoint-host parsing.
* audit batch 34 (sandbox): prove network-policy wiring end-to-end + document opt-in flags
Verification follow-up to batches 31-33; no behaviour change.
- Adds a registry-level test that exercises the FULL chain for the scoped-network
enforcement: registry.RunWithOptions -> engine.Evaluate (passes a scoped,
egress-capable sandbox-exec backend) -> routes web_fetch through RunWithSandbox ->
per-host allowlist blocks an unlisted host before any dial. This proves the
Tool-interface value actually satisfies sandboxAwareTool at the registry boundary,
not just in a direct method call.
- Documents the two opt-in hardening flags (sandbox.blockUnixSockets,
sandbox.monitorDenials) and the scoped web_fetch/web_search behaviour in the README
so the config keys are discoverable. Both remain off by default.
Confirmed wiring: agent loop forwards options.Sandbox to the registry; the registry
routes sandbox-aware tools (web_fetch, web_search) through RunWithSandbox when an
engine is present; MonitorTag is set only on the macOS sandbox-exec plan; the
seccomp helper prefix is added only on the bubblewrap plan when BlockUnixSockets is
set; both config flags map onto the policy via applyConfiguredSandboxPolicy.
* audit batch 35 (sessions): fix Windows CI failure in TestWriteFileSyncRoundTrips
The exact-permission assertion (perm == 0o600) added in batch 17 fails on
windows-latest because Windows does not honor Unix permission bits: Go reports
0o666 for a writable file regardless of the mode passed to OpenFile. This was the
sole cause of the failing Smoke (windows-latest) CI job; macOS and Ubuntu passed.
Guard the perm check with runtime.GOOS != "windows" so it still verifies the mode
on the platforms that enforce it, while the round-trip (write + read-back) is still
asserted on every platform.
* audit batch 36: address CodeRabbit review (5 findings)
- background/process_posix: after SIGKILL, poll until the process group is
actually gone (SIGKILL is async) and return an error if it survives past a
second deadline, instead of returning nil while descendants still race.
- sandbox/safe_command: only treat ')' as a substitution boundary when it closes
an active $(...). A literal ')' (e.g. echo "a) less") no longer splits into a
fake `less` segment and is no longer misflagged as interactive. Depth-tracked so
nested $(...) still isolate their inner command. Direct splitShellSegments test
(fails pre-fix: yields ["echo \"a", "less\""]).
- cron/schedule: the DST fall-back collapse now only applies when `after` sits
exactly on the minute boundary (its "last fire time" form). With sub-minute
precision (e.g. 01:30:30 EDT) the first 01:30 fire already passed, so the repeated
01:30 EST is the legitimate next fire and is returned — restoring strictly-after
semantics. New test fails pre-fix (skipped to next day).
- config/types: maxTurns error message now reads "must be >= 0" to match the check
(0 is accepted; only < 0 is rejected). Negative-maxTurns test asserts the message.
- docs/audit: header date/branch metadata now matches the body (2026-06-13,
fix/audit-batch) instead of the stale 2026-06-11/fix/audit-critical-high snapshot.
Verified: gofmt clean; go vet clean; host + linux + windows builds; affected suites
green. Each behavioural fix ships with a test confirmed to fail without the change.
* audit batch 37: address CodeRabbit re-review (12 findings)
Findings on the audit code from batches 21-36; each behavioural fix ships with a
regression test (several confirmed to fail without the change).
sandbox/analyzer (AnalyzeCommand):
- Unwrap launcher prefixes (sudo/env/nice/bash -c/...) and recurse into `sh -c`
payloads so `sudo rm -rf`, `env curl …`, `bash -c 'vim x'` classify on the real
command, not the launcher. Mirrors DetectInteractiveCommand.
- Honor rm's `--` end-of-options marker so `rm -- -rf` (delete a file named -rf) is
not flagged destructive.
sandbox/engine: NetworkHostAllowed now shares one backend-aware effective-mode
helper with Evaluate, so scoped collapses to deny when the backend can't enforce
scoped egress — the per-tool gate can no longer permit traffic the engine-level
decision would fail closed.
sandbox/egress: cache domain-prompt decisions by host:port, not host, so one
"allow" can't authorize other ports on the same host.
sandbox/safe_command: the segment splitter is now escape-aware (\| , \" ) so an
escaped operator/quote can't manufacture a fake interactive segment.
sandbox/log_monitor (darwin): Wait() the `log stream` child after draining stdout
so the cancelled process isn't leaked as a zombie.
tools/web_search: confine the search backend to same-host redirects (with a limit)
so a redirect chain can't egress to a host the scoped/deny policy never authorized.
cli/sandbox: an explicit empty --path (`--path=` / `--path ""`) now fails closed
instead of silently widening an allow to tool-wide or a revoke to all-grants.
cli/skills: warn on stderr when duplicate-skill detection fails instead of
swallowing the error.
specialist/exec: don't mask a failed orphan-kill — join the TerminateProcess error
with the SetPID error so a surviving process is visible.
background/process_posix: resolve the PGID and only group-signal when pid leads its
own group; otherwise signal the individual PID. Avoids both a false-success on a
non-leader ESRCH and the hazard of signalling an unrelated (possibly our own) group.
Verified: gofmt + go vet clean; host + linux + windows builds; full go test ./...
green.
* audit batch 38: address CodeRabbit re-review round 2 (3 findings)
Follow-ups to batch 37, each with a regression test:
- safe_command (splitter): a ')' is now only a substitution boundary when it
closes an active, UNQUOTED $(...). A quote-state stack saved at each '$(' lets a
substitution run in a fresh quoting context and restores the outer quoting on
its matching ')'. Fixes `echo $(printf "a) less")` splitting into a fake `less`
segment, while still isolating the inner command of `echo "$(vim x)"`.
- egress: deduplicate in-flight domain prompts per host:port. Concurrent requests
for the same unknown target now wait on the first prompt and reuse its cached
decision instead of each firing a prompt (a prompt storm) and racing to write
the final decision. Race-tested with 25 concurrent authorizers → one prompt.
- analyzer + safe_command: wrapper option-value consumption is now wrapper-specific
(wrapperValueOptionsByProg) instead of a global option set. A valueless flag like
`sudo -n` no longer swallows the real payload, so `sudo -n rm -rf` /
`sudo -n curl …` are correctly classified destructive/network; `sudo -u root vim`
still consumes its value. Shared by the AST analyzer and the regex detector so
they can't diverge.
Verified: gofmt + go vet clean; host + linux + windows builds; go test ./... and
-race on sandbox/background all green.
* audit batch 39: address CodeRabbit re-review round 3 (12 findings)
All verified against current code; each behavioural fix has a regression test.
Security / correctness:
- tools/web_search: redirect policy now also refuses a scheme change, blocking a
same-host https→http downgrade that would send the query + bearer token over
plaintext.
- sandbox/safe_command: a ')' is only a substitution boundary when it closes an
active UNQUOTED $(...), and backticks now get the same quote-state save/restore
as $( — so `echo "`a | less`"` and `echo "$(true | less)"` no longer hide the
inner pager from interactive detection.
- sandbox/analyzer + safe_command: wrapper option-value consumption stays
wrapper-specific, and unwrapping no longer aborts on a dynamic ($x) wrapper arg
once a wrapper is active, so `env "$opts" curl` / `sudo "$x" rm -rf` are still
classified.
- sandbox/egress: in-flight prompts deduped per host:port (carried from batch 38);
the prompt callback is now cancellable (context cancelled on timeout) so a stuck
prompt can abort instead of leaking; signature threaded through.
- sandbox/runner: per-plan unique denial tag (pid+counter) so concurrent monitored
sandbox-exec runs can't ingest each other's denials; helper selection now
requires an executable regular file so a non-exec sibling degrades gracefully.
- mcp/network_client: aggregate SSE event-size cap (not just per-line) so a server
can't grow memory with many unterminated data: lines.
- specialist/accounting: a previously-recorded stop/usage with an empty runId is a
catch-all and now suppresses a later runId-bearing duplicate.
- sessions/lineage: Tree fetches the root via Get() first so a corrupt root
surfaces its real error instead of degrading to "not found".
Docs/tests:
- background/terminate: docstring now states the leader-only group-kill (PID-only
fallback for non-leaders); test child-liveness probe uses ESRCH specifically.
- cli/sandbox: grant usage strings include --path.
Verified: gofmt + go vet clean; host + linux + windows builds; go test ./... and
-race on sandbox/background all green.
* audit batch 40: address CodeRabbit re-review round 4 (8 fixed, 1 skipped)
Verified each against current code.
Fixed:
- sessions/lineage: PANIC FIX — store.Get returns (nil, nil) for a missing
session, so the batch-39 root prefetch could nil-deref. Guard root == nil and
return a not-found error. Regression test added (was a panic).
- agent/compaction: the reactive recover() now records lowWaterMark in the SAME
combined (messages + tool-defs) token domain maybeCompact uses, so the proactive
shrink-guard compares like with like. recover() takes tools; callers pass
request.Tools.
- mcp/network_client: SSE aggregate counter no longer counts a joining newline for
the first data line (only when prior content exists), avoiding premature rejects.
- sessions/store: fsync the parent directory after the metadata rename so the
rename is durable across a crash (no-op on Windows, which can't fsync a dir).
- sandbox/egress: parse the target port with strconv.Atoi instead of
net.LookupPort (no context-unaware resolver use).
- sandbox/runner: escape the per-plan denial tag through sandboxProfileString when
embedding it in the seatbelt profile.
- sandbox/seccomp_linux: guard against an empty filter program before taking
&kernelFilters[0] (defensive; the program is fixed non-empty today).
- cli/cron_run_test: assert last-error-wins with two error events.
- background/terminate, cli/sandbox usage, process_posix_test ESRCH (carried).
Skipped (with reason):
- zerogit worktree-rename (Y='R') test: git porcelain v1 -z only emits the
rename-source field when the INDEX column is R/C; the worktree column never
carries 'R' with a source field in v1, so the fixture would test output git
doesn't produce, and making parseStatus consume on code[1]=='R' would risk
mis-consuming the following entry for a plain ` R` status. Left as-is.
Verified: gofmt + go vet clean; host + linux + windows builds; go test ./... and
-race on sandbox/background all green.
* audit batch 41: address CodeRabbit re-review round 5 (4 findings)
All verified against current code; code fixes have regression tests.
- agent/compaction: summarizeWithFallback's reduce pass no longer swallows ALL
errors. A context-limit error still falls back to the joined partial summaries
(documented "extreme" case), but a non-context failure (auth/network/provider)
now surfaces unchanged, per the function's contract. Two tests (propagate vs.
joined-fallback); the propagate test fails without the fix.
- sandbox/safe_command + analyzer: wrapper value-consuming options now include the
long spellings (sudo --user root, env --unset NAME, timeout --signal …), so a
long flag's value is no longer mis-read as the program (interactive/destructive/
network detection bypass). A `--flag=value` token carries its own value and never
consumes the next token. Tests for space and = forms in both the AST analyzer and
the regex detector.
- docs/audit: the status-ledger summary table's open/partial cells (64/14) were the
lone outlier — the ledger's own narrative and 2026-06-13-reverification.md both
state 66/12. Corrected to 66/12 (66+12+97 = 175); derivation: body 69 open + 18
partial minus the 9 already-fixed-on-main (3 open + 6 partial) = 66/12 = 78
pending, fixed 88+9 = 97. Added a comment documenting the summary-vs-body
relationship. The reverification doc already states 66/12, so both now agree.
Verified: gofmt + go vet clean; host + linux + windows builds; full go test ./...
green.
* audit batch 42: address CodeRabbit re-review round 6 (1 fixed, 1 kept w/ proof)
CodeRabbit re-surfaced two "duplicate" refinements; CI was already green (its 6th
pass was a COMMENT with no new findings).
- cli/sandbox_test (TestRunSandboxGrantsRejectsEmptyPath): now seeds a tool-wide
grant and asserts grant-store immutability after EACH rejected empty-`--path`
call (reflect.DeepEqual before/after), so a mutation in one call can't be masked
by a later one — and a buggy "revoke all" is actually observable (it's a no-op on
an empty store). Stronger than the suggested patch.
- zerogit.parseStatus: KEPT consuming the rename source on the index column only
(code[0]), with an explanatory comment. Verified empirically that git porcelain
v1 -z reports a rename/copy (and emits the extra source field) only in the index
column: `git mv` → "R new\0old\0"; a worktree-only rename → " D old\0?? new\0"
(delete + untracked, never "R" in code[1]). Consuming on code[1]=='R'/'C' would
match no real git output and would only risk mis-consuming the next entry on
malformed input, so the suggestion is declined as incorrect for v1.
Verified: gofmt + go vet clean; host + linux + windows builds; affected suites green.
1931 lines
67 KiB
Go
1931 lines
67 KiB
Go
package cli
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"encoding/json"
|
|
"errors"
|
|
"io"
|
|
"net/http"
|
|
"net/http/httptest"
|
|
"os"
|
|
"path/filepath"
|
|
"slices"
|
|
"strings"
|
|
"testing"
|
|
|
|
"github.com/Gitlawb/zero/internal/agent"
|
|
"github.com/Gitlawb/zero/internal/config"
|
|
"github.com/Gitlawb/zero/internal/modelregistry"
|
|
"github.com/Gitlawb/zero/internal/sessions"
|
|
"github.com/Gitlawb/zero/internal/zeroruntime"
|
|
)
|
|
|
|
func TestRunExecHelpDocumentsM1Flags(t *testing.T) {
|
|
for _, args := range [][]string{
|
|
{"exec", "--help"},
|
|
{"exec", "--help", "--model", "m1"},
|
|
} {
|
|
t.Run(strings.Join(args, " "), func(t *testing.T) {
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
|
|
exitCode := Run(args, &stdout, &stderr)
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d", exitCode)
|
|
}
|
|
for _, want := range []string{
|
|
"-f, --file",
|
|
"--image <path>",
|
|
"--mode <name>",
|
|
"-m, --model",
|
|
"--max-turns",
|
|
"--profile <profile>",
|
|
"-r, --reasoning-effort <effort>",
|
|
"-C, --cwd",
|
|
"-o, --output-format text|json",
|
|
"--prompt",
|
|
"--calling-session-id",
|
|
"--calling-tool-use-id",
|
|
"--tag <tag>",
|
|
"--depth <number>",
|
|
"--session-title",
|
|
"--init-session-id",
|
|
"--skip-permissions-unsafe",
|
|
} {
|
|
if !strings.Contains(stdout.String(), want) {
|
|
t.Fatalf("expected exec help to contain %q, got %q", want, stdout.String())
|
|
}
|
|
}
|
|
if stderr.Len() != 0 {
|
|
t.Fatalf("expected empty stderr, got %q", stderr.String())
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestRunExecRejectsInvalidMaxTurnsBeforeRuntime(t *testing.T) {
|
|
for _, tc := range []struct {
|
|
value string
|
|
want string
|
|
}{
|
|
{value: "nope", want: "invalid --max-turns"},
|
|
{value: "-1", want: "invalid --max-turns"},
|
|
{value: "0", want: "invalid --max-turns"},
|
|
} {
|
|
t.Run(tc.value, func(t *testing.T) {
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
|
|
exitCode := Run([]string{"exec", "--max-turns", tc.value, "hello"}, &stdout, &stderr)
|
|
|
|
if exitCode != exitUsage {
|
|
t.Fatalf("expected exit code %d, got %d", exitUsage, exitCode)
|
|
}
|
|
if stdout.Len() != 0 {
|
|
t.Fatalf("expected empty stdout before runtime, got %q", stdout.String())
|
|
}
|
|
if got := stderr.String(); !strings.Contains(got, tc.want) {
|
|
t.Fatalf("expected max-turns validation error containing %q, got %q", tc.want, got)
|
|
}
|
|
})
|
|
}
|
|
|
|
t.Run("equals-empty", func(t *testing.T) {
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
|
|
exitCode := Run([]string{"exec", "--max-turns=", "hello"}, &stdout, &stderr)
|
|
|
|
if exitCode != exitUsage {
|
|
t.Fatalf("expected exit code %d, got %d", exitUsage, exitCode)
|
|
}
|
|
if stdout.Len() != 0 {
|
|
t.Fatalf("expected empty stdout before runtime, got %q", stdout.String())
|
|
}
|
|
if got := stderr.String(); !strings.Contains(got, "--max-turns requires a value") {
|
|
t.Fatalf("expected empty max-turns validation error, got %q", got)
|
|
}
|
|
})
|
|
}
|
|
|
|
func TestRunExecMaxTurnsReachesConfigOverrides(t *testing.T) {
|
|
cwd := t.TempDir()
|
|
var gotMaxTurns int
|
|
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{"exec", "--max-turns", "7", "hello"}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
gotMaxTurns = overrides.MaxTurns
|
|
return config.ResolvedConfig{}, errors.New("stop before provider")
|
|
},
|
|
})
|
|
|
|
if exitCode != exitProvider {
|
|
t.Fatalf("expected provider exit %d, got %d", exitProvider, exitCode)
|
|
}
|
|
if gotMaxTurns != 7 {
|
|
t.Fatalf("overrides.MaxTurns = %d, want 7", gotMaxTurns)
|
|
}
|
|
}
|
|
|
|
func TestParseExecSpecialistMetadataFlags(t *testing.T) {
|
|
options, help, err := parseExecArgs([]string{
|
|
"--calling-session-id", "parent_session",
|
|
"--calling-tool-use-id=toolu_123",
|
|
"--tag", "specialist",
|
|
"--depth=2",
|
|
"--session-title", "Explorer child",
|
|
"--init-session-id", "child_session",
|
|
"--output-format", "debug",
|
|
"inspect the parser",
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("parseExecArgs returned error: %v", err)
|
|
}
|
|
if help {
|
|
t.Fatal("help = true, want false")
|
|
}
|
|
if options.callingSessionID != "parent_session" ||
|
|
options.callingToolUseID != "toolu_123" ||
|
|
options.tag != "specialist" ||
|
|
options.depth != 2 ||
|
|
options.sessionTitle != "Explorer child" ||
|
|
options.initSessionID != "child_session" {
|
|
t.Fatalf("metadata flags did not parse correctly: %#v", options)
|
|
}
|
|
if options.outputFormat != execOutputStreamJSON {
|
|
t.Fatalf("outputFormat = %q, want stream-json debug alias", options.outputFormat)
|
|
}
|
|
if strings.Join(options.promptParts, " ") != "inspect the parser" {
|
|
t.Fatalf("promptParts = %#v", options.promptParts)
|
|
}
|
|
}
|
|
|
|
func TestParseExecSpecialistMetadataRejectsInvalidValues(t *testing.T) {
|
|
tests := []struct {
|
|
name string
|
|
args []string
|
|
want string
|
|
}{
|
|
{name: "negative depth", args: []string{"--depth=-1", "hello"}, want: "invalid --depth"},
|
|
{name: "non numeric depth", args: []string{"--depth", "many", "hello"}, want: "invalid --depth"},
|
|
{name: "empty tag", args: []string{"--tag=", "hello"}, want: "--tag requires a value"},
|
|
{name: "bad init session", args: []string{"--init-session-id", "../escape", "hello"}, want: "invalid --init-session-id"},
|
|
{name: "init with resume", args: []string{"--init-session-id", "child", "--resume", "parent", "hello"}, want: "Use --init-session-id only"},
|
|
}
|
|
for _, tc := range tests {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
_, _, err := parseExecArgs(tc.args)
|
|
if err == nil || !strings.Contains(err.Error(), tc.want) {
|
|
t.Fatalf("expected %q error, got %v", tc.want, err)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestRunExecRegistersTaskOnlyForUnsafeTopLevelRuns(t *testing.T) {
|
|
tests := []struct {
|
|
name string
|
|
args []string
|
|
wantTask bool
|
|
}{
|
|
{name: "default headless", args: []string{"exec", "--list-tools"}, wantTask: false},
|
|
{name: "unsafe headless", args: []string{"exec", "--auto", "high", "--list-tools"}, wantTask: true},
|
|
{name: "specialist child", args: []string{"exec", "--auto", "high", "--tag", "specialist", "--list-tools"}, wantTask: false},
|
|
}
|
|
for _, tc := range tests {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps(tc.args, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return t.TempDir(), nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
for _, toolName := range []string{"Task", "TaskOutput", "TaskStop", "GenerateSpecialist"} {
|
|
hasTool := strings.Contains(stdout.String(), " "+toolName+" ")
|
|
if hasTool != tc.wantTask {
|
|
t.Fatalf("%s visibility = %v, want %v; output:\n%s", toolName, hasTool, tc.wantTask, stdout.String())
|
|
}
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestRunExecUsesInitSessionIDAndSessionTitle(t *testing.T) {
|
|
dataHome := t.TempDir()
|
|
t.Setenv("XDG_DATA_HOME", dataHome)
|
|
cwd := t.TempDir()
|
|
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{
|
|
"exec",
|
|
"--init-session-id", "specialist_child",
|
|
"--session-title", "Explorer child",
|
|
"--tag", "specialist",
|
|
"--depth", "1",
|
|
"hello",
|
|
}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, _ config.Overrides) (config.ResolvedConfig, error) {
|
|
return execResolvedConfig(), nil
|
|
},
|
|
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
return echoExecProvider{}, nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
|
|
store := sessions.NewStore(sessions.StoreOptions{RootDir: filepath.Join(dataHome, "zero", "sessions")})
|
|
session, err := store.Get("specialist_child")
|
|
if err != nil {
|
|
t.Fatalf("Get session returned error: %v", err)
|
|
}
|
|
if session == nil {
|
|
t.Fatal("expected initialized session specialist_child")
|
|
}
|
|
if session.Title != "Explorer child" || session.Tag != "specialist" || session.Depth != 1 {
|
|
t.Fatalf("session metadata = %#v, want title/tag/depth", session)
|
|
}
|
|
if session.Cwd != cwd {
|
|
t.Fatalf("session cwd = %q, want %q", session.Cwd, cwd)
|
|
}
|
|
}
|
|
|
|
func TestRunExecPersistsCallingSessionChildMetadata(t *testing.T) {
|
|
dataHome := t.TempDir()
|
|
t.Setenv("XDG_DATA_HOME", dataHome)
|
|
store := sessions.NewStore(sessions.StoreOptions{RootDir: filepath.Join(dataHome, "zero", "sessions")})
|
|
parent, err := store.Create(sessions.CreateInput{SessionID: "parent_session", Title: "Parent", Cwd: "/repo", ModelID: "gpt-parent", Provider: "openai"})
|
|
if err != nil {
|
|
t.Fatalf("Create parent returned error: %v", err)
|
|
}
|
|
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{
|
|
"exec",
|
|
"--output-format", "stream-json",
|
|
"--init-session-id", "child_session",
|
|
"--session-title", "worker: Auth check",
|
|
"--tag", "specialist",
|
|
"--depth", "1",
|
|
"--calling-session-id", parent.SessionID,
|
|
"--calling-tool-use-id", "toolu_123",
|
|
"hello",
|
|
}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return t.TempDir(), nil
|
|
},
|
|
resolveConfig: func(_ string, _ config.Overrides) (config.ResolvedConfig, error) {
|
|
return execResolvedConfig(), nil
|
|
},
|
|
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
return echoExecProvider{}, nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
|
|
child, err := store.Get("child_session")
|
|
if err != nil {
|
|
t.Fatalf("Get child returned error: %v", err)
|
|
}
|
|
if child == nil {
|
|
t.Fatal("expected child session metadata")
|
|
}
|
|
if child.SessionKind != sessions.SessionKindChild ||
|
|
child.ParentSessionID != parent.SessionID ||
|
|
child.RootSessionID != parent.SessionID ||
|
|
child.AgentName != "worker" ||
|
|
child.TaskID != "child_session" ||
|
|
child.SpawnedFromEventID != "toolu_123" ||
|
|
child.Tag != "specialist" ||
|
|
child.Depth != 1 {
|
|
t.Fatalf("unexpected child metadata: %#v", child)
|
|
}
|
|
}
|
|
|
|
func TestRunExecModeSeedsModelAndTurnOverrides(t *testing.T) {
|
|
cwd := t.TempDir()
|
|
var gotModel string
|
|
var gotMaxTurns int
|
|
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{"exec", "--mode", "deep", "hello"}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
gotModel = overrides.Provider.Model
|
|
gotMaxTurns = overrides.MaxTurns
|
|
return config.ResolvedConfig{}, errors.New("stop before provider")
|
|
},
|
|
})
|
|
|
|
if exitCode != exitProvider {
|
|
t.Fatalf("expected provider exit %d, got %d", exitProvider, exitCode)
|
|
}
|
|
if gotModel != "claude-opus-4.1" {
|
|
t.Fatalf("overrides.Provider.Model = %q, want claude-opus-4.1", gotModel)
|
|
}
|
|
if gotMaxTurns != 50 {
|
|
t.Fatalf("overrides.MaxTurns = %d, want 50", gotMaxTurns)
|
|
}
|
|
}
|
|
|
|
func TestRunExecExplicitModelOverridesMode(t *testing.T) {
|
|
cwd := t.TempDir()
|
|
var gotModel string
|
|
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{"exec", "--mode", "deep", "--model", "gpt-4.1", "hello"}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
gotModel = overrides.Provider.Model
|
|
return config.ResolvedConfig{}, errors.New("stop before provider")
|
|
},
|
|
})
|
|
|
|
if exitCode != exitProvider {
|
|
t.Fatalf("expected provider exit %d, got %d", exitProvider, exitCode)
|
|
}
|
|
if gotModel != "gpt-4.1" {
|
|
t.Fatalf("explicit --model should override mode: got %q, want gpt-4.1", gotModel)
|
|
}
|
|
}
|
|
|
|
func TestRunExecModeRoutesModelThroughRegistry(t *testing.T) {
|
|
cwd := t.TempDir()
|
|
var gotModel string
|
|
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
// "smart" maps to claude-sonnet-4.5; the mode's model must be routed through
|
|
// the registry (Resolve) so the canonical id reaches the overrides.
|
|
exitCode := runWithDeps([]string{"exec", "--mode", "smart", "hi"}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
gotModel = overrides.Provider.Model
|
|
return config.ResolvedConfig{}, errors.New("stop before provider")
|
|
},
|
|
})
|
|
|
|
if exitCode != exitProvider {
|
|
t.Fatalf("expected provider exit %d, got %d", exitProvider, exitCode)
|
|
}
|
|
if gotModel != "claude-sonnet-4.5" {
|
|
t.Fatalf("expected mode smart to select claude-sonnet-4.5, got %q", gotModel)
|
|
}
|
|
}
|
|
|
|
func TestRunExecUnknownModeErrors(t *testing.T) {
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
|
|
exitCode := Run([]string{"exec", "--mode", "turbo", "hello"}, &stdout, &stderr)
|
|
|
|
if exitCode != exitUsage {
|
|
t.Fatalf("expected usage exit %d, got %d", exitUsage, exitCode)
|
|
}
|
|
if !strings.Contains(stderr.String(), "unknown mode") {
|
|
t.Fatalf("expected unknown mode error, got %q", stderr.String())
|
|
}
|
|
for _, want := range []string{"smart", "deep", "fast", "large", "precise"} {
|
|
if !strings.Contains(stderr.String(), want) {
|
|
t.Fatalf("expected error to list valid mode %q, got %q", want, stderr.String())
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestApplyExecModeLeavesRawModelForSharedResolution(t *testing.T) {
|
|
// applyExecMode must NOT pre-resolve the mode's model: leaving the raw id/alias
|
|
// lets the shared --model resolution path resolve it AND surface any deprecation
|
|
// notice on stderr (the bug was that applyExecMode resolved it and discarded the
|
|
// notice). The raw alias must be the preset's exact Model value.
|
|
mode, ok := modelregistry.LookupMode("deep")
|
|
if !ok {
|
|
t.Fatal("expected built-in mode deep")
|
|
}
|
|
options := execOptions{mode: "deep"}
|
|
if err := applyExecMode(&options); err != nil {
|
|
t.Fatalf("applyExecMode returned error: %v", err)
|
|
}
|
|
if options.model != mode.Model {
|
|
t.Fatalf("options.model = %q, want raw mode model %q (resolution must be delegated)", options.model, mode.Model)
|
|
}
|
|
}
|
|
|
|
func TestRunExecModeModelSurfacesDeprecationNoticeViaSharedPath(t *testing.T) {
|
|
// A mode-supplied model must flow through the same resolution path as an
|
|
// explicit --model, so a deprecated id redirects AND prints a notice. No
|
|
// built-in mode references a deprecated model, so emulate one by setting the
|
|
// raw mode model directly through applyExecMode and threading it through the
|
|
// shared resolver, exactly as runExec does after the reorder.
|
|
options := execOptions{mode: "smart"}
|
|
if err := applyExecMode(&options); err != nil {
|
|
t.Fatalf("applyExecMode returned error: %v", err)
|
|
}
|
|
// Sanity: the shared resolver surfaces a notice + redirect for a deprecated id,
|
|
// which is the path the mode model now feeds into.
|
|
registry, err := modelregistry.DefaultRegistry()
|
|
if err != nil {
|
|
t.Fatalf("DefaultRegistry: %v", err)
|
|
}
|
|
resolved, notice := resolveSelectedModel(registry, "gpt-4-turbo")
|
|
if resolved != "gpt-4.1" {
|
|
t.Fatalf("expected deprecated model to redirect to gpt-4.1, got %q", resolved)
|
|
}
|
|
if !strings.Contains(notice, "deprecated") {
|
|
t.Fatalf("expected shared resolver to surface a deprecation notice, got %q", notice)
|
|
}
|
|
}
|
|
|
|
func TestRunExecListToolsAppliesModeBeforeListing(t *testing.T) {
|
|
// applyExecMode now runs before tool-filter validation and the --list-tools
|
|
// branch, so a --mode preset is expanded for --list-tools. Combining a mode
|
|
// with --list-tools must still succeed and never resolve a provider.
|
|
cwd := t.TempDir()
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
|
|
exitCode := runWithDeps([]string{"exec", "--list-tools", "--mode", "deep"}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(string, config.Overrides) (config.ResolvedConfig, error) {
|
|
return config.ResolvedConfig{}, errors.New("provider should not be resolved for --list-tools")
|
|
},
|
|
})
|
|
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("expected exit code %d, got %d: %s", exitSuccess, exitCode, stderr.String())
|
|
}
|
|
if !strings.Contains(stdout.String(), "Tools visible to model") {
|
|
t.Fatalf("expected --list-tools --mode to list tools, got %q", stdout.String())
|
|
}
|
|
}
|
|
|
|
func TestRunExecModeToolFilterReflectedInListTools(t *testing.T) {
|
|
// The reorder also means a mode-injected tool filter would be reflected in
|
|
// --list-tools. No built-in mode ships a tool filter, so drive the equivalent
|
|
// surface through applyExecMode + formatExecToolList: a filter seeded onto the
|
|
// options before the listing must narrow the tools the model can see.
|
|
options := execOptions{enabledTools: []string{"read_file", "grep"}}
|
|
registry := newCoreRegistry(t.TempDir())
|
|
list := formatExecToolList(registry, options, agent.PermissionModeAuto)
|
|
for _, want := range []string{"read_file", "grep"} {
|
|
if !strings.Contains(list, want) {
|
|
t.Fatalf("expected tool list to contain %q, got %q", want, list)
|
|
}
|
|
}
|
|
if strings.Contains(list, "bash") {
|
|
t.Fatalf("expected mode-style tool filter to hide bash, got %q", list)
|
|
}
|
|
}
|
|
|
|
func TestRunExecAcceptsLegacyModelProfileFlags(t *testing.T) {
|
|
exitCode, stdout, stderr := runExecWithEcho(t, []string{
|
|
"exec",
|
|
"--profile",
|
|
"fast",
|
|
"--reasoning-effort",
|
|
"low",
|
|
"hello",
|
|
})
|
|
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("expected exit code %d, got %d: %s", exitSuccess, exitCode, stderr)
|
|
}
|
|
if !strings.Contains(stdout, "hello") {
|
|
t.Fatalf("expected prompt output, got %q", stdout)
|
|
}
|
|
if stderr != "" {
|
|
t.Fatalf("expected empty stderr, got %q", stderr)
|
|
}
|
|
}
|
|
|
|
func TestRunExecJSONRunStartWriteFailureSkipsAgent(t *testing.T) {
|
|
cwd := t.TempDir()
|
|
called := false
|
|
|
|
exitCode := runWithDeps([]string{"exec", "-o", "json", "hello"}, failingWriter{}, io.Discard, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, _ config.Overrides) (config.ResolvedConfig, error) {
|
|
return execResolvedConfig(), nil
|
|
},
|
|
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
return recordingExecProvider{called: &called}, nil
|
|
},
|
|
})
|
|
|
|
if exitCode != exitCrash {
|
|
t.Fatalf("expected exit code %d, got %d", exitCrash, exitCode)
|
|
}
|
|
if called {
|
|
t.Fatal("expected agent provider not to run after run_start write failure")
|
|
}
|
|
}
|
|
|
|
func TestRunExecUnsafeWarningWriteFailureSkipsAgent(t *testing.T) {
|
|
cwd := t.TempDir()
|
|
called := false
|
|
|
|
exitCode := runWithDeps([]string{"exec", "--skip-permissions-unsafe", "hello"}, io.Discard, failingWriter{}, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, _ config.Overrides) (config.ResolvedConfig, error) {
|
|
return execResolvedConfig(), nil
|
|
},
|
|
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
return recordingExecProvider{called: &called}, nil
|
|
},
|
|
})
|
|
|
|
if exitCode != exitCrash {
|
|
t.Fatalf("expected exit code %d, got %d", exitCrash, exitCode)
|
|
}
|
|
if called {
|
|
t.Fatal("expected agent provider not to run after warning write failure")
|
|
}
|
|
}
|
|
|
|
func TestRunExecJSONProviderErrorWriteFailureReturnsCrash(t *testing.T) {
|
|
cwd := t.TempDir()
|
|
|
|
exitCode := runWithDeps([]string{"exec", "-o", "json", "hello"}, failingWriter{}, io.Discard, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, _ config.Overrides) (config.ResolvedConfig, error) {
|
|
return config.ResolvedConfig{}, errors.New("provider config failed")
|
|
},
|
|
})
|
|
|
|
if exitCode != exitCrash {
|
|
t.Fatalf("expected exit code %d, got %d", exitCrash, exitCode)
|
|
}
|
|
}
|
|
|
|
func execResolvedConfig() config.ResolvedConfig {
|
|
return config.ResolvedConfig{
|
|
ActiveProvider: "echo",
|
|
Provider: config.ProviderProfile{
|
|
Name: "echo",
|
|
ProviderKind: config.ProviderKindOpenAICompatible,
|
|
BaseURL: "http://127.0.0.1/v1",
|
|
Model: "echo-model",
|
|
},
|
|
}
|
|
}
|
|
|
|
type recordingExecProvider struct {
|
|
called *bool
|
|
}
|
|
|
|
func (provider recordingExecProvider) StreamCompletion(context.Context, zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
|
*provider.called = true
|
|
return nil, errors.New("provider should not run")
|
|
}
|
|
|
|
func TestRunPromptFlagRoutesToExecRunner(t *testing.T) {
|
|
execExitCode, execStdout, execStderr := runExecWithEcho(t, []string{"exec", "hello zero"})
|
|
|
|
for _, args := range [][]string{
|
|
{"-p", "hello zero"},
|
|
{"--prompt", "hello zero"},
|
|
} {
|
|
t.Run(args[0], func(t *testing.T) {
|
|
exitCode, stdout, stderr := runExecWithEcho(t, args)
|
|
|
|
if exitCode != execExitCode {
|
|
t.Fatalf("expected exit code %d, got %d", execExitCode, exitCode)
|
|
}
|
|
if stdout != execStdout {
|
|
t.Fatalf("expected stdout %q, got %q", execStdout, stdout)
|
|
}
|
|
if stderr != execStderr {
|
|
t.Fatalf("expected stderr %q, got %q", execStderr, stderr)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestRunExecAssemblesInlineAndFilePromptRelativeToCwd(t *testing.T) {
|
|
root := t.TempDir()
|
|
if err := os.WriteFile(filepath.Join(root, "prompt.txt"), []byte("file prompt\n"), 0o600); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
|
|
exitCode, stdout, stderr := runExecWithEcho(t, []string{"exec", "--cwd", root, "--file", "prompt.txt", "inline prompt"})
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d: %s", exitCode, stderr)
|
|
}
|
|
if !strings.Contains(stdout, "inline prompt\n\nfile prompt") {
|
|
t.Fatalf("expected inline and file prompt joined by blank line, got %q", stdout)
|
|
}
|
|
if stderr != "" {
|
|
t.Fatalf("expected empty stderr, got %q", stderr)
|
|
}
|
|
}
|
|
|
|
func TestRunExecAcceptsFileOnlyPrompt(t *testing.T) {
|
|
root := t.TempDir()
|
|
if err := os.WriteFile(filepath.Join(root, "prompt.txt"), []byte("file only prompt\n"), 0o600); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
|
|
exitCode, stdout, stderr := runExecWithEcho(t, []string{"exec", "-C", root, "-f", "prompt.txt"})
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d: %s", exitCode, stderr)
|
|
}
|
|
if !strings.Contains(stdout, "file only prompt") {
|
|
t.Fatalf("expected file prompt output, got %q", stdout)
|
|
}
|
|
if stderr != "" {
|
|
t.Fatalf("expected empty stderr, got %q", stderr)
|
|
}
|
|
}
|
|
|
|
func TestRunExecRejectsInvalidCwdBeforeRuntime(t *testing.T) {
|
|
root := t.TempDir()
|
|
filePath := filepath.Join(root, "not-a-directory.txt")
|
|
if err := os.WriteFile(filePath, []byte("nope"), 0o600); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
|
|
for _, tc := range []struct {
|
|
name string
|
|
cwd string
|
|
}{
|
|
{name: "missing", cwd: filepath.Join(root, "missing")},
|
|
{name: "file", cwd: filePath},
|
|
} {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
|
|
exitCode := Run([]string{"exec", "--cwd", tc.cwd, "hello"}, &stdout, &stderr)
|
|
|
|
if exitCode != 2 {
|
|
t.Fatalf("expected exit code 2, got %d", exitCode)
|
|
}
|
|
if stdout.Len() != 0 {
|
|
t.Fatalf("expected empty stdout before runtime, got %q", stdout.String())
|
|
}
|
|
if got := stderr.String(); !strings.Contains(got, "cwd must be an existing directory") {
|
|
t.Fatalf("expected cwd validation error, got %q", got)
|
|
}
|
|
if strings.Contains(stdout.String()+stderr.String(), "Go agent runtime ready") {
|
|
t.Fatalf("expected validation before runtime, got stdout %q stderr %q", stdout.String(), stderr.String())
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestRunExecRejectsInvalidOutputFormatBeforeRuntime(t *testing.T) {
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
|
|
exitCode := Run([]string{"exec", "-o", "yaml", "hello"}, &stdout, &stderr)
|
|
|
|
if exitCode != 2 {
|
|
t.Fatalf("expected exit code 2, got %d", exitCode)
|
|
}
|
|
if stdout.Len() != 0 {
|
|
t.Fatalf("expected empty stdout before runtime, got %q", stdout.String())
|
|
}
|
|
if got := stderr.String(); !strings.Contains(got, `invalid output format "yaml"`) || !strings.Contains(got, "debug") {
|
|
t.Fatalf("expected output format validation error, got %q", got)
|
|
}
|
|
if strings.Contains(stdout.String()+stderr.String(), "Go agent runtime ready") {
|
|
t.Fatalf("expected validation before runtime, got stdout %q stderr %q", stdout.String(), stderr.String())
|
|
}
|
|
}
|
|
|
|
func TestRunExecUnsafeTextModeWarns(t *testing.T) {
|
|
exitCode, stdout, stderr := runExecWithEcho(t, []string{"exec", "--skip-permissions-unsafe", "hello"})
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d: %s", exitCode, stderr)
|
|
}
|
|
if !strings.Contains(stdout, "hello") {
|
|
t.Fatalf("expected prompt in stdout, got %q", stdout)
|
|
}
|
|
if got := stderr; !strings.Contains(got, "WARNING") || !strings.Contains(got, "--skip-permissions-unsafe") {
|
|
t.Fatalf("expected unsafe warning, got %q", got)
|
|
}
|
|
}
|
|
|
|
func TestRunExecJSONOutputsNDJSONEvents(t *testing.T) {
|
|
root := t.TempDir()
|
|
|
|
exitCode, stdout, stderr := runExecWithEcho(t, []string{"exec", "--cwd", root, "-m", "m1-test", "-o", "json", "hello json"})
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d: %s", exitCode, stderr)
|
|
}
|
|
if stderr != "" {
|
|
t.Fatalf("expected empty stderr, got %q", stderr)
|
|
}
|
|
|
|
events := decodeJSONLines(t, stdout)
|
|
eventTypes := jsonEventTypes(events)
|
|
for _, want := range []string{"run_start", "text", "final", "done"} {
|
|
if !slices.Contains(eventTypes, want) {
|
|
t.Fatalf("expected JSON event %q in %v; output %q", want, eventTypes, stdout)
|
|
}
|
|
}
|
|
if got := events[0]["type"]; got != "run_start" {
|
|
t.Fatalf("expected first event run_start, got %v", got)
|
|
}
|
|
if got := events[0]["model"]; got != "m1-test" {
|
|
t.Fatalf("expected run_start model m1-test, got %v", got)
|
|
}
|
|
if got := events[0]["cwd"]; got != root {
|
|
t.Fatalf("expected run_start cwd %q, got %v", root, got)
|
|
}
|
|
if got := events[0]["permission_mode"]; got != "auto" {
|
|
t.Fatalf("expected default permission_mode auto, got %v", got)
|
|
}
|
|
}
|
|
|
|
func TestRunExecResolvesCanonicalModelAlias(t *testing.T) {
|
|
root := t.TempDir()
|
|
|
|
// "openai:gpt-4.1" is a registry alias for the canonical gpt-4.1 id; the
|
|
// selection boundary should normalize it before the provider sees it.
|
|
exitCode, stdout, stderr := runExecWithEcho(t, []string{"exec", "--cwd", root, "-m", "openai:gpt-4.1", "-o", "json", "hi"})
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d: %s", exitCode, stderr)
|
|
}
|
|
if stderr != "" {
|
|
t.Fatalf("expected empty stderr for active model, got %q", stderr)
|
|
}
|
|
events := decodeJSONLines(t, stdout)
|
|
if got := events[0]["model"]; got != "gpt-4.1" {
|
|
t.Fatalf("expected alias to resolve to gpt-4.1, got %v", got)
|
|
}
|
|
}
|
|
|
|
func TestRunExecRedirectsDeprecatedModelWithNotice(t *testing.T) {
|
|
root := t.TempDir()
|
|
|
|
exitCode, stdout, stderr := runExecWithEcho(t, []string{"exec", "--cwd", root, "-m", "gpt-4-turbo", "-o", "json", "hi"})
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d: %s", exitCode, stderr)
|
|
}
|
|
if !strings.Contains(stderr, "deprecated") || !strings.Contains(stderr, "gpt-4.1") {
|
|
t.Fatalf("expected deprecation notice on stderr, got %q", stderr)
|
|
}
|
|
events := decodeJSONLines(t, stdout)
|
|
if got := events[0]["model"]; got != "gpt-4.1" {
|
|
t.Fatalf("expected deprecated model to redirect to gpt-4.1, got %v", got)
|
|
}
|
|
}
|
|
|
|
func TestRunExecReasoningEffortNoticeForNonReasoningModel(t *testing.T) {
|
|
root := t.TempDir()
|
|
|
|
exitCode, stdout, stderr := runExecWithEcho(t, []string{"exec", "--cwd", root, "-m", "gpt-4.1", "-r", "high", "-o", "json", "hi"})
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d: %s", exitCode, stderr)
|
|
}
|
|
if !strings.Contains(stderr, "does not support reasoning effort") {
|
|
t.Fatalf("expected non-reasoning effort notice on stderr, got %q", stderr)
|
|
}
|
|
if stdout == "" {
|
|
t.Fatal("expected run output on stdout")
|
|
}
|
|
}
|
|
|
|
func TestReasoningEffortNoticeCoercesUnsupportedEffort(t *testing.T) {
|
|
registry, err := modelregistry.DefaultRegistry()
|
|
if err != nil {
|
|
t.Fatalf("DefaultRegistry: %v", err)
|
|
}
|
|
// claude-sonnet-4.5 supports low/medium/high with a medium default; xhigh is
|
|
// unsupported and should be coerced to the model default.
|
|
notice := reasoningEffortNotice(registry, "claude-sonnet-4.5", "xhigh")
|
|
if !strings.Contains(notice, "not supported") || !strings.Contains(notice, "medium") {
|
|
t.Fatalf("expected coercion notice to default medium, got %q", notice)
|
|
}
|
|
if got := reasoningEffortNotice(registry, "claude-sonnet-4.5", "high"); got != "" {
|
|
t.Fatalf("expected no notice for a supported effort, got %q", got)
|
|
}
|
|
if got := reasoningEffortNotice(registry, "gpt-4.1", "high"); !strings.Contains(got, "does not support") {
|
|
t.Fatalf("expected unsupported-model notice, got %q", got)
|
|
}
|
|
}
|
|
|
|
func TestForwardedReasoningEffortGating(t *testing.T) {
|
|
registry, err := modelregistry.DefaultRegistry()
|
|
if err != nil {
|
|
t.Fatalf("DefaultRegistry: %v", err)
|
|
}
|
|
cases := []struct {
|
|
name string
|
|
model string
|
|
requested string
|
|
want string
|
|
}{
|
|
{"empty request", "claude-sonnet-4.5", "", ""},
|
|
{"supported reasoning model", "claude-sonnet-4.5", "high", "high"},
|
|
{"unsupported effort coerced to default", "claude-sonnet-4.5", "xhigh", "medium"},
|
|
{"known non-reasoning model suppressed", "gpt-4.1", "high", ""},
|
|
{"unknown model forwards as-is", "custom-endpoint-model", "high", "high"},
|
|
}
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
if got := forwardedReasoningEffort(registry, tc.model, tc.requested); got != tc.want {
|
|
t.Fatalf("forwardedReasoningEffort(%q, %q) = %q, want %q", tc.model, tc.requested, got, tc.want)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestRunExecJSONUnsafeOutputsWarningEvent(t *testing.T) {
|
|
exitCode, stdout, stderr := runExecWithEcho(t, []string{"exec", "--skip-permissions-unsafe", "-o", "json", "hello"})
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d: %s", exitCode, stderr)
|
|
}
|
|
if stderr != "" {
|
|
t.Fatalf("expected empty stderr, got %q", stderr)
|
|
}
|
|
|
|
events := decodeJSONLines(t, stdout)
|
|
eventTypes := jsonEventTypes(events)
|
|
if !slices.Contains(eventTypes, "warning") {
|
|
t.Fatalf("expected JSON warning event in %v; output %q", eventTypes, stdout)
|
|
}
|
|
if got := events[0]["permission_mode"]; got != "unsafe" {
|
|
t.Fatalf("expected run_start permission_mode unsafe, got %v", got)
|
|
}
|
|
}
|
|
|
|
func TestRunExecUsesProjectConfigAndOpenAICompatibleProvider(t *testing.T) {
|
|
clearProviderEnv(t)
|
|
root := t.TempDir()
|
|
configDir := filepath.Join(root, ".zero")
|
|
if err := os.MkdirAll(configDir, 0o700); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
|
|
var gotAuth string
|
|
var gotMethod string
|
|
var gotPath string
|
|
var gotBody map[string]any
|
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
gotMethod = r.Method
|
|
gotPath = r.URL.Path
|
|
gotAuth = r.Header.Get("Authorization")
|
|
if err := json.NewDecoder(r.Body).Decode(&gotBody); err != nil {
|
|
t.Fatalf("decode provider request: %v", err)
|
|
}
|
|
_, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"provider ok\"}}]}\n\n")
|
|
_, _ = io.WriteString(w, "data: [DONE]\n\n")
|
|
}))
|
|
defer server.Close()
|
|
|
|
writeConfig := `{
|
|
"activeProvider": "local",
|
|
"providers": [{
|
|
"name": "local",
|
|
"provider_kind": "openai-compatible",
|
|
"base_url": "` + server.URL + `",
|
|
"api_key": "sk-local",
|
|
"model": "local-model"
|
|
}]
|
|
}`
|
|
if err := os.WriteFile(filepath.Join(configDir, "config.json"), []byte(writeConfig), 0o600); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := Run([]string{"exec", "--cwd", root, "hello provider"}, &stdout, &stderr)
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d: %s", exitCode, stderr.String())
|
|
}
|
|
if strings.TrimSpace(stdout.String()) != "provider ok" {
|
|
t.Fatalf("stdout = %q, want provider response", stdout.String())
|
|
}
|
|
if stderr.Len() != 0 {
|
|
t.Fatalf("expected empty stderr, got %q", stderr.String())
|
|
}
|
|
if gotAuth != "Bearer sk-local" {
|
|
t.Fatalf("Authorization = %q, want project config token", gotAuth)
|
|
}
|
|
if gotMethod != http.MethodPost {
|
|
t.Fatalf("method = %q, want %q", gotMethod, http.MethodPost)
|
|
}
|
|
if !strings.HasSuffix(gotPath, "/chat/completions") {
|
|
t.Fatalf("path = %q, want suffix /chat/completions", gotPath)
|
|
}
|
|
if gotBody["model"] != "local-model" {
|
|
t.Fatalf("provider model = %v, want local-model", gotBody["model"])
|
|
}
|
|
messages, ok := gotBody["messages"].([]any)
|
|
if !ok || len(messages) == 0 {
|
|
t.Fatalf("messages = %#v, want non-empty []any", gotBody["messages"])
|
|
}
|
|
lastMessage, ok := messages[len(messages)-1].(map[string]any)
|
|
if !ok {
|
|
t.Fatalf("last message = %#v, want map[string]any", messages[len(messages)-1])
|
|
}
|
|
if lastMessage["content"] != "hello provider" {
|
|
t.Fatalf("last provider message = %#v, want prompt", lastMessage)
|
|
}
|
|
}
|
|
|
|
func runExecWithEcho(t *testing.T, args []string) (int, string, string) {
|
|
t.Helper()
|
|
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
cwd := t.TempDir()
|
|
exitCode := runWithDeps(args, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
model := "echo-model"
|
|
if overrides.Provider.Model != "" {
|
|
model = overrides.Provider.Model
|
|
}
|
|
return config.ResolvedConfig{
|
|
ActiveProvider: "echo",
|
|
Provider: config.ProviderProfile{
|
|
Name: "echo",
|
|
ProviderKind: config.ProviderKindOpenAICompatible,
|
|
BaseURL: "http://127.0.0.1/v1",
|
|
Model: model,
|
|
},
|
|
MaxTurns: 3,
|
|
}, nil
|
|
},
|
|
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
return echoExecProvider{}, nil
|
|
},
|
|
})
|
|
return exitCode, stdout.String(), stderr.String()
|
|
}
|
|
|
|
func TestExecSessionRecorderWarnsOnRecordingFailure(t *testing.T) {
|
|
// A latched session-append failure must be surfaced once to stderr so a user
|
|
// is not misled into believing the session was persisted; a clean recorder
|
|
// stays silent.
|
|
var failed bytes.Buffer
|
|
(&execSessionRecorder{err: errors.New("disk full")}).warnIfRecordingFailed(&failed)
|
|
if !strings.Contains(failed.String(), "session not fully recorded") || !strings.Contains(failed.String(), "disk full") {
|
|
t.Fatalf("expected a recording-failure warning, got %q", failed.String())
|
|
}
|
|
|
|
var clean bytes.Buffer
|
|
(&execSessionRecorder{}).warnIfRecordingFailed(&clean)
|
|
if clean.Len() != 0 {
|
|
t.Fatalf("expected silence when recording succeeded, got %q", clean.String())
|
|
}
|
|
}
|
|
|
|
// canceledExecProvider fails its first call with context.Canceled, simulating a
|
|
// signal-interrupted run (agent.Run returns the error verbatim).
|
|
type canceledExecProvider struct{}
|
|
|
|
func (canceledExecProvider) StreamCompletion(context.Context, zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
|
return nil, context.Canceled
|
|
}
|
|
|
|
type echoExecProvider struct{}
|
|
|
|
func (echoExecProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
|
prompt := ""
|
|
for index := len(request.Messages) - 1; index >= 0; index-- {
|
|
if request.Messages[index].Role == zeroruntime.MessageRoleUser {
|
|
prompt = request.Messages[index].Content
|
|
break
|
|
}
|
|
}
|
|
ch := make(chan zeroruntime.StreamEvent, 2)
|
|
select {
|
|
case <-ctx.Done():
|
|
close(ch)
|
|
return ch, ctx.Err()
|
|
case ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventText, Content: prompt}:
|
|
}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
|
close(ch)
|
|
return ch, nil
|
|
}
|
|
|
|
func clearProviderEnv(t *testing.T) {
|
|
t.Helper()
|
|
|
|
for _, key := range []string{
|
|
"ZERO_PROVIDER_COMMAND",
|
|
"ZERO_PROVIDER",
|
|
"OPENAI_API_KEY",
|
|
"OPENAI_BASE_URL",
|
|
"OPENAI_MODEL",
|
|
} {
|
|
t.Setenv(key, "")
|
|
}
|
|
}
|
|
|
|
func decodeJSONLines(t *testing.T, output string) []map[string]any {
|
|
t.Helper()
|
|
|
|
lines := strings.Split(strings.TrimSpace(output), "\n")
|
|
if len(lines) == 0 || lines[0] == "" {
|
|
t.Fatalf("expected JSON lines, got %q", output)
|
|
}
|
|
|
|
events := make([]map[string]any, 0, len(lines))
|
|
for _, line := range lines {
|
|
var event map[string]any
|
|
if err := json.Unmarshal([]byte(line), &event); err != nil {
|
|
t.Fatalf("expected JSON object line, got %q: %v", line, err)
|
|
}
|
|
events = append(events, event)
|
|
}
|
|
return events
|
|
}
|
|
|
|
func jsonEventTypes(events []map[string]any) []string {
|
|
types := make([]string, 0, len(events))
|
|
for _, event := range events {
|
|
eventType, _ := event["type"].(string)
|
|
types = append(types, eventType)
|
|
}
|
|
return types
|
|
}
|
|
|
|
// runExecWithEffectiveModel runs exec with the echo provider but forces the
|
|
// resolved (effective) model regardless of any --model flag, so tests can
|
|
// exercise behavior that depends on the effective model rather than the
|
|
// override-supplied one.
|
|
func runExecWithEffectiveModel(t *testing.T, effectiveModel string, args []string) (int, string, string) {
|
|
t.Helper()
|
|
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
cwd := t.TempDir()
|
|
exitCode := runWithDeps(args, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, _ config.Overrides) (config.ResolvedConfig, error) {
|
|
return config.ResolvedConfig{
|
|
ActiveProvider: "echo",
|
|
Provider: config.ProviderProfile{
|
|
Name: "echo",
|
|
ProviderKind: config.ProviderKindOpenAICompatible,
|
|
BaseURL: "http://127.0.0.1/v1",
|
|
Model: effectiveModel,
|
|
},
|
|
MaxTurns: 3,
|
|
}, nil
|
|
},
|
|
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
return echoExecProvider{}, nil
|
|
},
|
|
})
|
|
return exitCode, stdout.String(), stderr.String()
|
|
}
|
|
|
|
// TestRunExecReasoningEffortNoticeUsesEffectiveModel asserts that
|
|
// --reasoning-effort is validated against the EFFECTIVE (resolved) model even
|
|
// when --model is omitted, so the advisory notice still surfaces.
|
|
func TestRunExecReasoningEffortNoticeUsesEffectiveModel(t *testing.T) {
|
|
// gpt-4.1 is a registry-known non-reasoning model. With no --model the
|
|
// override model is empty, so the notice must be evaluated against the
|
|
// effective model resolved by resolveConfig.
|
|
exitCode, stdout, stderr := runExecWithEffectiveModel(t, "gpt-4.1", []string{
|
|
"exec", "-r", "high", "hi",
|
|
})
|
|
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("expected exit code %d, got %d: %s", exitSuccess, exitCode, stderr)
|
|
}
|
|
if !strings.Contains(stderr, "does not support reasoning effort") {
|
|
t.Fatalf("expected effort notice for effective model on stderr, got %q", stderr)
|
|
}
|
|
if stdout == "" {
|
|
t.Fatal("expected run output on stdout")
|
|
}
|
|
}
|
|
|
|
// TestRunExecAutoHighEmitsUnsafeWarning asserts that --auto high (which resolves
|
|
// to PermissionModeUnsafe) surfaces the same unsafe warning as
|
|
// --skip-permissions-unsafe.
|
|
func TestRunExecAutoHighEmitsUnsafeWarning(t *testing.T) {
|
|
exitCode, stdout, stderr := runExecWithEcho(t, []string{
|
|
"exec", "--auto", "high", "-o", "json", "hello",
|
|
})
|
|
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("expected exit code %d, got %d: %s", exitSuccess, exitCode, stderr)
|
|
}
|
|
if stderr != "" {
|
|
t.Fatalf("expected empty stderr, got %q", stderr)
|
|
}
|
|
|
|
events := decodeJSONLines(t, stdout)
|
|
if !slices.Contains(jsonEventTypes(events), "warning") {
|
|
t.Fatalf("expected JSON warning event for --auto high, got %v; output %q", jsonEventTypes(events), stdout)
|
|
}
|
|
if got := events[0]["permission_mode"]; got != "unsafe" {
|
|
t.Fatalf("expected run_start permission_mode unsafe, got %v", got)
|
|
}
|
|
}
|
|
|
|
// TestRunExecInvalidAutoValidatedWithSkipPermissions asserts that an invalid
|
|
// --auto value is still rejected even when --skip-permissions-unsafe is also
|
|
// passed (the unsafe path must not short-circuit --auto validation).
|
|
func TestRunExecInvalidAutoValidatedWithSkipPermissions(t *testing.T) {
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
|
|
exitCode := Run([]string{"exec", "--auto", "bogus", "--skip-permissions-unsafe", "hello"}, &stdout, &stderr)
|
|
|
|
if exitCode != exitUsage {
|
|
t.Fatalf("expected exit code %d, got %d", exitUsage, exitCode)
|
|
}
|
|
if got := stderr.String(); !strings.Contains(got, "Invalid autonomy level") {
|
|
t.Fatalf("expected autonomy validation error, got %q", got)
|
|
}
|
|
}
|
|
|
|
func TestParseExecAllowEscalationFlag(t *testing.T) {
|
|
t.Run("absent defaults to false", func(t *testing.T) {
|
|
options, help, err := parseExecArgs([]string{"hello"})
|
|
if err != nil {
|
|
t.Fatalf("parseExecArgs returned error: %v", err)
|
|
}
|
|
if help {
|
|
t.Fatal("help = true, want false")
|
|
}
|
|
if options.allowEscalation {
|
|
t.Fatal("allowEscalation = true, want false by default")
|
|
}
|
|
})
|
|
|
|
t.Run("flag sets true", func(t *testing.T) {
|
|
options, _, err := parseExecArgs([]string{"--allow-escalation", "hello"})
|
|
if err != nil {
|
|
t.Fatalf("parseExecArgs returned error: %v", err)
|
|
}
|
|
if !options.allowEscalation {
|
|
t.Fatal("allowEscalation = false, want true")
|
|
}
|
|
if strings.Join(options.promptParts, " ") != "hello" {
|
|
t.Fatalf("promptParts = %#v, want [hello]", options.promptParts)
|
|
}
|
|
})
|
|
}
|
|
|
|
func TestRunExecHelpDocumentsAllowEscalation(t *testing.T) {
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
|
|
exitCode := Run([]string{"exec", "--help"}, &stdout, &stderr)
|
|
|
|
if exitCode != 0 {
|
|
t.Fatalf("expected exit code 0, got %d", exitCode)
|
|
}
|
|
if !strings.Contains(stdout.String(), "--allow-escalation") {
|
|
t.Fatalf("expected exec help to document --allow-escalation, got %q", stdout.String())
|
|
}
|
|
if stderr.Len() != 0 {
|
|
t.Fatalf("expected empty stderr, got %q", stderr.String())
|
|
}
|
|
}
|
|
|
|
func TestRunExecRegistersEscalateModelOnlyWithFlag(t *testing.T) {
|
|
for _, tc := range []struct {
|
|
name string
|
|
args []string
|
|
wantTool bool
|
|
}{
|
|
{name: "absent", args: []string{"exec", "--list-tools"}, wantTool: false},
|
|
{name: "present", args: []string{"exec", "--allow-escalation", "--list-tools"}, wantTool: true},
|
|
} {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps(tc.args, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return t.TempDir(), nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
hasTool := strings.Contains(stdout.String(), " escalate_model ")
|
|
if hasTool != tc.wantTool {
|
|
t.Fatalf("escalate_model visibility = %v, want %v; output:\n%s", hasTool, tc.wantTool, stdout.String())
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// escalatingExecProvider drives an exec run through a mid-run switch without a
|
|
// live model. Each instance tracks how many turns IT served (turns) so a test
|
|
// can assert exactly which provider handled which turn across a switch.
|
|
//
|
|
// escalateOnce controls behavior: when true, the FIRST turn this instance serves
|
|
// emits an escalate_model tool call (later turns answer); when false, every turn
|
|
// emits a final text answer. Pointing escalateOnce at distinct instances lets a
|
|
// test prove the post-switch turn lands on the SECOND (escalated) provider.
|
|
type escalatingExecProvider struct {
|
|
// turns counts how many StreamCompletion calls THIS instance handled.
|
|
turns int
|
|
// escalateOnce makes this instance's first served turn request an escalation.
|
|
escalateOnce bool
|
|
}
|
|
|
|
func (provider *escalatingExecProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
|
turn := provider.turns
|
|
provider.turns++
|
|
ch := make(chan zeroruntime.StreamEvent, 4)
|
|
if provider.escalateOnce && turn == 0 {
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call_escalate", ToolName: "escalate_model"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call_escalate", ArgumentsFragment: "{}"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call_escalate"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
|
close(ch)
|
|
return ch, nil
|
|
}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventText, Content: "done"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
|
close(ch)
|
|
return ch, nil
|
|
}
|
|
|
|
func TestRunExecWiresModelSwitcherUnderFlag(t *testing.T) {
|
|
dataHome := t.TempDir()
|
|
t.Setenv("XDG_DATA_HOME", dataHome)
|
|
cwd := t.TempDir()
|
|
|
|
var providerModels []string
|
|
// Each newProvider build returns a DISTINCT provider instance with its OWN
|
|
// turn counter, mirroring the agent-layer firstProvider/secondProvider split.
|
|
// The first build (escalation source) must handle exactly the escalation turn;
|
|
// the second build (escalation target) must handle exactly the post-switch
|
|
// answer turn. Sharing a single counter would hide a dropped `provider =
|
|
// newProvider` in loop.go — here it is mutation-proven by the per-instance
|
|
// turn assertions below.
|
|
var builtProviders []*escalatingExecProvider
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{"exec", "--allow-escalation", "--model", "claude-haiku-4.5", "escalate please"}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
model := "claude-haiku-4.5"
|
|
if overrides.Provider.Model != "" {
|
|
model = overrides.Provider.Model
|
|
}
|
|
cfg := execResolvedConfig()
|
|
// The escalation chain (haiku -> sonnet) is anthropic; declare the
|
|
// provider kind to match so provider-metadata resolution accepts it.
|
|
cfg.Provider.ProviderKind = config.ProviderKindAnthropic
|
|
cfg.Provider.Model = model
|
|
cfg.MaxTurns = 3
|
|
return cfg, nil
|
|
},
|
|
newProvider: func(profile config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
providerModels = append(providerModels, profile.Model)
|
|
// First build escalates; every later build answers. Each instance owns
|
|
// its turn counter so we can assert exactly-one-turn per provider.
|
|
provider := &escalatingExecProvider{escalateOnce: len(builtProviders) == 0}
|
|
builtProviders = append(builtProviders, provider)
|
|
return provider, nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
// Initial provider build + one rebuild for the escalation target.
|
|
if len(providerModels) != 2 {
|
|
t.Fatalf("newProvider called for models %#v, want exactly 2 builds (initial + escalated)", providerModels)
|
|
}
|
|
if providerModels[0] != "claude-haiku-4.5" {
|
|
t.Fatalf("initial provider model = %q, want claude-haiku-4.5", providerModels[0])
|
|
}
|
|
target, ok := mustUpgradeTarget(t, "claude-haiku-4.5")
|
|
if !ok {
|
|
t.Skip("registry has no upgrade target for claude-haiku-4.5")
|
|
}
|
|
if providerModels[1] != target {
|
|
t.Fatalf("escalated provider model = %q, want %q", providerModels[1], target)
|
|
}
|
|
if len(builtProviders) != 2 {
|
|
t.Fatalf("built %d providers, want 2", len(builtProviders))
|
|
}
|
|
// The original (escalation-source) provider handled ONLY the escalation turn;
|
|
// the escalated (target) provider handled ONLY the post-switch answer turn.
|
|
// This FAILS if loop.go drops `provider = newProvider` (the second provider
|
|
// would then handle zero turns and the first would handle both).
|
|
if builtProviders[0].turns != 1 {
|
|
t.Fatalf("first (source) provider handled %d turns, want exactly 1 (the escalation turn)", builtProviders[0].turns)
|
|
}
|
|
if builtProviders[1].turns != 1 {
|
|
t.Fatalf("second (escalated) provider handled %d turns, want exactly 1 (the post-switch answer turn)", builtProviders[1].turns)
|
|
}
|
|
}
|
|
|
|
func mustUpgradeTarget(t *testing.T, id string) (string, bool) {
|
|
t.Helper()
|
|
registry, err := modelregistry.DefaultRegistry()
|
|
if err != nil {
|
|
t.Fatalf("DefaultRegistry: %v", err)
|
|
}
|
|
entry, ok := registry.UpgradeTarget(id)
|
|
return entry.ID, ok
|
|
}
|
|
|
|
func TestRunExecNoSwitcherWithoutFlag(t *testing.T) {
|
|
dataHome := t.TempDir()
|
|
t.Setenv("XDG_DATA_HOME", dataHome)
|
|
cwd := t.TempDir()
|
|
|
|
var providerModels []string
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{"exec", "--model", "claude-haiku-4.5", "escalate please"}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
model := "claude-haiku-4.5"
|
|
if overrides.Provider.Model != "" {
|
|
model = overrides.Provider.Model
|
|
}
|
|
cfg := execResolvedConfig()
|
|
// The escalation chain (haiku -> sonnet) is anthropic; declare the
|
|
// provider kind to match so provider-metadata resolution accepts it.
|
|
cfg.Provider.ProviderKind = config.ProviderKindAnthropic
|
|
cfg.Provider.Model = model
|
|
cfg.MaxTurns = 3
|
|
return cfg, nil
|
|
},
|
|
newProvider: func(profile config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
providerModels = append(providerModels, profile.Model)
|
|
return &escalatingExecProvider{escalateOnce: true}, nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
if len(providerModels) != 1 {
|
|
t.Fatalf("newProvider called for %#v, want exactly 1 build (no switcher wired)", providerModels)
|
|
}
|
|
if providerModels[0] != "claude-haiku-4.5" {
|
|
t.Fatalf("provider model = %q, want claude-haiku-4.5", providerModels[0])
|
|
}
|
|
}
|
|
|
|
func TestRunExecAttributesUsageToEscalatedModel(t *testing.T) {
|
|
dataHome := t.TempDir()
|
|
t.Setenv("XDG_DATA_HOME", dataHome)
|
|
cwd := t.TempDir()
|
|
|
|
target, ok := mustUpgradeTarget(t, "claude-haiku-4.5")
|
|
if !ok {
|
|
t.Skip("registry has no upgrade target for claude-haiku-4.5")
|
|
}
|
|
|
|
// The escalation turn (turn 0, pre-switch) emits usage with distinct tokens so
|
|
// it can be told apart from the post-switch usage; the first build escalates,
|
|
// the second answers.
|
|
builds := 0
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{
|
|
"exec",
|
|
"--allow-escalation",
|
|
"--model", "claude-haiku-4.5",
|
|
"--init-session-id", "escalation_run",
|
|
"escalate please",
|
|
}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
model := "claude-haiku-4.5"
|
|
if overrides.Provider.Model != "" {
|
|
model = overrides.Provider.Model
|
|
}
|
|
cfg := execResolvedConfig()
|
|
// The escalation chain (haiku -> sonnet) is anthropic; declare the
|
|
// provider kind to match so provider-metadata resolution accepts it.
|
|
cfg.Provider.ProviderKind = config.ProviderKindAnthropic
|
|
cfg.Provider.Model = model
|
|
cfg.MaxTurns = 3
|
|
return cfg, nil
|
|
},
|
|
newProvider: func(profile config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
// First build = escalation source (escalate + usage 3/4 pre-switch);
|
|
// second build = escalation target (answer + usage 5/7 post-switch).
|
|
escalate := builds == 0
|
|
builds++
|
|
return &usageEmittingEscalatingProvider{escalate: escalate}, nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
|
|
store := sessions.NewStore(sessions.StoreOptions{RootDir: filepath.Join(dataHome, "zero", "sessions")})
|
|
events, err := store.ReadEvents("escalation_run")
|
|
if err != nil {
|
|
t.Fatalf("ReadEvents returned error: %v", err)
|
|
}
|
|
// Collect every usage model attribution IN ORDER. The pre-switch usage must
|
|
// attribute to the ORIGINAL model and the post-switch usage to the escalated
|
|
// target — proving the loop's ordering guarantee (usage is attributed to the
|
|
// model that produced it, not retroactively reassigned after the switch).
|
|
var usageModels []string
|
|
for _, event := range events {
|
|
if event.Type != sessions.EventUsage {
|
|
continue
|
|
}
|
|
var payload map[string]any
|
|
if err := json.Unmarshal(event.Payload, &payload); err != nil {
|
|
t.Fatalf("unmarshal usage payload: %v", err)
|
|
}
|
|
model, _ := payload["model"].(string)
|
|
usageModels = append(usageModels, model)
|
|
}
|
|
if len(usageModels) != 2 {
|
|
t.Fatalf("recorded usage model attributions = %#v, want exactly 2 (pre- and post-switch)", usageModels)
|
|
}
|
|
if usageModels[0] != "claude-haiku-4.5" {
|
|
t.Fatalf("first (pre-switch) usage model = %q, want claude-haiku-4.5", usageModels[0])
|
|
}
|
|
if usageModels[1] != target {
|
|
t.Fatalf("second (post-switch) usage model = %q, want escalated target %q", usageModels[1], target)
|
|
}
|
|
}
|
|
|
|
func TestRunExecNilSwitchProviderKeepsOriginalAttribution(t *testing.T) {
|
|
dataHome := t.TempDir()
|
|
t.Setenv("XDG_DATA_HOME", dataHome)
|
|
cwd := t.TempDir()
|
|
|
|
if _, ok := mustUpgradeTarget(t, "claude-haiku-4.5"); !ok {
|
|
t.Skip("registry has no upgrade target for claude-haiku-4.5")
|
|
}
|
|
|
|
// The escalation rebuild returns (nil, nil): the loop must NOT swap the
|
|
// provider, and the CLI switcher must leave currentModel unchanged so usage
|
|
// after the declined switch stays attributed to the ORIGINAL model. Without
|
|
// the switchedProvider != nil guard, currentModel would advance to the target
|
|
// even though no switch happened, misattributing the post-turn usage.
|
|
builds := 0
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{
|
|
"exec",
|
|
"--allow-escalation",
|
|
"--model", "claude-haiku-4.5",
|
|
"--init-session-id", "nil_switch_run",
|
|
"escalate please",
|
|
}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) { return cwd, nil },
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
model := "claude-haiku-4.5"
|
|
if overrides.Provider.Model != "" {
|
|
model = overrides.Provider.Model
|
|
}
|
|
cfg := execResolvedConfig()
|
|
cfg.Provider.ProviderKind = config.ProviderKindAnthropic
|
|
cfg.Provider.Model = model
|
|
cfg.MaxTurns = 3
|
|
return cfg, nil
|
|
},
|
|
newProvider: func(profile config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
builds++
|
|
// First build = the original (haiku). The escalation rebuild returns
|
|
// (nil, nil), so the loop keeps the original provider for every turn.
|
|
if builds == 1 {
|
|
return &escalateThenAnswerProvider{}, nil
|
|
}
|
|
return nil, nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
|
|
store := sessions.NewStore(sessions.StoreOptions{RootDir: filepath.Join(dataHome, "zero", "sessions")})
|
|
events, err := store.ReadEvents("nil_switch_run")
|
|
if err != nil {
|
|
t.Fatalf("ReadEvents returned error: %v", err)
|
|
}
|
|
sawUsage := false
|
|
for _, event := range events {
|
|
if event.Type != sessions.EventUsage {
|
|
continue
|
|
}
|
|
var payload map[string]any
|
|
if err := json.Unmarshal(event.Payload, &payload); err != nil {
|
|
t.Fatalf("unmarshal usage payload: %v", err)
|
|
}
|
|
sawUsage = true
|
|
if model, _ := payload["model"].(string); model != "claude-haiku-4.5" {
|
|
t.Fatalf("usage attributed to %q after a (nil,nil) switch, want original claude-haiku-4.5", model)
|
|
}
|
|
}
|
|
if !sawUsage {
|
|
t.Fatal("expected at least one usage event")
|
|
}
|
|
}
|
|
|
|
// escalateThenAnswerProvider escalates on its first turn and answers afterward,
|
|
// so a single instance can serve an entire run when no provider swap occurs
|
|
// (e.g. a (nil,nil) switcher). It emits usage on every turn for attribution.
|
|
type escalateThenAnswerProvider struct {
|
|
turns int
|
|
}
|
|
|
|
func (provider *escalateThenAnswerProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
|
turn := provider.turns
|
|
provider.turns++
|
|
ch := make(chan zeroruntime.StreamEvent, 6)
|
|
if turn == 0 {
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call_escalate", ToolName: "escalate_model"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call_escalate", ArgumentsFragment: "{}"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call_escalate"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventUsage, Usage: zeroruntime.Usage{InputTokens: 3, OutputTokens: 4}}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
|
close(ch)
|
|
return ch, nil
|
|
}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventText, Content: "done"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventUsage, Usage: zeroruntime.Usage{InputTokens: 5, OutputTokens: 7}}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
|
close(ch)
|
|
return ch, nil
|
|
}
|
|
|
|
// usageEmittingEscalatingProvider emits a usage event on every turn it serves so
|
|
// usage attribution can be traced across a mid-run switch. When escalate is set,
|
|
// its turn requests an escalation (and emits pre-switch usage tokens 3/4);
|
|
// otherwise it answers (post-switch usage tokens 5/7).
|
|
type usageEmittingEscalatingProvider struct {
|
|
escalate bool
|
|
}
|
|
|
|
func (provider *usageEmittingEscalatingProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
|
ch := make(chan zeroruntime.StreamEvent, 6)
|
|
if provider.escalate {
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call_escalate", ToolName: "escalate_model"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call_escalate", ArgumentsFragment: "{}"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call_escalate"}
|
|
// Pre-switch usage: still attributed to the ORIGINAL model.
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventUsage, Usage: zeroruntime.Usage{InputTokens: 3, OutputTokens: 4}}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
|
close(ch)
|
|
return ch, nil
|
|
}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventText, Content: "done"}
|
|
// Post-switch usage: attributed to the escalated model.
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventUsage, Usage: zeroruntime.Usage{InputTokens: 5, OutputTokens: 7}}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
|
close(ch)
|
|
return ch, nil
|
|
}
|
|
|
|
// usageEmittingEchoProvider answers in a single turn and emits a usage event. It
|
|
// has no escalation behavior, so it exercises a plain (flag-OFF) run that still
|
|
// records usage.
|
|
type usageEmittingEchoProvider struct{}
|
|
|
|
func (usageEmittingEchoProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
|
ch := make(chan zeroruntime.StreamEvent, 3)
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventText, Content: "done"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventUsage, Usage: zeroruntime.Usage{InputTokens: 5, OutputTokens: 7}}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
|
close(ch)
|
|
return ch, nil
|
|
}
|
|
|
|
// TestRunExecUsageOmitsModelKeyWithoutEscalationFlag verifies the back-compat
|
|
// guarantee: a run WITHOUT --allow-escalation persists EventUsage payloads that
|
|
// carry NO "model" key (byte-identical to before the escalation feature), since
|
|
// the model can never change mid-run when escalation is off. The flag-ON,
|
|
// post-escalation case is covered by TestRunExecAttributesUsageToEscalatedModel.
|
|
func TestRunExecUsageOmitsModelKeyWithoutEscalationFlag(t *testing.T) {
|
|
dataHome := t.TempDir()
|
|
t.Setenv("XDG_DATA_HOME", dataHome)
|
|
cwd := t.TempDir()
|
|
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{
|
|
"exec",
|
|
"--model", "claude-haiku-4.5",
|
|
"--init-session-id", "no_escalation_run",
|
|
"hello",
|
|
}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
model := "claude-haiku-4.5"
|
|
if overrides.Provider.Model != "" {
|
|
model = overrides.Provider.Model
|
|
}
|
|
cfg := execResolvedConfig()
|
|
cfg.Provider.ProviderKind = config.ProviderKindAnthropic
|
|
cfg.Provider.Model = model
|
|
cfg.MaxTurns = 3
|
|
return cfg, nil
|
|
},
|
|
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
return usageEmittingEchoProvider{}, nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
|
|
store := sessions.NewStore(sessions.StoreOptions{RootDir: filepath.Join(dataHome, "zero", "sessions")})
|
|
events, err := store.ReadEvents("no_escalation_run")
|
|
if err != nil {
|
|
t.Fatalf("ReadEvents returned error: %v", err)
|
|
}
|
|
var sawUsage bool
|
|
for _, event := range events {
|
|
if event.Type != sessions.EventUsage {
|
|
continue
|
|
}
|
|
sawUsage = true
|
|
var payload map[string]any
|
|
if err := json.Unmarshal(event.Payload, &payload); err != nil {
|
|
t.Fatalf("unmarshal usage payload: %v", err)
|
|
}
|
|
if _, ok := payload["model"]; ok {
|
|
t.Fatalf("flag-OFF usage payload must NOT carry a model key, got %#v", payload)
|
|
}
|
|
}
|
|
if !sawUsage {
|
|
t.Fatal("expected at least one usage event to be recorded")
|
|
}
|
|
}
|
|
|
|
// usageThenAnswerProvider serves a two-turn run on ONE instance: turn 0 requests
|
|
// an escalation and emits usage, turn 1 answers and emits usage. It is used by
|
|
// the switch-error test where no second provider is ever built, so a single
|
|
// instance must serve both turns on the original model.
|
|
type usageThenAnswerProvider struct {
|
|
turns int
|
|
}
|
|
|
|
func (provider *usageThenAnswerProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
|
turn := provider.turns
|
|
provider.turns++
|
|
ch := make(chan zeroruntime.StreamEvent, 6)
|
|
if turn == 0 {
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call_escalate", ToolName: "escalate_model"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call_escalate", ArgumentsFragment: "{}"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call_escalate"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventUsage, Usage: zeroruntime.Usage{InputTokens: 3, OutputTokens: 4}}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
|
close(ch)
|
|
return ch, nil
|
|
}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventText, Content: "done"}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventUsage, Usage: zeroruntime.Usage{InputTokens: 5, OutputTokens: 7}}
|
|
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
|
close(ch)
|
|
return ch, nil
|
|
}
|
|
|
|
// TestRunExecSwitcherErrorKeepsOriginalModelAttribution verifies that when the
|
|
// ModelSwitcher fails on the rebuild (deps.newProvider errors for the escalation
|
|
// target), the run is NON-FATAL and stays on the original provider — and every
|
|
// usage event, including the one AFTER the failed switch, stays attributed to the
|
|
// ORIGINAL model (currentModel is never reassigned on a failed switch).
|
|
func TestRunExecSwitcherErrorKeepsOriginalModelAttribution(t *testing.T) {
|
|
dataHome := t.TempDir()
|
|
t.Setenv("XDG_DATA_HOME", dataHome)
|
|
cwd := t.TempDir()
|
|
|
|
if _, ok := mustUpgradeTarget(t, "claude-haiku-4.5"); !ok {
|
|
t.Skip("registry has no upgrade target for claude-haiku-4.5")
|
|
}
|
|
|
|
builds := 0
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{
|
|
"exec",
|
|
"--allow-escalation",
|
|
"--model", "claude-haiku-4.5",
|
|
"--init-session-id", "switch_error_run",
|
|
"escalate please",
|
|
}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
model := "claude-haiku-4.5"
|
|
if overrides.Provider.Model != "" {
|
|
model = overrides.Provider.Model
|
|
}
|
|
cfg := execResolvedConfig()
|
|
cfg.Provider.ProviderKind = config.ProviderKindAnthropic
|
|
cfg.Provider.Model = model
|
|
cfg.MaxTurns = 3
|
|
return cfg, nil
|
|
},
|
|
newProvider: func(profile config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
builds++
|
|
// The first build (original model) succeeds; the rebuild on escalation
|
|
// FAILS, so the switcher returns an error and the run continues on the
|
|
// original provider.
|
|
if builds >= 2 {
|
|
return nil, errors.New("provider rebuild failed")
|
|
}
|
|
return &usageThenAnswerProvider{}, nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("expected non-fatal switch error, exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
if builds != 2 {
|
|
t.Fatalf("newProvider builds = %d, want 2 (initial + one failed rebuild attempt)", builds)
|
|
}
|
|
|
|
store := sessions.NewStore(sessions.StoreOptions{RootDir: filepath.Join(dataHome, "zero", "sessions")})
|
|
events, err := store.ReadEvents("switch_error_run")
|
|
if err != nil {
|
|
t.Fatalf("ReadEvents returned error: %v", err)
|
|
}
|
|
var usageModels []string
|
|
for _, event := range events {
|
|
if event.Type != sessions.EventUsage {
|
|
continue
|
|
}
|
|
var payload map[string]any
|
|
if err := json.Unmarshal(event.Payload, &payload); err != nil {
|
|
t.Fatalf("unmarshal usage payload: %v", err)
|
|
}
|
|
model, _ := payload["model"].(string)
|
|
usageModels = append(usageModels, model)
|
|
}
|
|
if len(usageModels) != 2 {
|
|
t.Fatalf("recorded usage attributions = %#v, want 2", usageModels)
|
|
}
|
|
// Both usages (pre- AND post-failed-switch) stay on the ORIGINAL model.
|
|
for i, model := range usageModels {
|
|
if model != "claude-haiku-4.5" {
|
|
t.Fatalf("usage[%d] model = %q, want claude-haiku-4.5 (currentModel must not change on a failed switch)", i, model)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestRunExecTopTierDeclineNoSwitch verifies an end-to-end flag-ON run where the
|
|
// agent calls escalate_model while ALREADY on a top-tier model (claude-opus-4.1):
|
|
// the tool returns the informational no-meta result, NO switch happens
|
|
// (newProvider is called exactly once), and the run succeeds.
|
|
func TestParseExecNotifyFlag(t *testing.T) {
|
|
opts, _, err := parseExecArgs([]string{"--notify", "both", "hello"})
|
|
if err != nil {
|
|
t.Fatalf("parse: %v", err)
|
|
}
|
|
if opts.notifyMode != "both" || opts.noNotify {
|
|
t.Fatalf("got mode=%q noNotify=%v", opts.notifyMode, opts.noNotify)
|
|
}
|
|
}
|
|
|
|
func TestParseExecNoNotifyFlag(t *testing.T) {
|
|
opts, _, err := parseExecArgs([]string{"--no-notify", "hello"})
|
|
if err != nil {
|
|
t.Fatalf("parse: %v", err)
|
|
}
|
|
if !opts.noNotify {
|
|
t.Fatal("expected noNotify=true")
|
|
}
|
|
}
|
|
|
|
func TestParseExecNotifyConflict(t *testing.T) {
|
|
if _, _, err := parseExecArgs([]string{"--notify", "bell", "--no-notify", "hi"}); err == nil {
|
|
t.Fatal("expected error for --notify with --no-notify")
|
|
}
|
|
}
|
|
|
|
func TestParseExecNotifyInvalidValue(t *testing.T) {
|
|
for _, arg := range [][]string{{"--notify", "buzz", "hi"}, {"--notify=loud", "hi"}} {
|
|
if _, _, err := parseExecArgs(arg); err == nil {
|
|
t.Fatalf("expected error for invalid notify value in %v", arg)
|
|
}
|
|
}
|
|
// Valid values still parse.
|
|
for _, mode := range []string{"off", "bell", "notify", "both"} {
|
|
if _, _, err := parseExecArgs([]string{"--notify", mode, "hi"}); err != nil {
|
|
t.Fatalf("valid --notify %s rejected: %v", mode, err)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestExecNotifyModeResolution(t *testing.T) {
|
|
resolved := config.ResolvedConfig{Notify: config.NotifyConfig{Mode: "bell"}}
|
|
if got := execNotifyMode(execOptions{}, resolved); got != "bell" {
|
|
t.Fatalf("config passthrough got %q", got)
|
|
}
|
|
if got := execNotifyMode(execOptions{notifyMode: "both"}, resolved); got != "both" {
|
|
t.Fatalf("flag override got %q", got)
|
|
}
|
|
if got := execNotifyMode(execOptions{noNotify: true}, resolved); got != "off" {
|
|
t.Fatalf("--no-notify got %q", got)
|
|
}
|
|
}
|
|
|
|
func TestRunExecTopTierDeclineNoSwitch(t *testing.T) {
|
|
dataHome := t.TempDir()
|
|
t.Setenv("XDG_DATA_HOME", dataHome)
|
|
cwd := t.TempDir()
|
|
|
|
// Sanity: claude-opus-4.1 must genuinely be top-tier (no upgrade target), or
|
|
// the scenario is meaningless.
|
|
if _, ok := mustUpgradeTarget(t, "claude-opus-4.1"); ok {
|
|
t.Skip("registry unexpectedly has an upgrade target for claude-opus-4.1")
|
|
}
|
|
|
|
var providerModels []string
|
|
var stdout bytes.Buffer
|
|
var stderr bytes.Buffer
|
|
exitCode := runWithDeps([]string{
|
|
"exec",
|
|
"--allow-escalation",
|
|
"--model", "claude-opus-4.1",
|
|
"escalate please",
|
|
}, &stdout, &stderr, appDeps{
|
|
getwd: func() (string, error) {
|
|
return cwd, nil
|
|
},
|
|
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
|
model := "claude-opus-4.1"
|
|
if overrides.Provider.Model != "" {
|
|
model = overrides.Provider.Model
|
|
}
|
|
cfg := execResolvedConfig()
|
|
cfg.Provider.ProviderKind = config.ProviderKindAnthropic
|
|
cfg.Provider.Model = model
|
|
cfg.MaxTurns = 3
|
|
return cfg, nil
|
|
},
|
|
newProvider: func(profile config.ProviderProfile) (zeroruntime.Provider, error) {
|
|
providerModels = append(providerModels, profile.Model)
|
|
return &escalatingExecProvider{escalateOnce: true}, nil
|
|
},
|
|
})
|
|
if exitCode != exitSuccess {
|
|
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
|
}
|
|
// The escalate_model call resolved to a no-target informational result, so the
|
|
// loop never requested a switch and newProvider was built exactly once.
|
|
if len(providerModels) != 1 {
|
|
t.Fatalf("newProvider built for %#v, want exactly 1 (top-tier decline performs no switch)", providerModels)
|
|
}
|
|
if providerModels[0] != "claude-opus-4.1" {
|
|
t.Fatalf("provider model = %q, want claude-opus-4.1", providerModels[0])
|
|
}
|
|
}
|