mirror of
https://github.com/Twigpine/zero.git
synced 2026-11-08 09:49:34 -10:00
Compare commits
24
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
37ea2a7be5 | ||
|
|
95fad4f0bc | ||
|
|
b0905a6278 | ||
|
|
35dfe9fda3 | ||
|
|
08904dfcdf | ||
|
|
f75cace483 | ||
|
|
a404352f84 | ||
|
|
9a57c1559e | ||
|
|
6c8532b91f | ||
|
|
30e2c3f7ff | ||
|
|
18cce358a2 | ||
|
|
e5670c427f | ||
|
|
31d45d5f14 | ||
|
|
727ad4d321 | ||
|
|
e666395d47 | ||
|
|
9043baedcf | ||
|
|
4d8c31cf16 | ||
|
|
824ecdbcf9 | ||
|
|
6b9c2f0c08 | ||
|
|
8ef8576df7 | ||
|
|
1c5c6e78a8 | ||
|
|
7f669f4550 | ||
|
|
2b42cd567b | ||
|
|
665bd4b3d0 |
No files matched your search
@@ -1 +1 @@
|
||||
{".":"0.3.0"}
|
||||
{".":"0.4.0"}
|
||||
+60
-4
@@ -6,6 +6,60 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html) once the first release is
|
||||
tagged. Until then, source builds report the version `dev`.
|
||||
|
||||
## [0.4.0](https://github.com/Gitlawb/zero/compare/v0.3.0...v0.4.0) (2026-07-17)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **aimlapi:** AI/ML API provider with guided onboarding (top-up + key issuance) ([#655](https://github.com/Gitlawb/zero/issues/655)) ([6b9c2f0](https://github.com/Gitlawb/zero/commit/6b9c2f0c083e02bcd3aad68fb7af0501a9e7cd61))
|
||||
* **cli:** show ZERO wordmark on --version ([#673](https://github.com/Gitlawb/zero/issues/673)) ([9acb411](https://github.com/Gitlawb/zero/commit/9acb4113cc3337b3f361a16278e9cf11ca105e34))
|
||||
* **cli:** wire MCP serve WorkspaceRoot and --add-dir scope ([#694](https://github.com/Gitlawb/zero/issues/694)) ([75a78e7](https://github.com/Gitlawb/zero/commit/75a78e715bf23154f69c8c79cb58eb4b535b2a2a))
|
||||
* **npm:** ship the native binary as platform optionalDependencies ([#626](https://github.com/Gitlawb/zero/issues/626)) ([5e1405d](https://github.com/Gitlawb/zero/commit/5e1405d0b7abff5b3ccb3cfdb66d64d6d3322922))
|
||||
* **perf:** emit prompt-prefix hash fingerprint per turn ([#704](https://github.com/Gitlawb/zero/issues/704)) ([1c5c6e7](https://github.com/Gitlawb/zero/commit/1c5c6e78a8a0e228bdf53d7be90934fbce9d98c3))
|
||||
* **providers:** add AI/ML API preset (rebased onto main) ([#621](https://github.com/Gitlawb/zero/issues/621)) ([d66a9dd](https://github.com/Gitlawb/zero/commit/d66a9dda69c32aa59f4bea903cfefe00d4b7adef))
|
||||
* **providers:** refresh MiniMax model coverage ([#665](https://github.com/Gitlawb/zero/issues/665)) ([fa3052a](https://github.com/Gitlawb/zero/commit/fa3052a1422a4ad30a3a6295829564f42dee31a8))
|
||||
* **skills:** discover shared ~/.agents/skills with multi-root skill loading ([#696](https://github.com/Gitlawb/zero/issues/696)) ([7d57999](https://github.com/Gitlawb/zero/commit/7d579996b43741a2e57e3e38fcdd8484fcfc34e9))
|
||||
* **tui:** Ctrl+X leader chords and emacs menu navigation ([#699](https://github.com/Gitlawb/zero/issues/699)) ([7f669f4](https://github.com/Gitlawb/zero/commit/7f669f455021be51319ee3b8298cd48a17f745c7))
|
||||
* **tui:** press up to edit queued messages ([#656](https://github.com/Gitlawb/zero/issues/656)) ([4c986d3](https://github.com/Gitlawb/zero/commit/4c986d327b5ed26338d295c23bea5839681db8d0))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **acp:** make truncateHint rune-safe ([#614](https://github.com/Gitlawb/zero/issues/614)) ([ddc4927](https://github.com/Gitlawb/zero/commit/ddc4927aac5544bf4dd2c46615aeda5a81c96576))
|
||||
* **agent,tui:** resolve git branch detection when starting Zero in subdirectories ([#613](https://github.com/Gitlawb/zero/issues/613)) ([0184581](https://github.com/Gitlawb/zero/commit/0184581ec234ea414e0a47bc33e8b0f4ddfb497b))
|
||||
* **agent:** raise default and deep-mode turn budgets ([#650](https://github.com/Gitlawb/zero/issues/650)) ([635c93a](https://github.com/Gitlawb/zero/commit/635c93af51ebc20f3e0917917e55a79edfe27c35))
|
||||
* **cli:** prevent consuming positional arguments as flag values ([#619](https://github.com/Gitlawb/zero/issues/619)) ([5b4f48d](https://github.com/Gitlawb/zero/commit/5b4f48d2dcb66402c13bde0c3cfe9c9371da19fb))
|
||||
* **config:** enforce MCP trust boundary so project config cannot override user disable ([#609](https://github.com/Gitlawb/zero/issues/609)) ([4d8c31c](https://github.com/Gitlawb/zero/commit/4d8c31cf16a3080344e3be7039408fcae71c075d)), closes [#512](https://github.com/Gitlawb/zero/issues/512)
|
||||
* **config:** surface unknown/typo'd config fields instead of silently dropping them ([#645](https://github.com/Gitlawb/zero/issues/645)) ([893b7b4](https://github.com/Gitlawb/zero/commit/893b7b424cc203a2fcf92327a4e25c84286a90e0))
|
||||
* **cron:** prevent cron job Mutate from clobbering concurrent updates ([#630](https://github.com/Gitlawb/zero/issues/630)) ([e4bd703](https://github.com/Gitlawb/zero/commit/e4bd703cfb28dab2dfa3c2ddba46237e1bb2e164))
|
||||
* **daemon:** handle os.ErrPermission as collision during O_EXCL lock creation ([#616](https://github.com/Gitlawb/zero/issues/616)) ([8ea5384](https://github.com/Gitlawb/zero/commit/8ea53841a5d54c37a153779ae86bea010659433c))
|
||||
* **exec:** stop false INCOMPLETE downgrades on conversational final messages ([#608](https://github.com/Gitlawb/zero/issues/608)) ([b6117af](https://github.com/Gitlawb/zero/commit/b6117af86d6bc87a4ee66910e99d76cb16b03fed))
|
||||
* harden MCP credential boundaries ([#597](https://github.com/Gitlawb/zero/issues/597)) ([fdddb05](https://github.com/Gitlawb/zero/commit/fdddb05ba84b1600ae6c3a20028bf83afe474c44))
|
||||
* **hooks:** fail closed on launch failures for beforeTool hooks ([#629](https://github.com/Gitlawb/zero/issues/629)) ([dc06fe7](https://github.com/Gitlawb/zero/commit/dc06fe72caf45f72d2cba1e8a835c0f5b405c1e8))
|
||||
* **hooks:** run sessionEnd hooks after Esc/Ctrl+C interrupts ([#606](https://github.com/Gitlawb/zero/issues/606)) ([824ecdb](https://github.com/Gitlawb/zero/commit/824ecdbcf9c467c35ef4e2666770fdadcb5bf402))
|
||||
* **keyring:** pass generic password via stdin on macOS ([#574](https://github.com/Gitlawb/zero/issues/574)) ([91ea6de](https://github.com/Gitlawb/zero/commit/91ea6ded7503538834a84d090f78670a363c62d3))
|
||||
* **lock:** prevent POSIX lock file overwrite and leak on Windows/Unix ([#628](https://github.com/Gitlawb/zero/issues/628)) ([da41c3a](https://github.com/Gitlawb/zero/commit/da41c3a75b782d6e0836fe13346321e40a90fbb4))
|
||||
* **openai:** omit prompt_cache_key for openai-compatible providers ([#636](https://github.com/Gitlawb/zero/issues/636)) ([1af5882](https://github.com/Gitlawb/zero/commit/1af58828eb3c22567599c000736c913a290959d2)), closes [#624](https://github.com/Gitlawb/zero/issues/624)
|
||||
* **plugins:** resolve relative executable paths against plugin root ([#627](https://github.com/Gitlawb/zero/issues/627)) ([2efe6d5](https://github.com/Gitlawb/zero/commit/2efe6d539e29374b3ef39c2290bdea81f33a228b))
|
||||
* **sandbox:** remove windowsWriteRestricted flag to fix DenyRead bypass ([#612](https://github.com/Gitlawb/zero/issues/612)) ([3d96ac7](https://github.com/Gitlawb/zero/commit/3d96ac7e55c760a97f28c0e6ceaf1ec3b4ab717a))
|
||||
* **sandbox:** scrub dynamic credential env vars ([#682](https://github.com/Gitlawb/zero/issues/682)) ([9043bae](https://github.com/Gitlawb/zero/commit/9043baedcff7776c7373645b57563cac06b31847))
|
||||
* **sandbox:** scrub sensitive credentials from sandbox environment ([#660](https://github.com/Gitlawb/zero/issues/660)) ([6fc1220](https://github.com/Gitlawb/zero/commit/6fc1220f6ac66fb3ae67b637cbbed7068d2213c0))
|
||||
* **sandbox:** unblock git fetch/commit/add under the write-restricted sandbox ([#654](https://github.com/Gitlawb/zero/issues/654)) ([5c4815a](https://github.com/Gitlawb/zero/commit/5c4815a66ed07d9cf90b825adfd936d3ac07639d))
|
||||
* **sandbox:** use WRITE_RESTRICTED token when no DenyRead paths are configured ([#658](https://github.com/Gitlawb/zero/issues/658)) ([a5d2e32](https://github.com/Gitlawb/zero/commit/a5d2e327c8681671aa8a9e5378801215b747edcf))
|
||||
* **securefile,credstore:** call Sync on temp file before close and rename ([#631](https://github.com/Gitlawb/zero/issues/631)) ([212734a](https://github.com/Gitlawb/zero/commit/212734adf3b5f982e22385161205aa1afe4634fe))
|
||||
* **securefile:** reclaim stale lock files to prevent permanent DOS ([#615](https://github.com/Gitlawb/zero/issues/615)) ([8536cc8](https://github.com/Gitlawb/zero/commit/8536cc87f7a885f9e436d6ef28f5f325201623dc))
|
||||
* **swarm:** wait for job.Runs directly in scheduler skip test ([#667](https://github.com/Gitlawb/zero/issues/667)) ([1bb6b57](https://github.com/Gitlawb/zero/commit/1bb6b5745af90321d1e657a12a1976cded5dd1bd))
|
||||
* **tools:** classify silent wrapped Windows command failures as sandbox denials ([#659](https://github.com/Gitlawb/zero/issues/659)) ([8bd9742](https://github.com/Gitlawb/zero/commit/8bd9742fa95c41b93ea4e718628aed0ff3ae9dd0))
|
||||
* **tools:** preserve SysProcAttr during PTY fallback ([#618](https://github.com/Gitlawb/zero/issues/618)) ([f78b36c](https://github.com/Gitlawb/zero/commit/f78b36c770daa4577a9f99265b18a354454e36eb))
|
||||
* **tui:** resolve pending askUser callbacks to prevent runner hangs ([#620](https://github.com/Gitlawb/zero/issues/620)) ([aa73a76](https://github.com/Gitlawb/zero/commit/aa73a76f1bd1b6fe97bac2fbff2d61b7474139f2))
|
||||
* **tui:** stop the composer cursor blinking while typing or unfocused ([#672](https://github.com/Gitlawb/zero/issues/672)) ([2b42cd5](https://github.com/Gitlawb/zero/commit/2b42cd567b96d6f7e0818594e53ff31cce1e42e9))
|
||||
* **windows:** resolve absolute path for taskkill to prevent hijacking ([#617](https://github.com/Gitlawb/zero/issues/617)) ([2db00ee](https://github.com/Gitlawb/zero/commit/2db00ee3d57db97e0fbe23cb8628e8bcb47f6f09))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* **tools:** add explicit effect metadata for safe concurrency ([#705](https://github.com/Gitlawb/zero/issues/705)) ([8ef8576](https://github.com/Gitlawb/zero/commit/8ef8576df7d0775a5b815bc0776a794cadb75c34))
|
||||
|
||||
## [0.3.0](https://github.com/Gitlawb/zero/compare/v0.2.0...v0.3.0) (2026-07-09)
|
||||
|
||||
|
||||
@@ -134,10 +188,12 @@ tagged. Until then, source builds report the version `dev`.
|
||||
GitHub issue/PR templates.
|
||||
- Interactive `/theme` picker: bare `/theme` opens a popup that live-previews each palette as you move
|
||||
and applies on select (Esc reverts).
|
||||
- Ten built-in color themes alongside the `dark`/`light` built-ins — `dracula`, `nord`, `gruvbox`,
|
||||
`tokyo-night`, `catppuccin`, `one-dark`, `solarized-dark`, `rose-pine`, `everforest`, and
|
||||
`solarized-light` — selectable via `/theme <name>`, `--theme <name>`, or `ZERO_THEME`. Every palette
|
||||
is contrast-audited to WCAG AA. The built-in light theme was reworked for legibility.
|
||||
- Twelve built-in color themes alongside the `dark`/`light` built-ins — `dracula`, `nord`, `gruvbox`,
|
||||
`tokyo-night`, `catppuccin`, `one-dark`, `solarized-dark`, `rose-pine`, `everforest`,
|
||||
`solarized-light`, `dune`, and `neon` — selectable via `/theme <name>`, `--theme <name>`, or
|
||||
`ZERO_THEME`. Every palette is contrast-audited to WCAG AA, and the new presets are additionally
|
||||
audited after xterm-256 downsampling; see [docs/THEMES.md](docs/THEMES.md). The built-in light
|
||||
theme was reworked for legibility.
|
||||
- `--theme <name>` flag for the TUI, accepting `auto` or any registered theme (previously only the
|
||||
`ZERO_THEME` env var existed).
|
||||
- "Accessibility / Appearance" section in the README documenting `NO_COLOR`, `ZERO_THEME`, `/theme`,
|
||||
|
||||
@@ -191,6 +191,8 @@ Useful controls:
|
||||
|---|---|
|
||||
| `Enter` | send the prompt |
|
||||
| `/` | open slash-command suggestions |
|
||||
| `Ctrl+X` then letter | common slash commands (e.g. `m` → `/model`; `Ctrl+X` `?` for full list) |
|
||||
| `Ctrl+P` / `Ctrl+N` | previous / next item in menus (arrows still work) |
|
||||
| `Shift+Tab` | cycle permission mode |
|
||||
| `Ctrl+B` | show/hide the sidebar |
|
||||
| `Ctrl+C` | cancel or exit |
|
||||
@@ -339,7 +341,7 @@ manifest.
|
||||
| Control | Effect |
|
||||
|---|---|
|
||||
| `NO_COLOR=<anything>` | disables color output |
|
||||
| `ZERO_THEME=<name>` | selects the startup theme (`auto`, `dark`, `light`, or a color theme like `dracula`, `nord`, `gruvbox`, `tokyo-night`, `catppuccin`, `one-dark`, `solarized-dark`, `rose-pine`, `everforest`, `solarized-light`) |
|
||||
| `ZERO_THEME=<name>` | selects the startup theme (`auto`, `dark`, `light`, or a color theme like `dracula`, `nord`, `gruvbox`, `tokyo-night`, `catppuccin`, `one-dark`, `solarized-dark`, `rose-pine`, `everforest`, `neon`, `solarized-light`, `dune`) |
|
||||
| `--theme <name>` | selects the TUI theme from the CLI (same names) |
|
||||
| `/theme` | opens the theme picker inside the TUI (live preview; `/theme <name>` switches directly) |
|
||||
| `ZERO_NO_FADE=1` | disables streaming fade animation |
|
||||
@@ -356,6 +358,12 @@ go run ./cmd/zero-release smoke
|
||||
go run ./cmd/zero-perf-bench
|
||||
```
|
||||
|
||||
Experimental: `ZERO_OPENAI_TURN_SESSION=1` enables the optimized OpenAI turn
|
||||
session (background connection prewarm + request-prefix telemetry) for headless
|
||||
`zero exec` runs against official OpenAI profiles. Off by default; `0`/`false`
|
||||
disable. A/B-benchmark it by running the same `zero-perf-bench` suite with the
|
||||
variable unset and set.
|
||||
|
||||
### Code Quality and Security Checks
|
||||
|
||||
Before committing any changes, ensure all Go code quality and security checks pass. Pinned `go run` commands matching CI constraints can be used directly without prior installation:
|
||||
@@ -386,6 +394,7 @@ go run ./cmd/zero-release build --goos windows --goarch amd64 --output dist/zero
|
||||
|
||||
- [Install](docs/INSTALL.md)
|
||||
- [Update flow](docs/UPDATE.md)
|
||||
- [Themes](docs/THEMES.md)
|
||||
- [Stream-JSON protocol](docs/STREAM_JSON_PROTOCOL.md)
|
||||
- [Specialists](docs/SPECIALISTS.md)
|
||||
- [GitHub Action](docs/GITHUB_ACTION.md)
|
||||
|
||||
+2
-1
@@ -234,7 +234,7 @@ zero update 检查更新版本
|
||||
| 控制 | 效果 |
|
||||
|---|---|
|
||||
| `NO_COLOR=<任意值>` | 禁用颜色输出 |
|
||||
| `ZERO_THEME=<名称>` | 选择启动主题(`auto`、`dark`、`light`,或颜色主题如 `dracula`、`nord`、`gruvbox`、`tokyo-night`、`catppuccin`、`one-dark`、`solarized-dark`、`rose-pine`、`everforest`、`solarized-light`) |
|
||||
| `ZERO_THEME=<名称>` | 选择启动主题(`auto`、`dark`、`light`,或颜色主题如 `dracula`、`nord`、`gruvbox`、`tokyo-night`、`catppuccin`、`one-dark`、`solarized-dark`、`rose-pine`、`everforest`、`neon`、`solarized-light`、`dune`) |
|
||||
| `--theme <名称>` | 从 CLI 选择 TUI 主题(相同名称) |
|
||||
| `/theme` | 在 TUI 中打开主题选择器(实时预览;`/theme <名称>` 直接切换) |
|
||||
| `ZERO_NO_FADE=1` | 禁用流式淡入动画 |
|
||||
@@ -261,6 +261,7 @@ go run ./cmd/zero-release build --goos windows --goarch amd64 --output dist/zero
|
||||
|
||||
- [安装](docs/INSTALL.md)
|
||||
- [更新流程](docs/UPDATE.md)
|
||||
- [主题](docs/THEMES.md)
|
||||
- [Stream-JSON 协议](docs/STREAM_JSON_PROTOCOL.md)
|
||||
- [专家](docs/SPECIALISTS.md)
|
||||
- [GitHub Action](docs/GITHUB_ACTION.md)
|
||||
|
||||
+16
-3
@@ -272,7 +272,8 @@ sequenceDiagram
|
||||
Agent->>Tool: execute(ctx, args)
|
||||
Tool-->>Agent: output, error, metadata
|
||||
Agent->>Hooks: afterTool hooks
|
||||
Agent->>Agent: redact, truncate, classify success/failure
|
||||
Agent->>Agent: redact secrets, apply semantic output budget, enforce byte ceiling
|
||||
Agent->>Agent: classify success/failure
|
||||
Agent-->>Surface: OnToolResult callback
|
||||
Agent->>Transcript: append tool result message
|
||||
end
|
||||
@@ -604,10 +605,22 @@ flowchart TD
|
||||
Prompt -- Yes --> Decision[Permission callback]
|
||||
Prompt -- No --> Run[Registry.RunWithOptions]
|
||||
Decision --> Run
|
||||
Run --> Redact[Redact secrets + enforce output ceiling]
|
||||
Redact --> Message[Return tool result to model]
|
||||
Run --> Redact[Redact secrets]
|
||||
Redact --> Budget[Token-aware semantic output budget]
|
||||
Budget --> Ceiling[Existing hard byte ceiling]
|
||||
Ceiling --> Message[Return one tool result to model]
|
||||
```
|
||||
|
||||
Oversized results use deterministic, provider-neutral estimated-token budgets.
|
||||
Policies retain useful structure for files, search matches, tests, process logs,
|
||||
diffs, and worker conclusions; tools without a declared category use a UTF-8-safe
|
||||
head/tail fallback. The estimate is intentionally conservative for non-ASCII
|
||||
text and is not exact provider tokenization. Existing byte ceilings remain the
|
||||
authoritative safety limit. When the existing spill mechanism can persist the
|
||||
complete redacted text received by the budgeting layer, the result includes its
|
||||
safe spill reference. This does not imply capture of subprocess bytes already
|
||||
discarded by a tool's established internal buffer.
|
||||
|
||||
Core tool groups include:
|
||||
|
||||
- **Read-only tools**: file reads, directory listing, glob, grep, LSP navigation,
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
# TUI Themes
|
||||
|
||||
Zero's TUI ships a set of built-in color themes. Pick one with `--theme <name>`,
|
||||
the `ZERO_THEME` environment variable, or the `/theme <name>` command while
|
||||
running. `auto` (the default) follows the terminal's detected background.
|
||||
|
||||
Run `/theme` with no argument to open a picker: move through the list to
|
||||
live-preview each theme, press Enter to apply the highlighted one, or Esc to
|
||||
cancel and restore the previously active theme. Run `/theme list` to print
|
||||
the active theme and the registered names without opening the picker.
|
||||
|
||||
## Dune (`dune`)
|
||||
|
||||
A warm sand-and-cream palette: sand/cream surface, charcoal ink, and a soft
|
||||
amber accent.
|
||||
|
||||
## Neon (`neon`)
|
||||
|
||||
A neon-on-black palette: pitch-black surface, bright green ink, and a cyan accent.
|
||||
|
||||
## Adding a theme
|
||||
|
||||
Every theme is a `palette` (a table of color hex tokens) plus one entry in
|
||||
`themeRegistry`, both in `internal/tui/theme_palettes.go`. `buildTheme` in
|
||||
`internal/tui/theme.go` turns a `palette` into the resolved `lipgloss.Style`
|
||||
set every renderer reads from the active `zeroTheme`. Adding a theme means
|
||||
adding a new `palette{...}` literal, a `themeRegistry` entry, and test
|
||||
coverage for the new palette (see below).
|
||||
|
||||
Registry-wide tests in `internal/tui/theme_select_test.go` assert the basic
|
||||
WCAG AA text tokens, the gray-ramp order, the diff word-span pairs, and the
|
||||
selected-row band for every entry. The rendered-surface invariants beyond
|
||||
those (permission surfaces, selected-row secondary text, diff gutters, and
|
||||
the xterm-256 downsampling checks) are asserted per palette, not against the
|
||||
whole registry: `TestExtendedThemeContrastInvariants` and
|
||||
`TestExtendedThemeANSI256Contrast` enumerate the palettes they cover. A new
|
||||
theme must be added to those tests (or given equivalent palette-specific
|
||||
assertions), or CI can stay green while its permission, selected-row, and
|
||||
diff surfaces ship unreadable.
|
||||
@@ -141,9 +141,11 @@ func TestRunAskUserCancellationAbortsRun(t *testing.T) {
|
||||
registry := registryWithAskUser()
|
||||
args := `{"questions":[{"question":"Which framework?"}]}`
|
||||
provider := providerCallingAskUserThenAnswer(args, "done")
|
||||
var toolResults []ToolResult
|
||||
|
||||
result, err := Run(context.Background(), "clarify", provider, Options{
|
||||
Registry: registry,
|
||||
Registry: registry,
|
||||
OnToolResult: func(result ToolResult) { toolResults = append(toolResults, result) },
|
||||
OnAskUser: func(_ context.Context, _ AskUserRequest) (AskUserResponse, error) {
|
||||
return AskUserResponse{}, context.Canceled
|
||||
},
|
||||
@@ -158,6 +160,9 @@ func TestRunAskUserCancellationAbortsRun(t *testing.T) {
|
||||
if len(provider.requests) != 1 {
|
||||
t.Fatalf("expected the run to stop after the canceled ask_user (1 turn), got %d", len(provider.requests))
|
||||
}
|
||||
if len(toolResults) != 1 || toolResults[0].ToolCallID == "" {
|
||||
t.Fatalf("cancellation must emit exactly one result for the call, got %#v", toolResults)
|
||||
}
|
||||
// The recorded tool result must reflect cancellation, not a synthetic answer.
|
||||
var toolMsg string
|
||||
for _, m := range result.Messages {
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
package agent
|
||||
|
||||
// CompletionDecision is the deterministic outcome of evaluating a text-only
|
||||
// assistant turn. Uncertain means the loop needs one bounded follow-up action
|
||||
// before it can finalize the run.
|
||||
type CompletionDecision string
|
||||
|
||||
const (
|
||||
CompletionUncertain CompletionDecision = "uncertain"
|
||||
CompletionComplete CompletionDecision = "complete"
|
||||
CompletionIncomplete CompletionDecision = "incomplete"
|
||||
)
|
||||
|
||||
type completionAction string
|
||||
|
||||
const (
|
||||
completionActionNone completionAction = ""
|
||||
completionActionContinue completionAction = "continue"
|
||||
completionActionSemanticCheck completionAction = "semantic_check"
|
||||
)
|
||||
|
||||
type completionEvaluation struct {
|
||||
Decision CompletionDecision
|
||||
Action completionAction
|
||||
Reason string
|
||||
}
|
||||
|
||||
// completionPolicy owns the small amount of state needed to keep uncertain
|
||||
// decisions bounded. requireSemanticCheck is set only for run profiles that
|
||||
// already opted into self-correction; ordinary completion remains entirely
|
||||
// local and adds no model call.
|
||||
type completionPolicy struct {
|
||||
continueNudges int
|
||||
requireSemanticCheck bool
|
||||
semanticCheckRequested bool
|
||||
}
|
||||
|
||||
func newCompletionPolicy(requireSemanticCheck bool) *completionPolicy {
|
||||
return &completionPolicy{requireSemanticCheck: requireSemanticCheck}
|
||||
}
|
||||
|
||||
func (policy *completionPolicy) evaluate(text string, planPending bool) completionEvaluation {
|
||||
// A direct admission is strong local evidence and takes precedence over all
|
||||
// weaker signals, avoiding both continuation nudges and semantic checks.
|
||||
if reason := selfReportedIncompletion(text); reason != "" {
|
||||
return completionEvaluation{Decision: CompletionIncomplete, Reason: reason}
|
||||
}
|
||||
|
||||
// A continuation cue is strong unfinished evidence; a pending plan is only
|
||||
// weak evidence because bookkeeping can be stale. Both get a bounded chance
|
||||
// to continue, but only a persisted cue is ultimately incomplete.
|
||||
cue := endsWithContinuationCue(text)
|
||||
if cue || planPending {
|
||||
if policy.continueNudges < maxContinueNudges {
|
||||
policy.continueNudges++
|
||||
reason := "your message ended mid-step"
|
||||
if !cue {
|
||||
reason = "pending plan items remain — finish them, or mark them complete with update_plan if you are done"
|
||||
}
|
||||
return completionEvaluation{
|
||||
Decision: CompletionUncertain,
|
||||
Action: completionActionContinue,
|
||||
Reason: reason,
|
||||
}
|
||||
}
|
||||
if cue {
|
||||
return completionEvaluation{
|
||||
Decision: CompletionIncomplete,
|
||||
Reason: "your message ended mid-step",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Profiles with self-correction enabled require one task-grounded semantic
|
||||
// check. It is requested at most once per run; the next locally complete turn
|
||||
// is accepted without another model call.
|
||||
if policy.requireSemanticCheck && !policy.semanticCheckRequested {
|
||||
policy.semanticCheckRequested = true
|
||||
return completionEvaluation{
|
||||
Decision: CompletionUncertain,
|
||||
Action: completionActionSemanticCheck,
|
||||
}
|
||||
}
|
||||
|
||||
return completionEvaluation{Decision: CompletionComplete, Action: completionActionNone}
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package agent
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestCompletionPolicyLocalEvidenceDecidesWithoutSemanticCheck(t *testing.T) {
|
||||
policy := newCompletionPolicy(false)
|
||||
|
||||
complete := policy.evaluate("Done. All required checks pass.", false)
|
||||
if complete.Decision != CompletionComplete {
|
||||
t.Fatalf("confident completion decision = %q, want %q", complete.Decision, CompletionComplete)
|
||||
}
|
||||
|
||||
incomplete := policy.evaluate("I couldn't verify the result, so this is my best guess.", false)
|
||||
if incomplete.Decision != CompletionIncomplete {
|
||||
t.Fatalf("admitted failure decision = %q, want %q", incomplete.Decision, CompletionIncomplete)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompletionPolicyPreservesBoundedPlanStallProtection(t *testing.T) {
|
||||
policy := newCompletionPolicy(false)
|
||||
for attempt := 0; attempt < maxContinueNudges; attempt++ {
|
||||
got := policy.evaluate("Let me inspect the remaining configuration:", true)
|
||||
if got.Decision != CompletionUncertain || got.Action != completionActionContinue {
|
||||
t.Fatalf("attempt %d = (%q, %q), want uncertain continue", attempt+1, got.Decision, got.Action)
|
||||
}
|
||||
}
|
||||
|
||||
got := policy.evaluate("Let me inspect the remaining configuration:", true)
|
||||
if got.Decision != CompletionIncomplete {
|
||||
t.Fatalf("decision after nudge budget = %q, want %q", got.Decision, CompletionIncomplete)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompletionPolicyTreatsPendingPlanAsWeakEvidence(t *testing.T) {
|
||||
policy := newCompletionPolicy(false)
|
||||
for attempt := 0; attempt < maxContinueNudges; attempt++ {
|
||||
got := policy.evaluate("All set.", true)
|
||||
if got.Decision != CompletionUncertain || got.Action != completionActionContinue {
|
||||
t.Fatalf("attempt %d = (%q, %q), want uncertain continue", attempt+1, got.Decision, got.Action)
|
||||
}
|
||||
}
|
||||
|
||||
got := policy.evaluate("All set.", true)
|
||||
if got.Decision != CompletionComplete {
|
||||
t.Fatalf("stale plan decision after nudge budget = %q, want %q", got.Decision, CompletionComplete)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompletionPolicyAllowsExactlyOneRequiredSemanticCheck(t *testing.T) {
|
||||
policy := newCompletionPolicy(true)
|
||||
|
||||
first := policy.evaluate("Implemented and tested.", false)
|
||||
if first.Decision != CompletionUncertain || first.Action != completionActionSemanticCheck {
|
||||
t.Fatalf("first decision = (%q, %q), want uncertain semantic check", first.Decision, first.Action)
|
||||
}
|
||||
|
||||
second := policy.evaluate("PASS. The result meets the task criterion.", false)
|
||||
if second.Decision != CompletionComplete {
|
||||
t.Fatalf("post-check decision = %q, want %q", second.Decision, CompletionComplete)
|
||||
}
|
||||
if second.Action == completionActionSemanticCheck {
|
||||
t.Fatal("semantic check was requested more than once")
|
||||
}
|
||||
}
|
||||
+145
-78
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
@@ -135,6 +136,33 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
}
|
||||
}
|
||||
|
||||
// Establish the run's turn session — the seam an optimized provider session
|
||||
// (connection reuse, prewarm, native compaction) plugs into without touching
|
||||
// this loop. Default: wrap the passed provider so the session's Stream IS
|
||||
// provider.StreamCompletion, byte-identical to calling it directly. Opened
|
||||
// before dispatchSessionStart so a failed open is a clean run-start error
|
||||
// with no session hooks fired and nothing to unwind.
|
||||
turnSessions := options.TurnSessionProvider
|
||||
if turnSessions == nil {
|
||||
turnSessions = zeroruntime.NewProviderTurnSessionProvider(provider, zeroruntime.ProviderCapabilities{})
|
||||
}
|
||||
session, sessionErr := turnSessions.OpenTurnSession(ctx)
|
||||
if sessionErr != nil {
|
||||
return Result{}, fmt.Errorf("agent: open turn session: %w", sessionErr)
|
||||
}
|
||||
// Closed via closure (not `defer session.Close()`) so after a mid-run model
|
||||
// swap the CURRENT session is the one closed at teardown — the swap closes
|
||||
// the old one inline. Close errors are advisory and never fail the run.
|
||||
defer func() { _ = session.Close() }()
|
||||
// Best-effort prewarm: the default adapter no-ops; a real session may prime
|
||||
// a connection. Failure is ignored by contract (never fatal).
|
||||
_ = session.Prewarm(ctx)
|
||||
// Route ALL provider I/O — per-turn streams, mid-stream retries, the
|
||||
// compaction summarizer, and the final-answer request — through the session
|
||||
// by reassigning the loop's provider local. Every existing call site reads
|
||||
// this local, so no signatures change anywhere downstream.
|
||||
provider = sessionProvider{session: session}
|
||||
|
||||
maxTurns := options.MaxTurns
|
||||
if maxTurns <= 0 {
|
||||
maxTurns = 12
|
||||
@@ -169,13 +197,10 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// loaded tool's full schema; it lives only for the run (v1 within-run scope).
|
||||
loaded := map[string]bool{}
|
||||
|
||||
// continueNudges counts how many times the headless completion gate
|
||||
// (Options.RequireCompletionSignal) has re-prompted a no-tool-call turn that
|
||||
// stopped with work still unfinished. Bounded by maxContinueNudges.
|
||||
continueNudges := 0
|
||||
// acceptanceRequested records that the one-time task-grounded acceptance check
|
||||
// has already been demanded this run, so it fires at most once.
|
||||
acceptanceRequested := false
|
||||
// The feature-gated completion policy keeps local completion decisions and its
|
||||
// bounded follow-up state out of the central loop. Self-correcting profiles
|
||||
// opt into the one permitted task-grounded semantic check.
|
||||
completionPolicy := newCompletionPolicy(options.SelfCorrect != nil)
|
||||
|
||||
// toolDefCache memoizes each tool's rendered JSON-schema definition across
|
||||
// turns (a tool's advertised schema is stable for the run), so partitionTools
|
||||
@@ -185,7 +210,11 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
result = Result{Messages: copyMessages(messages)}
|
||||
dispatchSessionStart(ctx, options)
|
||||
defer func() {
|
||||
dispatchSessionEnd(ctx, options, result, err)
|
||||
// Use a context detached from ctx's cancellation: on Esc/Ctrl+C the run
|
||||
// context is already canceled by the time this defer runs, and a canceled
|
||||
// parent would make Hooks.Dispatch's context.WithTimeout derive an
|
||||
// already-expired context, so the sessionEnd hook would never actually run.
|
||||
dispatchSessionEnd(context.WithoutCancel(ctx), options, result, err)
|
||||
}()
|
||||
for turn := 0; turn < maxTurns; turn++ {
|
||||
result.Turns = turn + 1
|
||||
@@ -213,6 +242,26 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
exposed, _ := partitionToolsCached(registry, permissionMode, options, loaded, toolDefCache)
|
||||
toolPartitionSpan.End()
|
||||
|
||||
// Prompt-prefix fingerprint: hash the seven cacheable sub-components
|
||||
// of this turn's request (system prompt sections, project context,
|
||||
// skills, partitioned tool list, tool schemas) and emit one trace
|
||||
// event. Stable across turns => cacheable. Drift in any sub-hash
|
||||
// names the sub-component that broke the cache for this turn. The
|
||||
// computation is pure and cheap (seven SHA-256s of small strings)
|
||||
// and is a no-op when tracing is off.
|
||||
if options.Trace != nil {
|
||||
fp := ComputePrefixFingerprint(options, exposed)
|
||||
options.Trace.EmitPrefixHash(trace.PrefixHash{
|
||||
BaseInstructionsHash: fp.BaseInstructionsHash,
|
||||
ConfirmationPolicyHash: fp.ConfirmationPolicyHash,
|
||||
ProjectContextHash: fp.ProjectContextHash,
|
||||
SkillsHash: fp.SkillsHash,
|
||||
ToolsHash: fp.ToolsHash,
|
||||
SchemaHash: fp.SchemaHash,
|
||||
CompletePrefixHash: fp.CompletePrefixHash,
|
||||
})
|
||||
}
|
||||
|
||||
// PROACTIVE compaction: if the history is approaching the model's
|
||||
// context window, summarize the oldest middle before building the
|
||||
// request. A no-op when ContextWindow == 0 (compaction disabled).
|
||||
@@ -496,68 +545,32 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// model's final answer ONLY when the work is actually done. Default off
|
||||
// (RequireCompletionSignal), so interactive runs stay byte-identical.
|
||||
if options.RequireCompletionSignal {
|
||||
// (1) Self-report downgrade (strongest, unambiguous): the model's own
|
||||
// final message admits it guessed / could not meet the objective. Checked
|
||||
// FIRST so an admitted-impossible task is downgraded immediately (no wasted
|
||||
// continue-nudges) and reports the accurate reason.
|
||||
if reason := selfReportedIncompletion(collected.Text); reason != "" {
|
||||
evaluation := completionPolicy.evaluate(collected.Text, guards.pendingPlanItems())
|
||||
switch evaluation.Decision {
|
||||
case CompletionIncomplete:
|
||||
result.Incomplete = true
|
||||
result.IncompleteReason = reason
|
||||
result.IncompleteReason = evaluation.Reason
|
||||
result.FinalAnswer = collected.Text
|
||||
result.Messages = copyMessages(messages)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// (2) The model stopped without a tool call while work may be unfinished:
|
||||
// - a continuation cue ("…Let me check the config:") is an unambiguous
|
||||
// mid-step stop;
|
||||
// - pending update_plan items are a WEAK, ambiguous signal (the model may
|
||||
// have finished without re-marking the last step).
|
||||
// Nudge to continue (bounded). After the budget: a persisted continuation
|
||||
// cue finalizes INCOMPLETE; pending-plan WITHOUT a cue does NOT (that would
|
||||
// false-fail a completed run with stale bookkeeping) — fall through to the
|
||||
// acceptance check / success.
|
||||
cue := endsWithContinuationCue(collected.Text)
|
||||
planPending := guards.pendingPlanItems()
|
||||
if cue || planPending {
|
||||
if continueNudges < maxContinueNudges {
|
||||
continueNudges++
|
||||
case CompletionUncertain:
|
||||
switch evaluation.Action {
|
||||
case completionActionContinue:
|
||||
options.Trace.Counter(trace.CounterCompletionNudges, 1)
|
||||
reason := "your message ended mid-step"
|
||||
if !cue {
|
||||
reason = "pending plan items remain — finish them, or mark them complete with update_plan if you are done"
|
||||
}
|
||||
messages = append(messages, zeroruntime.Message{
|
||||
Role: zeroruntime.MessageRoleUser,
|
||||
Content: continueNudge(reason),
|
||||
Content: continueNudge(evaluation.Reason),
|
||||
})
|
||||
case completionActionSemanticCheck:
|
||||
options.Trace.Counter(trace.CounterAcceptanceChecks, 1)
|
||||
messages = append(messages, zeroruntime.Message{
|
||||
Role: zeroruntime.MessageRoleUser,
|
||||
Content: acceptanceVerificationNudge(),
|
||||
})
|
||||
continue
|
||||
}
|
||||
if cue {
|
||||
result.Incomplete = true
|
||||
result.IncompleteReason = "your message ended mid-step"
|
||||
result.FinalAnswer = collected.Text
|
||||
result.Messages = copyMessages(messages)
|
||||
return result, nil
|
||||
}
|
||||
// pending-plan only, budget spent: trust the model's completion claim
|
||||
// over stale plan bookkeeping; fall through.
|
||||
}
|
||||
|
||||
// (3) Task-grounded acceptance: before accepting a "done" turn as success,
|
||||
// require ONE acceptance check grounded in the task's stated criterion
|
||||
// (only when self-correct is on). Rejects "well-formed == correct",
|
||||
// "existing-tests-pass == objective met", and "result == baseline" false
|
||||
// successes. Bounded to a single pass; a genuine post-check completion
|
||||
// (no admission, no cue) then finalizes as success on the next turn.
|
||||
if options.SelfCorrect != nil && !acceptanceRequested {
|
||||
acceptanceRequested = true
|
||||
options.Trace.Counter(trace.CounterAcceptanceChecks, 1)
|
||||
messages = append(messages, zeroruntime.Message{
|
||||
Role: zeroruntime.MessageRoleUser,
|
||||
Content: acceptanceVerificationNudge(),
|
||||
})
|
||||
continue
|
||||
case CompletionComplete:
|
||||
// Local evidence is sufficient; proceed to final diagnostics.
|
||||
}
|
||||
}
|
||||
// Finalization diagnostics gate: edits from this run may still have
|
||||
@@ -608,10 +621,9 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// The scan happens lazily at the run's first index — never ahead of a
|
||||
// pending mutating call — so read-after-write ordering is preserved.
|
||||
if index >= precomputedEnd {
|
||||
runEnd := index
|
||||
for runEnd < len(collected.ToolCalls) && parallelSafeToolCall(registry, collected.ToolCalls[runEnd], options) {
|
||||
runEnd++
|
||||
}
|
||||
// Capability-safe consecutive run (ReadOnly+ThreadSafe, no
|
||||
// resource-key conflicts). See extendParallelRun.
|
||||
runEnd := extendParallelRun(registry, collected.ToolCalls, index, options)
|
||||
if runEnd-index >= 2 {
|
||||
batchSpan := options.Trace.Span(trace.SpanToolExecution)
|
||||
precomputed = executeParallelReadBatch(ctx, registry, collected.ToolCalls, index, runEnd, permissionMode, options)
|
||||
@@ -633,6 +645,7 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
toolSpan.End()
|
||||
}
|
||||
options.Trace.Counter(trace.CounterToolCalls, 1)
|
||||
recordOutputBudgetTrace(options.Trace, toolResult)
|
||||
if options.OnToolResult != nil {
|
||||
options.OnToolResult(toolResult)
|
||||
}
|
||||
@@ -724,25 +737,53 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// of the run. At most one switch per turn (first request wins). A switcher
|
||||
// error is NON-FATAL — record a brief note and continue on the current
|
||||
// model. nil switcher ⇒ requests are ignored entirely (escalation off).
|
||||
if turnRequestedModel != "" && options.ModelSwitcher != nil {
|
||||
newProvider, switchErr := options.ModelSwitcher(ctx, turnRequestedModel)
|
||||
if turnRequestedModel != "" && (options.ModelSessionSwitcher != nil || options.ModelSwitcher != nil) {
|
||||
// Resolve the escalated model's session source. The target-aware
|
||||
// ModelSessionSwitcher is preferred: its TurnSessionProvider carries an
|
||||
// optimized session (and capabilities) across the swap. The legacy
|
||||
// ModelSwitcher fallback returns a bare Provider, wrapped in the
|
||||
// default no-op session — identical to pre-seam behavior.
|
||||
var newSessions zeroruntime.TurnSessionProvider
|
||||
var switchErr error
|
||||
if options.ModelSessionSwitcher != nil {
|
||||
newSessions, switchErr = options.ModelSessionSwitcher(ctx, turnRequestedModel)
|
||||
} else {
|
||||
var newProvider Provider
|
||||
newProvider, switchErr = options.ModelSwitcher(ctx, turnRequestedModel)
|
||||
if switchErr == nil && newProvider != nil {
|
||||
newSessions = zeroruntime.NewProviderTurnSessionProvider(newProvider, zeroruntime.ProviderCapabilities{})
|
||||
}
|
||||
}
|
||||
if switchErr != nil {
|
||||
messages = append(messages, zeroruntime.Message{
|
||||
Role: zeroruntime.MessageRoleUser,
|
||||
Content: escalationFailedNoticePrefix + " (" + turnRequestedModel + "): " + switchErr.Error() + ". Continuing on " + options.Model + ".",
|
||||
})
|
||||
} else if newProvider != nil {
|
||||
// Reassign the local provider so the next turn's StreamCompletion and
|
||||
// compaction use it; update options.Model so subsequent RunOptions.Model,
|
||||
// context-window sizing, and usage attribution follow the new model.
|
||||
provider = newProvider
|
||||
options.Model = turnRequestedModel
|
||||
options.Trace.Counter(trace.CounterModelSwitches, 1)
|
||||
// KNOWN LIMITATION (deferred): the compactor's context-window budget
|
||||
// is fixed at run start from options.ContextWindow and is NOT updated
|
||||
// here, so a switch to a model with a different window keeps compacting
|
||||
// against the original budget. Fixing it needs a ModelSwitcher contract
|
||||
// change (return the new window) — out of scope for this change.
|
||||
} else if newSessions != nil {
|
||||
// Open the new session and close the old one, so the next turn's
|
||||
// streams and compaction use the new model. For the default adapter
|
||||
// the open never errors and Close is a no-op, so this stays
|
||||
// byte-identical to a direct provider reassignment; the guarded open
|
||||
// and prewarm matter once a session holds real state.
|
||||
newSession, openErr := newSessions.OpenTurnSession(ctx)
|
||||
if openErr != nil {
|
||||
messages = append(messages, zeroruntime.Message{
|
||||
Role: zeroruntime.MessageRoleUser,
|
||||
Content: escalationFailedNoticePrefix + " (" + turnRequestedModel + "): " + openErr.Error() + ". Continuing on " + options.Model + ".",
|
||||
})
|
||||
} else {
|
||||
_ = session.Close()
|
||||
session = newSession
|
||||
_ = session.Prewarm(ctx)
|
||||
provider = sessionProvider{session: session}
|
||||
options.Model = turnRequestedModel
|
||||
options.Trace.Counter(trace.CounterModelSwitches, 1)
|
||||
// KNOWN LIMITATION (deferred): the compactor's context-window budget
|
||||
// is fixed at run start from options.ContextWindow and is NOT updated
|
||||
// here, so a switch to a model with a different window keeps compacting
|
||||
// against the original budget. Fixing it needs the switcher to also
|
||||
// report the new window — out of scope for this change.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -818,6 +859,28 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func recordOutputBudgetTrace(recorder *trace.Recorder, result ToolResult) {
|
||||
if recorder == nil || result.Meta["output_budget_category"] == "" {
|
||||
return
|
||||
}
|
||||
parseInt := func(key string) int {
|
||||
value, _ := strconv.Atoi(result.Meta[key])
|
||||
return value
|
||||
}
|
||||
spillCreated, _ := strconv.ParseBool(result.Meta["output_budget_spill_created"])
|
||||
recorder.EmitOutputBudget(trace.OutputBudgetEvent{
|
||||
Tool: result.Name,
|
||||
Category: result.Meta["output_budget_category"],
|
||||
OriginalBytes: parseInt("output_budget_original_bytes"),
|
||||
RetainedBytes: parseInt("output_budget_retained_bytes"),
|
||||
EstimatedOriginalTokens: parseInt("output_budget_estimated_original_tokens"),
|
||||
EstimatedRetainedTokens: parseInt("output_budget_estimated_retained_tokens"),
|
||||
Truncated: result.Truncated,
|
||||
Reason: result.Meta["output_budget_reason"],
|
||||
SpillCreated: spillCreated,
|
||||
})
|
||||
}
|
||||
|
||||
func finalAnswerAfterMaxTurns(ctx context.Context, provider Provider, messages []zeroruntime.Message, toolDefs []zeroruntime.ToolDefinition, options Options) (string, []zeroruntime.Message, string) {
|
||||
finalMessages := copyMessages(messages)
|
||||
finalMessages = append(finalMessages, zeroruntime.Message{
|
||||
@@ -1272,6 +1335,7 @@ func executeToolCall(ctx context.Context, registry *tools.Registry, call ToolCal
|
||||
if didRedact {
|
||||
result.Redacted = true
|
||||
}
|
||||
result = registry.RebudgetAfterHook(call.Name, args, result)
|
||||
}
|
||||
}
|
||||
// Secret scrubbing happens at the registry boundary (the single point both
|
||||
@@ -1282,6 +1346,7 @@ func executeToolCall(ctx context.Context, registry *tools.Registry, call ToolCal
|
||||
Name: call.Name,
|
||||
Status: result.Status,
|
||||
Output: result.Output,
|
||||
Truncated: result.Truncated,
|
||||
Meta: result.Meta,
|
||||
Redacted: result.Redacted,
|
||||
ChangedFiles: result.ChangedFiles,
|
||||
@@ -1592,6 +1657,7 @@ func toolResultFromPrePermissionReject(call ToolCall, result tools.Result) ToolR
|
||||
Name: call.Name,
|
||||
Status: result.Status,
|
||||
Output: output,
|
||||
Truncated: result.Truncated,
|
||||
Meta: meta,
|
||||
Redacted: result.Redacted || outputRedacted || summaryRedacted || metaRedacted,
|
||||
ChangedFiles: result.ChangedFiles,
|
||||
@@ -1861,6 +1927,7 @@ func askUserFallbackResult(ctx context.Context, registry *tools.Registry, call T
|
||||
Name: call.Name,
|
||||
Status: result.Status,
|
||||
Output: result.Output,
|
||||
Truncated: result.Truncated,
|
||||
Meta: result.Meta,
|
||||
Redacted: result.Redacted,
|
||||
ChangedFiles: result.ChangedFiles,
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
@@ -93,6 +94,87 @@ func TestRunDispatchesSessionLifecycleHooks(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// cancelingProvider simulates an Esc/Ctrl+C abort: it cancels the run's own
|
||||
// context mid-stream (as the TUI's interrupt handler would) and returns no
|
||||
// events, so Run observes ctx.Err() and returns early with an already-canceled
|
||||
// context in scope for the deferred sessionEnd dispatch.
|
||||
type cancelingProvider struct {
|
||||
cancel context.CancelFunc
|
||||
}
|
||||
|
||||
func (provider *cancelingProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
||||
provider.cancel()
|
||||
ch := make(chan zeroruntime.StreamEvent)
|
||||
close(ch)
|
||||
return ch, nil
|
||||
}
|
||||
|
||||
// TestRunDispatchesSessionEndHookAfterContextCancellation is a regression test
|
||||
// for a bug where an interrupted run (Esc/Ctrl+C) canceled ctx before the
|
||||
// deferred sessionEnd dispatch ran, so Hooks.Dispatch's context.WithTimeout(ctx, ...)
|
||||
// derived an already-canceled context and the hook command never actually launched.
|
||||
func TestRunDispatchesSessionEndHookAfterContextCancellation(t *testing.T) {
|
||||
goBinary, err := exec.LookPath("go")
|
||||
if err != nil {
|
||||
// The test binary runs on the build machine, so its build-time GOROOT
|
||||
// still identifies the toolchain that produced it.
|
||||
goRoot := runtime.GOROOT() //nolint:staticcheck // Safe for this non-portable test binary.
|
||||
goBinary = filepath.Join(goRoot, "bin", "go")
|
||||
if runtime.GOOS == "windows" {
|
||||
goBinary += ".exe"
|
||||
}
|
||||
if _, statErr := os.Stat(goBinary); statErr != nil {
|
||||
t.Skipf("go binary unavailable on PATH or in GOROOT: %v", statErr)
|
||||
}
|
||||
}
|
||||
audit, err := hooks.NewAuditStore(hooks.AuditStoreOptions{AuditPath: filepath.Join(t.TempDir(), "audit.jsonl")})
|
||||
if err != nil {
|
||||
t.Fatalf("NewAuditStore: %v", err)
|
||||
}
|
||||
dispatcher := hooks.NewDispatcher(hooks.DispatcherOptions{
|
||||
Config: hooks.Config{
|
||||
Enabled: true,
|
||||
Hooks: []hooks.Definition{
|
||||
{ID: "zero.session-end", Event: hooks.EventSessionEnd, Command: goBinary, Args: []string{"version"}, Enabled: true},
|
||||
},
|
||||
},
|
||||
Audit: audit,
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
provider := &cancelingProvider{cancel: cancel}
|
||||
|
||||
_, err = Run(ctx, "hi", provider, Options{
|
||||
SessionID: "session-cancel",
|
||||
Cwd: t.TempDir(),
|
||||
ProviderName: "test-provider",
|
||||
Model: "test-model",
|
||||
Hooks: dispatcher,
|
||||
})
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("Run err = %v, want context.Canceled", err)
|
||||
}
|
||||
|
||||
events, err := audit.ReadEvents()
|
||||
if err != nil {
|
||||
t.Fatalf("ReadEvents: %v", err)
|
||||
}
|
||||
var completedStatus hooks.AuditStatus
|
||||
found := false
|
||||
for _, event := range events {
|
||||
if event.Type == "hook_execution_completed" && event.Event == hooks.EventSessionEnd {
|
||||
completedStatus = event.Status
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("no sessionEnd hook_execution_completed audit event, events=%#v", events)
|
||||
}
|
||||
if completedStatus != hooks.AuditCompleted {
|
||||
t.Fatalf("sessionEnd hook status = %q, want %q (hook must actually run despite ctx cancellation)", completedStatus, hooks.AuditCompleted)
|
||||
}
|
||||
}
|
||||
|
||||
type recordingWebSearchTool struct {
|
||||
calls []map[string]any
|
||||
}
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/hooks"
|
||||
"github.com/Gitlawb/zero/internal/tools"
|
||||
"github.com/Gitlawb/zero/internal/trace"
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
type propagationOutputTool struct {
|
||||
output string
|
||||
}
|
||||
|
||||
func TestOutputBudgetHookHelperProcess(t *testing.T) {
|
||||
for index, arg := range os.Args {
|
||||
if arg != "--zero-output-budget-hook" || index+1 >= len(os.Args) {
|
||||
continue
|
||||
}
|
||||
if _, err := os.Stdout.WriteString(os.Args[index+1]); err != nil {
|
||||
os.Exit(2)
|
||||
}
|
||||
os.Exit(0)
|
||||
}
|
||||
}
|
||||
|
||||
func largeOutputBudgetHookDispatcher() *hooks.Dispatcher {
|
||||
feedback := strings.Repeat("hook feedback ", 200)
|
||||
return hooks.NewDispatcher(hooks.DispatcherOptions{Config: hooks.Config{
|
||||
Enabled: true,
|
||||
Hooks: []hooks.Definition{{
|
||||
ID: "large-feedback",
|
||||
Event: hooks.EventAfterTool,
|
||||
Matcher: "propagation_output",
|
||||
Command: os.Args[0],
|
||||
Args: []string{
|
||||
"-test.run=TestOutputBudgetHookHelperProcess",
|
||||
"--",
|
||||
"--zero-output-budget-hook",
|
||||
feedback,
|
||||
},
|
||||
Enabled: true,
|
||||
}},
|
||||
}})
|
||||
}
|
||||
|
||||
func (tool propagationOutputTool) Name() string { return "propagation_output" }
|
||||
func (tool propagationOutputTool) Description() string { return "returns output for propagation tests" }
|
||||
func (tool propagationOutputTool) Parameters() tools.Schema {
|
||||
return tools.Schema{Type: "object", AdditionalProperties: false}
|
||||
}
|
||||
func (tool propagationOutputTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow, Reason: "test read"}
|
||||
}
|
||||
func (tool propagationOutputTool) Run(context.Context, map[string]any) tools.Result {
|
||||
return tools.Result{Status: tools.StatusOK, Output: tool.output}
|
||||
}
|
||||
|
||||
func TestExecuteToolCallPropagatesOutputTruncation(t *testing.T) {
|
||||
t.Setenv("TMPDIR", t.TempDir())
|
||||
t.Setenv("ZERO_TOOL_OUTPUT_CEILING_TOKENS", "80")
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(propagationOutputTool{output: strings.Repeat("large output\n", 1000)})
|
||||
|
||||
result, abortErr := executeToolCall(context.Background(), registry, ToolCall{
|
||||
ID: "call-budget",
|
||||
Name: "propagation_output",
|
||||
Arguments: `{}`,
|
||||
}, PermissionModeAuto, Options{Cwd: t.TempDir()})
|
||||
if abortErr != nil {
|
||||
t.Fatalf("executeToolCall abort error: %v", abortErr)
|
||||
}
|
||||
if !result.Truncated {
|
||||
t.Fatalf("agent ToolResult lost tools.Result.Truncated: %#v", result)
|
||||
}
|
||||
if result.Meta["spill_path"] == "" {
|
||||
t.Fatalf("agent ToolResult lost spill metadata: %#v", result.Meta)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRecordOutputBudgetTraceUsesOnlyCompactMetadata(t *testing.T) {
|
||||
recorder := trace.NewRecorder("session", "run", "")
|
||||
recorder.Start()
|
||||
recordOutputBudgetTrace(recorder, ToolResult{
|
||||
Name: "grep",
|
||||
Truncated: true,
|
||||
Output: "SECRET OUTPUT MUST NOT ENTER TRACE",
|
||||
Meta: map[string]string{
|
||||
"output_budget_category": "search",
|
||||
"output_budget_original_bytes": "1000",
|
||||
"output_budget_retained_bytes": "100",
|
||||
"output_budget_estimated_original_tokens": "250",
|
||||
"output_budget_estimated_retained_tokens": "25",
|
||||
"output_budget_reason": "semantic_search_budget",
|
||||
"output_budget_spill_created": "true",
|
||||
"spill_path": "/secret/path/not-for-trace",
|
||||
},
|
||||
})
|
||||
events := recorder.Finish().OutputBudgets
|
||||
if len(events) != 1 {
|
||||
t.Fatalf("events = %#v", events)
|
||||
}
|
||||
event := events[0]
|
||||
if event.Tool != "grep" || event.Category != "search" || event.OriginalBytes != 1000 || event.RetainedBytes != 100 || !event.SpillCreated {
|
||||
t.Fatalf("unexpected trace event: %#v", event)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteToolCallRebudgetsOversizedAfterToolFeedback(t *testing.T) {
|
||||
t.Setenv("ZERO_TOOL_OUTPUT_CEILING_TOKENS", "80")
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(propagationOutputTool{output: "tool output"})
|
||||
dispatcher := largeOutputBudgetHookDispatcher()
|
||||
|
||||
result, abortErr := executeToolCall(context.Background(), registry, ToolCall{
|
||||
ID: "call-hook-budget",
|
||||
Name: "propagation_output",
|
||||
Arguments: `{}`,
|
||||
}, PermissionModeAuto, Options{Hooks: dispatcher})
|
||||
if abortErr != nil {
|
||||
t.Fatalf("executeToolCall abort error: %v", abortErr)
|
||||
}
|
||||
if !result.Truncated || len(result.Output) > 80*4 {
|
||||
t.Fatalf("afterTool feedback bypassed output budget: truncated=%t bytes=%d meta=%#v", result.Truncated, len(result.Output), result.Meta)
|
||||
}
|
||||
if result.Meta["output_budget_category"] == "" || result.Meta["output_budget_retained_bytes"] != strconv.Itoa(len(result.Output)) {
|
||||
t.Fatalf("post-hook budget metadata does not describe final output: %#v", result.Meta)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunTraceReflectsPostHookBudget(t *testing.T) {
|
||||
t.Setenv("ZERO_TOOL_OUTPUT_CEILING_TOKENS", "80")
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(propagationOutputTool{output: "tool output"})
|
||||
dispatcher := largeOutputBudgetHookDispatcher()
|
||||
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
|
||||
{
|
||||
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call-hook-trace", ToolName: "propagation_output"},
|
||||
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call-hook-trace", ArgumentsFragment: `{}`},
|
||||
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call-hook-trace"},
|
||||
{Type: zeroruntime.StreamEventDone},
|
||||
},
|
||||
{{Type: zeroruntime.StreamEventText, Content: "done"}, {Type: zeroruntime.StreamEventDone}},
|
||||
}}
|
||||
recorder := trace.NewRecorder("session", "run", "")
|
||||
var toolResults []ToolResult
|
||||
if _, err := Run(context.Background(), "budget hook", provider, Options{
|
||||
Registry: registry,
|
||||
Hooks: dispatcher,
|
||||
Trace: recorder,
|
||||
OnToolResult: func(result ToolResult) { toolResults = append(toolResults, result) },
|
||||
}); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if len(toolResults) != 1 || !toolResults[0].Truncated {
|
||||
t.Fatalf("tool result = %#v, want one truncated post-hook result", toolResults)
|
||||
}
|
||||
events := recorder.Finish().OutputBudgets
|
||||
if len(events) != 1 || !events[0].Truncated || events[0].RetainedBytes != len(toolResults[0].Output) {
|
||||
t.Fatalf("trace does not describe final post-hook output: events=%#v result=%#v", events, toolResults[0])
|
||||
}
|
||||
}
|
||||
@@ -10,12 +10,23 @@ import (
|
||||
// Parallel read-ahead for tool batches. When a turn requests several
|
||||
// independent lookups (read_file + grep + glob is the common shape), executing
|
||||
// them one after another serializes pure I/O waits. A consecutive run of
|
||||
// auto-allowed read-only calls is executed concurrently instead; results are
|
||||
// capability-safe read-only calls is executed concurrently instead; results are
|
||||
// then consumed in the original call order, so guard counters, message
|
||||
// ordering, abort semantics, and the surface's call/result event pairing are
|
||||
// byte-identical to sequential execution. Runs never span a mutating call: a
|
||||
// read that follows a write must observe the write, so eligibility is decided
|
||||
// per consecutive run, not per batch.
|
||||
//
|
||||
// Eligibility uses the PR5 tool-effect contract (tools.CapabilitiesOf):
|
||||
//
|
||||
// Effect == ReadOnly
|
||||
// AND ThreadSafe == true
|
||||
// AND auto-allowed (no interactive permission prompt on the hot path)
|
||||
// AND no resource-key conflict with an earlier call in the same concurrent window
|
||||
//
|
||||
// Unknown, mutators, interactive tools, and non-thread-safe reads stay
|
||||
// sequential. Empty resource keys do not conflict (ThreadSafe is the safety
|
||||
// gate); shared non-empty keys force a batch boundary.
|
||||
|
||||
// maxParallelReadTools bounds concurrent read-only tool executions in a turn.
|
||||
const maxParallelReadTools = 8
|
||||
@@ -28,28 +39,117 @@ type precomputedToolResult struct {
|
||||
}
|
||||
|
||||
// parallelSafeToolCall reports whether call may run concurrently with its
|
||||
// neighbors: the tool must exist, be side-effect-free (SideEffectRead), and be
|
||||
// auto-allowed for these args, so no interactive prompt or workspace mutation
|
||||
// is on the hot path. Loop-intercepted tools (ask_user, request_permissions)
|
||||
// and tool_search (mutates the deferred-tool set) stay sequential.
|
||||
// neighbors under the PR5 capability contract. Loop-intercepted tools
|
||||
// (ask_user, request_permissions) and tool_search (mutates the deferred-tool
|
||||
// set) always stay sequential.
|
||||
func parallelSafeToolCall(registry *tools.Registry, call ToolCall, options Options) bool {
|
||||
switch call.Name {
|
||||
case "ask_user", tools.RequestPermissionsToolName, tools.ToolSearchToolName:
|
||||
return false
|
||||
}
|
||||
tool, found := registry.Get(call.Name)
|
||||
if !found || tool.Safety().SideEffect != tools.SideEffectRead {
|
||||
if !found {
|
||||
return false
|
||||
}
|
||||
args := map[string]any{}
|
||||
if call.Arguments != "" {
|
||||
if err := decodeToolArguments(call.Arguments, &args); err != nil {
|
||||
return false
|
||||
}
|
||||
caps := tools.CapabilitiesOf(tool)
|
||||
// Fail-closed: only audited concurrent-safe pure reads.
|
||||
if caps.Effect != tools.EffectReadOnly || !caps.ThreadSafe {
|
||||
return false
|
||||
}
|
||||
args, ok := decodeCallArgs(call)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
return effectivePermission(tool, args) == tools.PermissionAllow
|
||||
}
|
||||
|
||||
// decodeCallArgs decodes tool call JSON arguments. Returns false on malformed
|
||||
// input so the call stays sequential (never panics into the parallel path).
|
||||
func decodeCallArgs(call ToolCall) (map[string]any, bool) {
|
||||
args := map[string]any{}
|
||||
if call.Arguments == "" {
|
||||
return args, true
|
||||
}
|
||||
if err := decodeToolArguments(call.Arguments, &args); err != nil {
|
||||
return nil, false
|
||||
}
|
||||
return args, true
|
||||
}
|
||||
|
||||
// resourceKeysForCall returns the conflict keys for a call, or nil when the
|
||||
// tool has no ResourceKeys function / no keys for these args.
|
||||
func resourceKeysForCall(registry *tools.Registry, call ToolCall) []string {
|
||||
tool, found := registry.Get(call.Name)
|
||||
if !found {
|
||||
return nil
|
||||
}
|
||||
caps := tools.CapabilitiesOf(tool)
|
||||
if caps.ResourceKeys == nil {
|
||||
return nil
|
||||
}
|
||||
args, ok := decodeCallArgs(call)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
return caps.ResourceKeys(args)
|
||||
}
|
||||
|
||||
// resourceKeysConflict reports whether two key sets share any non-empty key.
|
||||
// Empty key sets never conflict: ThreadSafe is the eligibility gate; keys only
|
||||
// refine conflict detection among tools that declare specific resources.
|
||||
func resourceKeysConflict(a, b []string) bool {
|
||||
if len(a) == 0 || len(b) == 0 {
|
||||
return false
|
||||
}
|
||||
seen := make(map[string]struct{}, len(a))
|
||||
for _, key := range a {
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
}
|
||||
for _, key := range b {
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[key]; ok {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// extendParallelRun returns the exclusive end index of a consecutive parallel-
|
||||
// safe run starting at start. The run stops before the first call that is not
|
||||
// parallel-safe or that conflicts on resource keys with any earlier call in
|
||||
// the same window (so two read_file calls on the same path stay sequential).
|
||||
func extendParallelRun(registry *tools.Registry, calls []ToolCall, start int, options Options) int {
|
||||
if start >= len(calls) || !parallelSafeToolCall(registry, calls[start], options) {
|
||||
return start
|
||||
}
|
||||
end := start + 1
|
||||
keysWindow := [][]string{resourceKeysForCall(registry, calls[start])}
|
||||
for end < len(calls) {
|
||||
if !parallelSafeToolCall(registry, calls[end], options) {
|
||||
break
|
||||
}
|
||||
nextKeys := resourceKeysForCall(registry, calls[end])
|
||||
conflict := false
|
||||
for _, prev := range keysWindow {
|
||||
if resourceKeysConflict(prev, nextKeys) {
|
||||
conflict = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if conflict {
|
||||
break
|
||||
}
|
||||
keysWindow = append(keysWindow, nextKeys)
|
||||
end++
|
||||
}
|
||||
return end
|
||||
}
|
||||
|
||||
// executeParallelReadBatch runs calls[start:end] concurrently (bounded by
|
||||
// maxParallelReadTools) and returns results indexed relative to start. All
|
||||
// execution-side callbacks that can fire inside executeToolCall are serialized
|
||||
|
||||
@@ -57,6 +57,20 @@ func (tool *probeTool) Parameters() tools.Schema {
|
||||
func (tool *probeTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tool.sideEffect, Permission: tools.PermissionAllow, Reason: "test"}
|
||||
}
|
||||
|
||||
// Capabilities maps the probe's SideEffect into the PR5 capability contract so
|
||||
// parallelSafeToolCall (CapabilitiesOf) can classify reads vs mutators. Without
|
||||
// this, probes default to EffectUnknown and never enter the concurrent path.
|
||||
func (tool *probeTool) Capabilities() tools.ToolCapabilities {
|
||||
switch tool.sideEffect {
|
||||
case tools.SideEffectRead:
|
||||
return tools.ToolCapabilities{Effect: tools.EffectReadOnly, ThreadSafe: true}
|
||||
case tools.SideEffectWrite:
|
||||
return tools.ToolCapabilities{Effect: tools.EffectWorkspaceWrite, ThreadSafe: false}
|
||||
default:
|
||||
return tools.UnknownCapabilities()
|
||||
}
|
||||
}
|
||||
func (tool *probeTool) Run(_ context.Context, args map[string]any) tools.Result {
|
||||
id, _ := args["id"].(string)
|
||||
tool.mu.Lock()
|
||||
@@ -92,6 +106,11 @@ func TestParallelSafeToolCall(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(&probeTool{name: "probe_read", sideEffect: tools.SideEffectRead})
|
||||
registry.Register(&probeTool{name: "probe_write", sideEffect: tools.SideEffectWrite})
|
||||
// Read-only but not ThreadSafe: must stay sequential under the PR5 gate.
|
||||
registry.Register(&keyedProbeTool{
|
||||
probeTool: probeTool{name: "probe_read_serial", sideEffect: tools.SideEffectRead},
|
||||
threadSafe: false,
|
||||
})
|
||||
|
||||
call := func(name, args string) ToolCall { return ToolCall{ID: "c", Name: name, Arguments: args} }
|
||||
if !parallelSafeToolCall(registry, call("probe_read", `{"id":"a"}`), Options{}) {
|
||||
@@ -109,6 +128,91 @@ func TestParallelSafeToolCall(t *testing.T) {
|
||||
if parallelSafeToolCall(registry, call("ask_user", `{}`), Options{}) {
|
||||
t.Fatal("loop-intercepted tools must stay sequential")
|
||||
}
|
||||
if parallelSafeToolCall(registry, call("probe_read_serial", `{"id":"a"}`), Options{}) {
|
||||
t.Fatal("ReadOnly without ThreadSafe must not be parallel-safe")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourceKeysConflict(t *testing.T) {
|
||||
if resourceKeysConflict(nil, []string{"file:a"}) {
|
||||
t.Fatal("empty vs non-empty must not conflict")
|
||||
}
|
||||
if resourceKeysConflict([]string{"file:a"}, nil) {
|
||||
t.Fatal("non-empty vs empty must not conflict")
|
||||
}
|
||||
if resourceKeysConflict([]string{"file:a"}, []string{"file:b"}) {
|
||||
t.Fatal("distinct keys must not conflict")
|
||||
}
|
||||
if !resourceKeysConflict([]string{"file:a", "file:b"}, []string{"file:b"}) {
|
||||
t.Fatal("shared key must conflict")
|
||||
}
|
||||
if resourceKeysConflict([]string{"", "file:a"}, []string{""}) {
|
||||
t.Fatal("empty-string keys must be ignored")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtendParallelRunResourceKeyBoundary(t *testing.T) {
|
||||
// Two keyed reads on the same path must not share a concurrent window;
|
||||
// distinct paths may batch together.
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(&keyedProbeTool{
|
||||
probeTool: probeTool{name: "keyed_read", sideEffect: tools.SideEffectRead},
|
||||
threadSafe: true,
|
||||
keys: func(args map[string]any) []string {
|
||||
id, _ := args["id"].(string)
|
||||
if id == "" {
|
||||
return nil
|
||||
}
|
||||
return []string{"file:" + id}
|
||||
},
|
||||
})
|
||||
calls := []ToolCall{
|
||||
{ID: "1", Name: "keyed_read", Arguments: `{"id":"a"}`},
|
||||
{ID: "2", Name: "keyed_read", Arguments: `{"id":"b"}`},
|
||||
{ID: "3", Name: "keyed_read", Arguments: `{"id":"a"}`}, // conflicts with call 0
|
||||
}
|
||||
// start=0: a and b share no key → run covers [0,2)
|
||||
if end := extendParallelRun(registry, calls, 0, Options{}); end != 2 {
|
||||
t.Fatalf("extend from 0 = %d, want 2 (stop before second file:a)", end)
|
||||
}
|
||||
// start=2: single remaining call
|
||||
if end := extendParallelRun(registry, calls, 2, Options{}); end != 3 {
|
||||
t.Fatalf("extend from 2 = %d, want 3", end)
|
||||
}
|
||||
// Empty keys never conflict: three keyless safe reads form one window.
|
||||
registry.Register(&probeTool{name: "keyless_read", sideEffect: tools.SideEffectRead})
|
||||
keyless := []ToolCall{
|
||||
{ID: "1", Name: "keyless_read", Arguments: `{"id":"x"}`},
|
||||
{ID: "2", Name: "keyless_read", Arguments: `{"id":"y"}`},
|
||||
{ID: "3", Name: "keyless_read", Arguments: `{"id":"z"}`},
|
||||
}
|
||||
if end := extendParallelRun(registry, keyless, 0, Options{}); end != 3 {
|
||||
t.Fatalf("keyless extend = %d, want 3", end)
|
||||
}
|
||||
}
|
||||
|
||||
// keyedProbeTool is a probe with explicit ResourceKeys / ThreadSafe for
|
||||
// planner unit tests.
|
||||
type keyedProbeTool struct {
|
||||
probeTool
|
||||
threadSafe bool
|
||||
keys func(args map[string]any) []string
|
||||
}
|
||||
|
||||
func (tool *keyedProbeTool) Capabilities() tools.ToolCapabilities {
|
||||
caps := tools.ToolCapabilities{
|
||||
Effect: tools.EffectReadOnly,
|
||||
ThreadSafe: tool.threadSafe,
|
||||
}
|
||||
if tool.keys != nil {
|
||||
caps.ResourceKeys = tool.keys
|
||||
}
|
||||
// Honor mutator side effects from the embedded probe when used as write.
|
||||
if tool.sideEffect == tools.SideEffectWrite {
|
||||
caps.Effect = tools.EffectWorkspaceWrite
|
||||
caps.ThreadSafe = false
|
||||
}
|
||||
return caps
|
||||
}
|
||||
|
||||
func TestRunExecutesConsecutiveReadsConcurrently(t *testing.T) {
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// promptSubstrings are the seven cacheable sub-components of the prompt a run
|
||||
// sends to a model. Each is the literal content the sub-component would emit
|
||||
// into the system prompt or tool list, hashed so a downstream observer can
|
||||
// detect drift turn-over-turn. The substrings are produced by
|
||||
// buildPromptSubstrings and consumed by ComputePrefixFingerprint.
|
||||
type promptSubstrings struct {
|
||||
baseInstructions string
|
||||
confirmationPolicy string
|
||||
projectContext string
|
||||
skills string
|
||||
tools string
|
||||
schema string
|
||||
}
|
||||
|
||||
// buildPromptSubstrings assembles the seven cacheable sub-components of the
|
||||
// prompt without joining them. The system prompt sections are produced by the
|
||||
// same private builders that buildSystemPrompt uses, so the substrings are
|
||||
// byte-identical to what would have appeared in the joined prompt. The tools
|
||||
// and schema substrings are derived from the partitioned tool list (caller-
|
||||
// provided) so this helper does not need to re-run the partition.
|
||||
//
|
||||
// This is the seam ComputePrefixFingerprint reads from. It exists as a
|
||||
// separate function (rather than returning both prompt and substrings from
|
||||
// buildSystemPrompt) so existing callers of buildSystemPrompt are unaffected
|
||||
// and the substrings helper is independently testable.
|
||||
func buildPromptSubstrings(options Options, exposed []zeroruntime.ToolDefinition) promptSubstrings {
|
||||
core := strings.TrimSpace(options.SystemPrompt)
|
||||
if core == "" {
|
||||
core = strings.TrimSpace(coreSystemPrompt)
|
||||
}
|
||||
if core == "" {
|
||||
core = fallbackSystemPrompt
|
||||
}
|
||||
|
||||
// The confirmation policy is appended unconditionally in buildSystemPrompt;
|
||||
// the substring here is the same constant.
|
||||
policy := strings.TrimSpace(confirmationPolicy)
|
||||
|
||||
// projectContext is the joined output of workspaceContext, which itself
|
||||
// walks the AGENTS.md / ZERO.md / .zero/AGENTS.md chain. Hashing the
|
||||
// joined string captures every guideline file's contribution and the
|
||||
// ordering between them.
|
||||
project := workspaceContext(options.Cwd)
|
||||
|
||||
// skills is the joined output of skillsContext, with each skill on its own
|
||||
// line. The substring is the literal text the model would see.
|
||||
skills := skillsContext(options)
|
||||
|
||||
// tools and schema are derived from the partitioned tool list. The list
|
||||
// passed in is already in stable order (partitionToolsCached is the
|
||||
// canonical stable partitioner) so concatenating in slice order is
|
||||
// sufficient — we sort names again defensively in case a future caller
|
||||
// hands us a list from a different partitioner.
|
||||
toolsSubstr, schemaSubstr := toolSubstrings(exposed)
|
||||
|
||||
return promptSubstrings{
|
||||
baseInstructions: core,
|
||||
confirmationPolicy: policy,
|
||||
projectContext: project,
|
||||
skills: skills,
|
||||
tools: toolsSubstr,
|
||||
schema: schemaSubstr,
|
||||
}
|
||||
}
|
||||
|
||||
// toolSubstrings extracts a stable, name-sorted digest of the partitioned tool
|
||||
// list. The tools substring is "name1\nname2\n..." (sorted) so a reordering of
|
||||
// the partition produces a different hash. The schema substring is the
|
||||
// concatenation of each tool's JSON schema in name order, so a schema change
|
||||
// is detected independently of the tool ordering.
|
||||
func toolSubstrings(exposed []zeroruntime.ToolDefinition) (toolsSubstr, schemaSubstr string) {
|
||||
if len(exposed) == 0 {
|
||||
return "", ""
|
||||
}
|
||||
names := make([]string, 0, len(exposed))
|
||||
byName := make(map[string]zeroruntime.ToolDefinition, len(exposed))
|
||||
for _, def := range exposed {
|
||||
names = append(names, def.Name)
|
||||
byName[def.Name] = def
|
||||
}
|
||||
sort.Strings(names)
|
||||
var toolsSB, schemaSB strings.Builder
|
||||
for i, name := range names {
|
||||
if i > 0 {
|
||||
toolsSB.WriteByte('\n')
|
||||
}
|
||||
toolsSB.WriteString(name)
|
||||
def := byName[name]
|
||||
// Parameters is rendered to a canonical JSON string per tool. The
|
||||
// schema-render cache in internal/agent (used by partitionToolsCached)
|
||||
// guarantees the same render is byte-identical across turns for the
|
||||
// same tool, so this substring is a stable hash input.
|
||||
schemaSB.WriteString(name)
|
||||
schemaSB.WriteByte('\n')
|
||||
schemaSB.WriteString(canonicalSchemaString(def.Parameters))
|
||||
schemaSB.WriteByte('\n')
|
||||
}
|
||||
return toolsSB.String(), schemaSB.String()
|
||||
}
|
||||
|
||||
// canonicalSchemaString renders a tool's parameter schema to a stable string.
|
||||
// tools.ToolDef.Parameters is a map[string]any in the common case; Go's
|
||||
// fmt.Sprintf("%v", m) iterates a map in random order, which would produce
|
||||
// a different hash for the same Parameters value across calls and defeat
|
||||
// the fingerprint. encoding/json marshals maps with keys sorted
|
||||
// alphabetically, so json.Marshal is the primary stable render. The
|
||||
// fallback for the rare non-JSON-compatible value (functions, channels,
|
||||
// NaN/Inf floats, cyclic references) is a stable key-sorted stringifier
|
||||
// that walks the value with sorted keys for maps and a leading
|
||||
// "__non_json:" prefix so a future schema change is visible in the trace
|
||||
// (a SchemaHash collision from the same value) rather than a silent hash
|
||||
// drift.
|
||||
//
|
||||
// Note: SchemaHash is a stability signal, not a wire-identity signal. The
|
||||
// bytes json.Marshal produces for a Go map[string]any may differ from
|
||||
// the bytes the provider actually sends on the wire (provider encoders
|
||||
// use their own JSON conventions; an Anthropic schema may serialize with
|
||||
// different whitespace, key order, or number formatting). A consumer
|
||||
// must NOT assume SchemaHash matches the provider's wire schema — only
|
||||
// that two turns with the same Go-side Parameters produce the same
|
||||
// SchemaHash. This is sufficient for the trace's purpose (drift
|
||||
// detection) but not for a content-based equality check.
|
||||
func canonicalSchemaString(params map[string]any) string {
|
||||
if len(params) == 0 {
|
||||
return ""
|
||||
}
|
||||
data, err := json.Marshal(params)
|
||||
if err != nil {
|
||||
return "__non_json:" + stableStringify(params)
|
||||
}
|
||||
return string(data)
|
||||
}
|
||||
|
||||
// stableStringify renders v to a deterministic string. Maps have their keys
|
||||
// sorted alphabetically; slices are walked in order; primitives use their
|
||||
// natural Go format. The function is recursive but bounded by the size of
|
||||
// the input, so a cyclic reference is not reachable here (the json.Marshal
|
||||
// fallback is hit when json.Marshal itself returns an error, which for the
|
||||
// common case is a non-JSON-compatible value, not a cycle — cycles are
|
||||
// rare in tool schemas and acceptable to mis-render in the fallback path
|
||||
// since the trace's contract is "the hash is stable for the same input,"
|
||||
// not "the fallback is lossless"). Used only when json.Marshal fails.
|
||||
func stableStringify(v any) string {
|
||||
var sb strings.Builder
|
||||
writeStable(&sb, v)
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
func writeStable(sb *strings.Builder, v any) {
|
||||
switch x := v.(type) {
|
||||
case nil:
|
||||
sb.WriteString("null")
|
||||
case bool:
|
||||
if x {
|
||||
sb.WriteString("true")
|
||||
} else {
|
||||
sb.WriteString("false")
|
||||
}
|
||||
case string:
|
||||
sb.WriteByte('"')
|
||||
sb.WriteString(x)
|
||||
sb.WriteByte('"')
|
||||
case float64:
|
||||
// Use %g for compact, deterministic float rendering. NaN and Inf are
|
||||
// not JSON-encodable (which is why we are in the fallback path) and
|
||||
// render as "NaN" / "+Inf" / "-Inf" — distinct, stable strings.
|
||||
fmt.Fprintf(sb, "%g", x)
|
||||
case float32:
|
||||
fmt.Fprintf(sb, "%g", float64(x))
|
||||
case int:
|
||||
fmt.Fprintf(sb, "%d", x)
|
||||
case int64:
|
||||
fmt.Fprintf(sb, "%d", x)
|
||||
case int32:
|
||||
fmt.Fprintf(sb, "%d", x)
|
||||
case uint:
|
||||
fmt.Fprintf(sb, "%d", x)
|
||||
case uint64:
|
||||
fmt.Fprintf(sb, "%d", x)
|
||||
case uint32:
|
||||
fmt.Fprintf(sb, "%d", x)
|
||||
case map[string]any:
|
||||
sb.WriteByte('{')
|
||||
keys := make([]string, 0, len(x))
|
||||
for k := range x {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for i, k := range keys {
|
||||
if i > 0 {
|
||||
sb.WriteByte(',')
|
||||
}
|
||||
sb.WriteByte('"')
|
||||
sb.WriteString(k)
|
||||
sb.WriteString(`":`)
|
||||
writeStable(sb, x[k])
|
||||
}
|
||||
sb.WriteByte('}')
|
||||
case []any:
|
||||
sb.WriteByte('[')
|
||||
for i, item := range x {
|
||||
if i > 0 {
|
||||
sb.WriteByte(',')
|
||||
}
|
||||
writeStable(sb, item)
|
||||
}
|
||||
sb.WriteByte(']')
|
||||
default:
|
||||
// Last-resort: include the type so distinct values produce distinct
|
||||
// strings even if their default format collides. Without the type
|
||||
// tag, fmt.Sprintf("%v", x) for two different types could produce
|
||||
// the same bytes (rare, but the type prefix makes it impossible).
|
||||
fmt.Fprintf(sb, "<%T:%v>", x, x)
|
||||
}
|
||||
}
|
||||
|
||||
// ComputePrefixFingerprint returns a trace.PrefixHash (a 7-field fingerprint
|
||||
// of the prompt prefix) for one turn of a run. The seven sub-hashes are
|
||||
// independent SHA-256s of the corresponding sub-component; the complete-prefix
|
||||
// hash is a SHA-256 of the canonical concatenation of the other six, so any
|
||||
// sub-component drift is observable both individually and in aggregate.
|
||||
//
|
||||
// I/O side effects: buildPromptSubstrings calls workspaceContext, which runs
|
||||
// git (gitBranchForPrompt, FindProjectGitRoot) and reads the AGENTS.md /
|
||||
// ZERO.md / .zero/AGENTS.md chain plus the repo map. The same workspace-
|
||||
// context work is performed separately by buildSystemPrompt a few lines
|
||||
// later in the request-build path, so on every traced turn the file and
|
||||
// git reads happen twice. The duplication is intentional for this PR (a
|
||||
// follow-up will pass substrings through) and is a known perf cost only on
|
||||
// the opt-in trace path.
|
||||
//
|
||||
// A stable CompletePrefixHash across turns means the four captured
|
||||
// sub-components (baseInstructions, confirmationPolicy, projectContext,
|
||||
// skills) are byte-identical. It does NOT rule out drift in the seven
|
||||
// uncaptured sections of buildSystemPrompt (modelPromptAddendum,
|
||||
// sessionRuntimeContext, approvedCommandPrefixContext, workspaceSeedContext,
|
||||
// userGuidelines, specialistDelegationContext, responseStyleContext). For
|
||||
// default Options the four captured substrings are the full prompt; for
|
||||
// non-default Options the seven uncaptured sections may contribute, and a
|
||||
// consumer correlating CompletePrefix stability with cached_input_tokens
|
||||
// must cross-check the model_switches counter (modelPromptAddendum changes
|
||||
// on a model switch) to disambiguate "no drift" from "drift in an
|
||||
// uncaptured section."
|
||||
|
||||
func ComputePrefixFingerprint(options Options, exposed []zeroruntime.ToolDefinition) prefixFingerprint {
|
||||
subs := buildPromptSubstrings(options, exposed)
|
||||
base := sha256hex(subs.baseInstructions)
|
||||
policy := sha256hex(subs.confirmationPolicy)
|
||||
project := sha256hex(subs.projectContext)
|
||||
skills := sha256hex(subs.skills)
|
||||
toolsH := sha256hex(subs.tools)
|
||||
schema := sha256hex(subs.schema)
|
||||
complete := sha256hex(strings.Join([]string{
|
||||
base, policy, project, skills, toolsH, schema,
|
||||
}, "|"))
|
||||
return prefixFingerprint{
|
||||
BaseInstructionsHash: base,
|
||||
ConfirmationPolicyHash: policy,
|
||||
ProjectContextHash: project,
|
||||
SkillsHash: skills,
|
||||
ToolsHash: toolsH,
|
||||
SchemaHash: schema,
|
||||
CompletePrefixHash: complete,
|
||||
}
|
||||
}
|
||||
|
||||
// prefixFingerprint is the agent-side shape of a prompt-prefix fingerprint. It
|
||||
// is converted to a trace.PrefixHash at the loop boundary (see EmitPrefixHash
|
||||
// in loop.go) so the trace package does not need to import this one. The field
|
||||
// names match the trace.PrefixHash JSON tags 1:1.
|
||||
type prefixFingerprint struct {
|
||||
BaseInstructionsHash string
|
||||
ConfirmationPolicyHash string
|
||||
ProjectContextHash string
|
||||
SkillsHash string
|
||||
ToolsHash string
|
||||
SchemaHash string
|
||||
CompletePrefixHash string
|
||||
}
|
||||
|
||||
// sha256hex returns the hex-encoded SHA-256 of s. Empty input produces the
|
||||
// hash of the empty string, which is a constant; callers that want to
|
||||
// distinguish "absent" from "empty" should check s before calling.
|
||||
func sha256hex(s string) string {
|
||||
sum := sha256.Sum256([]byte(s))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// TestComputePrefixFingerprintStableAcrossCalls asserts the headline property
|
||||
// the trace is meant to expose: two ComputePrefixFingerprint calls over the
|
||||
// same Options and the same exposed tool list produce byte-identical hashes.
|
||||
// This is the regression catch for "did anyone introduce non-determinism into
|
||||
// the prompt prefix."
|
||||
func TestComputePrefixFingerprintStableAcrossCalls(t *testing.T) {
|
||||
opts := Options{Cwd: t.TempDir(), SystemPrompt: "core"}
|
||||
exposed := []zeroruntime.ToolDefinition{
|
||||
{Name: "read_file", Parameters: map[string]any{"type": "object"}},
|
||||
{Name: "grep", Parameters: map[string]any{"type": "object"}},
|
||||
}
|
||||
first := ComputePrefixFingerprint(opts, exposed)
|
||||
second := ComputePrefixFingerprint(opts, exposed)
|
||||
if first != second {
|
||||
t.Fatalf("fingerprint must be stable across calls with the same input:\n first=%+v\n second=%+v", first, second)
|
||||
}
|
||||
if first.CompletePrefixHash == "" {
|
||||
t.Fatalf("CompletePrefixHash must be non-empty for a non-trivial prompt")
|
||||
}
|
||||
}
|
||||
|
||||
// TestComputePrefixFingerprintToolsHashIsNameSetSensitive asserts the
|
||||
// name-set sensitivity of the ToolsHash. The test name used to read as
|
||||
// "reorder changes hash" but the assertion is the opposite: the hash is
|
||||
// name-set sensitive, not name-order sensitive (toolSubstrings sorts
|
||||
// names before hashing). The partitioner is expected to produce a stable
|
||||
// order, so a name-order change is not a hash signal — a name-set change
|
||||
// is. The test asserts both: permutation of the same tools produces the
|
||||
// same hash, and adding a tool produces a different hash.
|
||||
func TestComputePrefixFingerprintToolsHashIsNameSetSensitive(t *testing.T) {
|
||||
opts := Options{Cwd: t.TempDir(), SystemPrompt: "core"}
|
||||
a := []zeroruntime.ToolDefinition{
|
||||
{Name: "read_file", Parameters: map[string]any{"schema": "schema-read"}},
|
||||
{Name: "grep", Parameters: map[string]any{"schema": "schema-grep"}},
|
||||
}
|
||||
b := []zeroruntime.ToolDefinition{
|
||||
{Name: "grep", Parameters: map[string]any{"schema": "schema-grep"}},
|
||||
{Name: "read_file", Parameters: map[string]any{"schema": "schema-read"}},
|
||||
}
|
||||
// toolSubstrings sorts names internally, so the ToolsHash is identical
|
||||
// for a permutation of the same tools. We assert that here to document
|
||||
// the contract: name-order independence, name-set sensitivity.
|
||||
fpA := ComputePrefixFingerprint(opts, a)
|
||||
fpB := ComputePrefixFingerprint(opts, b)
|
||||
if fpA.ToolsHash != fpB.ToolsHash {
|
||||
t.Fatalf("ToolsHash must be name-set sensitive, not name-order sensitive: a=%s b=%s", fpA.ToolsHash, fpB.ToolsHash)
|
||||
}
|
||||
// A different tool set must produce a different ToolsHash.
|
||||
c := append([]zeroruntime.ToolDefinition{}, a...)
|
||||
c = append(c, zeroruntime.ToolDefinition{Name: "bash", Parameters: map[string]any{"schema": "schema-bash"}})
|
||||
fpC := ComputePrefixFingerprint(opts, c)
|
||||
if fpA.ToolsHash == fpC.ToolsHash {
|
||||
t.Fatalf("ToolsHash must change when the tool set changes: a=%s c=%s", fpA.ToolsHash, fpC.ToolsHash)
|
||||
}
|
||||
}
|
||||
|
||||
// TestComputePrefixFingerprintSchemaChangeChangesSchemaHash asserts that a
|
||||
// change to a tool's Parameters schema is visible in the SchemaHash. A schema
|
||||
// edit that should be cache-stable (e.g. field reorder, doc tweak) will move
|
||||
// the hash and surface in the trace; a schema edit that legitimately
|
||||
// invalidates the cache also moves the hash, which is the right behavior.
|
||||
func TestComputePrefixFingerprintSchemaChangeChangesSchemaHash(t *testing.T) {
|
||||
opts := Options{Cwd: t.TempDir(), SystemPrompt: "core"}
|
||||
a := []zeroruntime.ToolDefinition{
|
||||
{Name: "read_file", Parameters: map[string]any{"version": "v1"}},
|
||||
}
|
||||
b := []zeroruntime.ToolDefinition{
|
||||
{Name: "read_file", Parameters: map[string]any{"version": "v2"}},
|
||||
}
|
||||
fpA := ComputePrefixFingerprint(opts, a)
|
||||
fpB := ComputePrefixFingerprint(opts, b)
|
||||
if fpA.SchemaHash == fpB.SchemaHash {
|
||||
t.Fatalf("SchemaHash must change when tool Parameters change: a=%s b=%s", fpA.SchemaHash, fpB.SchemaHash)
|
||||
}
|
||||
}
|
||||
|
||||
// TestComputePrefixFingerprintSystemPromptChangeChangesBaseHash asserts the
|
||||
// most common drift path: a system-prompt edit (e.g. an agent.md update, a
|
||||
// model addendum change) must move the base-instructions hash, which moves
|
||||
// the complete-prefix hash, which is what the trace is for.
|
||||
func TestComputePrefixFingerprintSystemPromptChangeChangesBaseHash(t *testing.T) {
|
||||
optsA := Options{Cwd: t.TempDir(), SystemPrompt: "core v1"}
|
||||
optsB := Options{Cwd: t.TempDir(), SystemPrompt: "core v2"}
|
||||
exposed := []zeroruntime.ToolDefinition{{Name: "noop", Parameters: map[string]any{}}}
|
||||
fpA := ComputePrefixFingerprint(optsA, exposed)
|
||||
fpB := ComputePrefixFingerprint(optsB, exposed)
|
||||
if fpA.BaseInstructionsHash == fpB.BaseInstructionsHash {
|
||||
t.Fatalf("BaseInstructionsHash must change when SystemPrompt changes: a=%s b=%s", fpA.BaseInstructionsHash, fpB.BaseInstructionsHash)
|
||||
}
|
||||
if fpA.CompletePrefixHash == fpB.CompletePrefixHash {
|
||||
t.Fatalf("CompletePrefixHash must change when any sub-hash changes: a=%s b=%s", fpA.CompletePrefixHash, fpB.CompletePrefixHash)
|
||||
}
|
||||
}
|
||||
|
||||
// TestComputePrefixFingerprintCompleteIsAggregateOfSubHashes asserts the
|
||||
// canonical-join property: CompletePrefixHash must depend on every sub-hash.
|
||||
// This is the regression catch for "did anyone reorder or drop a sub-hash
|
||||
// from the canonical join without updating CompletePrefixHash."
|
||||
func TestComputePrefixFingerprintCompleteIsAggregateOfSubHashes(t *testing.T) {
|
||||
opts := Options{Cwd: t.TempDir(), SystemPrompt: "core"}
|
||||
exposed := []zeroruntime.ToolDefinition{{Name: "read_file", Parameters: map[string]any{"k": "v"}}}
|
||||
fp := ComputePrefixFingerprint(opts, exposed)
|
||||
// Manually compute what the canonical join should be.
|
||||
expected := sha256hex(strings.Join([]string{
|
||||
fp.BaseInstructionsHash,
|
||||
fp.ConfirmationPolicyHash,
|
||||
fp.ProjectContextHash,
|
||||
fp.SkillsHash,
|
||||
fp.ToolsHash,
|
||||
fp.SchemaHash,
|
||||
}, "|"))
|
||||
if fp.CompletePrefixHash != expected {
|
||||
t.Fatalf("CompletePrefixHash must equal sha256hex of the canonical join of the other six:\n got: %s\n expected: %s", fp.CompletePrefixHash, expected)
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildPromptSubstringsDefaultOptions asserts the invariants of the
|
||||
// substrings helper for default Options: which substrings are non-empty,
|
||||
// which are empty, and that the substring-to-hash round-trip is lossless.
|
||||
func TestBuildPromptSubstringsDefaultOptions(t *testing.T) {
|
||||
opts := Options{
|
||||
Cwd: t.TempDir(),
|
||||
SystemPrompt: "test core",
|
||||
}
|
||||
subs := buildPromptSubstrings(opts, nil)
|
||||
// Invariants the trace depends on (default Options):
|
||||
// 1. baseInstructions substring equals the core system prompt bytes.
|
||||
// 2. confirmationPolicy substring is non-empty (the embedded policy
|
||||
// is always present, post TrimSpace).
|
||||
// 3. skills, tools, and schema substrings are empty (no skills or
|
||||
// tools configured for default Options).
|
||||
if subs.baseInstructions != "test core" {
|
||||
t.Fatalf("baseInstructions substring must equal the core system prompt: got %q want %q", subs.baseInstructions, "test core")
|
||||
}
|
||||
if subs.confirmationPolicy == "" {
|
||||
t.Fatalf("confirmationPolicy substring must be non-empty (the embedded policy is always present)")
|
||||
}
|
||||
if subs.skills != "" {
|
||||
t.Fatalf("skills substring must be empty for default Options: got %q", subs.skills)
|
||||
}
|
||||
// Round-trip: the corresponding fingerprint hashes must equal
|
||||
// sha256hex of the substrings, with no truncation or reformatting
|
||||
// between the two layers.
|
||||
fp := ComputePrefixFingerprint(opts, nil)
|
||||
if fp.BaseInstructionsHash != sha256hex(subs.baseInstructions) {
|
||||
t.Fatalf("BaseInstructionsHash in fingerprint must equal sha256hex of the substring")
|
||||
}
|
||||
if fp.ConfirmationPolicyHash != sha256hex(subs.confirmationPolicy) {
|
||||
t.Fatalf("ConfirmationPolicyHash in fingerprint must equal sha256hex of the substring")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCanonicalSchemaStringStableForMapParameters asserts the headline
|
||||
// property the trace needs: the same map[string]any produces the same
|
||||
// canonical string across calls, even though Go's fmt.Sprintf("%v", m)
|
||||
// iterates a map in random order. json.Marshal sorts map keys
|
||||
// alphabetically, which is the fix.
|
||||
func TestCanonicalSchemaStringStableForMapParameters(t *testing.T) {
|
||||
params := map[string]any{
|
||||
"description": "read a file",
|
||||
"properties": map[string]any{
|
||||
"path": map[string]any{"type": "string"},
|
||||
},
|
||||
"required": []any{"path"},
|
||||
"type": "object",
|
||||
}
|
||||
first := canonicalSchemaString(params)
|
||||
second := canonicalSchemaString(params)
|
||||
if first != second {
|
||||
t.Fatalf("canonicalSchemaString must be stable across calls for the same Parameters:\n first=%s\n second=%s", first, second)
|
||||
}
|
||||
if first == "" {
|
||||
t.Fatal("canonicalSchemaString must produce a non-empty string for a non-empty map")
|
||||
}
|
||||
// Two maps with the same key/value pairs (different declaration
|
||||
// order in source) must produce identical canonical strings. This
|
||||
// is the property that defeats Go's map iteration randomization.
|
||||
reordered := map[string]any{
|
||||
"type": "object",
|
||||
"required": []any{"path"},
|
||||
"properties": map[string]any{
|
||||
"path": map[string]any{"type": "string"},
|
||||
},
|
||||
"description": "read a file",
|
||||
}
|
||||
if canonicalSchemaString(params) != canonicalSchemaString(reordered) {
|
||||
t.Fatal("canonicalSchemaString must produce identical output for maps with the same key/value pairs in different declaration orders")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCanonicalSchemaStringFallbackStableForNonJSONValue exercises the
|
||||
// non-JSON fallback in canonicalSchemaString: when Parameters contains a
|
||||
// value json.Marshal cannot encode (here, a channel), the function must
|
||||
// fall back to a key-sorted stringification that is stable across calls
|
||||
// and across map-iteration order. The previous fallback used
|
||||
// fmt.Sprintf("%v", params), which iterates the map in random order and
|
||||
// produced a different hash for the same value across calls — defeating
|
||||
// the fingerprint. The current fallback is stable by construction.
|
||||
func TestCanonicalSchemaStringFallbackStableForNonJSONValue(t *testing.T) {
|
||||
ch := make(chan int) // channels are not JSON-encodable
|
||||
params := map[string]any{
|
||||
"description": "a tool with a non-JSON value",
|
||||
"channel": ch,
|
||||
"z_first": 1,
|
||||
"a_second": 2,
|
||||
}
|
||||
first := canonicalSchemaString(params)
|
||||
second := canonicalSchemaString(params)
|
||||
if first == "" {
|
||||
t.Fatal("canonicalSchemaString must produce a non-empty string for the fallback path")
|
||||
}
|
||||
if first != second {
|
||||
t.Fatalf("canonicalSchemaString fallback must be stable across calls:\n first=%s\n second=%s", first, second)
|
||||
}
|
||||
// A re-ordered map (same key/value pairs, different declaration order)
|
||||
// must produce the same fallback string. This is the property the
|
||||
// key-sorted stringifier provides that fmt.Sprintf("%v", m) does not.
|
||||
reordered := map[string]any{
|
||||
"a_second": 2,
|
||||
"z_first": 1,
|
||||
"channel": ch,
|
||||
"description": "a tool with a non-JSON value",
|
||||
}
|
||||
if canonicalSchemaString(params) != canonicalSchemaString(reordered) {
|
||||
t.Fatalf("canonicalSchemaString fallback must produce identical output for maps with the same key/value pairs in different declaration orders:\n first=%s\n second=%s", first, canonicalSchemaString(reordered))
|
||||
}
|
||||
// The fallback must include the "__non_json:" prefix so a consumer
|
||||
// can tell the bytes are not a JSON-marshaled schema (a hash collision
|
||||
// between a json.Marshal result and a stableStringify result is
|
||||
// possible in theory; the prefix makes the source observable).
|
||||
if !strings.HasPrefix(first, "__non_json:") {
|
||||
t.Fatalf("canonicalSchemaString fallback must start with the __non_json: prefix, got: %s", first)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// sessionProvider adapts a TurnSession back to the Provider interface so the
|
||||
// loop's existing Provider-shaped call sites (streamWithReconnect, the
|
||||
// compaction summarizer, finalAnswerAfterMaxTurns) route their stream I/O
|
||||
// through the session without any signature change. For the default adapter
|
||||
// this reduces to the wrapped provider's StreamCompletion, so behavior is
|
||||
// byte-identical; an optimized session (PR8) takes effect here transparently.
|
||||
type sessionProvider struct {
|
||||
session zeroruntime.TurnSession
|
||||
}
|
||||
|
||||
func (s sessionProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
||||
return s.session.Stream(ctx, request)
|
||||
}
|
||||
@@ -0,0 +1,367 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/tools"
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// fakeTurnSession wraps an inner provider for streaming while counting the
|
||||
// session lifecycle, so tests can observe exactly how the loop drives it.
|
||||
type fakeTurnSession struct {
|
||||
inner Provider
|
||||
prewarms int
|
||||
streams int
|
||||
closes int
|
||||
closeErr error
|
||||
}
|
||||
|
||||
func (s *fakeTurnSession) Prewarm(context.Context) error { s.prewarms++; return nil }
|
||||
|
||||
func (s *fakeTurnSession) Stream(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
||||
s.streams++
|
||||
return s.inner.StreamCompletion(ctx, request)
|
||||
}
|
||||
|
||||
func (s *fakeTurnSession) Compact(context.Context, zeroruntime.CompletionRequest) ([]zeroruntime.Message, error) {
|
||||
return nil, zeroruntime.ErrCompactionUnsupported
|
||||
}
|
||||
|
||||
func (s *fakeTurnSession) Close() error { s.closes++; return s.closeErr }
|
||||
|
||||
// fakeTurnSessionProvider opens a scripted session (or fails to).
|
||||
type fakeTurnSessionProvider struct {
|
||||
session *fakeTurnSession
|
||||
openErr error
|
||||
opens int
|
||||
}
|
||||
|
||||
func (p *fakeTurnSessionProvider) OpenTurnSession(context.Context) (zeroruntime.TurnSession, error) {
|
||||
p.opens++
|
||||
if p.openErr != nil {
|
||||
return nil, p.openErr
|
||||
}
|
||||
return p.session, nil
|
||||
}
|
||||
|
||||
func (p *fakeTurnSessionProvider) Capabilities() zeroruntime.ProviderCapabilities {
|
||||
return zeroruntime.ProviderCapabilities{}
|
||||
}
|
||||
|
||||
func singleAnswerTurns(answer string) [][]zeroruntime.StreamEvent {
|
||||
return [][]zeroruntime.StreamEvent{{
|
||||
{Type: zeroruntime.StreamEventText, Content: answer},
|
||||
{Type: zeroruntime.StreamEventDone},
|
||||
}}
|
||||
}
|
||||
|
||||
// TestRunOpenTurnSessionFailureSurfaces verifies a failed session open is a
|
||||
// clean run-start error: no panic, no stream ever issued, nothing to close.
|
||||
func TestRunOpenTurnSessionFailureSurfaces(t *testing.T) {
|
||||
positional := &mockProvider{turns: singleAnswerTurns("never reached")}
|
||||
tsp := &fakeTurnSessionProvider{openErr: errors.New("handshake refused")}
|
||||
|
||||
_, err := Run(context.Background(), "go", positional, Options{
|
||||
MaxTurns: 2,
|
||||
TurnSessionProvider: tsp,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("expected an error from a failed session open")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "open turn session") || !strings.Contains(err.Error(), "handshake refused") {
|
||||
t.Fatalf("expected a wrapped open-turn-session error, got %v", err)
|
||||
}
|
||||
if len(positional.requests) != 0 {
|
||||
t.Fatalf("expected no provider request after a failed open, got %d", len(positional.requests))
|
||||
}
|
||||
if tsp.opens != 1 {
|
||||
t.Fatalf("expected exactly one open attempt, got %d", tsp.opens)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunCloseFailureIsSafe verifies a Close error at teardown is swallowed:
|
||||
// the run still returns its normal result.
|
||||
func TestRunCloseFailureIsSafe(t *testing.T) {
|
||||
inner := &mockProvider{turns: singleAnswerTurns("done")}
|
||||
session := &fakeTurnSession{inner: inner, closeErr: errors.New("close blew up")}
|
||||
tsp := &fakeTurnSessionProvider{session: session}
|
||||
|
||||
result, err := Run(context.Background(), "go", inner, Options{
|
||||
MaxTurns: 2,
|
||||
TurnSessionProvider: tsp,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected Close failure to be swallowed, got %v", err)
|
||||
}
|
||||
if result.FinalAnswer != "done" {
|
||||
t.Fatalf("expected normal result despite Close error, got %q", result.FinalAnswer)
|
||||
}
|
||||
if session.closes != 1 {
|
||||
t.Fatalf("expected exactly one Close at teardown, got %d", session.closes)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunStreamsThroughTurnSession verifies that when a TurnSessionProvider is
|
||||
// wired, every model request of the run flows through the session (and the
|
||||
// session is prewarmed once and closed once).
|
||||
func TestRunStreamsThroughTurnSession(t *testing.T) {
|
||||
inner := &mockProvider{turns: singleAnswerTurns("via session")}
|
||||
session := &fakeTurnSession{inner: inner}
|
||||
tsp := &fakeTurnSessionProvider{session: session}
|
||||
|
||||
// The positional provider is a DIFFERENT mock: it must stay untouched,
|
||||
// proving the session (not the positional provider) carried the run.
|
||||
positional := &mockProvider{turns: singleAnswerTurns("wrong path")}
|
||||
|
||||
result, err := Run(context.Background(), "go", positional, Options{
|
||||
MaxTurns: 2,
|
||||
TurnSessionProvider: tsp,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if result.FinalAnswer != "via session" {
|
||||
t.Fatalf("expected the session-backed answer, got %q", result.FinalAnswer)
|
||||
}
|
||||
if len(positional.requests) != 0 {
|
||||
t.Fatalf("expected the positional provider to be bypassed, got %d requests", len(positional.requests))
|
||||
}
|
||||
if session.streams == 0 || session.streams != len(inner.requests) {
|
||||
t.Fatalf("expected every request to flow through the session: streams=%d inner=%d", session.streams, len(inner.requests))
|
||||
}
|
||||
if session.prewarms != 1 {
|
||||
t.Fatalf("expected exactly one prewarm, got %d", session.prewarms)
|
||||
}
|
||||
if session.closes != 1 {
|
||||
t.Fatalf("expected exactly one close, got %d", session.closes)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunDefaultTurnSessionMatchesRawProvider verifies the default (nil
|
||||
// TurnSessionProvider) path is behavior-identical to passing an explicit
|
||||
// default adapter over the same provider: same result, same request count.
|
||||
func TestRunDefaultTurnSessionMatchesRawProvider(t *testing.T) {
|
||||
run := func(explicit bool) (Result, *mockProvider) {
|
||||
provider := &mockProvider{turns: escalateThenAnswerTurns("identical")}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(escalatingTool{target: ""})
|
||||
options := Options{Registry: registry, MaxTurns: 4}
|
||||
if explicit {
|
||||
options.TurnSessionProvider = zeroruntime.NewProviderTurnSessionProvider(provider, zeroruntime.ProviderCapabilities{})
|
||||
}
|
||||
result, err := Run(context.Background(), "go", provider, options)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return result, provider
|
||||
}
|
||||
|
||||
defaultResult, defaultProvider := run(false)
|
||||
explicitResult, explicitProvider := run(true)
|
||||
|
||||
if defaultResult.FinalAnswer != explicitResult.FinalAnswer {
|
||||
t.Fatalf("final answers diverged: default=%q explicit=%q", defaultResult.FinalAnswer, explicitResult.FinalAnswer)
|
||||
}
|
||||
if defaultResult.Turns != explicitResult.Turns {
|
||||
t.Fatalf("turn counts diverged: default=%d explicit=%d", defaultResult.Turns, explicitResult.Turns)
|
||||
}
|
||||
if len(defaultProvider.requests) != len(explicitProvider.requests) {
|
||||
t.Fatalf("request counts diverged: default=%d explicit=%d", len(defaultProvider.requests), len(explicitProvider.requests))
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunModelSwitchClosesSessionAndContinues verifies a mid-run model switch
|
||||
// closes the active session exactly once and streams the rest of the run on
|
||||
// the new provider.
|
||||
func TestRunModelSwitchClosesSessionAndContinues(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(escalatingTool{target: "claude-opus-4.1"})
|
||||
|
||||
firstInner := &mockProvider{turns: escalateThenAnswerTurns("unused")}
|
||||
firstSession := &fakeTurnSession{inner: firstInner}
|
||||
tsp := &fakeTurnSessionProvider{session: firstSession}
|
||||
|
||||
secondProvider := &mockProvider{turns: singleAnswerTurns("answered after switch")}
|
||||
|
||||
result, err := Run(context.Background(), "go", firstInner, Options{
|
||||
Registry: registry,
|
||||
Model: "claude-sonnet-4.5",
|
||||
MaxTurns: 4,
|
||||
TurnSessionProvider: tsp,
|
||||
ModelSwitcher: func(_ context.Context, _ string) (Provider, error) {
|
||||
return secondProvider, nil
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if result.FinalAnswer != "answered after switch" {
|
||||
t.Fatalf("expected the post-switch answer, got %q", result.FinalAnswer)
|
||||
}
|
||||
// The original session handled exactly the escalation turn, then was closed
|
||||
// once by the swap (the teardown defer closes the NEW session, not this one).
|
||||
if firstSession.streams != 1 {
|
||||
t.Fatalf("expected the original session to stream exactly the escalation turn, got %d", firstSession.streams)
|
||||
}
|
||||
if firstSession.closes != 1 {
|
||||
t.Fatalf("expected the swap to close the original session exactly once, got %d", firstSession.closes)
|
||||
}
|
||||
if len(secondProvider.requests) != 1 {
|
||||
t.Fatalf("expected the post-switch turn on the new provider, got %d requests", len(secondProvider.requests))
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunModelSessionSwitcherCarriesOptimizedSession verifies the target-aware
|
||||
// switcher: the switched-to TurnSessionProvider's DISTINCT session receives the
|
||||
// post-switch streams, is prewarmed exactly once, and is closed exactly once at
|
||||
// teardown — and the legacy ModelSwitcher is never consulted when both are set.
|
||||
func TestRunModelSessionSwitcherCarriesOptimizedSession(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(escalatingTool{target: "claude-opus-4.1"})
|
||||
|
||||
firstInner := &mockProvider{turns: escalateThenAnswerTurns("unused")}
|
||||
firstSession := &fakeTurnSession{inner: firstInner}
|
||||
initial := &fakeTurnSessionProvider{session: firstSession}
|
||||
|
||||
switchedInner := &mockProvider{turns: singleAnswerTurns("answered on the switched session")}
|
||||
switchedSession := &fakeTurnSession{inner: switchedInner}
|
||||
switched := &fakeTurnSessionProvider{session: switchedSession}
|
||||
|
||||
var switchedTo string
|
||||
legacyCalled := false
|
||||
result, err := Run(context.Background(), "go", firstInner, Options{
|
||||
Registry: registry,
|
||||
Model: "claude-sonnet-4.5",
|
||||
MaxTurns: 4,
|
||||
TurnSessionProvider: initial,
|
||||
ModelSessionSwitcher: func(_ context.Context, modelID string) (zeroruntime.TurnSessionProvider, error) {
|
||||
switchedTo = modelID
|
||||
return switched, nil
|
||||
},
|
||||
ModelSwitcher: func(_ context.Context, _ string) (Provider, error) {
|
||||
legacyCalled = true
|
||||
return nil, errors.New("legacy switcher must not be consulted")
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if result.FinalAnswer != "answered on the switched session" {
|
||||
t.Fatalf("expected the switched session's answer, got %q", result.FinalAnswer)
|
||||
}
|
||||
if switchedTo != "claude-opus-4.1" {
|
||||
t.Fatalf("expected switch to claude-opus-4.1, got %q", switchedTo)
|
||||
}
|
||||
if legacyCalled {
|
||||
t.Fatal("legacy ModelSwitcher was consulted despite ModelSessionSwitcher being set")
|
||||
}
|
||||
if switched.opens != 1 {
|
||||
t.Fatalf("expected exactly one open of the switched provider, got %d", switched.opens)
|
||||
}
|
||||
// The switched session carries the rest of the run: it streams the answer
|
||||
// turn, is prewarmed exactly once at the swap, and is closed exactly once by
|
||||
// the teardown defer.
|
||||
if switchedSession.streams != 1 || len(switchedInner.requests) != 1 {
|
||||
t.Fatalf("expected the switched session to stream the post-switch turn, streams=%d inner=%d", switchedSession.streams, len(switchedInner.requests))
|
||||
}
|
||||
if switchedSession.prewarms != 1 {
|
||||
t.Fatalf("expected the switched session to be prewarmed exactly once, got %d", switchedSession.prewarms)
|
||||
}
|
||||
if switchedSession.closes != 1 {
|
||||
t.Fatalf("expected the switched session to be closed exactly once at teardown, got %d", switchedSession.closes)
|
||||
}
|
||||
// The original session ends with the swap: one stream, one close.
|
||||
if firstSession.streams != 1 || firstSession.closes != 1 {
|
||||
t.Fatalf("expected the original session to end at the swap (streams=%d closes=%d)", firstSession.streams, firstSession.closes)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunModelSessionSwitcherOpenFailureIsNonFatal verifies the swap-time open
|
||||
// failure: the switcher returns a TurnSessionProvider whose OpenTurnSession
|
||||
// fails. The run stays non-fatal on the ORIGINAL session (closed only at
|
||||
// teardown), and the switch-failure note reaches the next request.
|
||||
func TestRunModelSessionSwitcherOpenFailureIsNonFatal(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(escalatingTool{target: "claude-opus-4.1"})
|
||||
|
||||
inner := &mockProvider{turns: escalateThenAnswerTurns("recovered after failed open")}
|
||||
session := &fakeTurnSession{inner: inner}
|
||||
failingOpen := &fakeTurnSessionProvider{openErr: errors.New("switched handshake refused")}
|
||||
|
||||
result, err := Run(context.Background(), "go", inner, Options{
|
||||
Registry: registry,
|
||||
Model: "claude-sonnet-4.5",
|
||||
MaxTurns: 4,
|
||||
TurnSessionProvider: &fakeTurnSessionProvider{session: session},
|
||||
ModelSessionSwitcher: func(_ context.Context, _ string) (zeroruntime.TurnSessionProvider, error) {
|
||||
return failingOpen, nil
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected a failed swap open to be non-fatal, got %v", err)
|
||||
}
|
||||
if result.FinalAnswer != "recovered after failed open" {
|
||||
t.Fatalf("expected the run to recover on the original session, got %q", result.FinalAnswer)
|
||||
}
|
||||
if failingOpen.opens != 1 {
|
||||
t.Fatalf("expected exactly one open attempt on the switched provider, got %d", failingOpen.opens)
|
||||
}
|
||||
// The original session serves both turns and is closed only at teardown —
|
||||
// never by the failed swap.
|
||||
if session.streams != 2 || session.closes != 1 {
|
||||
t.Fatalf("expected the original session to serve both turns and close once (streams=%d closes=%d)", session.streams, session.closes)
|
||||
}
|
||||
var sawNote bool
|
||||
for _, message := range inner.requests[1].Messages {
|
||||
if message.Role == zeroruntime.MessageRoleUser && strings.Contains(strings.ToLower(message.Content), "could not switch") {
|
||||
sawNote = true
|
||||
}
|
||||
}
|
||||
if !sawNote {
|
||||
t.Fatalf("expected a switch-failure note on the next turn, messages: %+v", inner.requests[1].Messages)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunModelSessionSwitcherErrorIsNonFatal verifies a ModelSessionSwitcher
|
||||
// error keeps the run on the current session with a transcript note — the same
|
||||
// non-fatal contract as ModelSwitcher.
|
||||
func TestRunModelSessionSwitcherErrorIsNonFatal(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(escalatingTool{target: "claude-opus-4.1"})
|
||||
|
||||
inner := &mockProvider{turns: escalateThenAnswerTurns("recovered on original session")}
|
||||
session := &fakeTurnSession{inner: inner}
|
||||
|
||||
result, err := Run(context.Background(), "go", inner, Options{
|
||||
Registry: registry,
|
||||
Model: "claude-sonnet-4.5",
|
||||
MaxTurns: 4,
|
||||
TurnSessionProvider: &fakeTurnSessionProvider{session: session},
|
||||
ModelSessionSwitcher: func(_ context.Context, _ string) (zeroruntime.TurnSessionProvider, error) {
|
||||
return nil, errors.New("session build blew up")
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected session-switcher error to be non-fatal, got %v", err)
|
||||
}
|
||||
if result.FinalAnswer != "recovered on original session" {
|
||||
t.Fatalf("expected the run to recover on the original session, got %q", result.FinalAnswer)
|
||||
}
|
||||
// Both turns stream through the original session; it closes once at teardown.
|
||||
if session.streams != 2 || session.closes != 1 {
|
||||
t.Fatalf("expected the original session to serve both turns and close once (streams=%d closes=%d)", session.streams, session.closes)
|
||||
}
|
||||
var sawNote bool
|
||||
for _, request := range inner.requests[1].Messages {
|
||||
if request.Role == zeroruntime.MessageRoleUser && strings.Contains(strings.ToLower(request.Content), "could not switch") {
|
||||
sawNote = true
|
||||
}
|
||||
}
|
||||
if !sawNote {
|
||||
t.Fatalf("expected a non-fatal switch-failure note on the next turn, messages: %+v", inner.requests[1].Messages)
|
||||
}
|
||||
}
|
||||
+27
-6
@@ -61,10 +61,13 @@ const (
|
||||
)
|
||||
|
||||
type ToolResult struct {
|
||||
ToolCallID string
|
||||
Name string
|
||||
Status tools.Status
|
||||
Output string
|
||||
ToolCallID string
|
||||
Name string
|
||||
Status tools.Status
|
||||
Output string
|
||||
// Truncated reports that the tool's model-visible output omitted content.
|
||||
// The full result may be recoverable through Meta["spill_path"].
|
||||
Truncated bool
|
||||
Meta map[string]string
|
||||
Redacted bool
|
||||
ChangedFiles []string
|
||||
@@ -280,6 +283,21 @@ type Options struct {
|
||||
// request), so every existing caller is unaffected. A returned error is
|
||||
// non-fatal: the run continues on the current model.
|
||||
ModelSwitcher func(ctx context.Context, modelID string) (Provider, error)
|
||||
// TurnSessionProvider, when set, supplies the turn session the run streams
|
||||
// through — the seam an optimized provider session (connection reuse,
|
||||
// prewarm, native compaction) plugs into without touching the loop. nil
|
||||
// keeps the default: the loop wraps the passed provider in a no-op session
|
||||
// whose Stream IS provider.StreamCompletion, so behavior is byte-identical
|
||||
// and every existing caller is unaffected.
|
||||
TurnSessionProvider zeroruntime.TurnSessionProvider
|
||||
// ModelSessionSwitcher, when set, is the target-aware escalation hook: the
|
||||
// loop prefers it over ModelSwitcher, and its TurnSessionProvider keeps an
|
||||
// optimized session (and its capabilities) across a mid-run model switch.
|
||||
// nil falls back to ModelSwitcher, whose bare Provider is wrapped in the
|
||||
// default no-op session — today's behavior, unchanged. Same non-fatal error
|
||||
// contract as ModelSwitcher: a returned error records a note and the run
|
||||
// continues on the current model and session.
|
||||
ModelSessionSwitcher func(ctx context.Context, modelID string) (zeroruntime.TurnSessionProvider, error)
|
||||
// Trace, when set, records per-turn timing for the run: the loop stamps
|
||||
// spans (prompt build, generation, tool execution, permission wait,
|
||||
// compaction, provider connect) and counters (model requests, tool calls,
|
||||
@@ -312,8 +330,11 @@ type Options struct {
|
||||
// continuation cue ("…Let me check the config:"). The loop then nudges the
|
||||
// model to continue instead, bounded by maxContinueNudges (and still by
|
||||
// MaxTurns and the run deadline); if the model keeps stalling, the run
|
||||
// finalizes as INCOMPLETE (Result.Incomplete) rather than success. Default
|
||||
// false leaves the loop byte-identical, so the interactive TUI is unaffected.
|
||||
// finalizes as INCOMPLETE (Result.Incomplete) rather than success. When the
|
||||
// run's profile also enables SelfCorrect, an otherwise-complete turn gets one
|
||||
// task-grounded semantic check before success; profiles without SelfCorrect
|
||||
// add no model call. Default false leaves the loop byte-identical, so the
|
||||
// interactive TUI is unaffected.
|
||||
RequireCompletionSignal bool
|
||||
|
||||
runPermissions *permissionRunState
|
||||
|
||||
@@ -0,0 +1,259 @@
|
||||
package aimlapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// PaymentMethod selects how a partner-checkout top-up is paid for.
|
||||
type PaymentMethod string
|
||||
|
||||
const (
|
||||
PaymentMethodCard PaymentMethod = "card"
|
||||
PaymentMethodCrypto PaymentMethod = "crypto"
|
||||
)
|
||||
|
||||
// SessionStatus is the lifecycle state of a partner-checkout session, as reported
|
||||
// by the aimlapi.com backend while polling for payment.
|
||||
type SessionStatus string
|
||||
|
||||
const (
|
||||
SessionStatusPendingAuth SessionStatus = "pending_auth"
|
||||
SessionStatusPendingPayment SessionStatus = "pending_payment"
|
||||
SessionStatusPaid SessionStatus = "paid"
|
||||
SessionStatusExchanging SessionStatus = "exchanging"
|
||||
SessionStatusExchanged SessionStatus = "exchanged"
|
||||
SessionStatusCancelled SessionStatus = "cancelled"
|
||||
SessionStatusExpired SessionStatus = "expired"
|
||||
SessionStatusFailed SessionStatus = "failed"
|
||||
)
|
||||
|
||||
// AuthResult is the bearer token minted by a passwordless sign-in / sign-up.
|
||||
type AuthResult struct {
|
||||
Token string `json:"token"`
|
||||
Exp int64 `json:"exp"`
|
||||
}
|
||||
|
||||
// PartnerCheckoutSession is a partner-attributed top-up session; its SessionToken
|
||||
// addresses it on later pay/exchange/poll calls.
|
||||
type PartnerCheckoutSession struct {
|
||||
ID string `json:"id"`
|
||||
SessionToken string `json:"sessionToken"`
|
||||
PartnerID string `json:"partnerId"`
|
||||
PartnerName *string `json:"partnerName"`
|
||||
UserID *int64 `json:"userId"`
|
||||
AmountUSDMinor *int64 `json:"amountUsdMinor"`
|
||||
Status SessionStatus `json:"status"`
|
||||
IssuedKeyID *string `json:"issuedKeyId"`
|
||||
ReturnURL *string `json:"returnUrl"`
|
||||
}
|
||||
|
||||
// PaymentSession is the hosted payment-provider checkout; PayURL is the page the
|
||||
// user opens to pay.
|
||||
type PaymentSession struct {
|
||||
ProviderSessionID string `json:"providerSessionId"`
|
||||
PayURL string `json:"payUrl"`
|
||||
}
|
||||
|
||||
// PayResult pairs the hosted checkout with the updated partner-checkout session
|
||||
// returned when a top-up payment is initiated.
|
||||
type PayResult struct {
|
||||
Checkout PaymentSession `json:"checkout"`
|
||||
PartnerCheckout PartnerCheckoutSession `json:"partnerCheckout"`
|
||||
}
|
||||
|
||||
// ExchangeResult holds the API key minted when a paid session is exchanged.
|
||||
type ExchangeResult struct {
|
||||
APIKey string `json:"apiKey"`
|
||||
APIKeyID string `json:"apiKeyId"`
|
||||
}
|
||||
|
||||
// TopUpByKeyRequest funds a partner-checkout session with the API key that owns
|
||||
// the account (Path A / env key), instead of an email session.
|
||||
type TopUpByKeyRequest struct {
|
||||
SessionToken string
|
||||
AmountUSDMinor int
|
||||
PaymentSessionID string // idempotency: same id → same checkout, never a second charge
|
||||
AutoTopUp bool // enroll the account in automatic top-up at checkout (card, saved off-session)
|
||||
SuccessURL string
|
||||
CancelURL string
|
||||
}
|
||||
|
||||
// TopUpByKeyResult is the hosted checkout returned by TopUpByKey.
|
||||
type TopUpByKeyResult struct {
|
||||
Checkout PaymentSession `json:"checkout"`
|
||||
}
|
||||
|
||||
// APIError is a non-2xx HTTP response from the aimlapi.com API. Status carries the
|
||||
// code so callers can branch (e.g. 401 = bad key, 5xx = retryable).
|
||||
type APIError struct {
|
||||
Message string
|
||||
Status int
|
||||
Body string
|
||||
}
|
||||
|
||||
// Error renders the method, endpoint, status, and (when present) the response body.
|
||||
func (e APIError) Error() string {
|
||||
if strings.TrimSpace(e.Body) != "" {
|
||||
return fmt.Sprintf("%s: HTTP %d: %s", e.Message, e.Status, strings.TrimSpace(e.Body))
|
||||
}
|
||||
return fmt.Sprintf("%s: HTTP %d", e.Message, e.Status)
|
||||
}
|
||||
|
||||
// Client talks to the aimlapi.com auth, app, and inference APIs for the onboarding
|
||||
// and partner-checkout top-up flows.
|
||||
type Client struct {
|
||||
endpoints Endpoints
|
||||
httpClient *http.Client
|
||||
}
|
||||
|
||||
func NewClient(endpoints Endpoints, httpClient *http.Client) *Client {
|
||||
if httpClient == nil {
|
||||
httpClient = &http.Client{Timeout: 60 * time.Second}
|
||||
}
|
||||
return &Client{endpoints: endpoints, httpClient: httpClient}
|
||||
}
|
||||
|
||||
// CreateSession opens a partner-checkout session attributed to partnerID; the
|
||||
// returned SessionToken addresses it on the pay/exchange/poll calls.
|
||||
func (c *Client) CreateSession(ctx context.Context, partnerID string, partnerName string, returnURL string) (PartnerCheckoutSession, error) {
|
||||
body := map[string]any{"partnerId": partnerID}
|
||||
if strings.TrimSpace(partnerName) != "" {
|
||||
body["partnerName"] = strings.TrimSpace(partnerName)
|
||||
}
|
||||
if strings.TrimSpace(returnURL) != "" {
|
||||
body["returnUrl"] = strings.TrimSpace(returnURL)
|
||||
}
|
||||
var result PartnerCheckoutSession
|
||||
err := c.request(ctx, http.MethodPost, strings.TrimRight(c.endpoints.AppBaseURL, "/")+"/v3/partner-checkout/sessions", "", body, &result)
|
||||
return result, err
|
||||
}
|
||||
|
||||
// GetSession fetches the current state of a partner-checkout session; used to poll
|
||||
// for payment completion.
|
||||
func (c *Client) GetSession(ctx context.Context, sessionToken string) (PartnerCheckoutSession, error) {
|
||||
var result PartnerCheckoutSession
|
||||
err := c.request(ctx, http.MethodGet, strings.TrimRight(c.endpoints.AppBaseURL, "/")+"/v3/partner-checkout/sessions/"+urlPathEscape(sessionToken), "", nil, &result)
|
||||
return result, err
|
||||
}
|
||||
|
||||
// Pay initiates payment for a session and returns the hosted checkout URL.
|
||||
// autoTopUp enrolls the account in automatic top-up at checkout time.
|
||||
func (c *Client) Pay(ctx context.Context, bearer string, sessionToken string, amountUSDMinor int, paymentSessionID string, method PaymentMethod, successURL string, cancelURL string, autoTopUp bool) (PayResult, error) {
|
||||
body := map[string]any{
|
||||
"amountUsdMinor": amountUSDMinor,
|
||||
"paymentSessionId": paymentSessionID,
|
||||
"method": method,
|
||||
}
|
||||
if strings.TrimSpace(successURL) != "" {
|
||||
body["successUrl"] = strings.TrimSpace(successURL)
|
||||
}
|
||||
if strings.TrimSpace(cancelURL) != "" {
|
||||
body["cancelUrl"] = strings.TrimSpace(cancelURL)
|
||||
}
|
||||
// Only sent when enabled: enrolls the account in automatic top-up. The backend
|
||||
// honours this field at checkout time.
|
||||
if autoTopUp {
|
||||
body["autoTopUp"] = true
|
||||
}
|
||||
var result PayResult
|
||||
err := c.request(ctx, http.MethodPost, strings.TrimRight(c.endpoints.AppBaseURL, "/")+"/v3/partner-checkout/sessions/"+urlPathEscape(sessionToken)+"/pay", bearer, body, &result)
|
||||
return result, err
|
||||
}
|
||||
|
||||
// Exchange converts a paid session into a freshly minted API key (new-account flow).
|
||||
func (c *Client) Exchange(ctx context.Context, bearer string, sessionToken string) (ExchangeResult, error) {
|
||||
var result ExchangeResult
|
||||
err := c.request(ctx, http.MethodPost, strings.TrimRight(c.endpoints.AppBaseURL, "/")+"/v3/partner-checkout/sessions/"+urlPathEscape(sessionToken)+"/exchange", bearer, nil, &result)
|
||||
return result, err
|
||||
}
|
||||
|
||||
// TopUpByKey funds a partner-checkout session using the raw API key that owns the
|
||||
// account (Path A / env key), returning the hosted checkout. It binds the session
|
||||
// to the key's account server-side, so no email session is needed and no key is
|
||||
// exchanged. req.PaymentSessionID makes a retry idempotent (same id → same
|
||||
// checkout, never a second charge).
|
||||
func (c *Client) TopUpByKey(ctx context.Context, apiKey string, req TopUpByKeyRequest) (TopUpByKeyResult, error) {
|
||||
body := map[string]any{
|
||||
"sessionToken": req.SessionToken,
|
||||
"amountUsdMinor": req.AmountUSDMinor,
|
||||
"paymentSessionId": req.PaymentSessionID,
|
||||
}
|
||||
if req.AutoTopUp {
|
||||
body["autoTopUp"] = true
|
||||
}
|
||||
if strings.TrimSpace(req.SuccessURL) != "" {
|
||||
body["successUrl"] = strings.TrimSpace(req.SuccessURL)
|
||||
}
|
||||
if strings.TrimSpace(req.CancelURL) != "" {
|
||||
body["cancelUrl"] = strings.TrimSpace(req.CancelURL)
|
||||
}
|
||||
var result TopUpByKeyResult
|
||||
err := c.request(ctx, http.MethodPost, c.billingV2URL("/billing/topup"), apiKey, body, &result)
|
||||
return result, err
|
||||
}
|
||||
|
||||
// billingV2URL builds a v2 billing API URL from the inference base. The top-up
|
||||
// endpoint lives next to the balance endpoint on the API gateway, one version up
|
||||
// (".../v1" → ".../v2/billing/..."), so it re-anchors the version while keeping any
|
||||
// host/prefix an override supplied.
|
||||
func (c *Client) billingV2URL(path string) string {
|
||||
base := strings.TrimRight(strings.TrimSpace(c.endpoints.InferenceBaseURL), "/")
|
||||
base = strings.TrimSuffix(base, "/v1")
|
||||
return strings.TrimRight(base, "/") + "/v2" + path
|
||||
}
|
||||
|
||||
func (c *Client) request(ctx context.Context, method string, endpoint string, bearer string, body any, out any) error {
|
||||
var reader io.Reader
|
||||
if body != nil {
|
||||
data, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
reader = bytes.NewReader(data)
|
||||
}
|
||||
request, err := http.NewRequestWithContext(ctx, method, endpoint, reader)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
request.Header.Set("Accept", "application/json")
|
||||
if body != nil {
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if strings.TrimSpace(bearer) != "" {
|
||||
request.Header.Set("Authorization", "Bearer "+strings.TrimSpace(bearer))
|
||||
}
|
||||
response, err := c.httpClient.Do(request)
|
||||
if err != nil {
|
||||
return fmt.Errorf("network request to %s failed: %w", endpoint, err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
text, readErr := io.ReadAll(io.LimitReader(response.Body, 1<<20))
|
||||
if readErr != nil {
|
||||
return readErr
|
||||
}
|
||||
if response.StatusCode < 200 || response.StatusCode >= 300 {
|
||||
return APIError{Message: method + " " + endpoint, Status: response.StatusCode, Body: string(text)}
|
||||
}
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
if len(bytes.TrimSpace(text)) == 0 {
|
||||
return APIError{Message: method + " " + endpoint + " returned empty body", Status: response.StatusCode}
|
||||
}
|
||||
if err := json.Unmarshal(text, out); err != nil {
|
||||
return APIError{Message: method + " " + endpoint + " returned non-JSON body", Status: response.StatusCode, Body: string(text)}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func urlPathEscape(value string) string {
|
||||
return url.PathEscape(value)
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
package aimlapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Additional client methods backing the interactive TUI onboarding flow. They
|
||||
// target the endpoints shipped in the aimlapi backend:
|
||||
// - GET {inference}/billing/balance (Path A: validate key + balance)
|
||||
// - PATCH{auth}/v1/auth/account (Path B: does the account exist?)
|
||||
// - POST {auth}/v1/auth/sign-in/code(/verify) (Path B existing: passwordless login)
|
||||
// - POST {auth}/v1/auth/account/passwordless (Path B new: email-only signup)
|
||||
// - POST {app}/v1/keys (mint a key once we hold a session)
|
||||
|
||||
// BalanceResult is the wallet balance returned by GetBalance, with the backend's
|
||||
// low-balance flag and threshold.
|
||||
type BalanceResult struct {
|
||||
Balance float64 `json:"balance"`
|
||||
LowBalance bool `json:"lowBalance"`
|
||||
LowBalanceThreshold float64 `json:"lowBalanceThreshold"`
|
||||
}
|
||||
|
||||
// CheckResult reports whether an email already has an account, driving the
|
||||
// sign-in vs sign-up branch of the onboarding flow.
|
||||
type CheckResult struct {
|
||||
// "sign-in" (account exists) or "sign-up" (no account yet).
|
||||
Action string `json:"action"`
|
||||
Provider *string `json:"provider"`
|
||||
}
|
||||
|
||||
// CreatedKey is an API key minted by CreateKey.
|
||||
type CreatedKey struct {
|
||||
Key string `json:"key"`
|
||||
ID string `json:"id"`
|
||||
}
|
||||
|
||||
func (c *Client) authURL(path string) string {
|
||||
return strings.TrimRight(c.endpoints.AuthBaseURL, "/") + path
|
||||
}
|
||||
|
||||
func (c *Client) appURL(path string) string {
|
||||
return strings.TrimRight(c.endpoints.AppBaseURL, "/") + path
|
||||
}
|
||||
|
||||
func (c *Client) inferenceURL(path string) string {
|
||||
return strings.TrimRight(c.endpoints.InferenceBaseURL, "/") + path
|
||||
}
|
||||
|
||||
// GetBalance doubles as key validation: a 401 means the key is invalid, a 2xx
|
||||
// returns the wallet balance. Auth is the raw API key itself.
|
||||
func (c *Client) GetBalance(ctx context.Context, apiKey string) (BalanceResult, error) {
|
||||
var result BalanceResult
|
||||
err := c.request(ctx, http.MethodGet, c.inferenceURL("/billing/balance"), apiKey, nil, &result)
|
||||
return result, err
|
||||
}
|
||||
|
||||
// CheckAccount reports whether an email already has an aimlapi.com account
|
||||
// (Action "sign-in") or not ("sign-up").
|
||||
func (c *Client) CheckAccount(ctx context.Context, email string) (CheckResult, error) {
|
||||
var result CheckResult
|
||||
err := c.request(ctx, http.MethodPatch, c.authURL("/v1/auth/account"), "", map[string]any{"email": email}, &result)
|
||||
return result, err
|
||||
}
|
||||
|
||||
// SendSignInCode emails a 6-digit sign-in code to an existing account.
|
||||
func (c *Client) SendSignInCode(ctx context.Context, email string) error {
|
||||
return c.request(ctx, http.MethodPost, c.authURL("/v1/auth/sign-in/code"), "", map[string]any{"email": email}, nil)
|
||||
}
|
||||
|
||||
// VerifySignInCode exchanges an emailed code for a session bearer token.
|
||||
func (c *Client) VerifySignInCode(ctx context.Context, email string, code string) (AuthResult, error) {
|
||||
var result AuthResult
|
||||
err := c.request(ctx, http.MethodPost, c.authURL("/v1/auth/sign-in/code/verify"), "", map[string]any{"email": email, "code": code}, &result)
|
||||
if err == nil && strings.TrimSpace(result.Token) == "" {
|
||||
err = fmt.Errorf("aimlapi.com did not return an auth token")
|
||||
}
|
||||
return result, err
|
||||
}
|
||||
|
||||
// CreatePasswordlessAccount registers a new email-only account and returns its
|
||||
// session bearer token.
|
||||
func (c *Client) CreatePasswordlessAccount(ctx context.Context, email string) (AuthResult, error) {
|
||||
var result AuthResult
|
||||
err := c.request(ctx, http.MethodPost, c.authURL("/v1/auth/account/passwordless"), "", map[string]any{"email": email}, &result)
|
||||
if err == nil && strings.TrimSpace(result.Token) == "" {
|
||||
err = fmt.Errorf("aimlapi.com did not return an auth token")
|
||||
}
|
||||
return result, err
|
||||
}
|
||||
|
||||
// CreateKey mints a fresh API key for a session-holding user.
|
||||
func (c *Client) CreateKey(ctx context.Context, bearer string, name string) (CreatedKey, error) {
|
||||
body := map[string]any{}
|
||||
if strings.TrimSpace(name) != "" {
|
||||
body["name"] = strings.TrimSpace(name)
|
||||
}
|
||||
var result CreatedKey
|
||||
err := c.request(ctx, http.MethodPost, c.appURL("/v1/keys"), bearer, body, &result)
|
||||
if err == nil && strings.TrimSpace(result.Key) == "" {
|
||||
err = fmt.Errorf("aimlapi.com did not return an API key")
|
||||
}
|
||||
return result, err
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
package aimlapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestCreateKeySendsAuthenticatedRequestAndDecodesKey(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost || r.URL.Path != "/v1/keys" {
|
||||
t.Fatalf("request = %s %s, want POST /v1/keys", r.Method, r.URL.Path)
|
||||
}
|
||||
if got := r.Header.Get("Authorization"); got != "Bearer session-token" {
|
||||
t.Fatalf("Authorization = %q", got)
|
||||
}
|
||||
var body map[string]string
|
||||
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if body["name"] != "Zero CLI" {
|
||||
t.Fatalf("name = %q", body["name"])
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"key":"sk-issued","id":"key-id"}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client := NewClient(Endpoints{AppBaseURL: server.URL}, server.Client())
|
||||
key, err := client.CreateKey(context.Background(), " session-token ", " Zero CLI ")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if key.Key != "sk-issued" || key.ID != "key-id" {
|
||||
t.Fatalf("key = %#v", key)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateKeyRejectsEmptyKeyResponse(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"id":"key-id"}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client := NewClient(Endpoints{AppBaseURL: server.URL}, server.Client())
|
||||
if _, err := client.CreateKey(context.Background(), "session-token", "Zero"); err == nil {
|
||||
t.Fatal("CreateKey accepted a response without an API key")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAuthMethodsRejectEmptyTokenResponses(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"token":""}`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client := NewClient(Endpoints{AuthBaseURL: server.URL}, server.Client())
|
||||
tests := []struct {
|
||||
name string
|
||||
call func() error
|
||||
}{
|
||||
{
|
||||
name: "verify sign-in code",
|
||||
call: func() error {
|
||||
_, err := client.VerifySignInCode(context.Background(), "[email protected]", "123456")
|
||||
return err
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "create passwordless account",
|
||||
call: func() error {
|
||||
_, err := client.CreatePasswordlessAccount(context.Background(), "[email protected]")
|
||||
return err
|
||||
},
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if err := test.call(); err == nil {
|
||||
t.Fatal("accepted a response without an auth token")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientReturnsTypedAPIError(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
http.Error(w, "invalid key", http.StatusUnauthorized)
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client := NewClient(Endpoints{InferenceBaseURL: server.URL}, server.Client())
|
||||
_, err := client.GetBalance(context.Background(), "bad-key")
|
||||
var apiErr APIError
|
||||
if !errors.As(err, &apiErr) {
|
||||
t.Fatalf("error = %T %v, want APIError", err, err)
|
||||
}
|
||||
if apiErr.Status != http.StatusUnauthorized || apiErr.Body != "invalid key\n" {
|
||||
t.Fatalf("APIError = %#v", apiErr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTypedResponseRejectsEmptySuccessBody(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client := NewClient(Endpoints{InferenceBaseURL: server.URL}, server.Client())
|
||||
if _, err := client.GetBalance(context.Background(), "key"); err == nil {
|
||||
t.Fatal("GetBalance accepted an empty successful response")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBodylessOperationAcceptsEmptySuccessBody(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client := NewClient(Endpoints{AuthBaseURL: server.URL}, server.Client())
|
||||
if err := client.SendSignInCode(context.Background(), "[email protected]"); err != nil {
|
||||
t.Fatalf("SendSignInCode() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseAmountUSD(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
value string
|
||||
want int
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "default", want: DefaultAmountUSDMinor},
|
||||
{name: "minimum", value: "20", want: 2000},
|
||||
{name: "cents", value: "25.49", want: 2549},
|
||||
{name: "below minimum", value: "19.99", wantErr: true},
|
||||
{name: "above maximum", value: "10001", wantErr: true},
|
||||
{name: "not a number", value: "twenty", wantErr: true},
|
||||
{name: "nan", value: "NaN", wantErr: true},
|
||||
{name: "infinity", value: "Inf", wantErr: true},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
got, err := ParseAmountUSD(test.value)
|
||||
if (err != nil) != test.wantErr {
|
||||
t.Fatalf("ParseAmountUSD(%q) error = %v", test.value, err)
|
||||
}
|
||||
if got != test.want {
|
||||
t.Fatalf("ParseAmountUSD(%q) = %d, want %d", test.value, got, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package aimlapi
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Endpoints struct {
|
||||
AuthBaseURL string
|
||||
AppBaseURL string
|
||||
InferenceBaseURL string
|
||||
PayBaseURL string
|
||||
// VerificationBaseURL is the aimlapi web-app frontend base. The browser is sent
|
||||
// back here after the partner checkout completes (see BuildPartnerReturnURL), so
|
||||
// it must point at the same environment's front as the one the CLI is talking to
|
||||
// to land the user there. Empty falls back to DefaultReturnURL.
|
||||
VerificationBaseURL string
|
||||
}
|
||||
|
||||
const (
|
||||
DefaultPartnerID = "part_62yQoGYDq4Yqnrj2R1iGrDNJ"
|
||||
DefaultPartnerName = "Gitlawb"
|
||||
PartnerHeaderName = "X-AIMLAPI-Partner-ID"
|
||||
// DefaultReturnURL is the https fallback the browser is sent to after the
|
||||
// checkout / OAuth consent finishes when no frontend base is known. It must be
|
||||
// a real web page the browser can open — NOT a custom scheme like
|
||||
// zero://aimlapi/complete, which fails with "the scheme does not have a
|
||||
// registered handler" because the CLI registers no OS protocol handler. The
|
||||
// CLI learns of success by polling; this URL is purely the browser's landing.
|
||||
DefaultReturnURL = "https://aimlapi.com/app"
|
||||
DefaultModel = "anthropic/claude-sonnet-5"
|
||||
|
||||
MinAmountUSDMinor = 2000
|
||||
MaxAmountUSDMinor = 1000000
|
||||
DefaultAmountUSDMinor = 2500
|
||||
)
|
||||
|
||||
func ResolveEndpoints() Endpoints {
|
||||
return Endpoints{
|
||||
AuthBaseURL: envOrDefault("AIMLAPI_AUTH_URL", "https://auth.aimlapi.com"),
|
||||
AppBaseURL: envOrDefault("AIMLAPI_APP_URL", "https://app.aimlapi.com"),
|
||||
InferenceBaseURL: envOrDefault("AIMLAPI_INFERENCE_URL", "https://api.aimlapi.com/v1"),
|
||||
PayBaseURL: envOrDefault("AIMLAPI_PAY_URL", "https://pay.aimlapi.com"),
|
||||
VerificationBaseURL: envOrDefault("AIMLAPI_VERIFICATION_BASE_URL", "https://aimlapi.com/app"),
|
||||
}
|
||||
}
|
||||
|
||||
// ResolvePartnerID applies the same explicit-option, environment, and default
|
||||
// precedence everywhere partner attribution is used. Keeping checkout creation
|
||||
// and saved inference profiles on one resolver prevents them from attributing
|
||||
// the same user to different partners.
|
||||
func ResolvePartnerID(explicit string) string {
|
||||
if value := strings.TrimSpace(explicit); value != "" {
|
||||
return value
|
||||
}
|
||||
return envOrDefault("AIMLAPI_PARTNER_ID", DefaultPartnerID)
|
||||
}
|
||||
|
||||
// WithResolvedPartnerHeader returns a copy with the effective partner ID set.
|
||||
// Header matching is case-insensitive so an existing spelling is replaced rather
|
||||
// than duplicated. Callers remain responsible for applying catalog attribution
|
||||
// only to the canonical AIMLAPI endpoint.
|
||||
func WithResolvedPartnerHeader(headers map[string]string) map[string]string {
|
||||
resolved := make(map[string]string, len(headers)+1)
|
||||
for key, value := range headers {
|
||||
if strings.EqualFold(strings.TrimSpace(key), PartnerHeaderName) {
|
||||
continue
|
||||
}
|
||||
resolved[key] = value
|
||||
}
|
||||
resolved[PartnerHeaderName] = ResolvePartnerID("")
|
||||
return resolved
|
||||
}
|
||||
|
||||
func BuildPartnerCheckoutReturnURLs(appBaseURL string, sessionToken string) (successURL string, cancelURL string) {
|
||||
base := strings.TrimRight(strings.TrimSpace(appBaseURL), "/")
|
||||
if base == "" {
|
||||
// With no base there is nothing to anchor an absolute URL to. Return empty
|
||||
// so the caller omits the return URLs entirely (client.Pay skips empty ones,
|
||||
// leaving the backend's generic checkout) instead of emitting a broken
|
||||
// relative "/checkout?...".
|
||||
return "", ""
|
||||
}
|
||||
token := url.QueryEscape(sessionToken)
|
||||
return base + "/checkout?checkout=success&partnerCheckout=1&sessionToken=" + token,
|
||||
base + "/checkout?checkout=cancel&partnerCheckout=1&sessionToken=" + token
|
||||
}
|
||||
|
||||
// BuildPartnerReturnURL returns the https page the browser is sent back to once
|
||||
// the partner checkout or OAuth consent completes — the aimlapi web app the user
|
||||
// opened the flow from. frontendBaseURL is normally endpoints.VerificationBaseURL
|
||||
// (the front that hosts the login/consent page), so the return follows the same
|
||||
// environment. AIMLAPI_RETURN_URL overrides it outright; an empty base falls back
|
||||
// to DefaultReturnURL. It is deliberately NOT a custom scheme (zero://…): no
|
||||
// browser can hand a custom scheme off without an OS-registered handler, which a
|
||||
// CLI does not install.
|
||||
func BuildPartnerReturnURL(frontendBaseURL string) string {
|
||||
if override := strings.TrimSpace(os.Getenv("AIMLAPI_RETURN_URL")); override != "" {
|
||||
return override
|
||||
}
|
||||
if base := strings.TrimRight(strings.TrimSpace(frontendBaseURL), "/"); base != "" {
|
||||
return base
|
||||
}
|
||||
return DefaultReturnURL
|
||||
}
|
||||
|
||||
func envOrDefault(name string, fallback string) string {
|
||||
if value := strings.TrimSpace(os.Getenv(name)); value != "" {
|
||||
return value
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
package aimlapi
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestResolveEndpointsDefaults(t *testing.T) {
|
||||
t.Setenv("AIMLAPI_AUTH_URL", "")
|
||||
t.Setenv("AIMLAPI_APP_URL", "")
|
||||
t.Setenv("AIMLAPI_INFERENCE_URL", "")
|
||||
t.Setenv("AIMLAPI_PAY_URL", "")
|
||||
|
||||
endpoints := ResolveEndpoints()
|
||||
if endpoints.AuthBaseURL != "https://auth.aimlapi.com" ||
|
||||
endpoints.AppBaseURL != "https://app.aimlapi.com" ||
|
||||
endpoints.InferenceBaseURL != "https://api.aimlapi.com/v1" ||
|
||||
endpoints.PayBaseURL != "https://pay.aimlapi.com" {
|
||||
t.Fatalf("unexpected endpoints: %#v", endpoints)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvedPartnerHeaderUsesEnvironmentOverrideWithoutDuplicates(t *testing.T) {
|
||||
t.Setenv("AIMLAPI_PARTNER_ID", "part_override")
|
||||
|
||||
headers := WithResolvedPartnerHeader(map[string]string{
|
||||
"x-aimlapi-partner-id": "part_catalog",
|
||||
"X-AIMLAPI-Integration-Repo": "Gitlawb/zero",
|
||||
})
|
||||
|
||||
if got := headers[PartnerHeaderName]; got != "part_override" {
|
||||
t.Fatalf("partner header = %q, want part_override", got)
|
||||
}
|
||||
if _, ok := headers["x-aimlapi-partner-id"]; ok {
|
||||
t.Fatalf("case-insensitive partner header was duplicated: %#v", headers)
|
||||
}
|
||||
if got := headers["X-AIMLAPI-Integration-Repo"]; got != "Gitlawb/zero" {
|
||||
t.Fatalf("unrelated header changed: %#v", headers)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPartnerReturnURL(t *testing.T) {
|
||||
t.Setenv("AIMLAPI_RETURN_URL", "")
|
||||
// The frontend base (the web app the flow was opened from) is returned as-is,
|
||||
// trailing slash trimmed — an https page the browser can actually land on.
|
||||
if got := BuildPartnerReturnURL("https://aimlapi.com/app/"); got != "https://aimlapi.com/app" {
|
||||
t.Fatalf("return URL = %q, want the frontend base", got)
|
||||
}
|
||||
// An empty base falls back to the https default, never a custom scheme.
|
||||
if got := BuildPartnerReturnURL(""); got != DefaultReturnURL {
|
||||
t.Fatalf("empty base return URL = %q, want %q", got, DefaultReturnURL)
|
||||
}
|
||||
|
||||
t.Setenv("AIMLAPI_RETURN_URL", "https://example.test/done")
|
||||
if got := BuildPartnerReturnURL("https://aimlapi.com/app"); got != "https://example.test/done" {
|
||||
t.Fatalf("env override return URL = %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package aimlapi
|
||||
|
||||
// UI strings shared by the first-run and provider-wizard onboarding surfaces.
|
||||
|
||||
const (
|
||||
// Path A — existing API key. Pick-path prompt + option labels.
|
||||
MsgAPIKeyInputPrompt = "Enter your aimlapi.com key."
|
||||
MsgAPIKeyInvalid = "API key is invalid. Please make sure you enter a valid aimlapi.com key."
|
||||
MsgPickPathPrompt = "Do you have aimlapi.com key?"
|
||||
MsgPickPathHaveKey = "I already have aimlapi.com key"
|
||||
MsgPickPathHaveHint = "Proceed to paste the key"
|
||||
MsgPickPathNewUser = "I am a new user"
|
||||
MsgPickPathNewHint = "One click set up"
|
||||
// Path B — email.
|
||||
MsgEnterEmail = "Enter your email.\nTo access aimlapi.com dashboard."
|
||||
MsgEmailInvalid = "Email format is incorrect."
|
||||
MsgAccountActionInvalid = "aimlapi.com returned an unsupported account action. Please try again."
|
||||
// %s = email
|
||||
MsgCodeSent = "We sent a 6-digit code to %s.\nEnter it below to continue."
|
||||
MsgCodeIncorrect = "Code you've entered is incorrect."
|
||||
|
||||
// Balance check.
|
||||
MsgLowBalance = "Your aimlapi.com balance is running low.\nIt is recommended to top up your balance."
|
||||
MsgEverythingRuns = "Everything is ready."
|
||||
// Low-balance choices (spec wording).
|
||||
MsgLowBalanceTopUp = "Sure, let's do that"
|
||||
MsgLowBalanceSkip = "I'll skip topping up the balance for now"
|
||||
|
||||
// Top-up workflow.
|
||||
MsgTopUpPrompt = "Add credits.\nEnter an amount (min $20)."
|
||||
MsgAmountRequired = "Please enter a top-up amount."
|
||||
// %s = checkout URL
|
||||
MsgTopUpBrowserFallback = "If the browser did not open automatically please use this link to top up your account: %s"
|
||||
MsgTopUpFailed = "Top up failed. Please try again."
|
||||
MsgTopUpSuccess = "Top-up successful."
|
||||
|
||||
// Success — key delivered.
|
||||
// %s = amount in dollars (e.g. "25")
|
||||
MsgTopUpAddedFmt = "$%s has been added to your balance."
|
||||
// %s = email
|
||||
MsgSuccessMagicLink = "We've emailed you a magic link to %s. Use it to access your aimlapi.com account and review your usage and balance."
|
||||
)
|
||||
@@ -0,0 +1,26 @@
|
||||
package aimlapi
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestValidEmailRejectsIncompleteDomains(t *testing.T) {
|
||||
tests := []struct {
|
||||
value string
|
||||
valid bool
|
||||
}{
|
||||
{value: "[email protected]", valid: true},
|
||||
{value: "[email protected]", valid: true},
|
||||
{value: "123@123.", valid: false},
|
||||
{value: "123@123", valid: false},
|
||||
{value: "[email protected]", valid: false},
|
||||
{value: "[email protected]", valid: false},
|
||||
{value: "[email protected]", valid: false},
|
||||
{value: "[email protected]", valid: false},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.value, func(t *testing.T) {
|
||||
if got := ValidEmail(test.value); got != test.valid {
|
||||
t.Fatalf("ValidEmail(%q) = %v, want %v", test.value, got, test.valid)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
package aimlapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// StreamTopUpOptions drives a session-based partner-checkout top-up for the
|
||||
// interactive (TUI) onboarding. The caller already holds a user session —
|
||||
// obtained from an email-code sign-in (Path B existing) or a
|
||||
// passwordless new account (Path B sign-up) — so this skips authentication and
|
||||
// runs create-session → pay → wait-for-payment, optionally exchanging the paid
|
||||
// session into a fresh key (new accounts) versus keeping the caller's key.
|
||||
type StreamTopUpOptions struct {
|
||||
SessionToken string // user session bearer (required)
|
||||
AmountUSD string // dollars; parsed via ParseAmountUSD (min $20)
|
||||
// InferenceBaseURL pins key-bound billing and the returned provider profile to
|
||||
// the endpoint that validated the credential. Empty uses ResolveEndpoints.
|
||||
InferenceBaseURL string
|
||||
Method PaymentMethod
|
||||
AutoTopUp bool // enroll the account in automatic top-up at checkout time
|
||||
Exchange bool // exchange the paid session into a new key (new accounts)
|
||||
Model string
|
||||
PartnerID string
|
||||
PartnerName string
|
||||
NoOpen bool
|
||||
|
||||
// ResumeSessionToken, when set, resumes that partner-checkout session instead
|
||||
// of opening a new one, so a retry after an ambiguous failure (a lost Pay/
|
||||
// Exchange response) can never double-charge or strand an already-paid session.
|
||||
ResumeSessionToken string
|
||||
// PaymentSessionID is generated once per amount/auto-top-up intent and reused
|
||||
// on retry so an ambiguous Pay response cannot create a second checkout.
|
||||
PaymentSessionID string
|
||||
|
||||
HTTPClient *http.Client
|
||||
OpenBrowser func(string) error
|
||||
OnStatus func(Status, string)
|
||||
// OnSession reports the partner-checkout session token the moment it exists (a
|
||||
// fresh CreateSession, or the resumed one), so the caller can retain it and
|
||||
// resume on a later failure. It fires with "" when the session is dead and the
|
||||
// retained token should be dropped (a fresh run must start over).
|
||||
OnSession func(sessionToken string)
|
||||
}
|
||||
|
||||
// topupPhase is where StreamTopUp enters its create-checkout → poll → exchange
|
||||
// pipeline. A fresh run starts at phasePay; a resume skips ahead to whatever the
|
||||
// existing session's status makes safe.
|
||||
type topupPhase int
|
||||
|
||||
const (
|
||||
phasePay topupPhase = iota // (re-)open the Stripe checkout, then poll + exchange
|
||||
phasePoll // checkout already open; poll for payment, never re-pay
|
||||
phaseExchange // already paid; exchange only (recovers a lost key)
|
||||
phaseWaitExchange // an exchange claim is in flight; poll, never call Exchange twice
|
||||
)
|
||||
|
||||
// StreamTopUp performs the browser-checkout top-up and reports progress via
|
||||
// OnStatus.
|
||||
func StreamTopUp(ctx context.Context, options StreamTopUpOptions) (ProvisionedKey, error) {
|
||||
endpoints := ResolveEndpoints()
|
||||
if baseURL := strings.TrimSpace(options.InferenceBaseURL); baseURL != "" {
|
||||
endpoints.InferenceBaseURL = baseURL
|
||||
}
|
||||
client := NewClient(endpoints, options.HTTPClient)
|
||||
partnerID := ResolvePartnerID(options.PartnerID)
|
||||
partnerName := firstNonEmpty(options.PartnerName, DefaultPartnerName)
|
||||
method := options.Method
|
||||
if method != PaymentMethodCrypto {
|
||||
method = PaymentMethodCard
|
||||
}
|
||||
amount, err := ParseAmountUSD(options.AmountUSD)
|
||||
if err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
if strings.TrimSpace(options.SessionToken) == "" {
|
||||
return ProvisionedKey{}, fmt.Errorf("a session is required to top up")
|
||||
}
|
||||
if strings.TrimSpace(options.PaymentSessionID) == "" {
|
||||
return ProvisionedKey{}, fmt.Errorf("a payment session id is required to top up")
|
||||
}
|
||||
|
||||
status(options.OnStatus, StatusCreatingSession, "")
|
||||
sessionToken, phase, err := resolveTopupSession(ctx, client, options, partnerID, partnerName, endpoints)
|
||||
if err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
// Retain the live token so an ambiguous failure below can resume this session
|
||||
// rather than open a second checkout on the next attempt.
|
||||
reportSession(options.OnSession, sessionToken)
|
||||
|
||||
if phase <= phasePay {
|
||||
successURL, cancelURL := BuildPartnerCheckoutReturnURLs(endpoints.PayBaseURL, sessionToken)
|
||||
status(options.OnStatus, StatusOpeningCheckout, "")
|
||||
pay, err := client.Pay(ctx, options.SessionToken, sessionToken, amount, options.PaymentSessionID, method, successURL, cancelURL, options.AutoTopUp)
|
||||
if err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
checkoutURL, err := validatedCheckoutURL(pay.Checkout.PayURL)
|
||||
if err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
announceCheckout(options.OnStatus, options.OpenBrowser, options.NoOpen, checkoutURL)
|
||||
}
|
||||
|
||||
paidToken := sessionToken
|
||||
if phase <= phasePoll {
|
||||
status(options.OnStatus, StatusWaitingPayment, "")
|
||||
paid, err := pollUntilPaid(ctx, client, sessionToken, options.OnSession)
|
||||
if err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
paidToken = paid.SessionToken
|
||||
if paid.Status == SessionStatusExchanging {
|
||||
// Another/lost request claimed exchange while we were polling payment.
|
||||
// Carry that fact into the provisioning branch so it follows the claim
|
||||
// instead of issuing a second single-flight Exchange request.
|
||||
phase = phaseWaitExchange
|
||||
}
|
||||
}
|
||||
|
||||
result := ProvisionedKey{
|
||||
BaseURL: endpoints.InferenceBaseURL,
|
||||
Model: firstNonEmpty(options.Model, DefaultModel),
|
||||
}
|
||||
if options.Exchange {
|
||||
status(options.OnStatus, StatusProvisioningKey, "")
|
||||
if phase == phaseWaitExchange {
|
||||
return ProvisionedKey{}, pollUntilExchangeSettled(ctx, client, sessionToken, options.OnSession)
|
||||
}
|
||||
exchange, err := client.Exchange(ctx, options.SessionToken, paidToken)
|
||||
if err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
if strings.TrimSpace(exchange.APIKey) == "" {
|
||||
return ProvisionedKey{}, fmt.Errorf("aimlapi.com did not return an API key")
|
||||
}
|
||||
result.APIKey = exchange.APIKey
|
||||
result.APIKeyID = exchange.APIKeyID
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// resolveTopupSession returns the partner-checkout session token to drive and the
|
||||
// phase to enter. A fresh run opens a new session at phasePay. A resume inspects
|
||||
// the existing session and enters the earliest phase that cannot re-charge:
|
||||
// - pending_auth → phasePay: payment never started, so (re-)Pay is safe.
|
||||
// - pending_payment → phasePoll: a checkout is already open; poll, never re-Pay.
|
||||
// - paid → phaseExchange: exchange the paid session once.
|
||||
// - exchanging → phaseWaitExchange: another/lost request owns the atomic
|
||||
// exchange claim; poll it, never issue a second Exchange request.
|
||||
// - exchanged → terminal: the one-shot raw key is gone; point at dashboard
|
||||
// recovery and keep the token so a re-run repeats that guidance, not a charge.
|
||||
// - cancelled/expired/failed → terminal: drop the dead token so a re-run is fresh.
|
||||
func resolveTopupSession(ctx context.Context, client *Client, options StreamTopUpOptions, partnerID, partnerName string, endpoints Endpoints) (string, topupPhase, error) {
|
||||
resume := strings.TrimSpace(options.ResumeSessionToken)
|
||||
if resume == "" {
|
||||
session, err := client.CreateSession(ctx, partnerID, partnerName, BuildPartnerReturnURL(endpoints.VerificationBaseURL))
|
||||
if err != nil {
|
||||
return "", phasePay, err
|
||||
}
|
||||
return session.SessionToken, phasePay, nil
|
||||
}
|
||||
session, err := client.GetSession(ctx, resume)
|
||||
if err != nil {
|
||||
if isTerminalSessionAPIError(err) {
|
||||
reportSession(options.OnSession, "")
|
||||
}
|
||||
return "", phasePay, err
|
||||
}
|
||||
switch session.Status {
|
||||
case SessionStatusPendingAuth:
|
||||
return resume, phasePay, nil
|
||||
case SessionStatusPendingPayment:
|
||||
return resume, phasePoll, nil
|
||||
case SessionStatusPaid:
|
||||
return resume, phaseExchange, nil
|
||||
case SessionStatusExchanging:
|
||||
return resume, phaseWaitExchange, nil
|
||||
case SessionStatusExchanged:
|
||||
return "", phasePay, fmt.Errorf("session was already exchanged; rotate the key from the aimlapi.com dashboard")
|
||||
default:
|
||||
reportSession(options.OnSession, "") // drop the dead token → next run starts fresh
|
||||
return "", phasePay, fmt.Errorf("payment %s; re-run the top-up to try again", session.Status)
|
||||
}
|
||||
}
|
||||
|
||||
func reportSession(callback func(string), sessionToken string) {
|
||||
if callback != nil {
|
||||
callback(sessionToken)
|
||||
}
|
||||
}
|
||||
|
||||
// announceCheckout opens the hosted checkout in the browser (unless suppressed)
|
||||
// and reports the URL either way, so the TUI can always show the direct link.
|
||||
func announceCheckout(onStatus func(Status, string), open func(string) error, noOpen bool, checkoutURL string) {
|
||||
if noOpen || open == nil {
|
||||
status(onStatus, StatusOpeningCheckout, checkoutURL)
|
||||
return
|
||||
}
|
||||
if err := open(checkoutURL); err != nil {
|
||||
status(onStatus, StatusOpeningCheckout, "Open manually: "+checkoutURL)
|
||||
return
|
||||
}
|
||||
status(onStatus, StatusOpeningCheckout, checkoutURL)
|
||||
}
|
||||
|
||||
func validatedCheckoutURL(value string) (string, error) {
|
||||
checkoutURL := strings.TrimSpace(value)
|
||||
parsed, err := url.Parse(checkoutURL)
|
||||
if err != nil || !parsed.IsAbs() || !strings.EqualFold(parsed.Scheme, "https") || parsed.Hostname() == "" || parsed.User != nil {
|
||||
return "", fmt.Errorf("payment provider did not return a valid HTTPS checkout URL")
|
||||
}
|
||||
return checkoutURL, nil
|
||||
}
|
||||
|
||||
// StreamTopUpByKeyOptions drives a top-up funded with the API key that already
|
||||
// owns the account (Path A pasted key, or AIMLAPI_API_KEY). Unlike StreamTopUp it
|
||||
// needs no email session and never exchanges a key — the caller already holds one,
|
||||
// so it only funds the wallet: create-session → pay-by-key → wait-for-payment.
|
||||
type StreamTopUpByKeyOptions struct {
|
||||
APIKey string // raw key that owns the account (required)
|
||||
AmountUSD string // dollars; parsed via ParseAmountUSD (min $20)
|
||||
InferenceBaseURL string // validated key endpoint; empty uses ResolveEndpoints
|
||||
AutoTopUp bool // enroll the account in automatic top-up at checkout time
|
||||
PartnerID string
|
||||
PartnerName string
|
||||
NoOpen bool
|
||||
|
||||
// ResumeSessionToken resumes an existing partner-checkout session instead of
|
||||
// opening a new one. PaymentSessionID is the idempotency handle carried across
|
||||
// retries so a re-issued pay returns the same checkout, never a second charge.
|
||||
ResumeSessionToken string
|
||||
PaymentSessionID string // required; generate once via NewPaymentSessionID, reuse on retry
|
||||
|
||||
HTTPClient *http.Client
|
||||
OpenBrowser func(string) error
|
||||
OnStatus func(Status, string)
|
||||
OnSession func(sessionToken string)
|
||||
}
|
||||
|
||||
// StreamTopUpByKey funds the account behind APIKey via the hosted checkout and
|
||||
// reports progress through OnStatus. It returns an empty ProvisionedKey: the
|
||||
// pasted/env key is unchanged, only the balance grows.
|
||||
func StreamTopUpByKey(ctx context.Context, options StreamTopUpByKeyOptions) (ProvisionedKey, error) {
|
||||
endpoints := ResolveEndpoints()
|
||||
if baseURL := strings.TrimSpace(options.InferenceBaseURL); baseURL != "" {
|
||||
endpoints.InferenceBaseURL = baseURL
|
||||
}
|
||||
client := NewClient(endpoints, options.HTTPClient)
|
||||
partnerID := ResolvePartnerID(options.PartnerID)
|
||||
partnerName := firstNonEmpty(options.PartnerName, DefaultPartnerName)
|
||||
amount, err := ParseAmountUSD(options.AmountUSD)
|
||||
if err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
if strings.TrimSpace(options.APIKey) == "" {
|
||||
return ProvisionedKey{}, fmt.Errorf("an API key is required to top up")
|
||||
}
|
||||
if strings.TrimSpace(options.PaymentSessionID) == "" {
|
||||
return ProvisionedKey{}, fmt.Errorf("a payment session id is required to top up")
|
||||
}
|
||||
|
||||
status(options.OnStatus, StatusCreatingSession, "")
|
||||
sessionToken, phase, err := resolveByKeySession(ctx, client, options, partnerID, partnerName, endpoints)
|
||||
if err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
reportSession(options.OnSession, sessionToken)
|
||||
|
||||
if phase <= phasePay {
|
||||
successURL, cancelURL := BuildPartnerCheckoutReturnURLs(endpoints.PayBaseURL, sessionToken)
|
||||
status(options.OnStatus, StatusOpeningCheckout, "")
|
||||
pay, err := client.TopUpByKey(ctx, options.APIKey, TopUpByKeyRequest{
|
||||
SessionToken: sessionToken,
|
||||
AmountUSDMinor: amount,
|
||||
PaymentSessionID: options.PaymentSessionID,
|
||||
AutoTopUp: options.AutoTopUp,
|
||||
SuccessURL: successURL,
|
||||
CancelURL: cancelURL,
|
||||
})
|
||||
if err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
checkoutURL, err := validatedCheckoutURL(pay.Checkout.PayURL)
|
||||
if err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
announceCheckout(options.OnStatus, options.OpenBrowser, options.NoOpen, checkoutURL)
|
||||
}
|
||||
|
||||
if phase <= phasePoll {
|
||||
status(options.OnStatus, StatusWaitingPayment, "")
|
||||
if _, err := pollUntilPaid(ctx, client, sessionToken, options.OnSession); err != nil {
|
||||
return ProvisionedKey{}, err
|
||||
}
|
||||
}
|
||||
// No exchange: the account is funded and the caller keeps its existing key.
|
||||
return ProvisionedKey{}, nil
|
||||
}
|
||||
|
||||
// resolveByKeySession mirrors resolveTopupSession for the by-key flow. Because
|
||||
// by-key never exchanges, a paid/exchanging/exchanged session all mean "already
|
||||
// funded, done" (phaseExchange skips straight past pay + poll to the empty result).
|
||||
func resolveByKeySession(ctx context.Context, client *Client, options StreamTopUpByKeyOptions, partnerID, partnerName string, endpoints Endpoints) (string, topupPhase, error) {
|
||||
resume := strings.TrimSpace(options.ResumeSessionToken)
|
||||
if resume == "" {
|
||||
session, err := client.CreateSession(ctx, partnerID, partnerName, BuildPartnerReturnURL(endpoints.VerificationBaseURL))
|
||||
if err != nil {
|
||||
return "", phasePay, err
|
||||
}
|
||||
return session.SessionToken, phasePay, nil
|
||||
}
|
||||
session, err := client.GetSession(ctx, resume)
|
||||
if err != nil {
|
||||
if isTerminalSessionAPIError(err) {
|
||||
reportSession(options.OnSession, "")
|
||||
}
|
||||
return "", phasePay, err
|
||||
}
|
||||
switch session.Status {
|
||||
case SessionStatusPendingAuth:
|
||||
return resume, phasePay, nil
|
||||
case SessionStatusPendingPayment:
|
||||
return resume, phasePoll, nil
|
||||
case SessionStatusPaid, SessionStatusExchanging, SessionStatusExchanged:
|
||||
return resume, phaseExchange, nil
|
||||
default:
|
||||
reportSession(options.OnSession, "") // drop the dead token → next run starts fresh
|
||||
return "", phasePay, fmt.Errorf("payment %s; re-run the top-up to try again", session.Status)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,457 @@
|
||||
package aimlapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestStreamTopUpByKeyFundsWithoutExchange(t *testing.T) {
|
||||
var topupCalls, exchangeCalls int
|
||||
var topupBody map[string]any
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/partner-checkout/sessions"):
|
||||
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_tok", Status: SessionStatusPendingAuth})
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/v2/billing/topup"):
|
||||
topupCalls++
|
||||
_ = json.NewDecoder(r.Body).Decode(&topupBody)
|
||||
_ = json.NewEncoder(w).Encode(TopUpByKeyResult{Checkout: PaymentSession{PayURL: "https://pay/checkout"}})
|
||||
case r.Method == http.MethodGet && strings.Contains(r.URL.Path, "/partner-checkout/sessions/"):
|
||||
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_tok", Status: SessionStatusPaid})
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/exchange"):
|
||||
exchangeCalls++
|
||||
t.Errorf("by-key must never exchange a key")
|
||||
default:
|
||||
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
t.Setenv("AIMLAPI_APP_URL", server.URL)
|
||||
t.Setenv("AIMLAPI_INFERENCE_URL", server.URL+"/v1")
|
||||
|
||||
var opened string
|
||||
result, err := StreamTopUpByKey(context.Background(), StreamTopUpByKeyOptions{
|
||||
APIKey: "sk-user",
|
||||
AmountUSD: "20",
|
||||
AutoTopUp: true,
|
||||
PaymentSessionID: "pay-1",
|
||||
OpenBrowser: func(u string) error { opened = u; return nil },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("StreamTopUpByKey = %v", err)
|
||||
}
|
||||
if result.APIKey != "" {
|
||||
t.Fatalf("by-key must not mint a key, got %q", result.APIKey)
|
||||
}
|
||||
if topupCalls != 1 || exchangeCalls != 0 {
|
||||
t.Fatalf("topup=%d exchange=%d, want 1/0", topupCalls, exchangeCalls)
|
||||
}
|
||||
if opened != "https://pay/checkout" {
|
||||
t.Fatalf("opened checkout = %q", opened)
|
||||
}
|
||||
if topupBody["autoTopUp"] != true || topupBody["paymentSessionId"] != "pay-1" {
|
||||
t.Fatalf("topup body missing autoTopUp/paymentSessionId: %#v", topupBody)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidatedCheckoutURLRequiresAbsoluteHTTPS(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
value string
|
||||
want string
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "https", value: " https://pay.example.test/checkout?id=1 ", want: "https://pay.example.test/checkout?id=1"},
|
||||
{name: "empty", wantErr: true},
|
||||
{name: "relative", value: "/checkout", wantErr: true},
|
||||
{name: "http", value: "http://pay.example.test/checkout", wantErr: true},
|
||||
{name: "file", value: "file:///tmp/checkout", wantErr: true},
|
||||
{name: "custom protocol", value: "wallet://checkout/123", wantErr: true},
|
||||
{name: "userinfo", value: "https://[email protected]/checkout", wantErr: true},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
got, err := validatedCheckoutURL(test.value)
|
||||
if (err != nil) != test.wantErr {
|
||||
t.Fatalf("validatedCheckoutURL(%q) error = %v", test.value, err)
|
||||
}
|
||||
if got != test.want {
|
||||
t.Fatalf("validatedCheckoutURL(%q) = %q, want %q", test.value, got, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamTopUpByKeyPinsValidatedInferenceEndpoint(t *testing.T) {
|
||||
validatedCalls := 0
|
||||
validated := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/partner-checkout/sessions"):
|
||||
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_tok", Status: SessionStatusPendingAuth})
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/v2/billing/topup"):
|
||||
validatedCalls++
|
||||
_ = json.NewEncoder(w).Encode(TopUpByKeyResult{Checkout: PaymentSession{PayURL: "https://pay/checkout"}})
|
||||
case r.Method == http.MethodGet && strings.Contains(r.URL.Path, "/partner-checkout/sessions/"):
|
||||
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_tok", Status: SessionStatusPaid})
|
||||
default:
|
||||
t.Errorf("unexpected validated-endpoint request: %s %s", r.Method, r.URL.Path)
|
||||
}
|
||||
}))
|
||||
defer validated.Close()
|
||||
override := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
t.Errorf("validated key leaked to process override: %s %s", r.Method, r.URL.Path)
|
||||
}))
|
||||
defer override.Close()
|
||||
t.Setenv("AIMLAPI_APP_URL", validated.URL)
|
||||
t.Setenv("AIMLAPI_INFERENCE_URL", override.URL)
|
||||
|
||||
_, err := StreamTopUpByKey(context.Background(), StreamTopUpByKeyOptions{
|
||||
APIKey: "production-key",
|
||||
AmountUSD: "20",
|
||||
InferenceBaseURL: validated.URL,
|
||||
PaymentSessionID: "payment-1",
|
||||
OpenBrowser: func(string) error { return nil },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("StreamTopUpByKey() error = %v", err)
|
||||
}
|
||||
if validatedCalls != 1 {
|
||||
t.Fatalf("validated endpoint top-up calls = %d, want 1", validatedCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamTopUpEmailRetrySendsStablePaymentSessionID(t *testing.T) {
|
||||
getCalls := 0
|
||||
var paymentSessionID string
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodGet && strings.Contains(r.URL.Path, "/partner-checkout/sessions/"):
|
||||
status := SessionStatusPendingAuth
|
||||
if getCalls > 0 {
|
||||
status = SessionStatusPaid
|
||||
}
|
||||
getCalls++
|
||||
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_retry", Status: status})
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/pay"):
|
||||
var body map[string]any
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
paymentSessionID, _ = body["paymentSessionId"].(string)
|
||||
_ = json.NewEncoder(w).Encode(PayResult{Checkout: PaymentSession{PayURL: "https://pay/checkout"}})
|
||||
default:
|
||||
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
t.Setenv("AIMLAPI_APP_URL", server.URL)
|
||||
|
||||
_, err := StreamTopUp(context.Background(), StreamTopUpOptions{
|
||||
SessionToken: "bearer",
|
||||
ResumeSessionToken: "pcs_retry",
|
||||
PaymentSessionID: "payment-stable",
|
||||
AmountUSD: "20",
|
||||
NoOpen: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("StreamTopUp() error = %v", err)
|
||||
}
|
||||
if paymentSessionID != "payment-stable" {
|
||||
t.Fatalf("paymentSessionId = %q, want stable retry id", paymentSessionID)
|
||||
}
|
||||
}
|
||||
|
||||
// resumeServer is a partner-checkout backend fake that fails the test if the
|
||||
// re-charging endpoints (create-session / pay) are ever hit, and serves a
|
||||
// scripted GetSession status sequence so a resume can be driven deterministically.
|
||||
type resumeServer struct {
|
||||
t *testing.T
|
||||
statuses []string // GetSession status per call, last value repeats
|
||||
getCalls int
|
||||
payCalls int
|
||||
createCall int
|
||||
exchanged int
|
||||
}
|
||||
|
||||
func (rs *resumeServer) handler() http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/partner-checkout/sessions"):
|
||||
rs.createCall++
|
||||
rs.t.Errorf("resume must not create a new session")
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/pay"):
|
||||
rs.payCalls++
|
||||
rs.t.Errorf("resume must not re-pay (double charge)")
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/exchange"):
|
||||
rs.exchanged++
|
||||
_ = json.NewEncoder(w).Encode(ExchangeResult{APIKey: "sk-resumed", APIKeyID: "key_1"})
|
||||
case r.Method == http.MethodGet:
|
||||
status := rs.statuses[len(rs.statuses)-1]
|
||||
if rs.getCalls < len(rs.statuses) {
|
||||
status = rs.statuses[rs.getCalls]
|
||||
}
|
||||
rs.getCalls++
|
||||
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pc_tok", Status: SessionStatus(status)})
|
||||
default:
|
||||
rs.t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestStreamTopUpResumePaidExchangesWithoutRecharge(t *testing.T) {
|
||||
rs := &resumeServer{t: t, statuses: []string{"paid"}}
|
||||
server := httptest.NewServer(rs.handler())
|
||||
defer server.Close()
|
||||
t.Setenv("AIMLAPI_APP_URL", server.URL)
|
||||
|
||||
var seen string
|
||||
result, err := StreamTopUp(context.Background(), StreamTopUpOptions{
|
||||
SessionToken: "bearer",
|
||||
ResumeSessionToken: "pc_tok",
|
||||
PaymentSessionID: "payment-1",
|
||||
AmountUSD: "20",
|
||||
Exchange: true,
|
||||
NoOpen: true,
|
||||
OnSession: func(token string) { seen = token },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("StreamTopUp resume = %v", err)
|
||||
}
|
||||
if result.APIKey != "sk-resumed" {
|
||||
t.Fatalf("APIKey = %q, want recovered key", result.APIKey)
|
||||
}
|
||||
if rs.exchanged != 1 {
|
||||
t.Fatalf("exchange calls = %d, want 1", rs.exchanged)
|
||||
}
|
||||
if seen != "pc_tok" {
|
||||
t.Fatalf("OnSession token = %q, want retained pc_tok", seen)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamTopUpResumePendingPaymentPollsNeverRepays(t *testing.T) {
|
||||
// First GetSession (resume resolution) sees the still-open checkout; the next
|
||||
// (poll) sees it settle. Pay must never be called — the fake fails the test if
|
||||
// it is.
|
||||
rs := &resumeServer{t: t, statuses: []string{"pending_payment", "paid"}}
|
||||
server := httptest.NewServer(rs.handler())
|
||||
defer server.Close()
|
||||
t.Setenv("AIMLAPI_APP_URL", server.URL)
|
||||
|
||||
result, err := StreamTopUp(context.Background(), StreamTopUpOptions{
|
||||
SessionToken: "bearer",
|
||||
ResumeSessionToken: "pc_tok",
|
||||
PaymentSessionID: "payment-1",
|
||||
AmountUSD: "20",
|
||||
Exchange: true,
|
||||
NoOpen: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("StreamTopUp resume = %v", err)
|
||||
}
|
||||
if result.APIKey != "sk-resumed" {
|
||||
t.Fatalf("APIKey = %q, want recovered key", result.APIKey)
|
||||
}
|
||||
if rs.payCalls != 0 {
|
||||
t.Fatalf("pay calls = %d, want 0", rs.payCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamTopUpPollingDeadSessionClearsRetainedToken(t *testing.T) {
|
||||
rs := &resumeServer{t: t, statuses: []string{"pending_payment", "expired"}}
|
||||
server := httptest.NewServer(rs.handler())
|
||||
defer server.Close()
|
||||
t.Setenv("AIMLAPI_APP_URL", server.URL)
|
||||
|
||||
seen := []string{}
|
||||
_, err := StreamTopUp(context.Background(), StreamTopUpOptions{
|
||||
SessionToken: "bearer",
|
||||
ResumeSessionToken: "pc_tok",
|
||||
PaymentSessionID: "payment-1",
|
||||
AmountUSD: "20",
|
||||
Exchange: true,
|
||||
NoOpen: true,
|
||||
OnSession: func(token string) { seen = append(seen, token) },
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "payment expired") {
|
||||
t.Fatalf("err = %v, want expired payment error", err)
|
||||
}
|
||||
if len(seen) == 0 || seen[len(seen)-1] != "" {
|
||||
t.Fatalf("OnSession events = %#v, want terminal clear", seen)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPollUntilPaidTerminal4xxClearsRetainedToken(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "session rejected", http.StatusNotFound)
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
seen := "retained"
|
||||
_, err := pollUntilPaid(context.Background(), NewClient(Endpoints{AppBaseURL: server.URL}, server.Client()), "pc_missing", func(token string) {
|
||||
seen = token
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("pollUntilPaid() error = nil, want terminal 4xx")
|
||||
}
|
||||
if seen != "" {
|
||||
t.Fatalf("retained session = %q, want cleared", seen)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPollUntilExchangeSettledClearsDeadSession(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
statusCode int
|
||||
status SessionStatus
|
||||
}{
|
||||
{name: "terminal API rejection", statusCode: http.StatusNotFound},
|
||||
{name: "expired exchange claim", status: SessionStatusExpired},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
if test.statusCode != 0 {
|
||||
http.Error(w, "session rejected", test.statusCode)
|
||||
return
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(PartnerCheckoutSession{SessionToken: "pcs_dead", Status: test.status})
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
seen := "retained"
|
||||
err := pollUntilExchangeSettled(context.Background(), NewClient(Endpoints{AppBaseURL: server.URL}, server.Client()), "pcs_dead", func(token string) {
|
||||
seen = token
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("pollUntilExchangeSettled() error = nil")
|
||||
}
|
||||
if seen != "" {
|
||||
t.Fatalf("retained session = %q, want cleared", seen)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResumeHelpersClearOnlyRejectedSessions(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
status int
|
||||
wantClear bool
|
||||
}{
|
||||
{name: "terminal 4xx", status: http.StatusNotFound, wantClear: true},
|
||||
{name: "retryable 5xx", status: http.StatusServiceUnavailable, wantClear: false},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
http.Error(w, "session lookup failed", test.status)
|
||||
}))
|
||||
defer server.Close()
|
||||
client := NewClient(Endpoints{AppBaseURL: server.URL}, server.Client())
|
||||
|
||||
for _, flow := range []struct {
|
||||
name string
|
||||
resolve func(func(string)) error
|
||||
}{
|
||||
{
|
||||
name: "session bearer",
|
||||
resolve: func(onSession func(string)) error {
|
||||
_, _, err := resolveTopupSession(context.Background(), client, StreamTopUpOptions{
|
||||
ResumeSessionToken: "pc_rejected", OnSession: onSession,
|
||||
}, DefaultPartnerID, DefaultPartnerName, Endpoints{})
|
||||
return err
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "api key",
|
||||
resolve: func(onSession func(string)) error {
|
||||
_, _, err := resolveByKeySession(context.Background(), client, StreamTopUpByKeyOptions{
|
||||
ResumeSessionToken: "pc_rejected", OnSession: onSession,
|
||||
}, DefaultPartnerID, DefaultPartnerName, Endpoints{})
|
||||
return err
|
||||
},
|
||||
},
|
||||
} {
|
||||
t.Run(flow.name, func(t *testing.T) {
|
||||
seen := "retained"
|
||||
if err := flow.resolve(func(token string) { seen = token }); err == nil {
|
||||
t.Fatal("resolve error = nil")
|
||||
}
|
||||
if cleared := seen == ""; cleared != test.wantClear {
|
||||
t.Fatalf("session cleared = %v, want %v", cleared, test.wantClear)
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamTopUpResumeExchangedReturnsRecoveryError(t *testing.T) {
|
||||
rs := &resumeServer{t: t, statuses: []string{"exchanged"}}
|
||||
server := httptest.NewServer(rs.handler())
|
||||
defer server.Close()
|
||||
t.Setenv("AIMLAPI_APP_URL", server.URL)
|
||||
|
||||
_, err := StreamTopUp(context.Background(), StreamTopUpOptions{
|
||||
SessionToken: "bearer",
|
||||
ResumeSessionToken: "pc_tok",
|
||||
PaymentSessionID: "payment-1",
|
||||
AmountUSD: "20",
|
||||
Exchange: true,
|
||||
NoOpen: true,
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "already exchanged") {
|
||||
t.Fatalf("err = %v, want an already-exchanged recovery error", err)
|
||||
}
|
||||
if rs.exchanged != 0 {
|
||||
t.Fatalf("exchange calls = %d, want 0 (no second key)", rs.exchanged)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamTopUpResumeExchangingPollsWithoutSecondExchange(t *testing.T) {
|
||||
rs := &resumeServer{t: t, statuses: []string{"exchanging", "exchanged"}}
|
||||
server := httptest.NewServer(rs.handler())
|
||||
defer server.Close()
|
||||
t.Setenv("AIMLAPI_APP_URL", server.URL)
|
||||
|
||||
_, err := StreamTopUp(context.Background(), StreamTopUpOptions{
|
||||
SessionToken: "bearer",
|
||||
ResumeSessionToken: "pc_tok",
|
||||
PaymentSessionID: "payment-1",
|
||||
AmountUSD: "20",
|
||||
Exchange: true,
|
||||
NoOpen: true,
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "already exchanged") {
|
||||
t.Fatalf("err = %v, want dashboard recovery guidance", err)
|
||||
}
|
||||
if rs.exchanged != 0 {
|
||||
t.Fatalf("exchange calls = %d, want 0 while an exchange claim is in flight", rs.exchanged)
|
||||
}
|
||||
if rs.createCall != 0 || rs.payCalls != 0 {
|
||||
t.Fatalf("resume created/paid a new checkout: create=%d pay=%d", rs.createCall, rs.payCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamTopUpPollObservesExchangingWithoutSecondExchange(t *testing.T) {
|
||||
rs := &resumeServer{t: t, statuses: []string{"pending_payment", "exchanging", "exchanged"}}
|
||||
server := httptest.NewServer(rs.handler())
|
||||
defer server.Close()
|
||||
t.Setenv("AIMLAPI_APP_URL", server.URL)
|
||||
|
||||
_, err := StreamTopUp(context.Background(), StreamTopUpOptions{
|
||||
SessionToken: "bearer",
|
||||
ResumeSessionToken: "pc_tok",
|
||||
PaymentSessionID: "payment-1",
|
||||
AmountUSD: "20",
|
||||
Exchange: true,
|
||||
NoOpen: true,
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "already exchanged") {
|
||||
t.Fatalf("err = %v, want dashboard recovery guidance", err)
|
||||
}
|
||||
if rs.exchanged != 0 {
|
||||
t.Fatalf("exchange calls = %d, want 0 after poll observed an existing claim", rs.exchanged)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
package aimlapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Status is a progress phase reported by StreamTopUp through its OnStatus callback.
|
||||
type Status string
|
||||
|
||||
const (
|
||||
StatusCreatingSession Status = "creating-session"
|
||||
StatusOpeningCheckout Status = "opening-checkout"
|
||||
StatusWaitingPayment Status = "waiting-payment"
|
||||
StatusProvisioningKey Status = "provisioning-key"
|
||||
)
|
||||
|
||||
// ProvisionedKey is the outcome of a top-up: the (optionally minted) API key plus
|
||||
// the inference base URL and model to write into the provider profile.
|
||||
type ProvisionedKey struct {
|
||||
APIKey string
|
||||
APIKeyID string
|
||||
BaseURL string
|
||||
Model string
|
||||
}
|
||||
|
||||
const (
|
||||
pollInterval = 3 * time.Second
|
||||
pollTimeout = 20 * time.Minute
|
||||
)
|
||||
|
||||
// ParseAmountUSD parses a dollar string into USD minor units (cents), enforcing the
|
||||
// min/max top-up bounds. An empty value yields the default amount.
|
||||
func ParseAmountUSD(value string) (int, error) {
|
||||
value = strings.TrimSpace(value)
|
||||
if value == "" {
|
||||
return DefaultAmountUSDMinor, nil
|
||||
}
|
||||
dollars, err := strconv.ParseFloat(value, 64)
|
||||
// NaN/±Inf parse without error and slip past the min/max comparisons below
|
||||
// (every ordered comparison with NaN is false), so reject them explicitly
|
||||
// before the cents conversion turns them into a garbage minor-unit amount.
|
||||
if err != nil || math.IsNaN(dollars) || math.IsInf(dollars, 0) || dollars <= 0 {
|
||||
return 0, fmt.Errorf("invalid amount %q; pass a positive USD amount", value)
|
||||
}
|
||||
minor := int(dollars*100 + 0.5)
|
||||
if minor < MinAmountUSDMinor {
|
||||
return 0, fmt.Errorf("minimum top-up is $%d", MinAmountUSDMinor/100)
|
||||
}
|
||||
if minor > MaxAmountUSDMinor {
|
||||
return 0, fmt.Errorf("maximum top-up is $%d", MaxAmountUSDMinor/100)
|
||||
}
|
||||
return minor, nil
|
||||
}
|
||||
|
||||
// NewPaymentSessionID returns a random UUIDv4-shaped idempotency id for a top-up.
|
||||
// The caller generates it once per top-up intent and reuses it on retry so the
|
||||
// backend (and Stripe) coalesce retries onto the same checkout instead of a
|
||||
// second charge.
|
||||
func NewPaymentSessionID() (string, error) {
|
||||
var b [16]byte
|
||||
if _, err := rand.Read(b[:]); err != nil {
|
||||
return "", err
|
||||
}
|
||||
b[6] = (b[6] & 0x0f) | 0x40 // version 4
|
||||
b[8] = (b[8] & 0x3f) | 0x80 // variant 10
|
||||
return fmt.Sprintf("%x-%x-%x-%x-%x", b[0:4], b[4:6], b[6:8], b[8:10], b[10:16]), nil
|
||||
}
|
||||
|
||||
func pollUntilPaid(ctx context.Context, client *Client, sessionToken string, onSession func(string)) (PartnerCheckoutSession, error) {
|
||||
deadline := time.Now().Add(pollTimeout)
|
||||
for time.Now().Before(deadline) {
|
||||
session, err := client.GetSession(ctx, sessionToken)
|
||||
if err != nil {
|
||||
// Context cancellation/deadline is terminal (also covers an in-flight
|
||||
// request cancelled mid-poll, whose transport error wraps ctx.Err()).
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
return PartnerCheckoutSession{}, err
|
||||
}
|
||||
// A 4xx API response is terminal. Everything else — a 5xx or a transient
|
||||
// transport error (non-APIError, e.g. a dropped connection during the
|
||||
// wait) — is retried until the poll deadline.
|
||||
if isTerminalSessionAPIError(err) {
|
||||
reportSession(onSession, "")
|
||||
return PartnerCheckoutSession{}, err
|
||||
}
|
||||
if err := sleepContext(ctx, pollInterval); err != nil {
|
||||
return PartnerCheckoutSession{}, err
|
||||
}
|
||||
continue
|
||||
}
|
||||
switch session.Status {
|
||||
case SessionStatusPaid, SessionStatusExchanging:
|
||||
return session, nil
|
||||
case SessionStatusExchanged:
|
||||
return PartnerCheckoutSession{}, fmt.Errorf("session was already exchanged; rotate the key from the aimlapi.com dashboard")
|
||||
case SessionStatusCancelled, SessionStatusExpired, SessionStatusFailed:
|
||||
// The checkout cannot recover from these states. Drop the retained token
|
||||
// (and the TUI's paired by-key idempotency ID) now so the very next retry
|
||||
// creates a fresh payment intent.
|
||||
reportSession(onSession, "")
|
||||
return PartnerCheckoutSession{}, fmt.Errorf("payment %s; re-run the top-up to try again", session.Status)
|
||||
default:
|
||||
if err := sleepContext(ctx, pollInterval); err != nil {
|
||||
return PartnerCheckoutSession{}, err
|
||||
}
|
||||
}
|
||||
}
|
||||
return PartnerCheckoutSession{}, fmt.Errorf("timed out waiting for payment; re-run once the payment clears")
|
||||
}
|
||||
|
||||
func isTerminalSessionAPIError(err error) bool {
|
||||
var apiErr APIError
|
||||
return errors.As(err, &apiErr) && apiErr.Status >= http.StatusBadRequest && apiErr.Status < http.StatusInternalServerError
|
||||
}
|
||||
|
||||
// pollUntilExchangeSettled follows an exchange claim that was already in flight
|
||||
// when a retry resumed the session. Exchange is single-flight and the raw key is
|
||||
// returned only to the winning request, so issuing Exchange again is never safe.
|
||||
// Once the claim completes, direct the user to the dashboard recovery path.
|
||||
func pollUntilExchangeSettled(ctx context.Context, client *Client, sessionToken string, onSession func(string)) error {
|
||||
deadline := time.Now().Add(pollTimeout)
|
||||
for time.Now().Before(deadline) {
|
||||
session, err := client.GetSession(ctx, sessionToken)
|
||||
if err != nil {
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
return err
|
||||
}
|
||||
if isTerminalSessionAPIError(err) {
|
||||
reportSession(onSession, "")
|
||||
return err
|
||||
}
|
||||
if err := sleepContext(ctx, pollInterval); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
}
|
||||
switch session.Status {
|
||||
case SessionStatusExchanged:
|
||||
return fmt.Errorf("session was already exchanged; rotate the key from the aimlapi.com dashboard")
|
||||
case SessionStatusExchanging:
|
||||
if err := sleepContext(ctx, pollInterval); err != nil {
|
||||
return err
|
||||
}
|
||||
case SessionStatusCancelled, SessionStatusExpired, SessionStatusFailed:
|
||||
reportSession(onSession, "")
|
||||
return fmt.Errorf("key provisioning %s; rotate the key from the aimlapi.com dashboard", session.Status)
|
||||
default:
|
||||
return fmt.Errorf("key provisioning returned to %s; re-run the top-up", session.Status)
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("timed out waiting for key provisioning; retry to check the same session")
|
||||
}
|
||||
|
||||
func sleepContext(ctx context.Context, delay time.Duration) error {
|
||||
timer := time.NewTimer(delay)
|
||||
defer timer.Stop()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-timer.C:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func status(callback func(Status, string), value Status, detail string) {
|
||||
if callback != nil {
|
||||
callback(value, detail)
|
||||
}
|
||||
}
|
||||
|
||||
func firstNonEmpty(values ...string) string {
|
||||
for _, value := range values {
|
||||
if value = strings.TrimSpace(value); value != "" {
|
||||
return value
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
package aimlapi
|
||||
|
||||
import (
|
||||
"net/mail"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ValidEmail performs the lightweight validation used before starting the
|
||||
// passwordless onboarding request.
|
||||
func ValidEmail(value string) bool {
|
||||
value = strings.TrimSpace(value)
|
||||
address, err := mail.ParseAddress(value)
|
||||
if err != nil || address.Address != value {
|
||||
return false
|
||||
}
|
||||
at := strings.LastIndexByte(value, '@')
|
||||
if at <= 0 || at == len(value)-1 {
|
||||
return false
|
||||
}
|
||||
domain := value[at+1:]
|
||||
if strings.HasPrefix(domain, ".") || strings.HasSuffix(domain, ".") || strings.Contains(domain, "..") {
|
||||
return false
|
||||
}
|
||||
dot := strings.LastIndexByte(domain, '.')
|
||||
if dot <= 0 || dot == len(domain)-1 {
|
||||
return false
|
||||
}
|
||||
tld := domain[dot+1:]
|
||||
if len(tld) < 2 {
|
||||
return false
|
||||
}
|
||||
for _, char := range tld {
|
||||
if char < 'A' || char > 'Z' && char < 'a' || char > 'z' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
+6
-5
@@ -766,11 +766,12 @@ func runInteractiveTUIWithSetup(stderr io.Writer, deps appDeps, permissionMode a
|
||||
}
|
||||
sandboxBackend := deps.selectSandboxBackend(sandbox.BackendOptions{})
|
||||
sandboxEngine := sandbox.NewEngine(sandbox.EngineOptions{
|
||||
WorkspaceRoot: workspaceRoot,
|
||||
Policy: applyConfiguredSandboxPolicy(sandbox.DefaultPolicy(), resolved.Sandbox),
|
||||
Store: sandboxStore,
|
||||
Backend: sandboxBackend,
|
||||
Scope: scope,
|
||||
WorkspaceRoot: workspaceRoot,
|
||||
Policy: applyConfiguredSandboxPolicy(sandbox.DefaultPolicy(), resolved.Sandbox),
|
||||
Store: sandboxStore,
|
||||
Backend: sandboxBackend,
|
||||
Scope: scope,
|
||||
SensitiveEnvKeys: providerSensitiveEnvKeys(resolved),
|
||||
})
|
||||
lastKnownMCPConfig := mcpConfig
|
||||
fileTracker := tools.NewFileTracker()
|
||||
|
||||
@@ -348,6 +348,83 @@ func TestRunProvidersAddWritesCatalogProfile(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunProvidersAddAimlapiWritesDefaultHeaders(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
|
||||
|
||||
exitCode := runWithDeps([]string{"providers", "add", "aimlapi"}, &stdout, &stderr, providerSetupDeps(configPath))
|
||||
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("expected exit code %d, got %d: %s", exitSuccess, exitCode, stderr.String())
|
||||
}
|
||||
cfg := readFileConfig(t, configPath)
|
||||
if len(cfg.Providers) != 1 {
|
||||
t.Fatalf("providers = %#v, want one provider", cfg.Providers)
|
||||
}
|
||||
profile := cfg.Providers[0]
|
||||
if profile.Name != "aimlapi" ||
|
||||
profile.CatalogID != "aimlapi" ||
|
||||
profile.BaseURL != "https://api.aimlapi.com/v1" ||
|
||||
profile.Model != "anthropic/claude-sonnet-5" ||
|
||||
profile.APIKeyEnv != "AIMLAPI_API_KEY" {
|
||||
t.Fatalf("unexpected provider profile: %#v", profile)
|
||||
}
|
||||
if profile.CustomHeaders["X-AIMLAPI-Partner-ID"] != "part_62yQoGYDq4Yqnrj2R1iGrDNJ" ||
|
||||
profile.CustomHeaders["X-AIMLAPI-Integration-Repo"] != "Gitlawb/zero" {
|
||||
t.Fatalf("missing aimlapi.com default headers: %#v", profile.CustomHeaders)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunProvidersAddAimlapiMixedCaseHeaderOverride(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
|
||||
|
||||
// A differently-cased override must replace the catalog header in place, not
|
||||
// leave both spellings behind to race when request construction canonicalizes.
|
||||
exitCode := runWithDeps([]string{
|
||||
"providers", "add", "aimlapi",
|
||||
"--header", "x-aimlapi-partner-id=part_override",
|
||||
}, &stdout, &stderr, providerSetupDeps(configPath))
|
||||
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("expected exit code %d, got %d: %s", exitSuccess, exitCode, stderr.String())
|
||||
}
|
||||
profile := readFileConfig(t, configPath).Providers[0]
|
||||
if _, ok := profile.CustomHeaders["x-aimlapi-partner-id"]; ok {
|
||||
t.Fatalf("lowercase override left a duplicate key: %#v", profile.CustomHeaders)
|
||||
}
|
||||
if profile.CustomHeaders["X-AIMLAPI-Partner-ID"] != "part_override" {
|
||||
t.Fatalf("override did not win on the canonical key: %#v", profile.CustomHeaders)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunProvidersAddAimlapiOverrideDropsCatalogHeaders(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
|
||||
|
||||
exitCode := runWithDeps([]string{
|
||||
"providers", "add", "aimlapi",
|
||||
"--base-url", "https://staging.example/v1",
|
||||
"--header", "X-Environment=staging",
|
||||
}, &stdout, &stderr, providerSetupDeps(configPath))
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("expected exit code %d, got %d: %s", exitSuccess, exitCode, stderr.String())
|
||||
}
|
||||
profile := readFileConfig(t, configPath).Providers[0]
|
||||
if _, ok := profile.CustomHeaders["X-AIMLAPI-Partner-ID"]; ok {
|
||||
t.Fatalf("partner attribution leaked to overridden endpoint: %#v", profile.CustomHeaders)
|
||||
}
|
||||
if _, ok := profile.CustomHeaders["X-AIMLAPI-Integration-Repo"]; ok {
|
||||
t.Fatalf("integration attribution leaked to overridden endpoint: %#v", profile.CustomHeaders)
|
||||
}
|
||||
if profile.CustomHeaders["X-Environment"] != "staging" {
|
||||
t.Fatalf("explicit user header was dropped: %#v", profile.CustomHeaders)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunProvidersAddWritesCustomHeaders(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
+77
-28
@@ -412,6 +412,36 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
}
|
||||
}
|
||||
|
||||
// Optimized OpenAI turn sessions (ZERO_OPENAI_TURN_SESSION, default off).
|
||||
// nil when gated off or the profile is ineligible: agent.Run then wraps the
|
||||
// provider in its default adapter — the exact code path of today. The
|
||||
// session switcher is installed only when the run START is optimized, so
|
||||
// the legacy ModelSwitcher path above stays untouched otherwise.
|
||||
turnSessions, _ := providers.OptimizedTurnSessions(resolved.Provider, provider, providers.Options{})
|
||||
var modelSessionSwitcher func(context.Context, string) (zeroruntime.TurnSessionProvider, error)
|
||||
if options.allowEscalation && turnSessions != nil {
|
||||
modelSessionSwitcher = func(_ context.Context, modelID string) (zeroruntime.TurnSessionProvider, error) {
|
||||
switchedProfile := resolved.Provider
|
||||
switchedProfile.Model = modelID
|
||||
switchedProvider, err := deps.newProvider(switchedProfile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if switchedProvider == nil {
|
||||
// The loop treats a nil session source as "no swap" — mirror the
|
||||
// legacy closure's (nil, nil) contract.
|
||||
return nil, nil
|
||||
}
|
||||
currentModel = modelID
|
||||
if optimized, ok := providers.OptimizedTurnSessions(switchedProfile, switchedProvider, providers.Options{}); ok {
|
||||
return optimized, nil
|
||||
}
|
||||
// Ineligible switch target: default adapter, but with the switched
|
||||
// model's resolved capability projection preserved.
|
||||
return providers.DefaultTurnSessions(switchedProfile, switchedProvider, providers.Options{}), nil
|
||||
}
|
||||
}
|
||||
|
||||
runMetadata, err := resolveExecRunMetadata(resolved.Provider)
|
||||
if err != nil {
|
||||
return writeExecProviderError(stdout, stderr, options.outputFormat, "provider_error", err.Error())
|
||||
@@ -572,29 +602,31 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
hookDispatcher, hookSkip := newHookDispatcherWithExtra(workspaceRoot, pluginActivation.hooks, trustRoot)
|
||||
emitTrustNotice(stderr, hookSkip, pluginActivation.trustSkip, mcpSkip)
|
||||
result, err := agent.Run(runCtx, agentPrompt, provider, agent.Options{
|
||||
MaxTurns: resolved.MaxTurns,
|
||||
ContextWindow: resolveAgentContextWindow(runCtx, modelRegistry, resolved.Provider),
|
||||
DeferThreshold: effectiveDeferThreshold,
|
||||
Specialists: specialistRuntime.specialistInfos(),
|
||||
Skills: pluginActivation.skillInfos(deps.skillsDir()),
|
||||
SessionID: preparedSession.Session.SessionID,
|
||||
CallingSessionID: options.callingSessionID,
|
||||
CallingToolUseID: options.callingToolUseID,
|
||||
Tag: options.tag,
|
||||
Depth: options.depth,
|
||||
SessionTitle: sessionTitle,
|
||||
ProviderName: resolved.Provider.Name,
|
||||
Model: resolved.Provider.Model,
|
||||
ModelSwitcher: modelSwitcher,
|
||||
ReasoningEffort: forwardEffort,
|
||||
Trace: traceRecorder,
|
||||
Cwd: workspaceRoot,
|
||||
Images: images,
|
||||
Registry: registry,
|
||||
PermissionMode: permissionMode,
|
||||
Autonomy: options.autonomy,
|
||||
SelfCorrect: selfCorrector,
|
||||
FileDiagnostics: fileDiagnostics,
|
||||
MaxTurns: resolved.MaxTurns,
|
||||
ContextWindow: resolveAgentContextWindow(runCtx, modelRegistry, resolved.Provider),
|
||||
DeferThreshold: effectiveDeferThreshold,
|
||||
Specialists: specialistRuntime.specialistInfos(),
|
||||
Skills: pluginActivation.skillInfos(deps.skillsDir()),
|
||||
SessionID: preparedSession.Session.SessionID,
|
||||
CallingSessionID: options.callingSessionID,
|
||||
CallingToolUseID: options.callingToolUseID,
|
||||
Tag: options.tag,
|
||||
Depth: options.depth,
|
||||
SessionTitle: sessionTitle,
|
||||
ProviderName: resolved.Provider.Name,
|
||||
Model: resolved.Provider.Model,
|
||||
ModelSwitcher: modelSwitcher,
|
||||
TurnSessionProvider: turnSessions,
|
||||
ModelSessionSwitcher: modelSessionSwitcher,
|
||||
ReasoningEffort: forwardEffort,
|
||||
Trace: traceRecorder,
|
||||
Cwd: workspaceRoot,
|
||||
Images: images,
|
||||
Registry: registry,
|
||||
PermissionMode: permissionMode,
|
||||
Autonomy: options.autonomy,
|
||||
SelfCorrect: selfCorrector,
|
||||
FileDiagnostics: fileDiagnostics,
|
||||
// Headless exec: don't accept a no-tool-call turn as "done" while work
|
||||
// clearly remains (pending plan items / a mid-step continuation cue) —
|
||||
// nudge to continue, and finalize as INCOMPLETE rather than false success
|
||||
@@ -636,6 +668,9 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
if len(result.Meta) > 0 {
|
||||
payload["meta"] = result.Meta
|
||||
}
|
||||
if result.Truncated {
|
||||
payload["truncated"] = true
|
||||
}
|
||||
if result.Redacted {
|
||||
payload["redacted"] = true
|
||||
}
|
||||
@@ -853,14 +888,28 @@ func buildExecSandboxEngine(workspaceRoot string, resolved config.ResolvedConfig
|
||||
policy := applyConfiguredSandboxPolicy(sandbox.DefaultPolicy(), resolved.Sandbox)
|
||||
backend := deps.selectSandboxBackend(sandbox.BackendOptions{})
|
||||
return sandbox.NewEngine(sandbox.EngineOptions{
|
||||
WorkspaceRoot: workspaceRoot,
|
||||
Policy: policy,
|
||||
Store: store,
|
||||
Backend: backend,
|
||||
Scope: scope,
|
||||
WorkspaceRoot: workspaceRoot,
|
||||
Policy: policy,
|
||||
Store: store,
|
||||
Backend: backend,
|
||||
Scope: scope,
|
||||
SensitiveEnvKeys: providerSensitiveEnvKeys(resolved),
|
||||
}), nil
|
||||
}
|
||||
|
||||
func providerSensitiveEnvKeys(resolved config.ResolvedConfig) []string {
|
||||
keys := make([]string, 0, len(resolved.Providers)+1)
|
||||
for _, profile := range resolved.Providers {
|
||||
if key := strings.TrimSpace(profile.APIKeyEnv); key != "" {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
}
|
||||
if key := strings.TrimSpace(resolved.Provider.APIKeyEnv); key != "" {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
// applyConfiguredSandboxPolicy overlays every config-sourced sandbox knob onto
|
||||
// the default policy.
|
||||
func applyConfiguredSandboxPolicy(policy sandbox.Policy, cfg config.SandboxConfig) sandbox.Policy {
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/config"
|
||||
"github.com/Gitlawb/zero/internal/providers/openai"
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// TestRunExecOptimizedSessionUnderGate proves the end-to-end wiring: with
|
||||
// ZERO_OPENAI_TURN_SESSION on and a real official-OpenAI provider, a headless
|
||||
// run streams through the optimized session — the server sees the prewarm HEAD
|
||||
// probe in addition to the turn's POST.
|
||||
func TestRunExecOptimizedSessionUnderGate(t *testing.T) {
|
||||
t.Setenv("XDG_DATA_HOME", t.TempDir())
|
||||
t.Setenv("ZERO_OPENAI_TURN_SESSION", "1")
|
||||
cwd := t.TempDir()
|
||||
|
||||
var heads, posts atomic.Int64
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodHead:
|
||||
heads.Add(1)
|
||||
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/chat/completions"):
|
||||
posts.Add(1)
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
_, _ = w.Write([]byte("data: {\"choices\":[{\"delta\":{\"content\":\"done\"}}]}\n\n"))
|
||||
_, _ = w.Write([]byte("data: [DONE]\n\n"))
|
||||
default:
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
exitCode := runWithDeps([]string{"exec", "say done"}, &stdout, &stderr, appDeps{
|
||||
getwd: func() (string, error) { return cwd, nil },
|
||||
resolveConfig: func(string, config.Overrides) (config.ResolvedConfig, error) {
|
||||
return config.ResolvedConfig{
|
||||
ActiveProvider: "official-openai",
|
||||
Provider: config.ProviderProfile{
|
||||
Name: "official-openai",
|
||||
ProviderKind: config.ProviderKindOpenAI,
|
||||
BaseURL: server.URL,
|
||||
APIKey: "sk-exec-test",
|
||||
Model: "pr8-exec-model",
|
||||
},
|
||||
}, nil
|
||||
},
|
||||
newProvider: func(profile config.ProviderProfile) (zeroruntime.Provider, error) {
|
||||
// Pin a keep-alive transport so the probe fires on every platform:
|
||||
// the shared default transport disables keep-alives on macOS, where
|
||||
// the session correctly skips the prewarm.
|
||||
transport := http.DefaultTransport.(*http.Transport).Clone()
|
||||
transport.DisableKeepAlives = false
|
||||
return openai.New(openai.Options{
|
||||
APIKey: profile.APIKey,
|
||||
BaseURL: profile.BaseURL,
|
||||
Model: profile.Model,
|
||||
HTTPClient: &http.Client{Transport: transport},
|
||||
})
|
||||
},
|
||||
})
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
||||
}
|
||||
if got := posts.Load(); got < 1 {
|
||||
t.Fatalf("server saw %d chat-completions POSTs, want >= 1", got)
|
||||
}
|
||||
// The prewarm probe is asynchronous; it fires well before the model turn
|
||||
// finishes, but poll rather than assuming scheduling order. The deadline
|
||||
// exceeds the production prewarmTimeout (3s) so a slow probe cannot flake
|
||||
// this assertion.
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for heads.Load() == 0 && time.Now().Before(deadline) {
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if got := heads.Load(); got != 1 {
|
||||
t.Fatalf("server saw %d prewarm HEAD probes, want exactly 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunExecGateOnFallsBackForFakeProvider proves fallback safety: with the
|
||||
// gate on but a provider that is not the concrete *openai.Provider, the run
|
||||
// proceeds on the default path exactly as today.
|
||||
func TestRunExecGateOnFallsBackForFakeProvider(t *testing.T) {
|
||||
t.Setenv("XDG_DATA_HOME", t.TempDir())
|
||||
t.Setenv("ZERO_OPENAI_TURN_SESSION", "1")
|
||||
cwd := t.TempDir()
|
||||
|
||||
var builds int
|
||||
var stdout, stderr bytes.Buffer
|
||||
exitCode := runWithDeps([]string{"exec", "--model", "claude-haiku-4.5", "hello"}, &stdout, &stderr, appDeps{
|
||||
getwd: func() (string, error) { return cwd, nil },
|
||||
resolveConfig: func(_ string, overrides config.Overrides) (config.ResolvedConfig, error) {
|
||||
model := "claude-haiku-4.5"
|
||||
if overrides.Provider.Model != "" {
|
||||
model = overrides.Provider.Model
|
||||
}
|
||||
cfg := execResolvedConfig()
|
||||
cfg.Provider.ProviderKind = config.ProviderKindAnthropic
|
||||
cfg.Provider.Model = model
|
||||
cfg.MaxTurns = 3
|
||||
return cfg, nil
|
||||
},
|
||||
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
||||
builds++
|
||||
return &escalatingExecProvider{}, nil
|
||||
},
|
||||
})
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("exitCode = %d stdout=%s stderr=%s", exitCode, stdout.String(), stderr.String())
|
||||
}
|
||||
if builds != 1 {
|
||||
t.Fatalf("newProvider called %d times, want 1 (default path, no optimized session)", builds)
|
||||
}
|
||||
}
|
||||
@@ -153,6 +153,9 @@ func (writer *execEventWriter) toolResult(result agent.ToolResult) {
|
||||
if len(result.Meta) > 0 {
|
||||
payload["meta"] = result.Meta
|
||||
}
|
||||
if result.Truncated {
|
||||
payload["truncated"] = true
|
||||
}
|
||||
if result.Redacted {
|
||||
payload["redacted"] = true
|
||||
}
|
||||
@@ -166,7 +169,8 @@ func (writer *execEventWriter) toolResult(result agent.ToolResult) {
|
||||
return
|
||||
}
|
||||
if writer.format == execOutputStreamJSON {
|
||||
output, truncated := truncateForStreamJSONOutput(result.Output)
|
||||
output, surfaceTruncated := truncateForStreamJSONOutput(result.Output)
|
||||
truncated := result.Truncated || surfaceTruncated
|
||||
event := streamjson.Event{
|
||||
Type: streamjson.EventToolResult,
|
||||
RunID: writer.runID,
|
||||
|
||||
@@ -1,8 +1,12 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/agent"
|
||||
"github.com/Gitlawb/zero/internal/tools"
|
||||
)
|
||||
|
||||
@@ -16,3 +20,35 @@ func TestStreamJSONSideEffectReportsNoneForControlTool(t *testing.T) {
|
||||
t.Fatalf("streamJSONSideEffect(escalate_model) = %q, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecWriterPropagatesToolResultTruncation(t *testing.T) {
|
||||
for _, format := range []execOutputFormat{execOutputJSON, execOutputStreamJSON} {
|
||||
t.Run(string(format), func(t *testing.T) {
|
||||
var stdout, stderr bytes.Buffer
|
||||
writer := execEventWriter{
|
||||
stdout: &stdout,
|
||||
stderr: &stderr,
|
||||
format: format,
|
||||
runID: "run_budget",
|
||||
streamedText: &strings.Builder{},
|
||||
}
|
||||
writer.toolResult(agent.ToolResult{
|
||||
ToolCallID: "call_budget",
|
||||
Name: "read_file",
|
||||
Status: tools.StatusOK,
|
||||
Output: "bounded output",
|
||||
Truncated: true,
|
||||
})
|
||||
if writer.err != nil {
|
||||
t.Fatalf("toolResult: %v", writer.err)
|
||||
}
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(bytes.TrimSpace(stdout.Bytes()), &payload); err != nil {
|
||||
t.Fatalf("decode output %q: %v", stdout.String(), err)
|
||||
}
|
||||
if payload["truncated"] != true {
|
||||
t.Fatalf("truncated = %#v, want true; payload=%#v", payload["truncated"], payload)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -17,7 +17,6 @@ func TestProviderCatalogRuntimeProvidersShowOnboardingSetupHints(t *testing.T) {
|
||||
defaultModel string
|
||||
}{
|
||||
{id: "openai", name: "OpenAI", transport: "openai", defaultModel: "gpt-4.1"},
|
||||
{id: "aimlapi", name: "AI/ML API", transport: "openai-compatible", defaultModel: "openai/gpt-5-chat"},
|
||||
{id: "groq", name: "Groq", transport: "openai-compatible", defaultModel: "llama-3.3-70b-versatile"},
|
||||
{id: "longcat", name: "LongCat", transport: "openai-compatible", defaultModel: "LongCat-2.0"},
|
||||
{id: "ollama-cloud", name: "Ollama Cloud", transport: "openai-compatible", defaultModel: "qwen3-coder:480b"},
|
||||
@@ -48,21 +47,6 @@ func TestProviderCatalogRuntimeProvidersShowOnboardingSetupHints(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderCatalogAIMLAPIRequiresAPIKey(t *testing.T) {
|
||||
output := runProviderCatalogOnboarding(t)
|
||||
block := providerCatalogOnboardingBlock(t, output, "aimlapi")
|
||||
|
||||
for _, want := range []string{
|
||||
"requiresAuth=true",
|
||||
"authEnvVars=AIMLAPI_API_KEY",
|
||||
"setup: zero providers setup aimlapi --set-active",
|
||||
} {
|
||||
if !strings.Contains(block, want) {
|
||||
t.Fatalf("expected AIMLAPI catalog block to contain %q, got:\n%s", want, block)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderCatalogLongCatRequiresAPIKey(t *testing.T) {
|
||||
output := runProviderCatalogOnboarding(t)
|
||||
block := providerCatalogOnboardingBlock(t, output, "longcat")
|
||||
|
||||
@@ -3,8 +3,10 @@ package cli
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/aimlapi"
|
||||
"github.com/Gitlawb/zero/internal/config"
|
||||
"github.com/Gitlawb/zero/internal/providercatalog"
|
||||
"github.com/Gitlawb/zero/internal/providerhealth"
|
||||
@@ -424,22 +426,39 @@ func providerProfileForAdd(options providerAddOptions) (config.ProviderProfile,
|
||||
if apiKeyEnv == "" && len(descriptor.AuthEnvVars) > 0 {
|
||||
apiKeyEnv = descriptor.AuthEnvVars[0]
|
||||
}
|
||||
baseURL := firstNonEmptyCLI(options.baseURL, descriptor.DefaultBaseURL)
|
||||
var catalogHeaders map[string]string
|
||||
if sameProviderSetupBaseURL(baseURL, descriptor.DefaultBaseURL) {
|
||||
catalogHeaders = descriptor.CustomHeaders
|
||||
if strings.EqualFold(strings.TrimSpace(descriptor.ID), "aimlapi") {
|
||||
catalogHeaders = aimlapi.WithResolvedPartnerHeader(catalogHeaders)
|
||||
}
|
||||
}
|
||||
profile := config.ProviderProfile{
|
||||
Name: name,
|
||||
ProviderKind: providerKindForDescriptor(descriptor),
|
||||
CatalogID: descriptor.ID,
|
||||
BaseURL: firstNonEmptyCLI(options.baseURL, descriptor.DefaultBaseURL),
|
||||
BaseURL: baseURL,
|
||||
APIKeyEnv: apiKeyEnv,
|
||||
APIFormat: firstAPIFormat(descriptor),
|
||||
AuthHeader: strings.TrimSpace(options.authHeader),
|
||||
AuthScheme: normalizeAuthScheme(options.authScheme),
|
||||
AuthHeaderValue: strings.TrimSpace(options.authHeaderValue),
|
||||
CustomHeaders: copyProviderHeaders(options.customHeaders),
|
||||
Model: firstNonEmptyCLI(options.model, descriptor.DefaultModel),
|
||||
// Catalog attribution belongs only to the catalog endpoint. Explicit
|
||||
// user headers still apply to an override/staging/proxy endpoint.
|
||||
CustomHeaders: mergeProviderHeaders(catalogHeaders, options.customHeaders),
|
||||
Model: firstNonEmptyCLI(options.model, descriptor.DefaultModel),
|
||||
}
|
||||
return profile, nil
|
||||
}
|
||||
|
||||
func sameProviderSetupBaseURL(left string, right string) bool {
|
||||
return strings.EqualFold(
|
||||
strings.TrimRight(strings.TrimSpace(left), "/"),
|
||||
strings.TrimRight(strings.TrimSpace(right), "/"),
|
||||
)
|
||||
}
|
||||
|
||||
func selectProviderForCheck(resolved config.ResolvedConfig, name string) (config.ProviderProfile, error) {
|
||||
name = strings.TrimSpace(name)
|
||||
if name == "" {
|
||||
@@ -544,3 +563,43 @@ func copyProviderHeaders(headers map[string]string) map[string]string {
|
||||
}
|
||||
return copied
|
||||
}
|
||||
|
||||
func mergeProviderHeaders(base map[string]string, overrides map[string]string) map[string]string {
|
||||
merged := copyProviderHeaders(base)
|
||||
if len(overrides) == 0 {
|
||||
return merged
|
||||
}
|
||||
if merged == nil {
|
||||
merged = map[string]string{}
|
||||
}
|
||||
for key, value := range overrides {
|
||||
// HTTP header names are case-insensitive and request construction
|
||||
// canonicalizes them (http.Header.Set), so an override that differs from a
|
||||
// base key only in case must replace that key's value in place — otherwise
|
||||
// both survive as distinct map entries and race non-deterministically at
|
||||
// build time. Keep the existing spelling (the catalog's, which the resolver
|
||||
// re-merges verbatim); only a genuinely new header is added as typed.
|
||||
if existing, ok := headerKeyFold(merged, key); ok {
|
||||
merged[existing] = value
|
||||
continue
|
||||
}
|
||||
merged[key] = value
|
||||
}
|
||||
return merged
|
||||
}
|
||||
|
||||
// headerKeyFold returns the key in headers whose canonical HTTP form matches key,
|
||||
// so a case-insensitive override lands on the existing entry instead of adding a
|
||||
// colliding duplicate.
|
||||
func headerKeyFold(headers map[string]string, key string) (string, bool) {
|
||||
if _, ok := headers[key]; ok {
|
||||
return key, true
|
||||
}
|
||||
canonical := http.CanonicalHeaderKey(key)
|
||||
for existing := range headers {
|
||||
if http.CanonicalHeaderKey(existing) == canonical {
|
||||
return existing, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/config"
|
||||
)
|
||||
|
||||
func TestProviderSensitiveEnvKeys(t *testing.T) {
|
||||
resolved := config.ResolvedConfig{
|
||||
Providers: []config.ProviderProfile{
|
||||
{Name: "custom", APIKeyEnv: " COMPANY_LLM_SECRET "},
|
||||
{Name: "catalog", APIKeyEnv: "OPENAI_API_KEY"},
|
||||
{Name: "inline"},
|
||||
},
|
||||
Provider: config.ProviderProfile{Name: "active", APIKeyEnv: "ACTIVE_PROVIDER_SECRET"},
|
||||
}
|
||||
want := []string{"COMPANY_LLM_SECRET", "OPENAI_API_KEY", "ACTIVE_PROVIDER_SECRET"}
|
||||
if got := providerSensitiveEnvKeys(resolved); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("providerSensitiveEnvKeys() = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
@@ -200,6 +200,54 @@ func TestSaveSetupProviderStoresPastedAPIKey(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSaveSetupProviderAimlapiEnvReferenceAndOverrideHeaders(t *testing.T) {
|
||||
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
|
||||
result, err := saveSetupProvider(appDeps{
|
||||
userConfigPath: func() (string, error) { return configPath, nil },
|
||||
}, tui.SetupSelection{
|
||||
CatalogID: "aimlapi",
|
||||
BaseURL: "https://staging.example/v1",
|
||||
Model: "anthropic/claude-sonnet-5",
|
||||
}, setupSaveOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("saveSetupProvider() error = %v", err)
|
||||
}
|
||||
if result.Provider.APIKey != "" || result.Provider.APIKeyEnv != "AIMLAPI_API_KEY" {
|
||||
t.Fatalf("runtime profile did not retain AIMLAPI_API_KEY reference: %+v", result.Provider)
|
||||
}
|
||||
if len(result.Provider.CustomHeaders) != 0 {
|
||||
t.Fatalf("catalog headers leaked to first-run override: %#v", result.Provider.CustomHeaders)
|
||||
}
|
||||
persisted := readFileConfig(t, configPath).Providers[0]
|
||||
if persisted.APIKey != "" || persisted.APIKeyEnv != "AIMLAPI_API_KEY" || persisted.APIKeyStored {
|
||||
t.Fatalf("persisted credential source changed: %+v", persisted)
|
||||
}
|
||||
if len(persisted.CustomHeaders) != 0 {
|
||||
t.Fatalf("persisted override contains catalog headers: %#v", persisted.CustomHeaders)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSaveSetupProviderAimlapiUsesResolvedPartnerOverride(t *testing.T) {
|
||||
t.Setenv("AIMLAPI_PARTNER_ID", "part_override")
|
||||
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
|
||||
result, err := saveSetupProvider(appDeps{
|
||||
userConfigPath: func() (string, error) { return configPath, nil },
|
||||
}, tui.SetupSelection{
|
||||
CatalogID: "aimlapi",
|
||||
BaseURL: "https://api.aimlapi.com/v1",
|
||||
Model: "anthropic/claude-sonnet-5",
|
||||
}, setupSaveOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("saveSetupProvider() error = %v", err)
|
||||
}
|
||||
if got := result.Provider.CustomHeaders["X-AIMLAPI-Partner-ID"]; got != "part_override" {
|
||||
t.Fatalf("runtime partner header = %q, want part_override", got)
|
||||
}
|
||||
if got := readFileConfig(t, configPath).Providers[0].CustomHeaders["X-AIMLAPI-Partner-ID"]; got != "part_override" {
|
||||
t.Fatalf("persisted partner header = %q, want part_override", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSaveSetupProviderStoresCustomEndpointSelection(t *testing.T) {
|
||||
configPath := filepath.Join(t.TempDir(), "zero", "config.json")
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
func mergeMCPConfig(dst *MCPConfig, src MCPConfig) {
|
||||
func mergeMCPConfig(dst *MCPConfig, src MCPConfig, canReenable bool) {
|
||||
if len(src.Servers) == 0 {
|
||||
return
|
||||
}
|
||||
@@ -14,10 +14,14 @@ func mergeMCPConfig(dst *MCPConfig, src MCPConfig) {
|
||||
dst.Servers = map[string]MCPServerConfig{}
|
||||
}
|
||||
for name, server := range src.Servers {
|
||||
dst.Servers[name] = mergeMCPServer(dst.Servers[name], server)
|
||||
dst.Servers[name] = mergeMCPServer(dst.Servers[name], server, canReenable)
|
||||
}
|
||||
}
|
||||
|
||||
// mergeProjectMCPConfig merges a project-scoped (lower-trust) MCP config into
|
||||
// dst. It never re-enables a server the user explicitly disabled or disabled a
|
||||
// server the user explicitly enabled: it merges with canReenable=false so the
|
||||
// trust-boundary guard in mergeMCPServer keeps the user's decision authoritative.
|
||||
func mergeProjectMCPConfig(dst *MCPConfig, src MCPConfig) error {
|
||||
if len(src.Servers) == 0 {
|
||||
return nil
|
||||
@@ -27,7 +31,7 @@ func mergeProjectMCPConfig(dst *MCPConfig, src MCPConfig) error {
|
||||
}
|
||||
for name, server := range src.Servers {
|
||||
base := dst.Servers[name]
|
||||
candidate := mergeMCPServer(base, server)
|
||||
candidate := mergeMCPServer(base, server, false)
|
||||
if projectMCPServerTargetChanges(base, server) && hasInheritedMCPCredentialMaterial(server, candidate) {
|
||||
return fmt.Errorf("project MCP server %q cannot override target while inheriting user credentials; set headers/env/oauth explicitly or use a new server name", name)
|
||||
}
|
||||
@@ -37,7 +41,11 @@ func mergeProjectMCPConfig(dst *MCPConfig, src MCPConfig) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func mergeMCPServer(base MCPServerConfig, next MCPServerConfig) MCPServerConfig {
|
||||
// mergeMCPServer merges a later config layer's MCP server into the base. The
|
||||
// canReenable flag marks the CLI-override scope, the only layer that is allowed
|
||||
// to lift a sticky user-level disable/enable decision (see the disabled-handling
|
||||
// block).
|
||||
func mergeMCPServer(base MCPServerConfig, next MCPServerConfig, canReenable bool) MCPServerConfig {
|
||||
if strings.TrimSpace(next.Type) != "" {
|
||||
base.Type = next.Type
|
||||
}
|
||||
@@ -62,8 +70,26 @@ func mergeMCPServer(base MCPServerConfig, next MCPServerConfig) MCPServerConfig
|
||||
if next.OAuth != nil {
|
||||
base.OAuth = next.OAuth
|
||||
}
|
||||
// Capture the higher-trust scope's prior decision before folding in
|
||||
// next.disabledSet. The trust boundary must be evaluated against the state
|
||||
// the higher-trust layer actually left behind, not against next's own flag.
|
||||
baseDisabledSet := base.disabledSet
|
||||
baseDisabled := base.Disabled
|
||||
if next.disabledSet {
|
||||
base.disabledSet = true
|
||||
}
|
||||
if next.disabledSet || next.Disabled {
|
||||
base.Disabled = next.Disabled
|
||||
// Trust boundary: once a higher-trust scope has explicitly set the
|
||||
// disabled flag, a lower-trust layer (project config) cannot override
|
||||
// that decision in either direction — the user's choice (whether to
|
||||
// enable or disable a server) wins over the repo. This blocks a repo
|
||||
// from re-enabling a server the user disabled, and from disabling a
|
||||
// server the user explicitly enabled. The only layer allowed to
|
||||
// override an explicit higher-scope decision is the CLI override scope
|
||||
// (canReenable=true).
|
||||
if canReenable || !(baseDisabledSet && baseDisabled != next.Disabled) {
|
||||
base.Disabled = next.Disabled
|
||||
}
|
||||
}
|
||||
if next.configured {
|
||||
base.configured = true
|
||||
@@ -75,7 +101,7 @@ func mergeMCPServer(base MCPServerConfig, next MCPServerConfig) MCPServerConfig
|
||||
}
|
||||
|
||||
func projectMCPServerTargetChanges(base MCPServerConfig, next MCPServerConfig) bool {
|
||||
if strings.TrimSpace(next.Type) != "" && mcpServerTransportKind(base) != mcpServerTransportKind(mergeMCPServer(base, next)) {
|
||||
if strings.TrimSpace(next.Type) != "" && mcpServerTransportKind(base) != mcpServerTransportKind(mergeMCPServer(base, next, true)) {
|
||||
return true
|
||||
}
|
||||
if strings.TrimSpace(next.URL) != "" && strings.TrimSpace(next.URL) != strings.TrimSpace(base.URL) {
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/aimlapi"
|
||||
"github.com/Gitlawb/zero/internal/modelregistry"
|
||||
"github.com/Gitlawb/zero/internal/notify"
|
||||
"github.com/Gitlawb/zero/internal/providercatalog"
|
||||
@@ -170,7 +171,10 @@ func ResolveMCP(options ResolveOptions) (MCPConfig, error) {
|
||||
if err != nil {
|
||||
return MCPConfig{}, err
|
||||
}
|
||||
mergeMCPConfig(&cfg.MCP, fileConfig.MCP)
|
||||
// User config is higher-trust than project config: it may re-enable a
|
||||
// server the user disabled (a user-level disable is sticky, but the user
|
||||
// scope itself may lift it).
|
||||
mergeMCPConfig(&cfg.MCP, fileConfig.MCP, true)
|
||||
}
|
||||
// Drop the project layer when the workspace is untrusted, so a cloned repo's
|
||||
// ./.zero/config.json cannot register (and spawn) MCP servers. Fail-closed:
|
||||
@@ -184,7 +188,7 @@ func ResolveMCP(options ResolveOptions) (MCPConfig, error) {
|
||||
return MCPConfig{}, err
|
||||
}
|
||||
}
|
||||
mergeMCPConfig(&cfg.MCP, options.Overrides.MCP)
|
||||
mergeMCPConfig(&cfg.MCP, options.Overrides.MCP, true)
|
||||
return cfg.MCP, nil
|
||||
}
|
||||
|
||||
@@ -211,7 +215,7 @@ func mergeConfig(dst *FileConfig, src FileConfig) {
|
||||
for _, provider := range src.Providers {
|
||||
mergeProvider(dst, provider)
|
||||
}
|
||||
mergeMCPConfig(&dst.MCP, src.MCP)
|
||||
mergeMCPConfig(&dst.MCP, src.MCP, true)
|
||||
if network := strings.TrimSpace(src.Sandbox.Network); network != "" {
|
||||
dst.Sandbox.Network = network
|
||||
}
|
||||
@@ -709,7 +713,7 @@ func applyOverrides(cfg *FileConfig, overrides Overrides) {
|
||||
if hasProviderFields(overrides.Provider) {
|
||||
mergeProvider(cfg, overrides.Provider)
|
||||
}
|
||||
mergeMCPConfig(&cfg.MCP, overrides.MCP)
|
||||
mergeMCPConfig(&cfg.MCP, overrides.MCP, true)
|
||||
}
|
||||
|
||||
func mergeLocalControlConfig(dst *LocalControlConfig, src LocalControlConfig) {
|
||||
@@ -1039,6 +1043,45 @@ func applyCatalogDescriptor(profile *ProviderProfile, descriptor providercatalog
|
||||
if profile.APIKeyEnv == "" && len(descriptor.AuthEnvVars) > 0 && (!explicitBaseURL || sameBaseURL(profile.BaseURL, descriptor.DefaultBaseURL)) {
|
||||
profile.APIKeyEnv = descriptor.AuthEnvVars[0]
|
||||
}
|
||||
canonicalCatalogEndpoint := !explicitBaseURL || sameBaseURL(profile.BaseURL, descriptor.DefaultBaseURL)
|
||||
if len(descriptor.CustomHeaders) > 0 && canonicalCatalogEndpoint {
|
||||
catalogHeaders := descriptor.CustomHeaders
|
||||
if strings.EqualFold(strings.TrimSpace(descriptor.ID), "aimlapi") {
|
||||
catalogHeaders = aimlapi.WithResolvedPartnerHeader(catalogHeaders)
|
||||
}
|
||||
merged := copyStringMap(catalogHeaders)
|
||||
for key, value := range profile.CustomHeaders {
|
||||
// Header names are case-insensitive. Preserve the catalog spelling while
|
||||
// replacing its value so request construction cannot see two colliding
|
||||
// map entries whose eventual winner depends on iteration order.
|
||||
existingKey := ""
|
||||
for candidate := range merged {
|
||||
if strings.EqualFold(candidate, key) {
|
||||
existingKey = candidate
|
||||
break
|
||||
}
|
||||
}
|
||||
if existingKey != "" {
|
||||
merged[existingKey] = value
|
||||
continue
|
||||
}
|
||||
merged[key] = value
|
||||
}
|
||||
profile.CustomHeaders = merged
|
||||
} else if strings.EqualFold(strings.TrimSpace(descriptor.ID), "aimlapi") && !canonicalCatalogEndpoint {
|
||||
// AIMLAPI attribution is owned by the catalog endpoint. A profile can retain
|
||||
// those generated headers after its base URL is edited; strip their names
|
||||
// before sending requests to an arbitrary staging/proxy host while preserving
|
||||
// unrelated headers explicitly supplied by the user.
|
||||
for profileKey := range profile.CustomHeaders {
|
||||
for catalogKey := range descriptor.CustomHeaders {
|
||||
if strings.EqualFold(profileKey, catalogKey) {
|
||||
delete(profile.CustomHeaders, profileKey)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func sameBaseURL(left string, right string) bool {
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/modelregistry"
|
||||
"github.com/Gitlawb/zero/internal/providercatalog"
|
||||
)
|
||||
|
||||
func TestResolveAppliesLayerPrecedence(t *testing.T) {
|
||||
@@ -554,7 +555,7 @@ func TestResolveReplacesMCPServerOverlayCollections(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveMCPServerLayersCanClearAndReenable(t *testing.T) {
|
||||
func TestResolveMCPServerLayersCannotReenableUserDisabled(t *testing.T) {
|
||||
userPath := writeConfig(t, `{
|
||||
"mcp": {
|
||||
"servers": {
|
||||
@@ -588,9 +589,11 @@ func TestResolveMCPServerLayersCanClearAndReenable(t *testing.T) {
|
||||
}
|
||||
|
||||
docs := resolved.MCP.Servers["docs"]
|
||||
if docs.Disabled {
|
||||
t.Fatalf("docs.Disabled = true, want project layer to re-enable")
|
||||
// A user-level disable is sticky: project config must not re-enable it.
|
||||
if !docs.Disabled {
|
||||
t.Fatal("docs.Disabled = false, want user-level disable to remain sticky")
|
||||
}
|
||||
// Non-disable fields from the project layer still merge normally.
|
||||
if len(docs.Args) != 0 {
|
||||
t.Fatalf("docs.Args = %#v, want project layer to clear inherited args", docs.Args)
|
||||
}
|
||||
@@ -599,6 +602,125 @@ func TestResolveMCPServerLayersCanClearAndReenable(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeMCPStickyDisableReenable(t *testing.T) {
|
||||
disabled := MCPServerConfig{Disabled: true, disabledSet: true}
|
||||
|
||||
// A lower-trust (project) layer must not lift a sticky user disable.
|
||||
project := MCPServerConfig{Disabled: false, disabledSet: true}
|
||||
if got := mergeMCPServer(disabled, project, false); !got.Disabled {
|
||||
t.Fatal("project layer re-enabled a sticky user disable")
|
||||
}
|
||||
|
||||
// The user scope (explicit mcp enable command / CLI override) may re-enable.
|
||||
override := MCPServerConfig{Disabled: false, disabledSet: true}
|
||||
if got := mergeMCPServer(disabled, override, true); got.Disabled {
|
||||
t.Fatal("user-scope override must be able to re-enable")
|
||||
}
|
||||
|
||||
// A user-scope disable stacked on a project attempt still sticks.
|
||||
if got := mergeMCPServer(disabled, project, false); !got.Disabled {
|
||||
t.Fatal("sticky disable must survive a project re-enable attempt")
|
||||
}
|
||||
|
||||
// Baseline: a project layer may disable a default-enabled server when no
|
||||
// higher-trust scope has explicitly set disabled (the trust boundary must
|
||||
// not block legitimate project disables). Regression guard for the
|
||||
// disabledSet-before-check ordering bug.
|
||||
defaultEnabled := MCPServerConfig{Disabled: false, disabledSet: false}
|
||||
projectDisable := MCPServerConfig{Disabled: true, disabledSet: true}
|
||||
if got := mergeMCPServer(defaultEnabled, projectDisable, false); !got.Disabled {
|
||||
t.Fatal("project layer should be able to disable a default-enabled server")
|
||||
}
|
||||
|
||||
// And an empty project layer must not flip an unconfigured server to disabled.
|
||||
if got := mergeMCPServer(MCPServerConfig{}, MCPServerConfig{}, false); got.Disabled {
|
||||
t.Fatal("an empty project layer must not disable an unconfigured server")
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolveMCPCannotReenableUserDisabled exercises the trust boundary at the
|
||||
// actual ResolveMCP entry point: a project config (project scope) must not lift
|
||||
// a disable the user set in their higher-trust user config.
|
||||
func TestResolveMCPCannotReenableUserDisabled(t *testing.T) {
|
||||
userPath := writeConfig(t, `{
|
||||
"mcp": {
|
||||
"servers": {
|
||||
"docs": {
|
||||
"type": "stdio",
|
||||
"command": "docs-mcp",
|
||||
"disabled": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}`)
|
||||
// Project config tries to re-enable it.
|
||||
projectPath := writeConfig(t, `{
|
||||
"mcpServers": {
|
||||
"docs": {
|
||||
"disabled": false
|
||||
}
|
||||
}
|
||||
}`)
|
||||
|
||||
resolved, err := ResolveMCP(ResolveOptions{
|
||||
UserConfigPath: userPath,
|
||||
ProjectConfigPath: projectPath,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveMCP() error = %v", err)
|
||||
}
|
||||
|
||||
docs, ok := resolved.Servers["docs"]
|
||||
if !ok {
|
||||
t.Fatal("docs server missing from resolved config")
|
||||
}
|
||||
if !docs.Disabled {
|
||||
t.Fatalf("docs.Disabled = false, want user-level disable to remain sticky across project re-enable")
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolveMCPUserLiftsProjectDisabled asserts the reverse direction is open:
|
||||
// a user config (higher-trust scope) re-enabling a server the project config
|
||||
// disabled must be permitted, since user > project in the trust hierarchy.
|
||||
func TestResolveMCPUserLiftsProjectDisabled(t *testing.T) {
|
||||
// Project config disables the server.
|
||||
projectPath := writeConfig(t, `{
|
||||
"mcpServers": {
|
||||
"docs": {
|
||||
"type": "stdio",
|
||||
"command": "docs-mcp",
|
||||
"disabled": true
|
||||
}
|
||||
}
|
||||
}`)
|
||||
// User config re-enables it.
|
||||
userPath := writeConfig(t, `{
|
||||
"mcp": {
|
||||
"servers": {
|
||||
"docs": {
|
||||
"disabled": false
|
||||
}
|
||||
}
|
||||
}
|
||||
}`)
|
||||
|
||||
resolved, err := ResolveMCP(ResolveOptions{
|
||||
UserConfigPath: userPath,
|
||||
ProjectConfigPath: projectPath,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveMCP() error = %v", err)
|
||||
}
|
||||
|
||||
docs, ok := resolved.Servers["docs"]
|
||||
if !ok {
|
||||
t.Fatal("docs server missing from resolved config")
|
||||
}
|
||||
if docs.Disabled {
|
||||
t.Fatalf("docs.Disabled = true, want user scope to lift the project-level disable")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveRejectsDuplicateMCPRootAliases(t *testing.T) {
|
||||
path := writeConfig(t, `{
|
||||
"mcpServers": {
|
||||
@@ -1374,6 +1496,71 @@ func TestResolveProviderProfileExtendedJSONAliases(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyCatalogDescriptorFoldsCustomHeadersCaseInsensitively(t *testing.T) {
|
||||
t.Setenv("AIMLAPI_PARTNER_ID", "part_env")
|
||||
descriptor, err := providercatalog.Require("aimlapi")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
profile := ProviderProfile{
|
||||
BaseURL: descriptor.DefaultBaseURL,
|
||||
CustomHeaders: map[string]string{
|
||||
"x-aimlapi-partner-id": "part_override",
|
||||
},
|
||||
}
|
||||
|
||||
applyCatalogDescriptor(&profile, descriptor, true)
|
||||
|
||||
if got := profile.CustomHeaders["X-AIMLAPI-Partner-ID"]; got != "part_override" {
|
||||
t.Fatalf("canonical header value = %q, want override", got)
|
||||
}
|
||||
if _, duplicate := profile.CustomHeaders["x-aimlapi-partner-id"]; duplicate {
|
||||
t.Fatalf("case-colliding header survived merge: %#v", profile.CustomHeaders)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyCatalogDescriptorUsesAimlapiPartnerEnvironmentOverride(t *testing.T) {
|
||||
t.Setenv("AIMLAPI_PARTNER_ID", "part_env")
|
||||
descriptor, err := providercatalog.Require("aimlapi")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
profile := ProviderProfile{BaseURL: descriptor.DefaultBaseURL}
|
||||
|
||||
applyCatalogDescriptor(&profile, descriptor, true)
|
||||
|
||||
if got := profile.CustomHeaders["X-AIMLAPI-Partner-ID"]; got != "part_env" {
|
||||
t.Fatalf("partner header = %q, want part_env", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyCatalogDescriptorStripsAimlapiAttributionFromRetargetedProfile(t *testing.T) {
|
||||
descriptor, err := providercatalog.Require("aimlapi")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
profile := ProviderProfile{
|
||||
BaseURL: "https://proxy.example.test/v1",
|
||||
CustomHeaders: map[string]string{
|
||||
"x-aimlapi-partner-id": "persisted-partner",
|
||||
"X-AIMLAPI-Integration-Repo": "Gitlawb/zero",
|
||||
"X-AIMLAPI-Integration-Version": "zero",
|
||||
"X-Environment": "staging",
|
||||
},
|
||||
}
|
||||
|
||||
applyCatalogDescriptor(&profile, descriptor, true)
|
||||
|
||||
for key := range profile.CustomHeaders {
|
||||
if strings.HasPrefix(strings.ToLower(key), "x-aimlapi-") {
|
||||
t.Fatalf("catalog attribution survived retargeting: %#v", profile.CustomHeaders)
|
||||
}
|
||||
}
|
||||
if profile.CustomHeaders["X-Environment"] != "staging" {
|
||||
t.Fatalf("user header was removed: %#v", profile.CustomHeaders)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveProviderProfileParseThinkTagsFalseAlias(t *testing.T) {
|
||||
path := writeConfig(t, `{
|
||||
"activeProvider": "custom",
|
||||
|
||||
+120
-35
@@ -10,16 +10,19 @@ matters for months, so the contract is written down here.
|
||||
|
||||
48 tasks across seven classes:
|
||||
|
||||
| class | count | oracle | tier |
|
||||
|-----------|-------|-----------------------------|-------------|
|
||||
| nav | 10 | none | latency |
|
||||
| edit | 10 | substring grep | correctness |
|
||||
| fix | 8 | scoped `go test -run <name>` | correctness |
|
||||
| refactor | 6 | `go build ./...` | build |
|
||||
| longproc | 4 | none | latency |
|
||||
| longctx | 4 | none | latency |
|
||||
| parallel | 6 | none | latency |
|
||||
| **total** | **48**| | |
|
||||
| class | count | oracle | tier |
|
||||
|-----------|-------|-----------------------------------------|-------------|
|
||||
| nav | 10 | grep on captured final answer | correctness |
|
||||
| edit | 10 | stamped `oracle_test.go` + `go test` | correctness |
|
||||
| fix | 8 | scoped `go test -run <name>` | correctness |
|
||||
| refactor | 6 | stamped `oracle_test.go` + `go test` | correctness |
|
||||
| longproc | 4 | none | latency |
|
||||
| longctx | 4 | none | latency |
|
||||
| parallel | 6 | none | latency |
|
||||
| **total** | **48**| | |
|
||||
|
||||
Tier counts: **correctness 34** (10 edit + 8 fix + 10 nav + 6 refactor),
|
||||
**build 0**, **latency 14** (4 longproc + 4 longctx + 6 parallel).
|
||||
|
||||
## Oracle tiers
|
||||
|
||||
@@ -27,16 +30,21 @@ Pass/fail is reported per tier so the report cannot be misread as a blanket
|
||||
correctness verdict. The tier is decided per task from oracle presence and the
|
||||
manifest's `buildOnlyClasses` list:
|
||||
|
||||
- **Correctness** (18 tasks: 10 edit + 8 fix) — a positive oracle (substring
|
||||
grep that the requested change landed, or a scoped `go test -run` that the
|
||||
bug fix passes). This is the only pass rate that can move with model quality:
|
||||
`tasksVerified` / `tasksPassed` / `correctnessPassRate`.
|
||||
- **Build-only** (6 tasks: refactor) — `go build ./...` proves the edit
|
||||
compiles, not that the refactor achieved its goal (a no-op refactor passes).
|
||||
The manifest declares `buildOnlyClasses: ["refactor"]`. Reported as
|
||||
`buildCheckedTasks` / `buildPassedTasks` / `buildPassRate`, never in
|
||||
`correctnessPassRate`.
|
||||
- **Latency-only** (24 tasks: nav, longproc, longctx, parallel) — no
|
||||
- **Correctness** (34 tasks: edit, fix, nav, refactor) — a positive oracle. For
|
||||
edit/refactor the harness stamps an `oracle_test.go` (from the task's
|
||||
`oracleTest` field) into the fixture copy **after** the agent run, then runs
|
||||
`go test ./...`: the Go compiler is the structural verifier, so a no-op
|
||||
refactor, a missing field, or a reworded-but-not-removed line fails to
|
||||
compile/run. fix uses a scoped `go test -run <name>`. nav greps the agent's
|
||||
captured final answer (`.zero-answer.txt`) for determinable facts. This is
|
||||
the only pass rate that can move with model quality: `tasksVerified` /
|
||||
`tasksPassed` / `correctnessPassRate`.
|
||||
- **Build-only** (0 tasks) — the tier is empty. refactor used to live here with
|
||||
a non-positive `go build ./...` (a no-op refactor passed); PR #712 gave it a
|
||||
structural oracle, graduating it to correctness. `buildOnlyClasses` is `[]`.
|
||||
`buildCheckedTasks` / `buildPassedTasks` / `buildPassRate` are reported as 0
|
||||
and never in `correctnessPassRate`.
|
||||
- **Latency-only** (14 tasks: longproc, longctx, parallel) — no
|
||||
`verificationCommand`. An exit 0 only proves the turn ran, not that the answer
|
||||
was right. They contribute to latency and span attribution only and are
|
||||
counted in `latencyOnlyTasks`, never in any pass rate.
|
||||
@@ -45,22 +53,89 @@ A task's tier is driven by oracle **presence** first: a task with no
|
||||
`verificationCommand` is latency-only even if its class is listed in
|
||||
`buildOnlyClasses`, so a missing oracle can never silently pass on exit 0.
|
||||
|
||||
## Known limitations (deferred)
|
||||
### How the stamped oracle and captured answer work
|
||||
|
||||
These are accepted for the Phase 0 baseline and tracked in a follow-up issue;
|
||||
they do not block the baseline because the tier split keeps the report honest:
|
||||
The production runner (`NewTurnExecRunner` in `turn_bench.go`) copies the fixture
|
||||
into an isolated temp dir, runs `zero exec --trace`, then — before running the
|
||||
`verificationCommand` — stamps two files into the copy:
|
||||
|
||||
- The edit grep oracles are substring checks. A rename oracle asserts the new
|
||||
name is present AND the old name is gone (compound `bash -c`), but an
|
||||
add-field oracle (`grep -R Label .`) only proves the string appears, not that
|
||||
it landed on the right struct. Strengthening these to `go vet`/`go build` +
|
||||
structural assertions is follow-up work.
|
||||
- The refactor build oracle is non-positive (a no-op refactor passes). A
|
||||
structural verifier that proves the refactor happened is follow-up work;
|
||||
until then refactor is `buildPassRate`, not `correctnessPassRate`.
|
||||
- The 24 read-only tasks have no oracle. Adding deterministic oracles for nav
|
||||
(diff against an expected answer) and documenting longproc/longctx/parallel
|
||||
as permanently latency-only is follow-up work.
|
||||
- `oracle_test.go` from `task.OracleTest` (edit/refactor). It is stamped *after*
|
||||
the agent run so it can't interfere with the agent's own `go build`/`go test`
|
||||
during the task (refactor-03's `package zeroapp` test would break a pre-rename
|
||||
build) and can't be pre-seen or tampered with. The `verificationCommand`
|
||||
(`go test ./...`) then compiles and runs it.
|
||||
- `.zero-answer.txt` from the `{"type":"final","text":...}` event in the
|
||||
stream-json output (nav). The `verificationCommand` is a compound `bash -c`
|
||||
grep requiring the determinable facts (e.g. nav-09 requires the answer to
|
||||
mention `port`, `name`, and `retries`). An empty file (no `final` event) makes
|
||||
the grep fail, which is the correct outcome for a run that produced no answer.
|
||||
|
||||
The `oracleTest` source uses compile-time references where the assertion is
|
||||
structural (`var _ = Config{}.Label` for edit-03, `var _ = formatGreeting` for
|
||||
refactor-01) and behavioral `Test…` functions where the assertion is a value
|
||||
(edit-04 `Version != "1.1.0"`, edit-10 `greet("x") != "hello, world"`, edit-09
|
||||
`load("nonexistent")` wrapping). Tasks whose oracle is a plain grep/build with
|
||||
no stamped test (edit-02, refactor-02) carry no `oracleTest`.
|
||||
|
||||
## Known limitations
|
||||
|
||||
These are accepted for the Phase 0 baseline; the tier split keeps the report
|
||||
honest where they apply:
|
||||
|
||||
- **longproc / longctx / parallel are permanently latency-only.** There is no
|
||||
deterministic oracle for "report whether the build succeeds", "summarize a
|
||||
large generated file", or "report the first line of each of six files" — the
|
||||
answer is free-form prose and a contains-check would rubber-stamp. They stay
|
||||
in `latencyOnlyTasks` and never enter a pass rate.
|
||||
- **nav count tasks (nav-01, nav-04, nav-05, nav-08) require a structured,
|
||||
line-anchored `count: N` token with the fixture's exact expected count AND a
|
||||
workspace-derived fact that proves the agent actually inspected the fixture —
|
||||
not the count alone.** Their prompts ask the agent to state the count as
|
||||
`count: N` on its own line and to name the thing(s) counted, and the oracle
|
||||
greps for an anchored `^count: N$` with the exact expected N plus a named fact:
|
||||
nav-01 requires `count: 5` and names `main.go`, `config.json`, and `README.md`
|
||||
(the three the oracle anchors on — a correct answer lists all five files, but
|
||||
the verification contract only requires those three be named); nav-04 names
|
||||
the one test function (`TestGreet` from `main_test.go`); nav-05 names the file
|
||||
holding the one TODO (`main.go`); nav-08 names both stdlib imports (`fmt` from
|
||||
`main.go`, `testing` from `main_test.go`). A substring like "the count: 1" or a
|
||||
wrong count like "count: 3" fails the anchor; a bare "count: 1" with no named
|
||||
fact fails the inspection check. Non-zero ground truth alone was not enough —
|
||||
it closes the always-guess-zero game but opens an always-guess-one game (an
|
||||
agent that never opens the workspace and emits "count: 1" would pass a
|
||||
count-only oracle). The named-fact requirement closes that surface: a blind
|
||||
count is no longer sufficient, the agent must commit to a fact only readable
|
||||
from the fixture. nav-04 and nav-05 were given a real `TestGreet` and a real
|
||||
TODO so the named fact exists to require; nav-08 stays at count 0 (a real
|
||||
third-party import would need a network fetch and break the offline suite), so
|
||||
its named fact is the stdlib imports the agent must enumerate across every file
|
||||
to conclude 0 third-party. The named facts are not all equally tight. nav-04's
|
||||
`TestGreet` is a specific, non-generic identifier a blind agent cannot produce
|
||||
without reading `main.go` to learn a greet function exists, so it fully closes
|
||||
the blind-count-one surface. nav-05's `main.go` is the conventional Go
|
||||
entrypoint and thus the most guessable filename, so it *reduces* rather than
|
||||
fully closes that surface — a blind "count: 1, in main.go" can still pass.
|
||||
Accepted for a Phase 0 baseline: the blind floor on nav-05 is below 100% (a
|
||||
bare "count: 1" fails), so a model still moves the pass rate by actually
|
||||
inspecting; strengthening it further (e.g. requiring the TODO's surrounding
|
||||
symbol) would trade output freedom for determinism and risks false-failing a
|
||||
terse-but-correct answer. nav-08's `testing` half is the tighter of its two:
|
||||
`fmt` alone is the most common Go stdlib import and could be named blind, but
|
||||
`testing` only appears in the test file, so naming it proves the agent read more
|
||||
than `main.go`. This trades a little output freedom for determinism (a loose
|
||||
prose-contains on "0"/"1" would match almost any answer, and an unanchored
|
||||
`count: N` would match inside a longer line).
|
||||
- **nav-03 / nav-06 / nav-07 are lenient contains-oracles on real facts.** A
|
||||
wrong-but-plausible answer that happens to mention the real symbols (e.g.
|
||||
"Config" and "MaxRetries" for nav-06) could in principle pass. Accepted for a
|
||||
latency-first baseline; the oracle still catches an agent that didn't look at
|
||||
the code at all.
|
||||
- **Rename oracles use a declaration-site negative grep** (`! grep -RIn
|
||||
'MaxRetries =' .`) rather than a broad `! grep -R MaxRetries .`, so a correct
|
||||
rename that leaves a doc-comment mention of the old name is not false-failed.
|
||||
The negative grep is still required: without it, a no-op that *adds* the new
|
||||
name and *keeps* the old would pass the compile-time `var _ = RetryLimit` and
|
||||
`go test`.
|
||||
|
||||
## Fixtures
|
||||
|
||||
@@ -68,4 +143,14 @@ Each task's `workspaceFixture` points at a small self-contained workspace under
|
||||
`testdata/` so the suite runs offline and repeatably. Mutating tasks (edit,
|
||||
fix, refactor) run against a per-invocation **copy** of their fixture, so the
|
||||
checked-in fixtures stay clean and one task's edits can't bleed into the next
|
||||
iteration or a later task.
|
||||
iteration or a later task.
|
||||
|
||||
Each mutating fixture (and nav) carries a `go.mod` (`module <pkg>; go 1.22`) so
|
||||
`go test ./...` / `go build ./...` oracles work in the copy: `copyFixture`
|
||||
creates a unique, 0700 parent under the system temp root
|
||||
(`os.MkdirTemp(os.TempDir(), "zero-turn-bench-*")`) and nests the copy beneath
|
||||
it, so the copy is a *grandchild* of the temp root. Go ignores `go.mod` in
|
||||
direct children of the system temp dir (a hijack guard), so the copy must sit
|
||||
one level below that root or every compiler-backed oracle would fail with
|
||||
"cannot find main module". The unique parent also avoids a predictable
|
||||
shared directory and is removed wholesale on cleanup.
|
||||
@@ -1,62 +1,545 @@
|
||||
{
|
||||
"id": "zero-baseline-turn",
|
||||
"name": "Zero per-turn baseline (Phase 0)",
|
||||
"description": "Latency-first baseline: measures where a turn spends wall time, with pass/fail reported per oracle tier so it cannot be misread as a blanket correctness verdict. Read-only classes (nav, longproc, longctx, parallel) carry no verification oracle — a zero-exit only proves the turn ran, so they count as latency-only and never in any pass rate. Correctness classes (edit, fix) carry a positive oracle (substring grep or a scoped `go test`) and are the only tasks in correctnessPassRate. Build-only classes (refactor) carry `go build ./...`, which proves the edit compiles but not that the refactor achieved its goal, so they are reported as buildPassRate and excluded from correctnessPassRate.",
|
||||
"buildOnlyClasses": ["refactor"],
|
||||
"description": "Latency-first baseline: measures where a turn spends wall time, with pass/fail reported per oracle tier so it cannot be misread as a blanket correctness verdict. Correctness classes (edit, fix, nav, refactor) carry a positive oracle: edit/refactor stamp an oracle_test.go compiled by `go test ./...` (the Go compiler is the structural verifier \u2014 a no-op refactor or a missing field fails to compile), fix runs a scoped `go test -run`, and nav greps the agent's captured final answer (.zero-answer.txt) for determinable facts. tasksVerified/tasksPassed/correctnessPassRate cover exactly these. Latency-only classes (longproc, longctx, parallel) carry no verification oracle \u2014 a zero-exit only proves the turn ran, so they count as latency-only and never in any pass rate. The build tier is empty: refactor graduated to correctness once its oracle became structural-positive. See MANIFEST.md for the per-task oracle specs and known limitations.",
|
||||
"buildOnlyClasses": [],
|
||||
"tasks": [
|
||||
{"id": "nav-01", "name": "list repo files", "class": "nav", "prompt": "List the files in this repository and report their count. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
|
||||
{"id": "nav-02", "name": "find a function", "class": "nav", "prompt": "Find the function named 'greet' and report the file and line it is defined on. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
|
||||
{"id": "nav-03", "name": "summarize module", "class": "nav", "prompt": "Read main.go and summarize what this program does in two sentences. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
|
||||
{"id": "nav-04", "name": "count tests", "class": "nav", "prompt": "Count the test functions (functions whose names start with Test) in this repository and report the number. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
|
||||
{"id": "nav-05", "name": "locate a string", "class": "nav", "prompt": "Find every occurrence of the string 'TODO' in the repository and report the files. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
|
||||
{"id": "nav-06", "name": "describe package", "class": "nav", "prompt": "Describe the exported API of this package by reading the source. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
|
||||
{"id": "nav-07", "name": "trace a call", "class": "nav", "prompt": "Trace which function calls which other function starting from main. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
|
||||
{"id": "nav-08", "name": "find imports", "class": "nav", "prompt": "List every third-party import used by this repository. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
|
||||
{"id": "nav-09", "name": "report config", "class": "nav", "prompt": "Read config.json and report the keys it contains. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
|
||||
{"id": "nav-10", "name": "doc lookup", "class": "nav", "prompt": "Find the doc comment on the greet function and quote it. Do not modify anything.", "workspaceFixture": "../testdata/nav"},
|
||||
|
||||
{"id": "edit-01", "name": "rename a constant", "class": "edit", "prompt": "Rename the constant MaxRetries to RetryLimit everywhere it appears.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R RetryLimit . && ! grep -R MaxRetries ."]},
|
||||
{"id": "edit-02", "name": "fix a typo", "class": "edit", "prompt": "Fix the typo in the comment above the greet function ('receieves' should be 'receives').", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "! grep -R receieves ."]},
|
||||
{"id": "edit-03", "name": "add a field", "class": "edit", "prompt": "Add a string field named Label to the Config struct in main.go.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "Label", "."]},
|
||||
{"id": "edit-04", "name": "bump a version", "class": "edit", "prompt": "Change the version string in version.go from 1.0.0 to 1.1.0.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R 1.1.0 . && ! grep -R 1.0.0 ."]},
|
||||
{"id": "edit-05", "name": "remove a log line", "class": "edit", "prompt": "Remove the line that prints 'debug: starting'.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "! grep -R 'debug: starting' ."]},
|
||||
{"id": "edit-06", "name": "add a license header", "class": "edit", "prompt": "Add a comment line '// SPDX-License-Identifier: MIT' at the top of main.go.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "SPDX-License-Identifier", "."]},
|
||||
{"id": "edit-07", "name": "change a default", "class": "edit", "prompt": "Change the default value of the Port constant from 8080 to 9090.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R 9090 . && ! grep -R 8080 ."]},
|
||||
{"id": "edit-08", "name": "add a getter", "class": "edit", "prompt": "Add a method func (c *Config) GetLabel() string that returns c.Label.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["bash", "-c", "grep -R 'func (c *Config) GetLabel' ."]},
|
||||
{"id": "edit-09", "name": "wrap an error", "class": "edit", "prompt": "In the load function, wrap the returned error with fmt.Errorf so it includes 'load failed'.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "load failed", "."]},
|
||||
{"id": "edit-10", "name": "update a message", "class": "edit", "prompt": "Change the printed greeting from 'hello' to 'hello, world'.", "workspaceFixture": "../testdata/edit", "verificationCommand": ["grep", "-R", "hello, world", "."]},
|
||||
|
||||
{"id": "fix-01", "name": "off-by-one", "class": "fix", "prompt": "In bugs.go the Sum function has an off-by-one error that skips the last element; fix the loop so it sums all elements.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestSumIncludesLastElement", "./..."]},
|
||||
{"id": "fix-02", "name": "nil deref", "class": "fix", "prompt": "In bugs.go DefaultConfig returns nil and Port() dereferences it; make DefaultConfig return a valid Config with Port 8080.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestPortReturnsDefault", "./..."]},
|
||||
{"id": "fix-03", "name": "wrong operator", "class": "fix", "prompt": "In bugs.go Max uses the wrong inequality and returns the smaller value; fix it to return the larger.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestMaxReturnsLarger", "./..."]},
|
||||
{"id": "fix-04", "name": "truncated read", "class": "fix", "prompt": "In bugs.go ReadFirst returns only a single character instead of the full content; fix it to return the whole string.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestReadFirstReturnsFullContent", "./..."]},
|
||||
{"id": "fix-05", "name": "swapped args", "class": "fix", "prompt": "In bugs.go Divide swaps its operands; fix the order so Divide(10,2) returns 5.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestDivideOrder", "./..."]},
|
||||
{"id": "fix-06", "name": "missing return", "class": "fix", "prompt": "In bugs.go Classify is missing a return for the high case; add it so Classify(500) returns \"high\".", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestClassifyHigh", "./..."]},
|
||||
{"id": "fix-07", "name": "wrong label prefix", "class": "fix", "prompt": "In bugs.go Label returns 'admin-<name>' but the test expects 'user-<name>'; change the prefix from 'admin-' to 'user-'.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-run", "TestLabelUsesString", "./..."]},
|
||||
{"id": "fix-08", "name": "race on counter", "class": "fix", "prompt": "In bugs.go the global counter is incremented without synchronization; guard it with a mutex.", "workspaceFixture": "../testdata/fix", "verificationCommand": ["go", "test", "-race", "-run", "TestCounterRace", "./..."]},
|
||||
|
||||
{"id": "refactor-01", "name": "extract helper", "class": "refactor", "prompt": "Extract the duplicated greeting logic into a single helper function and call it from both sites.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
|
||||
{"id": "refactor-02", "name": "split a file", "class": "refactor", "prompt": "Split main.go into two files: one for Config, one for the rest, keeping the package building.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
|
||||
{"id": "refactor-03", "name": "rename package", "class": "refactor", "prompt": "Rename the package from app to zeroapp everywhere and keep it building.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
|
||||
{"id": "refactor-04", "name": "introduce a type", "class": "refactor", "prompt": "Introduce a named type for the map[string]int used in main.go and update its users.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
|
||||
{"id": "refactor-05", "name": "inline a wrapper", "class": "refactor", "prompt": "Inline the thin wrapper function into its single caller and remove it.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
|
||||
{"id": "refactor-06", "name": "consolidate errors", "class": "refactor", "prompt": "Consolidate the repeated error-construction blocks into a single helper.", "workspaceFixture": "../testdata/refactor", "verificationCommand": ["go", "build", "./..."]},
|
||||
|
||||
{"id": "longproc-01", "name": "long build", "class": "longproc", "prompt": "Run the build command and report whether it succeeds. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
|
||||
{"id": "longproc-02", "name": "long test", "class": "longproc", "prompt": "Run go test ./... and report the result. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
|
||||
{"id": "longproc-03", "name": "bench run", "class": "longproc", "prompt": "Run go test -bench=. -benchtime=1x and report the benchmark numbers. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
|
||||
{"id": "longproc-04", "name": "vet run", "class": "longproc", "prompt": "Run go vet ./... and report the output. Do not modify anything.", "workspaceFixture": "../testdata/longproc"},
|
||||
|
||||
{"id": "longctx-01", "name": "summarize large file", "class": "longctx", "prompt": "Read the large generated file big.go and summarize its structure. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
|
||||
{"id": "longctx-02", "name": "find in large file", "class": "longctx", "prompt": "In big.go, find every function that returns an error and list them. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
|
||||
{"id": "longctx-03", "name": "cross-file question", "class": "longctx", "prompt": "Read all .go files and report which two functions share the same name across files. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
|
||||
{"id": "longctx-04", "name": "trace through large file", "class": "longctx", "prompt": "Starting from Handle, trace the call path through big.go and report the chain. Do not modify anything.", "workspaceFixture": "../testdata/longctx"},
|
||||
|
||||
{"id": "parallel-01", "name": "read six files", "class": "parallel", "prompt": "Read a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt, then report the first line of each. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
|
||||
{"id": "parallel-02", "name": "grep six patterns", "class": "parallel", "prompt": "Search for 'alpha', 'beta', 'gamma', 'delta', 'epsilon', and 'zeta' across the repo and report which files contain each. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
|
||||
{"id": "parallel-03", "name": "list six dirs", "class": "parallel", "prompt": "List the contents of dir1, dir2, dir3, dir4, dir5, and dir6. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
|
||||
{"id": "parallel-04", "name": "stat six files", "class": "parallel", "prompt": "Report the line count of a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
|
||||
{"id": "parallel-05", "name": "read configs", "class": "parallel", "prompt": "Read config1.json, config2.json, config3.json, config4.json, config5.json, and config6.json and report each top-level key. Do not modify anything.", "workspaceFixture": "../testdata/parallel"},
|
||||
{"id": "parallel-06", "name": "summarize six", "class": "parallel", "prompt": "Read the six .txt files and give a one-line summary of each. Do not modify anything.", "workspaceFixture": "../testdata/parallel"}
|
||||
{
|
||||
"id": "nav-01",
|
||||
"name": "list repo files",
|
||||
"class": "nav",
|
||||
"prompt": "List the files in this repository and report their count. State the count as 'count: N' on its own line. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/nav",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"grep -iq 'main.go' .zero-answer.txt && grep -iq 'config.json' .zero-answer.txt && grep -iq 'readme' .zero-answer.txt && grep -iqE '^count:[[:space:]]*5$' .zero-answer.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nav-02",
|
||||
"name": "find a function",
|
||||
"class": "nav",
|
||||
"prompt": "Find the function named 'greet' and report the file and line it is defined on. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/nav",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"grep -iq 'main.go' .zero-answer.txt && grep -iqE '\\b1[5-7]\\b' .zero-answer.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nav-03",
|
||||
"name": "summarize module",
|
||||
"class": "nav",
|
||||
"prompt": "Read main.go and summarize what this program does in two sentences. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/nav",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"grep -iqE 'greet|hello|Config' .zero-answer.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nav-04",
|
||||
"name": "count tests",
|
||||
"class": "nav",
|
||||
"prompt": "Count the test functions (functions whose names start with Test) in this repository. Name each one you find and report the number. State the count as 'count: N' on its own line. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/nav",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"grep -iq 'TestGreet' .zero-answer.txt && grep -iqE '^count:[[:space:]]*1$' .zero-answer.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nav-05",
|
||||
"name": "locate a string",
|
||||
"class": "nav",
|
||||
"prompt": "Find every occurrence of the string 'TODO' in the repository and report the file(s) they appear in. State the count as 'count: N' on its own line. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/nav",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"grep -iq 'main.go' .zero-answer.txt && grep -iqE '^count:[[:space:]]*1$' .zero-answer.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nav-06",
|
||||
"name": "describe package",
|
||||
"class": "nav",
|
||||
"prompt": "Describe the exported API of this package by reading the source. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/nav",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"grep -iq config .zero-answer.txt && grep -iq maxretries .zero-answer.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nav-07",
|
||||
"name": "trace a call",
|
||||
"class": "nav",
|
||||
"prompt": "Trace which function calls which other function starting from main. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/nav",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"grep -iq main .zero-answer.txt && grep -iq greet .zero-answer.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nav-08",
|
||||
"name": "find imports",
|
||||
"class": "nav",
|
||||
"prompt": "List every import used by this repository and classify each as standard library or third-party. State the number of third-party (non-stdlib) imports as 'count: N' on its own line. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/nav",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"grep -iq 'fmt' .zero-answer.txt && grep -iq 'testing' .zero-answer.txt && grep -iqE '^count:[[:space:]]*0$' .zero-answer.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nav-09",
|
||||
"name": "report config",
|
||||
"class": "nav",
|
||||
"prompt": "Read config.json and report the keys it contains. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/nav",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"grep -iq port .zero-answer.txt && grep -iq name .zero-answer.txt && grep -iq retries .zero-answer.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nav-10",
|
||||
"name": "doc lookup",
|
||||
"class": "nav",
|
||||
"prompt": "Find the doc comment on the greet function and quote it. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/nav",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"grep -iq 'greet returns a greeting' .zero-answer.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "edit-01",
|
||||
"name": "rename a constant",
|
||||
"class": "edit",
|
||||
"prompt": "Rename the constant MaxRetries to RetryLimit everywhere it appears.",
|
||||
"workspaceFixture": "../testdata/edit",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"go test ./... && ! grep -RIn 'MaxRetries =' ."
|
||||
],
|
||||
"oracleTest": "package edit\n\nvar _ = RetryLimit\n"
|
||||
},
|
||||
{
|
||||
"id": "edit-02",
|
||||
"name": "fix a typo",
|
||||
"class": "edit",
|
||||
"prompt": "Fix the typo in the comment above the greet function ('receieves' should be 'receives').",
|
||||
"workspaceFixture": "../testdata/edit",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"go test ./... && ! grep -R receieves . && grep -R receives ."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "edit-03",
|
||||
"name": "add a field",
|
||||
"class": "edit",
|
||||
"prompt": "Add a string field named Label to the Config struct in main.go.",
|
||||
"workspaceFixture": "../testdata/edit",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"./..."
|
||||
],
|
||||
"oracleTest": "package edit\n\nvar _ string = Config{}.Label\n"
|
||||
},
|
||||
{
|
||||
"id": "edit-04",
|
||||
"name": "bump a version",
|
||||
"class": "edit",
|
||||
"prompt": "Change the version string in version.go from 1.0.0 to 1.1.0.",
|
||||
"workspaceFixture": "../testdata/edit",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"./..."
|
||||
],
|
||||
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestVersion(t *testing.T) {\n\tif Version != \"1.1.0\" {\n\t\tt.Fatalf(\"Version=%q\", Version)\n\t}\n}\n"
|
||||
},
|
||||
{
|
||||
"id": "edit-05",
|
||||
"name": "remove a log line",
|
||||
"class": "edit",
|
||||
"prompt": "Remove the line that prints 'debug: starting'.",
|
||||
"workspaceFixture": "../testdata/edit",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"./..."
|
||||
],
|
||||
"oracleTest": "package edit\n\nimport (\n\t\"os\"\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestNoDebugPrintln(t *testing.T) {\n\tb, err := os.ReadFile(\"main.go\")\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\t// The greet print is fmt.Println(greet(...)) with no string literal, so a\n\t// remaining fmt.Println(\"...\") call means the debug line was not removed\n\t// (or was only reworded, not deleted).\n\tif strings.Contains(string(b), `fmt.Println(\"`) {\n\t\tt.Fatal(\"main.go still has a fmt.Println string-literal call; the debug line was not removed\")\n\t}\n\t// The task is to remove the line that prints \"debug: starting\", so the\n\t// message text itself must be gone -- this catches a reword onto a different\n\t// call form (fmt.Printf, backticks, log.Println, println) that the\n\t// string-literal check above would miss.\n\tif strings.Contains(string(b), \"debug: starting\") {\n\t\tt.Fatal(\"main.go still prints 'debug: starting'; the line was not removed (only reworded or respelled)\")\n\t}\n}\n"
|
||||
},
|
||||
{
|
||||
"id": "edit-06",
|
||||
"name": "add a license header",
|
||||
"class": "edit",
|
||||
"prompt": "Add a comment line '// SPDX-License-Identifier: MIT' at the top of main.go.",
|
||||
"workspaceFixture": "../testdata/edit",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"./..."
|
||||
],
|
||||
"oracleTest": "package edit\n\nimport (\n\t\"os\"\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestSPDXHeader(t *testing.T) {\n\tb, err := os.ReadFile(\"main.go\")\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\tif !strings.HasPrefix(string(b), \"// SPDX-License-Identifier: MIT\") {\n\t\tt.Fatal(\"main.go must start with the SPDX-License-Identifier: MIT header\")\n\t}\n}\n"
|
||||
},
|
||||
{
|
||||
"id": "edit-07",
|
||||
"name": "change a default",
|
||||
"class": "edit",
|
||||
"prompt": "Change the default value of the Port constant from 8080 to 9090.",
|
||||
"workspaceFixture": "../testdata/edit",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"./..."
|
||||
],
|
||||
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestPort(t *testing.T) {\n\tif Port != 9090 {\n\t\tt.Fatalf(\"Port=%d, want 9090\", Port)\n\t}\n}\n"
|
||||
},
|
||||
{
|
||||
"id": "edit-08",
|
||||
"name": "add a getter",
|
||||
"class": "edit",
|
||||
"prompt": "Add a method func (c *Config) GetName() string that returns c.Name.",
|
||||
"workspaceFixture": "../testdata/edit",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"./..."
|
||||
],
|
||||
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestGetName(t *testing.T) {\n\tc := Config{Name: \"x\"}\n\tif got := c.GetName(); got != \"x\" {\n\t\tt.Fatalf(\"GetName()=%q, want %q\", got, \"x\")\n\t}\n}\n"
|
||||
},
|
||||
{
|
||||
"id": "edit-09",
|
||||
"name": "wrap an error",
|
||||
"class": "edit",
|
||||
"prompt": "In the load function, wrap the returned error with fmt.Errorf so it includes 'load failed'.",
|
||||
"workspaceFixture": "../testdata/edit",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"./..."
|
||||
],
|
||||
"oracleTest": "package edit\n\nimport (\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestLoadWrapsError(t *testing.T) {\n\t_, err := load(\"nonexistent-file-xyz\")\n\tif err == nil {\n\t\tt.Fatal(\"want a non-nil error for a missing file\")\n\t}\n\tif !strings.Contains(err.Error(), \"load failed\") {\n\t\tt.Fatalf(\"err=%v, want it to contain 'load failed'\", err)\n\t}\n}\n"
|
||||
},
|
||||
{
|
||||
"id": "edit-10",
|
||||
"name": "update a message",
|
||||
"class": "edit",
|
||||
"prompt": "Change the printed greeting from 'hello' to 'hello, world'.",
|
||||
"workspaceFixture": "../testdata/edit",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"./..."
|
||||
],
|
||||
"oracleTest": "package edit\n\nimport \"testing\"\n\nfunc TestGreet(t *testing.T) {\n\tif got := greet(\"x\"); got != \"hello, world\" {\n\t\tt.Fatalf(\"greet()=%q, want %q\", got, \"hello, world\")\n\t}\n}\n"
|
||||
},
|
||||
{
|
||||
"id": "fix-01",
|
||||
"name": "off-by-one",
|
||||
"class": "fix",
|
||||
"prompt": "In bugs.go the Sum function has an off-by-one error that skips the last element; fix the loop so it sums all elements.",
|
||||
"workspaceFixture": "../testdata/fix",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"-run",
|
||||
"TestSumIncludesLastElement",
|
||||
"./..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fix-02",
|
||||
"name": "nil deref",
|
||||
"class": "fix",
|
||||
"prompt": "In bugs.go DefaultConfig returns nil and Port() dereferences it; make DefaultConfig return a valid Config with Port 8080.",
|
||||
"workspaceFixture": "../testdata/fix",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"-run",
|
||||
"TestPortReturnsDefault",
|
||||
"./..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fix-03",
|
||||
"name": "wrong operator",
|
||||
"class": "fix",
|
||||
"prompt": "In bugs.go Max uses the wrong inequality and returns the smaller value; fix it to return the larger.",
|
||||
"workspaceFixture": "../testdata/fix",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"-run",
|
||||
"TestMaxReturnsLarger",
|
||||
"./..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fix-04",
|
||||
"name": "truncated read",
|
||||
"class": "fix",
|
||||
"prompt": "In bugs.go ReadFirst returns only a single character instead of the full content; fix it to return the whole string.",
|
||||
"workspaceFixture": "../testdata/fix",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"-run",
|
||||
"TestReadFirstReturnsFullContent",
|
||||
"./..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fix-05",
|
||||
"name": "swapped args",
|
||||
"class": "fix",
|
||||
"prompt": "In bugs.go Divide swaps its operands; fix the order so Divide(10,2) returns 5.",
|
||||
"workspaceFixture": "../testdata/fix",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"-run",
|
||||
"TestDivideOrder",
|
||||
"./..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fix-06",
|
||||
"name": "missing return",
|
||||
"class": "fix",
|
||||
"prompt": "In bugs.go Classify is missing a return for the high case; add it so Classify(500) returns \"high\".",
|
||||
"workspaceFixture": "../testdata/fix",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"-run",
|
||||
"TestClassifyHigh",
|
||||
"./..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fix-07",
|
||||
"name": "wrong label prefix",
|
||||
"class": "fix",
|
||||
"prompt": "In bugs.go Label returns 'admin-<name>' but the test expects 'user-<name>'; change the prefix from 'admin-' to 'user-'.",
|
||||
"workspaceFixture": "../testdata/fix",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"-run",
|
||||
"TestLabelUsesString",
|
||||
"./..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fix-08",
|
||||
"name": "race on counter",
|
||||
"class": "fix",
|
||||
"prompt": "In bugs.go the global counter is incremented without synchronization; guard it with a mutex.",
|
||||
"workspaceFixture": "../testdata/fix",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"-race",
|
||||
"-run",
|
||||
"TestCounterRace",
|
||||
"./..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "refactor-01",
|
||||
"name": "extract helper",
|
||||
"class": "refactor",
|
||||
"prompt": "Extract the duplicated fmt.Sprintf(\"hello, %s\", ...) greeting logic into a single helper function named formatGreeting and call it from both GreetFromConfig and GreetByName.",
|
||||
"workspaceFixture": "../testdata/refactor",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"go test ./... && [ $(grep -c 'hello, %s' main.go) -eq 1 ]"
|
||||
],
|
||||
"oracleTest": "package refactor\n\nvar _ = formatGreeting\n"
|
||||
},
|
||||
{
|
||||
"id": "refactor-02",
|
||||
"name": "split a file",
|
||||
"class": "refactor",
|
||||
"prompt": "Move the Config type into a new file named config.go, keeping the package building.",
|
||||
"workspaceFixture": "../testdata/refactor",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"go test ./... && grep -q 'type Config' config.go && ! grep -q 'type Config' main.go"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "refactor-03",
|
||||
"name": "rename package",
|
||||
"class": "refactor",
|
||||
"prompt": "Rename the package from refactor to zeroapp everywhere and keep it building.",
|
||||
"workspaceFixture": "../testdata/refactor",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"go test ./... && ! grep -R 'package refactor' . && grep -R 'package zeroapp' ."
|
||||
],
|
||||
"oracleTest": "package zeroapp\n\nvar _ = Config{}\n"
|
||||
},
|
||||
{
|
||||
"id": "refactor-04",
|
||||
"name": "introduce a type",
|
||||
"class": "refactor",
|
||||
"prompt": "Introduce a named type `type Stats map[string]int` for the bare map used by stats, and update its users (Record, Lookup).",
|
||||
"workspaceFixture": "../testdata/refactor",
|
||||
"verificationCommand": [
|
||||
"go",
|
||||
"test",
|
||||
"./..."
|
||||
],
|
||||
"oracleTest": "package refactor\n\nimport (\n\t\"os\"\n\t\"regexp\"\n\t\"strings\"\n\t\"testing\"\n)\n\n// TestStatsNamedType proves the bare map was replaced by the named Stats type:\n// the old `var stats = map[string]int` (or `stats := map[string]int{}`) form is\n// gone -- matched with a spacing-tolerant regex so a `var stats=map[string]int`\n// or `stats :=` variant can't dodge it -- and the Stats type is actually\n// referenced in main.go, so a no-op that only adds an unused `type Stats` while\n// keeping the bare map fails. `go test ./...` compiling the fixture proves the\n// type declaration is valid.\nfunc TestStatsNamedType(t *testing.T) {\n\tb, err := os.ReadFile(\"main.go\")\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\tsrc := string(b)\n\tif regexp.MustCompile(`var\\s+stats\\s*=\\s*map\\[string\\]int`).MatchString(src) ||\n\t\tregexp.MustCompile(`stats\\s*:=\\s*map\\[string\\]int`).MatchString(src) {\n\t\tt.Fatal(\"main.go still declares stats as a bare map[string]int; the named Stats type was not adopted\")\n\t}\n\tif !strings.Contains(src, \"Stats\") {\n\t\tt.Fatal(\"main.go does not reference the Stats type; refactor-04 was a no-op\")\n\t}\n}\n"
|
||||
},
|
||||
{
|
||||
"id": "refactor-05",
|
||||
"name": "inline a wrapper",
|
||||
"class": "refactor",
|
||||
"prompt": "Inline the Wrapper function into greetWrapped (its single caller) and remove the Wrapper function.",
|
||||
"workspaceFixture": "../testdata/refactor",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"go test ./... && ! grep -R 'func Wrapper' ."
|
||||
],
|
||||
"oracleTest": "package refactor\n\nimport \"testing\"\n\n// TestGreetWrapped proves greetWrapped still exists and behaves as the inlined\n// Wrapper did: deleting both Wrapper and greetWrapped (or stubbing greetWrapped\n// to return \"\") fails to compile or run, so the oracle can't be passed by\n// removing the target along with its caller. Paired with the ! grep 'func\n// Wrapper' verificationCommand that proves Wrapper itself is gone.\nfunc TestGreetWrapped(t *testing.T) {\n\tif got := greetWrapped(\"x\"); got != \"hello, x\" {\n\t\tt.Fatalf(\"greetWrapped(x) = %q, want %q\", got, \"hello, x\")\n\t}\n}\n"
|
||||
},
|
||||
{
|
||||
"id": "refactor-06",
|
||||
"name": "consolidate errors",
|
||||
"class": "refactor",
|
||||
"prompt": "Consolidate the three error-construction functions (firstError, secondError, buildError) into a single helper func makeLoadError(ctx string) error that returns fmt.Errorf(\"load failed: %s\", ctx), and call it from all three.",
|
||||
"workspaceFixture": "../testdata/refactor",
|
||||
"verificationCommand": [
|
||||
"bash",
|
||||
"-c",
|
||||
"go test ./... && [ $(grep -c 'fmt.Errorf(\"load failed' main.go) -eq 1 ]"
|
||||
],
|
||||
"oracleTest": "package refactor\n\nvar _ = makeLoadError\n"
|
||||
},
|
||||
{
|
||||
"id": "longproc-01",
|
||||
"name": "long build",
|
||||
"class": "longproc",
|
||||
"prompt": "Run the build command and report whether it succeeds. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/longproc"
|
||||
},
|
||||
{
|
||||
"id": "longproc-02",
|
||||
"name": "long test",
|
||||
"class": "longproc",
|
||||
"prompt": "Run go test ./... and report the result. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/longproc"
|
||||
},
|
||||
{
|
||||
"id": "longproc-03",
|
||||
"name": "bench run",
|
||||
"class": "longproc",
|
||||
"prompt": "Run go test -bench=. -benchtime=1x and report the benchmark numbers. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/longproc"
|
||||
},
|
||||
{
|
||||
"id": "longproc-04",
|
||||
"name": "vet run",
|
||||
"class": "longproc",
|
||||
"prompt": "Run go vet ./... and report the output. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/longproc"
|
||||
},
|
||||
{
|
||||
"id": "longctx-01",
|
||||
"name": "summarize large file",
|
||||
"class": "longctx",
|
||||
"prompt": "Read the large generated file big.go and summarize its structure. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/longctx"
|
||||
},
|
||||
{
|
||||
"id": "longctx-02",
|
||||
"name": "find in large file",
|
||||
"class": "longctx",
|
||||
"prompt": "In big.go, find every function that returns an error and list them. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/longctx"
|
||||
},
|
||||
{
|
||||
"id": "longctx-03",
|
||||
"name": "cross-file question",
|
||||
"class": "longctx",
|
||||
"prompt": "Read all .go files and report which two functions share the same name across files. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/longctx"
|
||||
},
|
||||
{
|
||||
"id": "longctx-04",
|
||||
"name": "trace through large file",
|
||||
"class": "longctx",
|
||||
"prompt": "Starting from Handle, trace the call path through big.go and report the chain. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/longctx"
|
||||
},
|
||||
{
|
||||
"id": "parallel-01",
|
||||
"name": "read six files",
|
||||
"class": "parallel",
|
||||
"prompt": "Read a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt, then report the first line of each. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/parallel"
|
||||
},
|
||||
{
|
||||
"id": "parallel-02",
|
||||
"name": "grep six patterns",
|
||||
"class": "parallel",
|
||||
"prompt": "Search for 'alpha', 'beta', 'gamma', 'delta', 'epsilon', and 'zeta' across the repo and report which files contain each. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/parallel"
|
||||
},
|
||||
{
|
||||
"id": "parallel-03",
|
||||
"name": "list six dirs",
|
||||
"class": "parallel",
|
||||
"prompt": "List the contents of dir1, dir2, dir3, dir4, dir5, and dir6. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/parallel"
|
||||
},
|
||||
{
|
||||
"id": "parallel-04",
|
||||
"name": "stat six files",
|
||||
"class": "parallel",
|
||||
"prompt": "Report the line count of a.txt, b.txt, c.txt, d.txt, e.txt, and f.txt. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/parallel"
|
||||
},
|
||||
{
|
||||
"id": "parallel-05",
|
||||
"name": "read configs",
|
||||
"class": "parallel",
|
||||
"prompt": "Read config1.json, config2.json, config3.json, config4.json, config5.json, and config6.json and report each top-level key. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/parallel"
|
||||
},
|
||||
{
|
||||
"id": "parallel-06",
|
||||
"name": "summarize six",
|
||||
"class": "parallel",
|
||||
"prompt": "Read the six .txt files and give a one-line summary of each. Do not modify anything.",
|
||||
"workspaceFixture": "../testdata/parallel"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -51,6 +51,14 @@ type BenchTask struct {
|
||||
Prompt string `json:"prompt"`
|
||||
WorkspaceFixture string `json:"workspaceFixture,omitempty"`
|
||||
VerificationCommand []string `json:"verificationCommand,omitempty"`
|
||||
// OracleTest is Go source stamped into <fixtureCopy>/oracle_test.go AFTER
|
||||
// the agent run (so it can't interfere with the agent's own go build/test
|
||||
// during the task and can't be pre-seen). The verificationCommand then runs
|
||||
// `go test ./...`, making the Go compiler the structural verifier: a
|
||||
// compile-time reference (e.g. `var _ = Config{}.Label`) or a behavioral call
|
||||
// fails to compile/run when the task wasn't actually done. Empty for tasks
|
||||
// whose verificationCommand is a plain grep/build with no stamped test.
|
||||
OracleTest string `json:"oracleTest,omitempty"`
|
||||
}
|
||||
|
||||
// TaskConfig is the recorded configuration of a benchmark run. The honest
|
||||
@@ -389,14 +397,26 @@ func runVerification(ctx context.Context, task BenchTask) TaskOutcome {
|
||||
return TaskOutcome{Passed: true}
|
||||
}
|
||||
|
||||
// streamJSONExitCode scans stream-json output for the terminal run_end event and
|
||||
// returns its exit code. ZERO emits one JSON object per line; the last run_end
|
||||
// (or final) event carries the exit code. Lines that don't parse are skipped.
|
||||
func streamJSONExitCode(output []byte) (int, bool) {
|
||||
// streamJSONResult holds the terminal events parsed from one pass over a
|
||||
// stream-json output: the last run_end exit code (if any) and the last final
|
||||
// event's text (the agent's answer, if any). streamJSONExitCode and
|
||||
// streamJSONFinalText both consume this so the scanner limits and
|
||||
// malformed-event handling live in exactly one place.
|
||||
type streamJSONResult struct {
|
||||
exitCode int
|
||||
haveExit bool
|
||||
finalText string
|
||||
}
|
||||
|
||||
// parseStreamJSON scans stream-json output once and returns the terminal
|
||||
// run_end exit code and the final event's text. ZERO emits one JSON object per
|
||||
// line; the last run_end carries the exit code and the last final carries the
|
||||
// answer. Lines that don't parse are skipped. The scanner buffer is raised so a
|
||||
// large single-line event (e.g. a big final answer) doesn't error out.
|
||||
func parseStreamJSON(output []byte) streamJSONResult {
|
||||
scanner := bufio.NewScanner(bytes.NewReader(output))
|
||||
scanner.Buffer(make([]byte, 0, 64*1024), 4*1024*1024)
|
||||
exitCode := 0
|
||||
found := false
|
||||
var r streamJSONResult
|
||||
for scanner.Scan() {
|
||||
line := bytes.TrimSpace(scanner.Bytes())
|
||||
if len(line) == 0 || line[0] != '{' {
|
||||
@@ -405,16 +425,40 @@ func streamJSONExitCode(output []byte) (int, bool) {
|
||||
var event struct {
|
||||
Type string `json:"type"`
|
||||
ExitCode *int `json:"exitCode"`
|
||||
Text string `json:"text"`
|
||||
}
|
||||
if err := json.Unmarshal(line, &event); err != nil {
|
||||
continue
|
||||
}
|
||||
if event.Type == "run_end" && event.ExitCode != nil {
|
||||
exitCode = *event.ExitCode
|
||||
found = true
|
||||
r.exitCode = *event.ExitCode
|
||||
r.haveExit = true
|
||||
}
|
||||
if event.Type == "final" {
|
||||
r.finalText = event.Text
|
||||
}
|
||||
}
|
||||
return exitCode, found
|
||||
return r
|
||||
}
|
||||
|
||||
// streamJSONExitCode scans stream-json output for the terminal run_end event and
|
||||
// returns its exit code. The last run_end event carries the exit code. Lines
|
||||
// that don't parse are skipped.
|
||||
func streamJSONExitCode(output []byte) (int, bool) {
|
||||
r := parseStreamJSON(output)
|
||||
return r.exitCode, r.haveExit
|
||||
}
|
||||
|
||||
// streamJSONFinalText scans stream-json output for the terminal "final" event
|
||||
// and returns its text (the agent's final answer). ZERO emits one final event
|
||||
// per run on both the success and incomplete paths (exec.go writer.final), so
|
||||
// the last one wins. The turn benchmark writes this to .zero-answer.txt in the
|
||||
// fixture copy so nav answer-oracles can grep the captured answer rather than
|
||||
// the raw stream. Returns "" when no final event was emitted (e.g. a provider
|
||||
// error path) — the caller writes an empty file and a nav grep then fails,
|
||||
// which is the correct outcome for a run that produced no answer.
|
||||
func streamJSONFinalText(output []byte) string {
|
||||
return parseStreamJSON(output).finalText
|
||||
}
|
||||
|
||||
func firstLine(text string) string {
|
||||
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
module edit
|
||||
|
||||
go 1.22
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
module fix
|
||||
|
||||
go 1.22
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
module nav
|
||||
|
||||
go 1.22
|
||||
+4
@@ -22,3 +22,7 @@ func greet(name string) string {
|
||||
func main() {
|
||||
fmt.Println(greet("world"))
|
||||
}
|
||||
|
||||
// TODO: replace the demo greet with the real client call before shipping.
|
||||
// (Present so nav-05's find-the-markers task has a non-zero, inspection-
|
||||
// required answer; an agent that always emits "count: 0" fails.)
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
package nav
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestGreet is a real test so nav-04's "count the test functions" has a
|
||||
// non-zero, inspection-required answer: an agent that never opens the workspace
|
||||
// and always emits "count: 0" fails, which is the whole point of graduating nav
|
||||
// into the correctness tier. It runs green offline (no third-party deps).
|
||||
func TestGreet(t *testing.T) {
|
||||
if got := greet("x"); got != "hello, x" {
|
||||
t.Fatalf("greet(x) = %q, want %q", got, "hello, x")
|
||||
}
|
||||
}
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
module refactor
|
||||
|
||||
go 1.22
|
||||
@@ -28,6 +28,12 @@ func Wrapper(name string) string {
|
||||
return GreetByName(name)
|
||||
}
|
||||
|
||||
// greetWrapped is Wrapper's single caller — refactor-05 inlines Wrapper here
|
||||
// and removes the Wrapper function.
|
||||
func greetWrapped(name string) string {
|
||||
return Wrapper(name)
|
||||
}
|
||||
|
||||
// stats is a bare map used across the file — refactor-04 introduces a named type.
|
||||
var stats = map[string]int{}
|
||||
|
||||
|
||||
@@ -27,7 +27,20 @@ import (
|
||||
// buildPassedTasks/buildPassRate cover the non-positive build-check classes
|
||||
// (refactor); latencyOnlyTasks covers the no-oracle classes (nav/longproc/
|
||||
// longctx/parallel). tasksAttempted is still every task run.
|
||||
const TurnSchemaVersion = 2
|
||||
//
|
||||
// v3 (Phase 0 oracle hardening) strengthens the oracles so the tiers are honest: edit/refactor
|
||||
// now carry a stamped oracle_test.go compiled by `go test ./...` (the Go
|
||||
// compiler is the structural verifier — a no-op refactor or a missing field
|
||||
// fails to compile), and nav carries an answer-oracle grepping the agent's
|
||||
// captured final answer. refactor graduates from build to correctness (its
|
||||
// oracle is now positive), nav graduates from latency-only to correctness, and
|
||||
// the build tier is empty (nothing is only-build-checked anymore). The result
|
||||
// SHAPE is unchanged — only class-list membership shifts — but
|
||||
// correctnessPassRate now means something strictly stronger, so the bump
|
||||
// prevents a v2→v2 cross-version comparison from misreading the jump as a model
|
||||
// improvement. New counts: correctness 34 (edit 10 + fix 8 + nav 10 + refactor
|
||||
// 6), build 0, latency 14 (longproc 4 + longctx 4 + parallel 6).
|
||||
const TurnSchemaVersion = 3
|
||||
|
||||
// TurnRunner runs one benchmark task and reports its outcome plus the captured
|
||||
// per-turn trace. A non-nil Err means the run failed to execute (process crash);
|
||||
@@ -89,10 +102,11 @@ type LatencySource struct {
|
||||
// ClassSummary is the per-class (task group) roll-up.
|
||||
//
|
||||
// Passed is the count that passed THIS class's oracle: a correctness pass for
|
||||
// edit/fix, a build pass for refactor, and always 0 for the no-oracle classes
|
||||
// (nav/longproc/longctx/parallel). Verified is how many tasks in the class
|
||||
// carry an oracle (correctness or build); LatencyOnly is how many carry none.
|
||||
// A latency-only class therefore reports Passed=0, Verified=0, LatencyOnly=Tasks.
|
||||
// the positive-oracle classes (edit/fix/nav/refactor), and always 0 for the
|
||||
// no-oracle classes (longproc/longctx/parallel) — the build tier is empty in v3
|
||||
// (refactor graduated to correctness). Verified is how many tasks in the class
|
||||
// carry an oracle; LatencyOnly is how many carry none. A latency-only class
|
||||
// therefore reports Passed=0, Verified=0, LatencyOnly=Tasks.
|
||||
type ClassSummary struct {
|
||||
Tasks int `json:"tasks"`
|
||||
Verified int `json:"verified"`
|
||||
@@ -107,13 +121,14 @@ type ClassSummary struct {
|
||||
// Pass/fail is split into three oracle tiers so it cannot be misread as a
|
||||
// blanket correctness verdict:
|
||||
// - Correctness (tasksVerified / tasksPassed / correctnessPassRate): tasks
|
||||
// with a positive oracle (edit/fix). This is the only pass rate that can
|
||||
// move with model quality.
|
||||
// - Build-only (buildCheckedTasks / buildPassedTasks / buildPassRate): tasks
|
||||
// whose oracle is a non-positive build check (refactor `go build`). A pass
|
||||
// means it compiles, not that the refactor is correct.
|
||||
// - Latency-only (latencyOnlyTasks): tasks with no oracle (nav/longproc/
|
||||
// longctx/parallel). They ran for latency and span attribution only and are
|
||||
// with a positive oracle (edit/fix/nav/refactor). This is the only pass
|
||||
// rate that can move with model quality.
|
||||
// - Build-only (buildCheckedTasks / buildPassedTasks / buildPassRate): the
|
||||
// non-positive build-check tier. Empty in v3 — refactor graduated to
|
||||
// correctness (its stamped oracle is positive) — so buildCheckedTasks is
|
||||
// always 0 and buildPassRate is 0. The fields remain for schema stability.
|
||||
// - Latency-only (latencyOnlyTasks): tasks with no oracle (longproc/longctx/
|
||||
// parallel). They ran for latency and span attribution only and are
|
||||
// excluded from every pass rate.
|
||||
//
|
||||
// tasksAttempted is still the total number of tasks run across all three tiers.
|
||||
@@ -432,8 +447,9 @@ func FormatTurnBenchSummary(result TurnBenchResult) string {
|
||||
"model: " + displayOrUnknown(result.Model),
|
||||
// The headline separates the three oracle tiers so an exit-0 read-only
|
||||
// task can never inflate a "pass rate" that reads as correctness:
|
||||
// correctness (positive oracle, edit/fix), build (non-positive `go build`,
|
||||
// refactor), and latency-only (no oracle, nav/longproc/longctx/parallel).
|
||||
// correctness (positive oracle: edit/fix/nav/refactor), build (empty in
|
||||
// v3 — refactor graduated to correctness), and latency-only (no oracle:
|
||||
// longproc/longctx/parallel).
|
||||
fmt.Sprintf("tasks: %d total | correctness %d/%d (%.0f%%) | build %d/%d (%.0f%%) | latency-only %d | %d iter",
|
||||
result.TasksAttempted,
|
||||
result.TasksPassed, result.TasksVerified, result.CorrectnessPassRate*100,
|
||||
@@ -509,11 +525,13 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
|
||||
// fixture or bleed into a later iteration of the same task. When no
|
||||
// fixture is configured the agent runs in the caller's cwd as before.
|
||||
if fixture := strings.TrimSpace(task.WorkspaceFixture); fixture != "" {
|
||||
copyDir, cerr := copyFixture(fixture)
|
||||
copyDir, parent, cerr := copyFixture(fixture)
|
||||
if cerr != nil {
|
||||
return TurnTaskOutcome{Err: fmt.Errorf("isolate fixture: %w", cerr)}
|
||||
}
|
||||
defer os.RemoveAll(copyDir)
|
||||
// Clean the whole unique parent (which owns copyDir) so the
|
||||
// per-invocation scratch dir never leaks.
|
||||
defer os.RemoveAll(parent)
|
||||
task.WorkspaceFixture = copyDir
|
||||
}
|
||||
|
||||
@@ -577,6 +595,22 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
|
||||
if outcome.VerifyErr != "" {
|
||||
return outcome
|
||||
}
|
||||
// Stamp the compiler-backed oracle and capture the agent's final answer
|
||||
// BEFORE running the verification command. Both write into the fixture
|
||||
// copy (task.WorkspaceFixture, set to copyDir above) so runVerification —
|
||||
// which uses that same dir as cmd.Dir — sees them. Stamping happens only
|
||||
// after the agent run so the oracle test can't interfere with the agent's
|
||||
// own go build/test during the task (e.g. refactor-03's package-zeroapp
|
||||
// test would break a pre-rename build) and can't be pre-seen or tampered
|
||||
// with.
|
||||
if err := stampOracleAndAnswer(task, outBuf.Bytes()); err != nil {
|
||||
// Preserve the already-parsed trace, trace issue, and wall time: a
|
||||
// stamp failure is a harness error, but the run still produced a
|
||||
// trace worth attributing, so mutate the existing outcome rather than
|
||||
// returning a fresh one that drops Trace/TraceIssue.
|
||||
outcome.Err = fmt.Errorf("stamp oracle: %w", err)
|
||||
return outcome
|
||||
}
|
||||
if len(task.VerificationCommand) > 0 {
|
||||
if vOutcome := runVerification(ctx, task); !vOutcome.Passed {
|
||||
outcome.VerifyErr = strings.TrimSpace(vOutcome.Detail)
|
||||
@@ -584,7 +618,7 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
|
||||
}
|
||||
// A positive oracle (grep/test/build) passed: this is the only path
|
||||
// that sets Passed. A task with no verificationCommand is latency-only
|
||||
// (read-only nav/longproc/longctx/parallel): its exit 0 proves the turn
|
||||
// (read-only longproc/longctx/parallel): its exit 0 proves the turn
|
||||
// ran, not that the answer was right, so it never reports Passed and the
|
||||
// harness counts it in latencyOnlyTasks rather than any pass rate.
|
||||
outcome.Passed = true
|
||||
@@ -594,22 +628,41 @@ func NewTurnExecRunner(binary string, extraArgs ...string) TurnRunner {
|
||||
}
|
||||
|
||||
// copyFixture copies the fixture directory at src into a fresh temp dir and
|
||||
// returns the copy's path. Used to give each benchmark invocation an isolated
|
||||
// workspace so mutating tasks (edit/fix/refactor) can't dirty the checked-in
|
||||
// fixture or a later iteration.
|
||||
func copyFixture(src string) (string, error) {
|
||||
// returns the copy's path (dst) plus the unique parent dir that owns it (parent,
|
||||
// which the caller must RemoveAll to clean up dst and its sibling scratch space
|
||||
// together). Used to give each benchmark invocation an isolated workspace so
|
||||
// mutating tasks (edit/fix/refactor) can't dirty the checked-in fixture or a
|
||||
// later iteration.
|
||||
//
|
||||
// The copy is placed two levels BELOW the system temp root: a unique 0700
|
||||
// parent (created fresh per invocation via os.MkdirTemp, so concurrent runs and
|
||||
// different users can't share or traverse a predictable scratch dir) and the
|
||||
// fixture copy beneath it. Go ignores go.mod files in DIRECT children of the
|
||||
// system temp dir (a hijack guard), which would make the fixture's go.mod —
|
||||
// and thus every `go test ./...`/`go build ./...` oracle — fail with "cannot
|
||||
// find main module". Keeping the copy a grandchild of the temp root makes Go
|
||||
// respect the copied go.mod so the compiler-backed oracles actually run.
|
||||
func copyFixture(src string) (dst, parent string, err error) {
|
||||
info, err := os.Stat(src)
|
||||
if err != nil {
|
||||
return "", err
|
||||
return "", "", err
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return "", fmt.Errorf("fixture %q is not a directory", src)
|
||||
return "", "", fmt.Errorf("fixture %q is not a directory", src)
|
||||
}
|
||||
dst, err := os.MkdirTemp("", "zero-turn-fixture-*")
|
||||
// Unique, 0700-per-invocation parent directly under the temp root. The
|
||||
// fixture copy is created BENEATH it, so the copy is a grandchild of the
|
||||
// temp root and its go.mod is respected (see the doc comment above).
|
||||
parent, err = os.MkdirTemp(os.TempDir(), "zero-turn-bench-*")
|
||||
if err != nil {
|
||||
return "", err
|
||||
return "", "", err
|
||||
}
|
||||
err = filepath.WalkDir(src, func(path string, d fs.DirEntry, werr error) error {
|
||||
dst, err = os.MkdirTemp(parent, "zero-turn-fixture-*")
|
||||
if err != nil {
|
||||
os.RemoveAll(parent)
|
||||
return "", "", err
|
||||
}
|
||||
werr := filepath.WalkDir(src, func(path string, d fs.DirEntry, werr error) error {
|
||||
if werr != nil {
|
||||
return werr
|
||||
}
|
||||
@@ -627,11 +680,47 @@ func copyFixture(src string) (string, error) {
|
||||
}
|
||||
return os.WriteFile(target, data, 0o644)
|
||||
})
|
||||
if err != nil {
|
||||
os.RemoveAll(dst)
|
||||
return "", err
|
||||
if werr != nil {
|
||||
os.RemoveAll(parent) // removes dst too
|
||||
return "", "", werr
|
||||
}
|
||||
return dst, nil
|
||||
return dst, parent, nil
|
||||
}
|
||||
|
||||
// stampOracleAndAnswer writes the compiler-backed oracle test (when configured)
|
||||
// and the agent's captured final answer into the fixture copy, so the
|
||||
// verification command can compile the oracle and grep the answer. Both files
|
||||
// land in task.WorkspaceFixture, which the runner has already set to the
|
||||
// fixture copy and which runVerification uses as cmd.Dir. The answer file is
|
||||
// always written (empty when no final event was captured) so a nav answer-oracle
|
||||
// fails cleanly on a run that produced no answer rather than reading a stale or
|
||||
// absent file.
|
||||
//
|
||||
// A task that needs an oracle — a stamped oracle_test.go OR a verification
|
||||
// command that reads .zero-answer.txt — but has no workspace fixture is
|
||||
// rejected: there is nowhere safe to stamp, so runVerification would otherwise
|
||||
// run in the caller's cwd with the oracle never compiled (or the answer never
|
||||
// captured), which can read as a false pass. A fixtureless task with no oracle
|
||||
// and no verification (a pure latency-only run in the caller's cwd) is left
|
||||
// alone.
|
||||
func stampOracleAndAnswer(task BenchTask, outBuf []byte) error {
|
||||
dir := strings.TrimSpace(task.WorkspaceFixture)
|
||||
if dir == "" {
|
||||
if task.OracleTest != "" || len(task.VerificationCommand) > 0 {
|
||||
return fmt.Errorf("stamp oracle: task %q has an oracle or verification but no workspace fixture to stamp into", task.ID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if ot := task.OracleTest; ot != "" {
|
||||
if err := os.WriteFile(filepath.Join(dir, "oracle_test.go"), []byte(ot), 0o644); err != nil {
|
||||
return fmt.Errorf("write oracle_test.go: %w", err)
|
||||
}
|
||||
}
|
||||
answer := streamJSONFinalText(outBuf)
|
||||
if err := os.WriteFile(filepath.Join(dir, ".zero-answer.txt"), []byte(answer), 0o644); err != nil {
|
||||
return fmt.Errorf("write .zero-answer.txt: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func buildTurnExecArgs(task BenchTask, rc RunContext, tracePath string, extraArgs []string) []string {
|
||||
|
||||
@@ -51,21 +51,26 @@ func cannedTrace(genMs, toolMs int, tokens int64) *trace.TurnTrace {
|
||||
}
|
||||
|
||||
func TestRunTurnBenchAggregation(t *testing.T) {
|
||||
// v3 tier model: nav and refactor both carry a verificationCommand and are
|
||||
// NOT in buildOnlyClasses, so they count as correctness; longproc has no
|
||||
// oracle and is latency-only. The build tier is empty (buildOnlyClasses is
|
||||
// nil), matching the v3 baseline manifest where refactor graduated from
|
||||
// build to correctness.
|
||||
set := TaskSet{
|
||||
ID: "fake-suite",
|
||||
Tasks: []BenchTask{
|
||||
{ID: "t1", Class: "nav", Prompt: "p1"}, // latency-only
|
||||
{ID: "t2", Class: "nav", Prompt: "p2"}, // latency-only
|
||||
{ID: "t3", Class: "edit", Prompt: "p3", VerificationCommand: []string{"true"}}, // correctness
|
||||
{ID: "t4", Class: "refactor", Prompt: "p4", VerificationCommand: []string{"true"}}, // build-only
|
||||
{ID: "t1", Class: "nav", Prompt: "p1", VerificationCommand: []string{"true"}}, // correctness
|
||||
{ID: "t2", Class: "edit", Prompt: "p2", VerificationCommand: []string{"true"}}, // correctness
|
||||
{ID: "t3", Class: "refactor", Prompt: "p3", VerificationCommand: []string{"true"}}, // correctness
|
||||
{ID: "t4", Class: "longproc", Prompt: "p4"}, // latency-only
|
||||
},
|
||||
BuildOnlyClasses: []string{"refactor"},
|
||||
BuildOnlyClasses: nil,
|
||||
}
|
||||
canned := map[string]*trace.TurnTrace{
|
||||
"t1": cannedTrace(100, 10, 1000),
|
||||
"t2": cannedTrace(300, 10, 1000),
|
||||
"t3": cannedTrace(200, 50, 2000),
|
||||
"t4": cannedTrace(150, 20, 1500),
|
||||
"t2": cannedTrace(200, 50, 2000),
|
||||
"t3": cannedTrace(150, 20, 1500),
|
||||
"t4": cannedTrace(300, 10, 1000),
|
||||
}
|
||||
cfg := TurnBenchConfig{
|
||||
Model: "fake-model",
|
||||
@@ -77,33 +82,36 @@ func TestRunTurnBenchAggregation(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("RunTurnBench: %v", err)
|
||||
}
|
||||
// 4 tasks attempted; the tier split is 2 latency-only, 1 correctness, 1 build.
|
||||
// 4 tasks attempted; the v3 tier split is 3 correctness (nav, edit, refactor),
|
||||
// 1 latency-only (longproc), and 0 build (the build tier is empty).
|
||||
if result.TasksAttempted != 4 {
|
||||
t.Fatalf("attempted=%d, want 4", result.TasksAttempted)
|
||||
}
|
||||
if result.TasksVerified != 1 || result.TasksPassed != 1 {
|
||||
t.Fatalf("correctness verified=%d passed=%d, want 1/1", result.TasksVerified, result.TasksPassed)
|
||||
if result.TasksVerified != 3 || result.TasksPassed != 3 {
|
||||
t.Fatalf("correctness verified=%d passed=%d, want 3/3", result.TasksVerified, result.TasksPassed)
|
||||
}
|
||||
if result.LatencyOnlyTasks != 2 {
|
||||
t.Fatalf("latencyOnly=%d, want 2", result.LatencyOnlyTasks)
|
||||
if result.LatencyOnlyTasks != 1 {
|
||||
t.Fatalf("latencyOnly=%d, want 1", result.LatencyOnlyTasks)
|
||||
}
|
||||
if result.BuildCheckedTasks != 1 || result.BuildPassedTasks != 1 {
|
||||
t.Fatalf("build checked=%d passed=%d, want 1/1", result.BuildCheckedTasks, result.BuildPassedTasks)
|
||||
if result.BuildCheckedTasks != 0 || result.BuildPassedTasks != 0 {
|
||||
t.Fatalf("build checked=%d passed=%d, want 0/0 (empty build tier)", result.BuildCheckedTasks, result.BuildPassedTasks)
|
||||
}
|
||||
if result.CorrectnessPassRate != 1.0 {
|
||||
t.Fatalf("correctnessPassRate=%v, want 1.0", result.CorrectnessPassRate)
|
||||
}
|
||||
if result.BuildPassRate != 1.0 {
|
||||
t.Fatalf("buildPassRate=%v, want 1.0", result.BuildPassRate)
|
||||
// passRate returns 0 for a 0/0 denominator, so an empty build tier reports 0
|
||||
// (not NaN) and cannot be misread as a perfect build score.
|
||||
if result.BuildPassRate != 0 {
|
||||
t.Fatalf("buildPassRate=%v, want 0 (empty build tier is 0/0 -> 0)", result.BuildPassRate)
|
||||
}
|
||||
if len(result.CorrectnessClasses) != 1 || result.CorrectnessClasses[0] != "edit" {
|
||||
t.Fatalf("correctnessClasses=%v, want [edit]", result.CorrectnessClasses)
|
||||
if len(result.CorrectnessClasses) != 3 || result.CorrectnessClasses[0] != "edit" || result.CorrectnessClasses[1] != "nav" || result.CorrectnessClasses[2] != "refactor" {
|
||||
t.Fatalf("correctnessClasses=%v, want [edit nav refactor]", result.CorrectnessClasses)
|
||||
}
|
||||
if len(result.BuildOnlyClasses) != 1 || result.BuildOnlyClasses[0] != "refactor" {
|
||||
t.Fatalf("buildOnlyClasses=%v, want [refactor]", result.BuildOnlyClasses)
|
||||
if len(result.BuildOnlyClasses) != 0 {
|
||||
t.Fatalf("buildOnlyClasses=%v, want empty (v3: build tier empty)", result.BuildOnlyClasses)
|
||||
}
|
||||
if len(result.LatencyOnlyClasses) != 1 || result.LatencyOnlyClasses[0] != "nav" {
|
||||
t.Fatalf("latencyOnlyClasses=%v, want [nav]", result.LatencyOnlyClasses)
|
||||
if len(result.LatencyOnlyClasses) != 1 || result.LatencyOnlyClasses[0] != "longproc" {
|
||||
t.Fatalf("latencyOnlyClasses=%v, want [longproc]", result.LatencyOnlyClasses)
|
||||
}
|
||||
if result.SchemaVersion != TurnSchemaVersion {
|
||||
t.Fatalf("schemaVersion = %d, want %d", result.SchemaVersion, TurnSchemaVersion)
|
||||
@@ -112,8 +120,8 @@ func TestRunTurnBenchAggregation(t *testing.T) {
|
||||
t.Fatalf("date = %q", result.Date)
|
||||
}
|
||||
|
||||
// Per-span: generation appears in all four (100+300+200+150=750ms), tool in
|
||||
// all four (10+10+50+20=90ms). Count must equal the number of tasks * iterations.
|
||||
// Per-span: generation appears in all four (100+200+150+300=750ms), tool in
|
||||
// all four (10+50+20+10=90ms). Count must equal the number of tasks * iterations.
|
||||
gen := result.PerSpan[trace.SpanGeneration]
|
||||
if gen.Count != 4 {
|
||||
t.Fatalf("generation count = %d, want 4", gen.Count)
|
||||
@@ -135,7 +143,7 @@ func TestRunTurnBenchAggregation(t *testing.T) {
|
||||
t.Fatalf("top latency not ranked by share: %+v", result.TopLatency)
|
||||
}
|
||||
|
||||
// Totals: 4 model requests, 4 tool calls, input tokens 1000+1000+2000+1500=5500.
|
||||
// Totals: 4 model requests, 4 tool calls, input tokens 1000+2000+1500+1000=5500.
|
||||
if result.Totals.ModelRequests != 4 {
|
||||
t.Fatalf("modelRequests = %d, want 4", result.Totals.ModelRequests)
|
||||
}
|
||||
@@ -149,10 +157,10 @@ func TestRunTurnBenchAggregation(t *testing.T) {
|
||||
t.Fatalf("outputTokens = %d, want 2750", result.Totals.OutputTokens)
|
||||
}
|
||||
|
||||
// Per-class tier roll-up: nav is latency-only (0 verified, 2 latency-only),
|
||||
// edit is correctness (1/1 verified passed), refactor is build (1/1 passed).
|
||||
// Per-class tier roll-up: nav/edit/refactor are correctness (1/1 verified
|
||||
// passed, 0 latency-only); longproc is latency-only (0 verified, 1 latency).
|
||||
nav := result.PerClass["nav"]
|
||||
if nav.Tasks != 2 || nav.Verified != 0 || nav.Passed != 0 || nav.LatencyOnly != 2 {
|
||||
if nav.Tasks != 1 || nav.Verified != 1 || nav.Passed != 1 || nav.LatencyOnly != 0 {
|
||||
t.Fatalf("nav class = %+v", nav)
|
||||
}
|
||||
edit := result.PerClass["edit"]
|
||||
@@ -163,6 +171,10 @@ func TestRunTurnBenchAggregation(t *testing.T) {
|
||||
if refactor.Tasks != 1 || refactor.Verified != 1 || refactor.Passed != 1 || refactor.LatencyOnly != 0 {
|
||||
t.Fatalf("refactor class = %+v", refactor)
|
||||
}
|
||||
longproc := result.PerClass["longproc"]
|
||||
if longproc.Tasks != 1 || longproc.Verified != 0 || longproc.Passed != 0 || longproc.LatencyOnly != 1 {
|
||||
t.Fatalf("longproc class = %+v", longproc)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTurnBenchLatencyOnlyNeverPassed asserts the honesty gate: a task with
|
||||
@@ -424,17 +436,31 @@ func TestCopyFixtureIsolatesSourceFromMutation(t *testing.T) {
|
||||
t.Fatalf("read source main.go: %v", err)
|
||||
}
|
||||
|
||||
copyDir, err := copyFixture(src)
|
||||
copyDir, parent, err := copyFixture(src)
|
||||
if err != nil {
|
||||
t.Fatalf("copyFixture: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(copyDir)
|
||||
defer os.RemoveAll(parent) // cleans copyDir too, since copyDir lives under parent
|
||||
|
||||
// The copy must be a real copy, not a symlink/alias of the source, and
|
||||
// mutating it must not touch the source.
|
||||
if copyDir == src {
|
||||
t.Fatalf("copyFixture returned the source dir, not a copy: %s", copyDir)
|
||||
}
|
||||
// The copy must be a grandchild of the system temp root (parent is a direct
|
||||
// child of Temp, copyDir is a direct child of parent). Go ignores go.mod in
|
||||
// direct children of Temp (a hijack guard), so this nesting is what lets the
|
||||
// compiler-backed oracles actually run in the copy. Compare via filepath.Clean
|
||||
// because os.TempDir() may carry a trailing slash (macOS $TMPDIR ends in
|
||||
// "/T/") while filepath.Dir strips it; the raw != would false-fail on macOS
|
||||
// even though the parent is genuinely a direct child.
|
||||
tempRoot := filepath.Clean(os.TempDir())
|
||||
if filepath.Dir(parent) != tempRoot {
|
||||
t.Fatalf("parent %q is not a direct child of temp root %q", parent, tempRoot)
|
||||
}
|
||||
if filepath.Dir(copyDir) != parent {
|
||||
t.Fatalf("copyDir %q is not nested under parent %q", copyDir, parent)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(copyDir, "main.go"), []byte("package main\n\n// mutated\nfunc main() {}\n"), 0o644); err != nil {
|
||||
t.Fatalf("mutate copy: %v", err)
|
||||
}
|
||||
@@ -453,3 +479,591 @@ func TestCopyFixtureIsolatesSourceFromMutation(t *testing.T) {
|
||||
t.Fatalf("file added to copy appeared in source: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTurnBenchBuildOnlyMechanism exercises the build-only tier code path in
|
||||
// isolation. The v3 baseline manifest leaves this tier empty (refactor graduated
|
||||
// to correctness), but the mechanism is still supported for future use, so a
|
||||
// focused test keeps it covered: a class listed in buildOnlyClasses with a
|
||||
// verificationCommand counts in buildCheckedTasks/buildPassedTasks — never in
|
||||
// tasksVerified/correctnessPassRate — and a build-pass cannot leak into the
|
||||
// correctness number.
|
||||
func TestRunTurnBenchBuildOnlyMechanism(t *testing.T) {
|
||||
set := TaskSet{
|
||||
ID: "build-suite",
|
||||
Tasks: []BenchTask{
|
||||
{ID: "b1", Class: "buildcheck", Prompt: "p", VerificationCommand: []string{"true"}},
|
||||
},
|
||||
BuildOnlyClasses: []string{"buildcheck"},
|
||||
}
|
||||
canned := map[string]*trace.TurnTrace{"b1": cannedTrace(120, 30, 900)}
|
||||
result, err := RunTurnBench(context.Background(), set, TurnBenchConfig{
|
||||
Model: "fake-model",
|
||||
Runner: fakeTurnRunner(canned),
|
||||
Now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("RunTurnBench: %v", err)
|
||||
}
|
||||
if result.BuildCheckedTasks != 1 || result.BuildPassedTasks != 1 {
|
||||
t.Fatalf("build checked=%d passed=%d, want 1/1", result.BuildCheckedTasks, result.BuildPassedTasks)
|
||||
}
|
||||
if result.BuildPassRate != 1.0 {
|
||||
t.Fatalf("buildPassRate=%v, want 1.0", result.BuildPassRate)
|
||||
}
|
||||
// A build-only task must NOT count toward correctness.
|
||||
if result.TasksVerified != 0 || result.TasksPassed != 0 {
|
||||
t.Fatalf("build-only leaked into correctness: verified=%d passed=%d, want 0/0",
|
||||
result.TasksVerified, result.TasksPassed)
|
||||
}
|
||||
if result.CorrectnessPassRate != 0 {
|
||||
t.Fatalf("correctnessPassRate=%v, want 0 with only build-only tasks", result.CorrectnessPassRate)
|
||||
}
|
||||
if len(result.BuildOnlyClasses) != 1 || result.BuildOnlyClasses[0] != "buildcheck" {
|
||||
t.Fatalf("buildOnlyClasses=%v, want [buildcheck]", result.BuildOnlyClasses)
|
||||
}
|
||||
if len(result.CorrectnessClasses) != 0 {
|
||||
t.Fatalf("correctnessClasses=%v, want empty", result.CorrectnessClasses)
|
||||
}
|
||||
bc := result.PerClass["buildcheck"]
|
||||
if bc.Tasks != 1 || bc.Verified != 1 || bc.Passed != 1 || bc.LatencyOnly != 0 {
|
||||
t.Fatalf("buildcheck class = %+v, want 1/1 verified passed", bc)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTurnSchemaVersion3 pins the schema bump. v3 records the tier
|
||||
// reclassification (refactor structural-positive and nav answer-oracles moved
|
||||
// into correctnessPassRate), so a v2->v3 cross-version comparison cannot
|
||||
// silently misread the jump as a model improvement — exactly the misread the
|
||||
// tier system exists to prevent.
|
||||
func TestTurnSchemaVersion3(t *testing.T) {
|
||||
if TurnSchemaVersion != 3 {
|
||||
t.Fatalf("TurnSchemaVersion = %d, want 3", TurnSchemaVersion)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStreamJSONFinalTextExtractsAnswer, TestStreamJSONFinalTextEmptyWhenNoFinal,
|
||||
// and TestStreamJSONFinalTextLastWins cover the pure capture helper that the
|
||||
// nav answer-oracle depends on: it scans stream-json for the terminal "final"
|
||||
// event and returns its text, "" when none was emitted, and the last when
|
||||
// multiple appear (the success path emits exactly one, but the incomplete path
|
||||
// can also emit one, so last-wins matches streamJSONExitCode's tie-break).
|
||||
func TestStreamJSONFinalTextExtractsAnswer(t *testing.T) {
|
||||
out := []byte(`{"type":"text","text":"thinking..."}
|
||||
{"type":"final","text":"the keys are port, name, and retries"}
|
||||
{"type":"run_end","exitCode":0}
|
||||
`)
|
||||
if got := streamJSONFinalText(out); got != "the keys are port, name, and retries" {
|
||||
t.Fatalf("streamJSONFinalText = %q, want the final text", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamJSONFinalTextEmptyWhenNoFinal(t *testing.T) {
|
||||
out := []byte(`{"type":"text","text":"hi"}
|
||||
{"type":"run_end","exitCode":0}
|
||||
`)
|
||||
if got := streamJSONFinalText(out); got != "" {
|
||||
t.Fatalf("streamJSONFinalText = %q, want empty when no final event", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamJSONFinalTextLastWins(t *testing.T) {
|
||||
out := []byte(`{"type":"final","text":"first"}
|
||||
{"type":"final","text":"second"}
|
||||
`)
|
||||
if got := streamJSONFinalText(out); got != "second" {
|
||||
t.Fatalf("streamJSONFinalText = %q, want second (last final wins)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// loadBaselineTask returns the named task from the checked-in baseline manifest.
|
||||
// The stub-binary oracle tests below run the REAL manifest oracles against the
|
||||
// REAL fixtures through the REAL NewTurnExecRunner, so a manifest edit that
|
||||
// breaks an oracle is caught here rather than only in production.
|
||||
func loadBaselineTask(t *testing.T, id string) BenchTask {
|
||||
t.Helper()
|
||||
set, err := LoadTaskSet(filepath.Join("manifests", "baseline.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("LoadTaskSet: %v", err)
|
||||
}
|
||||
for _, task := range set.Tasks {
|
||||
if task.ID == id {
|
||||
return task
|
||||
}
|
||||
}
|
||||
t.Fatalf("manifest has no task %q", id)
|
||||
return BenchTask{}
|
||||
}
|
||||
|
||||
// runTurnStub runs one manifest task through the production NewTurnExecRunner
|
||||
// with a stub "zero" binary whose body is a POSIX sh script (writeExecStub
|
||||
// skips on Windows). The stub is invoked with cmd.Dir set to the fixture copy,
|
||||
// so it can both emit canned stream-json AND mutate the copy (apply or omit the
|
||||
// fix) before the runner stamps the oracle and runs the verification command.
|
||||
func runTurnStub(t *testing.T, task BenchTask, stubBody string) TurnTaskOutcome {
|
||||
t.Helper()
|
||||
stub := writeExecStub(t, stubBody)
|
||||
return NewTurnExecRunner(stub)(context.Background(), task, RunContext{Model: "fake-model"})
|
||||
}
|
||||
|
||||
// assertVerifyFailed asserts an outcome failed specifically because the oracle
|
||||
// rejected the work — Passed is false, there is no harness error (Err nil), and
|
||||
// VerifyErr carries the surfaced failure detail. This is stronger than merely
|
||||
// checking Passed=false: it distinguishes a genuine oracle rejection from a
|
||||
// harness error (Err set) or a silent non-pass (Passed=false with no VerifyErr,
|
||||
// which would mean the failure never went through verification at all). The
|
||||
// gating stubs all emit a clean run_end (exitCode 0), so a non-empty VerifyErr
|
||||
// here can only come from the verification command failing.
|
||||
func assertVerifyFailed(t *testing.T, label string, outcome TurnTaskOutcome) {
|
||||
t.Helper()
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("%s: want a verify fail, got harness error: %v", label, outcome.Err)
|
||||
}
|
||||
if outcome.Passed {
|
||||
t.Fatalf("%s: want verify fail, got Passed=true", label)
|
||||
}
|
||||
if strings.TrimSpace(outcome.VerifyErr) == "" {
|
||||
t.Fatalf("%s: want verify fail, but VerifyErr is empty (failure did not come from verification): %+v", label, outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// --- Gating tests: the oracle FAILS the wrong thing (no-op / wrong answer) ---
|
||||
|
||||
// TestStampedOracleRejectsNoOpRefactor is the core #712 fix: refactor used to
|
||||
// live in the build tier where a no-op `go build ./...` passed. Now a no-op
|
||||
// agent (the stub emits a clean run_end but touches nothing) leaves the fixture
|
||||
// with no formatGreeting helper, so the stamped `var _ = formatGreeting` fails
|
||||
// to compile and `go test ./...` fails — the task is failed, not passed.
|
||||
func TestStampedOracleRejectsNoOpRefactor(t *testing.T) {
|
||||
task := loadBaselineTask(t, "refactor-01")
|
||||
outcome := runTurnStub(t, task, `echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "no-op refactor", outcome)
|
||||
}
|
||||
|
||||
// TestStampedOracleRejectsMissingField proves edit-03's oracle is structural: a
|
||||
// no-op leaves Config with no Label field, so `var _ string = Config{}.Label`
|
||||
// fails to compile and the task fails.
|
||||
func TestStampedOracleRejectsMissingField(t *testing.T) {
|
||||
task := loadBaselineTask(t, "edit-03")
|
||||
outcome := runTurnStub(t, task, `echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "missing-field edit-03", outcome)
|
||||
}
|
||||
|
||||
// TestNavAnswerOracleRejectsWrongAnswer proves the nav oracle greps the CAPTURED
|
||||
// answer, not the raw stream: the stub emits a final answer missing the
|
||||
// required keys, the harness writes it to .zero-answer.txt, and the compound
|
||||
// grep fails — so a plausible-but-wrong answer cannot pass nav-09.
|
||||
func TestNavAnswerOracleRejectsWrongAnswer(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-09")
|
||||
outcome := runTurnStub(t, task, `echo '{"type":"final","text":"the keys are foo and bar"}'
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "wrong nav-09 answer", outcome)
|
||||
}
|
||||
|
||||
// TestNavNoFinalTextFails proves a run that produced no answer fails nav: with
|
||||
// no "final" event, .zero-answer.txt is empty and the grep finds nothing.
|
||||
func TestNavNoFinalTextFails(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-09")
|
||||
outcome := runTurnStub(t, task, `echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "nav-09 no final", outcome)
|
||||
}
|
||||
|
||||
// TestEdit05RejectsRewordedDebugPrint proves edit-05's oracle catches a reword,
|
||||
// not just a deletion: changing `fmt.Println("debug: starting")` to
|
||||
// `fmt.Println("starting up")` leaves a fmt.Println string-literal call, so the
|
||||
// stamped oracle's `strings.Contains(main.go, fmt.Println(")` check fails. A
|
||||
// plain `! grep 'debug: starting'` would have rubber-stamped this reword.
|
||||
func TestEdit05RejectsRewordedDebugPrint(t *testing.T) {
|
||||
task := loadBaselineTask(t, "edit-05")
|
||||
outcome := runTurnStub(t, task, `sed 's/debug: starting/starting up/' main.go > .zero-tmp && mv .zero-tmp main.go
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "reworded edit-05 debug print", outcome)
|
||||
}
|
||||
|
||||
// TestEdit05RejectsRespelledDebugPrint proves the second edit-05 check (the
|
||||
// message text "debug: starting" must be absent) catches a respell onto a
|
||||
// different call form that the fmt.Println(" string-literal check misses: the
|
||||
// agent rewrites the line to fmt.Printf("debug: starting\n"), which still prints
|
||||
// the message but is no longer a fmt.Println string-literal. The
|
||||
// strings.Contains("debug: starting") assertion fails the task.
|
||||
func TestEdit05RejectsRespelledDebugPrint(t *testing.T) {
|
||||
task := loadBaselineTask(t, "edit-05")
|
||||
outcome := runTurnStub(t, task, `sed 's/fmt.Println("debug: starting")/fmt.Printf("debug: starting\\n")/' main.go > .zero-tmp && mv .zero-tmp main.go
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "respelled edit-05 debug print", outcome)
|
||||
}
|
||||
|
||||
// TestRefactor05RejectsStubbedCaller proves refactor-05's behavioral oracle
|
||||
// catches a wrong edit the negative grep alone would pass: the agent deletes
|
||||
// Wrapper (so ! grep 'func Wrapper' succeeds) but stubs greetWrapped to return
|
||||
// "" instead of inlining. The stamped TestGreetWrapped runs greetWrapped("x"),
|
||||
// gets "" not "hello, x", and fails — so deleting the target along with its
|
||||
// caller (or stubbing the caller) can no longer pass the oracle.
|
||||
func TestRefactor05RejectsStubbedCaller(t *testing.T) {
|
||||
task := loadBaselineTask(t, "refactor-05")
|
||||
outcome := runTurnStub(t, task, `sed -e 's/return Wrapper(name)/return ""/' -e '/^func Wrapper(name string) string {/,/^}/d' main.go > .zero-tmp && mv .zero-tmp main.go
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "stubbed refactor-05 caller", outcome)
|
||||
}
|
||||
|
||||
// TestRefactor04RejectsBareMapKept proves refactor-04's file-reading oracle
|
||||
// catches a no-op that the old exact-text shell grep would miss: the agent adds
|
||||
// an unused `type Stats map[string]int` (so the compile-time `var _ Stats`
|
||||
// passes) but keeps the bare `var stats = map[string]int{}` under the original
|
||||
// spacing. The spacing-tolerant regex in TestStatsNamedType still matches the
|
||||
// bare-map declaration, so the task fails — Record/Lookup never moved onto the
|
||||
// named type.
|
||||
func TestRefactor04RejectsBareMapKept(t *testing.T) {
|
||||
task := loadBaselineTask(t, "refactor-04")
|
||||
outcome := runTurnStub(t, task, `sed 's/type Config struct {/type Stats map[string]int\n\ntype Config struct {/' main.go > .zero-tmp && mv .zero-tmp main.go
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "refactor-04 bare-map kept", outcome)
|
||||
}
|
||||
|
||||
// TestEdit01IgnoresDocComment proves edit-01's scoped negative grep does NOT
|
||||
// false-fail a correct rename that leaves the doc comment mentioning the old
|
||||
// name. The stub renames the declaration (`MaxRetries =` -> `RetryLimit =`) but
|
||||
// leaves `// MaxRetries is the maximum...`; the scoped `! grep -RIn 'MaxRetries ='`
|
||||
// matches declaration sites only, so the comment is ignored, `var _ = RetryLimit`
|
||||
// compiles, and the task passes.
|
||||
func TestEdit01IgnoresDocComment(t *testing.T) {
|
||||
task := loadBaselineTask(t, "edit-01")
|
||||
outcome := runTurnStub(t, task, `sed 's/const MaxRetries = 3/const RetryLimit = 3/' main.go > .zero-tmp && mv .zero-tmp main.go
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("correct rename with kept doc comment should pass, got harness error: %v", outcome.Err)
|
||||
}
|
||||
if !outcome.Passed {
|
||||
t.Fatalf("edit-01 oracle must not false-fail a correct rename that leaves the doc comment: %+v", outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// --- Satisfiable tests: the oracle PASSES the right thing (real fix applied) ---
|
||||
|
||||
// TestStampedOraclePassesWhenRefactorHappened proves the refactor-01 oracle is
|
||||
// not an always-fail gate: when the stub applies the real refactor (extract
|
||||
// formatGreeting, call it from both callers), `var _ = formatGreeting` compiles
|
||||
// and the `hello, %s` grep-count is 1 (only the helper holds the literal), so
|
||||
// the task passes.
|
||||
func TestStampedOraclePassesWhenRefactorHappened(t *testing.T) {
|
||||
task := loadBaselineTask(t, "refactor-01")
|
||||
outcome := runTurnStub(t, task, `sed -e 's/return fmt.Sprintf("hello, %s", c.Name)/return formatGreeting(c.Name)/' -e 's/return fmt.Sprintf("hello, %s", name)/return formatGreeting(name)/' main.go > .zero-tmp
|
||||
cat >> .zero-tmp <<'EOF'
|
||||
|
||||
func formatGreeting(name string) string {
|
||||
return fmt.Sprintf("hello, %s", name)
|
||||
}
|
||||
EOF
|
||||
mv .zero-tmp main.go
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("real refactor should pass, got harness error: %v", outcome.Err)
|
||||
}
|
||||
if !outcome.Passed {
|
||||
t.Fatalf("refactor-01 oracle must pass when formatGreeting is extracted: %+v", outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEdit03PassesWhenFieldAdded proves edit-03's oracle passes when the field
|
||||
// is actually added: Config gets a Label string field, so `var _ string =
|
||||
// Config{}.Label` compiles and the task passes.
|
||||
func TestEdit03PassesWhenFieldAdded(t *testing.T) {
|
||||
task := loadBaselineTask(t, "edit-03")
|
||||
outcome := runTurnStub(t, task, `awk '/Name string/ && !d {print; print "Label string"; d=1; next} {print}' main.go > .zero-tmp && mv .zero-tmp main.go
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("real field-add should pass, got harness error: %v", outcome.Err)
|
||||
}
|
||||
if !outcome.Passed {
|
||||
t.Fatalf("edit-03 oracle must pass when Label is added: %+v", outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNav09PassesWithCorrectAnswer proves the nav-09 oracle passes when the
|
||||
// captured answer names all three config keys.
|
||||
func TestNav09PassesWithCorrectAnswer(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-09")
|
||||
outcome := runTurnStub(t, task, `echo '{"type":"final","text":"config.json has three keys: port, name, and retries."}'
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("correct nav answer should pass, got harness error: %v", outcome.Err)
|
||||
}
|
||||
if !outcome.Passed {
|
||||
t.Fatalf("nav-09 oracle must pass when the answer names port/name/retries: %+v", outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRefactor05PassesWhenInlined proves refactor-05's oracle is not an
|
||||
// always-fail gate: when the stub applies the real inline (greetWrapped calls
|
||||
// GreetByName directly, Wrapper removed), TestGreetWrapped gets "hello, x" and
|
||||
// the ! grep 'func Wrapper' check passes, so the task passes.
|
||||
func TestRefactor05PassesWhenInlined(t *testing.T) {
|
||||
task := loadBaselineTask(t, "refactor-05")
|
||||
outcome := runTurnStub(t, task, `sed -e 's/return Wrapper(name)/return GreetByName(name)/' -e '/^func Wrapper(name string) string {/,/^}/d' main.go > .zero-tmp && mv .zero-tmp main.go
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("real inline should pass, got harness error: %v", outcome.Err)
|
||||
}
|
||||
if !outcome.Passed {
|
||||
t.Fatalf("refactor-05 oracle must pass when Wrapper is inlined into greetWrapped: %+v", outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRefactor04PassesWhenTyped proves refactor-04's oracle passes when the
|
||||
// bare map is actually replaced by the named type: the stub introduces
|
||||
// `type Stats map[string]int` and retypes the var to `var stats = Stats{}`,
|
||||
// so the bare-map regex no longer matches, Stats is referenced, `var _ Stats`
|
||||
// compiles, and go test passes.
|
||||
func TestRefactor04PassesWhenTyped(t *testing.T) {
|
||||
task := loadBaselineTask(t, "refactor-04")
|
||||
outcome := runTurnStub(t, task, `sed -e 's/type Config struct {/type Stats map[string]int\n\ntype Config struct {/' -e 's/var stats = map\[string\]int{}/var stats = Stats{}/' main.go > .zero-tmp && mv .zero-tmp main.go
|
||||
echo '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("real typed refactor should pass, got harness error: %v", outcome.Err)
|
||||
}
|
||||
if !outcome.Passed {
|
||||
t.Fatalf("refactor-04 oracle must pass when stats is typed as Stats: %+v", outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStampOracleRejectsFixturelessOracleTask covers the fail-closed path in
|
||||
// stampOracleAndAnswer: a task that carries an oracle (a stamped OracleTest or a
|
||||
// verification command) but has no workspace fixture has nowhere safe to stamp,
|
||||
// so it must error rather than let runVerification run in the caller's cwd with
|
||||
// the oracle never compiled (or the answer never captured). A fixtureless task
|
||||
// with no oracle and no verification (a pure latency-only run in the caller's
|
||||
// cwd) is left alone. This is a direct unit test (no stub), so it runs on all
|
||||
// platforms.
|
||||
func TestStampOracleRejectsFixturelessOracleTask(t *testing.T) {
|
||||
// OracleTest set, no verificationCommand: isolates the `OracleTest != ""`
|
||||
// reject branch so a regression that only checked VerificationCommand would
|
||||
// fail here (the both-set case below would still pass it).
|
||||
withOracleTest := BenchTask{
|
||||
ID: "t",
|
||||
OracleTest: "package x\n\nvar _ = Missing\n",
|
||||
}
|
||||
if err := stampOracleAndAnswer(withOracleTest, nil); err == nil {
|
||||
t.Fatal("fixtureless task with an OracleTest must be rejected, got nil error")
|
||||
}
|
||||
// verificationCommand set, no OracleTest: isolates the other reject branch.
|
||||
withVerify := BenchTask{
|
||||
ID: "t",
|
||||
VerificationCommand: []string{"bash", "-c", "grep x .zero-answer.txt"},
|
||||
}
|
||||
if err := stampOracleAndAnswer(withVerify, nil); err == nil {
|
||||
t.Fatal("fixtureless task with a verificationCommand must be rejected, got nil error")
|
||||
}
|
||||
// Both set: the `||` rejects, and this guards against a regression that
|
||||
// inverted the logic to `&&`.
|
||||
withBoth := BenchTask{
|
||||
ID: "t",
|
||||
OracleTest: "package x\n\nvar _ = Missing\n",
|
||||
VerificationCommand: []string{"go", "test", "./..."},
|
||||
}
|
||||
if err := stampOracleAndAnswer(withBoth, nil); err == nil {
|
||||
t.Fatal("fixtureless task with both an OracleTest and a verificationCommand must be rejected, got nil error")
|
||||
}
|
||||
latencyOnly := BenchTask{ID: "t"}
|
||||
if err := stampOracleAndAnswer(latencyOnly, nil); err != nil {
|
||||
t.Fatalf("fixtureless latency-only task must not be rejected, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// --- Count-oracle tests: anchored `^count: N$` with the exact expected count ---
|
||||
|
||||
// TestNav01CountOracleAcceptsExactCount proves nav-01's anchored count oracle
|
||||
// passes when the answer names the files AND states the exact count (5 — the
|
||||
// fixture has README.md, config.json, go.mod, main.go, main_test.go) on its own
|
||||
// line. printf emits the JSON with a literal \n escape so the captured answer
|
||||
// has "count: 5" on its own line.
|
||||
func TestNav01CountOracleAcceptsExactCount(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-01")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"main.go, config.json, go.mod, README.md, main_test.go\ncount: 5"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("correct nav-01 answer should pass, got harness error: %v", outcome.Err)
|
||||
}
|
||||
if !outcome.Passed {
|
||||
t.Fatalf("nav-01 oracle must pass when the answer names the files and states count: 5: %+v", outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNav01CountOracleRejectsWrongCount proves the anchored count oracle rejects
|
||||
// a wrong count: the file names are all present, but "count: 3" does not match
|
||||
// `^count: 5$`, so the task fails at verification.
|
||||
func TestNav01CountOracleRejectsWrongCount(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-01")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"main.go, config.json, go.mod, README.md, main_test.go\ncount: 3"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "nav-01 wrong count", outcome)
|
||||
}
|
||||
|
||||
// TestNav04CountOracleRejectsSubstring proves the line anchor rejects a
|
||||
// substring: "the count: 1 is the number" contains "count: 1" (the right
|
||||
// value) but the line does not start with "count:", so `^count:[[:space:]]*1$`
|
||||
// does not match. A pre-anchor oracle (`count:\s*1`) would have rubber-stamped
|
||||
// this.
|
||||
func TestNav04CountOracleRejectsSubstring(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-04")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"the count: 1 is the number of test functions"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "nav-04 substring count", outcome)
|
||||
}
|
||||
|
||||
// TestNav04CountOracleAcceptsExactCount proves the nav-04 oracle passes for a
|
||||
// real answer: the count is anchored on its own line AND the answer names the
|
||||
// one Test* function (TestGreet, from main_test.go). The fixture's ground truth
|
||||
// is exactly one test function, so count: 1 is right, and naming TestGreet is
|
||||
// the inspection-proof that the agent actually read main_test.go.
|
||||
func TestNav04CountOracleAcceptsExactCount(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-04")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"There is one test function: TestGreet in main_test.go.\ncount: 1"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("correct nav-04 answer should pass, got harness error: %v", outcome.Err)
|
||||
}
|
||||
if !outcome.Passed {
|
||||
t.Fatalf("nav-04 oracle must pass when the answer names TestGreet and states count: 1 on its own line: %+v", outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNav04CountOracleRejectsZeroGuess is the count=0 gameability gate: an agent
|
||||
// that never opens the workspace and always emits "count: 0" used to pass nav-04
|
||||
// when the ground truth was 0. The fixture now has one real Test* function, so
|
||||
// the ground truth is 1 and a clean "count: 0" fails.
|
||||
func TestNav04CountOracleRejectsZeroGuess(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-04")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 0"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "nav-04 zero-guess", outcome)
|
||||
}
|
||||
|
||||
// TestNav04CountOracleRejectsBlindCountOne is the count=1 gameability gate: once
|
||||
// the ground truth became 1, an agent that never opens the workspace and always
|
||||
// emits "count: 1" would pass a count-only oracle. nav-04's oracle also requires
|
||||
// the answer to name the test function (TestGreet), so a bare "count: 1" with no
|
||||
// named fact fails — proving the blind-count-one surface is closed.
|
||||
func TestNav04CountOracleRejectsBlindCountOne(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-04")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 1"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "nav-04 blind-count-one", outcome)
|
||||
}
|
||||
|
||||
// TestNav05CountOracleAcceptsExactCount proves nav-05's oracle passes for a real
|
||||
// answer: the count is anchored on its own line AND the answer names the file
|
||||
// holding the one TODO (main.go). Naming main.go is the inspection-proof that the
|
||||
// agent actually located the TODO rather than guessing the count.
|
||||
func TestNav05CountOracleAcceptsExactCount(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-05")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"One TODO in main.go.\ncount: 1"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("correct nav-05 answer should pass, got harness error: %v", outcome.Err)
|
||||
}
|
||||
if !outcome.Passed {
|
||||
t.Fatalf("nav-05 oracle must pass when the answer names main.go and states count: 1: %+v", outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNav05CountOracleRejectsZeroGuess is nav-05's count=0 gameability gate: the
|
||||
// fixture now has one real TODO, so a clean "count: 0" (the always-guess-zero
|
||||
// answer) fails.
|
||||
func TestNav05CountOracleRejectsZeroGuess(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-05")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 0"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "nav-05 zero-guess", outcome)
|
||||
}
|
||||
|
||||
// TestNav05CountOracleRejectsBlindCountOne is nav-05's count=1 gameability gate:
|
||||
// once the ground truth became 1, an agent that never opens the workspace and
|
||||
// always emits "count: 1" would pass a count-only oracle. nav-05's oracle also
|
||||
// requires the answer to name the file (main.go), so a bare "count: 1" with no
|
||||
// named file fails — proving the blind-count-one surface is closed.
|
||||
func TestNav05CountOracleRejectsBlindCountOne(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-05")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"count: 1"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "nav-05 blind-count-one", outcome)
|
||||
}
|
||||
|
||||
// TestNav05CountOracleRejectsSubstring proves the line anchor still bites even
|
||||
// when the named fact is present: "the count: 1 is the number of TODOs in
|
||||
// main.go" names main.go and contains "count: 1" (the right value), but the
|
||||
// count does not stand on its own line, so `^count:[[:space:]]*1$` does not
|
||||
// match and the oracle fails. A pre-anchor oracle (`count:\s*1`) would have
|
||||
// rubber-stamped this. Mirrors TestNav04CountOracleRejectsSubstring.
|
||||
func TestNav05CountOracleRejectsSubstring(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-05")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"the count: 1 is the number of TODOs in main.go"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "nav-05 substring count", outcome)
|
||||
}
|
||||
|
||||
// TestNav08CountOracleAcceptsStdlibAnswer proves nav-08's oracle passes when the
|
||||
// agent enumerated the imports across BOTH .go files and named the stdlib ones
|
||||
// (fmt from main.go, testing from main_test.go): the answer states the only
|
||||
// imports are stdlib so there are 0 third-party, with "count: 0" on its own
|
||||
// line. The fmt+testing requirement is the inspection-proof — it forces the
|
||||
// agent to have read all the files rather than blindly emitting "count: 0".
|
||||
func TestNav08CountOracleAcceptsStdlibAnswer(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-08")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"The only imports are fmt (main.go) and testing (main_test.go), both standard library, so there are no third-party imports.\ncount: 0"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
if outcome.Err != nil {
|
||||
t.Fatalf("correct nav-08 answer should pass, got harness error: %v", outcome.Err)
|
||||
}
|
||||
if !outcome.Passed {
|
||||
t.Fatalf("nav-08 oracle must pass when the answer names fmt+testing and states count: 0: %+v", outcome)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNav08CountOracleRejectsAnswerWithoutFmt proves the inspection-proof bites:
|
||||
// an answer that states the right count (0) but never names the stdlib imports it
|
||||
// examined ("No third-party imports. count: 0") fails, so an agent that didn't
|
||||
// actually look at the imports can't pass by guessing the count alone.
|
||||
func TestNav08CountOracleRejectsAnswerWithoutFmt(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-08")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"No third-party imports.\ncount: 0"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "nav-08 no-fmt", outcome)
|
||||
}
|
||||
|
||||
// TestNav08CountOracleRejectsFmtOnlyNoTesting proves the tighter inspection-proof:
|
||||
// an answer that names fmt (from main.go) but misses testing (from main_test.go)
|
||||
// fails, so an agent that only read main.go — or a guesser that names the one
|
||||
// most-common stdlib import blind — can't pass. The agent must enumerate every
|
||||
// file's imports, which is what the prompt asks for.
|
||||
func TestNav08CountOracleRejectsFmtOnlyNoTesting(t *testing.T) {
|
||||
task := loadBaselineTask(t, "nav-08")
|
||||
outcome := runTurnStub(t, task, `printf '%s\n' '{"type":"final","text":"The only import is fmt, which is standard library. count: 0"}'
|
||||
printf '%s\n' '{"type":"run_end","exitCode":0}'
|
||||
`)
|
||||
assertVerifyFailed(t, "nav-08 fmt-only-no-testing", outcome)
|
||||
}
|
||||
@@ -34,7 +34,7 @@ const (
|
||||
|
||||
var ErrUnknownProvider = errors.New("unknown provider")
|
||||
|
||||
// AIMLAPIID is the provider catalog identifier for the AI/ML API preset.
|
||||
// AIMLAPIID is the provider catalog identifier for the aimlapi.com preset.
|
||||
const AIMLAPIID = "aimlapi"
|
||||
|
||||
type Descriptor struct {
|
||||
@@ -44,6 +44,7 @@ type Descriptor struct {
|
||||
DefaultBaseURL string
|
||||
DefaultModel string
|
||||
AuthEnvVars []string
|
||||
CustomHeaders map[string]string
|
||||
RequiresAuth bool
|
||||
UsesAmbientAuth bool
|
||||
Public bool
|
||||
@@ -70,9 +71,9 @@ type Descriptor struct {
|
||||
// headless / SSH use) in addition to the browser flow.
|
||||
OAuthDeviceFlow bool
|
||||
|
||||
// Recommended marks the provider that should be surfaced first and badged
|
||||
// (★ … (recommended)) in every catalog-ordered list and picker. At most one
|
||||
// descriptor should set this.
|
||||
// Recommended marks a provider surfaced at the top and badged in
|
||||
// catalog-ordered lists and pickers. The recommended descriptors are the first
|
||||
// entries in the catalog, in order.
|
||||
Recommended bool
|
||||
}
|
||||
|
||||
@@ -103,6 +104,11 @@ var descriptors = []Descriptor{
|
||||
// minimax, qwen, google, nvidia, tencent, z-ai). Flat /v1/chat/completions
|
||||
// with a Bearer ogw_live_… key; listed first and badged in every picker.
|
||||
recommended(openAICompat("gitlawb-opengateway", "GitLawb OpenGateway", "https://opengateway.gitlawb.com/v1", "mimo-v2.5-pro", []string{"GITLAWB_OPENGATEWAY_API_KEY"}, "gitlawb opengateway", "opengateway")),
|
||||
// aimlapi.com — OpenAI-compatible aggregating gateway, also badged as
|
||||
// recommended (second, right after the OpenGateway default). The built-in TUI
|
||||
// onboarding can register/top up an account and save the issued key; the partner
|
||||
// headers attribute usage for the rebate.
|
||||
recommended(aimlapi()),
|
||||
openAI("openai", "OpenAI", "https://api.openai.com/v1", "gpt-4.1", []string{"OPENAI_API_KEY"}),
|
||||
anthropic("anthropic", "Anthropic", "https://api.anthropic.com", "claude-sonnet-4.5", []string{"ANTHROPIC_API_KEY"}),
|
||||
google("google", "Google", "https://generativelanguage.googleapis.com", "gemini-2.5-pro", []string{"GEMINI_API_KEY", "GOOGLE_API_KEY"}, "gemini"),
|
||||
@@ -110,7 +116,6 @@ var descriptors = []Descriptor{
|
||||
localOpenAI("ollama", "Ollama Local", "http://localhost:11434/v1", "llama3.1", "ollama local"),
|
||||
localOpenAI("lmstudio", "LM Studio", "http://localhost:1234/v1", "local-model", "lm-studio", "lm studio"),
|
||||
oauthProvider(openAICompat("openrouter", "OpenRouter", "https://openrouter.ai/api/v1", "openai/gpt-4.1", []string{"OPENROUTER_API_KEY"}), true, false),
|
||||
openAICompat(AIMLAPIID, "AI/ML API", "https://api.aimlapi.com/v1", "openai/gpt-5-chat", []string{"AIMLAPI_API_KEY"}, "aiml api", "ai/ml api"),
|
||||
// Hugging Face Inference Providers — OpenAI-compatible router at
|
||||
// https://router.huggingface.co/v1 exposes hundreds of OSS models. OAuth
|
||||
// requires a one-time app registration at huggingface.co/settings/applications/new
|
||||
@@ -302,8 +307,21 @@ func google(id string, name string, baseURL string, model string, env []string,
|
||||
}
|
||||
}
|
||||
|
||||
// recommended marks a descriptor as the recommended default so list/picker
|
||||
// surfaces sort it first and render the ★ … (recommended) badge.
|
||||
func aimlapi() Descriptor {
|
||||
descriptor := openAICompat(AIMLAPIID, "aimlapi.com", "https://api.aimlapi.com/v1", "anthropic/claude-sonnet-5", []string{"AIMLAPI_API_KEY"}, "aimlapi", "aiml api", "ai/ml api", "ai ml api")
|
||||
descriptor.CustomHeaders = map[string]string{
|
||||
"X-AIMLAPI-Partner-ID": "part_62yQoGYDq4Yqnrj2R1iGrDNJ",
|
||||
"X-AIMLAPI-Integration-Repo": "Gitlawb/zero",
|
||||
"X-AIMLAPI-Integration-Version": "zero",
|
||||
}
|
||||
// Onboarding is key-based only: the TUI sub-flow takes an existing key or an
|
||||
// email top-up (internal/aimlapi Paths A/B). Not flagged
|
||||
// OAuth-capable, so it does not appear in the wizard's "Sign in with OAuth" list.
|
||||
return descriptor
|
||||
}
|
||||
|
||||
// recommended marks a descriptor as a recommended default so list/picker
|
||||
// surfaces sort it to the top and render a provider-specific badge.
|
||||
func recommended(descriptor Descriptor) Descriptor {
|
||||
descriptor.Recommended = true
|
||||
return descriptor
|
||||
@@ -384,5 +402,19 @@ func cloneDescriptor(descriptor Descriptor) Descriptor {
|
||||
descriptor.AuthEnvVars = append([]string{}, descriptor.AuthEnvVars...)
|
||||
descriptor.SupportedAPIFormats = append([]APIFormat{}, descriptor.SupportedAPIFormats...)
|
||||
descriptor.Aliases = append([]string{}, descriptor.Aliases...)
|
||||
if descriptor.CustomHeaders != nil {
|
||||
descriptor.CustomHeaders = copyStringMap(descriptor.CustomHeaders)
|
||||
}
|
||||
return descriptor
|
||||
}
|
||||
|
||||
func copyStringMap(values map[string]string) map[string]string {
|
||||
if values == nil {
|
||||
return nil
|
||||
}
|
||||
copied := make(map[string]string, len(values))
|
||||
for key, value := range values {
|
||||
copied[key] = value
|
||||
}
|
||||
return copied
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
|
||||
var expectedCatalogIDs = []string{
|
||||
"gitlawb-opengateway",
|
||||
"aimlapi",
|
||||
"openai",
|
||||
"anthropic",
|
||||
"google",
|
||||
@@ -16,7 +17,6 @@ var expectedCatalogIDs = []string{
|
||||
"ollama",
|
||||
"lmstudio",
|
||||
"openrouter",
|
||||
"aimlapi",
|
||||
"huggingface",
|
||||
"chatgpt",
|
||||
"groq",
|
||||
@@ -69,25 +69,31 @@ func TestAllHasStableUniqueIDs(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRecommendedProviderIsFirstAndUnique(t *testing.T) {
|
||||
func TestRecommendedProvidersAreTopOfCatalog(t *testing.T) {
|
||||
descriptors := All()
|
||||
if len(descriptors) == 0 {
|
||||
t.Fatal("All() returned no descriptors")
|
||||
// The recommended providers are badged and pinned to the top of the catalog,
|
||||
// in this order (OpenGateway remains the default; aimlapi.com is also badged).
|
||||
wantTop := []string{"gitlawb-opengateway", "aimlapi"}
|
||||
if len(descriptors) < len(wantTop) {
|
||||
t.Fatalf("All() returned %d descriptors, want at least %d", len(descriptors), len(wantTop))
|
||||
}
|
||||
if !descriptors[0].Recommended {
|
||||
t.Fatalf("All()[0] = %q, want it to be the recommended provider", descriptors[0].ID)
|
||||
}
|
||||
if descriptors[0].ID != "gitlawb-opengateway" {
|
||||
t.Fatalf("recommended provider = %q, want %q", descriptors[0].ID, "gitlawb-opengateway")
|
||||
for index, id := range wantTop {
|
||||
if descriptors[index].ID != id {
|
||||
t.Fatalf("descriptors[%d] = %q, want %q", index, descriptors[index].ID, id)
|
||||
}
|
||||
if !descriptors[index].Recommended {
|
||||
t.Fatalf("descriptors[%d] (%q) should be recommended", index, id)
|
||||
}
|
||||
}
|
||||
// Exactly those are recommended, and they are contiguous at the top.
|
||||
recommended := 0
|
||||
for _, descriptor := range descriptors {
|
||||
if descriptor.Recommended {
|
||||
recommended++
|
||||
}
|
||||
}
|
||||
if recommended != 1 {
|
||||
t.Fatalf("recommended descriptor count = %d, want exactly 1", recommended)
|
||||
if recommended != len(wantTop) {
|
||||
t.Fatalf("recommended descriptor count = %d, want %d", recommended, len(wantTop))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -109,14 +115,14 @@ func TestAIMLAPIDescriptor(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Require(aimlapi) error = %v", err)
|
||||
}
|
||||
if descriptor.Name != "AI/ML API" {
|
||||
t.Fatalf("Name = %q, want AI/ML API", descriptor.Name)
|
||||
if descriptor.Name != "aimlapi.com" {
|
||||
t.Fatalf("Name = %q, want aimlapi.com", descriptor.Name)
|
||||
}
|
||||
if descriptor.DefaultBaseURL != "https://api.aimlapi.com/v1" {
|
||||
t.Fatalf("DefaultBaseURL = %q, want AI/ML API endpoint", descriptor.DefaultBaseURL)
|
||||
t.Fatalf("DefaultBaseURL = %q, want aimlapi.com endpoint", descriptor.DefaultBaseURL)
|
||||
}
|
||||
if descriptor.DefaultModel != "openai/gpt-5-chat" {
|
||||
t.Fatalf("DefaultModel = %q, want openai/gpt-5-chat", descriptor.DefaultModel)
|
||||
if descriptor.DefaultModel != "anthropic/claude-sonnet-5" {
|
||||
t.Fatalf("DefaultModel = %q, want anthropic/claude-sonnet-5", descriptor.DefaultModel)
|
||||
}
|
||||
if descriptor.Transport != TransportOpenAICompatible {
|
||||
t.Fatalf("Transport = %q, want %q", descriptor.Transport, TransportOpenAICompatible)
|
||||
@@ -316,7 +322,7 @@ func TestListByTransportPreservesCatalogOrder(t *testing.T) {
|
||||
TransportBedrock: {"bedrock"},
|
||||
TransportVertex: {"vertex"},
|
||||
TransportAnthropicCompat: {"minimax", "minimaxi-cn", "opencode-go-anthropic-compatible", "custom-anthropic-compatible"},
|
||||
TransportOpenAICompat: {"gitlawb-opengateway", "ollama-cloud", "ollama", "lmstudio", "openrouter", "aimlapi", "huggingface", "chatgpt", "groq", "deepseek", "together", "dashscope", "moonshot", "longcat", "nvidia-nim", "mistral", "github", "xai", "venice", "xiaomi-mimo", "bankr", "zai", "zai-cn", "kilocode", "opencode", "opencode-go", "atomic-chat", "chatgpt-proxy", "custom-openai-compatible"},
|
||||
TransportOpenAICompat: {"gitlawb-opengateway", "aimlapi", "ollama-cloud", "ollama", "lmstudio", "openrouter", "huggingface", "chatgpt", "groq", "deepseek", "together", "dashscope", "moonshot", "longcat", "nvidia-nim", "mistral", "github", "xai", "venice", "xiaomi-mimo", "bankr", "zai", "zai-cn", "kilocode", "opencode", "opencode-go", "atomic-chat", "chatgpt-proxy", "custom-openai-compatible"},
|
||||
}
|
||||
|
||||
for transport, wantIDs := range cases {
|
||||
|
||||
@@ -61,6 +61,17 @@ var zaiCuratedModels = []Model{
|
||||
}
|
||||
|
||||
var curatedModels = map[string][]Model{
|
||||
// aimlapi.com aggregates every upstream provider behind one OpenAI-compatible
|
||||
// endpoint, so the curated defaults are a cross-provider "best of" flagship
|
||||
// list. The live /v1/models discovery (when a key is present) replaces this;
|
||||
// these are the offline/pre-key fallback and the picker's default order.
|
||||
"aimlapi": {
|
||||
{ID: "anthropic/claude-sonnet-5", Description: "catalog default (Anthropic Claude Sonnet 5)"},
|
||||
{ID: "google/gemini-3.5-flash", Description: "Google Gemini Flash 3.5"},
|
||||
{ID: "openai/gpt-5.5-2026-04-23", Description: "OpenAI GPT-5.5"},
|
||||
{ID: "qwen/qwen-3.7-max", Description: "Qwen 3.7 Max"},
|
||||
{ID: "deepseek/deepseek-v4-pro", Description: "DeepSeek V4"},
|
||||
},
|
||||
"ollama-cloud": {
|
||||
{ID: "qwen3-coder:480b", Description: "catalog default"},
|
||||
{ID: "gpt-oss:120b", Description: "agentic coding model"},
|
||||
|
||||
@@ -33,6 +33,11 @@ func TestModelsAreProviderScoped(t *testing.T) {
|
||||
want: []string{"gpt-5.5", "gpt-5.4", "gpt-5.4-mini", "gpt-5.3-codex-spark"},
|
||||
notWant: []string{"gpt-5", "gpt-4.1", "openai/gpt-4.1"},
|
||||
},
|
||||
{
|
||||
provider: "aimlapi",
|
||||
want: []string{"anthropic/claude-sonnet-5", "google/gemini-3.5-flash", "openai/gpt-5.5-2026-04-23", "qwen/qwen-3.7-max", "deepseek/deepseek-v4-pro"},
|
||||
notWant: []string{"gpt-4o", "openai/gpt-4o", "anthropic/claude-opus-4.8", "x-ai/grok-4-5"},
|
||||
},
|
||||
{
|
||||
provider: "mistral",
|
||||
want: []string{"mistral-large-latest", "codestral-latest"},
|
||||
|
||||
@@ -108,6 +108,57 @@ func New(profile config.ProviderProfile, options Options) (zeroruntime.Provider,
|
||||
}
|
||||
}
|
||||
|
||||
// NewTurnSessionProvider builds the default TurnSessionProvider for a resolved
|
||||
// profile: it constructs the provider exactly as New does, then wraps it with a
|
||||
// ProviderCapabilities projection computed from the same model-registry entry
|
||||
// New already resolves. The default session's Stream is the provider's
|
||||
// StreamCompletion, so runtime behavior is identical to using New directly.
|
||||
func NewTurnSessionProvider(profile config.ProviderProfile, options Options) (zeroruntime.TurnSessionProvider, error) {
|
||||
provider, err := New(profile, options)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
caps, err := resolveCapabilities(profile, options)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return zeroruntime.NewProviderTurnSessionProvider(provider, caps), nil
|
||||
}
|
||||
|
||||
// resolveCapabilities projects the resolved profile's model-registry entry into
|
||||
// the flat zeroruntime.ProviderCapabilities (zeroruntime cannot reference
|
||||
// modelregistry types — modelregistry imports zeroruntime, so a typed field
|
||||
// would form an import cycle). A model absent from the registry yields only the
|
||||
// resolved API model id and max-output tokens; the rest stays zero (unknown).
|
||||
func resolveCapabilities(profile config.ProviderProfile, options Options) (zeroruntime.ProviderCapabilities, error) {
|
||||
resolved, err := resolveProfile(profile, options)
|
||||
if err != nil {
|
||||
return zeroruntime.ProviderCapabilities{}, err
|
||||
}
|
||||
caps := zeroruntime.ProviderCapabilities{
|
||||
Model: resolved.apiModel,
|
||||
MaxOutputTokens: resolved.maxOutputTokens,
|
||||
}
|
||||
registry, err := defaultRegistry(options.ModelRegistry)
|
||||
if err != nil {
|
||||
return zeroruntime.ProviderCapabilities{}, err
|
||||
}
|
||||
if entry, ok := registry.Get(strings.TrimSpace(profile.Model)); ok {
|
||||
caps.ContextWindow = entry.ContextLimits.ContextWindow
|
||||
caps.SupportsVision = entry.Supports(modelregistry.ModelCapabilityVision)
|
||||
caps.SupportsReasoning = entry.Supports(modelregistry.ModelCapabilityReasoning)
|
||||
caps.SupportsPromptCache = entry.Supports(modelregistry.ModelCapabilityPromptCache)
|
||||
// Read efforts through Registry.ReasoningEfforts (not the raw entry) so
|
||||
// the projection matches what the /effort picker and the run-time
|
||||
// resolver advertise — including the name-based fallback for catalog
|
||||
// entries that enumerate no efforts of their own.
|
||||
for _, effort := range registry.ReasoningEfforts(entry.ID) {
|
||||
caps.ReasoningEfforts = append(caps.ReasoningEfforts, string(effort))
|
||||
}
|
||||
}
|
||||
return caps, nil
|
||||
}
|
||||
|
||||
func parseThinkTagsForProfile(profile config.ProviderProfile, resolved resolvedProfile) bool {
|
||||
if profile.ParseThinkTags != nil {
|
||||
return *profile.ParseThinkTags
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
package providers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/config"
|
||||
"github.com/Gitlawb/zero/internal/modelregistry"
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// TestNewTurnSessionProviderForEveryKind asserts every current provider kind
|
||||
// wraps into the default TurnSessionProvider: construction succeeds, a session
|
||||
// opens, and Compact reports unsupported (the default adapter contract).
|
||||
func TestNewTurnSessionProviderForEveryKind(t *testing.T) {
|
||||
kinds := []config.ProviderKind{
|
||||
config.ProviderKindOpenAI,
|
||||
config.ProviderKindOpenAICompatible,
|
||||
config.ProviderKindAnthropic,
|
||||
config.ProviderKindAnthropicCompat,
|
||||
config.ProviderKindGoogle,
|
||||
}
|
||||
for _, kind := range kinds {
|
||||
t.Run(string(kind), func(t *testing.T) {
|
||||
tsp, err := NewTurnSessionProvider(config.ProviderProfile{
|
||||
Name: "pr7-" + string(kind),
|
||||
ProviderKind: kind,
|
||||
BaseURL: "https://provider.example/v1",
|
||||
APIKey: "sk-turn-session-test",
|
||||
// A model absent from the registry: resolution falls back to the
|
||||
// raw id, so this test stays hermetic across catalog changes.
|
||||
Model: "pr7-unregistered-model",
|
||||
}, Options{UserAgent: "zero-turn-session-test"})
|
||||
if err != nil {
|
||||
t.Fatalf("NewTurnSessionProvider(%s): %v", kind, err)
|
||||
}
|
||||
session, err := tsp.OpenTurnSession(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("OpenTurnSession(%s): %v", kind, err)
|
||||
}
|
||||
defer func() {
|
||||
if closeErr := session.Close(); closeErr != nil {
|
||||
t.Fatalf("Close(%s): %v", kind, closeErr)
|
||||
}
|
||||
}()
|
||||
if err := session.Prewarm(context.Background()); err != nil {
|
||||
t.Fatalf("Prewarm(%s): %v", kind, err)
|
||||
}
|
||||
if _, compactErr := session.Compact(context.Background(), zeroruntime.CompletionRequest{}); !errors.Is(compactErr, zeroruntime.ErrCompactionUnsupported) {
|
||||
t.Fatalf("Compact(%s) = %v, want ErrCompactionUnsupported", kind, compactErr)
|
||||
}
|
||||
caps := tsp.Capabilities()
|
||||
if caps.Model != "pr7-unregistered-model" {
|
||||
t.Fatalf("Capabilities(%s).Model = %q, want the resolved api model", kind, caps.Model)
|
||||
}
|
||||
if caps.NativeCompaction {
|
||||
t.Fatalf("Capabilities(%s).NativeCompaction = true, want false for the default adapter", kind)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewTurnSessionProviderProjectsRegistryCapabilities asserts the factory
|
||||
// projects the model-registry entry (context limits, capability flags,
|
||||
// reasoning efforts) into the flat ProviderCapabilities.
|
||||
func TestNewTurnSessionProviderProjectsRegistryCapabilities(t *testing.T) {
|
||||
registry, err := modelregistry.NewRegistry([]modelregistry.ModelEntry{{
|
||||
ID: "pr7-caps-model",
|
||||
DisplayName: "PR7 Capability Probe",
|
||||
APIModel: "pr7-caps-api-model",
|
||||
Provider: modelregistry.ProviderOpenAI,
|
||||
ContextLimits: modelregistry.ContextLimits{
|
||||
ContextWindow: 200_000,
|
||||
MaxOutputTokens: 64_000,
|
||||
},
|
||||
ReasoningEfforts: []modelregistry.ReasoningEffort{
|
||||
modelregistry.ReasoningEffortLow,
|
||||
modelregistry.ReasoningEffortMedium,
|
||||
modelregistry.ReasoningEffortHigh,
|
||||
},
|
||||
Capabilities: modelregistry.ModelCapabilities{
|
||||
modelregistry.ModelCapabilityChat,
|
||||
modelregistry.ModelCapabilityVision,
|
||||
modelregistry.ModelCapabilityReasoning,
|
||||
modelregistry.ModelCapabilityPromptCache,
|
||||
},
|
||||
Cost: modelregistry.ModelCost{
|
||||
Currency: "USD",
|
||||
Unit: "per_1m_tokens",
|
||||
InputPerMillion: 1,
|
||||
OutputPerMillion: 2,
|
||||
Source: "https://provider.example/pricing (test fixture)",
|
||||
SourceLastVerified: "2026-07-18",
|
||||
},
|
||||
Status: modelregistry.ModelStatusActive,
|
||||
Aliases: []string{"pr7-caps"},
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatalf("NewRegistry: %v", err)
|
||||
}
|
||||
|
||||
tsp, err := NewTurnSessionProvider(config.ProviderProfile{
|
||||
Name: "pr7-caps",
|
||||
ProviderKind: config.ProviderKindOpenAI,
|
||||
BaseURL: "https://provider.example/v1",
|
||||
APIKey: "sk-turn-session-test",
|
||||
Model: "pr7-caps-model",
|
||||
}, Options{
|
||||
UserAgent: "zero-turn-session-test",
|
||||
ModelRegistry: ®istry,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewTurnSessionProvider: %v", err)
|
||||
}
|
||||
|
||||
caps := tsp.Capabilities()
|
||||
if caps.Model != "pr7-caps-api-model" {
|
||||
t.Fatalf("Model = %q, want the registry APIModel", caps.Model)
|
||||
}
|
||||
if caps.ContextWindow != 200_000 || caps.MaxOutputTokens != 64_000 {
|
||||
t.Fatalf("limits = (%d, %d), want (200000, 64000)", caps.ContextWindow, caps.MaxOutputTokens)
|
||||
}
|
||||
if !caps.SupportsVision || !caps.SupportsReasoning || !caps.SupportsPromptCache {
|
||||
t.Fatalf("capability flags = %+v, want vision/reasoning/prompt-cache all true", caps)
|
||||
}
|
||||
if len(caps.ReasoningEfforts) != 3 || caps.ReasoningEfforts[0] != "low" || caps.ReasoningEfforts[2] != "high" {
|
||||
t.Fatalf("ReasoningEfforts = %v, want [low medium high]", caps.ReasoningEfforts)
|
||||
}
|
||||
if caps.NativeCompaction {
|
||||
t.Fatal("NativeCompaction = true, want false for the default adapter")
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewTurnSessionProviderUsesEffectiveReasoningEfforts asserts the
|
||||
// projection reads efforts through Registry.ReasoningEfforts, so a catalog
|
||||
// entry that enumerates no efforts of its own still reports the name-inferred
|
||||
// effective tiers the /effort picker and run-time resolver advertise.
|
||||
func TestNewTurnSessionProviderUsesEffectiveReasoningEfforts(t *testing.T) {
|
||||
registry, err := modelregistry.NewRegistry([]modelregistry.ModelEntry{{
|
||||
// A gpt-5-family id with NO ReasoningEfforts listed: the effective
|
||||
// efforts come from name inference, differing from the raw entry.
|
||||
ID: "gpt-5-pr7-probe",
|
||||
DisplayName: "PR7 Effective Efforts Probe",
|
||||
APIModel: "gpt-5-pr7-probe-api",
|
||||
Provider: modelregistry.ProviderOpenAI,
|
||||
ContextLimits: modelregistry.ContextLimits{
|
||||
ContextWindow: 400_000,
|
||||
MaxOutputTokens: 128_000,
|
||||
},
|
||||
Capabilities: modelregistry.ModelCapabilities{
|
||||
modelregistry.ModelCapabilityChat,
|
||||
modelregistry.ModelCapabilityReasoning,
|
||||
},
|
||||
Cost: modelregistry.ModelCost{
|
||||
Currency: "USD",
|
||||
Unit: "per_1m_tokens",
|
||||
InputPerMillion: 1,
|
||||
OutputPerMillion: 2,
|
||||
Source: "https://provider.example/pricing (test fixture)",
|
||||
SourceLastVerified: "2026-07-18",
|
||||
},
|
||||
Status: modelregistry.ModelStatusActive,
|
||||
Aliases: []string{"pr7-effective-efforts"},
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatalf("NewRegistry: %v", err)
|
||||
}
|
||||
|
||||
tsp, err := NewTurnSessionProvider(config.ProviderProfile{
|
||||
Name: "pr7-effective",
|
||||
ProviderKind: config.ProviderKindOpenAI,
|
||||
BaseURL: "https://provider.example/v1",
|
||||
APIKey: "sk-turn-session-test",
|
||||
Model: "gpt-5-pr7-probe",
|
||||
}, Options{
|
||||
UserAgent: "zero-turn-session-test",
|
||||
ModelRegistry: ®istry,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewTurnSessionProvider: %v", err)
|
||||
}
|
||||
|
||||
got := tsp.Capabilities().ReasoningEfforts
|
||||
want := []string{"minimal", "low", "medium", "high"}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("ReasoningEfforts = %v, want %v (name-inferred fallback)", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("ReasoningEfforts[%d] = %q, want %q (full: %v)", i, got[i], want[i], got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,205 @@
|
||||
package openai
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/trace"
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// prewarmTimeout bounds the best-effort prewarm probe. The probe runs in the
|
||||
// background, so this cap only limits how long the goroutine may linger — it
|
||||
// can never delay the first turn.
|
||||
const prewarmTimeout = 3 * time.Second
|
||||
|
||||
// NewTurnSessionProvider wraps an already-constructed *Provider in the
|
||||
// optimized OpenAI turn session: a best-effort connection prewarm at run start
|
||||
// plus request-prefix stability telemetry. Stream is Provider.StreamCompletion
|
||||
// verbatim, so runtime request behavior is identical to the default adapter —
|
||||
// the session adds observation and pool priming, never a different request.
|
||||
func NewTurnSessionProvider(provider *Provider, caps zeroruntime.ProviderCapabilities) zeroruntime.TurnSessionProvider {
|
||||
return &turnSessionProvider{provider: provider, caps: caps}
|
||||
}
|
||||
|
||||
type turnSessionProvider struct {
|
||||
provider *Provider
|
||||
caps zeroruntime.ProviderCapabilities
|
||||
}
|
||||
|
||||
func (p *turnSessionProvider) OpenTurnSession(context.Context) (zeroruntime.TurnSession, error) {
|
||||
return &turnSession{provider: p.provider}, nil
|
||||
}
|
||||
|
||||
func (p *turnSessionProvider) Capabilities() zeroruntime.ProviderCapabilities {
|
||||
return p.caps
|
||||
}
|
||||
|
||||
// turnSession is one run's optimized OpenAI session. The agent loop serializes
|
||||
// all provider I/O through its session shim, so fields need no mutex. The
|
||||
// fingerprint counters exist so a future stateful-reuse session (Responses API)
|
||||
// inherits a proven prefix-stability detector; on chat completions there is no
|
||||
// server-side response state to invalidate, so drift is telemetry only.
|
||||
type turnSession struct {
|
||||
provider *Provider
|
||||
lastFingerprint string
|
||||
prewarmOnce sync.Once
|
||||
// prewarmDone closes when the background probe settles; tests wait on it.
|
||||
prewarmDone chan struct{}
|
||||
}
|
||||
|
||||
// Prewarm launches one bounded, unauthenticated HEAD probe to the provider's
|
||||
// base URL in the background so the TCP+TLS handshake lands in the shared
|
||||
// connection pool while the loop assembles the first prompt. One attempt, no
|
||||
// retries, no bearer token — the goal is the handshake, not a 2xx (a 401/404/
|
||||
// 405 response still primes the pool). Always returns nil: prewarm is advisory
|
||||
// by contract and never required for correctness.
|
||||
//
|
||||
// The probe is issued directly through the client (NOT providerio.SendWithRetry)
|
||||
// so it stamps ONLY the provider_prewarm span — a nested provider_connect stamp
|
||||
// would contaminate the exact A/B metric this feature is measured by.
|
||||
//
|
||||
// When the client's transport cannot retain idle connections (macOS disables
|
||||
// keep-alives on the shared transport because degraded pooled connections are
|
||||
// indistinguishable from backend slowness there), the probe is skipped entirely
|
||||
// rather than spending a request that can never be reused. The pool's 30s idle
|
||||
// timeout bounds the warm window; a slower start simply falls back to a cold
|
||||
// dial, never worse than today.
|
||||
func (s *turnSession) Prewarm(ctx context.Context) error {
|
||||
s.prewarmOnce.Do(func() {
|
||||
done := make(chan struct{})
|
||||
s.prewarmDone = done
|
||||
provider := s.provider
|
||||
if !transportRetainsIdleConns(provider.httpClient) {
|
||||
close(done)
|
||||
return
|
||||
}
|
||||
recorder := trace.FromContext(ctx)
|
||||
go func() {
|
||||
defer close(done)
|
||||
probeCtx, cancel := context.WithTimeout(ctx, prewarmTimeout)
|
||||
defer cancel()
|
||||
span := recorder.Span(trace.SpanProviderPrewarm)
|
||||
defer span.End()
|
||||
recorder.Counter(trace.CounterPrewarmAttempts, 1)
|
||||
request, err := http.NewRequestWithContext(probeCtx, http.MethodHead, provider.baseURL, nil)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if provider.userAgent != "" {
|
||||
request.Header.Set("User-Agent", provider.userAgent)
|
||||
}
|
||||
response, err := provider.httpClient.Do(request)
|
||||
if err == nil {
|
||||
_ = response.Body.Close()
|
||||
}
|
||||
}()
|
||||
})
|
||||
return nil
|
||||
}
|
||||
|
||||
// transportRetainsIdleConns reports whether a warmed connection can outlive the
|
||||
// probe. A *http.Transport with DisableKeepAlives (the shared transport on
|
||||
// macOS) closes every connection after its request, so a prewarm there is pure
|
||||
// overhead. A nil transport is net/http's default (keep-alives on); an unknown
|
||||
// custom transport is assumed reusable — the probe stays a harmless best-effort.
|
||||
func transportRetainsIdleConns(client *http.Client) bool {
|
||||
if client == nil {
|
||||
return false
|
||||
}
|
||||
if transport, ok := client.Transport.(*http.Transport); ok {
|
||||
return !transport.DisableKeepAlives
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *turnSession) Stream(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
||||
s.observePrefix(ctx, request)
|
||||
return s.provider.StreamCompletion(ctx, request)
|
||||
}
|
||||
|
||||
func (s *turnSession) Compact(context.Context, zeroruntime.CompletionRequest) ([]zeroruntime.Message, error) {
|
||||
return nil, zeroruntime.ErrCompactionUnsupported
|
||||
}
|
||||
|
||||
// Close is an idempotent no-op: the shared transport pool owns the connections.
|
||||
func (s *turnSession) Close() error { return nil }
|
||||
|
||||
// observePrefix tracks whether the request-prefix parameters stayed stable
|
||||
// between this session's streams. The first stream seeds the fingerprint and
|
||||
// counts as neither stable nor drift.
|
||||
func (s *turnSession) observePrefix(ctx context.Context, request zeroruntime.CompletionRequest) {
|
||||
fingerprint := s.computeFingerprint(request)
|
||||
if s.lastFingerprint == "" {
|
||||
s.lastFingerprint = fingerprint
|
||||
return
|
||||
}
|
||||
recorder := trace.FromContext(ctx)
|
||||
if fingerprint == s.lastFingerprint {
|
||||
recorder.Counter(trace.CounterPrefixStable, 1)
|
||||
return
|
||||
}
|
||||
recorder.Counter(trace.CounterPrefixDrift, 1)
|
||||
s.lastFingerprint = fingerprint
|
||||
}
|
||||
|
||||
// computeFingerprint digests the wire-affecting request parameters: the
|
||||
// session's model and max-tokens (fixed at construction — CompletionRequest
|
||||
// carries no model field), the normalized reasoning effort as it would appear
|
||||
// on the wire, the prompt-cache key (constant within a session, and part of
|
||||
// the request the provider serializes), and an order-preserving digest of the
|
||||
// advertised tools (request serialization preserves tool order, so a reorder
|
||||
// changes the wire bytes and counts as drift). Messages are excluded — they
|
||||
// grow every turn; this fingerprints request parameters, not conversation
|
||||
// content.
|
||||
//
|
||||
// SCOPE: this is per-provider request-parameter TELEMETRY, not a complete
|
||||
// compatibility fingerprint. It deliberately does not incorporate the prompt
|
||||
// prefix hashes (base instructions / project context / skills — the
|
||||
// complete_prefix signal the agent loop already emits per turn). Any future
|
||||
// stateful response reuse MUST combine both signals plus the remaining
|
||||
// wire-affecting fields before reusing provider-side state; these counters
|
||||
// alone are not sufficient compatibility protection.
|
||||
func (s *turnSession) computeFingerprint(request zeroruntime.CompletionRequest) string {
|
||||
var builder strings.Builder
|
||||
builder.WriteString(s.provider.model)
|
||||
builder.WriteByte('|')
|
||||
fmt.Fprintf(&builder, "%d", s.provider.maxTokens)
|
||||
builder.WriteByte('|')
|
||||
builder.WriteString(openAIReasoningEffort(request.ReasoningEffort))
|
||||
builder.WriteByte('|')
|
||||
builder.WriteString(request.PromptCacheKey)
|
||||
builder.WriteByte('|')
|
||||
builder.WriteString(wireToolsDigest(request.Tools))
|
||||
sum := sha256.Sum256([]byte(builder.String()))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// wireToolsDigest renders the tool set in WIRE ORDER: each tool as name +
|
||||
// description + its JSON-marshaled parameters (encoding/json sorts map keys,
|
||||
// so each render is stable). Order is preserved deliberately — the OpenAI
|
||||
// request serializes tools in the order given, so a reorder changes the
|
||||
// request bytes and must count as drift. Descriptions are model-visible
|
||||
// request bytes too. A schema that fails to marshal is recorded under a
|
||||
// stable sentinel — this digest feeds telemetry counters, so cruder handling
|
||||
// of that pathological case is acceptable.
|
||||
func wireToolsDigest(tools []zeroruntime.ToolDefinition) string {
|
||||
rendered := make([]string, 0, len(tools))
|
||||
for _, tool := range tools {
|
||||
schema, err := json.Marshal(tool.Parameters)
|
||||
if err != nil {
|
||||
rendered = append(rendered, tool.Name+"\n"+tool.Description+"\n__non_json:"+tool.Name)
|
||||
continue
|
||||
}
|
||||
rendered = append(rendered, tool.Name+"\n"+tool.Description+"\n"+string(schema))
|
||||
}
|
||||
sum := sha256.Sum256([]byte(strings.Join(rendered, "\x00")))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
@@ -0,0 +1,387 @@
|
||||
package openai
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/trace"
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// keepAliveClient returns a client whose transport retains idle connections on
|
||||
// EVERY platform. The shared default transport disables keep-alives on macOS,
|
||||
// where the session correctly skips the probe — tests that assert the probe
|
||||
// fires must therefore pin a reusable transport to stay platform-deterministic.
|
||||
func keepAliveClient() *http.Client {
|
||||
transport := http.DefaultTransport.(*http.Transport).Clone()
|
||||
transport.DisableKeepAlives = false
|
||||
return &http.Client{Transport: transport}
|
||||
}
|
||||
|
||||
// newPrewarmTestProvider builds a provider whose transport always allows the
|
||||
// prewarm probe, regardless of host platform.
|
||||
func newPrewarmTestProvider(t *testing.T, handler http.HandlerFunc) *Provider {
|
||||
t.Helper()
|
||||
return newTestProviderWithOptions(t, Options{APIKey: "sk-test", HTTPClient: keepAliveClient()}, handler)
|
||||
}
|
||||
|
||||
func waitPrewarmDone(t *testing.T, session *turnSession) {
|
||||
t.Helper()
|
||||
if session.prewarmDone == nil {
|
||||
t.Fatal("prewarm was never launched")
|
||||
}
|
||||
select {
|
||||
case <-session.prewarmDone:
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("prewarm probe did not settle in time")
|
||||
}
|
||||
}
|
||||
|
||||
func openOptimizedSession(t *testing.T, provider *Provider) *turnSession {
|
||||
t.Helper()
|
||||
session, err := NewTurnSessionProvider(provider, zeroruntime.ProviderCapabilities{}).OpenTurnSession(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("OpenTurnSession: %v", err)
|
||||
}
|
||||
return session.(*turnSession)
|
||||
}
|
||||
|
||||
func TestTurnSessionPrewarmSendsSingleHEAD(t *testing.T) {
|
||||
var requests atomic.Int64
|
||||
var lastMethod atomic.Value
|
||||
provider := newPrewarmTestProvider(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
requests.Add(1)
|
||||
lastMethod.Store(r.Method)
|
||||
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||
})
|
||||
|
||||
session := openOptimizedSession(t, provider)
|
||||
if err := session.Prewarm(context.Background()); err != nil {
|
||||
t.Fatalf("Prewarm: %v", err)
|
||||
}
|
||||
waitPrewarmDone(t, session)
|
||||
|
||||
if got := requests.Load(); got != 1 {
|
||||
t.Fatalf("prewarm sent %d requests, want exactly 1 (no retries even on 405)", got)
|
||||
}
|
||||
if method, _ := lastMethod.Load().(string); method != http.MethodHead {
|
||||
t.Fatalf("prewarm method = %q, want HEAD", method)
|
||||
}
|
||||
|
||||
// A second Prewarm (e.g. after a mid-run swap re-open) must not re-probe.
|
||||
if err := session.Prewarm(context.Background()); err != nil {
|
||||
t.Fatalf("second Prewarm: %v", err)
|
||||
}
|
||||
waitPrewarmDone(t, session)
|
||||
if got := requests.Load(); got != 1 {
|
||||
t.Fatalf("second Prewarm re-probed: %d requests, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnSessionPrewarmNonFatalOnConnectError(t *testing.T) {
|
||||
// A server that is immediately closed leaves a port that refuses connections.
|
||||
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
|
||||
deadURL := server.URL
|
||||
server.Close()
|
||||
|
||||
provider, err := New(Options{APIKey: "sk-test", BaseURL: deadURL, Model: "gpt-test", HTTPClient: keepAliveClient()})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
session := openOptimizedSession(t, provider)
|
||||
if err := session.Prewarm(context.Background()); err != nil {
|
||||
t.Fatalf("Prewarm must be advisory even when the probe cannot connect, got %v", err)
|
||||
}
|
||||
waitPrewarmDone(t, session)
|
||||
}
|
||||
|
||||
func TestTurnSessionPrewarmTraceStamps(t *testing.T) {
|
||||
provider := newPrewarmTestProvider(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||
})
|
||||
recorder := trace.NewRecorder("session-test", "run-1", "test")
|
||||
recorder.Start()
|
||||
ctx := trace.WithContext(context.Background(), recorder)
|
||||
|
||||
session := openOptimizedSession(t, provider)
|
||||
if err := session.Prewarm(ctx); err != nil {
|
||||
t.Fatalf("Prewarm: %v", err)
|
||||
}
|
||||
waitPrewarmDone(t, session)
|
||||
|
||||
tr := recorder.Finish()
|
||||
if got := tr.Counter(trace.CounterPrewarmAttempts); got != 1 {
|
||||
t.Fatalf("prewarm_attempts = %d, want 1", got)
|
||||
}
|
||||
var sawSpan bool
|
||||
for _, span := range tr.Spans {
|
||||
if span.Name == trace.SpanProviderPrewarm {
|
||||
sawSpan = true
|
||||
}
|
||||
}
|
||||
if !sawSpan {
|
||||
t.Fatalf("missing %s span in %+v", trace.SpanProviderPrewarm, tr.Spans)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnSessionStreamDelegatesToProvider(t *testing.T) {
|
||||
handler := func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
_, _ = w.Write([]byte("data: {\"choices\":[{\"delta\":{\"content\":\"hello\"}}]}\n\n"))
|
||||
_, _ = w.Write([]byte("data: [DONE]\n\n"))
|
||||
}
|
||||
|
||||
direct := newTestProvider(t, handler)
|
||||
viaSession := newTestProvider(t, handler)
|
||||
session := openOptimizedSession(t, viaSession)
|
||||
|
||||
request := zeroruntime.CompletionRequest{
|
||||
Messages: []zeroruntime.Message{{Role: zeroruntime.MessageRoleUser, Content: "hi"}},
|
||||
}
|
||||
collect := func(stream <-chan zeroruntime.StreamEvent) []zeroruntime.StreamEvent {
|
||||
var events []zeroruntime.StreamEvent
|
||||
for event := range stream {
|
||||
events = append(events, event)
|
||||
}
|
||||
return events
|
||||
}
|
||||
|
||||
directStream, err := direct.StreamCompletion(context.Background(), request)
|
||||
if err != nil {
|
||||
t.Fatalf("direct StreamCompletion: %v", err)
|
||||
}
|
||||
sessionStream, err := session.Stream(context.Background(), request)
|
||||
if err != nil {
|
||||
t.Fatalf("session Stream: %v", err)
|
||||
}
|
||||
|
||||
directEvents := collect(directStream)
|
||||
sessionEvents := collect(sessionStream)
|
||||
if len(directEvents) != len(sessionEvents) {
|
||||
t.Fatalf("event counts diverged: direct=%d session=%d", len(directEvents), len(sessionEvents))
|
||||
}
|
||||
for i := range directEvents {
|
||||
if directEvents[i].Type != sessionEvents[i].Type || directEvents[i].Content != sessionEvents[i].Content {
|
||||
t.Fatalf("event %d diverged: direct=%+v session=%+v", i, directEvents[i], sessionEvents[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnSessionFingerprintIgnoresMessages(t *testing.T) {
|
||||
provider := newTestProvider(t, func(http.ResponseWriter, *http.Request) {})
|
||||
session := openOptimizedSession(t, provider)
|
||||
|
||||
tools := []zeroruntime.ToolDefinition{{Name: "read_file", Parameters: map[string]any{"type": "object"}}}
|
||||
base := session.computeFingerprint(zeroruntime.CompletionRequest{
|
||||
Messages: []zeroruntime.Message{{Role: zeroruntime.MessageRoleUser, Content: "turn one"}},
|
||||
Tools: tools,
|
||||
PromptCacheKey: "session-a",
|
||||
})
|
||||
grown := session.computeFingerprint(zeroruntime.CompletionRequest{
|
||||
Messages: []zeroruntime.Message{
|
||||
{Role: zeroruntime.MessageRoleUser, Content: "turn one"},
|
||||
{Role: zeroruntime.MessageRoleAssistant, Content: "reply"},
|
||||
{Role: zeroruntime.MessageRoleUser, Content: "turn two"},
|
||||
},
|
||||
Tools: tools,
|
||||
PromptCacheKey: "session-a",
|
||||
})
|
||||
if base != grown {
|
||||
t.Fatal("fingerprint changed when only messages grew — conversation content must not be an input")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnSessionFingerprintDriftOnCacheKey(t *testing.T) {
|
||||
provider := newTestProvider(t, func(http.ResponseWriter, *http.Request) {})
|
||||
session := openOptimizedSession(t, provider)
|
||||
|
||||
tools := []zeroruntime.ToolDefinition{{Name: "read_file", Parameters: map[string]any{"type": "object"}}}
|
||||
base := session.computeFingerprint(zeroruntime.CompletionRequest{Tools: tools, PromptCacheKey: "session-a"})
|
||||
changed := session.computeFingerprint(zeroruntime.CompletionRequest{Tools: tools, PromptCacheKey: "session-b"})
|
||||
if base == changed {
|
||||
t.Fatal("fingerprint did not drift on a changed prompt-cache key — it is a serialized request field")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnSessionFingerprintDriftOnToolsAndEffort(t *testing.T) {
|
||||
provider := newTestProvider(t, func(http.ResponseWriter, *http.Request) {})
|
||||
session := openOptimizedSession(t, provider)
|
||||
|
||||
base := session.computeFingerprint(zeroruntime.CompletionRequest{
|
||||
Tools: []zeroruntime.ToolDefinition{{Name: "read_file", Parameters: map[string]any{"type": "object"}}},
|
||||
})
|
||||
changedTools := session.computeFingerprint(zeroruntime.CompletionRequest{
|
||||
Tools: []zeroruntime.ToolDefinition{{Name: "write_file", Parameters: map[string]any{"type": "object"}}},
|
||||
})
|
||||
if base == changedTools {
|
||||
t.Fatal("fingerprint did not drift on a changed tool set")
|
||||
}
|
||||
changedEffort := session.computeFingerprint(zeroruntime.CompletionRequest{
|
||||
Tools: []zeroruntime.ToolDefinition{{Name: "read_file", Parameters: map[string]any{"type": "object"}}},
|
||||
ReasoningEffort: "high",
|
||||
})
|
||||
if base == changedEffort {
|
||||
t.Fatal("fingerprint did not drift on a changed reasoning effort")
|
||||
}
|
||||
// Unrecognized efforts normalize to omitted — same as empty.
|
||||
droppedEffort := session.computeFingerprint(zeroruntime.CompletionRequest{
|
||||
Tools: []zeroruntime.ToolDefinition{{Name: "read_file", Parameters: map[string]any{"type": "object"}}},
|
||||
ReasoningEffort: "xhigh",
|
||||
})
|
||||
if base != droppedEffort {
|
||||
t.Fatal("an effort the wire would omit must fingerprint like no effort")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnSessionFingerprintDriftOnToolDescription(t *testing.T) {
|
||||
provider := newTestProvider(t, func(http.ResponseWriter, *http.Request) {})
|
||||
session := openOptimizedSession(t, provider)
|
||||
|
||||
base := session.computeFingerprint(zeroruntime.CompletionRequest{
|
||||
Tools: []zeroruntime.ToolDefinition{{Name: "read_file", Description: "Reads a file.", Parameters: map[string]any{"type": "object"}}},
|
||||
})
|
||||
changedDescription := session.computeFingerprint(zeroruntime.CompletionRequest{
|
||||
Tools: []zeroruntime.ToolDefinition{{Name: "read_file", Description: "Reads a file, now with ranges.", Parameters: map[string]any{"type": "object"}}},
|
||||
})
|
||||
if base == changedDescription {
|
||||
t.Fatal("fingerprint did not drift on a description-only tool change — descriptions are model-visible request bytes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnSessionFingerprintDriftOnToolReorder(t *testing.T) {
|
||||
provider := newTestProvider(t, func(http.ResponseWriter, *http.Request) {})
|
||||
session := openOptimizedSession(t, provider)
|
||||
|
||||
forward := session.computeFingerprint(zeroruntime.CompletionRequest{Tools: []zeroruntime.ToolDefinition{
|
||||
{Name: "alpha", Parameters: map[string]any{"type": "object"}},
|
||||
{Name: "beta", Parameters: map[string]any{"type": "object"}},
|
||||
}})
|
||||
reversed := session.computeFingerprint(zeroruntime.CompletionRequest{Tools: []zeroruntime.ToolDefinition{
|
||||
{Name: "beta", Parameters: map[string]any{"type": "object"}},
|
||||
{Name: "alpha", Parameters: map[string]any{"type": "object"}},
|
||||
}})
|
||||
if forward == reversed {
|
||||
t.Fatal("fingerprint did not drift on a tool reorder — request serialization preserves tool order, so a reorder changes the wire bytes")
|
||||
}
|
||||
}
|
||||
|
||||
// TestTurnSessionPrewarmDoesNotStampProviderConnect guards the A/B metric: the
|
||||
// prewarm probe must appear ONLY under provider_prewarm — a provider_connect
|
||||
// stamp from the probe would contaminate the exact span the benchmark compares.
|
||||
func TestTurnSessionPrewarmDoesNotStampProviderConnect(t *testing.T) {
|
||||
provider := newPrewarmTestProvider(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||
})
|
||||
recorder := trace.NewRecorder("session-test", "run-1", "test")
|
||||
recorder.Start()
|
||||
ctx := trace.WithContext(context.Background(), recorder)
|
||||
|
||||
session := openOptimizedSession(t, provider)
|
||||
if err := session.Prewarm(ctx); err != nil {
|
||||
t.Fatalf("Prewarm: %v", err)
|
||||
}
|
||||
waitPrewarmDone(t, session)
|
||||
|
||||
tr := recorder.Finish()
|
||||
for _, span := range tr.Spans {
|
||||
if span.Name == trace.SpanProviderConnect {
|
||||
t.Fatalf("prewarm stamped a %s span — it must stay solely under %s", trace.SpanProviderConnect, trace.SpanProviderPrewarm)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTurnSessionPrewarmSkippedWhenKeepAlivesDisabled verifies the probe is not
|
||||
// spent when the transport cannot retain idle connections (the macOS shared
|
||||
// transport): no request, no trace stamps, and the done channel still closes.
|
||||
func TestTurnSessionPrewarmSkippedWhenKeepAlivesDisabled(t *testing.T) {
|
||||
var requests atomic.Int64
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
requests.Add(1)
|
||||
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
transport := http.DefaultTransport.(*http.Transport).Clone()
|
||||
transport.DisableKeepAlives = true
|
||||
provider, err := New(Options{
|
||||
APIKey: "sk-test",
|
||||
BaseURL: server.URL,
|
||||
Model: "gpt-test",
|
||||
HTTPClient: &http.Client{Transport: transport},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
|
||||
recorder := trace.NewRecorder("session-test", "run-1", "test")
|
||||
recorder.Start()
|
||||
ctx := trace.WithContext(context.Background(), recorder)
|
||||
|
||||
session := openOptimizedSession(t, provider)
|
||||
if err := session.Prewarm(ctx); err != nil {
|
||||
t.Fatalf("Prewarm: %v", err)
|
||||
}
|
||||
waitPrewarmDone(t, session)
|
||||
|
||||
if got := requests.Load(); got != 0 {
|
||||
t.Fatalf("keep-alive-disabled transport still received %d probe requests, want 0", got)
|
||||
}
|
||||
tr := recorder.Finish()
|
||||
if got := tr.Counter(trace.CounterPrewarmAttempts); got != 0 {
|
||||
t.Fatalf("prewarm_attempts = %d, want 0 for a skipped probe", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnSessionPrefixCounters(t *testing.T) {
|
||||
handler := func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
_, _ = w.Write([]byte("data: [DONE]\n\n"))
|
||||
}
|
||||
provider := newTestProvider(t, handler)
|
||||
session := openOptimizedSession(t, provider)
|
||||
|
||||
recorder := trace.NewRecorder("session-test", "run-1", "test")
|
||||
recorder.Start()
|
||||
ctx := trace.WithContext(context.Background(), recorder)
|
||||
|
||||
stable := zeroruntime.CompletionRequest{Tools: []zeroruntime.ToolDefinition{{Name: "read_file", Parameters: map[string]any{"type": "object"}}}}
|
||||
drain := func(request zeroruntime.CompletionRequest) {
|
||||
stream, err := session.Stream(ctx, request)
|
||||
if err != nil {
|
||||
t.Fatalf("Stream: %v", err)
|
||||
}
|
||||
for range stream {
|
||||
}
|
||||
}
|
||||
|
||||
drain(stable) // seeds — neither stable nor drift
|
||||
drain(stable) // stable
|
||||
drain(zeroruntime.CompletionRequest{Tools: []zeroruntime.ToolDefinition{{Name: "write_file", Parameters: map[string]any{"type": "object"}}}}) // drift
|
||||
|
||||
tr := recorder.Finish()
|
||||
if got := tr.Counter(trace.CounterPrefixStable); got != 1 {
|
||||
t.Fatalf("prefix_stable = %d, want 1", got)
|
||||
}
|
||||
if got := tr.Counter(trace.CounterPrefixDrift); got != 1 {
|
||||
t.Fatalf("prefix_drift = %d, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTurnSessionCloseIdempotentAndCompactUnsupported(t *testing.T) {
|
||||
provider := newTestProvider(t, func(http.ResponseWriter, *http.Request) {})
|
||||
session := openOptimizedSession(t, provider)
|
||||
if err := session.Close(); err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
if err := session.Close(); err != nil {
|
||||
t.Fatalf("second Close: %v", err)
|
||||
}
|
||||
if _, err := session.Compact(context.Background(), zeroruntime.CompletionRequest{}); !errors.Is(err, zeroruntime.ErrCompactionUnsupported) {
|
||||
t.Fatalf("Compact error = %v, want ErrCompactionUnsupported", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
package providers
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/config"
|
||||
"github.com/Gitlawb/zero/internal/providers/openai"
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// openaiTurnSessionEnv gates the optimized OpenAI turn session (prewarm +
|
||||
// prefix telemetry). Default OFF: unset, "0", or "false" (any case) leave the
|
||||
// agent loop's default adapter path untouched. Set to any other non-empty
|
||||
// value (e.g. "1") to enable — the same boolean idiom as ZERO_FORMAT_ON_WRITE.
|
||||
const openaiTurnSessionEnv = "ZERO_OPENAI_TURN_SESSION"
|
||||
|
||||
func openaiTurnSessionEnabled() bool {
|
||||
value := strings.TrimSpace(os.Getenv(openaiTurnSessionEnv))
|
||||
return value != "" && value != "0" && !strings.EqualFold(value, "false")
|
||||
}
|
||||
|
||||
// OptimizedTurnSessions returns the gated optimized TurnSessionProvider for an
|
||||
// already-built provider, or (nil, false) when the gate is off or the profile
|
||||
// is ineligible. Callers leave agent Options.TurnSessionProvider nil on false,
|
||||
// so the loop's default wrap keeps today's behavior byte-identical.
|
||||
//
|
||||
// Eligibility is deliberately narrow: the resolved provider kind must be
|
||||
// official OpenAI (NOT openai-compatible gateways — the constructor merges the
|
||||
// two, so this branches on the resolved kind explicitly), not the ChatGPT
|
||||
// Codex catalog, and the provider value must be the concrete *openai.Provider
|
||||
// (a fake, a Codex provider, or nil falls back to the default path safely).
|
||||
// No base-URL check on top of the kind: prewarm and fingerprint telemetry are
|
||||
// harmless against any host, and kind==openai mirrors the existing
|
||||
// official-OpenAI precedent used for prompt_cache_key.
|
||||
func OptimizedTurnSessions(profile config.ProviderProfile, provider zeroruntime.Provider, options Options) (zeroruntime.TurnSessionProvider, bool) {
|
||||
if !openaiTurnSessionEnabled() {
|
||||
return nil, false
|
||||
}
|
||||
resolved, err := resolveProfile(profile, options)
|
||||
if err != nil || resolved.providerKind != config.ProviderKindOpenAI || isCodexCatalog(profile, resolved) {
|
||||
return nil, false
|
||||
}
|
||||
concrete, ok := provider.(*openai.Provider)
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
caps, err := resolveCapabilities(profile, options)
|
||||
if err != nil {
|
||||
// resolveProfile above already succeeded, so this is effectively
|
||||
// unreachable — but never ship an optimized session with unknown
|
||||
// capabilities; the default path is the safe answer.
|
||||
return nil, false
|
||||
}
|
||||
return openai.NewTurnSessionProvider(concrete, caps), true
|
||||
}
|
||||
|
||||
// DefaultTurnSessions wraps an already-built provider in the DEFAULT (no-op)
|
||||
// turn-session adapter with the profile's resolved capability projection. It is
|
||||
// the non-optimized sibling of OptimizedTurnSessions for callers that still
|
||||
// want capabilities populated — e.g. the mid-run model-switch fallback when the
|
||||
// switched model is not eligible for the optimized session. Capability
|
||||
// resolution failure degrades to an unknown (zero) projection rather than
|
||||
// failing the wrap: the default session has no behavior that depends on
|
||||
// capabilities, so a swap must never be blocked by a projection error.
|
||||
func DefaultTurnSessions(profile config.ProviderProfile, provider zeroruntime.Provider, options Options) zeroruntime.TurnSessionProvider {
|
||||
caps, err := resolveCapabilities(profile, options)
|
||||
if err != nil {
|
||||
caps = zeroruntime.ProviderCapabilities{}
|
||||
}
|
||||
return zeroruntime.NewProviderTurnSessionProvider(provider, caps)
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package providers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Gitlawb/zero/internal/config"
|
||||
"github.com/Gitlawb/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
func openaiEligibleProfile() config.ProviderProfile {
|
||||
return config.ProviderProfile{
|
||||
Name: "official-openai",
|
||||
ProviderKind: config.ProviderKindOpenAI,
|
||||
BaseURL: "https://provider.example/v1",
|
||||
APIKey: "sk-gate-test",
|
||||
Model: "pr8-unregistered-model",
|
||||
}
|
||||
}
|
||||
|
||||
func buildGateProvider(t *testing.T, profile config.ProviderProfile) zeroruntime.Provider {
|
||||
t.Helper()
|
||||
provider, err := New(profile, Options{UserAgent: "zero-gate-test"})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
return provider
|
||||
}
|
||||
|
||||
type fakeGateProvider struct{}
|
||||
|
||||
func (fakeGateProvider) StreamCompletion(context.Context, zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func TestOptimizedTurnSessionsDefaultOff(t *testing.T) {
|
||||
// No env set: the gate must be off even for a fully eligible profile.
|
||||
profile := openaiEligibleProfile()
|
||||
tsp, ok := OptimizedTurnSessions(profile, buildGateProvider(t, profile), Options{})
|
||||
if ok || tsp != nil {
|
||||
t.Fatal("optimized turn sessions must be disabled by default")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOptimizedTurnSessionsFalseyValues(t *testing.T) {
|
||||
profile := openaiEligibleProfile()
|
||||
provider := buildGateProvider(t, profile)
|
||||
for _, value := range []string{"0", "false", "FALSE", "False", " ", ""} {
|
||||
t.Setenv(openaiTurnSessionEnv, value)
|
||||
if _, ok := OptimizedTurnSessions(profile, provider, Options{}); ok {
|
||||
t.Fatalf("gate enabled for falsey value %q", value)
|
||||
}
|
||||
}
|
||||
for _, value := range []string{"1", "true", "on"} {
|
||||
t.Setenv(openaiTurnSessionEnv, value)
|
||||
if _, ok := OptimizedTurnSessions(profile, provider, Options{}); !ok {
|
||||
t.Fatalf("gate disabled for truthy value %q", value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOptimizedTurnSessionsOpenAIEligible(t *testing.T) {
|
||||
t.Setenv(openaiTurnSessionEnv, "1")
|
||||
profile := openaiEligibleProfile()
|
||||
tsp, ok := OptimizedTurnSessions(profile, buildGateProvider(t, profile), Options{})
|
||||
if !ok || tsp == nil {
|
||||
t.Fatal("expected the optimized session for an official-OpenAI profile with the gate on")
|
||||
}
|
||||
session, err := tsp.OpenTurnSession(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("OpenTurnSession: %v", err)
|
||||
}
|
||||
if err := session.Close(); err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
if caps := tsp.Capabilities(); caps.Model != "pr8-unregistered-model" {
|
||||
t.Fatalf("Capabilities().Model = %q, want the resolved api model", caps.Model)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOptimizedTurnSessionsRejectsCompatible(t *testing.T) {
|
||||
t.Setenv(openaiTurnSessionEnv, "1")
|
||||
profile := openaiEligibleProfile()
|
||||
profile.ProviderKind = config.ProviderKindOpenAICompatible
|
||||
if _, ok := OptimizedTurnSessions(profile, buildGateProvider(t, profile), Options{}); ok {
|
||||
t.Fatal("openai-compatible gateways must not get the optimized session")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOptimizedTurnSessionsRejectsCodexCatalog(t *testing.T) {
|
||||
t.Setenv(openaiTurnSessionEnv, "1")
|
||||
profile := openaiEligibleProfile()
|
||||
profile.CatalogID = "chatgpt"
|
||||
// Build the provider from the base profile: the point under test is the
|
||||
// catalog check, not Codex construction (which needs OAuth wiring).
|
||||
if _, ok := OptimizedTurnSessions(profile, buildGateProvider(t, openaiEligibleProfile()), Options{}); ok {
|
||||
t.Fatal("the ChatGPT Codex catalog must not get the optimized session")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultTurnSessionsPreservesResolvedCapabilities(t *testing.T) {
|
||||
profile := openaiEligibleProfile()
|
||||
tsp := DefaultTurnSessions(profile, buildGateProvider(t, profile), Options{})
|
||||
if tsp == nil {
|
||||
t.Fatal("DefaultTurnSessions returned nil")
|
||||
}
|
||||
caps := tsp.Capabilities()
|
||||
if caps.Model != "pr8-unregistered-model" {
|
||||
t.Fatalf("Capabilities().Model = %q, want the resolved api model", caps.Model)
|
||||
}
|
||||
session, err := tsp.OpenTurnSession(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("OpenTurnSession: %v", err)
|
||||
}
|
||||
if err := session.Close(); err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOptimizedTurnSessionsRejectsForeignProviderValue(t *testing.T) {
|
||||
t.Setenv(openaiTurnSessionEnv, "1")
|
||||
if _, ok := OptimizedTurnSessions(openaiEligibleProfile(), fakeGateProvider{}, Options{}); ok {
|
||||
t.Fatal("a non-*openai.Provider value must fall back to the default path")
|
||||
}
|
||||
if _, ok := OptimizedTurnSessions(openaiEligibleProfile(), nil, Options{}); ok {
|
||||
t.Fatal("a nil provider must fall back to the default path")
|
||||
}
|
||||
}
|
||||
+23
-18
@@ -14,18 +14,22 @@ type EngineOptions struct {
|
||||
Store *GrantStore
|
||||
Backend Backend
|
||||
Scope *Scope
|
||||
// SensitiveEnvKeys adds config-derived credential variable names to the
|
||||
// catalog and namespace secrets scrubbed from sandboxed commands.
|
||||
SensitiveEnvKeys []string
|
||||
}
|
||||
|
||||
type Engine struct {
|
||||
workspaceRoot string
|
||||
policy Policy
|
||||
store *GrantStore
|
||||
backend Backend
|
||||
scope *Scope
|
||||
sessionGrants *memoryGrantSet
|
||||
sessionProfiles *permissionProfileGrantSet
|
||||
turnProfiles *permissionProfileGrantSet
|
||||
commandPrefixes *commandPrefixGrantSet
|
||||
workspaceRoot string
|
||||
policy Policy
|
||||
store *GrantStore
|
||||
backend Backend
|
||||
scope *Scope
|
||||
sensitiveEnvKeys []string
|
||||
sessionGrants *memoryGrantSet
|
||||
sessionProfiles *permissionProfileGrantSet
|
||||
turnProfiles *permissionProfileGrantSet
|
||||
commandPrefixes *commandPrefixGrantSet
|
||||
}
|
||||
|
||||
func NewEngine(options EngineOptions) *Engine {
|
||||
@@ -49,15 +53,16 @@ func NewEngine(options EngineOptions) *Engine {
|
||||
scope = newScopeBestEffort(workspaceRoot)
|
||||
}
|
||||
return &Engine{
|
||||
workspaceRoot: workspaceRoot,
|
||||
policy: policy,
|
||||
store: options.Store,
|
||||
backend: options.Backend,
|
||||
scope: scope,
|
||||
sessionGrants: newMemoryGrantSet(),
|
||||
sessionProfiles: newPermissionProfileGrantSet(),
|
||||
turnProfiles: newPermissionProfileGrantSet(),
|
||||
commandPrefixes: newCommandPrefixGrantSet(),
|
||||
workspaceRoot: workspaceRoot,
|
||||
policy: policy,
|
||||
store: options.Store,
|
||||
backend: options.Backend,
|
||||
scope: scope,
|
||||
sensitiveEnvKeys: normalizeSensitiveEnvKeys(options.SensitiveEnvKeys),
|
||||
sessionGrants: newMemoryGrantSet(),
|
||||
sessionProfiles: newPermissionProfileGrantSet(),
|
||||
turnProfiles: newPermissionProfileGrantSet(),
|
||||
commandPrefixes: newCommandPrefixGrantSet(),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -29,6 +29,9 @@ type CommandSpec struct {
|
||||
Args []string
|
||||
Dir string
|
||||
Env []string
|
||||
// sensitiveEnvKeys is populated by Engine from config-derived credential
|
||||
// variable names and carried through SandboxManager to the platform plan.
|
||||
sensitiveEnvKeys []string
|
||||
}
|
||||
|
||||
type CommandPlan struct {
|
||||
@@ -122,6 +125,7 @@ func (engine *Engine) BuildCommandPlan(spec CommandSpec) (CommandPlan, error) {
|
||||
return CommandPlan{}, errors.New("sandbox command name is required")
|
||||
}
|
||||
spec.Dir = commandDir
|
||||
spec.sensitiveEnvKeys = engine.sensitiveEnvKeys
|
||||
|
||||
backend := engine.backend
|
||||
if backend.Name == "" {
|
||||
@@ -207,7 +211,7 @@ func linuxSandboxHelperCommandPlan(execRequest SandboxExecutionRequest, policy P
|
||||
if err != nil {
|
||||
return CommandPlan{}, err
|
||||
}
|
||||
env := sandboxEnvironmentForCommand(spec.Env, policy, BackendLinuxBwrap, "")
|
||||
env := sandboxEnvironmentForCommandWithSensitiveEnv(spec.Env, policy, BackendLinuxBwrap, "", spec.sensitiveEnvKeys)
|
||||
planDir := spec.Dir
|
||||
if helper.Dir != "" {
|
||||
planDir = helper.Dir
|
||||
@@ -253,10 +257,27 @@ func directCommandPlan(spec CommandSpec, backend Backend, policy Policy, workspa
|
||||
Name: spec.Name,
|
||||
Args: cloneStrings(spec.Args),
|
||||
Dir: spec.Dir,
|
||||
Env: cloneStrings(spec.Env),
|
||||
Env: directCommandEnv(spec),
|
||||
}
|
||||
}
|
||||
|
||||
// directCommandEnv scrubs sensitive credentials from the environment for a
|
||||
// direct (unwrapped) command plan: the platform sandbox backend is
|
||||
// unavailable, disabled, or not required, so this is the actual environment
|
||||
// the child process inherits. Without scrubbing here, config-derived
|
||||
// apiKeyEnv and dynamic OAuth client-secret variables would leak into
|
||||
// commands that fall back to this path (e.g. EnforcementDegraded).
|
||||
func directCommandEnv(spec CommandSpec) []string {
|
||||
env := cloneStrings(spec.Env)
|
||||
if spec.Env == nil {
|
||||
// Match the wrapped-plan behavior: an unset spec.Env means "inherit the
|
||||
// caller's environment," so scrub a snapshot of it rather than passing
|
||||
// nil through to exec.Cmd, which would skip scrubbing entirely.
|
||||
env = os.Environ()
|
||||
}
|
||||
return scrubSensitiveEnv(env, spec.sensitiveEnvKeys...)
|
||||
}
|
||||
|
||||
func (engine *Engine) resolveCommandDir(dir string, policy Policy) (string, string, error) {
|
||||
workspaceRoot := strings.TrimSpace(engine.workspaceRoot)
|
||||
if workspaceRoot == "" {
|
||||
@@ -318,7 +339,7 @@ func seatbeltCommandPlanWithProfile(spec CommandSpec, workspaceRoot string, prof
|
||||
if envBackend == "" {
|
||||
envBackend = BackendMacOSSeatbelt
|
||||
}
|
||||
env := sandboxEnvironmentForCommand(spec.Env, policy, envBackend, "")
|
||||
env := sandboxEnvironmentForCommandWithSensitiveEnv(spec.Env, policy, envBackend, "", spec.sensitiveEnvKeys)
|
||||
plan := CommandPlan{
|
||||
Backend: backend,
|
||||
TargetBackend: backend.TargetBackend(),
|
||||
@@ -377,6 +398,10 @@ func sandboxEnvironment(policy Policy, backend BackendName, workspaceRoot string
|
||||
}
|
||||
|
||||
func sandboxEnvironmentForCommand(specEnv []string, policy Policy, backend BackendName, workspaceRoot string) []string {
|
||||
return sandboxEnvironmentForCommandWithSensitiveEnv(specEnv, policy, backend, workspaceRoot, nil)
|
||||
}
|
||||
|
||||
func sandboxEnvironmentForCommandWithSensitiveEnv(specEnv []string, policy Policy, backend BackendName, workspaceRoot string, sensitiveEnvKeys []string) []string {
|
||||
env := cloneStrings(specEnv)
|
||||
if specEnv == nil {
|
||||
// Preserve the caller environment for sandboxed commands. The sandbox
|
||||
@@ -384,7 +409,7 @@ func sandboxEnvironmentForCommand(specEnv []string, policy Policy, backend Backe
|
||||
// command env values still replace inherited values below.
|
||||
env = os.Environ()
|
||||
}
|
||||
env = scrubSensitiveEnv(env)
|
||||
env = scrubSensitiveEnv(env, sensitiveEnvKeys...)
|
||||
pathValue := envListValue(env, "PATH", defaultPath())
|
||||
if runtime.GOOS == "darwin" {
|
||||
// Preserve standard user tool locations so a bare `python3`/`node`
|
||||
@@ -967,7 +992,7 @@ func regexpQuoteMeta(value string) string {
|
||||
return replacer.Replace(value)
|
||||
}
|
||||
|
||||
func scrubSensitiveEnv(env []string) []string {
|
||||
func scrubSensitiveEnv(env []string, additionalKeys ...string) []string {
|
||||
// Secrets not covered by the provider catalog: cloud/VCS credentials and
|
||||
// providers Zero talks to through generic OpenAI-compatible endpoints.
|
||||
sensitiveKeys := []string{
|
||||
@@ -997,6 +1022,7 @@ func scrubSensitiveEnv(env []string) []string {
|
||||
sensitiveKeys = append(sensitiveKeys, key)
|
||||
}
|
||||
}
|
||||
sensitiveKeys = append(sensitiveKeys, normalizeSensitiveEnvKeys(additionalKeys)...)
|
||||
|
||||
out := make([]string, 0, len(env))
|
||||
for _, kv := range env {
|
||||
@@ -1005,7 +1031,7 @@ func scrubSensitiveEnv(env []string) []string {
|
||||
out = append(out, kv)
|
||||
continue
|
||||
}
|
||||
drop := false
|
||||
drop := isDynamicSensitiveEnvKey(key)
|
||||
for _, sensitive := range sensitiveKeys {
|
||||
if strings.EqualFold(key, sensitive) {
|
||||
drop = true
|
||||
@@ -1018,3 +1044,37 @@ func scrubSensitiveEnv(env []string) []string {
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func normalizeSensitiveEnvKeys(keys []string) []string {
|
||||
out := make([]string, 0, len(keys))
|
||||
seen := make(map[string]struct{}, len(keys))
|
||||
for _, key := range keys {
|
||||
key = strings.TrimSpace(key)
|
||||
// A misconfigured value like "COMPANY_LLM_SECRET=..." would never
|
||||
// match a real env key during scrubbing; keep only the name part.
|
||||
if name, _, found := strings.Cut(key, "="); found {
|
||||
key = strings.TrimSpace(name)
|
||||
}
|
||||
folded := strings.ToUpper(key)
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[folded]; ok {
|
||||
continue
|
||||
}
|
||||
seen[folded] = struct{}{}
|
||||
out = append(out, key)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func isDynamicSensitiveEnvKey(key string) bool {
|
||||
const (
|
||||
prefix = "ZERO_OAUTH_"
|
||||
suffix = "_CLIENT_SECRET"
|
||||
)
|
||||
key = strings.ToUpper(strings.TrimSpace(key))
|
||||
return strings.HasPrefix(key, prefix) &&
|
||||
strings.HasSuffix(key, suffix) &&
|
||||
len(key) > len(prefix)+len(suffix)
|
||||
}
|
||||
@@ -122,6 +122,47 @@ func TestBuildCommandPlanDegradesUnavailableFallback(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildCommandPlanDegradedFallbackScrubsInheritedEnv covers the
|
||||
// EnforcementDegraded fallback path: when the native backend is
|
||||
// unavailable, BuildCommandPlan falls back to a direct (unwrapped) plan
|
||||
// whose spec.Env is nil, so exec.Cmd would otherwise inherit the caller's
|
||||
// environment — including configured and dynamically named credentials —
|
||||
// unscrubbed.
|
||||
func TestBuildCommandPlanDegradedFallbackScrubsInheritedEnv(t *testing.T) {
|
||||
t.Setenv("COMPANY_LLM_SECRET", "custom-secret")
|
||||
t.Setenv("ZERO_OAUTH_ACME_CLIENT_SECRET", "oauth-secret")
|
||||
t.Setenv("SAFE_VAR", "hello")
|
||||
|
||||
root := t.TempDir()
|
||||
engine := NewEngine(EngineOptions{
|
||||
WorkspaceRoot: root,
|
||||
Policy: DefaultPolicy(),
|
||||
Backend: Backend{Name: BackendUnavailable, Message: "native sandbox unavailable"},
|
||||
SensitiveEnvKeys: []string{"COMPANY_LLM_SECRET"},
|
||||
})
|
||||
|
||||
plan, err := engine.BuildCommandPlan(CommandSpec{
|
||||
Name: "/bin/sh",
|
||||
Args: []string{"-c", "pwd"},
|
||||
Dir: root,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildCommandPlan: %v", err)
|
||||
}
|
||||
if plan.Wrapped || plan.EnforcementLevel != EnforcementDegraded {
|
||||
t.Fatalf("plan = %#v, want degraded direct plan", plan)
|
||||
}
|
||||
for _, entry := range plan.Env {
|
||||
key, _, _ := strings.Cut(entry, "=")
|
||||
if strings.EqualFold(key, "COMPANY_LLM_SECRET") || strings.EqualFold(key, "ZERO_OAUTH_ACME_CLIENT_SECRET") {
|
||||
t.Fatalf("degraded plan.Env retained sensitive key %q: %v", key, plan.Env)
|
||||
}
|
||||
}
|
||||
if got := envListValue(plan.Env, "SAFE_VAR", ""); got != "hello" {
|
||||
t.Fatalf("SAFE_VAR = %q, want hello", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildCommandPlanRejectsOutsideDirectory(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
engine := NewEngine(EngineOptions{
|
||||
@@ -653,14 +694,19 @@ func TestScrubSensitiveEnv(t *testing.T) {
|
||||
"XAI_API_KEY=xai-12345",
|
||||
"HUGGINGFACE_API_KEY=hf_12345",
|
||||
"GOOGLE_APPLICATION_CREDENTIALS=/home/user/sa-key.json",
|
||||
"COMPANY_LLM_SECRET=custom-secret",
|
||||
"ZERO_OAUTH_MY_SVC_CLIENT_SECRET=oauth-secret",
|
||||
"zero_oauth_second_client_secret=case-insensitive-secret",
|
||||
"ZERO_OAUTH_CLIENT_SECRET=not-a-provider-secret",
|
||||
"AWS_PROFILE=staging",
|
||||
"SAFE_VAR=hello",
|
||||
}
|
||||
scrubbed := scrubSensitiveEnv(inputEnv)
|
||||
scrubbed := scrubSensitiveEnv(inputEnv, " COMPANY_LLM_SECRET ", "company_llm_secret", "GITHUB_TOKEN=ghp_pasted-assignment", "=", "")
|
||||
expected := map[string]string{
|
||||
"PATH": "/usr/bin",
|
||||
"SAFE_VAR": "hello",
|
||||
"AWS_PROFILE": "staging",
|
||||
"PATH": "/usr/bin",
|
||||
"SAFE_VAR": "hello",
|
||||
"AWS_PROFILE": "staging",
|
||||
"ZERO_OAUTH_CLIENT_SECRET": "not-a-provider-secret",
|
||||
}
|
||||
actual := make(map[string]string, len(scrubbed))
|
||||
for _, entry := range scrubbed {
|
||||
@@ -674,3 +720,34 @@ func TestScrubSensitiveEnv(t *testing.T) {
|
||||
t.Errorf("scrubSensitiveEnv() = %v, want %v", actual, expected)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEngineScrubsConfiguredSensitiveEnvKeys(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
engine := NewEngine(EngineOptions{
|
||||
WorkspaceRoot: workspace,
|
||||
Policy: DefaultPolicy(),
|
||||
Backend: Backend{Name: BackendLinuxBwrap, Available: true, Executable: "/usr/bin/zero-linux-sandbox"},
|
||||
SensitiveEnvKeys: []string{"COMPANY_LLM_SECRET"},
|
||||
})
|
||||
plan, err := engine.BuildCommandPlan(CommandSpec{
|
||||
Name: "true",
|
||||
Env: []string{
|
||||
"PATH=/usr/bin",
|
||||
"COMPANY_LLM_SECRET=custom-secret",
|
||||
"ZERO_OAUTH_CUSTOM_CLIENT_SECRET=oauth-secret",
|
||||
"SAFE_VAR=hello",
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildCommandPlan: %v", err)
|
||||
}
|
||||
for _, entry := range plan.Env {
|
||||
key, _, _ := strings.Cut(entry, "=")
|
||||
if strings.EqualFold(key, "COMPANY_LLM_SECRET") || strings.EqualFold(key, "ZERO_OAUTH_CUSTOM_CLIENT_SECRET") {
|
||||
t.Fatalf("plan.Env retained sensitive key %q: %v", key, plan.Env)
|
||||
}
|
||||
}
|
||||
if got := envListValue(plan.Env, "SAFE_VAR", ""); got != "hello" {
|
||||
t.Fatalf("SAFE_VAR = %q, want hello", got)
|
||||
}
|
||||
}
|
||||
@@ -326,7 +326,7 @@ func windowsRestrictedTokenCommandPlan(execRequest SandboxExecutionRequest, poli
|
||||
if err != nil {
|
||||
return CommandPlan{}, err
|
||||
}
|
||||
childEnv := sandboxEnvironmentForCommand(spec.Env, policy, BackendWindowsRestrictedToken, execRequest.WorkspaceRoot)
|
||||
childEnv := sandboxEnvironmentForCommandWithSensitiveEnv(spec.Env, policy, BackendWindowsRestrictedToken, execRequest.WorkspaceRoot, spec.sensitiveEnvKeys)
|
||||
// The unelevated enforcement tier maps to the runner's unelevated level: same
|
||||
// restricted token, but the runner applies the workspace ACLs itself instead
|
||||
// of requiring the elevated setup marker.
|
||||
|
||||
+41
-6
@@ -23,6 +23,15 @@ type bashTool struct {
|
||||
scope PathScope
|
||||
}
|
||||
|
||||
func (bashTool) outputCategory(args map[string]any) outputCategory {
|
||||
command, _ := bashCommandArg(args)
|
||||
return shellOutputCategory(command)
|
||||
}
|
||||
|
||||
func bashCommandArg(args map[string]any) (string, error) {
|
||||
return aliasedStringArg(args, []string{"command", "cmd", "script", "shell"}, "", true, false)
|
||||
}
|
||||
|
||||
func NewBashTool(workspaceRoot string) Tool {
|
||||
return NewScopedBashTool(workspaceRoot, nil)
|
||||
}
|
||||
@@ -63,15 +72,19 @@ func NewScopedBashTool(workspaceRoot string, scope PathScope) Tool {
|
||||
}
|
||||
|
||||
func (tool bashTool) Run(ctx context.Context, args map[string]any) Result {
|
||||
return tool.run(ctx, args, nil)
|
||||
return tool.run(ctx, args, nil, true)
|
||||
}
|
||||
|
||||
func (tool bashTool) RunWithSandbox(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine) Result {
|
||||
return tool.run(ctx, args, engine)
|
||||
return tool.run(ctx, args, engine, true)
|
||||
}
|
||||
|
||||
func (tool bashTool) run(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine) Result {
|
||||
commandText, err := aliasedStringArg(args, []string{"command", "cmd", "script", "shell"}, "", true, false)
|
||||
func (tool bashTool) RunWithOptions(ctx context.Context, args map[string]any, options RunOptions) Result {
|
||||
return tool.run(ctx, args, options.Sandbox, false)
|
||||
}
|
||||
|
||||
func (tool bashTool) run(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine, directBudget bool) Result {
|
||||
commandText, err := bashCommandArg(args)
|
||||
if err != nil {
|
||||
return errorResult("Error: Invalid arguments for bash: " + err.Error())
|
||||
}
|
||||
@@ -172,7 +185,7 @@ func (tool bashTool) run(ctx context.Context, args map[string]any, engine *zeroS
|
||||
}
|
||||
}
|
||||
markLikelySandboxDenial(meta, plan, exitCode, stdoutText, stderrText)
|
||||
outText, errText, truncated := budgetBashCapture(stdoutText, stdout.total, stderrText, stderrTotal, meta)
|
||||
outText, errText, truncated := prepareBashOutput(stdoutText, stdout.total, stderrText, stderrTotal, meta, directBudget)
|
||||
return Result{
|
||||
Status: StatusError,
|
||||
Output: formatBashOutputWithShellHint(outText, errText, exitCode, meta),
|
||||
@@ -182,7 +195,7 @@ func (tool bashTool) run(ctx context.Context, args map[string]any, engine *zeroS
|
||||
}
|
||||
|
||||
markLikelySandboxDenial(meta, plan, exitCode, stdoutText, stderrText)
|
||||
outText, errText, truncated := budgetBashCapture(stdoutText, stdout.total, stderrText, stderrTotal, meta)
|
||||
outText, errText, truncated := prepareBashOutput(stdoutText, stdout.total, stderrText, stderrTotal, meta, directBudget)
|
||||
if meta[SandboxLikelyDeniedMeta] == "true" {
|
||||
return Result{
|
||||
Status: StatusError,
|
||||
@@ -415,6 +428,28 @@ func budgetBashOutput(stdout string, stderr string, meta map[string]string) (str
|
||||
return budgetBashCapture(stdout, len(stdout), stderr, len(stderr), meta)
|
||||
}
|
||||
|
||||
// prepareBashOutput keeps direct callers on the established positional budget.
|
||||
// Registry calls retain the existing bounded capture but leave final semantic
|
||||
// reduction and spill creation to the post-redaction registry boundary.
|
||||
func prepareBashOutput(out string, outTotal int, errStr string, errTotal int, meta map[string]string, directBudget bool) (string, string, bool) {
|
||||
if directBudget {
|
||||
return budgetBashCapture(out, outTotal, errStr, errTotal, meta)
|
||||
}
|
||||
outText := sectionWithCaptureGap(out, outTotal)
|
||||
errText := sectionWithCaptureGap(errStr, errTotal)
|
||||
truncated := outTotal > len(out) || errTotal > len(errStr)
|
||||
if meta != nil {
|
||||
meta["raw_bytes"] = strconv.Itoa(outTotal + errTotal)
|
||||
meta["emitted_bytes"] = strconv.Itoa(len(outText) + len(errText))
|
||||
meta["estimated_tokens"] = strconv.Itoa(estimatedTokensFromBytes(len(outText) + len(errText)))
|
||||
if truncated {
|
||||
meta["truncated"] = "true"
|
||||
meta["truncation_reason"] = "capture_budget"
|
||||
}
|
||||
}
|
||||
return outText, errText, truncated
|
||||
}
|
||||
|
||||
// budgetBashCapture is budgetBashOutput for the streaming-capture path: outTotal
|
||||
// and errTotal are the true byte counts (from boundedBuffer.total), which may
|
||||
// exceed the retained strings when the middle was dropped during capture. Meta's
|
||||
|
||||
@@ -37,6 +37,18 @@ func TestBudgetBashOutputSmallPassesThrough(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareBashOutputDefersRegistrySemanticBudget(t *testing.T) {
|
||||
output := "START\n" + strings.Repeat("progress\n", 8_000) + "ERROR: retained\nEND"
|
||||
registryOutput, _, registryTruncated := prepareBashOutput(output, len(output), "", 0, map[string]string{}, false)
|
||||
if registryTruncated || registryOutput != output {
|
||||
t.Fatalf("registry preparation must preserve capture-bounded output for semantic reduction: truncated=%t bytes=%d", registryTruncated, len(registryOutput))
|
||||
}
|
||||
directOutput, _, directTruncated := prepareBashOutput(output, len(output), "", 0, map[string]string{}, true)
|
||||
if !directTruncated || len(directOutput) > bashOutputBudgetBytes+512 {
|
||||
t.Fatalf("direct path must preserve legacy positional budget: truncated=%t bytes=%d", directTruncated, len(directOutput))
|
||||
}
|
||||
}
|
||||
|
||||
// Oversized stdout is truncated head+tail: both the first and last lines survive,
|
||||
// the middle is dropped behind a marker, meta is flagged, and the captured text
|
||||
// is spilled to a re-readable file.
|
||||
|
||||
@@ -300,6 +300,18 @@ func TestInteractiveClassifications(t *testing.T) {
|
||||
|
||||
func TestReadOnlyClassifications(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
// PR6 audited concurrent-safe pure reads (mutex-guarded FileTracker,
|
||||
// independent scan I/O, independent HTTP). Others stay sequential until
|
||||
// their shared state is proven safe under concurrent Run.
|
||||
threadSafeReads := map[string]bool{
|
||||
"read_file": true,
|
||||
"read_minified_file": true,
|
||||
"list_directory": true,
|
||||
"glob": true,
|
||||
"grep": true,
|
||||
"skill": true,
|
||||
"web_fetch": true,
|
||||
}
|
||||
for _, name := range []string{"read_file", "read_minified_file", "list_directory", "glob", "grep", "lsp_navigate", "skill", "web_fetch", "web_search", "tool_search"} {
|
||||
var found Tool
|
||||
for _, tool := range BuiltinCatalog(root) {
|
||||
@@ -315,14 +327,38 @@ func TestReadOnlyClassifications(t *testing.T) {
|
||||
if caps.Effect != EffectReadOnly {
|
||||
t.Errorf("%s effect = %v, want ReadOnly", name, caps.Effect)
|
||||
}
|
||||
// Current catalog intentionally keeps read tools non-thread-safe
|
||||
// until each is audited for shared mutable state (FileTracker, etc.).
|
||||
if caps.ThreadSafe {
|
||||
t.Errorf("%s ThreadSafe=true without explicit audit", name)
|
||||
wantSafe := threadSafeReads[name]
|
||||
if caps.ThreadSafe != wantSafe {
|
||||
t.Errorf("%s ThreadSafe=%v, want %v", name, caps.ThreadSafe, wantSafe)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestScopedScanResourceKeys(t *testing.T) {
|
||||
// Workspace-wide scan → no keys (do not false-conflict every concurrent scan).
|
||||
if keys := scopedScanResourceKeys(nil); keys != nil {
|
||||
t.Fatalf("nil args = %v, want nil", keys)
|
||||
}
|
||||
if keys := scopedScanResourceKeys(map[string]any{}); keys != nil {
|
||||
t.Fatalf("empty args = %v, want nil", keys)
|
||||
}
|
||||
if keys := scopedScanResourceKeys(map[string]any{"cwd": "."}); keys != nil {
|
||||
t.Fatalf("cwd=. = %v, want nil", keys)
|
||||
}
|
||||
if keys := scopedScanResourceKeys(map[string]any{"path": "."}); keys != nil {
|
||||
t.Fatalf("path=. = %v, want nil", keys)
|
||||
}
|
||||
// Scoped path → directory: key (glob uses cwd; grep uses path).
|
||||
keys := scopedScanResourceKeys(map[string]any{"cwd": "internal/tools"})
|
||||
if len(keys) != 1 || keys[0] != ResourceKeyDirectory+NormalizeResourcePath("internal/tools") {
|
||||
t.Fatalf("cwd scoped = %v", keys)
|
||||
}
|
||||
keys = scopedScanResourceKeys(map[string]any{"path": "./pkg/../pkg/api"})
|
||||
if len(keys) != 1 || keys[0] != ResourceKeyDirectory+NormalizeResourcePath("./pkg/../pkg/api") {
|
||||
t.Fatalf("path scoped = %v", keys)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateBuiltinCatalogRejectsUnknownRaw(t *testing.T) {
|
||||
// Prove the gate is not a rubber stamp: raw Unknown+ThreadSafe is rejected.
|
||||
problems := ValidateCapabilities("ghost", ToolCapabilities{Effect: EffectUnknown, ThreadSafe: true})
|
||||
|
||||
@@ -427,6 +427,15 @@ type execCommandTool struct {
|
||||
manager *execSessionManager
|
||||
}
|
||||
|
||||
func (execCommandTool) outputCategory(args map[string]any) outputCategory {
|
||||
command, _ := execCommandArg(args)
|
||||
return shellOutputCategory(command)
|
||||
}
|
||||
|
||||
func execCommandArg(args map[string]any) (string, error) {
|
||||
return aliasedStringArg(args, []string{"cmd", "command", "script", "shell"}, "", true, false)
|
||||
}
|
||||
|
||||
func NewExecCommandTool(workspaceRoot string, manager *execSessionManager) Tool {
|
||||
return NewScopedExecCommandTool(workspaceRoot, nil, manager)
|
||||
}
|
||||
@@ -475,11 +484,15 @@ func NewScopedExecCommandTool(workspaceRoot string, scope PathScope, manager *ex
|
||||
}
|
||||
|
||||
func (tool execCommandTool) Run(ctx context.Context, args map[string]any) Result {
|
||||
return tool.run(ctx, args, nil)
|
||||
return tool.run(ctx, args, nil, true)
|
||||
}
|
||||
|
||||
func (tool execCommandTool) RunWithSandbox(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine) Result {
|
||||
return tool.run(ctx, args, engine)
|
||||
return tool.run(ctx, args, engine, true)
|
||||
}
|
||||
|
||||
func (tool execCommandTool) RunWithOptions(ctx context.Context, args map[string]any, options RunOptions) Result {
|
||||
return tool.run(ctx, args, options.Sandbox, false)
|
||||
}
|
||||
|
||||
func (tool execCommandTool) ExecSessions() []ExecSessionSnapshot {
|
||||
@@ -494,8 +507,8 @@ func (tool execCommandTool) StopAllExecSessions() []int {
|
||||
return tool.manager.stopAll()
|
||||
}
|
||||
|
||||
func (tool execCommandTool) run(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine) Result {
|
||||
commandText, err := aliasedStringArg(args, []string{"cmd", "command", "script", "shell"}, "", true, false)
|
||||
func (tool execCommandTool) run(ctx context.Context, args map[string]any, engine *zeroSandbox.Engine, directBudget bool) Result {
|
||||
commandText, err := execCommandArg(args)
|
||||
if err != nil {
|
||||
return errorResult("Error: Invalid arguments for exec_command: " + err.Error())
|
||||
}
|
||||
@@ -550,7 +563,7 @@ func (tool execCommandTool) run(ctx context.Context, args map[string]any, engine
|
||||
if exited {
|
||||
tool.manager.remove(session.id)
|
||||
}
|
||||
return execToolResult(execToolResultInput{
|
||||
return execToolResultWithBudget(execToolResultInput{
|
||||
commandText: commandText,
|
||||
output: output,
|
||||
outputBufferTruncated: outputTruncated,
|
||||
@@ -561,7 +574,7 @@ func (tool execCommandTool) run(ctx context.Context, args map[string]any, engine
|
||||
tty: session.tty,
|
||||
plan: session.plan,
|
||||
maxOutputTokens: maxOutputTokens,
|
||||
})
|
||||
}, directBudget)
|
||||
}
|
||||
|
||||
func (tool execCommandTool) startSession(commandText string, absoluteCwd string, relativeCwd string, ttyRequested bool, engine *zeroSandbox.Engine, sandboxPermissions SandboxPermissionOverride) (*execSession, error) {
|
||||
@@ -694,6 +707,10 @@ type writeStdinTool struct {
|
||||
manager *execSessionManager
|
||||
}
|
||||
|
||||
func (writeStdinTool) outputCategory(map[string]any) outputCategory {
|
||||
return outputCategoryProcess
|
||||
}
|
||||
|
||||
func NewWriteStdinTool(manager *execSessionManager) Tool {
|
||||
if manager == nil {
|
||||
manager = defaultExecSessionManager
|
||||
@@ -843,7 +860,15 @@ type execToolResultInput struct {
|
||||
}
|
||||
|
||||
func execToolResult(input execToolResultInput) Result {
|
||||
output, truncated := truncateExecOutput(input.output, input.maxOutputTokens)
|
||||
return execToolResultWithBudget(input, true)
|
||||
}
|
||||
|
||||
func execToolResultWithBudget(input execToolResultInput, directBudget bool) Result {
|
||||
output := input.output
|
||||
truncated := false
|
||||
if directBudget {
|
||||
output, truncated = truncateExecOutput(input.output, input.maxOutputTokens)
|
||||
}
|
||||
meta := map[string]string{
|
||||
"cwd": input.relativeCwd,
|
||||
"tty": strconv.FormatBool(input.tty),
|
||||
|
||||
+24
-15
@@ -18,6 +18,8 @@ type globTool struct {
|
||||
scope PathScope
|
||||
}
|
||||
|
||||
func (globTool) outputCategory(map[string]any) outputCategory { return outputCategorySearch }
|
||||
|
||||
func NewGlobTool(workspaceRoot string) Tool {
|
||||
return NewScopedGlobTool(workspaceRoot, nil)
|
||||
}
|
||||
@@ -39,7 +41,7 @@ func NewScopedGlobTool(workspaceRoot string, scope PathScope) Tool {
|
||||
AdditionalProperties: false,
|
||||
},
|
||||
safety: readOnlySafety("Finds matching paths without reading contents or modifying files."),
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: workspaceResourceKeys},
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: scopedScanResourceKeys},
|
||||
},
|
||||
workspaceRoot: normalizeWorkspaceRoot(workspaceRoot),
|
||||
scope: scope,
|
||||
@@ -47,17 +49,25 @@ func NewScopedGlobTool(workspaceRoot string, scope PathScope) Tool {
|
||||
}
|
||||
|
||||
func (tool globTool) Run(ctx context.Context, args map[string]any) Result {
|
||||
return tool.runWith(ctx, args, readExcluder{})
|
||||
return tool.runWith(ctx, args, readExcluder{}, true)
|
||||
}
|
||||
|
||||
func (tool globTool) RunWithOptions(ctx context.Context, args map[string]any, options RunOptions) Result {
|
||||
exclude := readExcluder{}
|
||||
if options.Sandbox != nil {
|
||||
exclude = sandboxReadExcluder(options.Sandbox)
|
||||
}
|
||||
return tool.runWith(ctx, args, exclude, false)
|
||||
}
|
||||
|
||||
// RunWithSandbox runs glob while skipping subtrees the sandbox policy denies
|
||||
// reads to (DenyRead). With no DenyRead configured the excluder is a no-op and
|
||||
// behavior is unchanged.
|
||||
func (tool globTool) RunWithSandbox(ctx context.Context, args map[string]any, engine *sandbox.Engine) Result {
|
||||
return tool.runWith(ctx, args, sandboxReadExcluder(engine))
|
||||
return tool.runWith(ctx, args, sandboxReadExcluder(engine), true)
|
||||
}
|
||||
|
||||
func (tool globTool) runWith(ctx context.Context, args map[string]any, exclude readExcluder) Result {
|
||||
func (tool globTool) runWith(ctx context.Context, args map[string]any, exclude readExcluder, directBudget bool) Result {
|
||||
pattern, err := aliasedStringArg(args, []string{"pattern", "glob", "match", "query", "expression"}, "", true, false)
|
||||
if err != nil {
|
||||
return errorResult("Error: Invalid arguments for glob: " + err.Error())
|
||||
@@ -110,24 +120,23 @@ func (tool globTool) runWith(ctx context.Context, args map[string]any, exclude r
|
||||
if truncated {
|
||||
output += fmt.Sprintf("\n\n[truncated: showing first %d of %d matches; increase limit or narrow cwd/pattern]", len(matches), totalMatches)
|
||||
}
|
||||
budgeted := applyOutputBudget(output, searchOutputBudgetBytes, "increase limit or narrow cwd/pattern")
|
||||
meta := outputBudgetMeta(budgeted)
|
||||
meta := map[string]string{}
|
||||
meta["pattern"] = pattern
|
||||
if truncated || budgeted.Truncated {
|
||||
if truncated {
|
||||
meta["truncated"] = "true"
|
||||
if budgeted.Truncated {
|
||||
meta["truncation_reason"] = "byte_budget"
|
||||
} else {
|
||||
meta["truncation_reason"] = "limit"
|
||||
}
|
||||
meta["truncation_reason"] = "limit"
|
||||
}
|
||||
|
||||
return Result{
|
||||
result := Result{
|
||||
Status: StatusOK,
|
||||
Output: budgeted.Output,
|
||||
Truncated: truncated || budgeted.Truncated,
|
||||
Output: output,
|
||||
Truncated: truncated,
|
||||
Meta: meta,
|
||||
}
|
||||
if directBudget {
|
||||
return applyLegacyByteBudgetToResult(result, searchOutputBudgetBytes, "increase limit or narrow cwd/pattern")
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func scanGlob(ctx context.Context, root string, displayRoot string, matcher *regexp.Regexp, includeDirs bool, exclude readExcluder) ([]string, error) {
|
||||
|
||||
+30
-24
@@ -21,6 +21,8 @@ type grepTool struct {
|
||||
scope PathScope
|
||||
}
|
||||
|
||||
func (grepTool) outputCategory(map[string]any) outputCategory { return outputCategorySearch }
|
||||
|
||||
type grepMatch struct {
|
||||
file string
|
||||
line int
|
||||
@@ -54,7 +56,7 @@ func NewScopedGrepTool(workspaceRoot string, scope PathScope) Tool {
|
||||
AdditionalProperties: false,
|
||||
},
|
||||
safety: readOnlySafety("Searches file paths and matching lines without modifying files."),
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: workspaceResourceKeys},
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: scopedScanResourceKeys},
|
||||
},
|
||||
workspaceRoot: normalizeWorkspaceRoot(workspaceRoot),
|
||||
scope: scope,
|
||||
@@ -62,7 +64,15 @@ func NewScopedGrepTool(workspaceRoot string, scope PathScope) Tool {
|
||||
}
|
||||
|
||||
func (tool grepTool) Run(ctx context.Context, args map[string]any) Result {
|
||||
return tool.runWith(ctx, args, readExcluder{})
|
||||
return tool.runWith(ctx, args, readExcluder{}, true)
|
||||
}
|
||||
|
||||
func (tool grepTool) RunWithOptions(ctx context.Context, args map[string]any, options RunOptions) Result {
|
||||
exclude := readExcluder{}
|
||||
if options.Sandbox != nil {
|
||||
exclude = sandboxReadExcluder(options.Sandbox)
|
||||
}
|
||||
return tool.runWith(ctx, args, exclude, false)
|
||||
}
|
||||
|
||||
// RunWithSandbox runs the search while skipping subtrees the sandbox policy
|
||||
@@ -70,10 +80,10 @@ func (tool grepTool) Run(ctx context.Context, args map[string]any) Result {
|
||||
// path. With no DenyRead configured the excluder is a no-op and behavior is
|
||||
// unchanged.
|
||||
func (tool grepTool) RunWithSandbox(ctx context.Context, args map[string]any, engine *sandbox.Engine) Result {
|
||||
return tool.runWith(ctx, args, sandboxReadExcluder(engine))
|
||||
return tool.runWith(ctx, args, sandboxReadExcluder(engine), true)
|
||||
}
|
||||
|
||||
func (tool grepTool) runWith(ctx context.Context, args map[string]any, exclude readExcluder) Result {
|
||||
func (tool grepTool) runWith(ctx context.Context, args map[string]any, exclude readExcluder, directBudget bool) Result {
|
||||
pattern, err := aliasedStringArg(args, []string{"pattern", "query", "regex", "search", "expression"}, "", true, false)
|
||||
if err != nil {
|
||||
return errorResult("Error: Invalid arguments for grep: " + err.Error())
|
||||
@@ -154,7 +164,7 @@ func (tool grepTool) runWith(ctx context.Context, args map[string]any, exclude r
|
||||
}
|
||||
return errorResult("Error running grep: " + err.Error())
|
||||
}
|
||||
return collector.result()
|
||||
return applyDirectSearchBudget(collector.result(), directBudget, "narrow path/glob/pattern to continue")
|
||||
case "files_with_matches":
|
||||
collector := &grepFileListCollector{}
|
||||
if err := scanGrepMatches(ctx, resolvedRoot, target, globMatcher, exclude, absolutePaths, presenceGrepLineMatcher(compiled), collector.collect); err != nil {
|
||||
@@ -163,7 +173,7 @@ func (tool grepTool) runWith(ctx context.Context, args map[string]any, exclude r
|
||||
}
|
||||
return errorResult("Error running grep: " + err.Error())
|
||||
}
|
||||
return collector.result()
|
||||
return applyDirectSearchBudget(collector.result(), directBudget, "narrow path/glob/pattern to continue")
|
||||
default:
|
||||
collector := &grepContentCollector{headLimit: headLimit}
|
||||
if err := scanGrepMatches(ctx, resolvedRoot, target, globMatcher, exclude, absolutePaths, presenceGrepLineMatcher(compiled), collector.collect); err != nil {
|
||||
@@ -172,10 +182,17 @@ func (tool grepTool) runWith(ctx context.Context, args map[string]any, exclude r
|
||||
}
|
||||
return errorResult("Error running grep: " + err.Error())
|
||||
}
|
||||
return collector.result()
|
||||
return applyDirectSearchBudget(collector.result(), directBudget, "narrow path/glob/pattern or increase head_limit")
|
||||
}
|
||||
}
|
||||
|
||||
func applyDirectSearchBudget(result Result, directBudget bool, hint string) Result {
|
||||
if !directBudget {
|
||||
return result
|
||||
}
|
||||
return applyLegacyByteBudgetToResult(result, searchOutputBudgetBytes, hint)
|
||||
}
|
||||
|
||||
// resolveGrepRoot picks the scope root whose EvalSymlinks-resolved path contains
|
||||
// the already-resolved target, so that confineGrepFile computes correct
|
||||
// workspace-relative paths even when the target lives in an extra root.
|
||||
@@ -455,13 +472,7 @@ func (collector *grepFileListCollector) result() Result {
|
||||
return okResult("No matches found.")
|
||||
}
|
||||
sort.Strings(collector.files)
|
||||
budgeted := applyOutputBudget(strings.Join(collector.files, "\n"), searchOutputBudgetBytes, "narrow path/glob/pattern to continue")
|
||||
meta := outputBudgetMeta(budgeted)
|
||||
if budgeted.Truncated {
|
||||
meta["truncated"] = "true"
|
||||
meta["truncation_reason"] = "byte_budget"
|
||||
}
|
||||
return Result{Status: StatusOK, Output: budgeted.Output, Truncated: budgeted.Truncated, Meta: meta}
|
||||
return Result{Status: StatusOK, Output: strings.Join(collector.files, "\n")}
|
||||
}
|
||||
|
||||
type grepContentCollector struct {
|
||||
@@ -494,20 +505,15 @@ func (collector *grepContentCollector) result() Result {
|
||||
if truncated {
|
||||
output += fmt.Sprintf("\n\n[truncated: showing first %d matches; narrow path/glob/pattern or increase head_limit]", len(lines))
|
||||
}
|
||||
budgeted := applyOutputBudget(output, searchOutputBudgetBytes, "narrow path/glob/pattern or increase head_limit")
|
||||
meta := outputBudgetMeta(budgeted)
|
||||
if truncated || budgeted.Truncated {
|
||||
meta := map[string]string{}
|
||||
if truncated {
|
||||
meta["truncated"] = "true"
|
||||
if budgeted.Truncated {
|
||||
meta["truncation_reason"] = "byte_budget"
|
||||
} else {
|
||||
meta["truncation_reason"] = "head_limit"
|
||||
}
|
||||
meta["truncation_reason"] = "head_limit"
|
||||
}
|
||||
return Result{
|
||||
Status: StatusOK,
|
||||
Output: budgeted.Output,
|
||||
Truncated: truncated || budgeted.Truncated,
|
||||
Output: output,
|
||||
Truncated: truncated,
|
||||
Meta: meta,
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,10 @@ type listDirectoryTool struct {
|
||||
scope PathScope
|
||||
}
|
||||
|
||||
func (listDirectoryTool) outputCategory(map[string]any) outputCategory {
|
||||
return outputCategorySearch
|
||||
}
|
||||
|
||||
func NewListDirectoryTool(workspaceRoot string) Tool {
|
||||
return NewScopedListDirectoryTool(workspaceRoot, nil)
|
||||
}
|
||||
@@ -34,7 +38,7 @@ func NewScopedListDirectoryTool(workspaceRoot string, scope PathScope) Tool {
|
||||
AdditionalProperties: false,
|
||||
},
|
||||
safety: readOnlySafety("Lists directory entries without modifying files."),
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: directoryResourceKeys},
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: directoryResourceKeys},
|
||||
},
|
||||
workspaceRoot: normalizeWorkspaceRoot(workspaceRoot),
|
||||
scope: scope,
|
||||
@@ -42,6 +46,14 @@ func NewScopedListDirectoryTool(workspaceRoot string, scope PathScope) Tool {
|
||||
}
|
||||
|
||||
func (tool listDirectoryTool) Run(_ context.Context, args map[string]any) Result {
|
||||
return tool.run(args, true)
|
||||
}
|
||||
|
||||
func (tool listDirectoryTool) RunWithOptions(_ context.Context, args map[string]any, _ RunOptions) Result {
|
||||
return tool.run(args, false)
|
||||
}
|
||||
|
||||
func (tool listDirectoryTool) run(args map[string]any, directBudget bool) Result {
|
||||
// Optional with a "." default: treat an explicit empty path (a common
|
||||
// weak-model quirk) the same as the key being absent rather than erroring.
|
||||
requestedPath, err := aliasedStringArg(args, []string{"path", "directory", "dir"}, ".", false, true)
|
||||
@@ -76,13 +88,11 @@ func (tool listDirectoryTool) Run(_ context.Context, args map[string]any) Result
|
||||
return okResult("Directory is empty: " + relativePath)
|
||||
}
|
||||
output := "Contents of " + relativePath + ":\n\n" + strings.Join(entries, "\n")
|
||||
budgeted := applyOutputBudget(output, searchOutputBudgetBytes, "use path, recursive=false, or a smaller max_depth to narrow the listing")
|
||||
meta := outputBudgetMeta(budgeted)
|
||||
if budgeted.Truncated {
|
||||
meta["truncated"] = "true"
|
||||
meta["truncation_reason"] = "byte_budget"
|
||||
result := Result{Status: StatusOK, Output: output}
|
||||
if directBudget {
|
||||
return applyLegacyByteBudgetToResult(result, searchOutputBudgetBytes, "use path, recursive=false, or a smaller max_depth to narrow the listing")
|
||||
}
|
||||
return Result{Status: StatusOK, Output: budgeted.Output, Truncated: budgeted.Truncated, Meta: meta}
|
||||
return result
|
||||
}
|
||||
|
||||
func listDirectoryEntries(path string, depth int, maxDepth int) ([]string, error) {
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
package tools
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const (
|
||||
outputBudgetCategoryMeta = "output_budget_category"
|
||||
outputBudgetOriginalBytesMeta = "output_budget_original_bytes"
|
||||
outputBudgetRetainedBytesMeta = "output_budget_retained_bytes"
|
||||
outputBudgetEstimatedOriginalTokensMeta = "output_budget_estimated_original_tokens"
|
||||
outputBudgetEstimatedRetainedTokensMeta = "output_budget_estimated_retained_tokens"
|
||||
outputBudgetReasonMeta = "output_budget_reason"
|
||||
outputBudgetSpillCreatedMeta = "output_budget_spill_created"
|
||||
)
|
||||
|
||||
// applyRegistryOutputBudget is the common post-redaction semantic budgeting
|
||||
// boundary for tools that do not already own a deliberate output budget.
|
||||
func applyRegistryOutputBudget(tool Tool, toolName string, args map[string]any, result Result) Result {
|
||||
budget := registryOutputBudget(toolName)
|
||||
if budget.maxEstimatedTokens <= 0 && budget.hardMaxBytes <= 0 {
|
||||
return result // preserve ZERO_TOOL_OUTPUT_CEILING_TOKENS=0 semantics
|
||||
}
|
||||
|
||||
category := resolveOutputCategory(tool, toolName, args)
|
||||
budgeted := budgetSemanticOutput(result.Output, category, budget)
|
||||
if budgeted.truncated {
|
||||
budgeted = attachExistingSpill(toolName, result.Output, budget, budgeted)
|
||||
}
|
||||
if result.Truncated {
|
||||
budgeted.truncated = true
|
||||
if budgeted.reason == "" {
|
||||
budgeted.reason = result.Meta["truncation_reason"]
|
||||
if budgeted.reason == "" {
|
||||
budgeted.reason = "upstream_tool_budget"
|
||||
}
|
||||
}
|
||||
if budgeted.spillPath == "" {
|
||||
budgeted.spillPath = result.Meta["spill_path"]
|
||||
}
|
||||
}
|
||||
result.Output = budgeted.text
|
||||
result.Truncated = result.Truncated || budgeted.truncated
|
||||
result.Meta = addOutputBudgetMetadata(result.Meta, budgeted)
|
||||
return result
|
||||
}
|
||||
|
||||
// applySelfManagedOutputBudget applies semantic retention to tools that retain
|
||||
// output with their own capture-aware limits. Their explicit budgets remain
|
||||
// authoritative, rather than being replaced by the registry default ceiling.
|
||||
func applySelfManagedOutputBudget(tool Tool, toolName string, args map[string]any, result Result) Result {
|
||||
budget := selfManagedOutputBudget(toolName, args)
|
||||
if budget.maxEstimatedTokens <= 0 && budget.hardMaxBytes <= 0 {
|
||||
return annotateSelfBudgetedOutput(tool, toolName, args, result)
|
||||
}
|
||||
|
||||
category := resolveOutputCategory(tool, toolName, args)
|
||||
budgeted := budgetSemanticOutput(result.Output, category, budget)
|
||||
if result.Truncated && !budgeted.truncated {
|
||||
budgeted.truncated = true
|
||||
budgeted.reason = result.Meta["truncation_reason"]
|
||||
if budgeted.reason == "" {
|
||||
budgeted.reason = "upstream_tool_budget"
|
||||
}
|
||||
}
|
||||
if budgeted.truncated {
|
||||
budgeted = attachExistingSpill(toolName, result.Output, budget, budgeted)
|
||||
}
|
||||
result.Output = budgeted.text
|
||||
result.Truncated = result.Truncated || budgeted.truncated
|
||||
result.Meta = addOutputBudgetMetadata(result.Meta, budgeted)
|
||||
return result
|
||||
}
|
||||
|
||||
func selfManagedOutputBudget(toolName string, args map[string]any) outputBudget {
|
||||
switch toolName {
|
||||
case "bash":
|
||||
// bash previously exposed at most 32 KiB from each stream. Keep that
|
||||
// combined 64 KiB ceiling while selecting its contents semantically.
|
||||
return outputBudget{maxEstimatedTokens: (bashOutputBudgetBytes * 2) / 4, hardMaxBytes: bashOutputBudgetBytes * 2}
|
||||
case ExecCommandToolName:
|
||||
maxOutputTokens := defaultMaxOutputTokens
|
||||
if parsed, err := intArg(args, "max_output_tokens", defaultMaxOutputTokens, 1, maxExecOutputTokenRequest); err == nil {
|
||||
maxOutputTokens = parsed
|
||||
}
|
||||
return outputBudget{maxEstimatedTokens: maxOutputTokens, hardMaxBytes: maxOutputTokens * 4}
|
||||
default:
|
||||
return outputBudget{}
|
||||
}
|
||||
}
|
||||
|
||||
// RebudgetAfterHook reapplies the registry's redaction and output limits after
|
||||
// an afterTool hook appends model-visible feedback to a completed result. The
|
||||
// initial registry pass has already run; this second pass is limited to the
|
||||
// newly combined result so hooks cannot bypass the established safety ceiling.
|
||||
func (registry *Registry) RebudgetAfterHook(toolName string, args map[string]any, result Result) Result {
|
||||
result = scrubResultSecrets(result)
|
||||
tool, _ := registry.Get(toolName)
|
||||
if _, ok := tool.(selfBudgeting); ok {
|
||||
// Match the primary registry boundary: self-managed tools keep their
|
||||
// call-specific capture/output budget instead of being tightened or
|
||||
// loosened to the generic registry ceiling after hook feedback.
|
||||
return applySelfManagedOutputBudget(tool, toolName, args, result)
|
||||
}
|
||||
result = applyRegistryOutputBudget(tool, toolName, args, result)
|
||||
return enforceOutputCeiling(toolName, result)
|
||||
}
|
||||
|
||||
func registryOutputBudget(toolName string) outputBudget {
|
||||
switch toolName {
|
||||
case "read_file", "read_minified_file":
|
||||
return outputBudget{maxEstimatedTokens: readOutputBudgetBytes / 4, hardMaxBytes: readOutputBudgetBytes}
|
||||
case "grep", "glob", "list_directory":
|
||||
return outputBudget{maxEstimatedTokens: searchOutputBudgetBytes / 4, hardMaxBytes: searchOutputBudgetBytes}
|
||||
default:
|
||||
ceilingTokens := resolveOutputCeilingTokens()
|
||||
if ceilingTokens <= 0 {
|
||||
return outputBudget{}
|
||||
}
|
||||
return outputBudget{maxEstimatedTokens: ceilingTokens, hardMaxBytes: ceilingTokens * 4}
|
||||
}
|
||||
}
|
||||
|
||||
func resolveOutputCategory(tool Tool, toolName string, args map[string]any) outputCategory {
|
||||
if provider, ok := tool.(outputPolicyProvider); ok {
|
||||
if category := provider.outputCategory(args); category != "" {
|
||||
return category
|
||||
}
|
||||
}
|
||||
switch toolName {
|
||||
case "Task", "swarm_collect":
|
||||
return outputCategoryWorker
|
||||
case "apply_patch":
|
||||
return outputCategoryDiff
|
||||
case "write_stdin":
|
||||
return outputCategoryProcess
|
||||
default:
|
||||
return outputCategoryDefault
|
||||
}
|
||||
}
|
||||
|
||||
// annotateSelfBudgetedOutput records the same compact metadata for tools whose
|
||||
// existing capture/budget implementation remains authoritative in PR11. It
|
||||
// does not re-budget their text or claim that raw_bytes represents every byte a
|
||||
// subprocess produced beyond its established capture limits.
|
||||
func annotateSelfBudgetedOutput(tool Tool, toolName string, args map[string]any, result Result) Result {
|
||||
retainedBytes := len(result.Output)
|
||||
originalBytes := retainedBytes
|
||||
if parsed, err := strconv.Atoi(result.Meta["raw_bytes"]); err == nil && parsed > originalBytes {
|
||||
originalBytes = parsed
|
||||
}
|
||||
retainedTokens := estimateOutputTokens(result.Output)
|
||||
originalTokens := retainedTokens
|
||||
if originalBytes > retainedBytes {
|
||||
originalTokens = max(originalTokens, estimatedTokensFromBytes(originalBytes))
|
||||
}
|
||||
reason := result.Meta["truncation_reason"]
|
||||
if result.Truncated && reason == "" {
|
||||
reason = "upstream_tool_budget"
|
||||
}
|
||||
observed := budgetedOutput{
|
||||
text: result.Output,
|
||||
originalBytes: originalBytes,
|
||||
retainedBytes: retainedBytes,
|
||||
estimatedOriginalTokens: originalTokens,
|
||||
estimatedRetainedTokens: retainedTokens,
|
||||
truncated: result.Truncated,
|
||||
category: resolveOutputCategory(tool, toolName, args),
|
||||
reason: reason,
|
||||
spillPath: result.Meta["spill_path"],
|
||||
}
|
||||
result.Meta = addOutputBudgetMetadata(result.Meta, observed)
|
||||
return result
|
||||
}
|
||||
|
||||
func shellOutputCategory(command string) outputCategory {
|
||||
normalized := strings.ToLower(strings.TrimSpace(command))
|
||||
if containsAny(normalized,
|
||||
"go test", "pytest", "python -m pytest", "cargo test", "npm test", "npm run test",
|
||||
"pnpm test", "yarn test", "bun test", "dotnet test", "mvn test", "gradle test", "phpunit") {
|
||||
return outputCategoryTest
|
||||
}
|
||||
if containsAny(normalized, "git diff", "git show", "git format-patch", " diff -u", "diff --git") || strings.HasPrefix(normalized, "diff ") {
|
||||
return outputCategoryDiff
|
||||
}
|
||||
return outputCategoryProcess
|
||||
}
|
||||
|
||||
// attachExistingSpill reuses Zero's hardened per-user spill directory. output
|
||||
// is the already-redacted text received by this layer; it may itself be a
|
||||
// capture-bounded view produced by a subprocess tool.
|
||||
func attachExistingSpill(toolName, output string, budget outputBudget, current budgetedOutput) budgetedOutput {
|
||||
path := spillTruncatedOutput(toolName, output)
|
||||
if path == "" {
|
||||
return current
|
||||
}
|
||||
notice := "[zero] full output received by budgeting layer saved to " + path + " (grep or read_file it instead of re-running)"
|
||||
reduced := outputBudget{
|
||||
maxEstimatedTokens: max(1, budget.maxEstimatedTokens-estimateOutputTokens("\n"+notice)),
|
||||
hardMaxBytes: max(1, budget.hardMaxBytes-len("\n"+notice)),
|
||||
}
|
||||
base := budgetSemanticOutput(output, current.category, reduced)
|
||||
text := strings.TrimRight(base.text, "\n") + "\n" + notice
|
||||
if !fitsOutputBudget(text, budget) {
|
||||
// An unusually long temp path or tiny configured ceiling can leave no
|
||||
// room for the full notice. Keep the safe bounded result; the spill still
|
||||
// exists but is intentionally not advertised with a chopped reference.
|
||||
current.spillPath = path
|
||||
return current
|
||||
}
|
||||
base.text = text
|
||||
base.retainedBytes = len(text)
|
||||
base.estimatedRetainedTokens = estimateOutputTokens(text)
|
||||
base.truncated = true
|
||||
if base.reason == "" {
|
||||
base.reason = current.reason
|
||||
}
|
||||
base.category = current.category
|
||||
base.spillPath = path
|
||||
return base
|
||||
}
|
||||
|
||||
func addOutputBudgetMetadata(meta map[string]string, output budgetedOutput) map[string]string {
|
||||
if meta == nil {
|
||||
meta = map[string]string{}
|
||||
}
|
||||
meta[outputBudgetCategoryMeta] = string(output.category)
|
||||
meta[outputBudgetOriginalBytesMeta] = strconv.Itoa(output.originalBytes)
|
||||
meta[outputBudgetRetainedBytesMeta] = strconv.Itoa(output.retainedBytes)
|
||||
meta[outputBudgetEstimatedOriginalTokensMeta] = strconv.Itoa(output.estimatedOriginalTokens)
|
||||
meta[outputBudgetEstimatedRetainedTokensMeta] = strconv.Itoa(output.estimatedRetainedTokens)
|
||||
meta[outputBudgetSpillCreatedMeta] = strconv.FormatBool(output.spillPath != "")
|
||||
// These established fields describe the final model-visible output. Refresh
|
||||
// them on every pass because redaction and after-tool hooks may change the
|
||||
// text even when this pass does not newly truncate it.
|
||||
meta["emitted_bytes"] = strconv.Itoa(output.retainedBytes)
|
||||
meta["estimated_tokens"] = strconv.Itoa(output.estimatedRetainedTokens)
|
||||
if output.reason != "" {
|
||||
meta[outputBudgetReasonMeta] = output.reason
|
||||
}
|
||||
if output.truncated {
|
||||
// Preserve the existing metadata vocabulary used by callers and tests.
|
||||
if _, exists := meta["raw_bytes"]; !exists {
|
||||
meta["raw_bytes"] = strconv.Itoa(output.originalBytes)
|
||||
}
|
||||
meta["truncated"] = "true"
|
||||
meta["truncation_reason"] = output.reason
|
||||
}
|
||||
if output.spillPath != "" {
|
||||
meta["spill_path"] = output.spillPath
|
||||
}
|
||||
return meta
|
||||
}
|
||||
@@ -0,0 +1,381 @@
|
||||
package tools
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
func TestRegistryBudgetRunsAfterRedactionAndSpillsRedactedOutput(t *testing.T) {
|
||||
setTestTempDir(t)
|
||||
secret := "ghp_" + strings.Repeat("a", 36)
|
||||
big := "HEAD\n" + secret + "\n" + strings.Repeat("🙂 noisy output\n", 20_000) + "TAIL"
|
||||
registry := NewRegistry()
|
||||
registry.Register(newCeilingFakeTool("redacted_big", big))
|
||||
|
||||
result := registry.Run(context.Background(), "redacted_big", map[string]any{})
|
||||
if !result.Truncated || result.Meta[outputBudgetSpillCreatedMeta] != "true" {
|
||||
t.Fatalf("unexpected budget result: truncated=%t meta=%#v", result.Truncated, result.Meta)
|
||||
}
|
||||
if strings.Contains(result.Output, secret) || !utf8.ValidString(result.Output) {
|
||||
t.Fatalf("exposed output leaked secret or invalid UTF-8: %q", result.Output)
|
||||
}
|
||||
spillPath := result.Meta["spill_path"]
|
||||
content, err := os.ReadFile(spillPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read spill: %v", err)
|
||||
}
|
||||
if strings.Contains(string(content), secret) {
|
||||
t.Fatal("spill contains unredacted secret")
|
||||
}
|
||||
if !strings.Contains(string(content), "HEAD") || !strings.Contains(string(content), "TAIL") {
|
||||
t.Fatal("spill does not contain the complete redacted output received by the budget layer")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryBudgetSpillFailureFallsBackToBoundedOutput(t *testing.T) {
|
||||
temp := t.TempDir()
|
||||
blockedTemp := filepath.Join(temp, "not-a-directory")
|
||||
if err := os.WriteFile(blockedTemp, []byte("x"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("TMPDIR", blockedTemp)
|
||||
t.Setenv("TMP", blockedTemp)
|
||||
t.Setenv("TEMP", blockedTemp)
|
||||
t.Setenv(outputCeilingEnv, "100")
|
||||
|
||||
registry := NewRegistry()
|
||||
registry.Register(newCeilingFakeTool("spill_failure", strings.Repeat("large output\n", 1000)))
|
||||
result := registry.Run(context.Background(), "spill_failure", map[string]any{})
|
||||
if !result.Truncated || len(result.Output) > 400 {
|
||||
t.Fatalf("fallback is not bounded: truncated=%t bytes=%d", result.Truncated, len(result.Output))
|
||||
}
|
||||
if result.Meta[outputBudgetSpillCreatedMeta] != "false" || result.Meta["spill_path"] != "" {
|
||||
t.Fatalf("spill failure incorrectly advertised: %#v", result.Meta)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryMissingPolicyUsesDefaultAndExactHardCeiling(t *testing.T) {
|
||||
setTestTempDir(t)
|
||||
t.Setenv(outputCeilingEnv, "80")
|
||||
input := "HEAD\n" + strings.Repeat("x", 3000) + "\nTAIL"
|
||||
registry := NewRegistry()
|
||||
registry.Register(newCeilingFakeTool("no_policy", input))
|
||||
result := registry.Run(context.Background(), "no_policy", map[string]any{})
|
||||
if got := result.Meta[outputBudgetCategoryMeta]; got != string(outputCategoryDefault) {
|
||||
t.Fatalf("category = %q, want default", got)
|
||||
}
|
||||
if len(result.Output) > 80*4 {
|
||||
t.Fatalf("output = %d bytes, hard ceiling %d", len(result.Output), 80*4)
|
||||
}
|
||||
if result.Meta[outputBudgetOriginalBytesMeta] != strconv.Itoa(len(input)) {
|
||||
t.Fatalf("original size metadata = %q", result.Meta[outputBudgetOriginalBytesMeta])
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistrySmallOutputRemainsByteIdentical(t *testing.T) {
|
||||
registry := NewRegistry()
|
||||
registry.Register(newCeilingFakeTool("small_identity", "hello\nworld\n"))
|
||||
result := registry.Run(context.Background(), "small_identity", map[string]any{})
|
||||
if result.Output != "hello\nworld\n" || result.Truncated {
|
||||
t.Fatalf("small output changed: %#v", result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDirectReadAndSearchToolsKeepLegacyByteBudgets(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(root, "large-read.txt"), []byte(strings.Repeat("source line with enough content\n", 7000)), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if result := NewReadFileTool(root).Run(context.Background(), map[string]any{"path": "large-read.txt"}); !result.Truncated || len(result.Output) > readOutputBudgetBytes {
|
||||
t.Fatalf("direct read_file output is not bounded: %#v", result)
|
||||
}
|
||||
|
||||
for index := 0; index < 800; index++ {
|
||||
name := fmt.Sprintf("%04d-%s.txt", index, strings.Repeat("n", 80))
|
||||
if err := os.WriteFile(filepath.Join(root, name), []byte("match\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, "large-content.txt"), []byte(strings.Repeat("match "+strings.Repeat("x", 2000)+"\n", 50)), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
for name, result := range map[string]Result{
|
||||
"list_directory": NewListDirectoryTool(root).Run(context.Background(), map[string]any{"recursive": false}),
|
||||
"glob": NewGlobTool(root).Run(context.Background(), map[string]any{"pattern": "*.txt", "limit": 1000}),
|
||||
"grep files": NewGrepTool(root).Run(context.Background(), map[string]any{"pattern": "match", "output_mode": "files_with_matches"}),
|
||||
"grep content": NewGrepTool(root).Run(context.Background(), map[string]any{"pattern": "match", "path": "large-content.txt", "head_limit": 50}),
|
||||
} {
|
||||
if !result.Truncated || len(result.Output) > searchOutputBudgetBytes {
|
||||
t.Fatalf("direct %s output is not bounded: truncated=%t bytes=%d meta=%#v", name, result.Truncated, len(result.Output), result.Meta)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryLargeFileUsesSemanticFilePolicy(t *testing.T) {
|
||||
setTestTempDir(t)
|
||||
root := t.TempDir()
|
||||
var content strings.Builder
|
||||
content.WriteString("HEAD_MARK\n")
|
||||
for line := 0; line < 7000; line++ {
|
||||
content.WriteString("ordinary source line with enough text to fill the output budget\n")
|
||||
}
|
||||
content.WriteString("TAIL_MARK\n")
|
||||
if err := os.WriteFile(filepath.Join(root, "large.txt"), []byte(content.String()), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
registry := NewRegistry()
|
||||
registry.Register(NewReadFileTool(root))
|
||||
result := registry.Run(context.Background(), "read_file", map[string]any{"path": "large.txt"})
|
||||
if !result.Truncated || result.Meta[outputBudgetCategoryMeta] != string(outputCategoryFile) {
|
||||
t.Fatalf("large file was not semantically budgeted: truncated=%t meta=%#v", result.Truncated, result.Meta)
|
||||
}
|
||||
if len(result.Output) > readOutputBudgetBytes {
|
||||
t.Fatalf("large file output = %d bytes, ceiling %d", len(result.Output), readOutputBudgetBytes)
|
||||
}
|
||||
for _, want := range []string{"File: large.txt", "HEAD_MARK", "TAIL_MARK"} {
|
||||
if !strings.Contains(result.Output, want) {
|
||||
t.Fatalf("large file output missing %q", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryLargeSingleLineFileRetainsHeadAndTail(t *testing.T) {
|
||||
setTestTempDir(t)
|
||||
root := t.TempDir()
|
||||
content := "HEAD_SINGLE_LINE_MARK" + strings.Repeat("x", readOutputBudgetBytes*2) + "TAIL_SINGLE_LINE_MARK"
|
||||
if err := os.WriteFile(filepath.Join(root, "large.min.js"), []byte(content), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
registry := NewRegistry()
|
||||
registry.Register(NewReadFileTool(root))
|
||||
result := registry.Run(context.Background(), "read_file", map[string]any{"path": "large.min.js"})
|
||||
if !result.Truncated || result.Meta[outputBudgetCategoryMeta] != string(outputCategoryFile) {
|
||||
t.Fatalf("single-line file was not semantically budgeted: truncated=%t meta=%#v", result.Truncated, result.Meta)
|
||||
}
|
||||
if len(result.Output) > readOutputBudgetBytes || !utf8.ValidString(result.Output) {
|
||||
t.Fatalf("single-line file output is not safely bounded: bytes=%d valid=%t", len(result.Output), utf8.ValidString(result.Output))
|
||||
}
|
||||
if rawBytes, err := strconv.Atoi(result.Meta["raw_bytes"]); err != nil || rawBytes <= len(result.Output) {
|
||||
t.Fatalf("single-line file capture was not bounded head/tail output: raw=%q emitted=%d", result.Meta["raw_bytes"], len(result.Output))
|
||||
}
|
||||
for _, want := range []string{"File: large.min.js", "HEAD_SINGLE_LINE_MARK", "TAIL_SINGLE_LINE_MARK"} {
|
||||
if !strings.Contains(result.Output, want) {
|
||||
t.Fatalf("single-line file output missing %q", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryMinifiedFileUpdatesEmittedBytesAfterBudgeting(t *testing.T) {
|
||||
setTestTempDir(t)
|
||||
root := t.TempDir()
|
||||
content := "HEAD_MINIFIED_MARK" + strings.Repeat("x", readOutputBudgetBytes*2) + "TAIL_MINIFIED_MARK"
|
||||
if err := os.WriteFile(filepath.Join(root, "large.txt"), []byte(content), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
registry := NewRegistry()
|
||||
registry.Register(NewReadMinifiedFileTool(root))
|
||||
result := registry.Run(context.Background(), "read_minified_file", map[string]any{"path": "large.txt"})
|
||||
if !result.Truncated || result.Meta["emitted_bytes"] != strconv.Itoa(len(result.Output)) {
|
||||
t.Fatalf("minified metadata does not describe final output: truncated=%t emitted=%q bytes=%d", result.Truncated, result.Meta["emitted_bytes"], len(result.Output))
|
||||
}
|
||||
for _, want := range []string{"HEAD_MINIFIED_MARK", "TAIL_MINIFIED_MARK"} {
|
||||
if !strings.Contains(result.Output, want) {
|
||||
t.Fatalf("minified single-line file output missing %q", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryGrepUsesSemanticMultiFileCoverage(t *testing.T) {
|
||||
setTestTempDir(t)
|
||||
root := t.TempDir()
|
||||
for _, name := range []string{"a.txt", "b.txt", "c.txt", "d.txt"} {
|
||||
body := strings.Repeat("needle "+strings.Repeat(name, 15)+"\n", 350)
|
||||
if err := os.WriteFile(filepath.Join(root, name), []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
registry := NewRegistry()
|
||||
registry.Register(NewGrepTool(root))
|
||||
result := registry.Run(context.Background(), "grep", map[string]any{
|
||||
"pattern": "needle",
|
||||
"path": ".",
|
||||
"head_limit": 1400,
|
||||
})
|
||||
if !result.Truncated || result.Meta[outputBudgetCategoryMeta] != string(outputCategorySearch) {
|
||||
t.Fatalf("grep was not semantically budgeted: truncated=%t meta=%#v", result.Truncated, result.Meta)
|
||||
}
|
||||
for _, name := range []string{"a.txt:", "b.txt:", "c.txt:", "d.txt:"} {
|
||||
if !strings.Contains(result.Output, name) {
|
||||
t.Fatalf("search output lost %s coverage", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestShellOutputCategoryClassification(t *testing.T) {
|
||||
tests := []struct {
|
||||
command string
|
||||
want outputCategory
|
||||
}{
|
||||
{"go test ./...", outputCategoryTest},
|
||||
{"python -m pytest -q", outputCategoryTest},
|
||||
{"git diff --stat main...HEAD", outputCategoryDiff},
|
||||
{"git show HEAD", outputCategoryDiff},
|
||||
{"make build", outputCategoryProcess},
|
||||
}
|
||||
for _, test := range tests {
|
||||
if got := shellOutputCategory(test.command); got != test.want {
|
||||
t.Errorf("shellOutputCategory(%q) = %q, want %q", test.command, got, test.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestShellToolOutputCategoryUsesAllCommandAliases(t *testing.T) {
|
||||
providers := []struct {
|
||||
name string
|
||||
provider outputPolicyProvider
|
||||
}{
|
||||
{name: "bash", provider: NewBashTool(t.TempDir()).(outputPolicyProvider)},
|
||||
{name: "exec_command", provider: NewExecCommandTool(t.TempDir(), newExecSessionManager()).(outputPolicyProvider)},
|
||||
}
|
||||
for _, provider := range providers {
|
||||
for _, alias := range []string{"command", "cmd", "script", "shell"} {
|
||||
t.Run(provider.name+"/"+alias, func(t *testing.T) {
|
||||
if got := provider.provider.outputCategory(map[string]any{alias: "go test ./..."}); got != outputCategoryTest {
|
||||
t.Fatalf("category = %q, want test", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
if got := providers[0].provider.outputCategory(map[string]any{"command": "make build", "cmd": "go test ./..."}); got != outputCategoryProcess {
|
||||
t.Fatalf("bash alias precedence category = %q, want process", got)
|
||||
}
|
||||
if got := providers[1].provider.outputCategory(map[string]any{"cmd": "go test ./...", "command": "make build"}); got != outputCategoryTest {
|
||||
t.Fatalf("exec_command alias precedence category = %q, want test", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryBudgetPreservesExistingTruncationAndRefreshesMetadata(t *testing.T) {
|
||||
output := "post-hook output"
|
||||
result := applyRegistryOutputBudget(newCeilingFakeTool("existing_truncation", output), "existing_truncation", map[string]any{}, Result{
|
||||
Status: StatusOK,
|
||||
Output: output,
|
||||
Truncated: true,
|
||||
Meta: map[string]string{
|
||||
"emitted_bytes": "999",
|
||||
"estimated_tokens": "999",
|
||||
"truncation_reason": "capture_budget",
|
||||
},
|
||||
})
|
||||
if !result.Truncated || result.Meta["truncated"] != "true" || result.Meta[outputBudgetReasonMeta] != "capture_budget" {
|
||||
t.Fatalf("prior truncation state was lost: %#v", result)
|
||||
}
|
||||
if result.Meta["emitted_bytes"] != strconv.Itoa(len(output)) || result.Meta["estimated_tokens"] != strconv.Itoa(estimateOutputTokens(output)) {
|
||||
t.Fatalf("final output metadata is stale: %#v", result.Meta)
|
||||
}
|
||||
if result.Meta[outputBudgetSpillCreatedMeta] != "false" {
|
||||
t.Fatalf("existing truncation unexpectedly created a new spill: %#v", result.Meta)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelfManagedOutputBudgetUsesSemanticTestPolicy(t *testing.T) {
|
||||
setTestTempDir(t)
|
||||
input := "stdout:\n" + strings.Repeat("PASS progress\n", 8_000) +
|
||||
"--- FAIL: TestImportant (0.01s)\nthing_test.go:42: expected 7, got 9\nFAIL\nexit_code: 1"
|
||||
tests := []struct {
|
||||
name string
|
||||
tool Tool
|
||||
args map[string]any
|
||||
}{
|
||||
{name: "bash", tool: NewBashTool(t.TempDir()), args: map[string]any{"command": "go test ./..."}},
|
||||
{name: "exec", tool: NewExecCommandTool(t.TempDir(), newExecSessionManager()), args: map[string]any{"cmd": "go test ./...", "max_output_tokens": 16_000}},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
got := applySelfManagedOutputBudget(test.tool, test.tool.Name(), test.args, Result{Status: StatusError, Output: input, Meta: map[string]string{}})
|
||||
if !got.Truncated || got.Meta[outputBudgetCategoryMeta] != string(outputCategoryTest) {
|
||||
t.Fatalf("semantic budget was not applied: %#v", got)
|
||||
}
|
||||
for _, want := range []string{"TestImportant", "expected 7", "FAIL", "exit_code: 1"} {
|
||||
if !strings.Contains(got.Output, want) {
|
||||
t.Fatalf("semantic test output missing %q:\n%s", want, got.Output)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type semanticBashFakeTool struct{ ceilingFakeTool }
|
||||
|
||||
func (semanticBashFakeTool) managesOutputBudget() {}
|
||||
|
||||
func (semanticBashFakeTool) outputCategory(args map[string]any) outputCategory {
|
||||
command, _ := args["command"].(string)
|
||||
return shellOutputCategory(command)
|
||||
}
|
||||
|
||||
func TestRegistryAppliesSemanticBudgetToSelfManagedShellOutput(t *testing.T) {
|
||||
setTestTempDir(t)
|
||||
input := "startup\n" + strings.Repeat("PASS progress\n", 8_000) +
|
||||
"--- FAIL: TestRegistryImportant (0.01s)\nexpected 7, got 9\nFAIL\nexit_code: 1"
|
||||
registry := NewRegistry()
|
||||
registry.Register(semanticBashFakeTool{newCeilingFakeTool("bash", input)})
|
||||
result := registry.Run(context.Background(), "bash", map[string]any{"command": "go test ./..."})
|
||||
if !result.Truncated || result.Meta[outputBudgetReasonMeta] != "semantic_test_budget" {
|
||||
t.Fatalf("registry did not apply semantic test budget: %#v", result.Meta)
|
||||
}
|
||||
for _, want := range []string{"TestRegistryImportant", "expected 7", "FAIL", "exit_code: 1"} {
|
||||
if !strings.Contains(result.Output, want) {
|
||||
t.Fatalf("registry semantic test output missing %q:\n%s", want, result.Output)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryAppliesSemanticBudgetToSelfManagedProcessOutput(t *testing.T) {
|
||||
setTestTempDir(t)
|
||||
input := "starting build\n" + strings.Repeat("progress\n", 8_000) +
|
||||
"WARNING: cache is cold\nERROR: compilation failed\nexit_code: 1"
|
||||
registry := NewRegistry()
|
||||
registry.Register(semanticBashFakeTool{newCeilingFakeTool("bash", input)})
|
||||
result := registry.Run(context.Background(), "bash", map[string]any{"command": "make build"})
|
||||
if !result.Truncated || result.Meta[outputBudgetReasonMeta] != "semantic_process_budget" {
|
||||
t.Fatalf("registry did not apply semantic process budget: %#v", result.Meta)
|
||||
}
|
||||
for _, want := range []string{"starting build", "WARNING", "ERROR", "exit_code: 1"} {
|
||||
if !strings.Contains(result.Output, want) {
|
||||
t.Fatalf("registry semantic process output missing %q:\n%s", want, result.Output)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRebudgetAfterHookPreservesSelfManagedExecBudget(t *testing.T) {
|
||||
setTestTempDir(t)
|
||||
registry := NewRegistry()
|
||||
registry.Register(NewExecCommandTool(t.TempDir(), newExecSessionManager()))
|
||||
|
||||
t.Run("tight call budget remains authoritative", func(t *testing.T) {
|
||||
args := map[string]any{"cmd": "make build", "max_output_tokens": 10}
|
||||
result := registry.RebudgetAfterHook(ExecCommandToolName, args, Result{Status: StatusOK, Output: strings.Repeat("x", 1_000)})
|
||||
if !result.Truncated || len(result.Output) > 40 {
|
||||
t.Fatalf("post-hook output escaped call budget: truncated=%t bytes=%d meta=%#v", result.Truncated, len(result.Output), result.Meta)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("raised call budget is not replaced by generic ceiling", func(t *testing.T) {
|
||||
args := map[string]any{"cmd": "make build", "max_output_tokens": 20_000}
|
||||
output := strings.Repeat("x", 70_000)
|
||||
result := registry.RebudgetAfterHook(ExecCommandToolName, args, Result{Status: StatusOK, Output: output})
|
||||
if result.Truncated || result.Output != output {
|
||||
t.Fatalf("post-hook output was incorrectly limited by generic ceiling: truncated=%t bytes=%d", result.Truncated, len(result.Output))
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -11,6 +12,167 @@ const (
|
||||
searchOutputBudgetBytes = 64 * 1024
|
||||
)
|
||||
|
||||
// outputCategory selects the deterministic retention policy used for a tool
|
||||
// result. It is deliberately internal: categories are an implementation detail
|
||||
// of the tools boundary, not part of Zero's public tool protocol.
|
||||
type outputCategory string
|
||||
|
||||
const (
|
||||
outputCategoryDefault outputCategory = "default"
|
||||
outputCategoryFile outputCategory = "file"
|
||||
outputCategorySearch outputCategory = "search"
|
||||
outputCategoryTest outputCategory = "test"
|
||||
outputCategoryProcess outputCategory = "process"
|
||||
outputCategoryDiff outputCategory = "diff"
|
||||
outputCategoryWorker outputCategory = "worker"
|
||||
)
|
||||
|
||||
// outputBudget combines a provider-neutral estimated-token target with the
|
||||
// existing byte ceiling. The byte ceiling is always authoritative: semantic
|
||||
// retention may use fewer bytes, but can never use more.
|
||||
type outputBudget struct {
|
||||
maxEstimatedTokens int
|
||||
hardMaxBytes int
|
||||
}
|
||||
|
||||
// budgetedOutput describes the text retained by a semantic output policy. The
|
||||
// original size is the output received by this layer; it does not claim to be
|
||||
// every byte an underlying subprocess may have produced before its own capture
|
||||
// limits were applied.
|
||||
type budgetedOutput struct {
|
||||
text string
|
||||
originalBytes int
|
||||
retainedBytes int
|
||||
estimatedOriginalTokens int
|
||||
estimatedRetainedTokens int
|
||||
truncated bool
|
||||
category outputCategory
|
||||
reason string
|
||||
spillPath string
|
||||
}
|
||||
|
||||
// outputPolicyProvider optionally assigns a semantic category to a tool call.
|
||||
// Tools that do not implement it use outputCategoryDefault.
|
||||
type outputPolicyProvider interface {
|
||||
outputCategory(args map[string]any) outputCategory
|
||||
}
|
||||
|
||||
// estimateOutputTokens is a deterministic, provider-neutral estimate used only
|
||||
// for output budgeting. It is not exact provider tokenization. ASCII non-space
|
||||
// text uses the repository's established four-bytes-per-token approximation;
|
||||
// each byte of non-ASCII UTF-8 is counted as a token, intentionally
|
||||
// overestimating multilingual text and emoji rather than letting them bypass a
|
||||
// budget. Existing hard byte ceilings remain the final safety limit.
|
||||
func estimateOutputTokens(value string) int {
|
||||
asciiNonSpace, nonASCIIBytes := outputTokenComponents(value)
|
||||
return (asciiNonSpace+3)/4 + nonASCIIBytes
|
||||
}
|
||||
|
||||
func outputTokenComponents(value string) (asciiNonSpace int, nonASCIIBytes int) {
|
||||
for index := 0; index < len(value); {
|
||||
r, size := utf8.DecodeRuneInString(value[index:])
|
||||
if r == utf8.RuneError && size == 1 {
|
||||
// Invalid input is charged conservatively one token per byte. Budget
|
||||
// slicing itself remains rune-safe for valid UTF-8 tool output.
|
||||
nonASCIIBytes++
|
||||
index++
|
||||
continue
|
||||
}
|
||||
if r < utf8.RuneSelf {
|
||||
switch r {
|
||||
case ' ', '\t', '\n', '\r', '\f', '\v':
|
||||
default:
|
||||
asciiNonSpace++
|
||||
}
|
||||
} else {
|
||||
nonASCIIBytes += size
|
||||
}
|
||||
index += size
|
||||
}
|
||||
return asciiNonSpace, nonASCIIBytes
|
||||
}
|
||||
|
||||
const defaultSemanticTruncationNotice = "\n[zero] output truncated\n"
|
||||
|
||||
// budgetDefaultOutput applies the safe fallback policy: retain a rune-safe head
|
||||
// and tail around a stable truncation notice. Small output is returned
|
||||
// byte-identically. Spill creation is intentionally handled by the shared
|
||||
// boundary integration so every semantic policy uses the existing spill path.
|
||||
func budgetDefaultOutput(output string, budget outputBudget) budgetedOutput {
|
||||
result := budgetedOutput{
|
||||
text: output,
|
||||
originalBytes: len(output),
|
||||
retainedBytes: len(output),
|
||||
estimatedOriginalTokens: estimateOutputTokens(output),
|
||||
estimatedRetainedTokens: estimateOutputTokens(output),
|
||||
category: outputCategoryDefault,
|
||||
}
|
||||
if fitsOutputBudget(output, budget) {
|
||||
return result
|
||||
}
|
||||
|
||||
reason := outputBudgetReason(result.originalBytes, result.estimatedOriginalTokens, budget)
|
||||
maxContentBytes := len(output)
|
||||
if budget.hardMaxBytes > 0 {
|
||||
maxContentBytes = min(maxContentBytes, max(0, budget.hardMaxBytes-len(defaultSemanticTruncationNotice)))
|
||||
}
|
||||
|
||||
// Find the largest deterministic head+tail window that satisfies both the
|
||||
// estimate and the hard byte ceiling. fitsOutputBudget is monotonic as this
|
||||
// window grows, so binary search avoids repeatedly trimming one rune at a time.
|
||||
low, high := 0, maxContentBytes
|
||||
best := ""
|
||||
if fitsOutputBudget(defaultSemanticTruncationNotice, budget) {
|
||||
best = defaultSemanticTruncationNotice
|
||||
}
|
||||
for low <= high {
|
||||
window := low + (high-low)/2
|
||||
candidate := defaultHeadTail(output, window)
|
||||
if fitsOutputBudget(candidate, budget) {
|
||||
best = candidate
|
||||
low = window + 1
|
||||
} else {
|
||||
high = window - 1
|
||||
}
|
||||
}
|
||||
|
||||
result.text = best
|
||||
result.retainedBytes = len(best)
|
||||
result.estimatedRetainedTokens = estimateOutputTokens(best)
|
||||
result.truncated = true
|
||||
result.reason = reason
|
||||
return result
|
||||
}
|
||||
|
||||
func defaultHeadTail(output string, contentBytes int) string {
|
||||
if contentBytes <= 0 {
|
||||
return defaultSemanticTruncationNotice
|
||||
}
|
||||
headBytes := contentBytes / 2
|
||||
tailBytes := contentBytes - headBytes
|
||||
return utf8Prefix(output, headBytes) + defaultSemanticTruncationNotice + utf8Suffix(output, tailBytes)
|
||||
}
|
||||
|
||||
func fitsOutputBudget(output string, budget outputBudget) bool {
|
||||
if budget.hardMaxBytes > 0 && len(output) > budget.hardMaxBytes {
|
||||
return false
|
||||
}
|
||||
return budget.maxEstimatedTokens <= 0 || estimateOutputTokens(output) <= budget.maxEstimatedTokens
|
||||
}
|
||||
|
||||
func outputBudgetReason(originalBytes int, estimatedTokens int, budget outputBudget) string {
|
||||
overTokens := budget.maxEstimatedTokens > 0 && estimatedTokens > budget.maxEstimatedTokens
|
||||
overBytes := budget.hardMaxBytes > 0 && originalBytes > budget.hardMaxBytes
|
||||
switch {
|
||||
case overTokens && overBytes:
|
||||
return "token_and_byte_budget"
|
||||
case overTokens:
|
||||
return "estimated_token_budget"
|
||||
default:
|
||||
return "hard_byte_ceiling"
|
||||
}
|
||||
}
|
||||
|
||||
type outputBudgetResult struct {
|
||||
Output string
|
||||
Truncated bool
|
||||
@@ -56,6 +218,7 @@ func estimatedTokensFromBytes(bytes int) int {
|
||||
|
||||
type outputBudgetBuilder struct {
|
||||
builder strings.Builder
|
||||
capture *boundedBuffer
|
||||
rawBytes int
|
||||
maxBytes int
|
||||
hint string
|
||||
@@ -65,17 +228,60 @@ func newOutputBudgetBuilder(maxBytes int, hint string) *outputBudgetBuilder {
|
||||
return &outputBudgetBuilder{maxBytes: maxBytes, hint: hint}
|
||||
}
|
||||
|
||||
// newHeadTailOutputBudgetBuilder bounds capture while retaining both ends of
|
||||
// an input for a later semantic policy. It is used for registry-driven file
|
||||
// reads; direct Tool.Run calls keep the legacy prefix-only builder above.
|
||||
func newHeadTailOutputBudgetBuilder(maxBytes int, hint string) *outputBudgetBuilder {
|
||||
headBytes := maxBytes / 2
|
||||
return &outputBudgetBuilder{
|
||||
capture: newBoundedBuffer(headBytes, maxBytes-headBytes),
|
||||
maxBytes: maxBytes,
|
||||
hint: hint,
|
||||
}
|
||||
}
|
||||
|
||||
func (builder *outputBudgetBuilder) WriteString(value string) {
|
||||
builder.rawBytes += len(value)
|
||||
if builder.maxBytes <= 0 || builder.builder.Len() >= builder.maxBytes {
|
||||
if builder.capture != nil {
|
||||
_, _ = builder.capture.Write([]byte(value))
|
||||
return
|
||||
}
|
||||
if builder.maxBytes <= 0 {
|
||||
builder.builder.WriteString(value)
|
||||
return
|
||||
}
|
||||
if builder.builder.Len() >= builder.maxBytes {
|
||||
return
|
||||
}
|
||||
remaining := builder.maxBytes - builder.builder.Len()
|
||||
builder.builder.WriteString(utf8Prefix(value, remaining))
|
||||
}
|
||||
|
||||
// applyLegacyByteBudgetToResult preserves direct Tool.Run behavior for callers
|
||||
// that intentionally bypass Registry.RunWithOptions. Agent/MCP execution uses
|
||||
// the registry's post-redaction semantic boundary instead.
|
||||
func applyLegacyByteBudgetToResult(result Result, maxBytes int, hint string) Result {
|
||||
budgeted := applyOutputBudget(result.Output, maxBytes, hint)
|
||||
result.Output = budgeted.Output
|
||||
result.Truncated = result.Truncated || budgeted.Truncated
|
||||
if result.Meta == nil {
|
||||
result.Meta = map[string]string{}
|
||||
}
|
||||
for key, value := range outputBudgetMeta(budgeted) {
|
||||
result.Meta[key] = value
|
||||
}
|
||||
if budgeted.Truncated {
|
||||
result.Meta["truncated"] = "true"
|
||||
result.Meta["truncation_reason"] = "byte_budget"
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func (builder *outputBudgetBuilder) Result() outputBudgetResult {
|
||||
output := builder.builder.String()
|
||||
if builder.capture != nil {
|
||||
output = builder.capture.retained()
|
||||
}
|
||||
result := outputBudgetResult{
|
||||
Output: output,
|
||||
RawBytes: builder.rawBytes,
|
||||
@@ -90,7 +296,12 @@ func (builder *outputBudgetBuilder) Result() outputBudgetResult {
|
||||
if budget < 0 {
|
||||
budget = 0
|
||||
}
|
||||
result.Output = utf8Prefix(output, budget) + marker
|
||||
if builder.capture != nil {
|
||||
headBytes := budget / 2
|
||||
result.Output = utf8Prefix(output, headBytes) + marker + utf8Suffix(output, budget-headBytes)
|
||||
} else {
|
||||
result.Output = utf8Prefix(output, budget) + marker
|
||||
}
|
||||
result.Truncated = true
|
||||
result.EmittedBytes = len(result.Output)
|
||||
return result
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
package tools
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
func TestEstimateOutputTokensASCIIAndUnicode(t *testing.T) {
|
||||
if got := estimateOutputTokens("abcd efgh"); got != 2 {
|
||||
t.Fatalf("ASCII estimate = %d, want 2", got)
|
||||
}
|
||||
unicodeText := "🙂界"
|
||||
if got := estimateOutputTokens(unicodeText); got < len([]byte(unicodeText)) {
|
||||
t.Fatalf("Unicode estimate = %d, want conservative estimate >= %d", got, len([]byte(unicodeText)))
|
||||
}
|
||||
if first, second := estimateOutputTokens(unicodeText), estimateOutputTokens(unicodeText); first != second {
|
||||
t.Fatalf("estimator is not deterministic: %d != %d", first, second)
|
||||
}
|
||||
if input := strings.Repeat("\u0080", 4); estimateOutputTokens(input) < len([]byte(input)) {
|
||||
t.Fatalf("U+0080 estimate = %d, want conservative estimate >= %d", estimateOutputTokens(input), len([]byte(input)))
|
||||
}
|
||||
}
|
||||
|
||||
func TestBudgetDefaultOutputLeavesSmallOutputByteIdentical(t *testing.T) {
|
||||
input := "small output\nwith spacing\n"
|
||||
got := budgetDefaultOutput(input, outputBudget{maxEstimatedTokens: 100, hardMaxBytes: 1024})
|
||||
if got.text != input {
|
||||
t.Fatalf("small output changed: got %q want %q", got.text, input)
|
||||
}
|
||||
if got.truncated || got.originalBytes != len(input) || got.retainedBytes != len(input) {
|
||||
t.Fatalf("unexpected small-output metadata: %#v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBudgetDefaultOutputKeepsUTF8HeadTailWithinHardCeiling(t *testing.T) {
|
||||
input := "HEAD🙂\n" + strings.Repeat("界", 200) + "\nTAIL🙂"
|
||||
const hardMax = 120
|
||||
got := budgetDefaultOutput(input, outputBudget{maxEstimatedTokens: 10_000, hardMaxBytes: hardMax})
|
||||
if !got.truncated {
|
||||
t.Fatal("large output was not truncated")
|
||||
}
|
||||
if len(got.text) > hardMax {
|
||||
t.Fatalf("retained %d bytes, hard ceiling %d", len(got.text), hardMax)
|
||||
}
|
||||
if !utf8.ValidString(got.text) {
|
||||
t.Fatalf("budgeted output is invalid UTF-8: %q", got.text)
|
||||
}
|
||||
for _, want := range []string{"HEAD", "TAIL", "output truncated"} {
|
||||
if !strings.Contains(got.text, want) {
|
||||
t.Fatalf("budgeted output missing %q: %q", want, got.text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBudgetDefaultOutputHonorsEstimatedTokenBudget(t *testing.T) {
|
||||
input := strings.Repeat("🙂", 100)
|
||||
got := budgetDefaultOutput(input, outputBudget{maxEstimatedTokens: 60, hardMaxBytes: 1024})
|
||||
if !got.truncated || got.reason != "estimated_token_budget" {
|
||||
t.Fatalf("unexpected token-budget result: %#v", got)
|
||||
}
|
||||
if got.estimatedRetainedTokens > 60 {
|
||||
t.Fatalf("retained estimate = %d, budget 60", got.estimatedRetainedTokens)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBudgetDefaultOutputDeterministic(t *testing.T) {
|
||||
input := "head\n" + strings.Repeat("same line\n", 1000) + "tail\n"
|
||||
budget := outputBudget{maxEstimatedTokens: 80, hardMaxBytes: 512}
|
||||
want := budgetDefaultOutput(input, budget)
|
||||
for iteration := 0; iteration < 20; iteration++ {
|
||||
got := budgetDefaultOutput(input, budget)
|
||||
if got != want {
|
||||
t.Fatalf("iteration %d differs:\n got %#v\nwant %#v", iteration, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBudgetDefaultOutputTinyBudgetsNeverExceedTheirCeilings(t *testing.T) {
|
||||
input := strings.Repeat("oversized output ", 100)
|
||||
for _, budget := range []outputBudget{
|
||||
{hardMaxBytes: len(defaultSemanticTruncationNotice) - 1},
|
||||
{maxEstimatedTokens: estimateOutputTokens(defaultSemanticTruncationNotice) - 1},
|
||||
} {
|
||||
got := budgetDefaultOutput(input, budget)
|
||||
if !got.truncated || !fitsOutputBudget(got.text, budget) {
|
||||
t.Fatalf("tiny budget result does not fit: budget=%#v result=%#v", budget, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -25,23 +25,19 @@ const defaultOutputCeilingTokens = 16_000
|
||||
// disables the ceiling entirely; unset or unparsable keeps the default.
|
||||
const outputCeilingEnv = "ZERO_TOOL_OUTPUT_CEILING_TOKENS"
|
||||
|
||||
// selfBudgeting marks a tool that enforces its own deliberate output budget —
|
||||
// possibly model-raisable (exec_command) — which the registry ceiling must not
|
||||
// second-guess. The method is unexported on purpose: only tools in this
|
||||
// package can opt out, so an MCP-served tool can never exempt itself.
|
||||
// selfBudgeting marks a tool with a deliberate capture-aware output budget —
|
||||
// possibly model-raisable (exec_command). The registry applies semantic
|
||||
// retention within that explicit budget rather than replacing it with the
|
||||
// default ceiling. The method is unexported so an MCP-served tool cannot opt
|
||||
// into this path.
|
||||
type selfBudgeting interface{ managesOutputBudget() }
|
||||
|
||||
// The exemption list, kept in one place. Each of these applies its own budget
|
||||
// before returning: bash (bashOutputBudgetBytes per stream + spill),
|
||||
// exec_command (model-raisable token budget + spill), read tools (128 KiB),
|
||||
// search tools (64 KiB).
|
||||
func (bashTool) managesOutputBudget() {}
|
||||
func (execCommandTool) managesOutputBudget() {}
|
||||
func (readFileTool) managesOutputBudget() {}
|
||||
func (readMinifiedFileTool) managesOutputBudget() {}
|
||||
func (grepTool) managesOutputBudget() {}
|
||||
func (globTool) managesOutputBudget() {}
|
||||
func (listDirectoryTool) managesOutputBudget() {}
|
||||
// The list is kept in one place. Shell/process tools retain their established
|
||||
// capture-aware budgets: bash (per-stream bounded capture) and exec_command
|
||||
// (model-raisable bounded session output). File/search tools use the standard
|
||||
// shared post-redaction boundary.
|
||||
func (bashTool) managesOutputBudget() {}
|
||||
func (execCommandTool) managesOutputBudget() {}
|
||||
|
||||
func resolveOutputCeilingTokens() int {
|
||||
raw := strings.TrimSpace(os.Getenv(outputCeilingEnv))
|
||||
@@ -61,6 +57,12 @@ func resolveOutputCeilingTokens() int {
|
||||
// and the spill file agree on what was hidden.
|
||||
func enforceOutputCeiling(toolName string, result Result) Result {
|
||||
ceiling := resolveOutputCeilingTokens()
|
||||
switch toolName {
|
||||
case "read_file", "read_minified_file":
|
||||
ceiling = readOutputBudgetBytes / 4
|
||||
case "grep", "glob", "list_directory":
|
||||
ceiling = searchOutputBudgetBytes / 4
|
||||
}
|
||||
if ceiling <= 0 {
|
||||
return result
|
||||
}
|
||||
|
||||
@@ -55,10 +55,10 @@ func TestOutputCeilingCapsUnbudgetedTool(t *testing.T) {
|
||||
if result.Meta["raw_bytes"] != strconv.Itoa(len(big)) {
|
||||
t.Fatalf("raw_bytes = %s, want %d", result.Meta["raw_bytes"], len(big))
|
||||
}
|
||||
if !strings.Contains(result.Output, "full output saved to ") {
|
||||
if !strings.Contains(result.Output, "full output received by budgeting layer saved to ") {
|
||||
t.Fatalf("ceiling truncation must include a spill hint: %q", result.Output[:200])
|
||||
}
|
||||
start := strings.Index(result.Output, "full output saved to ") + len("full output saved to ")
|
||||
start := strings.Index(result.Output, "full output received by budgeting layer saved to ") + len("full output received by budgeting layer saved to ")
|
||||
end := strings.Index(result.Output[start:], " (grep")
|
||||
content, err := os.ReadFile(result.Output[start : start+end])
|
||||
if err != nil {
|
||||
@@ -123,11 +123,6 @@ func TestSelfBudgetingExemptionList(t *testing.T) {
|
||||
exempt := []Tool{
|
||||
NewBashTool(dir),
|
||||
NewExecCommandTool(dir, newExecSessionManager()),
|
||||
NewReadFileTool(dir),
|
||||
NewReadMinifiedFileTool(dir),
|
||||
NewGrepTool(dir),
|
||||
NewGlobTool(dir),
|
||||
NewListDirectoryTool(dir),
|
||||
}
|
||||
for _, tool := range exempt {
|
||||
if _, ok := tool.(selfBudgeting); !ok {
|
||||
@@ -137,4 +132,9 @@ func TestSelfBudgetingExemptionList(t *testing.T) {
|
||||
if _, ok := NewWebFetchTool().(selfBudgeting); ok {
|
||||
t.Error("web_fetch must NOT be exempt — the ceiling is its only budget")
|
||||
}
|
||||
for _, tool := range []Tool{NewReadFileTool(dir), NewReadMinifiedFileTool(dir), NewGrepTool(dir), NewGlobTool(dir), NewListDirectoryTool(dir)} {
|
||||
if _, ok := tool.(selfBudgeting); ok {
|
||||
t.Errorf("%s must use the shared post-redaction budget boundary", tool.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,447 @@
|
||||
package tools
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// budgetSemanticOutput applies the deterministic policy for category. The
|
||||
// caller supplies output after secret redaction. Small results pass through
|
||||
// byte-identically; only oversized results are semantically reduced.
|
||||
func budgetSemanticOutput(output string, category outputCategory, budget outputBudget) budgetedOutput {
|
||||
if category == "" {
|
||||
category = outputCategoryDefault
|
||||
}
|
||||
if fitsOutputBudget(output, budget) {
|
||||
return unchangedBudgetedOutput(output, category)
|
||||
}
|
||||
|
||||
var retained string
|
||||
switch category {
|
||||
case outputCategoryFile:
|
||||
retained = budgetFileLines(output, budget)
|
||||
case outputCategorySearch:
|
||||
retained = budgetSearchLines(output, budget)
|
||||
case outputCategoryTest:
|
||||
retained = budgetTestLines(output, budget)
|
||||
case outputCategoryProcess:
|
||||
retained = budgetProcessLines(output, budget)
|
||||
case outputCategoryDiff:
|
||||
retained = budgetDiffUnits(output, budget)
|
||||
case outputCategoryWorker:
|
||||
retained = budgetWorkerLines(output, budget)
|
||||
default:
|
||||
return budgetDefaultOutput(output, budget)
|
||||
}
|
||||
|
||||
if retained == "" || !fitsOutputBudget(retained, budget) {
|
||||
fallback := budgetDefaultOutput(output, budget)
|
||||
fallback.category = category
|
||||
return fallback
|
||||
}
|
||||
return truncatedBudgetedOutput(output, retained, category, "semantic_"+string(category)+"_budget")
|
||||
}
|
||||
|
||||
func unchangedBudgetedOutput(output string, category outputCategory) budgetedOutput {
|
||||
estimated := estimateOutputTokens(output)
|
||||
return budgetedOutput{
|
||||
text: output,
|
||||
originalBytes: len(output),
|
||||
retainedBytes: len(output),
|
||||
estimatedOriginalTokens: estimated,
|
||||
estimatedRetainedTokens: estimated,
|
||||
category: category,
|
||||
}
|
||||
}
|
||||
|
||||
func truncatedBudgetedOutput(original, retained string, category outputCategory, reason string) budgetedOutput {
|
||||
return budgetedOutput{
|
||||
text: retained,
|
||||
originalBytes: len(original),
|
||||
retainedBytes: len(retained),
|
||||
estimatedOriginalTokens: estimateOutputTokens(original),
|
||||
estimatedRetainedTokens: estimateOutputTokens(retained),
|
||||
truncated: true,
|
||||
category: category,
|
||||
reason: reason,
|
||||
}
|
||||
}
|
||||
|
||||
func budgetFileLines(output string, budget outputBudget) string {
|
||||
lines := outputLines(output)
|
||||
priorities := make([]int, 0, len(lines))
|
||||
// Keep the existing file/range header first, then alternate from the start
|
||||
// and end so both requested-location boundaries survive.
|
||||
if len(lines) > 0 {
|
||||
priorities = append(priorities, 0)
|
||||
}
|
||||
for left, right := 1, len(lines)-1; left <= right; left, right = left+1, right-1 {
|
||||
priorities = append(priorities, left)
|
||||
if right != left {
|
||||
priorities = append(priorities, right)
|
||||
}
|
||||
}
|
||||
units := lineDiffUnits(lines)
|
||||
selected := selectPrioritizedUnits(units, priorities, budget)
|
||||
for index := range selected {
|
||||
// Index zero is the file/range header. If no non-blank requested
|
||||
// content line fits as a complete line, let the common caller fall
|
||||
// back to its UTF-8-safe head/tail policy rather than returning only
|
||||
// a header and omission marker for a minified single-line file.
|
||||
if index > 0 && strings.TrimSpace(lines[index]) != "" {
|
||||
return renderSelectedUnits(units, selected)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func budgetSearchLines(output string, budget outputBudget) string {
|
||||
lines := collapseConsecutiveDuplicateLines(outputLines(output))
|
||||
priorities := make([]int, 0, len(lines))
|
||||
for index, line := range lines {
|
||||
lower := strings.ToLower(line)
|
||||
// A match body naturally contains the search term (often literally
|
||||
// "match"); treat only non-result lines as summaries so one busy file
|
||||
// cannot consume the budget before cross-file representatives are chosen.
|
||||
if searchResultFile(line) == "" && (strings.Contains(lower, "match") || strings.Contains(lower, "truncated") || strings.Contains(lower, "result")) {
|
||||
priorities = append(priorities, index)
|
||||
}
|
||||
}
|
||||
if len(lines) > 0 {
|
||||
priorities = append(priorities, 0, len(lines)-1)
|
||||
}
|
||||
seenFiles := map[string]bool{}
|
||||
for index, line := range lines {
|
||||
if file := searchResultFile(line); file != "" && !seenFiles[file] {
|
||||
seenFiles[file] = true
|
||||
priorities = append(priorities, index)
|
||||
}
|
||||
}
|
||||
priorities = append(priorities, sequence(len(lines))...)
|
||||
return retainPrioritizedLines(lines, priorities, budget)
|
||||
}
|
||||
|
||||
func budgetTestLines(output string, budget outputBudget) string {
|
||||
lines := collapseConsecutiveDuplicateLines(outputLines(output))
|
||||
priorities := make([]int, 0, len(lines))
|
||||
for index, line := range lines {
|
||||
if isTestFailureLine(line) {
|
||||
for contextIndex := max(0, index-2); contextIndex <= min(len(lines)-1, index+3); contextIndex++ {
|
||||
priorities = append(priorities, contextIndex)
|
||||
}
|
||||
}
|
||||
}
|
||||
// Final summaries and process status usually live at the tail.
|
||||
for index := max(0, len(lines)-16); index < len(lines); index++ {
|
||||
priorities = append(priorities, index)
|
||||
}
|
||||
for index := 0; index < min(8, len(lines)); index++ {
|
||||
priorities = append(priorities, index)
|
||||
}
|
||||
priorities = append(priorities, sequence(len(lines))...)
|
||||
return retainPrioritizedLines(lines, priorities, budget)
|
||||
}
|
||||
|
||||
func budgetProcessLines(output string, budget outputBudget) string {
|
||||
lines := collapseConsecutiveDuplicateLines(outputLines(output))
|
||||
priorities := make([]int, 0, len(lines))
|
||||
for index := 0; index < min(10, len(lines)); index++ {
|
||||
priorities = append(priorities, index)
|
||||
}
|
||||
seenDiagnostic := map[string]bool{}
|
||||
for index, line := range lines {
|
||||
lower := strings.ToLower(line)
|
||||
if containsAny(lower, "error", "warning", "warn:", "fatal", "panic", "failed", "denied") && !seenDiagnostic[line] {
|
||||
seenDiagnostic[line] = true
|
||||
priorities = append(priorities, index)
|
||||
}
|
||||
}
|
||||
for index := max(0, len(lines)-16); index < len(lines); index++ {
|
||||
priorities = append(priorities, index)
|
||||
}
|
||||
priorities = append(priorities, sequence(len(lines))...)
|
||||
return retainPrioritizedLines(lines, priorities, budget)
|
||||
}
|
||||
|
||||
func budgetWorkerLines(output string, budget outputBudget) string {
|
||||
lines := collapseConsecutiveDuplicateLines(outputLines(output))
|
||||
priorities := make([]int, 0, len(lines))
|
||||
for index, line := range lines {
|
||||
lower := strings.ToLower(line)
|
||||
if containsAny(lower, "status", "error", "failed", "failure", "session_id", "changed", "files", "tools executed", "conclusion", "result") {
|
||||
priorities = append(priorities, index)
|
||||
}
|
||||
}
|
||||
// A specialist's final conclusion is conventionally at the end.
|
||||
for index := max(0, len(lines)-20); index < len(lines); index++ {
|
||||
priorities = append(priorities, index)
|
||||
}
|
||||
for index := 0; index < min(6, len(lines)); index++ {
|
||||
priorities = append(priorities, index)
|
||||
}
|
||||
priorities = append(priorities, sequence(len(lines))...)
|
||||
return retainPrioritizedLines(lines, priorities, budget)
|
||||
}
|
||||
|
||||
type diffUnit struct {
|
||||
order int
|
||||
text string
|
||||
key bool
|
||||
}
|
||||
|
||||
func budgetDiffUnits(output string, budget outputBudget) string {
|
||||
units := splitDiffUnits(output)
|
||||
if len(units) == 0 {
|
||||
return ""
|
||||
}
|
||||
priority := make([]int, 0, len(units))
|
||||
for index, unit := range units {
|
||||
if unit.key {
|
||||
priority = append(priority, index)
|
||||
}
|
||||
}
|
||||
priority = append(priority, sequence(len(units))...)
|
||||
return retainPrioritizedUnits(units, priority, budget)
|
||||
}
|
||||
|
||||
// splitDiffUnits keeps each hunk indivisible. File headers/stat text are key
|
||||
// units so broad file coverage is selected before secondary hunks.
|
||||
func splitDiffUnits(output string) []diffUnit {
|
||||
lines := outputLines(output)
|
||||
units := make([]diffUnit, 0)
|
||||
var current []string
|
||||
currentKey := false
|
||||
flush := func() {
|
||||
if len(current) == 0 {
|
||||
return
|
||||
}
|
||||
units = append(units, diffUnit{order: len(units), text: strings.Join(current, "\n"), key: currentKey})
|
||||
current = nil
|
||||
currentKey = false
|
||||
}
|
||||
for _, line := range lines {
|
||||
if strings.HasPrefix(line, "diff --git ") {
|
||||
flush()
|
||||
currentKey = true
|
||||
current = append(current, line)
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(line, "@@ ") {
|
||||
flush()
|
||||
current = append(current, line)
|
||||
continue
|
||||
}
|
||||
if len(current) == 0 {
|
||||
currentKey = true // diff stat/summary or standalone file headers
|
||||
}
|
||||
current = append(current, line)
|
||||
}
|
||||
flush()
|
||||
return units
|
||||
}
|
||||
|
||||
func retainPrioritizedLines(lines []string, priorities []int, budget outputBudget) string {
|
||||
return retainPrioritizedUnits(lineDiffUnits(lines), priorities, budget)
|
||||
}
|
||||
|
||||
func lineDiffUnits(lines []string) []diffUnit {
|
||||
units := make([]diffUnit, 0, len(lines))
|
||||
for index, line := range lines {
|
||||
units = append(units, diffUnit{order: index, text: line})
|
||||
}
|
||||
return units
|
||||
}
|
||||
|
||||
func retainPrioritizedUnits(units []diffUnit, priorities []int, budget outputBudget) string {
|
||||
return renderSelectedUnits(units, selectPrioritizedUnits(units, priorities, budget))
|
||||
}
|
||||
|
||||
func selectPrioritizedUnits(units []diffUnit, priorities []int, budget outputBudget) map[int]bool {
|
||||
selected := map[int]bool{}
|
||||
indexes := make([]int, 0, len(priorities))
|
||||
cost := retainedUnitCost{}
|
||||
for _, index := range priorities {
|
||||
if index < 0 || index >= len(units) || selected[index] {
|
||||
continue
|
||||
}
|
||||
position := sort.SearchInts(indexes, index)
|
||||
previous, next := -1, len(units)
|
||||
if position > 0 {
|
||||
previous = indexes[position-1]
|
||||
}
|
||||
if position < len(indexes) {
|
||||
next = indexes[position]
|
||||
}
|
||||
|
||||
candidate := cost
|
||||
candidate.add(units[index].text)
|
||||
if len(indexes) == 0 {
|
||||
candidate.addOmission(index)
|
||||
candidate.addOmission(len(units) - index - 1)
|
||||
} else {
|
||||
candidate.removeOmission(next - previous - 1)
|
||||
candidate.addOmission(index - previous - 1)
|
||||
candidate.addOmission(next - index - 1)
|
||||
}
|
||||
if !candidate.fits(budget) {
|
||||
continue
|
||||
}
|
||||
selected[index] = true
|
||||
indexes = append(indexes, 0)
|
||||
copy(indexes[position+1:], indexes[position:])
|
||||
indexes[position] = index
|
||||
cost = candidate
|
||||
}
|
||||
return selected
|
||||
}
|
||||
|
||||
// retainedUnitCost tracks the exact rendered size of selected units without
|
||||
// repeatedly rebuilding the full candidate text. Newlines between rendered
|
||||
// parts add bytes but no estimated tokens, so the token components remain
|
||||
// additive even as priorities select units out of source order.
|
||||
type retainedUnitCost struct {
|
||||
textBytes int
|
||||
asciiNonSpace int
|
||||
nonASCIIBytes int
|
||||
parts int
|
||||
}
|
||||
|
||||
func (cost *retainedUnitCost) add(text string) {
|
||||
ascii, nonASCII := outputTokenComponents(text)
|
||||
cost.textBytes += len(text)
|
||||
cost.asciiNonSpace += ascii
|
||||
cost.nonASCIIBytes += nonASCII
|
||||
cost.parts++
|
||||
}
|
||||
|
||||
func (cost *retainedUnitCost) remove(text string) {
|
||||
ascii, nonASCII := outputTokenComponents(text)
|
||||
cost.textBytes -= len(text)
|
||||
cost.asciiNonSpace -= ascii
|
||||
cost.nonASCIIBytes -= nonASCII
|
||||
cost.parts--
|
||||
}
|
||||
|
||||
func (cost *retainedUnitCost) addOmission(count int) {
|
||||
if count > 0 {
|
||||
cost.add(omittedSectionsMarker(count))
|
||||
}
|
||||
}
|
||||
|
||||
func (cost *retainedUnitCost) removeOmission(count int) {
|
||||
if count > 0 {
|
||||
cost.remove(omittedSectionsMarker(count))
|
||||
}
|
||||
}
|
||||
|
||||
func (cost retainedUnitCost) fits(budget outputBudget) bool {
|
||||
bytes := cost.textBytes + max(0, cost.parts-1)
|
||||
tokens := (cost.asciiNonSpace+3)/4 + cost.nonASCIIBytes
|
||||
return (budget.hardMaxBytes <= 0 || bytes <= budget.hardMaxBytes) &&
|
||||
(budget.maxEstimatedTokens <= 0 || tokens <= budget.maxEstimatedTokens)
|
||||
}
|
||||
|
||||
func omittedSectionsMarker(count int) string {
|
||||
return fmt.Sprintf("[zero] ... %d section(s) omitted ...", count)
|
||||
}
|
||||
|
||||
func renderSelectedUnits(units []diffUnit, selected map[int]bool) string {
|
||||
indexes := make([]int, 0, len(selected))
|
||||
for index := range selected {
|
||||
indexes = append(indexes, index)
|
||||
}
|
||||
sort.Ints(indexes)
|
||||
if len(indexes) == 0 {
|
||||
return ""
|
||||
}
|
||||
parts := make([]string, 0, len(indexes)*2)
|
||||
previous := -1
|
||||
for _, index := range indexes {
|
||||
if previous >= 0 && index != previous+1 {
|
||||
omitted := index - previous - 1
|
||||
parts = append(parts, omittedSectionsMarker(omitted))
|
||||
} else if previous < 0 && index > 0 {
|
||||
parts = append(parts, omittedSectionsMarker(index))
|
||||
}
|
||||
parts = append(parts, units[index].text)
|
||||
previous = index
|
||||
}
|
||||
if previous < len(units)-1 {
|
||||
parts = append(parts, omittedSectionsMarker(len(units)-previous-1))
|
||||
}
|
||||
return strings.Join(parts, "\n")
|
||||
}
|
||||
|
||||
func outputLines(output string) []string {
|
||||
return strings.Split(strings.TrimRight(output, "\r\n"), "\n")
|
||||
}
|
||||
|
||||
func collapseConsecutiveDuplicateLines(lines []string) []string {
|
||||
if len(lines) < 2 {
|
||||
return lines
|
||||
}
|
||||
result := make([]string, 0, len(lines))
|
||||
for index := 0; index < len(lines); {
|
||||
end := index + 1
|
||||
for end < len(lines) && lines[end] == lines[index] {
|
||||
end++
|
||||
}
|
||||
result = append(result, lines[index])
|
||||
if count := end - index; count > 1 {
|
||||
result = append(result, fmt.Sprintf("[zero] previous line repeated %d more time(s)", count-1))
|
||||
}
|
||||
index = end
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func searchResultFile(line string) string {
|
||||
// Search output is path:line: text. Locate the first numeric field from
|
||||
// the left after an optional Windows drive prefix. Match text may itself
|
||||
// contain numeric colon-delimited fields, which are not record structure.
|
||||
searchFrom := 0
|
||||
if len(line) >= 2 && line[1] == ':' && ((line[0] >= 'A' && line[0] <= 'Z') || (line[0] >= 'a' && line[0] <= 'z')) {
|
||||
searchFrom = 2
|
||||
}
|
||||
for searchFrom < len(line) {
|
||||
lineStartOffset := strings.IndexByte(line[searchFrom:], ':')
|
||||
if lineStartOffset < 0 {
|
||||
return ""
|
||||
}
|
||||
lineStart := searchFrom + lineStartOffset
|
||||
lineEndOffset := strings.IndexByte(line[lineStart+1:], ':')
|
||||
if lineEndOffset < 0 {
|
||||
return ""
|
||||
}
|
||||
lineEnd := lineStart + 1 + lineEndOffset
|
||||
if _, err := strconv.Atoi(line[lineStart+1 : lineEnd]); err == nil {
|
||||
return strings.TrimSpace(line[:lineStart])
|
||||
}
|
||||
searchFrom = lineStart + 1
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func isTestFailureLine(line string) bool {
|
||||
lower := strings.ToLower(line)
|
||||
return containsAny(lower, "--- fail:", "failed", "failure", "panic", "assert", "error", "fatal", "expected", "actual:")
|
||||
}
|
||||
|
||||
func containsAny(value string, needles ...string) bool {
|
||||
for _, needle := range needles {
|
||||
if strings.Contains(value, needle) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func sequence(length int) []int {
|
||||
result := make([]int, length)
|
||||
for index := range result {
|
||||
result[index] = index
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
package tools
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
func semanticTestBudget() outputBudget {
|
||||
return outputBudget{maxEstimatedTokens: 140, hardMaxBytes: 900}
|
||||
}
|
||||
|
||||
func TestFileOutputPolicyKeepsCompleteHeadAndTailLines(t *testing.T) {
|
||||
lines := []string{"File: huge.go (500 lines)", "1 | package main"}
|
||||
for index := 2; index < 500; index++ {
|
||||
lines = append(lines, fmt.Sprintf("%d | value := %d", index, index))
|
||||
}
|
||||
lines = append(lines, "500 | // END_MARK")
|
||||
got := budgetSemanticOutput(strings.Join(lines, "\n"), outputCategoryFile, semanticTestBudget())
|
||||
if !got.truncated || !strings.Contains(got.text, "File: huge.go") || !strings.Contains(got.text, "END_MARK") {
|
||||
t.Fatalf("file policy lost boundaries: %#v", got)
|
||||
}
|
||||
if !utf8.ValidString(got.text) || strings.Contains(got.text, "value := 49\ufffd") {
|
||||
t.Fatalf("file policy produced invalid or split text: %q", got.text)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSearchOutputPolicyPreservesMultiFileCoverage(t *testing.T) {
|
||||
var lines []string
|
||||
for _, file := range []string{"a.go", "b.go", "c.go", "d.go"} {
|
||||
for line := 1; line <= 30; line++ {
|
||||
lines = append(lines, fmt.Sprintf("%s:%d: match value %d", file, line, line))
|
||||
}
|
||||
}
|
||||
lines = append(lines, "120 matches found")
|
||||
got := budgetSemanticOutput(strings.Join(lines, "\n"), outputCategorySearch, semanticTestBudget())
|
||||
for _, want := range []string{"a.go:", "b.go:", "c.go:", "d.go:", "120 matches"} {
|
||||
if !strings.Contains(got.text, want) {
|
||||
t.Fatalf("search policy missing %q:\n%s", want, got.text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSearchOutputPolicyPreservesWindowsPathCoverage(t *testing.T) {
|
||||
var lines []string
|
||||
for _, file := range []string{"a.go", "b.go", "c.go", "d.go"} {
|
||||
for line := 1; line <= 30; line++ {
|
||||
lines = append(lines, fmt.Sprintf(`C:\workspace\%s:%d: match value %d`, file, line, line))
|
||||
}
|
||||
}
|
||||
lines = append(lines, "120 matches found")
|
||||
got := budgetSemanticOutput(strings.Join(lines, "\n"), outputCategorySearch, semanticTestBudget())
|
||||
for _, want := range []string{`C:\workspace\a.go:`, `C:\workspace\b.go:`, `C:\workspace\c.go:`, `C:\workspace\d.go:`, "120 matches"} {
|
||||
if !strings.Contains(got.text, want) {
|
||||
t.Fatalf("search policy missing %q:\n%s", want, got.text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSearchResultFileIgnoresNumericMatchContent(t *testing.T) {
|
||||
tests := []struct {
|
||||
line string
|
||||
want string
|
||||
}{
|
||||
{line: `C:\workspace\a.go:12: error code:123: failed`, want: `C:\workspace\a.go`},
|
||||
{line: `/workspace/a.go:12: error code:123: failed`, want: `/workspace/a.go`},
|
||||
{line: `/workspace/name:with-colon.go:12: match`, want: `/workspace/name:with-colon.go`},
|
||||
}
|
||||
for _, test := range tests {
|
||||
if got := searchResultFile(test.line); got != test.want {
|
||||
t.Errorf("searchResultFile(%q) = %q, want %q", test.line, got, test.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessOutputPolicyCollapsesRepetitiveLogsAndKeepsDiagnostics(t *testing.T) {
|
||||
input := "starting server\n" + strings.Repeat("polling...\n", 300) + "WARNING: queue slow\nERROR: request failed\nshutdown complete"
|
||||
got := budgetSemanticOutput(input, outputCategoryProcess, semanticTestBudget())
|
||||
for _, want := range []string{"starting server", "repeated", "WARNING", "ERROR", "shutdown complete"} {
|
||||
if !strings.Contains(got.text, want) {
|
||||
t.Fatalf("process policy missing %q:\n%s", want, got.text)
|
||||
}
|
||||
}
|
||||
if strings.Count(got.text, "polling...") > 1 {
|
||||
t.Fatalf("repetitive line was not collapsed:\n%s", got.text)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTestOutputPolicyKeepsFailureAndFinalSummary(t *testing.T) {
|
||||
input := "=== RUN TestSuite\n" + strings.Repeat("ok progress\n", 300) +
|
||||
"--- FAIL: TestImportant (0.01s)\n thing_test.go:42: expected 7, got 9\nFAIL\nexit status 1"
|
||||
got := budgetSemanticOutput(input, outputCategoryTest, semanticTestBudget())
|
||||
for _, want := range []string{"TestImportant", "expected 7", "FAIL", "exit status 1"} {
|
||||
if !strings.Contains(got.text, want) {
|
||||
t.Fatalf("test policy missing %q:\n%s", want, got.text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiffOutputPolicyKeepsFilesAndCompleteHunks(t *testing.T) {
|
||||
var diff strings.Builder
|
||||
for file := 1; file <= 4; file++ {
|
||||
fmt.Fprintf(&diff, "diff --git a/f%d.go b/f%d.go\n--- a/f%d.go\n+++ b/f%d.go\n", file, file, file, file)
|
||||
for hunk := 1; hunk <= 8; hunk++ {
|
||||
fmt.Fprintf(&diff, "@@ -%d,2 +%d,2 @@\n-old%d_%d\n+new%d_%d\n", hunk, hunk, file, hunk, file, hunk)
|
||||
}
|
||||
}
|
||||
got := budgetSemanticOutput(diff.String(), outputCategoryDiff, outputBudget{maxEstimatedTokens: 220, hardMaxBytes: 1400})
|
||||
for file := 1; file <= 4; file++ {
|
||||
if !strings.Contains(got.text, fmt.Sprintf("diff --git a/f%d.go", file)) {
|
||||
t.Fatalf("diff policy lost file %d header:\n%s", file, got.text)
|
||||
}
|
||||
}
|
||||
for _, line := range strings.Split(got.text, "\n") {
|
||||
if strings.HasPrefix(line, "@@ ") {
|
||||
// Each retained hunk is an indivisible three-line unit.
|
||||
if !strings.Contains(got.text, line+"\n-old") {
|
||||
t.Fatalf("hunk header was sliced from its body: %q", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkerOutputPolicyKeepsStatusErrorsAndConclusion(t *testing.T) {
|
||||
input := "session_id: child_1\nstatus: error\n" + strings.Repeat("worker progress\n", 200) +
|
||||
"tools executed: grep, read_file\nchanged files: a.go, b.go\nerror: verification failed\nConclusion: fix the nil check before merging."
|
||||
got := budgetSemanticOutput(input, outputCategoryWorker, semanticTestBudget())
|
||||
for _, want := range []string{"session_id", "status: error", "tools executed", "changed files", "verification failed", "Conclusion"} {
|
||||
if !strings.Contains(got.text, want) {
|
||||
t.Fatalf("worker policy missing %q:\n%s", want, got.text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSemanticPoliciesAreDeterministic(t *testing.T) {
|
||||
input := "start\n" + strings.Repeat("tick\n", 500) + "ERROR: boom\nend"
|
||||
want := budgetSemanticOutput(input, outputCategoryProcess, semanticTestBudget())
|
||||
for iteration := 0; iteration < 20; iteration++ {
|
||||
if got := budgetSemanticOutput(input, outputCategoryProcess, semanticTestBudget()); got != want {
|
||||
t.Fatalf("iteration %d differs:\n got %#v\nwant %#v", iteration, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -18,6 +18,8 @@ type readFileTool struct {
|
||||
scope PathScope
|
||||
}
|
||||
|
||||
func (readFileTool) outputCategory(map[string]any) outputCategory { return outputCategoryFile }
|
||||
|
||||
func NewReadFileTool(workspaceRoot string) Tool {
|
||||
return NewScopedReadFileTool(workspaceRoot, nil)
|
||||
}
|
||||
@@ -39,8 +41,10 @@ func NewScopedReadFileTool(workspaceRoot string, scope PathScope) Tool {
|
||||
AdditionalProperties: false,
|
||||
},
|
||||
safety: readOnlySafety("Reads file contents without modifying files."),
|
||||
// ThreadSafe=false: may update FileTracker session state on read.
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: fileResourceKeys},
|
||||
// ThreadSafe: FileTracker is mutex-guarded; concurrent reads of
|
||||
// distinct paths are safe. Same-path calls still serialize via
|
||||
// resource-key conflict detection in the agent parallel planner.
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: fileResourceKeys},
|
||||
},
|
||||
workspaceRoot: normalizeWorkspaceRoot(workspaceRoot),
|
||||
scope: scope,
|
||||
@@ -48,10 +52,14 @@ func NewScopedReadFileTool(workspaceRoot string, scope PathScope) Tool {
|
||||
}
|
||||
|
||||
func (tool readFileTool) Run(ctx context.Context, args map[string]any) Result {
|
||||
return tool.RunWithOptions(ctx, args, RunOptions{})
|
||||
return tool.run(args, RunOptions{}, true)
|
||||
}
|
||||
|
||||
func (tool readFileTool) RunWithOptions(_ context.Context, args map[string]any, options RunOptions) Result {
|
||||
return tool.run(args, options, false)
|
||||
}
|
||||
|
||||
func (tool readFileTool) run(args map[string]any, options RunOptions, directBudget bool) Result {
|
||||
requestedPath, err := aliasedStringArg(args, []string{"path", "file", "file_path", "filepath", "filename"}, "", true, false)
|
||||
if err != nil {
|
||||
return errorResult("Error: Invalid arguments for read_file: " + err.Error())
|
||||
@@ -84,10 +92,14 @@ func (tool readFileTool) RunWithOptions(_ context.Context, args map[string]any,
|
||||
// not the authoritative content hash.
|
||||
options.FileTracker.RecordHash(absolutePath, stats.hash, stats.info)
|
||||
|
||||
return renderReadFileRange(absolutePath, relativePath, stats.lines, startLine, endLine, maxLines)
|
||||
maxBytes := 0
|
||||
if directBudget {
|
||||
maxBytes = readOutputBudgetBytes
|
||||
}
|
||||
return renderReadFileRange(absolutePath, relativePath, stats.lines, startLine, endLine, maxLines, maxBytes)
|
||||
}
|
||||
|
||||
func renderReadFileRange(absolutePath string, relativePath string, total int, startLine int, endLine int, maxLines int) Result {
|
||||
func renderReadFileRange(absolutePath string, relativePath string, total int, startLine int, endLine int, maxLines int, maxBytes int) Result {
|
||||
if startLine > total {
|
||||
return okResult(fmt.Sprintf("File: %s\n(start_line %d is past the end of the file, which has %d lines)", relativePath, startLine, total))
|
||||
}
|
||||
@@ -112,7 +124,16 @@ func renderReadFileRange(absolutePath string, relativePath string, total int, st
|
||||
header = fmt.Sprintf("File: %s (lines %d-%d of %d)", relativePath, startLine, lastLine, total)
|
||||
}
|
||||
|
||||
budgetedOutput := newOutputBudgetBuilder(readOutputBudgetBytes, "use start_line/end_line or max_lines to continue with a smaller range")
|
||||
// The shared registry boundary applies the file policy after redaction. Its
|
||||
// input still needs a bounded capture: retain the requested range's head and
|
||||
// tail instead of materializing an arbitrarily large file in memory. Direct
|
||||
// Tool.Run calls preserve the legacy prefix-only byte budget.
|
||||
var budgetedOutput *outputBudgetBuilder
|
||||
if maxBytes > 0 {
|
||||
budgetedOutput = newOutputBudgetBuilder(maxBytes, "use start_line/end_line or max_lines to continue with a smaller range")
|
||||
} else {
|
||||
budgetedOutput = newHeadTailOutputBudgetBuilder(readOutputBudgetBytes, "use start_line/end_line or max_lines to continue with a smaller range")
|
||||
}
|
||||
budgetedOutput.WriteString(header)
|
||||
budgetedOutput.WriteString("\n\n")
|
||||
if err := appendReadFileRange(budgetedOutput, absolutePath, startLine, selectedLines, width); err != nil {
|
||||
@@ -126,10 +147,19 @@ func renderReadFileRange(absolutePath string, relativePath string, total int, st
|
||||
}
|
||||
|
||||
budgeted := budgetedOutput.Result()
|
||||
meta := outputBudgetMeta(budgeted)
|
||||
if budgeted.Truncated {
|
||||
meta := map[string]string{}
|
||||
if maxBytes > 0 || budgeted.Truncated {
|
||||
for key, value := range outputBudgetMeta(budgeted) {
|
||||
meta[key] = value
|
||||
}
|
||||
}
|
||||
if truncated || budgeted.Truncated {
|
||||
meta["truncated"] = "true"
|
||||
meta["truncation_reason"] = "byte_budget"
|
||||
if budgeted.Truncated {
|
||||
meta["truncation_reason"] = "byte_budget"
|
||||
} else {
|
||||
meta["truncation_reason"] = "max_lines"
|
||||
}
|
||||
}
|
||||
return Result{
|
||||
Status: StatusOK,
|
||||
|
||||
@@ -16,6 +16,10 @@ type readMinifiedFileTool struct {
|
||||
scope PathScope
|
||||
}
|
||||
|
||||
func (readMinifiedFileTool) outputCategory(map[string]any) outputCategory {
|
||||
return outputCategoryFile
|
||||
}
|
||||
|
||||
func NewReadMinifiedFileTool(workspaceRoot string) Tool {
|
||||
return NewScopedReadMinifiedFileTool(workspaceRoot, nil)
|
||||
}
|
||||
@@ -34,7 +38,7 @@ func NewScopedReadMinifiedFileTool(workspaceRoot string, scope PathScope) Tool {
|
||||
AdditionalProperties: false,
|
||||
},
|
||||
safety: readOnlySafety("Reads a minified view of file contents without modifying files."),
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: fileResourceKeys},
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: fileResourceKeys},
|
||||
},
|
||||
workspaceRoot: normalizeWorkspaceRoot(workspaceRoot),
|
||||
scope: scope,
|
||||
@@ -42,10 +46,14 @@ func NewScopedReadMinifiedFileTool(workspaceRoot string, scope PathScope) Tool {
|
||||
}
|
||||
|
||||
func (tool readMinifiedFileTool) Run(ctx context.Context, args map[string]any) Result {
|
||||
return tool.RunWithOptions(ctx, args, RunOptions{})
|
||||
return tool.run(args, RunOptions{}, true)
|
||||
}
|
||||
|
||||
func (tool readMinifiedFileTool) RunWithOptions(_ context.Context, args map[string]any, options RunOptions) Result {
|
||||
return tool.run(args, options, false)
|
||||
}
|
||||
|
||||
func (tool readMinifiedFileTool) run(args map[string]any, options RunOptions, directBudget bool) Result {
|
||||
requestedPath, err := aliasedStringArg(args, []string{"path", "file", "file_path", "filepath", "filename"}, "", true, false)
|
||||
if err != nil {
|
||||
return errorResult("Error: Invalid arguments for read_minified_file: " + err.Error())
|
||||
@@ -92,23 +100,21 @@ func (tool readMinifiedFileTool) RunWithOptions(_ context.Context, args map[stri
|
||||
savedTokens = estimatedTokensFromBytes(savedBytes)
|
||||
}
|
||||
output := header + "\n\n" + result.Content
|
||||
budgeted := applyOutputBudget(output, readOutputBudgetBytes, "use read_file with start_line/end_line or max_lines for a smaller exact range")
|
||||
meta := outputBudgetMeta(budgeted)
|
||||
meta := map[string]string{}
|
||||
meta["path"] = relativePath
|
||||
meta["mode"] = result.Language
|
||||
meta["compacted"] = strconv.FormatBool(result.Applied)
|
||||
meta["raw_bytes"] = strconv.Itoa(rawBytes)
|
||||
meta["compact_bytes"] = strconv.Itoa(compactBytes)
|
||||
meta["emitted_bytes"] = strconv.Itoa(budgeted.EmittedBytes)
|
||||
meta["emitted_bytes"] = strconv.Itoa(len(output))
|
||||
meta["raw_lines"] = strconv.Itoa(rawLines)
|
||||
meta["emitted_lines"] = strconv.Itoa(minLines)
|
||||
meta["estimated_tokens_saved"] = strconv.Itoa(savedTokens)
|
||||
if budgeted.Truncated {
|
||||
meta["truncated"] = "true"
|
||||
meta["truncation_reason"] = "byte_budget"
|
||||
toolResult := Result{Status: StatusOK, Output: output, Meta: meta}
|
||||
if directBudget {
|
||||
return applyLegacyByteBudgetToResult(toolResult, readOutputBudgetBytes, "use read_file with start_line/end_line or max_lines for a smaller exact range")
|
||||
}
|
||||
|
||||
return Result{Status: StatusOK, Output: budgeted.Output, Truncated: budgeted.Truncated, Meta: meta}
|
||||
return toolResult
|
||||
}
|
||||
|
||||
// lineCount reports the number of newline-separated lines in s (an empty string
|
||||
|
||||
@@ -126,20 +126,25 @@ func (registry *Registry) RunWithOptions(ctx context.Context, name string, args
|
||||
// args/paths) are redacted at the boundary just like tool output. The output
|
||||
// ceiling runs after the scrub so the transcript and the spill file agree on
|
||||
// what was hidden.
|
||||
ceilingExempt := false
|
||||
selfManagedOutput := false
|
||||
var tool Tool
|
||||
var ok bool
|
||||
defer func() {
|
||||
result = scrubResultSecrets(result)
|
||||
if !ceilingExempt {
|
||||
if selfManagedOutput {
|
||||
result = applySelfManagedOutputBudget(tool, name, args, result)
|
||||
} else {
|
||||
result = applyRegistryOutputBudget(tool, name, args, result)
|
||||
result = enforceOutputCeiling(name, result)
|
||||
}
|
||||
}()
|
||||
|
||||
tool, ok := registry.Get(name)
|
||||
tool, ok = registry.Get(name)
|
||||
if !ok {
|
||||
return errorResult(`Error: Unknown tool "` + name + `".`)
|
||||
}
|
||||
if _, ok := tool.(selfBudgeting); ok {
|
||||
ceilingExempt = true
|
||||
selfManagedOutput = true
|
||||
}
|
||||
if rejecter, ok := tool.(PrePermissionRejecter); ok {
|
||||
if res, rejected := rejecter.RejectBeforePermission(args); rejected {
|
||||
|
||||
@@ -127,6 +127,20 @@ func workspaceResourceKeys(_ map[string]any) []string {
|
||||
return []string{ResourceKeyWorkspace + "root"}
|
||||
}
|
||||
|
||||
// scopedScanResourceKeys returns conflict keys for directory-scoped scan tools
|
||||
// (glob, grep). Workspace-wide scans (missing / empty / ".") return nil so
|
||||
// they do not force a false conflict with every other concurrent scan —
|
||||
// ThreadSafe is the concurrency gate; keys refine same-directory collisions.
|
||||
// A non-default path/cwd yields a single directory: key.
|
||||
func scopedScanResourceKeys(args map[string]any) []string {
|
||||
path := firstStringArg(args, "path", "cwd", "dir", "directory")
|
||||
normalized := NormalizeResourcePath(path)
|
||||
if normalized == "" || normalized == "." {
|
||||
return nil
|
||||
}
|
||||
return []string{ResourceKeyDirectory + normalized}
|
||||
}
|
||||
|
||||
// multiFileResourceKeys collects path and paths[] arguments into file: keys.
|
||||
func multiFileResourceKeys(args map[string]any) []string {
|
||||
var keys []string
|
||||
|
||||
@@ -49,7 +49,7 @@ func NewSkillTool(dir string) *skillTool {
|
||||
AdditionalProperties: false,
|
||||
},
|
||||
safety: readOnlySafety("Reads a local skill file; gathers reusable instructions only."),
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false},
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -141,8 +141,11 @@ func newWebFetchToolWithClientAndResolver(client *http.Client, resolver webFetch
|
||||
Reason: "Fetches remote URL content over the network.",
|
||||
AdvertiseInAuto: true,
|
||||
},
|
||||
// Network read; ThreadSafe=false until client proven concurrent-safe.
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: false, ResourceKeys: endpointResourceKeys},
|
||||
// Concurrent-safe: net/http.Client is documented as safe for
|
||||
// concurrent use; each call issues independent request I/O.
|
||||
// parallelSafeToolCall still requires PermissionAllow (prompted
|
||||
// network fetches stay sequential until auto-allowed).
|
||||
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: endpointResourceKeys},
|
||||
},
|
||||
client: client,
|
||||
resolver: resolver,
|
||||
|
||||
@@ -94,6 +94,48 @@ func WriteNDJSON(w io.Writer, t *TurnTrace) error {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// Prefix fingerprints are emitted after counters in insertion (turn)
|
||||
// order. The order is the order EmitPrefixHash was called, which is the
|
||||
// order the agent loop computed each turn's fingerprint, which is the
|
||||
// order a downstream consumer needs to correlate a prefix_hash event
|
||||
// with the cached_input_tokens counter for that turn. Sorting by
|
||||
// complete_prefix hash would destroy that correlation, so we do not
|
||||
// sort. The slice is already a deep copy from Finish (see
|
||||
// Recorder.Finish) so it is safe to range over without copying.
|
||||
for _, p := range t.PrefixHashes {
|
||||
if err := enc.Encode(map[string]any{
|
||||
"type": "prefix_hash",
|
||||
"base_instructions": p.BaseInstructionsHash,
|
||||
"confirmation_policy": p.ConfirmationPolicyHash,
|
||||
"project_context": p.ProjectContextHash,
|
||||
"skills": p.SkillsHash,
|
||||
"tools": p.ToolsHash,
|
||||
"schema": p.SchemaHash,
|
||||
"complete_prefix": p.CompletePrefixHash,
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
// Output budget events stay in tool-result emission order. Sorting them by
|
||||
// tool/category would destroy correlation with concurrent calls whose results
|
||||
// are deliberately consumed in original call order.
|
||||
for _, event := range t.OutputBudgets {
|
||||
if err := enc.Encode(map[string]any{
|
||||
"type": "output_budget",
|
||||
"tool": event.Tool,
|
||||
"category": event.Category,
|
||||
"original_bytes": event.OriginalBytes,
|
||||
"retained_bytes": event.RetainedBytes,
|
||||
"estimated_original_tokens": event.EstimatedOriginalTokens,
|
||||
"estimated_retained_tokens": event.EstimatedRetainedTokens,
|
||||
"truncated": event.Truncated,
|
||||
"reason": event.Reason,
|
||||
"spill_created": event.SpillCreated,
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -154,6 +196,15 @@ func WriteText(w io.Writer, t *TurnTrace) error {
|
||||
for _, c := range counters {
|
||||
write(" %-22s %d\n", c.Name, c.Value)
|
||||
}
|
||||
if len(t.OutputBudgets) > 0 {
|
||||
write("output budgets:\n")
|
||||
for _, event := range t.OutputBudgets {
|
||||
write(" tool=%s category=%s bytes=%d/%d tokens=%d/%d truncated=%t reason=%s spill=%t\n",
|
||||
event.Tool, event.Category, event.RetainedBytes, event.OriginalBytes,
|
||||
event.EstimatedRetainedTokens, event.EstimatedOriginalTokens,
|
||||
event.Truncated, event.Reason, event.SpillCreated)
|
||||
}
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
package trace
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestOutputBudgetTraceRoundTripContainsNoOutput(t *testing.T) {
|
||||
recorder := NewRecorder("session", "run", "")
|
||||
recorder.Start()
|
||||
event := OutputBudgetEvent{
|
||||
Tool: "grep",
|
||||
Category: "search",
|
||||
OriginalBytes: 10000,
|
||||
RetainedBytes: 1000,
|
||||
EstimatedOriginalTokens: 2500,
|
||||
EstimatedRetainedTokens: 250,
|
||||
Truncated: true,
|
||||
Reason: "semantic_search_budget",
|
||||
SpillCreated: true,
|
||||
}
|
||||
recorder.EmitOutputBudget(event)
|
||||
|
||||
var encoded bytes.Buffer
|
||||
if err := WriteNDJSON(&encoded, recorder.Finish()); err != nil {
|
||||
t.Fatalf("WriteNDJSON: %v", err)
|
||||
}
|
||||
const secretOutput = "secret output body"
|
||||
if strings.Contains(encoded.String(), secretOutput) {
|
||||
t.Fatal("output_budget trace contains raw secret output")
|
||||
}
|
||||
allowedKeys := map[string]bool{
|
||||
"type": true, "tool": true, "category": true, "original_bytes": true, "retained_bytes": true,
|
||||
"estimated_original_tokens": true, "estimated_retained_tokens": true, "truncated": true,
|
||||
"reason": true, "spill_created": true,
|
||||
}
|
||||
findUndocumentedKey := func(record map[string]any) string {
|
||||
for key := range record {
|
||||
if !allowedKeys[key] {
|
||||
return key
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
decoder := json.NewDecoder(strings.NewReader(encoded.String()))
|
||||
var outputBudgetRecord map[string]any
|
||||
for {
|
||||
var record map[string]any
|
||||
if err := decoder.Decode(&record); err != nil {
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
t.Fatalf("decode trace record: %v", err)
|
||||
}
|
||||
if record["type"] == "output_budget" {
|
||||
outputBudgetRecord = record
|
||||
break
|
||||
}
|
||||
}
|
||||
if outputBudgetRecord == nil {
|
||||
t.Fatal("missing output_budget trace record")
|
||||
}
|
||||
if key := findUndocumentedKey(outputBudgetRecord); key != "" {
|
||||
t.Fatalf("output_budget trace contains undocumented key %q", key)
|
||||
}
|
||||
for key := range allowedKeys {
|
||||
if _, exists := outputBudgetRecord[key]; !exists {
|
||||
t.Fatalf("output_budget trace missing documented key %q", key)
|
||||
}
|
||||
}
|
||||
recordJSON, err := json.Marshal(outputBudgetRecord)
|
||||
if err != nil {
|
||||
t.Fatalf("encode output_budget trace record: %v", err)
|
||||
}
|
||||
var serializedEvent OutputBudgetEvent
|
||||
if err := json.Unmarshal(recordJSON, &serializedEvent); err != nil {
|
||||
t.Fatalf("decode output_budget trace event: %v", err)
|
||||
}
|
||||
if serializedEvent != event {
|
||||
t.Fatalf("serialized output_budget event = %#v, want %#v", serializedEvent, event)
|
||||
}
|
||||
contaminated := make(map[string]any, len(outputBudgetRecord)+1)
|
||||
for key, value := range outputBudgetRecord {
|
||||
contaminated[key] = value
|
||||
}
|
||||
contaminated["output"] = secretOutput
|
||||
if key := findUndocumentedKey(contaminated); key == "" {
|
||||
t.Fatal("trace key validation would allow raw secret output")
|
||||
}
|
||||
parsed, err := ReadNDJSON(strings.NewReader(encoded.String()))
|
||||
if err != nil {
|
||||
t.Fatalf("ReadNDJSON: %v", err)
|
||||
}
|
||||
if len(parsed.OutputBudgets) != 1 || parsed.OutputBudgets[0] != event {
|
||||
t.Fatalf("unexpected round trip: %#v", parsed.OutputBudgets)
|
||||
}
|
||||
}
|
||||
+50
-2
@@ -97,6 +97,40 @@ func ReadNDJSON(r io.Reader) (*TurnTrace, error) {
|
||||
}
|
||||
name, _ := obj["name"].(string)
|
||||
t.Counters = append(t.Counters, Counter{Name: name, Value: parseInt64(obj["value"])})
|
||||
case "prefix_hash":
|
||||
if !sawTraceHeader {
|
||||
return nil, errors.New("parse trace: not a valid NDJSON trace (no type:trace header)")
|
||||
}
|
||||
// Round-trip the seven prefix-hash fields. Missing fields are
|
||||
// accepted as empty strings (a partially-written trace from a
|
||||
// crashed emitter must not fatal a parse). The decoder tolerates
|
||||
// any shape the encoder produced and the seven keys are the
|
||||
// contract — adding a new field is non-breaking; renaming one
|
||||
// requires a schema version bump.
|
||||
t.PrefixHashes = append(t.PrefixHashes, PrefixHash{
|
||||
BaseInstructionsHash: stringField(obj, "base_instructions"),
|
||||
ConfirmationPolicyHash: stringField(obj, "confirmation_policy"),
|
||||
ProjectContextHash: stringField(obj, "project_context"),
|
||||
SkillsHash: stringField(obj, "skills"),
|
||||
ToolsHash: stringField(obj, "tools"),
|
||||
SchemaHash: stringField(obj, "schema"),
|
||||
CompletePrefixHash: stringField(obj, "complete_prefix"),
|
||||
})
|
||||
case "output_budget":
|
||||
if !sawTraceHeader {
|
||||
return nil, errors.New("parse trace: not a valid NDJSON trace (no type:trace header)")
|
||||
}
|
||||
t.OutputBudgets = append(t.OutputBudgets, OutputBudgetEvent{
|
||||
Tool: stringField(obj, "tool"),
|
||||
Category: stringField(obj, "category"),
|
||||
OriginalBytes: int(parseInt64(obj["original_bytes"])),
|
||||
RetainedBytes: int(parseInt64(obj["retained_bytes"])),
|
||||
EstimatedOriginalTokens: int(parseInt64(obj["estimated_original_tokens"])),
|
||||
EstimatedRetainedTokens: int(parseInt64(obj["estimated_retained_tokens"])),
|
||||
Truncated: boolField(obj, "truncated"),
|
||||
Reason: stringField(obj, "reason"),
|
||||
SpillCreated: boolField(obj, "spill_created"),
|
||||
})
|
||||
default:
|
||||
// Unknown event type: tolerate (forward-compat) but only after a
|
||||
// header has been seen.
|
||||
@@ -117,12 +151,26 @@ func ReadNDJSON(r io.Reader) (*TurnTrace, error) {
|
||||
if !sawTraceHeader {
|
||||
return nil, errors.New("parse trace: non-empty input had no type:trace header")
|
||||
}
|
||||
if len(t.Spans) == 0 && len(t.Counters) == 0 {
|
||||
return nil, errors.New("parse trace: header present but no spans or counters recovered (corrupt or truncated)")
|
||||
if len(t.Spans) == 0 && len(t.Counters) == 0 && len(t.PrefixHashes) == 0 && len(t.OutputBudgets) == 0 {
|
||||
return nil, errors.New("parse trace: header present but no spans, counters, or prefix hashes recovered (corrupt or truncated)")
|
||||
}
|
||||
return t, nil
|
||||
}
|
||||
|
||||
func boolField(obj map[string]any, key string) bool {
|
||||
value, _ := obj[key].(bool)
|
||||
return value
|
||||
}
|
||||
|
||||
// stringField returns obj[key] as a string, or "" if the key is missing or
|
||||
// the value is not a string. JSON-marshaled trace events always emit
|
||||
// string fields as JSON strings, so the type assertion is the right
|
||||
// narrowing; a missing key produces an empty hash, which is what the
|
||||
// encoder would have produced for the absent value.
|
||||
func stringField(obj map[string]any, key string) string {
|
||||
s, _ := obj[key].(string)
|
||||
return s
|
||||
}
|
||||
func parseTime(v any) time.Time {
|
||||
s, _ := v.(string)
|
||||
if s == "" {
|
||||
|
||||
@@ -172,6 +172,37 @@ func (r *Recorder) StampFirstUsefulAction() {
|
||||
r.tr.FirstUsefulActionAt = time.Now()
|
||||
}
|
||||
|
||||
// EmitPrefixHash records one prompt-prefix fingerprint on the trace. Multiple
|
||||
// calls are allowed within a run (one per turn, typically) and accumulate in
|
||||
// order. The first call after Finish is a no-op; later calls are also no-ops
|
||||
// because the trace has been sealed.
|
||||
func (r *Recorder) EmitPrefixHash(p PrefixHash) {
|
||||
if r == nil {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.finished {
|
||||
return
|
||||
}
|
||||
r.tr.PrefixHashes = append(r.tr.PrefixHashes, p)
|
||||
}
|
||||
|
||||
// EmitOutputBudget records one content-free output budgeting decision. Calls
|
||||
// are retained in transcript emission order, including when the underlying
|
||||
// tools executed concurrently.
|
||||
func (r *Recorder) EmitOutputBudget(event OutputBudgetEvent) {
|
||||
if r == nil {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.finished {
|
||||
return
|
||||
}
|
||||
r.tr.OutputBudgets = append(r.tr.OutputBudgets, event)
|
||||
}
|
||||
|
||||
// Finish stamps CompletedAt, derives each span's parent (by interval
|
||||
// containment) and exclusive time, and returns a snapshot of the trace. Calling
|
||||
// Finish more than once returns the same snapshot.
|
||||
@@ -190,6 +221,8 @@ func (r *Recorder) Finish() *TurnTrace {
|
||||
// Copy the slices so callers cannot mutate the recorder's state.
|
||||
snap.Spans = append([]Span(nil), r.tr.Spans...)
|
||||
snap.Counters = append([]Counter(nil), r.tr.Counters...)
|
||||
snap.PrefixHashes = append([]PrefixHash(nil), r.tr.PrefixHashes...)
|
||||
snap.OutputBudgets = append([]OutputBudgetEvent(nil), r.tr.OutputBudgets...)
|
||||
return &snap
|
||||
}
|
||||
|
||||
|
||||
+71
-10
@@ -32,6 +32,7 @@ const (
|
||||
SpanToolPartition = "tool_partition" // partitioning the tool set for the prompt
|
||||
SpanProviderConnect = "provider_connect" // client.Do in the provider seam
|
||||
SpanProviderQueue = "provider_queue" // pre-send OAuth/token resolve
|
||||
SpanProviderPrewarm = "provider_prewarm" // best-effort turn-session prewarm probe
|
||||
SpanGeneration = "generation" // streaming a model completion
|
||||
SpanToolExecution = "tool_execution" // executing tool calls
|
||||
SpanPermissionWait = "permission_wait" // waiting on a permission prompt
|
||||
@@ -53,6 +54,9 @@ const (
|
||||
CounterInputTokens = "input_tokens"
|
||||
CounterCachedInputTokens = "cached_input_tokens"
|
||||
CounterOutputTokens = "output_tokens"
|
||||
CounterPrewarmAttempts = "prewarm_attempts"
|
||||
CounterPrefixStable = "prefix_stable"
|
||||
CounterPrefixDrift = "prefix_drift"
|
||||
)
|
||||
|
||||
// Span is one named wall interval attributed to part of a run. Each stamp is
|
||||
@@ -79,19 +83,71 @@ type Counter struct {
|
||||
Value int64 `json:"value"`
|
||||
}
|
||||
|
||||
// OutputBudgetEvent is compact, content-free metadata for one tool result's
|
||||
// output budgeting decision. It deliberately carries no output text, paths, or
|
||||
// arguments so tracing cannot become a secret-bearing side channel.
|
||||
type OutputBudgetEvent struct {
|
||||
Tool string `json:"tool"`
|
||||
Category string `json:"category"`
|
||||
OriginalBytes int `json:"original_bytes"`
|
||||
RetainedBytes int `json:"retained_bytes"`
|
||||
EstimatedOriginalTokens int `json:"estimated_original_tokens"`
|
||||
EstimatedRetainedTokens int `json:"estimated_retained_tokens"`
|
||||
Truncated bool `json:"truncated"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
SpillCreated bool `json:"spill_created"`
|
||||
}
|
||||
|
||||
// TurnTrace is the finished record for one agent.Run. It is the value
|
||||
// emitters serialize; it is not mutated after Finish returns a snapshot.
|
||||
type TurnTrace struct {
|
||||
SessionID string `json:"session_id"`
|
||||
RunID string `json:"run_id"`
|
||||
Profile string `json:"profile,omitempty"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
FirstVisibleEventAt time.Time `json:"first_visible_event_at,omitempty"`
|
||||
FirstUsefulActionAt time.Time `json:"first_useful_action_at,omitempty"`
|
||||
FirstTokenAt time.Time `json:"first_token_at,omitempty"`
|
||||
CompletedAt time.Time `json:"completed_at"`
|
||||
Spans []Span `json:"spans"`
|
||||
Counters []Counter `json:"counters"`
|
||||
SessionID string `json:"session_id"`
|
||||
RunID string `json:"run_id"`
|
||||
Profile string `json:"profile,omitempty"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
FirstVisibleEventAt time.Time `json:"first_visible_event_at,omitempty"`
|
||||
FirstUsefulActionAt time.Time `json:"first_useful_action_at,omitempty"`
|
||||
FirstTokenAt time.Time `json:"first_token_at,omitempty"`
|
||||
CompletedAt time.Time `json:"completed_at"`
|
||||
Spans []Span `json:"spans"`
|
||||
Counters []Counter `json:"counters"`
|
||||
PrefixHashes []PrefixHash `json:"prefix_hashes,omitempty"`
|
||||
OutputBudgets []OutputBudgetEvent `json:"output_budgets,omitempty"`
|
||||
}
|
||||
|
||||
// PrefixHash is one fingerprint of the prompt prefix emitted by an agent run.
|
||||
// The seven fields decompose the cacheable part of the request so a downstream
|
||||
// observer can detect which sub-component drifted turn-over-turn:
|
||||
// base_instructions (the embedded core system prompt), confirmation_policy,
|
||||
// project_context (AGENTS.md / ZERO.md chain), skills, tools, schema (tool
|
||||
// JSON schemas), and complete_prefix (SHA-256 of the canonical concatenation
|
||||
// of the other six). All hashes are hex-encoded SHA-256.
|
||||
//
|
||||
// Scope: the fingerprint covers 4 of 11 sections of buildSystemPrompt. The
|
||||
// seven sections NOT covered — modelPromptAddendum, sessionRuntimeContext,
|
||||
// approvedCommandPrefixContext, workspaceSeedContext, userGuidelines,
|
||||
// specialistDelegationContext, responseStyleContext — fire only for
|
||||
// non-default Options. For default Options (the common case), the four
|
||||
// captured substrings are the full prompt and the fingerprint is accurate.
|
||||
//
|
||||
// A run with a stable CompletePrefix across turns means the four captured
|
||||
// sub-components are byte-identical. A run where CompletePrefix changes
|
||||
// names the captured sub-component that drifted, but does NOT rule out
|
||||
// drift in the seven uncaptured sections. modelPromptAddendum in particular
|
||||
// changes on a model switch (a model_switches counter the trace already
|
||||
// emits), so a consumer correlating CompletePrefix stability with
|
||||
// cached_input_tokens must cross-check the model_switches counter to
|
||||
// disambiguate "no drift" from "drift in an uncaptured section." The fields
|
||||
// are emitted as a "prefix_hash" event in the NDJSON trace (see
|
||||
// WriteNDJSON).
|
||||
type PrefixHash struct {
|
||||
BaseInstructionsHash string `json:"base_instructions"`
|
||||
ConfirmationPolicyHash string `json:"confirmation_policy"`
|
||||
ProjectContextHash string `json:"project_context"`
|
||||
SkillsHash string `json:"skills"`
|
||||
ToolsHash string `json:"tools"`
|
||||
SchemaHash string `json:"schema"`
|
||||
CompletePrefixHash string `json:"complete_prefix"`
|
||||
}
|
||||
|
||||
// WallDuration is the total traced wall time of the run.
|
||||
@@ -229,5 +285,10 @@ func OptionalEventKeys() []string {
|
||||
"counter:" + CounterAcceptanceChecks,
|
||||
"counter:" + CounterPollingTurn,
|
||||
"counter:" + CounterModelSwitches,
|
||||
"span:" + SpanProviderPrewarm,
|
||||
"counter:" + CounterPrewarmAttempts,
|
||||
"counter:" + CounterPrefixStable,
|
||||
"counter:" + CounterPrefixDrift,
|
||||
"event:prefix_hash",
|
||||
}
|
||||
}
|
||||
@@ -370,6 +370,27 @@ func TestReadNDJSONRoundTrip(t *testing.T) {
|
||||
r.Counter(CounterModelRequests, 3)
|
||||
r.Counter(CounterToolCalls, 7)
|
||||
r.StampFirstToken()
|
||||
// Two prefix_hash events so the round-trip test covers the third
|
||||
// event type. Insertion order is preserved by the parser (no sort
|
||||
// is applied on read or write).
|
||||
r.EmitPrefixHash(PrefixHash{
|
||||
BaseInstructionsHash: "b1",
|
||||
ConfirmationPolicyHash: "c1",
|
||||
ProjectContextHash: "p1",
|
||||
SkillsHash: "s1",
|
||||
ToolsHash: "t1",
|
||||
SchemaHash: "x1",
|
||||
CompletePrefixHash: "complete1",
|
||||
})
|
||||
r.EmitPrefixHash(PrefixHash{
|
||||
BaseInstructionsHash: "b2",
|
||||
ConfirmationPolicyHash: "c2",
|
||||
ProjectContextHash: "p2",
|
||||
SkillsHash: "s2",
|
||||
ToolsHash: "t2",
|
||||
SchemaHash: "x2",
|
||||
CompletePrefixHash: "complete2",
|
||||
})
|
||||
original := r.Finish()
|
||||
|
||||
var buf bytes.Buffer
|
||||
@@ -398,6 +419,20 @@ func TestReadNDJSONRoundTrip(t *testing.T) {
|
||||
if parsed.FirstTokenAt.IsZero() {
|
||||
t.Fatal("first_token_at lost in round-trip")
|
||||
}
|
||||
// prefix_hash round-trip: two events, in insertion order, with all
|
||||
// seven sub-hash fields preserved exactly.
|
||||
if len(parsed.PrefixHashes) != 2 {
|
||||
t.Fatalf("expected 2 prefix_hash events after round-trip, got %d", len(parsed.PrefixHashes))
|
||||
}
|
||||
if parsed.PrefixHashes[0].CompletePrefixHash != "complete1" || parsed.PrefixHashes[1].CompletePrefixHash != "complete2" {
|
||||
t.Fatalf("prefix_hash insertion order lost: got %+v want [complete1, complete2]", parsed.PrefixHashes)
|
||||
}
|
||||
if parsed.PrefixHashes[0].BaseInstructionsHash != "b1" || parsed.PrefixHashes[0].SchemaHash != "x1" {
|
||||
t.Fatalf("prefix_hash sub-hashes lost on first event: got %+v", parsed.PrefixHashes[0])
|
||||
}
|
||||
if parsed.PrefixHashes[1].BaseInstructionsHash != "b2" || parsed.PrefixHashes[1].SchemaHash != "x2" {
|
||||
t.Fatalf("prefix_hash sub-hashes lost on second event: got %+v", parsed.PrefixHashes[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadNDJSONRejectsNonTrace(t *testing.T) {
|
||||
@@ -411,10 +446,12 @@ func TestReadNDJSONRejectsNonTrace(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestReadNDJSONRejectsHeaderOnly(t *testing.T) {
|
||||
// A header with no recoverable spans/counters is corrupt/truncated, not empty.
|
||||
// A header with no recoverable spans, counters, or prefix hashes is
|
||||
// corrupt/truncated, not empty. prefix_hash is now a third valid
|
||||
// event type, so a header with only prefix_hash events is accepted.
|
||||
header := `{"type":"trace","name":"run","session_id":"s","run_id":"r"}` + "\n"
|
||||
if _, err := ReadNDJSON(strings.NewReader(header)); err == nil {
|
||||
t.Fatal("expected error for a trace header with no spans or counters")
|
||||
t.Fatal("expected error for a trace header with no spans, counters, or prefix hashes")
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -63,6 +63,14 @@ func (m model) routePaste(content string) (tea.Model, tea.Cmd) {
|
||||
// do it once here, before any early return, so both the bracketed-paste
|
||||
// and right-click-paste paths are covered uniformly.
|
||||
m = m.disarmCancelConfirmation()
|
||||
// Same immediate-input transition as the keypress hook: text entering the
|
||||
// composer must land the caret in its solid-while-typing state right away
|
||||
// and refresh the typing timestamp. Without this, a paste right after the
|
||||
// blink phase hid the caret leaves the newly populated composer caret-less
|
||||
// until the next 530ms tick, and the stale timestamp makes that tick
|
||||
// toggle instead of going solid.
|
||||
m.lastCharTime = m.now()
|
||||
m.composerCursorVisible = true
|
||||
if content == "" {
|
||||
// Empty text clipboard — the user may have pasted a screenshot.
|
||||
// Probe the OS clipboard for image content asynchronously.
|
||||
|
||||
@@ -39,6 +39,8 @@ func (m model) buildKeybindingGroups() []keybindingGroup {
|
||||
{"Shift+Enter / Alt+Enter", "insert a newline (multi-line compose)"},
|
||||
{"Esc (\u00d72)", "cancel the run / dismiss a popup / clear the input"},
|
||||
{"Ctrl+C", "cancel the run, then quit"},
|
||||
{"Ctrl+X then letter", "common slash commands (m=/model, p=/provider, r=/resume, \u2026)"},
|
||||
{"Ctrl+X ?", "list every Ctrl+X slash shortcut"},
|
||||
{"?", "show this help (on an empty input)"},
|
||||
},
|
||||
},
|
||||
@@ -47,7 +49,7 @@ func (m model) buildKeybindingGroups() []keybindingGroup {
|
||||
bindings: []keybinding{
|
||||
{labelOr(m.keyBindings.cycleReasoning, "Ctrl+T"), "cycle reasoning effort (auto \u2192 low \u2192 medium \u2192 high)"},
|
||||
{"Shift+Tab", "cycle permission mode (auto \u2194 ask)"},
|
||||
{labelOr(m.keyBindings.togglePlan, "Ctrl+P"), "expand / collapse the plan panel"},
|
||||
{labelOr(m.keyBindings.togglePlan, "Ctrl+P"), "expand / collapse the plan panel (when no menu is open)"},
|
||||
},
|
||||
},
|
||||
{
|
||||
@@ -55,6 +57,7 @@ func (m model) buildKeybindingGroups() []keybindingGroup {
|
||||
bindings: []keybinding{
|
||||
{"PgUp / PgDn", "scroll the transcript by a page"},
|
||||
{"\u2191 / \u2193", "scroll, or move within a popup / multi-line input"},
|
||||
{"Ctrl+P / Ctrl+N", "previous / next item in menus (emacs-style)"},
|
||||
{labelOr(m.keyBindings.toggleDetailed, "Ctrl+O"), "toggle the detailed (full-screen) transcript"},
|
||||
{labelOr(m.keyBindings.toggleSidebar, "Ctrl+B"), "hide / show the right context sidebar"},
|
||||
{labelOr(m.keyBindings.toggleMouse, "Ctrl+E"), "release the mouse to drag-select & copy text"},
|
||||
|
||||
@@ -79,6 +79,7 @@ func TestHelpOverlayViewRendersGroupsAndKeys(t *testing.T) {
|
||||
"Ctrl+T", "cycle reasoning effort",
|
||||
"Shift+Tab", "Ctrl+P", "Ctrl+O",
|
||||
"drill into its sub-session",
|
||||
"Ctrl+X then letter", "/model",
|
||||
keybindingHelpFooter,
|
||||
} {
|
||||
if !strings.Contains(view, want) {
|
||||
@@ -224,7 +225,9 @@ func TestRemappedToggleBindingsIgnoreComposerGuard(t *testing.T) {
|
||||
m.height = 40
|
||||
m.transcript = append(m.transcript, transcriptRow{kind: rowToolCall, tool: "read_file", detail: "main.go"})
|
||||
m.keyBindings.toggleMouse = parseBinding("ctrl+m")
|
||||
m.keyBindings.toggleSidebar = parseBinding("ctrl+n")
|
||||
// Avoid Ctrl+N: idle Ctrl+N is reserved as an emacs menu no-op and never
|
||||
// reaches configurable global bindings.
|
||||
m.keyBindings.toggleSidebar = parseBinding("ctrl+y")
|
||||
if !m.sidebarToggleAllowed() {
|
||||
t.Fatal("sidebar toggle should be allowed")
|
||||
}
|
||||
@@ -245,10 +248,10 @@ func TestRemappedToggleBindingsIgnoreComposerGuard(t *testing.T) {
|
||||
}
|
||||
|
||||
initialSidebar := next.sidebarHidden
|
||||
updated, _ = next.Update(testKeyCtrl('n'))
|
||||
updated, _ = next.Update(testKeyCtrl('y'))
|
||||
next = updated.(model)
|
||||
if next.sidebarHidden == initialSidebar {
|
||||
t.Fatal("remapped Ctrl+N toggleSidebar binding should still fire with a non-empty composer")
|
||||
t.Fatal("remapped Ctrl+Y toggleSidebar binding should still fire with a non-empty composer")
|
||||
}
|
||||
if next.composerValue() != "hello" {
|
||||
t.Fatalf("remapped toggleSidebar binding should not fall through to composer input, got %q", next.composerValue())
|
||||
|
||||
Loaded 100 of 120 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user