mirror of
https://github.com/Twigpine/zero.git
synced 2026-11-09 18:08:31 -10:00
Compare commits
75
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b46e943e68 | ||
|
|
de4dfbe551 | ||
|
|
ae30755ce7 | ||
|
|
d6d0d0cc25 | ||
|
|
0121fc4f3c | ||
|
|
158d5ee62f | ||
|
|
4c945485d0 | ||
|
|
7f9ca0860a | ||
|
|
34dea45d58 | ||
|
|
1e0120c1a6 | ||
|
|
c84640ca60 | ||
|
|
ed355ca4c4 | ||
|
|
87124fb463 | ||
|
|
a4a248c15e | ||
|
|
1d75e35670 | ||
|
|
0a600a83d8 | ||
|
|
5e574c729a | ||
|
|
6235e85b5a | ||
|
|
b98cdde42c | ||
|
|
3eda0c1d3e | ||
|
|
72515e0c7f | ||
|
|
8bebff712c | ||
|
|
d7f047d2f6 | ||
|
|
3da7325349 | ||
|
|
ec85bd5849 | ||
|
|
37d08e3833 | ||
|
|
0504fa790c | ||
|
|
ec86c77abc | ||
|
|
bd031f0ec1 | ||
|
|
aaf681dec6 | ||
|
|
b314301ff0 | ||
|
|
4315bec802 | ||
|
|
ea56072536 | ||
|
|
00573812b2 | ||
|
|
bcbdd65c18 | ||
|
|
fcedabf2ed | ||
|
|
427e7161ee | ||
|
|
9a289b9c07 | ||
|
|
b3c0ac91cf | ||
|
|
7fb33c381a | ||
|
|
e7c23b3b92 | ||
|
|
c879333290 | ||
|
|
fbd6b47c64 | ||
|
|
8b63915ff1 | ||
|
|
da85c67244 | ||
|
|
5737cb3c8f | ||
|
|
4b2e10ad68 | ||
|
|
ddd236f045 | ||
|
|
25075d0a09 | ||
|
|
9f34f4ab3e | ||
|
|
c965339908 | ||
|
|
fb96d065c0 | ||
|
|
c73e47a025 | ||
|
|
29795b49b1 | ||
|
|
072d3fe260 | ||
|
|
b45b38d247 | ||
|
|
bff135a60e | ||
|
|
e280f0ebdd | ||
|
|
63322d059e | ||
|
|
b1166032f3 | ||
|
|
725967f85b | ||
|
|
62a835c2e0 | ||
|
|
e3a65bca32 | ||
|
|
5fc455b726 | ||
|
|
1811e7b775 | ||
|
|
851bf7b054 | ||
|
|
3d5cdefbc9 | ||
|
|
b57d3e4abf | ||
|
|
d64cc67f9d | ||
|
|
1b4f880cb6 | ||
|
|
6fcd5224ac | ||
|
|
2a71cafdcf | ||
|
|
4791a0a7d2 | ||
|
|
920e26e8ce | ||
|
|
e68fb87467 |
No files matched your search
@@ -1 +1 @@
|
||||
{".":"0.13.0"}
|
||||
{".":"0.11.0"}
|
||||
@@ -6,52 +6,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html) once the first release is
|
||||
tagged. Until then, source builds report the version `dev`.
|
||||
|
||||
## [0.13.0](https://github.com/Twigpine/zero/compare/v0.12.0...v0.13.0) (2026-11-09)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **mcp:** add AI/ML API as an opt-in built-in server ([#1193](https://github.com/Twigpine/zero/issues/1193)) ([16ccd78](https://github.com/Twigpine/zero/commit/16ccd78765b6a2a56b7370e3031710d718aa4757))
|
||||
* **plugins:** add CLI enable/disable for plugin.json ([#1202](https://github.com/Twigpine/zero/issues/1202)) ([05c1b5e](https://github.com/Twigpine/zero/commit/05c1b5e16098e1c60212316d20ae8e055a27b5ae))
|
||||
* **providers:** point OpenGateway at api.twigpine.com and trust its live model list ([#1208](https://github.com/Twigpine/zero/issues/1208)) ([47f78c6](https://github.com/Twigpine/zero/commit/47f78c66247aa96c8753d6bb95e935126ff1761e))
|
||||
* **web-fetch:** add an opt-in AI/ML API Toolbox scraping backend ([#1192](https://github.com/Twigpine/zero/issues/1192)) ([c9f1794](https://github.com/Twigpine/zero/commit/c9f179482a7f6d7111ee64fa9cb8ffa2517c2397))
|
||||
* **web-search:** add AI/ML API Toolbox as an opt-in backend ([#1189](https://github.com/Twigpine/zero/issues/1189)) ([9873645](https://github.com/Twigpine/zero/commit/98736457bdac6b0272e032728a5485cf557780fe))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **hooks:** dispatch specialistStart/Stop lifecycle hooks ([#1204](https://github.com/Twigpine/zero/issues/1204)) ([2ca84cc](https://github.com/Twigpine/zero/commit/2ca84cc127d9369765004b72cf95049fecc6c28b))
|
||||
* **tui:** make reasoning stream rendering linear (coalesced deltas, O(1) append, tail-bounded preview) ([#1198](https://github.com/Twigpine/zero/issues/1198)) ([bbf779e](https://github.com/Twigpine/zero/commit/bbf779e69168d62bb1103aefdd1b10e53433209e))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* **tui:** bounded file read, LRU rendering cache, and size caps for full-file view ([#1203](https://github.com/Twigpine/zero/issues/1203)) ([e034a93](https://github.com/Twigpine/zero/commit/e034a93815629c68976dcdab7dcc08bbe9b03693))
|
||||
|
||||
## [0.12.0](https://github.com/Twigpine/zero/compare/v0.11.0...v0.12.0) (2026-11-02)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **providers:** add MiniMax-M3.1-Flash to catalog ([#1180](https://github.com/Twigpine/zero/issues/1180)) ([d99f92b](https://github.com/Twigpine/zero/commit/d99f92b81a671f186cede81614c89b8e8a7f9451))
|
||||
* **sandbox:** support self-exec pattern to eliminate separate helper binary ([#946](https://github.com/Twigpine/zero/issues/946)) ([8c4e759](https://github.com/Twigpine/zero/commit/8c4e7597989ef9b3ca67b7703dfd15d92372aed2))
|
||||
* **skills:** install a skill's bundled files and point the skill tool at them ([#1167](https://github.com/Twigpine/zero/issues/1167)) ([1641506](https://github.com/Twigpine/zero/commit/164150633a8b2e6ae864ae02fd7105b201eeb68e))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **agent:** mark headless guardrail stops incomplete ([#1182](https://github.com/Twigpine/zero/issues/1182)) ([5aa54fe](https://github.com/Twigpine/zero/commit/5aa54fe1d2b419c9bbef73605a77919c9007cf7b))
|
||||
* bump Go to 1.26.10 for GO-2026-6809 ([#1168](https://github.com/Twigpine/zero/issues/1168)) ([6393a90](https://github.com/Twigpine/zero/commit/6393a90ed618a3536e09c02852f4ec829b97545b))
|
||||
* **chatgpt:** refresh subscription defaults and compatibility version ([#1179](https://github.com/Twigpine/zero/issues/1179)) ([f912f80](https://github.com/Twigpine/zero/commit/f912f80d002602e11ba0ef8548ecee58442874aa))
|
||||
* **deps:** refresh the lockfile to hono 4.13.13 ([#1171](https://github.com/Twigpine/zero/issues/1171)) ([de1443b](https://github.com/Twigpine/zero/commit/de1443b2edb44fd3747cc77200d54dc8cffcd0a7))
|
||||
* **oauth:** reuse the crash-safe secret store and sync encrypted publications ([#1165](https://github.com/Twigpine/zero/issues/1165)) ([84c1b15](https://github.com/Twigpine/zero/commit/84c1b1554076679a342e9cac61f8f8cdbaf3a429))
|
||||
* **redaction:** redact credentials whose trigger a greedy span swallowed ([#1158](https://github.com/Twigpine/zero/issues/1158)) ([a97db88](https://github.com/Twigpine/zero/commit/a97db8832fdba077f2d31bdf03ecef8d7c774fe1))
|
||||
* **sessions:** capture freeform apply_patch checkpoints ([#1183](https://github.com/Twigpine/zero/issues/1183)) ([6396208](https://github.com/Twigpine/zero/commit/63962081cf130c2171407936ce3beb7021d238b8))
|
||||
* **sessions:** preserve image attachment summaries on resume ([#1152](https://github.com/Twigpine/zero/issues/1152)) ([1e84849](https://github.com/Twigpine/zero/commit/1e848492e5c1919d187fb5e1eb5e7ef9bc6140dc)), closes [#1135](https://github.com/Twigpine/zero/issues/1135)
|
||||
* **tools:** grep stops where a file's line breaks stop ([#1160](https://github.com/Twigpine/zero/issues/1160)) ([aa9ba84](https://github.com/Twigpine/zero/commit/aa9ba8412eb8917daa2ca5cb3fdbd0a39276fa88))
|
||||
* **workspaceindex:** import gitsafe under the Twigpine module path ([#1153](https://github.com/Twigpine/zero/issues/1153)) ([ef4da13](https://github.com/Twigpine/zero/commit/ef4da1323f1097863ae278224b5f304c2fff909a))
|
||||
* **workspaceindex:** run Scan's git calls through gitsafe ([#1149](https://github.com/Twigpine/zero/issues/1149)) ([2ffe183](https://github.com/Twigpine/zero/commit/2ffe1831b0950b1463ec46d487d03a5fcb679cc6))
|
||||
|
||||
## [0.11.0](https://github.com/Twigpine/zero/compare/v0.10.0...v0.11.0) (2026-10-15)
|
||||
|
||||
|
||||
|
||||
@@ -15,7 +15,6 @@
|
||||
<img alt="Go 1.26.10+" src="https://img.shields.io/badge/Go-1.26.10+-00ADD8?logo=go&logoColor=white">
|
||||
<img alt="25+ providers" src="https://img.shields.io/badge/providers-25+-34E2EA">
|
||||
<a href="https://discord.gg/CaQDS6wdFn"><img alt="Discord" src="https://img.shields.io/badge/Discord-join-5865F2?logo=discord&logoColor=white"></a>
|
||||
<a href="https://vercel.com/open-source-program"><img alt="Vercel OSS Program" src="https://vercel.com/oss/program-badge-2026.svg"></a>
|
||||
<br>
|
||||
<strong>English</strong> | <a href="README_ZH.md">中文</a>
|
||||
</p>
|
||||
@@ -88,18 +87,9 @@ zero exec --output-format stream-json < turns.jsonl
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
<td align="center" width="150" height="80">
|
||||
<a href="https://vercel.com/">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/vercel-logo-dark.svg">
|
||||
<img src="docs/assets/vercel-logo.svg" alt="Vercel logo" width="120">
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center"><a href="https://www.gmicloud.ai/"><strong>GMI Cloud</strong></a></td>
|
||||
<td align="center"><a href="https://vercel.com/"><strong>Vercel</strong></a></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
@@ -397,76 +387,6 @@ and can rewrite every request through it, so this is the trust you accept by
|
||||
configuring one. Leave the variables unset for the checks to be enforced end to
|
||||
end.
|
||||
|
||||
The built-in `web_search` tool is registered only when a search backend is
|
||||
configured: `ZERO_WEBSEARCH_BASE_URL` (with an optional `ZERO_WEBSEARCH_API_KEY`)
|
||||
for a generic JSON endpoint, `ZERO_WEBSEARCH_PROVIDER=searxng` with a SearXNG
|
||||
base URL, or the AI/ML API Toolbox:
|
||||
|
||||
```bash
|
||||
export ZERO_WEBSEARCH_PROVIDER=aimlapi
|
||||
```
|
||||
|
||||
The AI/ML API backend runs the Toolbox tool `web/search`, and every search is
|
||||
billed to your AI/ML API balance: about $0.0013 per result, with a documented
|
||||
minimum of $0.0091 per search. Each search is sent with a $0.025 spending cap,
|
||||
which the Toolbox enforces. It is used only when
|
||||
`ZERO_WEBSEARCH_PROVIDER` is set to `aimlapi`; an AI/ML API key alone never
|
||||
enables it. The key is read from your user configuration only; provider
|
||||
profiles in a project `.zero` config are not used. It is the aimlapi profile's
|
||||
key (the active profile when it is aimlapi, otherwise the first aimlapi
|
||||
profile), including a key in the credential store, or `AIMLAPI_API_KEY` when no
|
||||
aimlapi profile exists. It is resolved again on every search;
|
||||
`ZERO_WEBSEARCH_API_KEY` is not used. When any aimlapi profile, or any profile
|
||||
that reads `AIMLAPI_API_KEY`, points at a base URL other than
|
||||
`https://api.aimlapi.com/v1`, or `AIMLAPI_INFERENCE_URL` is overridden, the key
|
||||
may belong to a proxy, so it is not sent and the search fails without a request.
|
||||
Requests go only to
|
||||
`https://tools.aimlapi.com` (`ZERO_WEBSEARCH_BASE_URL` is ignored and redirects
|
||||
are refused) and carry `X-AIMLAPI-Source: agent/zero` and an
|
||||
`X-AIMLAPI-Partner-ID` header. The `domains` argument is sent to the Toolbox as
|
||||
`include_domains`, and results are still filtered locally.
|
||||
|
||||
### AI/ML API Toolbox backend for `web_fetch`
|
||||
|
||||
`web_fetch` can fetch pages through the AI/ML API Toolbox tool
|
||||
`search1api/crawl` instead of a direct HTTP GET. It is off unless you choose it:
|
||||
|
||||
```bash
|
||||
export ZERO_WEBFETCH_PROVIDER=aimlapi
|
||||
```
|
||||
|
||||
Having an AI/ML API key never switches it on by itself, and any other value (or
|
||||
none) keeps the direct fetch. The variable is read on every fetch.
|
||||
|
||||
- **Billing:** Toolbox runs are billed to your AI/ML API balance, about
|
||||
$0.0013 per run. Each run is sent with `max_cost_usd` 0.005, so a run
|
||||
priced above that is refused before it starts and not charged. Rate-limited
|
||||
or gateway-failed requests are retried with the same idempotency key, so a
|
||||
retry is never billed twice.
|
||||
- **Key:** the key comes from your `aimlapi` provider profile (including a key
|
||||
in the credential store), or from `AIMLAPI_API_KEY` when there is no such
|
||||
profile. It is withheld, and `web_fetch` returns an error, when any `aimlapi`
|
||||
profile or any profile using `AIMLAPI_API_KEY` points at a base URL other than
|
||||
`https://api.aimlapi.com/v1`, or when `AIMLAPI_INFERENCE_URL` is overridden:
|
||||
the key may then belong to a proxy.
|
||||
- **Destination:** requests go only to the fixed `https://tools.aimlapi.com`
|
||||
endpoint, with `Authorization: Bearer <key>` and the attribution headers
|
||||
`X-AIMLAPI-Source: agent/zero` and `X-AIMLAPI-Partner-ID`. Redirects are
|
||||
refused. `HTTP_PROXY`/`HTTPS_PROXY` apply to these requests as to Zero's other
|
||||
aimlapi.com calls.
|
||||
- **No fallback:** without a usable key, or when the run fails, `web_fetch`
|
||||
returns an error and does not fetch the page directly. Unset the variable to
|
||||
go back to direct fetches.
|
||||
- **Remote fetch:** the page is fetched by a remote service, which may serve a
|
||||
cached copy, and the full URL (query string included) is sent to it. The URL
|
||||
checks above still run first, so loopback, private and metadata URLs are
|
||||
never handed to it. The result is Markdown with a
|
||||
`Backend: aimlapi (search1api/crawl)` line; `max_bytes` cuts the Markdown,
|
||||
and `format: "raw"` is refused because the original response is not
|
||||
available. The complete Toolbox run response is limited to 8 MiB before JSON
|
||||
decoding; larger responses return an error even if `max_bytes` is smaller.
|
||||
The remote run may already have been billed when this limit is detected.
|
||||
|
||||
For local dev servers, use shell commands such as `curl` through `exec_command`
|
||||
so the normal sandbox and permission policy applies. Long-running commands stay
|
||||
attached to a background terminal session and can be listed or stopped from the
|
||||
|
||||
@@ -15,7 +15,6 @@
|
||||
<img alt="Go 1.26.10+" src="https://img.shields.io/badge/Go-1.26.10+-00ADD8?logo=go&logoColor=white">
|
||||
<img alt="25+ providers" src="https://img.shields.io/badge/providers-25+-34E2EA">
|
||||
<a href="https://discord.gg/CaQDS6wdFn"><img alt="Discord" src="https://img.shields.io/badge/Discord-join-5865F2?logo=discord&logoColor=white"></a>
|
||||
<a href="https://vercel.com/open-source-program"><img alt="Vercel OSS Program" src="https://vercel.com/oss/program-badge-2026.svg"></a>
|
||||
<br>
|
||||
<a href="README.md">English</a> | <strong>中文</strong>
|
||||
</p>
|
||||
@@ -86,18 +85,9 @@ zero exec --output-format stream-json < turns.jsonl
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
<td align="center" width="150" height="80">
|
||||
<a href="https://vercel.com/">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/vercel-logo-dark.svg">
|
||||
<img src="docs/assets/vercel-logo.svg" alt="Vercel logo" width="120">
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center"><a href="https://www.gmicloud.ai/"><strong>GMI Cloud</strong></a></td>
|
||||
<td align="center"><a href="https://vercel.com/"><strong>Vercel</strong></a></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
|
||||
+1
-13
@@ -33,16 +33,4 @@ Please include, where possible:
|
||||
depend on a user explicitly disabling the sandbox (`--skip-permissions-unsafe`), are generally
|
||||
out of scope — but please report anything you are unsure about.
|
||||
- Zero sends no telemetry. Network calls go to the model/tool providers you configure (and, for
|
||||
keyless web search, to Exa's hosted MCP endpoint at `https://mcp.exa.ai/mcp`). The built-in
|
||||
AI/ML API MCP server (`https://mcp.aimlapi.com/mcp`) is off until you run
|
||||
`zero mcp enable aimlapi`. Once enabled, the Bearer key follows the aimlapi
|
||||
profile resolution order (inline key, then the profile's `apiKeyEnv` — default
|
||||
`AIMLAPI_API_KEY` — then the credential store), or `AIMLAPI_API_KEY` alone when
|
||||
no aimlapi profile exists; a non-empty `AIMLAPI_API_KEY` can override a stored
|
||||
profile key for MCP billing. The automatic key is withheld if an aimlapi profile
|
||||
(or a profile reading `AIMLAPI_API_KEY`) uses a base URL other than
|
||||
`https://api.aimlapi.com/v1`, `AIMLAPI_INFERENCE_URL` points elsewhere, a
|
||||
non-empty user Authorization header is configured, OAuth is selected, project
|
||||
config has touched the server, or the key is missing or not a single token
|
||||
(including Unicode whitespace). See
|
||||
[docs/EXTENDING.md](docs/EXTENDING.md#built-in-servers).
|
||||
keyless web search, to Exa's hosted MCP endpoint at `https://mcp.exa.ai/mcp`).
|
||||
+2
-69
@@ -110,8 +110,6 @@ Discovery roots (earlier wins on name collisions):
|
||||
|
||||
A missing directory is fine — Zero just omits it. Management commands (`zero skill add` / `zero skill remove`) always write to the primary Zero dir only; `list` / `info` search primary + `~/.agents/skills`. `skill info` reports lock source/hash when present and whether the on-disk `SKILL.md` has drifted from the recorded hash.
|
||||
|
||||
For a skill installed since Zero started installing bundled files, `skill info` also reports drift in any file installed with it.
|
||||
|
||||
```text
|
||||
~/.local/share/zero/skills/ # primary (install/remove target)
|
||||
run-benchmarks/
|
||||
@@ -142,15 +140,6 @@ Only `name` and `description` are recognized in the frontmatter today. The `name
|
||||
|
||||
The `skill` tool lets the agent load any discovered skill by name (primary, agents, or plugin).
|
||||
|
||||
A skill can ship files beside its `SKILL.md`, in the [agentskills.io](https://agentskills.io/specification) layout: `scripts/`, `references/`, `assets/` or anything else the instructions refer to. `zero skill add` installs the whole skill directory, leaving out hidden entries such as `.git` and not following or copying symlinks (it lists any it skipped). A skill larger than 1,000 files or 32 MiB is refused. When the `skill` tool loads a skill that has such files, its output ends with the skill's directory and a list of the files, so instructions like "run `scripts/fill.py`" can be followed. Running a bundled script goes through the normal tool permissions; installing or loading a skill never runs anything.
|
||||
|
||||
```text
|
||||
pdf/
|
||||
SKILL.md
|
||||
scripts/fill.py
|
||||
references/form-fields.md
|
||||
```
|
||||
|
||||
## 4. Hooks
|
||||
|
||||
Hooks fire shell commands on lifecycle events. Configure them in JSON:
|
||||
@@ -203,11 +192,7 @@ Events the agent emits (in dispatch order):
|
||||
| `specialistStart` | A sub-agent is spawned | yes (specialist name) |
|
||||
| `specialistStop` | A sub-agent ends | yes (specialist name) |
|
||||
|
||||
For swarm members the matcher subject is the **agent type** (`subagent`, `teammate`, …), not a registered specialist profile name — swarm members run from an inline manifest keyed by agent type.
|
||||
|
||||
`specialistStart` fires once the child process has actually started; a launch that fails before exec (missing binary, spawn error) records no public lifecycle hooks. `specialistStop` fires once per terminal stop: completion, error, explicit stop of a recovered task, and a background task reaped by a later `TaskOutput` all share one durable claim, so a hook that notifies or cleans up does not run twice for the same sub-agent run.
|
||||
|
||||
A hook's exit code decides what happens next: `0` continues, non-zero blocks the tool call (`beforeTool`) or surfaces an error (`afterTool`). `sessionStart` / `sessionEnd` / `specialistStart` / `specialistStop` hooks are advisory (failures are audited but do not interrupt the run). Hook execution is recorded in the audit log; the audit is reachable from the agent's view of past actions, not from a dedicated `zero doctor` check.
|
||||
A hook's exit code decides what happens next: `0` continues, non-zero blocks the tool call (`beforeTool`) or surfaces an error (`afterTool`). Hook execution is recorded in the audit log; the audit is reachable from the agent's view of past actions, not from a dedicated `zero doctor` check.
|
||||
|
||||
> **Roadmap.** An in-UI hooks manager is on the backlog. Today you edit the JSON directly.
|
||||
|
||||
@@ -261,56 +246,6 @@ or use a new server name. OAuth tokens are bound to the resolved server identity
|
||||
so changing a server target can require `zero mcp oauth login <server>` again.
|
||||
OAuth server names ending in `.<32 hex chars>` are reserved for token storage.
|
||||
|
||||
#### Built-in servers
|
||||
|
||||
Zero seeds two MCP servers before your config is merged. Both show up in
|
||||
`zero mcp list`, and any field can be overridden in your user config.
|
||||
|
||||
- `exa` (`https://mcp.exa.ai/mcp`) is on by default for keyless web search.
|
||||
`zero mcp disable exa` turns it off.
|
||||
- `aimlapi` (`https://mcp.aimlapi.com/mcp`, the AI/ML API Toolbox, model
|
||||
discovery, image/video/audio generation, chat, embeddings and balance tools)
|
||||
is **off by default**, because its tool runs and generations are billed to
|
||||
your AI/ML API balance. Turn it on with `zero mcp enable aimlapi`, which
|
||||
writes only `"disabled": false` to your user config, and off again with
|
||||
`zero mcp disable aimlapi`. A project config cannot turn it on.
|
||||
|
||||
When `aimlapi` connects, Zero sends your AI/ML API key as
|
||||
`Authorization: Bearer <key>`. The key is resolved again on every request and
|
||||
is never written to `config.json`. With an aimlapi.com provider profile
|
||||
(`zero providers setup aimlapi`), the key is chosen in this order:
|
||||
|
||||
1. An inline key on the profile (if set in config).
|
||||
2. Otherwise the environment variable named by the profile's `apiKeyEnv`
|
||||
(the catalog default is `AIMLAPI_API_KEY`).
|
||||
3. Otherwise the key held in the credential store when the profile uses
|
||||
`apiKeyStored`.
|
||||
|
||||
When there is no aimlapi profile, Zero uses `AIMLAPI_API_KEY` alone. If both
|
||||
`AIMLAPI_API_KEY` and a stored profile key exist, step 2 wins over the store
|
||||
whenever the inline key is empty — MCP requests may bill the environment key's
|
||||
account, not the stored key.
|
||||
|
||||
No automatic key is sent when an aimlapi profile, or any profile reading
|
||||
`AIMLAPI_API_KEY`, uses a base URL other than `https://api.aimlapi.com/v1`, or
|
||||
when `AIMLAPI_INFERENCE_URL` is overridden, because that credential may belong
|
||||
to a proxy. No key is sent when the resolved value is not a single token (for
|
||||
example a comma- or newline-separated pool), when you set your own
|
||||
non-empty `Authorization` header, when the server uses `"auth": "oauth"`, or
|
||||
when project config touched the server. Without a key the server answers 401.
|
||||
To sign in with your aimlapi.com account instead, set `"auth": "oauth"` on the
|
||||
server and run `zero mcp oauth login aimlapi`.
|
||||
|
||||
Every request to exactly `https://mcp.aimlapi.com` (https, default port,
|
||||
whatever the server is named, including one you add by hand) also carries the
|
||||
attribution headers `X-AIMLAPI-Source: agent/zero` and `X-AIMLAPI-Partner-ID`,
|
||||
and gets the key under the rules above. Other MCP servers never receive these
|
||||
headers or the key.
|
||||
|
||||
If the server echoes back the key Zero sent it, Zero replaces the key with
|
||||
`[REDACTED]` in that server's error messages and tool results before anything
|
||||
reads them, and reports an HTTP error status without the response body.
|
||||
|
||||
### As a server — expose Zero's tools to another agent
|
||||
|
||||
```bash
|
||||
@@ -355,12 +290,10 @@ Install and manage:
|
||||
zero plugins add ./github-pr-review # copy into ~/.config/zero/plugins/ or ./.zero/plugins/
|
||||
zero plugins list
|
||||
zero plugins info github-pr-review # show manifest path, enabled state, lock metadata
|
||||
zero plugins disable github-pr-review # sets "enabled": false in plugin.json
|
||||
zero plugins enable github-pr-review # sets "enabled": true in plugin.json
|
||||
zero plugins remove github-pr-review # alias: rm
|
||||
```
|
||||
|
||||
A plugin is active when it is present under a plugins directory and its `plugin.json` has `"enabled": true` (the default when the field is omitted). Use `zero plugins disable <id>` / `zero plugins enable <id>` to toggle that field without removing the plugin. Pass `--user` to target only user plugins and ignore `./.zero/plugins`; without it, a project plugin with the same id takes precedence. The toggle edits the `enabled` value in place — the rest of the manifest keeps its formatting, field order, and numeric literals — and serializes with `plugins add` / `plugins remove` on the same plugins root. Zero reads plugins when it starts, so the change applies to the next `zero exec` run or interactive session; an interactive session that is already running keeps the plugin tools, hooks, and skills it started with until you restart it.
|
||||
A plugin is enabled by being present in the plugins directory and disabled by removing it (or by the user setting `"enabled": false` in its `plugin.json`). Plugins are not enabled or disabled by a CLI subcommand today.
|
||||
|
||||
Plugin commands run with the plugin directory as their working directory. Use relative paths; the loader resolves them at activation time.
|
||||
|
||||
|
||||
@@ -634,10 +634,6 @@ Core tool groups include:
|
||||
`HTTP_PROXY`/`HTTPS_PROXY` set, the proxy is dialed instead and resolves the
|
||||
target itself, so destination enforcement ends at the proxy; see the note in
|
||||
the README.
|
||||
With `ZERO_WEBFETCH_PROVIDER=aimlapi` (opt-in; a key alone does not select
|
||||
it), the same URL checks run and the page is then fetched as Markdown by the
|
||||
billed AI/ML API Toolbox tool `search1api/crawl`, with no fallback to a
|
||||
direct fetch; see the README.
|
||||
- **Local-control tools**: config-gated wrappers for browser, terminal, desktop,
|
||||
and artifact helpers backed by packages such as `internal/localcontrol` and
|
||||
`internal/browser`.
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="261" height="52" fill="none" viewBox="0 0 261 52"><path fill="white" d="M59.8 52H0L29.9 0zm67.82-38.45q4.9 0 8.81 2.13a15.5 15.5 0 0 1 6.22 6.32q2.3 4.2 2.38 10.26v2.06h-26.35q.27 4.4 2.58 6.92 2.38 2.47 6.36 2.47a8.4 8.4 0 0 0 7.76-4.93l9.16.67q-1.68 4.98-6.29 7.99t-10.63 3q-5.53 0-9.64-2.27a16 16 0 0 1-6.43-6.46 20 20 0 0 1-2.31-9.72q0-5.52 2.3-9.72a16 16 0 0 1 6.44-6.46 20 20 0 0 1 9.64-2.26m62.55 0q4.47 0 8.18 1.66a15.3 15.3 0 0 1 6.15 4.6q2.38 3 2.87 7.05l-9.23.47a8 8 0 0 0-2.8-5 7.7 7.7 0 0 0-5.17-1.86q-4.33 0-6.7 3-2.4 3-2.39 8.52t2.38 8.52q2.37 3 6.71 3 3.15 0 5.39-1.87 2.24-1.92 2.72-5.46l9.3.4a14.7 14.7 0 0 1-2.87 7.33 16 16 0 0 1-6.15 4.86 21 21 0 0 1-8.39 1.66q-5.53 0-9.64-2.26a16 16 0 0 1-6.44-6.46 20 20 0 0 1-2.3-9.72q0-5.52 2.3-9.72a16 16 0 0 1 6.44-6.46 20 20 0 0 1 9.64-2.26m38.66 0q4.9 0 8.8 2.13a15.5 15.5 0 0 1 6.22 6.32q2.31 4.2 2.38 10.26v2.06h-26.35q.28 4.4 2.58 6.92 2.38 2.47 6.36 2.47a8.4 8.4 0 0 0 7.77-4.93l9.15.67q-1.68 5-6.29 7.99t-10.62 3q-5.53 0-9.65-2.27a16 16 0 0 1-6.43-6.46 20 20 0 0 1-2.31-9.72q0-5.52 2.3-9.72a16 16 0 0 1 6.44-6.46 20 20 0 0 1 9.64-2.26M86.9 36.69l17.24-34.33h10.8L89.96 49.63h-6.12L58.85 2.36h10.81zm71.62-15.55a11 11 0 0 1 2.47-4.48q2.28-2.31 6.38-2.31h3.4v7.26h-3.47q-2.91 0-4.79.8a5.8 5.8 0 0 0-2.77 2.5q-.9 1.72-.9 4.37v20.35h-8.89V14.35h8.33zm101.73 28.5h-8.95V2.35h8.95zM127.62 20.26q-3.7 0-6 2.2-2.32 2.2-2.87 6.2h17.05q-.48-4.34-2.72-6.33a7.8 7.8 0 0 0-5.46-2.07m101.2 0q-3.7 0-6 2.2-2.31 2.2-2.87 6.2H237q-.5-4.34-2.73-6.33a7.8 7.8 0 0 0-5.46-2.07" style="fill:white;fill-opacity:1;"/></svg>
|
||||
|
Before Width: | Height: | Size: 1.6 KiB |
@@ -1 +0,0 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="261" height="52" fill="none" viewBox="0 0 261 52"><path fill="black" d="M59.8 52H0L29.9 0zm67.82-38.45q4.9 0 8.81 2.13a15.5 15.5 0 0 1 6.22 6.32q2.3 4.2 2.38 10.26v2.06h-26.35q.27 4.4 2.58 6.92 2.38 2.47 6.36 2.47a8.4 8.4 0 0 0 7.76-4.93l9.16.67q-1.68 4.98-6.29 7.99t-10.63 3q-5.53 0-9.64-2.27a16 16 0 0 1-6.43-6.46 20 20 0 0 1-2.31-9.72q0-5.52 2.3-9.72a16 16 0 0 1 6.44-6.46 20 20 0 0 1 9.64-2.26m62.55 0q4.47 0 8.18 1.66a15.3 15.3 0 0 1 6.15 4.6q2.38 3 2.87 7.05l-9.23.47a8 8 0 0 0-2.8-5 7.7 7.7 0 0 0-5.17-1.86q-4.33 0-6.7 3-2.4 3-2.39 8.52t2.38 8.52q2.37 3 6.71 3 3.15 0 5.39-1.87 2.24-1.92 2.72-5.46l9.3.4a14.7 14.7 0 0 1-2.87 7.33 16 16 0 0 1-6.15 4.86 21 21 0 0 1-8.39 1.66q-5.53 0-9.64-2.26a16 16 0 0 1-6.44-6.46 20 20 0 0 1-2.3-9.72q0-5.52 2.3-9.72a16 16 0 0 1 6.44-6.46 20 20 0 0 1 9.64-2.26m38.66 0q4.9 0 8.8 2.13a15.5 15.5 0 0 1 6.22 6.32q2.31 4.2 2.38 10.26v2.06h-26.35q.28 4.4 2.58 6.92 2.38 2.47 6.36 2.47a8.4 8.4 0 0 0 7.77-4.93l9.15.67q-1.68 5-6.29 7.99t-10.62 3q-5.53 0-9.65-2.27a16 16 0 0 1-6.43-6.46 20 20 0 0 1-2.31-9.72q0-5.52 2.3-9.72a16 16 0 0 1 6.44-6.46 20 20 0 0 1 9.64-2.26M86.9 36.69l17.24-34.33h10.8L89.96 49.63h-6.12L58.85 2.36h10.81zm71.62-15.55a11 11 0 0 1 2.47-4.48q2.28-2.31 6.38-2.31h3.4v7.26h-3.47q-2.91 0-4.79.8a5.8 5.8 0 0 0-2.77 2.5q-.9 1.72-.9 4.37v20.35h-8.89V14.35h8.33zm101.73 28.5h-8.95V2.35h8.95zM127.62 20.26q-3.7 0-6 2.2-2.32 2.2-2.87 6.2h17.05q-.48-4.34-2.72-6.33a7.8 7.8 0 0 0-5.46-2.07m101.2 0q-3.7 0-6 2.2-2.31 2.2-2.87 6.2H237q-.5-4.34-2.73-6.33a7.8 7.8 0 0 0-5.46-2.07" style="fill:black;fill-opacity:1;"/></svg>
|
||||
|
Before Width: | Height: | Size: 1.6 KiB |
@@ -559,7 +559,6 @@ func (a *Agent) runTurn(ctx context.Context, sess *acpSession, userText string,
|
||||
Cwd: sess.cwd,
|
||||
SessionID: sess.id,
|
||||
ProviderName: resolved.Provider.Name,
|
||||
ModelFamily: providercatalog.ModelFamilyFor(resolved.Provider.CatalogID),
|
||||
Model: resolved.Provider.Model,
|
||||
Registry: registry,
|
||||
Sandbox: sandboxEngine,
|
||||
|
||||
@@ -1,143 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// A CONFIGURATION REFUSAL IS NOT A RETRIABLE FAILURE, EVEN WHEN IT ARRIVES
|
||||
// EARLY.
|
||||
//
|
||||
// capture_artifact rejects in RejectBeforePermission, which the registry returns
|
||||
// straight back before any of the gates that attach provenance. Its
|
||||
// valid-but-unavailable calls therefore reached the classifier with no denial
|
||||
// category, no permission metadata and no refusal marker, and were read as
|
||||
// ordinary retriable failures: the model got the schema hint and the call could
|
||||
// consume the profile failure-streak escalation, for a tool that never executed
|
||||
// and that no argument change can enable.
|
||||
func TestDisabledCaptureArtifactIsAPolicyRefusalNotARetriableFailure(t *testing.T) {
|
||||
// No artifacts directory and no enabled driver: valid arguments, unavailable
|
||||
// tool. This is the shape an operator produces by configuration alone.
|
||||
registry := tools.NewRegistry()
|
||||
for _, tool := range tools.NewLocalControlArtifactTools(tools.LocalControlArtifactOptions{}) {
|
||||
registry.Register(tool)
|
||||
}
|
||||
|
||||
result := registry.RunWithOptions(context.Background(), "capture_artifact", map[string]any{
|
||||
"action": "browser_screenshot",
|
||||
"name": "shot",
|
||||
}, tools.RunOptions{PermissionGranted: true})
|
||||
|
||||
if result.Status != tools.StatusError {
|
||||
t.Fatalf("SETUP INVALID: expected the disabled tool to refuse, got %s: %s", result.Status, result.Output)
|
||||
}
|
||||
if !tools.IsPolicyRefusalResult(result) {
|
||||
t.Fatalf("the refusal carries no provenance, so nothing downstream can tell it from a failed command: %#v", result.Meta)
|
||||
}
|
||||
|
||||
// Through the conversion the loop performs, which is where the marker has to
|
||||
// survive to be worth anything.
|
||||
converted := ToolResult{
|
||||
Status: result.Status,
|
||||
Output: result.Output,
|
||||
Meta: result.Meta,
|
||||
}
|
||||
if !isPolicyRefusal(converted) {
|
||||
t.Error("the marker did not survive into the agent-facing result")
|
||||
}
|
||||
if isRetriableToolError(converted) {
|
||||
t.Error("a tool that never executed, and that no argument change can enable, was marked retriable: the model gets a schema hint telling it to fix arguments that were already valid")
|
||||
}
|
||||
}
|
||||
|
||||
// The malformed-argument branch must stay retriable. That one IS fixable by
|
||||
// trying again differently, which is what the hint exists for, so marking every
|
||||
// early rejection would trade one wrong answer for another.
|
||||
func TestMalformedCaptureArtifactArgumentsStayRetriable(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
for _, tool := range tools.NewLocalControlArtifactTools(tools.LocalControlArtifactOptions{}) {
|
||||
registry.Register(tool)
|
||||
}
|
||||
|
||||
result := registry.RunWithOptions(context.Background(), "capture_artifact", map[string]any{
|
||||
"action": "not_a_real_action",
|
||||
}, tools.RunOptions{PermissionGranted: true})
|
||||
|
||||
if result.Status != tools.StatusError {
|
||||
t.Fatalf("SETUP INVALID: expected invalid arguments to fail, got %s", result.Status)
|
||||
}
|
||||
if tools.IsPolicyRefusalResult(result) {
|
||||
t.Fatalf("a malformed-argument error was marked a policy refusal, so the model is denied the hint that would let it fix the call: %q", result.Output)
|
||||
}
|
||||
converted := ToolResult{Status: result.Status, Output: result.Output, Meta: result.Meta}
|
||||
if !isRetriableToolError(converted) {
|
||||
t.Error("invalid arguments should stay retriable")
|
||||
}
|
||||
}
|
||||
|
||||
// alwaysRefusingCaptureTool stands in for the disabled tool at Run level, so the
|
||||
// loop consequence can be observed rather than inferred.
|
||||
type alwaysRefusingCaptureTool struct{ ran int }
|
||||
|
||||
func (tool *alwaysRefusingCaptureTool) Name() string { return "capture_artifact" }
|
||||
func (tool *alwaysRefusingCaptureTool) Description() string { return "test capture tool" }
|
||||
func (tool *alwaysRefusingCaptureTool) Parameters() tools.Schema {
|
||||
return tools.Schema{
|
||||
Type: "object",
|
||||
Properties: map[string]tools.PropertySchema{"action": {Type: "string"}},
|
||||
Required: []string{"action"},
|
||||
AdditionalProperties: false,
|
||||
}
|
||||
}
|
||||
func (tool *alwaysRefusingCaptureTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow, Reason: "captures artifacts"}
|
||||
}
|
||||
func (tool *alwaysRefusingCaptureTool) Run(context.Context, map[string]any) tools.Result {
|
||||
tool.ran++
|
||||
return tools.Result{}
|
||||
}
|
||||
func (tool *alwaysRefusingCaptureTool) RejectBeforePermission(map[string]any) (tools.Result, bool) {
|
||||
return tools.Result{
|
||||
Status: tools.StatusError,
|
||||
Output: "Error: capture_artifact is disabled because no artifact directory is configured.",
|
||||
Meta: map[string]string{tools.PolicyRefusalMeta: tools.PolicyRefusalToolNotEnabled},
|
||||
}, true
|
||||
}
|
||||
|
||||
// THE LOOP CONSEQUENCE. A refusal must not draw the retry hint, because the hint
|
||||
// tells the model to fix arguments that were already valid and the tool will
|
||||
// refuse identically next time.
|
||||
func TestRunDoesNotHintARefusedCaptureArtifact(t *testing.T) {
|
||||
tool := &alwaysRefusingCaptureTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
calls := toolFailureHintAt + 1
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, calls+1)
|
||||
for i := range calls {
|
||||
turns = append(turns, toolTurn("call-"+strconv.Itoa(i), "capture_artifact", `{"action":"browser_screenshot"}`))
|
||||
}
|
||||
turns = append(turns, textTurn("gave up on the screenshot"))
|
||||
|
||||
result, err := Run(context.Background(), "take a screenshot", &mockProvider{turns: turns}, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tool.ran != 0 {
|
||||
t.Fatalf("SETUP INVALID: the tool executed %d times; it must be refused before Run", tool.ran)
|
||||
}
|
||||
for _, message := range result.Messages {
|
||||
if strings.Contains(message.Content, toolFailureHintMarker) {
|
||||
t.Fatal("a refused, never-executed tool drew the retry hint, which tells the model to fix arguments that were already valid")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,112 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// alternatingRefusalCaptureTool refuses every call with the same CATEGORY and a
|
||||
// different MESSAGE, which is what a real disabled driver does: the refusal
|
||||
// names the action, and the model is free to alternate valid actions.
|
||||
type alternatingRefusalCaptureTool struct{ ran int }
|
||||
|
||||
func (tool *alternatingRefusalCaptureTool) Name() string { return "capture_artifact" }
|
||||
func (tool *alternatingRefusalCaptureTool) Description() string { return "test capture tool" }
|
||||
func (tool *alternatingRefusalCaptureTool) Parameters() tools.Schema {
|
||||
return tools.Schema{
|
||||
Type: "object",
|
||||
Properties: map[string]tools.PropertySchema{"action": {Type: "string"}},
|
||||
Required: []string{"action"},
|
||||
AdditionalProperties: false,
|
||||
}
|
||||
}
|
||||
func (tool *alternatingRefusalCaptureTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow, Reason: "captures artifacts"}
|
||||
}
|
||||
func (tool *alternatingRefusalCaptureTool) Run(context.Context, map[string]any) tools.Result {
|
||||
tool.ran++
|
||||
return tools.Result{}
|
||||
}
|
||||
func (tool *alternatingRefusalCaptureTool) RejectBeforePermission(args map[string]any) (tools.Result, bool) {
|
||||
action, _ := args["action"].(string)
|
||||
return tools.Result{
|
||||
Status: tools.StatusError,
|
||||
// Action-specific wording, same category. This is the whole point: the
|
||||
// prose differs on every call while the refusal never changes.
|
||||
Output: "Error: Local control driver for " + action + " is disabled.",
|
||||
Meta: map[string]string{tools.PolicyRefusalMeta: tools.PolicyRefusalToolNotEnabled},
|
||||
}, true
|
||||
}
|
||||
|
||||
// A REFUSAL KEYS ON ITS CATEGORY, INCLUDING WHEN THE CATEGORY IS ONLY A MARKER.
|
||||
//
|
||||
// The registry marks pre-execution refusals in metadata, and isPolicyRefusal
|
||||
// read that marker while observeToolResult keyed on DenialReason, which those
|
||||
// paths leave empty. The guard fell back to errorSignature(output), so two
|
||||
// refusals of the same category with different wording looked like two
|
||||
// different failures and the streak restarted at 1 every call.
|
||||
//
|
||||
// A model alternating capture_artifact's browser_screenshot and browser_pdf
|
||||
// against a disabled driver is refused identically each time and never tripped
|
||||
// the six-call halt. Only the generic twelve-error fallback stopped the run,
|
||||
// reporting varied errors rather than a repeated refusal.
|
||||
func TestAlternatingRefusedActionsStillTripTheRefusalHalt(t *testing.T) {
|
||||
tool := &alternatingRefusalCaptureTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
// More calls than the refusal halt but FEWER than the generic any-error
|
||||
// fallback, so only the category-keyed streak can stop this.
|
||||
calls := toolFailureAnyErrorStopAt - 1
|
||||
if calls <= toolFailureStopAt {
|
||||
t.Fatalf("SETUP INVALID: %d calls cannot distinguish the refusal halt from the generic fallback", calls)
|
||||
}
|
||||
actions := []string{"browser_screenshot", "browser_pdf"}
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, calls+1)
|
||||
for i := range calls {
|
||||
action := actions[i%len(actions)]
|
||||
turns = append(turns, toolTurn("call-"+strconv.Itoa(i), "capture_artifact", `{"action":"`+action+`"}`))
|
||||
}
|
||||
turns = append(turns, textTurn("gave up"))
|
||||
|
||||
result, err := Run(context.Background(), "capture something", &mockProvider{turns: turns}, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tool.ran != 0 {
|
||||
t.Fatalf("SETUP INVALID: the tool executed %d times; it must be refused before Run", tool.ran)
|
||||
}
|
||||
|
||||
refusals := 0
|
||||
for _, message := range result.Messages {
|
||||
if strings.Contains(message.Content, "is disabled") {
|
||||
refusals++
|
||||
}
|
||||
}
|
||||
if refusals > toolFailureStopAt {
|
||||
t.Errorf("the run made %d refused calls; the six-call refusal halt never tripped because the streak re-keyed on each action's wording", refusals)
|
||||
}
|
||||
|
||||
// And the halt has to read as a repeated refusal, not as varied errors.
|
||||
stop := strings.ToLower(strings.Join(messageContents(result.Messages), "\n"))
|
||||
if strings.Contains(stop, toolFailureHintMarker) {
|
||||
t.Error("a refused, never-executed tool drew the retry hint")
|
||||
}
|
||||
}
|
||||
|
||||
func messageContents(messages []zeroruntime.Message) []string {
|
||||
out := make([]string, 0, len(messages))
|
||||
for _, message := range messages {
|
||||
out = append(out, message.Content)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -1,93 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/localcontrol"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
)
|
||||
|
||||
// THE DISABLED-DRIVER BRANCH, REACHED FOR REAL.
|
||||
//
|
||||
// RejectBeforePermission refuses on two separate conditions, and its neighbour
|
||||
// test constructs empty options, so it returns at the FIRST one (no artifact
|
||||
// directory configured) and the second is never evaluated. Reverting the
|
||||
// disabled-driver branch in local_capture.go to a plain errorResult therefore
|
||||
// left the whole agent and tools suites green.
|
||||
//
|
||||
// This is the shape an operator actually produces: an artifact root IS
|
||||
// configured and one driver IS enabled, and the model asks for an action owned
|
||||
// by a different, disabled driver. Without provenance that call is read as a
|
||||
// retriable failure, so it collects a schema hint and can spend the profile's
|
||||
// failure-streak escalation, for a tool no argument change can enable.
|
||||
func TestCaptureArtifactDisabledDriverIsAPolicyRefusal(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
options := tools.LocalControlArtifactOptions{
|
||||
ArtifactsDir: t.TempDir(),
|
||||
// Configured and enabled, so the tool as a whole is available and the
|
||||
// missing-directory branch cannot fire.
|
||||
Browser: localcontrol.BrowserOptions{Enabled: true, Driver: "test-browser", HelperPath: "browser-helper"},
|
||||
// Terminal deliberately left disabled.
|
||||
}
|
||||
for _, tool := range tools.NewLocalControlArtifactTools(options) {
|
||||
registry.Register(tool)
|
||||
}
|
||||
|
||||
// An enabled driver's action must still be accepted, or the test below would
|
||||
// pass for a tool that refuses everything.
|
||||
if result := registry.RunWithOptions(context.Background(), "capture_artifact", map[string]any{
|
||||
"action": "browser_screenshot", "name": "shot",
|
||||
}, tools.RunOptions{PermissionGranted: true}); tools.IsPolicyRefusalResult(result) {
|
||||
t.Fatalf("SETUP INVALID: the enabled browser driver was refused as policy: %s", result.Output)
|
||||
}
|
||||
|
||||
result := registry.RunWithOptions(context.Background(), "capture_artifact", map[string]any{
|
||||
"action": "terminal_snapshot", "name": "snap", "session": "test-session",
|
||||
}, tools.RunOptions{PermissionGranted: true})
|
||||
|
||||
if result.Status != tools.StatusError {
|
||||
t.Fatalf("SETUP INVALID: the disabled terminal driver did not refuse: %s / %s", result.Status, result.Output)
|
||||
}
|
||||
if !tools.IsPolicyRefusalResult(result) {
|
||||
t.Fatalf("a driver disabled by configuration refuses without provenance, so it reads as a retriable failure: %#v", result.Meta)
|
||||
}
|
||||
if got := result.Meta["policy_refusal"]; got != tools.PolicyRefusalToolNotEnabled {
|
||||
t.Errorf("policy_refusal = %q, want %q", got, tools.PolicyRefusalToolNotEnabled)
|
||||
}
|
||||
if isRetriableToolError(ToolResult{Status: result.Status, Output: result.ModelOutput(), Meta: result.Meta}) {
|
||||
t.Error("the refusal is still classified as retriable, so the model gets a schema hint for a configuration decision")
|
||||
}
|
||||
}
|
||||
|
||||
// AND THE EARLY REJECTIONS MUST NOT COLLAPSE INTO ONE ANSWER.
|
||||
//
|
||||
// RejectBeforePermission refuses on configuration; argument validation refuses
|
||||
// on a fixable mistake, and it runs FIRST, which is how the disabled-driver
|
||||
// branch stayed uncovered. A malformed call must stay an ordinary retriable
|
||||
// error, because trying again differently is exactly the right response to it.
|
||||
func TestCaptureArtifactMalformedArgumentsStayRetriable(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
for _, tool := range tools.NewLocalControlArtifactTools(tools.LocalControlArtifactOptions{
|
||||
ArtifactsDir: t.TempDir(),
|
||||
Terminal: localcontrol.TerminalOptions{Enabled: true, Driver: "test-terminal", HelperPath: "terminal-helper"},
|
||||
}) {
|
||||
registry.Register(tool)
|
||||
}
|
||||
|
||||
// The driver IS enabled, so nothing here is a configuration decision. The
|
||||
// call is simply missing the session the action requires.
|
||||
result := registry.RunWithOptions(context.Background(), "capture_artifact", map[string]any{
|
||||
"action": "terminal_snapshot", "name": "snap",
|
||||
}, tools.RunOptions{PermissionGranted: true})
|
||||
|
||||
if result.Status != tools.StatusError {
|
||||
t.Fatalf("SETUP INVALID: the malformed call did not fail: %s", result.Output)
|
||||
}
|
||||
if tools.IsPolicyRefusalResult(result) {
|
||||
t.Errorf("a fixable argument mistake was marked a policy refusal, so the model loses the schema hint that would fix it: %s", result.Output)
|
||||
}
|
||||
if !isRetriableToolError(ToolResult{Status: result.Status, Output: result.ModelOutput(), Meta: result.Meta}) {
|
||||
t.Errorf("a fixable argument mistake is not retriable: %s", result.Output)
|
||||
}
|
||||
}
|
||||
@@ -1,34 +0,0 @@
|
||||
package agent
|
||||
|
||||
import "github.com/Twigpine/zero/internal/tools"
|
||||
|
||||
// CheckpointTargets returns the workspace-relative paths a tool call will mutate,
|
||||
// resolving its arguments exactly as executeToolCall does. Freeform apply_patch
|
||||
// input is normalized through the registered built-in tool, so absolute headers
|
||||
// are rewritten relative to the selected write root and cwd is set: checkpoint
|
||||
// capture and execution therefore agree on the files at stake.
|
||||
//
|
||||
// Best-effort: nil when the call has no capture targets or its arguments cannot
|
||||
// be resolved. Checkpoint capture never fails a run.
|
||||
func CheckpointTargets(workspaceRoot string, registry *tools.Registry, call ToolCall) []string {
|
||||
if registry == nil {
|
||||
return nil
|
||||
}
|
||||
args := map[string]any{}
|
||||
if call.Freeform {
|
||||
tool, ok := registry.Get(call.Name)
|
||||
if !ok || !tools.IsBuiltInApplyPatch(tool) {
|
||||
return nil
|
||||
}
|
||||
prepared, err := tools.PrepareFreeformApplyPatchArguments(tool, call.Arguments)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
args = prepared
|
||||
} else if call.Arguments != "" {
|
||||
if err := decodeToolArguments(call.Arguments, &args); err != nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return tools.MutationTargets(workspaceRoot, call.Name, args)
|
||||
}
|
||||
@@ -1,99 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
)
|
||||
|
||||
func TestCheckpointTargets(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
// Absolute headers must be compared against the physical root: t.TempDir
|
||||
// differs from it on macOS (/var -> /private/var) and via Windows short names.
|
||||
if resolved, err := filepath.EvalSymlinks(root); err == nil {
|
||||
root = resolved
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, "main.txt"), []byte("before\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
relativePatch := "*** Begin Patch\n*** Update File: main.txt\n@@\n-before\n+after\n*** End Patch\n"
|
||||
absolutePatch := "*** Begin Patch\n*** Update File: " + filepath.Join(root, "main.txt") + "\n@@\n-before\n+after\n*** End Patch\n"
|
||||
multiFilePatch := "*** Begin Patch\n*** Update File: main.txt\n@@\n-before\n+after\n*** Add File: notes.txt\n+new\n*** End Patch\n"
|
||||
jsonArgs, err := json.Marshal(map[string]any{"patch": relativePatch})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tools.NewScopedApplyPatchTool(root, nil))
|
||||
unbuiltinRegistry := tools.NewRegistry()
|
||||
unbuiltinRegistry.Register(&customApplyPatchTool{})
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
registry *tools.Registry
|
||||
call ToolCall
|
||||
want []string
|
||||
}{
|
||||
{
|
||||
name: "freeform relative header",
|
||||
call: ToolCall{Name: "apply_patch", Arguments: relativePatch, Freeform: true},
|
||||
want: []string{"main.txt"},
|
||||
},
|
||||
{
|
||||
name: "freeform absolute header stays workspace-relative",
|
||||
call: ToolCall{Name: "apply_patch", Arguments: absolutePatch, Freeform: true},
|
||||
want: []string{"main.txt"},
|
||||
},
|
||||
{
|
||||
name: "freeform multi-file patch",
|
||||
call: ToolCall{Name: "apply_patch", Arguments: multiFilePatch, Freeform: true},
|
||||
want: []string{"main.txt", "notes.txt"},
|
||||
},
|
||||
{
|
||||
name: "json arguments",
|
||||
call: ToolCall{Name: "apply_patch", Arguments: string(jsonArgs)},
|
||||
want: []string{"main.txt"},
|
||||
},
|
||||
{
|
||||
name: "freeform traversal is rejected",
|
||||
call: ToolCall{Name: "apply_patch", Arguments: "*** Begin Patch\n*** Update File: ../escape.txt\n@@\n-x\n+y\n*** End Patch\n", Freeform: true},
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "freeform custom apply_patch is not prepared",
|
||||
registry: unbuiltinRegistry,
|
||||
call: ToolCall{Name: "apply_patch", Arguments: relativePatch, Freeform: true},
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "unknown freeform tool",
|
||||
call: ToolCall{Name: "read_file", Arguments: "raw", Freeform: true},
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "malformed json arguments",
|
||||
call: ToolCall{Name: "apply_patch", Arguments: "{"},
|
||||
want: nil,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
reg := registry
|
||||
if tc.registry != nil {
|
||||
reg = tc.registry
|
||||
}
|
||||
got := CheckpointTargets(root, reg, tc.call)
|
||||
if !slices.Equal(got, tc.want) {
|
||||
t.Fatalf("CheckpointTargets() = %#v, want %#v", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
if got := CheckpointTargets(root, nil, ToolCall{Name: "apply_patch", Arguments: relativePatch}); got != nil {
|
||||
t.Fatalf("CheckpointTargets(nil registry) = %#v, want nil", got)
|
||||
}
|
||||
}
|
||||
@@ -1,135 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/streamjson"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
)
|
||||
|
||||
// progressProbeTool records whether the loop handed it a progress callback.
|
||||
// streams mirrors what a real tool declares via tools.ChildProgressStreamer.
|
||||
type progressProbeTool struct {
|
||||
name string
|
||||
streams bool
|
||||
got *bool
|
||||
}
|
||||
|
||||
func (t *progressProbeTool) Name() string { return t.name }
|
||||
func (t *progressProbeTool) Description() string { return "probe" }
|
||||
func (t *progressProbeTool) Parameters() tools.Schema {
|
||||
return tools.Schema{Type: "object", Properties: map[string]tools.PropertySchema{}}
|
||||
}
|
||||
func (t *progressProbeTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow}
|
||||
}
|
||||
func (t *progressProbeTool) Run(context.Context, map[string]any) tools.Result {
|
||||
*t.got = false
|
||||
return tools.Result{Status: tools.StatusOK, Output: "ok"}
|
||||
}
|
||||
func (t *progressProbeTool) RunWithOptions(_ context.Context, _ map[string]any, options tools.RunOptions) tools.Result {
|
||||
*t.got = options.Progress != nil
|
||||
return tools.Result{Status: tools.StatusOK, Output: "ok"}
|
||||
}
|
||||
|
||||
// StreamsChildProgress makes this type implement tools.ChildProgressStreamer;
|
||||
// the flag drives the answer. A tool that never declares at all is modelled by
|
||||
// silentProbeTool below, which genuinely does not implement the interface.
|
||||
func (t *progressProbeTool) StreamsChildProgress() bool { return t.streams }
|
||||
|
||||
// silentProbeTool does NOT implement tools.ChildProgressStreamer at all — the
|
||||
// state every tool in the tree is in today except Task and orchestrate.
|
||||
type silentProbeTool struct {
|
||||
name string
|
||||
got *bool
|
||||
}
|
||||
|
||||
func (t *silentProbeTool) Name() string { return t.name }
|
||||
func (t *silentProbeTool) Description() string { return "probe" }
|
||||
func (t *silentProbeTool) Parameters() tools.Schema {
|
||||
return tools.Schema{Type: "object", Properties: map[string]tools.PropertySchema{}}
|
||||
}
|
||||
func (t *silentProbeTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow}
|
||||
}
|
||||
func (t *silentProbeTool) Run(context.Context, map[string]any) tools.Result {
|
||||
*t.got = false
|
||||
return tools.Result{Status: tools.StatusOK, Output: "ok"}
|
||||
}
|
||||
func (t *silentProbeTool) RunWithOptions(_ context.Context, _ map[string]any, options tools.RunOptions) tools.Result {
|
||||
*t.got = options.Progress != nil
|
||||
return tools.Result{Status: tools.StatusOK, Output: "ok"}
|
||||
}
|
||||
|
||||
func runProbe(t *testing.T, tool tools.Tool, got *bool, withSink bool) bool {
|
||||
t.Helper()
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
options := Options{}
|
||||
if withSink {
|
||||
options.OnToolProgress = func(string, streamjson.Event) {}
|
||||
}
|
||||
result, err := executeToolCall(context.Background(),
|
||||
registry,
|
||||
ToolCall{ID: "call_1", Name: tool.Name(), Arguments: "{}"},
|
||||
PermissionModeUnsafe,
|
||||
options)
|
||||
if err != nil {
|
||||
t.Fatalf("executeToolCall: %v", err)
|
||||
}
|
||||
if result.Status == "" {
|
||||
t.Fatalf("probe produced no result")
|
||||
}
|
||||
return *got
|
||||
}
|
||||
|
||||
// THE PARITY OBLIGATION for un-gating the progress path.
|
||||
//
|
||||
// The relationship asserted here is EQUALITY, per RULES.md §3: a tool's
|
||||
// progress wiring before and after this change must be the same for every tool
|
||||
// that already had it and every tool that already lacked it. Only a tool that
|
||||
// newly DECLARES the interface may change.
|
||||
//
|
||||
// The name gate is gone, so the guarantee cannot be "Task still works" — it has
|
||||
// to be stated in terms of the declaration, which is what these cases do.
|
||||
func TestProgressCallbackFollowsTheDeclarationNotTheName(t *testing.T) {
|
||||
t.Run("a declaring tool receives it (Task's behaviour, preserved)", func(t *testing.T) {
|
||||
var got bool
|
||||
// Named something other than "Task" ON PURPOSE: under the old name gate
|
||||
// this case failed, which is the whole point of the change.
|
||||
if !runProbe(t, &progressProbeTool{name: "spawner", streams: true, got: &got}, &got, true) {
|
||||
t.Fatal("a tool declaring StreamsChildProgress must receive the callback")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a tool named Task that does NOT declare gets nothing", func(t *testing.T) {
|
||||
var got bool
|
||||
// The inverse of the old behaviour, and the reason a name is the wrong
|
||||
// key: identity is not capability.
|
||||
if runProbe(t, &silentProbeTool{name: "Task", got: &got}, &got, true) {
|
||||
t.Fatal("the callback must follow the declaration, not the name")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a non-declaring tool receives nothing (every other tool, unchanged)", func(t *testing.T) {
|
||||
var got bool
|
||||
if runProbe(t, &silentProbeTool{name: "read_file", got: &got}, &got, true) {
|
||||
t.Fatal("un-gating must not start handing a callback to tools that never had one")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a declaring tool that answers false gets nothing", func(t *testing.T) {
|
||||
var got bool
|
||||
if runProbe(t, &progressProbeTool{name: "spawner", streams: false, got: &got}, &got, true) {
|
||||
t.Fatal("StreamsChildProgress() == false must be honoured")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("no sink means no callback, whatever the tool declares", func(t *testing.T) {
|
||||
var got bool
|
||||
if runProbe(t, &progressProbeTool{name: "spawner", streams: true, got: &got}, &got, false) {
|
||||
t.Fatal("without OnToolProgress there is nothing to forward to")
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -2,7 +2,6 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
@@ -40,15 +39,3 @@ func parsePreservedState(summaryContent string) (string, []skillEntry) {
|
||||
func partitionTools(registry *tools.Registry, permissionMode PermissionMode, options Options, loaded map[string]bool) ([]zeroruntime.ToolDefinition, string) {
|
||||
return partitionToolsCached(registry, permissionMode, options, loaded, nil)
|
||||
}
|
||||
|
||||
// Phase 2 additivity-proof seam. The identity test uses the REAL orchestrate
|
||||
// tool rather than a stub, so it exercises the actual Deferred() contract that
|
||||
// enforces the posture-off constraint. internal/specialist does not import
|
||||
// internal/agent, so this direction creates no cycle.
|
||||
const phase2ToolName = specialist.OrchestrateToolName
|
||||
|
||||
// registerPhase2ToolForTest registers the real tool with the posture OFF, which
|
||||
// is the condition the identity test is about.
|
||||
func registerPhase2ToolForTest(registry *tools.Registry) {
|
||||
registry.Register(&specialist.OrchestrateTool{PostureActive: func() bool { return false }})
|
||||
}
|
||||
+16
-627
@@ -1,7 +1,6 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
@@ -56,22 +55,6 @@ const (
|
||||
// error, so this only affects true same-error loops.
|
||||
toolFailureStopAt = 6
|
||||
|
||||
// toolFailureAnyErrorStopAt halts a tool that keeps failing with DIFFERENT
|
||||
// errors. The streak above cannot see that case by construction: a changed
|
||||
// signature rebuilds the record at 1, so a tool whose error text varies every
|
||||
// call never reaches toolFailureStopAt however long it loops.
|
||||
//
|
||||
// That is not hypothetical. A headless run made 384 denied calls over 26
|
||||
// minutes without tripping a halt set to 6, because each denial named a
|
||||
// different path. Keying denials on their category (see observeToolResult)
|
||||
// fixes that case; this bound covers every error with no category to key on.
|
||||
//
|
||||
// Deliberately well above toolFailureStopAt. A model iterating on a genuinely
|
||||
// tricky edit legitimately fails several times with different errors while it
|
||||
// converges, and this must not cut those runs short — the same reasoning that
|
||||
// moved toolFailureStopAt from 4 to 6. Both counters reset on success.
|
||||
toolFailureAnyErrorStopAt = 12
|
||||
|
||||
// maxContinueNudges bounds how many times the headless completion gate
|
||||
// (Options.RequireCompletionSignal) re-prompts a model that stopped without a
|
||||
// tool call while work clearly remained. Once spent, the run finalizes as
|
||||
@@ -185,24 +168,7 @@ var selfReportPhrases = []string{
|
||||
var inabilityStems = []string{
|
||||
"i cannot ", "i can't ", "i can not ", "i could not ", "i couldn't ",
|
||||
"i am unable to", "i'm unable to", "i was unable to", "i wasn't able to",
|
||||
"i was not able to", "i do not have", "i don't have",
|
||||
"we are unable to", "we were unable to",
|
||||
// THE SUBJECTLESS STEM IS KEPT, and the heading it fired on is handled where
|
||||
// the heading is, not by deleting the stem.
|
||||
//
|
||||
// It was removed once because a completed audit's own section heading —
|
||||
// "**Unable to verify (1):** - MCP #3 claim was truncated" — reads as an
|
||||
// admission. But it is the ONLY stem that catches an admission with no
|
||||
// first-person subject, and removing it lost every one of those:
|
||||
//
|
||||
// "Unable to complete the task; the build never succeeded."
|
||||
// "The agent was unable to finish the migration."
|
||||
// "Unable to verify the fix, so the change is unverified."
|
||||
//
|
||||
// None name "i" or "we", so no other stem sees them. That traded one false
|
||||
// positive for three false negatives, in the direction this guard exists to
|
||||
// prevent. countedLabelSentence drops the heading shape instead.
|
||||
"unable to ",
|
||||
"i was not able to", "i do not have", "i don't have", "unable to ",
|
||||
"without being able to",
|
||||
}
|
||||
|
||||
@@ -213,19 +179,6 @@ var inabilityStems = []string{
|
||||
var successNegationTails = []string{
|
||||
"find any", "found any", "find a ", "see any", "detect any", "identify any",
|
||||
"reproduce", "spot any", "locate any",
|
||||
// A NEGATIVE SEARCH RESULT IS THE ANSWER, not a failure to produce one.
|
||||
//
|
||||
// The list above already encodes this — "I could not find any remaining
|
||||
// issues" is success — but only for the "any" phrasings. A finder reporting
|
||||
// "I could NOT find where AllowManifestToolAutoApproval is set to true in
|
||||
// production code" was marked INCOMPLETE after 53 tool calls and a 19,145
|
||||
// character audit, for doing precisely the job it was given: establishing
|
||||
// that something is not there.
|
||||
"find where", "find the", "find it", "find that", "find this",
|
||||
"found where", "found the",
|
||||
"locate where", "locate the", "locate it",
|
||||
"determine where", "identify where", "see where",
|
||||
"reproduce ", "confirm any", "observe any",
|
||||
}
|
||||
|
||||
// narrativeMarkers flag a sentence as RETELLING a past exchange rather than
|
||||
@@ -294,401 +247,11 @@ func admissionSentences(lower string) []string {
|
||||
// retells a past exchange (narrativeMarkers) is skipped entirely — an admission
|
||||
// must be the model's own report about the CURRENT objective, not general
|
||||
// language that merely resembles one.
|
||||
// toolGrantMarkers flag a sentence as reporting WHICH TOOLS this run was given,
|
||||
// not whether the work was done.
|
||||
//
|
||||
// A read-only plan task is SUPPOSED to say this. One wrote "I don't have an
|
||||
// update_plan tool available in this specialist context (only read-only
|
||||
// exploration tools were provided)" and then delivered the complete answer —
|
||||
// helper name, file, line 214, full source — and was marked INCOMPLETE on the
|
||||
// "i don't have" stem. The prompt asks tasks to name their limits plainly; a
|
||||
// detector that punishes exactly that teaches the opposite.
|
||||
//
|
||||
// NARROW ON PURPOSE: the sentence must name a TOOL or a GRANT. "I do not have
|
||||
// enough evidence" is still an admission and still fires.
|
||||
// NARROWED AFTER AN AUDIT OF THIS VERY FIX. The first version listed a bare
|
||||
// " tool", which exempts any inability sentence that merely mentions one —
|
||||
// measured at 5/5 on ordinary phrasings:
|
||||
//
|
||||
// "I cannot run the build tool, so the change is unverified"
|
||||
// "I could not use the migration tool and the data is untouched"
|
||||
// "I was unable to invoke the formatting tool on the output"
|
||||
//
|
||||
// Those are genuine admissions, and silently exempting them is the WORSE
|
||||
// direction: a false positive costs a re-run, a false negative reports
|
||||
// unfinished work as done. The markers now have to be about what the run WAS
|
||||
// GIVEN, not about a tool being mentioned at all.
|
||||
var toolGrantMarkers = []string{
|
||||
"tool available", "tools available", "no such tool", "not available in this",
|
||||
"read-only tools", "read only tools", "only read-only", "only read only",
|
||||
"tools were provided", "tools were given", "toolset provided",
|
||||
"in this specialist context", "in this context only",
|
||||
"is not in my toolset", "not in my toolset", "not in this toolset",
|
||||
}
|
||||
|
||||
// objectiveFailureMarkers name the OBJECTIVE rather than a capability. A
|
||||
// sentence carrying one is about whether the job got done, so the tool-grant
|
||||
// exemption above does not apply to it however many tools it mentions.
|
||||
// VERB-ANCHORED, not bare nouns. "this task" alone was too crude: a task that
|
||||
// finished wrote "so i could not record a plan; the task is a single
|
||||
// read-and-report step and is now complete" — it names the task in order to
|
||||
// report SUCCESS, and a bare-noun override read that as failure. The marker has
|
||||
// to be the objective NOT BEING DONE, which needs the verb.
|
||||
var objectiveFailureMarkers = []string{
|
||||
"complete this task", "complete the task", "completing this task", "completing the task",
|
||||
"finish this task", "finish the task", "finishing this task",
|
||||
"complete it", "completing it", "finish it", "finishing it",
|
||||
// The objective and the assignment need the verb too, for the same reason
|
||||
// "this task" did. "the objective is met" and "the assignment is complete"
|
||||
// name the objective in order to report SUCCESS, and a bare noun read both
|
||||
// as failure — so a finished answer carrying a tool caveat was told it had
|
||||
// not finished, which is the worst thing this detector can do.
|
||||
"complete the objective", "completing the objective",
|
||||
"finish the objective", "finishing the objective",
|
||||
"meet the objective", "meeting the objective", "achieve the objective",
|
||||
"complete the assignment", "completing the assignment",
|
||||
"finish the assignment", "finishing the assignment",
|
||||
"as requested", "what was asked",
|
||||
"do this task", "perform this task", "carry out this task",
|
||||
}
|
||||
|
||||
// blockedWorkMarkers are what turns an absence-establishing sentence back into
|
||||
// an admission of failure.
|
||||
//
|
||||
// MEASURED, NOT ARGUED. The successNegationTails list above exists so a finder
|
||||
// reporting "I could not find where X is set" is not marked incomplete for doing
|
||||
// its job. But the stems it added — "find the", "reproduce ", "confirm any",
|
||||
// "observe any" — also head the most ordinary way of admitting defeat, and the
|
||||
// allowance fired on the whole sentence regardless of how it ended. Measured
|
||||
// against eleven genuine admissions, TEN passed the detector undetected:
|
||||
//
|
||||
// "I could not reproduce the crash, so the fix is unverified."
|
||||
// "I could not find the root cause; someone else will need to pick this up."
|
||||
// "I could not locate the source of the regression and have run out of ideas."
|
||||
//
|
||||
// That is the guard's entire purpose defeated — it is the last thing between a
|
||||
// stalled run and a report that reads like success.
|
||||
//
|
||||
// So the allowance now yields when the sentence ALSO says the work is blocked.
|
||||
// "I could not find where X is set in production code" still passes, because
|
||||
// nothing in it claims the objective was left undone; the three above do not.
|
||||
// strongAbsenceTails are the allowance tails carrying an explicit "any": the
|
||||
// model asserting it looked and found NOTHING. That is a finding whatever the
|
||||
// rest of the sentence says, so blockedWorkMarkers does not override them.
|
||||
var strongAbsenceTails = []string{
|
||||
"find any", "found any", "see any", "detect any", "identify any",
|
||||
"spot any", "locate any", "confirm any", "observe any",
|
||||
// The OBSERVATION family. Looking for a failure and not producing one is the
|
||||
// same kind of result as looking for an issue and not finding one, and
|
||||
// leaving these out meant "I could not reproduce any failure in the parser"
|
||||
// was not a strong absence — so an unrelated blocked statement in the next
|
||||
// sentence flipped a clean negative result into an admission.
|
||||
"reproduce any", "trigger any", "produce any", "hit any",
|
||||
"encounter any", "provoke any", "surface any", "measure any",
|
||||
}
|
||||
|
||||
// strongAbsenceObjects are the things whose ABSENCE IS THE RESULT: you go
|
||||
// looking for them precisely so you can report there are none, and finding none
|
||||
// is the work succeeding.
|
||||
//
|
||||
// WHAT FOLLOWS "any" DECIDES, and treating every "find any" as success was too
|
||||
// broad:
|
||||
//
|
||||
// "I could not find any remaining issues" -> a finding, the search succeeded
|
||||
// "I could not find any solution" -> an admission, the work did not
|
||||
//
|
||||
// Both carry the explicit "any". Only the object separates them, so only the
|
||||
// object can classify them.
|
||||
//
|
||||
// AN ALLOW-LIST, because this grants the exemption. A deny-list of deliverables
|
||||
// would have to anticipate every noun a model might reach for, and everything
|
||||
// forgotten would be waved through as success — the failure direction this
|
||||
// detector exists to prevent. An unrecognised object is simply not strong, which
|
||||
// leaves the sentence to the ordinary blocked-work handling rather than
|
||||
// flagging it outright.
|
||||
var strongAbsenceObjects = []string{
|
||||
"issue", "issues", "problem", "problems", "bug", "bugs", "defect", "defects",
|
||||
"error", "errors", "failure", "failures", "regression", "regressions",
|
||||
"evidence", "example", "examples", "occurrence", "occurrences",
|
||||
"instance", "instances", "reference", "references", "match", "matches",
|
||||
"caller", "callers", "usage", "usages", "use", "uses", "case", "cases",
|
||||
"vulnerability", "vulnerabilities", "leak", "leaks", "race", "races",
|
||||
"sign", "signs", "trace", "traces", "mention", "mentions", "difference", "differences",
|
||||
"blocker", "blockers", "gap", "gaps", "omission", "omissions", "discrepancy", "discrepancies",
|
||||
"conflict", "conflicts", "violation", "violations", "warning", "warnings",
|
||||
"call", "calls", "site", "sites", "callsite", "callsites", "consumer", "consumers",
|
||||
"dependency", "dependencies", "user", "users", "path", "paths",
|
||||
}
|
||||
|
||||
// absenceQualifiers sit between "any" and the object without changing it.
|
||||
var absenceQualifiers = []string{
|
||||
"remaining", "other", "further", "more", "additional", "obvious", "such",
|
||||
"outstanding", "new", "existing", "actual", "real", "clear", "direct",
|
||||
}
|
||||
|
||||
// strongAbsence reports whether tail is an "any"-family absence whose OBJECT
|
||||
// makes finding nothing the result rather than the shortfall.
|
||||
func strongAbsence(tail string) bool {
|
||||
for _, prefix := range strongAbsenceTails {
|
||||
if !strings.HasPrefix(tail, prefix) {
|
||||
continue
|
||||
}
|
||||
rest := strings.TrimSpace(tail[len(prefix):])
|
||||
for trimmed := true; trimmed; {
|
||||
trimmed = false
|
||||
for _, qualifier := range absenceQualifiers {
|
||||
if word, remainder, ok := cutFirstWord(rest); ok && word == qualifier {
|
||||
rest, trimmed = remainder, true
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
word, _, ok := cutFirstWord(rest)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
for _, object := range strongAbsenceObjects {
|
||||
if word == object {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// cutFirstWord returns the first bare word of text, lowercased by the caller's
|
||||
// own normalisation, with surrounding punctuation removed.
|
||||
func cutFirstWord(text string) (word string, rest string, ok bool) {
|
||||
text = strings.TrimLeft(text, " \t")
|
||||
end := 0
|
||||
for end < len(text) {
|
||||
c := text[end]
|
||||
if (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' {
|
||||
end++
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
if end == 0 {
|
||||
return "", text, false
|
||||
}
|
||||
return text[:end], text[end:], true
|
||||
}
|
||||
|
||||
var blockedWorkMarkers = []string{
|
||||
"unverified", "not verified", "cannot verify", "could not verify",
|
||||
"someone else", "will need to", "needs someone", "handed off", "hand off",
|
||||
// "someone else" and "will need to" are the two that also appear in
|
||||
// SUCCESSFUL reports, describing somebody else's future work — "I could not
|
||||
// find any remaining issues, though a follow-up will need to cover the
|
||||
// Windows path" is a finding. They are kept because they do carry a genuine
|
||||
// handoff-because-blocked ("someone else will need to pick this up"), and the
|
||||
// strongAbsenceTails exemption above is what stops them flipping a finding.
|
||||
"ran out of", "run out of", "out of time", "in the time available",
|
||||
"stopped there", "stopping here", "gave up", "giving up",
|
||||
"nothing was modified", "no changes were made", "left unchanged", "left undone",
|
||||
"may be inert", "is unresolved", "remains unresolved", "still broken",
|
||||
"so i cannot", "so i could not",
|
||||
// Saying the work is BLOCKED, in as many words. "I could not find the root
|
||||
// cause, so the work is blocked" carries the statement in the same sentence
|
||||
// and still passed, because every marker above names a symptom of being
|
||||
// blocked and none named the thing itself.
|
||||
"is blocked", "are blocked", "remains blocked", "stays blocked", "still blocked",
|
||||
"cannot proceed", "could not proceed", "can not proceed", "unable to proceed",
|
||||
"is unfinished", "remains unfinished", "left unfinished", "still unfinished",
|
||||
"is incomplete", "remains incomplete",
|
||||
"still unresolved", "still unverified",
|
||||
}
|
||||
|
||||
// bareInabilityStems are the two entries above that are STEMS rather than
|
||||
// descriptions of a blocked state. The stem loop below already scans them with
|
||||
// its own tail logic, and a sentence can carry one on its way to reporting
|
||||
// success: "so i could not record a plan; the task is a single read-and-report
|
||||
// step and is now complete" is a finished task that did not need the plan.
|
||||
var bareInabilityStems = []string{"so i cannot", "so i could not"}
|
||||
|
||||
// unambiguousFailureStates say the work is in a bad state, with no reading on
|
||||
// which it is a result.
|
||||
//
|
||||
// AN OBJECT CANNOT OUTRANK AN EXPLICIT STATE. strongAbsence returns true for
|
||||
// "evidence", and that suppressed every blocked-work marker in the sentence — so
|
||||
// "I could not find any evidence supporting the fix, so it remains unverified"
|
||||
// reported success while saying in as many words that the work is unverified.
|
||||
// The absence protection exists for ownership and follow-up wording, where "I
|
||||
// could not find any remaining issues, though a follow-up will need to cover the
|
||||
// Windows path" really is a finding. It was never meant to cover a sentence that
|
||||
// states the outcome.
|
||||
//
|
||||
// So this is the SHORT list: "unverified" and "still broken" have one reading,
|
||||
// while "someone else", "will need to" and "nothing was modified" have two and
|
||||
// stay ambiguous. Same-sentence only — a state in the NEXT sentence may belong
|
||||
// to another subject, which is what the lookahead's topic-shift guard is for.
|
||||
var unambiguousFailureStates = []string{
|
||||
"unverified", "not verified",
|
||||
"is unresolved", "remains unresolved",
|
||||
"still broken",
|
||||
"is blocked", "are blocked", "remains blocked", "stays blocked", "still blocked",
|
||||
"is unfinished", "remains unfinished", "left unfinished", "still unfinished",
|
||||
"still unresolved", "still unverified",
|
||||
"is incomplete", "remains incomplete",
|
||||
"cannot proceed", "could not proceed", "unable to proceed",
|
||||
"gave up", "giving up", "ran out of", "run out of",
|
||||
"left undone",
|
||||
}
|
||||
|
||||
// consequenceBoundaries separate what was looked for from what followed.
|
||||
//
|
||||
// THE STATE HAS TO BE THE OUTCOME, NOT PART OF WHAT WAS NEGATED. Matching
|
||||
// "is unresolved" anywhere in the sentence read the negated PROPOSITION as the
|
||||
// reported result:
|
||||
//
|
||||
// "I could not find any evidence that the issue is unresolved."
|
||||
//
|
||||
// That is a successful negative finding — there is no evidence the issue remains
|
||||
// unresolved — and it was marked incomplete, which is the opposite polarity of
|
||||
// the case the override was added for. What separates them is position: after a
|
||||
// consequence boundary the state is being asserted, inside a "that…" clause it is
|
||||
// the thing being denied.
|
||||
var consequenceBoundaries = []string{", so ", "; ", ", but ", ", therefore", ", leaving ", ", which "}
|
||||
|
||||
// reportedConsequence returns the part of a sentence that states the OUTCOME,
|
||||
// or "" when the sentence never turns to one. Everything before the first
|
||||
// boundary is what was searched for.
|
||||
func reportedConsequence(sentence string) string {
|
||||
earliest := -1
|
||||
width := 0
|
||||
for _, boundary := range consequenceBoundaries {
|
||||
if index := strings.Index(sentence, boundary); index >= 0 && (earliest < 0 || index < earliest) {
|
||||
earliest, width = index, len(boundary)
|
||||
}
|
||||
}
|
||||
if earliest < 0 {
|
||||
return ""
|
||||
}
|
||||
return sentence[earliest+width:]
|
||||
}
|
||||
|
||||
// blockedStateMarkers describe WORK LEFT BLOCKED — unverified, unresolved,
|
||||
// handed off, abandoned for time. Only these break the tool-grant exemption:
|
||||
// they say something about the state the work is in, which a tool caveat cannot
|
||||
// excuse, whereas a bare stem says only that one step did not happen.
|
||||
//
|
||||
// DERIVED, not copied. Two hand-maintained lists that must stay in step drift,
|
||||
// and the drift here would be silent in both directions.
|
||||
var blockedStateMarkers = func() []string {
|
||||
bare := make(map[string]bool, len(bareInabilityStems))
|
||||
for _, stem := range bareInabilityStems {
|
||||
bare[stem] = true
|
||||
}
|
||||
out := make([]string, 0, len(blockedWorkMarkers))
|
||||
for _, marker := range blockedWorkMarkers {
|
||||
if !bare[marker] {
|
||||
out = append(out, marker)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}()
|
||||
|
||||
// carriesTheConsequence reports whether the sentence after an allowance should
|
||||
// be read as that allowance's consequence.
|
||||
//
|
||||
// The lookahead exists because the consequence usually IS the next sentence, but
|
||||
// "usually" is not "always" — a message can turn to something else, and reading
|
||||
// a blocked statement about a different subject as this one's consequence is the
|
||||
// cost of the lookahead. A sentence that announces the change of subject, or
|
||||
// disclaims the thing as out of scope, is taken at its word.
|
||||
//
|
||||
// This does not catch every unrelated follow-on, and deliberately errs toward
|
||||
// reading the next sentence: an admission reported as success is the failure this
|
||||
// guard exists to prevent, and a message that says something is unverified has
|
||||
// said it whether or not it is the same something.
|
||||
func carriesTheConsequence(next string) bool {
|
||||
return !containsAny(next, topicShiftMarkers)
|
||||
}
|
||||
|
||||
// topicShiftMarkers say the message has moved on to something else.
|
||||
var topicShiftMarkers = []string{
|
||||
"separately", "unrelatedly", "unrelated", "as an aside", "aside from",
|
||||
"out of scope", "outside the scope", "not in scope", "for a different",
|
||||
"in a different", "on another", "elsewhere in", "in other news",
|
||||
}
|
||||
|
||||
// countedLabelSentence reports whether a sentence is a markdown LABEL that
|
||||
// introduces and counts a bucket of findings, rather than a claim about the
|
||||
// objective.
|
||||
//
|
||||
// "**Unable to verify (1):** - MCP #3 claim was truncated" is a section heading
|
||||
// in a COMPLETED audit. Read as a sentence it is indistinguishable from an
|
||||
// admission, which is why the subjectless stem was once deleted to silence it.
|
||||
//
|
||||
// NARROW ON PURPOSE: the sentence must BEGIN with the inability phrase, after
|
||||
// markdown emphasis, AND carry a parenthesised count. "Unable to complete the
|
||||
// task; the build never succeeded" begins the same way and has no count, so it
|
||||
// still fires — which is the whole reason this is preferable to dropping the
|
||||
// stem.
|
||||
func countedLabelSentence(sentence string) bool {
|
||||
trimmed := strings.TrimLeft(strings.TrimSpace(sentence), "-*#> \t")
|
||||
if !strings.HasPrefix(trimmed, "unable to ") {
|
||||
return false
|
||||
}
|
||||
return countedLabelSuffix.MatchString(sentence)
|
||||
}
|
||||
|
||||
var countedLabelSuffix = regexp.MustCompile(`\(\s*\d+\s*\)`)
|
||||
|
||||
func selfReportedIncompletion(text string) string {
|
||||
sentences := admissionSentences(strings.ToLower(stripQuoted(text)))
|
||||
for index, sentence := range sentences {
|
||||
// THE CONSEQUENCE IS OFTEN THE NEXT SENTENCE. The blocked-work override
|
||||
// only ever saw the sentence the allowance fired in, so the same
|
||||
// admission escaped or was caught purely on its punctuation:
|
||||
//
|
||||
// "I could not reproduce the crash, so the fix is unverified." caught
|
||||
// "I could not reproduce the crash. The fix is unverified." missed
|
||||
//
|
||||
// A full stop is not a claim that the work finished, and writing the
|
||||
// consequence as its own sentence is how most people write. The
|
||||
// blocked-work question is asked of this sentence AND the one after it;
|
||||
// everything else is still decided on the sentence alone, so a stem in
|
||||
// one sentence cannot be paired with an allowance tail in another.
|
||||
blockedContext := sentence
|
||||
if index+1 < len(sentences) && carriesTheConsequence(sentences[index+1]) {
|
||||
blockedContext += " " + sentences[index+1]
|
||||
}
|
||||
for _, sentence := range admissionSentences(strings.ToLower(stripQuoted(text))) {
|
||||
if containsAny(sentence, narrativeMarkers) {
|
||||
continue
|
||||
}
|
||||
// A counted label is a heading, not a claim about the objective.
|
||||
if countedLabelSentence(sentence) {
|
||||
continue
|
||||
}
|
||||
// A sentence about the tool grant is about CAPABILITY, not about the
|
||||
// objective — UNLESS it also says the task itself could not be done.
|
||||
//
|
||||
// THE OVERRIDE IS NOT OPTIONAL. Without it the exemption swallowed a
|
||||
// genuine failure: "I am unable to complete this task with the current
|
||||
// tool set … Only write_file is enabled … so I cannot inspect the
|
||||
// codebase." That task really did fail, and it mentions tools, so a bare
|
||||
// tool-marker check waved it through. Naming the task is what separates
|
||||
// "I lack a tool I did not need" from "I lack the tools this needed".
|
||||
// BLOCKED WORK BREAKS THE EXEMPTION TOO, not just a named objective. The
|
||||
// override above asks whether the sentence names the task, which a whole
|
||||
// class of real admissions never does:
|
||||
//
|
||||
// "No write tool is available in this context, so the fix is unverified."
|
||||
// "There is no edit tool available here, so the change remains unapplied."
|
||||
// "Write tools are not available in this setup, so the tests were never run."
|
||||
//
|
||||
// Every one mentions tools, none names the objective, and all four were
|
||||
// waved through. The tool caveat is the REASON the work is blocked, not a
|
||||
// reason to stop reading — so a sentence that also says something is
|
||||
// unverified, unapplied or untested goes on to the blocked-work handling
|
||||
// below instead of being exempted here.
|
||||
if containsAny(sentence, toolGrantMarkers) &&
|
||||
!containsAny(sentence, objectiveFailureMarkers) &&
|
||||
!containsAny(sentence, blockedStateMarkers) {
|
||||
continue
|
||||
}
|
||||
for _, phrase := range selfReportPhrases {
|
||||
if strings.Contains(sentence, phrase) {
|
||||
return selfReportReason(phrase)
|
||||
@@ -705,18 +268,7 @@ func selfReportedIncompletion(text string) string {
|
||||
}
|
||||
abs := start + rel
|
||||
tail := strings.TrimSpace(sentence[abs+len(stem):])
|
||||
// The allowance yields to a sentence that also says the work is
|
||||
// blocked: "could not reproduce the crash" is a finding, "could not
|
||||
// reproduce the crash, so the fix is unverified" is an admission,
|
||||
// and the tail prefix alone cannot tell them apart.
|
||||
strong := strongAbsence(tail)
|
||||
// The object decides whether finding nothing is a result; an
|
||||
// explicit failure state in the SAME sentence decides that it is
|
||||
// not, whatever the object.
|
||||
if strong && containsAny(reportedConsequence(sentence), unambiguousFailureStates) {
|
||||
strong = false
|
||||
}
|
||||
if !hasAnyPrefix(tail, successNegationTails) || (!strong && containsAny(blockedContext, blockedWorkMarkers)) {
|
||||
if !hasAnyPrefix(tail, successNegationTails) {
|
||||
return selfReportReason(strings.TrimSpace(stem) + " …")
|
||||
}
|
||||
start = abs + len(stem)
|
||||
@@ -772,91 +324,15 @@ func acceptanceVerificationNudge(objective string) string {
|
||||
const toolFailureHintMarker = "kept failing with the same error"
|
||||
|
||||
type toolFailureRecord struct {
|
||||
count int
|
||||
identity failureIdentity
|
||||
// sawExecuted and sawRefused are the AGGREGATE over the whole streak, not
|
||||
// the current identity. The content-blind bound needs them because it fires
|
||||
// exactly when no identity repeated, so the last one is unrepresentative.
|
||||
sawExecuted bool
|
||||
sawRefused bool
|
||||
hintShown bool
|
||||
// anyErrorCount counts consecutive failures of this tool REGARDLESS of the
|
||||
// error, and is cleared only by a success. count above restarts whenever the
|
||||
// signature changes, which is exactly what a varying error message defeats;
|
||||
// this one cannot be reset by changing the text.
|
||||
anyErrorCount int
|
||||
count int
|
||||
errSig string
|
||||
hintShown bool
|
||||
}
|
||||
|
||||
// failureKind separates a failure the TOOL produced from one POLICY produced.
|
||||
//
|
||||
// The two used to share one string namespace, with provenance recovered
|
||||
// afterwards by testing that string for a "denial:" prefix. Trusted provenance
|
||||
// and untrusted content in one namespace is a namespace the untrusted side can
|
||||
// write into: a command that printed exactly "denial:permission_denied" and
|
||||
// exited non-zero acquired the identity of a real permission refusal, merged
|
||||
// its streak with one, and had the run report it as refused although it ran
|
||||
// every time. The kind is carried as its own field so no output can spell it.
|
||||
type failureKind uint8
|
||||
|
||||
const (
|
||||
// The zero value: no failure recorded.
|
||||
_ failureKind = iota
|
||||
// failureKindExecuted: the tool ran and returned an error. The key is the
|
||||
// normalized error signature, which is tool-controlled text.
|
||||
failureKindExecuted
|
||||
// failureKindRefused: policy refused the call before the tool ran. The key is
|
||||
// the denial category, a small closed enum the loop sets.
|
||||
failureKindRefused
|
||||
)
|
||||
|
||||
// failureIdentity is what the same-identity streak counts. Comparing the pair
|
||||
// means an executed error can never equal a refusal however it is spelled.
|
||||
type failureIdentity struct {
|
||||
kind failureKind
|
||||
key string
|
||||
}
|
||||
|
||||
// toolFailureCause is why the guard stopped, decided where the facts are rather
|
||||
// than reconstructed by the caller from overlapping booleans.
|
||||
type toolFailureCause uint8
|
||||
|
||||
const (
|
||||
// The zero value: the guard did not stop.
|
||||
_ toolFailureCause = iota
|
||||
// toolFailureCauseSameError: the same executed error signature, six times.
|
||||
toolFailureCauseSameError
|
||||
// toolFailureCauseSameRefusal: the same denial category, six times. The prose
|
||||
// differs on every call because it names what was refused, so this is NOT the
|
||||
// same-error case.
|
||||
toolFailureCauseSameRefusal
|
||||
// toolFailureCauseVariedExecuted: the content-blind bound, and every failure
|
||||
// was the tool executing and failing.
|
||||
toolFailureCauseVariedExecuted
|
||||
// toolFailureCauseVariedRefused: the content-blind bound, and every failure
|
||||
// was a policy refusal. Alternating two categories reaches twelve without
|
||||
// either reaching six, and the tool never ran once.
|
||||
toolFailureCauseVariedRefused
|
||||
// toolFailureCauseVariedMixed: the content-blind bound over both kinds.
|
||||
// Named deliberately rather than left to whichever field a switch tested
|
||||
// first.
|
||||
toolFailureCauseVariedMixed
|
||||
)
|
||||
|
||||
type toolFailureOutcome struct {
|
||||
InjectHint bool
|
||||
Stop bool
|
||||
Count int
|
||||
// Cause is set only when Stop is. It carries the aggregate fact about the
|
||||
// sequence that tripped a bound, which the last record alone cannot supply:
|
||||
// the content-blind bound is reached precisely when no single identity
|
||||
// repeated enough, so the identity present at the end says nothing about the
|
||||
// eleven before it.
|
||||
Cause toolFailureCause
|
||||
}
|
||||
|
||||
// Refused reports whether policy, rather than the tool, produced the failures.
|
||||
func (cause toolFailureCause) Refused() bool {
|
||||
return cause == toolFailureCauseSameRefusal || cause == toolFailureCauseVariedRefused
|
||||
}
|
||||
|
||||
// errorSignature normalizes a tool error to a short, comparable signature so
|
||||
@@ -882,47 +358,12 @@ func toolFailureHint(toolName, schemaJSON, errOutput string) string {
|
||||
|
||||
// toolFailureStopAnswer is the final answer when the repeated-failure guard halts
|
||||
// a run.
|
||||
func toolFailureStopAnswer(toolName string, count int, cause toolFailureCause) string {
|
||||
// Each branch claims only what its counter established, and the mixed case is
|
||||
// spelled out rather than decided by whichever field a switch happened to
|
||||
// test first. The wording used to overclaim in both directions: the
|
||||
// content-blind bound said every failure differed, which it does not track,
|
||||
// and it described a run of policy refusals as the tool failing, when the
|
||||
// tool had not run at all.
|
||||
verb, tail := " tool failed ", " times in a row with the same error, "
|
||||
switch cause {
|
||||
case toolFailureCauseSameRefusal:
|
||||
verb, tail = " tool was refused ", " times in a row, "
|
||||
case toolFailureCauseVariedExecuted:
|
||||
tail = " times in a row with varying errors, "
|
||||
case toolFailureCauseVariedRefused:
|
||||
verb, tail = " tool was refused ", " times in a row for different reasons, "
|
||||
case toolFailureCauseVariedMixed:
|
||||
verb, tail = " tool failed or was refused ", " times in a row, "
|
||||
}
|
||||
return "Agent stopped: the `" + toolName + "`" + verb + strconv.Itoa(count) +
|
||||
tail + "so I halted instead of looping further. " +
|
||||
func toolFailureStopAnswer(toolName string, count int) string {
|
||||
return "Agent stopped: the `" + toolName + "` tool failed " + strconv.Itoa(count) +
|
||||
" times in a row with the same error, so I halted instead of looping further. " +
|
||||
"Please check the request or adjust the tool arguments."
|
||||
}
|
||||
|
||||
// toolFailureIncompleteReason describes a guard halt for the headless
|
||||
// completion gate. A guard halt is BY DEFINITION unfinished work: the run
|
||||
// stopped because a tool would not stop failing, not because the task was done.
|
||||
//
|
||||
// It makes the same claim as toolFailureStopAnswer for the same halt, including
|
||||
// the mixed case: a headless caller reads this reason and an interactive one
|
||||
// reads the stop answer, and the two must not describe one halt differently.
|
||||
// Reported by @jatmn.
|
||||
func toolFailureIncompleteReason(toolName string, cause toolFailureCause) string {
|
||||
switch {
|
||||
case cause.Refused():
|
||||
return "halted after `" + toolName + "` was refused repeatedly, without completing the task"
|
||||
case cause == toolFailureCauseVariedMixed:
|
||||
return "halted after `" + toolName + "` failed or was refused repeatedly, without completing the task"
|
||||
}
|
||||
return "halted after `" + toolName + "` failed repeatedly, without completing the task"
|
||||
}
|
||||
|
||||
// The no-output stop answer is assembled from these fixed parts (only the turn
|
||||
// count varies). IsNoProgressStop matches all three so a legitimate message that
|
||||
// merely quotes the marker substring is not misclassified as a failed empty run.
|
||||
@@ -1031,68 +472,28 @@ func newGuardState() *guardState {
|
||||
// observeToolResult tracks repeated identical failures of a tool. A successful
|
||||
// result clears that tool's failure streak. Returns whether to inject a one-shot
|
||||
// corrective hint and/or stop the run.
|
||||
// hintable is separate from failed on purpose. A categorized denial MUST count
|
||||
// toward the streaks — that is the whole point of keying on the category — but a
|
||||
// schema hint is the wrong response to a policy refusal: the call shape is fine,
|
||||
// the answer was no. Collapsing the two is what made the earlier version of this
|
||||
// fix a no-op, since the caller could only pass a flag that excluded denials.
|
||||
func (state *guardState) observeToolResult(name string, failed bool, hintable bool, output string, denial DenialCategory) toolFailureOutcome {
|
||||
func (state *guardState) observeToolResult(name string, failed bool, output string) toolFailureOutcome {
|
||||
if state.toolFailures == nil {
|
||||
state.toolFailures = map[string]*toolFailureRecord{}
|
||||
}
|
||||
if !failed {
|
||||
delete(state.toolFailures, name) // success resets both counters
|
||||
delete(state.toolFailures, name) // success resets the streak
|
||||
return toolFailureOutcome{}
|
||||
}
|
||||
// A denial keys on its CATEGORY, not its prose. The message embeds the path
|
||||
// or command that was refused, so it differs on every call while describing
|
||||
// the same unchanging refusal — which rebuilt the record at 1 each time and
|
||||
// let a denied tool loop indefinitely under a halt set to 6. The category is
|
||||
// a small closed enum the loop already sets on the result.
|
||||
identity := failureIdentity{kind: failureKindExecuted, key: errorSignature(output)}
|
||||
if denial != DenialNone {
|
||||
identity = failureIdentity{kind: failureKindRefused, key: string(denial)}
|
||||
}
|
||||
sig := errorSignature(output)
|
||||
record := state.toolFailures[name]
|
||||
if record == nil {
|
||||
record = &toolFailureRecord{identity: identity}
|
||||
if record == nil || record.errSig != sig {
|
||||
record = &toolFailureRecord{count: 1, errSig: sig}
|
||||
state.toolFailures[name] = record
|
||||
}
|
||||
if record.identity != identity {
|
||||
// A different failure restarts the same-identity streak but NOT the
|
||||
// content-blind one: changing how a tool fails is not progress.
|
||||
record.count = 1
|
||||
record.identity = identity
|
||||
record.hintShown = false
|
||||
} else {
|
||||
record.count++
|
||||
}
|
||||
record.anyErrorCount++
|
||||
if identity.kind == failureKindRefused {
|
||||
record.sawRefused = true
|
||||
} else {
|
||||
record.sawExecuted = true
|
||||
}
|
||||
|
||||
outcome := toolFailureOutcome{Count: record.count}
|
||||
switch {
|
||||
case record.count >= toolFailureStopAt:
|
||||
if record.count >= toolFailureStopAt {
|
||||
outcome.Stop = true
|
||||
outcome.Cause = toolFailureCauseSameError
|
||||
if identity.kind == failureKindRefused {
|
||||
outcome.Cause = toolFailureCauseSameRefusal
|
||||
}
|
||||
return outcome
|
||||
case record.anyErrorCount >= toolFailureAnyErrorStopAt:
|
||||
// Report the counter that actually tripped. record.count is the
|
||||
// same-identity streak and is often 1 here, which would describe a tool
|
||||
// that failed a dozen different ways as having failed once.
|
||||
outcome.Stop = true
|
||||
outcome.Count = record.anyErrorCount
|
||||
outcome.Cause = record.variedCause()
|
||||
return outcome
|
||||
}
|
||||
if hintable && record.count >= toolFailureHintAt && !record.hintShown {
|
||||
if record.count >= toolFailureHintAt && !record.hintShown {
|
||||
record.hintShown = true
|
||||
outcome.InjectHint = true
|
||||
}
|
||||
@@ -1202,15 +603,3 @@ func (state *guardState) planReminder(turn int) string {
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
// variedCause classifies a content-blind stop by what the whole streak held.
|
||||
func (record *toolFailureRecord) variedCause() toolFailureCause {
|
||||
switch {
|
||||
case record.sawRefused && record.sawExecuted:
|
||||
return toolFailureCauseVariedMixed
|
||||
case record.sawRefused:
|
||||
return toolFailureCauseVariedRefused
|
||||
default:
|
||||
return toolFailureCauseVariedExecuted
|
||||
}
|
||||
}
|
||||
@@ -1,458 +0,0 @@
|
||||
package agent
|
||||
|
||||
import "testing"
|
||||
|
||||
// FOUR COMPLETED TASKS WERE MARKED INCOMPLETE, and the sentences below are
|
||||
// verbatim from those sessions.
|
||||
//
|
||||
// The detector reads message text — the one place in this feature that does —
|
||||
// and it is supposed to catch a model admitting it did not do the job. Instead
|
||||
// it caught four tasks that HAD done the job and were honestly naming a limit,
|
||||
// which is exactly what the plan-task prompt asks them to do. Two of them had
|
||||
// made 53 and 60 tool calls; one had written files.
|
||||
//
|
||||
// The cost of a false positive here is not cosmetic: the task is reported
|
||||
// failed, retried on another model, and rendered red in the plan panel.
|
||||
func TestAnHonestCaveatInsideDeliveredWorkIsNotAnAdmission(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
text string
|
||||
}{
|
||||
{
|
||||
// 53 tool calls, a 19,145-character audit. Establishing that
|
||||
// something is NOT there is the finding, not a failure to find one.
|
||||
name: "a negative search result is the answer",
|
||||
text: "I traced every code path where repo-committed files influence tool approval. " +
|
||||
"I could NOT find where `AllowManifestToolAutoApproval` is set to true in production code paths.",
|
||||
},
|
||||
{
|
||||
// A completed verification task categorising its own findings. The
|
||||
// bare "unable to" stem had no first-person subject and fired on a
|
||||
// markdown heading.
|
||||
name: "a report section heading",
|
||||
text: "## Verification Table\n\n**Verified (4):** all confirmed against source.\n" +
|
||||
"**Unable to verify (1):** - MCP #3 claim was truncated in the input.",
|
||||
},
|
||||
{
|
||||
// Delivered helper name, file, line 214 and full source. The task was
|
||||
// read-only by design and said so.
|
||||
name: "a statement about the tool grant",
|
||||
text: "I don't have an `update_plan` tool available in this specialist context " +
|
||||
"(only read-only exploration tools were provided). " +
|
||||
"The task is a single read-only assessment and I've already gathered all the needed evidence.",
|
||||
},
|
||||
{
|
||||
name: "read-only tools named plainly",
|
||||
text: "I did not run the tests because my tools are read-only, " +
|
||||
"so I can report only what the code and tests say statically.",
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if reason := selfReportedIncompletion(tc.text); reason != "" {
|
||||
t.Fatalf("a completed task was marked incomplete: %s", reason)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// AND THE DETECTOR MUST STILL BITE. Widening it to stop punishing honesty must
|
||||
// not turn it off — these are the admissions it exists for, several of them
|
||||
// verbatim from the same set of sessions.
|
||||
func TestGenuineAdmissionsAreStillCaught(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
text string
|
||||
}{
|
||||
{
|
||||
name: "the measured workspace failure",
|
||||
text: "I cannot complete this task. The target directory `/Users/kratos/zm-lab/pkg/execprofile` " +
|
||||
"is outside the workspace boundary.",
|
||||
},
|
||||
{
|
||||
name: "an empty worktree",
|
||||
text: "This task cannot be completed in the current workspace because the relevant source tree is absent. " +
|
||||
"I cannot find BuildFinalResult anywhere.",
|
||||
},
|
||||
{
|
||||
name: "guessing",
|
||||
text: "I guessed at the retry semantics because the code was not reachable.",
|
||||
},
|
||||
{
|
||||
name: "fabrication",
|
||||
text: "I fabricated the line numbers to fill the table.",
|
||||
},
|
||||
{
|
||||
name: "stated doubt about the result",
|
||||
text: "The patch may not be correct — I did not run it.",
|
||||
},
|
||||
{
|
||||
name: "not enough evidence is still an admission",
|
||||
text: "I do not have enough evidence to answer the question.",
|
||||
},
|
||||
{
|
||||
name: "first-person unable, with a subject",
|
||||
text: "I was unable to determine which branch actually runs.",
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if reason := selfReportedIncompletion(tc.text); reason == "" {
|
||||
t.Fatalf("a real admission was missed: %q", tc.text)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The tool-grant exemption is NARROW. A sentence has to be about tools or the
|
||||
// grant; "not enough evidence" is not, and must keep firing.
|
||||
func TestTheToolGrantExemptionDoesNotSwallowRealAdmissions(t *testing.T) {
|
||||
if reason := selfReportedIncompletion("I could not verify the claim and I do not have enough evidence."); reason == "" {
|
||||
t.Fatal("the tool-grant exemption swallowed an evidence admission")
|
||||
}
|
||||
if reason := selfReportedIncompletion("I don't have the file contents, so the answer is a guess."); reason == "" {
|
||||
t.Fatal("an admission about missing content was exempted as a tool statement")
|
||||
}
|
||||
}
|
||||
|
||||
// THE EXEMPTION MUST NOT SWALLOW A TASK THAT REALLY COULD NOT BE DONE.
|
||||
//
|
||||
// Verbatim from a session the first version of this fix regressed: the task
|
||||
// mentions tools, so a bare tool-marker check waved it through — but it names
|
||||
// the TASK as the thing that failed, which is the whole distinction.
|
||||
func TestATaskThatCouldNotBeDoneForLackOfToolsStillFails(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
text string
|
||||
}{
|
||||
{
|
||||
name: "the measured regression",
|
||||
text: "I am unable to complete this task with the current tool set. " +
|
||||
"Only `write_file` is enabled, so I cannot inspect the codebase to collect scan findings.",
|
||||
},
|
||||
{
|
||||
name: "named differently",
|
||||
text: "I could not finish the objective because no read tools were provided.",
|
||||
},
|
||||
{
|
||||
name: "cannot complete it",
|
||||
text: "I don't have the tools to complete it as requested.",
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if reason := selfReportedIncompletion(tc.text); reason == "" {
|
||||
t.Fatalf("a task that genuinely could not be done was passed as complete: %q", tc.text)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// And the exemption must still work for the case it exists for: a tool that
|
||||
// was not needed, alongside delivered work.
|
||||
exempt := "I don't have an `update_plan` tool available in this specialist context " +
|
||||
"(only read-only exploration tools were provided)."
|
||||
if reason := selfReportedIncompletion(exempt); reason != "" {
|
||||
t.Fatalf("the override broke the exemption it guards: %s", reason)
|
||||
}
|
||||
}
|
||||
|
||||
// NAMING THE TASK TO REPORT SUCCESS IS NOT NAMING IT TO REPORT FAILURE.
|
||||
//
|
||||
// Verbatim from the session that the first override regressed. A task that had
|
||||
// finished wrote a footnote about a tool it did not need, in the same sentence
|
||||
// as its own completion statement — and a bare-noun override read "the task" as
|
||||
// an admission. The marker has to be the objective NOT being done.
|
||||
func TestNamingTheTaskWhileReportingSuccessIsNotAnAdmission(t *testing.T) {
|
||||
done := "- **note:** the update_plan tool is not available in my current toolset " +
|
||||
"(only read-only file tools), so i could not record a plan; " +
|
||||
"the task is a single read-and-report step and is now complete"
|
||||
if reason := selfReportedIncompletion(done); reason != "" {
|
||||
t.Fatalf("a finished task was marked incomplete for saying so: %s", reason)
|
||||
}
|
||||
// The failure form, which must still fire, differs only in the verb.
|
||||
failed := "the update_plan tool is not available, so i could not complete this task"
|
||||
if reason := selfReportedIncompletion(failed); reason == "" {
|
||||
t.Fatal("a task that could not be completed was passed as complete")
|
||||
}
|
||||
}
|
||||
|
||||
// AN INABILITY THAT MERELY MENTIONS A TOOL IS STILL AN ADMISSION.
|
||||
//
|
||||
// THE AUDIT FINDING THIS PINS. The first version of the exemption listed a bare
|
||||
// " tool", so any inability sentence mentioning one escaped — measured at 5/5 on
|
||||
// ordinary phrasings. That is the worse direction: a false positive costs a
|
||||
// re-run, a false negative reports unfinished work as done.
|
||||
func TestMentioningAToolDoesNotExemptAnAdmission(t *testing.T) {
|
||||
for _, text := range []string{
|
||||
"I cannot run the build tool, so the change is unverified",
|
||||
"I could not use the migration tool and the data is untouched",
|
||||
"I was unable to invoke the formatting tool on the output",
|
||||
"I don't have a working compiler tool here",
|
||||
"I cannot finish because the deploy tools are broken",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(text); reason == "" {
|
||||
t.Errorf("a genuine admission was silently exempted: %q", text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// And the exemption still covers what it was built for: a run stating which
|
||||
// tools it was GIVEN, alongside delivered work.
|
||||
func TestTheGrantExemptionStillCoversWhatItWasBuiltFor(t *testing.T) {
|
||||
for _, text := range []string{
|
||||
"I don't have an `update_plan` tool available in this specialist context (only read-only exploration tools were provided).",
|
||||
"I did not run the tests because only read-only tools were provided, so I report what the code says statically.",
|
||||
"update_plan is not in my toolset, so I could not record a plan.",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(text); reason != "" {
|
||||
t.Errorf("a run naming its own grant was marked incomplete: %q -> %s", text, reason)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// NAMING THE OBJECTIVE TO REPORT SUCCESS, same trap as naming the task.
|
||||
//
|
||||
// "the objective" and "the assignment" were bare nouns in objectiveFailureMarkers
|
||||
// while "this task" had already been verb-anchored for exactly this reason. A
|
||||
// message that mentions a tool it lacked and then says the objective IS met was
|
||||
// read as saying the objective was not met — a finished answer told it had not
|
||||
// finished, which is the worst outcome this detector has.
|
||||
func TestNamingTheObjectiveWhileReportingSuccessIsNotAnAdmission(t *testing.T) {
|
||||
for _, done := range []string{
|
||||
"I do not have write tools available, but the objective is met: the config already sets the flag.",
|
||||
"No write tool is available to me, so the objective was achieved by reading alone.",
|
||||
"The assignment is complete; no shell was needed.",
|
||||
"I have no browser tool available here, yet the assignment is complete.",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(done); reason != "" {
|
||||
t.Errorf("a finished answer was marked incomplete for naming its objective: %q -> %s", done, reason)
|
||||
}
|
||||
}
|
||||
// The failure forms differ only in the verb, and must still fire.
|
||||
for _, failed := range []string{
|
||||
"I could not complete the objective because the build never succeeded.",
|
||||
"I was unable to finish the assignment.",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(failed); reason == "" {
|
||||
t.Errorf("an objective that was not met was passed as complete: %q", failed)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A TOOL CAVEAT IS THE REASON WORK IS BLOCKED, NOT A REASON TO STOP READING.
|
||||
//
|
||||
// The tool-grant exemption asked only whether the sentence named the objective,
|
||||
// so an admission that named none was waved through on the strength of
|
||||
// mentioning tools:
|
||||
//
|
||||
// "No write tools available, so I could not verify the change."
|
||||
//
|
||||
// It breaks now on a blocked STATE — unverified, unresolved, handed off — but
|
||||
// deliberately not on the two bare inability stems in that list, because a
|
||||
// sentence carrying one can still be on its way to reporting success (see
|
||||
// TestNamingTheTaskWhileReportingSuccessIsNotAnAdmission, which this broke when
|
||||
// the whole list was used).
|
||||
func TestAToolCaveatDoesNotExcuseBlockedWork(t *testing.T) {
|
||||
for _, admission := range []string{
|
||||
"No write tools available, so I could not verify the change.",
|
||||
"There is no edit tool available here, so I could not verify the fix.",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(admission); reason == "" {
|
||||
t.Errorf("a tool caveat excused blocked work: %q", admission)
|
||||
}
|
||||
}
|
||||
// The exemption still covers what it exists for: a tool that was not needed.
|
||||
for _, exempt := range []string{
|
||||
"No update_plan tool available in this specialist context, so I proceeded directly.",
|
||||
"There is no update_plan tool available in this specialist context (only read-only tools), so I have written the plan into this answer instead.",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(exempt); reason != "" {
|
||||
t.Errorf("the blocked-state check broke the exemption it guards: %q -> %s", exempt, reason)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// WHAT FOLLOWS "any" DECIDES WHETHER FINDING NOTHING IS THE RESULT.
|
||||
//
|
||||
// The "any"-family was treated as a finding whatever the object, so an admission
|
||||
// wearing the same words walked through:
|
||||
//
|
||||
// "I could not find any remaining issues" -> a finding, the search succeeded
|
||||
// "I could not find any solution" -> an admission, the work did not
|
||||
//
|
||||
// Both carry the explicit "any"; only the object separates them. The object list
|
||||
// is an ALLOW-LIST because it grants the exemption — a deny-list of deliverables
|
||||
// would wave through every noun nobody thought of, which is the wrong direction
|
||||
// for this detector to fail in.
|
||||
func TestAnAbsenceIsAFindingOnlyWhenTheObjectMakesItOne(t *testing.T) {
|
||||
for _, admission := range []string{
|
||||
"I could not find any solution, so the migration remains unresolved.",
|
||||
"I could not find any fix, so the build is still broken.",
|
||||
"I could not find any workaround; someone else will need to pick this up.",
|
||||
"I could not identify any approach, so this is unverified.",
|
||||
} {
|
||||
if selfReportedIncompletion(admission) == "" {
|
||||
t.Errorf("an admission wearing the words of a finding passed as complete: %q", admission)
|
||||
}
|
||||
}
|
||||
// The findings the allowance exists for are untouched, including the ones
|
||||
// carrying a blocked-work marker about somebody else's future work.
|
||||
for _, finding := range []string{
|
||||
"I could not find any remaining issues, though a follow-up will need to cover the Windows path.",
|
||||
"I could not find any blockers; someone else can take the release from here.",
|
||||
"I could not find any evidence of a leak, so nothing was modified.",
|
||||
"I did not see any further regressions, so someone else can ship it.",
|
||||
"I could not detect any races, and nothing was modified.",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(finding); reason != "" {
|
||||
t.Errorf("a finding was reported as an admission: %q -> %s", finding, reason)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A FULL STOP IS NOT A CLAIM THAT THE WORK FINISHED.
|
||||
//
|
||||
// The blocked-work override only ever saw the sentence the allowance fired in,
|
||||
// so the SAME admission was caught or missed on its punctuation alone:
|
||||
//
|
||||
// "I could not reproduce the crash, so the fix is unverified." caught
|
||||
// "I could not reproduce the crash. The fix is unverified." missed
|
||||
//
|
||||
// Writing the consequence as its own sentence is how most people write, and half
|
||||
// of a reviewer's corpus of ordinary admissions escaped through it.
|
||||
//
|
||||
// Only the blocked-work question looks ahead. A stem in one sentence is still
|
||||
// never paired with an allowance tail in another.
|
||||
func TestTheConsequenceMayBeTheNextSentence(t *testing.T) {
|
||||
for _, admission := range []string{
|
||||
"I could not reproduce the crash, so the fix is unverified.",
|
||||
"I could not reproduce the crash. The fix is unverified.",
|
||||
"I could not locate the source of the regression and have run out of ideas.",
|
||||
"I could not locate the source of the regression. I have run out of ideas.",
|
||||
// "the work is blocked" says it in as many words, and every marker was a
|
||||
// SYMPTOM of being blocked rather than the thing itself.
|
||||
"I could not find the root cause, so the work is blocked.",
|
||||
"I could not find the root cause. The work is blocked.",
|
||||
"I could not get the integration test to run. The behaviour is therefore unverified.",
|
||||
"I could not apply the patch cleanly. Nothing was modified.",
|
||||
} {
|
||||
if selfReportedIncompletion(admission) == "" {
|
||||
t.Errorf("an admission escaped by putting its consequence in a second sentence: %q", admission)
|
||||
}
|
||||
}
|
||||
|
||||
// THE LOOKAHEAD'S OWN COST, guarded. A message that turns to other work must
|
||||
// not have that work's blocked state read as this result's consequence.
|
||||
for _, finding := range []string{
|
||||
"I searched the tree and could not find any other call sites. The rename is complete.",
|
||||
"I could not find any remaining usages of the deprecated helper. A future cleanup is blocked until the API freeze lifts.",
|
||||
"I could not reproduce any failure in the parser. The CI flake in the network suite remains unresolved and belongs to another team.",
|
||||
"I could not find any issues in the diff. Documentation for the new flag is unfinished, which was never part of this request.",
|
||||
"I could not reproduce the crash. Separately, the deploy step is unverified because it needs prod access.",
|
||||
"I could not find any other call sites. The follow-up work is blocked on a design decision, which is out of scope here.",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(finding); reason != "" {
|
||||
t.Errorf("the lookahead read another subject's blocked state as this result's: %q -> %s", finding, reason)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// AN UNAMBIGUOUS STATE THAT IS NOT ALSO A MARKER IS DEAD.
|
||||
//
|
||||
// unambiguousFailureStates only decides whether a recognised absence object
|
||||
// stops protecting the sentence; something still has to FIRE, and that is
|
||||
// blockedWorkMarkers. Adding "still blocked" to the first list and not the
|
||||
// second left it doing nothing at all, and the case looked handled because the
|
||||
// phrase appeared in the code.
|
||||
//
|
||||
// Two hand-maintained lists that must agree is the shape that drifts, so the
|
||||
// agreement is asserted rather than remembered.
|
||||
func TestEveryUnambiguousStateIsAlsoABlockedWorkMarker(t *testing.T) {
|
||||
for _, state := range unambiguousFailureStates {
|
||||
found := false
|
||||
for _, marker := range blockedWorkMarkers {
|
||||
if state == marker {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("%q outranks a strong absence but is not a blockedWorkMarker, so nothing fires on it", state)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// EXPLICIT FAILURE STATES OUTRANK A RECOGNISED ABSENCE OBJECT.
|
||||
//
|
||||
// strongAbsence returns true for objects like "evidence", and that suppressed
|
||||
// every blocked-work marker in the sentence — so a message saying in as many
|
||||
// words that the work is unverified reported success. The absence protection is
|
||||
// for ownership and follow-up wording; it was never meant to cover a sentence
|
||||
// that states the outcome.
|
||||
func TestAnExplicitFailureStateOutranksTheAbsenceObject(t *testing.T) {
|
||||
for _, admission := range []string{
|
||||
"I could not find any evidence supporting the fix, so it remains unverified.",
|
||||
"I could not find any evidence for the cause, so the bug is unresolved.",
|
||||
"I could not find any way to make it work, so I gave up.",
|
||||
"I could not find any working approach; it is still broken.",
|
||||
"I could not find any issues, but the migration is still blocked.",
|
||||
} {
|
||||
if selfReportedIncompletion(admission) == "" {
|
||||
t.Errorf("an explicit failure state was overruled by the absence object: %q", admission)
|
||||
}
|
||||
}
|
||||
|
||||
// AMBIGUOUS wording still yields to the absence, which is the whole reason
|
||||
// the protection exists: ownership and follow-up read two ways, an explicit
|
||||
// state reads one.
|
||||
for _, finding := range []string{
|
||||
"I could not find any evidence that the flag is read in production.",
|
||||
"I could not find any remaining issues, though a follow-up will need to cover the Windows path.",
|
||||
"I could not find any blockers; someone else can take the release from here.",
|
||||
"I could not find any evidence of a leak, so nothing was modified.",
|
||||
"I could not detect any races, and nothing was modified.",
|
||||
// A state in the NEXT sentence may belong to another subject, so
|
||||
// same-sentence only.
|
||||
"I could not reproduce any failure in the parser. The CI flake in the network suite remains unresolved and belongs to another team.",
|
||||
"I could not find any issues in the diff. Documentation for the new flag is unfinished, which was never part of this request.",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(finding); reason != "" {
|
||||
t.Errorf("a finding was flagged by ambiguous or another subject's wording: %q -> %s", finding, reason)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// THE STATE MUST BE THE OUTCOME, NOT PART OF WHAT WAS NEGATED.
|
||||
//
|
||||
// The override matched a failure state anywhere in the sentence, which read the
|
||||
// negated PROPOSITION as the reported result:
|
||||
//
|
||||
// "I could not find any evidence that the issue is unresolved."
|
||||
//
|
||||
// That is a successful negative finding — there is no evidence the issue remains
|
||||
// unresolved — and it was marked incomplete. Exactly the opposite polarity of the
|
||||
// case the override was added for, and the two have to be tested together or
|
||||
// fixing one just moves the error.
|
||||
func TestAStateInsideTheNegatedPropositionIsNotTheOutcome(t *testing.T) {
|
||||
for _, finding := range []string{
|
||||
"I could not find any evidence that the issue is unresolved.",
|
||||
"I could not find any evidence that the build is still broken.",
|
||||
"I could not find any evidence that the migration is incomplete.",
|
||||
"I could not find any sign that the deploy is blocked.",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(finding); reason != "" {
|
||||
t.Errorf("a negated proposition was read as the reported outcome: %q -> %s", finding, reason)
|
||||
}
|
||||
}
|
||||
|
||||
// Both polarities together: after a consequence boundary the state IS the
|
||||
// outcome and must still fire.
|
||||
for _, admission := range []string{
|
||||
"I could not find any evidence supporting the fix, so it remains unverified.",
|
||||
"I could not find any evidence for the cause, so the bug is unresolved.",
|
||||
"I could not find any way to make it work, so I gave up.",
|
||||
"I could not find any working approach; it is still broken.",
|
||||
"I could not find any issues, but the migration is still blocked.",
|
||||
} {
|
||||
if selfReportedIncompletion(admission) == "" {
|
||||
t.Errorf("an outcome stated after a consequence boundary was missed: %q", admission)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2,8 +2,6 @@ package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -82,9 +80,6 @@ func TestRunStopsAfterConsecutiveEmptyTurns(t *testing.T) {
|
||||
if result.FinalAnswer == maxTurnsAnswer {
|
||||
t.Fatalf("no-output guard must stop before reaching maxTurns, got max-turns answer")
|
||||
}
|
||||
if result.Incomplete {
|
||||
t.Fatalf("interactive run must not be marked Incomplete; reason=%q", result.IncompleteReason)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunResetsEmptyTurnCounterOnVisibleOutput(t *testing.T) {
|
||||
@@ -194,7 +189,7 @@ func TestUnknownExecSessionProbingTripsFailureHalt(t *testing.T) {
|
||||
var state guardState
|
||||
var stoppedAt int
|
||||
for i := 1; i <= toolFailureStopAt; i++ {
|
||||
out := state.observeToolResult(tools.WriteStdinToolName, true, true, tools.UnknownExecSessionError(i), "")
|
||||
out := state.observeToolResult(tools.WriteStdinToolName, true, tools.UnknownExecSessionError(i))
|
||||
if out.Stop {
|
||||
stoppedAt = i
|
||||
break
|
||||
@@ -205,223 +200,6 @@ func TestUnknownExecSessionProbingTripsFailureHalt(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A permission denial repeats forever when the streak is keyed on the error
|
||||
// TEXT, because the denial message embeds the path or command that varies per
|
||||
// call.
|
||||
//
|
||||
// Observed, not theorised: a headless run made 384 denied calls over 26 minutes
|
||||
// without tripping a halt that stops at 6. Every denial carried the same typed
|
||||
// category and a different reason string, so errorSignature differed each time
|
||||
// and the record was rebuilt at count 1 on every call.
|
||||
//
|
||||
// TestUnknownExecSessionErrorSignatureIsIDInvariant fixed one message this way.
|
||||
// Keying on the category fixes the class, without needing every future denial
|
||||
// message to remember to be invariant.
|
||||
func TestPermissionDenialStreakSurvivesVaryingReasonText(t *testing.T) {
|
||||
var state guardState
|
||||
stoppedAt := 0
|
||||
for i := 1; i <= toolFailureStopAt; i++ {
|
||||
// The shape the loop actually produces: same tool, same category, a
|
||||
// different path every time.
|
||||
output := "Error: Permission denied for write_file: cannot write " +
|
||||
filepath.Join("C:", "ws", "pkg", "file"+strconv.Itoa(i)+".go")
|
||||
out := state.observeToolResult("write_file", true, true, output, DenialPermissionDenied)
|
||||
if out.Stop {
|
||||
stoppedAt = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if stoppedAt != toolFailureStopAt {
|
||||
t.Fatalf("denials with varying reason text stopped at %d, want %d", stoppedAt, toolFailureStopAt)
|
||||
}
|
||||
}
|
||||
|
||||
// The content-blind bound. A tool failing over and over with genuinely
|
||||
// DIFFERENT errors and no denial category is still a tool that is not working,
|
||||
// and the same-signature streak can never see it.
|
||||
//
|
||||
// Bounded well above toolFailureStopAt on purpose: a model iterating on a
|
||||
// tricky edit legitimately fails a few times with different errors while it
|
||||
// converges, and cutting that short is the regression
|
||||
// TestSuccessResetsBothFailureCounters guards.
|
||||
func TestToolFailingWithDifferentErrorsEveryTimeStillStops(t *testing.T) {
|
||||
var state guardState
|
||||
stoppedAt := 0
|
||||
for i := 1; i <= toolFailureAnyErrorStopAt; i++ {
|
||||
out := state.observeToolResult("bash", true, true, "distinct failure "+strconv.Itoa(i), "")
|
||||
if out.Stop {
|
||||
stoppedAt = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if stoppedAt != toolFailureAnyErrorStopAt {
|
||||
t.Fatalf("a tool failing with a new error each call stopped at %d, want %d", stoppedAt, toolFailureAnyErrorStopAt)
|
||||
}
|
||||
}
|
||||
|
||||
// Neither counter may outlive a success, or a long run that fails occasionally
|
||||
// and recovers would eventually halt for no reason. This is the property that
|
||||
// keeps the content-blind bound safe to add.
|
||||
func TestSuccessResetsBothFailureCounters(t *testing.T) {
|
||||
var state guardState
|
||||
for i := 1; i < toolFailureAnyErrorStopAt; i++ {
|
||||
if out := state.observeToolResult("bash", true, true, "distinct failure "+strconv.Itoa(i), ""); out.Stop {
|
||||
t.Fatalf("stopped at %d before the success that should reset it", i)
|
||||
}
|
||||
}
|
||||
state.observeToolResult("bash", false, false, "ok", "")
|
||||
|
||||
// Same again from zero. Reaching the bound a second time proves the counter
|
||||
// restarted rather than merely paused.
|
||||
stoppedAt := 0
|
||||
for i := 1; i <= toolFailureAnyErrorStopAt; i++ {
|
||||
if out := state.observeToolResult("bash", true, true, "later failure "+strconv.Itoa(i), ""); out.Stop {
|
||||
stoppedAt = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if stoppedAt != toolFailureAnyErrorStopAt {
|
||||
t.Fatalf("after a success the tool stopped at %d, want a full fresh %d", stoppedAt, toolFailureAnyErrorStopAt)
|
||||
}
|
||||
}
|
||||
|
||||
// Records are keyed per tool, so ANOTHER tool succeeding in between must not
|
||||
// clear the failing tool's streak.
|
||||
//
|
||||
// This is the realistic shape of the run that motivated the fix: the model kept
|
||||
// making progress elsewhere — reading files, updating its plan — while one tool
|
||||
// was refused over and over. A reset keyed on "something succeeded" rather than
|
||||
// "this tool succeeded" would make the halt unreachable in exactly the runs that
|
||||
// need it.
|
||||
func TestAnotherToolSucceedingDoesNotClearAFailingToolsStreak(t *testing.T) {
|
||||
var state guardState
|
||||
stoppedAt := 0
|
||||
for i := 1; i <= toolFailureStopAt; i++ {
|
||||
state.observeToolResult("read_file", false, false, "ok", "")
|
||||
out := state.observeToolResult("write_file", true, false,
|
||||
"Error: Permission denied for write_file: "+strconv.Itoa(i), DenialPermissionDenied)
|
||||
if out.Stop {
|
||||
stoppedAt = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if stoppedAt != toolFailureStopAt {
|
||||
t.Fatalf("denials interleaved with another tool's successes stopped at %d, want %d", stoppedAt, toolFailureStopAt)
|
||||
}
|
||||
}
|
||||
|
||||
// alwaysPromptingTool is never allowed to run: it exists so a Run-level test can
|
||||
// drive real permission denials through the loop.
|
||||
type alwaysPromptingTool struct{ ran int }
|
||||
|
||||
func (tool *alwaysPromptingTool) Name() string { return "bash" }
|
||||
func (tool *alwaysPromptingTool) Description() string { return "test shell tool" }
|
||||
func (tool *alwaysPromptingTool) Parameters() tools.Schema {
|
||||
return tools.Schema{
|
||||
Type: "object",
|
||||
Properties: map[string]tools.PropertySchema{"command": {Type: "string"}},
|
||||
Required: []string{"command"},
|
||||
AdditionalProperties: false,
|
||||
}
|
||||
}
|
||||
func (tool *alwaysPromptingTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectShell, Permission: tools.PermissionPrompt, Reason: "runs shell commands"}
|
||||
}
|
||||
func (tool *alwaysPromptingTool) Run(context.Context, map[string]any) tools.Result {
|
||||
tool.ran++
|
||||
return tools.Result{Status: tools.StatusOK, Output: "should never run"}
|
||||
}
|
||||
|
||||
// The regression for the gap this whole change existed to close, driven through
|
||||
// Run rather than through the guard helper.
|
||||
//
|
||||
// The first version of this fix was a no-op in production and every one of its
|
||||
// unit tests passed, because they called observeToolResult directly with
|
||||
// failed=true. The loop passes isRetriableToolError, which returns false for a
|
||||
// categorized denial, so the guard took its success branch and deleted the
|
||||
// record before it could key on the category. A denied tool still looped to the
|
||||
// turn limit.
|
||||
//
|
||||
// This drives the real path: a tool that always prompts, an approver that always
|
||||
// denies, and a different command each turn so the denial REASON — and therefore
|
||||
// the error text — varies exactly as it does in a real run.
|
||||
func TestRunStopsARepeatedlyDeniedToolAtTheFailureBound(t *testing.T) {
|
||||
tool := &alwaysPromptingTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
// Comfortably more turns than the bound, so reaching the bound is what stops
|
||||
// the run rather than exhausting the provider or MaxTurns.
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, toolFailureStopAt+4)
|
||||
for i := range toolFailureStopAt + 4 {
|
||||
turns = append(turns, toolTurn("call-"+strconv.Itoa(i), "bash",
|
||||
`{"command":"touch /etc/file`+strconv.Itoa(i)+`"}`))
|
||||
}
|
||||
provider := &mockProvider{turns: turns}
|
||||
|
||||
denials := 0
|
||||
result, err := Run(context.Background(), "do the thing", provider, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
OnPermissionRequest: func(_ context.Context, request PermissionRequest) (PermissionDecision, error) {
|
||||
denials++
|
||||
// The varying half. In production this is the refused path or command;
|
||||
// here it is the command, which lands in the denial message the guard
|
||||
// used to key on.
|
||||
return PermissionDecision{
|
||||
Action: PermissionDecisionDeny,
|
||||
Reason: "refused " + request.ToolName + " call " + strconv.Itoa(denials),
|
||||
}, nil
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if denials != toolFailureStopAt {
|
||||
t.Errorf("the run made %d denied calls, want it halted at %d", denials, toolFailureStopAt)
|
||||
}
|
||||
if tool.ran != 0 {
|
||||
t.Errorf("the denied tool executed %d times; the denial must precede execution", tool.ran)
|
||||
}
|
||||
want := toolFailureStopAnswer("bash", toolFailureStopAt, toolFailureCauseSameRefusal)
|
||||
if result.FinalAnswer != want {
|
||||
t.Errorf("final answer =\n %q\nwant\n %q", result.FinalAnswer, want)
|
||||
}
|
||||
}
|
||||
|
||||
// The stop message must describe the bound that actually tripped. When the
|
||||
// content-blind counter is what halts the run, the same-signature streak is
|
||||
// usually 1, and reporting that would tell the user a tool failed once after it
|
||||
// failed a dozen different ways.
|
||||
func TestVariedFailureStopAnswerReportsTheRightCounter(t *testing.T) {
|
||||
var state guardState
|
||||
var outcome toolFailureOutcome
|
||||
for i := 1; i <= toolFailureAnyErrorStopAt; i++ {
|
||||
outcome = state.observeToolResult("bash", true, true, "distinct failure "+strconv.Itoa(i), "")
|
||||
if outcome.Stop {
|
||||
break
|
||||
}
|
||||
}
|
||||
if !outcome.Stop {
|
||||
t.Fatal("never stopped")
|
||||
}
|
||||
if outcome.Cause != toolFailureCauseVariedExecuted {
|
||||
t.Errorf("Cause = %v for a stop driven by the content-blind counter over executed failures", outcome.Cause)
|
||||
}
|
||||
if outcome.Count != toolFailureAnyErrorStopAt {
|
||||
t.Errorf("Count = %d, want the counter that tripped (%d)", outcome.Count, toolFailureAnyErrorStopAt)
|
||||
}
|
||||
answer := toolFailureStopAnswer("bash", outcome.Count, outcome.Cause)
|
||||
if !strings.Contains(answer, "varying errors") {
|
||||
t.Errorf("stop answer describes the wrong cause: %q", answer)
|
||||
}
|
||||
if strings.Contains(answer, "with the same error") {
|
||||
t.Errorf("stop answer claims a same-error loop after distinct failures: %q", answer)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuardStateResetsToolOnlyStreakOnEmptyNonToolTurn(t *testing.T) {
|
||||
var state guardState
|
||||
toolOnly := zeroruntime.CollectedStream{
|
||||
@@ -735,22 +513,6 @@ func repeatedFlakyTurns(n int) [][]zeroruntime.StreamEvent {
|
||||
return turns
|
||||
}
|
||||
|
||||
// alwaysSucceedingTool is the successful counterpart to alwaysFailingTool, used
|
||||
// to put a productive tool turn before an empty-turn streak.
|
||||
type alwaysSucceedingTool struct{}
|
||||
|
||||
func (alwaysSucceedingTool) Name() string { return "ok" }
|
||||
func (alwaysSucceedingTool) Description() string { return "always succeeds for testing" }
|
||||
func (alwaysSucceedingTool) Parameters() tools.Schema {
|
||||
return tools.Schema{Type: "object", AdditionalProperties: false}
|
||||
}
|
||||
func (alwaysSucceedingTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow}
|
||||
}
|
||||
func (alwaysSucceedingTool) Run(context.Context, map[string]any) tools.Result {
|
||||
return tools.Result{Status: tools.StatusOK, Output: "ok"}
|
||||
}
|
||||
|
||||
func TestRunStopsAfterRepeatedToolFailures(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(alwaysFailingTool{})
|
||||
@@ -767,51 +529,6 @@ func TestRunStopsAfterRepeatedToolFailures(t *testing.T) {
|
||||
if len(provider.requests) != toolFailureStopAt {
|
||||
t.Fatalf("expected stop at %d failures, made %d requests", toolFailureStopAt, len(provider.requests))
|
||||
}
|
||||
if result.Incomplete {
|
||||
t.Fatalf("interactive run must not be marked Incomplete; reason=%q", result.IncompleteReason)
|
||||
}
|
||||
}
|
||||
|
||||
// Under the headless completion gate both guardrail stops are unfinished runs,
|
||||
// so they must set Result.Incomplete and a reason; exec maps that to exit 4.
|
||||
// Thresholds and stop answers stay pinned by the two tests above.
|
||||
func TestHeadlessGuardStopsAreIncomplete(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
turns [][]zeroruntime.StreamEvent
|
||||
wantTurns int
|
||||
wantReason string
|
||||
}{
|
||||
{"empty_turns", [][]zeroruntime.StreamEvent{emptyTurn(), emptyTurn(), emptyTurn()}, maxEmptyTurns, "3 consecutive turns with no output"},
|
||||
{"empty_turns_after_progress", [][]zeroruntime.StreamEvent{
|
||||
toolTurn("ok-1", "ok", "{}"),
|
||||
emptyTurn(), emptyTurn(), emptyTurn(),
|
||||
}, maxEmptyTurns + 1, "3 consecutive turns with no output"},
|
||||
{"repeated_tool_failures", repeatedFlakyTurns(10), toolFailureStopAt, "flaky"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(alwaysFailingTool{})
|
||||
registry.Register(alwaysSucceedingTool{})
|
||||
result, err := Run(context.Background(), "go", &mockProvider{turns: tc.turns}, Options{
|
||||
Registry: registry,
|
||||
MaxTurns: 12,
|
||||
RequireCompletionSignal: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if result.Turns != tc.wantTurns {
|
||||
t.Fatalf("guard did not trigger: turns=%d want=%d", result.Turns, tc.wantTurns)
|
||||
}
|
||||
if !result.Incomplete {
|
||||
t.Fatalf("guardrail stop under the headless gate must be Incomplete; final=%q", result.FinalAnswer)
|
||||
}
|
||||
if !strings.Contains(result.IncompleteReason, tc.wantReason) {
|
||||
t.Fatalf("IncompleteReason = %q, want it to mention %q", result.IncompleteReason, tc.wantReason)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunInjectsToolFailureHintWithSchema(t *testing.T) {
|
||||
@@ -837,91 +554,3 @@ func TestRunInjectsToolFailureHintWithSchema(t *testing.T) {
|
||||
t.Fatalf("expected a tool-failure hint on the 3rd turn, messages: %+v", provider.requests[2].Messages)
|
||||
}
|
||||
}
|
||||
|
||||
// An absence-establishing sentence is a finding; the same sentence that also
|
||||
// says the work is blocked is an admission. The allowance stems ("find the",
|
||||
// "reproduce ", "confirm any" …) head both, so the tail prefix alone cannot
|
||||
// separate them — measured against real phrasings, TEN of eleven genuine
|
||||
// admissions passed the detector before blockedWorkMarkers existed.
|
||||
func TestIncompletionAllowanceYieldsToBlockedWork(t *testing.T) {
|
||||
// Must FIRE: the inability is reported as leaving work undone.
|
||||
for _, admission := range []string{
|
||||
"I could not reproduce the crash, so the fix is unverified.",
|
||||
"I was unable to reproduce the reported behaviour and stopped there.",
|
||||
"I could not find the root cause; someone else will need to pick this up.",
|
||||
"I could not find the bug in the time available.",
|
||||
"I did not manage to reproduce the failure, so I cannot confirm the patch works.",
|
||||
"I could not locate the source of the regression and have run out of ideas.",
|
||||
"I could not find where to apply the change, so nothing was modified.",
|
||||
} {
|
||||
if selfReportedIncompletion(admission) == "" {
|
||||
t.Errorf("admission passed the detector: %q", admission)
|
||||
}
|
||||
}
|
||||
|
||||
// Must NOT fire: establishing that something is absent is the job, and the
|
||||
// motivating case is the audit that spent 53 tool calls proving a negative.
|
||||
for _, finding := range []string{
|
||||
"I could not find any remaining issues.",
|
||||
"I could NOT find where AllowManifestToolAutoApproval is set to true in production code.",
|
||||
"I could not find the flag being set anywhere outside tests, so the concern does not apply.",
|
||||
"I could not reproduce the reported exploit, which confirms the guard holds.",
|
||||
"I could not observe any regression across the suite.",
|
||||
"I could not confirm any leak; every path is bounded.",
|
||||
// Next steps and ownership belong in successful reports too. Both of
|
||||
// these were flagged when blockedWorkMarkers could override an explicit
|
||||
// "any", which is the model asserting exhaustive absence.
|
||||
"I could not find any remaining issues, though a follow-up will need to cover the Windows path.",
|
||||
"I could not find any blockers; someone else can take the release from here.",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(finding); reason != "" {
|
||||
t.Errorf("finding wrongly flagged as incomplete: %q -> %q", finding, reason)
|
||||
}
|
||||
}
|
||||
|
||||
// NOT CAUGHT, and recorded rather than hidden. An explicit "any" is treated
|
||||
// as exhaustive absence, so an admission phrased that way passes:
|
||||
//
|
||||
// "I could not observe any effect, so the change may be inert."
|
||||
//
|
||||
// Measured across eleven admissions, four still pass, all of them either
|
||||
// "any"-phrased or single-clause with no blocked-work signal ("I failed to
|
||||
// reproduce it locally"). Catching those needs a different signal than
|
||||
// substring matching; tightening this list to reach them re-broke the
|
||||
// findings above every way I tried.
|
||||
}
|
||||
|
||||
// An admission with NO first-person subject must still fire. The subjectless
|
||||
// "unable to " stem was once deleted to silence a completed audit's section
|
||||
// heading, which also lost every impersonal admission — none of these names "i"
|
||||
// or "we", so no other stem sees them.
|
||||
func TestImpersonalAdmissionsAreStillCaught(t *testing.T) {
|
||||
for _, admission := range []string{
|
||||
"Unable to complete the task; the build never succeeded.",
|
||||
"The agent was unable to finish the migration.",
|
||||
"Unable to verify the fix, so the change is unverified.",
|
||||
} {
|
||||
if selfReportedIncompletion(admission) == "" {
|
||||
t.Errorf("an impersonal admission passed the detector: %q", admission)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The heading that motivated deleting the stem must stay exempt. It is a label
|
||||
// counting a bucket of findings in a COMPLETED audit, not a claim about the
|
||||
// objective — which is why it is recognised by shape rather than by removing a
|
||||
// stem that catches real admissions.
|
||||
func TestACountedHeadingIsNotAnAdmission(t *testing.T) {
|
||||
for _, heading := range []string{
|
||||
"**Unable to verify (1):** - MCP #3 claim was truncated",
|
||||
"Unable to reproduce (3):",
|
||||
} {
|
||||
if reason := selfReportedIncompletion(heading); reason != "" {
|
||||
t.Errorf("a counted heading was read as an admission: %q -> %q", heading, reason)
|
||||
}
|
||||
}
|
||||
// But the same opening WITHOUT a count is a real admission.
|
||||
if selfReportedIncompletion("Unable to verify the deployment; it never started.") == "" {
|
||||
t.Error("a subjectless admission with no count was exempted as a heading")
|
||||
}
|
||||
}
|
||||
+30
-353
@@ -13,7 +13,6 @@ import (
|
||||
|
||||
"github.com/Twigpine/zero/internal/execution"
|
||||
"github.com/Twigpine/zero/internal/hooks"
|
||||
"github.com/Twigpine/zero/internal/measurements"
|
||||
"github.com/Twigpine/zero/internal/modelregistry"
|
||||
"github.com/Twigpine/zero/internal/redaction"
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
@@ -24,16 +23,8 @@ import (
|
||||
)
|
||||
|
||||
const maxTurnsAnswer = "Agent reached maximum number of turns without a final answer."
|
||||
const maxTokensAnswer = "Agent reached its token budget without a final answer."
|
||||
const maxTurnsFinalAnswerPrompt = "You have reached the tool-turn limit. Do not call tools. Give a concise final answer now: summarize what you completed, what you found, and any remaining blockers."
|
||||
|
||||
// maxTokensFinalAnswerPrompt is the same ending for a different reason, and it
|
||||
// says which. A run stopped for spend and a run stopped for round trips call for
|
||||
// the same next action from the model and a different explanation to the reader,
|
||||
// who otherwise reaches for the wrong lever — raising a turn count that was never
|
||||
// what ran out.
|
||||
const maxTokensFinalAnswerPrompt = "You have reached this run's token budget. Do not call tools. Give a concise final answer now: summarize what you completed, what you found, and any remaining blockers."
|
||||
|
||||
// maxStreamStallRetries bounds how many times a turn that timed out (idle/stall)
|
||||
// OR hit a mid-stream transport abort (#973) with no answer text yet is re-issued
|
||||
// on a fresh connection before giving up. Incomplete tool-call previews are
|
||||
@@ -187,10 +178,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// this local, so no signatures change anywhere downstream.
|
||||
provider = sessionProvider{session: session}
|
||||
|
||||
// The turn's raw user text, carried to tools so a gate can tell "the user
|
||||
// asked for this" from "the model read a sentence that said to".
|
||||
options.userMessage = prompt
|
||||
|
||||
maxTurns := options.MaxTurns
|
||||
if maxTurns <= 0 {
|
||||
maxTurns = 12
|
||||
@@ -269,18 +256,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// FileDiagnostics callback is wired; every method no-ops on nil.
|
||||
postEditDiagnostics := newAsyncDiagnostics(options.FileDiagnostics, options.Cwd)
|
||||
|
||||
// THE RUN'S OWN TIMINGS, so a final answer cannot report a number no command
|
||||
// here produced. Every tool result is read for `go test` durations; the final
|
||||
// answer is checked against them once before it is returned.
|
||||
//
|
||||
// POSTURE-GATED, and nil when the posture is off — every method no-ops on
|
||||
// nil, so a run that never heard of zeromaxing takes exactly the path it took
|
||||
// before this existed. See internal/measurements for the failure it is for.
|
||||
var measured *measurements.Ledger
|
||||
if options.Zeromaxing != ZeromaxingOff {
|
||||
measured = measurements.NewLedger()
|
||||
}
|
||||
|
||||
// loaded tracks deferred-eligible tools the model has pulled via tool_search
|
||||
// during THIS run. It is consulted by partitionTools each turn to expose a
|
||||
// loaded tool's full schema; it lives only for the run (v1 within-run scope).
|
||||
@@ -296,12 +271,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// doesn't re-run the recursive schema→map conversion for every tool every turn.
|
||||
toolDefCache := map[string]zeroruntime.ToolDefinition{}
|
||||
|
||||
// SPEND, accumulated from what the provider actually reported rather than
|
||||
// estimated. Only read when options.MaxTokens > 0, so an unbounded run adds
|
||||
// one integer and changes nothing else.
|
||||
spentTokens := 0
|
||||
stoppedOnTokens := false
|
||||
|
||||
result = Result{Messages: copyMessages(messages)}
|
||||
dispatchSessionStart(ctx, options)
|
||||
defer func() {
|
||||
@@ -314,27 +283,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
for turn := 0; turn < maxTurns; turn++ {
|
||||
result.Turns = turn + 1
|
||||
|
||||
// CHECKED HERE, at the top of a turn, for the same reason the turn count
|
||||
// is: the previous turn's tool calls have run and their results are in
|
||||
// hand. Stopping mid-turn would discard work already paid for.
|
||||
if options.MaxTokens > 0 && spentTokens >= options.MaxTokens {
|
||||
stoppedOnTokens = true
|
||||
break
|
||||
}
|
||||
|
||||
// The zeromaxing posture's reminders. Appended to the CONVERSATION tail
|
||||
// as user-role messages — the same channel as the diagnostics nudge
|
||||
// below and the failure/plan hints later in the turn — and never into
|
||||
// the system prompt, which is built once per run and must stay
|
||||
// byte-stable so the provider's cached prefix survives. See
|
||||
// internal/agent/zeromaxing.go.
|
||||
for _, reminder := range zeromaxingReminders(options.Zeromaxing, result.Turns, options.OrchestrateAvailable) {
|
||||
messages = append(messages, zeroruntime.Message{
|
||||
Role: zeroruntime.MessageRoleUser,
|
||||
Content: reminder,
|
||||
})
|
||||
}
|
||||
|
||||
// Deliver background post-edit diagnostics from the previous turn's edits
|
||||
// BEFORE compaction so the nudge is part of the request being budgeted.
|
||||
// A brief wait at most (asyncDiagnosticsDrainTimeout); an unfinished check
|
||||
@@ -346,19 +294,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
})
|
||||
}
|
||||
|
||||
// A BACKGROUND PLAN that finished since the last turn. Same channel and
|
||||
// the same point in the turn as the diagnostics nudge above: the model
|
||||
// was told the plan was not finished and must not report it as done, so
|
||||
// this is the message that makes that promise good.
|
||||
if options.PlanCompletions != nil {
|
||||
if finished := options.PlanCompletions(); finished != "" {
|
||||
messages = append(messages, zeroruntime.Message{
|
||||
Role: zeroruntime.MessageRoleUser,
|
||||
Content: finished,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Build the per-turn tool list first so proactive compaction can include
|
||||
// the tool-definition tokens (they ride on every request) in its estimate.
|
||||
// partitionTools depends only on registry/permissions/options/loaded, not on
|
||||
@@ -620,11 +555,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// assistant reply is appended below, after this, so `messages` is still the
|
||||
// sent request.
|
||||
compactor.calibrate(estimateTokens(messages)+estimateToolDefTokens(exposed), collected.Usage.InputTokens)
|
||||
// Accumulated at the same point the estimator is calibrated: past error
|
||||
// recovery, and past any reactive compaction that re-sent the request, so
|
||||
// this counts the exchange that actually happened rather than one that was
|
||||
// abandoned and replaced.
|
||||
spentTokens += collected.Usage.TotalTokens()
|
||||
|
||||
// Carry the turn's terminal stop reason so a final answer cut off at the
|
||||
// output token cap (or by a content filter) is reported as truncated. A
|
||||
@@ -657,14 +587,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// counted toward the runaway cap so we stop before burning maxTurns.
|
||||
if guards.observeTurn(collected) {
|
||||
result.FinalAnswer = noOutputStopAnswer(result.Turns)
|
||||
// Under the headless completion gate this stop is unfinished work,
|
||||
// not a successful final answer: the guard exists precisely because
|
||||
// the model made no progress. Interactive runs keep the gate off, so
|
||||
// their behavior is unchanged.
|
||||
if options.RequireCompletionSignal {
|
||||
result.Incomplete = true
|
||||
result.IncompleteReason = "stopped after " + strconv.Itoa(guards.emptyTurns) + " consecutive turns with no output"
|
||||
}
|
||||
result.Messages = copyMessages(messages)
|
||||
return result, nil
|
||||
}
|
||||
@@ -732,23 +654,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
})
|
||||
continue
|
||||
}
|
||||
// THE ANSWER IS CHECKED AGAINST THE TRANSCRIPT before it is returned.
|
||||
// Same channel and the same gate as the diagnostics drain above, and
|
||||
// for the same reason: a final answer means there is no later turn, so
|
||||
// a number that contradicts what actually ran has to be raised now or
|
||||
// never. Each name is raised at most once by the ledger, so an answer
|
||||
// that comes back unchanged is returned rather than asked again.
|
||||
// ACROSS RUNS, because a final answer may summarise several commands
|
||||
// and this caller cannot say which one any given number came from.
|
||||
// Holding it to a single run would accuse the model of inventing a
|
||||
// number one of the other commands really printed.
|
||||
if nudge := measurements.Nudge(measured.ConflictsAcrossRuns(collected.Text)); nudge != "" {
|
||||
messages = append(messages, zeroruntime.Message{
|
||||
Role: zeroruntime.MessageRoleUser,
|
||||
Content: nudge,
|
||||
})
|
||||
continue
|
||||
}
|
||||
result.FinalAnswer = collected.Text
|
||||
result.Messages = copyMessages(messages)
|
||||
return result, nil
|
||||
@@ -806,60 +711,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// executed concurrently, consumed strictly in order below.
|
||||
var precomputed []precomputedToolResult
|
||||
precomputedStart, precomputedEnd := 0, 0
|
||||
// closeOutRemaining terminates every advertised call from `from` onward
|
||||
// TRUTHFULLY, and every early return in this loop goes through it.
|
||||
//
|
||||
// Read-ahead broke the assumption the abort placeholders were written
|
||||
// under. executeParallelReadBatch runs an entire eligible run before the
|
||||
// loop consumes any of it, so "not consumed yet" stopped meaning "not
|
||||
// executed": a sibling can already have run, committed its side effects
|
||||
// and any authorization credit they carry, and still sit past the index
|
||||
// where a terminal branch fires. Recording it as aborted threw that result
|
||||
// away, so the transcript disagreed with what had actually happened, and
|
||||
// its callbacks, trace counter, task observation and images went with it.
|
||||
//
|
||||
// So each remaining call is put in the one state that is true of it:
|
||||
// completed, and finalized exactly once with the same bookkeeping the main
|
||||
// path performs, or unstarted, and aborted. The guard is deliberately NOT
|
||||
// consulted for a drained sibling: these results cannot reverse a stop
|
||||
// decision that has already been made, they are only owed an honest record.
|
||||
closeOutRemaining := func(from int) {
|
||||
for next := from; next < len(collected.ToolCalls); next++ {
|
||||
sibling, ran := precomputedResultFor(precomputed, precomputedStart, precomputedEnd, next)
|
||||
if !ran {
|
||||
messages = appendAbortedToolResults(messages, collected.ToolCalls[next:next+1])
|
||||
continue
|
||||
}
|
||||
nextCall := collected.ToolCalls[next]
|
||||
if options.OnToolCall != nil {
|
||||
options.OnToolCall(nextCall)
|
||||
}
|
||||
options.Trace.Counter(trace.CounterToolCalls, 1)
|
||||
recordOutputBudgetTrace(options.Trace, sibling)
|
||||
task.observe(taskStateEvent{kind: taskStateEventToolResult, arguments: nextCall.Arguments, toolResult: sibling})
|
||||
if options.OnToolResult != nil {
|
||||
options.OnToolResult(sibling)
|
||||
}
|
||||
for _, name := range sibling.LoadedTools {
|
||||
loaded[name] = true
|
||||
}
|
||||
// The same fields the consumption path below records, provider call
|
||||
// ID included: a strict replay pairs the result to its tool_use by
|
||||
// that ID, and a finalized sibling without it was the one result in
|
||||
// the turn that could not be paired. Reported by @jatmn.
|
||||
messages = append(messages, zeroruntime.Message{
|
||||
Role: zeroruntime.MessageRoleTool,
|
||||
Content: sibling.ModelOutput(),
|
||||
ToolCallID: sibling.ToolCallID,
|
||||
ToolCallProviderID: nextCall.ProviderCallID,
|
||||
IsError: sibling.Status == tools.StatusError,
|
||||
ChangedFiles: append([]string(nil), sibling.ChangedFiles...),
|
||||
})
|
||||
if len(sibling.Images) > 0 {
|
||||
toolResultsWithImages = append(toolResultsWithImages, sibling)
|
||||
}
|
||||
}
|
||||
}
|
||||
for index, call := range collected.ToolCalls {
|
||||
// When this call starts a consecutive run of >= 2 auto-allowed read-only
|
||||
// calls, execute the whole run concurrently now (see parallel_tools.go).
|
||||
@@ -891,14 +742,6 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
}
|
||||
options.Trace.Counter(trace.CounterToolCalls, 1)
|
||||
recordOutputBudgetTrace(options.Trace, toolResult)
|
||||
// Read before anything can truncate or summarize it: this is the only
|
||||
// place the command's own words are in hand — and the only place the
|
||||
// CALL that produced them is in hand too. Recording which tool and
|
||||
// which arguments printed a timing is what stops a claim about one
|
||||
// command being answered by a number a different one produced: a
|
||||
// `-race` run is routinely several times slower, which is the size of
|
||||
// discrepancy this check exists to catch.
|
||||
measured.Record(measurements.Run{Command: call.Name, Args: []string{call.Arguments}}, toolResult.Output)
|
||||
task.observe(taskStateEvent{kind: taskStateEventToolResult, arguments: call.Arguments, toolResult: toolResult})
|
||||
if options.OnToolResult != nil {
|
||||
options.OnToolResult(toolResult)
|
||||
@@ -932,13 +775,13 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
abortErr = ctx.Err()
|
||||
}
|
||||
if abortErr != nil {
|
||||
closeOutRemaining(index + 1)
|
||||
messages = appendAbortedToolResults(messages, collected.ToolCalls[index+1:])
|
||||
messages = append(messages, buildToolImageMessages()...)
|
||||
result.Messages = copyMessages(messages)
|
||||
return result, abortErr
|
||||
}
|
||||
if stopReason := stopReasonFromToolResult(toolResult); stopReason != "" {
|
||||
closeOutRemaining(index + 1)
|
||||
messages = appendAbortedToolResults(messages, collected.ToolCalls[index+1:])
|
||||
messages = append(messages, buildToolImageMessages()...)
|
||||
result.FinalAnswer = toolResult.ModelOutput()
|
||||
result.StopReason = stopReason
|
||||
@@ -953,29 +796,8 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// aren't fixed by reformatting the call, so a "match this schema" hint
|
||||
// would misdirect the model toward JSON shape or blocked behavior.
|
||||
retriableFailure := isRetriableToolError(toolResult)
|
||||
// A categorized denial is NOT retriable — retrying it verbatim is
|
||||
// pointless — but it is still a failure the streaks must count, or a
|
||||
// refused tool loops until the turn limit. Passing retriableFailure for
|
||||
// both is what let that happen: observeToolResult took its success
|
||||
// branch and deleted the record before it could key on the category.
|
||||
// isPolicyRefusal rather than DenialReason alone: the categories are
|
||||
// only attached where a typed denial is built, so a headless prompt
|
||||
// refusal and an uncategorized sandbox block were counted as SUCCESS
|
||||
// and cleared the record they were supposed to accumulate.
|
||||
policyRefusal := isPolicyRefusal(toolResult)
|
||||
countedFailure := retriableFailure || policyRefusal
|
||||
outcome := guards.observeToolResult(call.Name, countedFailure, retriableFailure, toolResult.ModelOutput(), toolResult.DenialReason)
|
||||
// The profile's failure-streak trigger is for RETRIABLE failures: it
|
||||
// restores the displaced turn budget and reasoning effort on the theory
|
||||
// that the tool is struggling and needs room. A policy refusal is not a
|
||||
// struggling tool, it is an answer, and spending the one-shot
|
||||
// escalation on one contradicts that trigger's own contract. Denials
|
||||
// still count for the halt above; they just do not buy more budget.
|
||||
if policyRefusal {
|
||||
posture.observeToolOutcome(toolFailureOutcome{}, toolResult)
|
||||
} else {
|
||||
posture.observeToolOutcome(outcome, toolResult)
|
||||
}
|
||||
outcome := guards.observeToolResult(call.Name, retriableFailure, toolResult.ModelOutput())
|
||||
posture.observeToolOutcome(outcome, toolResult)
|
||||
if outcome.Stop {
|
||||
// The assistant message advertised EVERY collected tool call, but
|
||||
// the guard halts mid-turn so the calls after this one never run.
|
||||
@@ -983,22 +805,10 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
// every tool_use has a matching tool_result and the recorded
|
||||
// messages stay valid for a strict provider replay (Anthropic
|
||||
// rejects a tool_use with no answering tool_result).
|
||||
closeOutRemaining(index + 1)
|
||||
messages = appendAbortedToolResults(messages, collected.ToolCalls[index+1:])
|
||||
messages = append(messages, buildToolImageMessages()...)
|
||||
result.FinalAnswer = toolFailureStopAnswer(call.Name, outcome.Count, outcome.Cause)
|
||||
result.FinalAnswer = toolFailureStopAnswer(call.Name, outcome.Count)
|
||||
result.Messages = copyMessages(messages)
|
||||
// THE HALT HAS TO CROSS THE TERMINAL-STATUS BOUNDARY TOO.
|
||||
//
|
||||
// This branch returns straight out, so it never reached the
|
||||
// completion gate that the max-turns paths below go through. Under
|
||||
// RequireCompletionSignal `zero exec` treats only Incomplete as exit
|
||||
// 4 and otherwise emits run_end("success", 0), so a task that was
|
||||
// denied six times, or failed through the twelve-call bound, became
|
||||
// a successful automation result having done none of the work.
|
||||
if options.RequireCompletionSignal {
|
||||
result.Incomplete = true
|
||||
result.IncompleteReason = toolFailureIncompleteReason(call.Name, outcome.Cause)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
if outcome.InjectHint && failureHint == "" {
|
||||
@@ -1167,31 +977,22 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
|
||||
})
|
||||
}
|
||||
finalExposed, _ := partitionToolsCached(registry, permissionMode, options, loaded, toolDefCache)
|
||||
// WHICH BOUND FIRED reaches the reader, because the two call for different
|
||||
// responses: a turn-count stop is a round-trip problem, a token stop is a
|
||||
// spend one, and reporting the first for the second sends someone to raise a
|
||||
// limit that was never what ran out.
|
||||
finalPrompt, exhaustedAnswer, limitName := maxTurnsFinalAnswerPrompt, maxTurnsAnswer, "max-turns limit"
|
||||
if stoppedOnTokens {
|
||||
finalPrompt, exhaustedAnswer, limitName = maxTokensFinalAnswerPrompt, maxTokensAnswer, "token budget"
|
||||
}
|
||||
|
||||
if answer, finalMessages, finishReason := finalAnswerAfterMaxTurns(ctx, provider, planner, messages, finalExposed, options, finalPrompt); strings.TrimSpace(answer) != "" {
|
||||
if answer, finalMessages, finishReason := finalAnswerAfterMaxTurns(ctx, provider, planner, messages, finalExposed, options); strings.TrimSpace(answer) != "" {
|
||||
result.FinalAnswer = answer
|
||||
result.FinishReason = finishReason
|
||||
result.Messages = copyMessages(finalMessages)
|
||||
if options.RequireCompletionSignal {
|
||||
result.Incomplete = true
|
||||
result.IncompleteReason = "reached the " + limitName + " without completing"
|
||||
result.IncompleteReason = "reached the max-turns limit without completing"
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
result.FinalAnswer = exhaustedAnswer
|
||||
result.FinalAnswer = maxTurnsAnswer
|
||||
result.Messages = copyMessages(messages)
|
||||
if options.RequireCompletionSignal {
|
||||
result.Incomplete = true
|
||||
result.IncompleteReason = "reached the " + limitName + " without a final answer"
|
||||
result.IncompleteReason = "reached the max-turns limit without a final answer"
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
@@ -1258,11 +1059,11 @@ func recordContextPlanTrace(recorder *trace.Recorder, plan contextPlan) {
|
||||
})
|
||||
}
|
||||
|
||||
func finalAnswerAfterMaxTurns(ctx context.Context, provider Provider, planner *contextPlanner, messages []zeroruntime.Message, toolDefs []zeroruntime.ToolDefinition, options Options, prompt string) (string, []zeroruntime.Message, string) {
|
||||
func finalAnswerAfterMaxTurns(ctx context.Context, provider Provider, planner *contextPlanner, messages []zeroruntime.Message, toolDefs []zeroruntime.ToolDefinition, options Options) (string, []zeroruntime.Message, string) {
|
||||
finalMessages := copyMessages(messages)
|
||||
finalMessages = append(finalMessages, zeroruntime.Message{
|
||||
Role: zeroruntime.MessageRoleUser,
|
||||
Content: prompt,
|
||||
Content: maxTurnsFinalAnswerPrompt,
|
||||
})
|
||||
// The max-turns final-answer call is a pre-content connect, often after a long
|
||||
// autonomous/cron run — route it through the reconnect helper so a single
|
||||
@@ -1673,17 +1474,10 @@ func executeToolCall(ctx context.Context, registry *tools.Registry, call ToolCal
|
||||
}
|
||||
args = shellExecutionArgsForApproval(call.Name, args, decisionAction, options)
|
||||
|
||||
// Wire the progress callback so the TUI sees live tool-call events from a
|
||||
// child agent process.
|
||||
//
|
||||
// Keyed on the TOOL'S OWN DECLARATION (tools.ChildProgressStreamer), not on
|
||||
// its name. This was `call.Name == "Task"`, which made the second
|
||||
// sub-agent-spawning tool run invisibly; `|| call.Name == "orchestrate"`
|
||||
// would have been the same defect one name later. A tool that spawns
|
||||
// children declares it, and every tool that does not keeps the nil callback
|
||||
// it has today.
|
||||
// Task tool: wire progress callback so the TUI sees live tool-call events
|
||||
// from the specialist child process.
|
||||
var progressCallback func(streamjson.Event)
|
||||
if options.OnToolProgress != nil && tools.StreamsChildProgress(tool) {
|
||||
if call.Name == "Task" && options.OnToolProgress != nil {
|
||||
toolCallID := call.ID
|
||||
onProgress := options.OnToolProgress
|
||||
progressCallback = func(event streamjson.Event) {
|
||||
@@ -1702,7 +1496,6 @@ func executeToolCall(ctx context.Context, registry *tools.Registry, call ToolCal
|
||||
ReasoningEffort: options.ReasoningEffort,
|
||||
Depth: options.Depth,
|
||||
Cwd: options.Cwd,
|
||||
UserMessage: options.userMessage,
|
||||
// Per-session file version tracker so write_file/edit_file refuse to clobber
|
||||
// a file that changed on disk outside Zero since it was last read.
|
||||
FileTracker: options.FileTracker,
|
||||
@@ -1797,41 +1590,9 @@ func executeToolCall(ctx context.Context, registry *tools.Registry, call ToolCal
|
||||
// the Run turn loop performs the actual provider switch. Empty for every
|
||||
// ordinary tool result.
|
||||
RequestedModel: result.Meta["escalate_to_model"],
|
||||
// ONE REFUSAL IDENTITY, derived here so classification and streak
|
||||
// accounting cannot disagree.
|
||||
//
|
||||
// The registry marks its pre-execution refusals in metadata, and
|
||||
// isPolicyRefusal read that marker while observeToolResult keyed on
|
||||
// DenialReason, which those paths leave empty. The guard therefore fell
|
||||
// back to errorSignature(output), and two refusals of the same category
|
||||
// with different wording looked like two different failures. A model
|
||||
// alternating capture_artifact's browser_screenshot and browser_pdf against
|
||||
// a disabled driver is refused identically each time, and never tripped the
|
||||
// six-call refusal halt: only the generic twelve-error fallback stopped it,
|
||||
// reporting varied errors rather than a repeated refusal.
|
||||
DenialReason: denialCategoryForResult(result),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// denialCategoryForResult gives every pre-execution refusal a stable category,
|
||||
// whether it was built as a typed denial or only marked in metadata.
|
||||
//
|
||||
// The mapping lives here, at the one boundary tools.Result becomes ToolResult,
|
||||
// rather than at each producer. A marker without a category is the shape that
|
||||
// caused this: it classified as a refusal and keyed as an error signature.
|
||||
func denialCategoryForResult(result tools.Result) DenialCategory {
|
||||
switch result.Meta[tools.PolicyRefusalMeta] {
|
||||
case tools.PolicyRefusalToolNotEnabled:
|
||||
return DenialFiltered
|
||||
case tools.PolicyRefusalPermissionDenied, tools.PolicyRefusalPermissionRequired:
|
||||
return DenialPermissionDenied
|
||||
case tools.PolicyRefusalSandboxDenied, tools.PolicyRefusalSandboxApproval:
|
||||
return DenialSandboxBlock
|
||||
default:
|
||||
return DenialNone
|
||||
}
|
||||
}
|
||||
|
||||
const sandboxNamespaceLimitedReason = "sandbox output is limited to the sandbox PID namespace; host/global state requires approval"
|
||||
|
||||
func maybeRetryUnsandboxedAfterSandboxRestriction(ctx context.Context, registry *tools.Registry, call ToolCall, tool tools.Tool, args map[string]any, result tools.Result, permissionMode PermissionMode, options Options, progressCallback func(streamjson.Event)) (tools.Result, *ToolResult, bool, PermissionDecisionAction, string, []string, error) {
|
||||
@@ -2162,10 +1923,6 @@ func toolResultFromPrePermissionReject(call ToolCall, result tools.Result) ToolR
|
||||
Display: display,
|
||||
LoadedTools: loadedToolsFromResult(meta),
|
||||
RequestedModel: meta["escalate_to_model"],
|
||||
// The SAME identity the executed path derives. This is the route a
|
||||
// RejectBeforePermission refusal takes, so leaving it empty here is what
|
||||
// made the marker classify as a refusal and key as an error signature.
|
||||
DenialReason: denialCategoryForResult(tools.Result{Meta: meta}),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2337,64 +2094,21 @@ func isRetriableToolError(result ToolResult) bool {
|
||||
// A categorized denial (filtered / permission / sandbox) is a policy decision,
|
||||
// not a transient failure — never retry it. This is robust to message wording
|
||||
// (the text checks below remain as a fallback for results lacking the field).
|
||||
return !isPolicyRefusal(result)
|
||||
}
|
||||
|
||||
// isPolicyRefusal reports a result the run refused on policy grounds: a
|
||||
// permission gate, a filter, a sandbox preflight, or a hook.
|
||||
//
|
||||
// Split out of isRetriableToolError so the two questions cannot drift apart,
|
||||
// which is exactly what they had done. The halt guard asked
|
||||
// `DenialReason != DenialNone`, and a category is only attached on the paths
|
||||
// that build a TYPED denial. A headless run leaves OnPermissionRequest nil, so
|
||||
// the loop never reaches its typed-denial branch and the registry returns a bare
|
||||
// `Error: Permission required ...` carrying no category. A sandbox preflight
|
||||
// denial on a non-shell tool loses its SandboxDecision converting to ToolResult
|
||||
// and arrives as an uncategorized `Sandbox block`.
|
||||
//
|
||||
// Both are refusals, and both were counted as SUCCESS, so observeToolResult
|
||||
// cleared the record and the same refused call could repeat to MaxTurns. That is
|
||||
// the loop this guard exists to stop. The text checks below already enumerated
|
||||
// these outcomes for the retriable question; they simply were not reachable from
|
||||
// the counting one.
|
||||
func isPolicyRefusal(result ToolResult) bool {
|
||||
// A REFUSAL IS A FAILURE. Everything below inspects metadata and then output
|
||||
// text, and neither is meaningful on a result the tool completed.
|
||||
//
|
||||
// isRetriableToolError gates on this before it calls here, so the boundary
|
||||
// held while that was the only caller. Extracting this helper and calling it
|
||||
// straight from the counting path dropped the gate: an allowed `bash`
|
||||
// printing "Sandbox block", or a read_file returning a document that quotes
|
||||
// one of these phrases, was counted as a denial. Six such successes tripped
|
||||
// the same-signature stop and ended a healthy run with a refusal answer.
|
||||
//
|
||||
// The gate belongs here rather than at each caller, because the next caller
|
||||
// will forget it too.
|
||||
if result.Status != tools.StatusError {
|
||||
if result.DenialReason != DenialNone {
|
||||
return false
|
||||
}
|
||||
if result.DenialReason != DenialNone {
|
||||
return true
|
||||
if result.Meta["permission_action"] == string(PermissionActionDeny) {
|
||||
return false
|
||||
}
|
||||
if result.Meta[tools.PermissionActionMeta] == string(PermissionActionDeny) {
|
||||
return true
|
||||
switch {
|
||||
case strings.Contains(result.Output, "is not enabled for this run"),
|
||||
strings.Contains(result.Output, "Permission denied for "),
|
||||
strings.Contains(result.Output, "Permission required for "),
|
||||
strings.Contains(result.Output, "Sandbox block"),
|
||||
strings.Contains(result.Output, "Sandbox approval required for "):
|
||||
return false
|
||||
}
|
||||
// STRUCTURED ONLY. This used to fall back to matching phrases in Output
|
||||
// ("Sandbox block", "Permission denied for ", and so on), which cannot work:
|
||||
// Output is tool-controlled. `bash` preserves arbitrary stdout and stderr on
|
||||
// a StatusError for any nonzero exit, so an ALLOWED command running
|
||||
// `printf 'Sandbox block\n' >&2; exit 1` has actually executed, carries no
|
||||
// DenialReason, and was still classified as refused. The loop then withheld
|
||||
// the schema hint and the profile failure-streak recovery, counted the
|
||||
// executed failure toward the refusal halt, and a later stop told the user a
|
||||
// tool had been refused when it had run.
|
||||
//
|
||||
// read_file returning a document that quotes one of those phrases did the
|
||||
// same thing, which is the more likely way a real session hits it.
|
||||
//
|
||||
// The registry now marks every path that returns BEFORE the tool runs, so
|
||||
// the question is answerable from provenance. Nothing below reads Output.
|
||||
return tools.IsPolicyRefusalResult(tools.Result{Meta: result.Meta})
|
||||
return true
|
||||
}
|
||||
|
||||
// scrubInterceptedOutput mirrors the registry's scrubResultSecrets boundary for
|
||||
@@ -2960,7 +2674,7 @@ func deniedPermissionResult(call ToolCall, reason string, requestEvent Permissio
|
||||
Output: "Error: Permission denied for " + call.Name + ": " + reason,
|
||||
DenialReason: denial,
|
||||
Meta: map[string]string{
|
||||
tools.PermissionActionMeta: string(event.Action),
|
||||
"permission_action": string(event.Action),
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -2986,7 +2700,7 @@ func canceledPermissionResult(call ToolCall, reason string, requestEvent Permiss
|
||||
Output: "Error: Permission approval cancelled for " + call.Name + ": " + reason,
|
||||
DenialReason: DenialApprovalCanceled,
|
||||
Meta: map[string]string{
|
||||
tools.PermissionActionMeta: string(event.Action),
|
||||
"permission_action": string(event.Action),
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -3210,7 +2924,7 @@ func availablePermissionDecisions(event PermissionEvent, args map[string]any, op
|
||||
decisions = append(decisions, PermissionDecisionAlwaysAllowPrefix)
|
||||
}
|
||||
}
|
||||
if options.Sandbox.CanPersistGrants() && permissionSupportsPersistentDecision(event.ToolName, options) && !filesystemSandboxPrompt(event) && !inlineAdditionalPermissions {
|
||||
if options.Sandbox.CanPersistGrants() && permissionSupportsPersistentDecision(event.ToolName) && !filesystemSandboxPrompt(event) && !inlineAdditionalPermissions {
|
||||
decisions = append(decisions, PermissionDecisionAlwaysAllow)
|
||||
}
|
||||
}
|
||||
@@ -3412,17 +3126,7 @@ func grantFilesystemForSandboxPrompt(event PermissionEvent, scope sandbox.Permis
|
||||
}, scope)
|
||||
}
|
||||
|
||||
func permissionSupportsPersistentDecision(toolName string, options Options) bool {
|
||||
// THE TOOL'S OWN DECLARATION FIRST. A name list cannot describe a tool whose
|
||||
// reach depends on its arguments, and it cannot cover a tool that RUNS the
|
||||
// ones already refused below — see tools.PersistentPermissionRefuser.
|
||||
if options.Registry != nil {
|
||||
if tool, found := options.Registry.Get(toolName); found {
|
||||
if refuser, ok := tool.(tools.PersistentPermissionRefuser); ok && refuser.RefusesPersistentPermission() {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
func permissionSupportsPersistentDecision(toolName string) bool {
|
||||
switch toolName {
|
||||
case "bash", "exec_command", "write_stdin", "apply_patch":
|
||||
return false
|
||||
@@ -3850,33 +3554,6 @@ func toolResultImageMessage(result ToolResult, options Options) (zeroruntime.Mes
|
||||
}, true
|
||||
}
|
||||
|
||||
// precomputedResultFor reports the read-ahead result for call index, and whether
|
||||
// that call actually produced one. A batch entry that never started is still
|
||||
// unstarted as far as the transcript is concerned.
|
||||
//
|
||||
// PRODUCING A RESULT AND ASKING THE RUN TO STOP ARE DIFFERENT FACTS. Treating a
|
||||
// non-nil abortErr as "unstarted" conflated them, and a cancelled permission
|
||||
// request is exactly the case where they disagree: executeToolCall builds the
|
||||
// cancellation result and returns it WITH ErrPermissionApprovalCanceled, so an
|
||||
// earlier sibling reaching a terminal branch discarded a real result and wrote
|
||||
// an aborted placeholder in its place. The transcript then denied that the call
|
||||
// had completed permission handling, and the cancellation was missing from
|
||||
// OnToolResult, the trace counters and the task observation even though the
|
||||
// permission event had already happened.
|
||||
//
|
||||
// The terminal decision is not affected: draining such a sibling only records
|
||||
// what occurred, it cannot reverse a stop already selected.
|
||||
func precomputedResultFor(precomputed []precomputedToolResult, start, end, index int) (ToolResult, bool) {
|
||||
if index < start || index >= end {
|
||||
return ToolResult{}, false
|
||||
}
|
||||
entry := precomputed[index-start]
|
||||
if !entry.completed {
|
||||
return ToolResult{}, false
|
||||
}
|
||||
return entry.result, true
|
||||
}
|
||||
|
||||
func modelAcceptsToolImages(options Options) bool {
|
||||
if options.SupportsVision != nil {
|
||||
return options.SupportsVision(options.Model)
|
||||
|
||||
+4
-152
@@ -685,23 +685,11 @@ func TestIsRetriableToolError(t *testing.T) {
|
||||
{"success", ToolResult{Status: tools.StatusOK}, false},
|
||||
{"bad arguments", ToolResult{Status: tools.StatusError, Output: "Error: Failed to parse arguments for x: bad json"}, true},
|
||||
{"execution failure", ToolResult{Status: tools.StatusError, Output: "Error: read foo.txt: no such file"}, true},
|
||||
// Each refusal carries the provenance its production path attaches: the
|
||||
// disabled-tool path sets a denial category, the registry gates set the
|
||||
// pre-execution marker. Text is never what decides.
|
||||
{"disabled tool", ToolResult{Status: tools.StatusError, Output: `Error: Tool "x" is not enabled for this run.`, DenialReason: DenialFiltered}, false},
|
||||
{"disabled tool", ToolResult{Status: tools.StatusError, Output: `Error: Tool "x" is not enabled for this run.`}, false},
|
||||
{"permission denied (meta)", ToolResult{Status: tools.StatusError, Output: "Error: Permission denied for x: needs approval", Meta: map[string]string{"permission_action": "deny"}}, false},
|
||||
{"permission required", ToolResult{Status: tools.StatusError, Output: "Error: Permission required for x: approve first", Meta: map[string]string{tools.PolicyRefusalMeta: tools.PolicyRefusalPermissionRequired}}, false},
|
||||
{"sandbox block", ToolResult{Status: tools.StatusError, Output: "Error: Sandbox block: outside_workspace", Meta: map[string]string{tools.PolicyRefusalMeta: tools.PolicyRefusalSandboxDenied}}, false},
|
||||
{"sandbox approval", ToolResult{Status: tools.StatusError, Output: "Error: Sandbox approval required for x: network", Meta: map[string]string{tools.PolicyRefusalMeta: tools.PolicyRefusalSandboxApproval}}, false},
|
||||
// THE INVERSE, which is the failure this classifier had. An allowed tool
|
||||
// that RAN, failed, and happened to print one of those phrases must stay
|
||||
// retriable: `bash` preserves arbitrary stdout and stderr on any nonzero
|
||||
// exit, and read_file returns whatever the file says.
|
||||
{"executed failure printing the sandbox phrase", ToolResult{Status: tools.StatusError, Output: "Sandbox block\n"}, true},
|
||||
{"executed failure printing a permission denial", ToolResult{Status: tools.StatusError, Output: "cp: Permission denied for /etc/hosts"}, true},
|
||||
{"executed failure printing a permission requirement", ToolResult{Status: tools.StatusError, Output: "Permission required for sudo"}, true},
|
||||
{"executed failure printing the approval phrase", ToolResult{Status: tools.StatusError, Output: "Sandbox approval required for curl"}, true},
|
||||
{"executed failure quoting the disabled-tool phrase", ToolResult{Status: tools.StatusError, Output: "grep: web_fetch is not enabled for this run"}, true},
|
||||
{"permission required", ToolResult{Status: tools.StatusError, Output: "Error: Permission required for x: approve first"}, false},
|
||||
{"sandbox block", ToolResult{Status: tools.StatusError, Output: "Error: Sandbox block: outside_workspace"}, false},
|
||||
{"sandbox approval", ToolResult{Status: tools.StatusError, Output: "Error: Sandbox approval required for x: network"}, false},
|
||||
// Structured denial categories are authoritative regardless of message text.
|
||||
{"denial: filtered", ToolResult{Status: tools.StatusError, Output: "anything", DenialReason: DenialFiltered}, false},
|
||||
{"denial: permission", ToolResult{Status: tools.StatusError, Output: "anything", DenialReason: DenialPermissionDenied}, false},
|
||||
@@ -4373,142 +4361,6 @@ func TestPlanModeHonorsBeforeToolVeto(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// (h) The zeromaxing variant of TestRunPreservesRequestPrefixAcrossTurns.
|
||||
//
|
||||
// This is the test the whole reminder design exists to satisfy. The system
|
||||
// prompt and tool definitions are the provider's CACHED PREFIX; #760 made them
|
||||
// build once per run so they stay byte-identical across turns. The posture adds
|
||||
// per-turn text, so it is exactly the kind of change that silently destroys the
|
||||
// cache — roughly doubling input cost with nothing detecting it.
|
||||
//
|
||||
// The reminders are appended to the conversation TAIL as user-role messages, so
|
||||
// each turn's request must still be an exact prefix-extension of the previous
|
||||
// one, with identical tools and cache key.
|
||||
func TestRunPreservesRequestPrefixAcrossTurnsUnderZeromaxing(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
writeAgentTestFile(t, filepath.Join(root, "notes.txt"), "alpha\n")
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tools.NewReadFileTool(root))
|
||||
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
|
||||
{
|
||||
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call-1", ToolName: "read_file"},
|
||||
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call-1", ArgumentsFragment: `{"path":"notes.txt"}`},
|
||||
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call-1"},
|
||||
{Type: zeroruntime.StreamEventDone},
|
||||
},
|
||||
{
|
||||
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call-2", ToolName: "read_file"},
|
||||
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call-2", ArgumentsFragment: `{"path":"notes.txt"}`},
|
||||
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call-2"},
|
||||
{Type: zeroruntime.StreamEventDone},
|
||||
},
|
||||
{
|
||||
{Type: zeroruntime.StreamEventText, Content: "done"},
|
||||
{Type: zeroruntime.StreamEventDone},
|
||||
},
|
||||
}}
|
||||
|
||||
if _, err := Run(context.Background(), "read notes", provider, Options{
|
||||
Cwd: root,
|
||||
Registry: registry,
|
||||
SessionID: "session-stable-prefix-zeromaxing",
|
||||
Zeromaxing: ZeromaxingEntering,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(provider.requests) != 3 {
|
||||
t.Fatalf("provider requests = %d, want 3", len(provider.requests))
|
||||
}
|
||||
|
||||
// Every request must extend the previous one exactly — nothing rewritten
|
||||
// above the append point, which is what keeps the cached prefix valid.
|
||||
for i := 1; i < len(provider.requests); i++ {
|
||||
prev, cur := provider.requests[i-1], provider.requests[i]
|
||||
if len(cur.Messages) < len(prev.Messages) ||
|
||||
!reflect.DeepEqual(cur.Messages[:len(prev.Messages)], prev.Messages) {
|
||||
t.Fatalf("request %d is not an exact prefix-extension of request %d:\nprev=%#v\ncur=%#v",
|
||||
i, i-1, prev.Messages, cur.Messages)
|
||||
}
|
||||
if !reflect.DeepEqual(prev.Tools, cur.Tools) {
|
||||
t.Fatalf("tool definitions drifted between turns %d and %d", i-1, i)
|
||||
}
|
||||
if cur.PromptCacheKey != prev.PromptCacheKey {
|
||||
t.Fatalf("prompt cache key drifted: %q -> %q", prev.PromptCacheKey, cur.PromptCacheKey)
|
||||
}
|
||||
}
|
||||
|
||||
// The system prompt — the cached prefix itself — must be byte-identical
|
||||
// across turns, and must contain NONE of the posture text.
|
||||
system := provider.requests[0].Messages[0]
|
||||
for i, request := range provider.requests {
|
||||
if !reflect.DeepEqual(request.Messages[0], system) {
|
||||
t.Fatalf("system message changed on turn %d — the cached prefix is broken", i+1)
|
||||
}
|
||||
}
|
||||
for _, forbidden := range []string{
|
||||
ZeromaxingEnterNotice, ZeromaxingBudgetNotice, ZeromaxingStillOnNotice, ZeromaxingExitNotice,
|
||||
ZeromaxingEvidenceNotice, ZeromaxingOrchestrateNotice,
|
||||
} {
|
||||
if strings.Contains(system.Content, forbidden) {
|
||||
t.Fatalf("posture reminder leaked into the SYSTEM PROMPT (above the cache breakpoint): %q", forbidden)
|
||||
}
|
||||
}
|
||||
|
||||
// ...and the reminders really did arrive, on schedule. Without this the
|
||||
// assertions above would pass just as happily if the feature did nothing.
|
||||
firstTurn := renderZeromaxingMessages(provider.requests[0].Messages)
|
||||
if !strings.Contains(firstTurn, ZeromaxingEnterNotice) || !strings.Contains(firstTurn, ZeromaxingBudgetNotice) {
|
||||
t.Fatalf("turn 1 must carry the enter + budget notices:\n%s", firstTurn)
|
||||
}
|
||||
if strings.Contains(firstTurn, ZeromaxingStillOnNotice) {
|
||||
t.Fatalf("turn 1 must NOT carry the still-on notice:\n%s", firstTurn)
|
||||
}
|
||||
secondOnly := renderZeromaxingMessages(provider.requests[1].Messages[len(provider.requests[0].Messages):])
|
||||
if !strings.Contains(secondOnly, ZeromaxingStillOnNotice) {
|
||||
t.Fatalf("turn 2 must carry the still-on notice:\n%s", secondOnly)
|
||||
}
|
||||
if strings.Contains(secondOnly, ZeromaxingEnterNotice) {
|
||||
t.Fatalf("turn 2 must NOT re-announce entry:\n%s", secondOnly)
|
||||
}
|
||||
}
|
||||
|
||||
// A run with the posture OFF must be byte-identical to one that never heard of
|
||||
// it — the no-regression half of the feature.
|
||||
func TestRunWithoutZeromaxingCarriesNoPostureText(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tools.NewReadFileTool(root))
|
||||
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
|
||||
{{Type: zeroruntime.StreamEventText, Content: "done"}, {Type: zeroruntime.StreamEventDone}},
|
||||
}}
|
||||
if _, err := Run(context.Background(), "hello", provider, Options{
|
||||
Cwd: root, Registry: registry, SessionID: "session-no-zeromaxing",
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rendered := renderZeromaxingMessages(provider.requests[0].Messages)
|
||||
for _, forbidden := range []string{
|
||||
ZeromaxingEnterNotice, ZeromaxingBudgetNotice, ZeromaxingStillOnNotice, ZeromaxingExitNotice,
|
||||
ZeromaxingEvidenceNotice, ZeromaxingOrchestrateNotice,
|
||||
} {
|
||||
if strings.Contains(rendered, forbidden) {
|
||||
t.Fatalf("a posture-free run must carry no posture text, found %q", forbidden)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// renderZeromaxingMessages flattens messages to one searchable string.
|
||||
func renderZeromaxingMessages(messages []zeroruntime.Message) string {
|
||||
var b strings.Builder
|
||||
for _, message := range messages {
|
||||
b.WriteString(string(message.Role))
|
||||
b.WriteString(": ")
|
||||
b.WriteString(message.Content)
|
||||
b.WriteString("\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// A cancelled permission on either sandbox-retry path must abort the run with a
|
||||
// recognisable cancellation, and must not run the escalated retry.
|
||||
//
|
||||
|
||||
@@ -1,149 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// goTestTool stands in for a shell that ran `go test` and printed its result.
|
||||
type goTestTool struct{ output string }
|
||||
|
||||
func (goTestTool) Name() string { return "run_tests" }
|
||||
func (goTestTool) Description() string { return "run the test suite" }
|
||||
func (goTestTool) Parameters() tools.Schema {
|
||||
return tools.Schema{Type: "object", AdditionalProperties: false}
|
||||
}
|
||||
func (goTestTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectNone, Permission: tools.PermissionAllow, AdvertiseInAuto: true}
|
||||
}
|
||||
func (t goTestTool) Run(context.Context, map[string]any) tools.Result {
|
||||
return tools.Result{Status: tools.StatusOK, Output: t.output}
|
||||
}
|
||||
|
||||
// callThenAnswer is a provider that calls run_tests, then gives finalAnswer, then
|
||||
// gives corrected — so a run that is sent back for a number gets somewhere to go.
|
||||
func callThenAnswer(finalAnswer, corrected string) *mockProvider {
|
||||
return &mockProvider{turns: [][]zeroruntime.StreamEvent{
|
||||
{
|
||||
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call-1", ToolName: "run_tests"},
|
||||
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call-1", ArgumentsFragment: `{}`},
|
||||
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call-1"},
|
||||
{Type: zeroruntime.StreamEventDone},
|
||||
},
|
||||
{{Type: zeroruntime.StreamEventText, Content: finalAnswer}, {Type: zeroruntime.StreamEventDone}},
|
||||
{{Type: zeroruntime.StreamEventText, Content: corrected}, {Type: zeroruntime.StreamEventDone}},
|
||||
}}
|
||||
}
|
||||
|
||||
const suiteOutput = `{"Action":"pass","Package":"github.com/x/y","Test":"TestChattyChild","Elapsed":0.86}
|
||||
{"Action":"pass","Package":"github.com/x/y","Elapsed":0.86}
|
||||
`
|
||||
|
||||
// A FINAL ANSWER THAT REPORTS A NUMBER THE RUN NEVER PRODUCED IS SENT BACK.
|
||||
//
|
||||
// The measured failure: a benchmark table where the same test read 0.86s in one
|
||||
// paste and 4.20s in the next, with nothing said about the difference. A prompt
|
||||
// rule cannot catch this — a model willing to write a number it did not measure
|
||||
// is equally willing to say it re-ran it. The transcript can.
|
||||
func TestAFinalAnswerContradictingTheTranscriptIsSentBack(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(goTestTool{output: suiteOutput})
|
||||
provider := callThenAnswer("github.com/x/y took 4.20s.", "Corrected: github.com/x/y took 0.86s.")
|
||||
|
||||
result, err := Run(context.Background(), "run the suite", provider, Options{
|
||||
Cwd: root, Registry: registry, SessionID: "session-measured",
|
||||
Zeromaxing: ZeromaxingActive,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if len(provider.requests) < 3 {
|
||||
t.Fatalf("the run ended after %d requests: the contradicted answer was accepted", len(provider.requests))
|
||||
}
|
||||
sentBack := renderZeromaxingMessages(provider.requests[2].Messages)
|
||||
for _, required := range []string{"github.com/x/y", "4.2s", "0.86s", "Re-run the command"} {
|
||||
if !strings.Contains(sentBack, required) {
|
||||
t.Errorf("the correction does not mention %q:\n%s", required, sentBack)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(result.FinalAnswer, "0.86s") {
|
||||
t.Errorf("the corrected answer was not the one returned: %q", result.FinalAnswer)
|
||||
}
|
||||
}
|
||||
|
||||
// AN HONEST ANSWER IS RETURNED UNTOUCHED. A tripwire that fires on a correct
|
||||
// report gets switched off, and then it catches nothing.
|
||||
func TestAnAnswerThatMatchesTheTranscriptIsReturnedAsIs(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(goTestTool{output: suiteOutput})
|
||||
provider := callThenAnswer("github.com/x/y took 0.86s.", "should never be reached")
|
||||
|
||||
result, err := Run(context.Background(), "run the suite", provider, Options{
|
||||
Cwd: root, Registry: registry, SessionID: "session-honest",
|
||||
Zeromaxing: ZeromaxingActive,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(provider.requests) != 2 {
|
||||
t.Fatalf("an honest answer cost %d requests, want 2: the check fired on a correct report", len(provider.requests))
|
||||
}
|
||||
if !strings.Contains(result.FinalAnswer, "0.86s") {
|
||||
t.Errorf("final answer = %q", result.FinalAnswer)
|
||||
}
|
||||
}
|
||||
|
||||
// THE CHECK IS PART OF THE POSTURE. With zeromaxing off the run must take
|
||||
// exactly the path it took before this existed — the same rule every other
|
||||
// posture behaviour follows.
|
||||
func TestThePostureOffRunIsNeverSentBackForANumber(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(goTestTool{output: suiteOutput})
|
||||
provider := callThenAnswer("github.com/x/y took 4.20s.", "should never be reached")
|
||||
|
||||
result, err := Run(context.Background(), "run the suite", provider, Options{
|
||||
Cwd: root, Registry: registry, SessionID: "session-posture-off",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(provider.requests) != 2 {
|
||||
t.Fatalf("a posture-off run cost %d requests, want 2: the check reached a path it must not", len(provider.requests))
|
||||
}
|
||||
if !strings.Contains(result.FinalAnswer, "4.20s") {
|
||||
t.Errorf("a posture-off run's answer was altered: %q", result.FinalAnswer)
|
||||
}
|
||||
}
|
||||
|
||||
// AN UNCORRECTED ANSWER IS RETURNED RATHER THAN ASKED AGAIN. The ledger raises
|
||||
// each name once, so a model that repeats the number ends the run instead of
|
||||
// looping until the turn budget is gone.
|
||||
func TestARepeatedNumberEndsTheRunRatherThanLooping(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(goTestTool{output: suiteOutput})
|
||||
// Says the same wrong number both times.
|
||||
provider := callThenAnswer("github.com/x/y took 4.20s.", "github.com/x/y took 4.20s.")
|
||||
|
||||
result, err := Run(context.Background(), "run the suite", provider, Options{
|
||||
Cwd: root, Registry: registry, SessionID: "session-stubborn",
|
||||
Zeromaxing: ZeromaxingActive,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(provider.requests) != 3 {
|
||||
t.Fatalf("the run cost %d requests, want exactly 3: one call, one answer, one correction", len(provider.requests))
|
||||
}
|
||||
if !strings.Contains(result.FinalAnswer, "4.20s") {
|
||||
t.Errorf("final answer = %q", result.FinalAnswer)
|
||||
}
|
||||
}
|
||||
@@ -1,74 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// mixedOutcomeTool needs permission on every call. The test denies every other
|
||||
// request, and a call that is allowed executes and fails with an error naming
|
||||
// its own item, so no single identity repeats and only the content-blind bound
|
||||
// can stop the run.
|
||||
type mixedOutcomeTool struct{ ran int }
|
||||
|
||||
func (tool *mixedOutcomeTool) Name() string { return "bash" }
|
||||
func (tool *mixedOutcomeTool) Description() string { return "probe" }
|
||||
func (tool *mixedOutcomeTool) Parameters() tools.Schema { return tools.Schema{Type: "object"} }
|
||||
func (tool *mixedOutcomeTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectShell, Permission: tools.PermissionPrompt, Reason: "runs a command"}
|
||||
}
|
||||
|
||||
func (tool *mixedOutcomeTool) Run(context.Context, map[string]any) tools.Result {
|
||||
tool.ran++
|
||||
return tools.Result{Status: tools.StatusError, Output: "Error: item " + strconv.Itoa(tool.ran) + " failed"}
|
||||
}
|
||||
|
||||
// A MIXED HALT READS THE SAME HEADLESS AS INTERACTIVE.
|
||||
//
|
||||
// The stop answer for the content-blind bound over executed failures and
|
||||
// refusals says the tool "failed or was refused". The headless reason for the
|
||||
// same halt said only that it "failed repeatedly", so zero exec and the
|
||||
// interactive answer disagreed about one stop. Reported by @jatmn.
|
||||
func TestAMixedGuardHaltIsReportedAsMixedUnderTheCompletionGate(t *testing.T) {
|
||||
tool := &mixedOutcomeTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, toolFailureAnyErrorStopAt+4)
|
||||
for i := range toolFailureAnyErrorStopAt + 4 {
|
||||
turns = append(turns, toolTurn("c"+strconv.Itoa(i), "bash", `{"item":`+strconv.Itoa(i)+`}`))
|
||||
}
|
||||
asked := 0
|
||||
result, err := Run(context.Background(), "run them", &mockProvider{turns: turns}, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
RequireCompletionSignal: true,
|
||||
OnPermissionRequest: func(context.Context, PermissionRequest) (PermissionDecision, error) {
|
||||
asked++
|
||||
if asked%2 == 0 {
|
||||
return PermissionDecision{Action: PermissionDecisionAllow}, nil
|
||||
}
|
||||
return PermissionDecision{Action: PermissionDecisionDeny}, nil
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tool.ran == 0 || asked <= tool.ran {
|
||||
t.Fatalf("the run did not mix executed failures with refusals (ran %d of %d asked), so this proves nothing", tool.ran, asked)
|
||||
}
|
||||
if !strings.Contains(result.FinalAnswer, "failed or was refused") {
|
||||
t.Fatalf("the halt was not the mixed content-blind bound: %q", result.FinalAnswer)
|
||||
}
|
||||
if !result.Incomplete {
|
||||
t.Fatal("a guard halt under the completion gate must be Incomplete")
|
||||
}
|
||||
if !strings.Contains(result.IncompleteReason, "failed or was refused") || !strings.Contains(result.IncompleteReason, "bash") {
|
||||
t.Errorf("IncompleteReason = %q, want it to say the tool failed or was refused, as the stop answer does", result.IncompleteReason)
|
||||
}
|
||||
}
|
||||
@@ -1,69 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// THE PROVIDER OUTRANKS THE MODEL ID, because the provider knows and the id only
|
||||
// hints.
|
||||
//
|
||||
// A ChatGPT OAuth session was classified correctly for one reason: "gpt-5.5"
|
||||
// happens to begin with "gpt". Nothing obliges the next id to — OpenAI has
|
||||
// already shipped o1, o3 and o4 — and the prefix list has to be amended each
|
||||
// time to keep a first-party provider working. Asking the provider does not.
|
||||
func TestTheProviderFamilyWinsOverTheModelId(t *testing.T) {
|
||||
// An id no prefix arm matches, from a provider that declares OpenAI.
|
||||
if got := modelPromptAddendum("openai", "some-unreleased-id"); got != openAIPromptAddendum {
|
||||
t.Error("a provider that declares openai did not get the openai addendum for an unrecognised id")
|
||||
}
|
||||
if got := modelPromptAddendum("gemini", "some-unreleased-id"); got != geminiPromptAddendum {
|
||||
t.Error("a provider that declares gemini did not get the gemini addendum")
|
||||
}
|
||||
// Anthropic is aligned with the core prompt and takes no addendum, declared
|
||||
// or guessed. Asserted so "no addendum" stays a decision, not an accident.
|
||||
if got := modelPromptAddendum("anthropic", "some-unreleased-id"); got != "" {
|
||||
t.Errorf("anthropic gained an addendum: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A GATEWAY DECLARES NOTHING, and must fall back to the id — it is the only
|
||||
// thing that distinguishes openai/gpt-4.1 from z-ai/glm-4.6 on one endpoint.
|
||||
func TestAGatewayFallsBackToTheModelId(t *testing.T) {
|
||||
for _, testCase := range []struct {
|
||||
model string
|
||||
want string
|
||||
}{
|
||||
{"openai/gpt-4.1", openAIPromptAddendum},
|
||||
{"google/gemini-2.5-pro", geminiPromptAddendum},
|
||||
{"anthropic/claude-sonnet-4.5", ""},
|
||||
// Unclassified, and honestly so: no family addendum exists for it.
|
||||
{"z-ai/glm-4.6", ""},
|
||||
{"qwen3-coder:480b", ""},
|
||||
} {
|
||||
if got := modelPromptAddendum("", testCase.model); got != testCase.want {
|
||||
t.Errorf("%s: addendum = %q, want %q", testCase.model, truncateForTest(got), truncateForTest(testCase.want))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The wiring is not decoration: a family that reaches Options must reach the
|
||||
// prompt. Asserted through the real assembly rather than the helper, because
|
||||
// asserting the helper is how a value gets threaded to a layer that drops it.
|
||||
func TestTheDeclaredFamilyReachesTheAssembledPrompt(t *testing.T) {
|
||||
withProvider := BuildSystemPromptPreview(Options{ModelFamily: "openai", Model: "some-unreleased-id"})
|
||||
if !strings.Contains(withProvider, "Persist until the task is fully handled this turn") {
|
||||
t.Error("the openai guidance did not reach the assembled prompt when the provider declared it")
|
||||
}
|
||||
withoutProvider := BuildSystemPromptPreview(Options{Model: "some-unreleased-id"})
|
||||
if strings.Contains(withoutProvider, "Persist until the task is fully handled this turn") {
|
||||
t.Error("an unclassified model gained openai guidance from nowhere")
|
||||
}
|
||||
}
|
||||
|
||||
func truncateForTest(text string) string {
|
||||
if len(text) > 40 {
|
||||
return text[:40] + "..."
|
||||
}
|
||||
return text
|
||||
}
|
||||
@@ -1,232 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// readAheadTool is read-only and thread-safe, so a consecutive run of its calls
|
||||
// is executed as one parallel batch before the loop consumes any of it.
|
||||
type readAheadTool struct {
|
||||
mu sync.Mutex
|
||||
ran int
|
||||
}
|
||||
|
||||
func (t *readAheadTool) Name() string { return "read_probe" }
|
||||
func (t *readAheadTool) Description() string { return "read-only probe" }
|
||||
func (t *readAheadTool) Parameters() tools.Schema {
|
||||
return tools.Schema{Type: "object", Properties: map[string]tools.PropertySchema{"id": {Type: "string"}}}
|
||||
}
|
||||
|
||||
func (t *readAheadTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow, Reason: "test"}
|
||||
}
|
||||
|
||||
func (t *readAheadTool) Capabilities() tools.ToolCapabilities {
|
||||
return tools.ToolCapabilities{Effect: tools.EffectReadOnly, ThreadSafe: true}
|
||||
}
|
||||
|
||||
func (t *readAheadTool) Run(_ context.Context, args map[string]any) tools.Result {
|
||||
t.mu.Lock()
|
||||
t.ran++
|
||||
t.mu.Unlock()
|
||||
id, _ := args["id"].(string)
|
||||
// Varying output, so no identity repeats and only the content-blind counter
|
||||
// can halt the run.
|
||||
return tools.Result{Status: tools.StatusError, Output: "failure for " + id}
|
||||
}
|
||||
|
||||
func readCall(id string) []zeroruntime.StreamEvent {
|
||||
return []zeroruntime.StreamEvent{
|
||||
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: id, ToolCallProviderID: "provider-" + id, ToolName: "read_probe"},
|
||||
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: id, ArgumentsFragment: `{"id":"` + id + `"}`},
|
||||
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: id},
|
||||
}
|
||||
}
|
||||
|
||||
// READ-AHEAD MEANS "NOT CONSUMED YET" NO LONGER MEANS "NOT EXECUTED".
|
||||
//
|
||||
// executeParallelReadBatch runs an entire eligible run of read calls before the
|
||||
// loop consumes any of them. Every terminal branch in the consumption loop used
|
||||
// to close out the calls after the current index with aborted placeholders, on
|
||||
// the assumption that they had not run. A sibling that had already executed was
|
||||
// therefore recorded as aborted: its real result was discarded, the transcript
|
||||
// disagreed with what had happened, and its callbacks, trace counter, task
|
||||
// observation and images were lost with it. Where such a sibling is a
|
||||
// successful read, execution may already have committed file-observation credit
|
||||
// for content the model never receives.
|
||||
func TestParallelSiblingsAreFinalizedWhenTheGuardHalts(t *testing.T) {
|
||||
tool := &readAheadTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
// Eleven single-call turns bring the content-blind counter to one below its
|
||||
// bound, so the first call of the pair below trips it.
|
||||
var turns [][]zeroruntime.StreamEvent
|
||||
for i := 0; i < toolFailureAnyErrorStopAt-1; i++ {
|
||||
id := "s" + strconv.Itoa(i)
|
||||
turns = append(turns, append(readCall(id), zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}))
|
||||
}
|
||||
// Then one turn advertising two parallel-eligible calls. Both execute in the
|
||||
// batch; the halt fires while consuming the first.
|
||||
pair := append(readCall("pA"), readCall("pB")...)
|
||||
turns = append(turns, append(pair, zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}))
|
||||
turns = append(turns, append(readCall("unreached"), zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}))
|
||||
|
||||
var reported []ToolResult
|
||||
var announced int
|
||||
result, err := Run(context.Background(), "read things", &mockProvider{turns: turns}, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAuto,
|
||||
MaxTurns: len(turns) + 5,
|
||||
OnToolCall: func(ToolCall) { announced++ },
|
||||
OnToolResult: func(r ToolResult) { reported = append(reported, r) },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasPrefix(result.FinalAnswer, "Agent stopped:") {
|
||||
t.Fatalf("the guard did not halt this run: %q", result.FinalAnswer)
|
||||
}
|
||||
|
||||
// Every execution is reported exactly once, and the announcements match.
|
||||
if tool.ran != len(reported) {
|
||||
t.Errorf("the tool executed %d times but %d results were reported; a completed call was recorded as aborted",
|
||||
tool.ran, len(reported))
|
||||
}
|
||||
if announced != len(reported) {
|
||||
t.Errorf("OnToolCall fired %d times against %d results; the two callbacks disagree", announced, len(reported))
|
||||
}
|
||||
|
||||
// Strict pairing: one tool result per advertised tool call, and the drained
|
||||
// sibling carries its REAL output rather than the aborted placeholder.
|
||||
byID := map[string]int{}
|
||||
var siblingContent string
|
||||
for _, message := range result.Messages {
|
||||
if message.Role != zeroruntime.MessageRoleTool {
|
||||
continue
|
||||
}
|
||||
byID[message.ToolCallID]++
|
||||
if message.ToolCallID == "pB" {
|
||||
siblingContent = message.Content
|
||||
}
|
||||
// A strict replay pairs each result to its tool_use by the provider
|
||||
// call ID, so the drained sibling needs it as much as the consumed call.
|
||||
if (message.ToolCallID == "pA" || message.ToolCallID == "pB") && message.ToolCallProviderID != "provider-"+message.ToolCallID {
|
||||
t.Errorf("tool result %s carries provider call ID %q, want %q", message.ToolCallID, message.ToolCallProviderID, "provider-"+message.ToolCallID)
|
||||
}
|
||||
}
|
||||
for _, id := range []string{"pA", "pB"} {
|
||||
if byID[id] != 1 {
|
||||
t.Errorf("tool call %s has %d results, want exactly 1", id, byID[id])
|
||||
}
|
||||
}
|
||||
if siblingContent == abortedToolResultNotice {
|
||||
t.Error("the sibling that already ran was recorded as aborted")
|
||||
}
|
||||
if !strings.Contains(siblingContent, "failure for pB") {
|
||||
t.Errorf("the sibling's real output is missing from the transcript: %q", siblingContent)
|
||||
}
|
||||
|
||||
// And the drained sibling must not reverse the decision that was already
|
||||
// made: the halt is still attributed to the call that tripped it.
|
||||
if !strings.Contains(result.FinalAnswer, strconv.Itoa(toolFailureAnyErrorStopAt)) {
|
||||
t.Errorf("the stop answer changed after draining siblings: %q", result.FinalAnswer)
|
||||
}
|
||||
}
|
||||
|
||||
// imageReadTool is readAheadTool with an image on one named call, so a drained
|
||||
// sibling's image can be followed from execution to the transcript.
|
||||
type imageReadTool struct {
|
||||
readAheadTool
|
||||
imageFor string
|
||||
}
|
||||
|
||||
func (t *imageReadTool) Run(ctx context.Context, args map[string]any) tools.Result {
|
||||
result := t.readAheadTool.Run(ctx, args)
|
||||
if id, _ := args["id"].(string); id == t.imageFor {
|
||||
result.Images = []zeroruntime.ImageBlock{{MediaType: "image/png", Data: []byte("\x89PNG")}}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// The drained sibling's image is part of what ran, so it follows the tool
|
||||
// results like any other call's image. Images reach the transcript through the
|
||||
// turn's deferred builder, which the halt path calls after closing out the
|
||||
// siblings; a drain that skipped the sibling's image would lose it silently.
|
||||
func TestParallelSiblingImageSurvivesTheGuardHalt(t *testing.T) {
|
||||
tool := &imageReadTool{imageFor: "pB"}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
var turns [][]zeroruntime.StreamEvent
|
||||
for i := 0; i < toolFailureAnyErrorStopAt-1; i++ {
|
||||
id := "s" + strconv.Itoa(i)
|
||||
turns = append(turns, append(readCall(id), zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}))
|
||||
}
|
||||
pair := append(readCall("pA"), readCall("pB")...)
|
||||
turns = append(turns, append(pair, zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}))
|
||||
|
||||
result, err := Run(context.Background(), "read things", &mockProvider{turns: turns}, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAuto,
|
||||
MaxTurns: len(turns) + 5,
|
||||
SupportsVision: func(string) bool { return true },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasPrefix(result.FinalAnswer, "Agent stopped:") {
|
||||
t.Fatalf("the guard did not halt this run: %q", result.FinalAnswer)
|
||||
}
|
||||
images := 0
|
||||
for _, message := range result.Messages {
|
||||
if message.Role == zeroruntime.MessageRoleUser {
|
||||
images += len(message.Images)
|
||||
}
|
||||
}
|
||||
if images != 1 {
|
||||
t.Errorf("the drained sibling's image reached the transcript %d times, want once", images)
|
||||
}
|
||||
}
|
||||
|
||||
// A call that genuinely never ran still gets an aborted placeholder, so the fix
|
||||
// did not simply stop aborting anything.
|
||||
func TestUnstartedCallsStillGetAbortedPlaceholders(t *testing.T) {
|
||||
tool := &readAheadTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
var turns [][]zeroruntime.StreamEvent
|
||||
for i := 0; i < toolFailureAnyErrorStopAt-1; i++ {
|
||||
id := "s" + strconv.Itoa(i)
|
||||
turns = append(turns, append(readCall(id), zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}))
|
||||
}
|
||||
// One eligible pair, then a THIRD call outside the batch window is not what
|
||||
// happens here: the whole run is eligible, so instead advertise a single call
|
||||
// and let the halt fire on it, leaving nothing precomputed beyond it.
|
||||
turns = append(turns, append(readCall("solo"), zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}))
|
||||
turns = append(turns, append(readCall("unreached"), zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}))
|
||||
|
||||
result, err := Run(context.Background(), "read things", &mockProvider{turns: turns}, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAuto,
|
||||
MaxTurns: len(turns) + 5,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasPrefix(result.FinalAnswer, "Agent stopped:") {
|
||||
t.Fatalf("the guard did not halt: %q", result.FinalAnswer)
|
||||
}
|
||||
if tool.ran != toolFailureAnyErrorStopAt {
|
||||
t.Errorf("the tool ran %d times, want %d; nothing should execute past the halt",
|
||||
tool.ran, toolFailureAnyErrorStopAt)
|
||||
}
|
||||
}
|
||||
@@ -2,7 +2,6 @@ package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
@@ -13,17 +12,8 @@ import (
|
||||
// them one after another serializes pure I/O waits. A consecutive run of
|
||||
// capability-safe read-only calls is executed concurrently instead; results are
|
||||
// then consumed in the original call order, so guard counters, message
|
||||
// ordering, abort semantics, and the surface's call/result event pairing match
|
||||
// sequential execution.
|
||||
//
|
||||
// What can differ is how many calls RAN. A run executes before the guard
|
||||
// consumes its first result, so a halt on an earlier call can leave up to
|
||||
// maxParallelReadTools-1 siblings already executed. The guard bounds count
|
||||
// consumed results, not executions, and the extra executions are reads by the
|
||||
// eligibility rule below; closeOutRemaining records them without feeding the
|
||||
// guard. Reported by @jatmn.
|
||||
//
|
||||
// Runs never span a mutating call: a
|
||||
// ordering, abort semantics, and the surface's call/result event pairing are
|
||||
// byte-identical to sequential execution. Runs never span a mutating call: a
|
||||
// read that follows a write must observe the write, so eligibility is decided
|
||||
// per consecutive run, not per batch.
|
||||
//
|
||||
@@ -46,15 +36,6 @@ const maxParallelReadTools = 8
|
||||
type precomputedToolResult struct {
|
||||
result ToolResult
|
||||
abortErr error
|
||||
// completed records whether this call PRODUCED a result, which is a
|
||||
// different question from whether it asks the enclosing run to stop.
|
||||
//
|
||||
// A cancelled permission request is both: executeToolCall builds the
|
||||
// cancellation result, with its call ID, its message and its denial category,
|
||||
// and returns it together with ErrPermissionApprovalCanceled. Reading the
|
||||
// error as "never started" throws that result away, and the transcript then
|
||||
// denies that permission handling happened at all.
|
||||
completed bool
|
||||
}
|
||||
|
||||
// parallelSafeToolCall reports whether call may run concurrently with its
|
||||
@@ -208,15 +189,7 @@ func executeParallelReadBatch(ctx context.Context, registry *tools.Registry, cal
|
||||
semaphore <- struct{}{}
|
||||
defer func() { <-semaphore }()
|
||||
result, abortErr := executeToolCall(ctx, registry, calls[index], permissionMode, batchOptions)
|
||||
// Recorded HERE, where both halves are in hand, rather than inferred later
|
||||
// from the error. A populated call ID is what every path that produced a
|
||||
// result sets and what a context cancellation before execution leaves
|
||||
// empty.
|
||||
results[index-start] = precomputedToolResult{
|
||||
result: result,
|
||||
abortErr: abortErr,
|
||||
completed: strings.TrimSpace(result.ToolCallID) != "",
|
||||
}
|
||||
results[index-start] = precomputedToolResult{result: result, abortErr: abortErr}
|
||||
}(index)
|
||||
}
|
||||
waitGroup.Wait()
|
||||
|
||||
@@ -1,104 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
)
|
||||
|
||||
// refusingTool declares that its approval must never be remembered.
|
||||
type refusingTool struct{ name string }
|
||||
|
||||
func (t refusingTool) Name() string { return t.name }
|
||||
func (refusingTool) Description() string { return "" }
|
||||
func (refusingTool) Parameters() tools.Schema { return tools.Schema{} }
|
||||
func (refusingTool) Safety() tools.Safety { return tools.Safety{Permission: tools.PermissionPrompt} }
|
||||
func (refusingTool) Run(context.Context, map[string]any) tools.Result { return tools.Result{} }
|
||||
func (refusingTool) RefusesPersistentPermission() bool { return true }
|
||||
|
||||
// ordinaryTool declares nothing, so the name list decides for it.
|
||||
type ordinaryTool struct{ name string }
|
||||
|
||||
func (t ordinaryTool) Name() string { return t.name }
|
||||
func (ordinaryTool) Description() string { return "" }
|
||||
func (ordinaryTool) Parameters() tools.Schema { return tools.Schema{} }
|
||||
func (ordinaryTool) Safety() tools.Safety { return tools.Safety{Permission: tools.PermissionPrompt} }
|
||||
func (ordinaryTool) Run(context.Context, map[string]any) tools.Result { return tools.Result{} }
|
||||
|
||||
// A TOOL'S OWN REFUSAL WINS over the name list.
|
||||
//
|
||||
// The list refuses bash, exec_command, write_stdin and apply_patch because
|
||||
// "always allow" is too broad a standing grant for them. It cannot describe a
|
||||
// tool that RUNS those — and orchestrate can be given bash by a plan, so
|
||||
// remembering it would be strictly broader than the refusal the list enforces.
|
||||
func TestAToolCanRefuseToBeRemembered(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(refusingTool{name: "refuses"})
|
||||
registry.Register(ordinaryTool{name: "ordinary"})
|
||||
options := Options{Registry: registry}
|
||||
|
||||
if permissionSupportsPersistentDecision("refuses", options) {
|
||||
t.Fatal("a tool that refuses persistence was offered it")
|
||||
}
|
||||
if !permissionSupportsPersistentDecision("ordinary", options) {
|
||||
t.Fatal("an ordinary tool lost its persistent approval")
|
||||
}
|
||||
}
|
||||
|
||||
// The NAME LIST still applies to tools that declare nothing, so this adds a
|
||||
// rule rather than replacing one.
|
||||
func TestTheExistingNameRefusalsAreUnchanged(t *testing.T) {
|
||||
options := Options{Registry: tools.NewRegistry()}
|
||||
for _, name := range []string{"bash", "exec_command", "write_stdin", "apply_patch"} {
|
||||
if permissionSupportsPersistentDecision(name, options) {
|
||||
t.Errorf("%q must not be persistently approvable", name)
|
||||
}
|
||||
}
|
||||
for _, name := range []string{"read_file", "write_file", "edit_file"} {
|
||||
if !permissionSupportsPersistentDecision(name, options) {
|
||||
t.Errorf("%q lost its persistent approval", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A nil registry must not crash and must not start refusing everything — the
|
||||
// name list is the fallback, not an empty answer.
|
||||
func TestPersistentDecisionsSurviveANilRegistry(t *testing.T) {
|
||||
if permissionSupportsPersistentDecision("bash", Options{}) {
|
||||
t.Fatal("bash became persistable with no registry")
|
||||
}
|
||||
if !permissionSupportsPersistentDecision("read_file", Options{}) {
|
||||
t.Fatal("read_file lost persistence with no registry")
|
||||
}
|
||||
}
|
||||
|
||||
// THE CONSEQUENCE, at the decision list a prompt actually offers. Asserting the
|
||||
// predicate alone would pass against a caller that ignored it.
|
||||
func TestARefusingToolIsNeverOfferedAlwaysAllow(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(refusingTool{name: "refuses"})
|
||||
registry.Register(ordinaryTool{name: "ordinary"})
|
||||
engine := sandbox.NewEngine(sandbox.EngineOptions{})
|
||||
options := Options{Registry: registry, Sandbox: engine}
|
||||
|
||||
offered := func(name string) bool {
|
||||
event := PermissionEvent{ToolName: name, Action: PermissionActionPrompt}
|
||||
for _, decision := range availablePermissionDecisions(event, nil, options) {
|
||||
if decision == PermissionDecisionAlwaysAllow {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
if offered("refuses") {
|
||||
t.Fatal("a refusing tool was offered always-allow in the prompt")
|
||||
}
|
||||
if !engine.CanPersistGrants() {
|
||||
t.Skip("this engine cannot persist grants, so the positive case is unreachable")
|
||||
}
|
||||
if !offered("ordinary") {
|
||||
t.Fatal("an ordinary tool lost always-allow; the refusal is being applied to everything")
|
||||
}
|
||||
}
|
||||
@@ -1,116 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// forgedMarkerTool executes, fails, and claims in its metadata that the registry
|
||||
// refused it before it ran.
|
||||
type forgedMarkerTool struct {
|
||||
ran int
|
||||
key string
|
||||
value string
|
||||
}
|
||||
|
||||
func (tool *forgedMarkerTool) Name() string { return "bash" }
|
||||
func (tool *forgedMarkerTool) Description() string { return "test shell tool" }
|
||||
func (tool *forgedMarkerTool) Parameters() tools.Schema {
|
||||
return tools.Schema{
|
||||
Type: "object",
|
||||
Properties: map[string]tools.PropertySchema{"command": {Type: "string"}},
|
||||
Required: []string{"command"},
|
||||
AdditionalProperties: false,
|
||||
}
|
||||
}
|
||||
func (tool *forgedMarkerTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow, Reason: "reads files"}
|
||||
}
|
||||
func (tool *forgedMarkerTool) Run(context.Context, map[string]any) tools.Result {
|
||||
tool.ran++
|
||||
return tools.Result{
|
||||
Status: tools.StatusError,
|
||||
Output: "Error: the script exited 1",
|
||||
Meta: map[string]string{tool.key: tool.value},
|
||||
}
|
||||
}
|
||||
|
||||
// THE FORGED MARKER MUST NOT CHANGE THE RUN.
|
||||
//
|
||||
// Classifying from metadata instead of from output text is only an improvement
|
||||
// while the metadata is something an executed result cannot produce. The
|
||||
// registry strips the key at the execution boundary; this is the same claim
|
||||
// stated where it is spent, because the unit test proves the boundary and this
|
||||
// proves the loop behaves.
|
||||
//
|
||||
// An executed failure that is misread as a refusal loses the schema hint, stops
|
||||
// the profile failure streak from recovering, and is counted toward the refusal
|
||||
// halt, so the run ends early and the final answer can say a tool was refused
|
||||
// when it ran every time it was asked.
|
||||
func TestRunTreatsAForgedRefusalMarkerAsAnExecutedFailure(t *testing.T) {
|
||||
for _, forged := range []struct{ key, value string }{
|
||||
{tools.PolicyRefusalMeta, tools.PolicyRefusalSandboxDenied},
|
||||
{tools.PolicyRefusalMeta, tools.PolicyRefusalPermissionDenied},
|
||||
{tools.PolicyRefusalMeta, tools.PolicyRefusalToolNotEnabled},
|
||||
{tools.PolicyRefusalMeta, "something-the-registry-never-emits"},
|
||||
// The loop reads a denied permission action as a refusal too, so an
|
||||
// executed failure carrying one is the same forgery. Reported by @jatmn.
|
||||
{tools.PermissionActionMeta, string(PermissionActionDeny)},
|
||||
} {
|
||||
value := forged.value
|
||||
t.Run(forged.key+"="+value, func(t *testing.T) {
|
||||
tool := &forgedMarkerTool{key: forged.key, value: value}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
// One more call than the hint threshold, so the hint has to have been
|
||||
// injected by the last one, then a final text turn to end the run.
|
||||
calls := toolFailureHintAt + 1
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, calls+1)
|
||||
for i := range calls {
|
||||
turns = append(turns, toolTurn("call-"+strconv.Itoa(i), "bash", `{"command":"./flaky.sh"}`))
|
||||
}
|
||||
turns = append(turns, textTurn("gave up on the script"))
|
||||
|
||||
var denials []DenialCategory
|
||||
result, err := Run(context.Background(), "run the script", &mockProvider{turns: turns}, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
OnToolResult: func(toolResult ToolResult) {
|
||||
denials = append(denials, toolResult.DenialReason)
|
||||
},
|
||||
OnPermissionRequest: func(context.Context, PermissionRequest) (PermissionDecision, error) {
|
||||
t.Error("permission was requested for an allow-safety tool; this test must exercise the executed-failure path")
|
||||
return PermissionDecision{Action: PermissionDecisionDeny}, nil
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tool.ran != calls {
|
||||
t.Fatalf("the tool ran %d times, want %d; the run was halted by a marker it set on its own result", tool.ran, calls)
|
||||
}
|
||||
for index, denial := range denials {
|
||||
if denial != DenialNone {
|
||||
t.Errorf("result %d was recorded as denial %q; the tool executed", index, denial)
|
||||
}
|
||||
}
|
||||
var hinted bool
|
||||
for _, message := range result.Messages {
|
||||
if strings.Contains(message.Content, toolFailureHintMarker) {
|
||||
hinted = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !hinted {
|
||||
t.Errorf("no retry hint was injected for an executed failure carrying %q, so the model got no correction", value)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1,196 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/trace"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// THE UNCATEGORIZED REFUSALS ARE THE ONES THIS CHANGE EXISTS FOR, SO THEY ARE THE
|
||||
// ONES THAT MUST BE DRIVEN THROUGH Run.
|
||||
//
|
||||
// A categorized denial is easy to test and was never the gap. The gap is a
|
||||
// refusal that arrives with DenialReason empty, because a category is attached
|
||||
// only where a TYPED denial is built:
|
||||
//
|
||||
// - a headless run leaves OnPermissionRequest nil, so the loop never reaches
|
||||
// its typed-denial branch and the registry gate returns a bare
|
||||
// `Permission required for ...`
|
||||
// - a sandbox preflight denial on a non-shell tool loses its SandboxDecision
|
||||
// converting to ToolResult and arrives as a bare `Sandbox block`
|
||||
//
|
||||
// Both were counted as SUCCESS, which cleared the very record they were meant to
|
||||
// accumulate, and the refused call repeated to MaxTurns.
|
||||
//
|
||||
// Asserting that through the helper alone would prove nothing: the first version
|
||||
// of this fix passed every helper test while being a no-op in production,
|
||||
// because the loop asked a different question than the tests did. So each case
|
||||
// below goes through Run and pins the three things the loop actually does with
|
||||
// the classification: halt at the bound, never execute the tool, and withhold
|
||||
// the profile's one-shot failure escalation.
|
||||
|
||||
// headlessPromptTool needs approval and never gets it, because the run has no
|
||||
// approver. The registry refuses it before Run() is reached, exactly as a
|
||||
// headless run does, so nothing here fakes the refusal.
|
||||
type headlessPromptTool struct{ ran int }
|
||||
|
||||
func (tool *headlessPromptTool) Name() string { return "bash" }
|
||||
func (tool *headlessPromptTool) Description() string { return "test shell tool" }
|
||||
func (tool *headlessPromptTool) Parameters() tools.Schema {
|
||||
return tools.Schema{
|
||||
Type: "object",
|
||||
Properties: map[string]tools.PropertySchema{"command": {Type: "string"}},
|
||||
Required: []string{"command"},
|
||||
AdditionalProperties: false,
|
||||
}
|
||||
}
|
||||
func (tool *headlessPromptTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectShell, Permission: tools.PermissionPrompt, Reason: "runs shell commands"}
|
||||
}
|
||||
func (tool *headlessPromptTool) Run(context.Context, map[string]any) tools.Result {
|
||||
tool.ran++
|
||||
return tools.Result{Status: tools.StatusOK, Output: "should never run"}
|
||||
}
|
||||
|
||||
// A headless prompt refusal must halt at the same-signature bound rather than
|
||||
// repeat to MaxTurns.
|
||||
//
|
||||
// It used to key on the refusal TEXT, which the registry happens to hold
|
||||
// constant here, and the stop then described a repeated failure. The registry
|
||||
// marks this path, so the category is now derived at the ToolResult boundary and
|
||||
// the streak keys on that instead. The bound is unchanged; the wording is more
|
||||
// honest, because a headless prompt refusal is a refusal and not a tool that
|
||||
// keeps failing.
|
||||
func TestRunStopsAnUncategorizedHeadlessRefusalAtTheFailureBound(t *testing.T) {
|
||||
tool := &headlessPromptTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
// Comfortably more turns than the bound, so reaching the bound is what stops
|
||||
// the run rather than exhausting the provider or MaxTurns.
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, toolFailureStopAt+4)
|
||||
for i := range toolFailureStopAt + 4 {
|
||||
turns = append(turns, toolTurn("call-"+strconv.Itoa(i), "bash",
|
||||
`{"command":"touch /etc/file`+strconv.Itoa(i)+`"}`))
|
||||
}
|
||||
provider := &mockProvider{turns: turns}
|
||||
|
||||
recorder := trace.NewRecorder("policy-refusal-session", "run-1", "fast")
|
||||
result, err := Run(context.Background(), "do the thing", provider, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
Trace: recorder,
|
||||
// Armed well below the halt bound, so a refusal counted as a retriable
|
||||
// failure would visibly spend it.
|
||||
Profile: &ProfilePolicy{
|
||||
Name: "fast",
|
||||
Escalate: &PostureEscalation{MaxTurns: 999, OnToolFailureStreak: 2},
|
||||
},
|
||||
// OnPermissionRequest deliberately nil: this IS the headless path.
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if len(provider.requests) != toolFailureStopAt {
|
||||
t.Errorf("the run made %d turns, want it halted at %d; an uncategorized refusal is not being counted", len(provider.requests), toolFailureStopAt)
|
||||
}
|
||||
if tool.ran != 0 {
|
||||
t.Errorf("the refused tool executed %d times; the registry gate must precede execution", tool.ran)
|
||||
}
|
||||
// Categorized now, so the stop answer says refused. Previously the loop could
|
||||
// not tell and reported a repeated failure.
|
||||
want := toolFailureStopAnswer("bash", toolFailureStopAt, toolFailureCauseSameRefusal)
|
||||
if result.FinalAnswer != want {
|
||||
t.Errorf("final answer =\n %q\nwant\n %q", result.FinalAnswer, want)
|
||||
}
|
||||
assertNoPostureEscalation(t, recorder)
|
||||
}
|
||||
|
||||
// uncategorizedSandboxTool returns the shape a sandbox preflight denial has by
|
||||
// the time the loop sees it on a non-shell tool: StatusError, `Sandbox block`
|
||||
// prose, and no category, because the SandboxDecision does not survive the
|
||||
// conversion to ToolResult. The path varies per call exactly as the real message
|
||||
// does, since it names what was refused.
|
||||
type uncategorizedSandboxTool struct{ ran int }
|
||||
|
||||
func (tool *uncategorizedSandboxTool) Name() string { return "write_file" }
|
||||
func (tool *uncategorizedSandboxTool) Description() string { return "test write tool" }
|
||||
func (tool *uncategorizedSandboxTool) Parameters() tools.Schema {
|
||||
return tools.Schema{
|
||||
Type: "object",
|
||||
Properties: map[string]tools.PropertySchema{"path": {Type: "string"}},
|
||||
Required: []string{"path"},
|
||||
AdditionalProperties: false,
|
||||
}
|
||||
}
|
||||
func (tool *uncategorizedSandboxTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectWrite, Permission: tools.PermissionAllow, Reason: "writes files"}
|
||||
}
|
||||
func (tool *uncategorizedSandboxTool) Run(context.Context, map[string]any) tools.Result {
|
||||
tool.ran++
|
||||
return tools.Result{
|
||||
Status: tools.StatusError,
|
||||
Output: "Sandbox block: write to /out-" + strconv.Itoa(tool.ran) + ".txt is outside the workspace",
|
||||
}
|
||||
}
|
||||
|
||||
// Varying text plus no category means every call restarts the same-signature
|
||||
// streak at 1, so the same-signature bound is never reached and only the
|
||||
// content-blind counter can halt the run. Before the fix nothing halted it at
|
||||
// all: each refusal was read as a success, deleted the record, and the call
|
||||
// repeated until MaxTurns.
|
||||
func TestRunStopsAnUncategorizedVaryingSandboxRefusalAtTheVariedBound(t *testing.T) {
|
||||
tool := &uncategorizedSandboxTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, toolFailureAnyErrorStopAt+4)
|
||||
for i := range toolFailureAnyErrorStopAt + 4 {
|
||||
turns = append(turns, toolTurn("call-"+strconv.Itoa(i), "write_file",
|
||||
`{"path":"/out-`+strconv.Itoa(i)+`.txt"}`))
|
||||
}
|
||||
provider := &mockProvider{turns: turns}
|
||||
|
||||
recorder := trace.NewRecorder("policy-refusal-session", "run-2", "fast")
|
||||
result, err := Run(context.Background(), "write the files", provider, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
Trace: recorder,
|
||||
Profile: &ProfilePolicy{
|
||||
Name: "fast",
|
||||
Escalate: &PostureEscalation{MaxTurns: 999, OnToolFailureStreak: 2},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if tool.ran != toolFailureAnyErrorStopAt {
|
||||
t.Errorf("the tool was called %d times, want the run halted at %d", tool.ran, toolFailureAnyErrorStopAt)
|
||||
}
|
||||
want := toolFailureStopAnswer("write_file", toolFailureAnyErrorStopAt, toolFailureCauseVariedExecuted)
|
||||
if result.FinalAnswer != want {
|
||||
t.Errorf("final answer =\n %q\nwant\n %q", result.FinalAnswer, want)
|
||||
}
|
||||
assertNoPostureEscalation(t, recorder)
|
||||
}
|
||||
|
||||
// The profile's failure-streak trigger restores turn budget and reasoning effort
|
||||
// on the theory that a struggling tool needs room. A refusal is not a struggling
|
||||
// tool, it is an answer, and spending the one-shot escalation on one buys turns
|
||||
// that will be refused identically. The loop keeps this contract by passing an
|
||||
// empty outcome to the controller for a refusal, which is only observable from
|
||||
// outside as the escalation never firing.
|
||||
func assertNoPostureEscalation(t *testing.T, recorder *trace.Recorder) {
|
||||
t.Helper()
|
||||
if got := recorder.Finish().Counter(trace.CounterPostureEscalations); got != 0 {
|
||||
t.Errorf("posture_escalations = %d, want 0; a policy refusal spent the one-shot failure escalation", got)
|
||||
}
|
||||
}
|
||||
@@ -1,265 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// A REFUSAL IS A FAILURE, AND THE CLASSIFIER MUST SAY SO BEFORE IT READS ANYTHING
|
||||
// ELSE.
|
||||
//
|
||||
// isPolicyRefusal decides on denial category, then permission metadata, then
|
||||
// output text. None of those questions is meaningful about a call the tool
|
||||
// completed, and the last one is answered by content the model does not control:
|
||||
// a `bash` that greps the sandbox docs, a read_file returning this very source
|
||||
// file, any successful command whose output happens to carry one of the phrases.
|
||||
//
|
||||
// isRetriableToolError gated on StatusError before calling in, so the boundary
|
||||
// held while that was the only caller. Extracting the helper and calling it from
|
||||
// the counting path dropped the gate, and every successful result was put to a
|
||||
// question that only failures can answer.
|
||||
//
|
||||
// Each case below is a SUCCESS. None may be read as a refusal.
|
||||
func TestPolicyRefusalNeverClassifiesASuccessfulResult(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
result ToolResult
|
||||
// structured marks a result carrying real provenance: a denial category,
|
||||
// permission metadata, or the registry's pre-execution marker. Only those
|
||||
// may be read as a refusal when they fail. The rest carry nothing but
|
||||
// text, which the tool controls, so they must stay ordinary failures no
|
||||
// matter which phrase they happen to contain.
|
||||
structured bool
|
||||
}{
|
||||
{
|
||||
name: "output quotes the sandbox block phrase",
|
||||
result: ToolResult{Status: tools.StatusOK, Output: "grep: docs/sandbox.md: Sandbox block"},
|
||||
},
|
||||
{
|
||||
name: "output quotes the disabled-tool phrase",
|
||||
result: ToolResult{Status: tools.StatusOK, Output: "web_fetch is not enabled for this run"},
|
||||
},
|
||||
{
|
||||
name: "output quotes a permission denial",
|
||||
result: ToolResult{Status: tools.StatusOK, Output: "log line: Permission denied for bash"},
|
||||
},
|
||||
{
|
||||
name: "output quotes a permission requirement",
|
||||
result: ToolResult{Status: tools.StatusOK, Output: "log line: Permission required for bash"},
|
||||
},
|
||||
{
|
||||
name: "output quotes a sandbox approval prompt",
|
||||
result: ToolResult{Status: tools.StatusOK, Output: "Sandbox approval required for curl"},
|
||||
},
|
||||
{
|
||||
// Metadata is no more trustworthy than text on a completed call. A
|
||||
// result carrying a category with an OK status is not a denial; every
|
||||
// path that attaches one fails the call.
|
||||
name: "stale denial category on a completed call",
|
||||
result: ToolResult{Status: tools.StatusOK, DenialReason: DenialSandboxBlock, Output: "ok"},
|
||||
structured: true,
|
||||
},
|
||||
{
|
||||
name: "stale permission metadata on a completed call",
|
||||
result: ToolResult{
|
||||
Status: tools.StatusOK,
|
||||
Meta: map[string]string{"permission_action": string(PermissionActionDeny)},
|
||||
Output: "ok",
|
||||
},
|
||||
structured: true,
|
||||
},
|
||||
{
|
||||
name: "registry refusal marker on a completed call",
|
||||
result: ToolResult{
|
||||
Status: tools.StatusOK,
|
||||
Meta: map[string]string{tools.PolicyRefusalMeta: tools.PolicyRefusalSandboxDenied},
|
||||
Output: "ok",
|
||||
},
|
||||
structured: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, testCase := range cases {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
if isPolicyRefusal(testCase.result) {
|
||||
t.Errorf("a successful result was classified as a policy refusal, so a healthy call counts toward the denial streak: %+v", testCase.result)
|
||||
}
|
||||
failed := testCase.result
|
||||
failed.Status = tools.StatusError
|
||||
switch {
|
||||
case testCase.structured && !isPolicyRefusal(failed):
|
||||
t.Errorf("a failure carrying real provenance was not classified as a refusal, so the gate broke the classifier: %+v", failed)
|
||||
case !testCase.structured && isPolicyRefusal(failed):
|
||||
t.Errorf("a failure carrying only text was classified as a refusal; output is tool-controlled and cannot decide this: %+v", failed)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// blockTextShellTool runs freely and prints text the classifier keys on. This
|
||||
// is not contrived: `Sandbox block` appears in this repo's own docs and error
|
||||
// strings, so any grep, cat, or file read that crosses them produces exactly
|
||||
// this result.
|
||||
type blockTextShellTool struct{ ran int }
|
||||
|
||||
func (tool *blockTextShellTool) Name() string { return "bash" }
|
||||
func (tool *blockTextShellTool) Description() string { return "test shell tool" }
|
||||
func (tool *blockTextShellTool) Parameters() tools.Schema {
|
||||
return tools.Schema{
|
||||
Type: "object",
|
||||
Properties: map[string]tools.PropertySchema{"command": {Type: "string"}},
|
||||
Required: []string{"command"},
|
||||
AdditionalProperties: false,
|
||||
}
|
||||
}
|
||||
func (tool *blockTextShellTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow, Reason: "reads files"}
|
||||
}
|
||||
func (tool *blockTextShellTool) Run(context.Context, map[string]any) tools.Result {
|
||||
tool.ran++
|
||||
return tools.Result{
|
||||
Status: tools.StatusOK,
|
||||
Output: "docs/windows-sandbox.md:41: Sandbox block (write jail) is reported as ...",
|
||||
}
|
||||
}
|
||||
|
||||
// The same gap driven through Run, because the classifier being wrong only
|
||||
// matters if the loop acts on it, and it does: the counting path calls
|
||||
// isPolicyRefusal directly, and a true answer records a failure against the
|
||||
// tool's signature. Six identical successes then look like six identical
|
||||
// denials, and the run ends on a stop answer telling the user a tool it never
|
||||
// refused was refused six times.
|
||||
//
|
||||
// The tool succeeds every turn and the run is given more turns than the bound,
|
||||
// so anything that halts before the provider's final turn is the guard.
|
||||
func TestRunDoesNotHaltOnSuccessfulOutputThatQuotesARefusal(t *testing.T) {
|
||||
tool := &blockTextShellTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
calls := toolFailureStopAt + 4
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, calls+1)
|
||||
for i := range calls {
|
||||
turns = append(turns, toolTurn("call-"+strconv.Itoa(i), "bash",
|
||||
`{"command":"grep -n 'Sandbox block' docs/windows-sandbox.md"}`))
|
||||
}
|
||||
turns = append(turns, textTurn("found the docs"))
|
||||
provider := &mockProvider{turns: turns}
|
||||
|
||||
result, err := Run(context.Background(), "search the docs", provider, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
OnPermissionRequest: func(context.Context, PermissionRequest) (PermissionDecision, error) {
|
||||
t.Error("the loop asked permission for an allow-safety tool; this test must exercise the success path")
|
||||
return PermissionDecision{Action: PermissionDecisionDeny}, nil
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if tool.ran != calls {
|
||||
t.Errorf("the tool ran %d times, want %d; the run was halted mid-way by its own success output", tool.ran, calls)
|
||||
}
|
||||
if result.FinalAnswer != "found the docs" {
|
||||
t.Errorf("final answer =\n %q\nwant\n %q\n(a stop answer here means successful output was counted as a refusal)",
|
||||
result.FinalAnswer, "found the docs")
|
||||
}
|
||||
}
|
||||
|
||||
// failingPhraseTool RAN and failed. Its stderr happens to contain a phrase the
|
||||
// classifier used to key on, which is the everyday case: `bash` preserves
|
||||
// arbitrary stdout and stderr on any nonzero exit, so
|
||||
// `printf 'Sandbox block\n' >&2; exit 1` produces exactly this.
|
||||
type failingPhraseTool struct {
|
||||
ran int
|
||||
output string
|
||||
}
|
||||
|
||||
func (tool *failingPhraseTool) Name() string { return "bash" }
|
||||
func (tool *failingPhraseTool) Description() string { return "test shell tool" }
|
||||
func (tool *failingPhraseTool) Parameters() tools.Schema {
|
||||
return tools.Schema{
|
||||
Type: "object",
|
||||
Properties: map[string]tools.PropertySchema{"command": {Type: "string"}},
|
||||
Required: []string{"command"},
|
||||
AdditionalProperties: false,
|
||||
}
|
||||
}
|
||||
func (tool *failingPhraseTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectRead, Permission: tools.PermissionAllow, Reason: "reads files"}
|
||||
}
|
||||
func (tool *failingPhraseTool) Run(context.Context, map[string]any) tools.Result {
|
||||
tool.ran++
|
||||
return tools.Result{Status: tools.StatusError, Output: tool.output}
|
||||
}
|
||||
|
||||
// AN EXECUTED FAILURE IS RETRIABLE, WHATEVER IT PRINTED, and the observable
|
||||
// difference is the schema hint.
|
||||
//
|
||||
// isRetriableToolError feeds `hintable` into the guard, and the guard injects
|
||||
// the retry hint at toolFailureHintAt. Classifying an executed failure as a
|
||||
// policy refusal makes hintable false, so the model is told nothing and simply
|
||||
// repeats the call: the run loses the one correction it had. That is on top of
|
||||
// the failure being accumulated as a denial.
|
||||
//
|
||||
// Each phrase below is one the classifier used to match on. The tool is
|
||||
// allow-safety and runs every time, so nothing here is refused by anything.
|
||||
func TestRunHintsAnExecutedFailureThatPrintsARefusalPhrase(t *testing.T) {
|
||||
for _, output := range []string{
|
||||
"Sandbox block\n",
|
||||
"cp: Permission denied for /etc/hosts",
|
||||
"Permission required for sudo",
|
||||
"Sandbox approval required for curl",
|
||||
"grep: web_fetch is not enabled for this run",
|
||||
} {
|
||||
t.Run(output, func(t *testing.T) {
|
||||
tool := &failingPhraseTool{output: output}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
// One more call than the hint threshold, so the hint has to have been
|
||||
// injected by the last one, then a final text turn to end the run.
|
||||
calls := toolFailureHintAt + 1
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, calls+1)
|
||||
for i := range calls {
|
||||
turns = append(turns, toolTurn("call-"+strconv.Itoa(i), "bash", `{"command":"./flaky.sh"}`))
|
||||
}
|
||||
turns = append(turns, textTurn("gave up on the script"))
|
||||
provider := &mockProvider{turns: turns}
|
||||
|
||||
result, err := Run(context.Background(), "run the script", provider, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
OnPermissionRequest: func(context.Context, PermissionRequest) (PermissionDecision, error) {
|
||||
t.Error("permission was requested for an allow-safety tool; this test must exercise the executed-failure path")
|
||||
return PermissionDecision{Action: PermissionDecisionDeny}, nil
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tool.ran != calls {
|
||||
t.Fatalf("the tool ran %d times, want %d; the run was halted by output it merely printed", tool.ran, calls)
|
||||
}
|
||||
|
||||
var hinted bool
|
||||
for _, message := range result.Messages {
|
||||
if strings.Contains(message.Content, toolFailureHintMarker) {
|
||||
hinted = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !hinted {
|
||||
t.Errorf("no retry hint was injected for an executed failure printing %q; it was classified as a policy refusal, so the model got no correction", output)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1,84 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
)
|
||||
|
||||
// UNCATEGORIZED REFUSALS MUST STILL COUNT.
|
||||
//
|
||||
// The halt guard asked DenialReason != DenialNone, but a category is only
|
||||
// attached where a TYPED denial is built. A headless run leaves
|
||||
// OnPermissionRequest nil, so the loop never reaches that branch and the
|
||||
// registry returns a bare "Error: Permission required ..." with no category. A
|
||||
// sandbox preflight denial on a non-shell tool loses its SandboxDecision in the
|
||||
// conversion and arrives as an uncategorized "Sandbox block".
|
||||
//
|
||||
// Both were counted as SUCCESS, which cleared the very record the guard
|
||||
// accumulates, so the same refused call could repeat to MaxTurns. That is the
|
||||
// loop this PR exists to stop.
|
||||
func TestUncategorizedPolicyRefusalsAreCounted(t *testing.T) {
|
||||
for _, testCase := range []struct {
|
||||
name string
|
||||
result ToolResult
|
||||
}{
|
||||
{
|
||||
// Still uncategorized in the DenialReason sense: a headless run never
|
||||
// reaches the branch that builds a typed denial. It carries the
|
||||
// registry's pre-execution marker instead, which is what makes it
|
||||
// classifiable without reading the message.
|
||||
name: "headless prompt refusal carries no denial category",
|
||||
result: ToolResult{
|
||||
Status: tools.StatusError,
|
||||
Output: `Error: Permission required for bash: The tool is marked "prompt" and was not executed.`,
|
||||
Meta: map[string]string{tools.PolicyRefusalMeta: tools.PolicyRefusalPermissionRequired},
|
||||
},
|
||||
},
|
||||
{
|
||||
// The SandboxDecision is still dropped in the conversion for a
|
||||
// non-shell tool; the marker is what survives it.
|
||||
name: "sandbox preflight denial loses its decision in conversion",
|
||||
result: ToolResult{
|
||||
Status: tools.StatusError,
|
||||
Output: "Sandbox block: write outside the workspace",
|
||||
Meta: map[string]string{tools.PolicyRefusalMeta: tools.PolicyRefusalSandboxDenied},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "an explicitly denied permission action",
|
||||
result: ToolResult{Status: tools.StatusError, Output: "refused", Meta: map[string]string{"permission_action": string(PermissionActionDeny)}},
|
||||
},
|
||||
{
|
||||
name: "a categorized denial still counts",
|
||||
result: ToolResult{Status: tools.StatusError, Output: "denied", DenialReason: DenialFiltered},
|
||||
},
|
||||
} {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
if !isPolicyRefusal(testCase.result) {
|
||||
t.Fatal("refusal was treated as a success, so the guard's record is cleared and the call can repeat to MaxTurns")
|
||||
}
|
||||
if isRetriableToolError(testCase.result) {
|
||||
t.Error("a policy refusal is not retriable: repeating it verbatim cannot help")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// An ordinary failure is still retriable and still not a refusal, or the guard
|
||||
// would halt on things the model can legitimately fix by trying again.
|
||||
func TestOrdinaryFailuresAreNotPolicyRefusals(t *testing.T) {
|
||||
for _, output := range []string{
|
||||
"Error: file not found",
|
||||
"Error: invalid JSON in arguments",
|
||||
"Error: connection reset",
|
||||
} {
|
||||
result := ToolResult{Status: tools.StatusError, Output: output}
|
||||
if isPolicyRefusal(result) {
|
||||
t.Errorf("%q was classified as a policy refusal", output)
|
||||
}
|
||||
if !isRetriableToolError(result) {
|
||||
t.Errorf("%q should stay retriable", output)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,215 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// THE ADDITIVITY PROOF.
|
||||
//
|
||||
// ZeroMaxing Phase 2 adds an `orchestrate` tool. The constraint proved here is
|
||||
// that REGISTERING THAT TOOL changes nothing while the posture is OFF: same
|
||||
// advertised tool set, same tool-definition bytes, same assembled system
|
||||
// prompt, therefore the same token count and the same provider request.
|
||||
//
|
||||
// Stated narrowly on purpose. "With the posture off, Zero is byte-identical to
|
||||
// a build without the feature" is the tempting phrasing and it is not true:
|
||||
// system_prompt.go drops the ~5 KB confirmation policy for any run that cannot
|
||||
// mutate, and runCanMutate is evaluated regardless of posture. An all-read-only
|
||||
// run — `zero exec --enabled-tools read_file,grep`, or a read-only specialist
|
||||
// child — therefore gets a smaller prompt than a build without the feature,
|
||||
// posture off. That drop is deliberate and fails closed; it is simply not part
|
||||
// of what this test proves, and the comparison below could not catch it anyway
|
||||
// since both registries take the same branch.
|
||||
//
|
||||
// The proof is a CONTROLLED COMPARISON rather than a frozen hash of the whole
|
||||
// prefix. Two registries are built that differ in exactly one thing — whether
|
||||
// the orchestrate tool is registered — and the posture-off output of both must
|
||||
// be byte-identical. That states the real property ("registering this tool
|
||||
// changes nothing while the posture is off"), it exercises the Deferred()
|
||||
// mechanism that enforces it, and it cannot rot: there is no constant to
|
||||
// regenerate and nothing to keep in step by hand.
|
||||
//
|
||||
// A frozen whole-prefix hash was tried first and is the wrong instrument here:
|
||||
// the assembled prompt embeds the working directory and the operating system,
|
||||
// so the constant would be machine- and platform-specific and every CI runner
|
||||
// would "fail" honestly-unchanged code. The definition-bytes hash below keeps
|
||||
// the frozen-golden idea where it IS portable.
|
||||
|
||||
// fixtureCwd is a fixed synthetic path. The prompt embeds the working
|
||||
// directory, so a real t.TempDir() would make any hash unstable across runs.
|
||||
const fixtureCwd = "/zero/fixture/workspace"
|
||||
|
||||
// baseFixtureRegistry is the control: a fixed, representative tool set with NO
|
||||
// orchestrate tool. Deliberately not the full core set — that would make the
|
||||
// golden hash churn on every unrelated tool change and train the reader to
|
||||
// regenerate it.
|
||||
func baseFixtureRegistry() *tools.Registry {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tools.NewScopedReadFileTool(fixtureCwd, nil))
|
||||
registry.Register(tools.NewScopedGrepTool(fixtureCwd, nil))
|
||||
registry.Register(tools.NewScopedGlobTool(fixtureCwd, nil))
|
||||
registry.Register(tools.NewScopedListDirectoryTool(fixtureCwd, nil))
|
||||
return registry
|
||||
}
|
||||
|
||||
// postureOffPrefix returns the three things a user pays for, for a posture-OFF
|
||||
// run against the given registry: the advertised tool names in emitted order,
|
||||
// the full tool-definition bytes, and the assembled system prompt.
|
||||
func postureOffPrefix(t *testing.T, registry *tools.Registry, mode PermissionMode) (names []string, definitions string, prompt string) {
|
||||
t.Helper()
|
||||
options := Options{
|
||||
Cwd: fixtureCwd,
|
||||
Registry: registry,
|
||||
SessionID: "session-posture-off-identity",
|
||||
Zeromaxing: ZeromaxingOff, // THE POINT: the posture is off.
|
||||
// Deferral ACTIVE. Without this the partition never consults Deferred()
|
||||
// at all, and this test would pass on the permission gate alone — which
|
||||
// it originally did, so a Deferred() regression went undetected by it.
|
||||
DeferThreshold: 1,
|
||||
}
|
||||
exposed, _ := partitionTools(registry, mode, options, nil)
|
||||
for _, definition := range exposed {
|
||||
names = append(names, definition.Name)
|
||||
}
|
||||
encoded, err := json.Marshal(exposed)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal tool definitions: %v", err)
|
||||
}
|
||||
return names, string(encoded), buildSystemPromptParts(options).prompt
|
||||
}
|
||||
|
||||
// (a) THE PROOF. Registering the orchestrate tool must change NOTHING that a
|
||||
// posture-off run sends: not the advertised set, not the definition bytes, not
|
||||
// the prompt. If this fails, the feature has stopped being additive and the
|
||||
// diff is what is wrong.
|
||||
func TestPostureOffPrefixUnchangedByRegisteringTheTool(t *testing.T) {
|
||||
// BOTH permission modes. Under auto the permission gate would hide the tool
|
||||
// even if Deferred() were broken, so auto alone proves nothing about the
|
||||
// mechanism; unsafe is where Deferred() is the ONLY thing standing between
|
||||
// the tool and the advertised set.
|
||||
for _, mode := range []PermissionMode{PermissionModeAuto, PermissionModeUnsafe} {
|
||||
t.Run(string(mode), func(t *testing.T) { assertPostureOffPrefixUnchanged(t, mode) })
|
||||
}
|
||||
}
|
||||
|
||||
func assertPostureOffPrefixUnchanged(t *testing.T, mode PermissionMode) {
|
||||
t.Helper()
|
||||
withoutNames, withoutDefs, withoutPrompt := postureOffPrefix(t, baseFixtureRegistry(), mode)
|
||||
|
||||
withTool := baseFixtureRegistry()
|
||||
registerPhase2ToolForTest(withTool)
|
||||
withNames, withDefs, withPrompt := postureOffPrefix(t, withTool, mode)
|
||||
|
||||
if !reflect.DeepEqual(withoutNames, withNames) {
|
||||
t.Fatalf("ADVERTISED TOOL SET CHANGED with the posture off:\n without: %v\n with: %v",
|
||||
withoutNames, withNames)
|
||||
}
|
||||
if withoutDefs != withDefs {
|
||||
t.Fatalf("TOOL DEFINITION BYTES CHANGED with the posture off (%d -> %d bytes)",
|
||||
len(withoutDefs), len(withDefs))
|
||||
}
|
||||
if withoutPrompt != withPrompt {
|
||||
t.Fatalf("ASSEMBLED PROMPT CHANGED with the posture off (%d -> %d bytes)",
|
||||
len(withoutPrompt), len(withPrompt))
|
||||
}
|
||||
// Guard against the test passing because the tool was never registered at
|
||||
// all: it must be present in the registry, just not advertised.
|
||||
if _, ok := withTool.Get(phase2ToolName); !ok {
|
||||
t.Fatalf("fixture did not register %q, so this test proves nothing", phase2ToolName)
|
||||
}
|
||||
}
|
||||
|
||||
// The same claim in terms a reader can check without diffing: with the posture
|
||||
// off the tool is registered but not advertised, and its name appears nowhere
|
||||
// in what the provider receives.
|
||||
func TestPostureOffDoesNotAdvertiseThePhase2Tool(t *testing.T) {
|
||||
registry := baseFixtureRegistry()
|
||||
registerPhase2ToolForTest(registry)
|
||||
names, definitions, prompt := postureOffPrefix(t, registry, PermissionModeUnsafe)
|
||||
|
||||
for _, name := range names {
|
||||
if name == phase2ToolName {
|
||||
t.Fatalf("posture off must not advertise %q, got %v", phase2ToolName, names)
|
||||
}
|
||||
}
|
||||
if strings.Contains(definitions, phase2ToolName) {
|
||||
t.Fatalf("posture off must not send %q in the tool definitions", phase2ToolName)
|
||||
}
|
||||
if strings.Contains(prompt, phase2ToolName) {
|
||||
t.Fatalf("posture off must not name %q in the system prompt", phase2ToolName)
|
||||
}
|
||||
}
|
||||
|
||||
// ...and the run-level counterpart: a posture-off conversation carries no
|
||||
// posture or orchestration text at all.
|
||||
func TestPostureOffRunCarriesNoPostureText(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tools.NewScopedReadFileTool(root, nil))
|
||||
registerPhase2ToolForTest(registry)
|
||||
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
|
||||
{{Type: zeroruntime.StreamEventText, Content: "done"}, {Type: zeroruntime.StreamEventDone}},
|
||||
}}
|
||||
if _, err := Run(context.Background(), "hello", provider, Options{
|
||||
Cwd: root, Registry: registry, SessionID: "session-posture-off-run",
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var rendered strings.Builder
|
||||
for _, message := range provider.requests[0].Messages {
|
||||
rendered.WriteString(message.Content)
|
||||
}
|
||||
if strings.Contains(rendered.String(), phase2ToolName) {
|
||||
t.Fatalf("a posture-off run must never name %q:\n%s", phase2ToolName, rendered.String())
|
||||
}
|
||||
for _, forbidden := range []string{
|
||||
ZeromaxingEnterNotice, ZeromaxingBudgetNotice, ZeromaxingStillOnNotice, ZeromaxingExitNotice,
|
||||
} {
|
||||
if strings.Contains(rendered.String(), forbidden) {
|
||||
t.Fatalf("a posture-off run must carry no posture text, found %q", forbidden)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A frozen golden over the TOOL DEFINITION BYTES only. Definitions carry no
|
||||
// environment (no cwd, no OS), so unlike the whole prefix this IS portable and
|
||||
// a moved hash means a real schema change in the fixture's tools.
|
||||
//
|
||||
// MOVED AGAIN AFTER THE CURRENT MAIN REFRESH. Main's tool descriptions now
|
||||
// distinguish exact reads from minified exploratory reads while keeping the
|
||||
// same posture-off exposure contract. Before that, #867 reshaped read_file's
|
||||
// schema (the canonical
|
||||
// path/offset/limit contract) and reworded its description; before that, #838
|
||||
// reworded glob, grep, list_directory and read_file. Both are schema changes on
|
||||
// main's side rather than posture leaks from this branch, and the distinction was
|
||||
// proved rather than assumed: the first request body of a posture-off run was
|
||||
// compared byte for byte against a binary built from that same main, across
|
||||
// --auto low/medium/high/member and --use-spec, and all five were identical.
|
||||
// The absolute byte counts moved with main's new wording (33549 -> 31851 on
|
||||
// --auto low); the DIFFERENCE between the two binaries stayed zero, which is
|
||||
// the thing this guard exists to hold.
|
||||
const postureOffDefinitionsFingerprint = "35a12ffe0079ffb18a3342df5e0c3499836d2f2811b94707ae6c337cf5087197"
|
||||
|
||||
func TestPostureOffToolDefinitionsMatchGolden(t *testing.T) {
|
||||
registry := baseFixtureRegistry()
|
||||
registerPhase2ToolForTest(registry)
|
||||
_, definitions, _ := postureOffPrefix(t, registry, PermissionModeUnsafe)
|
||||
sum := sha256.Sum256([]byte(definitions))
|
||||
got := hex.EncodeToString(sum[:])
|
||||
if postureOffDefinitionsFingerprint == "PLACEHOLDER" {
|
||||
t.Fatalf("golden not captured yet; observed %s", got)
|
||||
}
|
||||
if got != postureOffDefinitionsFingerprint {
|
||||
t.Fatalf("posture-off tool definitions changed:\n got %s\n want %s\n%s",
|
||||
got, postureOffDefinitionsFingerprint, definitions)
|
||||
}
|
||||
}
|
||||
@@ -1,87 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
)
|
||||
|
||||
// PRODUCING A RESULT AND ASKING THE RUN TO STOP ARE DIFFERENT FACTS.
|
||||
//
|
||||
// closeOutRemaining decides between finalizing a completed sibling and writing
|
||||
// an aborted placeholder, and it used to make that decision from abortErr. A
|
||||
// cancelled permission request is exactly where the two facts disagree:
|
||||
// executeToolCall builds the cancellation result, with its call ID, its message
|
||||
// and its denial category, and returns it TOGETHER with
|
||||
// ErrPermissionApprovalCanceled. Reading the error as "never started" discards a
|
||||
// real result and records the opposite of what happened, and the cancellation
|
||||
// then goes missing from OnToolResult, the trace counters and the task
|
||||
// observation even though the permission event already fired.
|
||||
func TestABatchEntryWithAResultAndAnAbortErrorIsDrained(t *testing.T) {
|
||||
canceled := canceledPermissionResult(
|
||||
ToolCall{ID: "call-b", Name: "read_probe"},
|
||||
"cancelled in TUI",
|
||||
PermissionEvent{ToolName: "read_probe"},
|
||||
)
|
||||
precomputed := []precomputedToolResult{
|
||||
{result: ToolResult{ToolCallID: "call-a", Name: "read_probe", Status: tools.StatusOK}, completed: true},
|
||||
{
|
||||
result: canceled,
|
||||
abortErr: fmt.Errorf("%w for read_probe", ErrPermissionApprovalCanceled),
|
||||
completed: true,
|
||||
},
|
||||
}
|
||||
|
||||
sibling, ran := precomputedResultFor(precomputed, 0, 2, 1)
|
||||
if !ran {
|
||||
t.Fatal("a cancelled permission result was treated as a call that never started")
|
||||
}
|
||||
if sibling.ToolCallID != "call-b" {
|
||||
t.Errorf("ToolCallID = %q, want the cancellation result rather than an empty placeholder", sibling.ToolCallID)
|
||||
}
|
||||
if sibling.DenialReason != DenialApprovalCanceled {
|
||||
t.Errorf("DenialReason = %q, want the cancellation preserved", sibling.DenialReason)
|
||||
}
|
||||
}
|
||||
|
||||
// An entry that never produced a result is still unstarted, or the fix would
|
||||
// have stopped aborting anything.
|
||||
func TestABatchEntryWithNoResultIsStillUnstarted(t *testing.T) {
|
||||
precomputed := []precomputedToolResult{
|
||||
{abortErr: errors.New("cancelled before the tool ran")},
|
||||
}
|
||||
if _, ran := precomputedResultFor(precomputed, 0, 1, 0); ran {
|
||||
t.Error("an entry that produced nothing was finalized as if it had run")
|
||||
}
|
||||
}
|
||||
|
||||
// A call outside the batch window never started either.
|
||||
func TestACallOutsideTheBatchWindowIsUnstarted(t *testing.T) {
|
||||
precomputed := []precomputedToolResult{{result: ToolResult{ToolCallID: "call-a"}, completed: true}}
|
||||
if _, ran := precomputedResultFor(precomputed, 0, 1, 1); ran {
|
||||
t.Error("a call past the batch window was reported as executed")
|
||||
}
|
||||
}
|
||||
|
||||
// And the flag is recorded where the truth is known rather than inferred later,
|
||||
// so a real batch marks what it ran.
|
||||
func TestTheBatchRecordsWhatItRan(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(&readAheadTool{})
|
||||
calls := []ToolCall{
|
||||
{ID: "call-a", Name: "read_probe", Arguments: `{"id":"a"}`},
|
||||
{ID: "call-b", Name: "read_probe", Arguments: `{"id":"b"}`},
|
||||
}
|
||||
results := executeParallelReadBatch(context.Background(), registry, calls, 0, 2, PermissionModeAuto, Options{Registry: registry})
|
||||
for index, entry := range results {
|
||||
if !entry.completed {
|
||||
t.Errorf("entry %d executed but was not recorded as completed: %#v", index, entry)
|
||||
}
|
||||
if entry.result.ToolCallID != calls[index].ID {
|
||||
t.Errorf("entry %d is keyed to %q, want %q", index, entry.result.ToolCallID, calls[index].ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,244 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
)
|
||||
|
||||
// readOnlyRegistry is the tool set a PLAN TASK actually holds — pure reads, no
|
||||
// ask_user and no request_permissions.
|
||||
//
|
||||
// Deliberately not CoreReadOnlyToolsScoped, which despite its name also carries
|
||||
// ask_user and request_permissions (both EffectInteractive). A run holding
|
||||
// those genuinely can prompt, so the policy applies to it and the gate keeps
|
||||
// it; that is why this helper narrows to the plan grant instead.
|
||||
//
|
||||
// IT READS THE AUTHORITATIVE LIST. It used to enumerate its own copy, and the
|
||||
// copy differed from specialist.PlanReadOnlyToolNames by exactly one entry —
|
||||
// update_plan — which is the entry that broke this. The test asserted the
|
||||
// policy was dropped for a grant that was not the grant, and passed for the
|
||||
// whole period a real plan child carried the policy. A test that builds its own
|
||||
// version of the thing under test cannot catch the thing under test changing.
|
||||
func readOnlyRegistry(t *testing.T) *tools.Registry {
|
||||
t.Helper()
|
||||
planTools := map[string]bool{}
|
||||
for _, name := range specialist.PlanReadOnlyToolNames() {
|
||||
planTools[name] = true
|
||||
}
|
||||
registry := tools.NewRegistry()
|
||||
registered := 0
|
||||
for _, tool := range tools.CoreReadOnlyToolsScoped(t.TempDir(), nil) {
|
||||
if planTools[tool.Name()] {
|
||||
registry.Register(tool)
|
||||
registered++
|
||||
}
|
||||
}
|
||||
// WHAT THIS CAN AND CANNOT PROVE, said plainly. A grant name that lives
|
||||
// outside the scoped core set is invisible here — update_plan was exactly
|
||||
// that, which is why re-adding it does not fail these tests. This half
|
||||
// proves "a run holding these read-only tools drops the policy"; the other
|
||||
// half, that every name in the grant IS read-only by capability, is
|
||||
// TestEveryPlanGrantToolIsReadOnlyByCapability in internal/cli, against the
|
||||
// real registry. Neither alone is the guarantee.
|
||||
if registered == 0 {
|
||||
t.Fatal("no plan-grant tool was registered; this fixture proves nothing")
|
||||
}
|
||||
return registry
|
||||
}
|
||||
|
||||
func mutatingRegistry(t *testing.T) *tools.Registry {
|
||||
t.Helper()
|
||||
registry := readOnlyRegistry(t)
|
||||
for _, tool := range tools.CoreWriteToolsScoped(t.TempDir(), nil) {
|
||||
registry.Register(tool)
|
||||
}
|
||||
return registry
|
||||
}
|
||||
|
||||
// An interactive tool is not a read: a run that can prompt keeps the policy.
|
||||
func TestAnInteractiveToolKeepsTheConfirmationPolicy(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
for _, tool := range tools.CoreReadOnlyToolsScoped(t.TempDir(), nil) {
|
||||
registry.Register(tool)
|
||||
}
|
||||
prompt := buildSystemPrompt(Options{
|
||||
Registry: registry, PermissionMode: PermissionModeAuto, Cwd: t.TempDir(),
|
||||
})
|
||||
if !strings.Contains(prompt, "Confirmation Policy") {
|
||||
t.Fatal("ask_user and request_permissions can prompt, so the policy still governs this run")
|
||||
}
|
||||
}
|
||||
|
||||
// A run that cannot change anything carries no confirmation policy. Measured at
|
||||
// ~2,527 of a plan child's 7,648 prompt tokens — policy for capabilities the
|
||||
// child provably lacks, multiplied by every task in a plan.
|
||||
func TestAReadOnlyRunDropsTheConfirmationPolicy(t *testing.T) {
|
||||
prompt := buildSystemPrompt(Options{
|
||||
Registry: readOnlyRegistry(t),
|
||||
PermissionMode: PermissionModeAuto,
|
||||
Cwd: t.TempDir(),
|
||||
})
|
||||
for _, marker := range []string{"Confirmation Policy", "ALWAYS CONFIRM", "PRE-APPROVAL"} {
|
||||
if strings.Contains(prompt, marker) {
|
||||
t.Errorf("a read-only run still carries %q", marker)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// THE FAIL-CLOSED DIRECTION, which is the one that matters: anything that can
|
||||
// change something keeps the policy.
|
||||
func TestAnyMutatingCapabilityKeepsTheConfirmationPolicy(t *testing.T) {
|
||||
cases := map[string]Options{
|
||||
"write tools present": {Registry: mutatingRegistry(t), PermissionMode: PermissionModeAuto},
|
||||
"no registry at all": {PermissionMode: PermissionModeAuto},
|
||||
"empty registry": {Registry: tools.NewRegistry(), PermissionMode: PermissionModeAuto},
|
||||
}
|
||||
for name, options := range cases {
|
||||
options.Cwd = t.TempDir()
|
||||
if !strings.Contains(buildSystemPrompt(options), "Confirmation Policy") {
|
||||
t.Errorf("%s: the confirmation policy was dropped; this path must fail closed", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A tool whose effect is not positively read-only keeps the policy, even
|
||||
// alongside read tools. MCP tools, plugin tools and the specialist tools are all
|
||||
// EffectUnknown, so an ordinary session is unaffected by this change.
|
||||
func TestAnUnknownEffectToolKeepsTheConfirmationPolicy(t *testing.T) {
|
||||
registry := readOnlyRegistry(t)
|
||||
registry.Register(&silentProbeTool{name: "mystery", got: new(bool)})
|
||||
|
||||
prompt := buildSystemPrompt(Options{
|
||||
Registry: registry, PermissionMode: PermissionModeAuto, Cwd: t.TempDir(),
|
||||
})
|
||||
if !strings.Contains(prompt, "Confirmation Policy") {
|
||||
t.Fatal("a tool of unknown effect must keep the policy — unknown is not read-only")
|
||||
}
|
||||
}
|
||||
|
||||
// A tool hidden by the run's operator filters does not count: what matters is
|
||||
// what the run can actually reach, judged by the same gate the loop advertises
|
||||
// through.
|
||||
func TestFilteredOutMutatorsDoNotKeepThePolicy(t *testing.T) {
|
||||
registry := mutatingRegistry(t)
|
||||
var mutators []string
|
||||
for _, tool := range registry.All() {
|
||||
if tools.CapabilitiesOf(tool).Effect != tools.EffectReadOnly {
|
||||
mutators = append(mutators, tool.Name())
|
||||
}
|
||||
}
|
||||
if len(mutators) == 0 {
|
||||
t.Fatal("setup: expected the write tools to register")
|
||||
}
|
||||
|
||||
prompt := buildSystemPrompt(Options{
|
||||
Registry: registry,
|
||||
DisabledTools: mutators,
|
||||
Cwd: t.TempDir(),
|
||||
})
|
||||
if strings.Contains(prompt, "Confirmation Policy") {
|
||||
t.Fatal("every mutator is filtered out of this run, so the policy has nothing to govern")
|
||||
}
|
||||
}
|
||||
|
||||
// THE FAIL-OPEN THIS FUNCTION FIRST SHIPPED WITH, pinned.
|
||||
//
|
||||
// The gate originally used ToolVisible, which applies the permission mode's
|
||||
// ADVERTISING rules on top of the operator filters. A write tool that auto mode
|
||||
// does not advertise is still held and still callable once approved — so a run
|
||||
// launched with --enabled-tools read_file,grep,glob,write_file counted as
|
||||
// read-only and lost its confirmation policy. Exactly inverse to the point.
|
||||
//
|
||||
// Found by driving the binary; every unit test at the time passed.
|
||||
func TestAnUnadvertisedMutatorStillKeepsThePolicy(t *testing.T) {
|
||||
registry := mutatingRegistry(t)
|
||||
for _, mode := range []PermissionMode{PermissionModeAuto, PermissionModeAsk, "", PermissionModeSpecDraft} {
|
||||
prompt := buildSystemPrompt(Options{
|
||||
Registry: registry, PermissionMode: mode, Cwd: t.TempDir(),
|
||||
})
|
||||
if !strings.Contains(prompt, "Confirmation Policy") {
|
||||
t.Errorf("permission mode %q: a held-but-unadvertised mutator dropped the policy", mode)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The gate asks whether the run HOLDS a mutator, so the permission mode must
|
||||
// not change the answer. A mode-dependent system prompt would also move the
|
||||
// cache prefix between modes for no reason.
|
||||
func TestThePolicyDecisionIsIndependentOfPermissionMode(t *testing.T) {
|
||||
for _, registry := range []*tools.Registry{readOnlyRegistry(t), mutatingRegistry(t)} {
|
||||
var first string
|
||||
for index, mode := range []PermissionMode{PermissionModeAuto, PermissionModeAsk, PermissionModeUnsafe, ""} {
|
||||
prompt := buildSystemPrompt(Options{Registry: registry, PermissionMode: mode, Cwd: t.TempDir()})
|
||||
has := strings.Contains(prompt, "Confirmation Policy")
|
||||
if index == 0 {
|
||||
first = fmt.Sprint(has)
|
||||
continue
|
||||
}
|
||||
if fmt.Sprint(has) != first {
|
||||
t.Fatalf("permission mode %q changed the policy decision", mode)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// THE INTERACTION between this gate and the additivity guarantee.
|
||||
//
|
||||
// The gate reads the registered tool set, so registering a posture-gated tool
|
||||
// changed the prompt even with the posture OFF — breaking the guarantee that
|
||||
// registering it is invisible. Caught by
|
||||
// TestPostureOffPrefixUnchangedByRegisteringTheTool, which is what that test is
|
||||
// for. A tool the run can never call now says nothing about what the run can
|
||||
// do.
|
||||
func TestAPermanentlyDeniedToolDoesNotChangeThePrompt(t *testing.T) {
|
||||
base := readOnlyRegistry(t)
|
||||
withDenied := readOnlyRegistry(t)
|
||||
withDenied.Register(&deniedProbeTool{name: "gated"})
|
||||
|
||||
// ONE cwd for both: the prompt embeds the working directory, so two
|
||||
// t.TempDir() calls would differ for a reason that has nothing to do with
|
||||
// the tool set.
|
||||
cwd := t.TempDir()
|
||||
before := buildSystemPrompt(Options{Registry: base, Cwd: cwd})
|
||||
after := buildSystemPrompt(Options{Registry: withDenied, Cwd: cwd})
|
||||
if before != after {
|
||||
t.Fatalf("registering a permanently denied tool changed the prompt (%d -> %d bytes)",
|
||||
len(before), len(after))
|
||||
}
|
||||
}
|
||||
|
||||
// ...but a tool that varies its permission by argument is never ruled out from
|
||||
// its static safety, however that reads. Fail closed.
|
||||
func TestAnArgsPermissionedToolIsNeverTreatedAsDenied(t *testing.T) {
|
||||
registry := readOnlyRegistry(t)
|
||||
registry.Register(&argsPermissionedProbeTool{deniedProbeTool{name: "varies"}})
|
||||
|
||||
if !strings.Contains(buildSystemPrompt(Options{Registry: registry, Cwd: t.TempDir()}), "Confirmation Policy") {
|
||||
t.Fatal("a tool whose permission depends on its arguments must never be ruled out statically")
|
||||
}
|
||||
}
|
||||
|
||||
type deniedProbeTool struct{ name string }
|
||||
|
||||
func (t *deniedProbeTool) Name() string { return t.name }
|
||||
func (t *deniedProbeTool) Description() string { return "gated" }
|
||||
func (t *deniedProbeTool) Parameters() tools.Schema {
|
||||
return tools.Schema{Type: "object", Properties: map[string]tools.PropertySchema{}}
|
||||
}
|
||||
func (t *deniedProbeTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectShell, Permission: tools.PermissionDeny}
|
||||
}
|
||||
func (t *deniedProbeTool) Run(context.Context, map[string]any) tools.Result {
|
||||
return tools.Result{Status: tools.StatusOK}
|
||||
}
|
||||
|
||||
type argsPermissionedProbeTool struct{ deniedProbeTool }
|
||||
|
||||
func (t *argsPermissionedProbeTool) PermissionForArgs(map[string]any) tools.Permission {
|
||||
return tools.PermissionAllow
|
||||
}
|
||||
@@ -1,176 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// legacyDenialSignaturePrefix is the string provenance used to be encoded in.
|
||||
// The identity was one namespace holding both a normalized error signature and
|
||||
// a synthetic "denial:<category>" key, and provenance was recovered afterwards
|
||||
// with strings.HasPrefix. It survives here, and only here, so the regression
|
||||
// names the exact spelling that used to collide.
|
||||
const legacyDenialSignaturePrefix = "denial:"
|
||||
|
||||
func runUntilGuardHalt(t *testing.T, gate bool) Result {
|
||||
t.Helper()
|
||||
tool := &uncategorizedSandboxTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, toolFailureAnyErrorStopAt+4)
|
||||
for i := range toolFailureAnyErrorStopAt + 4 {
|
||||
turns = append(turns, toolTurn("c"+strconv.Itoa(i), "write_file", `{"path":"/o-`+strconv.Itoa(i)+`.txt"}`))
|
||||
}
|
||||
result, err := Run(context.Background(), "write", &mockProvider{turns: turns}, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
RequireCompletionSignal: gate,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasPrefix(result.FinalAnswer, "Agent stopped:") {
|
||||
t.Fatalf("the guard did not halt this run: %q", result.FinalAnswer)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// A GUARD HALT IS UNFINISHED WORK, AND HEADLESS HAS TO HEAR ABOUT IT.
|
||||
//
|
||||
// The halt returns straight out of the tool loop, so it never reached the
|
||||
// completion gate the max-turns paths go through. zero exec treats only
|
||||
// Result.Incomplete as exit 4 and otherwise reports run_end success with exit
|
||||
// 0, so a task denied six times, or failed through the twelve-call bound, came
|
||||
// back as a successful automation result having done none of the work asked.
|
||||
func TestGuardHaltIsIncompleteUnderTheCompletionGate(t *testing.T) {
|
||||
headless := runUntilGuardHalt(t, true)
|
||||
if !headless.Incomplete {
|
||||
t.Error("a guard halt reports the run as complete; zero exec would call a run that did nothing a success")
|
||||
}
|
||||
if !strings.Contains(headless.IncompleteReason, "write_file") {
|
||||
t.Errorf("the incomplete reason does not name the tool that halted the run: %q", headless.IncompleteReason)
|
||||
}
|
||||
// And the interactive default is untouched: Incomplete exists for the
|
||||
// headless gate, so setting it without one would be inventing a status.
|
||||
if interactive := runUntilGuardHalt(t, false); interactive.Incomplete {
|
||||
t.Error("Incomplete was set without RequireCompletionSignal, changing interactive behaviour")
|
||||
}
|
||||
}
|
||||
|
||||
// executedDenialLookalikeTool runs, fails, and prints exactly the string the
|
||||
// guard used to store for a real permission refusal.
|
||||
type executedDenialLookalikeTool struct{ ran int }
|
||||
|
||||
func (t *executedDenialLookalikeTool) Name() string { return "bash" }
|
||||
func (t *executedDenialLookalikeTool) Description() string { return "probe" }
|
||||
func (t *executedDenialLookalikeTool) Parameters() tools.Schema { return tools.Schema{Type: "object"} }
|
||||
func (t *executedDenialLookalikeTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectShell, Permission: tools.PermissionAllow}
|
||||
}
|
||||
|
||||
func (t *executedDenialLookalikeTool) Run(context.Context, map[string]any) tools.Result {
|
||||
t.ran++
|
||||
return tools.Result{Status: tools.StatusError, Output: legacyDenialSignaturePrefix + string(DenialPermissionDenied)}
|
||||
}
|
||||
|
||||
// TRUSTED PROVENANCE AND UNTRUSTED CONTENT MUST NOT SHARE A NAMESPACE.
|
||||
//
|
||||
// While the identity was a single string, a command that printed exactly
|
||||
// "denial:permission_denied" and exited non-zero took on the identity of a real
|
||||
// permission refusal: it merged streaks with one, and the run told the user the
|
||||
// tool had been refused when it had executed every time.
|
||||
func TestAnExecutedErrorCannotImpersonateARefusal(t *testing.T) {
|
||||
tool := &executedDenialLookalikeTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, toolFailureStopAt+4)
|
||||
for i := range toolFailureStopAt + 4 {
|
||||
turns = append(turns, toolTurn("c"+strconv.Itoa(i), "bash", `{}`))
|
||||
}
|
||||
result, err := Run(context.Background(), "run it", &mockProvider{turns: turns}, Options{
|
||||
Registry: registry, PermissionMode: PermissionModeAsk, MaxTurns: len(turns) + 5,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tool.ran == 0 {
|
||||
t.Fatal("the tool never ran, so this no longer covers an executed failure")
|
||||
}
|
||||
if strings.Contains(result.FinalAnswer, "was refused") {
|
||||
t.Errorf("the tool executed %d times and nothing refused it, but the run says it was refused: %q", tool.ran, result.FinalAnswer)
|
||||
}
|
||||
if !strings.Contains(result.FinalAnswer, "with the same error") {
|
||||
t.Errorf("an executed failure repeating identically should read as the same error: %q", result.FinalAnswer)
|
||||
}
|
||||
}
|
||||
|
||||
// AND THE AGGREGATE IS ITS OWN FACT.
|
||||
//
|
||||
// The content-blind bound fires precisely when no single identity repeated
|
||||
// enough, so the identity present at the end says nothing about the eleven
|
||||
// before it. Alternating two refusal categories reaches twelve without either
|
||||
// reaching six; the tool never ran once, and the answer used to describe that
|
||||
// as the tool failing with varying errors.
|
||||
func TestTheContentBlindBoundKeepsRefusalProvenance(t *testing.T) {
|
||||
for _, testCase := range []struct {
|
||||
name string
|
||||
kinds []DenialCategory
|
||||
wantCause toolFailureCause
|
||||
wantAnswer string
|
||||
}{
|
||||
{
|
||||
name: "alternating refusal categories",
|
||||
kinds: []DenialCategory{DenialPermissionDenied, DenialSandboxBlock},
|
||||
wantCause: toolFailureCauseVariedRefused,
|
||||
wantAnswer: "was refused",
|
||||
},
|
||||
{
|
||||
// Mixed is DECIDED, not inherited from whichever field a switch
|
||||
// tested first. Neither "failed" alone nor "refused" alone is true of
|
||||
// this sequence, so the wording says both.
|
||||
name: "executed failures mixed with refusals",
|
||||
kinds: []DenialCategory{DenialNone, DenialPermissionDenied},
|
||||
wantCause: toolFailureCauseVariedMixed,
|
||||
wantAnswer: "failed or was refused",
|
||||
},
|
||||
{
|
||||
name: "executed failures only",
|
||||
kinds: []DenialCategory{DenialNone, DenialNone},
|
||||
wantCause: toolFailureCauseVariedExecuted,
|
||||
wantAnswer: "with varying errors",
|
||||
},
|
||||
} {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
state := newGuardState()
|
||||
var outcome toolFailureOutcome
|
||||
for index := 0; index < toolFailureAnyErrorStopAt; index++ {
|
||||
// Distinct prose every call, exactly as a real refusal or error
|
||||
// naming what it touched would produce.
|
||||
outcome = state.observeToolResult("bash", true, false,
|
||||
"failure on item "+strconv.Itoa(index), testCase.kinds[index%len(testCase.kinds)])
|
||||
if outcome.Stop {
|
||||
break
|
||||
}
|
||||
}
|
||||
if !outcome.Stop {
|
||||
t.Fatal("twelve consecutive failures did not trip the content-blind bound")
|
||||
}
|
||||
if outcome.Count != toolFailureAnyErrorStopAt {
|
||||
t.Errorf("Count = %d, want the counter that tripped (%d)", outcome.Count, toolFailureAnyErrorStopAt)
|
||||
}
|
||||
if outcome.Cause != testCase.wantCause {
|
||||
t.Errorf("Cause = %v, want %v", outcome.Cause, testCase.wantCause)
|
||||
}
|
||||
answer := toolFailureStopAnswer("bash", outcome.Count, outcome.Cause)
|
||||
if !strings.Contains(answer, testCase.wantAnswer) {
|
||||
t.Errorf("stop answer = %q, want it to contain %q", answer, testCase.wantAnswer)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1,103 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// deniedWriteTool records every execution. Its body must never run: the sandbox
|
||||
// refuses the call at the registry boundary, before the tool is reached.
|
||||
type deniedWriteTool struct{ ran int }
|
||||
|
||||
func (t *deniedWriteTool) Name() string { return "write_file" }
|
||||
func (t *deniedWriteTool) Description() string { return "test write tool" }
|
||||
func (t *deniedWriteTool) Parameters() tools.Schema {
|
||||
return tools.Schema{
|
||||
Type: "object",
|
||||
Properties: map[string]tools.PropertySchema{"path": {Type: "string"}},
|
||||
Required: []string{"path"},
|
||||
}
|
||||
}
|
||||
|
||||
func (t *deniedWriteTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectWrite, Permission: tools.PermissionAllow, Reason: "writes files"}
|
||||
}
|
||||
|
||||
func (t *deniedWriteTool) Run(context.Context, map[string]any) tools.Result {
|
||||
t.ran++
|
||||
return tools.Result{Status: tools.StatusOK, Output: "wrote"}
|
||||
}
|
||||
|
||||
// THROUGH THE REAL SANDBOX, NOT A FAKE STANDING WHERE IT WOULD BE.
|
||||
//
|
||||
// The neighbouring uncategorized test deliberately hand-builds the shape a
|
||||
// refusal has when no category survives, which is a fact about the LOOP. This
|
||||
// one covers the producer: a real sandbox.Engine evaluation denies the call,
|
||||
// the registry turns that into a categorized refusal, and the guard keys on the
|
||||
// category. Without it, reverting refusalResult(..., PolicyRefusalSandboxDenied)
|
||||
// in registry.go to a plain errorResult left both the agent and tools suites
|
||||
// green, so nothing depended on the marker at all.
|
||||
//
|
||||
// Every assertion here fails if a different link breaks: ran proves the body was
|
||||
// skipped, the request count proves the CATEGORY bound halted it rather than the
|
||||
// content-blind one, and the wording proves the provenance survived conversion.
|
||||
func TestRunHaltsARealSandboxDenialOnTheCategoryBound(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
forbidden := t.TempDir()
|
||||
policy := sandbox.DefaultPolicy()
|
||||
// Applies whenever the sandbox is enforcing, so this does not depend on the
|
||||
// workspace boundary being reachable from a synthetic request.
|
||||
policy.DenyWrite = []string{forbidden}
|
||||
|
||||
tool := &deniedWriteTool{}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tool)
|
||||
|
||||
// A different path every call. Note that errorSignature truncates at 80
|
||||
// characters and the message opens with a fixed 40-character prefix followed
|
||||
// by a long temp root, so the signatures collide here anyway: the request
|
||||
// count below pins WHICH bound halted the run, but it does not by itself
|
||||
// prove the streak was held by the category. The wording assertion is the one
|
||||
// that discriminates, and it is the one that fails when the marker is gone.
|
||||
turns := make([][]zeroruntime.StreamEvent, 0, toolFailureStopAt+4)
|
||||
for i := range toolFailureStopAt + 4 {
|
||||
target := filepath.Join(forbidden, "escape-"+strconv.Itoa(i)+".txt")
|
||||
turns = append(turns, toolTurn("c"+strconv.Itoa(i), "write_file", `{"path":`+strconv.Quote(target)+`}`))
|
||||
}
|
||||
provider := &mockProvider{turns: turns}
|
||||
|
||||
result, err := Run(context.Background(), "write the files", provider, Options{
|
||||
Registry: registry,
|
||||
PermissionMode: PermissionModeAsk,
|
||||
MaxTurns: len(turns) + 5,
|
||||
Cwd: workspace,
|
||||
Sandbox: sandbox.NewEngine(sandbox.EngineOptions{
|
||||
WorkspaceRoot: workspace,
|
||||
Policy: policy,
|
||||
}),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if tool.ran != 0 {
|
||||
t.Errorf("the tool body ran %d times; the sandbox must refuse before execution", tool.ran)
|
||||
}
|
||||
if len(provider.requests) != toolFailureStopAt {
|
||||
t.Errorf("the run made %d turns, want the category bound at %d; a categorized refusal whose prose varies must still halt on its category",
|
||||
len(provider.requests), toolFailureStopAt)
|
||||
}
|
||||
if want := toolFailureStopAnswer("write_file", toolFailureStopAt, toolFailureCauseSameRefusal); result.FinalAnswer != want {
|
||||
t.Errorf("final answer =\n %q\nwant\n %q", result.FinalAnswer, want)
|
||||
}
|
||||
if !strings.Contains(result.FinalAnswer, "was refused") {
|
||||
t.Errorf("the refusal provenance did not survive to the answer: %q", result.FinalAnswer)
|
||||
}
|
||||
}
|
||||
@@ -1,73 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// THE ANSWER MUST CLAIM ONLY WHAT THE COUNTER ESTABLISHED.
|
||||
//
|
||||
// anyErrorCount establishes that the tool failed consecutively without a
|
||||
// success. It does NOT establish that every failure differed: five A, five B and
|
||||
// two C reach the content-blind bound of 12 without any signature repeating six
|
||||
// times, and three of those errors were shared. The old wording called that
|
||||
// "each with a different error".
|
||||
//
|
||||
// The signature bound has the inverse problem. A denial streak keys on the
|
||||
// CATEGORY, and the prose behind it embeds the path or command refused, so it
|
||||
// differs on every call. Calling that "the same error" is false in the other
|
||||
// direction.
|
||||
func TestStopAnswerDoesNotOverclaimTheFailurePattern(t *testing.T) {
|
||||
varied := toolFailureStopAnswer("bash", 12, toolFailureCauseVariedExecuted)
|
||||
if strings.Contains(varied, "each with a different error") {
|
||||
t.Errorf("the content-blind bound does not track that every failure differed: %q", varied)
|
||||
}
|
||||
if !strings.Contains(varied, "varying errors") {
|
||||
t.Errorf("the varied stop should still say the errors varied: %q", varied)
|
||||
}
|
||||
|
||||
refused := toolFailureStopAnswer("bash", 6, toolFailureCauseSameRefusal)
|
||||
if strings.Contains(refused, "same error") {
|
||||
t.Errorf("a denial streak covers refusals of different paths, so it is not the same error: %q", refused)
|
||||
}
|
||||
if !strings.Contains(refused, "refused") {
|
||||
t.Errorf("a denial streak should say it was refused: %q", refused)
|
||||
}
|
||||
|
||||
// The one claim that IS justified: a signature streak really did repeat the
|
||||
// same error signature, so that wording stays.
|
||||
same := toolFailureStopAnswer("bash", 6, toolFailureCauseSameError)
|
||||
if !strings.Contains(same, "same error") {
|
||||
t.Errorf("a signature streak may still be described as the same error: %q", same)
|
||||
}
|
||||
}
|
||||
|
||||
// The mixed-signature case jatmn asked for, driven through the real counter
|
||||
// rather than asserted about the wording in isolation: a run whose errors vary
|
||||
// must trip the content-blind bound and must not be described as all-different.
|
||||
func TestMixedSignatureStreakTripsTheContentBlindBound(t *testing.T) {
|
||||
var state guardState
|
||||
var outcome toolFailureOutcome
|
||||
// Five of one error, five of another, two of a third: twelve failures, no
|
||||
// signature repeating six times in a row.
|
||||
for index, output := range []string{
|
||||
"Error: A", "Error: A", "Error: A", "Error: A", "Error: A",
|
||||
"Error: B", "Error: B", "Error: B", "Error: B", "Error: B",
|
||||
"Error: C", "Error: C",
|
||||
} {
|
||||
outcome = state.observeToolResult("bash", true, true, output, DenialNone)
|
||||
if outcome.Stop && index < 11 {
|
||||
t.Fatalf("halted early at failure %d", index+1)
|
||||
}
|
||||
}
|
||||
if !outcome.Stop {
|
||||
t.Fatal("twelve consecutive failures did not trip the content-blind bound")
|
||||
}
|
||||
if outcome.Cause != toolFailureCauseVariedExecuted {
|
||||
t.Errorf("a mixed-signature streak of executed failures should report varied-executed, got %v", outcome.Cause)
|
||||
}
|
||||
answer := toolFailureStopAnswer("bash", outcome.Count, outcome.Cause)
|
||||
if strings.Contains(answer, "each with a different error") {
|
||||
t.Errorf("three of these errors were shared, so they were not each different: %q", answer)
|
||||
}
|
||||
}
|
||||
@@ -11,7 +11,6 @@ import (
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/repomap"
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/workspaceseed"
|
||||
)
|
||||
@@ -67,72 +66,6 @@ const (
|
||||
workspaceSeedWidth = 100
|
||||
)
|
||||
|
||||
// runCanMutate reports whether this run holds any tool that could change
|
||||
// something — and therefore whether the confirmation policy applies to it.
|
||||
//
|
||||
// FAIL CLOSED at every step. A nil registry, a run with no visible tools, and
|
||||
// any tool whose effect is not explicitly read-only all answer TRUE, so the
|
||||
// policy is dropped ONLY when every visible tool is positively known to be a
|
||||
// pure read. EffectUnknown (MCP tools, plugin tools, the specialist tools)
|
||||
// keeps it, which is what makes an ordinary session byte-identical.
|
||||
//
|
||||
// Keyed on the RESOLVED TOOL SET, not on a specialist name or a manifest flag:
|
||||
// what a run can do is decided by the tools it actually holds, and a name is
|
||||
// not a capability.
|
||||
//
|
||||
// Filters, NOT advertising. ToolAllowedByFilters answers "does this run hold
|
||||
// the tool"; ToolVisible also applies the permission mode's advertising rules,
|
||||
// and a write tool that auto mode does not advertise is still held and still
|
||||
// callable once approved. Using ToolVisible here dropped the policy from a run
|
||||
// launched with --enabled-tools read_file,grep,glob,write_file — a fail-OPEN
|
||||
// exactly inverse to this function's purpose, caught by driving the binary.
|
||||
func runCanMutate(options Options) bool {
|
||||
if options.Registry == nil {
|
||||
return true
|
||||
}
|
||||
held := 0
|
||||
for _, tool := range options.Registry.All() {
|
||||
if !ToolAllowedByFilters(tool.Name(), options.EnabledTools, options.DisabledTools) {
|
||||
continue
|
||||
}
|
||||
if permanentlyDenied(tool) {
|
||||
// A tool the run can never call cannot act, so it says nothing
|
||||
// about what the run can do. This is what keeps registering a
|
||||
// posture-gated tool from changing the prompt while the posture is
|
||||
// off — the additivity guarantee, which its identity test caught
|
||||
// this function breaking.
|
||||
continue
|
||||
}
|
||||
held++
|
||||
if tools.CapabilitiesOf(tool).Effect != tools.EffectReadOnly {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return held == 0
|
||||
}
|
||||
|
||||
// permanentlyDenied reports whether a tool can never be invoked in this run.
|
||||
//
|
||||
// A tool that varies its permission by arguments is NEVER treated as denied,
|
||||
// however its static safety reads: the question is whether any call could
|
||||
// succeed, and only a tool with no per-argument override can be ruled out from
|
||||
// its static permission alone. Fail closed.
|
||||
func permanentlyDenied(tool tools.Tool) bool {
|
||||
// A tool that KNOWS it cannot fire is believed first, before the
|
||||
// argument-varying escape below. Otherwise a tool gated on something that is
|
||||
// not an argument — the zeromaxing posture is a session state, not a
|
||||
// parameter — would count as held in every run merely for implementing
|
||||
// ArgsPermissioner, and the confirmation policy would appear in read-only
|
||||
// runs that do not have the feature turned on at all.
|
||||
if denier, ok := tool.(tools.PermanentDenier); ok && denier.PermanentlyDenied() {
|
||||
return true
|
||||
}
|
||||
if _, varies := tool.(tools.ArgsPermissioner); varies {
|
||||
return false
|
||||
}
|
||||
return tool.Safety().Permission == tools.PermissionDeny
|
||||
}
|
||||
|
||||
// buildSystemPrompt assembles the full system prompt for a run: the core
|
||||
// coding-craft instructions, dynamic workspace context (cwd, git branch, project
|
||||
// guidelines), and the safety confirmation policy. It is built once per run so
|
||||
@@ -165,7 +98,7 @@ func buildSystemPromptParts(options Options) systemPromptParts {
|
||||
if transient := strings.TrimSpace(options.TransientSystemPrompt); transient != "" {
|
||||
sections = append(sections, transient)
|
||||
}
|
||||
if addendum := modelPromptAddendum(options.ModelFamily, options.Model); addendum != "" {
|
||||
if addendum := modelPromptAddendum(options.Model); addendum != "" {
|
||||
sections = append(sections, addendum)
|
||||
}
|
||||
if session := sessionRuntimeContext(options); session != "" {
|
||||
@@ -184,7 +117,7 @@ func buildSystemPromptParts(options Options) systemPromptParts {
|
||||
if user := userGuidelines(); user != "" {
|
||||
sections = append(sections, user)
|
||||
}
|
||||
project := workspaceContext(options.Cwd, options.Sandbox)
|
||||
project := workspaceContext(options.Cwd)
|
||||
if project != "" {
|
||||
sections = append(sections, project)
|
||||
}
|
||||
@@ -202,13 +135,6 @@ func buildSystemPromptParts(options Options) systemPromptParts {
|
||||
sections = append(sections, style)
|
||||
}
|
||||
policy := strings.TrimSpace(confirmationPolicy)
|
||||
if !runCanMutate(options) {
|
||||
// A run that cannot mutate anything has nothing to confirm. Carrying
|
||||
// ~5 KB of confirmation policy into such a run is pure cost: measured
|
||||
// at 2,527 of a plan child's 7,648 prompt tokens, multiplied by every
|
||||
// task in a plan and by every specialist sub-agent.
|
||||
policy = ""
|
||||
}
|
||||
if policy != "" {
|
||||
sections = append(sections, policy)
|
||||
}
|
||||
@@ -380,7 +306,7 @@ func sessionRuntimeContext(options Options) string {
|
||||
// directory, git branch, and any project guideline doc, so the model grounds its
|
||||
// work in the actual repo. Returns "" when cwd is unset (keeps headless/test
|
||||
// runs deterministic).
|
||||
func workspaceContext(cwd string, sandboxEngine *sandbox.Engine) string {
|
||||
func workspaceContext(cwd string) string {
|
||||
cwd = strings.TrimSpace(cwd)
|
||||
if cwd == "" {
|
||||
return ""
|
||||
@@ -389,7 +315,7 @@ func workspaceContext(cwd string, sandboxEngine *sandbox.Engine) string {
|
||||
b.WriteString("<environment>\n")
|
||||
b.WriteString("Working directory: " + cwd + "\n")
|
||||
b.WriteString("Operating system: " + runtime.GOOS + "\n")
|
||||
b.WriteString(tools.HostShellEnvironmentGuidanceForEngine(sandboxEngine, cwd) + "\n")
|
||||
b.WriteString(tools.HostShellEnvironmentGuidance() + "\n")
|
||||
if branch := gitBranchForPrompt(cwd); branch != "" {
|
||||
b.WriteString("Git branch: " + branch + "\n")
|
||||
}
|
||||
|
||||
@@ -22,13 +22,6 @@ const (
|
||||
|
||||
// modelFamily classifies a model id into a prompt-tuning family, or "" when
|
||||
// unknown (in which case no addendum is added).
|
||||
//
|
||||
// A GUESS, AND THE FALLBACK ONLY. Matching id prefixes answers correctly for
|
||||
// first-party ids and for the vendor-prefixed ones a gateway uses
|
||||
// ("openai/gpt-4.1"), and answers nothing at all for most of the market: of the
|
||||
// 37 providers in the catalog, 28 have a default model that matches none of
|
||||
// these arms — including the recommended default. Where the PROVIDER knows the
|
||||
// answer it is passed in instead; see modelPromptAddendum.
|
||||
func modelFamily(model string) string {
|
||||
m := strings.ToLower(strings.TrimSpace(model))
|
||||
switch {
|
||||
@@ -48,21 +41,8 @@ func modelFamily(model string) string {
|
||||
|
||||
// modelPromptAddendum returns the family-specific prompt block for a model id,
|
||||
// or "" when the family is unknown.
|
||||
// modelPromptAddendum returns the family-specific prompt block, preferring the
|
||||
// family the PROVIDER declared over anything guessed from the model id.
|
||||
//
|
||||
// The provider is authoritative where it is single-family, and the id is not: a
|
||||
// ChatGPT OAuth session was classified correctly only because "gpt-5.5" happens
|
||||
// to begin with "gpt", which no future id is obliged to do. Gateways serve
|
||||
// several families at once and declare none, so those still fall back to the id
|
||||
// — which is the right answer for them, since only the id distinguishes
|
||||
// "openai/gpt-4.1" from "z-ai/glm-4.6" on the same endpoint.
|
||||
func modelPromptAddendum(providerFamily, model string) string {
|
||||
family := strings.TrimSpace(providerFamily)
|
||||
if family == "" {
|
||||
family = modelFamily(model)
|
||||
}
|
||||
switch family {
|
||||
func modelPromptAddendum(model string) string {
|
||||
switch modelFamily(model) {
|
||||
case familyOpenAI:
|
||||
return openAIPromptAddendum
|
||||
case familyGemini:
|
||||
|
||||
@@ -42,7 +42,7 @@ func TestBuildSystemPromptAppendsModelAddendum(t *testing.T) {
|
||||
t.Fatalf("the claude prompt must not contain the OpenAI addendum")
|
||||
}
|
||||
// Unknown / unset model gets no family block.
|
||||
if got := modelPromptAddendum("", ""); got != "" {
|
||||
if got := modelPromptAddendum(""); got != "" {
|
||||
t.Fatalf("expected no addendum without a model, got %q", got)
|
||||
}
|
||||
if strings.Contains(buildSystemPrompt(Options{}), "<model_guidance>") {
|
||||
|
||||
@@ -1,20 +1,14 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
"github.com/Twigpine/zero/internal/testutil"
|
||||
)
|
||||
|
||||
// isolatedUserDirsRoot is where TestMain points every per-user directory.
|
||||
var isolatedUserDirsRoot string
|
||||
|
||||
// TestMain points userConfigDirForPrompt at an empty, package-wide temp
|
||||
// directory for every test in this package. Without this, buildSystemPrompt
|
||||
// falls back to the real config.UserConfigDir, so any developer with a
|
||||
@@ -23,56 +17,17 @@ var isolatedUserDirsRoot string
|
||||
// across contributor machines. Tests that specifically exercise user
|
||||
// guidelines stub userConfigDirForPrompt themselves via
|
||||
// withSystemPromptTestUserConfigDir(Func) and restore this default on cleanup.
|
||||
//
|
||||
// It also points every per-user directory root at the same temp directory:
|
||||
// sandboxed tool runs keep their runtime state and lease files under the user
|
||||
// cache dir, so without this they land in the developer's real cache.
|
||||
func TestMain(m *testing.M) {
|
||||
dir, err := os.MkdirTemp("", "zero-agent-tests-*")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
isolatedUserDirsRoot = dir
|
||||
restore, err := testutil.IsolateUserDirs(dir)
|
||||
if err != nil {
|
||||
os.RemoveAll(dir)
|
||||
panic(err)
|
||||
}
|
||||
userConfigDirForPrompt = func() (string, error) { return dir, nil }
|
||||
code := m.Run()
|
||||
if err := restore(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "agent tests: restore user dirs:", err)
|
||||
code = 1
|
||||
}
|
||||
os.RemoveAll(dir)
|
||||
os.Exit(code)
|
||||
}
|
||||
|
||||
// Guards TestMain: the per-user directories the package can read or write
|
||||
// must resolve inside the isolated root on every OS.
|
||||
func TestUserDirsAreIsolatedFromTheDeveloperHome(t *testing.T) {
|
||||
if isolatedUserDirsRoot == "" {
|
||||
t.Fatal("TestMain did not record an isolated root")
|
||||
}
|
||||
home, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
userConfig, err := config.UserConfigDir()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cache, err := os.UserCacheDir()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for name, dir := range map[string]string{"home": home, "config": userConfig, "cache": cache} {
|
||||
if !testutil.PathWithin(isolatedUserDirsRoot, dir) {
|
||||
t.Errorf("%s dir %q is outside the isolated root %q", name, dir, isolatedUserDirsRoot)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoreSystemPromptIncludesCodingQualityRules(t *testing.T) {
|
||||
prompt := strings.Join(strings.Fields(strings.ToLower(buildSystemPrompt(Options{}))), " ")
|
||||
if strings.Contains(prompt, "clickable file paths") {
|
||||
|
||||
@@ -1,169 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// spendingTurn is one provider turn that calls a tool and reports usage.
|
||||
func spendingTurn(id string, tokens int) []zeroruntime.StreamEvent {
|
||||
return []zeroruntime.StreamEvent{
|
||||
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: id, ToolName: "run_tests"},
|
||||
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: id, ArgumentsFragment: `{}`},
|
||||
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: id},
|
||||
{Type: zeroruntime.StreamEventUsage, Usage: zeroruntime.Usage{PromptTokens: tokens, CompletionTokens: 0}},
|
||||
{Type: zeroruntime.StreamEventDone},
|
||||
}
|
||||
}
|
||||
|
||||
func budgetProvider(turns int, perTurn int, finalText string) *mockProvider {
|
||||
streams := make([][]zeroruntime.StreamEvent, 0, turns+1)
|
||||
for i := 0; i < turns; i++ {
|
||||
streams = append(streams, spendingTurn("call-"+string(rune('a'+i)), perTurn))
|
||||
}
|
||||
streams = append(streams, []zeroruntime.StreamEvent{
|
||||
{Type: zeroruntime.StreamEventText, Content: finalText},
|
||||
{Type: zeroruntime.StreamEventDone},
|
||||
})
|
||||
return &mockProvider{turns: streams}
|
||||
}
|
||||
|
||||
func budgetRegistry() *tools.Registry {
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(goTestTool{output: "ok\tpkg\t0.10s\n"})
|
||||
return registry
|
||||
}
|
||||
|
||||
// A RUN MUST BE BOUNDED BY WHAT IT SPENDS, not only by how many round trips it
|
||||
// makes.
|
||||
//
|
||||
// MaxTurns is a proxy for cost that does not track cost: a measured heavy run
|
||||
// reached its 320-turn limit having spent 35,781,390 tokens, where a cheap run
|
||||
// reaches the same limit at roughly a tenth of that. The turn count bounds round
|
||||
// trips; nothing bounded spend.
|
||||
func TestARunStopsOnItsTokenBudget(t *testing.T) {
|
||||
provider := budgetProvider(6, 1000, "done")
|
||||
result, err := Run(context.Background(), "work", provider, Options{
|
||||
Cwd: t.TempDir(), Registry: budgetRegistry(), SessionID: "budget",
|
||||
MaxTurns: 50, MaxTokens: 2500,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// 1000 per turn against 2500: turns 1-3 run (3000 spent), turn 4 is refused.
|
||||
if len(provider.requests) > 5 {
|
||||
t.Fatalf("the run made %d requests; the token budget did not bind", len(provider.requests))
|
||||
}
|
||||
if strings.TrimSpace(result.FinalAnswer) == "" {
|
||||
t.Error("a budget expiry produced no final answer; it must end with a written summary, not silence")
|
||||
}
|
||||
}
|
||||
|
||||
// THE READER MUST BE TOLD WHICH BOUND FIRED. Reporting a turn limit for a spend
|
||||
// stop sends someone to raise a number that was never what ran out.
|
||||
func TestATokenStopSaysSoRatherThanBlamingTheTurnCount(t *testing.T) {
|
||||
provider := budgetProvider(6, 1000, "done")
|
||||
result, err := Run(context.Background(), "work", provider, Options{
|
||||
Cwd: t.TempDir(), Registry: budgetRegistry(), SessionID: "budget-reason",
|
||||
MaxTurns: 50, MaxTokens: 2500, RequireCompletionSignal: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(result.IncompleteReason, "token budget") {
|
||||
t.Errorf("IncompleteReason = %q; a spend stop must name the token budget", result.IncompleteReason)
|
||||
}
|
||||
if strings.Contains(result.IncompleteReason, "max-turns") {
|
||||
t.Errorf("a token stop was reported as a turn-limit stop: %q", result.IncompleteReason)
|
||||
}
|
||||
// ...and the model is told the same thing, so its summary is about the right
|
||||
// constraint.
|
||||
last := provider.requests[len(provider.requests)-1]
|
||||
rendered := renderZeromaxingMessages(last.Messages)
|
||||
if !strings.Contains(rendered, "token budget") {
|
||||
t.Errorf("the final-answer prompt does not mention the token budget:\n%s", rendered[max(0, len(rendered)-400):])
|
||||
}
|
||||
}
|
||||
|
||||
// A TURN-COUNT STOP STILL READS AS ONE. The new branch must not relabel the old
|
||||
// ending.
|
||||
func TestATurnStopStillSaysMaxTurns(t *testing.T) {
|
||||
provider := budgetProvider(6, 10, "done")
|
||||
result, err := Run(context.Background(), "work", provider, Options{
|
||||
Cwd: t.TempDir(), Registry: budgetRegistry(), SessionID: "turn-reason",
|
||||
MaxTurns: 3, RequireCompletionSignal: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(result.IncompleteReason, "max-turns") {
|
||||
t.Errorf("IncompleteReason = %q; a turn stop must still name the turn limit", result.IncompleteReason)
|
||||
}
|
||||
}
|
||||
|
||||
// AN UNSET BUDGET IS UNBOUNDED, so every caller that never sets it behaves
|
||||
// exactly as before.
|
||||
func TestAnUnsetTokenBudgetBoundsNothing(t *testing.T) {
|
||||
provider := budgetProvider(4, 1_000_000, "finished")
|
||||
result, err := Run(context.Background(), "work", provider, Options{
|
||||
Cwd: t.TempDir(), Registry: budgetRegistry(), SessionID: "unbounded",
|
||||
MaxTurns: 50,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(provider.requests) != 5 {
|
||||
t.Fatalf("an unbounded run made %d requests, want all 5", len(provider.requests))
|
||||
}
|
||||
if !strings.Contains(result.FinalAnswer, "finished") {
|
||||
t.Errorf("final answer = %q", result.FinalAnswer)
|
||||
}
|
||||
}
|
||||
|
||||
// THE BUDGET IS CHECKED AT A TURN BOUNDARY, so a turn whose tool calls have
|
||||
// already run is not discarded. Stopping mid-turn would throw away work the run
|
||||
// has already paid for.
|
||||
func TestTheBudgetDoesNotDiscardAnAlreadyPaidTurn(t *testing.T) {
|
||||
ran := 0
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(countingTool{count: &ran})
|
||||
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
|
||||
// One turn that blows the entire budget in a single exchange.
|
||||
{
|
||||
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "c1", ToolName: "counted"},
|
||||
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "c1", ArgumentsFragment: `{}`},
|
||||
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "c1"},
|
||||
{Type: zeroruntime.StreamEventUsage, Usage: zeroruntime.Usage{PromptTokens: 9999}},
|
||||
{Type: zeroruntime.StreamEventDone},
|
||||
},
|
||||
{{Type: zeroruntime.StreamEventText, Content: "summary"}, {Type: zeroruntime.StreamEventDone}},
|
||||
}}
|
||||
if _, err := Run(context.Background(), "work", provider, Options{
|
||||
Cwd: t.TempDir(), Registry: registry, SessionID: "boundary",
|
||||
MaxTurns: 50, MaxTokens: 100,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ran != 1 {
|
||||
t.Errorf("the tool ran %d times; the turn's already-issued call must still execute", ran)
|
||||
}
|
||||
}
|
||||
|
||||
type countingTool struct{ count *int }
|
||||
|
||||
func (countingTool) Name() string { return "counted" }
|
||||
func (countingTool) Description() string { return "counts" }
|
||||
func (countingTool) Parameters() tools.Schema {
|
||||
return tools.Schema{Type: "object", AdditionalProperties: false}
|
||||
}
|
||||
func (countingTool) Safety() tools.Safety {
|
||||
return tools.Safety{SideEffect: tools.SideEffectNone, Permission: tools.PermissionAllow, AdvertiseInAuto: true}
|
||||
}
|
||||
func (t countingTool) Run(context.Context, map[string]any) tools.Result {
|
||||
*t.count++
|
||||
return tools.Result{Status: tools.StatusOK, Output: "ok"}
|
||||
}
|
||||
+4
-46
@@ -325,24 +325,13 @@ type Options struct {
|
||||
SessionTitle string
|
||||
ProviderName string
|
||||
Model string
|
||||
// ModelFamily is the family the PROVIDER declares it serves, from the
|
||||
// provider catalog — "openai", "anthropic", "gemini", or "" when the provider
|
||||
// is a gateway serving several and cannot answer. Empty falls back to
|
||||
// classifying the model id, which is a guess that matches nothing for most of
|
||||
// the catalog. Prompt tuning only; it selects the family addendum and nothing
|
||||
// else.
|
||||
ModelFamily string
|
||||
ReasoningEffort string
|
||||
ServiceTier string
|
||||
Cwd string
|
||||
SystemPrompt string
|
||||
ReasoningEffort string
|
||||
ServiceTier string
|
||||
Cwd string
|
||||
SystemPrompt string
|
||||
// TransientSystemPrompt adds trusted runtime guidance for this run only.
|
||||
// Empty preserves the ordinary system prompt byte-for-byte.
|
||||
TransientSystemPrompt string
|
||||
// userMessage is this turn's raw user text, set by Run from its prompt
|
||||
// argument and forwarded to tools. Unexported: it is not a knob a caller
|
||||
// sets, it is what the caller already passed as `prompt`.
|
||||
userMessage string
|
||||
// ResponseStyle is the operator-selected reply style from the TUI /style
|
||||
// command (e.g. "concise", "explanatory", "review"). It is rendered into the
|
||||
// system prompt as a short directive. Empty or "balanced" adds nothing — the
|
||||
@@ -357,19 +346,6 @@ type Options struct {
|
||||
// boundary when this is false. nil uses the curated catalog plus the
|
||||
// name heuristic via modelregistry.SupportsVision.
|
||||
SupportsVision func(modelID string) bool
|
||||
// MaxTokens bounds what a whole run may SPEND, in provider-reported tokens.
|
||||
// 0 is unbounded, which is every caller that does not set it.
|
||||
//
|
||||
// MaxTurns IS NOT A COST BOUND, and treating it as one is how a run ends at
|
||||
// an arbitrary place. A cheap run reaches 320 turns having spent ~3M tokens; a
|
||||
// measured heavy run reached the same 320 having spent 35,781,390. The turn
|
||||
// count bounds round trips, not spend, and the two diverge by an order of
|
||||
// magnitude on exactly the runs worth bounding.
|
||||
//
|
||||
// Crossing it ends the run the same way MaxTurns does — one final call asking
|
||||
// for a summary — so a budget expiry produces a written report of what was
|
||||
// done rather than a run that simply stops.
|
||||
MaxTokens int
|
||||
// ContextWindow is the model's maximum input token budget. When > 0 the agent
|
||||
// loop compacts long conversations once the estimated size crosses a fraction
|
||||
// of this window. 0 DISABLES compaction entirely (every existing caller/test
|
||||
@@ -461,24 +437,6 @@ type Options struct {
|
||||
// byte-identical: no observation, no escalation, no counters. Same opt-in
|
||||
// convention as Trace and SelfCorrect.
|
||||
Profile *ProfilePolicy
|
||||
// Zeromaxing reports where this run sits in the zeromaxing execution
|
||||
// posture's lifecycle, driving the model-facing reminders the turn loop
|
||||
// appends. The zero value (ZeromaxingOff) suppresses every reminder, so a
|
||||
// caller that never heard of it is byte-identical to today. Deliberately
|
||||
// separate from Profile: zeromaxing arms no escalation triggers, so
|
||||
// Profile.Policy() returns nil for it and could not carry this.
|
||||
Zeromaxing Zeromaxing
|
||||
// PlanCompletions, when set, is drained once per turn for background plans
|
||||
// that finished since the last turn. Its text is appended to the
|
||||
// conversation TAIL as a user-role message, on the same channel as the
|
||||
// post-edit diagnostics nudge — background work reporting into a later turn
|
||||
// already has a channel here, and a second one would be a second ordering
|
||||
// and a second thing to budget. nil is a no-op.
|
||||
PlanCompletions func() string
|
||||
// OrchestrateAvailable reports whether the orchestrate tool is actually
|
||||
// advertised this run, so the enter notice names it only when it exists.
|
||||
// Zero value false keeps every existing caller's prompt unchanged.
|
||||
OrchestrateAvailable bool
|
||||
// Trace, when set, records per-turn timing for the run: the loop stamps
|
||||
// spans (prompt build, generation, tool execution, permission wait,
|
||||
// compaction, provider connect) and counters (model requests, tool calls,
|
||||
|
||||
@@ -1,167 +0,0 @@
|
||||
package agent
|
||||
|
||||
// The zeromaxing execution posture's model-facing reminders.
|
||||
//
|
||||
// WHERE THESE GO, AND WHY IT IS NOT NEGOTIABLE
|
||||
//
|
||||
// Every string here is appended to the CONVERSATION as a user-role message, at
|
||||
// the tail, exactly like the failure hint and the plan/progress reminders in
|
||||
// the turn loop. None of it ever reaches buildSystemPromptParts.
|
||||
//
|
||||
// The system prompt and the tool definitions are the provider's cached prefix
|
||||
// (the anthropic mapper puts its cache_control breakpoint on the last system
|
||||
// block and the last tool). #760 made that prefix build ONCE per run precisely
|
||||
// so it stays byte-identical across turns. A reminder that varies per turn and
|
||||
// lands above the breakpoint invalidates the cache on every single turn —
|
||||
// roughly doubling input cost — and nothing in the system reports it. Hence:
|
||||
// tail only.
|
||||
//
|
||||
// The same reasoning is why ZeromaxingStillOnNotice is a fixed literal with no
|
||||
// turn counter, no timestamp, and no accumulated state. It is repeated verbatim
|
||||
// on every continuing turn. If it ever grew a varying part and someone later
|
||||
// moved it into prompt assembly, that would be a silent cost bug rather than a
|
||||
// loud test failure. TestRunPreservesRequestPrefixAcrossTurnsUnderZeromaxing is
|
||||
// the tripwire.
|
||||
//
|
||||
// The wording is written for this project: plain, neutral, describing Zero's
|
||||
// own posture in Zero's own terms. It deliberately promises no orchestration,
|
||||
// no fan-out and no workflow tool — Phase 1 ships none of those, and a reminder
|
||||
// that implies capabilities the run does not have is a prompt-level lie.
|
||||
|
||||
// Zeromaxing is where a run sits in the zeromaxing posture's lifecycle. It is
|
||||
// set by whoever selected the posture (headless exec for a one-shot run, the
|
||||
// TUI across a session) and read only by the turn loop.
|
||||
type Zeromaxing int
|
||||
|
||||
const (
|
||||
// ZeromaxingOff: the posture is not active. Every reminder is suppressed and
|
||||
// the loop is byte-identical to a run that never heard of it. This is the
|
||||
// zero value, so every existing caller keeps today's behaviour.
|
||||
ZeromaxingOff Zeromaxing = iota
|
||||
// ZeromaxingEntering: this run is the first since the posture was turned on,
|
||||
// so its first turn carries the enter notice and the budget guideline.
|
||||
ZeromaxingEntering
|
||||
// ZeromaxingActive: the posture was already on before this run began, so
|
||||
// even the first turn gets the continuing notice rather than the enter one.
|
||||
ZeromaxingActive
|
||||
// ZeromaxingExiting: this run is the first since the posture was turned off.
|
||||
// Its first turn carries the exit notice and nothing after that.
|
||||
ZeromaxingExiting
|
||||
)
|
||||
|
||||
// The four messages. Exported as constants so tests assert on the same bytes
|
||||
// the model receives rather than on a paraphrase.
|
||||
const (
|
||||
// ZeromaxingEnterNotice announces the flip. It fires once, on the first turn
|
||||
// of the run in which the posture was selected.
|
||||
ZeromaxingEnterNotice = "The zeromaxing execution posture is now active for this session. " +
|
||||
"You have a substantially larger tool-turn budget than usual. Use it for depth on the task that was asked: " +
|
||||
"read before changing, run the checks that would catch a mistake, and follow up on anything you noticed but could not confirm."
|
||||
|
||||
// ZeromaxingBudgetNotice states the budget guideline alongside the enter
|
||||
// notice. Split from it so the two can be asserted independently, and so a
|
||||
// future budget change touches one string.
|
||||
ZeromaxingBudgetNotice = "Budget guideline under zeromaxing: the larger turn budget is for verifying one task thoroughly, not for starting extra work. " +
|
||||
"Finish what was asked, confirm it, and stop."
|
||||
|
||||
// ZeromaxingOrchestrateNotice is appended to the enter notice ONLY when the
|
||||
// orchestrate tool is actually available this run. It is the behavioral
|
||||
// directive that turns zeromaxing from "more turns" into "multi-agent depth".
|
||||
// Naming a tool the run does not have would be a prompt-level lie the model
|
||||
// would try to act on, which is why this is conditional rather than folded
|
||||
// into the enter notice.
|
||||
ZeromaxingOrchestrateNotice = "You also have the orchestrate tool and background task support. " +
|
||||
"For substantive work, split independent read/analyze pieces into a planned DAG with explicit depends_on, run them through the orchestrate tool, and verify the merged findings before acting. " +
|
||||
"For long-running commands, prefer Bash with run_in_background. " +
|
||||
"Work alone only on conversational turns or trivial single-file edits."
|
||||
|
||||
// ZeromaxingEvidenceNotice states what counts as evidence. It fires on the
|
||||
// FIRST TURN OF EVERY RUN while the posture is on — once per user message,
|
||||
// which is when a claim arrives and when a report is written. Repeating it on
|
||||
// every tool turn would cost hundreds of copies to say something that only
|
||||
// bears on the boundaries of a run.
|
||||
//
|
||||
// EARNED BY A MEASURED RUN, three failure classes in one submission:
|
||||
//
|
||||
// A property was scored top marks in five places on the strength of "the
|
||||
// test passes" — while the test consumed the lazy iterator it was meant to
|
||||
// prove lazy, discarded the value whose absence was the bug, and reported
|
||||
// avg × 2.5 as a p99. Passing was treated as proof without once asking what
|
||||
// the test would still pass with.
|
||||
//
|
||||
// A table of benchmark timings was reported that no command in the session
|
||||
// had produced. The same test read 0.86s in one paste and 4.20s in the
|
||||
// next with nothing said about the difference, and the column summed to an
|
||||
// exact total no real transcript lands on. That is the most serious of the
|
||||
// three, because it is not an oversight.
|
||||
//
|
||||
// Told its code had a defect, it agreed — and the claim was wrong. It
|
||||
// revised a correct answer down and wrote a proof that contradicted its own
|
||||
// preceding step. An agent that folds to whoever asserts most confidently
|
||||
// follows the user rather than the evidence, and is worth less the more it
|
||||
// is trusted.
|
||||
ZeromaxingEvidenceNotice = "Evidence discipline under zeromaxing. " +
|
||||
"A passing test is not proof that a property holds: whenever you rest a claim on one, name the test AND say in one sentence what it would still pass with. " +
|
||||
"Every measurement you report must come from a command you ran in this session — if a number differs from one you gave earlier, show both and account for the difference rather than quietly replacing it. " +
|
||||
"And when someone tells you your code has a defect, prove or refute it from the code before you agree: agreeing without a line-level trace is a wrong answer even when the claim turns out to be right."
|
||||
|
||||
// ZeromaxingStillOnNotice repeats on every continuing turn.
|
||||
//
|
||||
// It is intentionally short and FIXED. Do not add a turn number, a
|
||||
// remaining-budget count, a timestamp, or anything else that differs between
|
||||
// turns — see the file comment.
|
||||
ZeromaxingStillOnNotice = "The zeromaxing execution posture is still active."
|
||||
|
||||
// ZeromaxingExitNotice announces the flip back. It fires once, on the first
|
||||
// turn of the run after the posture was turned off.
|
||||
ZeromaxingExitNotice = "The zeromaxing execution posture is no longer active. " +
|
||||
"The tool-turn budget has returned to its normal value; work at the usual depth and avoid launching heavy multi-step side tasks."
|
||||
)
|
||||
|
||||
// zeromaxingReminders returns the reminder lines to append before the given
|
||||
// turn, oldest first, or nil when there is nothing to say. turn is 1-based, so
|
||||
// turn == 1 is the run's first provider request.
|
||||
//
|
||||
// The whole state machine is this function: it is pure, it depends only on
|
||||
// (posture, turn), and it has no memory. That is what makes "enter exactly
|
||||
// once" and "still-on never on the first turn" testable as data rather than as
|
||||
// a sequence of observed side effects.
|
||||
func zeromaxingReminders(posture Zeromaxing, turn int, orchestrateAvailable bool) []string {
|
||||
if turn < 1 {
|
||||
return nil
|
||||
}
|
||||
switch posture {
|
||||
case ZeromaxingEntering:
|
||||
if turn == 1 {
|
||||
notices := []string{ZeromaxingEnterNotice, ZeromaxingBudgetNotice}
|
||||
if orchestrateAvailable {
|
||||
notices = append(notices, ZeromaxingOrchestrateNotice)
|
||||
}
|
||||
// LAST, so it is the nearest instruction to the user's message.
|
||||
return append(notices, ZeromaxingEvidenceNotice)
|
||||
}
|
||||
return []string{ZeromaxingStillOnNotice}
|
||||
case ZeromaxingActive:
|
||||
// Already on when the run started: no enter notice, not even on turn 1.
|
||||
//
|
||||
// The evidence contract still repeats, because it is per-RUN rather than
|
||||
// per-posture-change: turn 1 is where the user's message sits, so it is
|
||||
// where a claim about the code arrives and where a report gets written.
|
||||
// Announcing it only when the posture flipped would leave every later run
|
||||
// of a long session without it — and the long sessions are the ones that
|
||||
// produce reports.
|
||||
if turn == 1 {
|
||||
return []string{ZeromaxingStillOnNotice, ZeromaxingEvidenceNotice}
|
||||
}
|
||||
return []string{ZeromaxingStillOnNotice}
|
||||
case ZeromaxingExiting:
|
||||
// One notice on the way out, then silence — the posture is off, so a
|
||||
// "still active" line every turn would be a lie.
|
||||
if turn == 1 {
|
||||
return []string{ZeromaxingExitNotice}
|
||||
}
|
||||
return nil
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
@@ -1,225 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// (i)(j)(k) The whole reminder state machine as data. zeromaxingReminders is
|
||||
// pure, so "enter exactly once" and "still-on never on the first turn" are
|
||||
// assertions about a table rather than about observed side effects.
|
||||
func TestZeromaxingReminderSchedule(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
posture Zeromaxing
|
||||
turn int
|
||||
want []string
|
||||
}{
|
||||
// (i) enter fires exactly once, on the first turn of the entering run.
|
||||
{"entering turn 1 announces, states the budget and the evidence contract", ZeromaxingEntering, 1,
|
||||
[]string{ZeromaxingEnterNotice, ZeromaxingBudgetNotice, ZeromaxingEvidenceNotice}},
|
||||
// (j) still-on from turn 2, and it is the ONLY thing from turn 2 on —
|
||||
// no second enter, no repeated budget notice.
|
||||
{"entering turn 2 continues", ZeromaxingEntering, 2, []string{ZeromaxingStillOnNotice}},
|
||||
{"entering turn 3 continues", ZeromaxingEntering, 3, []string{ZeromaxingStillOnNotice}},
|
||||
{"entering turn 99 continues", ZeromaxingEntering, 99, []string{ZeromaxingStillOnNotice}},
|
||||
|
||||
// (j) a run that began with the posture already on gets still-on even on
|
||||
// turn 1 — re-announcing entry every run would be wrong.
|
||||
// The evidence contract is per-RUN, not per-posture-change: turn 1 is
|
||||
// where the user's message sits, so it is where a claim about the code
|
||||
// arrives and where a report gets written.
|
||||
{"active turn 1 does not re-announce but restates the evidence contract", ZeromaxingActive, 1,
|
||||
[]string{ZeromaxingStillOnNotice, ZeromaxingEvidenceNotice}},
|
||||
{"active turn 2 continues", ZeromaxingActive, 2, []string{ZeromaxingStillOnNotice}},
|
||||
|
||||
// (k) exit fires exactly once, then silence — the posture is off, so a
|
||||
// "still active" line afterwards would be a lie.
|
||||
{"exiting turn 1 announces once", ZeromaxingExiting, 1, []string{ZeromaxingExitNotice}},
|
||||
{"exiting turn 2 is silent", ZeromaxingExiting, 2, nil},
|
||||
{"exiting turn 5 is silent", ZeromaxingExiting, 5, nil},
|
||||
|
||||
// Off is completely silent: an unaware caller's run is byte-identical.
|
||||
{"off turn 1 is silent", ZeromaxingOff, 1, nil},
|
||||
{"off turn 7 is silent", ZeromaxingOff, 7, nil},
|
||||
|
||||
// Defensive: a non-positive turn never emits.
|
||||
{"turn 0 is silent", ZeromaxingEntering, 0, nil},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := zeromaxingReminders(tc.posture, tc.turn, false)
|
||||
if !reflect.DeepEqual(got, tc.want) {
|
||||
t.Fatalf("zeromaxingReminders(%v, %d) = %#v, want %#v", tc.posture, tc.turn, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Counting across a whole run, the way a reader of the gate would check it.
|
||||
func TestZeromaxingEnterAndExitFireExactlyOncePerRun(t *testing.T) {
|
||||
count := func(posture Zeromaxing, needle string) int {
|
||||
n := 0
|
||||
for turn := 1; turn <= 50; turn++ {
|
||||
for _, line := range zeromaxingReminders(posture, turn, false) {
|
||||
if line == needle {
|
||||
n++
|
||||
}
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
if got := count(ZeromaxingEntering, ZeromaxingEnterNotice); got != 1 {
|
||||
t.Fatalf("enter fired %d times across 50 turns, want exactly 1", got)
|
||||
}
|
||||
if got := count(ZeromaxingEntering, ZeromaxingBudgetNotice); got != 1 {
|
||||
t.Fatalf("budget notice fired %d times across 50 turns, want exactly 1", got)
|
||||
}
|
||||
if got := count(ZeromaxingExiting, ZeromaxingExitNotice); got != 1 {
|
||||
t.Fatalf("exit fired %d times across 50 turns, want exactly 1", got)
|
||||
}
|
||||
if got := count(ZeromaxingActive, ZeromaxingEnterNotice); got != 0 {
|
||||
t.Fatalf("an already-active posture must never emit the enter notice, got %d", got)
|
||||
}
|
||||
if got := count(ZeromaxingEntering, ZeromaxingStillOnNotice); got != 49 {
|
||||
t.Fatalf("still-on fired on %d of turns 2..50, want 49", got)
|
||||
}
|
||||
}
|
||||
|
||||
// still-on repeats on every continuing turn, so it must be BYTE-IDENTICAL every
|
||||
// time. A turn counter, a remaining-budget number, or a timestamp here would be
|
||||
// invisible today (it rides below the cache breakpoint) and a silent cost bug
|
||||
// the moment anyone moved this text into the system prompt.
|
||||
func TestZeromaxingStillOnNoticeIsInvariant(t *testing.T) {
|
||||
first := zeromaxingReminders(ZeromaxingEntering, 2, false)
|
||||
for turn := 3; turn <= 40; turn++ {
|
||||
if got := zeromaxingReminders(ZeromaxingEntering, turn, false); !reflect.DeepEqual(got, first) {
|
||||
t.Fatalf("still-on drifted at turn %d: %#v vs %#v", turn, got, first)
|
||||
}
|
||||
}
|
||||
if strings.ContainsAny(ZeromaxingStillOnNotice, "0123456789") {
|
||||
t.Fatalf("still-on must carry no digits (no counter, no budget, no timestamp): %q",
|
||||
ZeromaxingStillOnNotice)
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 1 ships no orchestration, so no reminder may imply any. A prompt that
|
||||
// advertises capabilities the run does not have is a prompt-level lie, and the
|
||||
// model will try to use them.
|
||||
func TestZeromaxingRemindersPromiseNoOrchestration(t *testing.T) {
|
||||
all := []string{
|
||||
ZeromaxingEnterNotice, ZeromaxingBudgetNotice,
|
||||
ZeromaxingStillOnNotice, ZeromaxingExitNotice,
|
||||
}
|
||||
forbidden := []string{"orchestrat", "workflow", "fan out", "fan-out", "worker", "sub-agent", "subagent", "delegate", "parallel"}
|
||||
for _, notice := range all {
|
||||
lower := strings.ToLower(notice)
|
||||
for _, word := range forbidden {
|
||||
if strings.Contains(lower, word) {
|
||||
t.Fatalf("Phase 1 has no orchestration; reminder must not mention %q: %q", word, notice)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 1's no-orchestration rule, now CONDITIONAL rather than absolute.
|
||||
//
|
||||
// The rule was never "never mention a tool" — it was "never promise a
|
||||
// capability the run does not have". With the orchestrate tool actually
|
||||
// advertised, naming it is accurate; without it, naming it would be the
|
||||
// prompt-level lie the original test guarded against. Both directions asserted,
|
||||
// because deleting the guard and keeping only the positive case would lose
|
||||
// exactly the protection that mattered.
|
||||
func TestOrchestrateIsNamedOnlyWhenItExists(t *testing.T) {
|
||||
withTool := zeromaxingReminders(ZeromaxingEntering, 1, true)
|
||||
withoutTool := zeromaxingReminders(ZeromaxingEntering, 1, false)
|
||||
|
||||
joined := func(lines []string) string { return strings.Join(lines, "\n") }
|
||||
|
||||
if !strings.Contains(joined(withTool), "orchestrate") {
|
||||
t.Fatalf("with the tool available the enter notice must name it:\n%s", joined(withTool))
|
||||
}
|
||||
if strings.Contains(joined(withoutTool), "orchestrate") {
|
||||
t.Fatalf("without the tool NOTHING may mention it:\n%s", joined(withoutTool))
|
||||
}
|
||||
// The unavailable case keeps Phase 1's original vocabulary guard intact.
|
||||
for _, word := range []string{"orchestrat", "workflow", "fan-out", "worker", "delegate", "parallel"} {
|
||||
if strings.Contains(strings.ToLower(joined(withoutTool)), word) {
|
||||
t.Fatalf("without the tool the reminders must not mention %q:\n%s", word, joined(withoutTool))
|
||||
}
|
||||
}
|
||||
// The notice is one-shot like the rest: it rides the enter turn only.
|
||||
for turn := 2; turn <= 5; turn++ {
|
||||
if strings.Contains(joined(zeromaxingReminders(ZeromaxingEntering, turn, true)), "orchestrate") {
|
||||
t.Fatalf("the orchestrate notice must fire once, not on turn %d", turn)
|
||||
}
|
||||
}
|
||||
// Every OTHER posture state stays silent about it.
|
||||
for _, posture := range []Zeromaxing{ZeromaxingOff, ZeromaxingActive, ZeromaxingExiting} {
|
||||
if strings.Contains(joined(zeromaxingReminders(posture, 1, true)), "orchestrate") {
|
||||
t.Fatalf("posture %v must not name the tool", posture)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// THE EVIDENCE CONTRACT FIRES ONCE PER RUN, on the turn the user's message sits
|
||||
// on — not once per posture change, and not on every tool turn.
|
||||
//
|
||||
// Once per posture change would leave every later run of a long session without
|
||||
// it, and the long sessions are the ones that produce reports. Every tool turn
|
||||
// would pay hundreds of copies to say something that only bears on the
|
||||
// boundaries of a run.
|
||||
func TestTheEvidenceContractFiresOnceOnTheFirstTurnOfEveryRun(t *testing.T) {
|
||||
for _, posture := range []Zeromaxing{ZeromaxingEntering, ZeromaxingActive} {
|
||||
fired := 0
|
||||
for turn := 1; turn <= 50; turn++ {
|
||||
for _, line := range zeromaxingReminders(posture, turn, false) {
|
||||
if line == ZeromaxingEvidenceNotice {
|
||||
fired++
|
||||
if turn != 1 {
|
||||
t.Errorf("posture %v repeated the evidence contract on turn %d; it belongs on turn 1 only", posture, turn)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if fired != 1 {
|
||||
t.Errorf("posture %v fired the evidence contract %d times across 50 turns, want exactly 1", posture, fired)
|
||||
}
|
||||
}
|
||||
|
||||
// A posture that is OFF or on its way out says nothing: a contract about how
|
||||
// to weigh evidence is part of the posture, and claiming it after the posture
|
||||
// ended would be as wrong as the still-on notice would be.
|
||||
for _, posture := range []Zeromaxing{ZeromaxingOff, ZeromaxingExiting} {
|
||||
for turn := 1; turn <= 5; turn++ {
|
||||
for _, line := range zeromaxingReminders(posture, turn, true) {
|
||||
if line == ZeromaxingEvidenceNotice {
|
||||
t.Errorf("posture %v emitted the evidence contract on turn %d", posture, turn)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The contract has to actually carry all three rules it exists for. Asserted on
|
||||
// the substance, not the wording, so a rewrite is free but a DELETION is not:
|
||||
// each of these went missing in a measured run and cost a different thing.
|
||||
func TestTheEvidenceContractCoversAllThreeFailures(t *testing.T) {
|
||||
for _, required := range []string{
|
||||
// A passing test treated as proof of the property.
|
||||
"passing test is not proof",
|
||||
"what it would still pass with",
|
||||
// Numbers reported that no command in the session produced.
|
||||
"must come from a command you ran in this session",
|
||||
"show both",
|
||||
// Folding to whoever asserts a defect most confidently.
|
||||
"prove or refute it from the code before you agree",
|
||||
"even when the claim turns out to be right",
|
||||
} {
|
||||
if !strings.Contains(ZeromaxingEvidenceNotice, required) {
|
||||
t.Errorf("the evidence contract no longer says %q:\n%s", required, ZeromaxingEvidenceNotice)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,57 +0,0 @@
|
||||
package aimlapi
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const (
|
||||
// SourceHeaderName and SourceValue identify Zero as the calling agent on
|
||||
// requests to aimlapi.com services (the Toolbox and the MCP server).
|
||||
SourceHeaderName = "X-AIMLAPI-Source"
|
||||
SourceValue = "agent/zero"
|
||||
|
||||
// CanonicalInferenceBaseURL is the catalog inference endpoint. A provider
|
||||
// profile whose base URL differs is treated as a proxy: its key is never sent
|
||||
// to other aimlapi.com services.
|
||||
CanonicalInferenceBaseURL = "https://api.aimlapi.com/v1"
|
||||
// ToolboxBaseURL is the fixed AI/ML API Toolbox REST base. It is not
|
||||
// configurable, so the key and attribution can only reach tools.aimlapi.com.
|
||||
ToolboxBaseURL = "https://tools.aimlapi.com/v1/tools"
|
||||
// MCPServerURL is the AI/ML API MCP server (streamable HTTP).
|
||||
MCPServerURL = "https://mcp.aimlapi.com/mcp"
|
||||
)
|
||||
|
||||
// attributedHosts are the only hostnames that receive the AI/ML API key or the
|
||||
// attribution headers. Matching is exact; a suffix or substring match would
|
||||
// also accept hosts such as tools.aimlapi.com.example.net.
|
||||
var attributedHosts = [...]string{"api.aimlapi.com", "tools.aimlapi.com", "mcp.aimlapi.com"}
|
||||
|
||||
// IsAttributedURL reports whether target is an https URL on one of the
|
||||
// aimlapi.com service hosts, on the default port and without userinfo.
|
||||
func IsAttributedURL(target *url.URL) bool {
|
||||
if target == nil || !strings.EqualFold(target.Scheme, "https") || target.User != nil {
|
||||
return false
|
||||
}
|
||||
if port := target.Port(); port != "" && port != "443" {
|
||||
return false
|
||||
}
|
||||
host := strings.ToLower(target.Hostname())
|
||||
for _, candidate := range attributedHosts {
|
||||
if host == candidate {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ApplyAttributionHeaders sets the source and partner headers on header only
|
||||
// when target is an aimlapi.com service URL; any other target is left as is.
|
||||
func ApplyAttributionHeaders(header http.Header, target *url.URL) {
|
||||
if header == nil || !IsAttributedURL(target) {
|
||||
return
|
||||
}
|
||||
header.Set(SourceHeaderName, SourceValue)
|
||||
header.Set(PartnerHeaderName, ResolvePartnerID(""))
|
||||
}
|
||||
@@ -1,54 +0,0 @@
|
||||
package aimlapi
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestIsAttributedURLMatchesExactServiceHostsOnly(t *testing.T) {
|
||||
cases := map[string]bool{
|
||||
"https://tools.aimlapi.com/v1/tools/run": true,
|
||||
"https://mcp.aimlapi.com/mcp": true,
|
||||
"https://api.aimlapi.com/v1": true,
|
||||
"https://TOOLS.aimlapi.com:443/v1/tools": true,
|
||||
"http://tools.aimlapi.com/v1/tools/run": false,
|
||||
"https://tools.aimlapi.com:8443/v1/tools": false,
|
||||
"https://[email protected]/v1/tools": false,
|
||||
"https://tools.aimlapi.com.example.net/run": false,
|
||||
"https://eviltools.aimlapi.com/run": false,
|
||||
"https://aimlapi.com/app": false,
|
||||
"https://auth.aimlapi.com/": false,
|
||||
"https://example.com/?h=tools.aimlapi.com": false,
|
||||
}
|
||||
for raw, want := range cases {
|
||||
target, err := url.Parse(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("parse %q: %v", raw, err)
|
||||
}
|
||||
if got := IsAttributedURL(target); got != want {
|
||||
t.Errorf("IsAttributedURL(%q) = %v, want %v", raw, got, want)
|
||||
}
|
||||
}
|
||||
if IsAttributedURL(nil) {
|
||||
t.Fatal("IsAttributedURL(nil) = true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyAttributionHeadersSetsSourceAndPartnerOnServiceHosts(t *testing.T) {
|
||||
t.Setenv("AIMLAPI_PARTNER_ID", "")
|
||||
header := http.Header{}
|
||||
ApplyAttributionHeaders(header, &url.URL{Scheme: "https", Host: "tools.aimlapi.com", Path: "/v1/tools/run"})
|
||||
if got := header.Get(SourceHeaderName); got != "agent/zero" {
|
||||
t.Fatalf("%s = %q, want agent/zero", SourceHeaderName, got)
|
||||
}
|
||||
if got := header.Get(PartnerHeaderName); got != DefaultPartnerID {
|
||||
t.Fatalf("%s = %q, want %q", PartnerHeaderName, got, DefaultPartnerID)
|
||||
}
|
||||
|
||||
other := http.Header{}
|
||||
ApplyAttributionHeaders(other, &url.URL{Scheme: "https", Host: "tools.aimlapi.com.example.net"})
|
||||
if len(other) != 0 {
|
||||
t.Fatalf("attribution reached a non-aimlapi host: %#v", other)
|
||||
}
|
||||
}
|
||||
+12
-347
@@ -17,12 +17,10 @@ import (
|
||||
|
||||
"github.com/Twigpine/zero/internal/agent"
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/execprofile"
|
||||
"github.com/Twigpine/zero/internal/execution"
|
||||
"github.com/Twigpine/zero/internal/hooks"
|
||||
"github.com/Twigpine/zero/internal/localcontrol"
|
||||
"github.com/Twigpine/zero/internal/mcp"
|
||||
"github.com/Twigpine/zero/internal/memory"
|
||||
"github.com/Twigpine/zero/internal/modelregistry"
|
||||
"github.com/Twigpine/zero/internal/observability"
|
||||
"github.com/Twigpine/zero/internal/peermsg"
|
||||
@@ -790,19 +788,6 @@ func runInteractiveTUIWithSetup(stderr io.Writer, deps appDeps, permissionMode a
|
||||
}
|
||||
|
||||
registry := newCoreRegistryScoped(workspaceRoot, scope)
|
||||
// OPT-IN, and the default matters: registering these changes the advertised
|
||||
// tool set for EVERY run, which is exactly what this branch guarantees the
|
||||
// posture does not do when it is off. Off by default keeps that guarantee
|
||||
// true; a user who wants durable notes says so in their own config.
|
||||
if resolved.Profiles.Memory {
|
||||
registry.Register(tools.NewMemoryTool(memory.DefaultPaths(workspaceRoot)))
|
||||
registry.Register(tools.NewMemoryWriteTool(memory.DefaultPaths(workspaceRoot)))
|
||||
// Deletion is its own tool so the approval can disclose it. memory_write's
|
||||
// prompt says only that it saves, and deleting under that sentence is what
|
||||
// made an omitted content field destructive (#897) — registering the save
|
||||
// tool without this one would leave notes undeletable instead.
|
||||
registry.Register(tools.NewMemoryForgetTool(memory.DefaultPaths(workspaceRoot)))
|
||||
}
|
||||
registerLocalControlTools(registry, workspaceRoot, resolved.LocalControl)
|
||||
// Mid-run model escalation is opt-in on this surface too. The tool is present
|
||||
// only when the operator asked for it with --allow-escalation, and the
|
||||
@@ -832,103 +817,11 @@ func runInteractiveTUIWithSetup(stderr io.Writer, deps appDeps, permissionMode a
|
||||
SensitiveEnvKeys: providerSensitiveEnvKeys(resolved),
|
||||
})
|
||||
executionRunner.SetPreparer(sandboxEngine)
|
||||
// One gate shared by the registered tool and the TUI that flips it.
|
||||
zeromaxingGate := &specialist.PostureGate{}
|
||||
// The plan recorder. Registered once with the tool and re-attached to each
|
||||
// run by the model — without it the TUI recorded NONE of the five plan
|
||||
// lifecycle events, so a plan ran completely invisibly (audit finding 9).
|
||||
planProgress := tui.NewPlanProgressBridge()
|
||||
// A CANCELLABLE session root, replacing the context.Background() this used
|
||||
// to hand the TUI.
|
||||
//
|
||||
// Never cancelling it was harmless while nothing outlived a run. A
|
||||
// background plan does, and under an uncancellable root it would keep
|
||||
// running — and keep spending — after the session ended. Close() cancels
|
||||
// AND WAITS, so a plan is stopped and its terminal event written rather
|
||||
// than the process exiting out from under it.
|
||||
sessionCtx, cancelSession := context.WithCancel(context.Background())
|
||||
defer cancelSession()
|
||||
planLaunch := newPlanLauncher(sessionCtx, planProgress)
|
||||
// ITS defer IS REGISTERED LAST, further down, and that is load-bearing —
|
||||
// see the ordering note beside closeSpecialistRuntime.
|
||||
// Saved plans, resolved ONCE and handed to both consumers: the orchestrate
|
||||
// tool (which loads a plan named with `saved`) and the TUI (which saves,
|
||||
// lists and shows them). Two computations of the same pair of directories
|
||||
// would eventually disagree about where a plan lives, and the symptom would
|
||||
// be "I saved it" followed by "no saved plan named that".
|
||||
tuiUserConfigDir, _ := config.UserConfigDir()
|
||||
planPaths := specialist.DefaultPlanPaths(workspaceRoot, tuiUserConfigDir)
|
||||
// nil filters: the TUI has no --enabled-tools/--disabled-tools equivalent,
|
||||
// so the run's grant is every read-only tool the registry holds.
|
||||
specialistRuntime, err := registerSpecialistTools(registry, workspaceRoot, resolved.Swarm.MaxTeamSize, nil, nil, planProgress,
|
||||
orchestrateWiring{
|
||||
DiscoverModels: planModelDiscoverer(workspaceRoot, resolved.Provider),
|
||||
// Sizes each task's dependency briefing to the window of the model
|
||||
// that will READ it. Wired here because only this side knows the
|
||||
// provider profile and the session's own model.
|
||||
ContextWindows: planContextWindows(resolved.Provider, resolved.Provider.Model),
|
||||
// The LIVE scope, not a snapshot of it. A request_permissions grant
|
||||
// lands mid-session and must reach the children dispatched after it.
|
||||
// ExtraRoots, not Roots: this is what the run holds BEYOND its
|
||||
// workspace, which is what the field means and what a child needs.
|
||||
// Roots() also returns the workspace root itself, and for a plan task
|
||||
// running in an isolated worktree that arrived as --add-dir <parent
|
||||
// tree> — re-opening everything --cwd had just narrowed. A measured
|
||||
// run wrote ten times into the user's real tree that way, each write
|
||||
// allowed with the reason "workspace write is allowed", because by
|
||||
// then it genuinely was inside the child's write roots.
|
||||
//
|
||||
// withinTurn selects whether TURN-SCOPED grants may ride along: only
|
||||
// for a child that exits before this turn does (see childWriteRoots).
|
||||
ExtraWriteRoots: func(withinTurn bool) []string { return childWriteRoots(scope, withinTurn) },
|
||||
// The read counterpart: a request_permissions READ grant lands in the
|
||||
// scope's readRoots, which ExtraRoots (write grants) does not return, so
|
||||
// without this a plan auditing a granted path failed "outside the
|
||||
// workspace" in every task. LIVE scope, read at dispatch, same as above,
|
||||
// and handed to the executor too so an ordinary Task and a resume can
|
||||
// read the same grant. Same withinTurn rule as the write roots.
|
||||
ExtraReadRoots: func(withinTurn bool) []string { return childReadRoots(scope, withinTurn) },
|
||||
ProbeModel: planModelProber(workspaceRoot, resolved.Provider, deps.newProvider),
|
||||
ProbeIdentity: planModelProbeIdentity(workspaceRoot, resolved.Provider),
|
||||
ModelPrefs: planModelPreferences(resolved.Profiles.PlanModels),
|
||||
// Off unless the user's own config asks for it: on by default would
|
||||
// refuse a plan for everyone whose phrasing does not happen to match.
|
||||
RequirePlanKeyword: resolved.Profiles.RequirePlanKeyword,
|
||||
Gate: zeromaxingGate,
|
||||
// Depth stays 0: the TUI is always a root session — it has no
|
||||
// --depth and is never launched as a child — so zero is the
|
||||
// measured value here, not an unset one.
|
||||
PlanContext: specialist.PlanTaskContext{
|
||||
Cwd: workspaceRoot, Depth: 0,
|
||||
PostureReasoningEffort: string(execprofile.Zeromaxing.ReasoningEffort),
|
||||
},
|
||||
// The plan-size tier, from the SAME resolved config the rest of this
|
||||
// wiring reads. Project config may only have tightened it.
|
||||
Size: resolved.Profiles.PlanSizeTier(),
|
||||
Plans: planPaths,
|
||||
// The TUI can carry a background plan: it has a session that
|
||||
// outlives a turn. Headless exec supplies no launcher.
|
||||
Launch: planLaunch.Launch,
|
||||
// A write-capable plan gets a worktree of its own, or is refused.
|
||||
Isolate: newPlanIsolator(workspaceRoot),
|
||||
})
|
||||
specialistRuntime, err := registerSpecialistTools(registry, workspaceRoot, resolved.Swarm.MaxTeamSize)
|
||||
if err != nil {
|
||||
return writeAppError(stderr, "failed to initialize specialist tools: "+err.Error(), 1)
|
||||
}
|
||||
// SHUTDOWN ORDER, and defers run LIFO so the registration order here is the
|
||||
// REVERSE of what happens.
|
||||
//
|
||||
// Required: stop the plan, then close the runtime it was using, then cancel
|
||||
// the session. It was the exact opposite. planLaunch.Close() was registered
|
||||
// early (right where the launcher is built, which reads naturally), so it ran
|
||||
// AFTER closeSpecialistRuntime — and Close "cancels AND WAITS", so a
|
||||
// background plan was still being waited on with the specialist runtime it
|
||||
// needs already torn down.
|
||||
//
|
||||
// Registering Close here, last, is what puts it first at shutdown. Anything
|
||||
// added below this line runs BEFORE the plan is stopped; put it above.
|
||||
defer closeSpecialistRuntime(stderr, specialistRuntime)
|
||||
defer planLaunch.Close()
|
||||
// The TUI has no --worktree reassignment, so trustRoot == workspaceRoot here.
|
||||
// Gate the project MCP layer behind the workspace-trust check (fail-closed): an
|
||||
// untrusted workspace must not spawn its ./.zero/config.json stdio MCP servers.
|
||||
@@ -1071,13 +964,12 @@ func runInteractiveTUIWithSetup(stderr io.Writer, deps appDeps, permissionMode a
|
||||
// report can be combined with the plugin activation's, and emit at most one
|
||||
// notice when project hooks/plugins were dropped for an untrusted workspace.
|
||||
hookDispatcher, hookSkip := newHookDispatcherWithExtra(workspaceRoot, pluginActivation.hooks, trustRoot, executionRunner)
|
||||
specialistRuntime.attachLifecycleHooks(hookDispatcher)
|
||||
emitTrustNotice(stderr, hookSkip, pluginActivation.trustSkip, mcpSkip)
|
||||
// From the engine the session's commands run through, so the notice
|
||||
// describes the sandbox they actually get.
|
||||
startupNotices := []string{sandboxEngine.DegradedNotice()}
|
||||
modelRPM := zeroruntime.NewModelRPMLimiter(resolved.ModelRPM)
|
||||
return deps.runTUI(sessionCtx, tui.Options{
|
||||
return deps.runTUI(context.Background(), tui.Options{
|
||||
ModelRPM: modelRPM,
|
||||
Cwd: workspaceRoot,
|
||||
Version: version,
|
||||
@@ -1094,7 +986,6 @@ func runInteractiveTUIWithSetup(stderr io.Writer, deps appDeps, permissionMode a
|
||||
FavoriteModels: resolved.Preferences.FavoriteModels,
|
||||
RecentModels: resolved.Preferences.RecentModels,
|
||||
RecapsEnabled: resolved.Preferences.RecapsEnabled(),
|
||||
KeepFinishedAgents: resolved.Preferences.KeepsFinishedAgents(),
|
||||
CompactionModel: resolved.Preferences.CompactionModel,
|
||||
Provider: provider,
|
||||
NewProvider: deps.newProvider,
|
||||
@@ -1121,10 +1012,6 @@ func runInteractiveTUIWithSetup(stderr io.Writer, deps appDeps, permissionMode a
|
||||
PeerService: peerService,
|
||||
SandboxStore: sandboxStore,
|
||||
MCPConfig: mcpConfig,
|
||||
ZeromaxingDisabled: resolved.Profiles.DisableZeromaxing,
|
||||
ZeromaxingGate: zeromaxingGate,
|
||||
PlanProgress: planProgress,
|
||||
PlanPaths: planPaths,
|
||||
MCPPermissionStore: mcpPermissionStore,
|
||||
MCPTokenStore: mcpTokenStore,
|
||||
MCPCommand: func(ctx context.Context, args []string) tui.MCPCommandResult {
|
||||
@@ -1150,22 +1037,13 @@ func runInteractiveTUIWithSetup(stderr io.Writer, deps appDeps, permissionMode a
|
||||
MaxTurns: resolved.MaxTurns,
|
||||
Registry: registry,
|
||||
PermissionMode: permissionMode,
|
||||
// Set ONCE, here, rather than beside each of the three places the
|
||||
// TUI assigns Zeromaxing: the registry is built per session and does
|
||||
// not change per run, and three assignments of one fact is three
|
||||
// chances for the next one to be forgotten — which is precisely how
|
||||
// this field came to be set nowhere at all.
|
||||
OrchestrateAvailable: orchestrateAvailable(registry),
|
||||
// Background plans report into a later turn through this drain, on
|
||||
// the same channel as the post-edit diagnostics nudge.
|
||||
PlanCompletions: planProgress.DrainCompletedPlans,
|
||||
Autonomy: "low",
|
||||
Sandbox: sandboxEngine,
|
||||
FileTracker: fileTracker,
|
||||
Hooks: hookDispatcher,
|
||||
DeferThreshold: resolved.Tools.DeferThreshold,
|
||||
Specialists: specialistRuntime.specialists,
|
||||
Skills: pluginActivation.skillInfos(deps.skillsDir()),
|
||||
Autonomy: "low",
|
||||
Sandbox: sandboxEngine,
|
||||
FileTracker: fileTracker,
|
||||
Hooks: hookDispatcher,
|
||||
DeferThreshold: resolved.Tools.DeferThreshold,
|
||||
Specialists: specialistRuntime.specialists,
|
||||
Skills: pluginActivation.skillInfos(deps.skillsDir()),
|
||||
},
|
||||
AllowEscalation: allowEscalation,
|
||||
// LoadSkills backs /skills and direct /<skill-name> invocation in the TUI.
|
||||
@@ -1343,9 +1221,6 @@ type agentToolRuntime struct {
|
||||
// the orchestrator's system-prompt delegation section. Populated alongside the
|
||||
// tools, so it is present exactly when the Task tool is.
|
||||
specialists []agent.SpecialistInfo
|
||||
// lifecycleHooks is shared with the specialist executor so the hooks
|
||||
// dispatcher (built later in startup) can be attached before any specialist runs.
|
||||
lifecycleHooks *specialist.LifecycleHooks
|
||||
}
|
||||
|
||||
// specialistInfos returns the runtime's specialist summaries, nil-safe so the
|
||||
@@ -1357,164 +1232,12 @@ func (r *agentToolRuntime) specialistInfos() []agent.SpecialistInfo {
|
||||
return r.specialists
|
||||
}
|
||||
|
||||
func (r *agentToolRuntime) attachLifecycleHooks(dispatcher *hooks.Dispatcher) {
|
||||
if r == nil || r.lifecycleHooks == nil || dispatcher == nil {
|
||||
return
|
||||
}
|
||||
r.lifecycleHooks.Dispatch = func(ctx context.Context, event string, specialistName string, payload map[string]any) {
|
||||
dispatcher.Dispatch(ctx, hooks.DispatchInput{
|
||||
Event: hooks.Event(event),
|
||||
ToolName: specialistName,
|
||||
Payload: payload,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// orchestrateWiring carries what registerSpecialistTools needs to wire the
|
||||
// plan tool. A zero value leaves the tool registered but permanently off, which
|
||||
// is the posture-off behaviour every existing caller already gets.
|
||||
type orchestrateWiring struct {
|
||||
// RequirePlanKeyword carries Profiles.RequirePlanKeyword to the tool.
|
||||
RequirePlanKeyword bool
|
||||
// Gate is the shared posture flag. A POINTER, not a closure over caller
|
||||
// state: the TUI model is a value type copied on every update, so a closure
|
||||
// would freeze the posture as it was at registration.
|
||||
Gate *specialist.PostureGate
|
||||
// PlanContext supplies the run-invariant state a plan task inherits.
|
||||
PlanContext specialist.PlanTaskContext
|
||||
// ContextWindows sizes a task's dependency briefing to the model that will
|
||||
// read it. nil keeps the fixed caps.
|
||||
ContextWindows specialist.ContextWindowFunc
|
||||
// ExtraWriteRoots reports the run's non-workspace roots at LAUNCH time, so a
|
||||
// child's sandbox covers the same ground its parent's does. nil means the
|
||||
// workspace only, which is what every child got before this existed.
|
||||
// withinTurn: see specialist.Executor.ExtraWriteRoots.
|
||||
ExtraWriteRoots func(withinTurn bool) []string
|
||||
// ExtraReadRoots reports the paths the run may READ beyond its workspace — its
|
||||
// request_permissions grants — at DISPATCH time, so a plan's tasks can read a
|
||||
// granted external path instead of failing "outside the workspace". The read
|
||||
// counterpart of ExtraWriteRoots: a read grant lands in a separate scope list
|
||||
// that ExtraWriteRoots does not cover. nil means workspace-only reads.
|
||||
// withinTurn: see specialist.Executor.ExtraReadRoots. It reaches both the
|
||||
// executor (every Task, resume and plan task) and the orchestrate tool.
|
||||
ExtraReadRoots func(withinTurn bool) []string
|
||||
// ProbeModel proves a model will actually run before any task is assigned it.
|
||||
// nil skips proving, which is what every plan did before this existed.
|
||||
ProbeModel specialist.ModelProber
|
||||
// ProbeIdentity namespaces cached verdicts by the live provider/profile and
|
||||
// endpoint. nil uses the specialist package's private default namespace.
|
||||
ProbeIdentity func() string
|
||||
// Size is the configured plan-size tier. The zero value is the default tier,
|
||||
// so a call site that has no resolved config yet still gets a real ceiling
|
||||
// rather than none.
|
||||
Size config.PlanSize
|
||||
// Plans locates saved plans. Empty means a `saved` reference is refused with
|
||||
// a reason rather than searched for in nowhere.
|
||||
Plans specialist.PlanPaths
|
||||
// Isolate prepares a worktree for a write-capable plan. nil refuses one.
|
||||
Isolate specialist.PlanIsolator
|
||||
// Launch runs a plan in the background. nil — the headless default — makes
|
||||
// a background plan refuse rather than start one nothing can report.
|
||||
Launch func(run func(ctx context.Context)) bool
|
||||
// DiscoverModels lists what the active provider can serve, for auto_assign.
|
||||
// nil makes a plan asking for it refuse with a reason.
|
||||
DiscoverModels specialist.ModelDiscoverer
|
||||
// ModelPrefs carries the user's per-role model pins and exclusions.
|
||||
ModelPrefs specialist.ModelPreferences
|
||||
}
|
||||
|
||||
// planParentTools is the run's grant: the tools a plan task may inherit.
|
||||
//
|
||||
// Derived, never hand-written. The candidate set is specialist's own exported
|
||||
// list — nothing outside it can ever be granted — narrowed to what this
|
||||
// registry actually holds and what the run's operator filters allow. Keeping
|
||||
// one authoritative list rather than a second copy here is invariant 5; the
|
||||
// previous field was a hand-supplied []string that BOTH production call sites
|
||||
// left nil, which silently disabled the narrowing rule entirely.
|
||||
func planParentTools(registry *tools.Registry, enabledTools, disabledTools []string) []string {
|
||||
grant := []string{}
|
||||
// EVERY GRANTABLE NAME, read-only and write alike. The write tools are not
|
||||
// thereby granted to anything: a task inherits only the read-only ones
|
||||
// (planToolGrant) and must NAME a write tool to hold it. Narrowing this list
|
||||
// to read-only would instead make a named write tool undeliverable — the
|
||||
// task would validate and then run with less than it asked for.
|
||||
for _, name := range specialist.PlanGrantableToolNames() {
|
||||
if _, found := registry.Get(name); !found {
|
||||
continue
|
||||
}
|
||||
if !agent.ToolAllowedByFilters(name, enabledTools, disabledTools) {
|
||||
continue
|
||||
}
|
||||
grant = append(grant, name)
|
||||
}
|
||||
return grant
|
||||
}
|
||||
|
||||
// wireChildRoots gives the executor what every child it launches (Task, resume,
|
||||
// swarm member, plan task) may inherit from this run's sandbox scope.
|
||||
func wireChildRoots(executor *specialist.Executor, workspaceRoot string, wiring orchestrateWiring) {
|
||||
// The run's own roots, read at every launch. Without this a child is
|
||||
// confined more tightly than its parent: a run granted a directory
|
||||
// outside its workspace could create and populate it, then watch every
|
||||
// plan task be refused at that same directory.
|
||||
executor.ExtraWriteRoots = wiring.ExtraWriteRoots
|
||||
// The parent's own tree, so a child isolated in a worktree is never handed
|
||||
// a write root that overlaps it (an ancestor --add-dir, say).
|
||||
executor.WorkspaceRoot = workspaceRoot
|
||||
// The run's READ grants, for every child and not only plan tasks: an
|
||||
// ordinary Task or a resume must be able to read what the parent was just
|
||||
// granted to read. Emitted by the executor as --add-read-dir only.
|
||||
executor.ExtraReadRoots = wiring.ExtraReadRoots
|
||||
}
|
||||
|
||||
// orchestrateReadRoots adapts the run's read supplier to the orchestrate tool,
|
||||
// which snapshots it when a plan starts. The plan tool has always passed every
|
||||
// live read grant, turn-scoped ones included, and that is unchanged here.
|
||||
func orchestrateReadRoots(supplier func(withinTurn bool) []string) func() []string {
|
||||
if supplier == nil {
|
||||
return nil
|
||||
}
|
||||
return func() []string { return supplier(true) }
|
||||
}
|
||||
|
||||
// registerSpecialistTools registers the specialist, swarm and orchestrate tools.
|
||||
// The wiring argument carries what the orchestrate tool needs; a zero value
|
||||
// leaves it registered but permanently off, which is the posture-off behaviour.
|
||||
//
|
||||
// enabledTools/disabledTools are the run's operator filters, and recorder is the
|
||||
// plan lifecycle sink. All three are explicit PARAMETERS rather than wiring
|
||||
// fields on purpose: a call site cannot forget them without failing to compile.
|
||||
// As a field, Recorder was simply never set by the TUI, so a plan there recorded
|
||||
// none of its five lifecycle events (finding 9) — the same omission the parent
|
||||
// grant suffered. An explicit nil is a stated choice; an absent field is not.
|
||||
func registerSpecialistTools(registry *tools.Registry, workspaceRoot string, maxTeamSize int, enabledTools, disabledTools []string, recorder specialist.PlanRecorder, wiring orchestrateWiring) (*agentToolRuntime, error) {
|
||||
func registerSpecialistTools(registry *tools.Registry, workspaceRoot string, maxTeamSize int) (*agentToolRuntime, error) {
|
||||
paths, err := specialist.DefaultPaths(workspaceRoot)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
lifecycleHooks := &specialist.LifecycleHooks{}
|
||||
// One budget for the whole session, shared by every run and every plan in it
|
||||
// — which is the point: a per-run counter would reset on each message and
|
||||
// bound nothing a conversation does over time.
|
||||
sessionBudget := specialist.NewSessionBudget(specialist.DefaultSessionSubagents)
|
||||
executor := specialist.Executor{
|
||||
SessionBudget: sessionBudget,
|
||||
Paths: paths,
|
||||
// Dispatches specialistStart/specialistStop hooks once the session's
|
||||
// hook dispatcher is attached (attachLifecycleHooks).
|
||||
LifecycleHooks: lifecycleHooks,
|
||||
// So a delegated Task that names no model is routed to a role-appropriate
|
||||
// pin under the zeromaxing posture, the same way a plan task is. Off unless
|
||||
// the posture is active and auto-assign is configured. The discoverer is
|
||||
// the same one the plan tool uses: a pin fires only when the provider's
|
||||
// own listing carries it, so a stale pin after a provider switch degrades
|
||||
// to inherit instead of a child dead at spawn.
|
||||
ModelPrefs: wiring.ModelPrefs,
|
||||
PostureActive: wiring.Gate.Active,
|
||||
DiscoverModels: wiring.DiscoverModels,
|
||||
ServeCache: &specialist.ModelServeCache{},
|
||||
}
|
||||
wireChildRoots(&executor, workspaceRoot, wiring)
|
||||
executor := specialist.Executor{Paths: paths}
|
||||
runtime, err := specialist.RegisterTools(registry, executor)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -1533,47 +1256,7 @@ func registerSpecialistTools(registry *tools.Registry, workspaceRoot string, max
|
||||
return nil, err
|
||||
}
|
||||
swarm.RegisterTools(registry, sw)
|
||||
// The orchestrate tool. Registered ALWAYS and gated by Safety(): with the
|
||||
// posture off it reports PermissionDeny, so it is never advertised and the
|
||||
// prefix is byte-identical to a build without it. Registering conditionally
|
||||
// would not work — the TUI builds its registry once per session and clones
|
||||
// tool POINTERS per run, so a tool added on a posture flip would never
|
||||
// reach a run already holding a clone.
|
||||
planContext := wiring.PlanContext
|
||||
planContext.Executor = executor
|
||||
if strings.TrimSpace(planContext.Cwd) == "" {
|
||||
planContext.Cwd = workspaceRoot
|
||||
}
|
||||
if strings.TrimSpace(planContext.SpecialistName) == "" {
|
||||
planContext.SpecialistName = "explorer"
|
||||
}
|
||||
registry.Register(&specialist.OrchestrateTool{
|
||||
PostureActive: wiring.Gate.Active,
|
||||
RunTask: specialist.NewPlanRunner(planContext),
|
||||
Recorder: recorder,
|
||||
ParentTools: planParentTools(registry, enabledTools, disabledTools),
|
||||
Depth: planContext.Depth,
|
||||
Size: wiring.Size,
|
||||
// Refuse a plan the user's own turn did not ask for, when this session
|
||||
// asks to be protected that way. Off unless configured — see
|
||||
// ProfilesConfig.RequirePlanKeyword.
|
||||
RequirePlanKeyword: wiring.RequirePlanKeyword,
|
||||
Plans: wiring.Plans,
|
||||
Launch: wiring.Launch,
|
||||
Isolate: wiring.Isolate,
|
||||
DiscoverModels: wiring.DiscoverModels,
|
||||
ContextWindows: wiring.ContextWindows,
|
||||
ExtraReadRoots: orchestrateReadRoots(wiring.ExtraReadRoots),
|
||||
ProbeModel: wiring.ProbeModel,
|
||||
ProbeIdentity: wiring.ProbeIdentity,
|
||||
ModelPrefs: wiring.ModelPrefs,
|
||||
})
|
||||
return &agentToolRuntime{
|
||||
specialist: runtime,
|
||||
swarm: sw,
|
||||
specialists: specialistSummaries(paths),
|
||||
lifecycleHooks: lifecycleHooks,
|
||||
}, nil
|
||||
return &agentToolRuntime{specialist: runtime, swarm: sw, specialists: specialistSummaries(paths)}, nil
|
||||
}
|
||||
|
||||
// specialistSummaries loads the available specialists (built-ins + user/project
|
||||
@@ -2007,21 +1690,3 @@ func cachedSkillsLoader(load func() []skills.Skill) func() []skills.Skill {
|
||||
return cached
|
||||
}
|
||||
}
|
||||
|
||||
// orchestrateAvailable reports whether this run actually holds the orchestrate
|
||||
// tool, for the reminder that names it.
|
||||
//
|
||||
// DERIVED FROM THE REGISTRY, never hand-set. The field it feeds was declared,
|
||||
// documented and consumed by the reminder selector — and set by NEITHER
|
||||
// production call site, so the notice that tells the model the tool exists
|
||||
// could not fire. The model was handed an advertised tool it was never told
|
||||
// about, and did not use it. That is invariant 1 for the second time in this
|
||||
// feature, and a bool a caller has to remember to pass is how it happened; the
|
||||
// registry is the thing that actually knows.
|
||||
func orchestrateAvailable(registry *tools.Registry) bool {
|
||||
if registry == nil {
|
||||
return false
|
||||
}
|
||||
_, found := registry.Get(specialist.OrchestrateToolName)
|
||||
return found
|
||||
}
|
||||
@@ -1,51 +0,0 @@
|
||||
package cli
|
||||
|
||||
import "testing"
|
||||
|
||||
// TWO DEFINITIONS OF ONE NUMBER, and this is the only defence against them
|
||||
// drifting apart.
|
||||
//
|
||||
// internal/specialist decides whether to retry a task from the child's exit
|
||||
// code — a structural signal rather than its prose. It cannot import this
|
||||
// package to read exitIncomplete, because cli imports specialist and the
|
||||
// dependency cannot run both ways, so the constant is duplicated there.
|
||||
//
|
||||
// If this file's exitIncomplete ever moves and specialist's copy does not, a
|
||||
// declined task stops being recognised as declined and silently loses its one
|
||||
// retry — with nothing failing anywhere. That is the exact shape of defect this
|
||||
// session has produced repeatedly: a value that exists at one layer, is consumed
|
||||
// at another, and quietly stops agreeing.
|
||||
func TestTheChildIncompleteExitCodeAgreesWithWhatSpecialistExpects(t *testing.T) {
|
||||
// Kept as a literal on purpose. Reading specialist's unexported constant is
|
||||
// impossible from here, so this asserts the NUMBER both sides were written
|
||||
// against; changing one without the other now fails.
|
||||
const specialistChildExitIncomplete = 4
|
||||
if exitIncomplete != specialistChildExitIncomplete {
|
||||
t.Fatalf("exitIncomplete is %d but internal/specialist retries declines on %d — "+
|
||||
"a declined plan task will no longer be recognised, and will silently lose its retry",
|
||||
exitIncomplete, specialistChildExitIncomplete)
|
||||
}
|
||||
}
|
||||
|
||||
// THE SAME HAZARD, for the same reason, on the provider exit.
|
||||
//
|
||||
// specialist retries a plan task ONCE when the child died on the provider
|
||||
// rather than on the work, and it recognises that from this exit code. If
|
||||
// exitProvider moves and specialist's childExitProvider does not, a provider
|
||||
// failure stops being recognised, the task silently loses its retry, and a
|
||||
// transient 500 goes back to costing the task and everything downstream of it —
|
||||
// with nothing failing anywhere to say so.
|
||||
func TestTheChildProviderExitCodeAgreesWithWhatSpecialistExpects(t *testing.T) {
|
||||
const specialistChildExitProvider = 3
|
||||
if exitProvider != specialistChildExitProvider {
|
||||
t.Fatalf("exitProvider is %d but internal/specialist retries provider failures on %d — "+
|
||||
"a task killed by a provider error will no longer be retried",
|
||||
exitProvider, specialistChildExitProvider)
|
||||
}
|
||||
// And the two codes must stay DISTINCT: one number meaning both would make
|
||||
// every provider failure look like a decline (or the reverse), and the two
|
||||
// retries have different safety rules — the provider one refuses write tasks.
|
||||
if exitProvider == exitIncomplete {
|
||||
t.Fatal("exitProvider and exitIncomplete collided; the two retries would be indistinguishable")
|
||||
}
|
||||
}
|
||||
@@ -1,163 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
)
|
||||
|
||||
// childRootsFixture is the shape a write-capable plan run takes on disk: the
|
||||
// parent workspace below an ancestor the run was granted (--add-dir ~/dev), and
|
||||
// an isolated worktree somewhere else entirely.
|
||||
func childRootsFixture(t *testing.T) (ancestor, workspace, worktree string) {
|
||||
t.Helper()
|
||||
base, err := filepath.EvalSymlinks(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ancestor = filepath.Join(base, "dev")
|
||||
workspace = filepath.Join(ancestor, "zero")
|
||||
worktree = filepath.Join(base, "state", "worktrees", "plan-a")
|
||||
for _, dir := range []string{workspace, worktree} {
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return ancestor, workspace, worktree
|
||||
}
|
||||
|
||||
func flagValues(args []string, flag string) []string {
|
||||
var out []string
|
||||
for index, arg := range args {
|
||||
if arg == flag && index+1 < len(args) {
|
||||
out = append(out, args[index+1])
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func containsValue(values []string, want string) bool {
|
||||
for _, value := range values {
|
||||
if value == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// THE EXECUTOR THE CLI BUILDS must know the parent's workspace, or the
|
||||
// isolation filter in the argument builder has nothing to compare against and
|
||||
// an ancestor write root follows an isolated child into its worktree.
|
||||
func TestTheCLIExecutorWithholdsAnAncestorRootFromAnIsolatedChild(t *testing.T) {
|
||||
ancestor, workspace, worktree := childRootsFixture(t)
|
||||
scope, err := sandbox.NewScope(workspace, []string{ancestor})
|
||||
if err != nil {
|
||||
t.Fatalf("NewScope: %v", err)
|
||||
}
|
||||
executor := specialist.Executor{
|
||||
NewSessionID: func() (string, error) { return "specialist_00000000000000000000000a", nil },
|
||||
}
|
||||
wireChildRoots(&executor, workspace, orchestrateWiring{ExtraWriteRoots: func(withinTurn bool) []string { return childWriteRoots(scope, withinTurn) }})
|
||||
|
||||
isolated, err := executor.BuildArgs(specialist.BuildArgsInput{
|
||||
Manifest: specialist.Manifest{Metadata: specialist.Metadata{Name: "explorer"}},
|
||||
Prompt: "p",
|
||||
Cwd: worktree,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildArgs: %v", err)
|
||||
}
|
||||
if added := flagValues(isolated.Args, "--add-dir"); containsValue(added, ancestor) {
|
||||
t.Fatalf("the isolated child was handed the ancestor %q of the parent workspace as a write root: --add-dir %v", ancestor, added)
|
||||
}
|
||||
|
||||
// The same run's ordinary child, in the parent's own workspace, keeps it.
|
||||
ordinary, err := executor.BuildArgs(specialist.BuildArgsInput{
|
||||
Manifest: specialist.Manifest{Metadata: specialist.Metadata{Name: "explorer"}},
|
||||
Prompt: "p",
|
||||
Cwd: workspace,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildArgs: %v", err)
|
||||
}
|
||||
if added := flagValues(ordinary.Args, "--add-dir"); !containsValue(added, ancestor) {
|
||||
t.Fatalf("an ordinary child lost the run's ancestor root: --add-dir %v", added)
|
||||
}
|
||||
}
|
||||
|
||||
// THE CLI EXECUTOR CARRIES THE RUN'S READ GRANTS to ordinary and resumed
|
||||
// children, read-only. The live read supplier was wired into the orchestrate
|
||||
// tool only, so a Task or a resume could not read what the parent was just
|
||||
// granted to read.
|
||||
func TestTheCLIExecutorPassesReadGrantsReadOnlyToTaskAndResume(t *testing.T) {
|
||||
workspace, err := filepath.EvalSymlinks(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
scope, err := sandbox.NewScope(workspace, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("NewScope: %v", err)
|
||||
}
|
||||
// Outside the temp roots NewScope seeds as permanent WRITE roots, which
|
||||
// would cover a t.TempDir() grant and record nothing.
|
||||
readRoot, err := scope.AddRead(tempDirOutsideDefaultTemp(t))
|
||||
if err != nil {
|
||||
t.Fatalf("AddRead: %v", err)
|
||||
}
|
||||
executor := specialist.Executor{
|
||||
NewSessionID: func() (string, error) { return "specialist_00000000000000000000000a", nil },
|
||||
}
|
||||
wireChildRoots(&executor, workspace, orchestrateWiring{
|
||||
ExtraWriteRoots: func(withinTurn bool) []string { return childWriteRoots(scope, withinTurn) },
|
||||
ExtraReadRoots: func(withinTurn bool) []string { return childReadRoots(scope, withinTurn) },
|
||||
})
|
||||
manifest := specialist.Manifest{Metadata: specialist.Metadata{Name: "explorer"}}
|
||||
fresh, err := executor.BuildArgs(specialist.BuildArgsInput{Manifest: manifest, Prompt: "p", Cwd: workspace, WithinTurn: true})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildArgs: %v", err)
|
||||
}
|
||||
resumed, err := executor.BuildResumeArgs(specialist.BuildResumeArgsInput{
|
||||
SessionID: "specialist_00000000000000000000000a", Manifest: manifest, Prompt: "p", Cwd: workspace, WithinTurn: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildResumeArgs: %v", err)
|
||||
}
|
||||
for label, args := range map[string][]string{"fresh": fresh.Args, "resume": resumed.Args} {
|
||||
if !containsValue(flagValues(args, "--add-read-dir"), readRoot) {
|
||||
t.Errorf("%s: the read grant %q never reached the child as --add-read-dir: %v", label, readRoot, args)
|
||||
}
|
||||
if containsValue(flagValues(args, "--add-dir"), readRoot) {
|
||||
t.Errorf("%s: the read grant %q was emitted as --add-dir — a read grant escalated to write", label, readRoot)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// childReadRoots applies the write side's lifetime rule: a turn-scoped read
|
||||
// grant only for a child that ends within the turn.
|
||||
func TestChildReadRootsWithholdATurnGrantFromAChildThatOutlivesTheTurn(t *testing.T) {
|
||||
workspace, err := filepath.EvalSymlinks(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
scope, err := sandbox.NewScope(workspace, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("NewScope: %v", err)
|
||||
}
|
||||
root, release, err := scope.AddTemporaryRead(tempDirOutsideDefaultTemp(t))
|
||||
if err != nil {
|
||||
t.Fatalf("AddTemporaryRead: %v", err)
|
||||
}
|
||||
defer release()
|
||||
if !containsValue(childReadRoots(scope, true), root) {
|
||||
t.Fatalf("a child inside the turn lost the turn's read grant: %v", childReadRoots(scope, true))
|
||||
}
|
||||
if containsValue(childReadRoots(scope, false), root) {
|
||||
t.Fatalf("a child that may outlive the turn was handed the turn-scoped read grant %q", root)
|
||||
}
|
||||
if childReadRoots(nil, true) != nil {
|
||||
t.Fatal("a nil scope must yield no roots")
|
||||
}
|
||||
}
|
||||
@@ -92,25 +92,13 @@ func runSkillAdd(args []string, dir string, stdout io.Writer, stderr io.Writer)
|
||||
lines := []string{}
|
||||
if result.Updated {
|
||||
lines = append(lines, fmt.Sprintf("Updated skill %q.", result.Name))
|
||||
switch {
|
||||
case result.PreviousHash != result.Hash:
|
||||
lines = append(lines, fmt.Sprintf(" hash: %s -> %s", shortHash(result.PreviousHash), shortHash(result.Hash)))
|
||||
case result.PreviousTreeHash == "":
|
||||
// The install this replaced predates tree hashes and copied SKILL.md
|
||||
// alone, so the bundled files are new rather than changed.
|
||||
lines = append(lines, " SKILL.md is unchanged; the files bundled with it are now installed too")
|
||||
default:
|
||||
lines = append(lines, " SKILL.md is unchanged; the files bundled with it changed")
|
||||
}
|
||||
lines = append(lines, fmt.Sprintf(" hash: %s -> %s", shortHash(result.PreviousHash), shortHash(result.Hash)))
|
||||
} else {
|
||||
lines = append(lines, fmt.Sprintf("Installed skill %q.", result.Name))
|
||||
lines = append(lines, " hash: "+result.Hash)
|
||||
}
|
||||
lines = append(lines, " source: "+result.Source)
|
||||
lines = append(lines, " path: "+result.Path)
|
||||
if len(result.SkippedLinks) > 0 {
|
||||
lines = append(lines, " not copied (symlinks): "+strings.Join(result.SkippedLinks, ", "))
|
||||
}
|
||||
if _, err := fmt.Fprintln(stdout, redaction.RedactString(strings.Join(lines, "\n"), redaction.Options{})); err != nil {
|
||||
return exitCrash
|
||||
}
|
||||
|
||||
@@ -7,8 +7,6 @@ import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/skills"
|
||||
)
|
||||
|
||||
func writeSourceSkillDir(t *testing.T, dir string, content string) string {
|
||||
@@ -393,133 +391,3 @@ func TestRunDistributionCommandsDoNotLaunchTUI(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A reinstall that only changed a bundled script must still read as an update
|
||||
// (#584): the SKILL.md hash is the same, so "hash: X -> X" would say nothing.
|
||||
func TestRunSkillAddReportsAnUpdateToBundledFilesOnly(t *testing.T) {
|
||||
skillsDir := t.TempDir()
|
||||
src := writeSourceSkillDir(t, filepath.Join(t.TempDir(), "src"),
|
||||
"---\nname: pdf\ndescription: Fill forms.\n---\nRun scripts/fill.py.\n")
|
||||
script := filepath.Join(src, "scripts", "fill.py")
|
||||
if err := os.MkdirAll(filepath.Dir(script), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(script, []byte("print('v1')\n"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deps := appDeps{skillsDir: func() string { return skillsDir }}
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
if exit := runWithDeps([]string{"skill", "add", src}, &stdout, &stderr, deps); exit != 0 {
|
||||
t.Fatalf("first skill add exit = %d, stderr = %s", exit, stderr.String())
|
||||
}
|
||||
if err := os.WriteFile(script, []byte("print('v2')\n"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
if exit := runWithDeps([]string{"skill", "add", src}, &stdout, &stderr, deps); exit != 0 {
|
||||
t.Fatalf("second skill add exit = %d, stderr = %s", exit, stderr.String())
|
||||
}
|
||||
for _, want := range []string{`Updated skill "pdf".`, "SKILL.md is unchanged; the files bundled with it changed"} {
|
||||
if !strings.Contains(stdout.String(), want) {
|
||||
t.Fatalf("second add output does not say %q:\n%s", want, stdout.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An install from before tree hashes copied SKILL.md alone and recorded no tree
|
||||
// hash. Adding the skill again over that entry must not claim bundled files
|
||||
// changed: there are none, or they are installed for the first time.
|
||||
func TestRunSkillAddOverAnEntryWithoutATreeHash(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
script bool
|
||||
want []string
|
||||
notWant []string
|
||||
}{
|
||||
{
|
||||
name: "SKILL.md only",
|
||||
want: []string{`Installed skill "pdf".`},
|
||||
notWant: []string{"Updated skill", "files bundled with it"},
|
||||
},
|
||||
{
|
||||
name: "with a bundled script",
|
||||
script: true,
|
||||
want: []string{`Updated skill "pdf".`, "SKILL.md is unchanged; the files bundled with it are now installed too"},
|
||||
notWant: []string{"files bundled with it changed"},
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
skillsDir := t.TempDir()
|
||||
src := writeSourceSkillDir(t, filepath.Join(t.TempDir(), "src"),
|
||||
"---\nname: pdf\ndescription: Fill forms.\n---\nRun scripts/fill.py.\n")
|
||||
if tc.script {
|
||||
script := filepath.Join(src, "scripts", "fill.py")
|
||||
if err := os.MkdirAll(filepath.Dir(script), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(script, []byte("print('fill')\n"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
deps := appDeps{skillsDir: func() string { return skillsDir }}
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
if exit := runWithDeps([]string{"skill", "add", src}, &stdout, &stderr, deps); exit != 0 {
|
||||
t.Fatalf("first skill add exit = %d, stderr = %s", exit, stderr.String())
|
||||
}
|
||||
lock, err := skills.ReadLock(skillsDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entry := lock["pdf"]
|
||||
entry.TreeHash = ""
|
||||
lock["pdf"] = entry
|
||||
data, err := json.Marshal(lock)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(skillsDir, skills.LockFileName), data, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
if exit := runWithDeps([]string{"skill", "add", src}, &stdout, &stderr, deps); exit != 0 {
|
||||
t.Fatalf("second skill add exit = %d, stderr = %s", exit, stderr.String())
|
||||
}
|
||||
for _, want := range tc.want {
|
||||
if !strings.Contains(stdout.String(), want) {
|
||||
t.Fatalf("second add output does not say %q:\n%s", want, stdout.String())
|
||||
}
|
||||
}
|
||||
for _, notWant := range tc.notWant {
|
||||
if strings.Contains(stdout.String(), notWant) {
|
||||
t.Fatalf("second add output says %q:\n%s", notWant, stdout.String())
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunSkillAddListsTheSymlinksItDidNotCopy(t *testing.T) {
|
||||
skillsDir := t.TempDir()
|
||||
src := writeSourceSkillDir(t, filepath.Join(t.TempDir(), "src"),
|
||||
"---\nname: pdf\ndescription: Fill forms.\n---\nbody\n")
|
||||
outside := filepath.Join(t.TempDir(), "notes.md")
|
||||
if err := os.WriteFile(outside, []byte("outside\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink(outside, filepath.Join(src, "notes.md")); err != nil {
|
||||
t.Skipf("symlinks unavailable: %v", err)
|
||||
}
|
||||
var stdout, stderr bytes.Buffer
|
||||
deps := appDeps{skillsDir: func() string { return skillsDir }}
|
||||
if exit := runWithDeps([]string{"skill", "add", src}, &stdout, &stderr, deps); exit != 0 {
|
||||
t.Fatalf("skill add exit = %d, stderr = %s", exit, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "not copied (symlinks): notes.md") {
|
||||
t.Fatalf("add output does not list the skipped symlink:\n%s", stdout.String())
|
||||
}
|
||||
}
|
||||
+5
-275
@@ -24,7 +24,6 @@ import (
|
||||
"github.com/Twigpine/zero/internal/providers"
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
"github.com/Twigpine/zero/internal/sessions"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
"github.com/Twigpine/zero/internal/specmode"
|
||||
"github.com/Twigpine/zero/internal/streamjson"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
@@ -151,11 +150,6 @@ type execOptions struct {
|
||||
// additional write roots for this run. Unioned with
|
||||
// config.SandboxConfig.AdditionalWriteRoots at scope construction time.
|
||||
addDirs []string
|
||||
// readDirs holds directories passed via --add-read-dir: READ-ONLY roots added
|
||||
// with scope.AddRead after construction. This is how a specialist child
|
||||
// inherits the parent's request_permissions read grants without gaining write
|
||||
// access to them — a plan auditing a granted external path reads it here.
|
||||
readDirs []string
|
||||
// tracePath, when set, writes a per-turn NDJSON trace (agenteval-compatible)
|
||||
// to the given file path — or to stderr when the value is "-". Falls back to
|
||||
// the ZERO_TRACE env var when the flag is absent. Off by default: a run
|
||||
@@ -195,15 +189,6 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
// mode-supplied model flows through the same resolution (and deprecation
|
||||
// notice) path as an explicit --model. Explicit flags still win: applyExecMode
|
||||
// only fills fields the caller left unset.
|
||||
// --reasoning-effort zeromaxing selects the POSTURE, not a provider effort
|
||||
// level. Normalize it into the profile selection before mode/profile
|
||||
// expansion so /effort zeromaxing and --exec-profile zeromaxing resolve to
|
||||
// exactly the same state, and so the posture name can never survive into a
|
||||
// provider request. Runs before applyExecMode, leaving the documented
|
||||
// precedence ordering (flag > mode > profile) exactly as it was.
|
||||
if err := normalizeZeromaxingEffort(&options); err != nil {
|
||||
return writeExecFormatUsageError(stdout, stderr, options.outputFormat, err.Error())
|
||||
}
|
||||
if err := applyExecMode(&options); err != nil {
|
||||
return writeExecFormatUsageError(stdout, stderr, options.outputFormat, err.Error())
|
||||
}
|
||||
@@ -211,10 +196,6 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
// runs after it so the mode's fills count as "set" and win. MaxTurns is
|
||||
// deferred to config resolution below, where the displaced resolved budget
|
||||
// is known and becomes the escalation restore target.
|
||||
// Captured BEFORE the profile expands: applyExecProfile arms self-correction
|
||||
// as a side effect, so reading options.selfCorrect afterwards would always
|
||||
// report "already on" and the delta would describe a change it just made.
|
||||
selfCorrectBeforeProfile := options.selfCorrect
|
||||
execProfile, execProfileFilledEffort, err := applyExecProfile(&options)
|
||||
if err != nil {
|
||||
return writeExecFormatUsageError(stdout, stderr, options.outputFormat, err.Error())
|
||||
@@ -274,103 +255,17 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
registry.Register(tools.NewEscalateModelTool())
|
||||
}
|
||||
var specialistRuntime *agentToolRuntime
|
||||
var planGate *specialist.PostureGate
|
||||
// Created before registration so the tool can hold it; its inner exec
|
||||
// recorder is attached once the session exists (below). A nil inner is a
|
||||
// no-op, so events before that point are simply not recorded — recording is
|
||||
// best-effort and must never be the thing that fails a run.
|
||||
planRecorder := &planSessionRecorder{}
|
||||
// DECLARED HERE, built further down. The specialist wiring below closes over
|
||||
// it so a child launched later is confined to the same roots this run holds;
|
||||
// reading it at registration time would capture nil forever, which is the
|
||||
// same reason planGate is a pointer.
|
||||
var execScope *sandbox.Scope
|
||||
if shouldRegisterExecSpecialistTools(options) {
|
||||
// Specialist tools register before the full config resolve below (so
|
||||
// --list-tools stays offline). swarm.maxTeamSize is not affected by
|
||||
// overrides, so an empty-overrides resolve yields the same value; a resolve
|
||||
// error falls back to the swarm's built-in default (0 => 8).
|
||||
maxTeamSize := 0
|
||||
// planSize resolves to the DEFAULT tier when this early resolve fails —
|
||||
// the same ceiling an unset value gets, never "no ceiling". A config
|
||||
// error must not be the thing that removes a bound.
|
||||
planSize := config.DefaultPlanSize
|
||||
// The provider profile is captured alongside the other early config so
|
||||
// auto_assign can list this provider's models. Hoisted out of the if
|
||||
// because the wiring below needs it; the zero value simply means
|
||||
// discovery is unavailable and a plan asking for it is told so.
|
||||
var planProvider config.ProviderProfile
|
||||
var planModelPrefs config.PlanModelsConfig
|
||||
if earlyCfg, cfgErr := deps.resolveConfig(workspaceRoot, config.Overrides{}); cfgErr == nil {
|
||||
maxTeamSize = earlyCfg.Swarm.MaxTeamSize
|
||||
planSize = earlyCfg.Profiles.PlanSizeTier()
|
||||
planProvider = earlyCfg.Provider
|
||||
planModelPrefs = earlyCfg.Profiles.PlanModels
|
||||
if swarmCfg, cfgErr := deps.resolveConfig(workspaceRoot, config.Overrides{}); cfgErr == nil {
|
||||
maxTeamSize = swarmCfg.Swarm.MaxTeamSize
|
||||
}
|
||||
var err error
|
||||
// The posture is fixed for a headless run, so the gate is set once here
|
||||
// rather than flipping. It is still a POINTER for the same reason the
|
||||
// TUI needs one: the tool holds it for the process's life.
|
||||
planGate = &specialist.PostureGate{}
|
||||
planGate.Set(execProfile.IsZeromaxing())
|
||||
// The run's operator filters are final here: applyExecMode has already
|
||||
// expanded any --mode preset onto them. They bound the plan tool's
|
||||
// parent grant, so a task can never hold a tool this run was denied.
|
||||
specialistRuntime, err = registerSpecialistTools(registry, workspaceRoot, maxTeamSize,
|
||||
options.enabledTools, options.disabledTools, planRecorder, orchestrateWiring{
|
||||
DiscoverModels: planModelDiscoverer(workspaceRoot, planProvider),
|
||||
// The run's EXTRA write roots only — the grants beyond the
|
||||
// workspace — exactly as the TUI wires them (scope.ExtraRoots).
|
||||
// Passing execScope.Roots() here handed the child the PARENT
|
||||
// WORKSPACE ROOT as an extra writable --add-dir, and for an
|
||||
// ISOLATED plan that defeats the isolation outright: the
|
||||
// worktree exists so a write-capable plan cannot touch the
|
||||
// parent tree, and this line was handing the parent tree back.
|
||||
// A child's own workspace comes from its --cwd; only explicit
|
||||
// grants ride along.
|
||||
//
|
||||
// A CLOSURE OVER THE VARIABLE, not over its value: the scope is
|
||||
// built further down this function, and a child is launched long
|
||||
// after both. Reading it here would capture nil forever — the same
|
||||
// reason planGate above is a pointer.
|
||||
ExtraWriteRoots: func(withinTurn bool) []string {
|
||||
return childWriteRoots(execScope, withinTurn)
|
||||
},
|
||||
// The read counterpart: a request_permissions READ grant lands in a
|
||||
// separate scope list the write roots above do not cover, so without
|
||||
// this a plan auditing a granted path fails "outside the workspace" in
|
||||
// every task. Same live-scope closure, same reason.
|
||||
ExtraReadRoots: func(withinTurn bool) []string {
|
||||
return childReadRoots(execScope, withinTurn)
|
||||
},
|
||||
// Proves a model will actually run before a task is assigned it,
|
||||
// so the plan never dispatches onto something the provider only
|
||||
// advertises.
|
||||
ProbeModel: planModelProber(workspaceRoot, planProvider, deps.newProvider),
|
||||
ProbeIdentity: planModelProbeIdentity(workspaceRoot, planProvider),
|
||||
ModelPrefs: planModelPreferences(planModelPrefs),
|
||||
Gate: planGate,
|
||||
PlanContext: specialist.PlanTaskContext{
|
||||
PostureReasoningEffort: string(execprofile.Zeromaxing.ReasoningEffort),
|
||||
Cwd: workspaceRoot,
|
||||
// Resolved permission mode is not available this early; the
|
||||
// executor applies its own fail-safe mapping from an empty mode
|
||||
// (never unsafe), so a plan task can never exceed the parent.
|
||||
PermissionMode: "",
|
||||
// This run's nesting depth. Left unset, the admission
|
||||
// headroom check and the executor's own depth check both
|
||||
// measured a depth that was always zero, so neither could
|
||||
// ever fire — an inert guard someone would later trust.
|
||||
Depth: options.depth,
|
||||
},
|
||||
Size: planSize,
|
||||
// The SAME pair of directories the TUI uses, so a plan saved in
|
||||
// one surface is found by the other.
|
||||
Plans: execPlanPaths(workspaceRoot),
|
||||
// Headless runs isolate too: --worktree already exists here, and
|
||||
// a write-capable plan must not be the one path that skips it.
|
||||
Isolate: newPlanIsolator(workspaceRoot),
|
||||
})
|
||||
specialistRuntime, err = registerSpecialistTools(registry, workspaceRoot, maxTeamSize)
|
||||
if err != nil {
|
||||
return writeExecProviderError(stdout, stderr, options.outputFormat, "specialist_error", err.Error())
|
||||
}
|
||||
@@ -434,45 +329,12 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
return writeExecProviderError(stdout, stderr, options.outputFormat, "provider_error", err.Error())
|
||||
}
|
||||
resolved.MaxTurns = execTurnBudget(resolved)
|
||||
// Profile selection refusal, evaluated once config is resolved (the disable
|
||||
// flag lives there). execprofile.SelectionRefusal is the ONE rule; the TUI
|
||||
// /effort and /profile paths call the same function, so the selection
|
||||
// decision cannot drift between surfaces — see
|
||||
// TestSelectionRefusalAgreesAcrossPaths.
|
||||
if refusal := execprofile.SelectionRefusal(execProfile, resolved.Profiles.DisableZeromaxing); refusal != "" {
|
||||
return writeExecFormatUsageError(stdout, stderr, options.outputFormat,
|
||||
fmt.Sprintf("cannot use execution profile %q: %s.", execProfile.Name, refusal))
|
||||
}
|
||||
// State what it actually changes, at selection time. Burying the real delta
|
||||
// in a PR body is how a posture ends up documented as doing things it does
|
||||
// not do.
|
||||
if execProfile.IsZeromaxing() {
|
||||
delta := execprofile.Delta(execprofile.DeltaState{
|
||||
// resolved.MaxTurns is still the pre-posture budget here: the Delta
|
||||
// notice is printed BEFORE applyProfileTurnBudget displaces it.
|
||||
CurrentMaxTurns: resolved.MaxTurns,
|
||||
Effort: execEffortTransition(execProfileFilledEffort),
|
||||
SelfCorrect: execSelfCorrectTransition(selfCorrectBeforeProfile),
|
||||
})
|
||||
if _, err := fmt.Fprintln(stderr, delta); err != nil {
|
||||
return exitCrash
|
||||
}
|
||||
}
|
||||
var displacedMaxTurns int
|
||||
resolved.MaxTurns, displacedMaxTurns = applyProfileTurnBudget(execProfile, options.maxTurns, resolved.MaxTurns)
|
||||
execScope, err = sandbox.NewScope(workspaceRoot, append(append([]string{}, resolved.Sandbox.AdditionalWriteRoots...), options.addDirs...))
|
||||
execScope, err := sandbox.NewScope(workspaceRoot, append(append([]string{}, resolved.Sandbox.AdditionalWriteRoots...), options.addDirs...))
|
||||
if err != nil {
|
||||
return writeExecProviderError(stdout, stderr, options.outputFormat, "sandbox_error", err.Error())
|
||||
}
|
||||
// READ-ONLY roots from --add-read-dir, added after construction so they never
|
||||
// become write roots: this is how a specialist child reads the parent's
|
||||
// request_permissions grants without gaining write access. A bad path fails
|
||||
// closed (the run stops) rather than silently dropping the grant.
|
||||
for _, dir := range options.readDirs {
|
||||
if _, err := execScope.AddRead(dir); err != nil {
|
||||
return writeExecProviderError(stdout, stderr, options.outputFormat, "sandbox_error", fmt.Sprintf("--add-read-dir %q: %v", dir, err))
|
||||
}
|
||||
}
|
||||
for _, tool := range tools.CoreToolsScoped(workspaceRoot, execScope) {
|
||||
registry.Register(tool)
|
||||
}
|
||||
@@ -752,7 +614,6 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
}
|
||||
|
||||
sessionRecorder := execSessionRecorder{prepared: preparedSession}
|
||||
planRecorder.recorder = &sessionRecorder
|
||||
// Surface a best-effort session-recording failure once, on every exit path.
|
||||
defer sessionRecorder.warnIfRecordingFailed(stderr)
|
||||
userPayload := map[string]any{
|
||||
@@ -794,11 +655,9 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
// can be combined with the plugin activation's, and emit at most one notice when
|
||||
// project hooks/plugins were dropped for an untrusted workspace.
|
||||
hookDispatcher, hookSkip := newHookDispatcherWithExtra(workspaceRoot, pluginActivation.hooks, trustRoot, executionRunner)
|
||||
specialistRuntime.attachLifecycleHooks(hookDispatcher)
|
||||
emitTrustNotice(stderr, hookSkip, pluginActivation.trustSkip, mcpSkip)
|
||||
result, err := agent.Run(runCtx, agentPrompt, provider, agent.Options{
|
||||
MaxTurns: resolved.MaxTurns,
|
||||
MaxTokens: execProfile.MaxTokens,
|
||||
ContextWindow: resolveAgentContextWindow(runCtx, modelRegistry, resolved.Provider),
|
||||
DeferThreshold: effectiveDeferThreshold,
|
||||
Specialists: specialistRuntime.specialistInfos(),
|
||||
@@ -810,7 +669,6 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
Depth: options.depth,
|
||||
SessionTitle: sessionTitle,
|
||||
ProviderName: resolved.Provider.Name,
|
||||
ModelFamily: providercatalog.ModelFamilyFor(resolved.Provider.CatalogID),
|
||||
Model: resolved.Provider.Model,
|
||||
ModelSwitcher: modelSwitcher,
|
||||
TurnSessionProvider: turnSessions,
|
||||
@@ -832,13 +690,6 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
SelfCorrect: selfCorrector,
|
||||
FileDiagnostics: fileDiagnostics,
|
||||
Profile: execProfile.Policy(displacedMaxTurns, execProfileFilledEffort),
|
||||
// A headless run that selected the posture is by definition entering it:
|
||||
// the process starts, runs once, exits, so there is no earlier turn for
|
||||
// it to have been active across.
|
||||
Zeromaxing: execZeromaxing(execProfile),
|
||||
// Whether the enter notice may NAME the orchestrate tool. Derived from
|
||||
// the registry rather than passed as a flag — see orchestrateAvailable.
|
||||
OrchestrateAvailable: orchestrateAvailable(registry),
|
||||
// Headless exec: don't accept a no-tool-call turn as "done" while work
|
||||
// clearly remains (pending plan items / a mid-step continuation cue) —
|
||||
// nudge to continue, and finalize as INCOMPLETE rather than false success
|
||||
@@ -861,7 +712,7 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
"arguments": call.Arguments,
|
||||
})
|
||||
// Snapshot before-state of files this call will mutate (safe rewind).
|
||||
if checkpoint, ok := sessionRecorder.captureCheckpoint(workspaceRoot, registry, call); ok {
|
||||
if checkpoint, ok := sessionRecorder.captureCheckpoint(workspaceRoot, call); ok {
|
||||
writer.checkpoint(checkpoint)
|
||||
}
|
||||
},
|
||||
@@ -977,14 +828,6 @@ func runExec(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) in
|
||||
// {"type":"done","exit_code":0} that would otherwise mask the incomplete exit.
|
||||
// An `error` event (not just a warning) is emitted so log/cron consumers that
|
||||
// scan for type=="error" can recover the reason.
|
||||
// A plan that did not fully complete is work left undone, which is exactly
|
||||
// what exitIncomplete exists for. Folded into the EXISTING incomplete path
|
||||
// rather than a second exit route, so a plan and a stalled loop report the
|
||||
// same way. Does not override an incompleteness the loop already found.
|
||||
if reason, planIncomplete := planRecorder.Incomplete(); planIncomplete && !result.Incomplete {
|
||||
result.Incomplete = true
|
||||
result.IncompleteReason = reason
|
||||
}
|
||||
if result.Incomplete {
|
||||
reason := result.IncompleteReason
|
||||
if reason == "" {
|
||||
@@ -1400,64 +1243,6 @@ func applyExecProfile(options *execOptions) (execprofile.Profile, bool, error) {
|
||||
return profile, effortFilled, nil
|
||||
}
|
||||
|
||||
// normalizeZeromaxingEffort folds `--reasoning-effort zeromaxing` into
|
||||
// `--exec-profile zeromaxing`. The posture is one thing with one name, reachable
|
||||
// from either flag, so this is a rename rather than a second implementation.
|
||||
//
|
||||
// A conflicting explicit --exec-profile is a usage error rather than a silent
|
||||
// winner: the user asked for two different postures and deserves to be told,
|
||||
// not to have one quietly discarded.
|
||||
func normalizeZeromaxingEffort(options *execOptions) error {
|
||||
if !strings.EqualFold(strings.TrimSpace(options.reasoningEffort), execprofile.Name) {
|
||||
return nil
|
||||
}
|
||||
if selected := strings.TrimSpace(options.execProfile); selected != "" &&
|
||||
!strings.EqualFold(selected, execprofile.Name) {
|
||||
return execUsageError{fmt.Sprintf(
|
||||
"--reasoning-effort %s selects the %s execution profile, which conflicts with --exec-profile %s. Pass one of them.",
|
||||
execprofile.Name, execprofile.Name, selected)}
|
||||
}
|
||||
options.execProfile = execprofile.Name
|
||||
// Clear the effort so the profile FILLS it (with "high"). Leaving the
|
||||
// posture name here would carry it into forwardedReasoningEffort.
|
||||
options.reasoningEffort = ""
|
||||
return nil
|
||||
}
|
||||
|
||||
// execEffortTransition reports what the posture did to reasoning effort for
|
||||
// this run. applyExecProfile reports whether it actually filled the effort; it
|
||||
// backs off when the caller passed one explicitly. The headless path does not
|
||||
// gate the fill on model support (forwardedReasoningEffort coerces later, and
|
||||
// prints its own notice), so EffortNotSupported is not reachable here.
|
||||
func execEffortTransition(effortFilled bool) execprofile.EffortTransition {
|
||||
if effortFilled {
|
||||
return execprofile.EffortRaised
|
||||
}
|
||||
return execprofile.EffortKeptExplicit
|
||||
}
|
||||
|
||||
// execSelfCorrectTransition reports what the posture did to post-edit
|
||||
// verification for THIS run. A headless run has no way to override the profile
|
||||
// after it applies (--self-correct is presence-only and read before), so the
|
||||
// Overridden state is unreachable here — it is a TUI-only condition.
|
||||
func execSelfCorrectTransition(selfCorrectBefore bool) execprofile.SelfCorrectTransition {
|
||||
if selfCorrectBefore {
|
||||
return execprofile.SelfCorrectAlreadyOn
|
||||
}
|
||||
return execprofile.SelfCorrectRaised
|
||||
}
|
||||
|
||||
// execZeromaxing maps a selected profile onto the run's posture lifecycle. A
|
||||
// headless exec run is one process for one run, so selecting the posture means
|
||||
// entering it and anything else means off — the Active/Exiting states only
|
||||
// arise in a session that outlives a single run (the TUI).
|
||||
func execZeromaxing(profile execprofile.Profile) agent.Zeromaxing {
|
||||
if profile.IsZeromaxing() {
|
||||
return agent.ZeromaxingEntering
|
||||
}
|
||||
return agent.ZeromaxingOff
|
||||
}
|
||||
|
||||
// specProfileEffortFilled reports whether the profile's effort fill actually
|
||||
// governs the spec-draft run. An explicit --spec-reasoning-effort replaces the
|
||||
// filled effort for the draft, so the escalation's effort restore must not arm
|
||||
@@ -1589,16 +1374,6 @@ func forwardedReasoningEffort(registry modelregistry.Registry, modelID string, r
|
||||
if requested == "" {
|
||||
return ""
|
||||
}
|
||||
// LOAD-BEARING GUARD. "zeromaxing" is a Zero posture name, never a provider
|
||||
// effort level. normalizeZeromaxingEffort already converts it into a profile
|
||||
// selection long before this point, so reaching here means something
|
||||
// upstream regressed — and the failure mode would be a provider request
|
||||
// carrying a parameter value no provider defines. Refuse it at the boundary
|
||||
// rather than trusting the caller. Pinned by
|
||||
// TestZeromaxingIsNeverForwardedToAProvider.
|
||||
if strings.EqualFold(requested, execprofile.Name) {
|
||||
return ""
|
||||
}
|
||||
entry, ok := registry.Get(strings.TrimSpace(modelID))
|
||||
if !ok {
|
||||
return requested
|
||||
@@ -1729,48 +1504,3 @@ func writeTraceSnapshot(snapshot *trace.TurnTrace, dest string, stderr io.Writer
|
||||
defer file.Close()
|
||||
return trace.WriteNDJSON(file, snapshot)
|
||||
}
|
||||
|
||||
// execPlanPaths locates saved plans for a headless run, from the same pair of
|
||||
// directories the TUI uses. A resolve failure yields the project directory
|
||||
// alone rather than nothing: a plan checked into the repo must still run when
|
||||
// the user config directory is unavailable.
|
||||
func execPlanPaths(workspaceRoot string) specialist.PlanPaths {
|
||||
userConfigDir, _ := config.UserConfigDir()
|
||||
return specialist.DefaultPlanPaths(workspaceRoot, userConfigDir)
|
||||
}
|
||||
|
||||
// childWriteRoots is what a run's children may WRITE beyond their own
|
||||
// workspace: the run's granted extra roots, and never the parent workspace root
|
||||
// itself. It returned execScope.Roots() — workspace included — and for an
|
||||
// ISOLATED plan that defeats the isolation outright: the worktree exists so a
|
||||
// write-capable plan cannot touch the parent tree, and the wiring was handing
|
||||
// the parent tree back as a writable --add-dir. Both surfaces (TUI and headless)
|
||||
// use this one function.
|
||||
//
|
||||
// TURN-SCOPED GRANTS ONLY FOR A CHILD THAT ENDS WITHIN THE TURN. A root in a
|
||||
// child's argv is permanent for the child's life; the parent's end-of-turn
|
||||
// release cannot reach it. So a child that may outlive the turn (background
|
||||
// Task, plan task, swarm member) gets only the roots that outlive it as well.
|
||||
func childWriteRoots(scope *sandbox.Scope, withinTurn bool) []string {
|
||||
if scope == nil {
|
||||
return nil
|
||||
}
|
||||
if withinTurn {
|
||||
return scope.ExtraRoots()
|
||||
}
|
||||
return scope.SessionExtraRoots()
|
||||
}
|
||||
|
||||
// childReadRoots is childWriteRoots for READ grants: every path the run may
|
||||
// read beyond its workspace (write roots included, since they confer read),
|
||||
// turn-scoped grants only for a child that ends within the turn. The executor
|
||||
// emits these as --add-read-dir only.
|
||||
func childReadRoots(scope *sandbox.Scope, withinTurn bool) []string {
|
||||
if scope == nil {
|
||||
return nil
|
||||
}
|
||||
if withinTurn {
|
||||
return scope.ExtraReadRoots()
|
||||
}
|
||||
return scope.SessionExtraReadRoots()
|
||||
}
|
||||
@@ -1,206 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/agent"
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/sessions"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
func freeformApplyPatchArguments(root string, absolute bool) string {
|
||||
target := "main.txt"
|
||||
if absolute {
|
||||
target = filepath.Join(root, "main.txt")
|
||||
}
|
||||
return "*** Begin Patch\n*** Update File: " + target + "\n@@\n-before\n+after\n*** End Patch\n"
|
||||
}
|
||||
|
||||
// captureCheckpoint must resolve a freeform apply_patch call through the same
|
||||
// preparation execution uses, so the before-state is captured and rewind can
|
||||
// restore it. The JSON control pins the unchanged path.
|
||||
func TestCaptureCheckpointFreeformApplyPatch(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
freeform bool
|
||||
absolute bool
|
||||
jsonArgs bool
|
||||
}{
|
||||
{name: "freeform", freeform: true},
|
||||
{name: "freeform absolute header", freeform: true, absolute: true},
|
||||
{name: "json arguments", jsonArgs: true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Setenv("XDG_DATA_HOME", t.TempDir())
|
||||
t.Setenv("ZERO_CHECKPOINTS", "on")
|
||||
root := t.TempDir()
|
||||
// The absolute-header case needs the physical root spelling the tool
|
||||
// resolves (macOS /var vs /private/var, Windows short names).
|
||||
if resolved, err := filepath.EvalSymlinks(root); err == nil {
|
||||
root = resolved
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, "main.txt"), []byte("before\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
store := sessions.NewStore(sessions.StoreOptions{})
|
||||
session, err := store.Create(sessions.CreateInput{Title: "freeform checkpoint"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := store.AppendEvent(session.SessionID, sessions.AppendEventInput{
|
||||
Type: sessions.EventMessage,
|
||||
Payload: map[string]string{"role": "user", "content": "apply the patch"},
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
recorder := &execSessionRecorder{prepared: sessions.PreparedExec{Store: store, Session: session}}
|
||||
registry := tools.NewRegistry()
|
||||
registry.Register(tools.NewScopedApplyPatchTool(root, nil))
|
||||
|
||||
call := agent.ToolCall{ID: "call", Name: "apply_patch", Freeform: tc.freeform}
|
||||
if tc.jsonArgs {
|
||||
encoded, err := json.Marshal(map[string]string{"patch": freeformApplyPatchArguments(root, false)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
call.Arguments = string(encoded)
|
||||
} else {
|
||||
call.Arguments = freeformApplyPatchArguments(root, tc.absolute)
|
||||
}
|
||||
|
||||
event, ok := recorder.captureCheckpoint(root, registry, call)
|
||||
if !ok || event.Type != sessions.EventSessionCheckpoint {
|
||||
t.Fatalf("captureCheckpoint() = %#v, %t; want a checkpoint event", event, ok)
|
||||
}
|
||||
var payload sessions.CheckpointPayload
|
||||
if err := json.Unmarshal(event.Payload, &payload); err != nil {
|
||||
t.Fatalf("decode checkpoint payload: %v", err)
|
||||
}
|
||||
if len(payload.Files) != 1 || payload.Files[0].Path != "main.txt" || payload.Files[0].Blob == "" {
|
||||
t.Fatalf("checkpoint files = %#v, want main.txt with a blob", payload.Files)
|
||||
}
|
||||
|
||||
// Simulate the tool's mutation, then rewind past it.
|
||||
if err := os.WriteFile(filepath.Join(root, "main.txt"), []byte("after\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
report, err := store.ApplyRewind(session.SessionID, root, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("ApplyRewind: %v", err)
|
||||
}
|
||||
if report.FilesRestored != 1 {
|
||||
t.Fatalf("restore report = %#v, want one restored file", report)
|
||||
}
|
||||
content, err := os.ReadFile(filepath.Join(root, "main.txt"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(content) != "before\n" {
|
||||
t.Fatalf("restored content = %q, want before newline", content)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// freeformPatchExecProvider emits one native freeform apply_patch call, then a
|
||||
// final text turn.
|
||||
type freeformPatchExecProvider struct {
|
||||
patch string
|
||||
calls int
|
||||
}
|
||||
|
||||
func (provider *freeformPatchExecProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
||||
ch := make(chan zeroruntime.StreamEvent, 5)
|
||||
calls := provider.calls
|
||||
provider.calls++
|
||||
if calls == 0 {
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call_patch", ToolName: "apply_patch", ToolCallFreeform: true}
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call_patch", ArgumentsFragment: provider.patch}
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call_patch"}
|
||||
} else {
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventText, Content: "patched"}
|
||||
}
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
||||
close(ch)
|
||||
return ch, nil
|
||||
}
|
||||
|
||||
// End-to-end: exec captures the freeform checkpoint and `sessions rewind`
|
||||
// restores the pre-patch content.
|
||||
func TestRunExecFreeformApplyPatchCheckpointsForRewind(t *testing.T) {
|
||||
isolateCLIUserState(t)
|
||||
t.Setenv("ZERO_CHECKPOINTS", "on")
|
||||
root := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(root, "main.txt"), []byte("before\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
provider := &freeformPatchExecProvider{patch: freeformApplyPatchArguments(root, false)}
|
||||
var stdout, stderr bytes.Buffer
|
||||
exitCode := runWithDeps([]string{
|
||||
"exec", "--init-session-id", "freeform_rewind", "--skip-permissions-unsafe",
|
||||
"--output-format", "stream-json", "apply the patch",
|
||||
}, &stdout, &stderr, appDeps{
|
||||
getwd: func() (string, error) { return root, nil },
|
||||
resolveConfig: func(string, config.Overrides) (config.ResolvedConfig, error) {
|
||||
return execResolvedConfig(), nil
|
||||
},
|
||||
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
||||
return provider, nil
|
||||
},
|
||||
})
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("exit = %d, want %d; stderr=%q stdout=%q", exitCode, exitSuccess, stderr.String(), stdout.String())
|
||||
}
|
||||
content, err := os.ReadFile(filepath.Join(root, "main.txt"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(content) != "after\n" {
|
||||
t.Fatalf("patched content = %q, want after newline", content)
|
||||
}
|
||||
|
||||
store := sessions.NewStore(sessions.StoreOptions{})
|
||||
events, err := store.ReadEvents("freeform_rewind")
|
||||
if err != nil {
|
||||
t.Fatalf("ReadEvents: %v", err)
|
||||
}
|
||||
checkpoints := 0
|
||||
for _, event := range events {
|
||||
if event.Type == sessions.EventSessionCheckpoint {
|
||||
checkpoints++
|
||||
}
|
||||
}
|
||||
if checkpoints != 1 {
|
||||
t.Fatalf("checkpoint events = %d, want 1; events=%v", checkpoints, sessionEventTypes(events))
|
||||
}
|
||||
|
||||
report, err := store.ApplyRewind("freeform_rewind", root, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("ApplyRewind: %v", err)
|
||||
}
|
||||
if report.FilesRestored != 1 {
|
||||
t.Fatalf("restore report = %#v, want one restored file", report)
|
||||
}
|
||||
content, err = os.ReadFile(filepath.Join(root, "main.txt"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(content) != "before\n" {
|
||||
t.Fatalf("rewound content = %q, want before newline", content)
|
||||
}
|
||||
}
|
||||
|
||||
func sessionEventTypes(events []sessions.Event) []string {
|
||||
types := make([]string, 0, len(events))
|
||||
for _, event := range events {
|
||||
types = append(types, string(event.Type))
|
||||
}
|
||||
return types
|
||||
}
|
||||
@@ -1,82 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
)
|
||||
|
||||
// AN ISOLATED PLAN'S CHILD MUST NOT GET THE PARENT TREE BACK. The headless
|
||||
// wiring handed children execScope.Roots() — the parent workspace root
|
||||
// included — as extra WRITE roots. A worktree-isolated plan exists so a
|
||||
// write-capable task cannot touch the parent tree, and that line handed the
|
||||
// parent tree back as a writable --add-dir. Only the run's explicit extra
|
||||
// grants may ride along; the child's own workspace comes from its --cwd.
|
||||
func TestExecChildWriteRootsExcludeTheParentWorkspace(t *testing.T) {
|
||||
// The scope stores symlink-resolved paths (/var → /private/var on macOS),
|
||||
// so both sides of every comparison resolve first.
|
||||
resolve := func(path string) string {
|
||||
resolved, err := filepath.EvalSymlinks(path)
|
||||
if err != nil {
|
||||
t.Fatalf("EvalSymlinks(%s): %v", path, err)
|
||||
}
|
||||
return resolved
|
||||
}
|
||||
workspace := resolve(t.TempDir())
|
||||
granted := resolve(t.TempDir())
|
||||
scope, err := sandbox.NewScope(workspace, []string{granted})
|
||||
if err != nil {
|
||||
t.Fatalf("NewScope: %v", err)
|
||||
}
|
||||
for _, withinTurn := range []bool{true, false} {
|
||||
roots := childWriteRoots(scope, withinTurn)
|
||||
found := false
|
||||
for _, root := range roots {
|
||||
if root == workspace {
|
||||
t.Fatalf("the parent workspace root leaked into a child's write roots: %q", roots)
|
||||
}
|
||||
if root == granted {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("the explicit grant must still ride along (withinTurn=%v), got %q", withinTurn, roots)
|
||||
}
|
||||
if childWriteRoots(nil, withinTurn) != nil {
|
||||
t.Fatal("a nil scope must yield no roots")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A TURN-SCOPED WRITE GRANT REACHES ONLY A CHILD THAT ENDS WITHIN THE TURN.
|
||||
//
|
||||
// Scope.ExtraRoots carried AddTemporaryWrite roots like permanent ones, so a
|
||||
// turn-scoped request_permissions grant left as a plain --add-dir and a child
|
||||
// that outlived the turn kept writing there after the parent had released it.
|
||||
func TestChildWriteRootsWithholdATurnGrantFromAChildThatOutlivesTheTurn(t *testing.T) {
|
||||
workspace, err := filepath.EvalSymlinks(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
scope, err := sandbox.NewScope(workspace, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("NewScope: %v", err)
|
||||
}
|
||||
// Outside the temp roots NewScope seeds as permanent: a turn grant under one
|
||||
// of them would be covered already and recorded as nothing.
|
||||
temporary := tempDirOutsideDefaultTemp(t)
|
||||
root, release, err := scope.AddTemporaryWrite(temporary)
|
||||
if err != nil {
|
||||
t.Fatalf("AddTemporaryWrite: %v", err)
|
||||
}
|
||||
defer release()
|
||||
|
||||
if !slices.Contains(childWriteRoots(scope, true), root) {
|
||||
t.Fatalf("a child inside the turn lost the turn's grant: %v", childWriteRoots(scope, true))
|
||||
}
|
||||
if slices.Contains(childWriteRoots(scope, false), root) {
|
||||
t.Fatalf("a child that may outlive the turn was handed the turn-scoped grant %q: %v", root, childWriteRoots(scope, false))
|
||||
}
|
||||
}
|
||||
@@ -1,90 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// guardStopExecProvider emits turns with no visible text and no tool calls, so
|
||||
// the empty-turn guardrail stops the run after its consecutive-empty-turn cap.
|
||||
type guardStopExecProvider struct{}
|
||||
|
||||
func (guardStopExecProvider) StreamCompletion(ctx context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
||||
ch := make(chan zeroruntime.StreamEvent, 1)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
close(ch)
|
||||
return ch, ctx.Err()
|
||||
case ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}:
|
||||
}
|
||||
close(ch)
|
||||
return ch, nil
|
||||
}
|
||||
|
||||
// A guardrail stop in headless exec must use the existing INCOMPLETE contract
|
||||
// (exit 4, error event, run_end status), and --no-completion-gate must keep
|
||||
// reporting success. isolateCLIUserState also turns off the models.dev refresh
|
||||
// that runExec starts in the background (see the #1152/#1161 flake).
|
||||
func TestRunExecGuardrailStopIncompleteContract(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
args []string
|
||||
wantExit int
|
||||
wantStatus string
|
||||
}{
|
||||
{"gate on", []string{"exec", "--output-format", "stream-json", "complete the task"}, exitIncomplete, "incomplete"},
|
||||
{"gate off", []string{"exec", "--no-completion-gate", "--output-format", "stream-json", "complete the task"}, exitSuccess, "success"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
isolateCLIUserState(t)
|
||||
cwd := t.TempDir()
|
||||
var stdout, stderr bytes.Buffer
|
||||
exitCode := runWithDeps(tc.args, &stdout, &stderr, appDeps{
|
||||
getwd: func() (string, error) { return cwd, nil },
|
||||
resolveConfig: func(string, config.Overrides) (config.ResolvedConfig, error) {
|
||||
return execResolvedConfig(), nil
|
||||
},
|
||||
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
||||
return guardStopExecProvider{}, nil
|
||||
},
|
||||
})
|
||||
if exitCode != tc.wantExit {
|
||||
t.Fatalf("exit code = %d, want %d; stderr=%q", exitCode, tc.wantExit, stderr.String())
|
||||
}
|
||||
|
||||
events := decodeJSONLines(t, stdout.String())
|
||||
if len(events) == 0 || events[len(events)-1]["type"] != "run_end" {
|
||||
t.Fatalf("expected run_end as the last event, got %#v", events)
|
||||
}
|
||||
runEnd := events[len(events)-1]
|
||||
if runEnd["status"] != tc.wantStatus {
|
||||
t.Fatalf("run_end status = %v, want %q", runEnd["status"], tc.wantStatus)
|
||||
}
|
||||
if runEnd["exitCode"] != float64(tc.wantExit) {
|
||||
t.Fatalf("run_end exitCode = %v, want %d", runEnd["exitCode"], tc.wantExit)
|
||||
}
|
||||
finalEvent := findJSONEvent(t, events, "final")
|
||||
if text, _ := finalEvent["text"].(string); !strings.Contains(text, "no output") {
|
||||
t.Fatalf("final text = %q, want the guard stop answer", text)
|
||||
}
|
||||
|
||||
if tc.wantStatus == "incomplete" {
|
||||
errorEvent := findJSONEvent(t, events, "error")
|
||||
if errorEvent["code"] != "incomplete" {
|
||||
t.Fatalf("error code = %v, want incomplete", errorEvent["code"])
|
||||
}
|
||||
if message, _ := errorEvent["message"].(string); !strings.Contains(message, "no output") {
|
||||
t.Fatalf("error message = %q, want the guard reason", message)
|
||||
}
|
||||
} else if slices.Contains(jsonEventTypes(events), "error") {
|
||||
t.Fatalf("gate-off run must not emit an error event: %#v", events)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/Twigpine/zero/internal/execprofile"
|
||||
"github.com/Twigpine/zero/internal/sessions"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
)
|
||||
@@ -130,19 +129,6 @@ func parseExecArgs(args []string) (execOptions, bool, error) {
|
||||
return options, false, err
|
||||
}
|
||||
options.addDirs = append(options.addDirs, value)
|
||||
case arg == "--add-read-dir":
|
||||
value, next, err := nextFlagValue(args, index, arg)
|
||||
if err != nil {
|
||||
return options, false, err
|
||||
}
|
||||
options.readDirs = append(options.readDirs, value)
|
||||
index = next
|
||||
case strings.HasPrefix(arg, "--add-read-dir="):
|
||||
value, err := requiredInlineFlagValue(arg, "--add-read-dir")
|
||||
if err != nil {
|
||||
return options, false, err
|
||||
}
|
||||
options.readDirs = append(options.readDirs, value)
|
||||
case arg == "--mode":
|
||||
value, next, err := nextFlagValue(args, index, arg)
|
||||
if err != nil {
|
||||
@@ -567,33 +553,9 @@ func nextFlagValue(args []string, index int, flag string) (string, int, error) {
|
||||
default:
|
||||
return "", index, execUsageError{fmt.Sprintf("Invalid input format %q. Expected text or stream-json.", next)}
|
||||
}
|
||||
case "--reasoning-effort":
|
||||
// zeromaxing is accepted here as a POSTURE name, not a provider effort
|
||||
// level: normalizeZeromaxingEffort converts it into the equivalent
|
||||
// --exec-profile selection before anything reads it as an effort, and
|
||||
// forwardedReasoningEffort refuses to forward it even if that regressed.
|
||||
//
|
||||
// "max" is deliberately NOT accepted, exactly as before — the flag has
|
||||
// always rejected it and that spelling stays reserved for a real
|
||||
// provider rung. Adding it here would burn the name.
|
||||
switch strings.ToLower(next) {
|
||||
case "low", "medium", "high", execprofile.Name:
|
||||
default:
|
||||
return "", index, execUsageError{fmt.Sprintf("invalid %s %q. Expected low, medium, high, or %s.", flag, next, execprofile.Name)}
|
||||
}
|
||||
case "--spec-reasoning-effort":
|
||||
// The spec-draft effort is a plain provider level; the posture is a RUN
|
||||
// posture and has no meaning for a draft, so it is not accepted here.
|
||||
case "--reasoning-effort", "--spec-reasoning-effort":
|
||||
switch strings.ToLower(next) {
|
||||
case "low", "medium", "high":
|
||||
case execprofile.Name:
|
||||
// Rejecting this with the generic "Expected low, medium, or high"
|
||||
// reads like a bug to anyone who just learned the name works on
|
||||
// --reasoning-effort. Say why it does not apply, and name the two
|
||||
// flags that do what they were reaching for.
|
||||
return "", index, execUsageError{fmt.Sprintf(
|
||||
"invalid %s %q. %s is a run posture, not a reasoning level, so it does not apply to spec drafting. Use %s high for the draft, or --reasoning-effort %s to run under the posture.",
|
||||
flag, next, execprofile.Name, flag, execprofile.Name)}
|
||||
default:
|
||||
return "", index, execUsageError{fmt.Sprintf("invalid %s %q. Expected low, medium, or high.", flag, next)}
|
||||
}
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
package cli
|
||||
|
||||
import "testing"
|
||||
|
||||
// --add-read-dir collects READ-ONLY roots, the read counterpart of --add-dir.
|
||||
// A specialist child receives the parent's request_permissions read grants on
|
||||
// this flag so it can audit a granted path without gaining write access.
|
||||
func TestParseExecArgsCollectsAddReadDirs(t *testing.T) {
|
||||
options, _, err := parseExecArgs([]string{
|
||||
"--prompt", "hi",
|
||||
"--add-read-dir", "/one",
|
||||
"--add-read-dir=/two",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("parseExecArgs: %v", err)
|
||||
}
|
||||
if len(options.readDirs) != 2 || options.readDirs[0] != "/one" || options.readDirs[1] != "/two" {
|
||||
t.Fatalf("readDirs=%v want [/one /two]", options.readDirs)
|
||||
}
|
||||
// It must NOT bleed into addDirs — that would emit it as a write root.
|
||||
if len(options.addDirs) != 0 {
|
||||
t.Fatalf("a read-only root leaked into addDirs (write roots): %v", options.addDirs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseExecArgsAddReadDirRequiresValue(t *testing.T) {
|
||||
if _, _, err := parseExecArgs([]string{"--add-read-dir"}); err == nil {
|
||||
t.Fatal("bare --add-read-dir must error")
|
||||
}
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The plan-size tier, proved END TO END: a .zero/config.json setting has to
|
||||
// reach the ceiling a plan is actually rejected against.
|
||||
//
|
||||
// A test on registerSpecialistTools would pass while `zero exec` never read the
|
||||
// setting at all — that is the exact shape of this feature's recurring defect,
|
||||
// a field correctly threaded through everything except the one call site that
|
||||
// populates it. So this drives Run(), writes a real config file, and asserts on
|
||||
// what the model is told.
|
||||
func TestExecReadsThePlanSizeTierFromProjectConfig(t *testing.T) {
|
||||
clearProviderEnv(t)
|
||||
root := t.TempDir()
|
||||
configDir := filepath.Join(root, ".zero")
|
||||
if err := os.MkdirAll(configDir, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// BOTH probes are plans that get REJECTED, at DIFFERENT ceilings. An admitted
|
||||
// plan would spawn real child processes — six of them — so the control here
|
||||
// cannot be "and this one is admitted"; it is "and this one is rejected at
|
||||
// the other tier's number", which isolates the configured value just as well
|
||||
// and executes nothing.
|
||||
planArgs := func(n int) string {
|
||||
tasks := make([]map[string]any, 0, n)
|
||||
for i := 0; i < n; i++ {
|
||||
tasks = append(tasks, map[string]any{"id": "t" + strings.Repeat("z", i), "prompt": "look"})
|
||||
}
|
||||
encoded, err := json.Marshal(map[string]any{
|
||||
"name": "sweep",
|
||||
"tasks": tasks,
|
||||
"budget": map[string]any{"max_workers": 1},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(encoded)
|
||||
}
|
||||
|
||||
var plan string
|
||||
var toolOutput string
|
||||
turn := 0
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
var body map[string]any
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
turn++
|
||||
if turn == 1 {
|
||||
call, _ := json.Marshal(map[string]any{
|
||||
"choices": []any{map[string]any{"delta": map[string]any{
|
||||
"tool_calls": []any{map[string]any{
|
||||
"index": 0, "id": "call_1", "type": "function",
|
||||
"function": map[string]any{"name": "orchestrate", "arguments": plan},
|
||||
}},
|
||||
}}},
|
||||
})
|
||||
_, _ = io.WriteString(w, "data: "+string(call)+"\n\n")
|
||||
_, _ = io.WriteString(w, `data: {"choices":[{"delta":{},"finish_reason":"tool_calls"}]}`+"\n\n")
|
||||
_, _ = io.WriteString(w, "data: [DONE]\n\n")
|
||||
return
|
||||
}
|
||||
// The second turn carries the tool result back. That message is what the
|
||||
// model would have been told, and it is the assertion target.
|
||||
if messages, ok := body["messages"].([]any); ok {
|
||||
for _, raw := range messages {
|
||||
message, _ := raw.(map[string]any)
|
||||
if message["role"] == "tool" {
|
||||
if content, ok := message["content"].(string); ok {
|
||||
toolOutput += content
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
_, _ = io.WriteString(w, `data: {"choices":[{"delta":{"content":"done"}}]}`+"\n\n")
|
||||
_, _ = io.WriteString(w, "data: [DONE]\n\n")
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
providerConfig := func(planSize string) string {
|
||||
profiles := ""
|
||||
if planSize != "" {
|
||||
profiles = `"profiles": {"planSize": "` + planSize + `"},`
|
||||
}
|
||||
return `{
|
||||
` + profiles + `
|
||||
"activeProvider": "local",
|
||||
"providers": [{
|
||||
"name": "local",
|
||||
"provider_kind": "openai-compatible",
|
||||
"base_url": "` + server.URL + `",
|
||||
"api_key": "sk-local",
|
||||
"model": "local-model"
|
||||
}]
|
||||
}`
|
||||
}
|
||||
|
||||
run := func(planSize string, taskCount int) string {
|
||||
turn = 0
|
||||
toolOutput = ""
|
||||
plan = planArgs(taskCount)
|
||||
if err := os.WriteFile(filepath.Join(configDir, "config.json"), []byte(providerConfig(planSize)), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var stdout, stderr bytes.Buffer
|
||||
Run([]string{"exec", "--cwd", root, "--reasoning-effort", "zeromaxing", "--auto", "high", "sweep it"}, &stdout, &stderr)
|
||||
return toolOutput
|
||||
}
|
||||
|
||||
// small: a six-task plan is refused at 5, and the message names the tier that
|
||||
// refused it plus how to raise it.
|
||||
small := run("small", 6)
|
||||
for _, want := range []string{`"small" plan size`, "planSize", "exceeds the limit of 5"} {
|
||||
if !strings.Contains(small, want) {
|
||||
t.Fatalf("under planSize=small the tool output must contain %q; got:\n%s", want, small)
|
||||
}
|
||||
}
|
||||
|
||||
// THE OTHER DIRECTION, which is what makes the first half mean anything: with
|
||||
// no setting the ceiling is the DEFAULT tier's 20, not small's 5. Without
|
||||
// this, a run that ignored config entirely and always used one hard-coded
|
||||
// number would satisfy the assertion above.
|
||||
unset := run("", 25)
|
||||
for _, want := range []string{`"medium" plan size`, "exceeds the limit of 20"} {
|
||||
if !strings.Contains(unset, want) {
|
||||
t.Fatalf("with no planSize set the ceiling must be the default tier's; want %q, got:\n%s", want, unset)
|
||||
}
|
||||
}
|
||||
if strings.Contains(unset, "limit of 5") {
|
||||
t.Fatalf("an unset planSize applied small's ceiling:\n%s", unset)
|
||||
}
|
||||
}
|
||||
@@ -115,26 +115,6 @@ func runExecAddDirWriteProbe(t *testing.T, cwd string, args []string, target str
|
||||
// is known the scoped core tools are re-registered and Registry.Register
|
||||
// replaces the earlier instances BY NAME. The before/after write_file probes
|
||||
// prove both the overwrite and the scoped enforcement it ships.
|
||||
// --add-read-dir grants READ, not write, at the process boundary: a write_file
|
||||
// inside the read-only root is DENIED. This pins exec's readDirs loop to
|
||||
// scope.AddRead — had it used scope.Add (write), or emitted --add-dir, this write
|
||||
// would land, and a path the parent may only read would be writable in the child.
|
||||
func TestAddReadDirGrantsReadOnlyAtTheProcessBoundary(t *testing.T) {
|
||||
t.Setenv("XDG_DATA_HOME", t.TempDir())
|
||||
cwd := t.TempDir()
|
||||
readRoot := tempDirOutsideDefaultTemp(t)
|
||||
target := filepath.Join(readRoot, "must-not-write.txt")
|
||||
|
||||
args := []string{"exec", "--add-read-dir", readRoot, "--skip-permissions-unsafe", "write the note"}
|
||||
exitCode, stderr := runExecAddDirWriteProbe(t, cwd, args, target)
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("exit = %d, want %d; stderr = %q", exitCode, exitSuccess, stderr)
|
||||
}
|
||||
if _, err := os.Stat(target); !os.IsNotExist(err) {
|
||||
t.Fatalf("a write inside a --add-read-dir root landed — a read grant escalated to write (stat err=%v)", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecScopeReRegistrationSwapsCoreToolsByName(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
extra := tempDirOutsideDefaultTemp(t)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
@@ -18,11 +19,15 @@ type execSessionRecorder struct {
|
||||
// captureCheckpoint snapshots the before-state of files a tool call will mutate.
|
||||
// Best-effort: failures never affect the run. Returns the checkpoint event and
|
||||
// true when one was recorded.
|
||||
func (recorder *execSessionRecorder) captureCheckpoint(workspaceRoot string, registry *tools.Registry, call agent.ToolCall) (sessions.Event, bool) {
|
||||
func (recorder *execSessionRecorder) captureCheckpoint(workspaceRoot string, call agent.ToolCall) (sessions.Event, bool) {
|
||||
if recorder.prepared.Store == nil || recorder.prepared.Session.SessionID == "" {
|
||||
return sessions.Event{}, false
|
||||
}
|
||||
targets := agent.CheckpointTargets(workspaceRoot, registry, call)
|
||||
var args map[string]any
|
||||
if call.Arguments != "" {
|
||||
_ = json.Unmarshal([]byte(call.Arguments), &args)
|
||||
}
|
||||
targets := tools.MutationTargets(workspaceRoot, call.Name, args)
|
||||
if len(targets) == 0 {
|
||||
return sessions.Event{}, false
|
||||
}
|
||||
|
||||
@@ -12,7 +12,6 @@ import (
|
||||
"github.com/Twigpine/zero/internal/execution"
|
||||
"github.com/Twigpine/zero/internal/modelregistry"
|
||||
"github.com/Twigpine/zero/internal/notify"
|
||||
"github.com/Twigpine/zero/internal/providercatalog"
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
"github.com/Twigpine/zero/internal/sessions"
|
||||
"github.com/Twigpine/zero/internal/specmode"
|
||||
@@ -124,7 +123,6 @@ func runExecSpecDraft(run execSpecDraftRun) int {
|
||||
SessionID: draftSession.SessionID,
|
||||
SessionTitle: run.sessionTitle,
|
||||
ProviderName: run.resolved.Provider.Name,
|
||||
ModelFamily: providercatalog.ModelFamilyFor(run.resolved.Provider.CatalogID),
|
||||
Model: run.resolved.Provider.Model,
|
||||
SupportsVision: func(modelID string) bool {
|
||||
return modelregistry.SupportsVision(run.modelRegistry, modelID)
|
||||
@@ -150,7 +148,7 @@ func runExecSpecDraft(run execSpecDraftRun) int {
|
||||
"name": call.Name,
|
||||
"arguments": call.Arguments,
|
||||
})
|
||||
if checkpoint, ok := sessionRecorder.captureCheckpoint(run.workspaceRoot, run.registry, call); ok {
|
||||
if checkpoint, ok := sessionRecorder.captureCheckpoint(run.workspaceRoot, call); ok {
|
||||
writer.checkpoint(checkpoint)
|
||||
}
|
||||
},
|
||||
|
||||
@@ -1,88 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/agent"
|
||||
)
|
||||
|
||||
// THE CHILD MUST PUBLISH WHAT MAKES ITS TURN PRICEABLE.
|
||||
//
|
||||
// This writer is the only channel a parent has. Its session record already keeps
|
||||
// cachedInputTokens / cacheWriteTokens / reasoningTokens so a turn can be costed
|
||||
// exactly; the stream carried prompt/completion/total alone, so every sub-agent
|
||||
// turn rolled up to its parent was priced as if nothing had been cached.
|
||||
//
|
||||
// A measured plan task had 49,280 of 49,894 prompt tokens served from cache —
|
||||
// 98.8%. Plan tasks are the ideal cache case: one large stable prompt, re-sent
|
||||
// every turn.
|
||||
func TestTheChildPublishesCacheAndReasoningTokensOnTheUsageStream(t *testing.T) {
|
||||
var stdout, stderr bytes.Buffer
|
||||
writer := execEventWriter{
|
||||
stdout: &stdout,
|
||||
stderr: &stderr,
|
||||
format: execOutputStreamJSON,
|
||||
runID: "run_usage",
|
||||
streamedText: &strings.Builder{},
|
||||
}
|
||||
writer.usage(agent.Usage{
|
||||
PromptTokens: 30000,
|
||||
CompletionTokens: 500,
|
||||
CachedInputTokens: 29000,
|
||||
CacheWriteTokens: 1000,
|
||||
ReasoningTokens: 120,
|
||||
})
|
||||
if writer.err != nil {
|
||||
t.Fatalf("usage: %v", writer.err)
|
||||
}
|
||||
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal([]byte(strings.TrimSpace(stdout.String())), &payload); err != nil {
|
||||
t.Fatalf("decode stream event %q: %v", stdout.String(), err)
|
||||
}
|
||||
for key, want := range map[string]float64{
|
||||
"cachedInputTokens": 29000, "cacheWriteTokens": 1000, "reasoningTokens": 120,
|
||||
} {
|
||||
got, ok := payload[key]
|
||||
if !ok {
|
||||
t.Errorf("the stream omits %q, so a parent prices this turn as uncached: %v", key, payload)
|
||||
continue
|
||||
}
|
||||
if number, ok := got.(float64); !ok || number != want {
|
||||
t.Errorf("%s: want %v, got %v", key, want, got)
|
||||
}
|
||||
}
|
||||
// The three original fields must be exactly as before.
|
||||
for key, want := range map[string]float64{
|
||||
"promptTokens": 30000, "completionTokens": 500, "totalTokens": 30500,
|
||||
} {
|
||||
if got, ok := payload[key].(float64); !ok || got != want {
|
||||
t.Errorf("%s changed: want %v, got %v", key, want, payload[key])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// NON-ZERO ONLY, exactly like usage.EventUsagePayload. A provider reporting no
|
||||
// cache must produce the same three-field event it always did, or every existing
|
||||
// reader of this stream sees a shape it has never seen.
|
||||
func TestAProviderWithNoCacheStillEmitsTheOriginalUsageShape(t *testing.T) {
|
||||
var stdout, stderr bytes.Buffer
|
||||
writer := execEventWriter{
|
||||
stdout: &stdout, stderr: &stderr, format: execOutputStreamJSON,
|
||||
runID: "run_usage", streamedText: &strings.Builder{},
|
||||
}
|
||||
writer.usage(agent.Usage{PromptTokens: 100, CompletionTokens: 10})
|
||||
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal([]byte(strings.TrimSpace(stdout.String())), &payload); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
for _, key := range []string{"cachedInputTokens", "cacheWriteTokens", "reasoningTokens"} {
|
||||
if _, present := payload[key]; present {
|
||||
t.Errorf("%q was emitted as a zero, widening the event for every provider that reports no cache", key)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -282,25 +282,15 @@ func (writer *execEventWriter) usage(usage agent.Usage) {
|
||||
cachedInputTokens := usage.CachedInputTokens
|
||||
cacheWriteTokens := usage.CacheWriteTokens
|
||||
totalTokens := usage.TotalTokens()
|
||||
event := streamjson.Event{
|
||||
Type: streamjson.EventUsage,
|
||||
RunID: writer.runID,
|
||||
PromptTokens: &promptTokens,
|
||||
CompletionTokens: &completionTokens,
|
||||
TotalTokens: &totalTokens,
|
||||
}
|
||||
// Preserve the provider breakdown when it exists without widening the
|
||||
// event shape for providers that report no cache or reasoning usage.
|
||||
if cachedInputTokens > 0 {
|
||||
event.CachedInputTokens = &cachedInputTokens
|
||||
}
|
||||
if cacheWriteTokens > 0 {
|
||||
event.CacheWriteTokens = &cacheWriteTokens
|
||||
}
|
||||
if reasoning := usage.ReasoningTokens; reasoning > 0 {
|
||||
event.ReasoningTokens = &reasoning
|
||||
}
|
||||
writer.writeStreamJSON(event)
|
||||
writer.writeStreamJSON(streamjson.Event{
|
||||
Type: streamjson.EventUsage,
|
||||
RunID: writer.runID,
|
||||
PromptTokens: &promptTokens,
|
||||
CompletionTokens: &completionTokens,
|
||||
CachedInputTokens: &cachedInputTokens,
|
||||
CacheWriteTokens: &cacheWriteTokens,
|
||||
TotalTokens: &totalTokens,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,457 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/agent"
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/execprofile"
|
||||
"github.com/Twigpine/zero/internal/modelregistry"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
)
|
||||
|
||||
// (a) CLI: an explicit --reasoning-effort survives zeromaxing. The profile
|
||||
// fills only what the caller left unset, exactly as fast/thorough do.
|
||||
func TestZeromaxingDoesNotOverrideExplicitReasoningEffortCLI(t *testing.T) {
|
||||
options := execOptions{execProfile: execprofile.Name, reasoningEffort: "low"}
|
||||
profile, effortFilled, err := applyExecProfile(&options)
|
||||
if err != nil {
|
||||
t.Fatalf("applyExecProfile: %v", err)
|
||||
}
|
||||
if !profile.IsZeromaxing() {
|
||||
t.Fatalf("profile = %q, want %q", profile.Name, execprofile.Name)
|
||||
}
|
||||
if options.reasoningEffort != "low" {
|
||||
t.Fatalf("reasoningEffort = %q, the explicit low must survive the posture's high", options.reasoningEffort)
|
||||
}
|
||||
if effortFilled {
|
||||
t.Fatal("must report it did NOT fill the effort, or a mid-run escalation would clear a hand-pinned value")
|
||||
}
|
||||
}
|
||||
|
||||
// (b) CLI: --mode's fills survive zeromaxing. applyExecMode runs first, so its
|
||||
// values read as "set" by the time the profile arrives — precedence is enforced
|
||||
// by ordering, and this pins that the ordering still holds with a fourth profile.
|
||||
func TestZeromaxingDoesNotOverrideModeFillsCLI(t *testing.T) {
|
||||
options := execOptions{mode: "fast", execProfile: execprofile.Name}
|
||||
if err := applyExecMode(&options); err != nil {
|
||||
t.Fatalf("applyExecMode: %v", err)
|
||||
}
|
||||
modeEffort := options.reasoningEffort
|
||||
modeTurns := options.maxTurns
|
||||
profile, _, err := applyExecProfile(&options)
|
||||
if err != nil {
|
||||
t.Fatalf("applyExecProfile: %v", err)
|
||||
}
|
||||
if modeEffort != "" && options.reasoningEffort != modeEffort {
|
||||
t.Fatalf("reasoningEffort = %q, the mode's %q must win", options.reasoningEffort, modeEffort)
|
||||
}
|
||||
if modeTurns > 0 {
|
||||
// Mirror the real call site: a mode's --max-turns fill lands in
|
||||
// options.maxTurns AND flows through config overrides into
|
||||
// resolved.MaxTurns, so both arguments carry it. The profile must then
|
||||
// back off entirely — displaced 0, budget untouched.
|
||||
effective, displaced := applyProfileTurnBudget(profile, options.maxTurns, options.maxTurns)
|
||||
if effective != modeTurns || displaced != 0 {
|
||||
t.Fatalf("turn budget = (%d, displaced %d), the mode's %d must win with nothing displaced",
|
||||
effective, displaced, modeTurns)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// (c) CLI: fills only what is unset; an explicit --max-turns pins the budget so
|
||||
// the profile backs off with nothing displaced.
|
||||
func TestZeromaxingFillsOnlyUnsetCLI(t *testing.T) {
|
||||
options := execOptions{execProfile: execprofile.Name}
|
||||
profile, effortFilled, err := applyExecProfile(&options)
|
||||
if err != nil {
|
||||
t.Fatalf("applyExecProfile: %v", err)
|
||||
}
|
||||
if options.reasoningEffort != "high" || !effortFilled {
|
||||
t.Fatalf("unset effort must be filled with high, got %q filled=%v", options.reasoningEffort, effortFilled)
|
||||
}
|
||||
if !options.selfCorrect {
|
||||
t.Fatal("must arm self-correction when it was unset")
|
||||
}
|
||||
if effective, displaced := applyProfileTurnBudget(profile, 0, 80); effective != 480 || displaced != 80 {
|
||||
t.Fatalf("over resolved 80 = (%d, %d), want (480, 80)", effective, displaced)
|
||||
}
|
||||
if effective, displaced := applyProfileTurnBudget(profile, 50, 50); effective != 50 || displaced != 0 {
|
||||
t.Fatalf("with explicit --max-turns 50 = (%d, %d), want (50, 0)", effective, displaced)
|
||||
}
|
||||
}
|
||||
|
||||
// (g) CLI: --reasoning-effort zeromaxing and --exec-profile zeromaxing resolve
|
||||
// to IDENTICAL state. One posture, one name, reachable from either flag.
|
||||
func TestZeromaxingEffortFlagResolvesLikeProfileFlagCLI(t *testing.T) {
|
||||
viaEffort := execOptions{reasoningEffort: execprofile.Name}
|
||||
if err := normalizeZeromaxingEffort(&viaEffort); err != nil {
|
||||
t.Fatalf("normalize: %v", err)
|
||||
}
|
||||
viaProfile := execOptions{execProfile: execprofile.Name}
|
||||
if err := normalizeZeromaxingEffort(&viaProfile); err != nil {
|
||||
t.Fatalf("normalize: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(viaEffort, viaProfile) {
|
||||
t.Fatalf("the two entry points diverged:\n--reasoning-effort: %+v\n--exec-profile: %+v", viaEffort, viaProfile)
|
||||
}
|
||||
// ...and after profile expansion they are still identical.
|
||||
pe, fe, err := applyExecProfile(&viaEffort)
|
||||
if err != nil {
|
||||
t.Fatalf("applyExecProfile(effort path): %v", err)
|
||||
}
|
||||
pp, fp, err := applyExecProfile(&viaProfile)
|
||||
if err != nil {
|
||||
t.Fatalf("applyExecProfile(profile path): %v", err)
|
||||
}
|
||||
if pe != pp || fe != fp || !reflect.DeepEqual(viaEffort, viaProfile) {
|
||||
t.Fatalf("resolved state diverged after expansion:\n%+v %v\n%+v %v", pe, fe, pp, fp)
|
||||
}
|
||||
if viaEffort.reasoningEffort != "high" {
|
||||
t.Fatalf("both paths must resolve the effort to the profile's high, got %q", viaEffort.reasoningEffort)
|
||||
}
|
||||
}
|
||||
|
||||
// A conflicting pair is a usage error, not a silent winner.
|
||||
func TestZeromaxingEffortFlagConflictIsUsageError(t *testing.T) {
|
||||
options := execOptions{reasoningEffort: execprofile.Name, execProfile: "fast"}
|
||||
err := normalizeZeromaxingEffort(&options)
|
||||
if err == nil {
|
||||
t.Fatal("--reasoning-effort zeromaxing with --exec-profile fast must be a usage error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "conflicts") {
|
||||
t.Fatalf("the error must name the conflict: %v", err)
|
||||
}
|
||||
// The same profile on both flags is not a conflict.
|
||||
same := execOptions{reasoningEffort: execprofile.Name, execProfile: execprofile.Name}
|
||||
if err := normalizeZeromaxingEffort(&same); err != nil {
|
||||
t.Fatalf("naming the same posture twice must be accepted: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// (e) THE LOAD-BEARING GUARD. The posture name must never be forwarded to a
|
||||
// provider as an effort value — not by any path, not for any model.
|
||||
func TestZeromaxingIsNeverForwardedToAProvider(t *testing.T) {
|
||||
registry, err := modelregistry.DefaultRegistry()
|
||||
if err != nil {
|
||||
t.Fatalf("DefaultRegistry: %v", err)
|
||||
}
|
||||
// Across a reasoning model, a non-reasoning model, and a model the catalog
|
||||
// has never heard of (where unknown values are otherwise passed through).
|
||||
for _, model := range []string{"claude-sonnet-4.5", "gpt-4.1", "some-custom-endpoint-model"} {
|
||||
for _, spelling := range []string{"zeromaxing", "ZEROMAXING", " Zeromaxing "} {
|
||||
if got := forwardedReasoningEffort(registry, model, spelling); got != "" {
|
||||
t.Fatalf("forwardedReasoningEffort(%q, %q) = %q, want \"\" — the posture name is not a provider value",
|
||||
model, spelling, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
// The guard must not over-reach: real levels still forward.
|
||||
if got := forwardedReasoningEffort(registry, "claude-sonnet-4.5", "high"); got != "high" {
|
||||
t.Fatalf("a real level must still forward, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// (f) THE RESERVATION. /effort max and --reasoning-effort max are UNCHANGED by
|
||||
// this feature: "max" still parses as a raw provider level (ValidReasoningEffort
|
||||
// accepts ReasoningEffortMax) and still resolves to nothing usable on today's
|
||||
// models. The spelling stays free for a real provider rung.
|
||||
func TestEffortMaxReservationUnchangedCLI(t *testing.T) {
|
||||
if !modelregistry.ValidReasoningEffort(modelregistry.ReasoningEffortMax) {
|
||||
t.Fatal("ReasoningEffortMax must remain a valid raw effort value — the reservation depends on it")
|
||||
}
|
||||
if _, ok := execprofile.Lookup("max"); ok {
|
||||
t.Fatal("\"max\" must NOT resolve to a profile — it is reserved for a provider rung")
|
||||
}
|
||||
registry, err := modelregistry.DefaultRegistry()
|
||||
if err != nil {
|
||||
t.Fatalf("DefaultRegistry: %v", err)
|
||||
}
|
||||
// No curated model lists "max", so it coerces away rather than forwarding.
|
||||
if got := forwardedReasoningEffort(registry, "claude-sonnet-4.5", "max"); got == "max" {
|
||||
t.Fatal("no current model supports \"max\"; it must not be forwarded as-is")
|
||||
}
|
||||
// And it must NOT be swallowed by the zeromaxing normalizer.
|
||||
options := execOptions{reasoningEffort: "max"}
|
||||
if err := normalizeZeromaxingEffort(&options); err != nil {
|
||||
t.Fatalf("normalize: %v", err)
|
||||
}
|
||||
if options.reasoningEffort != "max" || options.execProfile != "" {
|
||||
t.Fatalf("\"max\" must pass through the normalizer untouched, got effort=%q profile=%q",
|
||||
options.reasoningEffort, options.execProfile)
|
||||
}
|
||||
}
|
||||
|
||||
// (l) CLI leg of the honesty rule: a model that cannot take the effort is TOLD.
|
||||
// reasoningEffortNotice is the existing coerce-and-tell helper, and it fires for
|
||||
// a PROFILE-FILLED effort, not just an explicitly flagged one.
|
||||
func TestZeromaxingUnsupportedEffortIsReportedCLI(t *testing.T) {
|
||||
registry, err := modelregistry.DefaultRegistry()
|
||||
if err != nil {
|
||||
t.Fatalf("DefaultRegistry: %v", err)
|
||||
}
|
||||
const nonReasoning = "gpt-4.1" // a catalog model with no reasoning capability
|
||||
notice := reasoningEffortNotice(registry, nonReasoning, "high")
|
||||
if notice == "" {
|
||||
t.Fatal("a non-reasoning model must produce a notice when an effort is requested")
|
||||
}
|
||||
if !strings.Contains(notice, "reasoning effort") {
|
||||
t.Fatalf("the notice must explain what was not applied: %q", notice)
|
||||
}
|
||||
if forwarded := forwardedReasoningEffort(registry, nonReasoning, "high"); forwarded != "" {
|
||||
t.Fatalf("forwarded effort = %q, want empty for a non-reasoning model", forwarded)
|
||||
}
|
||||
// ...and the rest of the posture still applies: the budget is unaffected.
|
||||
profile, _ := execprofile.Lookup(execprofile.Name)
|
||||
if effective, _ := applyProfileTurnBudget(profile, 0, 80); effective != 480 {
|
||||
t.Fatalf("the turn budget must still apply on an unsupported model, got %d", effective)
|
||||
}
|
||||
}
|
||||
|
||||
// (m)+(n) CLI leg of the config gate.
|
||||
func TestZeromaxingSelectionRefusalCLI(t *testing.T) {
|
||||
profile, _ := execprofile.Lookup(execprofile.Name)
|
||||
if refusal := execprofile.SelectionRefusal(profile, true); refusal == "" {
|
||||
t.Fatal("exec must refuse the posture when config disabled it")
|
||||
}
|
||||
if refusal := execprofile.SelectionRefusal(profile, false); refusal != "" {
|
||||
t.Fatalf("exec must allow it when config did not disable it, got %q", refusal)
|
||||
}
|
||||
}
|
||||
|
||||
// (o) The raised budget PROPAGATES to spawned children. Deliberate — this is a
|
||||
// maximal posture — so it is asserted explicitly rather than left to be
|
||||
// discovered. applyProfileTurnBudget's effective value becomes resolved.MaxTurns,
|
||||
// which is what the run exports as ZERO_MAX_TURNS for sub-agents.
|
||||
func TestZeromaxingTurnBudgetPropagatesToChildren(t *testing.T) {
|
||||
profile, _ := execprofile.Lookup(execprofile.Name)
|
||||
effective, _ := applyProfileTurnBudget(profile, 0, 80)
|
||||
if effective != 480 {
|
||||
t.Fatalf("effective budget = %d, want 480", effective)
|
||||
}
|
||||
if effective > config.MaxTurnsCeiling {
|
||||
t.Fatalf("the budget %d exceeds the shared ceiling %d", effective, config.MaxTurnsCeiling)
|
||||
}
|
||||
// The delta text promises exactly this, so the promise and the number must
|
||||
// not drift apart.
|
||||
delta := execprofile.Delta(execprofile.DeltaState{CurrentMaxTurns: 80})
|
||||
if !strings.Contains(delta, "sub-agents") {
|
||||
t.Fatalf("the delta must tell the user the budget reaches sub-agents: %q", delta)
|
||||
}
|
||||
}
|
||||
|
||||
// The headless posture mapping: selecting it enters; everything else is off.
|
||||
func TestExecZeromaxingMapping(t *testing.T) {
|
||||
profile, _ := execprofile.Lookup(execprofile.Name)
|
||||
if got := execZeromaxing(profile); got != agent.ZeromaxingEntering {
|
||||
t.Fatalf("execZeromaxing(zeromaxing) = %v, want ZeromaxingEntering", got)
|
||||
}
|
||||
for _, name := range []string{"balanced", "fast", "thorough"} {
|
||||
other, _ := execprofile.Lookup(name)
|
||||
if got := execZeromaxing(other); got != agent.ZeromaxingOff {
|
||||
t.Fatalf("execZeromaxing(%s) = %v, want ZeromaxingOff", name, got)
|
||||
}
|
||||
}
|
||||
if got := execZeromaxing(execprofile.Profile{}); got != agent.ZeromaxingOff {
|
||||
t.Fatalf("execZeromaxing(zero) = %v, want ZeromaxingOff", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The flag PARSER must accept the posture name, and must still reject "max".
|
||||
//
|
||||
// This test exists because the unit tests above call normalizeZeromaxingEffort
|
||||
// directly and never reach the parser — which rejected "zeromaxing" outright, so
|
||||
// the whole entry point was dead at the user surface while every unit test
|
||||
// passed. Driving the real binary is what found it; this is the regression.
|
||||
func TestReasoningEffortFlagAcceptsZeromaxingButNotMax(t *testing.T) {
|
||||
accepted := func(t *testing.T, flag, value string) error {
|
||||
t.Helper()
|
||||
_, _, err := parseExecArgs([]string{flag, value, "-p", "x"})
|
||||
return err
|
||||
}
|
||||
if err := accepted(t, "--reasoning-effort", execprofile.Name); err != nil {
|
||||
t.Fatalf("--reasoning-effort %s must be accepted by the parser: %v", execprofile.Name, err)
|
||||
}
|
||||
if err := accepted(t, "--reasoning-effort", "ZEROMAXING"); err != nil {
|
||||
t.Fatalf("the parser must be case-insensitive: %v", err)
|
||||
}
|
||||
for _, level := range []string{"low", "medium", "high"} {
|
||||
if err := accepted(t, "--reasoning-effort", level); err != nil {
|
||||
t.Fatalf("--reasoning-effort %s must still be accepted: %v", level, err)
|
||||
}
|
||||
}
|
||||
// THE RESERVATION: "max" was rejected before this feature and must stay
|
||||
// rejected. Accepting it here would burn the spelling.
|
||||
err := accepted(t, "--reasoning-effort", "max")
|
||||
if err == nil {
|
||||
t.Fatal("--reasoning-effort max must still be rejected — the spelling is reserved")
|
||||
}
|
||||
if !strings.Contains(err.Error(), execprofile.Name) {
|
||||
t.Fatalf("the usage error should name the accepted values: %v", err)
|
||||
}
|
||||
// The spec-draft effort is a plain provider level; the posture has no
|
||||
// meaning for a draft, so it is NOT accepted there.
|
||||
if err := accepted(t, "--spec-reasoning-effort", execprofile.Name); err == nil {
|
||||
t.Fatalf("--spec-reasoning-effort %s must be rejected; it is a run posture, not a draft level", execprofile.Name)
|
||||
}
|
||||
}
|
||||
|
||||
// The headless delta must describe the transition from the state the run was
|
||||
// invoked in, captured BEFORE the profile arms self-correction as a side
|
||||
// effect. Reading options.selfCorrect afterwards would always say "already on".
|
||||
func TestExecSelfCorrectTransitionUsesPreProfileState(t *testing.T) {
|
||||
if got := execSelfCorrectTransition(false); got != execprofile.SelfCorrectRaised {
|
||||
t.Fatalf("an unflagged run = %v, want SelfCorrectRaised", got)
|
||||
}
|
||||
if got := execSelfCorrectTransition(true); got != execprofile.SelfCorrectAlreadyOn {
|
||||
t.Fatalf("an explicit --self-correct run = %v, want SelfCorrectAlreadyOn", got)
|
||||
}
|
||||
// And the rendered text differs, so the distinction reaches the user.
|
||||
raised := execprofile.Delta(execprofile.DeltaState{CurrentMaxTurns: 80, SelfCorrect: execSelfCorrectTransition(false)})
|
||||
already := execprofile.Delta(execprofile.DeltaState{CurrentMaxTurns: 80, SelfCorrect: execSelfCorrectTransition(true)})
|
||||
if raised == already {
|
||||
t.Fatalf("both states render identically:\n%s", raised)
|
||||
}
|
||||
if !strings.Contains(raised, "lsp → tests") {
|
||||
t.Fatalf("an unflagged run must be told what changes:\n%s", raised)
|
||||
}
|
||||
}
|
||||
|
||||
// --spec-reasoning-effort zeromaxing stays rejected, but the message must
|
||||
// explain WHY and name the way forward rather than reading like a bug.
|
||||
func TestSpecReasoningEffortZeromaxingExplainsItself(t *testing.T) {
|
||||
_, _, err := parseExecArgs([]string{"--spec-reasoning-effort", execprofile.Name, "-p", "x"})
|
||||
if err == nil {
|
||||
t.Fatalf("--spec-reasoning-effort %s must still be rejected", execprofile.Name)
|
||||
}
|
||||
message := err.Error()
|
||||
for _, want := range []string{
|
||||
"run posture", // why it does not apply
|
||||
"spec drafting", // where the user is
|
||||
"--spec-reasoning-effort high", // the way forward for the draft
|
||||
"--reasoning-effort " + execprofile.Name, // ...and for the run
|
||||
} {
|
||||
if !strings.Contains(message, want) {
|
||||
t.Fatalf("the usage error must mention %q, got: %s", want, message)
|
||||
}
|
||||
}
|
||||
// A genuinely unknown value keeps the plain message — the explanation is
|
||||
// specific to the posture name, not a new blanket wording.
|
||||
_, _, err = parseExecArgs([]string{"--spec-reasoning-effort", "turbo", "-p", "x"})
|
||||
if err == nil || !strings.Contains(err.Error(), "Expected low, medium, or high.") {
|
||||
t.Fatalf("an unknown value must keep the plain message, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The CAPTURE POINT, exercised through the real exec path.
|
||||
//
|
||||
// TestExecSelfCorrectTransitionUsesPreProfileState covers the helper; it cannot
|
||||
// catch the capture being read at the wrong moment, because it never runs the
|
||||
// code that does the capturing. applyExecProfile arms self-correction as a side
|
||||
// effect, so reading options.selfCorrect after it would make every run report
|
||||
// "unchanged (tests)" — the exact wording this change exists to remove.
|
||||
func TestExecPrintsTheRaisedTransitionForAnUnflaggedRun(t *testing.T) {
|
||||
exitCode, _, stderr := runExecWithEcho(t, []string{
|
||||
"exec", "--exec-profile", execprofile.Name, "hello",
|
||||
})
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("expected exit %d, got %d: %s", exitSuccess, exitCode, stderr)
|
||||
}
|
||||
if !strings.Contains(stderr, "self-correct: lsp → tests") {
|
||||
t.Fatalf("an unflagged run starts LSP-only, so the delta must show the transition:\n%s", stderr)
|
||||
}
|
||||
if strings.Contains(stderr, "self-correct: unchanged") {
|
||||
t.Fatalf("the delta must not claim self-correction was already on:\n%s", stderr)
|
||||
}
|
||||
}
|
||||
|
||||
// ...and the other direction: an explicit --self-correct run genuinely was
|
||||
// already on, so it must say so.
|
||||
func TestExecPrintsUnchangedWhenSelfCorrectWasAlreadyOn(t *testing.T) {
|
||||
exitCode, _, stderr := runExecWithEcho(t, []string{
|
||||
"exec", "--exec-profile", execprofile.Name, "--self-correct", "hello",
|
||||
})
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("expected exit %d, got %d: %s", exitSuccess, exitCode, stderr)
|
||||
}
|
||||
if !strings.Contains(stderr, "self-correct: unchanged (tests)") {
|
||||
t.Fatalf("an explicit --self-correct run must be told nothing changed:\n%s", stderr)
|
||||
}
|
||||
if strings.Contains(stderr, "lsp → tests") {
|
||||
t.Fatalf("must not claim a transition that did not happen:\n%s", stderr)
|
||||
}
|
||||
}
|
||||
|
||||
// (3) The budget clause the USER sees must be caller-relative. Asserted through
|
||||
// the real exec path: a unit test on budgetLine cannot catch the caller passing
|
||||
// the wrong number into it, which is exactly what mutation found.
|
||||
func TestExecDeltaShowsTheCallersOwnTurnBudget(t *testing.T) {
|
||||
exitCode, _, stderr := runExecWithEcho(t, []string{
|
||||
"exec", "--exec-profile", execprofile.Name, "hello",
|
||||
})
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("expected exit %d, got %d: %s", exitSuccess, exitCode, stderr)
|
||||
}
|
||||
// Assert the SHAPE, not a specific origin: the harness's resolved budget is
|
||||
// a fixture value, and pinning it here would test the fixture rather than
|
||||
// the behaviour. What matters is that the origin is the caller's own
|
||||
// resolved budget and the destination is the posture's.
|
||||
transition := regexp.MustCompile(`turn budget: (\d+) → 480`)
|
||||
match := transition.FindStringSubmatch(stderr)
|
||||
if match == nil {
|
||||
t.Fatalf("the delta must state a caller-relative budget transition:\n%s", stderr)
|
||||
}
|
||||
if match[1] == "480" {
|
||||
t.Fatalf("origin and destination are the same; the clause should read \"unchanged\":\n%s", stderr)
|
||||
}
|
||||
if strings.Contains(stderr, "160") {
|
||||
t.Fatalf("the delta must not compare against thorough's budget:\n%s", stderr)
|
||||
}
|
||||
if n := strings.Count(stderr, "reasoning effort:"); n != 1 {
|
||||
t.Fatalf("exactly one reasoning-effort statement expected, found %d:\n%s", n, stderr)
|
||||
}
|
||||
}
|
||||
|
||||
// A partial plan must reach the PROCESS exit code, not just the tool result.
|
||||
//
|
||||
// Asserting the executor's status missed the tool's status; asserting the
|
||||
// tool's status would miss the exit code the same way. This is the third link,
|
||||
// asserted at the boundary that actually matters to a caller.
|
||||
func TestPlanPartialMapsToIncompleteExit(t *testing.T) {
|
||||
recorder := &planSessionRecorder{}
|
||||
if _, incomplete := recorder.Incomplete(); incomplete {
|
||||
t.Fatal("a recorder that saw no plan must not mark the run incomplete")
|
||||
}
|
||||
plan, err := specialist.ParsePlan(map[string]any{
|
||||
"name": "p",
|
||||
"tasks": []any{map[string]any{"id": "a", "prompt": "x"}},
|
||||
"budget": map[string]any{"max_workers": float64(1), "max_tokens": float64(100_000)},
|
||||
}, specialist.Limits{MaxTasks: 5})
|
||||
if err != nil {
|
||||
t.Fatalf("ParsePlan: %v", err)
|
||||
}
|
||||
// A COMPLETED plan leaves the run alone.
|
||||
recorder.PlanCompleted(plan, specialist.PlanReport{Status: specialist.PlanCompleted, Succeeded: 1})
|
||||
if _, incomplete := recorder.Incomplete(); incomplete {
|
||||
t.Fatal("a completed plan must not mark the run incomplete")
|
||||
}
|
||||
// A PARTIAL one does, and names the counts.
|
||||
recorder.PlanCompleted(plan, specialist.PlanReport{Status: specialist.PlanPartial, Succeeded: 1, Skipped: 2})
|
||||
reason, incomplete := recorder.Incomplete()
|
||||
if !incomplete {
|
||||
t.Fatal("a partial plan must mark the run incomplete, which is exit 4")
|
||||
}
|
||||
for _, want := range []string{"partial", "1 succeeded", "2 skipped"} {
|
||||
if !strings.Contains(reason, want) {
|
||||
t.Fatalf("the reason must carry %q: %q", want, reason)
|
||||
}
|
||||
}
|
||||
// FAILED outranks partial: if any plan failed outright, that is the story.
|
||||
recorder.PlanCompleted(plan, specialist.PlanReport{Status: specialist.PlanFailed})
|
||||
if reason, _ := recorder.Incomplete(); !strings.Contains(reason, "failed") {
|
||||
t.Fatalf("a failed plan must outrank a partial one: %q", reason)
|
||||
}
|
||||
}
|
||||
@@ -2,7 +2,6 @@ package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
@@ -83,105 +82,11 @@ func runPlugins(args []string, stdout io.Writer, stderr io.Writer, deps appDeps)
|
||||
return runPluginInfo(args[1:], stdout, stderr, deps)
|
||||
case "remove", "rm":
|
||||
return runPluginRemove(args[1:], deps.pluginsDir(), stdout, stderr)
|
||||
case "enable":
|
||||
return runPluginsToggle(args[1:], stdout, stderr, deps, false)
|
||||
case "disable":
|
||||
return runPluginsToggle(args[1:], stdout, stderr, deps, true)
|
||||
default:
|
||||
return writeExecUsageError(stderr, fmt.Sprintf("unknown plugins subcommand %q", args[0]))
|
||||
}
|
||||
}
|
||||
|
||||
type pluginToggleOptions struct {
|
||||
json bool
|
||||
user bool
|
||||
}
|
||||
|
||||
// runPluginsToggle implements `zero plugins enable|disable <id>`.
|
||||
//
|
||||
// Every message it writes, including usage errors that echo the caller's
|
||||
// arguments, passes through the redaction pipeline: an argument is user input
|
||||
// and can be a mistyped credential.
|
||||
func runPluginsToggle(args []string, stdout io.Writer, stderr io.Writer, deps appDeps, disabled bool) int {
|
||||
commandName := "enable"
|
||||
if disabled {
|
||||
commandName = "disable"
|
||||
}
|
||||
usageError := func(message string) int {
|
||||
return writeExecUsageError(stderr, redaction.RedactString(message, redaction.Options{}))
|
||||
}
|
||||
options, positional, help, err := parsePluginToggleArgs(args, commandName)
|
||||
if err != nil {
|
||||
return usageError(err.Error())
|
||||
}
|
||||
if help {
|
||||
if err := writePluginsToggleHelp(stdout, commandName); err != nil {
|
||||
return exitCrash
|
||||
}
|
||||
return exitSuccess
|
||||
}
|
||||
if len(positional) != 1 {
|
||||
return usageError(fmt.Sprintf("usage: zero plugins %s <id> [--user] [--json]", commandName))
|
||||
}
|
||||
pluginID := strings.TrimSpace(positional[0])
|
||||
if pluginID == "" {
|
||||
return usageError(fmt.Sprintf("usage: zero plugins %s <id> [--user] [--json]", commandName))
|
||||
}
|
||||
|
||||
cwd, err := deps.getwd()
|
||||
if err != nil {
|
||||
return writeAppError(stderr, "failed to resolve workspace: "+redaction.ErrorMessage(err, redaction.Options{}), exitCrash)
|
||||
}
|
||||
loadOptions := plugins.LoadOptions{Cwd: cwd, ExcludeProject: options.user}
|
||||
result, err := plugins.SetEnabledByID(loadOptions, pluginID, !disabled)
|
||||
if err != nil {
|
||||
if errors.Is(err, plugins.ErrNotInstalled) {
|
||||
return usageError(err.Error())
|
||||
}
|
||||
return writeAppError(stderr, redaction.ErrorMessage(err, redaction.Options{}), exitCrash)
|
||||
}
|
||||
|
||||
state := "enabled"
|
||||
if disabled {
|
||||
state = "disabled"
|
||||
}
|
||||
if options.json {
|
||||
if err := writePrettyJSON(stdout, redaction.RedactValue(result, redaction.Options{})); err != nil {
|
||||
return exitCrash
|
||||
}
|
||||
return exitSuccess
|
||||
}
|
||||
message := fmt.Sprintf("Plugin %s is now %s in %s.\n", result.ID, state, result.ManifestPath)
|
||||
if !result.Changed {
|
||||
message = fmt.Sprintf("Plugin %s was already %s in %s.\n", result.ID, state, result.ManifestPath)
|
||||
}
|
||||
if _, err := fmt.Fprint(stdout, redaction.RedactString(message, redaction.Options{})); err != nil {
|
||||
return exitCrash
|
||||
}
|
||||
return exitSuccess
|
||||
}
|
||||
|
||||
func parsePluginToggleArgs(args []string, command string) (pluginToggleOptions, []string, bool, error) {
|
||||
options := pluginToggleOptions{}
|
||||
positional := []string{}
|
||||
for _, arg := range args {
|
||||
switch arg {
|
||||
case "-h", "--help", "help":
|
||||
return options, positional, true, nil
|
||||
case "--json":
|
||||
options.json = true
|
||||
case "--user":
|
||||
options.user = true
|
||||
default:
|
||||
if strings.HasPrefix(arg, "-") {
|
||||
return options, positional, false, execUsageError{fmt.Sprintf("unknown plugins %s flag %q", command, arg)}
|
||||
}
|
||||
positional = append(positional, arg)
|
||||
}
|
||||
}
|
||||
return options, positional, false, nil
|
||||
}
|
||||
|
||||
func runHooks(args []string, stdout io.Writer, stderr io.Writer, deps appDeps) int {
|
||||
if len(args) == 0 {
|
||||
return writeExecUsageError(stderr, "hooks subcommand required. Use `zero hooks list`.")
|
||||
@@ -661,31 +566,10 @@ Commands:
|
||||
info <id> Show plugin details and lockfile metadata
|
||||
add <git-url|path> Install a plugin (manifest-validated, pinned in plugins.lock)
|
||||
remove <id> Remove an installed plugin and its lockfile entry
|
||||
enable <id> Enable a plugin via its plugin.json
|
||||
disable <id> Disable a plugin via its plugin.json
|
||||
`)
|
||||
return err
|
||||
}
|
||||
|
||||
func writePluginsToggleHelp(w io.Writer, command string) error {
|
||||
_, err := fmt.Fprintf(w, `Usage:
|
||||
zero plugins %s <id> [flags]
|
||||
|
||||
Flags:
|
||||
--user Target only user plugins (ignore ./.zero/plugins)
|
||||
--json Print command result as JSON
|
||||
-h, --help Show this help
|
||||
|
||||
Without --user, a project plugin with the same id takes precedence over a user
|
||||
plugin, matching the loader's discovery order.
|
||||
|
||||
Zero reads plugins when it starts, so the change applies to the next zero exec
|
||||
run or interactive session. A running interactive session keeps the plugins it
|
||||
started with until you restart it.
|
||||
`, command)
|
||||
return err
|
||||
}
|
||||
|
||||
func writePluginsListHelp(w io.Writer) error {
|
||||
_, err := fmt.Fprint(w, `Usage:
|
||||
zero plugins list [flags]
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -334,234 +333,3 @@ func TestRunMCPPermissionsHelpDoesNotOpenStore(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunPluginsEnableDisable(t *testing.T) {
|
||||
cwd := t.TempDir()
|
||||
home := t.TempDir()
|
||||
t.Setenv("XDG_CONFIG_HOME", home)
|
||||
t.Setenv("HOME", home)
|
||||
pluginDir := filepath.Join(cwd, ".zero", "plugins", "demo")
|
||||
if err := os.MkdirAll(pluginDir, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
manifestPath := filepath.Join(pluginDir, "plugin.json")
|
||||
if err := os.WriteFile(manifestPath, []byte(`{
|
||||
"schemaVersion": 1,
|
||||
"id": "zero.demo",
|
||||
"name": "Demo",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
deps := appDeps{getwd: func() (string, error) { return cwd, nil }}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
exitCode := runWithDeps([]string{"plugins", "disable", "zero.demo", "--json"}, &stdout, &stderr, deps)
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("disable exit=%d stderr=%s", exitCode, stderr.String())
|
||||
}
|
||||
var disablePayload plugins.SetEnabledResult
|
||||
if err := json.Unmarshal(stdout.Bytes(), &disablePayload); err != nil {
|
||||
t.Fatalf("decode disable JSON: %v\n%s", err, stdout.String())
|
||||
}
|
||||
if disablePayload.ID != "zero.demo" || disablePayload.Enabled || !disablePayload.Changed {
|
||||
t.Fatalf("unexpected disable payload: %#v", disablePayload)
|
||||
}
|
||||
|
||||
raw, err := os.ReadFile(manifestPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(raw), `"enabled": false`) {
|
||||
t.Fatalf("manifest not disabled: %s", raw)
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
exitCode = runWithDeps([]string{"plugins", "disable", "zero.demo"}, &stdout, &stderr, deps)
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("noop disable exit=%d stderr=%s", exitCode, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "was already disabled") {
|
||||
t.Fatalf("noop disable output: %s", stdout.String())
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
exitCode = runWithDeps([]string{"plugins", "enable", "zero.demo"}, &stdout, &stderr, deps)
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("enable exit=%d stderr=%s", exitCode, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "is now enabled") {
|
||||
t.Fatalf("enable output: %s", stdout.String())
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
exitCode = runWithDeps([]string{"plugins", "enable", "missing-plugin"}, &stdout, &stderr, deps)
|
||||
if exitCode != exitUsage {
|
||||
t.Fatalf("missing plugin exit=%d want %d stderr=%s", exitCode, exitUsage, stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
// The text-mode toggle output runs through the redaction pipeline, so a
|
||||
// manifest path that contains something secret-shaped is masked.
|
||||
func TestRunPluginsToggleTextOutputIsRedacted(t *testing.T) {
|
||||
cwd := t.TempDir()
|
||||
home := t.TempDir()
|
||||
t.Setenv("XDG_CONFIG_HOME", home)
|
||||
t.Setenv("HOME", home)
|
||||
secretDir := "sk-proj-abcdefghijklmnopqrstuvwxyz"
|
||||
pluginDir := filepath.Join(cwd, ".zero", "plugins", secretDir)
|
||||
if err := os.MkdirAll(pluginDir, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
manifestPath := filepath.Join(pluginDir, "plugin.json")
|
||||
if err := os.WriteFile(manifestPath, []byte(`{
|
||||
"schemaVersion": 1,
|
||||
"id": "zero.demo",
|
||||
"name": "Demo",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
deps := appDeps{getwd: func() (string, error) { return cwd, nil }}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
exitCode := runWithDeps([]string{"plugins", "disable", "zero.demo"}, &stdout, &stderr, deps)
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("disable exit=%d stderr=%s", exitCode, stderr.String())
|
||||
}
|
||||
if strings.Contains(stdout.String(), secretDir) {
|
||||
t.Fatalf("toggle text leaked a secret-shaped manifest path: %q", stdout.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "[REDACTED]") {
|
||||
t.Fatalf("toggle text was not redacted: %q", stdout.String())
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
exitCode = runWithDeps([]string{"plugins", "disable", "zero.demo"}, &stdout, &stderr, deps)
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("noop disable exit=%d stderr=%s", exitCode, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "was already disabled") {
|
||||
t.Fatalf("noop toggle output: %q", stdout.String())
|
||||
}
|
||||
if strings.Contains(stdout.String(), secretDir) {
|
||||
t.Fatalf("noop toggle text leaked a secret-shaped manifest path: %q", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunPluginsToggleRejectsEmptyID(t *testing.T) {
|
||||
deps := appDeps{getwd: func() (string, error) { return t.TempDir(), nil }}
|
||||
|
||||
for _, subcommand := range []string{"enable", "disable"} {
|
||||
t.Run(subcommand, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
exitCode := runWithDeps([]string{"plugins", subcommand, ""}, &stdout, &stderr, deps)
|
||||
if exitCode != exitUsage {
|
||||
t.Fatalf("exit=%d want %d stderr=%s", exitCode, exitUsage, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "usage: zero plugins "+subcommand) {
|
||||
t.Fatalf("stderr = %q", stderr.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunPluginsDisableUserIgnoresProject(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
cwd := t.TempDir()
|
||||
t.Setenv("XDG_CONFIG_HOME", home)
|
||||
t.Setenv("HOME", home)
|
||||
|
||||
userManifest := filepath.Join(home, "zero", "plugins", "demo", "plugin.json")
|
||||
projectManifest := filepath.Join(cwd, ".zero", "plugins", "demo", "plugin.json")
|
||||
for _, path := range []string{userManifest, projectManifest} {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(`{
|
||||
"schemaVersion": 1,
|
||||
"id": "zero.demo",
|
||||
"name": "Demo",
|
||||
"version": "1.0.0",
|
||||
"enabled": true
|
||||
}
|
||||
`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
deps := appDeps{getwd: func() (string, error) { return cwd, nil }}
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
exitCode := runWithDeps([]string{"plugins", "disable", "zero.demo", "--user"}, &stdout, &stderr, deps)
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("disable --user exit=%d stderr=%s", exitCode, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "is now disabled") {
|
||||
t.Fatalf("disable --user output: %s", stdout.String())
|
||||
}
|
||||
|
||||
userRaw, err := os.ReadFile(userManifest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(userRaw), `"enabled": false`) {
|
||||
t.Fatalf("user manifest not disabled: %s", userRaw)
|
||||
}
|
||||
projectRaw, err := os.ReadFile(projectManifest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(string(projectRaw), `"enabled": false`) {
|
||||
t.Fatalf("project manifest should stay enabled: %s", projectRaw)
|
||||
}
|
||||
}
|
||||
|
||||
// The toggle's usage errors echo the caller's arguments, which can be a
|
||||
// mistyped credential, so they pass through redaction like its other output.
|
||||
func TestRunPluginsToggleUsageErrorsAreRedacted(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
t.Setenv("XDG_CONFIG_HOME", home)
|
||||
t.Setenv("HOME", home)
|
||||
deps := appDeps{getwd: func() (string, error) { return t.TempDir(), nil }}
|
||||
const secret = "sk-proj-abcdefghijklmnopqrstuvwxyz"
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
{"unknown flag", []string{"zero.demo", "--token=" + secret}, "unknown plugins"},
|
||||
{"missing plugin", []string{secret}, "plugin is not installed"},
|
||||
}
|
||||
for _, subcommand := range []string{"enable", "disable"} {
|
||||
for _, tc := range cases {
|
||||
t.Run(subcommand+" "+tc.name, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
args := append([]string{"plugins", subcommand}, tc.args...)
|
||||
exitCode := runWithDeps(args, &stdout, &stderr, deps)
|
||||
if exitCode != exitUsage {
|
||||
t.Fatalf("exit=%d want %d stderr=%s", exitCode, exitUsage, stderr.String())
|
||||
}
|
||||
if strings.Contains(stderr.String(), secret) {
|
||||
t.Fatalf("usage error leaked the argument: %q", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), tc.want) || !strings.Contains(stderr.String(), "[REDACTED]") {
|
||||
t.Fatalf("stderr = %q, want a redacted %q error", stderr.String(), tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -319,7 +319,7 @@ Flags:
|
||||
--command <cmd> Command to run (required)
|
||||
--name <name> Human-readable hook name
|
||||
--description <text> Hook description
|
||||
--matcher <pattern> Tool/specialist matcher (beforeTool/afterTool/specialistStart/specialistStop)
|
||||
--matcher <pattern> Tool matcher (beforeTool/afterTool only)
|
||||
--arg <value> Command argument (repeatable)
|
||||
--user Write to user config instead of the project
|
||||
--json Print command result as JSON
|
||||
|
||||
@@ -1,32 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
)
|
||||
|
||||
// TestMain lets this package's test binary stand in for the zero binary as its
|
||||
// own sandbox helper.
|
||||
//
|
||||
// THE SANDBOX SELF-DISPATCHES. With no standalone helper beside the running
|
||||
// executable, both sandboxes launch os.Executable() with a hidden subcommand
|
||||
// (__windows-command-runner, __windows-sandbox-setup, __sandbox-helper). In a
|
||||
// test that runs this binary as the zero CLI — TestLocalAdoptionSavesAndReloads-
|
||||
// EligibleModel does — os.Executable() is the test binary. Without this routing
|
||||
// it treated the hidden subcommand as nothing at all and ran the WHOLE package
|
||||
// suite as the "helper", so every sandboxed command in that child hung until it
|
||||
// was killed. cmd/zero/main.go and runWithDeps route these before anything else;
|
||||
// this does the same, in the same order.
|
||||
func TestMain(m *testing.M) {
|
||||
if len(os.Args) > 1 {
|
||||
switch os.Args[1] {
|
||||
case sandbox.LinuxSandboxHelperSubcommand:
|
||||
os.Exit(sandbox.RunLinuxSandboxHelper(os.Args[1:], os.Stderr))
|
||||
case sandbox.WindowsCommandRunnerSubcommand, sandbox.WindowsSandboxSetupSubcommand:
|
||||
os.Exit(runWithDeps(os.Args[1:], os.Stdout, os.Stderr, appDeps{}))
|
||||
}
|
||||
}
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
@@ -741,13 +741,6 @@ func (cfg *mcpWritableConfig) upsertServer(name string, server config.MCPServerC
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if !mergedServer.Disabled && config.IsDefaultMCPServerDisabled(name) {
|
||||
// Adding a server under the name of a default that ships disabled means
|
||||
// the user wants it on; the omitted flag would otherwise keep it off.
|
||||
if data, err = setMCPServerRawDisabledFalse(data); err != nil {
|
||||
return false, err
|
||||
}
|
||||
}
|
||||
cfg.serverRaw[name] = data
|
||||
if _, err := cfg.removeLegacyServer(name); err != nil {
|
||||
return false, err
|
||||
@@ -807,29 +800,11 @@ func mergeMCPServerRaw(existing json.RawMessage, server config.MCPServerConfig)
|
||||
if key == "oauth" && preserveRawOAuth {
|
||||
continue
|
||||
}
|
||||
// `mcp add` replaces the entry: it writes only a disable, never a stale
|
||||
// explicit false carried over from the old entry (upsertServer adds the
|
||||
// false a default that ships disabled needs).
|
||||
if key == "disabled" && string(value) == "false" {
|
||||
continue
|
||||
}
|
||||
merged[key] = value
|
||||
}
|
||||
return json.Marshal(merged)
|
||||
}
|
||||
|
||||
func setMCPServerRawDisabledFalse(data json.RawMessage) (json.RawMessage, error) {
|
||||
var server map[string]json.RawMessage
|
||||
if err := json.Unmarshal(data, &server); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if server == nil {
|
||||
server = map[string]json.RawMessage{}
|
||||
}
|
||||
server["disabled"] = json.RawMessage("false")
|
||||
return json.Marshal(server)
|
||||
}
|
||||
|
||||
func (cfg *mcpWritableConfig) removeServer(name string) (bool, error) {
|
||||
cfg.ensureRaw()
|
||||
removed := false
|
||||
@@ -881,19 +856,15 @@ func (cfg *mcpWritableConfig) setServerDisabled(name string, disabled bool) (boo
|
||||
server = map[string]json.RawMessage{}
|
||||
}
|
||||
|
||||
// A default that ships disabled (aimlapi) stays off until an entry says
|
||||
// otherwise, so an absent flag reads as disabled and enabling it must be
|
||||
// written out explicitly: dropping the flag would leave it off.
|
||||
shippedDisabled := config.IsDefaultMCPServerDisabled(name)
|
||||
current := shippedDisabled
|
||||
current := false
|
||||
if rawDisabled, ok := server["disabled"]; ok && len(rawDisabled) > 0 && string(rawDisabled) != "null" {
|
||||
if err := json.Unmarshal(rawDisabled, ¤t); err != nil {
|
||||
return false, false, err
|
||||
}
|
||||
}
|
||||
changed := current != disabled
|
||||
if disabled || shippedDisabled {
|
||||
data, err := json.Marshal(disabled)
|
||||
if disabled {
|
||||
data, err := json.Marshal(true)
|
||||
if err != nil {
|
||||
return false, false, err
|
||||
}
|
||||
|
||||
@@ -2,18 +2,10 @@ package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/aimlapi"
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
)
|
||||
|
||||
@@ -52,255 +44,3 @@ func TestRunMCPDisableSeededExaDefault(t *testing.T) {
|
||||
t.Fatal("expected `mcp disable exa` to turn the seeded default off in the resolved config")
|
||||
}
|
||||
}
|
||||
|
||||
// aimlapiWire stands in for https://mcp.aimlapi.com/mcp at the transport layer:
|
||||
// requests keep their real URL, nothing is dialed, and each one is recorded.
|
||||
type aimlapiWire struct {
|
||||
mu sync.Mutex
|
||||
requests []*http.Request
|
||||
}
|
||||
|
||||
func (wire *aimlapiWire) RoundTrip(request *http.Request) (*http.Response, error) {
|
||||
var body []byte
|
||||
if request.Body != nil {
|
||||
body, _ = io.ReadAll(request.Body)
|
||||
_ = request.Body.Close()
|
||||
}
|
||||
wire.mu.Lock()
|
||||
wire.requests = append(wire.requests, request.Clone(context.Background()))
|
||||
wire.mu.Unlock()
|
||||
var message struct {
|
||||
ID json.RawMessage `json:"id"`
|
||||
Method string `json:"method"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &message)
|
||||
respond := func(status int, payload string) *http.Response {
|
||||
return &http.Response{StatusCode: status, Header: http.Header{"Content-Type": {"application/json"}}, Body: io.NopCloser(strings.NewReader(payload)), Request: request}
|
||||
}
|
||||
if request.URL.String() != aimlapi.MCPServerURL {
|
||||
return respond(http.StatusBadGateway, `{}`), nil
|
||||
}
|
||||
if len(message.ID) == 0 {
|
||||
return respond(http.StatusAccepted, ""), nil
|
||||
}
|
||||
result := `{"protocolVersion":"2024-11-05","capabilities":{},"serverInfo":{"name":"fake","version":"1"}}`
|
||||
if message.Method == "tools/list" {
|
||||
result = `{"tools":[{"name":"tools_search","description":"Search the Toolbox","inputSchema":{"type":"object"}}]}`
|
||||
}
|
||||
return respond(http.StatusOK, fmt.Sprintf(`{"jsonrpc":"2.0","id":%s,"result":%s}`, message.ID, result)), nil
|
||||
}
|
||||
|
||||
func (wire *aimlapiWire) recorded() []*http.Request {
|
||||
wire.mu.Lock()
|
||||
defer wire.mu.Unlock()
|
||||
return append([]*http.Request(nil), wire.requests...)
|
||||
}
|
||||
|
||||
// isolateAIMLAPIMCPCommand redirects every root the real resolve path reads to
|
||||
// a temporary home, sets an AI/ML API key in the environment, and routes the
|
||||
// default HTTP transport to a recorder. It returns the user config path.
|
||||
func isolateAIMLAPIMCPCommand(t *testing.T, key string) (string, *aimlapiWire, appDeps) {
|
||||
t.Helper()
|
||||
home := t.TempDir()
|
||||
t.Setenv("HOME", home)
|
||||
t.Setenv("USERPROFILE", home)
|
||||
t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config"))
|
||||
t.Setenv("XDG_STATE_HOME", filepath.Join(home, ".state"))
|
||||
t.Setenv("XDG_CACHE_HOME", filepath.Join(home, ".cache"))
|
||||
t.Setenv("APPDATA", filepath.Join(home, "AppData", "Roaming"))
|
||||
t.Setenv("LOCALAPPDATA", filepath.Join(home, "AppData", "Local"))
|
||||
t.Setenv("ZERO_CRED_STORAGE", "file")
|
||||
t.Setenv("ZERO_PROVIDER", "")
|
||||
t.Setenv("AIMLAPI_INFERENCE_URL", "")
|
||||
t.Setenv("AIMLAPI_PARTNER_ID", "")
|
||||
t.Setenv("AIMLAPI_API_KEY", key)
|
||||
configPath, err := config.DefaultUserConfigPath()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeMCPCommandRawConfig(t, configPath, `{}`)
|
||||
|
||||
wire := &aimlapiWire{}
|
||||
previous := http.DefaultTransport
|
||||
http.DefaultTransport = wire
|
||||
t.Cleanup(func() { http.DefaultTransport = previous })
|
||||
|
||||
workspace := t.TempDir()
|
||||
return configPath, wire, appDeps{getwd: func() (string, error) { return workspace, nil }}
|
||||
}
|
||||
|
||||
func TestRunMCPEnableAIMLAPIDefaultWritesOnlyTheToggle(t *testing.T) {
|
||||
configPath, wire, deps := isolateAIMLAPIMCPCommand(t, "aiml-secret-key")
|
||||
|
||||
var out, errBuf bytes.Buffer
|
||||
if code := runWithDeps([]string{"mcp", "disable", "aimlapi", "--json"}, &out, &errBuf, deps); code != exitSuccess {
|
||||
t.Fatalf("disable exit=%d stderr=%s", code, errBuf.String())
|
||||
}
|
||||
if !strings.Contains(out.String(), `"changed": false`) {
|
||||
t.Fatalf("disabling the off-by-default server should be a no-op: %s", out.String())
|
||||
}
|
||||
|
||||
out.Reset()
|
||||
errBuf.Reset()
|
||||
if code := runWithDeps([]string{"mcp", "enable", "aimlapi", "--json"}, &out, &errBuf, deps); code != exitSuccess {
|
||||
t.Fatalf("enable exit=%d stderr=%s", code, errBuf.String())
|
||||
}
|
||||
if !strings.Contains(out.String(), `"changed": true`) {
|
||||
t.Fatalf("enable should report a change: %s", out.String())
|
||||
}
|
||||
data, err := os.ReadFile(configPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var written struct {
|
||||
MCP struct {
|
||||
Servers map[string]map[string]any `json:"servers"`
|
||||
} `json:"mcp"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &written); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if entry := written.MCP.Servers["aimlapi"]; len(entry) != 1 || entry["disabled"] != false {
|
||||
t.Fatalf("enable must write only the toggle, got %#v in %s", entry, data)
|
||||
}
|
||||
if strings.Contains(string(data), "aiml-secret-key") || strings.Contains(string(data), "Authorization") {
|
||||
t.Fatalf("config holds credential material: %s", data)
|
||||
}
|
||||
|
||||
cfg, err := config.ResolveMCP(config.ResolveOptions{UserConfigPath: configPath})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if server := cfg.Servers["aimlapi"]; server.Disabled || config.IsUnconfiguredDefault("aimlapi", server) {
|
||||
t.Fatalf("enabled aimlapi = %#v, want enabled and user-configured", server)
|
||||
}
|
||||
// An unrelated save rewrites the whole file through FileConfig; the
|
||||
// explicit enable has to survive it or the shipped disable comes back.
|
||||
if _, err := config.SetTheme(configPath, "dark"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg, err = config.ResolveMCP(config.ResolveOptions{UserConfigPath: configPath})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.Servers["aimlapi"].Disabled {
|
||||
data, _ := os.ReadFile(configPath)
|
||||
t.Fatalf("an unrelated config save turned aimlapi off again: %s", data)
|
||||
}
|
||||
if len(wire.recorded()) != 0 {
|
||||
t.Fatal("enable must not connect")
|
||||
}
|
||||
|
||||
out.Reset()
|
||||
if code := runWithDeps([]string{"mcp", "disable", "aimlapi"}, &out, &errBuf, deps); code != exitSuccess {
|
||||
t.Fatalf("disable exit=%d stderr=%s", code, errBuf.String())
|
||||
}
|
||||
cfg, err = config.ResolveMCP(config.ResolveOptions{UserConfigPath: configPath})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !cfg.Servers["aimlapi"].Disabled {
|
||||
t.Fatal("disable after enable must turn aimlapi off again")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunMCPAddUnderAIMLAPIDefaultNameEnablesIt(t *testing.T) {
|
||||
configPath, _, deps := isolateAIMLAPIMCPCommand(t, "")
|
||||
var out, errBuf bytes.Buffer
|
||||
if code := runWithDeps([]string{"mcp", "add", "aimlapi", "--type", "http", "--url", aimlapi.MCPServerURL, "--auth", "oauth"}, &out, &errBuf, deps); code != exitSuccess {
|
||||
t.Fatalf("add exit=%d stderr=%s", code, errBuf.String())
|
||||
}
|
||||
cfg, err := config.ResolveMCP(config.ResolveOptions{UserConfigPath: configPath})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if server := cfg.Servers["aimlapi"]; server.Disabled || server.Auth != "oauth" {
|
||||
t.Fatalf("added aimlapi = %#v, want enabled with oauth", server)
|
||||
}
|
||||
if code := runWithDeps([]string{"mcp", "add", "aimlapi", "--type", "http", "--url", aimlapi.MCPServerURL, "--disabled"}, &out, &errBuf, deps); code != exitSuccess {
|
||||
t.Fatalf("add --disabled exit=%d stderr=%s", code, errBuf.String())
|
||||
}
|
||||
if cfg, err = config.ResolveMCP(config.ResolveOptions{UserConfigPath: configPath}); err != nil || !cfg.Servers["aimlapi"].Disabled {
|
||||
t.Fatalf("add --disabled left aimlapi enabled: %#v, %v", cfg.Servers["aimlapi"], err)
|
||||
}
|
||||
}
|
||||
|
||||
// Through the real resolve path (no injected resolveMCPConfig): the default is
|
||||
// listed as disabled, never connects, and once enabled it reaches session
|
||||
// startup and `zero mcp check`, which connects with the key on the wire.
|
||||
func TestAIMLAPIMCPDefaultThroughRealResolvePath(t *testing.T) {
|
||||
_, wire, deps := isolateAIMLAPIMCPCommand(t, "aiml-secret-key")
|
||||
|
||||
var out, errBuf bytes.Buffer
|
||||
if code := runWithDeps([]string{"mcp", "list"}, &out, &errBuf, deps); code != exitSuccess {
|
||||
t.Fatalf("list exit=%d stderr=%s", code, errBuf.String())
|
||||
}
|
||||
if !strings.Contains(out.String(), "aimlapi [http] disabled https://mcp.aimlapi.com/mcp") || strings.Contains(out.String(), "aiml-secret-key") {
|
||||
t.Fatalf("list output should show the disabled aimlapi default and no key:\n%s", out.String())
|
||||
}
|
||||
out.Reset()
|
||||
if code := runWithDeps([]string{"mcp", "list", "--json"}, &out, &errBuf, deps); code != exitSuccess {
|
||||
t.Fatalf("list --json exit=%d stderr=%s", code, errBuf.String())
|
||||
}
|
||||
if !strings.Contains(out.String(), `"aimlapi"`) || strings.Contains(out.String(), "aiml-secret-key") {
|
||||
t.Fatalf("list --json = %s", out.String())
|
||||
}
|
||||
out.Reset()
|
||||
errBuf.Reset()
|
||||
if code := runWithDeps([]string{"mcp", "check", "aimlapi"}, &out, &errBuf, deps); code == exitSuccess || !strings.Contains(errBuf.String(), "is disabled") {
|
||||
t.Fatalf("check of the disabled default: exit=%d stderr=%s", code, errBuf.String())
|
||||
}
|
||||
resolved, err := fillAppDeps(deps).resolveMCPConfig(t.TempDir(), true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
critical, optional := splitMCPStartupConfig(resolved)
|
||||
if _, ok := critical.Servers["aimlapi"]; ok {
|
||||
t.Fatal("the disabled default must not start with a session")
|
||||
}
|
||||
if _, ok := optional.Servers["aimlapi"]; ok {
|
||||
t.Fatal("the disabled default must not start with a session")
|
||||
}
|
||||
if len(wire.recorded()) != 0 {
|
||||
t.Fatalf("the disabled default connected: %d requests", len(wire.recorded()))
|
||||
}
|
||||
|
||||
if code := runWithDeps([]string{"mcp", "enable", "aimlapi"}, &out, &errBuf, deps); code != exitSuccess {
|
||||
t.Fatalf("enable exit=%d stderr=%s", code, errBuf.String())
|
||||
}
|
||||
resolved, err = fillAppDeps(deps).resolveMCPConfig(t.TempDir(), true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
critical, _ = splitMCPStartupConfig(resolved)
|
||||
if _, ok := critical.Servers["aimlapi"]; !ok {
|
||||
t.Fatal("an enabled aimlapi must start with the session as a user-configured server")
|
||||
}
|
||||
|
||||
out.Reset()
|
||||
errBuf.Reset()
|
||||
if code := runWithDeps([]string{"mcp", "check", "aimlapi", "--json"}, &out, &errBuf, deps); code != exitSuccess {
|
||||
t.Fatalf("check exit=%d stdout=%s stderr=%s", code, out.String(), errBuf.String())
|
||||
}
|
||||
if !strings.Contains(out.String(), `"status": "ok"`) || !strings.Contains(out.String(), `"toolCount": 1`) {
|
||||
t.Fatalf("check output = %s", out.String())
|
||||
}
|
||||
if strings.Contains(out.String()+errBuf.String(), "aiml-secret-key") {
|
||||
t.Fatalf("check output leaks the key: %s %s", out.String(), errBuf.String())
|
||||
}
|
||||
requests := wire.recorded()
|
||||
if len(requests) == 0 {
|
||||
t.Fatal("check did not connect")
|
||||
}
|
||||
for _, request := range requests {
|
||||
if request.URL.String() != aimlapi.MCPServerURL {
|
||||
t.Fatalf("request went to %s", request.URL)
|
||||
}
|
||||
if got := request.Header.Get("Authorization"); got != "Bearer aiml-secret-key" {
|
||||
t.Fatalf("Authorization = %q, want the AI/ML API key", got)
|
||||
}
|
||||
if request.Header.Get(aimlapi.SourceHeaderName) != aimlapi.SourceValue || request.Header.Get(aimlapi.PartnerHeaderName) == "" {
|
||||
t.Fatalf("attribution headers missing: %v", request.Header)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,193 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/mcp"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/tui"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// The orchestrate notice, wired at both surfaces.
|
||||
//
|
||||
// AgentOptions.OrchestrateAvailable was declared, documented and consumed by the
|
||||
// reminder selector — and set by NEITHER production call site. So the tool was
|
||||
// advertised in the tool list and the model was never told it existed, which is
|
||||
// exactly what a user found by running the binary: the posture turned on and
|
||||
// nothing orchestrated.
|
||||
//
|
||||
// A unit test on zeromaxingReminders PASSED throughout that whole period,
|
||||
// because it passes the flag in itself. So these tests drive the two
|
||||
// constructions instead — Run() to the wire for exec, and the captured
|
||||
// tui.Options for the TUI — which is the only level at which the defect was
|
||||
// visible. Invariant 1, for the second time in this feature.
|
||||
|
||||
// EXEC: the notice has to reach the actual request body.
|
||||
func TestExecTellsTheModelAboutOrchestrate(t *testing.T) {
|
||||
clearProviderEnv(t)
|
||||
root := t.TempDir()
|
||||
configDir := filepath.Join(root, ".zero")
|
||||
if err := os.MkdirAll(configDir, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var sent []byte
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
if sent == nil {
|
||||
sent = body
|
||||
}
|
||||
_, _ = io.WriteString(w, `data: {"choices":[{"delta":{"content":"ok"}}]}`+"\n\n")
|
||||
_, _ = io.WriteString(w, "data: [DONE]\n\n")
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
providerConfig := `{
|
||||
"activeProvider": "local",
|
||||
"providers": [{
|
||||
"name": "local",
|
||||
"provider_kind": "openai-compatible",
|
||||
"base_url": "` + server.URL + `",
|
||||
"api_key": "sk-local",
|
||||
"model": "local-model"
|
||||
}]
|
||||
}`
|
||||
if err := os.WriteFile(filepath.Join(configDir, "config.json"), []byte(providerConfig), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
run := func(args ...string) string {
|
||||
sent = nil
|
||||
var stdout, stderr bytes.Buffer
|
||||
Run(append([]string{"exec", "--cwd", root}, args...), &stdout, &stderr)
|
||||
return string(sent)
|
||||
}
|
||||
|
||||
// POSTURE ON, specialist tools registered: the model is TOLD.
|
||||
on := run("--reasoning-effort", "zeromaxing", "--auto", "high", "look at this")
|
||||
if !strings.Contains(on, "orchestrate tool") {
|
||||
t.Fatalf("the model was never told the orchestrate tool exists:\n%s", firstKB(on))
|
||||
}
|
||||
// ...and the tool really is advertised, so the notice is not a promise the
|
||||
// run cannot keep.
|
||||
if !advertisesOrchestrate(t, on) {
|
||||
t.Fatalf("the notice names a tool this run does not advertise:\n%s", firstKB(on))
|
||||
}
|
||||
|
||||
// POSTURE OFF: neither the notice nor the tool. This is the direction that
|
||||
// makes the first half mean something — a notice emitted unconditionally
|
||||
// would satisfy the assertion above.
|
||||
off := run("--auto", "high", "look at this")
|
||||
if strings.Contains(off, "orchestrate") {
|
||||
t.Fatalf("orchestrate leaked into a posture-off run:\n%s", firstKB(off))
|
||||
}
|
||||
|
||||
// POSTURE ON but at an autonomy that registers NO specialist tools: the
|
||||
// notice must not fire, or it promises a capability this run does not have.
|
||||
unavailable := run("--reasoning-effort", "zeromaxing", "--auto", "low", "look at this")
|
||||
if advertisesOrchestrate(t, unavailable) {
|
||||
t.Skip("this autonomy level registers orchestrate after all; the case cannot be exercised here")
|
||||
}
|
||||
if strings.Contains(unavailable, "orchestrate tool") {
|
||||
t.Fatalf("the notice fired for a run that does not hold the tool:\n%s", firstKB(unavailable))
|
||||
}
|
||||
}
|
||||
|
||||
func advertisesOrchestrate(t *testing.T, body string) bool {
|
||||
t.Helper()
|
||||
var request struct {
|
||||
Tools []struct {
|
||||
Function struct {
|
||||
Name string `json:"name"`
|
||||
} `json:"function"`
|
||||
} `json:"tools"`
|
||||
}
|
||||
if json.Unmarshal([]byte(body), &request) != nil {
|
||||
return false
|
||||
}
|
||||
for _, tool := range request.Tools {
|
||||
if tool.Function.Name == "orchestrate" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func firstKB(body string) string {
|
||||
if len(body) > 1200 {
|
||||
return body[:1200]
|
||||
}
|
||||
return body
|
||||
}
|
||||
|
||||
// TUI: the same fact, at the only seam a test can reach. The TUI cannot be
|
||||
// driven headlessly, so this asserts what it is HANDED — which is where the
|
||||
// field was missing.
|
||||
func TestTheTUIIsHandedTheOrchestrateAvailability(t *testing.T) {
|
||||
var stdout, stderr bytes.Buffer
|
||||
cwd := t.TempDir()
|
||||
setCLIUserConfigRoot(t)
|
||||
userConfigPath := filepath.Join(t.TempDir(), "zero", "config.json")
|
||||
var launched tui.Options
|
||||
|
||||
exitCode := runWithDeps([]string{}, &stdout, &stderr, appDeps{
|
||||
getwd: func() (string, error) { return cwd, nil },
|
||||
resolveConfig: func(string, config.Overrides) (config.ResolvedConfig, error) {
|
||||
return config.ResolvedConfig{
|
||||
MaxTurns: 12,
|
||||
ActiveProvider: "local",
|
||||
Provider: config.ProviderProfile{
|
||||
Name: "local", ProviderKind: config.ProviderKindOpenAICompatible,
|
||||
BaseURL: "http://127.0.0.1/v1", Model: "m",
|
||||
},
|
||||
}, nil
|
||||
},
|
||||
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) { return nil, nil },
|
||||
userConfigPath: func() (string, error) { return userConfigPath, nil },
|
||||
registerMCPTools: func(context.Context, *tools.Registry, config.MCPConfig, mcp.RegisterOptions) (mcpToolRuntime, error) {
|
||||
return noopMCPRuntime{}, nil
|
||||
},
|
||||
runTUI: func(_ context.Context, options tui.Options) int {
|
||||
launched = options
|
||||
return 3
|
||||
},
|
||||
})
|
||||
if exitCode != 3 {
|
||||
t.Fatalf("exit = %d, want 3: %s", exitCode, stderr.String())
|
||||
}
|
||||
|
||||
// The TUI always registers specialist tools, so it always holds orchestrate;
|
||||
// the flag must say so, or the notice can never fire in the surface where
|
||||
// the posture is most used.
|
||||
if !launched.AgentOptions.OrchestrateAvailable {
|
||||
t.Fatal("the TUI was handed OrchestrateAvailable=false while holding the tool, so the model is never told it exists")
|
||||
}
|
||||
// ...and it agrees with the registry it was handed, rather than being a
|
||||
// constant that happens to read true.
|
||||
if _, found := launched.AgentOptions.Registry.Get("orchestrate"); !found {
|
||||
t.Fatal("the TUI's registry does not hold orchestrate, so the flag is wrong in the other direction")
|
||||
}
|
||||
}
|
||||
|
||||
// The derivation itself: false when the tool is absent, so the flag can never
|
||||
// promise a capability a run does not have.
|
||||
func TestOrchestrateAvailabilityFollowsTheRegistry(t *testing.T) {
|
||||
if orchestrateAvailable(nil) {
|
||||
t.Fatal("a nil registry reported the tool available")
|
||||
}
|
||||
empty := tools.NewRegistry()
|
||||
if orchestrateAvailable(empty) {
|
||||
t.Fatal("an empty registry reported the tool available")
|
||||
}
|
||||
}
|
||||
@@ -1,28 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Both surfaces that launch plan tasks must wire ExtraReadRoots, or a
|
||||
// request_permissions READ grant never reaches those tasks and a plan auditing a
|
||||
// granted external path fails "outside the workspace" in every finder.
|
||||
//
|
||||
// This is the unwired-feature guard: an option that is built but never passed
|
||||
// does nothing, and mutation testing cannot catch it because deleting an unused
|
||||
// wiring breaks no test. So this asserts the wiring is present at the source, in
|
||||
// both the TUI (app.go) and headless (exec.go) paths, beside the write grant it
|
||||
// mirrors.
|
||||
func TestBothPathsWireExtraReadRootsBesideWrite(t *testing.T) {
|
||||
for _, file := range []string{"app.go", "exec.go"} {
|
||||
src, err := os.ReadFile(file)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", file, err)
|
||||
}
|
||||
if !strings.Contains(string(src), "ExtraReadRoots:") {
|
||||
t.Fatalf("%s does not wire ExtraReadRoots — a plan there cannot read a granted path", file)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,109 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
|
||||
"github.com/Twigpine/zero/internal/tui"
|
||||
)
|
||||
|
||||
// The background-plan launcher: who owns a background plan's LIFETIME.
|
||||
//
|
||||
// The tool does not, and that is the whole point of the seam. A background plan
|
||||
// must outlive the tool call that started it and must NOT outlive the session,
|
||||
// and the only thing that knows where the session ends is the surface that runs
|
||||
// it. So the surface supplies a launcher, keeps the context, and drains on exit;
|
||||
// the tool asks to be run and is told yes or no.
|
||||
//
|
||||
// It also makes headless exec's refusal fall out rather than be special-cased:
|
||||
// that process exits when the turn ends, so it supplies no launcher at all, and
|
||||
// a background plan there is refused with a reason instead of being started and
|
||||
// orphaned.
|
||||
//
|
||||
// ONE AT A TIME, deliberately. The panel is keyed by task id and the bridge
|
||||
// holds one plan's cancel; two concurrent background plans would report into
|
||||
// one panel and the second would overwrite the first's rows. Refusing the
|
||||
// second is a stated rule; letting the UI silently pick one is a defect.
|
||||
|
||||
// planLauncher runs background plans under a session-scoped context.
|
||||
type planLauncher struct {
|
||||
mu sync.Mutex
|
||||
// ctx is the SESSION root, not a tool call's. Cancelling it is what stops a
|
||||
// background plan at exit — the failure this exists to prevent is a plan
|
||||
// that outlives its session, spends budget nobody sees, and reports nothing.
|
||||
ctx context.Context
|
||||
// running holds the one in-flight plan's cancel; nil means none.
|
||||
running context.CancelFunc
|
||||
// closed stops new launches once shutdown has begun. Checked under the same
|
||||
// lock that starts a plan, so a launch cannot slip past a concurrent Close.
|
||||
closed bool
|
||||
// wait lets Close block until the plan actually stops, rather than
|
||||
// cancelling and exiting while a child process is still being torn down.
|
||||
wait sync.WaitGroup
|
||||
// bridge is told a background plan is starting, so every message it posts is
|
||||
// marked as one and survives the panel's stale-run guard.
|
||||
bridge *tui.PlanProgressBridge
|
||||
}
|
||||
|
||||
func newPlanLauncher(ctx context.Context, bridge *tui.PlanProgressBridge) *planLauncher {
|
||||
return &planLauncher{ctx: ctx, bridge: bridge}
|
||||
}
|
||||
|
||||
// Launch starts run on its own goroutine and reports whether it did.
|
||||
//
|
||||
// False is returned — never a silent no-op — when the session is closing or a
|
||||
// background plan is already running, so the tool can say which rather than
|
||||
// handing back a run id for a plan nobody started.
|
||||
func (launcher *planLauncher) Launch(run func(ctx context.Context)) bool {
|
||||
if launcher == nil || run == nil {
|
||||
return false
|
||||
}
|
||||
launcher.mu.Lock()
|
||||
if launcher.closed || launcher.running != nil {
|
||||
launcher.mu.Unlock()
|
||||
return false
|
||||
}
|
||||
ctx, cancel := context.WithCancel(launcher.ctx)
|
||||
launcher.running = cancel
|
||||
launcher.wait.Add(1)
|
||||
launcher.mu.Unlock()
|
||||
|
||||
launcher.bridge.SetBackground(true)
|
||||
go func() {
|
||||
defer func() {
|
||||
// Every exit converges on the bridge's idempotent terminal transition.
|
||||
// Normal execution has already called PlanCompleted; panic, cancellation,
|
||||
// or an early return is finalized here before the launcher slot opens.
|
||||
panicked := recover() != nil
|
||||
launcher.bridge.FinalizeBackgroundExit(panicked, ctx.Err() != nil)
|
||||
cancel()
|
||||
launcher.mu.Lock()
|
||||
launcher.running = nil
|
||||
launcher.mu.Unlock()
|
||||
launcher.wait.Done()
|
||||
}()
|
||||
run(ctx)
|
||||
}()
|
||||
return true
|
||||
}
|
||||
|
||||
// Close stops any background plan and WAITS for it.
|
||||
//
|
||||
// Waiting matters: cancelling and returning would let the process exit while a
|
||||
// child is still being torn down, which is how orphans happen. The recorder's
|
||||
// terminal event is written by the plan itself on the way out, so the wait is
|
||||
// also what makes a cancelled background plan land in the session log rather
|
||||
// than vanish.
|
||||
func (launcher *planLauncher) Close() {
|
||||
if launcher == nil {
|
||||
return
|
||||
}
|
||||
launcher.mu.Lock()
|
||||
launcher.closed = true
|
||||
cancel := launcher.running
|
||||
launcher.mu.Unlock()
|
||||
if cancel != nil {
|
||||
cancel()
|
||||
}
|
||||
launcher.wait.Wait()
|
||||
}
|
||||
@@ -1,195 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
"github.com/Twigpine/zero/internal/tui"
|
||||
)
|
||||
|
||||
// A background plan MUST NOT OUTLIVE ITS SESSION. That is the failure this
|
||||
// whole seam exists to prevent — a plan still spending after the session ended,
|
||||
// reporting to nobody.
|
||||
func TestClosingTheLauncherStopsAndWaitsForTheBackgroundPlan(t *testing.T) {
|
||||
launcher := newPlanLauncher(context.Background(), tui.NewPlanProgressBridge())
|
||||
|
||||
started := make(chan struct{})
|
||||
stopped := make(chan struct{})
|
||||
if !launcher.Launch(func(ctx context.Context) {
|
||||
close(started)
|
||||
<-ctx.Done()
|
||||
// A real plan tears a child down here; the sleep stands in for that, and
|
||||
// Close must not return until it is over.
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
close(stopped)
|
||||
}) {
|
||||
t.Fatal("Launch refused")
|
||||
}
|
||||
<-started
|
||||
|
||||
launcher.Close()
|
||||
select {
|
||||
case <-stopped:
|
||||
default:
|
||||
t.Fatal("Close returned while the plan was still stopping; that is how a child is orphaned")
|
||||
}
|
||||
}
|
||||
|
||||
// ONE AT A TIME. Two background plans report into one panel keyed by task id and
|
||||
// the second would overwrite the first's rows, so the second is refused rather
|
||||
// than silently winning.
|
||||
func TestASecondBackgroundPlanIsRefused(t *testing.T) {
|
||||
launcher := newPlanLauncher(context.Background(), tui.NewPlanProgressBridge())
|
||||
release := make(chan struct{})
|
||||
defer close(release)
|
||||
|
||||
if !launcher.Launch(func(context.Context) { <-release }) {
|
||||
t.Fatal("the first launch was refused")
|
||||
}
|
||||
// Give the goroutine a moment to be counted; the refusal is decided under
|
||||
// the lock at Launch, so this is about the test not the code.
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
if launcher.Launch(func(context.Context) {}) {
|
||||
t.Fatal("a second background plan was launched over a running one")
|
||||
}
|
||||
}
|
||||
|
||||
// ...and once the first finishes, the slot frees.
|
||||
func TestTheSlotFreesWhenABackgroundPlanEnds(t *testing.T) {
|
||||
launcher := newPlanLauncher(context.Background(), tui.NewPlanProgressBridge())
|
||||
done := make(chan struct{})
|
||||
if !launcher.Launch(func(context.Context) { close(done) }) {
|
||||
t.Fatal("the first launch was refused")
|
||||
}
|
||||
<-done
|
||||
|
||||
deadline := time.After(3 * time.Second)
|
||||
for {
|
||||
if launcher.Launch(func(context.Context) {}) {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatal("the slot never freed after the plan ended")
|
||||
case <-time.After(10 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A launch after Close must be REFUSED, not started into a cancelled context.
|
||||
// Starting it would record an admission and a cancellation for a plan the user
|
||||
// never saw.
|
||||
func TestLaunchingAfterCloseIsRefused(t *testing.T) {
|
||||
launcher := newPlanLauncher(context.Background(), tui.NewPlanProgressBridge())
|
||||
launcher.Close()
|
||||
if launcher.Launch(func(context.Context) { t.Error("a plan ran after Close") }) {
|
||||
t.Fatal("Launch succeeded after Close")
|
||||
}
|
||||
}
|
||||
|
||||
// A PANIC IN A BACKGROUND PLAN MUST NOT TAKE THE SESSION DOWN, and must still
|
||||
// free the slot — a crashed plan that held the slot forever would block every
|
||||
// later one with "already running".
|
||||
func TestAPanickingBackgroundPlanIsContainedAndFreesTheSlot(t *testing.T) {
|
||||
bridge := tui.NewPlanProgressBridge()
|
||||
launcher := newPlanLauncher(context.Background(), bridge)
|
||||
if !launcher.Launch(func(context.Context) { panic("boom") }) {
|
||||
t.Fatal("Launch refused")
|
||||
}
|
||||
|
||||
deadline := time.After(3 * time.Second)
|
||||
for bridge.BackgroundPlanLive() {
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatal("a panicking plan left the bridge busy")
|
||||
case <-time.After(10 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
if completion := bridge.DrainCompletedPlans(); !strings.Contains(completion, "failed") || !strings.Contains(completion, "panicked") {
|
||||
t.Fatalf("panic produced no explicit failed completion: %q", completion)
|
||||
}
|
||||
if !launcher.Launch(func(context.Context) {}) {
|
||||
t.Fatal("a panicking plan held the launcher slot forever")
|
||||
}
|
||||
}
|
||||
|
||||
// Cancelling the SESSION context cancels the plan, which is what makes the
|
||||
// session root load-bearing rather than decorative.
|
||||
func TestCancellingTheSessionCancelsTheBackgroundPlan(t *testing.T) {
|
||||
sessionCtx, cancelSession := context.WithCancel(context.Background())
|
||||
launcher := newPlanLauncher(sessionCtx, tui.NewPlanProgressBridge())
|
||||
|
||||
observed := make(chan error, 1)
|
||||
if !launcher.Launch(func(ctx context.Context) {
|
||||
<-ctx.Done()
|
||||
observed <- ctx.Err()
|
||||
}) {
|
||||
t.Fatal("Launch refused")
|
||||
}
|
||||
cancelSession()
|
||||
select {
|
||||
case err := <-observed:
|
||||
if err == nil {
|
||||
t.Fatal("the plan's context was not cancelled")
|
||||
}
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("cancelling the session did not reach the background plan")
|
||||
}
|
||||
}
|
||||
|
||||
// The bridge is told BEFORE the plan runs, so its very first message is already
|
||||
// marked background. Marking it after the goroutine started would race the
|
||||
// plan's own admission message, and a dropped admission is a panel that never
|
||||
// shows the plan at all.
|
||||
func TestTheBridgeIsMarkedBackgroundBeforeThePlanRuns(t *testing.T) {
|
||||
bridge := tui.NewPlanProgressBridge()
|
||||
launcher := newPlanLauncher(context.Background(), bridge)
|
||||
|
||||
var once sync.Once
|
||||
markedAtStart := make(chan bool, 1)
|
||||
if !launcher.Launch(func(context.Context) {
|
||||
once.Do(func() { markedAtStart <- bridge.PlanIsBackground() })
|
||||
}) {
|
||||
t.Fatal("Launch refused")
|
||||
}
|
||||
select {
|
||||
case marked := <-markedAtStart:
|
||||
if !marked {
|
||||
t.Fatal("the plan started before the bridge was marked background")
|
||||
}
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("the plan never ran")
|
||||
}
|
||||
}
|
||||
|
||||
// HEADLESS EXEC SUPPLIES NO LAUNCHER, which is what makes its refusal of a
|
||||
// background plan fall out of the wiring rather than out of a special case.
|
||||
// Asserted at the registered tool, because a comment saying "exec passes nil"
|
||||
// is not evidence that exec passes nil.
|
||||
func TestHeadlessExecSuppliesNoLauncher(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
registry := newCoreRegistry(workspace)
|
||||
runtime, err := registerSpecialistTools(registry, workspace, 0, nil, nil, nil, orchestrateWiring{
|
||||
Gate: &specialist.PostureGate{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("registerSpecialistTools: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { closeSpecialistRuntime(nil, runtime) })
|
||||
|
||||
registered, found := registry.Get(specialist.OrchestrateToolName)
|
||||
if !found {
|
||||
t.Fatal("orchestrate was not registered")
|
||||
}
|
||||
tool, ok := registered.(*specialist.OrchestrateTool)
|
||||
if !ok {
|
||||
t.Fatalf("orchestrate is %T", registered)
|
||||
}
|
||||
if tool.Launch != nil {
|
||||
t.Fatal("a run wired without a launcher was given one; a background plan there could never report")
|
||||
}
|
||||
}
|
||||
@@ -1,384 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
)
|
||||
|
||||
// The plan tool's parent grant was declared, documented and validated against —
|
||||
// and left nil by BOTH production call sites, which made the "a task may narrow
|
||||
// the parent's grant, never widen it" rule inert. These tests drive
|
||||
// registerSpecialistTools, the function both call sites go through, rather than
|
||||
// the helper it uses: asserting the helper would have passed throughout the
|
||||
// period the wiring was missing.
|
||||
|
||||
func registeredOrchestrateTool(t *testing.T, enabled, disabled []string) *specialist.OrchestrateTool {
|
||||
t.Helper()
|
||||
workspace := t.TempDir()
|
||||
registry := newCoreRegistry(workspace)
|
||||
runtime, err := registerSpecialistTools(registry, workspace, 0, enabled, disabled, nil, orchestrateWiring{
|
||||
Gate: &specialist.PostureGate{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("registerSpecialistTools: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { closeSpecialistRuntime(nil, runtime) })
|
||||
|
||||
registered, found := registry.Get(specialist.OrchestrateToolName)
|
||||
if !found {
|
||||
t.Fatal("orchestrate was not registered")
|
||||
}
|
||||
tool, ok := registered.(*specialist.OrchestrateTool)
|
||||
if !ok {
|
||||
t.Fatalf("orchestrate is %T, not *specialist.OrchestrateTool", registered)
|
||||
}
|
||||
return tool
|
||||
}
|
||||
|
||||
// An unfiltered run grants every read-only tool a plan task may hold. The
|
||||
// failure this pins is a grant of length zero, which is what both call sites
|
||||
// produced.
|
||||
func TestRegisteredOrchestrateToolCarriesTheRunsGrant(t *testing.T) {
|
||||
tool := registeredOrchestrateTool(t, nil, nil)
|
||||
if len(tool.ParentTools) == 0 {
|
||||
t.Fatal("the registered orchestrate tool holds no parent grant, so the narrowing rule cannot fire")
|
||||
}
|
||||
// The grant is every GRANTABLE name this registry holds — read-only plus the
|
||||
// write tools a task may name — intersected with what the registry actually
|
||||
// has. Comparing against the full list would fail for a registry missing one
|
||||
// of them, so it is asserted as a subset with the read-only half required.
|
||||
grantable := map[string]bool{}
|
||||
for _, name := range specialist.PlanGrantableToolNames() {
|
||||
grantable[name] = true
|
||||
}
|
||||
for _, name := range tool.ParentTools {
|
||||
if !grantable[name] {
|
||||
t.Fatalf("grant contains %q, which a plan task may never hold", name)
|
||||
}
|
||||
}
|
||||
held := map[string]bool{}
|
||||
for _, name := range tool.ParentTools {
|
||||
held[name] = true
|
||||
}
|
||||
for _, name := range specialist.PlanReadOnlyToolNames() {
|
||||
if _, found := newCoreRegistry(t.TempDir()).Get(name); found && !held[name] {
|
||||
t.Fatalf("the unfiltered grant is missing read-only tool %q", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --enabled-tools narrows the grant. This is the reproduction from the audit,
|
||||
// at the registration boundary: a parent holding only grep must not hand a plan
|
||||
// task read_file.
|
||||
func TestRegisteredOrchestrateGrantHonoursEnabledTools(t *testing.T) {
|
||||
tool := registeredOrchestrateTool(t, []string{"grep", specialist.OrchestrateToolName}, nil)
|
||||
if strings.Join(tool.ParentTools, ",") != "grep" {
|
||||
t.Fatalf("grant = %v, want exactly [grep]: a plan task must not hold what --enabled-tools denied the run", tool.ParentTools)
|
||||
}
|
||||
}
|
||||
|
||||
// --disabled-tools narrows it too, through the same filter gate the agent loop
|
||||
// uses, so the two cannot disagree about what the run holds.
|
||||
func TestRegisteredOrchestrateGrantHonoursDisabledTools(t *testing.T) {
|
||||
tool := registeredOrchestrateTool(t, nil, []string{"read_file", "glob"})
|
||||
for _, name := range tool.ParentTools {
|
||||
if name == "read_file" || name == "glob" {
|
||||
t.Fatalf("grant %v still contains a tool --disabled-tools removed from the run", tool.ParentTools)
|
||||
}
|
||||
}
|
||||
if len(tool.ParentTools) == 0 {
|
||||
t.Fatal("grant collapsed to empty; only the two disabled tools should have been removed")
|
||||
}
|
||||
}
|
||||
|
||||
// A run holding no read-only tools grants nothing. The grant must be EMPTY
|
||||
// rather than falling back to a default set — an empty grant is refused
|
||||
// downstream, and the bug was that it expanded instead.
|
||||
func TestRegisteredOrchestrateGrantIsEmptyWhenTheRunHoldsNoReadTools(t *testing.T) {
|
||||
tool := registeredOrchestrateTool(t, []string{specialist.OrchestrateToolName}, nil)
|
||||
if len(tool.ParentTools) != 0 {
|
||||
t.Fatalf("grant = %v, want empty: this run holds no read-only tools at all", tool.ParentTools)
|
||||
}
|
||||
}
|
||||
|
||||
// The candidate set comes from specialist's exported list, not a second copy
|
||||
// maintained here. Two duplicated lists drift (invariant 5).
|
||||
func TestPlanParentToolsNeverExceedsWhatAPlanMayHold(t *testing.T) {
|
||||
// The bound is PlanGrantableToolNames, not PlanReadOnlyToolNames, and the
|
||||
// difference is the point of the write-task arc: a plan task inherits only
|
||||
// read-only tools, and may NAME one of a short write allow-list. The grant
|
||||
// has to be able to deliver a named write tool or the task would validate
|
||||
// and then run with less than it asked for.
|
||||
allowed := map[string]bool{}
|
||||
for _, name := range specialist.PlanGrantableToolNames() {
|
||||
allowed[name] = true
|
||||
}
|
||||
for _, name := range planParentTools(newCoreRegistry(t.TempDir()), nil, nil) {
|
||||
if !allowed[name] {
|
||||
t.Fatalf("grant contains %q, which is not a tool a plan task may ever hold", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The depth the plan tool measures must be the RUN's depth, not a constant.
|
||||
// Left unset it was always 0, so the admission headroom check and the
|
||||
// executor's own nesting check could never fire — an inert guard that reads
|
||||
// like a live one.
|
||||
func TestRegisteredOrchestrateToolCarriesTheRunsDepth(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
registry := newCoreRegistry(workspace)
|
||||
runtime, err := registerSpecialistTools(registry, workspace, 0, nil, nil, nil, orchestrateWiring{
|
||||
Gate: &specialist.PostureGate{},
|
||||
PlanContext: specialist.PlanTaskContext{Cwd: workspace, Depth: 3},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("registerSpecialistTools: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { closeSpecialistRuntime(nil, runtime) })
|
||||
|
||||
registered, _ := registry.Get(specialist.OrchestrateToolName)
|
||||
tool, ok := registered.(*specialist.OrchestrateTool)
|
||||
if !ok {
|
||||
t.Fatalf("orchestrate is %T", registered)
|
||||
}
|
||||
if tool.Depth != 3 {
|
||||
t.Fatalf("Depth = %d, want the run's depth 3: an always-zero depth makes the headroom check inert", tool.Depth)
|
||||
}
|
||||
}
|
||||
|
||||
// The plan recorder is a REQUIRED parameter, not a wiring field. As a field the
|
||||
// TUI simply never set it, so a plan run there recorded none of its five
|
||||
// lifecycle events — finding 9, and the same omission the parent grant
|
||||
// suffered. Making it positional turns forgetting it into a compile error;
|
||||
// this test pins that a supplied recorder actually reaches the tool.
|
||||
func TestRegisteredOrchestrateToolCarriesTheSuppliedRecorder(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
registry := newCoreRegistry(workspace)
|
||||
recorder := &countingPlanRecorder{}
|
||||
runtime, err := registerSpecialistTools(registry, workspace, 0, nil, nil, recorder, orchestrateWiring{
|
||||
Gate: &specialist.PostureGate{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("registerSpecialistTools: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { closeSpecialistRuntime(nil, runtime) })
|
||||
|
||||
registered, _ := registry.Get(specialist.OrchestrateToolName)
|
||||
tool, ok := registered.(*specialist.OrchestrateTool)
|
||||
if !ok {
|
||||
t.Fatalf("orchestrate is %T", registered)
|
||||
}
|
||||
if tool.Recorder != recorder {
|
||||
t.Fatal("the supplied recorder did not reach the tool, so plan events go nowhere")
|
||||
}
|
||||
}
|
||||
|
||||
type countingPlanRecorder struct{ dispatched int }
|
||||
|
||||
func (r *countingPlanRecorder) TaskDispatched(specialist.Task) { r.dispatched++ }
|
||||
func (r *countingPlanRecorder) TaskCompleted(specialist.TaskResult) {}
|
||||
func (r *countingPlanRecorder) TaskFailed(specialist.TaskResult) {}
|
||||
|
||||
// The plan-size tier must survive registerSpecialistTools. Both call sites pass
|
||||
// it through this function, and a tier read from config but dropped here would
|
||||
// leave every run on the default ceiling while the config file said otherwise —
|
||||
// the same shape as the grant defect above, one layer along.
|
||||
func TestRegisteredOrchestrateToolCarriesTheConfiguredTier(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
registry := newCoreRegistry(workspace)
|
||||
runtime, err := registerSpecialistTools(registry, workspace, 0, nil, nil, nil, orchestrateWiring{
|
||||
Gate: &specialist.PostureGate{},
|
||||
Size: config.PlanSizeSmall,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("registerSpecialistTools: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { closeSpecialistRuntime(nil, runtime) })
|
||||
|
||||
registered, _ := registry.Get(specialist.OrchestrateToolName)
|
||||
tool, ok := registered.(*specialist.OrchestrateTool)
|
||||
if !ok {
|
||||
t.Fatalf("orchestrate is %T", registered)
|
||||
}
|
||||
if tool.Size != config.PlanSizeSmall {
|
||||
t.Fatalf("registered tier = %q; want %q — the wiring dropped it", tool.Size, config.PlanSizeSmall)
|
||||
}
|
||||
}
|
||||
|
||||
// EVERY TOOL A PLAN TASK MAY HOLD MUST BE CLASSIFIED READ-ONLY BY CAPABILITY,
|
||||
// not merely by its safety string.
|
||||
//
|
||||
// This is the invariant that broke, and it broke silently. update_plan declared
|
||||
// readOnlySafety and set no capabilities, so CapabilitiesOf reported
|
||||
// EffectUnknown — the fail-closed default — while its safety said read-only.
|
||||
// runCanMutate reads the CAPABILITY, so one unclassified tool in an otherwise
|
||||
// read-only grant made the whole child look mutating, and ~6,500 characters of
|
||||
// confirmation policy it can never need rode along on every task of every plan.
|
||||
//
|
||||
// Asserted against the REAL registry, because the defect was that two
|
||||
// descriptions of one tool disagreed — checking the list against itself would
|
||||
// have found nothing.
|
||||
func TestEveryPlanGrantToolIsReadOnlyByCapability(t *testing.T) {
|
||||
registry := newCoreRegistry(t.TempDir())
|
||||
checked := 0
|
||||
for _, name := range specialist.PlanReadOnlyToolNames() {
|
||||
tool, found := registry.Get(name)
|
||||
if !found {
|
||||
// Not every read-only name is registered in a bare core registry;
|
||||
// what matters is that the ones that are, classify correctly.
|
||||
continue
|
||||
}
|
||||
checked++
|
||||
if effect := tools.CapabilitiesOf(tool).Effect; effect != tools.EffectReadOnly {
|
||||
t.Errorf("plan grant holds %q whose capability is %v, not EffectReadOnly; "+
|
||||
"a plan child would carry the confirmation policy it can never need", name, effect)
|
||||
}
|
||||
}
|
||||
if checked == 0 {
|
||||
t.Fatal("no plan-grant tool was found in the registry; this test checked nothing")
|
||||
}
|
||||
}
|
||||
|
||||
// BOTH SURFACES SUPPLY AN ISOLATOR. A write-capable plan is refused where none
|
||||
// exists, so a surface that forgot to wire one would refuse every write plan
|
||||
// with "this run cannot isolate" — correct, and useless.
|
||||
func TestBothSurfacesWireAnIsolator(t *testing.T) {
|
||||
if newPlanIsolator("") != nil {
|
||||
t.Fatal("an empty workspace root produced an isolator")
|
||||
}
|
||||
if newPlanIsolator(t.TempDir()) == nil {
|
||||
t.Fatal("a real workspace root produced no isolator")
|
||||
}
|
||||
|
||||
workspace := t.TempDir()
|
||||
registry := newCoreRegistry(workspace)
|
||||
runtime, err := registerSpecialistTools(registry, workspace, 0, nil, nil, nil, orchestrateWiring{
|
||||
Gate: &specialist.PostureGate{},
|
||||
Isolate: newPlanIsolator(workspace),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("registerSpecialistTools: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { closeSpecialistRuntime(nil, runtime) })
|
||||
registered, _ := registry.Get(specialist.OrchestrateToolName)
|
||||
tool, ok := registered.(*specialist.OrchestrateTool)
|
||||
if !ok {
|
||||
t.Fatalf("orchestrate is %T", registered)
|
||||
}
|
||||
if tool.Isolate == nil {
|
||||
t.Fatal("the wiring dropped the isolator; every write-capable plan would be refused")
|
||||
}
|
||||
}
|
||||
|
||||
// The worktree NAME is derived, not taken. A plan name is model-supplied and
|
||||
// reaches a filesystem path, so what comes out has to be a name — the allow-list
|
||||
// in worktrees.Prepare is the guarantee, and this is what feeds it something it
|
||||
// will accept rather than something it will reject.
|
||||
func TestAPlanNameBecomesAUsableWorktreeName(t *testing.T) {
|
||||
for _, tc := range []struct{ in, want string }{
|
||||
{"sweep", "plan-sweep"},
|
||||
{"pre release", "plan-prerelease"},
|
||||
{"../../etc/passwd", "plan-etcpasswd"},
|
||||
{"", "plan"},
|
||||
{"!!!", "plan"},
|
||||
{"a/b\\c;d", "plan-abcd"},
|
||||
} {
|
||||
if got := planWorktreeName(tc.in); got != tc.want {
|
||||
t.Errorf("planWorktreeName(%q) = %q, want %q", tc.in, got, tc.want)
|
||||
}
|
||||
}
|
||||
// Long names are bounded, so a plan cannot produce a path component the
|
||||
// filesystem refuses.
|
||||
if got := planWorktreeName(strings.Repeat("x", 300)); len(got) > 64 {
|
||||
t.Fatalf("a long plan name produced a %d-character worktree name", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
// THE HOOK MUST ACTUALLY BE WIRED. DiscoverModels is consulted only when a plan
|
||||
// sets auto_assign, so a nil left in the wiring compiles, ships, and refuses
|
||||
// every such plan with "not available in this run" — the feature present in the
|
||||
// schema and reachable from nowhere. That is this branch's most repeated defect,
|
||||
// and it is checked here rather than trusted.
|
||||
func TestOrchestrateGetsAModelDiscoverer(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
gate := &specialist.PostureGate{}
|
||||
gate.Set(true)
|
||||
workspace := t.TempDir()
|
||||
|
||||
if _, err := registerSpecialistTools(registry, workspace, 0, nil, nil, nil, orchestrateWiring{
|
||||
Gate: gate,
|
||||
PlanContext: specialist.PlanTaskContext{Cwd: workspace},
|
||||
DiscoverModels: planModelDiscoverer(t.TempDir(), config.ProviderProfile{Name: "test", Model: "gpt-4.1"}),
|
||||
}); err != nil {
|
||||
t.Fatalf("register: %v", err)
|
||||
}
|
||||
|
||||
tool, ok := registry.Get(specialist.OrchestrateToolName)
|
||||
if !ok {
|
||||
t.Fatal("the orchestrate tool was not registered")
|
||||
}
|
||||
orchestrate, ok := tool.(*specialist.OrchestrateTool)
|
||||
if !ok {
|
||||
t.Fatalf("registered tool is %T", tool)
|
||||
}
|
||||
if orchestrate.DiscoverModels == nil {
|
||||
t.Error("DiscoverModels is nil on the registered tool; auto_assign would refuse every plan")
|
||||
}
|
||||
}
|
||||
|
||||
// The user's model pins and exclusions must reach the tool. Configured and not
|
||||
// wired, they are a config key that silently does nothing — the same shape as
|
||||
// DiscoverModels above, which shipped nil.
|
||||
func TestOrchestrateGetsTheConfiguredModelPreferences(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
gate := &specialist.PostureGate{}
|
||||
gate.Set(true)
|
||||
workspace := t.TempDir()
|
||||
|
||||
if _, err := registerSpecialistTools(registry, workspace, 0, nil, nil, nil, orchestrateWiring{
|
||||
Gate: gate,
|
||||
PlanContext: specialist.PlanTaskContext{Cwd: workspace},
|
||||
ModelPrefs: planModelPreferences(config.PlanModelsConfig{
|
||||
Verify: "deepseek-v4-pro",
|
||||
Exclude: []string{"grok-build-0.1"},
|
||||
}),
|
||||
}); err != nil {
|
||||
t.Fatalf("register: %v", err)
|
||||
}
|
||||
tool, _ := registry.Get(specialist.OrchestrateToolName)
|
||||
orchestrate, ok := tool.(*specialist.OrchestrateTool)
|
||||
if !ok {
|
||||
t.Fatalf("registered tool is %T", tool)
|
||||
}
|
||||
if orchestrate.ModelPrefs.Verify != "deepseek-v4-pro" {
|
||||
t.Errorf("the verify pin did not reach the tool: %+v", orchestrate.ModelPrefs)
|
||||
}
|
||||
if len(orchestrate.ModelPrefs.Exclude) != 1 {
|
||||
t.Errorf("the exclusion list did not reach the tool: %+v", orchestrate.ModelPrefs)
|
||||
}
|
||||
}
|
||||
|
||||
// And the adapter really produces the shape specialist chooses between, rather
|
||||
// than an empty struct that would make every tier unassignable.
|
||||
func TestTheModelDiscovererCarriesWhatAPlanChoosesBetween(t *testing.T) {
|
||||
discover := planModelDiscoverer(t.TempDir(), config.ProviderProfile{Name: "test", Model: "gpt-4.1"})
|
||||
if discover == nil {
|
||||
t.Fatal("adapter returned nil")
|
||||
}
|
||||
// The network call is expected to fail in a test environment; what matters is
|
||||
// that a failure is REPORTED rather than silently yielding no models, which
|
||||
// would make auto_assign a no-op nobody could diagnose.
|
||||
models, err := discover(context.Background())
|
||||
if err == nil && len(models) > 0 {
|
||||
for _, model := range models {
|
||||
if strings.TrimSpace(model.ID) == "" {
|
||||
t.Errorf("a discovered model arrived with no id: %+v", model)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,162 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
"github.com/Twigpine/zero/internal/worktrees"
|
||||
)
|
||||
|
||||
// The isolator: a git worktree for a plan that may write.
|
||||
//
|
||||
// It reuses internal/worktrees, the same package `zero exec --worktree` already
|
||||
// uses, rather than shelling out to git again. That matters beyond tidiness —
|
||||
// worktrees.Prepare already handles name validation, the base directory, repo
|
||||
// discovery from a subdirectory, and the difference between a checkout and a
|
||||
// worktree. A second implementation would eventually disagree with the first
|
||||
// about which of those a plan gets.
|
||||
//
|
||||
// NOT DELETED ON RELEASE. A plan that wrote something produced work the user has
|
||||
// not seen yet, and removing the tree would remove the only copy of it. Release
|
||||
// therefore keeps the tree and drops only the lease, exactly as
|
||||
// `zero exec --worktree` does when it exits: the lease protects a tree while a
|
||||
// run is USING it, and worktrees.Clean never removes an unlocked tree that holds
|
||||
// uncommitted work (and preserves an unreachable HEAD before removing a clean
|
||||
// one), so the retained output survives without a lock nobody can clear.
|
||||
//
|
||||
// ONE TREE PER EXECUTION. The name carries a per-run suffix, so a second run of
|
||||
// the same plan neither collides with the first run's tree nor starts on top of
|
||||
// output the user has not reviewed yet. With a fixed name and a PID-less lease
|
||||
// that Release never dropped, every rerun of a write-capable plan was refused as
|
||||
// "locked by another active run" until someone released it by hand.
|
||||
|
||||
// planWorktreeName derives a worktree name from a plan name.
|
||||
//
|
||||
// worktrees.Prepare enforces its own allow-list on the result, so this only has
|
||||
// to produce a candidate; anything it cannot clean is handed over as a prefix
|
||||
// and Prepare supplies the rest. Deliberately NOT a sanitiser that promises
|
||||
// safety — the guarantee lives in one place, where it is tested.
|
||||
func planWorktreeName(planName string) string {
|
||||
cleaned := strings.Map(func(r rune) rune {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9':
|
||||
return r
|
||||
case r == '-', r == '_':
|
||||
return r
|
||||
default:
|
||||
return -1
|
||||
}
|
||||
}, planName)
|
||||
cleaned = strings.TrimSpace(cleaned)
|
||||
if cleaned == "" {
|
||||
return "plan"
|
||||
}
|
||||
if len(cleaned) > 40 {
|
||||
cleaned = cleaned[:40]
|
||||
}
|
||||
return "plan-" + cleaned
|
||||
}
|
||||
|
||||
// newPlanIsolator returns an isolator rooted at the run's workspace, or nil when
|
||||
// this run cannot isolate at all.
|
||||
//
|
||||
// nil is the meaningful answer, not an error: specialist refuses a plan that
|
||||
// requires isolation when the isolator is nil, with a reason. Returning a
|
||||
// non-nil isolator that always fails would move the same refusal later and make
|
||||
// it read like a malfunction.
|
||||
func newPlanIsolator(workspaceRoot string) specialist.PlanIsolator {
|
||||
return newPlanIsolatorWith(workspaceRoot, planIsolatorDeps{})
|
||||
}
|
||||
|
||||
// planIsolatorDeps are the seams a test needs to drive the isolator against a
|
||||
// real repository without touching the user's state directory or the clock.
|
||||
type planIsolatorDeps struct {
|
||||
baseDir string
|
||||
now func() time.Time
|
||||
suffix func() string
|
||||
}
|
||||
|
||||
func newPlanIsolatorWith(workspaceRoot string, deps planIsolatorDeps) specialist.PlanIsolator {
|
||||
if strings.TrimSpace(workspaceRoot) == "" {
|
||||
return nil
|
||||
}
|
||||
now := deps.now
|
||||
if now == nil {
|
||||
now = time.Now
|
||||
}
|
||||
suffix := deps.suffix
|
||||
if suffix == nil {
|
||||
suffix = randomPlanWorktreeSuffix
|
||||
}
|
||||
return func(ctx context.Context, planName string) (specialist.PlanWorkspace, error) {
|
||||
prepared, err := worktrees.Prepare(ctx, worktrees.Options{
|
||||
Cwd: workspaceRoot,
|
||||
Name: planExecutionWorktreeName(planName, now(), suffix()),
|
||||
BaseDir: deps.baseDir,
|
||||
// The lease is bound to THIS process, like `zero exec --worktree`:
|
||||
// if Zero dies before Release runs, Clean can expire it instead of
|
||||
// skipping the tree forever.
|
||||
LeasePID: os.Getpid(),
|
||||
})
|
||||
if err != nil {
|
||||
return specialist.PlanWorkspace{}, err
|
||||
}
|
||||
if strings.TrimSpace(prepared.Path) == "" {
|
||||
// Defence in depth against a Prepare that reports success with no
|
||||
// path: specialist refuses an un-isolated workspace, and this makes
|
||||
// sure it sees one rather than an empty string it might read as the
|
||||
// parent's own directory.
|
||||
return specialist.PlanWorkspace{}, fmt.Errorf("prepared worktree has no path")
|
||||
}
|
||||
// Only the lease this call took is ours to drop. A tree another live
|
||||
// caller already held reports LockAcquired=false, and releasing it would
|
||||
// expose that caller's workspace to Clean.
|
||||
release := func() {}
|
||||
if prepared.LockAcquired {
|
||||
var once sync.Once
|
||||
release = func() {
|
||||
once.Do(func() {
|
||||
_ = worktrees.Release(context.Background(), worktrees.Options{Cwd: workspaceRoot}, prepared.Path)
|
||||
})
|
||||
}
|
||||
}
|
||||
return specialist.PlanWorkspace{
|
||||
Path: prepared.Path,
|
||||
Isolated: true,
|
||||
Describe: fmt.Sprintf("worktree %s at %s", prepared.Name, prepared.Path),
|
||||
// Release drops the lease and KEEPS the tree: the plan's output is
|
||||
// unreviewed work, and Clean leaves an unlocked tree with uncommitted
|
||||
// changes alone.
|
||||
Release: release,
|
||||
// Abort drops the lease on a tree the plan never received. It is
|
||||
// fresh and untouched, so leaving it unlocked lets Clean reclaim it.
|
||||
Abort: release,
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// planExecutionWorktreeName is the per-execution worktree name: the plan-derived
|
||||
// prefix plus a UTC timestamp and a random suffix, so two runs of one plan —
|
||||
// even in the same second — never share a tree.
|
||||
func planExecutionWorktreeName(planName string, at time.Time, suffix string) string {
|
||||
name := planWorktreeName(planName) + "-" + at.UTC().Format("20060102-150405")
|
||||
if suffix = strings.TrimSpace(suffix); suffix != "" {
|
||||
name += "-" + suffix
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
func randomPlanWorktreeSuffix() string {
|
||||
var b [4]byte
|
||||
if _, err := rand.Read(b[:]); err != nil {
|
||||
return ""
|
||||
}
|
||||
return hex.EncodeToString(b[:])
|
||||
}
|
||||
@@ -1,113 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// worktreeLockReason returns the lock reason git records for path, and whether
|
||||
// it is locked at all.
|
||||
func worktreeLockReason(t *testing.T, repo, path string) (string, bool) {
|
||||
t.Helper()
|
||||
out, err := exec.Command("git", "-C", repo, "worktree", "list", "--porcelain").CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git worktree list: %v\n%s", err, out)
|
||||
}
|
||||
want, _ := filepath.EvalSymlinks(path)
|
||||
var current string
|
||||
for _, line := range strings.Split(string(out), "\n") {
|
||||
switch {
|
||||
case strings.HasPrefix(line, "worktree "):
|
||||
current = strings.TrimPrefix(line, "worktree ")
|
||||
if resolved, err := filepath.EvalSymlinks(current); err == nil {
|
||||
current = resolved
|
||||
}
|
||||
case current == want && (line == "locked" || strings.HasPrefix(line, "locked ")):
|
||||
return strings.TrimSpace(strings.TrimPrefix(line, "locked")), true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// A WRITE-CAPABLE PLAN MUST BE ABLE TO RUN TWICE. The isolator used a fixed
|
||||
// worktree name per plan, a PID-less lease, and a Release that did nothing, so
|
||||
// the second run of the same plan found its tree "locked by another active run"
|
||||
// and was refused until someone ran `zero worktrees release` by hand.
|
||||
//
|
||||
// Driven against a real repository: each execution gets its own tree, the
|
||||
// lease names this process while the plan runs, Release drops the lease, and
|
||||
// the first run's unreviewed output is still on disk afterwards.
|
||||
func TestAWriteCapablePlanCanRunAgainAfterItFinishes(t *testing.T) {
|
||||
repo := initTempGitRepo(t)
|
||||
base := t.TempDir()
|
||||
clock := time.Date(2026, 11, 7, 12, 0, 0, 0, time.UTC)
|
||||
suffixes := []string{"aaaa0001", "aaaa0002"}
|
||||
isolate := newPlanIsolatorWith(repo, planIsolatorDeps{
|
||||
baseDir: base,
|
||||
// The SAME second for both runs, so only the suffix separates them.
|
||||
now: func() time.Time { return clock },
|
||||
suffix: func() string {
|
||||
next := suffixes[0]
|
||||
suffixes = suffixes[1:]
|
||||
return next
|
||||
},
|
||||
})
|
||||
|
||||
first, err := isolate(context.Background(), "sweep")
|
||||
if err != nil {
|
||||
t.Fatalf("first run: %v", err)
|
||||
}
|
||||
reason, locked := worktreeLockReason(t, repo, first.Path)
|
||||
if !locked {
|
||||
t.Fatal("the tree a running plan uses must be leased so Clean cannot prune it")
|
||||
}
|
||||
if !strings.Contains(reason, "(pid "+strconv.Itoa(os.Getpid())+")") {
|
||||
t.Fatalf("the lease must name this process so a crash is recoverable, got %q", reason)
|
||||
}
|
||||
output := filepath.Join(first.Path, "result.txt")
|
||||
if err := os.WriteFile(output, []byte("unreviewed plan output\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
first.Release()
|
||||
|
||||
if _, locked := worktreeLockReason(t, repo, first.Path); locked {
|
||||
t.Fatal("Release left the finished plan's tree locked; nothing would ever clear it")
|
||||
}
|
||||
if data, err := os.ReadFile(output); err != nil || string(data) != "unreviewed plan output\n" {
|
||||
t.Fatalf("Release must keep the plan's output for review: %q, %v", data, err)
|
||||
}
|
||||
|
||||
second, err := isolate(context.Background(), "sweep")
|
||||
if err != nil {
|
||||
t.Fatalf("a second run of the same plan was refused: %v", err)
|
||||
}
|
||||
defer second.Release()
|
||||
if second.Path == first.Path {
|
||||
t.Fatalf("the second run reused the first run's tree %s, on top of output nobody reviewed", first.Path)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(second.Path, "result.txt")); !os.IsNotExist(err) {
|
||||
t.Fatalf("the second run's tree carries the first run's output: %v", err)
|
||||
}
|
||||
if !strings.HasPrefix(filepath.Base(second.Path), "plan-sweep-") {
|
||||
t.Fatalf("the tree name no longer says which plan made it: %s", second.Path)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPlanExecutionWorktreeNameIsAcceptedAndBounded(t *testing.T) {
|
||||
at := time.Date(2026, 11, 7, 12, 0, 0, 0, time.UTC)
|
||||
got := planExecutionWorktreeName(strings.Repeat("x", 300), at, "deadbeef")
|
||||
// worktrees.Prepare's allow-list (worktreeNamePattern there).
|
||||
if !regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,80}$`).MatchString(got) {
|
||||
t.Fatalf("worktrees.Prepare would reject %q", got)
|
||||
}
|
||||
if got := planExecutionWorktreeName("sweep", at, ""); got != "plan-sweep-20261107-120000" {
|
||||
t.Fatalf("name without a suffix = %q", got)
|
||||
}
|
||||
}
|
||||
@@ -1,131 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
)
|
||||
|
||||
// pointConfigHomeAtATempDir redirects config.UserConfigDir at a scratch directory
|
||||
// and returns it.
|
||||
//
|
||||
// The variable that does the redirecting IS NOT THE SAME ON EVERY PLATFORM:
|
||||
// os.UserConfigDir reads %AppData% on Windows and $XDG_CONFIG_HOME elsewhere, and
|
||||
// config.UserConfigDir lets XDG win on macOS too. Setting only XDG_CONFIG_HOME
|
||||
// left Windows resolving against the real %AppData%, where this file was never
|
||||
// written — so every case below silently exercised the resolve-failed fallback
|
||||
// and the switch-following test could not fail on Windows for the right reason.
|
||||
// Mirrors internal/config/paths_test.go's setUserConfigRoot.
|
||||
func pointConfigHomeAtATempDir(t *testing.T) string {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
switch runtime.GOOS {
|
||||
case "windows":
|
||||
t.Setenv("APPDATA", root)
|
||||
default:
|
||||
t.Setenv("XDG_CONFIG_HOME", root)
|
||||
}
|
||||
base, err := config.UserConfigDir()
|
||||
if err != nil {
|
||||
t.Fatalf("resolve user config dir: %v", err)
|
||||
}
|
||||
// The redirect must have LANDED. Without this, setting the wrong variable
|
||||
// for the platform resolves back to the developer's real config directory:
|
||||
// the helper writes a provider file into it and the tests pass for the
|
||||
// wrong reason — which is exactly how this went unnoticed on Windows.
|
||||
if !strings.HasPrefix(base, root) {
|
||||
t.Fatalf("config home resolved to %q, outside the temp root %q: this test would read and write the real user config", base, root)
|
||||
}
|
||||
return base
|
||||
}
|
||||
|
||||
func writeTwoProviderConfig(t *testing.T, active string) {
|
||||
t.Helper()
|
||||
dir := filepath.Join(pointConfigHomeAtATempDir(t), "zero")
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The provider shape a real config uses — provider_kind + apiFormat matter:
|
||||
// without them "xai" is read as an OpenAI profile and refused for not using
|
||||
// the official base URL.
|
||||
body := `{
|
||||
"activeProvider": "` + active + `",
|
||||
"providers": [
|
||||
{"name": "ollama-cloud", "provider_kind": "openai-compatible", "catalogID": "ollama-cloud",
|
||||
"baseURL": "https://ollama.com/v1", "apiFormat": "chat-completions",
|
||||
"model": "glm-5.2", "apiKey": "k1"},
|
||||
{"name": "xai", "provider_kind": "openai-compatible", "catalogID": "xai",
|
||||
"baseURL": "https://api.x.ai/v1", "apiFormat": "chat-completions",
|
||||
"model": "grok-4.5", "apiKey": "k2"}
|
||||
]
|
||||
}`
|
||||
if err := os.WriteFile(filepath.Join(dir, "config.json"), []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// DISCOVERY MUST FOLLOW AN IN-SESSION PROVIDER SWITCH, because child execution
|
||||
// already does and the two must never describe different providers.
|
||||
//
|
||||
// The orchestrate tool is wired once at startup. The TUI's /model picker can
|
||||
// switch provider afterwards; switchProviderModel re-points the parent client and
|
||||
// exports ZERO_PROVIDER so spawned children follow. Before this, nothing
|
||||
// re-pointed model discovery — a session on xai/grok-4.5 had its plan assigned
|
||||
// from a nineteen-model Ollama list and the provider rejected every task.
|
||||
func TestPlanDiscoveryFollowsAnInSessionProviderSwitch(t *testing.T) {
|
||||
writeTwoProviderConfig(t, "ollama-cloud")
|
||||
captured := config.ProviderProfile{Name: "ollama-cloud", BaseURL: "https://ollama.com/v1", Model: "glm-5.2"}
|
||||
|
||||
// What switchProviderModel does when the user picks a model from another
|
||||
// saved provider.
|
||||
t.Setenv(config.ActiveProviderEnv, "xai")
|
||||
|
||||
got := livePlanProvider(t.TempDir(), captured)
|
||||
if got.Name != "xai" {
|
||||
t.Fatalf("discovery stayed on the launch-time provider %q after a switch to xai", got.Name)
|
||||
}
|
||||
if got.BaseURL != "https://api.x.ai/v1" {
|
||||
t.Errorf("the switched profile carries the wrong endpoint: %q", got.BaseURL)
|
||||
}
|
||||
}
|
||||
|
||||
// THE CAPTURED PROFILE WINS WHEN NOTHING SWITCHED, and this is what keeps
|
||||
// headless runs correct. --provider, --base-url and an inline --api-key are
|
||||
// applied by the caller on top of Resolve, so re-resolving would silently discard
|
||||
// them and probe the wrong endpoint with the wrong key.
|
||||
func TestPlanDiscoveryKeepsFlagOverridesWhenTheProviderDidNotChange(t *testing.T) {
|
||||
writeTwoProviderConfig(t, "xai")
|
||||
// A profile that exists nowhere in the config file: exactly what a
|
||||
// --base-url / --api-key override produces.
|
||||
captured := config.ProviderProfile{
|
||||
Name: "xai",
|
||||
BaseURL: "https://gateway.internal/v1",
|
||||
Model: "grok-4.5",
|
||||
APIKey: "flag-only-key",
|
||||
}
|
||||
// Empty reads as unset (applyEnv TrimSpaces it) and t.Setenv restores the
|
||||
// real value afterwards — os.Unsetenv here would leak into sibling tests.
|
||||
t.Setenv(config.ActiveProviderEnv, "")
|
||||
|
||||
got := livePlanProvider(t.TempDir(), captured)
|
||||
if got.BaseURL != "https://gateway.internal/v1" || got.APIKey != "flag-only-key" {
|
||||
t.Fatalf("re-resolution discarded the caller's overrides: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// An unreadable or absent config must not blank the profile discovery probes
|
||||
// with. Falling back to the captured one leaves behaviour exactly as it was.
|
||||
func TestPlanDiscoveryFallsBackToTheCapturedProfileWhenResolutionFails(t *testing.T) {
|
||||
// A config home with no zero/config.json in it: resolution finds nothing.
|
||||
pointConfigHomeAtATempDir(t)
|
||||
t.Setenv(config.ActiveProviderEnv, "")
|
||||
captured := config.ProviderProfile{Name: "xai", BaseURL: "https://api.x.ai/v1", Model: "grok-4.5"}
|
||||
|
||||
if got := livePlanProvider(t.TempDir(), captured); got.Name != "xai" || got.BaseURL != captured.BaseURL {
|
||||
t.Fatalf("a failed resolve must leave the captured profile intact, got %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -1,83 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/providermodeldiscovery"
|
||||
)
|
||||
|
||||
// A PLAN MUST NOT BE OFFERED A MODEL ITS OWN PROVIDER WILL REFUSE.
|
||||
//
|
||||
// providers.New applies providermodelcatalog's per-provider allow-list at client
|
||||
// creation (validateModelAllowedForProvider), and `providers models` applies it
|
||||
// before printing. The plan discoverer did not, so on a scoped provider the
|
||||
// router was handed candidates that could never run: assignment succeeded, the
|
||||
// panel showed the model, and the task died at dispatch with "provider X does
|
||||
// not allow model Y" — the same invisible-until-dispatch failure as being
|
||||
// assigned from another provider's list altogether.
|
||||
func TestPlanDiscoveryDropsModelsTheProviderWillRefuse(t *testing.T) {
|
||||
// opencode-go-anthropic-compatible is scoped to Qwen and MiniMax families.
|
||||
found := []providermodeldiscovery.Model{
|
||||
{ID: "qwen3.5-coder", ToolCall: true},
|
||||
{ID: "claude-sonnet-5", ToolCall: true},
|
||||
{ID: "minimax-m2", ToolCall: true},
|
||||
{ID: "gpt-5", ToolCall: true},
|
||||
}
|
||||
|
||||
got := planModelsFromDiscovered("opencode-go-anthropic-compatible", found)
|
||||
|
||||
ids := map[string]bool{}
|
||||
for _, model := range got {
|
||||
ids[model.ID] = true
|
||||
}
|
||||
for _, refused := range []string{"claude-sonnet-5", "gpt-5"} {
|
||||
if ids[refused] {
|
||||
t.Errorf("%q reached the plan; providers.New refuses it for this provider, so every task assigned it dies at dispatch", refused)
|
||||
}
|
||||
}
|
||||
for _, allowed := range []string{"qwen3.5-coder", "minimax-m2"} {
|
||||
if !ids[allowed] {
|
||||
t.Errorf("%q was dropped, but this provider serves it", allowed)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An UNSCOPED provider keeps its whole list. The allow-list defaults to
|
||||
// permitting everything, and a filter that quietly narrowed an ordinary provider
|
||||
// would remove models a plan is entitled to use.
|
||||
func TestPlanDiscoveryKeepsEveryModelOnAnUnscopedProvider(t *testing.T) {
|
||||
found := []providermodeldiscovery.Model{
|
||||
{ID: "grok-4.5", ToolCall: true},
|
||||
{ID: "grok-code-fast", ToolCall: true},
|
||||
{ID: "claude-sonnet-5", ToolCall: true},
|
||||
}
|
||||
for _, catalogID := range []string{"xai", "", " "} {
|
||||
got := planModelsFromDiscovered(catalogID, found)
|
||||
if len(got) != len(found) {
|
||||
t.Errorf("catalogID %q narrowed an unscoped provider from %d models to %d", catalogID, len(found), len(got))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The translation must carry every field the plan chooses between — a model that
|
||||
// arrives with its costs zeroed is routed as if it were free.
|
||||
func TestPlanDiscoveryCarriesTheFieldsRoutingReads(t *testing.T) {
|
||||
got := planModelsFromDiscovered("xai", []providermodeldiscovery.Model{{
|
||||
ID: "grok-4.5", Description: "frontier", ToolCall: true, Reasoning: true,
|
||||
InputCost: 3, OutputCost: 15, OutputModalities: []string{"text"},
|
||||
}})
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("got %d models, want 1", len(got))
|
||||
}
|
||||
model := got[0]
|
||||
switch {
|
||||
case model.ID != "grok-4.5", model.Description != "frontier":
|
||||
t.Errorf("identity lost: %+v", model)
|
||||
case !model.ToolCall, !model.Reasoning:
|
||||
t.Errorf("capabilities lost, so routing cannot tell this model apart: %+v", model)
|
||||
case model.InputCost != 3, model.OutputCost != 15:
|
||||
t.Errorf("costs lost, so the cheapest-model rule ranks this as free: %+v", model)
|
||||
case len(model.OutputModalities) != 1:
|
||||
t.Errorf("modalities lost: %+v", model)
|
||||
}
|
||||
}
|
||||
@@ -1,63 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
)
|
||||
|
||||
// EVERY CONFIGURED PREFERENCE MUST SURVIVE THE TRANSLATION, and this is checked
|
||||
// by reflection rather than field by field on purpose.
|
||||
//
|
||||
// planModelPreferences is a hand-written copy between two structs that are
|
||||
// deliberately kept apart, so specialist need not import config. Hand-written
|
||||
// copies rot silently: a field gets added to both sides and to nothing in the
|
||||
// middle, and the symptom is a setting the user wrote in their config having no
|
||||
// effect — with nothing logged, nothing failing, and no way to tell it from the
|
||||
// feature simply not working. RouterGuidance was added this way; the next one
|
||||
// will be too.
|
||||
//
|
||||
// Naming the fields here would only pin today's set. Walking them means a new
|
||||
// field is caught the moment it exists.
|
||||
func TestEveryPlanModelPreferenceSurvivesTheCarryIntoSpecialist(t *testing.T) {
|
||||
source := config.PlanModelsConfig{
|
||||
Scan: "scan-model",
|
||||
Implement: "implement-model",
|
||||
Verify: "verify-model",
|
||||
Router: "router-model",
|
||||
RouterGuidance: "prefer kimi for judgement",
|
||||
AutoAssign: true,
|
||||
Exclude: []string{"never-this"},
|
||||
MinSize: 7,
|
||||
}
|
||||
carried := planModelPreferences(source)
|
||||
|
||||
from := reflect.ValueOf(source)
|
||||
to := reflect.ValueOf(carried)
|
||||
toType := to.Type()
|
||||
for i := 0; i < from.NumField(); i++ {
|
||||
name := from.Type().Field(i).Name
|
||||
if _, ok := toType.FieldByName(name); !ok {
|
||||
// A config field with no counterpart is a deliberate decision — say so
|
||||
// here when you make one, so the next reader knows it was not an
|
||||
// oversight. Today every field has one.
|
||||
t.Errorf("config field %q has no counterpart in specialist.ModelPreferences; "+
|
||||
"if that is intended, exempt it here with a reason", name)
|
||||
continue
|
||||
}
|
||||
want := from.Field(i).Interface()
|
||||
got := to.FieldByName(name).Interface()
|
||||
if !reflect.DeepEqual(want, got) {
|
||||
t.Errorf("%s was dropped or altered on the way into specialist: config had %#v, specialist got %#v",
|
||||
name, want, got)
|
||||
}
|
||||
}
|
||||
|
||||
// Guard the guard: a zero source would make every comparison trivially pass,
|
||||
// so prove the fixture actually set something.
|
||||
if reflect.DeepEqual(carried, specialist.ModelPreferences{}) {
|
||||
t.Fatal("the fixture carried nothing, so this test proves nothing")
|
||||
}
|
||||
}
|
||||
@@ -1,123 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// probeTimeout bounds one model's proof. Generous enough for a cold model to
|
||||
// load and short enough that proving nineteen of them concurrently costs one
|
||||
// pause rather than a stall — and a model too slow to answer this is not one a
|
||||
// plan wants either.
|
||||
const probeTimeout = 20 * time.Second
|
||||
|
||||
// planModelProber proves that a provider will actually RUN a model.
|
||||
//
|
||||
// DISCOVERY IS AN ADVERTISEMENT AND THIS IS THE RECEIPT. /v1/models reports what
|
||||
// a provider is willing to name; only a real request reports what it will serve.
|
||||
// Every model failure this feature has produced lived in that gap:
|
||||
// grok-4.20-multi-agent-0309 lists like anything else and answers "Multi Agent
|
||||
// requests are not allowed on chat completions"; a stale list from another
|
||||
// provider lists perfectly and serves nothing.
|
||||
//
|
||||
// The request is deliberately the smallest real one: a single word, one token
|
||||
// back. It proves the route end to end — credential, endpoint, model id,
|
||||
// entitlement — which is exactly the set of things a list cannot prove.
|
||||
func planModelProber(workspaceRoot string, profile config.ProviderProfile, newProvider func(config.ProviderProfile) (zeroruntime.Provider, error)) specialist.ModelProber {
|
||||
if newProvider == nil {
|
||||
return nil
|
||||
}
|
||||
return func(ctx context.Context, modelID string) specialist.ModelProbeResult {
|
||||
id := strings.TrimSpace(modelID)
|
||||
if id == "" {
|
||||
return specialist.ModelProbeResult{Verdict: specialist.ProbeUnknown}
|
||||
}
|
||||
// The LIVE profile, for the same reason discovery re-resolves it: a
|
||||
// session that switched provider must be proved against the one it is
|
||||
// actually on, not the one it started with.
|
||||
target := livePlanProvider(workspaceRoot, profile)
|
||||
target.Model = id
|
||||
provider, err := newProvider(discoveryCredentialProfile(target))
|
||||
if err != nil {
|
||||
// Could not even build a client: a fact about this run, not about
|
||||
// this model. Unknown keeps the model.
|
||||
return specialist.ModelProbeResult{Verdict: specialist.ProbeUnknown, Reason: err.Error()}
|
||||
}
|
||||
|
||||
probeCtx, cancel := context.WithTimeout(ctx, probeTimeout)
|
||||
defer cancel()
|
||||
stream, err := provider.StreamCompletion(probeCtx, zeroruntime.CompletionRequest{
|
||||
Messages: []zeroruntime.Message{{Role: zeroruntime.MessageRoleUser, Content: "ping"}},
|
||||
})
|
||||
if err != nil {
|
||||
return specialist.ClassifyProbeError(err)
|
||||
}
|
||||
collected := zeroruntime.CollectStream(probeCtx, stream)
|
||||
if collected.Error != "" {
|
||||
// A refusal usually arrives HERE rather than from StreamCompletion:
|
||||
// the request is accepted, the stream opens, and the provider's
|
||||
// complaint about the model comes back as the first event. Reading
|
||||
// only the call's own error is how "the model does not exist" was
|
||||
// mistaken for a working model.
|
||||
return specialist.ClassifyProbeError(errStub(collected.Error))
|
||||
}
|
||||
return specialist.ModelProbeResult{Verdict: specialist.ProbeServes}
|
||||
}
|
||||
}
|
||||
|
||||
// planModelProbeIdentity returns the non-secret routing identity that makes a
|
||||
// probe verdict true. Model IDs overlap across providers, profiles and custom
|
||||
// endpoints, so none of those dimensions may share a cache namespace.
|
||||
func planModelProbeIdentity(workspaceRoot string, profile config.ProviderProfile) func() string {
|
||||
return func() string {
|
||||
live := livePlanProvider(workspaceRoot, profile)
|
||||
return strings.Join([]string{
|
||||
strings.TrimSpace(live.Name),
|
||||
strings.TrimSpace(live.Provider),
|
||||
strings.TrimSpace(string(live.ProviderKind)),
|
||||
strings.TrimSpace(live.CatalogID),
|
||||
strings.TrimSpace(live.BaseURL),
|
||||
strings.TrimSpace(live.APIFormat),
|
||||
strings.TrimSpace(live.APIKeyEnv),
|
||||
}, "\x00")
|
||||
}
|
||||
}
|
||||
|
||||
// errStub carries a provider's message into the classifier, which takes an error
|
||||
// because every other caller has one.
|
||||
type errStub string
|
||||
|
||||
func (e errStub) Error() string { return string(e) }
|
||||
|
||||
// probeStatusWord names a verdict for a person. The three words are deliberately
|
||||
// distinct: "unreachable" is a fact about the connection, not about the model,
|
||||
// and reading it as a refusal is how a working model gets dropped on a flaky
|
||||
// network.
|
||||
func probeStatusWord(v specialist.ModelProbeVerdict) string {
|
||||
switch v {
|
||||
case specialist.ProbeServes:
|
||||
return "serves"
|
||||
case specialist.ProbeRefuses:
|
||||
return "refuses"
|
||||
default:
|
||||
return "unreachable"
|
||||
}
|
||||
}
|
||||
|
||||
// firstProbeLine keeps a provider's complaint to one readable line; these arrive
|
||||
// as multi-line JSON often enough to matter.
|
||||
func firstProbeLine(s string) string {
|
||||
if index := strings.IndexAny(s, "\r\n"); index >= 0 {
|
||||
s = s[:index]
|
||||
}
|
||||
const limit = 120
|
||||
if len(s) > limit {
|
||||
s = s[:limit] + "…"
|
||||
}
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
@@ -1,85 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/Twigpine/zero/internal/sessions"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
)
|
||||
|
||||
// planSessionRecorder bridges plan lifecycle events onto the existing exec
|
||||
// session recorder.
|
||||
//
|
||||
// IT PRESERVES THE BEST-EFFORT CONTRACT EXACTLY. execSessionRecorder.append
|
||||
// returns nothing, latches its first error, and short-circuits every later
|
||||
// call; warnIfRecordingFailed surfaces that once at run end. This type adds no
|
||||
// error path of its own: every method returns nothing, so there is no way for a
|
||||
// recording failure to reach ExecutePlan, and therefore no way for it to abort
|
||||
// a plan mid-flight. A nil recorder is a no-op rather than a panic, for the
|
||||
// same reason.
|
||||
//
|
||||
// The payloads are deliberately small and structured — ids, counts, durations,
|
||||
// the terminal status and max_speedup — not whole task outputs. A plan's task
|
||||
// outputs already reach the transcript through the tool result; duplicating
|
||||
// them into the event log would multiply a large plan's on-disk size for no
|
||||
// added recoverability.
|
||||
type planSessionRecorder struct {
|
||||
recorder *execSessionRecorder
|
||||
// worst remembers the most serious non-completed plan status seen this run,
|
||||
// so the PROCESS exit code can reflect it. Asserting the executor's status
|
||||
// missed the tool's status, and asserting the tool's status would miss the
|
||||
// exit code the same way — this is the third link in that chain.
|
||||
worst specialist.PlanStatus
|
||||
worstReason string
|
||||
}
|
||||
|
||||
// Incomplete reports whether any plan this run did not fully complete, with a
|
||||
// reason for the run-end message. A partial plan is work left undone, which is
|
||||
// exactly what exitIncomplete exists for.
|
||||
func (bridge *planSessionRecorder) Incomplete() (string, bool) {
|
||||
if bridge == nil || bridge.worst == "" || bridge.worst == specialist.PlanCompleted {
|
||||
return "", false
|
||||
}
|
||||
return bridge.worstReason, true
|
||||
}
|
||||
|
||||
func (bridge *planSessionRecorder) append(eventType sessions.EventType, payload any) {
|
||||
// Nil-safe at every level: a nil bridge or a nil inner recorder simply does
|
||||
// not record. Recording must never be the thing that fails a run.
|
||||
if bridge == nil || bridge.recorder == nil {
|
||||
return
|
||||
}
|
||||
bridge.recorder.append(eventType, payload)
|
||||
}
|
||||
|
||||
// The payloads come from specialist's own builders, shared with the TUI's
|
||||
// recorder: resume is a reduction over these events and cannot be written
|
||||
// against two shapes.
|
||||
func (bridge *planSessionRecorder) PlanAdmitted(plan specialist.Plan) {
|
||||
bridge.append(specialist.PlanAdmittedEvent(plan))
|
||||
}
|
||||
|
||||
func (bridge *planSessionRecorder) TaskDispatched(task specialist.Task) {
|
||||
bridge.append(specialist.TaskDispatchedEvent(task))
|
||||
}
|
||||
|
||||
func (bridge *planSessionRecorder) TaskCompleted(result specialist.TaskResult) {
|
||||
bridge.append(specialist.TaskCompletedEvent(result))
|
||||
}
|
||||
|
||||
func (bridge *planSessionRecorder) TaskFailed(result specialist.TaskResult) {
|
||||
bridge.append(specialist.TaskFailedEvent(result))
|
||||
}
|
||||
|
||||
func (bridge *planSessionRecorder) PlanCompleted(plan specialist.Plan, report specialist.PlanReport) {
|
||||
if bridge != nil && report.Status != specialist.PlanCompleted {
|
||||
// PlanFailed outranks PlanPartial: if any plan failed outright, that is
|
||||
// the run's story.
|
||||
if bridge.worst != specialist.PlanFailed {
|
||||
bridge.worst = report.Status
|
||||
bridge.worstReason = fmt.Sprintf("plan %q ended %s: %d succeeded, %d failed, %d skipped",
|
||||
plan.Name(), report.Status, report.Succeeded, report.Failed, report.Skipped)
|
||||
}
|
||||
}
|
||||
bridge.append(specialist.PlanCompletedEvent(plan, report))
|
||||
}
|
||||
@@ -1,134 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Twigpine/zero/internal/sessions"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
)
|
||||
|
||||
// planEventPayloads runs the recorder against a REAL session store and returns
|
||||
// the payload of each plan/task event, keyed by event type. Reading the events
|
||||
// back is the point: asserting the map the recorder builds would not prove they
|
||||
// reached the log a user actually opens.
|
||||
func planEventPayloads(t *testing.T, record func(*planSessionRecorder)) map[sessions.EventType]map[string]any {
|
||||
t.Helper()
|
||||
store := sessions.NewStore(sessions.StoreOptions{RootDir: t.TempDir()})
|
||||
session, err := store.Create(sessions.CreateInput{Cwd: t.TempDir()})
|
||||
if err != nil {
|
||||
t.Fatalf("create session: %v", err)
|
||||
}
|
||||
inner := execSessionRecorder{prepared: sessions.PreparedExec{
|
||||
Mode: sessions.ModeNew,
|
||||
Session: session,
|
||||
Store: store,
|
||||
}}
|
||||
bridge := &planSessionRecorder{recorder: &inner}
|
||||
|
||||
record(bridge)
|
||||
if inner.err != nil {
|
||||
t.Fatalf("recording failed: %v", inner.err)
|
||||
}
|
||||
|
||||
events, err := store.ReadEvents(session.SessionID)
|
||||
if err != nil {
|
||||
t.Fatalf("read events: %v", err)
|
||||
}
|
||||
payloads := map[sessions.EventType]map[string]any{}
|
||||
for _, event := range events {
|
||||
raw, err := json.Marshal(event.Payload)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal %s payload: %v", event.Type, err)
|
||||
}
|
||||
decoded := map[string]any{}
|
||||
if err := json.Unmarshal(raw, &decoded); err != nil {
|
||||
t.Fatalf("decode %s payload: %v", event.Type, err)
|
||||
}
|
||||
payloads[event.Type] = decoded
|
||||
}
|
||||
return payloads
|
||||
}
|
||||
|
||||
// A FAILED task must be as drillable as a successful one.
|
||||
//
|
||||
// Executor.Run carries the child session id even on a post-start failure,
|
||||
// specifically so a failed child can still be opened. Recording it only on
|
||||
// task_completed meant the one task a user needs to investigate was the one
|
||||
// task the event log could not point them at.
|
||||
func TestTaskFailedRecordsTheChildSessionAndItsSpend(t *testing.T) {
|
||||
payloads := planEventPayloads(t, func(bridge *planSessionRecorder) {
|
||||
bridge.TaskFailed(specialist.TaskResult{
|
||||
ID: "b",
|
||||
Outcome: specialist.TaskFailed,
|
||||
Err: "Subagent failed (exit 3)",
|
||||
Duration: 1500 * time.Millisecond,
|
||||
SessionID: "specialist_0fd7e5a5b5e5c31063516119",
|
||||
Tokens: 1500,
|
||||
})
|
||||
})
|
||||
|
||||
failed, ok := payloads[sessions.EventTaskFailed]
|
||||
if !ok {
|
||||
t.Fatal("no task_failed event was recorded")
|
||||
}
|
||||
if failed["session_id"] != "specialist_0fd7e5a5b5e5c31063516119" {
|
||||
t.Fatalf("the failed child's session id was dropped: %v", failed["session_id"])
|
||||
}
|
||||
if failed["tokens"] != float64(1500) {
|
||||
t.Fatalf("the failed task's spend was dropped: %v", failed["tokens"])
|
||||
}
|
||||
}
|
||||
|
||||
// The two task events are compared against EACH OTHER rather than each against
|
||||
// its own expectation. That is how the asymmetry survived: task_completed had a
|
||||
// test, task_failed had a test, and neither noticed the missing fields.
|
||||
func TestTaskEventsCarryTheSameIdentityFields(t *testing.T) {
|
||||
completed := planEventPayloads(t, func(bridge *planSessionRecorder) {
|
||||
bridge.TaskCompleted(specialist.TaskResult{
|
||||
ID: "a", Outcome: specialist.TaskSucceeded,
|
||||
Duration: time.Second, SessionID: "specialist_aaa", Tokens: 150,
|
||||
})
|
||||
})[sessions.EventTaskCompleted]
|
||||
|
||||
failed := planEventPayloads(t, func(bridge *planSessionRecorder) {
|
||||
bridge.TaskFailed(specialist.TaskResult{
|
||||
ID: "b", Outcome: specialist.TaskFailed, Err: "boom",
|
||||
Duration: time.Second, SessionID: "specialist_bbb", Tokens: 150,
|
||||
})
|
||||
})[sessions.EventTaskFailed]
|
||||
|
||||
for _, field := range []string{"id", "duration_ms", "session_id", "tokens"} {
|
||||
if _, ok := completed[field]; !ok {
|
||||
t.Fatalf("task_completed is missing %q", field)
|
||||
}
|
||||
if _, ok := failed[field]; !ok {
|
||||
t.Fatalf("task_failed is missing %q, which task_completed records; "+
|
||||
"a failure must be at least as recoverable as a success", field)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A dependency or budget skip never started a child, so an empty session id is
|
||||
// the honest value there — the field is present and empty, not absent.
|
||||
func TestSkippedTaskRecordsAnEmptySessionRatherThanOmittingIt(t *testing.T) {
|
||||
payloads := planEventPayloads(t, func(bridge *planSessionRecorder) {
|
||||
bridge.TaskFailed(specialist.TaskResult{
|
||||
ID: "d",
|
||||
Outcome: specialist.TaskSkippedDependency,
|
||||
Err: `skipped: dependency "b" did not succeed`,
|
||||
})
|
||||
})
|
||||
failed := payloads[sessions.EventTaskFailed]
|
||||
value, ok := failed["session_id"]
|
||||
if !ok {
|
||||
t.Fatal("session_id must be present even for a task that never started")
|
||||
}
|
||||
if value != "" {
|
||||
t.Fatalf("a skipped task started no child, so its session id must be empty, got %v", value)
|
||||
}
|
||||
if failed["outcome"] != string(specialist.TaskSkippedDependency) {
|
||||
t.Fatalf("the skip outcome must survive: %v", failed["outcome"])
|
||||
}
|
||||
}
|
||||
@@ -6,22 +6,16 @@ import (
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/providercatalog"
|
||||
"github.com/Twigpine/zero/internal/providermodelcatalog"
|
||||
"github.com/Twigpine/zero/internal/providermodeldiscovery"
|
||||
"github.com/Twigpine/zero/internal/providers"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
)
|
||||
|
||||
type providerModelsOptions struct {
|
||||
name string
|
||||
json bool
|
||||
// verify asks the provider to actually RUN each model, rather than trusting
|
||||
// that listing it means it works.
|
||||
verify bool
|
||||
}
|
||||
|
||||
// runProvidersModels lists the models a saved provider actually serves by probing
|
||||
@@ -72,37 +66,6 @@ func runProvidersModels(args []string, stdout io.Writer, stderr io.Writer, deps
|
||||
models = filtered
|
||||
}
|
||||
|
||||
// PROVED, not merely listed. /v1/models is an advertisement: every model that
|
||||
// has broken a real plan appeared on it looking ordinary — one refused at
|
||||
// chat completions, one belonging to a provider the session had switched away
|
||||
// from, two that fail every task they touch. Only a real request settles it.
|
||||
verdicts := map[string]specialist.ModelProbeResult{}
|
||||
if options.verify {
|
||||
workspaceRoot, wderr := deps.getwd()
|
||||
if wderr != nil {
|
||||
return writeAppError(stderr, "resolve working directory: "+wderr.Error(), exitCrash)
|
||||
}
|
||||
probe := planModelProber(workspaceRoot, profile, deps.newProvider)
|
||||
if probe == nil {
|
||||
return writeAppError(stderr, "this build cannot open provider connections, so nothing can be proved", exitCrash)
|
||||
}
|
||||
results := make([]specialist.ModelProbeResult, len(models))
|
||||
var wait sync.WaitGroup
|
||||
// CONCURRENT: nineteen models one after another is a minute of staring at
|
||||
// nothing, and they are independent.
|
||||
for index, model := range models {
|
||||
wait.Add(1)
|
||||
go func(index int, id string) {
|
||||
defer wait.Done()
|
||||
results[index] = probe(ctx, id)
|
||||
}(index, model.ID)
|
||||
}
|
||||
wait.Wait()
|
||||
for index, model := range models {
|
||||
verdicts[model.ID] = results[index]
|
||||
}
|
||||
}
|
||||
|
||||
if options.json {
|
||||
items := make([]map[string]any, 0, len(models))
|
||||
for _, model := range models {
|
||||
@@ -110,12 +73,6 @@ func runProvidersModels(args []string, stdout io.Writer, stderr io.Writer, deps
|
||||
if description := strings.TrimSpace(model.Description); description != "" {
|
||||
entry["description"] = description
|
||||
}
|
||||
if verdict, ok := verdicts[model.ID]; ok {
|
||||
entry["status"] = probeStatusWord(verdict.Verdict)
|
||||
if reason := strings.TrimSpace(verdict.Reason); reason != "" {
|
||||
entry["details"] = reason
|
||||
}
|
||||
}
|
||||
if len(model.ReasoningEfforts) > 0 {
|
||||
entry["reasoning_efforts"] = append([]string{}, model.ReasoningEfforts...)
|
||||
}
|
||||
@@ -148,62 +105,15 @@ func runProvidersModels(args []string, stdout io.Writer, stderr io.Writer, deps
|
||||
if _, err := fmt.Fprintf(stdout, "Provider models (%s)\n", name); err != nil {
|
||||
return exitCrash
|
||||
}
|
||||
var serves, refuses, unreachable int
|
||||
for _, model := range models {
|
||||
line := strings.TrimSpace(model.ID)
|
||||
// The size where the id names one, so the directory reads as light/medium/
|
||||
// heavy at a glance — the classification routing uses, made visible. Absent
|
||||
// for a cloud id that carries none.
|
||||
if label := specialist.ModelSizeLabel(model.ID); label != "" {
|
||||
line += " [" + label + "]"
|
||||
}
|
||||
if verdict, ok := verdicts[model.ID]; ok {
|
||||
// The VERDICT LEADS, because it is the reason to run this at all: a
|
||||
// reader scanning for what is wrong should not have to reach the end
|
||||
// of each line to find out.
|
||||
switch verdict.Verdict {
|
||||
case specialist.ProbeServes:
|
||||
serves++
|
||||
line = "ok " + line
|
||||
case specialist.ProbeRefuses:
|
||||
refuses++
|
||||
line = "REFUSES " + line
|
||||
default:
|
||||
unreachable++
|
||||
line = "? " + line
|
||||
}
|
||||
if verdict.Verdict != specialist.ProbeServes {
|
||||
if reason := strings.TrimSpace(verdict.Reason); reason != "" {
|
||||
line += " — " + firstProbeLine(reason)
|
||||
}
|
||||
}
|
||||
} else if description := strings.TrimSpace(model.Description); description != "" {
|
||||
if description := strings.TrimSpace(model.Description); description != "" {
|
||||
line += " — " + description
|
||||
}
|
||||
if _, err := fmt.Fprintln(stdout, line); err != nil {
|
||||
return exitCrash
|
||||
}
|
||||
}
|
||||
if options.verify {
|
||||
if _, err := fmt.Fprintf(stdout, "\n%d serve, %d refuse, %d could not be reached.\n",
|
||||
serves, refuses, unreachable); err != nil {
|
||||
return exitCrash
|
||||
}
|
||||
if refuses > 0 {
|
||||
// The point of --verify, stated plainly: these are the ids that would
|
||||
// otherwise be found mid-plan, one dead task at a time.
|
||||
if _, err := fmt.Fprintln(stdout, "A model that REFUSES is listed by this provider but will not run. "+
|
||||
"Plans skip these automatically — nothing needs excluding by hand."); err != nil {
|
||||
return exitCrash
|
||||
}
|
||||
}
|
||||
if unreachable > 0 {
|
||||
if _, err := fmt.Fprintln(stdout, "A model that could not be reached is left in place: "+
|
||||
"that is a fact about the connection, not about the model."); err != nil {
|
||||
return exitCrash
|
||||
}
|
||||
}
|
||||
}
|
||||
suffix := "s"
|
||||
if len(models) == 1 {
|
||||
suffix = ""
|
||||
@@ -219,73 +129,6 @@ func runProvidersModels(args []string, stdout io.Writer, stderr io.Writer, deps
|
||||
return exitSuccess
|
||||
}
|
||||
|
||||
// livePlanProvider answers WHICH provider this plan's models should be listed
|
||||
// from, at the moment the plan runs rather than at the moment the tool was built.
|
||||
//
|
||||
// THE CAPTURED PROFILE GOES STALE. The orchestrate tool is wired once, at
|
||||
// startup, from the provider active then; the TUI's /model picker can switch
|
||||
// provider mid-session, and switchProviderModel re-points the parent client and
|
||||
// exports ZERO_PROVIDER so spawned children follow. Nothing re-pointed this. The
|
||||
// result was a session running xai/grok-4.5 whose plan was handed nineteen Ollama
|
||||
// model ids — the router assigned them, the children were dispatched with them,
|
||||
// and the provider rejected every one.
|
||||
//
|
||||
// So it re-resolves through the SAME path a freshly spawned child takes:
|
||||
// config plus ZERO_PROVIDER. Discovery and child execution now read one source of
|
||||
// truth, which is stronger than keeping two in sync — they cannot drift apart
|
||||
// again because there is no longer a second place to update.
|
||||
//
|
||||
// THE CAPTURED PROFILE WINS WHEN THE NAME MATCHES, and that is not an
|
||||
// optimisation. It carries what re-resolution cannot see: --provider, --base-url
|
||||
// and an inline --api-key are applied by the caller ON TOP of Resolve, so a
|
||||
// headless run that never switches must keep exactly the profile it was given.
|
||||
// Only a genuine change of provider — the one case that broke — takes the new one.
|
||||
func livePlanProvider(workspaceRoot string, captured config.ProviderProfile) config.ProviderProfile {
|
||||
options, err := config.DefaultResolveOptions(workspaceRoot)
|
||||
if err != nil {
|
||||
return captured
|
||||
}
|
||||
// Nothing to re-resolve FROM, which is not the same as a failure and is why
|
||||
// the error branches cannot catch it.
|
||||
//
|
||||
// With no config file on either path, no provider command and no
|
||||
// ZERO_PROVIDER, config.Resolve still SUCCEEDS: it falls back to a built-in
|
||||
// default provider. That default is not a decision anyone made. Treating it
|
||||
// as a provider switch discards a captured profile carrying --provider,
|
||||
// --base-url and an inline --api-key, and points discovery at the default
|
||||
// endpoint with none of the caller's credentials — this function's own
|
||||
// failure mode arriving from the other direction. A headless run on a machine
|
||||
// with no config file, a fresh CI container being the obvious one, hits it
|
||||
// every time.
|
||||
//
|
||||
// Read off the OPTIONS rather than by stat-ing again: DefaultResolveOptions
|
||||
// already leaves a path empty when the file is absent, so this uses the
|
||||
// answer it computed instead of asking a second time and risking a different
|
||||
// one.
|
||||
//
|
||||
// ZERO_PROVIDER is deliberately NOT part of this test, though it looks like
|
||||
// it belongs. With no config file there are no profiles to select from, so a
|
||||
// ZERO_PROVIDER naming one resolves to ErrNoActiveProvider and the error
|
||||
// branch below already returns the captured profile. Adding it here would
|
||||
// read as a meaningful case while changing no outcome.
|
||||
if strings.TrimSpace(options.UserConfigPath) == "" &&
|
||||
strings.TrimSpace(options.ProjectConfigPath) == "" &&
|
||||
strings.TrimSpace(options.ProviderCommand) == "" {
|
||||
return captured
|
||||
}
|
||||
// Env deliberately left nil: config.Resolve then reads the live process
|
||||
// environment, which is where switchProviderModel wrote ZERO_PROVIDER.
|
||||
resolved, err := config.Resolve(options)
|
||||
if err != nil {
|
||||
return captured
|
||||
}
|
||||
if strings.TrimSpace(resolved.Provider.Name) == "" ||
|
||||
strings.EqualFold(strings.TrimSpace(resolved.Provider.Name), strings.TrimSpace(captured.Name)) {
|
||||
return captured
|
||||
}
|
||||
return resolved.Provider
|
||||
}
|
||||
|
||||
// discoveryCredentialProfile resolves the profile's API key the same way the
|
||||
// runtime does — inline, then the stored credential, then the configured env var —
|
||||
// so a `providers models` probe authenticates exactly like a real request. Mirrors
|
||||
@@ -324,8 +167,6 @@ func parseProviderModelsArgs(args []string) (providerModelsOptions, bool, error)
|
||||
return options, true, nil
|
||||
case arg == "--json":
|
||||
options.json = true
|
||||
case arg == "--verify":
|
||||
options.verify = true
|
||||
case strings.HasPrefix(arg, "-"):
|
||||
return options, false, execUsageError{fmt.Sprintf("unknown flag %q", arg)}
|
||||
default:
|
||||
@@ -337,108 +178,3 @@ func parseProviderModelsArgs(args []string) (providerModelsOptions, bool, error)
|
||||
}
|
||||
return options, false, nil
|
||||
}
|
||||
|
||||
// planModelDiscoverer adapts provider discovery for the orchestrate tool's
|
||||
// auto_assign.
|
||||
//
|
||||
// THE ADAPTER LIVES HERE, not in specialist, and that is the point of the
|
||||
// narrower type. internal/specialist runs on the child-execution path; importing
|
||||
// config and providercatalog to describe a model would drag the whole provider
|
||||
// stack in behind it. The surface that already holds a provider profile does the
|
||||
// translation and hands over the four facts a plan actually chooses between.
|
||||
func planModelDiscoverer(workspaceRoot string, profile config.ProviderProfile) specialist.ModelDiscoverer {
|
||||
return func(ctx context.Context) ([]specialist.DiscoveredModel, error) {
|
||||
live := livePlanProvider(workspaceRoot, profile)
|
||||
found, err := defaultDiscoverProviderModels(ctx, discoveryCredentialProfile(live))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// The LIVE profile's catalog id, not the captured one: discovery already
|
||||
// follows an in-session provider switch, and filtering against the
|
||||
// launch-time provider would reintroduce the disagreement
|
||||
// livePlanProvider exists to remove.
|
||||
return planModelsFromDiscovered(live.CatalogID, found), nil
|
||||
}
|
||||
}
|
||||
|
||||
// planModelsFromDiscovered narrows a provider's list to the models a plan may
|
||||
// actually be assigned, and translates them into specialist's own type.
|
||||
//
|
||||
// FILTERED THROUGH THE SAME ALLOW-LIST THE CLIENT WILL APPLY.
|
||||
// defaultDiscoverProviderModels deliberately returns a provider's whole list —
|
||||
// right for `providers models`, which exists to show what is there, and which
|
||||
// applies this same filter itself before printing. A plan does not display these,
|
||||
// it RUNS them, and providers.New refuses a model outside its provider's scope
|
||||
// (validateModelAllowedForProvider). So the unfiltered list handed the router
|
||||
// candidates that could never be dispatched, and every task assigned one died at
|
||||
// client creation — the same shape as assigning from the wrong provider's list
|
||||
// entirely, and just as invisible until dispatch.
|
||||
func planModelsFromDiscovered(catalogID string, found []providermodeldiscovery.Model) []specialist.DiscoveredModel {
|
||||
catalogID = strings.TrimSpace(catalogID)
|
||||
out := make([]specialist.DiscoveredModel, 0, len(found))
|
||||
for _, model := range found {
|
||||
if !providermodelcatalog.ModelIDAllowedForProvider(catalogID, model.ID) {
|
||||
continue
|
||||
}
|
||||
out = append(out, specialist.DiscoveredModel{
|
||||
ID: model.ID,
|
||||
Description: model.Description,
|
||||
ToolCall: model.ToolCall,
|
||||
Reasoning: model.Reasoning,
|
||||
InputCost: model.InputCost,
|
||||
OutputCost: model.OutputCost,
|
||||
OutputModalities: model.OutputModalities,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// planModelPreferences carries the configured per-role pins and exclusions into
|
||||
// specialist's own narrow type, so that package keeps its distance from config.
|
||||
func planModelPreferences(cfg config.PlanModelsConfig) specialist.ModelPreferences {
|
||||
return specialist.ModelPreferences{
|
||||
Scan: cfg.Scan,
|
||||
Implement: cfg.Implement,
|
||||
Verify: cfg.Verify,
|
||||
Exclude: cfg.Exclude,
|
||||
Router: cfg.Router,
|
||||
RouterGuidance: cfg.RouterGuidance,
|
||||
AutoAssign: cfg.AutoAssign,
|
||||
MinSize: cfg.MinSize,
|
||||
TopModels: cfg.TopModels,
|
||||
}
|
||||
}
|
||||
|
||||
// planContextWindows reports the context window of the model a plan task will
|
||||
// run on, 0 when unknown.
|
||||
//
|
||||
// FROM THE CATALOGUE, NOT FROM DISCOVERY. The window is static metadata about a
|
||||
// model, so asking the provider for it over the network per task would pay a
|
||||
// round trip for an answer that cannot change — and most of this catalogue's
|
||||
// gateways do not report a window at all, in which case 0 is the honest answer
|
||||
// and the plan keeps its fixed briefing caps.
|
||||
//
|
||||
// AN EMPTY MODEL ID MEANS "the model this run is using". A plan task that names
|
||||
// no model inherits the parent's, and only this side of the seam knows what that
|
||||
// is — internal/specialist deliberately does not.
|
||||
func planContextWindows(profile config.ProviderProfile, sessionModel string) specialist.ContextWindowFunc {
|
||||
var once sync.Once
|
||||
windows := map[string]int{}
|
||||
return func(modelID string) int {
|
||||
once.Do(func() {
|
||||
descriptor, err := providercatalog.Require(profile.CatalogID)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
for _, model := range providermodelcatalog.Models(descriptor) {
|
||||
if model.ContextWindow > 0 {
|
||||
windows[model.ID] = model.ContextWindow
|
||||
}
|
||||
}
|
||||
})
|
||||
if strings.TrimSpace(modelID) == "" {
|
||||
modelID = sessionModel
|
||||
}
|
||||
return windows[strings.TrimSpace(modelID)]
|
||||
}
|
||||
}
|
||||
@@ -42,31 +42,6 @@ func TestRunProvidersModelsListsDiscoveredModels(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The directory shows each model's SIZE where the id names one, so a reader sees
|
||||
// light/medium/heavy at a glance — the classification routing uses, made visible.
|
||||
// A cloud id with no size gets no invented label.
|
||||
func TestRunProvidersModelsShowsModelSize(t *testing.T) {
|
||||
var stdout, stderr bytes.Buffer
|
||||
deps := commandCenterDeps(t)
|
||||
deps.discoverProviderModels = func(_ context.Context, _ config.ProviderProfile) ([]providermodeldiscovery.Model, error) {
|
||||
return []providermodeldiscovery.Model{
|
||||
{ID: "gpt-oss:20b"},
|
||||
{ID: "team/cloud-model"}, // no size in the id
|
||||
}, nil
|
||||
}
|
||||
|
||||
if exitCode := runWithDeps([]string{"providers", "models"}, &stdout, &stderr, deps); exitCode != exitSuccess {
|
||||
t.Fatalf("exit = %d: %s", exitCode, stderr.String())
|
||||
}
|
||||
out := stdout.String()
|
||||
if !strings.Contains(out, "gpt-oss:20b [20B]") {
|
||||
t.Fatalf("the directory did not show the model size:\n%s", out)
|
||||
}
|
||||
if strings.Contains(out, "team/cloud-model [") {
|
||||
t.Fatalf("a cloud model with no size got an invented size label:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunProvidersModelsJSON(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
+12
-3
@@ -174,7 +174,7 @@ func runSandboxSetup(args []string, stdout io.Writer, stderr io.Writer, deps app
|
||||
if !setupHelper.Available() {
|
||||
return writeAppError(stderr, "Windows sandbox setup helper is not available", exitProvider)
|
||||
}
|
||||
setupArgs, err := zeroSandbox.BuildWindowsSandboxSetupArgs(zeroSandbox.WindowsSandboxSetupArgsOptions{
|
||||
setupPlan, err := zeroSandbox.BuildWindowsSandboxSetupArgs(zeroSandbox.WindowsSandboxSetupArgsOptions{
|
||||
CommandCWD: workspaceRoot,
|
||||
WorkspaceRoots: []string{workspaceRoot},
|
||||
PermissionProfile: profile,
|
||||
@@ -182,9 +182,18 @@ func runSandboxSetup(args []string, stdout io.Writer, stderr io.Writer, deps app
|
||||
if err != nil {
|
||||
return writeAppError(stderr, err.Error(), exitCrash)
|
||||
}
|
||||
setupArgs = append(append([]string{}, setupHelper.ArgsPrefix...), setupArgs...)
|
||||
setupArgs := append(append([]string{}, setupHelper.ArgsPrefix...), setupPlan.Args...)
|
||||
// BUILDING THE ARGS ALREADY WROTE TO DISK. Selecting the runtime root takes a
|
||||
// lease, and taking a lease creates the runtime tree and the lease file when
|
||||
// they are not there. A helper that never publishes a marker leaves that tree
|
||||
// behind, attested by nothing, so the undo belongs on this side of the process
|
||||
// boundary where the record of it lives.
|
||||
if err := deps.runSandboxSetupHelper(setupHelper.Name, setupArgs, stdout, stderr); err != nil {
|
||||
return writeAppError(stderr, "Windows sandbox setup failed: "+err.Error(), exitProvider)
|
||||
message := "Windows sandbox setup failed: " + err.Error()
|
||||
if undo := setupPlan.Rollback(); undo != nil {
|
||||
message += "\nleftover sandbox runtime state could not be removed: " + undo.Error()
|
||||
}
|
||||
return writeAppError(stderr, message, exitProvider)
|
||||
}
|
||||
return writeSandboxSetupResult(stdout, options.json, sandboxSetupResult{
|
||||
Platform: "windows",
|
||||
|
||||
@@ -576,17 +576,6 @@ func assertSandboxPolicyJSONGolden(t *testing.T, mustList string) string {
|
||||
t.Setenv("XDG_CONFIG_HOME", filepath.Join(emptyHome, ".config"))
|
||||
t.Setenv("CLOUDSDK_CONFIG", "")
|
||||
t.Setenv("GOOGLE_APPLICATION_CREDENTIALS", "")
|
||||
// XDG_CONFIG_HOME as well, or HOME is not actually the answer to "where is
|
||||
// the config directory". credentialDenyReadPathsForEnvironment reads
|
||||
// XDG_CONFIG_HOME FIRST and only falls back to $HOME/.config, so on a host
|
||||
// that sets it — every ubuntu GitHub runner does — redirecting HOME alone
|
||||
// left configHome pointing at the real /home/runner/.config. Other packages
|
||||
// in the same `go test ./...` write zero/config.json and zero/trust.json
|
||||
// there, `go test` runs packages in parallel, and this golden then failed or
|
||||
// passed depending on who won the race. macOS and Windows runners leave
|
||||
// XDG_CONFIG_HOME unset, which is the whole reason it only ever went red on
|
||||
// one platform.
|
||||
t.Setenv("XDG_CONFIG_HOME", filepath.Join(emptyHome, ".config"))
|
||||
t.Setenv("NPM_CONFIG_USERCONFIG", "")
|
||||
t.Setenv("npm_config_userconfig", "")
|
||||
t.Setenv("GH_CONFIG_DIR", "")
|
||||
|
||||
@@ -1,50 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
)
|
||||
|
||||
// THE TEST BINARY ANSWERS THE SANDBOX'S SELF-DISPATCH LIKE THE REAL BINARY.
|
||||
// The sandbox launches os.Executable() with a hidden helper subcommand; under a
|
||||
// test that runs this binary as the zero CLI, that is this binary. Unrouted, the
|
||||
// "helper" ran the entire package suite instead, and the sandboxed command that
|
||||
// launched it hung until killed.
|
||||
//
|
||||
// Each hidden subcommand, given no helper arguments, must come straight back
|
||||
// with the helper's own usage failure — never a test run.
|
||||
func TestTheTestBinaryRoutesSandboxSelfDispatch(t *testing.T) {
|
||||
if os.Getenv("ZERO_TEST_SELF_DISPATCH_CHILD") == "1" {
|
||||
// Reached only when routing is broken and the child is running the
|
||||
// suite; skip so the failure is the parent's timeout, not a recursion.
|
||||
t.Skip("self-dispatch child")
|
||||
}
|
||||
for _, subcommand := range []string{
|
||||
sandbox.WindowsCommandRunnerSubcommand,
|
||||
sandbox.WindowsSandboxSetupSubcommand,
|
||||
sandbox.LinuxSandboxHelperSubcommand,
|
||||
} {
|
||||
t.Run(subcommand, func(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, os.Args[0], subcommand)
|
||||
cmd.Env = append(os.Environ(), "ZERO_TEST_SELF_DISPATCH_CHILD=1")
|
||||
output, err := cmd.CombinedOutput()
|
||||
if ctx.Err() != nil {
|
||||
t.Fatalf("%s did not return: the test binary ran its suite instead of the helper", subcommand)
|
||||
}
|
||||
if err == nil {
|
||||
t.Fatalf("%s with no helper arguments succeeded; expected the helper's usage failure:\n%s", subcommand, output)
|
||||
}
|
||||
if text := string(output); strings.Contains(text, "=== RUN") || strings.Contains(text, "\nPASS") || strings.Contains(text, "--- SKIP") {
|
||||
t.Fatalf("%s ran tests instead of the helper:\n%s", subcommand, text)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1,51 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// SHUTDOWN ORDER, guarded at the source because defers cannot be observed from
|
||||
// a test.
|
||||
//
|
||||
// Required at shutdown: stop the plan, then close the runtime it was using, then
|
||||
// cancel the session. Defers run LIFO, so registration order is the reverse —
|
||||
// which is subtle enough that it was already wrong once. planLaunch.Close() was
|
||||
// registered beside the launcher it belongs to, which reads naturally and meant
|
||||
// it ran AFTER closeSpecialistRuntime. Close "cancels AND WAITS", so a
|
||||
// background plan was waited on with the runtime it needs already torn down.
|
||||
//
|
||||
// A source-order assertion is a blunt instrument. It is here because the
|
||||
// alternative is a comment, and a comment did not stop this happening: the
|
||||
// ordering is invisible at the point where someone would add the next defer.
|
||||
func TestShutdownDefersAreRegisteredInReverseOfTheirRequiredOrder(t *testing.T) {
|
||||
source, err := os.ReadFile("app.go")
|
||||
if err != nil {
|
||||
t.Fatalf("read app.go: %v", err)
|
||||
}
|
||||
text := string(source)
|
||||
|
||||
// Registration order, top to bottom.
|
||||
positions := map[string]int{
|
||||
"cancelSession": strings.Index(text, "defer cancelSession()"),
|
||||
"closeSpecialistRuntime": strings.Index(text, "defer closeSpecialistRuntime(stderr, specialistRuntime)"),
|
||||
"planLaunch.Close": strings.Index(text, "defer planLaunch.Close()"),
|
||||
}
|
||||
for name, at := range positions {
|
||||
if at < 0 {
|
||||
t.Fatalf("could not find the %s defer; this guard has gone stale and must be re-pointed, not deleted", name)
|
||||
}
|
||||
}
|
||||
|
||||
// LIFO: later registration runs earlier. The plan must stop first, so its
|
||||
// defer must be registered last.
|
||||
if positions["planLaunch.Close"] < positions["closeSpecialistRuntime"] {
|
||||
t.Error("planLaunch.Close() is registered before closeSpecialistRuntime, so it RUNS after it — " +
|
||||
"a background plan would be waited on with its specialist runtime already closed")
|
||||
}
|
||||
if positions["closeSpecialistRuntime"] < positions["cancelSession"] {
|
||||
t.Error("closeSpecialistRuntime is registered before cancelSession, so it RUNS after it — " +
|
||||
"the session would be cancelled while the runtime is still open")
|
||||
}
|
||||
}
|
||||
@@ -1,145 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"io"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/hooks"
|
||||
"github.com/Twigpine/zero/internal/mcp"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/tui"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
func TestRegisterSpecialistToolsWiresLifecycleHooksIntoTaskOutput(t *testing.T) {
|
||||
registry := tools.NewRegistry()
|
||||
runtime, err := registerSpecialistTools(registry, t.TempDir(), 0, nil, nil, nil, orchestrateWiring{})
|
||||
if err != nil {
|
||||
t.Fatalf("registerSpecialistTools returned error: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { closeSpecialistRuntime(io.Discard, runtime) })
|
||||
|
||||
if runtime.lifecycleHooks == nil {
|
||||
t.Fatal("lifecycleHooks is nil after registerSpecialistTools")
|
||||
}
|
||||
raw, ok := registry.Get("TaskOutput")
|
||||
if !ok {
|
||||
t.Fatal("TaskOutput tool not registered")
|
||||
}
|
||||
outputTool, ok := raw.(*specialist.OutputTool)
|
||||
if !ok {
|
||||
t.Fatalf("TaskOutput type = %T, want *specialist.OutputTool", raw)
|
||||
}
|
||||
if outputTool.LifecycleHooks != runtime.lifecycleHooks {
|
||||
t.Fatal("TaskOutput LifecycleHooks is not the shared registerSpecialistTools bridge")
|
||||
}
|
||||
|
||||
dispatcher := hooks.NewDispatcher(hooks.DispatcherOptions{})
|
||||
runtime.attachLifecycleHooks(dispatcher)
|
||||
if runtime.lifecycleHooks.Dispatch == nil {
|
||||
t.Fatal("attachLifecycleHooks left Dispatch nil")
|
||||
}
|
||||
if outputTool.LifecycleHooks.Dispatch == nil {
|
||||
t.Fatal("TaskOutput did not observe Dispatch after attachLifecycleHooks")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRealEntryPointsAttachLifecycleHooks closes the gap the registration-level
|
||||
// test above cannot see: wiring the bridge when tools register is not enough
|
||||
// unless the real startup paths actually call attachLifecycleHooks before any
|
||||
// run. Both surfaces that can launch specialists must do so — the interactive
|
||||
// TUI and headless exec. attachLifecycleHooks mutates the same
|
||||
// *specialist.LifecycleHooks the TaskOutput tool holds, so a nil Dispatch on
|
||||
// that tool after a full startup proves the call never ran and the
|
||||
// specialistStart/specialistStop hooks would silently never fire.
|
||||
func TestRealEntryPointsAttachLifecycleHooks(t *testing.T) {
|
||||
requireTaskOutputDispatchAttached := func(t *testing.T, registry *tools.Registry) {
|
||||
t.Helper()
|
||||
if registry == nil {
|
||||
t.Fatal("startup did not reach the run registry")
|
||||
}
|
||||
raw, ok := registry.Get("TaskOutput")
|
||||
if !ok {
|
||||
t.Fatal("TaskOutput tool not registered by startup")
|
||||
}
|
||||
outputTool, ok := raw.(*specialist.OutputTool)
|
||||
if !ok {
|
||||
t.Fatalf("TaskOutput type = %T, want *specialist.OutputTool", raw)
|
||||
}
|
||||
if outputTool.LifecycleHooks == nil || outputTool.LifecycleHooks.Dispatch == nil {
|
||||
t.Fatal("startup left the specialist lifecycle bridge unattached: specialistStart/specialistStop hooks would never fire")
|
||||
}
|
||||
}
|
||||
|
||||
t.Run("interactive TUI", func(t *testing.T) {
|
||||
isolateCLIUserState(t)
|
||||
cwd := t.TempDir()
|
||||
var registry *tools.Registry
|
||||
|
||||
// Empty args route runWithDeps to the interactive TUI path; the injected
|
||||
// runTUI receives the same registry specialist tools registered into.
|
||||
exitCode := runWithDeps([]string{}, io.Discard, io.Discard, appDeps{
|
||||
getwd: func() (string, error) { return cwd, nil },
|
||||
resolveConfig: func(string, config.Overrides) (config.ResolvedConfig, error) {
|
||||
return config.ResolvedConfig{
|
||||
Provider: config.ProviderProfile{
|
||||
Name: "p",
|
||||
ProviderKind: config.ProviderKindOpenAICompatible,
|
||||
BaseURL: "http://127.0.0.1/v1",
|
||||
Model: "m",
|
||||
},
|
||||
}, nil
|
||||
},
|
||||
resolveMCPConfig: func(string, bool) (config.MCPConfig, error) { return config.MCPConfig{}, nil },
|
||||
runTUI: func(_ context.Context, options tui.Options) int {
|
||||
registry = options.Registry
|
||||
return exitSuccess
|
||||
},
|
||||
})
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("runWithDeps exit = %d, want %d", exitCode, exitSuccess)
|
||||
}
|
||||
requireTaskOutputDispatchAttached(t, registry)
|
||||
})
|
||||
|
||||
t.Run("headless exec", func(t *testing.T) {
|
||||
isolateCLIUserState(t)
|
||||
cwd := t.TempDir()
|
||||
var registry *tools.Registry
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
// --auto high is what registers specialist tools on the exec surface.
|
||||
// MCP registration is the one startup chokepoint that sees the run
|
||||
// registry before attachLifecycleHooks runs, so capture it there and
|
||||
// assert after the run completes.
|
||||
exitCode := runWithDeps([]string{
|
||||
"exec", "--auto", "high", "--cwd", cwd, "--init-session-id", "hook_wireup", "hello",
|
||||
}, &stdout, &stderr, appDeps{
|
||||
getwd: func() (string, error) { return cwd, nil },
|
||||
resolveConfig: func(string, config.Overrides) (config.ResolvedConfig, error) {
|
||||
return execResolvedConfig(), nil
|
||||
},
|
||||
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
||||
return echoExecProvider{}, nil
|
||||
},
|
||||
resolveMCPConfig: func(string, bool) (config.MCPConfig, error) {
|
||||
return config.MCPConfig{Servers: map[string]config.MCPServerConfig{
|
||||
"wireup": {Type: "stdio", Command: "wireup-mcp"},
|
||||
}}, nil
|
||||
},
|
||||
newMCPStore: func() (*mcp.PermissionStore, error) { return nil, nil },
|
||||
registerMCPTools: func(_ context.Context, captured *tools.Registry, _ config.MCPConfig, _ mcp.RegisterOptions) (mcpToolRuntime, error) {
|
||||
registry = captured
|
||||
return closeFunc(func() error { return nil }), nil
|
||||
},
|
||||
})
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("runWithDeps exit = %d, want %d (stderr=%q)", exitCode, exitSuccess, stderr.String())
|
||||
}
|
||||
requireTaskOutputDispatchAttached(t, registry)
|
||||
})
|
||||
}
|
||||
@@ -1,163 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Twigpine/zero/internal/config"
|
||||
"github.com/Twigpine/zero/internal/sandbox"
|
||||
"github.com/Twigpine/zero/internal/specialist"
|
||||
"github.com/Twigpine/zero/internal/streamjson"
|
||||
"github.com/Twigpine/zero/internal/tools"
|
||||
"github.com/Twigpine/zero/internal/zeroruntime"
|
||||
)
|
||||
|
||||
// echoToolResultProvider calls one tool, then answers with what the tool
|
||||
// returned, so the run's output shows whether the call succeeded.
|
||||
type echoToolResultProvider struct {
|
||||
toolName string
|
||||
arguments string
|
||||
}
|
||||
|
||||
func (provider echoToolResultProvider) StreamCompletion(_ context.Context, request zeroruntime.CompletionRequest) (<-chan zeroruntime.StreamEvent, error) {
|
||||
for _, message := range request.Messages {
|
||||
if message.Role == zeroruntime.MessageRoleTool {
|
||||
ch := make(chan zeroruntime.StreamEvent, 2)
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventText,
|
||||
Content: fmt.Sprintf("TOOL-RESULT error=%v: %s", message.IsError, message.Content)}
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
||||
close(ch)
|
||||
return ch, nil
|
||||
}
|
||||
}
|
||||
ch := make(chan zeroruntime.StreamEvent, 4)
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call_probe", ToolName: provider.toolName}
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call_probe", ArgumentsFragment: provider.arguments}
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call_probe"}
|
||||
ch <- zeroruntime.StreamEvent{Type: zeroruntime.StreamEventDone}
|
||||
close(ch)
|
||||
return ch, nil
|
||||
}
|
||||
|
||||
// AN ORDINARY TASK'S CHILD CAN READ THE PARENT'S READ GRANT AND CANNOT WRITE IT.
|
||||
//
|
||||
// End to end on one path: the parent's scope holds a read grant outside its
|
||||
// workspace; the Task tool launches a write-capable child through the executor
|
||||
// the CLI wires; and the argv that launch produced is run through the real exec
|
||||
// entry point, where read_file inside the grant succeeds and write_file inside
|
||||
// it is refused. Before the executor carried the read supplier, the read failed
|
||||
// "outside the workspace"; had the grant been emitted as --add-dir, the write
|
||||
// would land.
|
||||
//
|
||||
// The resumed launch is covered at the argv level only
|
||||
// (TestTheCLIExecutorPassesReadGrantsReadOnlyToTaskAndResume): exec --resume
|
||||
// refuses a child session ("not resumable") on this tree, independently of the
|
||||
// roots it is given, so a resumed child cannot be run in-process here.
|
||||
func TestAnOrdinaryTaskChildCanReadButNotWriteTheParentsReadGrant(t *testing.T) {
|
||||
// Session state for the in-process child, redirected on every platform
|
||||
// (HOME covers macOS and Windows).
|
||||
t.Setenv("XDG_DATA_HOME", t.TempDir())
|
||||
t.Setenv("HOME", t.TempDir())
|
||||
workspace, err := filepath.EvalSymlinks(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
readRoot := tempDirOutsideDefaultTemp(t)
|
||||
evidence := filepath.Join(readRoot, "evidence.txt")
|
||||
if err := os.WriteFile(evidence, []byte("READ-GRANT-EVIDENCE"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
parentScope, err := sandbox.NewScope(workspace, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("NewScope: %v", err)
|
||||
}
|
||||
if _, err := parentScope.AddRead(readRoot); err != nil {
|
||||
t.Fatalf("AddRead: %v", err)
|
||||
}
|
||||
|
||||
launches := 0
|
||||
var childArgs []string
|
||||
executor := specialist.Executor{
|
||||
BinaryPath: "/bin/true",
|
||||
NewSessionID: func() (string, error) {
|
||||
launches++
|
||||
return fmt.Sprintf("specialist_%024d", launches), nil
|
||||
},
|
||||
Load: func(specialist.LoadOptions) (specialist.LoadResult, error) {
|
||||
return specialist.LoadResult{Specialists: []specialist.Manifest{{
|
||||
Metadata: specialist.Metadata{Name: "auditor", Description: "reads and writes"},
|
||||
SystemPrompt: "x",
|
||||
ResolvedTools: []string{"read_file", "write_file"},
|
||||
ToolsResolved: true,
|
||||
Location: specialist.LocationBuiltin,
|
||||
}}}, nil
|
||||
},
|
||||
RunChild: func(_ context.Context, _ string, args []string, _ func(streamjson.Event)) (specialist.ChildRunResult, error) {
|
||||
childArgs = append([]string(nil), args...)
|
||||
return specialist.ChildRunResult{Started: true}, nil
|
||||
},
|
||||
}
|
||||
wireChildRoots(&executor, workspace, orchestrateWiring{
|
||||
ExtraWriteRoots: func(withinTurn bool) []string { return childWriteRoots(parentScope, withinTurn) },
|
||||
ExtraReadRoots: func(withinTurn bool) []string { return childReadRoots(parentScope, withinTurn) },
|
||||
})
|
||||
taskTool := specialist.NewTaskTool(executor)
|
||||
|
||||
// launchAndRun runs the Task tool with taskArgs and then the child it would
|
||||
// have started, with a provider that makes the given tool call.
|
||||
launchAndRun := func(taskArgs map[string]any, toolName string, arguments map[string]string) string {
|
||||
t.Helper()
|
||||
result := taskTool.RunWithOptions(context.Background(), taskArgs,
|
||||
// Unsafe, so the child's tool layer — not a permission prompt — is
|
||||
// what decides whether the write may land.
|
||||
tools.RunOptions{Cwd: workspace, PermissionMode: "unsafe"})
|
||||
if result.Status != tools.StatusOK {
|
||||
t.Fatalf("Task launch failed: %s", result.Output)
|
||||
}
|
||||
if len(childArgs) == 0 || childArgs[0] != "exec" {
|
||||
t.Fatalf("unexpected child argv: %v", childArgs)
|
||||
}
|
||||
encoded, err := json.Marshal(arguments)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var stdout, stderr bytes.Buffer
|
||||
exitCode := runWithDeps(childArgs, &stdout, &stderr, appDeps{
|
||||
getwd: func() (string, error) { return workspace, nil },
|
||||
resolveConfig: func(string, config.Overrides) (config.ResolvedConfig, error) {
|
||||
return execResolvedConfig(), nil
|
||||
},
|
||||
newProvider: func(config.ProviderProfile) (zeroruntime.Provider, error) {
|
||||
return echoToolResultProvider{toolName: toolName, arguments: string(encoded)}, nil
|
||||
},
|
||||
newSandboxStore: func() (*sandbox.GrantStore, error) {
|
||||
return sandbox.NewGrantStore(sandbox.StoreOptions{FilePath: filepath.Join(t.TempDir(), "sandbox-grants.json")})
|
||||
},
|
||||
})
|
||||
if exitCode != exitSuccess {
|
||||
t.Fatalf("child exit = %d; argv=%v stdout=%q stderr=%q", exitCode, childArgs, stdout.String(), stderr.String())
|
||||
}
|
||||
return stdout.String()
|
||||
}
|
||||
|
||||
task := map[string]any{"name": "auditor", "prompt": "probe the grant"}
|
||||
readOutput := launchAndRun(task, "read_file", map[string]string{"path": evidence})
|
||||
if !strings.Contains(readOutput, "READ-GRANT-EVIDENCE") || strings.Contains(readOutput, "error=true") {
|
||||
t.Fatalf("the child could not read the parent's read grant %q:\nargv=%v\n%s", readRoot, childArgs, readOutput)
|
||||
}
|
||||
target := filepath.Join(readRoot, "must-not-write.txt")
|
||||
writeOutput := launchAndRun(task, "write_file", map[string]string{"path": target, "content": "escalated"})
|
||||
if _, err := os.Stat(target); !os.IsNotExist(err) {
|
||||
t.Fatalf("the child WROTE inside the parent's read grant — a read grant escalated to write (stat err=%v):\nargv=%v\n%s",
|
||||
err, childArgs, writeOutput)
|
||||
}
|
||||
if !strings.Contains(writeOutput, "error=true") {
|
||||
t.Fatalf("the write inside the read grant was not reported as refused:\n%s", writeOutput)
|
||||
}
|
||||
}
|
||||
@@ -1,168 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"github.com/Twigpine/zero/internal/aimlapi"
|
||||
"github.com/Twigpine/zero/internal/providercatalog"
|
||||
)
|
||||
|
||||
const aimlapiKeyEnv = "AIMLAPI_API_KEY"
|
||||
|
||||
var (
|
||||
// ErrAIMLAPIKeyMissing means no AI/ML API key is configured.
|
||||
ErrAIMLAPIKeyMissing = errors.New("no AI/ML API key is configured; set up aimlapi.com with /provider or set AIMLAPI_API_KEY")
|
||||
// ErrAIMLAPIProxyProfile means the aimlapi profile points at a proxy, so its
|
||||
// key belongs to that proxy and is withheld from aimlapi.com services.
|
||||
ErrAIMLAPIProxyProfile = errors.New("the aimlapi provider points at a proxy base URL; its key is not sent to aimlapi.com services")
|
||||
// ErrAIMLAPIKeyInvalid means the key cannot be sent as a single Bearer token.
|
||||
ErrAIMLAPIKeyInvalid = errors.New("the AI/ML API key contains whitespace, a comma or a control character")
|
||||
)
|
||||
|
||||
// AIMLAPIServiceKey returns the AI/ML API key that may be sent to the
|
||||
// aimlapi.com Toolbox and MCP server. It is resolved again on every call, so a
|
||||
// provider change is seen by the next request.
|
||||
//
|
||||
// The key comes from the aimlapi provider profile (the active one when it is
|
||||
// aimlapi, otherwise the first), including a key held in the credential store,
|
||||
// or from AIMLAPI_API_KEY when no aimlapi profile exists. The key is withheld
|
||||
// when that profile, any aimlapi profile, or any profile reading
|
||||
// AIMLAPI_API_KEY points at a base URL other than the canonical
|
||||
// https://api.aimlapi.com/v1, and when AIMLAPI_INFERENCE_URL is overridden:
|
||||
// in those cases the credential may belong to a proxy.
|
||||
func AIMLAPIServiceKey(resolved ResolvedConfig, store APIKeyGetter, getenv func(string) string) (string, error) {
|
||||
if getenv == nil {
|
||||
getenv = os.Getenv
|
||||
}
|
||||
if override := strings.TrimSpace(getenv("AIMLAPI_INFERENCE_URL")); override != "" && !sameBaseURL(override, aimlapi.CanonicalInferenceBaseURL) {
|
||||
return "", ErrAIMLAPIProxyProfile
|
||||
}
|
||||
for _, profile := range resolved.Providers {
|
||||
if aimlapiKeyedProfile(profile) && !sameBaseURL(profile.BaseURL, aimlapi.CanonicalInferenceBaseURL) {
|
||||
return "", ErrAIMLAPIProxyProfile
|
||||
}
|
||||
}
|
||||
if profile, ok := aimlapiServiceProfile(resolved); ok {
|
||||
if !sameBaseURL(profile.BaseURL, aimlapi.CanonicalInferenceBaseURL) {
|
||||
return "", ErrAIMLAPIProxyProfile
|
||||
}
|
||||
if store != nil && profile.APIKeyStored && strings.TrimSpace(profile.APIKey) == "" && strings.TrimSpace(profile.Name) != "" {
|
||||
key, found, err := store.Get(strings.TrimSpace(profile.Name))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("read AI/ML API credential store: %w", err)
|
||||
}
|
||||
if found {
|
||||
profile.APIKey = key
|
||||
}
|
||||
}
|
||||
return validAIMLAPIServiceKey(profile.APIKey)
|
||||
}
|
||||
return validAIMLAPIServiceKey(getenv(aimlapiKeyEnv))
|
||||
}
|
||||
|
||||
// LoadAIMLAPIServiceKey resolves the user configuration, the environment and
|
||||
// the user credential store, then applies AIMLAPIServiceKey. Project config is
|
||||
// not read: provider keys are user-scoped. A config with no active provider
|
||||
// still resolves its profiles (ErrNoActiveProvider), so the key is looked up in
|
||||
// that partial config; every other resolve error is returned. Resolve drops a
|
||||
// non-active profile it cannot normalize, so the configured profiles are
|
||||
// checked as well: such a profile still withholds the key when it carries it.
|
||||
func LoadAIMLAPIServiceKey() (string, error) {
|
||||
userConfigPath, err := DefaultUserConfigPath()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
existing, err := existingConfigFile(userConfigPath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
resolved, err := Resolve(ResolveOptions{UserConfigPath: existing, ExcludeProject: true})
|
||||
if err != nil && !errors.Is(err, ErrNoActiveProvider) {
|
||||
return "", err
|
||||
}
|
||||
if existing != "" {
|
||||
fileConfig, err := loadConfigFile(existing)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var configured FileConfig
|
||||
mergeConfig(&configured, fileConfig)
|
||||
if droppedAIMLAPIProxyProfile(configured.Providers, resolved.Providers) {
|
||||
return "", ErrAIMLAPIProxyProfile
|
||||
}
|
||||
}
|
||||
// Check proxy eligibility and already-resolved keys before opening a store.
|
||||
key, keyErr := AIMLAPIServiceKey(resolved, nil, os.Getenv)
|
||||
if !errors.Is(keyErr, ErrAIMLAPIKeyMissing) {
|
||||
return key, keyErr
|
||||
}
|
||||
profile, ok := aimlapiServiceProfile(resolved)
|
||||
if !ok || !profile.APIKeyStored {
|
||||
return key, keyErr
|
||||
}
|
||||
store, err := ProviderKeyStoreAt(filepath.Dir(userConfigPath))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("open AI/ML API credential store: %w", err)
|
||||
}
|
||||
return AIMLAPIServiceKey(resolved, store, os.Getenv)
|
||||
}
|
||||
|
||||
func isAIMLAPIProfile(profile ProviderProfile) bool {
|
||||
return strings.EqualFold(providercatalog.NormalizeID(profile.CatalogID), providercatalog.AIMLAPIID)
|
||||
}
|
||||
|
||||
// aimlapiKeyedProfile reports whether profile carries an AI/ML API credential:
|
||||
// an aimlapi catalog profile, or any profile that reads AIMLAPI_API_KEY.
|
||||
func aimlapiKeyedProfile(profile ProviderProfile) bool {
|
||||
return isAIMLAPIProfile(profile) || strings.EqualFold(strings.TrimSpace(profile.APIKeyEnv), aimlapiKeyEnv)
|
||||
}
|
||||
|
||||
// droppedAIMLAPIProxyProfile reports whether a configured profile missing from
|
||||
// the resolved list (Resolve could not normalize it) carries an AI/ML API
|
||||
// credential and is not explicitly on the canonical base URL. Its endpoint was
|
||||
// never resolved, so an empty base URL counts as a proxy.
|
||||
func droppedAIMLAPIProxyProfile(configured, resolved []ProviderProfile) bool {
|
||||
kept := make(map[string]bool, len(resolved))
|
||||
for _, profile := range resolved {
|
||||
kept[profile.Name] = true
|
||||
}
|
||||
for _, profile := range configured {
|
||||
if kept[strings.TrimSpace(profile.Name)] || !aimlapiKeyedProfile(profile) {
|
||||
continue
|
||||
}
|
||||
if !sameBaseURL(profile.BaseURL, aimlapi.CanonicalInferenceBaseURL) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func aimlapiServiceProfile(resolved ResolvedConfig) (ProviderProfile, bool) {
|
||||
if isAIMLAPIProfile(resolved.Provider) {
|
||||
return resolved.Provider, true
|
||||
}
|
||||
for _, profile := range resolved.Providers {
|
||||
if isAIMLAPIProfile(profile) {
|
||||
return profile, true
|
||||
}
|
||||
}
|
||||
return ProviderProfile{}, false
|
||||
}
|
||||
|
||||
func validAIMLAPIServiceKey(raw string) (string, error) {
|
||||
key := strings.TrimSpace(raw)
|
||||
if key == "" {
|
||||
return "", ErrAIMLAPIKeyMissing
|
||||
}
|
||||
for _, r := range key {
|
||||
if unicode.IsSpace(r) || r <= ' ' || r == 0x7f || r == ',' || (r >= 0x80 && r <= 0x9f) {
|
||||
return "", ErrAIMLAPIKeyInvalid
|
||||
}
|
||||
}
|
||||
return key, nil
|
||||
}
|
||||
@@ -1,282 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func aimlapiServiceKeyEnv(values map[string]string) func(string) string {
|
||||
return func(name string) string { return values[name] }
|
||||
}
|
||||
|
||||
func TestAIMLAPIServiceKeyUsesCanonicalProfileKey(t *testing.T) {
|
||||
resolved := ResolvedConfig{Providers: []ProviderProfile{
|
||||
{Name: "openai", BaseURL: "https://api.openai.com/v1", APIKey: "sk-openai"},
|
||||
{Name: "aimlapi", CatalogID: "aimlapi", BaseURL: "https://api.aimlapi.com/v1/", APIKey: " aiml-key "},
|
||||
}}
|
||||
key, err := AIMLAPIServiceKey(resolved, nil, aimlapiServiceKeyEnv(nil))
|
||||
if err != nil || key != "aiml-key" {
|
||||
t.Fatalf("AIMLAPIServiceKey = %q, %v; want aiml-key", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAIMLAPIServiceKeyReadsCredentialStore(t *testing.T) {
|
||||
resolved := ResolvedConfig{Provider: ProviderProfile{Name: "aimlapi", CatalogID: "aimlapi", BaseURL: "https://api.aimlapi.com/v1", APIKeyStored: true}}
|
||||
store := fakeKeyGetter{keys: map[string]string{"aimlapi": "stored-key"}}
|
||||
key, err := AIMLAPIServiceKey(resolved, store, aimlapiServiceKeyEnv(nil))
|
||||
if err != nil || key != "stored-key" {
|
||||
t.Fatalf("AIMLAPIServiceKey = %q, %v; want stored-key", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAIMLAPIServiceKeyPrefersResolvedProfileKeyOverCredentialStore(t *testing.T) {
|
||||
resolved := ResolvedConfig{Provider: ProviderProfile{
|
||||
Name: "aimlapi", CatalogID: "aimlapi", BaseURL: "https://api.aimlapi.com/v1",
|
||||
APIKey: "env-key", APIKeyStored: true,
|
||||
}}
|
||||
store := fakeKeyGetter{keys: map[string]string{"aimlapi": "stored-key"}}
|
||||
key, err := AIMLAPIServiceKey(resolved, store, aimlapiServiceKeyEnv(map[string]string{"AIMLAPI_API_KEY": "env-key"}))
|
||||
if err != nil || key != "env-key" {
|
||||
t.Fatalf("AIMLAPIServiceKey = %q, %v; want env-key from resolved profile, not stored-key", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAIMLAPIServiceKeyFallsBackToEnvironmentWithoutProfile(t *testing.T) {
|
||||
key, err := AIMLAPIServiceKey(ResolvedConfig{}, nil, aimlapiServiceKeyEnv(map[string]string{"AIMLAPI_API_KEY": "env-key"}))
|
||||
if err != nil || key != "env-key" {
|
||||
t.Fatalf("AIMLAPIServiceKey = %q, %v; want env-key", key, err)
|
||||
}
|
||||
if _, err := AIMLAPIServiceKey(ResolvedConfig{}, nil, aimlapiServiceKeyEnv(nil)); !errors.Is(err, ErrAIMLAPIKeyMissing) {
|
||||
t.Fatalf("missing key error = %v, want ErrAIMLAPIKeyMissing", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAIMLAPIServiceKeyWithholdsProxyCredentials(t *testing.T) {
|
||||
env := aimlapiServiceKeyEnv(map[string]string{"AIMLAPI_API_KEY": "proxy-key"})
|
||||
cases := map[string]ResolvedConfig{
|
||||
"active aimlapi profile on a proxy": {Provider: ProviderProfile{Name: "aimlapi", CatalogID: "aimlapi", BaseURL: "https://proxy.example/v1", APIKey: "proxy-key"}},
|
||||
"a second aimlapi profile on a proxy": {Providers: []ProviderProfile{
|
||||
{Name: "aimlapi", CatalogID: "aimlapi", BaseURL: "https://api.aimlapi.com/v1", APIKey: "proxy-key"},
|
||||
{Name: "aimlapi-proxy", CatalogID: "aimlapi", BaseURL: "https://proxy.example/v1", APIKey: "proxy-key"},
|
||||
}},
|
||||
"a custom profile reading AIMLAPI_API_KEY": {Providers: []ProviderProfile{
|
||||
{Name: "gateway", BaseURL: "https://proxy.example/v1", APIKeyEnv: "AIMLAPI_API_KEY", APIKey: "proxy-key"},
|
||||
}},
|
||||
}
|
||||
for name, resolved := range cases {
|
||||
key, err := AIMLAPIServiceKey(resolved, nil, env)
|
||||
if !errors.Is(err, ErrAIMLAPIProxyProfile) || key != "" {
|
||||
t.Errorf("%s: AIMLAPIServiceKey = %q, %v; want ErrAIMLAPIProxyProfile", name, key, err)
|
||||
}
|
||||
}
|
||||
|
||||
overridden := aimlapiServiceKeyEnv(map[string]string{"AIMLAPI_API_KEY": "proxy-key", "AIMLAPI_INFERENCE_URL": "https://staging.example/v1"})
|
||||
if key, err := AIMLAPIServiceKey(ResolvedConfig{}, nil, overridden); !errors.Is(err, ErrAIMLAPIProxyProfile) || key != "" {
|
||||
t.Fatalf("inference URL override: AIMLAPIServiceKey = %q, %v; want ErrAIMLAPIProxyProfile", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAIMLAPIServiceKeyRejectsKeysThatAreNotOneToken(t *testing.T) {
|
||||
for _, raw := range []string{"first\u00a0second", "first\u2003second", "first,second", "first\nsecond", "first\r\nsecond", "a b", "key\x00tail", "key\u0085tail"} {
|
||||
resolved := ResolvedConfig{Provider: ProviderProfile{Name: "aimlapi", CatalogID: "aimlapi", BaseURL: "https://api.aimlapi.com/v1", APIKey: raw}}
|
||||
if key, err := AIMLAPIServiceKey(resolved, nil, aimlapiServiceKeyEnv(nil)); !errors.Is(err, ErrAIMLAPIKeyInvalid) || key != "" {
|
||||
t.Errorf("key %q: AIMLAPIServiceKey = %q, %v; want ErrAIMLAPIKeyInvalid", raw, key, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadAIMLAPIServiceKeyResolvesUserConfig(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
t.Setenv("HOME", home)
|
||||
t.Setenv("APPDATA", filepath.Join(home, "AppData"))
|
||||
t.Setenv("USERPROFILE", home)
|
||||
t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config"))
|
||||
t.Setenv("ZERO_CRED_STORAGE", "file")
|
||||
t.Setenv("ZERO_PROVIDER", "")
|
||||
t.Setenv("AIMLAPI_INFERENCE_URL", "")
|
||||
t.Setenv("AIMLAPI_API_KEY", "env-key")
|
||||
|
||||
if key, err := LoadAIMLAPIServiceKey(); err != nil || key != "env-key" {
|
||||
t.Fatalf("without config: LoadAIMLAPIServiceKey = %q, %v; want env-key", key, err)
|
||||
}
|
||||
|
||||
path, err := DefaultUserConfigPath()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
proxy := `{"activeProvider":"aimlapi","providers":[{"name":"aimlapi","catalogID":"aimlapi","baseURL":"https://proxy.example/v1","model":"m"}]}`
|
||||
if err := os.WriteFile(path, []byte(proxy), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if key, err := LoadAIMLAPIServiceKey(); !errors.Is(err, ErrAIMLAPIProxyProfile) || key != "" {
|
||||
t.Fatalf("proxy profile: LoadAIMLAPIServiceKey = %q, %v; want ErrAIMLAPIProxyProfile", key, err)
|
||||
}
|
||||
|
||||
canonical := `{"activeProvider":"aimlapi","providers":[{"name":"aimlapi","catalogID":"aimlapi"}]}`
|
||||
if err := os.WriteFile(path, []byte(canonical), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if key, err := LoadAIMLAPIServiceKey(); err != nil || key != "env-key" {
|
||||
t.Fatalf("canonical profile: LoadAIMLAPIServiceKey = %q, %v; want env-key", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Several profiles and no activeProvider make Resolve return
|
||||
// ErrNoActiveProvider with the normalized profiles; the interactive app keeps
|
||||
// running on that partial config, so the service key must still resolve.
|
||||
func TestLoadAIMLAPIServiceKeyWithoutActiveProvider(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
t.Setenv("HOME", home)
|
||||
t.Setenv("APPDATA", filepath.Join(home, "AppData"))
|
||||
t.Setenv("USERPROFILE", home)
|
||||
t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config"))
|
||||
t.Setenv("ZERO_CRED_STORAGE", "file")
|
||||
t.Setenv("ZERO_PROVIDER", "")
|
||||
t.Setenv("AIMLAPI_INFERENCE_URL", "")
|
||||
t.Setenv("AIMLAPI_API_KEY", "env-key")
|
||||
path, err := DefaultUserConfigPath()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
other := `{"name":"other","provider":"openai","model":"m","apiKey":"other-key"}`
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
active string
|
||||
profile string
|
||||
want string
|
||||
wantErr error
|
||||
}{
|
||||
{name: "environment key", profile: `{"name":"second","provider":"openai","model":"m","apiKey":"second-key"}`, want: "env-key"},
|
||||
{name: "stale active provider", active: "gone", profile: `{"name":"second","provider":"openai","model":"m","apiKey":"second-key"}`, want: "env-key"},
|
||||
{name: "aimlapi profile key", profile: `{"name":"aimlapi","catalogID":"aimlapi","model":"m","apiKey":"profile-key"}`, want: "profile-key"},
|
||||
{name: "proxy profile", profile: `{"name":"aimlapi","catalogID":"aimlapi","baseURL":"https://proxy.example/v1","model":"m"}`, wantErr: ErrAIMLAPIProxyProfile},
|
||||
} {
|
||||
data := `{"activeProvider":"` + tc.active + `","providers":[` + other + `,` + tc.profile + `]}`
|
||||
if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
key, err := LoadAIMLAPIServiceKey()
|
||||
if tc.wantErr != nil {
|
||||
if !errors.Is(err, tc.wantErr) || key != "" {
|
||||
t.Fatalf("%s: LoadAIMLAPIServiceKey = %q, %v; want %v", tc.name, key, err, tc.wantErr)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err != nil || key != tc.want {
|
||||
t.Fatalf("%s: LoadAIMLAPIServiceKey = %q, %v; want %q", tc.name, key, err, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Resolve drops a non-active profile it cannot normalize (here an unknown
|
||||
// catalog entry). Such a profile still shows that AIMLAPI_API_KEY belongs to a
|
||||
// proxy, so the key is withheld whether or not a provider is active.
|
||||
func TestLoadAIMLAPIServiceKeyWithholdsForDroppedProxyProfile(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
t.Setenv("HOME", home)
|
||||
t.Setenv("APPDATA", filepath.Join(home, "AppData"))
|
||||
t.Setenv("USERPROFILE", home)
|
||||
t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config"))
|
||||
t.Setenv("ZERO_CRED_STORAGE", "file")
|
||||
t.Setenv("ZERO_PROVIDER", "")
|
||||
t.Setenv("AIMLAPI_INFERENCE_URL", "")
|
||||
t.Setenv("AIMLAPI_API_KEY", "env-key")
|
||||
path, err := DefaultUserConfigPath()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
canonical := `{"name":"aimlapi","catalogID":"aimlapi","model":"m","apiKey":"profile-key"}`
|
||||
other := `{"name":"other","provider":"openai","model":"m","apiKey":"other-key"}`
|
||||
dropped := `{"name":"future","catalogID":"no-such-catalog","baseURL":"https://proxy.example/v1","apiKeyEnv":"AIMLAPI_API_KEY","model":"m"}`
|
||||
unrelated := `{"name":"future","catalogID":"no-such-catalog","baseURL":"https://proxy.example/v1","apiKeyEnv":"OTHER_API_KEY","model":"m"}`
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
data string
|
||||
want string
|
||||
wantErr error
|
||||
}{
|
||||
{name: "active aimlapi profile", data: `{"activeProvider":"aimlapi","providers":[` + canonical + `,` + dropped + `]}`, wantErr: ErrAIMLAPIProxyProfile},
|
||||
{name: "no active provider", data: `{"providers":[` + other + `,` + dropped + `]}`, wantErr: ErrAIMLAPIProxyProfile},
|
||||
{name: "dropped profile without the key", data: `{"activeProvider":"aimlapi","providers":[` + canonical + `,` + unrelated + `]}`, want: "profile-key"},
|
||||
} {
|
||||
if err := os.WriteFile(path, []byte(tc.data), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
key, err := LoadAIMLAPIServiceKey()
|
||||
if tc.wantErr != nil {
|
||||
if !errors.Is(err, tc.wantErr) || key != "" {
|
||||
t.Fatalf("%s: LoadAIMLAPIServiceKey = %q, %v; want %v", tc.name, key, err, tc.wantErr)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err != nil || key != tc.want {
|
||||
t.Fatalf("%s: LoadAIMLAPIServiceKey = %q, %v; want %q", tc.name, key, err, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadAIMLAPIServiceKeyStoreOpenFailure(t *testing.T) {
|
||||
home := t.TempDir()
|
||||
t.Setenv("HOME", home)
|
||||
t.Setenv("APPDATA", filepath.Join(home, "AppData"))
|
||||
t.Setenv("USERPROFILE", home)
|
||||
t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config"))
|
||||
t.Setenv("ZERO_PROVIDER", "")
|
||||
t.Setenv("AIMLAPI_INFERENCE_URL", "")
|
||||
t.Setenv("AIMLAPI_API_KEY", "env-key")
|
||||
t.Setenv("ZERO_CRED_STORAGE", "invalid-test-storage")
|
||||
if key, err := LoadAIMLAPIServiceKey(); err != nil || key != "env-key" {
|
||||
t.Fatalf("environment key must not require a store: %q, %v", key, err)
|
||||
}
|
||||
path, err := DefaultUserConfigPath()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("AIMLAPI_API_KEY", "")
|
||||
data := `{"activeProvider":"aimlapi","providers":[{"name":"aimlapi","catalogID":"aimlapi","model":"m","apiKeyStored":true}]}`
|
||||
if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if key, err := LoadAIMLAPIServiceKey(); key != "" || err == nil || !strings.Contains(err.Error(), "unknown storage") {
|
||||
t.Fatalf("stored key must report store failure: %q, %v", key, err)
|
||||
}
|
||||
t.Setenv("AIMLAPI_INFERENCE_URL", "https://proxy.example/v1")
|
||||
if key, err := LoadAIMLAPIServiceKey(); key != "" || !errors.Is(err, ErrAIMLAPIProxyProfile) {
|
||||
t.Fatalf("proxy rejection must precede store open: %q, %v", key, err)
|
||||
}
|
||||
t.Setenv("AIMLAPI_INFERENCE_URL", "")
|
||||
data = `{"activeProvider":"aimlapi","providers":[{"name":"aimlapi","catalogID":"aimlapi","model":"m","apiKeyStored":true,"apiKey":"inline-key"}]}`
|
||||
if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if key, err := LoadAIMLAPIServiceKey(); err != nil || key != "inline-key" {
|
||||
t.Fatalf("inline key must not require a store: %q, %v", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAIMLAPIServiceKeyReportsStoreReadFailure(t *testing.T) {
|
||||
want := errors.New("test credential read failure")
|
||||
resolved := ResolvedConfig{Provider: ProviderProfile{Name: "aimlapi", CatalogID: "aimlapi", BaseURL: "https://api.aimlapi.com/v1", APIKeyStored: true}}
|
||||
if key, err := AIMLAPIServiceKey(resolved, fakeKeyGetter{err: want}, aimlapiServiceKeyEnv(nil)); key != "" || !errors.Is(err, want) {
|
||||
t.Fatalf("read failure was hidden: %q, %v", key, err)
|
||||
}
|
||||
resolved.Provider.BaseURL = "https://proxy.example/v1"
|
||||
if _, err := AIMLAPIServiceKey(resolved, fakeKeyGetter{err: want}, aimlapiServiceKeyEnv(nil)); !errors.Is(err, ErrAIMLAPIProxyProfile) {
|
||||
t.Fatalf("proxy check must precede store read: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,51 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEnsureChatGPTDefaultPreservesSavedSelection(t *testing.T) {
|
||||
for _, saved := range []string{"", "gpt-explicit-saved"} {
|
||||
name := "new profile"
|
||||
if saved != "" {
|
||||
name = "saved selection"
|
||||
}
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "config.json")
|
||||
if saved != "" {
|
||||
body, err := json.Marshal(FileConfig{ActiveProvider: "codex", Providers: []ProviderProfile{{Name: "codex", CatalogID: "chatgpt", Model: saved}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, body, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
ensured, err := EnsureCatalogProvider(path, "chatgpt")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var cfg FileConfig
|
||||
if err := json.Unmarshal(body, &cfg); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := "gpt-6.1-sol"
|
||||
if saved != "" {
|
||||
want = saved
|
||||
}
|
||||
if len(cfg.Providers) != 1 || cfg.Providers[0].Model != want {
|
||||
t.Fatalf("ChatGPT default/selection = %#v, want model %q", cfg.Providers, want)
|
||||
}
|
||||
if saved != "" && (ensured.Name != "codex" || ensured.Created) {
|
||||
t.Fatalf("saved subscription profile was replaced: %#v", ensured)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1,101 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// A restart goes through Resolve, so that is where the persistence has to be
|
||||
// proven.
|
||||
//
|
||||
// The existing coverage (internal/tui/keep_finished_agents_test.go) round-trips
|
||||
// through SetKeepFinishedAgents and reads the file back, which proves the WRITE.
|
||||
// It stayed green for the whole life of the bug, because the half that was
|
||||
// broken is the read on the next launch, and Resolve is the only thing on that
|
||||
// path. mergeConfig copied FavoriteModels, RecentModels, Recaps and Theme and
|
||||
// never touched this key, so the value reached disk and died there.
|
||||
//
|
||||
// Recaps and Theme are asserted alongside on purpose. They are the siblings this
|
||||
// field is modelled on, so a future edit that breaks the merge for one of them
|
||||
// says which.
|
||||
func TestKeepFinishedAgentsSurvivesResolve(t *testing.T) {
|
||||
for _, testCase := range []struct {
|
||||
name string
|
||||
body string
|
||||
want bool
|
||||
}{
|
||||
{
|
||||
name: "an explicit keep survives the merge",
|
||||
body: `{"preferences":{"keepFinishedAgents":true,"recaps":false,"theme":"dracula"}}`,
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
// The reason the field is a pointer: with omitempty an explicit false
|
||||
// is indistinguishable from an absent key unless the merge copies the
|
||||
// pointer rather than the value.
|
||||
name: "an explicit drop survives it too",
|
||||
body: `{"preferences":{"keepFinishedAgents":false}}`,
|
||||
want: false,
|
||||
},
|
||||
} {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "config.json")
|
||||
if err := os.WriteFile(path, []byte(testCase.body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resolved, err := Resolve(ResolveOptions{UserConfigPath: path, Env: map[string]string{}})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v", err)
|
||||
}
|
||||
if got := resolved.Preferences.KeepsFinishedAgents(); got != testCase.want {
|
||||
t.Errorf("after Resolve, KeepsFinishedAgents() = %v, want %v; the persisted preference did not survive the merge", got, testCase.want)
|
||||
}
|
||||
if resolved.Preferences.KeepFinishedAgents == nil {
|
||||
t.Error("the merge dropped the pointer, so an explicit false is now indistinguishable from an unset key")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The neighbouring preferences must keep coming through. This is the assertion
|
||||
// that would have localised the bug: theme and recaps arriving while
|
||||
// keepFinishedAgents did not is what pointed at mergeConfig rather than at the
|
||||
// unmarshal or the writer.
|
||||
func TestNeighbouringPreferencesStillSurviveResolve(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "config.json")
|
||||
body := `{"preferences":{"keepFinishedAgents":true,"recaps":false,"theme":"dracula"}}`
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resolved, err := Resolve(ResolveOptions{UserConfigPath: path, Env: map[string]string{}})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v", err)
|
||||
}
|
||||
if resolved.Preferences.Theme != "dracula" {
|
||||
t.Errorf("theme = %q, want dracula", resolved.Preferences.Theme)
|
||||
}
|
||||
if resolved.Preferences.Recaps == nil || *resolved.Preferences.Recaps {
|
||||
t.Error("an explicit recaps=false did not survive Resolve")
|
||||
}
|
||||
}
|
||||
|
||||
// Absent stays absent. Without this, a merge that assigned the key
|
||||
// unconditionally would pass the tests above while quietly turning the product
|
||||
// default into "keep".
|
||||
func TestKeepFinishedAgentsStaysUnsetWhenTheConfigIsSilent(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "config.json")
|
||||
if err := os.WriteFile(path, []byte(`{"preferences":{"theme":"dracula"}}`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resolved, err := Resolve(ResolveOptions{UserConfigPath: path, Env: map[string]string{}})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v", err)
|
||||
}
|
||||
if resolved.Preferences.KeepFinishedAgents != nil {
|
||||
t.Error("a config that says nothing about the preference resolved to an explicit value")
|
||||
}
|
||||
if resolved.Preferences.KeepsFinishedAgents() {
|
||||
t.Error("the established default is to drop finished agents")
|
||||
}
|
||||
}
|
||||
@@ -1,76 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// PROJECT CONFIG MAY ONLY TIGHTEN THE TURN BUDGET.
|
||||
//
|
||||
// A cloned repo setting maxTurns to the ceiling raises the per-run cost for
|
||||
// whoever opens it — the same hazard PlanSize and DisableZeromaxing are
|
||||
// tighten-only for. User config stays free to raise: it is the user's own file.
|
||||
func TestProjectConfigMayLowerTheTurnBudgetButNeverRaiseIt(t *testing.T) {
|
||||
for name, tc := range map[string]struct {
|
||||
user, project, want int
|
||||
}{
|
||||
"cannot raise a user value": {user: 80, project: 500, want: 80},
|
||||
"may lower a user value": {user: 80, project: 50, want: 50},
|
||||
"cannot raise from the default": {user: 0, project: 500, want: defaultMaxTurns},
|
||||
"may lower below the default": {user: 0, project: 50, want: 50},
|
||||
"equal changes nothing": {user: 80, project: 80, want: 80},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
userPath := filepath.Join(dir, "user.json")
|
||||
projectPath := filepath.Join(dir, "project.json")
|
||||
userBody := `{"providers":[{"name":"p","provider_kind":"openai-compatible","catalogID":"xai",` +
|
||||
`"baseURL":"https://api.x.ai/v1","apiFormat":"chat-completions","model":"m","apiKey":"k"}],` +
|
||||
`"activeProvider":"p"`
|
||||
if tc.user > 0 {
|
||||
userBody += `,"maxTurns":` + strconv.Itoa(tc.user)
|
||||
}
|
||||
userBody += `}`
|
||||
if err := os.WriteFile(userPath, []byte(userBody), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(projectPath, []byte(`{"maxTurns":`+strconv.Itoa(tc.project)+`}`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
resolved, err := Resolve(ResolveOptions{UserConfigPath: userPath, ProjectConfigPath: projectPath})
|
||||
if err != nil {
|
||||
t.Fatalf("resolve: %v", err)
|
||||
}
|
||||
if resolved.MaxTurns != tc.want {
|
||||
t.Errorf("maxTurns = %d, want %d", resolved.MaxTurns, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ZERO MEANS "THE DEFAULT", NOT "NO LIMIT", and the distinction is the whole
|
||||
// defect: the fallback to defaultMaxTurns happens AFTER the project merge, so
|
||||
// treating an unset user value as unbounded lets a repo raise 80 to 500 for
|
||||
// every user who never wrote a config — the common case, and the one this rule
|
||||
// exists for.
|
||||
func TestAnUnsetUserBudgetIsNotTreatedAsUnbounded(t *testing.T) {
|
||||
dst := FileConfig{}
|
||||
if err := mergeProjectConfig(&dst, FileConfig{MaxTurns: 500}); err != nil {
|
||||
t.Fatalf("merge: %v", err)
|
||||
}
|
||||
if dst.MaxTurns != 0 {
|
||||
t.Fatalf("a project raised the budget to %d against an unset user value; it must stay 0 so the default applies", dst.MaxTurns)
|
||||
}
|
||||
}
|
||||
|
||||
// The user's own file may still raise it — higher trust, their own money.
|
||||
func TestUserConfigMayStillRaiseTheTurnBudget(t *testing.T) {
|
||||
dst := FileConfig{MaxTurns: 80}
|
||||
mergeConfig(&dst, FileConfig{MaxTurns: 300})
|
||||
if dst.MaxTurns != 300 {
|
||||
t.Errorf("user config could not raise its own budget: %d", dst.MaxTurns)
|
||||
}
|
||||
}
|
||||
@@ -3,48 +3,26 @@ package config
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
|
||||
"github.com/Twigpine/zero/internal/aimlapi"
|
||||
)
|
||||
|
||||
// DefaultMCPServers returns the MCP servers Zero ships built in. Exa is ENABLED
|
||||
// by default so web search and page fetching work out of the box with no setup
|
||||
// and no API key. They are seeded before user/project config is merged (see
|
||||
// ResolveMCP), so a user can override any field — for example add an API-key
|
||||
// header to lift Exa's anonymous rate limit — or disable it entirely with
|
||||
// DefaultMCPServers returns the MCP servers Zero ships ENABLED by default so
|
||||
// web search and page fetching work out of the box with no setup and no API
|
||||
// key. They are seeded before user/project config is merged (see ResolveMCP),
|
||||
// so a user can override any field — for example add an API-key header to lift
|
||||
// Exa's anonymous rate limit — or disable it entirely with
|
||||
// `zero mcp disable <name>` (which writes `"disabled": true`).
|
||||
//
|
||||
// Exa's hosted MCP server works anonymously with rate limits. Users can add an
|
||||
// Exa API key for higher limits.
|
||||
//
|
||||
// The AI/ML API server ships DISABLED: its tool runs and generations are billed
|
||||
// to the user's AI/ML API balance, so it only connects after
|
||||
// `zero mcp enable aimlapi` (which writes `"disabled": false`). The disable is
|
||||
// marked as explicitly set so the lower-trust project layer cannot lift it
|
||||
// (see mergeMCPServer); only the user and CLI-override scopes can. The key is
|
||||
// attached per request by the MCP client and is never part of this config.
|
||||
func DefaultMCPServers() map[string]MCPServerConfig {
|
||||
return map[string]MCPServerConfig{
|
||||
"exa": {
|
||||
Type: "http",
|
||||
URL: "https://mcp.exa.ai/mcp",
|
||||
},
|
||||
"aimlapi": {
|
||||
Type: "http",
|
||||
URL: aimlapi.MCPServerURL,
|
||||
Disabled: true,
|
||||
disabledSet: true,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// IsDefaultMCPServerDisabled reports whether name is a built-in default that
|
||||
// ships disabled, so enabling it has to be written out as `"disabled": false`
|
||||
// rather than by dropping the flag.
|
||||
func IsDefaultMCPServerDisabled(name string) bool {
|
||||
return DefaultMCPServers()[strings.TrimSpace(name)].Disabled
|
||||
}
|
||||
|
||||
// IsDefaultMCPServer reports whether name is one of Zero's built-in default MCP
|
||||
// servers. The config commands use it so a default can be disabled/enabled even
|
||||
// though it is not written to the user's config file until overridden.
|
||||
|
||||
@@ -291,92 +291,3 @@ func TestResolveMCPDisabledRetiredEntryStaysReEnableable(t *testing.T) {
|
||||
t.Fatal("the disable must still carry to the successor")
|
||||
}
|
||||
}
|
||||
|
||||
func writeMCPDefaultsTestConfig(t *testing.T, data string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "config.json")
|
||||
if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestResolveMCPSeedsDisabledAIMLAPIDefault(t *testing.T) {
|
||||
if !IsDefaultMCPServer("aimlapi") || !IsDefaultMCPServerDisabled("aimlapi") || IsDefaultMCPServerDisabled("exa") {
|
||||
t.Fatal("aimlapi must be a built-in default that ships disabled, exa one that ships enabled")
|
||||
}
|
||||
// No user config at all, and a user config that never mentions aimlapi.
|
||||
for name, options := range map[string]ResolveOptions{
|
||||
"no config": {},
|
||||
"unrelated user file": {UserConfigPath: writeMCPDefaultsTestConfig(t, `{"mcp":{"servers":{"exa":{"disabled":true}}}}`)},
|
||||
"user entry with only headers": {UserConfigPath: writeMCPDefaultsTestConfig(t,
|
||||
`{"mcp":{"servers":{"aimlapi":{"headers":{"X-Trace":"1"}}}}}`)},
|
||||
} {
|
||||
cfg, err := ResolveMCP(options)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: ResolveMCP: %v", name, err)
|
||||
}
|
||||
server, ok := cfg.Servers["aimlapi"]
|
||||
if !ok {
|
||||
t.Fatalf("%s: the aimlapi default must be visible in the resolved config", name)
|
||||
}
|
||||
if server.Type != "http" || server.URL != "https://mcp.aimlapi.com/mcp" || !server.Disabled {
|
||||
t.Fatalf("%s: aimlapi default = %#v, want a disabled http server at the MCP URL", name, server)
|
||||
}
|
||||
}
|
||||
cfg, err := ResolveMCP(ResolveOptions{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !IsUnconfiguredDefault("aimlapi", cfg.Servers["aimlapi"]) {
|
||||
t.Fatal("the untouched aimlapi default should be reported as unconfigured")
|
||||
}
|
||||
if exa := cfg.Servers["exa"]; exa.Disabled || !IsUnconfiguredDefault("exa", exa) {
|
||||
t.Fatalf("exa default changed: %#v", exa)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveMCPUserEnablesAIMLAPIDefault(t *testing.T) {
|
||||
path := writeMCPDefaultsTestConfig(t, `{"mcp":{"servers":{"aimlapi":{"disabled":false}}}}`)
|
||||
cfg, err := ResolveMCP(ResolveOptions{UserConfigPath: path})
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveMCP: %v", err)
|
||||
}
|
||||
server := cfg.Servers["aimlapi"]
|
||||
if server.Disabled || server.URL != "https://mcp.aimlapi.com/mcp" || server.Type != "http" {
|
||||
t.Fatalf("user enable did not turn the default on: %#v", server)
|
||||
}
|
||||
if IsUnconfiguredDefault("aimlapi", server) {
|
||||
t.Fatal("an enabled aimlapi is user-configured, so its failures must be reported")
|
||||
}
|
||||
|
||||
enable := MCPServerConfig{Disabled: false, disabledSet: true}
|
||||
cfg, err = ResolveMCP(ResolveOptions{Overrides: Overrides{MCP: MCPConfig{Servers: map[string]MCPServerConfig{"aimlapi": enable}}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.Servers["aimlapi"].Disabled {
|
||||
t.Fatal("the CLI override scope must be able to enable the aimlapi default")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveMCPProjectCannotEnableAIMLAPIDefault(t *testing.T) {
|
||||
project := writeMCPDefaultsTestConfig(t, `{"mcp":{"servers":{"aimlapi":{"disabled":false}}}}`)
|
||||
cfg, err := ResolveMCP(ResolveOptions{ProjectConfigPath: project})
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveMCP: %v", err)
|
||||
}
|
||||
if !cfg.Servers["aimlapi"].Disabled {
|
||||
t.Fatal("a project config must not turn on the billed aimlapi default")
|
||||
}
|
||||
|
||||
user := writeMCPDefaultsTestConfig(t, `{"mcp":{"servers":{"aimlapi":{"disabled":false}}}}`)
|
||||
projectDisable := writeMCPDefaultsTestConfig(t, `{"mcp":{"servers":{"aimlapi":{"disabled":true}}}}`)
|
||||
cfg, err = ResolveMCP(ResolveOptions{UserConfigPath: user, ProjectConfigPath: projectDisable})
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveMCP: %v", err)
|
||||
}
|
||||
if cfg.Servers["aimlapi"].Disabled {
|
||||
t.Fatal("the user's explicit enable wins over the project layer, as for any server")
|
||||
}
|
||||
}
|
||||
Loaded 100 of 531 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user