mirror of
https://github.com/Twigpine/zero.git
synced 2026-11-10 05:01:28 -10:00
* feat(tools): give tool results an image channel and add view_image Closes #824. tools.Result carried only text, so no tool could hand the model a picture. The sharpest edge was our own capture tooling: browser_screenshot, desktop_screenshot and browser_pdf wrote a PNG, set Meta["artifact_path"] and returned "Artifact captured: <path>". Nothing read that path afterwards, so the model asked for a screenshot, was told one existed, and could not look at it. Zero was never image-blind on the way in — zeroruntime.ImageBlock, Message.Images and internal/imageinput are mature. What was missing is the model-initiated leg. Result.Images is delivered as a following USER message, not on the tool result itself. Every provider drops images on a tool-role message: Anthropic maps a tool result to a tool_result block whose content is a string, Gemini to a functionResponse, and OpenAI guards its image content-parts to the user role with an explicit comment saying so. Attaching them to the tool result would have plumbed the whole path and delivered nothing. A separate message also keeps one tool result per tool call, which the providers validate. Blocks with no data, or a media type outside the provider allow-list, are dropped rather than sent — a rejected image would fail the whole turn, and the tool's text still stands on its own. The message names the tool, since the images arrive detached from their result and nothing else says which call produced them when several ran in one turn. view_image resolves through resolveScopedReadPath before imageinput.LoadFile. That scoping is load-bearing: LoadFile joins a relative path to the workspace root and otherwise takes the path as given, so without it the tool would read anywhere on disk. It is registered with the other scoped readers and declared read-only and thread-safe, matching read_file. The capture tools now attach what they wrote. A PDF or an unreadable file yields no image and the text stands alone, so a capture is never failed over its attachment. Co-Authored-By: Claude Opus 5 <[email protected]> * fix(agent): keep tool results contiguous when a tool returns an image The image messages are user-role, and the first version appended each one immediately after its own tool result. With two tool calls in a turn that produces tool_result, user, tool_result — and a user message between two tool results breaks strict provider replay. Anthropic makes it concrete. appendUserBlocks coalesces consecutive user content into one block list, and tool_result blocks must come first in that message, so the interleaved shape maps to: [tool_result, text, image, tool_result] which the API rejects. loop.go already documents this exact rule three lines below the change, for the self-correction feedback it defers for the same reason. Images are now collected during the tool loop and appended once every tool_result is recorded — including the aborted placeholders on the abort, stop-reason and repeated-failure paths, so an early exit cannot strand them mid-sequence either. The original test used a single tool call, which cannot exhibit the bug. The new one runs two calls where the first returns an image and asserts the image message does not fall between the first and last tool result. Also from review: view_image joins the "execute" tool category, which grants read_file but not view_image, so a specialist declared with tools: [execute] could not look at a screenshot it had just captured. Covered by a test that holds for every category granting read_file rather than naming one. And the core read-only set is now asserted by tool name, not only by count — a count still passes when one reader is swapped for another. Co-Authored-By: Claude Opus 5 <[email protected]> * test(agent): pin the tool-result/image ordering contract The assertions only tracked the first and last tool-result index, so four distinct broken orderings passed: a message wedged between the two tool results, a duplicated image flush, a duplicated tool result, and both results answering the same tool_call_id. Mutation-testing each of those against the real loop confirmed the old assertions were green on all four. Now it collects every tool-result index and id and requires exactly two, in c1-then-c2 order, at adjacent indexes, with the image following the last one and delivered exactly once. Adjacency is the assertion that carries the contract: Anthropic needs every tool_result block ahead of any other block in the user message they coalesce into, so anything between the two results is the 400 this change exists to avoid. The image is deliberately NOT pinned to exactly last+1. appendUserBlocks merges consecutive user-role content, so [tool_result, tool_result, text, image] maps to tool_result blocks first and replays fine; pinning would fail that valid shape while killing nothing the adjacency and count assertions do not already catch. messageShape reports the actual role ordering on failure, and the loop uses two ifs rather than a switch so an image left on a tool-role message is reported as the misplacement it is instead of "the image never reached the model". --------- Co-authored-by: Claude Opus 5 <[email protected]>
101 lines
3.4 KiB
Go
101 lines
3.4 KiB
Go
package tools
|
|
|
|
import (
|
|
"context"
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
"testing"
|
|
)
|
|
|
|
// A 1x1 PNG. Small enough to inline, real enough to pass media-type sniffing.
|
|
var onePixelPNG = []byte{
|
|
0x89, 'P', 'N', 'G', 0x0d, 0x0a, 0x1a, 0x0a,
|
|
0x00, 0x00, 0x00, 0x0d, 'I', 'H', 'D', 'R',
|
|
0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
|
|
0x08, 0x06, 0x00, 0x00, 0x00, 0x1f, 0x15, 0xc4,
|
|
0x89, 0x00, 0x00, 0x00, 0x0a, 'I', 'D', 'A', 'T',
|
|
0x78, 0x9c, 0x63, 0x00, 0x01, 0x00, 0x00, 0x05,
|
|
0x00, 0x01, 0x0d, 0x0a, 0x2d, 0xb4, 0x00, 0x00,
|
|
0x00, 0x00, 'I', 'E', 'N', 'D', 0xae, 0x42, 0x60, 0x82,
|
|
}
|
|
|
|
func TestViewImageReturnsTheImage(t *testing.T) {
|
|
root := t.TempDir()
|
|
if err := os.WriteFile(filepath.Join(root, "shot.png"), onePixelPNG, 0o600); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
result := NewViewImageTool(root).Run(context.Background(), map[string]any{"path": "shot.png"})
|
|
|
|
if result.Status != StatusOK {
|
|
t.Fatalf("status = %v: %s", result.Status, result.Output)
|
|
}
|
|
if len(result.Images) != 1 {
|
|
t.Fatalf("got %d images, want 1 — the text alone is useless to the model", len(result.Images))
|
|
}
|
|
if result.Images[0].MediaType != "image/png" {
|
|
t.Errorf("media type = %q, want image/png", result.Images[0].MediaType)
|
|
}
|
|
if len(result.Images[0].Data) != len(onePixelPNG) {
|
|
t.Errorf("image data is %d bytes, want %d", len(result.Images[0].Data), len(onePixelPNG))
|
|
}
|
|
}
|
|
|
|
// The tool must not reach outside the workspace. imageinput.LoadFile does no
|
|
// containment check of its own, so the scoping is entirely this tool's job.
|
|
func TestViewImageRefusesPathsOutsideTheWorkspace(t *testing.T) {
|
|
root := t.TempDir()
|
|
outside := t.TempDir()
|
|
secret := filepath.Join(outside, "outside.png")
|
|
if err := os.WriteFile(secret, onePixelPNG, 0o600); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
tool := NewViewImageTool(root)
|
|
|
|
for name, path := range map[string]string{
|
|
"absolute path outside": secret,
|
|
"parent traversal": filepath.Join("..", filepath.Base(outside), "outside.png"),
|
|
} {
|
|
t.Run(name, func(t *testing.T) {
|
|
result := tool.Run(context.Background(), map[string]any{"path": path})
|
|
if result.Status == StatusOK {
|
|
t.Fatalf("read an image outside the workspace: %s", result.Output)
|
|
}
|
|
if len(result.Images) != 0 {
|
|
t.Error("a refused read still returned image bytes")
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestViewImageRejectsNonImages(t *testing.T) {
|
|
root := t.TempDir()
|
|
if err := os.WriteFile(filepath.Join(root, "notes.txt"), []byte("not an image"), 0o600); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
result := NewViewImageTool(root).Run(context.Background(), map[string]any{"path": "notes.txt"})
|
|
if result.Status == StatusOK {
|
|
t.Fatalf("accepted a text file as an image: %s", result.Output)
|
|
}
|
|
if len(result.Images) != 0 {
|
|
t.Error("a rejected file still returned image bytes")
|
|
}
|
|
}
|
|
|
|
func TestViewImageReportsAMissingFileClearly(t *testing.T) {
|
|
result := NewViewImageTool(t.TempDir()).Run(context.Background(), map[string]any{"path": "nope.png"})
|
|
if result.Status == StatusOK {
|
|
t.Fatal("a missing file reported success")
|
|
}
|
|
// The error names the path the caller asked for, matching read_file. Both
|
|
// also carry the resolved absolute path from the underlying syscall error;
|
|
// that is pre-existing behaviour shared with read_file, not something this
|
|
// tool introduces, so it is not asserted either way here.
|
|
if !strings.Contains(result.Output, "nope.png") {
|
|
t.Errorf("error should name the file, got %q", result.Output)
|
|
}
|
|
if len(result.Images) != 0 {
|
|
t.Error("a failed read still returned image bytes")
|
|
}
|
|
}
|