feat(tools): give tool results an image channel and add view_image (#843)

* feat(tools): give tool results an image channel and add view_image

Closes #824.

tools.Result carried only text, so no tool could hand the model a picture.
The sharpest edge was our own capture tooling: browser_screenshot,
desktop_screenshot and browser_pdf wrote a PNG, set Meta["artifact_path"]
and returned "Artifact captured: <path>". Nothing read that path
afterwards, so the model asked for a screenshot, was told one existed, and
could not look at it.

Zero was never image-blind on the way in — zeroruntime.ImageBlock,
Message.Images and internal/imageinput are mature. What was missing is the
model-initiated leg.

Result.Images is delivered as a following USER message, not on the tool
result itself. Every provider drops images on a tool-role message:
Anthropic maps a tool result to a tool_result block whose content is a
string, Gemini to a functionResponse, and OpenAI guards its image
content-parts to the user role with an explicit comment saying so. Attaching
them to the tool result would have plumbed the whole path and delivered
nothing. A separate message also keeps one tool result per tool call, which
the providers validate.

Blocks with no data, or a media type outside the provider allow-list, are
dropped rather than sent — a rejected image would fail the whole turn, and
the tool's text still stands on its own. The message names the tool, since
the images arrive detached from their result and nothing else says which
call produced them when several ran in one turn.

view_image resolves through resolveScopedReadPath before imageinput.LoadFile.
That scoping is load-bearing: LoadFile joins a relative path to the
workspace root and otherwise takes the path as given, so without it the tool
would read anywhere on disk. It is registered with the other scoped readers
and declared read-only and thread-safe, matching read_file.

The capture tools now attach what they wrote. A PDF or an unreadable file
yields no image and the text stands alone, so a capture is never failed over
its attachment.

Co-Authored-By: Claude Opus 5 <[email protected]>

* fix(agent): keep tool results contiguous when a tool returns an image

The image messages are user-role, and the first version appended each one
immediately after its own tool result. With two tool calls in a turn that
produces tool_result, user, tool_result — and a user message between two
tool results breaks strict provider replay.

Anthropic makes it concrete. appendUserBlocks coalesces consecutive user
content into one block list, and tool_result blocks must come first in that
message, so the interleaved shape maps to:

    [tool_result, text, image, tool_result]

which the API rejects. loop.go already documents this exact rule three lines
below the change, for the self-correction feedback it defers for the same
reason.

Images are now collected during the tool loop and appended once every
tool_result is recorded — including the aborted placeholders on the abort,
stop-reason and repeated-failure paths, so an early exit cannot strand them
mid-sequence either.

The original test used a single tool call, which cannot exhibit the bug. The
new one runs two calls where the first returns an image and asserts the image
message does not fall between the first and last tool result.

Also from review: view_image joins the "execute" tool category, which grants
read_file but not view_image, so a specialist declared with tools: [execute]
could not look at a screenshot it had just captured. Covered by a test that
holds for every category granting read_file rather than naming one. And the
core read-only set is now asserted by tool name, not only by count — a count
still passes when one reader is swapped for another.

Co-Authored-By: Claude Opus 5 <[email protected]>

* test(agent): pin the tool-result/image ordering contract

The assertions only tracked the first and last tool-result index, so four
distinct broken orderings passed: a message wedged between the two tool results,
a duplicated image flush, a duplicated tool result, and both results answering
the same tool_call_id. Mutation-testing each of those against the real loop
confirmed the old assertions were green on all four.

Now it collects every tool-result index and id and requires exactly two, in
c1-then-c2 order, at adjacent indexes, with the image following the last one and
delivered exactly once. Adjacency is the assertion that carries the contract:
Anthropic needs every tool_result block ahead of any other block in the user
message they coalesce into, so anything between the two results is the 400 this
change exists to avoid.

The image is deliberately NOT pinned to exactly last+1. appendUserBlocks merges
consecutive user-role content, so [tool_result, tool_result, text, image] maps to
tool_result blocks first and replays fine; pinning would fail that valid shape
while killing nothing the adjacency and count assertions do not already catch.

messageShape reports the actual role ordering on failure, and the loop uses two
ifs rather than a switch so an image left on a tool-role message is reported as
the misplacement it is instead of "the image never reached the model".

---------

Co-authored-by: Claude Opus 5 <[email protected]>
This commit is contained in:
Vasanth TandClaude Opus 5 authored and GitHub committed 2026-08-06 21:35:41 +05:30
1 parent 4ca4fa75d3
commit cd0eb194e4
13 files changed
+526 -12

No files matched your search

+59
View File
@@ -656,6 +656,13 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// between tool_results breaks strict provider replay) — same after-batch
// rationale as turnRequestedModel above.
var changedFilesThisBatch []string
// Images produced by tools this turn, held until every tool_result is
// recorded. They travel as user messages, and a user message between two
// tool_results breaks strict provider replay — Anthropic coalesces them
// into one user block list and requires the tool_result blocks first, so
// interleaving yields [tool_result, text, image, tool_result] and a 400.
// Same reason the self-correction feedback below is deferred.
var toolImageMessages []zeroruntime.Message
// Parallel read-ahead state: results for calls[precomputedStart:precomputedEnd]
// executed concurrently, consumed strictly in order below.
var precomputed []precomputedToolResult
@@ -709,6 +716,16 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
Content: toolResult.Output,
ToolCallID: toolResult.ToolCallID,
})
// Images ride a following USER message rather than the tool result
// above. Every provider drops images on a tool-role message —
// Anthropic's tool_result content is a string, Gemini's is a
// functionResponse, and OpenAI guards its image parts to the user role
// — so attaching them there would silently deliver nothing. A separate
// message also keeps the one-tool-result-per-tool-call pairing intact,
// which the providers validate.
if imageMessage, ok := toolResultImageMessage(toolResult); ok {
toolImageMessages = append(toolImageMessages, imageMessage)
}
// A tool may demand the run ABORT — a canceled/timed-out ask_user prompt
// returns context.Canceled rather than fabricating a headless answer. Stop
@@ -719,11 +736,13 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
}
if abortErr != nil {
messages = appendAbortedToolResults(messages, collected.ToolCalls[index+1:])
messages = append(messages, toolImageMessages...)
result.Messages = copyMessages(messages)
return result, abortErr
}
if stopReason := stopReasonFromToolResult(toolResult); stopReason != "" {
messages = appendAbortedToolResults(messages, collected.ToolCalls[index+1:])
messages = append(messages, toolImageMessages...)
result.FinalAnswer = toolResult.Output
result.StopReason = stopReason
result.Messages = copyMessages(messages)
@@ -747,6 +766,7 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
// messages stay valid for a strict provider replay (Anthropic
// rejects a tool_use with no answering tool_result).
messages = appendAbortedToolResults(messages, collected.ToolCalls[index+1:])
messages = append(messages, toolImageMessages...)
result.FinalAnswer = toolFailureStopAnswer(call.Name, outcome.Count)
result.Messages = copyMessages(messages)
return result, nil
@@ -765,6 +785,10 @@ func Run(ctx context.Context, prompt string, provider Provider, options Options)
postEditDiagnostics.enqueue(ctx, toolResult.ChangedFiles)
}
}
// Every tool_result for this turn is now recorded, including aborted
// placeholders, so the images can follow without splitting them.
messages = append(messages, toolImageMessages...)
toolImageMessages = nil
// Run post-edit self-correction once over the union of files this turn
// changed, then append any feedback after every tool_result is recorded so
@@ -1436,6 +1460,7 @@ func executeToolCall(ctx context.Context, registry *tools.Registry, call ToolCal
Output: result.Output,
Truncated: result.Truncated,
Meta: result.Meta,
Images: result.Images,
Redacted: result.Redacted,
ChangedFiles: result.ChangedFiles,
ChangeSummaries: result.ChangeSummaries,
@@ -3289,3 +3314,37 @@ func copyMessages(messages []Message) []Message {
}
return copied
}
// toolResultImageMessage builds the user message that carries a tool's images
// to the model, or reports false when the tool produced none.
//
// The text names the tool so the model can tell which call an image came from
// when several ran in one turn — the images arrive detached from their tool
// result, so nothing else associates them.
func toolResultImageMessage(result ToolResult) (zeroruntime.Message, bool) {
images := make([]zeroruntime.ImageBlock, 0, len(result.Images))
for _, image := range result.Images {
if len(image.Data) == 0 {
continue
}
mediaType := zeroruntime.NormalizeImageMediaType(image.MediaType)
if mediaType == "" {
// Outside the provider allow-list. Dropping it beats sending bytes a
// provider will reject and failing the whole turn.
continue
}
images = append(images, zeroruntime.ImageBlock{MediaType: mediaType, Data: image.Data})
}
if len(images) == 0 {
return zeroruntime.Message{}, false
}
label := result.Name
if label == "" {
label = "tool"
}
return zeroruntime.Message{
Role: zeroruntime.MessageRoleUser,
Content: "Image output from " + label + ":",
Images: images,
}, true
}
+207
View File
@@ -0,0 +1,207 @@
package agent
import (
"context"
"fmt"
"strings"
"testing"
"github.com/Gitlawb/zero/internal/tools"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
// imageTool returns a tool result carrying an image, the way a screenshot tool
// should be able to.
type imageTool struct {
media string
data []byte
}
func (imageTool) Name() string { return "capture" }
func (imageTool) Description() string { return "Captures an image." }
func (imageTool) Parameters() tools.Schema {
return tools.Schema{Type: "object", Properties: map[string]tools.PropertySchema{}}
}
func (imageTool) Safety() tools.Safety {
return tools.Safety{Permission: tools.PermissionAllow}
}
func (t imageTool) Run(context.Context, map[string]any) tools.Result {
return tools.Result{
Status: tools.StatusOK,
Output: "Captured a screenshot.",
Images: []zeroruntime.ImageBlock{{MediaType: t.media, Data: t.data}},
}
}
// A tool that produces an image must get that image in front of the model.
//
// It cannot ride the tool-result message: every provider drops images there.
// Anthropic maps a tool result to a tool_result block with string content,
// Gemini to a functionResponse, and OpenAI guards its image parts to the user
// role explicitly. So the image has to arrive as a following user message,
// which is also the only shape that keeps one tool result per tool call.
func TestRunDeliversToolResultImagesToTheModel(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(imageTool{media: "image/png", data: []byte("\x89PNG\r\n\x1a\nfake")})
provider := &mockProvider{
turns: [][]zeroruntime.StreamEvent{
{
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "call_1", ToolName: "capture"},
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "call_1", ArgumentsFragment: `{}`},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "call_1"},
{Type: zeroruntime.StreamEventDone},
},
{
{Type: zeroruntime.StreamEventText, Content: "I can see it."},
{Type: zeroruntime.StreamEventDone},
},
},
}
result, err := Run(context.Background(), "screenshot please", provider, Options{
Registry: registry,
MaxTurns: 2,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
var carrier *zeroruntime.Message
for index := range result.Messages {
if len(result.Messages[index].Images) > 0 {
carrier = &result.Messages[index]
break
}
}
if carrier == nil {
t.Fatal("no message carried the tool's image; the model never saw it")
}
if carrier.Role != zeroruntime.MessageRoleUser {
t.Errorf("image rode a %q message; every provider only serializes images on the user role", carrier.Role)
}
if got := carrier.Images[0].MediaType; got != "image/png" {
t.Errorf("media type = %q, want image/png", got)
}
// The tool-result pairing must be untouched: one tool result per tool call.
toolResults := 0
for _, message := range result.Messages {
if message.Role == zeroruntime.MessageRoleTool {
toolResults++
}
}
if toolResults != 1 {
t.Errorf("got %d tool-result messages for 1 tool call", toolResults)
}
}
// textTool is a second, image-free tool so a turn can carry two calls.
type textTool struct{}
func (textTool) Name() string { return "note" }
func (textTool) Description() string { return "Notes something." }
func (textTool) Parameters() tools.Schema {
return tools.Schema{Type: "object", Properties: map[string]tools.PropertySchema{}}
}
func (textTool) Safety() tools.Safety { return tools.Safety{Permission: tools.PermissionAllow} }
func (textTool) Run(context.Context, map[string]any) tools.Result {
return tools.Result{Status: tools.StatusOK, Output: "noted"}
}
// Tool results for one assistant turn must stay contiguous. The image messages
// are user-role, and a user message between two tool_results breaks strict
// provider replay: Anthropic coalesces consecutive user content into one block
// list and requires tool_result blocks first, so interleaving produces
// [tool_result, text, image, tool_result] and the request is rejected.
//
// The first version of this feature appended each image immediately after its
// own tool result, which is exactly that shape. A single-tool-call test passed
// and said nothing about it.
func TestRunKeepsToolResultsContiguousWhenAToolReturnsAnImage(t *testing.T) {
registry := tools.NewRegistry()
registry.Register(imageTool{media: "image/png", data: []byte("\x89PNG\r\n\x1a\nfake")})
registry.Register(textTool{})
provider := &mockProvider{turns: [][]zeroruntime.StreamEvent{
{
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "c1", ToolName: "capture"},
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "c1", ArgumentsFragment: `{}`},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "c1"},
{Type: zeroruntime.StreamEventToolCallStart, ToolCallID: "c2", ToolName: "note"},
{Type: zeroruntime.StreamEventToolCallDelta, ToolCallID: "c2", ArgumentsFragment: `{}`},
{Type: zeroruntime.StreamEventToolCallEnd, ToolCallID: "c2"},
{Type: zeroruntime.StreamEventDone},
},
{{Type: zeroruntime.StreamEventText, Content: "done"}, {Type: zeroruntime.StreamEventDone}},
}}
result, err := Run(context.Background(), "two calls", provider, Options{Registry: registry, MaxTurns: 2})
if err != nil {
t.Fatalf("Run: %v", err)
}
// Two separate ifs rather than a switch: a tool-role message that also
// carried images would match only the first arm of a switch, so a regression
// that put the image back on the tool result would read as "no image at all"
// instead of naming what actually happened.
type toolResult struct {
index int
id string
}
var toolResults []toolResult
imageIndex := -1
for index, message := range result.Messages {
if message.Role == zeroruntime.MessageRoleTool {
toolResults = append(toolResults, toolResult{index: index, id: message.ToolCallID})
}
if len(message.Images) > 0 {
if imageIndex >= 0 {
t.Fatalf("image delivered twice, at %d and %d; recorded %s", imageIndex, index, messageShape(result.Messages))
}
imageIndex = index
}
}
if imageIndex < 0 {
t.Fatalf("the image never reached the model; recorded %s", messageShape(result.Messages))
}
if len(toolResults) != 2 {
t.Fatalf("got %d tool-result messages for 2 tool calls; recorded %s", len(toolResults), messageShape(result.Messages))
}
if toolResults[0].id != "c1" || toolResults[1].id != "c2" {
t.Errorf("tool results answered %q then %q, want c1 then c2", toolResults[0].id, toolResults[1].id)
}
// The contract Anthropic enforces: every tool_result block has to come before
// any other block in the user message it lands in. Anything wedged between
// the two results breaks that, which is the 400 this PR exists to avoid.
if toolResults[1].index != toolResults[0].index+1 {
t.Errorf("tool results at %d and %d are not adjacent; %q message at %d splits them; recorded %s",
toolResults[0].index, toolResults[1].index, result.Messages[toolResults[0].index+1].Role,
toolResults[0].index+1, messageShape(result.Messages))
}
// Follows the last tool result, deliberately NOT pinned to exactly last+1.
// appendUserBlocks coalesces consecutive user-role content, so
// [tool_result, tool_result, text, image] still maps to tool_result blocks
// first and replays fine; pinning would redden that valid shape for nothing.
if imageIndex <= toolResults[1].index {
t.Errorf("image message at %d does not follow the last tool result at %d; recorded %s",
imageIndex, toolResults[1].index, messageShape(result.Messages))
}
}
// messageShape renders the conversation as a compact role list so a failure says
// what the ordering actually was instead of dumping whole messages.
func messageShape(messages []zeroruntime.Message) string {
parts := make([]string, 0, len(messages))
for index, message := range messages {
part := fmt.Sprintf("%d:%s", index, message.Role)
if message.ToolCallID != "" {
part += "(" + message.ToolCallID + ")"
}
if len(message.Images) > 0 {
part += "+image"
}
parts = append(parts, part)
}
return "[" + strings.Join(parts, " ") + "]"
}
+5 -2
View File
@@ -68,8 +68,11 @@ type ToolResult struct {
Output string
// Truncated reports that the tool's model-visible output omitted content.
// The full result may be recoverable through Meta["spill_path"].
Truncated bool
Meta map[string]string
Truncated bool
Meta map[string]string
// Images the tool produced, delivered to the model as a following user
// message rather than on this result. See tools.Result.Images.
Images []zeroruntime.ImageBlock
Redacted bool
ChangedFiles []string
// ChangeSummaries are non-selectable generated-tree summaries emitted by
+7 -4
View File
@@ -167,10 +167,13 @@ const permissionModeUnsafe = "unsafe"
var readOnlySpecialistTools = map[string]bool{
"read_file": true,
"read_minified_file": true,
"list_directory": true,
"grep": true,
"glob": true,
"update_plan": true,
// Reads one image file through read_file's own path scoping and mutates
// nothing, so it carries the same auto-approval as the other pure readers.
"view_image": true,
"list_directory": true,
"grep": true,
"glob": true,
"update_plan": true,
}
// IsReadOnlySpecialist reports whether the named specialist resolves to a
+1 -1
View File
@@ -255,7 +255,7 @@ func TestBuildArgsDefaultsToReadOnlyToolAllowlist(t *testing.T) {
if err != nil {
t.Fatalf("BuildArgs returned error: %v", err)
}
if !containsSequence(result.Args, []string{"--enabled-tools", "glob,grep,list_directory,read_file,read_minified_file"}) {
if !containsSequence(result.Args, []string{"--enabled-tools", "glob,grep,list_directory,read_file,read_minified_file,view_image"}) {
t.Fatalf("args missing default read-only allowlist: %#v", result.Args)
}
}
+4 -3
View File
@@ -82,9 +82,9 @@ var knownMetadataKeys = map[string]bool{
}
var toolCategories = map[string][]string{
"read-only": {"read_file", "read_minified_file", "list_directory", "grep", "glob"},
"edit": {"read_file", "read_minified_file", "list_directory", "grep", "glob", "write_file", "edit_file", "apply_patch"},
"execute": {"read_file", "read_minified_file", "list_directory", "grep", "glob", "exec_command", "write_stdin", "bash"},
"read-only": {"read_file", "read_minified_file", "view_image", "list_directory", "grep", "glob"},
"edit": {"read_file", "read_minified_file", "view_image", "list_directory", "grep", "glob", "write_file", "edit_file", "apply_patch"},
"execute": {"read_file", "read_minified_file", "view_image", "list_directory", "grep", "glob", "exec_command", "write_stdin", "bash"},
"plan": {"update_plan"},
}
@@ -102,6 +102,7 @@ var defaultToolSelection = []string{"read-only"}
var knownToolNames = map[string]bool{
"read_file": true,
"read_minified_file": true,
"view_image": true,
"list_directory": true,
"glob": true,
"grep": true,
@@ -0,0 +1,31 @@
package specialist
import (
"slices"
"testing"
)
// view_image is a pure workspace-scoped read, so every category that already
// grants read_file should grant it too. Categories are what specialist authors
// actually write ("tools: [execute]"), so an omission here silently denies a
// specialist the ability to look at a screenshot it just captured.
func TestToolCategoriesIncludeViewImageWhereverReadFileIsGranted(t *testing.T) {
for category, names := range toolCategories {
if !slices.Contains(names, "read_file") {
continue
}
if !slices.Contains(names, "view_image") {
t.Errorf("category %q grants read_file but not view_image", category)
}
}
}
func TestResolveToolsExpandsExecuteToIncludeViewImage(t *testing.T) {
resolved, err := ResolveTools([]string{"execute"})
if err != nil {
t.Fatalf("ResolveTools: %v", err)
}
if !slices.Contains(resolved, "view_image") {
t.Errorf("execute resolved to %v, missing view_image", resolved)
}
}
+11
View File
@@ -4,6 +4,8 @@ import (
"context"
"encoding/json"
"fmt"
"github.com/Gitlawb/zero/internal/imageinput"
"github.com/Gitlawb/zero/internal/zeroruntime"
"os"
"path/filepath"
"regexp"
@@ -345,9 +347,18 @@ func captureOKResult(driver string, request captureArtifactRequest, path string,
if helperOutput != "" {
output += "\n\n" + helperOutput
}
// Attach the artifact itself. Without this the model is told a screenshot
// exists and cannot look at it, which is the whole point of asking for one.
// A PDF or an unreadable file simply yields no image and the text stands on
// its own, so a capture is never failed over the attachment.
var images []zeroruntime.ImageBlock
if image, err := imageinput.LoadFile(path, ""); err == nil {
images = []zeroruntime.ImageBlock{image}
}
return Result{
Status: StatusOK,
Output: output,
Images: images,
Meta: map[string]string{
"artifact_path": path,
"action": request.action,
+3
View File
@@ -299,6 +299,9 @@ func CoreReadOnlyToolsScoped(workspaceRoot string, scope PathScope) []Tool {
return []Tool{
NewScopedReadFileTool(workspaceRoot, scope),
NewScopedReadMinifiedFileTool(workspaceRoot, scope),
// view_image is read-only and shares read_file's path scoping, so it
// belongs with the other readers rather than behind a gate.
NewScopedViewImageTool(workspaceRoot, scope),
NewScopedListDirectoryTool(workspaceRoot, scope),
NewScopedGlobTool(workspaceRoot, scope),
NewScopedGrepTool(workspaceRoot, scope),
+7 -2
View File
@@ -30,8 +30,8 @@ func tempDirOutsideDefaultTemp(t *testing.T) string {
func TestCoreReadOnlyToolsExposeSafeMetadata(t *testing.T) {
toolset := CoreReadOnlyToolsScoped(t.TempDir(), nil)
if len(toolset) != 9 {
t.Fatalf("expected 9 core read-only tools, got %d", len(toolset))
if len(toolset) != 10 {
t.Fatalf("expected 10 core read-only tools, got %d", len(toolset))
}
seen := map[string]bool{}
@@ -71,6 +71,11 @@ func TestCoreReadOnlyToolsExposeSafeMetadata(t *testing.T) {
if !seen[RequestPermissionsToolName] {
t.Fatalf("expected %s in core read-only tools", RequestPermissionsToolName)
}
// By name as well as by count: the count alone still passes if one reader is
// swapped for another.
if !seen[ViewImageToolName] {
t.Fatalf("expected %s in core read-only tools", ViewImageToolName)
}
}
func TestRegistryAllReturnsToolsSortedByName(t *testing.T) {
+11
View File
@@ -5,6 +5,7 @@ import (
"github.com/Gitlawb/zero/internal/execution"
"github.com/Gitlawb/zero/internal/sandbox"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
type SideEffect string
@@ -102,6 +103,16 @@ type Result struct {
// while callers migrate away from parsing text and string metadata.
ExecutionRequest *execution.Request `json:"-"`
ExecutionOutcome *execution.Outcome `json:"-"`
// Images are pictures the tool is handing the model — a screenshot it just
// captured, a file it was asked to look at. Text-only tools leave it nil.
//
// They are NOT delivered on this result's own tool-result message. Every
// provider drops images there: Anthropic maps a tool result to a tool_result
// block whose content is a string, Gemini to a functionResponse, and OpenAI
// guards its image content-parts to the user role. The agent loop therefore
// emits them as a following user message, which is also the only shape that
// keeps one tool result per tool call.
Images []zeroruntime.ImageBlock `json:"-"`
// Redacted is set when secret scrubbing altered Output before it left the
// tool-execution boundary.
Redacted bool
+80
View File
@@ -0,0 +1,80 @@
package tools
import (
"context"
"github.com/Gitlawb/zero/internal/imageinput"
"github.com/Gitlawb/zero/internal/zeroruntime"
)
// ViewImageToolName is the canonical registry name of the image viewer.
const ViewImageToolName = "view_image"
// NewViewImageTool builds the workspace-scoped image viewer.
func NewViewImageTool(workspaceRoot string) Tool {
return NewScopedViewImageTool(workspaceRoot, nil)
}
// NewScopedViewImageTool builds the image viewer bound to a path scope.
//
// The scope is not optional decoration. imageinput.LoadFile resolves a relative
// path by joining it to the workspace root and otherwise takes the path as
// given, so on its own it would happily read outside the workspace. Every path
// goes through resolveScopedReadPath first, exactly like read_file, so this tool
// grants no reach that read_file does not already have.
func NewScopedViewImageTool(workspaceRoot string, scope PathScope) Tool {
return viewImageTool{
baseTool: baseTool{
name: ViewImageToolName,
description: "Look at an image file. Use this to see a screenshot, diagram, mockup or photo " +
"the user mentioned but did not attach, or one a previous tool wrote to disk. " +
"Accepts PNG, JPEG, GIF and WebP up to 10 MiB.",
parameters: Schema{
Type: "object",
Properties: map[string]PropertySchema{
"path": {Type: "string", Description: "Path of the image to look at."},
},
Required: []string{"path"},
AdditionalProperties: false,
},
safety: readOnlySafety("Reads an image file without modifying anything."),
// Same shape as read_file: reads one file, mutates nothing, and is
// safe to run concurrently.
capabilities: ToolCapabilities{Effect: EffectReadOnly, ThreadSafe: true, ResourceKeys: fileResourceKeys},
},
workspaceRoot: workspaceRoot,
scope: scope,
}
}
type viewImageTool struct {
baseTool
workspaceRoot string
scope PathScope
}
func (tool viewImageTool) Run(_ context.Context, args map[string]any) Result {
requestedPath, err := aliasedStringArg(args, []string{"path", "file", "file_path", "filepath", "filename", "image"}, "", true, false)
if err != nil {
return errorResult("Error: Invalid arguments for " + ViewImageToolName + ": " + err.Error())
}
absolutePath, relativePath, err := resolveScopedReadPath(tool.workspaceRoot, tool.scope, requestedPath)
if err != nil {
return errorResult("Error: " + err.Error())
}
image, err := imageinput.LoadFile(absolutePath, tool.workspaceRoot)
if err != nil {
// LoadFile's errors already name the problem (missing, too large,
// unsupported type) and are safe to show: they carry the path the caller
// asked for, which it already knows.
return errorResult("Error viewing " + relativePath + ": " + err.Error())
}
return Result{
Status: StatusOK,
// The text says what arrived; the image itself reaches the model through
// the agent loop, which delivers Images on a following user message.
Output: "Viewing " + relativePath + " (" + image.MediaType + ").",
Images: []zeroruntime.ImageBlock{image},
Meta: map[string]string{"media_type": image.MediaType},
}
}
+100
View File
@@ -0,0 +1,100 @@
package tools
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
)
// A 1x1 PNG. Small enough to inline, real enough to pass media-type sniffing.
var onePixelPNG = []byte{
0x89, 'P', 'N', 'G', 0x0d, 0x0a, 0x1a, 0x0a,
0x00, 0x00, 0x00, 0x0d, 'I', 'H', 'D', 'R',
0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
0x08, 0x06, 0x00, 0x00, 0x00, 0x1f, 0x15, 0xc4,
0x89, 0x00, 0x00, 0x00, 0x0a, 'I', 'D', 'A', 'T',
0x78, 0x9c, 0x63, 0x00, 0x01, 0x00, 0x00, 0x05,
0x00, 0x01, 0x0d, 0x0a, 0x2d, 0xb4, 0x00, 0x00,
0x00, 0x00, 'I', 'E', 'N', 'D', 0xae, 0x42, 0x60, 0x82,
}
func TestViewImageReturnsTheImage(t *testing.T) {
root := t.TempDir()
if err := os.WriteFile(filepath.Join(root, "shot.png"), onePixelPNG, 0o600); err != nil {
t.Fatal(err)
}
result := NewViewImageTool(root).Run(context.Background(), map[string]any{"path": "shot.png"})
if result.Status != StatusOK {
t.Fatalf("status = %v: %s", result.Status, result.Output)
}
if len(result.Images) != 1 {
t.Fatalf("got %d images, want 1 — the text alone is useless to the model", len(result.Images))
}
if result.Images[0].MediaType != "image/png" {
t.Errorf("media type = %q, want image/png", result.Images[0].MediaType)
}
if len(result.Images[0].Data) != len(onePixelPNG) {
t.Errorf("image data is %d bytes, want %d", len(result.Images[0].Data), len(onePixelPNG))
}
}
// The tool must not reach outside the workspace. imageinput.LoadFile does no
// containment check of its own, so the scoping is entirely this tool's job.
func TestViewImageRefusesPathsOutsideTheWorkspace(t *testing.T) {
root := t.TempDir()
outside := t.TempDir()
secret := filepath.Join(outside, "outside.png")
if err := os.WriteFile(secret, onePixelPNG, 0o600); err != nil {
t.Fatal(err)
}
tool := NewViewImageTool(root)
for name, path := range map[string]string{
"absolute path outside": secret,
"parent traversal": filepath.Join("..", filepath.Base(outside), "outside.png"),
} {
t.Run(name, func(t *testing.T) {
result := tool.Run(context.Background(), map[string]any{"path": path})
if result.Status == StatusOK {
t.Fatalf("read an image outside the workspace: %s", result.Output)
}
if len(result.Images) != 0 {
t.Error("a refused read still returned image bytes")
}
})
}
}
func TestViewImageRejectsNonImages(t *testing.T) {
root := t.TempDir()
if err := os.WriteFile(filepath.Join(root, "notes.txt"), []byte("not an image"), 0o600); err != nil {
t.Fatal(err)
}
result := NewViewImageTool(root).Run(context.Background(), map[string]any{"path": "notes.txt"})
if result.Status == StatusOK {
t.Fatalf("accepted a text file as an image: %s", result.Output)
}
if len(result.Images) != 0 {
t.Error("a rejected file still returned image bytes")
}
}
func TestViewImageReportsAMissingFileClearly(t *testing.T) {
result := NewViewImageTool(t.TempDir()).Run(context.Background(), map[string]any{"path": "nope.png"})
if result.Status == StatusOK {
t.Fatal("a missing file reported success")
}
// The error names the path the caller asked for, matching read_file. Both
// also carry the resolved absolute path from the underlying syscall error;
// that is pre-existing behaviour shared with read_file, not something this
// tool introduces, so it is not asserted either way here.
if !strings.Contains(result.Output, "nope.png") {
t.Errorf("error should name the file, got %q", result.Output)
}
if len(result.Images) != 0 {
t.Error("a failed read still returned image bytes")
}
}