anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-23 02:36:57 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

[PR #4311] feat: add vision support for custom providers #10938

New Issue
Closed
opened 2026-02-16 18:15:42 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 18:15:42 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/anomalyco/opencode/pull/4311

State: closed
Merged: No

Problem Statement

OpenCode does not support vision capabilities for custom providers, even when they are compatible with OpenAI's vision API format. This affects users who want to use local or self-hosted vision models (like Qwen-VL, LLaVA, etc.) through OpenCode CLI.

Root Cause

Two bugs prevented custom providers from using vision:

  1. Read Tool Bug: The Read tool only checked modalities metadata (from models.dev) for vision support, ignoring the user-configured attachment flag in custom provider configs
  2. Message Format Bug: Image attachments were sent as generic "file" type instead of "image" type, which AI SDK requires for vision

Solution

1. Fix Read Tool - Allow images based on attachment flag

File: packages/opencode/src/tool/read.ts

When providerID/modelID context isn't available (e.g., -f flag usage), allow images through and let AI SDK handle validation. Also check the attachment flag as fallback when modalities metadata isn't available.

2. Fix Message Builder - Convert images to proper format

File: packages/opencode/src/session/message-v2.ts

Convert image attachments to AI SDK's "image" type instead of generic "file" type for both user messages and tool-returned attachments.

Benefits

  1. Generic Solution: Works with any OpenAI-compatible vision API (local vLLM, LM Studio, Ollama, etc.)
  2. No Breaking Changes: Existing functionality unchanged; only enables new capabilities
  3. Follows OpenCode Patterns: Uses existing attachment flag, similar to how official models declare vision support
  4. User-Friendly: Simple config change enables vision: 
    {
  "models": {
    "my-vision-model": {
      "attachment": true,
      ...
    }
  }
}

Testing

Test Configuration Example

{
  "provider": {
    "local-vision": {
      "npm": "@ai-sdk/openai",
      "api": "http://localhost:8000/v1",
      "models": {
        "vision-model": {
          "attachment": true,
          "limit": {
            "context": 32768,
            "output": 2048
          }
        }
      }
    }
  }
}

Test Command

opencode run "Describe this image" -f /path/to/image.png -m local-vision/vision-model

Files Modified

  1. packages/opencode/src/tool/read.ts - Lines 72-86
  2. packages/opencode/src/session/message-v2.ts - Lines 544-563, 599-614

Compatibility

  • ✅ Works with @ai-sdk/openai
  • ✅ Works with @ai-sdk/openai-compatible
  • ✅ Backward compatible (no breaking changes)
  • ✅ Works with official OpenCode providers (unchanged behavior)
**Original Pull Request:** https://github.com/anomalyco/opencode/pull/4311 **State:** closed **Merged:** No --- ## Problem Statement OpenCode does not support vision capabilities for custom providers, even when they are compatible with OpenAI's vision API format. This affects users who want to use local or self-hosted vision models (like Qwen-VL, LLaVA, etc.) through OpenCode CLI. ## Root Cause Two bugs prevented custom providers from using vision: 1. **Read Tool Bug**: The Read tool only checked `modalities` metadata (from models.dev) for vision support, ignoring the user-configured `attachment` flag in custom provider configs 2. **Message Format Bug**: Image attachments were sent as generic "file" type instead of "image" type, which AI SDK requires for vision ## Solution ### 1. Fix Read Tool - Allow images based on `attachment` flag **File**: `packages/opencode/src/tool/read.ts` When `providerID`/`modelID` context isn't available (e.g., `-f` flag usage), allow images through and let AI SDK handle validation. Also check the `attachment` flag as fallback when `modalities` metadata isn't available. ### 2. Fix Message Builder - Convert images to proper format **File**: `packages/opencode/src/session/message-v2.ts` Convert image attachments to AI SDK's "image" type instead of generic "file" type for both user messages and tool-returned attachments. ## Benefits 1. **Generic Solution**: Works with any OpenAI-compatible vision API (local vLLM, LM Studio, Ollama, etc.) 2. **No Breaking Changes**: Existing functionality unchanged; only enables new capabilities 3. **Follows OpenCode Patterns**: Uses existing `attachment` flag, similar to how official models declare vision support 4. **User-Friendly**: Simple config change enables vision: ```json { "models": { "my-vision-model": { "attachment": true, ... } } } ``` ## Testing ### Test Configuration Example ```json { "provider": { "local-vision": { "npm": "@ai-sdk/openai", "api": "http://localhost:8000/v1", "models": { "vision-model": { "attachment": true, "limit": { "context": 32768, "output": 2048 } } } } } } ``` ### Test Command ```bash opencode run "Describe this image" -f /path/to/image.png -m local-vision/vision-model ``` ## Files Modified 1. `packages/opencode/src/tool/read.ts` - Lines 72-86 2. `packages/opencode/src/session/message-v2.ts` - Lines 544-563, 599-614 ## Compatibility - ✅ Works with @ai-sdk/openai - ✅ Works with @ai-sdk/openai-compatible - ✅ Backward compatible (no breaking changes) - ✅ Works with official OpenCode providers (unchanged behavior)
yindo added the pull-request label 2026-02-16 18:15:42 -05:00
yindo closed this issue 2026-02-16 18:15:42 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#10938
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?