anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 18:25:32 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

Add metadata awareness and data type validation/blacklists for LLM tool execution #7725

New Issue
Open
opened 2026-02-16 18:08:03 -05:00 by yindo · 2 comments
yindo commented 2026-02-16 18:08:03 -05:00
Owner
Copy Link
Copy Source

Originally created by @davidbernat on GitHub (Jan 27, 2026).

Originally assigned to: @thdxr on GitHub.

Description

What are we missing here about reliability standards in this industry? Thanks.
This is a blocker before adopting LLMs for client use re: maintenance of these problems goes exponential.

Summary

OpenCode currently lacks awareness of LLM metadata (names, limitations, supported data types) and doesn't validate tool inputs/outputs against LLM capabilities. This causes session-breaking failures when tools return unsupported data types, particularly for binary data like images.

Problem Description

1. No Metadata Awareness

  • OpenCode doesn't know LLM names (e.g., "BigPickle")
  • OpenCode isn't aware of LLM data type limitations (i.e., BigPickle will interpret base64 encoded image data as text prompts, and begin spooling out similar gibberish)
  • No capability matching between tools and LLMs (i.e., LLMs cannot be informed to manually override its choice of a catastrophic tool)

2. No Data Type Validation

  • Tools can execute without checking if LLM can handle the output
  • No validation that an LLM can handle tool return types prior to execution
  • e.g. Binary data can be passed to text-only LLMs

3. Session-Breaking Failures

  • Binary image data causes catastrophic failures
  • Sessions end abruptly and require restart
  • No graceful degradation or error handling

4. No Manual Control of Limitations

  • MCP method blacklist to disallow breaking methods on third-party MCPs
  • (Advanced) No hooks for manual processing of data (i.e., catch an MCP returning bytes data with a bespoke handler to write to disk, return locla filename instead)

Use Cases & Examples

Playwright MCP Screenshot Scenario

  1. User requests: "Take a screenshot of this URL"
  2. OpenCode correctly identifies and executes Playwright MCP screenshot method
  3. MCP returns binary image data
  4. BigPickle receives binary data, treats it as text/JSON prompt
  5. Session crashes and requires restart, smoke emerges from computer, lightning crashes in distance, evil released from the depths Barad-dûr (typical)

OCR MCP Server Scenario

  1. User wants to OCR an image using MCP server
  2. API expects base64 encoded data URIs
  3. LLM cannot handle binary image data directly
  4. OpenCode cannot be provided a hook to execute beforehand; to modify MCP interface to accept filename
  5. Requires custom self-owned wrapper to handle file conversion; cannot "turn off" dangerous MCP methods

Proposed Solutions

Solution 1: Metadata Awareness

{
  "llm_metadata": {
    "name": "BigPickle",
    "capabilities": ["text/plain", "text/json"],
    // simple
    "tool_blacklists": {
        "playwright": ["browser_take_screenshot"]
    }
    // sensible, though will get numerous quickly
    "tool_replacements": {
        "playwright": {
            "browser_take_screenshot": "my_alternative_version"
        }
    }
    // advanced
    "tool_handlers": {
        "playwright": {
            "browser_take_screenshot": "my_handlers.py:ClassName"
        }    
    }
    "max_tokens": 50000
  }
}

Solution 2: Pre-Execution Validation

  • Check tool return types against LLM capabilities
  • Block incompatible tool executions (from being selected)
  • Provide user-friendly error messages (if no other solution)
    • "[Thinking]The Playwright MCP has a browser-take-screenshot method but BigPickle cannot use its image data return value.[/Thinking] It looks like Playwright MCP is supposed to handle this situation but as BigPickle I lack this functionality. Would you like to switch to a different LLM or should we write session code to handle these datatypes manually?"

Solution 3: Automatic Wrapper Generation

  • Detect incompatible tool/LLM combinations
  • OpenCode create wrapper methods automatically
  • Handle data type conversions transparently

Current Workarounds

Users must build and support custom MCP wrappers that:

  1. Execute the original MCP method
  2. Handle incompatible data types (save to disk, convert, etc.)
  3. Return only JSON-compatible results
  4. Clean up temporary files

This overhead is because of the limitation of the MCP Protocol and the lack of transparency into the OpenCode tool use, which makes this particularly suitable for resolution for the OpenCode company.

Impact & Rationale

Why This Matters

  • Session Reliability: Prevents catastrophic failures
  • User Experience: Eliminates confusing crashes
  • Developer Productivity: No need for high volume of monkeypatch wrappers, which must be supported for each MCP method, by each developer.
  • Company Growth: Makes OpenCode more robust, and in line with consumer expectations.

Additional Context

This issue represents a fundamental gap in LLM tool orchestration. While OpenCode correctly identifies and executes tools, it lacks the awareness to ensure tool outputs are compatible with the executing LLM. The current approach puts the burden on users to create custom workarounds for what should be a platform-level concern.

The fact that this can be solved with relatively simple metadata awareness and validation rules, against an infinite variety of monkeypatches supported by third-party users, makes it an ideal candidate for immediate implementation.

Thanks for handling the request this text is just lax writing as my brain is up with the sun here and still groggy. Using OpenCode v1.1.36.

Originally created by @davidbernat on GitHub (Jan 27, 2026). Originally assigned to: @thdxr on GitHub. ### Description What are we missing here about reliability standards in this industry? Thanks. This is a blocker before adopting LLMs for client use re: maintenance of these problems goes exponential. ## Summary OpenCode currently lacks awareness of LLM metadata (names, limitations, supported data types) and doesn't validate tool inputs/outputs against LLM capabilities. This causes session-breaking failures when tools return unsupported data types, particularly for binary data like images. ## Problem Description ### 1. No Metadata Awareness - OpenCode doesn't know LLM names (e.g., "BigPickle") - OpenCode isn't aware of LLM data type limitations (i.e., BigPickle will interpret base64 encoded image data as text prompts, and begin spooling out similar gibberish) - No capability matching between tools and LLMs (i.e., LLMs cannot be informed to manually override its choice of a catastrophic tool) ### 2. No Data Type Validation - Tools can execute without checking if LLM can handle the output - No validation that an LLM can handle tool return types prior to execution - e.g. Binary data can be passed to text-only LLMs ### 3. Session-Breaking Failures - Binary image data causes catastrophic failures - Sessions end abruptly and require restart - No graceful degradation or error handling ### 4. No Manual Control of Limitations - MCP method blacklist to disallow breaking methods on third-party MCPs - (Advanced) No hooks for manual processing of data (i.e., catch an MCP returning bytes data with a bespoke handler to write to disk, return locla filename instead) ## Use Cases & Examples ### Playwright MCP Screenshot Scenario 1. User requests: "Take a screenshot of this URL" 2. OpenCode correctly identifies and executes Playwright MCP screenshot method 3. MCP returns binary image data 4. BigPickle receives binary data, treats it as text/JSON prompt 5. Session crashes and requires restart, smoke emerges from computer, lightning crashes in distance, evil released from the depths Barad-dûr (typical) ### OCR MCP Server Scenario 1. User wants to OCR an image using MCP server 2. API expects base64 encoded data URIs 3. LLM cannot handle binary image data directly 4. OpenCode cannot be provided a hook to execute beforehand; to modify MCP interface to accept filename 4. Requires custom self-owned wrapper to handle file conversion; cannot "turn off" dangerous MCP methods ## Proposed Solutions ### Solution 1: Metadata Awareness ```json { "llm_metadata": { "name": "BigPickle", "capabilities": ["text/plain", "text/json"], // simple "tool_blacklists": { "playwright": ["browser_take_screenshot"] } // sensible, though will get numerous quickly "tool_replacements": { "playwright": { "browser_take_screenshot": "my_alternative_version" } } // advanced "tool_handlers": { "playwright": { "browser_take_screenshot": "my_handlers.py:ClassName" } } "max_tokens": 50000 } } ``` ### Solution 2: Pre-Execution Validation - Check tool return types against LLM capabilities - Block incompatible tool executions (from being selected) - Provide user-friendly error messages (if no other solution) - "[Thinking]The Playwright MCP has a browser-take-screenshot method but BigPickle cannot use its image data return value.[/Thinking] It looks like Playwright MCP is supposed to handle this situation but as BigPickle I lack this functionality. Would you like to switch to a different LLM or should we write session code to handle these datatypes manually?" ### Solution 3: Automatic Wrapper Generation - Detect incompatible tool/LLM combinations - OpenCode create wrapper methods automatically - Handle data type conversions transparently ## Current Workarounds Users must build and support custom MCP wrappers that: 1. Execute the original MCP method 2. Handle incompatible data types (save to disk, convert, etc.) 3. Return only JSON-compatible results 4. Clean up temporary files **This overhead is because of the limitation of the MCP Protocol and the lack of transparency into the OpenCode tool use, which makes this particularly suitable for resolution for the OpenCode company.** ## Impact & Rationale ### Why This Matters - **Session Reliability**: Prevents catastrophic failures - **User Experience**: Eliminates confusing crashes - **Developer Productivity**: No need for high volume of monkeypatch wrappers, which must be supported for each MCP method, by each developer. - **Company Growth**: Makes OpenCode more robust, and in line with consumer expectations. ### Additional Context This issue represents a fundamental gap in LLM tool orchestration. While OpenCode correctly identifies and executes tools, it lacks the awareness to ensure tool outputs are compatible with the executing LLM. The current approach puts the burden on users to create custom workarounds for what should be a platform-level concern. The fact that this can be solved with relatively simple metadata awareness and validation rules, against an infinite variety of monkeypatches supported by third-party users, makes it an ideal candidate for immediate implementation. Thanks for handling the request this text is just lax writing as my brain is up with the sun here and still groggy. Using OpenCode v1.1.36.
yindo added the bug label 2026-02-16 18:08:03 -05:00
yindo commented 2026-02-16 18:08:04 -05:00
Author
Owner
Copy Link
Copy Source

@github-actions[bot] commented on GitHub (Jan 27, 2026):

This issue might be a duplicate of existing issues or closely related to them. Please check:

  • #10730: ENOENT when image URL is treated as local file path — relates to data validation and binary data handling failures
  • #10667: OpenCode Validation Fails on AI SDK V3 Stream Parts — addresses data type validation for tool outputs
  • #10306: Screenshots from Chrome DevTools MCP are larger than GitHub Copilot model context window — tool output (binary images) exceeding LLM capabilities
  • #10150: Azure gpt-5.2-codex unable to read screenshots — image data incompatibility with LLM capabilities
  • #10802: Parent session appears stuck "loading" when subagent is blocked — session-breaking failures related to tool execution
  • #10813: todowrite tool call parameter error — tool input validation issues
  • #10803: Invalid tool role value error — tool-level data type validation
  • #10412: Tool calls fail with all models (Abacus AI provider) — provider/LLM/tool compatibility issues

These issues collectively address the problems described in this feature request: data type validation, LLM capability awareness, and session reliability during tool execution. They may benefit from being resolved as part of the same metadata awareness and validation system.

Feel free to ignore if your specific case differs from these existing issues.

@github-actions[bot] commented on GitHub (Jan 27, 2026): This issue might be a duplicate of existing issues or closely related to them. Please check: - #10730: ENOENT when image URL is treated as local file path — relates to data validation and binary data handling failures - #10667: OpenCode Validation Fails on AI SDK V3 Stream Parts — addresses data type validation for tool outputs - #10306: Screenshots from Chrome DevTools MCP are larger than GitHub Copilot model context window — tool output (binary images) exceeding LLM capabilities - #10150: Azure gpt-5.2-codex unable to read screenshots — image data incompatibility with LLM capabilities - #10802: Parent session appears stuck "loading" when subagent is blocked — session-breaking failures related to tool execution - #10813: todowrite tool call parameter error — tool input validation issues - #10803: Invalid tool role value error — tool-level data type validation - #10412: Tool calls fail with all models (Abacus AI provider) — provider/LLM/tool compatibility issues These issues collectively address the problems described in this feature request: data type validation, LLM capability awareness, and session reliability during tool execution. They may benefit from being resolved as part of the same metadata awareness and validation system. Feel free to ignore if your specific case differs from these existing issues.
yindo commented 2026-02-16 18:08:04 -05:00
Author
Owner
Copy Link
Copy Source

@davidbernat commented on GitHub (Feb 12, 2026):

@thdxr Is there any movement on this? This capacity for an agent to collapse itself is still a problematic universal use case.

@davidbernat commented on GitHub (Feb 12, 2026): @thdxr Is there any movement on this? This capacity for an agent to collapse itself is still a problematic universal use case.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#7725
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?