anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 10:15:31 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

[Feature] Add Fetch tool that saves to file (like Claude Code) #4341

New Issue
Closed
opened 2026-02-16 17:43:30 -05:00 by yindo · 7 comments
yindo commented 2026-02-16 17:43:30 -05:00
Owner
Copy Link
Copy Source

Originally created by @jdillon on GitHub (Jan 7, 2026).

Originally assigned to: @rekram1-node on GitHub.

Problem

The built-in webfetch tool returns entire page content directly into conversation context. For documentation pages (TanStack, MDN, React docs, etc.), this frequently exceeds the 200k token limit:

prompt is too long: 207654 tokens > 200000 maximum

This happens within a single tool response - the page itself is too large. Mid-turn compaction (#6480) can't help because overflow occurs before compaction can trigger.

Proposed Solution

Add a Fetch tool similar to Claude Code's implementation:

  1. Downloads content to a temp file (e.g., .opencode/tmp/fetch-{hash}.md)
  2. Returns only metadata to context:
    • File path
    • Title
    • Content length
    • First ~500 chars as preview
  3. Agent uses read tool to access content in chunks as needed

Claude Code's approach:

  • Fetch downloads to temp, returns path + summary
  • Agent reads sections as needed using standard file tools
  • Large pages never blow context because content lives on disk

Benefits

  • Any page size works - content is on disk, not in context
  • Agent reads selectively - only loads what's relevant
  • Consistent with existing read tool patterns for large files
  • Solves a class of problems, not just individual pages

Workaround Question

Is there an existing custom tool or plugin that implements this pattern?

I see custom tools can be created in .opencode/tool/ - has anyone published a fetch-to-file tool that could serve as a workaround until this is built-in? If not, is there a recommended approach for implementing this as a custom tool?

Related Issues

  • #4845 (prompt too long)
  • #5360, #5478 (duplicates)
  • #6480 (mid-turn compaction - helps multi-turn but not single large fetches)
Originally created by @jdillon on GitHub (Jan 7, 2026). Originally assigned to: @rekram1-node on GitHub. ### Problem The built-in `webfetch` tool returns entire page content directly into conversation context. For documentation pages (TanStack, MDN, React docs, etc.), this frequently exceeds the 200k token limit: ``` prompt is too long: 207654 tokens > 200000 maximum ``` This happens within a single tool response - the page itself is too large. Mid-turn compaction (#6480) can't help because overflow occurs before compaction can trigger. ### Proposed Solution Add a `Fetch` tool similar to Claude Code's implementation: 1. **Downloads content to a temp file** (e.g., `.opencode/tmp/fetch-{hash}.md`) 2. **Returns only metadata to context:** - File path - Title - Content length - First ~500 chars as preview 3. **Agent uses `read` tool** to access content in chunks as needed Claude Code's approach: - Fetch downloads to temp, returns path + summary - Agent reads sections as needed using standard file tools - Large pages never blow context because content lives on disk ### Benefits - Any page size works - content is on disk, not in context - Agent reads selectively - only loads what's relevant - Consistent with existing `read` tool patterns for large files - Solves a class of problems, not just individual pages ### Workaround Question Is there an existing custom tool or plugin that implements this pattern? I see custom tools can be created in `.opencode/tool/` - has anyone published a `fetch-to-file` tool that could serve as a workaround until this is built-in? If not, is there a recommended approach for implementing this as a custom tool? ### Related Issues - #4845 (prompt too long) - #5360, #5478 (duplicates) - #6480 (mid-turn compaction - helps multi-turn but not single large fetches)
yindo closed this issue 2026-02-16 17:43:30 -05:00
yindo commented 2026-02-16 17:43:30 -05:00
Author
Owner
Copy Link
Copy Source

@github-actions[bot] commented on GitHub (Jan 7, 2026):

This issue mentions several related issues that address the same core problem of context/token overflow:

  • #4845: prompt is too long unrecoverable
  • #5360: prompt is too long
  • #5478: prompt is too long when replacing in a file
  • #6480: Mid-turn compaction fix (helps with the context overflow issue)

The current solution via mid-turn compaction (#6480) helps with multi-turn scenarios, but as noted in this issue, it doesn't solve the single large fetch problem where the content itself exceeds the token limit before it can be processed.

@github-actions[bot] commented on GitHub (Jan 7, 2026): This issue mentions several related issues that address the same core problem of context/token overflow: - #4845: prompt is too long unrecoverable - #5360: prompt is too long - #5478: prompt is too long when replacing in a file - #6480: Mid-turn compaction fix (helps with the context overflow issue) The current solution via mid-turn compaction (#6480) helps with multi-turn scenarios, but as noted in this issue, it doesn't solve the single large fetch problem where the content itself exceeds the token limit before it can be processed.
yindo commented 2026-02-16 17:43:30 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Jan 7, 2026):

Lol im working on this rn

@rekram1-node commented on GitHub (Jan 7, 2026): Lol im working on this rn
yindo commented 2026-02-16 17:43:30 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Jan 7, 2026):

@jdillon can u show me how claude code does it for reference? I haven't seen any examples before?

@rekram1-node commented on GitHub (Jan 7, 2026): @jdillon can u show me how claude code does it for reference? I haven't seen any examples before?
yindo commented 2026-02-16 17:43:30 -05:00
Author
Owner
Copy Link
Copy Source

@jdillon commented on GitHub (Jan 7, 2026):

@rekram1-node I can't find any docs, I may have been confused about WebFetch vs. it showing Fetch. Though I can say that it does have some internal handling of this tool which doesn't cause the session to blow out when it tries to look at a page. I have seen it download urls to tmp files and refer to those tmp files later for reading.

I have run into this several times, and the exact same prompts work fine in claude.

I wonder if I slipped through to an alternative universe because I really did think it had 2 different modes to fetch URL content, but now I can't find it.

I imagine though, that a solution might be to always stream content to a tmp file, then if the content is too large, to provide the llm the file ref and how big it is and then let it decide if it needs to call another tool to parse out something or if it should do something else to summarize it before adding it to the conversation. or if its was small to just include it and rm the tmp file?

I also wonder if it might be using a sub-agent to handle fetching and summarizing, but at this point i'm just wildy guessing.

@jdillon commented on GitHub (Jan 7, 2026): @rekram1-node I can't find any docs, I may have been confused about WebFetch vs. it showing Fetch. Though I can say that it does have some internal handling of this tool which doesn't cause the session to blow out when it tries to look at a page. I have seen it download urls to tmp files and refer to those tmp files later for reading. I have run into this several times, and the exact same prompts work fine in claude. I wonder if I slipped through to an alternative universe because I really did think it had 2 different modes to fetch URL content, but now I can't find it. I imagine though, that a solution might be to always stream content to a tmp file, then if the content is too large, to provide the llm the file ref and how big it is and then let it decide if it needs to call another tool to parse out something or if it should do something else to summarize it before adding it to the conversation. or if its was small to just include it and rm the tmp file? I also wonder if it might be using a sub-agent to handle fetching and summarizing, but at this point i'm just wildy guessing.
yindo commented 2026-02-16 17:43:30 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Jan 7, 2026):

I also wonder if it might be using a sub-agent to handle fetching and summarizing, but at this point i'm just wildy guessing.

I think that it's doing

@rekram1-node commented on GitHub (Jan 7, 2026): > I also wonder if it might be using a sub-agent to handle fetching and summarizing, but at this point i'm just wildy guessing. I think that it's doing
yindo commented 2026-02-16 17:43:30 -05:00
Author
Owner
Copy Link
Copy Source

@davidbernat commented on GitHub (Jan 7, 2026):

I second this, for its implicit cache and persistence of the HTML data. If I may expand this slightly, with only novice experience with WebFetch and Playwright, I found myself wishing that every HTML page traversed was automatically being cached on the filesystem in a permanent (or temporary) location. Not only for research and reconstitution efforts afterwards, but I found that the agent would occasionally need to revisit a page again (instead of pulling its recent-enough HTML contents from its proposed disk) and also, crucially, requests on how to use a specific HTML file may adapt. For instance, to read an HTML document to acquire all the product titles and prices in a first pass, and then, on the basis of quality or other factors, request further information from that same page, such as their product ids, etc. In some regards this is a specific to the downstream MCPs (i.e., Playwright) but more generally this is an agent design decision that spans multiple kinds of document usage.

Side note: is OpenCode using its own WebFetch or is OpenCode using a configuration which uses, say, this one.
https://github.com/manooll/webfetch-mcp

Thanks.

@davidbernat commented on GitHub (Jan 7, 2026): I second this, for its implicit cache and persistence of the HTML data. If I may expand this slightly, with only novice experience with WebFetch and Playwright, I found myself wishing that every HTML page traversed was automatically being cached on the filesystem in a permanent (or temporary) location. Not only for research and reconstitution efforts afterwards, but I found that the agent would occasionally need to revisit a page again (instead of pulling its recent-enough HTML contents from its proposed disk) and also, crucially, requests on how to use a specific HTML file may adapt. For instance, to read an HTML document to acquire all the product titles and prices in a first pass, and then, on the basis of quality or other factors, request further information from that same page, such as their product ids, etc. In some regards this is a specific to the downstream MCPs (i.e., Playwright) but more generally this is an agent design decision that spans multiple kinds of document usage. Side note: is OpenCode using its own WebFetch or is OpenCode using a configuration which uses, say, this one. https://github.com/manooll/webfetch-mcp Thanks.
yindo commented 2026-02-16 17:43:30 -05:00
Author
Owner
Copy Link
Copy Source

@davidbernat commented on GitHub (Jan 8, 2026):

Apologies, @rekram1-node. I would appreciate for you to comment on my mention above, as your commit newest commit (#7239) does not directly address the concerns of my statement. I briefly read through the total commit changelist of #7239 (and tickets #6234 by @ben-vargas #6048 by @shantur) and from my understanding of your implementation what I see in your updates appears to be focused on context management (see further), not [necessarily] archival reconstruction (my discussion).

For your context management, you are concerned about large output from either WebFetch and tools clogging the context, and so your new change now writes large length outputs to a file on disk, instead of context, then selectively reads portions of the file into context as needed. This is great work!

But, can one enable this mechanism all the time regardless of output length? Direct its output to long term filesystem locations (not temporary)? Moreover, since you three know so much about this mechanism now, there seems to be no framework or set of guidelines within OpenCode for archival reconstruction of sessions. This would appear to be a massive feature needed for longevity and self-healing or self-learning hyper-use of OpenCode.

In a perfect world, I would decide to set a flag at the agent level to archive to file every agent-tool and download and access of any non-local files (or at least their checksums). This is because AI needs a mechanism for total reconstruction of its thinking (see also https://github.com/anomalyco/opencode/pull/6243#issuecomment-3721474461), and for repeated reads efficiencies.

Thanks. (Great work!!)

@davidbernat commented on GitHub (Jan 8, 2026): Apologies, @rekram1-node. I would appreciate for you to comment on my mention above, as your commit newest commit (#7239) does not directly address the concerns of my statement. I briefly read through the total commit changelist of #7239 (and tickets #6234 by @ben-vargas #6048 by @shantur) and from my understanding of your implementation what I see in your updates appears to be focused on context management (see further), not [necessarily] archival reconstruction (my discussion). For your context management, you are concerned about large output from either WebFetch and tools clogging the context, and so your new change now writes large length outputs to a file on disk, instead of context, then selectively reads portions of the file into context as needed. This is great work! But, can one enable this mechanism all the time regardless of output length? Direct its output to long term filesystem locations (not temporary)? Moreover, since you three know so much about this mechanism now, there seems to be no framework or set of guidelines within OpenCode for archival reconstruction of sessions. This would appear to be a massive feature needed for longevity and self-healing or self-learning hyper-use of OpenCode. In a perfect world, I would decide to set a flag at the agent level to archive to file every agent-tool and download and access of any non-local files (or at least their checksums). This is because AI needs a mechanism for total reconstruction of its thinking (see also https://github.com/anomalyco/opencode/pull/6243#issuecomment-3721474461), and for repeated reads efficiencies. Thanks. (Great work!!)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#4341
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?