anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-23 02:36:57 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

Webfetch and user agent #1474

New Issue
Closed
opened 2026-02-16 17:31:09 -05:00 by yindo · 2 comments
yindo commented 2026-02-16 17:31:09 -05:00
Owner
Copy Link
Copy Source

Originally created by @rmoriz on GitHub (Aug 24, 2025).

Originally assigned to: @thdxr on GitHub.

First: I don't want to start a political discussion, just point out an issue. I don't want to start or fuel a discussion about Anubus or if the problem others see with AI scraping is real or not or if this approach is good, bad or useless.

Issue:
Some sites deploy Anubis to "block requests by AI spiders". For example kernel.org or codeberg.org.

Our webfetch uses bun's fetch but "impersonates" as another web browser.
https://github.com/sst/opencode/blob/dev/packages/opencode/src/tool/webfetch.ts#L51

Accessing Anubis-running sites requires "proof of work" (kind of client side crypto mining). Bun's fetch is not a browser implementation, so this results in a block.

I was told in reply:

The blocking is targeted specifically at requests that try to disguise themselves as regular users. Whether the tool is cURL or some other library doesn't matter. Whether it comes from your local machine or some remote server doesn't matter.
Whatever your thing did, it sent a browser-like user agent, which triggered Anubis. But since it isn't actually a fully capable browser, it doesn't get past the proof-of-work.

This act of spoofing is exactly what's intended to be blocked.
If your AI tool were to not try to hide behind the disguise of a regular user and instead were to identify itself with a unique, unambiguous user agent, you wouldn't be blocked.

Try plain curl in a terminal yourself. Then try again with something like -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0' (impersonating Firefox).

Source

I'm not sure if other opencode users are having trouble or if it's just a minor issues. I have no opinion if something should be changed or not. I just want to point out, that this may or may not become an issue for users.

Originally created by @rmoriz on GitHub (Aug 24, 2025). Originally assigned to: @thdxr on GitHub. *First: I don't want to start a political discussion, just point out an issue. I don't want to start or fuel a discussion about Anubus or if the problem others see with AI scraping is real or not or if this approach is good, bad or useless.* **Issue:** Some sites deploy [Anubis](https://github.com/TecharoHQ/anubis) to "block requests by AI spiders". For example kernel.org or codeberg.org. Our webfetch uses bun's fetch but "impersonates" as another web browser. https://github.com/sst/opencode/blob/dev/packages/opencode/src/tool/webfetch.ts#L51 Accessing Anubis-running sites requires "proof of work" (kind of client side crypto mining). Bun's fetch is not a browser implementation, so this results in a block. I was told in reply: >The blocking is targeted specifically at requests that try to disguise themselves as regular users. Whether the tool is cURL or some other library doesn't matter. Whether it comes from your local machine or some remote server doesn't matter. Whatever your thing did, it sent a browser-like user agent, which triggered Anubis. But since it isn't actually a fully capable browser, it doesn't get past the proof-of-work. >This act of spoofing is exactly what's intended to be blocked. If your AI tool were to not try to hide behind the disguise of a regular user and instead were to identify itself with a unique, unambiguous user agent, you wouldn't be blocked. >Try plain curl in a terminal yourself. Then try again with something like -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0' (impersonating Firefox). [Source](https://codeberg.org/forgejo/discussions/issues/319#issuecomment-6694543) I'm not sure if other opencode users are having trouble or if it's just a minor issues. I have no opinion if something should be changed or not. I just want to point out, that this may or may not become an issue for users.
yindo closed this issue 2026-02-16 17:31:09 -05:00
yindo commented 2026-02-16 17:31:09 -05:00
Author
Owner
Copy Link
Copy Source

@rmoriz commented on GitHub (Aug 27, 2025):

While this example is unrelated to opencode, some other agents like Atlassian Rovo Dev CLI, go berserk on curl calls probably due to inefficient tool_calls: https://gist.githubusercontent.com/rmoriz/097b2adcde78bdd81322feb2916c7fa0/raw/dfe8692621226864d7d5d68e8d3225cbb6e80ade/agent.txt

So the identification of the agent besides the faked "User Agent" seems to be a valid claim by the site operators. This would allow selective blocking and long term fixed agents.

@rmoriz commented on GitHub (Aug 27, 2025): While this example is unrelated to opencode, some other agents like Atlassian Rovo Dev CLI, go berserk on curl calls probably due to inefficient tool_calls: https://gist.githubusercontent.com/rmoriz/097b2adcde78bdd81322feb2916c7fa0/raw/dfe8692621226864d7d5d68e8d3225cbb6e80ade/agent.txt So the identification of the agent besides the faked "User Agent" seems to be a valid claim by the site operators. This would allow selective blocking and long term fixed agents.
yindo commented 2026-02-16 17:31:09 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Dec 27, 2025):

[automated] Closing due to 90+ days of inactivity. Feel free to reopen if you still need this!

@rekram1-node commented on GitHub (Dec 27, 2025): [automated] Closing due to 90+ days of inactivity. Feel free to reopen if you still need this!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#1474
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?