anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 18:25:32 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

When calling to vllm server with hermes tool parser opencode doesn't call the tools #793

New Issue
Closed
opened 2026-02-16 17:28:20 -05:00 by yindo · 7 comments
yindo commented 2026-02-16 17:28:20 -05:00
Owner
Copy Link
Copy Source

Originally created by @Anodyine on GitHub (Jul 18, 2025).

I'm trying to get this working with my local vllm server on fedora 41. Here's the current server command I'm running:

vllm serve Qwen/Qwen2.5-Coder-32B-Instruct --tensor-parallel-size 4 --enable-auto-tool-choice --tool-call-parser hermes --gpu-memory-utilization 0.8 --max-model-len 30000 --trust-remote-code

It seems that the model generates the tool calls fine, but but opencode just shows them as text instead of calling them. Is support for this feature coming later? Am I missing an easy workaround?

Image
Originally created by @Anodyine on GitHub (Jul 18, 2025). I'm trying to get this working with my local vllm server on fedora 41. Here's the current server command I'm running: vllm serve Qwen/Qwen2.5-Coder-32B-Instruct --tensor-parallel-size 4 --enable-auto-tool-choice --tool-call-parser hermes --gpu-memory-utilization 0.8 --max-model-len 30000 --trust-remote-code It seems that the model generates the tool calls fine, but but opencode just shows them as text instead of calling them. Is support for this feature coming later? Am I missing an easy workaround? <img width="1214" height="1330" alt="Image" src="https://github.com/user-attachments/assets/c9438719-3b7b-4cf9-b874-baff8d8c4122" />
yindo closed this issue 2026-02-16 17:28:20 -05:00
yindo commented 2026-02-16 17:28:20 -05:00
Author
Owner
Copy Link
Copy Source

@Anodyine commented on GitHub (Jul 18, 2025):

I actually got it working with a local model that is served with vllm 0.9.2. It took a few messages to get it to use the tools, but eventually it made an edit that worked. So at least it proves that the tool calling can work with a local model for opencode 0.3.20. However, it is not practical for use yet. For more complex tasks it just thinks and then doesn't respond. I wonder if it is timing out. I would appreciate any help and will be looking out for updates that help us use local models better.

Image

Here is the vllm serve command that worked on my system with 4 3090 gpus:

vllm serve Qwen/Qwen3-32B --enable-reasoning --reasoning-parser deepseek_r1 --tensor-parallel-size 4 --gpu-memory-utilization 0.8 --max-model-len 20000 --enable-auto-tool-choice --tool-call-parser hermes --trust-remote-code

@Anodyine commented on GitHub (Jul 18, 2025): I actually got it working with a local model that is served with vllm 0.9.2. It took a few messages to get it to use the tools, but eventually it made an edit that worked. So at least it proves that the tool calling can work with a local model for opencode 0.3.20. However, it is not practical for use yet. For more complex tasks it just thinks and then doesn't respond. I wonder if it is timing out. I would appreciate any help and will be looking out for updates that help us use local models better. <img width="1218" height="1385" alt="Image" src="https://github.com/user-attachments/assets/afff459d-d86e-4fb5-9cc4-8aabc13d67ed" /> Here is the vllm serve command that worked on my system with 4 3090 gpus: vllm serve Qwen/Qwen3-32B --enable-reasoning --reasoning-parser deepseek_r1 --tensor-parallel-size 4 --gpu-memory-utilization 0.8 --max-model-len 20000 --enable-auto-tool-choice --tool-call-parser hermes --trust-remote-code
yindo commented 2026-02-16 17:28:20 -05:00
Author
Owner
Copy Link
Copy Source

@peaster commented on GitHub (Aug 8, 2025):

@Anodyine did you have any further progress with this? I'm trying to do the same with Qwen3 30B Coder on dual 3090s and i'm seeing roughly the same issues. It just doesn't process tool calls

@peaster commented on GitHub (Aug 8, 2025): @Anodyine did you have any further progress with this? I'm trying to do the same with Qwen3 30B Coder on dual 3090s and i'm seeing roughly the same issues. It just doesn't process tool calls
yindo commented 2026-02-16 17:28:21 -05:00
Author
Owner
Copy Link
Copy Source

@Anodyine commented on GitHub (Aug 18, 2025):

@peaster Yeah actually I had some great success with vllm 10.0 since it supports the qwen3_coder tool parser. I used the qwen cli client and it is very fast and effective.

Here is the relevant article:

https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3-Coder-480B-A35B.html

Of course, I was using this model not the 480B one the article mentions.

https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct

@Anodyine commented on GitHub (Aug 18, 2025): @peaster Yeah actually I had some great success with vllm 10.0 since it supports the qwen3_coder tool parser. I used the qwen cli client and it is very fast and effective. Here is the relevant article: https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3-Coder-480B-A35B.html Of course, I was using this model not the 480B one the article mentions. https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct
yindo commented 2026-02-16 17:28:21 -05:00
Author
Owner
Copy Link
Copy Source

@Anodyine commented on GitHub (Aug 19, 2025):

@peaster Here is my start server command for reference. You need to reduce tensor-parallel-size to 2 and may need to reduce max model length and/or gpu-memory-utilization if you run out of memory.

/opt/vllm/.venv/bin/vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct
--tensor-parallel-size 4
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--gpu-memory-utilization 0.9
--max-model-len 200000
--max-num-seqs 2
--max-num-batched-tokens 16384
--enable-chunked-prefill
--enable-prefix-caching
--swap-space 32
--trust-remote-code
--host 127.0.0.1
--port 8000

@Anodyine commented on GitHub (Aug 19, 2025): @peaster Here is my start server command for reference. You need to reduce tensor-parallel-size to 2 and may need to reduce max model length and/or gpu-memory-utilization if you run out of memory. /opt/vllm/.venv/bin/vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \ --tensor-parallel-size 4 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --gpu-memory-utilization 0.9 \ --max-model-len 200000 \ --max-num-seqs 2 \ --max-num-batched-tokens 16384 \ --enable-chunked-prefill \ --enable-prefix-caching \ --swap-space 32 \ --trust-remote-code \ --host 127.0.0.1 \ --port 8000
yindo commented 2026-02-16 17:28:21 -05:00
Author
Owner
Copy Link
Copy Source

@peaster commented on GitHub (Aug 19, 2025):

Thanks! i did get it mostly working using the new tool call parser. In opencode specifically, it's struggling on a couple of the tools though. I commented in this thread and that mostly works https://github.com/sst/opencode/issues/1809#issuecomment-3197891457 but it would be nice to have this working natively too

@peaster commented on GitHub (Aug 19, 2025): Thanks! i did get it mostly working using the new tool call parser. In opencode specifically, it's struggling on a couple of the tools though. I commented in this thread and that mostly works https://github.com/sst/opencode/issues/1809#issuecomment-3197891457 but it would be nice to have this working natively too
yindo commented 2026-02-16 17:28:21 -05:00
Author
Owner
Copy Link
Copy Source

@loligapta commented on GitHub (Oct 15, 2025):

Sir, how did you connect opencode to local running vLLM server? can share config please sir?

@loligapta commented on GitHub (Oct 15, 2025): Sir, how did you connect opencode to local running vLLM server? can share config please sir?
yindo commented 2026-02-16 17:28:21 -05:00
Author
Owner
Copy Link
Copy Source

@zaidorx commented on GitHub (Jan 19, 2026):

@loligapta Not sure if this is still relevant but I recently discovered and set up opencode with vllm and here is the section of my config.json

{
"$schema": "https://opencode.ai/config.json",
"provider": {
"local-vllm": {
"npm": "@ai-sdk/openai-compatible",
"name": "vLLM (Local)",
"options": {
"baseURL": "http://192.168.99.23:8000/v1",
"apiKey": "dummy-key"
},
"models": {
"Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8": {
"name": "Qwen3-Coder-30B"
}
}
}
}

@zaidorx commented on GitHub (Jan 19, 2026): @loligapta Not sure if this is still relevant but I recently discovered and set up opencode with vllm and here is the section of my config.json { "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "vLLM (Local)", "options": { "baseURL": "http://192.168.99.23:8000/v1", "apiKey": "dummy-key" }, "models": { "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8": { "name": "Qwen3-Coder-30B" } } } }
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#793
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?