vllm plugin can't not handle the case when my vllm serve 2 models, each on its own GPU #161

New Issue

2026-02-22T17:20:38-05:00

yindo commented

2026-02-22 17:20:38 -05:00

Originally created by @qiulang on GitHub (Jul 10, 2025).

My machine has 2 A10 GPU, on GPU 0 vllm serve a model, say Qwen3-8B, on GPU 1 vllm serve another model, Qwen-1.7B. Although I can add 2 models to the setting, but I can't use them because the app will complain it can't find the model, while sometimes it can.

I think it is because the request sends to my two GPUs randomly, so if it sends the request to GPU 0, while I set using Qwen3-8B then it will work, but if I it sends to GPU 1 it will complain 404.

When it works

When it does not work

Originally created by @qiulang on GitHub (Jul 10, 2025). My machine has 2 A10 GPU, on GPU 0 vllm serve a model, say Qwen3-8B, on GPU 1 vllm serve another model, Qwen-1.7B. Although I can add 2 models to the setting, but I can't use them because the app will complain it can't find the model, while sometimes it can. I think it is because the request sends to my two GPUs randomly, so if it sends the request to GPU 0, while I set using Qwen3-8B then it will work, but if I it sends to GPU 1 it will complain 404. **When it works** <img width="649" height="647" alt="Image" src="https://github.com/user-attachments/assets/039a1abf-08c7-4b42-bf4b-70a1c76d8b6b" /> **When it does not work** <img width="644" height="666" alt="Image" src="https://github.com/user-attachments/assets/cb884c29-c123-4615-b92a-1b14cc5fd479" />

yindo added the bug label 2026-02-22 17:20:38 -05:00

yindo referenced this issue

2026-02-22 17:35:37 -05:00

[PR #161] [MERGED] update quicksandzn/elasticsearch to 0.0.3 #669

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langgenius/dify-plugins#161