langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 12:36:14 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Question Classifier doesn't work completely with vLLM #13367

New Issue
Closed
opened 2026-02-21 19:11:50 -05:00 by yindo · 4 comments
yindo commented 2026-02-21 19:11:50 -05:00
Owner
Copy Link
Copy Source

Originally created by @StellaContrail on GitHub (Apr 25, 2025).

Originally assigned to: @StellaContrail on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.3.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Launch a model with vLLM (in this case, google/gemma3-1b-it)

    $ vllm serve "google/gemma-3-1b-it" --device cuda

  2. In Dify's settings, go to "Model Provider" and set up OpenAI-API-compatible plugin for the vLLM endpoint.

  3. Create a chatflow application and workflow with a question classifier using the vLLM model.

  4. Run the workflow.

My workflow for reference:

Image

It seems like this issue was addressed before (#17773), but was closed as it's "This is fixed in later version. Please upgrade to the latest version instead." So I assume this is a regression and the issue is happening again.

I think the issue is caused by this part: question_classifier_node.py#L82-L92. There are two successive user prompts in the messages because both self._get_prompt_template and self._fetch_prompt_messages append a user prompt at the end of the list. Since vLLM only allows alternating user/assistant messages, the question classifier fails to execute.

You can set sys_query=None when calling the self._fetch_prompt_messages method to address the issue. If needed, I can make a pull request.

✔️ Expected Behavior

Prerequisite:

  • vLLM model is configured in Question Classifier block

Expected Behavior:

  • Question Classifier block works without an error. (= user/assistant prompts alternate in the request messages)

❌ Actual Behavior

Prerequisite:

  • vLLM model is configured in Question Classifier block

Actual Behavior:

  • Question Classifier block fails with an error.

[openai_api_compatible] Error: PluginInvokeError: {"args":{"description":"[models] Error: API request failed with status code 400: {"object":"error","message":"Conversation roles must alternate user/assistant/user/assistant/...","type":"BadRequestError","param":null,"code":400}"},"error_type":"InvokeError","message":"[models] Error: API request failed with status code 400: {"object":"error","message":"Conversation roles must alternate user/assistant/user/assistant/...","type":"BadRequestError","param":null,"code":400}"}

Image

Originally created by @StellaContrail on GitHub (Apr 25, 2025). Originally assigned to: @StellaContrail on GitHub. ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.3.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Launch a model with vLLM (in this case, google/gemma3-1b-it) ```bash $ vllm serve "google/gemma-3-1b-it" --device cuda ``` 2. In Dify's settings, go to "Model Provider" and set up OpenAI-API-compatible plugin for the vLLM endpoint. 3. Create a chatflow application and workflow with a question classifier using the vLLM model. 4. Run the workflow. My workflow for reference: ![Image](https://github.com/user-attachments/assets/0ad30009-3b91-4470-95b1-aa1dafa860d2) It seems like this issue was addressed before (#17773), but was closed as it's "This is fixed in later version. Please upgrade to the latest version instead." So I assume this is a regression and the issue is happening again. I think the issue is caused by this part: [question_classifier_node.py#L82-L92](https://github.com/langgenius/dify/blob/main/api/core/workflow/nodes/question_classifier/question_classifier_node.py#L82-L92). There are two successive user prompts in the messages because both `self._get_prompt_template` and `self._fetch_prompt_messages` append a user prompt at the end of the list. Since vLLM only allows alternating user/assistant messages, the question classifier fails to execute. You can set `sys_query=None` when calling the `self._fetch_prompt_messages` method to address the issue. If needed, I can make a pull request. ### ✔️ Expected Behavior Prerequisite: - vLLM model is configured in Question Classifier block Expected Behavior: - Question Classifier block works without an error. (= user/assistant prompts alternate in the request messages) ### ❌ Actual Behavior Prerequisite: - vLLM model is configured in Question Classifier block Actual Behavior: - Question Classifier block fails with an error. > [openai_api_compatible] Error: PluginInvokeError: {"args":{"description":"[models] Error: API request failed with status code 400: {\"object\":\"error\",\"message\":\"Conversation roles must alternate user/assistant/user/assistant/...\",\"type\":\"BadRequestError\",\"param\":null,\"code\":400}"},"error_type":"InvokeError","message":"[models] Error: API request failed with status code 400: {\"object\":\"error\",\"message\":\"Conversation roles must alternate user/assistant/user/assistant/...\",\"type\":\"BadRequestError\",\"param\":null,\"code\":400}"} ![Image](https://github.com/user-attachments/assets/2c8af9e2-2e7d-455e-8278-f86cc4609a1f)
yindo added the 🐞 buggood first issue labels 2026-02-21 19:11:50 -05:00
yindo closed this issue 2026-02-21 19:11:50 -05:00
yindo commented 2026-02-21 19:11:50 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 25, 2025):

Hello @StellaContrail! I'm Dosu, your friendly bot here to help you with bugs, questions, or becoming a contributor. I'm ready to assist you while we wait for a human maintainer. Let's tackle this together!

A similar issue was previously discussed in this thread, where the problem was resolved by upgrading to a newer version of Dify. The issue was caused by non-alternating user/assistant prompts due to insufficient condition checks in the def_fetch_prompt_messages function, which allowed two consecutive 'user' messages in the API request. This was fixed in later versions by ensuring proper alternation between 'user' and 'assistant' messages [1].

In your case, setting sys_query=None when calling the self._fetch_prompt_messages method can help prevent the addition of an extra user prompt, which should resolve the issue with the vLLM model requiring alternating user/assistant messages. This approach aligns with the changes made in newer versions to ensure proper message alternation [2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 25, 2025): <!-- Greeting --> Hello @StellaContrail! I'm [Dosu](https://dosu.dev), your friendly bot here to help you with bugs, questions, or becoming a contributor. I'm ready to assist you while we wait for a human maintainer. Let's tackle this together! <!-- Answer --> A similar issue was previously discussed in [this thread](https://github.com/langgenius/dify/issues/17773), where the problem was resolved by upgrading to a newer version of Dify. The issue was caused by non-alternating user/assistant prompts due to insufficient condition checks in the `def_fetch_prompt_messages` function, which allowed two consecutive 'user' messages in the API request. This was fixed in later versions by ensuring proper alternation between 'user' and 'assistant' messages <sup>[[1]](https://github.com/langgenius/dify/issues/17773)</sup>. In your case, setting `sys_query=None` when calling the `self._fetch_prompt_messages` method can help prevent the addition of an extra user prompt, which should resolve the issue with the vLLM model requiring alternating user/assistant messages. This approach aligns with the changes made in newer versions to ensure proper message alternation <sup>[[2]](https://github.com/langgenius/dify/blob/main/api/core/workflow/nodes/llm/node.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/f1cc8aec-d437-4103-af73-bd033731014d?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/f1cc8aec-d437-4103-af73-bd033731014d?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/f1cc8aec-d437-4103-af73-bd033731014d?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/f1cc8aec-d437-4103-af73-bd033731014d?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/f1cc8aec-d437-4103-af73-bd033731014d?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/f1cc8aec-d437-4103-af73-bd033731014d?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/f1cc8aec-d437-4103-af73-bd033731014d?feedback_type=other)</sup>
yindo commented 2026-02-21 19:11:50 -05:00
Author
Owner
Copy Link
Copy Source

@StellaContrail commented on GitHub (Apr 25, 2025):

Given vLLM’s popularity, I imagine this fix is highly requested.
If a fix is underway, could you let me know when it will be released? Thanks!

@StellaContrail commented on GitHub (Apr 25, 2025): Given vLLM’s popularity, I imagine this fix is highly requested. If a fix is underway, could you let me know when it will be released? Thanks!
yindo commented 2026-02-21 19:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Apr 25, 2025):

@StellaContrail Hello, feel free to open a pr, I think I might messed with other issues.

It seems like this issue was addressed before (https://github.com/langgenius/dify/issues/17773)

BTW, have you tried this provider https://marketplace.dify.ai/plugins/yangyaofei/vllm?

@crazywoola commented on GitHub (Apr 25, 2025): @StellaContrail Hello, feel free to open a pr, I think I might messed with other issues. > It seems like this issue was addressed before (https://github.com/langgenius/dify/issues/17773) BTW, have you tried this provider https://marketplace.dify.ai/plugins/yangyaofei/vllm?
yindo commented 2026-02-21 19:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@StellaContrail commented on GitHub (Apr 26, 2025):

Hi @crazywoola, thank you for the quick response!

The issue still persists even after switching to yangyaofei's vllm plugin, so I'll go ahead and open a new PR!

[vllm] Error: PluginInvokeError: {"args":{"description":"[models] Error: API request failed with status code 400: {"object":"error","message":"Conversation roles must alternate user/assistant/user/assistant/...","type":"BadRequestError","param":null,"code":400}"},"error_type":"InvokeError","message":"[models] Error: API request failed with status code 400: {"object":"error","message":"Conversation roles must alternate user/assistant/user/assistant/...","type":"BadRequestError","param":null,"code":400}"}

Image

@StellaContrail commented on GitHub (Apr 26, 2025): Hi @crazywoola, thank you for the quick response! The issue still persists even after switching to yangyaofei's vllm plugin, so I'll go ahead and open a new PR! > [vllm] Error: PluginInvokeError: {"args":{"description":"[models] Error: API request failed with status code 400: {\"object\":\"error\",\"message\":\"Conversation roles must alternate user/assistant/user/assistant/...\",\"type\":\"BadRequestError\",\"param\":null,\"code\":400}"},"error_type":"InvokeError","message":"[models] Error: API request failed with status code 400: {\"object\":\"error\",\"message\":\"Conversation roles must alternate user/assistant/user/assistant/...\",\"type\":\"BadRequestError\",\"param\":null,\"code\":400}"} ![Image](https://github.com/user-attachments/assets/979c74fe-f3e2-45f4-9adf-222edef90acf)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label good first issue 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#13367
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?