langgenius/dify-official-plugins
1
0
Fork 0
mirror of https://github.com/langgenius/dify-official-plugins.git synced 2026-07-21 17:45:23 -04:00
Code Issues 54 Packages Projects Releases Wiki Activity

llm node using OpenAI-API-compatible/Vllm model plugin return text "" #176

New Issue
Closed
opened 2026-02-16 10:18:21 -05:00 by yindo · 5 comments
yindo commented 2026-02-16 10:18:21 -05:00
Owner
Copy Link
Copy Source

Originally created by @rainvstar on GitHub (Apr 7, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.1.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I deploy llm model locally and provide openai format interface. the interface is /v1/chat/completions . I use vllm node in my workflow， and send promopt like '你是谁' to the llm node。In my llm service, I accept the request and response. but in dify, the llm out put is {"text":""}，I can't see the request and response logs to the llm service. How to find the problem.

✔️ Expected Behavior

fix bug or tell me how to find out the problem.

❌ Actual Behavior

No response

Originally created by @rainvstar on GitHub (Apr 7, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.1.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce I deploy llm model locally and provide openai format interface. the interface is /v1/chat/completions . I use vllm node in my workflow， and send promopt like '你是谁' to the llm node。In my llm service, I accept the request and response. but in dify, the llm out put is {"text":""}，I can't see the request and response logs to the llm service. How to find the problem. ### ✔️ Expected Behavior fix bug or tell me how to find out the problem. ### ❌ Actual Behavior _No response_
yindo closed this issue 2026-02-16 10:18:21 -05:00
yindo commented 2026-02-16 10:18:21 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 7, 2025):

I found some similar threads that might help you troubleshoot the issue:

  1. Why is my output empty - This discussion is unsolved but might have relevant insights.
  2. The frontend cannot show the generated text by llm - This is another unsolved discussion that could provide useful information.
  3. The frontend cannot show the generated text by llm - This open issue suggests checking the Docker network configuration. Ensure your LLM service is accessible by binding it to 0.0.0.0 instead of localhost or 127.0.0.1. Also, verify that the Dify configuration uses the correct IP address and settings for your LLM service.

These threads might offer some guidance on resolving the issue with the LLM node returning an empty text.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 7, 2025): <!-- Answer --> I found some similar threads that might help you troubleshoot the issue: 1. [Why is my output empty](https://github.com/langgenius/dify/discussions/14030) - This discussion is unsolved but might have relevant insights. 2. [The frontend cannot show the generated text by llm](https://github.com/langgenius/dify/discussions/4081) - This is another unsolved discussion that could provide useful information. 3. [The frontend cannot show the generated text by llm](https://github.com/langgenius/dify/issues/4072) - This open issue suggests checking the Docker network configuration. Ensure your LLM service is accessible by binding it to `0.0.0.0` instead of `localhost` or `127.0.0.1`. Also, verify that the Dify configuration uses the correct IP address and settings for your LLM service. These threads might offer some guidance on resolving the issue with the LLM node returning an empty text. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/cd06cad0-63b4-4ce5-a842-7e33886121ed?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/cd06cad0-63b4-4ce5-a842-7e33886121ed?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/cd06cad0-63b4-4ce5-a842-7e33886121ed?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/cd06cad0-63b4-4ce5-a842-7e33886121ed?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/cd06cad0-63b4-4ce5-a842-7e33886121ed?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/cd06cad0-63b4-4ce5-a842-7e33886121ed?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/cd06cad0-63b4-4ce5-a842-7e33886121ed?feedback_type=other)</sup>
yindo commented 2026-02-16 10:18:21 -05:00
Author
Owner
Copy Link
Copy Source

@rainvstar commented on GitHub (Apr 7, 2025):

#17589
I am not using streaming, where to find the response structure matches what Dify expects. I can't find my mock result problem, it is opanai standard. can you show me an example just like this？

{
  "id": "1337",
  "object": "chat.completion",
  "created": 1744077337.1139085,
  "model": "mock-gpt-model",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "作为一个模拟的AI助手，我只能回复您的最后一条消息：你是谁"
      }
    }
  ]
}

@dosu @crazywoola

@rainvstar commented on GitHub (Apr 7, 2025): [#17589](https://github.com/langgenius/dify/issues/17589) I am not using streaming, where to find the response structure matches what Dify expects. I can't find my mock result problem, it is opanai standard. can you show me an example just like this？ ``` { "id": "1337", "object": "chat.completion", "created": 1744077337.1139085, "model": "mock-gpt-model", "choices": [ { "message": { "role": "assistant", "content": "作为一个模拟的AI助手，我只能回复您的最后一条消息：你是谁" } } ] } ``` @dosu @crazywoola
yindo commented 2026-02-16 10:18:21 -05:00
Author
Owner
Copy Link
Copy Source

@rainvstar commented on GitHub (Apr 7, 2025):

I find out the problem, the workflow request llm node and using stream = true. how to forbidden llm reusing stream=true ？ @dosu @crazywoola @dosubot

@rainvstar commented on GitHub (Apr 7, 2025): I find out the problem, the workflow request llm node and using stream = true. how to forbidden llm reusing stream=true ？ @dosu @crazywoola @dosubot
yindo commented 2026-02-16 10:18:21 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 20, 2025):

Hi, @rainvstar. I'm Dosu, and I'm helping the dify-official-plugins team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported that using a local LLM model with an OpenAI-compatible API and the vllm node results in empty output text in Dify.
  • You confirmed that requests and responses are visible in your LLM service, indicating the service is running.
  • The issue appears related to the workflow using stream=true with the LLM node.
  • You asked how to disable streaming to fix the empty output problem and tagged maintainers for guidance.
  • No resolution or further updates have been provided since.

Next Steps:

  • Please let me know if this issue is still relevant with the latest version of dify-official-plugins by commenting here.
  • If I don’t hear back within 5 days, I will automatically close this issue.

Thanks for your understanding and contribution!

@dosubot[bot] commented on GitHub (Aug 20, 2025): Hi, @rainvstar. I'm [Dosu](https://dosu.dev), and I'm helping the dify-official-plugins team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported that using a local LLM model with an OpenAI-compatible API and the vllm node results in empty output text in Dify. - You confirmed that requests and responses are visible in your LLM service, indicating the service is running. - The issue appears related to the workflow using `stream=true` with the LLM node. - You asked how to disable streaming to fix the empty output problem and tagged maintainers for guidance. - No resolution or further updates have been provided since. **Next Steps:** - Please let me know if this issue is still relevant with the latest version of dify-official-plugins by commenting here. - If I don’t hear back within 5 days, I will automatically close this issue. Thanks for your understanding and contribution!
yindo commented 2026-02-16 10:18:21 -05:00
Author
Owner
Copy Link
Copy Source

@ninechi143 commented on GitHub (Aug 27, 2025):

@rainvstar
@wenHK
Hello, though it was an old issue you reported on April, and now it has already been 4 months passed,
I still see no answer/response on the community to address this problem.
I am not sure whether you have already solved this issue, no matter what, I still want to share my solution even if it's too late or useless.

First of all, here is the version I am using (I use docker to deploy dify on WSL):
Image

From another "same" issue your reported at langgenius/dify#17589,
I see that you develop a web api in Python by using uvicorn, and register it as a standard openai-api-compatible,
and then you construct a workflow of ChatFlow/ChatApp on dify framework, with the block form of START->LLM->ANSWER.

Actually, it's exactly the same as what I am doing, and here is how I solve this "blank text" issue, I will directly provide the python code I am using:

import uvicorn
from typing import List, Optional
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
import time, uuid
import json
from API_Llama3_chat_ChatHuggingFace import chatbot_get_response

class Message(BaseModel):
    role: str
    content: str

class ChatCompletionRequest(BaseModel):
    model: Optional[str] = None
    messages: List[Message]
    temperature: Optional[float] = 0.7
    max_tokens: Optional[int] = None
    stream: Optional[bool] = False
    user: Optional[str] = None

app = FastAPI()

@app.post("/v1/chat/completions")
def chat_completions(request: ChatCompletionRequest):

    print(request)
    message = request.messages[-1]
    user_msg = message.content if message.role == "user" else "None"
    user_id = request.user or "default"
    response, inference_time = chatbot_get_response(user_id, user_msg)

    now = int(time.time())
    print(f"user msg: {user_msg}")
    print(f"response: {response}")
    print(f"inference_time: {inference_time:.4f}")
    print("----------------------------------------")

    if request.stream:
        async def event_stream():
            data = {
                "id": f"chatcmpl-{uuid.uuid4()}",
                "object": "chat.completion.chunk",
                "created": now,
                "model": request.model or "None",
                "choices": [{
                    "index": 0,
                    "delta": {"role": "assistant", "content": response},
                    "finish_reason": None
                }],
                # "usage": {"prompt_tokens": 111, "completion_tokens": 222, "currency": "NTD"},
            }
            yield f"data: {json.dumps(data)}\n\n"

            # end signal
            yield "data: [DONE]\n\n"

        return StreamingResponse(event_stream(), media_type="text/event-stream")

    return {
        "id": f"chatcmpl-{uuid.uuid4()}",
        "object": "chat.completion",
        "created": now,
        "model": request.model or "None",
        "choices": [{
            "index": 0,
            "message": {"role": "assistant", "content": response},
            "finish_reason": "stop"
        }],
        # "usage": {"prompt_tokens": 111, "completion_tokens": 222},
    }

def main():
    uvicorn.run(app, host="0.0.0.0", port=8888)

if __name__ == "__main__":
    main()

Here, please ignore from API_Llama3_chat_ChatHuggingFace import chatbot_get_response, you need to replace it with any other function you're developing to inference the local LLM (btw, I use Huggingface and Langchain to deploy local LLM).
From the part of print(request), we will notice that there is a key-value pair: stream-True in the request, and it's just the key point, as you said.

The following figure is the corresponding workflow on dify:

Image

Note: we need to put {x}sys.query of START block in the USER column of LLM block, such that we can get the user input from chatapp web page, and feed it into the LLM block (i.e., your "openai-api-compatible").

With all these modifications, I am able to show the response on the chatapp web page.
Hope this helps.

@ninechi143 commented on GitHub (Aug 27, 2025): @rainvstar @wenHK Hello, though it was an old issue you reported on April, and now it has already been 4 months passed, I still see no answer/response on the community to address this problem. I am not sure whether you have already solved this issue, no matter what, I still want to share my solution even if it's too late or useless. First of all, here is the version I am using (I use docker to deploy dify on WSL): <img width="756" height="102" alt="Image" src="https://github.com/user-attachments/assets/7166eaf2-741d-4871-b0d4-2f9236dd7e60" /> From another "same" issue your reported at langgenius/dify#17589, I see that you develop a web api in Python by using uvicorn, and register it as a standard openai-api-compatible, and then you construct a workflow of ChatFlow/ChatApp on dify framework, with the block form of START->LLM->ANSWER. Actually, it's exactly the same as what I am doing, and here is how I solve this "blank text" issue, I will directly provide the python code I am using: ```python import uvicorn from typing import List, Optional from fastapi import FastAPI from fastapi.responses import StreamingResponse from pydantic import BaseModel import time, uuid import json from API_Llama3_chat_ChatHuggingFace import chatbot_get_response class Message(BaseModel): role: str content: str class ChatCompletionRequest(BaseModel): model: Optional[str] = None messages: List[Message] temperature: Optional[float] = 0.7 max_tokens: Optional[int] = None stream: Optional[bool] = False user: Optional[str] = None app = FastAPI() @app.post("/v1/chat/completions") def chat_completions(request: ChatCompletionRequest): print(request) message = request.messages[-1] user_msg = message.content if message.role == "user" else "None" user_id = request.user or "default" response, inference_time = chatbot_get_response(user_id, user_msg) now = int(time.time()) print(f"user msg: {user_msg}") print(f"response: {response}") print(f"inference_time: {inference_time:.4f}") print("----------------------------------------") if request.stream: async def event_stream(): data = { "id": f"chatcmpl-{uuid.uuid4()}", "object": "chat.completion.chunk", "created": now, "model": request.model or "None", "choices": [{ "index": 0, "delta": {"role": "assistant", "content": response}, "finish_reason": None }], # "usage": {"prompt_tokens": 111, "completion_tokens": 222, "currency": "NTD"}, } yield f"data: {json.dumps(data)}\n\n" # end signal yield "data: [DONE]\n\n" return StreamingResponse(event_stream(), media_type="text/event-stream") return { "id": f"chatcmpl-{uuid.uuid4()}", "object": "chat.completion", "created": now, "model": request.model or "None", "choices": [{ "index": 0, "message": {"role": "assistant", "content": response}, "finish_reason": "stop" }], # "usage": {"prompt_tokens": 111, "completion_tokens": 222}, } def main(): uvicorn.run(app, host="0.0.0.0", port=8888) if __name__ == "__main__": main() ``` Here, please ignore `from API_Llama3_chat_ChatHuggingFace import chatbot_get_response`, you need to replace it with any other function you're developing to inference the local LLM (btw, I use Huggingface and Langchain to deploy local LLM). From the part of `print(request)`, we will notice that there is a key-value pair: stream-True in the request, and it's just the key point, as you said. The following figure is the corresponding workflow on dify: <img width="1522" height="784" alt="Image" src="https://github.com/user-attachments/assets/97a70e69-1a0e-4105-87e9-d8b3a7240cfb" /> Note: we need to put {x}sys.query of START block in the USER column of LLM block, such that we can get the user input from chatapp web page, and feed it into the LLM block (i.e., your "openai-api-compatible"). With all these modifications, I am able to show the response on the chatapp web page. Hope this helps.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
bug
dependencies
documentation
enhancement
good first issue
invalid
pull-request
Mirrored from GitHub Pull Request
 question
stale
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-official-plugins#176
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?