mirror of
https://github.com/langgenius/dify-official-plugins.git
synced 2026-07-21 17:45:23 -04:00
Local deployment of dify cannot use vllm's streaming return #408
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @froukee on GitHub (Jun 29, 2025).
Self Checks
Dify version
1.4.2 1.5.0
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
The Docker version of dify is deployed locally and on the server, and the model is deployed on another server using vllm. No streaming returns when calling the locally deployed model using vllm and OpenAI-API-compatible plug-ins. Upgrading the local dify to 1.5.0 still does not return streaming. Using cloud's dify to test the locally deployed model, using the vllm plug-in, the streaming returns are normal, and using code to test the locally deployed model, the streaming returns are normal.
✔️ Expected Behavior
No response
❌ Actual Behavior
No response
@dosubot[bot] commented on GitHub (Jun 29, 2025):
Dify's codebase and Docker setup both support streaming with vLLM, and there’s no logic that disables streaming for self-hosted or Docker deployments—the streaming flag is propagated all the way through the stack see code reference. If streaming works with Dify Cloud and direct API calls, but not with your local Docker deployment, the issue is almost certainly environmental.
Check your Docker NGINX proxy config: make sure
proxy_buffering off;,proxy_http_version 1.1;, andproxy_set_header Connection "";are set see nginx config. Also, confirm there are no extra proxies or firewalls between Dify, the plugin daemon, and vLLM that might buffer or block chunked responses.Verify that the plugin daemon and vLLM backend are running with streaming-capable servers (like gunicorn/uvicorn in async mode, not WSGI-only). If possible, try a direct streaming test (e.g., with curl or Python requests using
stream=True) from inside the Docker container to the plugin daemon and vLLM endpoints.If you can share logs or config snippets, I can help you dig deeper.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
@froukee commented on GitHub (Jun 29, 2025):
@dosu. I used the official docker local deployment without any changes, which means that nginx uses the configuration in the dify code repository. At the same time, I entered the container and used curl to test the service deployed by vllm, which was returned in a normal streaming manner. I used the plug-ins provided by other model vendors and they were returned in a normal streaming manner. This is what confuses me.
@dosubot[bot] commented on GitHub (Jun 29, 2025):
This is definitely a tricky one, since your environment and config look correct and other plugins stream fine. Dify treats vLLM just like any other OpenAI-compatible provider—there’s no special handling or code path that would block streaming for vLLM specifically, and the streaming flag is always passed through the stack (reference).
To dig deeper, I recommend enabling detailed logging to capture what’s happening during a failing vLLM streaming request. In your
.envfile, set:Then restart your containers and try a vLLM streaming request. Check the logs for any errors, warnings, or differences in how the vLLM plugin is invoked compared to other plugins. If possible, also capture the HTTP request/response headers and payloads for a streaming call to vLLM via Dify and compare them to a successful curl/plugin call (using
curl -vor similar tools).If you can share (with sensitive info redacted) your vLLM provider config from Dify and a snippet of the logs from a failed streaming attempt, I can help analyze further. This should help pinpoint whether the issue is in the request Dify sends, the plugin daemon, or something else in the stack.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
@froukee commented on GitHub (Jun 30, 2025):
@dosu. Thank u. I enabling detailed logging to capture what’s happening during a failing vLLM streaming request.I found that it was streaming output, but the first token took a very long time, almost as long as the non-streaming output time, and then all the content was output instantly. I mistakenly thought it was non-streaming. How can I locate this?
@dosubot[bot] commented on GitHub (Jun 30, 2025):
What you’re seeing—a long wait for the first token, then all tokens arriving at once—is almost always caused by something upstream of Dify holding back the stream until the first token is ready. Dify’s backend and HTTP stack are designed to relay each chunk as soon as it’s received from the plugin daemon, with no buffering or batching in between (code reference, streaming logic). So, the delay is likely in the plugin daemon’s vLLM handler or vLLM itself—possibly due to batching, request queueing, or model warmup.
To pinpoint where the delay happens, enable detailed logging (with timestamps) in Dify, the plugin daemon, and vLLM if possible. For a single streaming request, record when: (a) Dify receives the request, (b) Dify sends it to the plugin daemon, (c) the plugin daemon sends it to vLLM, (d) vLLM emits the first token, (e) the plugin daemon relays the first token, and (f) Dify relays it to the client. Compare these timings to a direct streaming call to vLLM from inside the Dify container. If the delay is before vLLM emits the first token, it’s likely a vLLM or plugin daemon config issue (like batching or cold start). If the delay is after, check for network or proxy buffering.
If you share anonymized logs or timing data, I can help interpret them and narrow down the bottleneck.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
@froukee commented on GitHub (Jul 1, 2025):
@dosu.
vllm logs：
WARNING 06-30 20:29:00 [protocol.py:58]
The following fields were present in the request but ignored: {'enable_thinking'}
INFO 06-30 20:29:00 [logger.py:43] Received request chatcmpl-2f4435c276364964ba27675dca982a36: prompt: '<｜begin▁of▁sentence｜>general 1000 words content<｜User｜>666<｜Assistant｜>\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=16369, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
INFO 06-30 20:29:00 [async_llm.py:271] Added request chatcmpl-2f4435c276364964ba27675dca982a36.
INFO: 172.19.0.1:50414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 06-30 20:29:10 [loggers.py:118] Engine 000: Avg prompt throughput: 1.5 tokens/s, Avg generation throughput: 65.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.0%, Prefix cache hit rate: 54.4%
INFO 06-30 20:29:20 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 65.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.7%, Prefix cache hit rate: 54.4%
INFO 06-30 20:29:30 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 63.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 54.4%
INFO 06-30 20:29:40 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 53.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.7%, Prefix cache hit rate: 54.4%
dify logs：
2025/07/01 03:29:00 stdio.go:182: [INFO]plugin yangyaofei/vllm:0.1.5: Model parameters: {'enable_thinking': True}
2025-07-01 11:29:00
2025/07/01 03:29:00 stdio.go:182: [INFO]plugin yangyaofei/vllm:0.1.5: Prompt messages: [SystemPromptMessage(role=<PromptMessageRole.SYSTEM: 'system'>, content='生成1000字文案', name=''), UserPromptMessage(role=<PromptMessageRole.USER: 'user'>, content='666', name='')]
2025-07-01 11:29:00
2025/07/01 03:29:00 stdio.go:182: [INFO]plugin yangyaofei/vllm:0.1.5: Request Model parameters: {'enable_thinking': True, 'chat_template_kwargs': {'enable_thinking': True}}
2025-07-01 11:29:37
[GIN] 2025/07/01 - 03:29:37 | 200 | 37.198022099s | 172.19.0.8 | POST "/plugin/4e174c88-dbf4-4705-8c9c-58910402ac0c/dispatch/llm/invoke
@dosubot[bot] commented on GitHub (Aug 20, 2025):
Hi, @froukee. I'm Dosu, and I'm helping the dify-official-plugins team manage their backlog and am marking this issue as stale.
Issue Summary:
Next Steps:
Thank you for your understanding and contribution!
@ViPenn commented on GitHub (Oct 13, 2025):
@froukee I also encountered a similar problem. I would like to ask if you have solved this issue?