mirror of
https://github.com/langgenius/dify-official-plugins.git
synced 2026-07-22 01:55:27 -04:00
When using local deployment dify, calling vllm deployment returns no streaming #405
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @froukee on GitHub (Jun 27, 2025).
Self Checks
Dify version
1.4.2；1.5.0
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
I have two dify servers deployed locally and on the cloud, both are 1.4.2. When I use the vllm plugin and the OpenAI-API-compatible plugin to call the model deployed locally, there is no streaming return at the end node, whether it is the workflow or the dialogue flow. I upgraded the local server to 1.5.0 and it still happend. At the same time, I tested the code calling interface and the dify cloud service, and both returned streaming normally. Is there any solution or how can I further locate the problem?
✔️ Expected Behavior
No response
❌ Actual Behavior
No response
@dosubot[bot] commented on GitHub (Jun 27, 2025):
Streaming support in Dify is always enabled at the application level for OpenAI-compatible providers like vllm, and the codebase passes the streaming option all the way through the stack—there are no toggles or config options that would disable it in Docker or self-hosted setups. Since streaming works with the cloud service and direct API calls, but not in your local Docker deployment, this almost always points to an environment or deployment issue rather than a Dify bug or config problem.
Common causes include reverse proxy buffering (Nginx, Traefik, etc.), Docker networking, or WSGI/gunicorn settings that interfere with HTTP chunked responses or Server-Sent Events. Check if your local setup has any proxies or load balancers in front of Dify or vllm, and make sure buffering is disabled for streaming endpoints. Also, verify that your vllm endpoint is actually returning chunked responses when accessed through Docker and any proxies.
To further isolate the problem, try:
curl --no-bufferor browser dev tools to watch the network response and see if chunks arrive or if the response is buffered.If you can share logs, network traces, or config snippets from your local deployment, I can help you dig deeper.
For reference, here’s how streaming is handled in the code: the
response_modeparameter controls streaming, and it’s passed through the API controller, service layer, and model client, which always enables streaming if requested. The plugin client sends requests to the plugin daemon with streaming enabled, and there’s no code path that disables this in Docker or self-hosted mode. So, focus on the deployment environment and network/proxy layers for the root cause.See code details
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
