langgenius/dify-official-plugins
1
0
Fork 0
mirror of https://github.com/langgenius/dify-official-plugins.git synced 2026-07-21 17:45:23 -04:00
Code Issues 54 Packages Projects Releases Wiki Activity

[OpenAI-API-compatible] llm stream output is not smooth problem #894

New Issue
Closed
opened 2026-02-16 10:20:52 -05:00 by yindo · 2 comments
yindo commented 2026-02-16 10:20:52 -05:00
Owner
Copy Link
Copy Source

Originally created by @ran411285752 on GitHub (Dec 22, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues Dify issues & Dify Official Plugins, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

V1.10.0

Plugin version

V0.0.27

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Background: I have a vllm model of a fastapi service agent written in python that is compatible with openai format, and the standard output has been manually processed to return 2-3 characters at a time

Problem: After using OpenAI-API-compatible configuration model, streaming output is not smooth. It feels like waiting for a certain amount of characters to accumulate and output at one time. The same interface is configured to CherryStudio or ChatBox without problems. It can be output normally like streaming output of typewriters. I have asked many friends who have similar situations.

Repeat steps:

  1. Use vllm to reason about a qwen3-8B model, and then write an openai-compatible fastapi service yourself, which will return 2-3 characters at a time for streaming output;
  2. Configure fastapi service using OpenAI-API-compatible;
  3. In chatflow, configure a model output for testing, and you will find the model called by OpenAI-API-compatible, and the streaming output will also accumulate to 140 characters for one-time output;
  4. Configure fastapi service through cherrystudio or other clients, and stream output can be output smoothly in the form of normal typewriter;
  5. Similarly, the model of xinference reasoning can be streamed smoothly by using the plug-in provided by xinference. By configuring the model api provided by xinference through OpenAI-API-compatible, the streaming output will also accumulate to 140 characters for one-time output.

This problem seriously affected the user experience

✔️ Error log

No response

Originally created by @ran411285752 on GitHub (Dec 22, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [Dify issues](https://github.com/langgenius/dify/issues) & [Dify Official Plugins](https://github.com/langgenius/dify-official-plugins/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version V1.10.0 ### Plugin version V0.0.27 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce **Background:** I have a vllm model of a fastapi service agent written in python that is compatible with openai format, and the standard output has been manually processed to return 2-3 characters at a time **Problem:** After using OpenAI-API-compatible configuration model, streaming output is not smooth. It feels like waiting for a certain amount of characters to accumulate and output at one time. The same interface is configured to CherryStudio or ChatBox without problems. It can be output normally like streaming output of typewriters. I have asked many friends who have similar situations. **Repeat steps:** 1. Use vllm to reason about a qwen3-8B model, and then write an openai-compatible fastapi service yourself, which will return 2-3 characters at a time for streaming output; 2. Configure fastapi service using OpenAI-API-compatible; 3. In chatflow, configure a model output for testing, and you will find the model called by OpenAI-API-compatible, and the streaming output will also accumulate to 140 characters for one-time output; 4. Configure fastapi service through cherrystudio or other clients, and stream output can be output smoothly in the form of normal typewriter; 5. Similarly, the model of xinference reasoning can be streamed smoothly by using the plug-in provided by xinference. By configuring the model api provided by xinference through OpenAI-API-compatible, the streaming output will also accumulate to 140 characters for one-time output. This problem seriously affected the user experience ### ✔️ Error log _No response_
yindo added the bug label 2026-02-16 10:20:52 -05:00
yindo closed this issue 2026-02-16 10:20:52 -05:00
yindo commented 2026-02-16 10:20:53 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 22, 2025):

Hi @ran411285752! I'm Dosu and I’m helping the dify-official-plugins team.

This streaming delay isn't caused by Dify or its plugins buffering output by character count—there's no 140-character buffer in the code. The most likely causes are buffering in your HTTP server (Gunicorn), Docker networking, or a reverse proxy (like Nginx or Traefik) in your deployment stack. These layers can accumulate data before flushing it to the client, which explains why other clients (like CherryStudio) stream smoothly but Dify (when running in Docker or behind a proxy) does not [source].

Here are some concrete steps and workarounds:

  • Make sure your Gunicorn worker class is set to gevent and tune worker count, connections, and keep-alive settings for optimal streaming. For example:
    gunicorn --worker-class gevent --workers 2 --worker-connections 1000 --timeout 120 app:app
    This improves concurrency and reduces buffering [source].
  • If you use a reverse proxy (Nginx, Traefik), disable buffering for streaming endpoints. For Nginx, use proxy_buffering off; for your /v1/chat/completions route.
  • Try changing the stream_mode_delimiter in your Dify OpenAI-compatible model config from \n\n to \n. This has resolved similar streaming issues for other users with FastAPI backends [source].
  • Test your FastAPI endpoint directly with curl --no-buffer or browser dev tools to confirm that it streams as expected outside Dify. If it does, the issue is almost certainly in the deployment stack, not your code.
  • Review logs and network traces for signs of buffering or delayed chunk delivery.

If you share more about your deployment (proxy, server config, Docker setup), I can help you pinpoint where the buffering is happening.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Dec 22, 2025): <!-- Greeting --> Hi @ran411285752! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the dify-official-plugins team. <!-- Answer --> This streaming delay isn't caused by Dify or its plugins buffering output by character count—there's no 140-character buffer in the code. The most likely causes are buffering in your HTTP server (Gunicorn), Docker networking, or a reverse proxy (like Nginx or Traefik) in your deployment stack. These layers can accumulate data before flushing it to the client, which explains why other clients (like CherryStudio) stream smoothly but Dify (when running in Docker or behind a proxy) does not [[source](https://github.com/langgenius/dify-official-plugins/issues/1172)]. Here are some concrete steps and workarounds: - Make sure your Gunicorn worker class is set to `gevent` and tune worker count, connections, and keep-alive settings for optimal streaming. For example: `gunicorn --worker-class gevent --workers 2 --worker-connections 1000 --timeout 120 app:app` This improves concurrency and reduces buffering [[source](https://github.com/langgenius/dify/blob/eabdc5f0ebd806ede0c8f8695d54efb24e61748b/scripts/stress-test/README.md#L101-L386)]. - If you use a reverse proxy (Nginx, Traefik), disable buffering for streaming endpoints. For Nginx, use `proxy_buffering off;` for your `/v1/chat/completions` route. - Try changing the `stream_mode_delimiter` in your Dify OpenAI-compatible model config from `\n\n` to `\n`. This has resolved similar streaming issues for other users with FastAPI backends [[source](https://github.com/langgenius/dify-official-plugins/issues/1858)]. - Test your FastAPI endpoint directly with `curl --no-buffer` or browser dev tools to confirm that it streams as expected outside Dify. If it does, the issue is almost certainly in the deployment stack, not your code. - Review logs and network traces for signs of buffering or delayed chunk delivery. If you share more about your deployment (proxy, server config, Docker setup), I can help you pinpoint where the buffering is happening. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/57bcb92b-4e1f-4faa-aa4a-2aa75efd520e?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/57bcb92b-4e1f-4faa-aa4a-2aa75efd520e?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/57bcb92b-4e1f-4faa-aa4a-2aa75efd520e?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/57bcb92b-4e1f-4faa-aa4a-2aa75efd520e?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/57bcb92b-4e1f-4faa-aa4a-2aa75efd520e?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/57bcb92b-4e1f-4faa-aa4a-2aa75efd520e?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/57bcb92b-4e1f-4faa-aa4a-2aa75efd520e?feedback_type=other)</sup> [![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=This%20streaming%20delay%20isn%27t%20caused%20by%20Dify%20or%20its%20plugins%20buffering%20output%20by%20character%20count%E2%80%94there%27s%20no%20140-character%20buffer%20in%20the%20code.%20The%20most%20likely%20causes%20are%20buffering%20in%20your%20HTTP%20server%20%28Gunicorn%29%2C%20Docker%20networking%2C%20or%20a%20reverse%20proxy%20%28like%20Nginx%20or%20Traefik%29%20in%20your%20deployment%20stack.%20These%20layers%20can%20accumulate%20data%20before%20flushing%20it%20to%20the%20client%2C%20which%20explains%20why%20other%20clients%20%28like%20CherryStudio%29%20stream%20smoothly%20but%20Dify%20%28when%20running%20in%20Docker%20or%20behind%20a%20proxy%29%20does%20not%20%5B%5Bsource%5D%28https%3A//github.com/langgenius/dify-official-plugins/issues/1172%29%5D.%0A%0AHere%20are%20some%20concrete%20steps%20and%20workarounds%3A%0A%0A-%20Make%20sure%20your%20Gunicorn%20worker%20class%20is%20set%20to%20%60gevent%60%20and%20tune%20worker%20count%2C%20connections%2C%20and%20keep-alive%20settings%20for%20optimal%20streaming.%20For%20example%3A%20%20%0A%20%20%60gunicorn%20--worker-class%20gevent%20--workers%202%20--worker-connections%201000%20--timeout%20120%20app%3Aapp%60%20%20%0A%20%20This%20improves%20concurrency%20and%20reduces%20buffering%20%5B%5Bsource%5D%28https%3A//github.com/langgenius/dify/blob/eabdc5f0ebd806ede0c8f8695d54efb24e61748b/scripts/stress-test/README.md%23L101-L386%29%5D.%0A-%20If%20you%20use%20a%20reverse%20proxy%20%28Nginx%2C%20Traefik%29%2C%20disable%20buffering%20for%20streaming%20endpoints.%20For%20Nginx%2C%20use%20%60proxy_buffering%20off%3B%60%20for%20your%20%60/v1/chat/completions%60%20route.%0A-%20Try%20changing%20the%20%60stream_mode_delimiter%60%20in%20your%20Dify%20OpenAI-compatible%20model%20config%20from%20%60%5Cn%5Cn%60%20to%20%60%5Cn%60.%20This%20has%20resolved%20similar%20streaming%20issues%20for%20other%20users%20with%20FastAPI%20backends%20%5B%5Bsource%5D%28https%3A//github.com/langgenius/dify-official-plugins/issues/1858%29%5D.%0A-%20Test%20your%20FastAPI%20endpoint%20directly%20with%20%60curl%20--no-buffer%60%20or%20browser%20dev%20tools%20to%20confirm%20that%20it%20streams%20as%20expected%20outside%20Dify.%20If%20it%20does%2C%20the%20issue%20is%20almost%20certainly%20in%20the%20deployment%20stack%2C%20not%20your%20code.%0A-%20Review%20logs%20and%20network%20traces%20for%20signs%20of%20buffering%20or%20delayed%20chunk%20delivery.%0A%0AIf%20you%20share%20more%20about%20your%20deployment%20%28proxy%2C%20server%20config%2C%20Docker%20setup%29%2C%20I%20can%20help%20you%20pinpoint%20where%20the%20buffering%20is%20happening.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify-official-plugins/issues/2312)
yindo commented 2026-02-16 10:20:53 -05:00
Author
Owner
Copy Link
Copy Source

@ran411285752 commented on GitHub (Dec 26, 2025):

After the operation and maintenance engineer checked, it was because nginx did not close the buffer policy. After closing it, it returned to normal.

@ran411285752 commented on GitHub (Dec 26, 2025): After the operation and maintenance engineer checked, it was because nginx did not close the buffer policy. After closing it, it returned to normal.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
bug
dependencies
documentation
enhancement
good first issue
invalid
pull-request
Mirrored from GitHub Pull Request
 question
stale
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-official-plugins#894
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?