langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-20 19:50:06 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[Cloud] Severe Latency Instability and Discrepancy between Node Execution Time and LLM Stats #21300

New Issue
Open
opened 2026-02-21 20:11:50 -05:00 by yindo · 8 comments
yindo commented 2026-02-21 20:11:50 -05:00
Owner
Copy Link
Copy Source

Originally created by @leozanee on GitHub (Dec 26, 2025).

Originally assigned to: @fatelei on GitHub.

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

v1.11.2

Cloud or Self Hosted

Cloud

Steps to reproduce

Description:
I am writing to report severe latency issues and statistical discrepancies experienced on the Cloud service. The performance is highly unstable even under identical testing conditions.

Details of the Issue:

1. High and Unstable Latency
I performed two consecutive tests in the exact same environment. The total execution time fluctuated significantly:

  • Test A: ~22 seconds
  • Test B: ~47 seconds
    This degree of variance makes the service unreliable for production use.

2. Abnormal Code Node Latency
I noticed that the latency for Code Nodes is unreasonably high.

  • Expected: ~200ms
  • Actual: Spiking to 5 seconds
    There were no changes to the code logic, yet the execution time increased drastically.

3. Discrepancy in LLM Node Statistics
There is a major inconsistency between the reported Node Execution Time and the internal LLM output statistics.

  • Node Execution Time (UI/Logs): ~30 seconds
  • LLM Output Metadata:
"latency": 9.08,
"time_to_first_token": 1.133,
"time_to_generate": 7.946
  • Problem: The LLM apparently finished generation in ~9 seconds, but the node held the process for 30 seconds. There is an unexplained ~21-second overhead/delay within the system processing that is not accounted for by the model generation itself.

Environment:

  • Platform: Dify Cloud Service
Image Image Image

✔️ Expected Behavior

  1. Consistent latency across identical runs.
  2. Code nodes should execute within milliseconds for simple logic.
  3. Node execution time should closely match the sum of internal processing + LLM generation time (latency).

❌ Actual Behavior

No response

Originally created by @leozanee on GitHub (Dec 26, 2025). Originally assigned to: @fatelei on GitHub. ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version v1.11.2 ### Cloud or Self Hosted Cloud ### Steps to reproduce **Description:** I am writing to report severe latency issues and statistical discrepancies experienced on the Cloud service. The performance is highly unstable even under identical testing conditions. **Details of the Issue:** **1. High and Unstable Latency** I performed two consecutive tests in the exact same environment. The total execution time fluctuated significantly: * **Test A:** ~22 seconds * **Test B:** ~47 seconds This degree of variance makes the service unreliable for production use. **2. Abnormal Code Node Latency** I noticed that the latency for `Code Nodes` is unreasonably high. * **Expected:** ~200ms * **Actual:** Spiking to **5 seconds** There were no changes to the code logic, yet the execution time increased drastically. **3. Discrepancy in LLM Node Statistics** There is a major inconsistency between the reported Node Execution Time and the internal LLM output statistics. * **Node Execution Time (UI/Logs):** ~30 seconds * **LLM Output Metadata:** ```json "latency": 9.08, "time_to_first_token": 1.133, "time_to_generate": 7.946 ``` * **Problem:** The LLM apparently finished generation in ~9 seconds, but the node held the process for 30 seconds. There is an unexplained ~21-second overhead/delay within the system processing that is not accounted for by the model generation itself. **Environment:** * **Platform:** Dify Cloud Service <img width="568" height="775" alt="Image" src="https://github.com/user-attachments/assets/e06ea7b4-28af-49f1-953d-4e8085c8bdb3" /> <img width="583" height="253" alt="Image" src="https://github.com/user-attachments/assets/705ec150-fdf0-49eb-988d-b3bd4a1d32c7" /> <img width="597" height="257" alt="Image" src="https://github.com/user-attachments/assets/442ef211-378b-4f45-9b83-239d9fa46181" /> ### ✔️ Expected Behavior 1. Consistent latency across identical runs. 2. Code nodes should execute within milliseconds for simple logic. 3. Node execution time should closely match the sum of internal processing + LLM generation time (latency). ### ❌ Actual Behavior _No response_
yindo added the 🐞 bugstalecloud labels 2026-02-21 20:11:50 -05:00
yindo commented 2026-02-21 20:11:50 -05:00
Author
Owner
Copy Link
Copy Source

@shenjiekoh commented on GitHub (Dec 26, 2025):

Same problem here, the chatbot consumes a lot of time to load before starting the node execution, please fix this issue ASAP

@shenjiekoh commented on GitHub (Dec 26, 2025): Same problem here, the chatbot consumes a lot of time to load before starting the node execution, please fix this issue ASAP
yindo commented 2026-02-21 20:11:50 -05:00
Author
Owner
Copy Link
Copy Source

@leozanee commented on GitHub (Jan 5, 2026):

Are there any updates on this bug? The latency is unacceptable and is severely affecting the customer experience. @laipz8200

Image
@leozanee commented on GitHub (Jan 5, 2026): Are there any updates on this bug? The latency is unacceptable and is severely affecting the customer experience. @laipz8200 <img width="571" height="715" alt="Image" src="https://github.com/user-attachments/assets/45357290-8cbd-4944-92ab-a367a5bd2b66" />
yindo commented 2026-02-21 20:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@fatelei commented on GitHub (Jan 5, 2026):

Self Checks

  • I have read the Contributing Guide and Language Policy.[x] This is only for bug report, if you would like to ask a question, please head to Discussions.[x] I have searched for existing issues search for existing issues, including closed ones.[x] I confirm that I am using English to submit this report, otherwise it will be closed.[x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）[x] Please do not modify this template :) and fill in all the required fields.

Dify version

v1.11.2

Cloud or Self Hosted

Cloud

Steps to reproduce

Description: I am writing to report severe latency issues and statistical discrepancies experienced on the Cloud service. The performance is highly unstable even under identical testing conditions.

Details of the Issue:

1. High and Unstable Latency I performed two consecutive tests in the exact same environment. The total execution time fluctuated significantly:

  • Test A: ~22 seconds
  • Test B: ~47 seconds
    This degree of variance makes the service unreliable for production use.

2. Abnormal Code Node Latency I noticed that the latency for Code Nodes is unreasonably high.

  • Expected: ~200ms
  • Actual: Spiking to 5 seconds
    There were no changes to the code logic, yet the execution time increased drastically.

3. Discrepancy in LLM Node Statistics There is a major inconsistency between the reported Node Execution Time and the internal LLM output statistics.

  • Node Execution Time (UI/Logs): ~30 seconds
  • LLM Output Metadata:

"latency": 9.08,
"time_to_first_token": 1.133,
"time_to_generate": 7.946

  • Problem: The LLM apparently finished generation in ~9 seconds, but the node held the process for 30 seconds. There is an unexplained ~21-second overhead/delay within the system processing that is not accounted for by the model generation itself.

Environment:

  • Platform: Dify Cloud Service

Image Image Image

✔️ Expected Behavior

  1. Consistent latency across identical runs.
  2. Code nodes should execute within milliseconds for simple logic.
  3. Node execution time should closely match the sum of internal processing + LLM generation time (latency).

❌ Actual Behavior

No response

thread join timeout = 10s

@fatelei commented on GitHub (Jan 5, 2026): > ### Self Checks > * [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).[x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).[x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.[x] I confirm that I am using English to submit this report, otherwise it will be closed.[x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）[x] Please do not modify this template :) and fill in all the required fields. > > ### Dify version > v1.11.2 > > ### Cloud or Self Hosted > Cloud > > ### Steps to reproduce > **Description:** I am writing to report severe latency issues and statistical discrepancies experienced on the Cloud service. The performance is highly unstable even under identical testing conditions. > > **Details of the Issue:** > > **1. High and Unstable Latency** I performed two consecutive tests in the exact same environment. The total execution time fluctuated significantly: > > * **Test A:** ~22 seconds > * **Test B:** ~47 seconds > This degree of variance makes the service unreliable for production use. > > **2. Abnormal Code Node Latency** I noticed that the latency for `Code Nodes` is unreasonably high. > > * **Expected:** ~200ms > * **Actual:** Spiking to **5 seconds** > There were no changes to the code logic, yet the execution time increased drastically. > > **3. Discrepancy in LLM Node Statistics** There is a major inconsistency between the reported Node Execution Time and the internal LLM output statistics. > > * **Node Execution Time (UI/Logs):** ~30 seconds > * **LLM Output Metadata:** > > "latency": 9.08, > "time_to_first_token": 1.133, > "time_to_generate": 7.946 > * **Problem:** The LLM apparently finished generation in ~9 seconds, but the node held the process for 30 seconds. There is an unexplained ~21-second overhead/delay within the system processing that is not accounted for by the model generation itself. > > **Environment:** > > * **Platform:** Dify Cloud Service > > <img alt="Image" width="568" height="775" src="https://private-user-images.githubusercontent.com/178130348/530290487-e06ea7b4-28af-49f1-953d-4e8085c8bdb3.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Njc2MTkzNzUsIm5iZiI6MTc2NzYxOTA3NSwicGF0aCI6Ii8xNzgxMzAzNDgvNTMwMjkwNDg3LWUwNmVhN2I0LTI4YWYtNDlmMS05NTNkLTRlODA4NWM4YmRiMy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjYwMTA1JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI2MDEwNVQxMzE3NTVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hNDVmYWQ1M2E1ZmIwNWNmZjgyNWU0OGI0MTc5OGRmZDg2MWM3YzMzZTA5ZjhhY2ZhODRhMDc5Yzc2YTlmYzg4JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.h0-TWAO9AZG2nGcEmM4isVIjuIW1QsdOWyp8gZh89pk"> <img alt="Image" width="583" height="253" src="https://private-user-images.githubusercontent.com/178130348/530290596-705ec150-fdf0-49eb-988d-b3bd4a1d32c7.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Njc2MTkzNzUsIm5iZiI6MTc2NzYxOTA3NSwicGF0aCI6Ii8xNzgxMzAzNDgvNTMwMjkwNTk2LTcwNWVjMTUwLWZkZjAtNDllYi05ODhkLWIzYmQ0YTFkMzJjNy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjYwMTA1JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI2MDEwNVQxMzE3NTVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yMjFmZDZiNzYxYjFjMjNlY2M5ZWRmZTRiZTVlYmQ1YmIxYjI5NDgxOWNhYTVkNjAzM2Q1ZTZkNjU5NjNiZDFiJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.CkV7VwMV8tTDUSLgMz0Ug0wccpgPc7zyPwAINjAkloo"> <img alt="Image" width="597" height="257" src="https://private-user-images.githubusercontent.com/178130348/530290683-442ef211-378b-4f45-9b83-239d9fa46181.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Njc2MTkzNzUsIm5iZiI6MTc2NzYxOTA3NSwicGF0aCI6Ii8xNzgxMzAzNDgvNTMwMjkwNjgzLTQ0MmVmMjExLTM3OGItNGY0NS05YjgzLTIzOWQ5ZmE0NjE4MS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjYwMTA1JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI2MDEwNVQxMzE3NTVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1jMzZjN2JkMWFmMjJiOTBiZjNjYzEyY2UzY2JlNjc0NDgyNzhhNDI4NzY4YmIwNDU2ZTQ0NDMwOTY3NjQ5Y2U4JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.BVRUizL7KkY6ToJ-ng0r75B7OdPbkyojBBdLjepL-xU"> > ### ✔️ Expected Behavior > 1. Consistent latency across identical runs. > 2. Code nodes should execute within milliseconds for simple logic. > 3. Node execution time should closely match the sum of internal processing + LLM generation time (latency). > > ### ❌ Actual Behavior > _No response_ thread join timeout = 10s
yindo commented 2026-02-21 20:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@fatelei commented on GitHub (Jan 6, 2026):

it has been optimized, the new version will decrease thread join timeout and unified the stop event

@fatelei commented on GitHub (Jan 6, 2026): it has been optimized, the new version will decrease thread join timeout and unified the stop event
yindo commented 2026-02-21 20:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@fatelei commented on GitHub (Jan 6, 2026):

you check in the next version

@fatelei commented on GitHub (Jan 6, 2026): you check in the next version
yindo commented 2026-02-21 20:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@shenjiekoh commented on GitHub (Jan 6, 2026):

@fatelei Around when will the new version be released on cloud?

@shenjiekoh commented on GitHub (Jan 6, 2026): @fatelei Around when will the new version be released on cloud?
yindo commented 2026-02-21 20:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@fatelei commented on GitHub (Jan 6, 2026):

@fatelei Around when will the new version be released on cloud?

yes, in the next version

@fatelei commented on GitHub (Jan 6, 2026): > [@fatelei](https://github.com/fatelei) Around when will the new version be released on cloud? yes, in the next version
yindo commented 2026-02-21 20:11:51 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 7, 2026):

Hi, @leozanee. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported severe and unstable latency issues on the Cloud service in version v1.11.2, including wide variations in execution times and abnormal delays in Code Node execution.
  • You also observed a significant discrepancy between Node Execution Time and LLM output latency, suggesting unexplained overhead.
  • Another user confirmed similar slow chatbot loading problems.
  • The maintainer responded that optimizations were made to reduce thread join timeout and unify stop events.
  • These fixes were expected to be included in the next Cloud version release.

What's Next:

  • Please let me know if you are still experiencing these latency issues with the latest version of Dify by commenting on this issue.
  • If I do not hear back within 15 days, I will automatically close this issue.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Feb 7, 2026): Hi, @leozanee. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported severe and unstable latency issues on the Cloud service in version v1.11.2, including wide variations in execution times and abnormal delays in Code Node execution. - You also observed a significant discrepancy between Node Execution Time and LLM output latency, suggesting unexplained overhead. - Another user confirmed similar slow chatbot loading problems. - The maintainer responded that optimizations were made to reduce thread join timeout and unify stop events. - These fixes were expected to be included in the next Cloud version release. **What's Next:** - Please let me know if you are still experiencing these latency issues with the latest version of Dify by commenting on this issue. - If I do not hear back within 15 days, I will automatically close this issue. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label cloud stale 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#21300
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?