langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Disconnection of network for workflow execution causes status stock as 'running' due to main thread kill while actual execution completes at backend thread #7758

New Issue
Closed
opened 2026-02-21 18:22:13 -05:00 by yindo · 15 comments
yindo commented 2026-02-21 18:22:13 -05:00
Owner
Copy Link
Copy Source

Originally created by @kazuhisa-wada on GitHub (Jan 16, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

v.0.15.1

Cloud or Self Hosted

Cloud, Self Hosted (Docker), Self Hosted (Source)

Steps to reproduce

Disconnecting network(*) between client and dify In the execution of workflow via API or web UI (both debug run and webApp run) kills main thread and then causes status stuck as 'running', while its execution, generated by main thread, actually completes. Once workflow falls into this situation, there is no way to stop the workflow execution e.g. stop API doesn't work.

Disconnecting networks here means as follows:

  • close browser or close a tab of browser in the execution of workflow via web UI(both debug run and webApp run)
  • killing a process for API calling application by any means

This issue then causes following bad behaviors:

  • there is no logging for that workflow execution on UI even if large tokens are consumed in the executions. It of course causes that users can't see any results of the executions. This is very confusing.
  • LLM tokens are not counted although workflow completes.

This issue seems serious as the actions which causes this issue can very easily be conducted and actually this issue frequently happens in our use.

I believe this is the result of not perfect handling of network disconnection which can cause main thread killing right now.

✔️ Expected Behavior

  • Regardless of network disconnection, actual workflow execution result should be logged in the database so uses can see the results if workflow normally ends.
  • Regardless of network disconnection, workflow execution can be stopped via workflow stop API

❌ Actual Behavior

Described above.

Originally created by @kazuhisa-wada on GitHub (Jan 16, 2025). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version v.0.15.1 ### Cloud or Self Hosted Cloud, Self Hosted (Docker), Self Hosted (Source) ### Steps to reproduce Disconnecting network(*) between client and dify In the execution of workflow via API or web UI (both debug run and webApp run) kills main thread and then causes status stuck as 'running', while its execution, generated by main thread, actually completes. Once workflow falls into this situation, there is no way to stop the workflow execution e.g. stop API doesn't work. **Disconnecting networks here means as follows:** - close browser or close a tab of browser in the execution of workflow via web UI(both debug run and webApp run) - killing a process for API calling application by any means This issue then causes following bad behaviors: - there is no logging for that workflow execution on UI even if large tokens are consumed in the executions. It of course causes that users can't see any results of the executions. This is very confusing. - LLM tokens are not counted although workflow completes. This issue seems serious as the actions which causes this issue can very easily be conducted and actually this issue frequently happens in our use. I believe this is the result of not perfect handling of network disconnection which can cause main thread killing right now. ### ✔️ Expected Behavior - Regardless of network disconnection, actual workflow execution result should be logged in the database so uses can see the results if workflow normally ends. - Regardless of network disconnection, workflow execution can be stopped via workflow stop API ### ❌ Actual Behavior Described above.
yindo added the 🌊 feat:workflow label 2026-02-21 18:22:13 -05:00
yindo closed this issue 2026-02-21 18:22:13 -05:00
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@kazuhisa-wada commented on GitHub (Jan 16, 2025):

I believe the root causes of this problem are as follows:

  1. db update when each node run finishes exists at main thread side. DB status update on each node run is trigerred once event message is received which is published by backend thread.
  2. main thread is killed when network disconnection happens(maybe TCP FIN is received)
  3. log info is not displayed in log tab on UI if status is still running. This spec is a bit strange and so it should be displayed separately from 1-2 above.
@kazuhisa-wada commented on GitHub (Jan 16, 2025): I believe the root causes of this problem are as follows: 1. db update when each node run finishes exists at main thread side. DB status update on each node run is trigerred once event message is received which is published by backend thread. 2. main thread is killed when network disconnection happens(maybe TCP FIN is received) 3. log info is not displayed in log tab on UI if status is still running. This spec is a bit strange and so it should be displayed separately from 1-2 above.
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@kazuhisa-wada commented on GitHub (Jan 16, 2025):

Sorry I forgot to mention about if stream or not for API case. What I tested was only streaming mode. I have not tested for non-streaming case but I believe it works well as non-streaming mode doesn't write anything on streaming channel.

@kazuhisa-wada commented on GitHub (Jan 16, 2025): Sorry I forgot to mention about if stream or not for API case. What I tested was only streaming mode. I have not tested for non-streaming case but I believe it works well as non-streaming mode doesn't write anything on streaming channel.
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@WeakX commented on GitHub (Jan 17, 2025):

I've ran into this issue too. Connection may be closed during stream resonding, then main thread will raise a GeneratorExit by flask

For the reason that the main thread is also responsible for DB operations, once it exits the workflow record(as well as workflow_node_execution record, message record, etc) will no longer be updated.

This only happens in streaming mode.

I think it will be better to decouple the db-operation and response from the main thread.

@WeakX commented on GitHub (Jan 17, 2025): I've ran into this issue too. Connection may be closed during stream resonding, then main thread will raise a GeneratorExit by flask For the reason that the main thread is also responsible for DB operations, once it exits the workflow record(as well as workflow_node_execution record, message record, etc) will no longer be updated. This only happens in streaming mode. I think it will be better to decouple the db-operation and response from the main thread.
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@kazuhisa-wada commented on GitHub (Jan 17, 2025):

Sounds good. Here is my understandings:

main thread

  1. create threads for node execution
  2. listen message from backend execution to receive event via python standard queue
  3. update the message when message is received
  4. write the response when message is received or ping event happens

backend thread

  1. execute node execution
  2. publish message(i.e. create event and throw it) via python standard queue

Better way is to move no.3 to additional other new thread.

@kazuhisa-wada commented on GitHub (Jan 17, 2025): Sounds good. Here is my understandings: **main thread** 1. create threads for node execution 2. listen message from backend execution to receive event via python standard queue 3. update the message when message is received 4. write the response when message is received or ping event happens **backend thread** 5. execute node execution 6. publish message(i.e. create event and throw it) via python standard queue **Better way is to move no.3 to additional other new thread.**
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@WeakX commented on GitHub (Jan 17, 2025):

Sounds good. Here is my understandings:

main thread

  1. create threads for node execution
  2. listen message from backend execution to receive event via python standard queue
  3. update the message when message is received
  4. write the response when message is received or ping event happens

backend thread

  1. execute node execution
  2. publish message(i.e. create event and throw it) via python standard queue

Better way is to move no.3 to additional other new thread.

agreed

@WeakX commented on GitHub (Jan 17, 2025): > Sounds good. Here is my understandings: > > **main thread** > > 1. create threads for node execution > 2. listen message from backend execution to receive event via python standard queue > 3. update the message when message is received > 4. write the response when message is received or ping event happens > > **backend thread** > > 5. execute node execution > 6. publish message(i.e. create event and throw it) via python standard queue > > **Better way is to move no.3 to additional other new thread.** agreed
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 16, 2025):

Hi, @kazuhisa-wada. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary:

  • Network disconnection during workflow execution causes the main thread to terminate.
  • Workflow status remains 'running' despite backend completion, with no logging or API control.
  • @WeakX confirmed the issue, noting it occurs in streaming mode due to main thread handling.
  • Suggested resolution: Decouple DB operations from the main thread.

Next Steps:

  • Is this issue still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open.
  • Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Feb 16, 2025): Hi, @kazuhisa-wada. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog. I'm marking this issue as stale. **Issue Summary:** - Network disconnection during workflow execution causes the main thread to terminate. - Workflow status remains 'running' despite backend completion, with no logging or API control. - @WeakX confirmed the issue, noting it occurs in streaming mode due to main thread handling. - Suggested resolution: Decouple DB operations from the main thread. **Next Steps:** - Is this issue still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open. - Otherwise, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@kazuhisa-wada commented on GitHub (Feb 17, 2025):

Is this issue still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open.

Definitely Yes. I also wanted to emphasize that this issue is a bit critical in terms of customer experience and so should be dealt with for v1.0.0 although this may be a difficult problem.

@kazuhisa-wada commented on GitHub (Feb 17, 2025): >Is this issue still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open. Definitely Yes. I also wanted to emphasize that this issue is a bit critical in terms of customer experience and so should be dealt with for v1.0.0 although this may be a difficult problem.
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 17, 2025):

@takatost, the user @kazuhisa-wada has confirmed that this issue is still relevant and critical for customer experience, especially for v1.0.0. Could you please assist them with this matter?

@dosubot[bot] commented on GitHub (Feb 17, 2025): @takatost, the user @kazuhisa-wada has confirmed that this issue is still relevant and critical for customer experience, especially for v1.0.0. Could you please assist them with this matter?
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@Zerglingzl commented on GitHub (Feb 25, 2025):

I encountered the same issue and sincerely hope that Dify can resolve it as soon as possible; otherwise, it will significantly impact our product’s user experience.

My situation might be somewhat unique, yet it makes reproducing the issue even easier—basically, it occurs over 50% of the time.

The business logic is as follows:
1. I use WeChat cloud functions to call Dify’s workflow API. Once the workflow starts running, to avoid waiting too long for a result, the cloud function exits and is terminated.
2. I use WeChat cloud functions to poll the workflow status periodically. If the status becomes “succeed,” I retrieve the result.

As a result, the problem mentioned in “thread #12798” keeps occurring, causing the workflow status to remain stuck on “running.” In reality, however, the workflow has completed (log analysis shows that the final step has produced results), and the tokens for the large model have already been consumed, which is extremely frustrating.

BTW, I'm using SaaS Dify 0.15.3.

@kazuhisa-wada @WeakX @takatost

@Zerglingzl commented on GitHub (Feb 25, 2025): I encountered the same issue and sincerely hope that Dify can resolve it as soon as possible; otherwise, it will significantly impact our product’s user experience. My situation might be somewhat unique, yet it makes reproducing the issue even easier—basically, it occurs over 50% of the time. The business logic is as follows: 1. I use WeChat cloud functions to call Dify’s workflow API. Once the workflow starts running, to avoid waiting too long for a result, the cloud function exits and is terminated. 2. I use WeChat cloud functions to poll the workflow status periodically. If the status becomes “succeed,” I retrieve the result. As a result, the problem mentioned in “thread #12798” keeps occurring, causing the workflow status to remain stuck on “running.” In reality, however, the workflow has completed (log analysis shows that the final step has produced results), and the tokens for the large model have already been consumed, which is extremely frustrating. BTW, I'm using SaaS Dify 0.15.3. @kazuhisa-wada @WeakX @takatost
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@kazuhisa-wada commented on GitHub (Feb 26, 2025):

Confirmation just in case, closed as it's been consolidated to #14362 ???

@kazuhisa-wada commented on GitHub (Feb 26, 2025): Confirmation just in case, closed as it's been consolidated to #14362 ???
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@Zerglingzl commented on GitHub (Feb 26, 2025):

so far as I experienced, the issue remains in Cloud version 0.15.3.

@Zerglingzl commented on GitHub (Feb 26, 2025): so far as I experienced, the issue remains in Cloud version 0.15.3.
yindo commented 2026-02-21 18:22:14 -05:00
Author
Owner
Copy Link
Copy Source

@Zerglingzl commented on GitHub (Feb 26, 2025):

For my understanding, "running" forever is not a good ending nevertheless. when the API caller END the connection (abnormal or in purperse), the workflow status should reflex the TURE state of workflow. which means, if it runs to the end (in fact it is, and tokens have been consumed), the status should be "succeed".

@Zerglingzl commented on GitHub (Feb 26, 2025): For my understanding, "running" forever is not a good ending nevertheless. when the API caller END the connection (abnormal or in purperse), the workflow status should reflex the TURE state of workflow. which means, if it runs to the end (in fact it is, and tokens have been consumed), the status should be "succeed".
yindo commented 2026-02-21 18:22:15 -05:00
Author
Owner
Copy Link
Copy Source

@Zerglingzl commented on GitHub (Feb 26, 2025):

update: I tried Ali cloud functions yesterday, unfortunately the issue REMAINS the same. it seems that this issue affects ALL BAAS type of backend caller, including weChat cloud functions, Ali cloud functions, Amazon cloud functions, etc. Our business stuck in that bug now, please fix it or any suggestions , thanks a lot!

@Zerglingzl commented on GitHub (Feb 26, 2025): update: I tried Ali cloud functions yesterday, unfortunately the issue REMAINS the same. it seems that this issue affects ALL BAAS type of backend caller, including weChat cloud functions, Ali cloud functions, Amazon cloud functions, etc. Our business stuck in that bug now, please fix it or any suggestions , thanks a lot!
yindo commented 2026-02-21 18:22:15 -05:00
Author
Owner
Copy Link
Copy Source

@Zerglingzl commented on GitHub (Feb 26, 2025):

update2: call back (when workflow is done) would be the perfect solution. Of cause that means a lot of work for dify engineer...

@Zerglingzl commented on GitHub (Feb 26, 2025): update2: call back (when workflow is done) would be the perfect solution. Of cause that means a lot of work for dify engineer...
yindo commented 2026-02-21 18:22:15 -05:00
Author
Owner
Copy Link
Copy Source

@kazuhisa-wada commented on GitHub (Mar 2, 2025):

@crazywoola
Could you please clarify why you marked this issue as closed ?
WIll this issue be resolved together with #14362 ?

@kazuhisa-wada commented on GitHub (Mar 2, 2025): @crazywoola Could you please clarify why you marked this issue as closed ? WIll this issue be resolved together with #14362 ?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🌊 feat:workflow
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7758
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?