langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Workflow log lost when sse connection close #20541

New Issue
Closed
opened 2026-02-21 20:07:53 -05:00 by yindo · 4 comments
yindo commented 2026-02-21 20:07:53 -05:00
Owner
Copy Link
Copy Source

Originally created by @CrabSAMA on GitHub (Nov 25, 2025).

Originally assigned to: @laipz8200 on GitHub.

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.10.0

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

  1. run a workflow via api request, use streaming mode
  2. when request start response and not finish, refresh the page or manual abort the connect(in browser)
  3. workflow whole process will run normally(we can add log in code to observed), but workflow /logs nothing, the log of this request is lost

https://github.com/langgenius/dify/blob/eed38c8b2a89835b6b34d34963995781b36adde0/api/core/app/apps/workflow_app_runner.py#L347-L355

we can add log in here to observed, such as

Image

test record here(please ignore 01:10-01:37, first test maybe have some problem):

https://github.com/user-attachments/assets/43c5d352-9a4e-4046-93e0-29b3d920d51e

✔️ Expected Behavior

One request corresponding one log, if I request to run workflow, it will build one log in /logs.

❌ Actual Behavior

Corresponding log is lost.

Originally created by @CrabSAMA on GitHub (Nov 25, 2025). Originally assigned to: @laipz8200 on GitHub. ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.10.0 ### Cloud or Self Hosted Self Hosted (Source) ### Steps to reproduce 1. run a workflow via api request, use streaming mode 2. when request start response and not finish, refresh the page or manual abort the connect(in browser) 3. workflow whole process will run normally(we can add log in code to observed), but workflow `/logs` nothing, the log of this request is lost https://github.com/langgenius/dify/blob/eed38c8b2a89835b6b34d34963995781b36adde0/api/core/app/apps/workflow_app_runner.py#L347-L355 we can add log in here to observed, such as <img width="704" height="346" alt="Image" src="https://github.com/user-attachments/assets/873c6e36-e979-4ce0-8ff4-33fffae809c7" /> test record here(please ignore 01:10-01:37, first test maybe have some problem): https://github.com/user-attachments/assets/43c5d352-9a4e-4046-93e0-29b3d920d51e ### ✔️ Expected Behavior One request corresponding one log, if I request to run workflow, it will build one log in `/logs`. ### ❌ Actual Behavior Corresponding log is lost.
yindo added the 🐞 bug label 2026-02-21 20:07:53 -05:00
yindo closed this issue 2026-02-21 20:07:53 -05:00
yindo commented 2026-02-21 20:07:54 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Nov 25, 2025):

This is a known architectural limitation in Dify: when a client disconnects during a streaming (SSE) workflow run, the main thread handling the response is interrupted (often by a GeneratorExit), which prevents workflow status and logs from being updated or persisted—even though the workflow itself completes in the background. This means logs for that run are lost if the connection closes before completion, as you've observed. There is no built-in mechanism to flush or persist logs on disconnect in the current implementation, including in the relevant code section of workflow_app_runner.py (source).

A practical workaround is to replace yield from with a for loop in your streaming generator and explicitly handle exceptions like GeneratorExit. This allows you to perform cleanup and ensure logs and statuses are updated even after a disconnect (example and discussion). For a more robust solution, consider decoupling log and status updates from the streaming thread—using a message queue or worker system—so that logs are always persisted regardless of client connection state (architectural recommendation).

There is currently no merged fix for this in recent Dify releases, so manual workarounds or architectural changes are required to ensure logs are not lost on disconnect.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Nov 25, 2025): <!-- Answer --> This is a known architectural limitation in Dify: when a client disconnects during a streaming (SSE) workflow run, the main thread handling the response is interrupted (often by a `GeneratorExit`), which prevents workflow status and logs from being updated or persisted—even though the workflow itself completes in the background. This means logs for that run are lost if the connection closes before completion, as you've observed. There is no built-in mechanism to flush or persist logs on disconnect in the current implementation, including in the relevant code section of `workflow_app_runner.py` [(source)](https://github.com/langgenius/dify/blob/57b405c4c2724b63be8b82525a23c1575db26aef/api/core/app/apps/workflow_app_runner.py#L347-L573). A practical workaround is to replace `yield from` with a `for` loop in your streaming generator and explicitly handle exceptions like `GeneratorExit`. This allows you to perform cleanup and ensure logs and statuses are updated even after a disconnect [(example and discussion)](https://github.com/langgenius/dify/issues/20237#issuecomment-2909247859). For a more robust solution, consider decoupling log and status updates from the streaming thread—using a message queue or worker system—so that logs are always persisted regardless of client connection state [(architectural recommendation)](https://github.com/langgenius/dify/issues/23654#issuecomment-3169875252). There is currently no merged fix for this in recent Dify releases, so manual workarounds or architectural changes are required to ensure logs are not lost on disconnect. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/67f11ee1-c810-4b2e-97b8-bda08a7b3891?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/67f11ee1-c810-4b2e-97b8-bda08a7b3891?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/67f11ee1-c810-4b2e-97b8-bda08a7b3891?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/67f11ee1-c810-4b2e-97b8-bda08a7b3891?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/67f11ee1-c810-4b2e-97b8-bda08a7b3891?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/67f11ee1-c810-4b2e-97b8-bda08a7b3891?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/67f11ee1-c810-4b2e-97b8-bda08a7b3891?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=This%20is%20a%20known%20architectural%20limitation%20in%20Dify%3A%20when%20a%20client%20disconnects%20during%20a%20streaming%20%28SSE%29%20workflow%20run%2C%20the%20main%20thread%20handling%20the%20response%20is%20interrupted%20%28often%20by%20a%20%60GeneratorExit%60%29%2C%20which%20prevents%20workflow%20status%20and%20logs%20from%20being%20updated%20or%20persisted%E2%80%94even%20though%20the%20workflow%20itself%20completes%20in%20the%20background.%20This%20means%20logs%20for%20that%20run%20are%20lost%20if%20the%20connection%20closes%20before%20completion%2C%20as%20you%27ve%20observed.%20There%20is%20no%20built-in%20mechanism%20to%20flush%20or%20persist%20logs%20on%20disconnect%20in%20the%20current%20implementation%2C%20including%20in%20the%20relevant%20code%20section%20of%20%60workflow_app_runner.py%60%20%5B%28source%29%5D%28https%3A//github.com/langgenius/dify/blob/57b405c4c2724b63be8b82525a23c1575db26aef/api/core/app/apps/workflow_app_runner.py%23L347-L573%29.%0A%0AA%20practical%20workaround%20is%20to%20replace%20%60yield%20from%60%20with%20a%20%60for%60%20loop%20in%20your%20streaming%20generator%20and%20explicitly%20handle%20exceptions%20like%20%60GeneratorExit%60.%20This%20allows%20you%20to%20perform%20cleanup%20and%20ensure%20logs%20and%20statuses%20are%20updated%20even%20after%20a%20disconnect%20%5B%28example%20and%20discussion%29%5D%28https%3A//github.com/langgenius/dify/issues/20237%23issuecomment-2909247859%29.%20For%20a%20more%20robust%20solution%2C%20consider%20decoupling%20log%20and%20status%20updates%20from%20the%20streaming%20thread%E2%80%94using%20a%20message%20queue%20or%20worker%20system%E2%80%94so%20that%20logs%20are%20always%20persisted%20regardless%20of%20client%20connection%20state%20%5B%28architectural%20recommendation%29%5D%28https%3A//github.com/langgenius/dify/issues/23654%23issuecomment-3169875252%29.%0A%0AThere%20is%20currently%20no%20merged%20fix%20for%20this%20in%20recent%20Dify%20releases%2C%20so%20manual%20workarounds%20or%20architectural%20changes%20are%20required%20to%20ensure%20logs%20are%20not%20lost%20on%20disconnect.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28640)
yindo commented 2026-02-21 20:07:54 -05:00
Author
Owner
Copy Link
Copy Source

@CrabSAMA commented on GitHub (Nov 25, 2025):

@dosu Let me introduce my further analysis about this issue:

copliot help me analyze the running flow, like this:

┌─────────────────────────────────────────────────────────────────────────┐
│                           Main thread (HTTP response)                   │
│  WorkflowAppGenerateTaskPipeline._process_stream_response()             │
│  - Consume events from the queue                                        │
│  - Generate SSE response                                                │
│  - Invoke _save_workflow_app_log()                                      │
└─────────────────────────────────────────────────────────────────────────┘
                                    ↑ Consumer event
                                    │
                              [Queue]
                                    │
                                    ↓ Publish Event
┌─────────────────────────────────────────────────────────────────────────┐
│                          Worker thread (actual execution)               │
│  WorkflowAppRunner.run()                                                │
│  - Execute graph engine                                                 │
│  - Publish the event to the queue                                       │
└─────────────────────────────────────────────────────────────────────────┘

Now save workflow log in cosumer thread, when an error occurs in the message consumption, it will affect the writing of the log.I think it's a strange design to put the method of saving log together with message consumption. Essentially, the event has already run successfully, and it should be written to the log.

Maybe we should decouple the log writing from the message consumption? Just like move _save_workflow_app_log to publish event side, because runner know the real status of workflow or node, IMO it is ok.

@CrabSAMA commented on GitHub (Nov 25, 2025): @dosu Let me introduce my further analysis about this issue: copliot help me analyze the running flow, like this: ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Main thread (HTTP response) │ │ WorkflowAppGenerateTaskPipeline._process_stream_response() │ │ - Consume events from the queue │ │ - Generate SSE response │ │ - Invoke _save_workflow_app_log() │ └─────────────────────────────────────────────────────────────────────────┘ ↑ Consumer event │ [Queue] │ ↓ Publish Event ┌─────────────────────────────────────────────────────────────────────────┐ │ Worker thread (actual execution) │ │ WorkflowAppRunner.run() │ │ - Execute graph engine │ │ - Publish the event to the queue │ └─────────────────────────────────────────────────────────────────────────┘ ``` Now save workflow log in cosumer thread, when an error occurs in the message consumption, it will affect the writing of the log.I think it's a strange design to put the method of saving log together with message consumption. Essentially, the event has already run successfully, and it should be written to the log. Maybe we should decouple the log writing from the message consumption? Just like move `_save_workflow_app_log ` to publish event side, because runner know the real status of workflow or node, IMO it is ok.
yindo commented 2026-02-21 20:07:54 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Nov 25, 2025):

Your analysis is spot on and matches both the technical findings and community consensus. The current design—where log writing (_save_workflow_app_log) happens in the main (consumer) thread after queue event consumption—means that if the HTTP connection is interrupted (like a client disconnect), logs and status updates can be lost, even though the workflow itself finishes in the worker thread. This is a known architectural limitation and has been reported as a critical issue by others as well (discussion, another example).

Moving log writing to the publisher (worker) side is feasible. The _save_workflow_app_log method only needs a database session and workflow_run_id, both of which are available in the worker context. The worker thread knows the real status of the workflow and could persist logs immediately upon completion or failure, decoupling log persistence from the consumer and making it resilient to client disconnects (code reference).

If you go this route, you'll need to ensure the worker has access to the necessary context (tenant/app/user info, workflow_run_id) and a way to safely open a DB session. You might also want to update the event structure to carry any extra metadata needed for logging, or have the worker write logs directly after publishing the final event.

This change would align with best practices for queue-based architectures and address the log loss problem on disconnect. There’s no merged fix for this yet, but your suggestion is well-supported by both the code and community feedback.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Nov 25, 2025): <!-- Answer --> Your analysis is spot on and matches both the technical findings and community consensus. The current design—where log writing (_save_workflow_app_log) happens in the main (consumer) thread after queue event consumption—means that if the HTTP connection is interrupted (like a client disconnect), logs and status updates can be lost, even though the workflow itself finishes in the worker thread. This is a known architectural limitation and has been reported as a critical issue by others as well ([discussion](https://github.com/langgenius/dify/issues/12798), [another example](https://github.com/langgenius/dify/issues/26169)). Moving log writing to the publisher (worker) side is feasible. The _save_workflow_app_log method only needs a database session and workflow_run_id, both of which are available in the worker context. The worker thread knows the real status of the workflow and could persist logs immediately upon completion or failure, decoupling log persistence from the consumer and making it resilient to client disconnects ([code reference](https://github.com/langgenius/dify/blob/57b405c4c2724b63be8b82525a23c1575db26aef/api/core/app/apps/workflow/generate_task_pipeline.py#L399-L659)). If you go this route, you'll need to ensure the worker has access to the necessary context (tenant/app/user info, workflow_run_id) and a way to safely open a DB session. You might also want to update the event structure to carry any extra metadata needed for logging, or have the worker write logs directly after publishing the final event. This change would align with best practices for queue-based architectures and address the log loss problem on disconnect. There’s no merged fix for this yet, but your suggestion is well-supported by both the code and community feedback. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/d5708807-51dd-4bab-9b31-1e5de6468a1c?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/d5708807-51dd-4bab-9b31-1e5de6468a1c?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/d5708807-51dd-4bab-9b31-1e5de6468a1c?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/d5708807-51dd-4bab-9b31-1e5de6468a1c?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d5708807-51dd-4bab-9b31-1e5de6468a1c?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/d5708807-51dd-4bab-9b31-1e5de6468a1c?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/d5708807-51dd-4bab-9b31-1e5de6468a1c?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=Your%20analysis%20is%20spot%20on%20and%20matches%20both%20the%20technical%20findings%20and%20community%20consensus.%20The%20current%20design%E2%80%94where%20log%20writing%20%28_save_workflow_app_log%29%20happens%20in%20the%20main%20%28consumer%29%20thread%20after%20queue%20event%20consumption%E2%80%94means%20that%20if%20the%20HTTP%20connection%20is%20interrupted%20%28like%20a%20client%20disconnect%29%2C%20logs%20and%20status%20updates%20can%20be%20lost%2C%20even%20though%20the%20workflow%20itself%20finishes%20in%20the%20worker%20thread.%20This%20is%20a%20known%20architectural%20limitation%20and%20has%20been%20reported%20as%20a%20critical%20issue%20by%20others%20as%20well%20%28%5Bdiscussion%5D%28https%3A//github.com/langgenius/dify/issues/12798%29%2C%20%5Banother%20example%5D%28https%3A//github.com/langgenius/dify/issues/26169%29%29.%0A%0AMoving%20log%20writing%20to%20the%20publisher%20%28worker%29%20side%20is%20feasible.%20The%20_save_workflow_app_log%20method%20only%20needs%20a%20database%20session%20and%20workflow_run_id%2C%20both%20of%20which%20are%20available%20in%20the%20worker%20context.%20The%20worker%20thread%20knows%20the%20real%20status%20of%20the%20workflow%20and%20could%20persist%20logs%20immediately%20upon%20completion%20or%20failure%2C%20decoupling%20log%20persistence%20from%20the%20consumer%20and%20making%20it%20resilient%20to%20client%20disconnects%20%28%5Bcode%20reference%5D%28https%3A//github.com/langgenius/dify/blob/57b405c4c2724b63be8b82525a23c1575db26aef/api/core/app/apps/workflow/generate_task_pipeline.py%23L399-L659%29%29.%0A%0AIf%20you%20go%20this%20route%2C%20you%27ll%20need%20to%20ensure%20the%20worker%20has%20access%20to%20the%20necessary%20context%20%28tenant/app/user%20info%2C%20workflow_run_id%29%20and%20a%20way%20to%20safely%20open%20a%20DB%20session.%20You%20might%20also%20want%20to%20update%20the%20event%20structure%20to%20carry%20any%20extra%20metadata%20needed%20for%20logging%2C%20or%20have%20the%20worker%20write%20logs%20directly%20after%20publishing%20the%20final%20event.%0A%0AThis%20change%20would%20align%20with%20best%20practices%20for%20queue-based%20architectures%20and%20address%20the%20log%20loss%20problem%20on%20disconnect.%20There%E2%80%99s%20no%20merged%20fix%20for%20this%20yet%2C%20but%20your%20suggestion%20is%20well-supported%20by%20both%20the%20code%20and%20community%20feedback.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28640)
yindo commented 2026-02-21 20:07:54 -05:00
Author
Owner
Copy Link
Copy Source

@CrabSAMA commented on GitHub (Nov 27, 2025):

Maybe close with https://github.com/langgenius/dify/pull/28701.

@CrabSAMA commented on GitHub (Nov 27, 2025): Maybe close with https://github.com/langgenius/dify/pull/28701.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#20541
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?