langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

TimeoutError / BlockingSwitchOutError in gevent-related callbacks #16255

New Issue
Closed
opened 2026-02-21 19:25:19 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 19:25:19 -05:00
Owner
Copy Link
Copy Source

Originally created by @AJinloser on GitHub (Aug 18, 2025).

Originally assigned to: @crazywoola on GitHub.

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

Dify 1.7.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I produce an app based on dify api, when I take many requests to dify api in the meanwhile, I rarely get situation that dify chatflow stuck on one node, keep running and there is no logs in the dify web. I check docker logs and find two bugs that may be cause this error.
I cannot reproduce this bug in specific steps, I only know it may happen frequently with many requests simultaneously

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/weakref.py", line 590, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1174, in _close_pool_connections
    conn = pool.get(block=False)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/queue.py", line 182, in get
    self.not_full.notify()
  File "/usr/local/lib/python3.12/threading.py", line 404, in notify
    if not self._is_owned():
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/threading.py", line 317, in _is_owned
    if self._lock.acquire(False):
       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/gevent/thread.py", line 296, in acquire
    sleep()
  File "/app/api/.venv/lib/python3.12/site-packages/gevent/hub.py", line 159, in sleep
    waiter.get()
  File "src/gevent/_waiter.py", line 143, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
TimeoutError: timed out

2025-08-15 19:30:22.263 ERROR [Dummy-207] [base.py:1009] - Exception during reset or similar
Traceback (most recent call last):
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 987, in _finalize_fairy
    fairy._reset(
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 1433, in _reset
    pool._dialect.do_rollback(self)
  File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 703, in do_rollback
    dbapi_connection.rollback()
  File "/app/api/.venv/lib/python3.12/site-packages/psycogreen/gevent.py", line 36, in gevent_wait_callback
    wait_read(conn.fileno(), timeout=timeout)
  File "src/gevent/_hub_primitives.py", line 353, in gevent._gevent_c_hub_primitives.wait_read
  File "src/gevent/_hub_primitives.py", line 367, in gevent._gevent_c_hub_primitives.wait_read
  File "src/gevent/_hub_primitives.py", line 350, in gevent._gevent_c_hub_primitives.wait_on_watcher
  File "src/gevent/_hub_primitives.py", line 304, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 64, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 67, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch_out
  File "src/gevent/_greenlet_primitives.py", line 68, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch_out
gevent.exceptions.BlockingSwitchOutError: Impossible to call blocking function in the event loop callback

and the web screenshot is like

Image

✔️ Expected Behavior

dify chatflow run smoothly or report log when it run error.

❌ Actual Behavior

dify chatflow keep running in one node without any error logs and api POST /chat-messages/:task_id/stop is useless.

Originally created by @AJinloser on GitHub (Aug 18, 2025). Originally assigned to: @crazywoola on GitHub. ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version Dify 1.7.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce I produce an app based on dify api, when I take many requests to dify api in the meanwhile, I rarely get situation that dify chatflow stuck on one node, keep running and there is no logs in the dify web. I check docker logs and find two bugs that may be cause this error. I cannot reproduce this bug in specific steps, I only know it may happen frequently with many requests simultaneously ``` Traceback (most recent call last): File "/usr/local/lib/python3.12/weakref.py", line 590, in __call__ return info.func(*info.args, **(info.kwargs or {})) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1174, in _close_pool_connections conn = pool.get(block=False) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/queue.py", line 182, in get self.not_full.notify() File "/usr/local/lib/python3.12/threading.py", line 404, in notify if not self._is_owned(): ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/threading.py", line 317, in _is_owned if self._lock.acquire(False): ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/gevent/thread.py", line 296, in acquire sleep() File "/app/api/.venv/lib/python3.12/site-packages/gevent/hub.py", line 159, in sleep waiter.get() File "src/gevent/_waiter.py", line 143, in gevent._gevent_c_waiter.Waiter.get File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch TimeoutError: timed out ``` ``` 2025-08-15 19:30:22.263 ERROR [Dummy-207] [base.py:1009] - Exception during reset or similar Traceback (most recent call last): File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 987, in _finalize_fairy fairy._reset( File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 1433, in _reset pool._dialect.do_rollback(self) File "/app/api/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 703, in do_rollback dbapi_connection.rollback() File "/app/api/.venv/lib/python3.12/site-packages/psycogreen/gevent.py", line 36, in gevent_wait_callback wait_read(conn.fileno(), timeout=timeout) File "src/gevent/_hub_primitives.py", line 353, in gevent._gevent_c_hub_primitives.wait_read File "src/gevent/_hub_primitives.py", line 367, in gevent._gevent_c_hub_primitives.wait_read File "src/gevent/_hub_primitives.py", line 350, in gevent._gevent_c_hub_primitives.wait_on_watcher File "src/gevent/_hub_primitives.py", line 304, in gevent._gevent_c_hub_primitives._primitive_wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 64, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 67, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch_out File "src/gevent/_greenlet_primitives.py", line 68, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch_out gevent.exceptions.BlockingSwitchOutError: Impossible to call blocking function in the event loop callback ``` and the web screenshot is like <img width="700" height="752" alt="Image" src="https://github.com/user-attachments/assets/712ba914-960d-41d6-b130-fec0cfe17549" /> ### ✔️ Expected Behavior dify chatflow run smoothly or report log when it run error. ### ❌ Actual Behavior dify chatflow keep running in one node without any error logs and api POST /chat-messages/:task_id/stop is useless.
yindo added the 🐞 bug label 2026-02-21 19:25:19 -05:00
yindo closed this issue 2026-02-21 19:25:19 -05:00
yindo commented 2026-02-21 19:25:19 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Aug 18, 2025):

https://github.com/langgenius/dify/issues/23654#issuecomment-3170082307

Things might be mitigated after this refactor https://github.com/langgenius/dify/pull/24068

@crazywoola commented on GitHub (Aug 18, 2025): https://github.com/langgenius/dify/issues/23654#issuecomment-3170082307 Things might be mitigated after this refactor https://github.com/langgenius/dify/pull/24068
yindo commented 2026-02-21 19:25:19 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 18, 2025):

Your issue is caused by conflicts between gevent's concurrency model and Dify's use of thread pools and synchronous libraries like urllib3 and SQLAlchemy. When running many simultaneous requests, gevent's monkey-patching can lead to thread contention, deadlocks, and errors such as TimeoutError and BlockingSwitchOutError—especially when blocking functions are called in the event loop context. This is a known problem when using SERVER_WORKER_CLASS=gevent with Dify; the recommended fix is to switch your worker class to uvicorn, which aligns better with Dify's concurrency model and avoids these deadlocks and errors (source).

The reason your chatflow gets stuck without error logs and the stop API is ineffective is due to how Dify handles task cancellation. The stop API sets a flag in Redis, but chatflow execution is cooperative: the running thread must periodically check this flag and raise a GenerateTaskStoppedError to halt execution. If the thread is blocked in a synchronous or deadlocked operation (like a stuck DB or HTTP call), it cannot check the flag, so the stop API can't interrupt it (source). This is a limitation of Python's threading and event loop model—true preemptive interruption isn't supported.

Recent updates in Dify (PR #20050, August 2025) introduced Celery-based asynchronous repositories for workflow execution, which offload database operations to background workers and may help reduce blocking and improve throughput (source). If possible, consider enabling these async repositories in your deployment.

Recommended actions:

  1. Change your worker class from gevent to uvicorn in your deployment configuration.
  2. If you are running into persistent blocking or deadlocks, review your model provider plugins for compatibility issues, as some plugins can cause connection persistence problems (source).
  3. Consider adopting the new Celery-based async repositories if available in your version.
  4. For better error visibility, ensure your logging configuration captures warnings and exceptions from all threads and greenlets, but be aware that some deep stack errors may still evade logs.

Let me know if you need help with the configuration steps or want more details on enabling Celery async repositories.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Aug 18, 2025): <!-- Answer --> Your issue is caused by conflicts between gevent's concurrency model and Dify's use of thread pools and synchronous libraries like urllib3 and SQLAlchemy. When running many simultaneous requests, gevent's monkey-patching can lead to thread contention, deadlocks, and errors such as `TimeoutError` and `BlockingSwitchOutError`—especially when blocking functions are called in the event loop context. This is a known problem when using `SERVER_WORKER_CLASS=gevent` with Dify; the recommended fix is to switch your worker class to `uvicorn`, which aligns better with Dify's concurrency model and avoids these deadlocks and errors ([source](https://github.com/langgenius/dify/issues/23223)). The reason your chatflow gets stuck without error logs and the stop API is ineffective is due to how Dify handles task cancellation. The stop API sets a flag in Redis, but chatflow execution is cooperative: the running thread must periodically check this flag and raise a `GenerateTaskStoppedError` to halt execution. If the thread is blocked in a synchronous or deadlocked operation (like a stuck DB or HTTP call), it cannot check the flag, so the stop API can't interrupt it ([source](https://github.com/langgenius/dify/blob/75199442c16fb2ad1758d1f813b0c9249fc150e9/api/core/app/apps/base_app_queue_manager.py)). This is a limitation of Python's threading and event loop model—true preemptive interruption isn't supported. Recent updates in Dify (PR #20050, August 2025) introduced Celery-based asynchronous repositories for workflow execution, which offload database operations to background workers and may help reduce blocking and improve throughput ([source](https://github.com/langgenius/dify/pull/20050)). If possible, consider enabling these async repositories in your deployment. **Recommended actions:** 1. Change your worker class from `gevent` to `uvicorn` in your deployment configuration. 2. If you are running into persistent blocking or deadlocks, review your model provider plugins for compatibility issues, as some plugins can cause connection persistence problems ([source](https://github.com/langgenius/dify/issues/23311)). 3. Consider adopting the new Celery-based async repositories if available in your version. 4. For better error visibility, ensure your logging configuration captures warnings and exceptions from all threads and greenlets, but be aware that some deep stack errors may still evade logs. Let me know if you need help with the configuration steps or want more details on enabling Celery async repositories. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/6ae9a6af-d281-4825-b114-dc7c0356f56b?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/6ae9a6af-d281-4825-b114-dc7c0356f56b?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/6ae9a6af-d281-4825-b114-dc7c0356f56b?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/6ae9a6af-d281-4825-b114-dc7c0356f56b?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/6ae9a6af-d281-4825-b114-dc7c0356f56b?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/6ae9a6af-d281-4825-b114-dc7c0356f56b?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/6ae9a6af-d281-4825-b114-dc7c0356f56b?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/24143)
yindo commented 2026-02-21 19:25:20 -05:00
Author
Owner
Copy Link
Copy Source

@AJinloser commented on GitHub (Aug 18, 2025):

#23654 (comment)

Things might be mitigated after this refactor #24068

should I pull this new branch or wait? I already pull main branch one week ago.

@AJinloser commented on GitHub (Aug 18, 2025): > [#23654 (comment)](https://github.com/langgenius/dify/issues/23654#issuecomment-3170082307) > > Things might be mitigated after this refactor [#24068](https://github.com/langgenius/dify/pull/24068) should I pull this new branch or wait? I already pull main branch one week ago.
yindo commented 2026-02-21 19:25:20 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Aug 18, 2025):

#23654 (comment)
Things might be mitigated after this refactor #24068

should I pull this new branch or wait? I already pull main branch one week ago.

This is a draft PR at this moment, we still need some time to test it out. Please wait for a bit. :)

@crazywoola commented on GitHub (Aug 18, 2025): > > [#23654 (comment)](https://github.com/langgenius/dify/issues/23654#issuecomment-3170082307) > > Things might be mitigated after this refactor [#24068](https://github.com/langgenius/dify/pull/24068) > > should I pull this new branch or wait? I already pull main branch one week ago. This is a draft PR at this moment, we still need some time to test it out. Please wait for a bit. :)
yindo commented 2026-02-21 19:25:20 -05:00
Author
Owner
Copy Link
Copy Source

@AJinloser commented on GitHub (Aug 18, 2025):

#23654 (comment)
Things might be mitigated after this refactor #24068

should I pull this new branch or wait? I already pull main branch one week ago.

This is a draft PR at this moment, we still need some time to test it out. Please wait for a bit. :)

Thank you for your reply, and thank you for your hard work on the development. I will wait patiently：）

@AJinloser commented on GitHub (Aug 18, 2025): > > > [#23654 (comment)](https://github.com/langgenius/dify/issues/23654#issuecomment-3170082307) > > > Things might be mitigated after this refactor [#24068](https://github.com/langgenius/dify/pull/24068) > > > > > > should I pull this new branch or wait? I already pull main branch one week ago. > > This is a draft PR at this moment, we still need some time to test it out. Please wait for a bit. :) Thank you for your reply, and thank you for your hard work on the development. I will wait patiently：）
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#16255
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?