langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-27 11:11:21 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

When using Milvus as a vector database and uploading files to the knowledge base, the dify worker container seems to be hanging on #12665

New Issue
Closed
opened 2026-02-21 19:08:27 -05:00 by yindo · 7 comments
yindo commented 2026-02-21 19:08:27 -05:00
Owner
Copy Link
Copy Source

Originally created by @YatogamiKid on GitHub (Apr 8, 2025).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.1.3 && 0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

When using Milvus as a vector database and uploading files to the knowledge base, the dify worker container seems to be hanging on. After debugging I find out that the worker hanged on since it goes to create a client from pymilvus's MilvusClient in milvus_vector.py. Even I set the timeout for 5 seconds, it still hang there. The code and logs are going down here.

Image

Image

Then I tried to execute the same code to create a MilvusClient in the worker container directly, it works.

Image

So I was confused why it couldn't work in dify worker. Then I tried to using Dify v0.15.3, it still has the same problem. This time i tried to use Milvus instead of MilvusClient to see if the client could be created successfully or not in milvus_vector.py though the interfaces of such two different objects are totally different.
Here's the logs that I tired in Dify v0.15.3 as the pymilvus version in 0.15.3 is 2.5.3 that has the Milvus object that in 2.5.5 has been removed. It seems to be created successfully.

Image

At the end I tried to create the connections directly in milvus_vector.py which is executed when initialized MilvusClient in pymilvus. Just to find out wheather the connection has been created or not. Then I got logs going down here, which caused the exception.

Image

Image

Is it a bug when creating MilvusClient that the dify worker hang on there?

✔️ Expected Behavior

Files which uploaded should be successfully insert into knowledge base.
The dify worker should be worked well instead of hang on there.

❌ Actual Behavior

The dify worker hang on there and could not get any response or logs.

Originally created by @YatogamiKid on GitHub (Apr 8, 2025). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.1.3 && 0.15.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce When using Milvus as a vector database and uploading files to the knowledge base, the dify worker container seems to be hanging on. After debugging I find out that the worker hanged on since it goes to create a client from **pymilvus's MilvusClient** in **milvus_vector.py**. Even I set the timeout for 5 seconds, it still hang there. The code and logs are going down here. ![Image](https://github.com/user-attachments/assets/dab7e16e-b05f-4808-96b1-c4af7f75ef2f) ![Image](https://github.com/user-attachments/assets/736d7b48-65ba-4948-af32-5b514333ebf0) Then I tried to execute the same code to create a MilvusClient in the worker container directly, it works. ![Image](https://github.com/user-attachments/assets/8dc9a04e-9668-4ee2-b83d-c7a42b3649df) So I was confused why it couldn't work in dify worker. Then I tried to using Dify v0.15.3, it still has the same problem. This time i tried to use Milvus instead of MilvusClient to see if the client could be created successfully or not in **milvus_vector.py** though the interfaces of such two different objects are totally different. Here's the logs that I tired in Dify v0.15.3 as the pymilvus version in 0.15.3 is 2.5.3 that has the Milvus object that in 2.5.5 has been removed. It seems to be created successfully. ![Image](https://github.com/user-attachments/assets/5a84af61-429a-4060-ad2b-80bde2096575) At the end I tried to create the connections directly in milvus_vector.py which is executed when initialized MilvusClient in pymilvus. Just to find out wheather the connection has been created or not. Then I got logs going down here, which caused the exception. ![Image](https://github.com/user-attachments/assets/0f268c62-623b-4e71-b695-ebd6fafae740) ![Image](https://github.com/user-attachments/assets/5451972b-a06c-4edb-a5c0-0b8d0e24ab43) Is it a bug when creating MilvusClient that the dify worker hang on there? ### ✔️ Expected Behavior Files which uploaded should be successfully insert into knowledge base. The dify worker should be worked well instead of hang on there. ### ❌ Actual Behavior The dify worker hang on there and could not get any response or logs.
yindo added the 🐞 bug👻 feat:ragstale labels 2026-02-21 19:08:27 -05:00
yindo closed this issue 2026-02-21 19:08:27 -05:00
yindo commented 2026-02-21 19:08:27 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 8, 2025):

Hello @YatogamiKid! I'm here to help you with any bugs or questions you might have while you wait for a human maintainer. If you need assistance, feel free to ask!

I found a similar unsolved discussion titled "can't connect to the vector database" which might be related to your issue. It was last updated on April 1, 2025, and is still open [1]. You might find some insights or additional troubleshooting steps in that thread.

Regarding your issue, it seems like there might be a problem with the asynchronous gRPC setup when initializing the MilvusClient in the Dify worker. Since you mentioned that executing the same code directly in the worker container works, it could be related to how the Dify worker handles the event loop or manages asynchronous operations. You might want to check if there are any differences in the environment or configurations between the direct execution and the Dify worker execution. Additionally, ensure that all dependencies and versions are consistent across both setups.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 8, 2025): <!-- Greeting --> Hello @YatogamiKid! I'm here to help you with any bugs or questions you might have while you wait for a human maintainer. If you need assistance, feel free to ask! <!-- Answer --> I found a similar unsolved discussion titled "can't connect to the vector database" which might be related to your issue. It was last updated on April 1, 2025, and is still open <sup>[[1]](https://github.com/langgenius/dify/discussions/17278)</sup>. You might find some insights or additional troubleshooting steps in that thread. Regarding your issue, it seems like there might be a problem with the asynchronous gRPC setup when initializing the `MilvusClient` in the Dify worker. Since you mentioned that executing the same code directly in the worker container works, it could be related to how the Dify worker handles the event loop or manages asynchronous operations. You might want to check if there are any differences in the environment or configurations between the direct execution and the Dify worker execution. Additionally, ensure that all dependencies and versions are consistent across both setups. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/4e83029f-9b2b-4b60-992a-e70883e3c5da?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/4e83029f-9b2b-4b60-992a-e70883e3c5da?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/4e83029f-9b2b-4b60-992a-e70883e3c5da?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/4e83029f-9b2b-4b60-992a-e70883e3c5da?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/4e83029f-9b2b-4b60-992a-e70883e3c5da?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/4e83029f-9b2b-4b60-992a-e70883e3c5da?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/4e83029f-9b2b-4b60-992a-e70883e3c5da?feedback_type=other)</sup>
yindo commented 2026-02-21 19:08:27 -05:00
Author
Owner
Copy Link
Copy Source

@YatogamiKid commented on GitHub (Apr 13, 2025):

Hi @JohnJyong & @crazywoola, is there any updating on this issue？Thanks a lot!

@YatogamiKid commented on GitHub (Apr 13, 2025): Hi @JohnJyong & @crazywoola, is there any updating on this issue？Thanks a lot!
yindo commented 2026-02-21 19:08:27 -05:00
Author
Owner
Copy Link
Copy Source

@vxzhong commented on GitHub (May 23, 2025):

Add api and worker to the network milvus，then set MILVUS_URI=http://milvus-standalone:19530。
Otherwise，Api service cannot access milvus port, and host.docker.internal is not valid for Linux docker.

@vxzhong commented on GitHub (May 23, 2025): Add `api `and `worker `to the network `milvus`，then set `MILVUS_URI=http://milvus-standalone:19530`。 Otherwise，Api service cannot access milvus port, and `host.docker.internal` is not valid for Linux docker.
yindo commented 2026-02-21 19:08:28 -05:00
Author
Owner
Copy Link
Copy Source

@labslibs commented on GitHub (Jun 29, 2025):

I also have encountered such a problem. Could you please tell me if there is a solution?
The local Milvus is working normally, but this problem occurs whenever connecting to the cloud-based Milvus provided by Alibaba.

When accessing the remote Milvus vector database, the underlying library used by Dify might have I/O blocking operations. After switching the gevent model to the prefork model, this problem was resolved.

@labslibs commented on GitHub (Jun 29, 2025): I also have encountered such a problem. Could you please tell me if there is a solution? The local Milvus is working normally, but this problem occurs whenever connecting to the cloud-based Milvus provided by Alibaba. When accessing the remote Milvus vector database, the underlying library used by Dify might have I/O blocking operations. After switching the gevent model to the prefork model, this problem was resolved.
yindo commented 2026-02-21 19:08:28 -05:00
Author
Owner
Copy Link
Copy Source

@Rajendrazepto commented on GitHub (Aug 1, 2025):

I am getting the same issue while using the Milvus vector DB with gevent for the Celery job worker.

Temporary fix to change env variable:

CELERY_WORKER_CLASS=prefork

Permanent fix:

Using AsyncMilvusClient() might fix the issue. https://milvus.io/docs/use-async-milvus-client-with-asyncio.md

Why Prefork Solves the Issue?

🔄 Gevent Problem:
Monkey patching replaces Python's standard networking/threading
gRPC background threads in Milvus conflict with gevent's cooperative scheduling
Shared memory space means all threads are affected by monkey patching
🚀 Prefork Solution:
Separate processes instead of threads/greenlets
Isolated memory spaces - each worker process is independent
No monkey patching conflicts - gRPC runs normally in each process
True parallelism instead of cooperative multitasking

Logs:

`_2025-07-31 04:36:29,806.806 DEBUG [ThreadPoolExecutor-3_0] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "GET /plugin/x/management/models?page=1&page_size=256 HTTP/1.1" 200 None
2025-07-31 04:36:29,792.792 DEBUG [ThreadPoolExecutor-3_0] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "POST /plugin/x/dispatch/text_embedding/num_tokens HTTP/1.1" 200 None

2025-07-31 04:36:29,736.736 DEBUG [Dummy-5] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "POST /plugin/x/dispatch/text_embedding/num_tokens HTTP/1.1" 200 None
2025-07-31 04:36:29,541.541 DEBUG [Dummy-5] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "GET /plugin/x/management/models?page=1&page_size=256 HTTP/1.1" 200 None

host:dify-svc-bucket.s3.ap-north.amazonaws.com
/upload_files/x/x.txt_

2025-07-31 04:36:29,327.327 INFO [Dummy-5] [document_indexing_task.py:63] - Start process document: 0667d900--

2025-07-31 04:36:29,246.246 DEBUG [MainThread] [base.py:149] - TaskPool: Apply <function fast_trace_task at 0x7fde28be0ea0> (args:('tasks.document_indexing_task.document_indexing_task', '0667d900--c2211f675309'...}
2025-07-31 04:36:29,246.246 INFO [MainThread] [strategy.py:161] - Task tasks.document_indexing_task.document_indexing_task[1653c6a3--*-57abf5] received**

2025-07-31 04:33:09,546.546 INFO [Dummy-4] [gossip.py:145] - missed heartbeat from celery@ZLMP90

2025-07-31 04:32:57,692.692 DEBUG [MainThread] [gossip.py:137] - celery@ZLMP90 joined the party`

@Rajendrazepto commented on GitHub (Aug 1, 2025): I am getting the same issue while using the `Milvus` vector DB with `gevent` for the Celery job worker. ### Temporary fix to change env variable: `CELERY_WORKER_CLASS=prefork` ### Permanent fix: Using AsyncMilvusClient() might fix the issue. https://milvus.io/docs/use-async-milvus-client-with-asyncio.md ### Why Prefork Solves the Issue? 🔄 Gevent Problem: Monkey patching replaces Python's standard networking/threading gRPC background threads in Milvus conflict with gevent's cooperative scheduling Shared memory space means all threads are affected by monkey patching 🚀 Prefork Solution: Separate processes instead of threads/greenlets Isolated memory spaces - each worker process is independent No monkey patching conflicts - gRPC runs normally in each process True parallelism instead of cooperative multitasking ### Logs: `_2025-07-31 04:36:29,806.806 DEBUG [ThreadPoolExecutor-3_0] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "GET /plugin/x/management/models?page=1&page_size=256 HTTP/1.1" 200 None 2025-07-31 04:36:29,792.792 DEBUG [ThreadPoolExecutor-3_0] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "POST /plugin/x/dispatch/text_embedding/num_tokens HTTP/1.1" 200 None 2025-07-31 04:36:29,736.736 DEBUG [Dummy-5] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "POST /plugin/x/dispatch/text_embedding/num_tokens HTTP/1.1" 200 None 2025-07-31 04:36:29,541.541 DEBUG [Dummy-5] [connectionpool.py:544] - http://dify-plugin-daemon:5002 "GET /plugin/x/management/models?page=1&page_size=256 HTTP/1.1" 200 None host:dify-svc-bucket.s3.ap-north.amazonaws.com /upload_files/x/x.txt_ **2025-07-31 04:36:29,327.327 INFO [Dummy-5] [document_indexing_task.py:63] - Start process document: 0667d900--** 2025-07-31 04:36:29,246.246 DEBUG [MainThread] [base.py:149] - TaskPool: Apply <function fast_trace_task at 0x7fde28be0ea0> (args:('tasks.document_indexing_task.document_indexing_task', '0667d900-****-c2211f675309'...} **2025-07-31 04:36:29,246.246 INFO [MainThread] [strategy.py:161] - Task tasks.document_indexing_task.document_indexing_task[1653c6a3-***-****-57abf5] received** 2025-07-31 04:33:09,546.546 INFO [Dummy-4] [gossip.py:145] - missed heartbeat from celery@ZLMP90 2025-07-31 04:32:57,692.692 DEBUG [MainThread] [gossip.py:137] - celery@ZLMP90 joined the party`
yindo commented 2026-02-21 19:08:28 -05:00
Author
Owner
Copy Link
Copy Source

@Rajendrazepto commented on GitHub (Aug 1, 2025):

Hi @JohnJyong & @crazywoola , Anyone working on it or open to pick this issue?

@Rajendrazepto commented on GitHub (Aug 1, 2025): Hi @JohnJyong & @crazywoola , Anyone working on it or open to pick this issue?
yindo commented 2026-02-21 19:08:28 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 1, 2025):

Hi, @YatogamiKid. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported that the Dify worker container hangs when creating a MilvusClient via pymilvus in a self-hosted Docker setup using Milvus as the vector database.
  • The hang blocks file uploads and appears related to conflicts between gRPC and gevent's cooperative scheduling in the worker.
  • Temporary workarounds include setting CELERY_WORKER_CLASS=prefork or using AsyncMilvusClient for async support.
  • Other users have confirmed similar issues and noted that switching from gevent to prefork resolves the problem.
  • The issue remains unresolved and maintainers have not yet provided a response.

Next Steps:

  • Please let me know if this issue is still relevant with the latest version of Dify by commenting here to keep the discussion open.
  • Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Sep 1, 2025): Hi, @YatogamiKid. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported that the Dify worker container hangs when creating a MilvusClient via pymilvus in a self-hosted Docker setup using Milvus as the vector database. - The hang blocks file uploads and appears related to conflicts between gRPC and gevent's cooperative scheduling in the worker. - Temporary workarounds include setting `CELERY_WORKER_CLASS=prefork` or using AsyncMilvusClient for async support. - Other users have confirmed similar issues and noted that switching from gevent to prefork resolves the problem. - The issue remains unresolved and maintainers have not yet provided a response. **Next Steps:** - Please let me know if this issue is still relevant with the latest version of Dify by commenting here to keep the discussion open. - Otherwise, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label stale 🐞 bug 👻 feat:rag
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#12665
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?