langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Dify worker crash while processing documents for indexing in weaviate #782

New Issue
Closed
opened 2026-02-21 17:28:24 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 17:28:24 -05:00
Owner
Copy Link
Copy Source

Originally created by @perzeuss on GitHub (Dec 5, 2023).

Originally assigned to: @JohnJyong on GitHub.

Dify version

0.3.32

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Hint: I was not yet able to reproduce it on a non-k8s environment

✔️ Expected Behavior

  • The worker processes all documents successfully

❌ Actual Behavior

  • The worker crashes without any information (no logs, no sentry issues)
Originally created by @perzeuss on GitHub (Dec 5, 2023). Originally assigned to: @JohnJyong on GitHub. ### Dify version 0.3.32 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce - Setup dify in k8s using the helm chart from https://github.com/BorisPolonsky/dify-helm - Configure s3 storage - Create a Dataset and add multiple documents to it - Watch the worker crash *Hint: I was not yet able to reproduce it on a non-k8s environment* ### ✔️ Expected Behavior - The worker processes all documents successfully ### ❌ Actual Behavior - The worker crashes without any information (no logs, no sentry issues)
yindo added the 🐞 bug label 2026-02-21 17:28:24 -05:00
yindo closed this issue 2026-02-21 17:28:24 -05:00
yindo commented 2026-02-21 17:28:25 -05:00
Author
Owner
Copy Link
Copy Source

@perzeuss commented on GitHub (Dec 5, 2023):

Here are some debug logs from the worker process: https://gist.github.com/perzeuss/48bd4a8f8e88c43632704e22667c376c

And here is a screenshot of the dify console after the crash:
grafik

After restart of the worker, it did not resume the indexing.

@perzeuss commented on GitHub (Dec 5, 2023): Here are some debug logs from the worker process: https://gist.github.com/perzeuss/48bd4a8f8e88c43632704e22667c376c And here is a screenshot of the dify console after the crash: ![grafik](https://github.com/langgenius/dify/assets/11357019/46ad3657-dfec-4d0e-ae83-5291765be302) After restart of the worker, it did not resume the indexing.
yindo commented 2026-02-21 17:28:25 -05:00
Author
Owner
Copy Link
Copy Source

@perzeuss commented on GitHub (Dec 5, 2023):

Any hints about where I could add additional debug logging would be helpful. I already made some adjustments to the worker code for more logging, but that did not help to find the source of the problem.

@perzeuss commented on GitHub (Dec 5, 2023): Any hints about where I could add additional debug logging would be helpful. I already made some adjustments to the worker code for more logging, but that did not help to find the source of the problem.
yindo commented 2026-02-21 17:28:25 -05:00
Author
Owner
Copy Link
Copy Source

@perzeuss commented on GitHub (Dec 5, 2023):

While testing, I found another issue which I might have fixed by this change: https://github.com/langgenius/dify/commit/cd96d8ec57056e4e91f76cdd07f71bd0d195da21 let me know if you can confirm that's a bug.

@perzeuss commented on GitHub (Dec 5, 2023): While testing, I found another issue which I might have fixed by this change: https://github.com/langgenius/dify/commit/cd96d8ec57056e4e91f76cdd07f71bd0d195da21 let me know if you can confirm that's a bug.
yindo commented 2026-02-21 17:28:25 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Dec 6, 2023):

Your commit did fix a bug, but it probably wasn't the cause of the worker service crash.

@JohnJyong commented on GitHub (Dec 6, 2023): Your commit did fix a bug, but it probably wasn't the cause of the worker service crash.
yindo commented 2026-02-21 17:28:25 -05:00
Author
Owner
Copy Link
Copy Source

@perzeuss commented on GitHub (Dec 6, 2023):

The issue has been resolved after further analysis. The k8s pod was killed for reaching its memory limit (OOMKilled). Memory limits have been adjusted to prevent future occurrences.

I'll open a PR for fixing the small bug I found while investigating.

@perzeuss commented on GitHub (Dec 6, 2023): The issue has been resolved after further analysis. The k8s pod was killed for reaching its memory limit (OOMKilled). Memory limits have been adjusted to prevent future occurrences. I'll open a PR for fixing the small bug I found while investigating.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#782
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?