langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 20:45:26 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Knowledge Base Embedding Indexing Process Stops Unexpectedly in Version 0.10.2 #6516

New Issue
Closed
opened 2026-02-21 18:16:05 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 18:16:05 -05:00
Owner
Copy Link
Copy Source

Originally created by @lukefan on GitHub (Nov 2, 2024).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.10.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Create a new knowledge base.
Upload 300 PDF documents from the internet in one go.
While indexing, add another 300 PDF documents in two separate batches.
Use the SiliconFlow NetEase embedding model for embeddings.
weaviate vector database

✔️ Expected Behavior

finish all progress.

❌ Actual Behavior

After some time, the indexing process stops. At this point:

Some documents are successfully indexed.
Some documents are skipped.
Others remain in the queue.
If I manually stop and restart the indexing process for documents stuck in the "indexing" status, they complete indexing normally.

Originally created by @lukefan on GitHub (Nov 2, 2024). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.10.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Create a new knowledge base. Upload 300 PDF documents from the internet in one go. While indexing, add another 300 PDF documents in two separate batches. Use the SiliconFlow NetEase embedding model for embeddings. weaviate vector database ### ✔️ Expected Behavior finish all progress. ### ❌ Actual Behavior After some time, the indexing process stops. At this point: Some documents are successfully indexed. Some documents are skipped. Others remain in the queue. If I manually stop and restart the indexing process for documents stuck in the "indexing" status, they complete indexing normally.
yindo added the 👻 feat:rag label 2026-02-21 18:16:05 -05:00
yindo closed this issue 2026-02-21 18:16:05 -05:00
yindo commented 2026-02-21 18:16:05 -05:00
Author
Owner
Copy Link
Copy Source

@lukefan commented on GitHub (Nov 2, 2024):

Suggested Solution:
Please consider implementing a feature that allows manual management of the document status within the knowledge base. This would help in cases where certain documents get stuck during the indexing process.

@lukefan commented on GitHub (Nov 2, 2024): Suggested Solution: Please consider implementing a feature that allows manual management of the document status within the knowledge base. This would help in cases where certain documents get stuck during the indexing process.
yindo commented 2026-02-21 18:16:05 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Nov 2, 2024):

The issue you're experiencing with the indexing process stopping unexpectedly in Dify version 0.10.2 could be related to several factors, including batch size limits and error handling mechanisms. Here are some insights and suggestions:

  1. Batch Upload Limit: Dify has a batch upload limit that, if exceeded, can cause issues with the indexing process. Ensure that the number of documents being processed does not exceed this limit [1].

  2. Error Handling: The indexing process includes error handling mechanisms that update the document's status to "error" if issues occur. This might be why some documents remain in the queue or are skipped [2].

  3. Manual Management of Document Status: Dify does support manual management of document statuses, including disabling and archiving documents. This can help manage documents that get stuck during the indexing process [3][4].

  4. Retry Mechanism: If documents are stuck in the "indexing" status, manually stopping and restarting the process, as you've done, can help complete the indexing. This suggests that implementing a more robust retry mechanism could be beneficial.

Consider reviewing the batch size and ensuring it aligns with the configured limits. Additionally, utilizing the manual management features can help address documents that get stuck. If the problem persists, further investigation into the specific error logs might be necessary to identify underlying issues.

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Nov 2, 2024): <!-- Answer --> The issue you're experiencing with the indexing process stopping unexpectedly in Dify version 0.10.2 could be related to several factors, including batch size limits and error handling mechanisms. Here are some insights and suggestions: 1. **Batch Upload Limit**: Dify has a batch upload limit that, if exceeded, can cause issues with the indexing process. Ensure that the number of documents being processed does not exceed this limit <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/tasks/document_indexing_task.py)</sup>. 2. **Error Handling**: The indexing process includes error handling mechanisms that update the document's status to "error" if issues occur. This might be why some documents remain in the queue or are skipped <sup>[[2]](https://github.com/langgenius/dify/blob/main/api/core/indexing_runner.py)</sup>. 3. **Manual Management of Document Status**: Dify does support manual management of document statuses, including disabling and archiving documents. This can help manage documents that get stuck during the indexing process <sup>[[3]](https://github.com/langgenius/dify-docs/blob/main/jp/guides/knowledge-base/knowledge-and-documents-maintenance.md)</sup><sup>[[4]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/maintain-dataset-via-api.md)</sup>. 4. **Retry Mechanism**: If documents are stuck in the "indexing" status, manually stopping and restarting the process, as you've done, can help complete the indexing. This suggests that implementing a more robust retry mechanism could be beneficial. Consider reviewing the batch size and ensuring it aligns with the configured limits. Additionally, utilizing the manual management features can help address documents that get stuck. If the problem persists, further investigation into the specific error logs might be necessary to identify underlying issues. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:16:05 -05:00
Author
Owner
Copy Link
Copy Source

@lukefan commented on GitHub (Nov 3, 2024):

I have been using the knowledge base feature and have encountered some issues. After uploading a large number of documents to the knowledge base, especially when multiple people are using it simultaneously, it is difficult to avoid indexing errors.
Currently, although it is possible to pause and resume documents one by one, this is unacceptable for a knowledge base with thousands of documents. I have resorted to using the dify API to read the document status and then use puppeteer to automatically pause and then resume all documents that are pending in the indexing state. However, this process is also unacceptable.
Therefore, I request a status check and reset function for the entire knowledge base. This would greatly improve the usability and efficiency of managing large knowledge bases.
Also, I would like to ask for some advice. What is a good limit for the number of documents in a knowledge base? For example, if there are 10,000 documents, should they be placed in one knowledge base or divided into 10 knowledge bases?
Looking forward to your response and solutions.

@lukefan commented on GitHub (Nov 3, 2024): I have been using the knowledge base feature and have encountered some issues. After uploading a large number of documents to the knowledge base, especially when multiple people are using it simultaneously, it is difficult to avoid indexing errors. Currently, although it is possible to pause and resume documents one by one, this is unacceptable for a knowledge base with thousands of documents. I have resorted to using the dify API to read the document status and then use puppeteer to automatically pause and then resume all documents that are pending in the indexing state. However, this process is also unacceptable. Therefore, I request a status check and reset function for the entire knowledge base. This would greatly improve the usability and efficiency of managing large knowledge bases. Also, I would like to ask for some advice. What is a good limit for the number of documents in a knowledge base? For example, if there are 10,000 documents, should they be placed in one knowledge base or divided into 10 knowledge bases? Looking forward to your response and solutions.
yindo commented 2026-02-21 18:16:05 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Nov 8, 2024):

could you pls provide error logs of the unexpected stop?
emmm, we have the retry funtion in the knowledge
image
@lukefan

@JohnJyong commented on GitHub (Nov 8, 2024): could you pls provide error logs of the unexpected stop? emmm, we have the retry funtion in the knowledge ![image](https://github.com/user-attachments/assets/dfe67f70-ccff-47d9-8484-f0c7c1b45bc4) @lukefan
yindo commented 2026-02-21 18:16:05 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Nov 12, 2024):

If you have any doubts， pls reopen the issue ， thanks

@JohnJyong commented on GitHub (Nov 12, 2024): If you have any doubts， pls reopen the issue ， thanks
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 👻 feat:rag
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#6516
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?