langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-19 08:35:20 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

doc of knowledge base indexing is very slow #12194

New Issue
Closed
opened 2026-02-21 19:06:18 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 19:06:18 -05:00
Owner
Copy Link
Copy Source

Originally created by @AlexiaChen on GitHub (Mar 26, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.0.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Use use APIs to upload to knowledge base

✔️ Expected Behavior

This uploading progress should not be such slow.

❌ Actual Behavior

most of Docs always is queueing not completed when doc uploading to knowledge base . and some of errors. only a small part of docs is available.

Image

Image

Originally created by @AlexiaChen on GitHub (Mar 26, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.0.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Use use APIs to upload to knowledge base ### ✔️ Expected Behavior This uploading progress should not be such slow. ### ❌ Actual Behavior most of Docs always is queueing not completed when doc uploading to knowledge base . and some of errors. only a small part of docs is available. ![Image](https://github.com/user-attachments/assets/ec96b095-37f4-4522-a607-7ddd7f74cdb5) ![Image](https://github.com/user-attachments/assets/bfacb1e9-bda9-4fd8-8c51-ba8e1b96de99)
yindo added the 🐞 bug label 2026-02-21 19:06:18 -05:00
yindo closed this issue 2026-02-21 19:06:18 -05:00
yindo commented 2026-02-21 19:06:18 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 26, 2025):

@AlexiaChen Hello there! I'm here to help you with bug reports, answer questions, and guide you to become a contributor. If you need any assistance while waiting for a human maintainer, feel free to ask.

I found some relevant discussions and issues that might help address the slow document indexing problem you're experiencing with Dify:

  1. Closed Issue: A similar issue was reported where documents were perpetually queued. The suggested solutions included ensuring the Celery worker service is running correctly, checking the rate limit for the Embedding model API key, and verifying that all necessary services are up and running [1].

  2. Open Issue: Another open issue suggests splitting Celery tasks to optimize performance and ensuring the Celery worker is started and running correctly. It also recommends checking the system embedding models and ensuring the .env file is correctly configured [2].

  3. Closed Issue: It was suggested to upgrade to the latest version of Dify, restart the worker container, and perform a full cleanup and restart. Additionally, checking the connection between the worker container and your instance, and reviewing the worker container logs for errors, were recommended [3].

To address your issue, consider the following steps:

  • Upgrade Dify: Ensure you are using the latest version, as updates may contain fixes for known issues.
  • Check Celery Worker: Make sure the Celery worker service is running properly. Use docker compose up -d worker to start it and check the logs for any errors.
  • Verify Services: Use docker compose ps to confirm that all services (api, worker, web, weaviate, db, redis, nginx) are up.
  • Review Indexing Process: Check for any exceptions or errors logged during the indexing process, especially in the _extract, _transform, or _load methods of the IndexingRunner class [4].
  • Batch Upload Limit: Ensure the number of documents being uploaded does not exceed the default batch upload limit of 50 pages [5].

These steps should help improve the document indexing speed and resolve errors in your self-hosted Dify environment.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 26, 2025): <!-- Greeting --> @AlexiaChen Hello there! I'm here to help you with bug reports, answer questions, and guide you to become a contributor. If you need any assistance while waiting for a human maintainer, feel free to ask. <!-- Answer --> I found some relevant discussions and issues that might help address the slow document indexing problem you're experiencing with Dify: 1. **Closed Issue**: A similar issue was reported where documents were perpetually queued. The suggested solutions included ensuring the Celery worker service is running correctly, checking the rate limit for the Embedding model API key, and verifying that all necessary services are up and running <sup>[[1]](https://github.com/langgenius/dify/issues/11911)</sup>. 2. **Open Issue**: Another open issue suggests splitting Celery tasks to optimize performance and ensuring the Celery worker is started and running correctly. It also recommends checking the system embedding models and ensuring the `.env` file is correctly configured <sup>[[2]](https://github.com/langgenius/dify/issues/16178)</sup>. 3. **Closed Issue**: It was suggested to upgrade to the latest version of Dify, restart the worker container, and perform a full cleanup and restart. Additionally, checking the connection between the worker container and your instance, and reviewing the worker container logs for errors, were recommended <sup>[[3]](https://github.com/langgenius/dify/issues/15312)</sup>. To address your issue, consider the following steps: - **Upgrade Dify**: Ensure you are using the latest version, as updates may contain fixes for known issues. - **Check Celery Worker**: Make sure the Celery worker service is running properly. Use `docker compose up -d worker` to start it and check the logs for any errors. - **Verify Services**: Use `docker compose ps` to confirm that all services (api, worker, web, weaviate, db, redis, nginx) are up. - **Review Indexing Process**: Check for any exceptions or errors logged during the indexing process, especially in the `_extract`, `_transform`, or `_load` methods of the `IndexingRunner` class <sup>[[4]](https://github.com/langgenius/dify/blob/main/api/core/indexing_runner.py)</sup>. - **Batch Upload Limit**: Ensure the number of documents being uploaded does not exceed the default batch upload limit of 50 pages <sup>[[5]](https://github.com/langgenius/dify/issues/11659)</sup>. These steps should help improve the document indexing speed and resolve errors in your self-hosted Dify environment. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/c2dc3950-4f62-48cf-80e7-1cd92dd5cfa0?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/c2dc3950-4f62-48cf-80e7-1cd92dd5cfa0?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/c2dc3950-4f62-48cf-80e7-1cd92dd5cfa0?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/c2dc3950-4f62-48cf-80e7-1cd92dd5cfa0?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/c2dc3950-4f62-48cf-80e7-1cd92dd5cfa0?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/c2dc3950-4f62-48cf-80e7-1cd92dd5cfa0?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/c2dc3950-4f62-48cf-80e7-1cd92dd5cfa0?feedback_type=other)</sup>
yindo commented 2026-02-21 19:06:18 -05:00
Author
Owner
Copy Link
Copy Source

@GuYunWei commented on GitHub (Mar 28, 2025):

have the same issue in v1.1.3

@GuYunWei commented on GitHub (Mar 28, 2025): have the same issue in v1.1.3
yindo commented 2026-02-21 19:06:18 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 28, 2025):

Hi, @AlexiaChen. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported slow document indexing in a self-hosted Dify version 1.0.0 setup.
  • Most documents remain queued and incomplete.
  • I suggested troubleshooting steps like upgrading Dify and checking the Celery worker service.
  • GuYunWei mentioned experiencing the same issue in version 1.1.3, indicating persistence in newer versions.

Next Steps:

  • Please let us know if this issue is still relevant to the latest version of the Dify repository by commenting here.
  • If no further updates are provided, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Apr 28, 2025): Hi, @AlexiaChen. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported slow document indexing in a self-hosted Dify version 1.0.0 setup. - Most documents remain queued and incomplete. - I suggested troubleshooting steps like upgrading Dify and checking the Celery worker service. - GuYunWei mentioned experiencing the same issue in version 1.1.3, indicating persistence in newer versions. **Next Steps:** - Please let us know if this issue is still relevant to the latest version of the Dify repository by commenting here. - If no further updates are provided, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
yindo commented 2026-02-21 19:06:18 -05:00
Author
Owner
Copy Link
Copy Source

@smy3321 commented on GitHub (Jun 22, 2025):

Same. indexing speed of 1.4.3 is almost 100 time slower than 0.15.2

@smy3321 commented on GitHub (Jun 22, 2025): Same. indexing speed of 1.4.3 is almost 100 time slower than 0.15.2
yindo commented 2026-02-21 19:06:19 -05:00
Author
Owner
Copy Link
Copy Source

@shenrenguo commented on GitHub (Jun 27, 2025):

Same. 1.5.0

Task tasks.document_indexing_task.document_indexing_task[bec59fcf-fcb3-4d8f-aa77-60278ebf758b] succeeded in 21.60580722987652s: None

@shenrenguo commented on GitHub (Jun 27, 2025): Same. 1.5.0 Task tasks.document_indexing_task.document_indexing_task[bec59fcf-fcb3-4d8f-aa77-60278ebf758b] succeeded in 21.60580722987652s: None
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#12194
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?