langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

A large amount of data in the knowledge base is in indexed and queued states #7865

New Issue
Closed
opened 2026-02-21 18:22:45 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 18:22:45 -05:00
Owner
Copy Link
Copy Source

Originally created by @KingPuiWong on GitHub (Jan 23, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.13.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

After uploading a batch of knowledge base documents in batches, it was found that many of them were in the queue. After switching the Embeeding model, the newly uploaded documents were still in the queue, and some old data were in the index, and the number in the index would not be reduced.

The values ​​of the error field in the document table of pgsql are all null, and there is no error message.

Image

Image

Image

✔️ Expected Behavior

I want to know what causes this and how I should deal with it.
The vector database used is VikingDB.

❌ Actual Behavior

No response

Originally created by @KingPuiWong on GitHub (Jan 23, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.13.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce After uploading a batch of knowledge base documents in batches, it was found that many of them were in the queue. After switching the Embeeding model, the newly uploaded documents were still in the queue, and some old data were in the index, and the number in the index would not be reduced. The values ​​of the error field in the document table of pgsql are all null, and there is no error message. ![Image](https://github.com/user-attachments/assets/b629ca77-a400-4bb7-9c0e-4d8a5eb62097) ![Image](https://github.com/user-attachments/assets/c43ce551-2c1f-4791-9366-9d0cae0a1ff9) ![Image](https://github.com/user-attachments/assets/0bed1782-30b6-476e-8c57-5b06286083a5) ### ✔️ Expected Behavior I want to know what causes this and how I should deal with it. The vector database used is VikingDB. ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 18:22:45 -05:00
yindo closed this issue 2026-02-21 18:22:45 -05:00
yindo commented 2026-02-21 18:22:46 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 23, 2025):

I found a similar issue that might be relevant to your situation. In a closed issue, it was suggested to use a production-oriented startup command with gevent for the Celery worker, which might help if the problem persists:

celery -A app.celery worker -P gevent -c 1 --loglevel INFO

Regarding your current issue with documents being stuck in 'waiting' or 'indexing' states, several factors could be contributing to this:

  1. Rate Limit: Ensure that the rate limit for the Embedding model API key is not being exceeded, as this can delay processing.

  2. Celery Worker Service: Verify that the Celery worker service is running properly. You can start it with docker compose up -d worker and check the logs for any errors.

  3. Service Status: Make sure all necessary services (api, worker, web, weaviate, db, redis, nginx) are running. Use docker compose ps to confirm that all services are 'Up'.

  4. Pending Processes: Check for any pending segmentation or embedding processes. Ensure that all dependencies, such as the database and Redis, are functioning correctly to avoid bottlenecks.

  5. Indexing Technique and Process Rules: The choice of indexing technique and process rules can affect processing time. For example, using a "high_quality" indexing technique might take longer.

  6. Switching Embedding Models: When you switch the embedding model, all documents need to be re-embedded, which can increase the queue size and processing time [1][2].

These steps should help you diagnose and address the issue with documents being stuck in the queue or indexing states.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Jan 23, 2025): <!-- Answer --> I found a similar issue that might be relevant to your situation. In a closed issue, it was suggested to use a production-oriented startup command with gevent for the Celery worker, which might help if the problem persists: ```bash celery -A app.celery worker -P gevent -c 1 --loglevel INFO ``` Regarding your current issue with documents being stuck in 'waiting' or 'indexing' states, several factors could be contributing to this: 1. **Rate Limit**: Ensure that the rate limit for the Embedding model API key is not being exceeded, as this can delay processing. 2. **Celery Worker Service**: Verify that the Celery worker service is running properly. You can start it with `docker compose up -d worker` and check the logs for any errors. 3. **Service Status**: Make sure all necessary services (api, worker, web, weaviate, db, redis, nginx) are running. Use `docker compose ps` to confirm that all services are 'Up'. 4. **Pending Processes**: Check for any pending segmentation or embedding processes. Ensure that all dependencies, such as the database and Redis, are functioning correctly to avoid bottlenecks. 5. **Indexing Technique and Process Rules**: The choice of indexing technique and process rules can affect processing time. For example, using a "high_quality" indexing technique might take longer. 6. **Switching Embedding Models**: When you switch the embedding model, all documents need to be re-embedded, which can increase the queue size and processing time <sup>[[1]](https://github.com/langgenius/dify/issues/11657)</sup><sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/knowledge-and-documents-maintenance.md)</sup>. These steps should help you diagnose and address the issue with documents being stuck in the queue or indexing states. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/4f5adb1c-9db3-4c48-830a-da8c97b87d1f?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/4f5adb1c-9db3-4c48-830a-da8c97b87d1f?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/4f5adb1c-9db3-4c48-830a-da8c97b87d1f?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/4f5adb1c-9db3-4c48-830a-da8c97b87d1f?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/4f5adb1c-9db3-4c48-830a-da8c97b87d1f?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/4f5adb1c-9db3-4c48-830a-da8c97b87d1f?feedback_type=other)</sup>
yindo commented 2026-02-21 18:22:46 -05:00
Author
Owner
Copy Link
Copy Source

@KingPuiWong commented on GitHub (Jan 23, 2025):

I have restarted dify-worker. After the restart, the number of indexings has been reduced to the number before the restart and will not continue to decrease. Other services are running normally.

@KingPuiWong commented on GitHub (Jan 23, 2025): I have restarted dify-worker. After the restart, the number of indexings has been reduced to the number before the restart and will not continue to decrease. Other services are running normally.
yindo commented 2026-02-21 18:22:46 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Jan 23, 2025):

Please upgrade to latest version.

@crazywoola commented on GitHub (Jan 23, 2025): Please upgrade to latest version.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7865
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?