langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[Bug] Documents stuck in "waiting" status when RAG Pipeline is enabled (runtime_mode = 'rag_pipeline') #19960

New Issue
Closed
opened 2026-02-21 20:05:08 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 20:05:08 -05:00
Owner
Copy Link
Copy Source

Originally created by @JackyX1996 on GitHub (Oct 27, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.9.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Steps

  1. Create a new knowledge base
  2. Enable Pipeline / Retrieval Settings for the knowledge base
  3. Upload any document to this knowledge base
  4. Document gets stuck in "Queuing" status forever

Root Cause
File: /app/api/services/dataset_service.py Line: ~559
python

if dataset.runtime_mode != "rag_pipeline":
    document_indexing_task.delay(dataset_id, [document.id])

When pipeline is enabled, runtime_mode is set to 'rag_pipeline', causing the condition to be False. The indexing task is never triggered.

Database Evidence
All waiting documents have runtime_mode = 'rag_pipeline':
sql

SELECT d.runtime_mode, doc.indexing_status, COUNT(*)
FROM datasets d
JOIN documents doc ON d.id = doc.dataset_id
WHERE doc.indexing_status = 'waiting'
GROUP BY d.runtime_mode, doc.indexing_status;

Result: All have runtime_mode = 'rag_pipeline'
Successfully indexed documents have runtime_mode = 'general' or were uploaded before enabling pipeline.

✔️ Expected Behavior

Documents should be indexed automatically after upload, regardless of whether Pipeline is enabled or not.

❌ Actual Behavior

  • Documents remain in "Queuing/Waiting" status forever
  • No indexing task is sent to Celery worker (verified by checking Redis queues and worker logs)
  • Documents cannot be used for retrieval
  • No error message shown to user

Verification
Worker is functioning correctly - when manually triggering the task, it processes documents successfully:
python

document_indexing_task.apply_async(
    args=[dataset_id, [document_id]],
    queue='dataset'
)

Works perfectly - proves the issue is the task not being sent

Suggested Fix
Remove the conditional check in line ~559:
python

# Change from:
if dataset.runtime_mode != "rag_pipeline":
    document_indexing_task.delay(dataset_id, [document.id])

# To:
document_indexing_task.delay(dataset_id, [document.id])
Or implement proper pipeline-specific indexing logic if different handling is intended.

Workaround
Manually trigger indexing via Python script:
python

from app_factory import create_app
app = create_app()
with app.app_context():
    from tasks.document_indexing_task import document_indexing_task
    from models.dataset import Document
    from extensions.ext_database import db
    
    waiting_docs = db.session.query(Document).filter(
        Document.indexing_status == 'waiting'
    ).all()
    
    for doc in waiting_docs:
        document_indexing_task.apply_async(
            args=[str(doc.dataset_id), [str(doc.id)]],
            queue='dataset'
        )

Impact

Severity: High - Pipeline feature completely unusable
Affected: All users who enable Pipeline/Retrieval Settings
Workaround: Requires database access

Originally created by @JackyX1996 on GitHub (Oct 27, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.9.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce **Steps** 1. Create a new knowledge base 2. Enable Pipeline / Retrieval Settings for the knowledge base 3. Upload any document to this knowledge base 4. Document gets stuck in "Queuing" status forever **Root Cause** File: /app/api/services/dataset_service.py Line: ~559 python ``` if dataset.runtime_mode != "rag_pipeline": document_indexing_task.delay(dataset_id, [document.id]) ``` When pipeline is enabled, runtime_mode is set to 'rag_pipeline', causing the condition to be False. The indexing task is never triggered. **Database Evidence** All waiting documents have runtime_mode = 'rag_pipeline': sql ``` SELECT d.runtime_mode, doc.indexing_status, COUNT(*) FROM datasets d JOIN documents doc ON d.id = doc.dataset_id WHERE doc.indexing_status = 'waiting' GROUP BY d.runtime_mode, doc.indexing_status; ``` Result: All have runtime_mode = 'rag_pipeline' Successfully indexed documents have runtime_mode = 'general' or were uploaded before enabling pipeline. ### ✔️ Expected Behavior Documents should be indexed automatically after upload, regardless of whether Pipeline is enabled or not. ### ❌ Actual Behavior - Documents remain in "Queuing/Waiting" status forever - No indexing task is sent to Celery worker (verified by checking Redis queues and worker logs) - Documents cannot be used for retrieval - No error message shown to user **Verification** Worker is functioning correctly - when manually triggering the task, it processes documents successfully: python ``` document_indexing_task.apply_async( args=[dataset_id, [document_id]], queue='dataset' ) ``` # Works perfectly - proves the issue is the task not being sent Suggested Fix Remove the conditional check in line ~559: python ``` # Change from: if dataset.runtime_mode != "rag_pipeline": document_indexing_task.delay(dataset_id, [document.id]) # To: document_indexing_task.delay(dataset_id, [document.id]) Or implement proper pipeline-specific indexing logic if different handling is intended. ``` Workaround Manually trigger indexing via Python script: python ``` from app_factory import create_app app = create_app() with app.app_context(): from tasks.document_indexing_task import document_indexing_task from models.dataset import Document from extensions.ext_database import db waiting_docs = db.session.query(Document).filter( Document.indexing_status == 'waiting' ).all() for doc in waiting_docs: document_indexing_task.apply_async( args=[str(doc.dataset_id), [str(doc.id)]], queue='dataset' ) ``` **Impact** Severity: High - Pipeline feature completely unusable Affected: All users who enable Pipeline/Retrieval Settings Workaround: Requires database access
yindo added the 🐞 bug label 2026-02-21 20:05:08 -05:00
yindo closed this issue 2026-02-21 20:05:08 -05:00
yindo commented 2026-02-21 20:05:08 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Oct 27, 2025):

Hi @JackyX1996! I'm Dosu and I’m helping the dify team.

This bug is caused by a conditional check in the code that prevents document indexing tasks from being triggered when runtime_mode is set to 'rag_pipeline', leaving documents stuck in "waiting" status and unusable for retrieval. There is no alternative indexing logic for pipeline mode in the affected code section, so removing the check will allow standard indexing to trigger and resolve the issue, but be mindful of possible side effects if pipeline-specific logic is added elsewhere in the future. As a workaround, manually triggering the indexing task via Python works.

A recent merged pull request has addressed a related queuing issue by updating Celery worker queue configurations to include priority_pipeline, which should help resolve stuck tasks in newer versions. For persistent session errors in pipeline indexing, upgrading to Dify 2.0.0 beta or newer is recommended, as it includes fixes for session management and pipeline document processing. If you cannot upgrade, ensure you only pass primary key IDs between pipeline steps and always re-query ORM objects inside a new session context.

For more details, see:

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Oct 27, 2025): <!-- Greeting --> Hi @JackyX1996! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the dify team. <!-- Answer --> This bug is caused by a conditional check in the code that prevents document indexing tasks from being triggered when `runtime_mode` is set to `'rag_pipeline'`, leaving documents stuck in "waiting" status and unusable for retrieval. There is no alternative indexing logic for pipeline mode in the affected code section, so removing the check will allow standard indexing to trigger and resolve the issue, but be mindful of possible side effects if pipeline-specific logic is added elsewhere in the future. As a workaround, manually triggering the indexing task via Python works. A recent merged pull request has addressed a related queuing issue by updating Celery worker queue configurations to include `priority_pipeline`, which should help resolve stuck tasks in newer versions. For persistent session errors in pipeline indexing, upgrading to Dify 2.0.0 beta or newer is recommended, as it includes fixes for session management and pipeline document processing. If you cannot upgrade, ensure you only pass primary key IDs between pipeline steps and always re-query ORM objects inside a new session context. For more details, see: - [Conditional check root cause and code context](https://github.com/langgenius/dify/blob/dc1ae57dc669f4dbce1daa08b37f2e78068d5074/api/services/dataset_service.py) - [Celery queue fix PR #27416](https://github.com/langgenius/dify/pull/27416) - [Pipeline session management and upgrade guidance](https://github.com/langgenius/dify/discussions/27329#discussioncomment-14758516) <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/bfa1db4e-9289-43fe-b354-ba6905d35483?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/bfa1db4e-9289-43fe-b354-ba6905d35483?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/bfa1db4e-9289-43fe-b354-ba6905d35483?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/bfa1db4e-9289-43fe-b354-ba6905d35483?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/bfa1db4e-9289-43fe-b354-ba6905d35483?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/bfa1db4e-9289-43fe-b354-ba6905d35483?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/bfa1db4e-9289-43fe-b354-ba6905d35483?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/27515)
yindo commented 2026-02-21 20:05:08 -05:00
Author
Owner
Copy Link
Copy Source

@water-in-stone commented on GitHub (Nov 9, 2025):

It seems this issue has been resolved in the pull request at https://github.com/langgenius/dify/pull/27416/files#diff-c7b1c23e00a1fd0378f6ea41a7dfe1c9e043d766f738dba26860c783b82e8c6a

@water-in-stone commented on GitHub (Nov 9, 2025): It seems this issue has been resolved in the pull request at https://github.com/langgenius/dify/pull/27416/files#diff-c7b1c23e00a1fd0378f6ea41a7dfe1c9e043d766f738dba26860c783b82e8c6a
yindo commented 2026-02-21 20:05:08 -05:00
Author
Owner
Copy Link
Copy Source

@huixiangyang commented on GitHub (Nov 12, 2025):

It seems this issue has been resolved in the pull request at https://github.com/langgenius/dify/pull/27416/files#diff-c7b1c23e00a1fd0378f6ea41a7dfe1c9e043d766f738dba26860c783b82e8c6a

What is the pattern for releasing docker images? I pulled an image and deployed a service today, but found that this PR hasn't been released to the docker image yet.

@huixiangyang commented on GitHub (Nov 12, 2025): > It seems this issue has been resolved in the pull request at https://github.com/langgenius/dify/pull/27416/files#diff-c7b1c23e00a1fd0378f6ea41a7dfe1c9e043d766f738dba26860c783b82e8c6a What is the pattern for releasing docker images? I pulled an image and deployed a service today, but found that this PR hasn't been released to the docker image yet.
yindo commented 2026-02-21 20:05:09 -05:00
Author
Owner
Copy Link
Copy Source

@ChillPilot commented on GitHub (Dec 27, 2025):

@JackyX1996 This may help you！！ #30237

@ChillPilot commented on GitHub (Dec 27, 2025): @JackyX1996 This may help you！！ #30237
yindo commented 2026-02-21 20:05:09 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 27, 2026):

Hi, @JackyX1996. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported a bug in version 1.9.2 where documents uploaded with runtime_mode='rag_pipeline' get stuck in "waiting" because indexing tasks do not trigger.
  • The root cause was identified as a conditional check blocking indexing in pipeline mode.
  • Workarounds included manual indexing and upgrading to version 2.0.0 beta where the fix is implemented.
  • PR #27416 addressed the issue, but the fix may not yet be available in the Docker image.
  • Related issue #30237 might provide additional context or help.

Next Steps:

  • Please confirm if this issue is still relevant with the latest version of Dify; if so, you can keep the discussion open by commenting here.
  • Otherwise, I will automatically close this issue in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Jan 27, 2026): Hi, @JackyX1996. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported a bug in version 1.9.2 where documents uploaded with `runtime_mode='rag_pipeline'` get stuck in "waiting" because indexing tasks do not trigger. - The root cause was identified as a conditional check blocking indexing in pipeline mode. - Workarounds included manual indexing and upgrading to version 2.0.0 beta where the fix is implemented. - PR #27416 addressed the issue, but the fix may not yet be available in the Docker image. - Related issue #30237 might provide additional context or help. **Next Steps:** - Please confirm if this issue is still relevant with the latest version of Dify; if so, you can keep the discussion open by commenting here. - Otherwise, I will automatically close this issue in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#19960
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?