langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Segments added through API get stucked at 'indexing' state. #4707

New Issue
Closed
opened 2026-02-21 18:07:39 -05:00 by yindo · 9 comments
yindo commented 2026-02-21 18:07:39 -05:00
Owner
Copy Link
Copy Source

Originally created by @racerxdl on GitHub (Jul 21, 2024).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.12

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Create a document with any text (via API)
  2. Add segments to it (via API)
  3. Wait for the document to change from Queued/Indexing -> Available
  4. Try enable/disable a added segment or check the API for segment status

✔️ Expected Behavior

Segments to be indexed/ready after some time, being able to enable/disable a segment.

❌ Actual Behavior

All segments stuck at "indexing" state.

image
image
image
image

This doesn't happen if I add the entire text to process when creating the document. But then the chunker messes it up and doesn't create the segments properly. So I manually separated the segments for QA and added them via API.

Originally created by @racerxdl on GitHub (Jul 21, 2024). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.6.12 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Create a document with any text (via API) 2. Add segments to it (via API) 3. Wait for the document to change from Queued/Indexing -> Available 4. Try enable/disable a added segment or check the API for segment status ### ✔️ Expected Behavior Segments to be indexed/ready after some time, being able to enable/disable a segment. ### ❌ Actual Behavior All segments stuck at "indexing" state. ![image](https://github.com/user-attachments/assets/059594c3-d274-4b57-9ee1-59e6ae1a34e1) ![image](https://github.com/user-attachments/assets/896dac99-04b7-42d6-a344-78c3f0529c42) ![image](https://github.com/user-attachments/assets/275349d7-b1ce-4f1e-b8c9-5c86e02d50db) ![image](https://github.com/user-attachments/assets/0c481651-4bbc-41dc-9fb6-036b90d1999e) This doesn't happen if I add the entire text to process when creating the document. But then the chunker messes it up and doesn't create the segments properly. So I manually separated the segments for QA and added them via API.
yindo added the 🐞 bug label 2026-02-21 18:07:39 -05:00
yindo closed this issue 2026-02-21 18:07:39 -05:00
yindo commented 2026-02-21 18:07:39 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 21, 2024):

Hey @racerxdl! 👋 I'm here to help you with bugs, questions, and becoming a contributor. Let's tackle this issue together!

To resolve the issue where segments added via API get stuck in the "indexing" state, you need to ensure that the indexing process completes successfully and updates the segment status to "completed". Here is a possible solution:

  1. Ensure that the _load method is called after the segments are loaded.
  2. Update the segment status to "completed" after the _load method completes.

Here is the relevant code snippet with the necessary changes:

def _load(self, index_processor: BaseIndexProcessor, dataset: Dataset,
          dataset_document: DatasetDocument, documents: list[Document]) -> None:
    """
    Insert index and update document/segment status to completed
    """
    embedding_model_instance = None
    if dataset.indexing_technique == 'high_quality':
        embedding_model_instance = self.model_manager.get_model_instance(
            tenant_id=dataset.tenant_id,
            provider=dataset.embedding_model_provider,
            model_type=ModelType.TEXT_EMBEDDING,
            model=dataset.embedding_model
        )

    # chunk nodes by chunk size
    indexing_start_at = time.perf_counter()
    tokens = 0
    chunk_size = 10

    # Assuming the indexing process happens here
    documents = index_processor.transform(text_docs, embedding_model_instance=embedding_model_instance,
                                          process_rule=process_rule, tenant_id=dataset.tenant_id,
                                          doc_language=doc_language)

    # Update segment status to completed
    self._update_segments_by_document(
        dataset_document_id=dataset_document.id,
        update_params={
            DocumentSegment.status: "completed",
            DocumentSegment.indexing_at: datetime.datetime.now(datetime.timezone.utc).replace(tzinfo=None)
        }
    )

    # Update document status to completed
    self._update_document_index_status(
        document_id=dataset_document.id,
        after_indexing_status="completed",
        extra_update_params={
            DatasetDocument.indexing_completed_at: datetime.datetime.now(datetime.timezone.utc).replace(tzinfo=None)
        }
    )

Additionally, you can use the duplicate_document_indexing_task function to handle retries and ensure that the indexing process completes successfully:

from duplicate_document_indexing_task import duplicate_document_indexing_task

# Retry indexing for a specific document ID
duplicate_document_indexing_task.delay(dataset_id, [document_id])

Make sure to call this _load method after the segments are loaded and ensure that the status is updated to "completed" once the indexing process is done. This should resolve the issue of segments getting stuck in the "indexing" state [1][2][3][4].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jul 21, 2024): <!-- Greeting --> Hey @racerxdl! 👋 I'm here to help you with bugs, questions, and becoming a contributor. Let's tackle this issue together! <!-- Answer --> To resolve the issue where segments added via API get stuck in the "indexing" state, you need to ensure that the indexing process completes successfully and updates the segment status to "completed". Here is a possible solution: 1. Ensure that the `_load` method is called after the segments are loaded. 2. Update the segment status to "completed" after the `_load` method completes. Here is the relevant code snippet with the necessary changes: ```python def _load(self, index_processor: BaseIndexProcessor, dataset: Dataset, dataset_document: DatasetDocument, documents: list[Document]) -> None: """ Insert index and update document/segment status to completed """ embedding_model_instance = None if dataset.indexing_technique == 'high_quality': embedding_model_instance = self.model_manager.get_model_instance( tenant_id=dataset.tenant_id, provider=dataset.embedding_model_provider, model_type=ModelType.TEXT_EMBEDDING, model=dataset.embedding_model ) # chunk nodes by chunk size indexing_start_at = time.perf_counter() tokens = 0 chunk_size = 10 # Assuming the indexing process happens here documents = index_processor.transform(text_docs, embedding_model_instance=embedding_model_instance, process_rule=process_rule, tenant_id=dataset.tenant_id, doc_language=doc_language) # Update segment status to completed self._update_segments_by_document( dataset_document_id=dataset_document.id, update_params={ DocumentSegment.status: "completed", DocumentSegment.indexing_at: datetime.datetime.now(datetime.timezone.utc).replace(tzinfo=None) } ) # Update document status to completed self._update_document_index_status( document_id=dataset_document.id, after_indexing_status="completed", extra_update_params={ DatasetDocument.indexing_completed_at: datetime.datetime.now(datetime.timezone.utc).replace(tzinfo=None) } ) ``` Additionally, you can use the `duplicate_document_indexing_task` function to handle retries and ensure that the indexing process completes successfully: ```python from duplicate_document_indexing_task import duplicate_document_indexing_task # Retry indexing for a specific document ID duplicate_document_indexing_task.delay(dataset_id, [document_id]) ``` Make sure to call this `_load` method after the segments are loaded and ensure that the status is updated to "completed" once the indexing process is done. This should resolve the issue of segments getting stuck in the "indexing" state <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/indexing_runner.py)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/tasks/retry_document_indexing_task.py)</sup><sup>[[3]](https://github.com/langgenius/dify/blob/main/api/tasks/create_segment_to_index_task.py)</sup><sup>[[4]](https://github.com/langgenius/dify/blob/main/api/tasks/duplicate_document_indexing_task.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:07:39 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Jul 22, 2024):

pls share the api curl , so we can check the real promble. @racerxdl

@JohnJyong commented on GitHub (Jul 22, 2024): pls share the api curl , so we can check the real promble. @racerxdl
yindo commented 2026-02-21 18:07:39 -05:00
Author
Owner
Copy Link
Copy Source

@racerxdl commented on GitHub (Jul 28, 2024):

Sorry for the delay @JohnJyong, here is a small example:

curl -X POST https://dify/v1/datasets/b384e948-2a1a-4c6a-9379-e05094d54ef5/documents/26a7498f-9539-4de4-a894-7a96139dafc4/segments \
    -H "Authorization: Bearer dataset-XXXXXXXXXX" \
    -H "Content-Type: application/json" \
    -d '{
    "segments": [
        {
            "content": "Tell me about Spell - Precision Shot",
            "answer": "# Spell - Precision Shot\nChannels for 2.5 seconds to shoot a powerful arrow. Deals 300% Weapon Power weapon damage to the target. Casting time is reduced by 0.5 seconds per 10 Aether consumed.\n\n* Agressive: True\n* Archetype: Archery\n* Channeling: Yes\n* Cooldown: 30.0 seconds\n* Cost: 5% of mana\n* Needs Target: Yes\n* Skill Points: 3\n* Range: 6 tiles\n* Aether: -50\n* Self Target: No",
            "keywords": [
                "ravendawn",
                "skill",
                "spell",
                "precision shot"
            ]
        }
    ]
}'
@racerxdl commented on GitHub (Jul 28, 2024): Sorry for the delay @JohnJyong, here is a small example: ```bash curl -X POST https://dify/v1/datasets/b384e948-2a1a-4c6a-9379-e05094d54ef5/documents/26a7498f-9539-4de4-a894-7a96139dafc4/segments \ -H "Authorization: Bearer dataset-XXXXXXXXXX" \ -H "Content-Type: application/json" \ -d '{ "segments": [ { "content": "Tell me about Spell - Precision Shot", "answer": "# Spell - Precision Shot\nChannels for 2.5 seconds to shoot a powerful arrow. Deals 300% Weapon Power weapon damage to the target. Casting time is reduced by 0.5 seconds per 10 Aether consumed.\n\n* Agressive: True\n* Archetype: Archery\n* Channeling: Yes\n* Cooldown: 30.0 seconds\n* Cost: 5% of mana\n* Needs Target: Yes\n* Skill Points: 3\n* Range: 6 tiles\n* Aether: -50\n* Self Target: No", "keywords": [ "ravendawn", "skill", "spell", "precision shot" ] } ] }' ```
yindo commented 2026-02-21 18:07:39 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Aug 28, 2024):

It should be fix in latest version.

@crazywoola commented on GitHub (Aug 28, 2024): It should be fix in latest version.
yindo commented 2026-02-21 18:07:39 -05:00
Author
Owner
Copy Link
Copy Source

@racerxdl commented on GitHub (Sep 3, 2024):

Hey @crazywoola which version it has been fixed? I'm testing on 0.7.3 and its still happens

image
image

@racerxdl commented on GitHub (Sep 3, 2024): Hey @crazywoola which version it has been fixed? I'm testing on 0.7.3 and its still happens ![image](https://github.com/user-attachments/assets/9b8d1999-983b-4c1e-8038-ec5d8cdf443e) ![image](https://github.com/user-attachments/assets/7839dd9d-b418-4c1f-9d99-8f130a464c49)
yindo commented 2026-02-21 18:07:39 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Sep 3, 2024):

Hello, it seems to be another problem, I will reopen this issue.

@crazywoola commented on GitHub (Sep 3, 2024): Hello, it seems to be another problem, I will reopen this issue.
yindo commented 2026-02-21 18:07:39 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Sep 3, 2024):

There is a conflict between segment creation during document generation and create segment by api . New segment creation will be prohibited during document processing.

@JohnJyong commented on GitHub (Sep 3, 2024): There is a conflict between segment creation during document generation and create segment by api . New segment creation will be prohibited during document processing.
yindo commented 2026-02-21 18:07:39 -05:00
Author
Owner
Copy Link
Copy Source

@racerxdl commented on GitHub (Sep 3, 2024):

Just to be sure @crazywoola is the patch for this issue applied in 0.7.3? (Not the issue @JohnJyong mentioned). Because if I check the API, its still stuck at indexing:
image

Thats stuff I added after upgrading to 0.7.3

@racerxdl commented on GitHub (Sep 3, 2024): Just to be sure @crazywoola is the patch for this issue applied in 0.7.3? (Not the issue @JohnJyong mentioned). Because if I check the API, its still stuck at indexing: ![image](https://github.com/user-attachments/assets/787e9057-fef1-4914-a3dc-0309cc25c26d) Thats stuff I added after upgrading to 0.7.3
yindo commented 2026-02-21 18:07:39 -05:00
Author
Owner
Copy Link
Copy Source

@racerxdl commented on GitHub (Sep 11, 2024):

Seens to be working on 0.8.0

@racerxdl commented on GitHub (Sep 11, 2024): Seens to be working on 0.8.0
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4707
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?