langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Segmented embedding stuck at 100% #4810

New Issue
Closed
opened 2026-02-21 18:08:06 -05:00 by yindo · 6 comments
yindo commented 2026-02-21 18:08:06 -05:00
Owner
Copy Link
Copy Source

Originally created by @Lxx-c on GitHub (Jul 28, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.15

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

The embedding is stuck at 100%, and I have several documents stuck at 100% for a long time. I have checked the worker's logs and there are no error messages in them.The specific lag state is shown in the following figure:
segement

✔️ Expected Behavior

No response

❌ Actual Behavior

It should become usable once embedded

Originally created by @Lxx-c on GitHub (Jul 28, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.6.15 ### Cloud or Self Hosted Self Hosted (Source) ### Steps to reproduce The embedding is stuck at 100%, and I have several documents stuck at 100% for a long time. I have checked the worker's logs and there are no error messages in them.The specific lag state is shown in the following figure: ![segement](https://github.com/user-attachments/assets/f5f015d3-0075-49b8-8acd-22b6473a1cf7) ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior It should become usable once embedded
yindo added the 🐞 bugstale labels 2026-02-21 18:08:06 -05:00
yindo closed this issue 2026-02-21 18:08:06 -05:00
yindo commented 2026-02-21 18:08:06 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 28, 2024):

The issue with the segmented embedding process being stuck at 100% in Dify version 0.6.15 could be related to the state management and fetching of the indexing status. Here are some areas to investigate and debug:

  1. Fetching Indexing Status: Ensure that the fetchIndexingStatus function is correctly fetching the status and updating the state. If the status is not being updated correctly, the UI might show the process as stuck at 100%.
const fetchIndexingStatus = async () => {
    const status = await doFetchIndexingStatus({ datasetId, batchId })
    setIndexingStatusDetail(status.data)
    return status.data
}

  1. State Update: Verify that the state indexingStatusBatchDetail is being updated correctly and that the component is re-rendering with the new status.

  2. Completion Check: Ensure that the logic to determine if the embedding process is completed is correct. The snippet below shows how the status is displayed:

<div className={s.embeddingStatus}>
    {isEmbedding && t('datasetDocuments.embedding.processing')}
    {isEmbeddingCompleted && t('datasetDocuments.embedding.completed')}
</div>

Make sure that isEmbeddingCompleted is being set correctly based on the fetched status.

  1. Logs and Errors: Since there are no error messages in the worker's logs, consider adding more detailed logging around the status fetching and state update process to identify where it might be getting stuck.

If these areas do not resolve the issue, further investigation into the backend processing logic and the specific implementation of doFetchIndexingStatus might be necessary.

const fetchIndexingStatus = async () => {
    try {
        const status = await doFetchIndexingStatus({ datasetId, batchId })
        setIndexingStatusDetail(status.data)
        return status.data
    } catch (error) {
        console.error("Error fetching indexing status:", error)
    }
}

This additional logging can help identify if there are any issues with the API call or state update process [1].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jul 28, 2024): <!-- Answer --> The issue with the segmented embedding process being stuck at 100% in Dify version 0.6.15 could be related to the state management and fetching of the indexing status. Here are some areas to investigate and debug: 1. **Fetching Indexing Status**: Ensure that the `fetchIndexingStatus` function is correctly fetching the status and updating the state. If the status is not being updated correctly, the UI might show the process as stuck at 100%. ```tsx const fetchIndexingStatus = async () => { const status = await doFetchIndexingStatus({ datasetId, batchId }) setIndexingStatusDetail(status.data) return status.data } ``` 2. **State Update**: Verify that the state `indexingStatusBatchDetail` is being updated correctly and that the component is re-rendering with the new status. 3. **Completion Check**: Ensure that the logic to determine if the embedding process is completed is correct. The snippet below shows how the status is displayed: ```tsx <div className={s.embeddingStatus}> {isEmbedding && t('datasetDocuments.embedding.processing')} {isEmbeddingCompleted && t('datasetDocuments.embedding.completed')} </div> ``` Make sure that `isEmbeddingCompleted` is being set correctly based on the fetched status. 4. **Logs and Errors**: Since there are no error messages in the worker's logs, consider adding more detailed logging around the status fetching and state update process to identify where it might be getting stuck. If these areas do not resolve the issue, further investigation into the backend processing logic and the specific implementation of `doFetchIndexingStatus` might be necessary. ```tsx const fetchIndexingStatus = async () => { try { const status = await doFetchIndexingStatus({ datasetId, batchId }) setIndexingStatusDetail(status.data) return status.data } catch (error) { console.error("Error fetching indexing status:", error) } } ``` This additional logging can help identify if there are any issues with the API call or state update process <sup>[[1]](https://github.com/langgenius/dify/blob/main/web/app/components/datasets/create/embedding-process/index.tsx)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:08:06 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Jul 29, 2024):

Please provide the logs of the celery worker.

@crazywoola commented on GitHub (Jul 29, 2024): Please provide the logs of the celery worker.
yindo commented 2026-02-21 18:08:07 -05:00
Author
Owner
Copy Link
Copy Source

@Lxx-c commented on GitHub (Jul 29, 2024):

@crazywoola This is my worker log file, unfortunately, there is no valid information inside
worker.log

@Lxx-c commented on GitHub (Jul 29, 2024): @crazywoola This is my worker log file, unfortunately, there is no valid information inside [worker.log](https://github.com/user-attachments/files/16410898/worker.log)
yindo commented 2026-02-21 18:08:07 -05:00
Author
Owner
Copy Link
Copy Source

@Lxx-c commented on GitHub (Jul 29, 2024):

@crazywoola He sometimes reports the following error, he calls the API http://172.16.0.14:8002/v1/chat/completions, but I haven't set up this API yet.The QDrant database I am using.

credentials, prompt_messages, model_parameters, tools, stop, stream, user)
  File "/data/apps/dify/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py", line 69, in _invoke
    return self._generate(
  File "/data/apps/dify/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py", line 351, in _generate
    raise InvokeError(f"API request failed with status code {response.status_code}: {response.text}")
core.model_runtime.errors.invoke.InvokeError: API request failed with status code 500: {"error":{"message":"do request failed: Post \"http://172.16.0.14:8002/v1/chat/completions\": EOF (request id: 2024072918091163744306624055307)","type":"one_api_error","param":"","code":"do_request_failed"}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/apps/dify/api/core/rag/index_processor/processor/qa_index_processor.py", line 133, in _format_qa_document
    response = LLMGenerator.generate_qa_document(tenant_id, document_node.page_content, document_language)
  File "/data/apps/dify/api/core/llm_generator/llm_generator.py", line 291, in generate_qa_document
    response = model_instance.invoke_llm(
  File "/data/apps/dify/api/core/model_manager.py", line 123, in invoke_llm
    return self._round_robin_invoke(
  File "/data/apps/dify/api/core/model_manager.py", line 302, in _round_robin_invoke
    return function(*args, **kwargs)
  File "/data/apps/dify/api/core/model_runtime/model_providers/__base/large_language_model.py", line 117, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [openai_api_compatible] Error: API request failed with status code 500: {"error":{"message":"do request failed: Post \"http://172.16.0.14:8002/v1/chat/completions\": EOF (request id: 2024072918091163744306624055307)","type":"one_api_error","param":"","code":"do_request_failed"}}
[2024-07-29 10:10:03,734: ERROR/MainProcess] [openai_api_compatible] Error: API request failed with status code 500: {"error":{"message":"do request failed: Post \"http://172.16.0.14:8002/v1/chat/completions\": EOF (request id: 2024072918100373768554533196363)","type":"one_api_error","param":"","code":"do_request_failed"}}
Traceback (most recent call last):
  File "/data/apps/dify/api/core/model_runtime/model_providers/__base/large_language_model.py", line 102, in invoke
    result = self._invoke(model, credentials, prompt_messages, model_parameters, tools, stop, stream, user)
  File "/data/apps/dify/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py", line 69, in _invoke
    return self._generate(
  File "/data/apps/dify/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py", line 351, in _generate
    raise InvokeError(f"API request failed with status code {response.status_code}: {response.text}")
core.model_runtime.errors.invoke.InvokeError: API request failed with status code 500: {"error":{"message":"do request failed: Post \"http://172.16.0.14:8002/v1/chat/completions\": EOF (request id: 2024072918100373768554533196363)","type":"one_api_error","param":"","code":"do_request_failed"}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/apps/dify/api/core/rag/index_processor/processor/qa_index_processor.py", line 133, in _format_qa_document
    response = LLMGenerator.generate_qa_document(tenant_id, document_node.page_content, document_language)
  File "/data/apps/dify/api/core/llm_generator/llm_generator.py", line 291, in generate_qa_document
    response = model_instance.invoke_llm(
  File "/data/apps/dify/api/core/model_manager.py", line 123, in invoke_llm
    return self._round_robin_invoke(
  File "/data/apps/dify/api/core/model_manager.py", line 302, in _round_robin_invoke
    return function(*args, **kwargs)
  File "/data/apps/dify/api/core/model_runtime/model_providers/__base/large_language_model.py", line 117, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [openai_api_compatible] Error: API request failed with status code 500: {"error":{"message":"do request failed: Post \"http://172.16.0.14:8002/v1/chat/completions\": EOF (request id: 2024072918100373768554533196363)","type":"one_api_error","param":"","code":"do_request_failed"}}
@Lxx-c commented on GitHub (Jul 29, 2024): @crazywoola He sometimes reports the following error, he calls the API `http://172.16.0.14:8002/v1/chat/completions`, but I haven't set up this API yet.The QDrant database I am using. ```log credentials, prompt_messages, model_parameters, tools, stop, stream, user) File "/data/apps/dify/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py", line 69, in _invoke return self._generate( File "/data/apps/dify/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py", line 351, in _generate raise InvokeError(f"API request failed with status code {response.status_code}: {response.text}") core.model_runtime.errors.invoke.InvokeError: API request failed with status code 500: {"error":{"message":"do request failed: Post \"http://172.16.0.14:8002/v1/chat/completions\": EOF (request id: 2024072918091163744306624055307)","type":"one_api_error","param":"","code":"do_request_failed"}} During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/data/apps/dify/api/core/rag/index_processor/processor/qa_index_processor.py", line 133, in _format_qa_document response = LLMGenerator.generate_qa_document(tenant_id, document_node.page_content, document_language) File "/data/apps/dify/api/core/llm_generator/llm_generator.py", line 291, in generate_qa_document response = model_instance.invoke_llm( File "/data/apps/dify/api/core/model_manager.py", line 123, in invoke_llm return self._round_robin_invoke( File "/data/apps/dify/api/core/model_manager.py", line 302, in _round_robin_invoke return function(*args, **kwargs) File "/data/apps/dify/api/core/model_runtime/model_providers/__base/large_language_model.py", line 117, in invoke raise self._transform_invoke_error(e) core.model_runtime.errors.invoke.InvokeError: [openai_api_compatible] Error: API request failed with status code 500: {"error":{"message":"do request failed: Post \"http://172.16.0.14:8002/v1/chat/completions\": EOF (request id: 2024072918091163744306624055307)","type":"one_api_error","param":"","code":"do_request_failed"}} [2024-07-29 10:10:03,734: ERROR/MainProcess] [openai_api_compatible] Error: API request failed with status code 500: {"error":{"message":"do request failed: Post \"http://172.16.0.14:8002/v1/chat/completions\": EOF (request id: 2024072918100373768554533196363)","type":"one_api_error","param":"","code":"do_request_failed"}} Traceback (most recent call last): File "/data/apps/dify/api/core/model_runtime/model_providers/__base/large_language_model.py", line 102, in invoke result = self._invoke(model, credentials, prompt_messages, model_parameters, tools, stop, stream, user) File "/data/apps/dify/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py", line 69, in _invoke return self._generate( File "/data/apps/dify/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py", line 351, in _generate raise InvokeError(f"API request failed with status code {response.status_code}: {response.text}") core.model_runtime.errors.invoke.InvokeError: API request failed with status code 500: {"error":{"message":"do request failed: Post \"http://172.16.0.14:8002/v1/chat/completions\": EOF (request id: 2024072918100373768554533196363)","type":"one_api_error","param":"","code":"do_request_failed"}} During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/data/apps/dify/api/core/rag/index_processor/processor/qa_index_processor.py", line 133, in _format_qa_document response = LLMGenerator.generate_qa_document(tenant_id, document_node.page_content, document_language) File "/data/apps/dify/api/core/llm_generator/llm_generator.py", line 291, in generate_qa_document response = model_instance.invoke_llm( File "/data/apps/dify/api/core/model_manager.py", line 123, in invoke_llm return self._round_robin_invoke( File "/data/apps/dify/api/core/model_manager.py", line 302, in _round_robin_invoke return function(*args, **kwargs) File "/data/apps/dify/api/core/model_runtime/model_providers/__base/large_language_model.py", line 117, in invoke raise self._transform_invoke_error(e) core.model_runtime.errors.invoke.InvokeError: [openai_api_compatible] Error: API request failed with status code 500: {"error":{"message":"do request failed: Post \"http://172.16.0.14:8002/v1/chat/completions\": EOF (request id: 2024072918100373768554533196363)","type":"one_api_error","param":"","code":"do_request_failed"}} ```
yindo commented 2026-02-21 18:08:07 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Jul 29, 2024):

image Could you check the system model settings here. Because when you turn on the QA format, it will use this model to split the documents.
@crazywoola commented on GitHub (Jul 29, 2024): <img width="1035" alt="image" src="https://github.com/user-attachments/assets/878de1cd-e22b-4ca7-b209-6c64ae13cd3f"> Could you check the system model settings here. Because when you turn on the QA format, it will use this model to split the documents.
yindo commented 2026-02-21 18:08:07 -05:00
Author
Owner
Copy Link
Copy Source

@Lxx-c commented on GitHub (Jul 29, 2024):

@crazywoola I am using the Qwen2-7B-Instruct model, which I deployed myself. Today, I checked the status of these documents and there was still no change (it was still 100%). I manually updated its status to completed in the database, and the documents with a 100% status became available. I am not sure if there are any other implications of this action.And I encountered a strange problem yesterday. The document that had been indexed suddenly became in the process of indexing. Before the change, I operated the vector database and switched from Weaviate to Qdrant.

@Lxx-c commented on GitHub (Jul 29, 2024): @crazywoola I am using the `Qwen2-7B-Instruct` model, which I deployed myself. Today, I checked the status of these documents and there was still no change (it was still 100%). I manually updated its status to completed in the database, and the documents with a 100% status became available. I am not sure if there are any other implications of this action.And I encountered a strange problem yesterday. The document that had been indexed suddenly became in the process of indexing. Before the change, I operated the vector database and switched from Weaviate to Qdrant.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label stale 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4810
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?