langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Knowledge file indexing 500 error #7100

New Issue
Closed
opened 2026-02-21 18:18:56 -05:00 by yindo · 14 comments
yindo commented 2026-02-21 18:18:56 -05:00
Owner
Copy Link
Copy Source

Originally created by @sugoh on GitHub (Dec 7, 2024).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

v0.13.1

Cloud or Self Hosted

Cloud

Steps to reproduce

Upload document via API https://api.dify.ai/v1/datasets/{inputs.dataset_id}/document/create-by-text
with this request body:

{
   "name": "xxx",
   "text": "yyy",
   "indexing_technique": "high_quality",
   "process_rule": {
      "mode": "custom",
      "rules": {
         "pre_processing_rules":  [],
         "segmentation": {
            "separator": "\n",
            "max_tokens": 1000,
         }
      }     
   }
}

✔️ Expected Behavior

  1. New file uploaded to Knowledge
  2. File indexing complete

❌ Actual Behavior

  1. New file uploaded to Knowledge
  2. 500 error on indexing
    Screenshot 2024-12-07 at 10 56 20 PM
Originally created by @sugoh on GitHub (Dec 7, 2024). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version v0.13.1 ### Cloud or Self Hosted Cloud ### Steps to reproduce Upload document via API `https://api.dify.ai/v1/datasets/{inputs.dataset_id}/document/create-by-text` with this request body: ``` { "name": "xxx", "text": "yyy", "indexing_technique": "high_quality", "process_rule": { "mode": "custom", "rules": { "pre_processing_rules": [], "segmentation": { "separator": "\n", "max_tokens": 1000, } } } } ``` ### ✔️ Expected Behavior 1. New file uploaded to Knowledge 2. File indexing complete ### ❌ Actual Behavior 1. New file uploaded to Knowledge 2. 500 error on indexing <img width="271" alt="Screenshot 2024-12-07 at 10 56 20 PM" src="https://github.com/user-attachments/assets/5664c88f-14fa-443c-9fa1-06fb126956a3">
yindo added the 🐞 bug🤔 cant-reproducecloud labels 2026-02-21 18:18:56 -05:00
yindo closed this issue 2026-02-21 18:18:56 -05:00
yindo commented 2026-02-21 18:18:56 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 7, 2024):

I found a similar closed issue regarding a 500 Internal Server Error during document indexing in the Dify platform, but no solution was provided in the comments [1].

For the 'high_quality' indexing technique, ensure that you have configured an Embedding model and specified both the embedding model and the embedding model provider in the Settings -> Model Provider. This configuration is necessary for high-quality indexing, and if not set, it can lead to errors [2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Dec 7, 2024): <!-- Answer --> I found a similar closed issue regarding a 500 Internal Server Error during document indexing in the Dify platform, but no solution was provided in the comments <sup>[[1]](https://github.com/langgenius/dify/issues/526)</sup>. For the 'high_quality' indexing technique, ensure that you have configured an Embedding model and specified both the embedding model and the embedding model provider in the Settings -> Model Provider. This configuration is necessary for high-quality indexing, and if not set, it can lead to errors <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents.md)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/bdb0df29-6f64-4dac-95e6-ebdaa2ba94c2?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/bdb0df29-6f64-4dac-95e6-ebdaa2ba94c2?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/bdb0df29-6f64-4dac-95e6-ebdaa2ba94c2?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/bdb0df29-6f64-4dac-95e6-ebdaa2ba94c2?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/bdb0df29-6f64-4dac-95e6-ebdaa2ba94c2?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/bdb0df29-6f64-4dac-95e6-ebdaa2ba94c2?feedback_type=other)</sup>
yindo commented 2026-02-21 18:18:56 -05:00
Author
Owner
Copy Link
Copy Source

@sugoh commented on GitHub (Dec 7, 2024):

To clarify, Model Provider has been setup and the API key is functioning. Our code was working all day until Dify suddenly stopped working

@sugoh commented on GitHub (Dec 7, 2024): To clarify, Model Provider has been setup and the API key is functioning. Our code was working all day until Dify suddenly stopped working
yindo commented 2026-02-21 18:18:56 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Dec 7, 2024):

I can not reproduce this in our cloud environment. And I saw the original version you mentioned is v0.11.2, however we are using v0.13.1 in our cloud environment right now.
Can you clarify this?

image
@crazywoola commented on GitHub (Dec 7, 2024): I can not reproduce this in our cloud environment. And I saw the original version you mentioned is v0.11.2, however we are using v0.13.1 in our cloud environment right now. Can you clarify this? <img width="841" alt="image" src="https://github.com/user-attachments/assets/68c1921c-502e-4524-8c97-8903587fdaac">
yindo commented 2026-02-21 18:18:56 -05:00
Author
Owner
Copy Link
Copy Source

@sugoh commented on GitHub (Dec 7, 2024):

@crazywoola sorry, copypasta error. it's v0.13.1 indeed.

Network shows that the error response I'm getting is (I've scrubbed the exact dataset id):

{
    "data": [
        {
            "id": "[xxx]",
            "indexing_status": "error",
            "processing_started_at": 1733626333,
            "parsing_completed_at": 1733626333,
            "cleaning_completed_at": 1733626334,
            "splitting_completed_at": 1733626334,
            "completed_at": 1733585155,
            "paused_at": null,
            "error": "Unexpected Response: 500 (Internal Server Error)\nRaw response content:\nb'{\"time\":4.1e-8,\"status\":{\"error\":\"Internal server error\"}}'",
            "stopped_at": 1733626338,
            "completed_segments": 0,
            "total_segments": 0
        },
        {
            "id": "[xxx]",
            "indexing_status": "error",
            "processing_started_at": 1733626347,
            "parsing_completed_at": 1733626347,
            "cleaning_completed_at": 1733626348,
            "splitting_completed_at": 1733626348,
            "completed_at": 1733585159,
            "paused_at": null,
            "error": "Unexpected Response: 500 (Internal Server Error)\nRaw response content:\nb'{\"time\":4.1e-8,\"status\":{\"error\":\"Internal server error\"}}'",
            "stopped_at": 1733626350,
            "completed_segments": 0,
            "total_segments": 0
        }
    ],
    "total": 2
}

Further maybe helpful information:

  • Indexing only fails in high quality mode. Economical works just fine.
  • We tried attaching a fresh Cohere API key but we're still facing this issue.
  • Have tried uploading the file via your GUI, that fails too
  • Indexing estimate endpoint works just fine (https://cloud.dify.ai/console/api/datasets/indexing-estimate)
  • Existing (already indexed) knowledge files can be retrieved just fine with hybrid_search

GET https://cloud.dify.ai/console/api/datasets/[xxx]

{
    "id": "[xxx]",
    "name": "[xxx]",
    "description": "[xxx]",
    "provider": "vendor",
    "permission": "only_me",
    "data_source_type": "upload_file",
    "indexing_technique": "high_quality",
    "app_count": 0,
    "document_count": 3,
    "word_count": 232354,
    "created_by": "[xxx]",
    "created_at": 1733570312,
    "updated_by": "[xxx]",
    "updated_at": 1733585173,
    "embedding_model": "text-embedding-3-large",
    "embedding_model_provider": "openai",
    "embedding_available": true,
    "retrieval_model_dict": {
        "search_method": "hybrid_search",
        "reranking_enable": true,
        "reranking_mode": "weighted_score",
        "reranking_model": {
            "reranking_provider_name": "cohere",
            "reranking_model_name": "rerank-english-v3.0"
        },
        "weights": {
            "keyword_setting": {
                "keyword_weight": 0.3
            },
            "vector_setting": {
                "vector_weight": 0.7,
                "embedding_model_name": "",
                "embedding_provider_name": ""
            }
        },
        "top_k": 5,
        "score_threshold_enabled": false,
        "score_threshold": 0.0
    },
    "tags": [],
    "external_knowledge_info": {
        "external_knowledge_id": null,
        "external_knowledge_api_id": null,
        "external_knowledge_api_name": null,
        "external_knowledge_api_endpoint": null
    },
    "external_retrieval_model": {
        "top_k": 5,
        "score_threshold": 0.0,
        "score_threshold_enabled": false
    }
}
@sugoh commented on GitHub (Dec 7, 2024): @crazywoola sorry, copypasta error. it's `v0.13.1` indeed. Network shows that the error response I'm getting is (I've scrubbed the exact dataset id): ```json { "data": [ { "id": "[xxx]", "indexing_status": "error", "processing_started_at": 1733626333, "parsing_completed_at": 1733626333, "cleaning_completed_at": 1733626334, "splitting_completed_at": 1733626334, "completed_at": 1733585155, "paused_at": null, "error": "Unexpected Response: 500 (Internal Server Error)\nRaw response content:\nb'{\"time\":4.1e-8,\"status\":{\"error\":\"Internal server error\"}}'", "stopped_at": 1733626338, "completed_segments": 0, "total_segments": 0 }, { "id": "[xxx]", "indexing_status": "error", "processing_started_at": 1733626347, "parsing_completed_at": 1733626347, "cleaning_completed_at": 1733626348, "splitting_completed_at": 1733626348, "completed_at": 1733585159, "paused_at": null, "error": "Unexpected Response: 500 (Internal Server Error)\nRaw response content:\nb'{\"time\":4.1e-8,\"status\":{\"error\":\"Internal server error\"}}'", "stopped_at": 1733626350, "completed_segments": 0, "total_segments": 0 } ], "total": 2 } ``` Further maybe helpful information: - Indexing only fails in high quality mode. Economical works just fine. - We tried attaching a fresh Cohere API key but we're still facing this issue. - Have tried uploading the file via your GUI, that fails too - Indexing estimate endpoint works just fine (https://cloud.dify.ai/console/api/datasets/indexing-estimate) - Existing (already indexed) knowledge files can be retrieved just fine with `hybrid_search` GET `https://cloud.dify.ai/console/api/datasets/[xxx]` ```json { "id": "[xxx]", "name": "[xxx]", "description": "[xxx]", "provider": "vendor", "permission": "only_me", "data_source_type": "upload_file", "indexing_technique": "high_quality", "app_count": 0, "document_count": 3, "word_count": 232354, "created_by": "[xxx]", "created_at": 1733570312, "updated_by": "[xxx]", "updated_at": 1733585173, "embedding_model": "text-embedding-3-large", "embedding_model_provider": "openai", "embedding_available": true, "retrieval_model_dict": { "search_method": "hybrid_search", "reranking_enable": true, "reranking_mode": "weighted_score", "reranking_model": { "reranking_provider_name": "cohere", "reranking_model_name": "rerank-english-v3.0" }, "weights": { "keyword_setting": { "keyword_weight": 0.3 }, "vector_setting": { "vector_weight": 0.7, "embedding_model_name": "", "embedding_provider_name": "" } }, "top_k": 5, "score_threshold_enabled": false, "score_threshold": 0.0 }, "tags": [], "external_knowledge_info": { "external_knowledge_id": null, "external_knowledge_api_id": null, "external_knowledge_api_name": null, "external_knowledge_api_endpoint": null }, "external_retrieval_model": { "top_k": 5, "score_threshold": 0.0, "score_threshold_enabled": false } } ```
yindo commented 2026-02-21 18:18:56 -05:00
Author
Owner
Copy Link
Copy Source

@sugoh commented on GitHub (Dec 7, 2024):

@JohnJyong can confirm it's not an issue with our Cohere rate limit. It's only been hit once.
Screenshot 2024-12-08 at 11 54 53 AM

@sugoh commented on GitHub (Dec 7, 2024): @JohnJyong can confirm it's not an issue with our Cohere rate limit. It's only been hit once. <img width="1445" alt="Screenshot 2024-12-08 at 11 54 53 AM" src="https://github.com/user-attachments/assets/122edcd6-86be-4cab-bb6d-ad1ca44ebbbf">
yindo commented 2026-02-21 18:18:56 -05:00
Author
Owner
Copy Link
Copy Source

@sugoh commented on GitHub (Dec 7, 2024):

@JohnJyong further update:

  1. This is working for new knowledge bases that we create.
  2. One knowledge based is cursed. the id is f6d65374-31f3-4e86-a710-37721f8d33f0. Suggestion: can you add more verbose error logging to determine why this is happening?
@sugoh commented on GitHub (Dec 7, 2024): @JohnJyong further update: 1. This is working for new knowledge bases that we create. 2. One knowledge based is cursed. the id is `f6d65374-31f3-4e86-a710-37721f8d33f0`. Suggestion: can you add more verbose error logging to determine why this is happening?
yindo commented 2026-02-21 18:18:56 -05:00
Author
Owner
Copy Link
Copy Source

@sugoh commented on GitHub (Dec 11, 2024):

@JohnJyong @crazywoola any updates?

Now every document in the knowledge base is stuck at queuing
Screenshot 2024-12-12 at 12 43 18 PM

This issue is also happening for a new knowledge base we set up.

This is the same issue as reported in Ticket 11414 that is also unresolved

@sugoh commented on GitHub (Dec 11, 2024): @JohnJyong @crazywoola any updates? Now every document in the knowledge base is stuck at queuing <img width="293" alt="Screenshot 2024-12-12 at 12 43 18 PM" src="https://github.com/user-attachments/assets/2be101a8-a217-4937-9239-e8d40b0a1986" /> This issue is also happening for a new knowledge base we set up. This is the same issue as reported in [Ticket 11414]( https://github.com/langgenius/dify/issues/11414) that is also unresolved
yindo commented 2026-02-21 18:18:56 -05:00
Author
Owner
Copy Link
Copy Source

@OrWestSide commented on GitHub (Dec 12, 2024):

Any solution to this? I am facing the same issue

@OrWestSide commented on GitHub (Dec 12, 2024): Any solution to this? I am facing the same issue
yindo commented 2026-02-21 18:18:56 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Dec 12, 2024):

@sugoh Are you using OpenAI's embedding model yesterday, their API service was down yesterday.

@crazywoola commented on GitHub (Dec 12, 2024): @sugoh Are you using OpenAI's embedding model yesterday, their API service was down yesterday.
yindo commented 2026-02-21 18:18:56 -05:00
Author
Owner
Copy Link
Copy Source

@sugoh commented on GitHub (Dec 13, 2024):

@crazywoola I was. However, this issue has existed since before they went down.

@sugoh commented on GitHub (Dec 13, 2024): @crazywoola I was. However, this issue has existed since before they went down.
yindo commented 2026-02-21 18:18:57 -05:00
Author
Owner
Copy Link
Copy Source

@ToX82 commented on GitHub (Dec 18, 2024):

I am having the same issue with embedding, both with cohere and huggingface. Those are both new accounts, so the rate limit is not being the issue

Edit: I'm having this issue since v0.13.1, I'm now on v0.14.1. I didn't try previous versions

@ToX82 commented on GitHub (Dec 18, 2024): I am having the same issue with embedding, both with cohere and huggingface. Those are both new accounts, so the rate limit is not being the issue Edit: I'm having this issue since `v0.13.1`, I'm now on `v0.14.1`. I didn't try previous versions
yindo commented 2026-02-21 18:18:57 -05:00
Author
Owner
Copy Link
Copy Source

@sugoh commented on GitHub (Dec 21, 2024):

@ToX82 Can confirm, I'm still facing the same issue in v0.14.1

It seems that once a knowledge base gets "corrupted" somehow, the only way to work around it is to create a new knowledge base and re-upload + re-index everything.

Am hoping @JohnJyong or @crazywoola would either:

  1. Investigate the server logs or
  2. Upgrade the error messages being passed to the client so we can report back.
    • The current error message is just so unhelpfully generic.
@sugoh commented on GitHub (Dec 21, 2024): @ToX82 Can confirm, I'm still facing the same issue in `v0.14.1` It seems that once a knowledge base gets "corrupted" somehow, the only way to work around it is to create a new knowledge base and re-upload + re-index everything. Am hoping @JohnJyong or @crazywoola would either: 1. Investigate the server logs or 2. Upgrade the error messages being passed to the client so we can report back. - The current error message is just so unhelpfully generic.
yindo commented 2026-02-21 18:18:57 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Dec 23, 2024):

This should be fixed now. We have some issues with TiDB instances. The resources were running out at that time.

@crazywoola commented on GitHub (Dec 23, 2024): This should be fixed now. We have some issues with TiDB instances. The resources were running out at that time.
yindo commented 2026-02-21 18:18:57 -05:00
Author
Owner
Copy Link
Copy Source

@sugoh commented on GitHub (Dec 23, 2024):

@crazywoola thanks for investigating. however, the issue still persists. No files can be indexed for corrupted knowledge bases. An example is f6d65374-31f3-4e86-a710-37721f8d33f0

@sugoh commented on GitHub (Dec 23, 2024): @crazywoola thanks for investigating. however, the issue still persists. No files can be indexed for corrupted knowledge bases. An example is `f6d65374-31f3-4e86-a710-37721f8d33f0`
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label cloud 🐞 bug 🤔 cant-reproduce
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7100
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?