Knowledge dataset: document created with empty content #20598

New Issue

Closed

opened 2026-02-21 20:08:06 -05:00 by yindo · 11 comments

yindo commented 2026-02-21 20:08:06 -05:00

Owner

Copy Link

Copy Source

Originally created by @talenhon on GitHub (Nov 26, 2025).

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• Please do not modify this template :) and fill in all the required fields.

Dify version

v1.10.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

1. Using Postman to upload create_by_text to the dify knowledge.
   curl --location --globoff 'https://api.dify.com/v1/datasets/{data_set_id}/document/create_by_text' \ --header 'Authorization: Bearer {auth}' \ --header 'Content-Type: application/json' \ --data '{"name": "text","text": "text","indexing_technique": "high_quality","process_rule": {"mode": "automatic"}, "doc_form": "hierarchical_model", "doc_language": "English"}'

2. Go to dify knowledge, showing 0 in the words

✔️ Expected Behavior

When using Dify API create_by_text / upload_by_file, document is created with content.

❌ Actual Behavior

When using Dify API create_by_text / upload_by_file, document is created with empty content (words).

Originally created by @talenhon on GitHub (Nov 26, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version v1.10.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Using Postman to upload create_by_text to the dify knowledge. `curl --location --globoff 'https://api.dify.com/v1/datasets/{data_set_id}/document/create_by_text' \ --header 'Authorization: Bearer {auth}' \ --header 'Content-Type: application/json' \ --data '{"name": "text","text": "text","indexing_technique": "high_quality","process_rule": {"mode": "automatic"}, "doc_form": "hierarchical_model", "doc_language": "English"}'` 3. Go to dify knowledge, showing 0 in the words <img width="889" height="579" alt="Image" src="https://github.com/user-attachments/assets/a3acd71a-9eac-4f55-bea5-f565741a86fa" /> ### ✔️ Expected Behavior When using Dify API create_by_text / upload_by_file, document is created with content. ### ❌ Actual Behavior When using Dify API create_by_text / upload_by_file, document is created with empty content (words). <img width="1367" height="179" alt="Image" src="https://github.com/user-attachments/assets/12b8929c-f04b-44df-8d56-6a440bca2c9a" />

yindo added the 🐞 bug label 2026-02-21 20:08:06 -05:00

yindo closed this issue 2026-02-21 20:08:06 -05:00

yindo commented 2026-02-21 20:08:06 -05:00

Author

Owner

Copy Link

Copy Source

@dosubot[bot] commented on GitHub (Nov 26, 2025):

This issue is almost always caused by a problem in the asynchronous document processing pipeline. When you use the API to create a document, Dify saves the file and creates a document entry immediately, but the actual content extraction and word count update happen later in a background Celery worker. If that worker fails, is not running, or can't access required resources, the document stays empty (zero words) in the UI.

Common causes include: the vector database container (like Weaviate) is missing or misconfigured, the Celery worker is not running, Python or locale dependencies are missing in the worker container, or the embedding provider API key (e.g., OPENAI_API_KEY) is not set. Any of these will break the indexing pipeline and result in empty documents, even though the API returns 200 OK and the file is uploaded successfully. See the detailed breakdown and troubleshooting steps here: dify-docs issue #397.

To fix:

• Make sure your vector DB container is running and healthy, and that VECTOR_STORE env vars match across all services.
• Confirm the Celery worker is running (docker compose ps should show it as healthy).
• Check that all Python and locale dependencies are installed in the worker container (no locale or Python errors in logs).
• Ensure OPENAI_API_KEY is set in the API and worker containers.
• Inspect logs for errors in the API, worker, and vector DB containers, especially around document indexing and extraction.
• Try uploading a small file and watch the logs and UI for indexing progress.

If the problem persists after these checks, collect logs from the worker and API containers during a document upload and share them for further diagnosis.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

@dosubot[bot] commented on GitHub (Nov 26, 2025): <!-- Answer --> This issue is almost always caused by a problem in the asynchronous document processing pipeline. When you use the API to create a document, Dify saves the file and creates a document entry immediately, but the actual content extraction and word count update happen later in a background Celery worker. If that worker fails, is not running, or can't access required resources, the document stays empty (zero words) in the UI. Common causes include: the vector database container (like Weaviate) is missing or misconfigured, the Celery worker is not running, Python or locale dependencies are missing in the worker container, or the embedding provider API key (e.g., `OPENAI_API_KEY`) is not set. Any of these will break the indexing pipeline and result in empty documents, even though the API returns 200 OK and the file is uploaded successfully. See the detailed breakdown and troubleshooting steps here: [dify-docs issue #397](https://github.com/langgenius/dify-docs/issues/397). To fix: - Make sure your vector DB container is running and healthy, and that `VECTOR_STORE` env vars match across all services. - Confirm the Celery worker is running (`docker compose ps` should show it as healthy). - Check that all Python and locale dependencies are installed in the worker container (no locale or Python errors in logs). - Ensure `OPENAI_API_KEY` is set in the API and worker containers. - Inspect logs for errors in the API, worker, and vector DB containers, especially around document indexing and extraction. - Try uploading a small file and watch the logs and UI for indexing progress. If the problem persists after these checks, collect logs from the worker and API containers during a document upload and share them for further diagnosis. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- ^{How did I do? [Good](https://app.dosu.dev/response-feedback/43431e13-c089-4c57-851b-263aee03499c?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/43431e13-c089-4c57-851b-263aee03499c?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/43431e13-c089-4c57-851b-263aee03499c?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/43431e13-c089-4c57-851b-263aee03499c?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/43431e13-c089-4c57-851b-263aee03499c?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/43431e13-c089-4c57-851b-263aee03499c?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/43431e13-c089-4c57-851b-263aee03499c?feedback_type=other)} [](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[](https://cursor.com/link/prompt?text=This%20issue%20is%20almost%20always%20caused%20by%20a%20problem%20in%20the%20asynchronous%20document%20processing%20pipeline.%20When%20you%20use%20the%20API%20to%20create%20a%20document%2C%20Dify%20saves%20the%20file%20and%20creates%20a%20document%20entry%20immediately%2C%20but%20the%20actual%20content%20extraction%20and%20word%20count%20update%20happen%20later%20in%20a%20background%20Celery%20worker.%20If%20that%20worker%20fails%2C%20is%20not%20running%2C%20or%20can%27t%20access%20required%20resources%2C%20the%20document%20stays%20empty%20%28zero%20words%29%20in%20the%20UI.%0A%0ACommon%20causes%20include%3A%20the%20vector%20database%20container%20%28like%20Weaviate%29%20is%20missing%20or%20misconfigured%2C%20the%20Celery%20worker%20is%20not%20running%2C%20Python%20or%20locale%20dependencies%20are%20missing%20in%20the%20worker%20container%2C%20or%20the%20embedding%20provider%20API%20key%20%28e.g.%2C%20%60OPENAI_API_KEY%60%29%20is%20not%20set.%20Any%20of%20these%20will%20break%20the%20indexing%20pipeline%20and%20result%20in%20empty%20documents%2C%20even%20though%20the%20API%20returns%20200%20OK%20and%20the%20file%20is%20uploaded%20successfully.%20See%20the%20detailed%20breakdown%20and%20troubleshooting%20steps%20here%3A%20%5Bdify-docs%20issue%20%23397%5D%28https%3A//github.com/langgenius/dify-docs/issues/397%29.%0A%0ATo%20fix%3A%0A-%20Make%20sure%20your%20vector%20DB%20container%20is%20running%20and%20healthy%2C%20and%20that%20%60VECTOR_STORE%60%20env%20vars%20match%20across%20all%20services.%0A-%20Confirm%20the%20Celery%20worker%20is%20running%20%28%60docker%20compose%20ps%60%20should%20show%20it%20as%20healthy%29.%0A-%20Check%20that%20all%20Python%20and%20locale%20dependencies%20are%20installed%20in%20the%20worker%20container%20%28no%20locale%20or%20Python%20errors%20in%20logs%29.%0A-%20Ensure%20%60OPENAI_API_KEY%60%20is%20set%20in%20the%20API%20and%20worker%20containers.%0A-%20Inspect%20logs%20for%20errors%20in%20the%20API%2C%20worker%2C%20and%20vector%20DB%20containers%2C%20especially%20around%20document%20indexing%20and%20extraction.%0A-%20Try%20uploading%20a%20small%20file%20and%20watch%20the%20logs%20and%20UI%20for%20indexing%20progress.%0A%0AIf%20the%20problem%20persists%20after%20these%20checks%2C%20collect%20logs%20from%20the%20worker%20and%20API%20containers%20during%20a%20document%20upload%20and%20share%20them%20for%20further%20diagnosis.)&nbsp;[](https://go.dosu.dev/discord-bot)&nbsp;[](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28763)

yindo commented 2026-02-21 20:08:06 -05:00

Author

Owner

Copy Link

Copy Source

@talenhon commented on GitHub (Nov 27, 2025):

@dosu
Tried below two scenarios,

1. Upload file within UI: Upload a txt file to the same dataset, document successfully uploaded, with chunks and results are able to capture.
2. Update text using api: upload the text to the same dataset. document is created with empty content (0 words).

@talenhon commented on GitHub (Nov 27, 2025): @dosu Tried below two scenarios, 1. Upload file within UI: Upload a txt file to the same dataset, document successfully uploaded, with chunks and results are able to capture. 2. Update text using api: upload the text to the same dataset. document is created with empty content (0 words). <img width="1354" height="215" alt="Image" src="https://github.com/user-attachments/assets/5eb907a6-07e7-4445-ab52-d022f030cea6" />

yindo commented 2026-02-21 20:08:06 -05:00

Author

Owner

Copy Link

Copy Source

@dosubot[bot] commented on GitHub (Nov 27, 2025):

Your UI and API uploads both use the same backend pipeline, so if UI uploads work but API uploads create empty documents, it's almost always due to either (1) a problem with the API payload or (2) the Celery worker not listening on the correct queues.

First, double-check your API payload: both the "name" and "text" fields must be present and non-empty. Recent backend changes strictly validate these fields—if "name" is missing when "text" is provided, the backend will silently skip content extraction, resulting in an empty document even if the API call returns 200 OK. Make sure your JSON looks like:

{
  "name": "your_filename.txt",
  "text": "your file content here",
  "indexing_technique": "high_quality",
  "process_rule": {"mode": "automatic"},
  "doc_form": "hierarchical_model",
  "doc_language": "English"
}

See the stricter validation logic here: source.

Second, confirm your Celery worker is running and listening on both the dataset and priority_dataset queues. If the worker is not started with these queues, document indexing tasks from API uploads will not be processed, and the documents will stay empty. The default Docker setup should include these queues, but if you override CELERY_QUEUES or CELERY_WORKER_QUEUES, make sure they include at least "dataset,priority_dataset" (source).

If both the payload and worker queues are correct, check your worker logs for any validation or processing errors right after an API upload. If you see errors about missing fields or failed extraction, that's the root cause.

Let me know if you want help reviewing your exact API payload or worker logs.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

@dosubot[bot] commented on GitHub (Nov 27, 2025): <!-- Answer --> Your UI and API uploads both use the same backend pipeline, so if UI uploads work but API uploads create empty documents, it's almost always due to either (1) a problem with the API payload or (2) the Celery worker not listening on the correct queues. First, double-check your API payload: both the "name" and "text" fields must be present and non-empty. Recent backend changes strictly validate these fields—if "name" is missing when "text" is provided, the backend will silently skip content extraction, resulting in an empty document even if the API call returns 200 OK. Make sure your JSON looks like: ```json { "name": "your_filename.txt", "text": "your file content here", "indexing_technique": "high_quality", "process_rule": {"mode": "automatic"}, "doc_form": "hierarchical_model", "doc_language": "English" } ``` See the stricter validation logic here: [source](https://github.com/langgenius/dify/pull/28236). Second, confirm your Celery worker is running and listening on both the dataset and priority_dataset queues. If the worker is not started with these queues, document indexing tasks from API uploads will not be processed, and the documents will stay empty. The default Docker setup should include these queues, but if you override CELERY_QUEUES or CELERY_WORKER_QUEUES, make sure they include at least "dataset,priority_dataset" ([source](https://github.com/langgenius/dify/blob/58f448a926174fa90a2d971432dacb218a990c11/api/docker/entrypoint.sh#L22-L68)). If both the payload and worker queues are correct, check your worker logs for any validation or processing errors right after an API upload. If you see errors about missing fields or failed extraction, that's the root cause. Let me know if you want help reviewing your exact API payload or worker logs. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- ^{How did I do? [Good](https://app.dosu.dev/response-feedback/4e3afaeb-0843-48db-80d9-87b9e10eb5dd?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/4e3afaeb-0843-48db-80d9-87b9e10eb5dd?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/4e3afaeb-0843-48db-80d9-87b9e10eb5dd?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/4e3afaeb-0843-48db-80d9-87b9e10eb5dd?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/4e3afaeb-0843-48db-80d9-87b9e10eb5dd?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/4e3afaeb-0843-48db-80d9-87b9e10eb5dd?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/4e3afaeb-0843-48db-80d9-87b9e10eb5dd?feedback_type=other)} [](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[](https://cursor.com/link/prompt?text=Your%20UI%20and%20API%20uploads%20both%20use%20the%20same%20backend%20pipeline%2C%20so%20if%20UI%20uploads%20work%20but%20API%20uploads%20create%20empty%20documents%2C%20it%27s%20almost%20always%20due%20to%20either%20%281%29%20a%20problem%20with%20the%20API%20payload%20or%20%282%29%20the%20Celery%20worker%20not%20listening%20on%20the%20correct%20queues.%0A%0AFirst%2C%20double-check%20your%20API%20payload%3A%20both%20the%20%22name%22%20and%20%22text%22%20fields%20must%20be%20present%20and%20non-empty.%20Recent%20backend%20changes%20strictly%20validate%20these%20fields%E2%80%94if%20%22name%22%20is%20missing%20when%20%22text%22%20is%20provided%2C%20the%20backend%20will%20silently%20skip%20content%20extraction%2C%20resulting%20in%20an%20empty%20document%20even%20if%20the%20API%20call%20returns%20200%20OK.%20Make%20sure%20your%20JSON%20looks%20like%3A%0A%60%60%60json%0A%7B%0A%20%20%22name%22%3A%20%22your_filename.txt%22%2C%0A%20%20%22text%22%3A%20%22your%20file%20content%20here%22%2C%0A%20%20%22indexing_technique%22%3A%20%22high_quality%22%2C%0A%20%20%22process_rule%22%3A%20%7B%22mode%22%3A%20%22automatic%22%7D%2C%0A%20%20%22doc_form%22%3A%20%22hierarchical_model%22%2C%0A%20%20%22doc_language%22%3A%20%22English%22%0A%7D%0A%60%60%60%0ASee%20the%20stricter%20validation%20logic%20here%3A%20%5Bsource%5D%28https%3A//github.com/langgenius/dify/pull/28236%29.%0A%0ASecond%2C%20confirm%20your%20Celery%20worker%20is%20running%20and%20listening%20on%20both%20the%20dataset%20and%20priority_dataset%20queues.%20If%20the%20worker%20is%20not%20started%20with%20these%20queues%2C%20document%20indexing%20tasks%20from%20API%20uploads%20will%20not%20be%20processed%2C%20and%20the%20documents%20will%20stay%20empty.%20The%20default%20Docker%20setup%20should%20include%20these%20queues%2C%20but%20if%20you%20override%20CELERY_QUEUES%20or%20CELERY_WORKER_QUEUES%2C%20make%20sure%20they%20include%20at%20least%20%22dataset%2Cpriority_dataset%22%20%28%5Bsource%5D%28https%3A//github.com/langgenius/dify/blob/58f448a926174fa90a2d971432dacb218a990c11/api/docker/entrypoint.sh%23L22-L68%29%29.%0A%0AIf%20both%20the%20payload%20and%20worker%20queues%20are%20correct%2C%20check%20your%20worker%20logs%20for%20any%20validation%20or%20processing%20errors%20right%20after%20an%20API%20upload.%20If%20you%20see%20errors%20about%20missing%20fields%20or%20failed%20extraction%2C%20that%27s%20the%20root%20cause.%0A%0ALet%20me%20know%20if%20you%20want%20help%20reviewing%20your%20exact%20API%20payload%20or%20worker%20logs.)&nbsp;[](https://go.dosu.dev/discord-bot)&nbsp;[](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28763)

yindo commented 2026-02-21 20:08:06 -05:00

Author

Owner

Copy Link

Copy Source

@talenhon commented on GitHub (Nov 27, 2025):

@dosu
Here's the request body: both the "name" and "text" fields must be present and non-empty

{
    "name": "text",
    "text": "text",
    "indexing_technique": "high_quality",
    "process_rule": {"mode": "automatic"},
    "doc_form": "hierarchical_model",
    "doc_language": "English"
}

And the response: 200 with norms

{
    "document": {
        "id": "dc065bc3-e1da-443d-b369-fad58cd184fd",
        "position": 4,
        "data_source_type": "upload_file",
        "data_source_info": {
            "upload_file_id": "6fce037a-2002-4067-8358-bd688db7a6b2"
        },
        "data_source_detail_dict": {
            "upload_file": {
                "id": "6fce037a-2002-4067-8358-bd688db7a6b2",
                "name": "text",
                "size": 4,
                "extension": "txt",
                "mime_type": "text/plain",
                "created_by": "72c3d0a8-7f75-4420-93d1-f874225dbcbd",
                "created_at": 1764225816.066173
            }
        },
        "dataset_process_rule_id": "1c531502-4338-4f19-917b-d4ccb545d381",
        "name": "text",
        "created_from": "api",
        "created_by": "72c3d0a8-7f75-4420-93d1-f874225dbcbd",
        "created_at": 1764213476,
        "tokens": 0,
        "indexing_status": "parsing",
        "error": null,
        "enabled": true,
        "disabled_at": null,
        "disabled_by": null,
        "archived": false,
        "display_status": "indexing",
        "word_count": 0,
        "hit_count": 0,
        "doc_form": "hierarchical_model",
        "doc_metadata": [
            {
                "id": "built-in",
                "name": "document_name",
                "type": "string",
                "value": "text"
            },
            {
                "id": "built-in",
                "name": "uploader",
                "type": "string",
                "value": "talenhon"
            },
            {
                "id": "built-in",
                "name": "upload_date",
                "type": "time",
                "value": "1764213476.0"
            },
            {
                "id": "built-in",
                "name": "last_update_date",
                "type": "time",
                "value": "1764225816.188019"
            },
            {
                "id": "built-in",
                "name": "source",
                "type": "string",
                "value": "file_upload"
            }
        ]
    },
    "batch": "20251127064336633034"
}

Worker Log -- No error

2025-11-27 10:30:55.699 INFO [MainThread] [strategy.py:161] - Task tasks.document_indexing_task.priority_document_indexing_task[cf250bc9-b9c4-4c8d-854a-32a5ac08a01e] received
2025-11-27 10:30:55.700 INFO [Dummy-450] [document_indexing_task.py:167] - priority document indexing task received: 12763198-a92e-4b38-b350-9555e78f8aba - d55317b7-627a-4442-a09b-b1a74b9dd4c0 - ['1635f1fa-3338-46b1-b3e1-ca96efb86c62']
2025-11-27 10:30:55.702 INFO [Dummy-450] [document_indexing_task.py:85] - Start process document: 1635f1fa-3338-46b1-b3e1-ca96efb86c62
2025-11-27 10:30:55.850 INFO [Dummy-450] [_client.py:1038] - HTTP Request: GET http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/management/models?page=1&page_size=256 "HTTP/1.1 200 OK"
2025-11-27 10:30:55.925 INFO [Dummy-450] [_client.py:1038] - HTTP Request: POST http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/dispatch/text_embedding/num_tokens "HTTP/1.1 200 OK"
2025-11-27 10:30:55.981 INFO [Dummy-450] [document_indexing_task.py:102] - Processed dataset: d55317b7-627a-4442-a09b-b1a74b9dd4c0 latency: 0.2808062300318852
2025-11-27 10:30:55.981 INFO [Dummy-450] [document_indexing_task.py:125] - document indexing tenant isolation queue next tasks: []
2025-11-27 10:30:56.015 INFO [Dummy-450] [trace.py:128] - Task tasks.document_indexing_task.priority_document_indexing_task[cf250bc9-b9c4-4c8d-854a-32a5ac08a01e] succeeded in 0.3156577539630234s: None
2025-11-27 10:31:07.682 INFO [MainThread] [strategy.py:161] - Task schedule.workflow_schedule_task.poll_workflow_schedules[79bc64ba-3bc9-4463-84bc-8a0ec735f446] received
2025-11-27 10:31:07.719 INFO [Dummy-451] [trace.py:128] - Task schedule.workflow_schedule_task.poll_workflow_schedules[79bc64ba-3bc9-4463-84bc-8a0ec735f446] succeeded in 0.03594772703945637s: None

Here's From the UI console

#	NAME	CHUNKING MODE	WORDS	RETRIEVAL COUNT	UPLOAD TIME	STATUS	ACTION
1	text	Parent-child	0	0	11/27/2025 06:30 PM	Available

@talenhon commented on GitHub (Nov 27, 2025): @dosu Here's the request body: both the "name" and "text" fields must be present and non-empty ``` { "name": "text", "text": "text", "indexing_technique": "high_quality", "process_rule": {"mode": "automatic"}, "doc_form": "hierarchical_model", "doc_language": "English" } ``` And the response: 200 with norms ``` { "document": { "id": "dc065bc3-e1da-443d-b369-fad58cd184fd", "position": 4, "data_source_type": "upload_file", "data_source_info": { "upload_file_id": "6fce037a-2002-4067-8358-bd688db7a6b2" }, "data_source_detail_dict": { "upload_file": { "id": "6fce037a-2002-4067-8358-bd688db7a6b2", "name": "text", "size": 4, "extension": "txt", "mime_type": "text/plain", "created_by": "72c3d0a8-7f75-4420-93d1-f874225dbcbd", "created_at": 1764225816.066173 } }, "dataset_process_rule_id": "1c531502-4338-4f19-917b-d4ccb545d381", "name": "text", "created_from": "api", "created_by": "72c3d0a8-7f75-4420-93d1-f874225dbcbd", "created_at": 1764213476, "tokens": 0, "indexing_status": "parsing", "error": null, "enabled": true, "disabled_at": null, "disabled_by": null, "archived": false, "display_status": "indexing", "word_count": 0, "hit_count": 0, "doc_form": "hierarchical_model", "doc_metadata": [ { "id": "built-in", "name": "document_name", "type": "string", "value": "text" }, { "id": "built-in", "name": "uploader", "type": "string", "value": "talenhon" }, { "id": "built-in", "name": "upload_date", "type": "time", "value": "1764213476.0" }, { "id": "built-in", "name": "last_update_date", "type": "time", "value": "1764225816.188019" }, { "id": "built-in", "name": "source", "type": "string", "value": "file_upload" } ] }, "batch": "20251127064336633034" } ``` Worker Log -- No error ``` 2025-11-27 10:30:55.699 INFO [MainThread] [strategy.py:161] - Task tasks.document_indexing_task.priority_document_indexing_task[cf250bc9-b9c4-4c8d-854a-32a5ac08a01e] received 2025-11-27 10:30:55.700 INFO [Dummy-450] [document_indexing_task.py:167] - priority document indexing task received: 12763198-a92e-4b38-b350-9555e78f8aba - d55317b7-627a-4442-a09b-b1a74b9dd4c0 - ['1635f1fa-3338-46b1-b3e1-ca96efb86c62'] 2025-11-27 10:30:55.702 INFO [Dummy-450] [document_indexing_task.py:85] - Start process document: 1635f1fa-3338-46b1-b3e1-ca96efb86c62 2025-11-27 10:30:55.850 INFO [Dummy-450] [_client.py:1038] - HTTP Request: GET http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/management/models?page=1&page_size=256 "HTTP/1.1 200 OK" 2025-11-27 10:30:55.925 INFO [Dummy-450] [_client.py:1038] - HTTP Request: POST http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/dispatch/text_embedding/num_tokens "HTTP/1.1 200 OK" 2025-11-27 10:30:55.981 INFO [Dummy-450] [document_indexing_task.py:102] - Processed dataset: d55317b7-627a-4442-a09b-b1a74b9dd4c0 latency: 0.2808062300318852 2025-11-27 10:30:55.981 INFO [Dummy-450] [document_indexing_task.py:125] - document indexing tenant isolation queue next tasks: [] 2025-11-27 10:30:56.015 INFO [Dummy-450] [trace.py:128] - Task tasks.document_indexing_task.priority_document_indexing_task[cf250bc9-b9c4-4c8d-854a-32a5ac08a01e] succeeded in 0.3156577539630234s: None 2025-11-27 10:31:07.682 INFO [MainThread] [strategy.py:161] - Task schedule.workflow_schedule_task.poll_workflow_schedules[79bc64ba-3bc9-4463-84bc-8a0ec735f446] received 2025-11-27 10:31:07.719 INFO [Dummy-451] [trace.py:128] - Task schedule.workflow_schedule_task.poll_workflow_schedules[79bc64ba-3bc9-4463-84bc-8a0ec735f446] succeeded in 0.03594772703945637s: None ``` Here's From the UI console | # | NAME | CHUNKING MODE | WORDS | RETRIEVAL COUNT | UPLOAD TIME | STATUS | ACTION | | -- | -- | -- | -- | -- | -- | -- | -- | | 1 | text | Parent-child | 0 | 0 | 11/27/2025 06:30 PM | Available | |

yindo commented 2026-02-21 20:08:06 -05:00

Author

Owner

Copy Link

Copy Source

@dosubot[bot] commented on GitHub (Nov 27, 2025):

This behavior is expected with such a minimal payload. Dify's indexing pipeline skips segments if the cleaned or split content is empty or trivial—very short text like "text" often results in zero segments, so the document appears empty even though the API call succeeds and the worker logs show no errors. This is by design: the segmenter drops nodes where the content is empty or only whitespace after cleaning and splitting.

To confirm, try uploading a longer, more substantial text via the API. If the document then shows a nonzero word count, the system is working as intended for non-trivial content. If longer texts also result in empty documents, let me know and we can dig into configuration or environment issues (like storage, FILES_URL, or worker queues).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

@dosubot[bot] commented on GitHub (Nov 27, 2025): <!-- Answer --> This behavior is expected with such a minimal payload. Dify's indexing pipeline skips segments if the cleaned or split content is empty or trivial—very short text like "text" often results in zero segments, so the document appears empty even though the API call succeeds and the worker logs show no errors. This is by design: the segmenter drops nodes where the content is empty or only whitespace after cleaning and splitting. To confirm, try uploading a longer, more substantial text via the API. If the document then shows a nonzero word count, the system is working as intended for non-trivial content. If longer texts also result in empty documents, let me know and we can dig into configuration or environment issues (like storage, FILES_URL, or worker queues). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- ^{How did I do? [Good](https://app.dosu.dev/response-feedback/93c268ba-df5c-4409-987a-927aa5df2373?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/93c268ba-df5c-4409-987a-927aa5df2373?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/93c268ba-df5c-4409-987a-927aa5df2373?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/93c268ba-df5c-4409-987a-927aa5df2373?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/93c268ba-df5c-4409-987a-927aa5df2373?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/93c268ba-df5c-4409-987a-927aa5df2373?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/93c268ba-df5c-4409-987a-927aa5df2373?feedback_type=other)} [](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[](https://cursor.com/link/prompt?text=This%20behavior%20is%20expected%20with%20such%20a%20minimal%20payload.%20Dify%27s%20indexing%20pipeline%20skips%20segments%20if%20the%20cleaned%20or%20split%20content%20is%20empty%20or%20trivial%E2%80%94very%20short%20text%20like%20%22text%22%20often%20results%20in%20zero%20segments%2C%20so%20the%20document%20appears%20empty%20even%20though%20the%20API%20call%20succeeds%20and%20the%20worker%20logs%20show%20no%20errors.%20This%20is%20by%20design%3A%20the%20segmenter%20drops%20nodes%20where%20the%20content%20is%20empty%20or%20only%20whitespace%20after%20cleaning%20and%20splitting.%0A%0ATo%20confirm%2C%20try%20uploading%20a%20longer%2C%20more%20substantial%20text%20via%20the%20API.%20If%20the%20document%20then%20shows%20a%20nonzero%20word%20count%2C%20the%20system%20is%20working%20as%20intended%20for%20non-trivial%20content.%20If%20longer%20texts%20also%20result%20in%20empty%20documents%2C%20let%20me%20know%20and%20we%20can%20dig%20into%20configuration%20or%20environment%20issues%20%28like%20storage%2C%20FILES_URL%2C%20or%20worker%20queues%29.)&nbsp;[](https://go.dosu.dev/discord-bot)&nbsp;[](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28763)

yindo commented 2026-02-21 20:08:06 -05:00

Author

Owner

Copy Link

Copy Source

@talenhon commented on GitHub (Nov 27, 2025):

@dosu Tried a long one

{"name": "IELTS","text": "We are all explorers. Our desire to discover, and then share that new-found knowledge, is part of what makes us human – indeed, this has played an important part in our success as a species. Long before the first caveman slumped down beside the fire and grunted news that there were plenty of wildebeest over yonder, our ancestors had learnt the value of sending out scouts to investigate the unknown. This questing nature of ours undoubtedly helped our species spread around the globe, just as it nowadays no doubt helps the last nomadic Penan maintain their existence in the depleted forests of Borneo, and a visitor negotiate the subways of New York.\n\nOver the years, we’ve come to think of explorers as a peculiar breed – different from the rest of us, different from those of us who are merely ‘well travelled’, even; and perhaps there is a type of person more suited to seeking out the new, a type of caveman more inclined to risk venturing out. That, however, doesn’t take away from the fact that we all have this enquiring instinct, even today; and that in all sorts of professions – whether artist, marine biologist or astronomer – borders of the unknown are being tested each day.\n\nThomas Hardy set some of his novels in Egdon Heath, a fictional area of uncultivated land, and used the landscape to suggest the desires and fears of his characters. He is delving into matters we all recognise because they are common to humanity. This is surely an act of exploration, and into a world as remote as the author chooses. Explorer and travel writer Peter Fleming talks of the moment when the explorer returns to the existence he has left behind with his loved ones. The traveller ‘who has for weeks or months seen himself only as a puny and irrelevant alien crawling laboriously over a country in which he has no roots and no background, suddenly encounters his other self, a relatively solid figure, with a place in the minds of certain people’. \n\n","indexing_technique": "high_quality","process_rule": {"mode": "automatic"},    "doc_form": "hierarchical_model",
    "doc_language": "English"}

HTTP response:

{
    "document": {
        "id": "3fea1f57-597b-413c-b8c8-b458f5f4260c",
        "position": 4,
        "data_source_type": "upload_file",
        "data_source_info": {
            "upload_file_id": "e8cc08fe-1b60-4d13-9b21-80c9baa79933"
        },
        "data_source_detail_dict": {
            "upload_file": {
                "id": "e8cc08fe-1b60-4d13-9b21-80c9baa79933",
                "name": "IELTS",
                "size": 1954,
                "extension": "txt",
                "mime_type": "text/plain",
                "created_by": "72c3d0a8-7f75-4420-93d1-f874225dbcbd",
                "created_at": 1764259897.735305
            }
        },
        "dataset_process_rule_id": "e93cc0b6-77c6-4733-903b-dd80d0737124",
        "name": "IELTS",
        "created_from": "api",
        "created_by": "72c3d0a8-7f75-4420-93d1-f874225dbcbd",
        "created_at": 1764259898,
        "tokens": null,
        "indexing_status": "parsing",
        "error": null,
        "enabled": true,
        "disabled_at": null,
        "disabled_by": null,
        "archived": false,
        "display_status": "indexing",
        "word_count": null,
        "hit_count": 0,
        "doc_form": "hierarchical_model",
        "doc_metadata": [
            {
                "id": "built-in",
                "name": "document_name",
                "type": "string",
                "value": "IELTS"
            },
            {
                "id": "built-in",
                "name": "uploader",
                "type": "string",
                "value": "talenhon"
            },
            {
                "id": "built-in",
                "name": "upload_date",
                "type": "time",
                "value": "1764259898.0"
            },
            {
                "id": "built-in",
                "name": "last_update_date",
                "type": "time",
                "value": "1764259897.759809"
            },
            {
                "id": "built-in",
                "name": "source",
                "type": "string",
                "value": "file_upload"
            }
        ]
    },
    "batch": "20251127161137795537"
}

Worker Log

2025-11-27 16:11:37.758 INFO [MainThread] [strategy.py:161] - Task tasks.document_indexing_task.priority_document_indexing_task[832d00cd-b1ed-4461-adf5-e398a9773f48] received
2025-11-27 16:11:37.758 INFO [Dummy-820] [document_indexing_task.py:167] - priority document indexing task received: 12763198-a92e-4b38-b350-9555e78f8aba - d55317b7-627a-4442-a09b-b1a74b9dd4c0 - ['3fea1f57-597b-413c-b8c8-b458f5f4260c']
2025-11-27 16:11:37.761 INFO [Dummy-820] [document_indexing_task.py:85] - Start process document: 3fea1f57-597b-413c-b8c8-b458f5f4260c
2025-11-27 16:11:37.921 INFO [Dummy-820] [_client.py:1038] - HTTP Request: GET http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/management/models?page=1&page_size=256 "HTTP/1.1 200 OK"
2025-11-27 16:11:37.998 INFO [Dummy-820] [_client.py:1038] - HTTP Request: POST http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/dispatch/text_embedding/num_tokens "HTTP/1.1 200 OK"
2025-11-27 16:11:38.045 INFO [Dummy-820] [document_indexing_task.py:102] - Processed dataset: d55317b7-627a-4442-a09b-b1a74b9dd4c0 latency: 0.2867944350000471
2025-11-27 16:11:38.046 INFO [Dummy-820] [document_indexing_task.py:125] - document indexing tenant isolation queue next tasks: []
2025-11-27 16:11:38.073 INFO [Dummy-820] [trace.py:128] - Task tasks.document_indexing_task.priority_document_indexing_task[832d00cd-b1ed-4461-adf5-e398a9773f48] succeeded in 0.315259046969004s: None

#	NAME	CHUNKING MODE	WORDS	RETRIEVAL COUNT	UPLOAD TIME	STATUS	ACTION
1	IELTS	Parent-child	0	0	11/28/2025 12:11 AM	Available

@talenhon commented on GitHub (Nov 27, 2025): @dosu Tried a long one ``` {"name": "IELTS","text": "We are all explorers. Our desire to discover, and then share that new-found knowledge, is part of what makes us human – indeed, this has played an important part in our success as a species. Long before the first caveman slumped down beside the fire and grunted news that there were plenty of wildebeest over yonder, our ancestors had learnt the value of sending out scouts to investigate the unknown. This questing nature of ours undoubtedly helped our species spread around the globe, just as it nowadays no doubt helps the last nomadic Penan maintain their existence in the depleted forests of Borneo, and a visitor negotiate the subways of New York.\n\nOver the years, we’ve come to think of explorers as a peculiar breed – different from the rest of us, different from those of us who are merely ‘well travelled’, even; and perhaps there is a type of person more suited to seeking out the new, a type of caveman more inclined to risk venturing out. That, however, doesn’t take away from the fact that we all have this enquiring instinct, even today; and that in all sorts of professions – whether artist, marine biologist or astronomer – borders of the unknown are being tested each day.\n\nThomas Hardy set some of his novels in Egdon Heath, a fictional area of uncultivated land, and used the landscape to suggest the desires and fears of his characters. He is delving into matters we all recognise because they are common to humanity. This is surely an act of exploration, and into a world as remote as the author chooses. Explorer and travel writer Peter Fleming talks of the moment when the explorer returns to the existence he has left behind with his loved ones. The traveller ‘who has for weeks or months seen himself only as a puny and irrelevant alien crawling laboriously over a country in which he has no roots and no background, suddenly encounters his other self, a relatively solid figure, with a place in the minds of certain people’. \n\n","indexing_technique": "high_quality","process_rule": {"mode": "automatic"}, "doc_form": "hierarchical_model", "doc_language": "English"} ``` HTTP response: ``` { "document": { "id": "3fea1f57-597b-413c-b8c8-b458f5f4260c", "position": 4, "data_source_type": "upload_file", "data_source_info": { "upload_file_id": "e8cc08fe-1b60-4d13-9b21-80c9baa79933" }, "data_source_detail_dict": { "upload_file": { "id": "e8cc08fe-1b60-4d13-9b21-80c9baa79933", "name": "IELTS", "size": 1954, "extension": "txt", "mime_type": "text/plain", "created_by": "72c3d0a8-7f75-4420-93d1-f874225dbcbd", "created_at": 1764259897.735305 } }, "dataset_process_rule_id": "e93cc0b6-77c6-4733-903b-dd80d0737124", "name": "IELTS", "created_from": "api", "created_by": "72c3d0a8-7f75-4420-93d1-f874225dbcbd", "created_at": 1764259898, "tokens": null, "indexing_status": "parsing", "error": null, "enabled": true, "disabled_at": null, "disabled_by": null, "archived": false, "display_status": "indexing", "word_count": null, "hit_count": 0, "doc_form": "hierarchical_model", "doc_metadata": [ { "id": "built-in", "name": "document_name", "type": "string", "value": "IELTS" }, { "id": "built-in", "name": "uploader", "type": "string", "value": "talenhon" }, { "id": "built-in", "name": "upload_date", "type": "time", "value": "1764259898.0" }, { "id": "built-in", "name": "last_update_date", "type": "time", "value": "1764259897.759809" }, { "id": "built-in", "name": "source", "type": "string", "value": "file_upload" } ] }, "batch": "20251127161137795537" } ``` Worker Log ``` 2025-11-27 16:11:37.758 INFO [MainThread] [strategy.py:161] - Task tasks.document_indexing_task.priority_document_indexing_task[832d00cd-b1ed-4461-adf5-e398a9773f48] received 2025-11-27 16:11:37.758 INFO [Dummy-820] [document_indexing_task.py:167] - priority document indexing task received: 12763198-a92e-4b38-b350-9555e78f8aba - d55317b7-627a-4442-a09b-b1a74b9dd4c0 - ['3fea1f57-597b-413c-b8c8-b458f5f4260c'] 2025-11-27 16:11:37.761 INFO [Dummy-820] [document_indexing_task.py:85] - Start process document: 3fea1f57-597b-413c-b8c8-b458f5f4260c 2025-11-27 16:11:37.921 INFO [Dummy-820] [_client.py:1038] - HTTP Request: GET http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/management/models?page=1&page_size=256 "HTTP/1.1 200 OK" 2025-11-27 16:11:37.998 INFO [Dummy-820] [_client.py:1038] - HTTP Request: POST http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/dispatch/text_embedding/num_tokens "HTTP/1.1 200 OK" 2025-11-27 16:11:38.045 INFO [Dummy-820] [document_indexing_task.py:102] - Processed dataset: d55317b7-627a-4442-a09b-b1a74b9dd4c0 latency: 0.2867944350000471 2025-11-27 16:11:38.046 INFO [Dummy-820] [document_indexing_task.py:125] - document indexing tenant isolation queue next tasks: [] 2025-11-27 16:11:38.073 INFO [Dummy-820] [trace.py:128] - Task tasks.document_indexing_task.priority_document_indexing_task[832d00cd-b1ed-4461-adf5-e398a9773f48] succeeded in 0.315259046969004s: None ``` | # | NAME | CHUNKING MODE | WORDS | RETRIEVAL COUNT | UPLOAD TIME | STATUS | ACTION | | -- | -- | -- | -- | -- | -- | -- | -- | 1 | IELTS | Parent-child | 0 | 0 | 11/28/2025 12:11 AM | Available |   |

yindo commented 2026-02-21 20:08:06 -05:00

Author

Owner

Copy Link

Copy Source

@dosubot[bot] commented on GitHub (Nov 27, 2025):

Since uploading via the UI works but API uploads create empty documents (even with substantial text), the most likely cause is a storage access/configuration issue. In Dify, the worker processes API-uploaded files by reading them directly from the storage backend using the file key, not via HTTP or FILES_URL. If the worker container can't access the storage (e.g., local volume not mounted, wrong STORAGE_TYPE, or missing credentials for S3/Azure), extraction silently fails and the document remains empty—even if logs show no error and the API returns 200 OK. This is a known issue and matches other user reports where API-created files are empty or 0 bytes due to storage misconfiguration or missing volume mounts, while UI uploads work because the web server and worker share the same storage context (source).

To resolve:

• Make sure both the API and worker containers have access to the same storage backend and, for local storage, the same mounted volume (e.g., ./volumes/app/storage:/app/api/storage).
• After an API upload, exec into the worker container and check that the uploaded file exists and is non-empty at the expected path.
• Double-check STORAGE_TYPE and related credentials in both containers.
• If using S3/Azure/etc., verify the worker can access the storage service.

If all storage settings are correct and the problem persists, enable debug logging in the worker and check for silent extraction or storage errors during document indexing.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

@dosubot[bot] commented on GitHub (Nov 27, 2025): <!-- Answer --> Since uploading via the UI works but API uploads create empty documents (even with substantial text), the most likely cause is a storage access/configuration issue. In Dify, the worker processes API-uploaded files by reading them directly from the storage backend using the file key, not via HTTP or FILES_URL. If the worker container can't access the storage (e.g., local volume not mounted, wrong STORAGE_TYPE, or missing credentials for S3/Azure), extraction silently fails and the document remains empty—even if logs show no error and the API returns 200 OK. This is a known issue and matches other user reports where API-created files are empty or 0 bytes due to storage misconfiguration or missing volume mounts, while UI uploads work because the web server and worker share the same storage context ([source](https://github.com/langgenius/dify/issues/22694)). To resolve: - Make sure both the API and worker containers have access to the same storage backend and, for local storage, the same mounted volume (e.g., `./volumes/app/storage:/app/api/storage`). - After an API upload, exec into the worker container and check that the uploaded file exists and is non-empty at the expected path. - Double-check STORAGE_TYPE and related credentials in both containers. - If using S3/Azure/etc., verify the worker can access the storage service. If all storage settings are correct and the problem persists, enable debug logging in the worker and check for silent extraction or storage errors during document indexing. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- ^{How did I do? [Good](https://app.dosu.dev/response-feedback/65637824-28ce-4fc5-8ced-ded06a474b50?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/65637824-28ce-4fc5-8ced-ded06a474b50?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/65637824-28ce-4fc5-8ced-ded06a474b50?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/65637824-28ce-4fc5-8ced-ded06a474b50?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/65637824-28ce-4fc5-8ced-ded06a474b50?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/65637824-28ce-4fc5-8ced-ded06a474b50?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/65637824-28ce-4fc5-8ced-ded06a474b50?feedback_type=other)} [](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[](https://cursor.com/link/prompt?text=Since%20uploading%20via%20the%20UI%20works%20but%20API%20uploads%20create%20empty%20documents%20%28even%20with%20substantial%20text%29%2C%20the%20most%20likely%20cause%20is%20a%20storage%20access/configuration%20issue.%20In%20Dify%2C%20the%20worker%20processes%20API-uploaded%20files%20by%20reading%20them%20directly%20from%20the%20storage%20backend%20using%20the%20file%20key%2C%20not%20via%20HTTP%20or%20FILES_URL.%20If%20the%20worker%20container%20can%27t%20access%20the%20storage%20%28e.g.%2C%20local%20volume%20not%20mounted%2C%20wrong%20STORAGE_TYPE%2C%20or%20missing%20credentials%20for%20S3/Azure%29%2C%20extraction%20silently%20fails%20and%20the%20document%20remains%20empty%E2%80%94even%20if%20logs%20show%20no%20error%20and%20the%20API%20returns%20200%20OK.%20This%20is%20a%20known%20issue%20and%20matches%20other%20user%20reports%20where%20API-created%20files%20are%20empty%20or%200%20bytes%20due%20to%20storage%20misconfiguration%20or%20missing%20volume%20mounts%2C%20while%20UI%20uploads%20work%20because%20the%20web%20server%20and%20worker%20share%20the%20same%20storage%20context%20%28%5Bsource%5D%28https%3A//github.com/langgenius/dify/issues/22694%29%29.%0A%0ATo%20resolve%3A%0A-%20Make%20sure%20both%20the%20API%20and%20worker%20containers%20have%20access%20to%20the%20same%20storage%20backend%20and%2C%20for%20local%20storage%2C%20the%20same%20mounted%20volume%20%28e.g.%2C%20%60./volumes/app/storage%3A/app/api/storage%60%29.%0A-%20After%20an%20API%20upload%2C%20exec%20into%20the%20worker%20container%20and%20check%20that%20the%20uploaded%20file%20exists%20and%20is%20non-empty%20at%20the%20expected%20path.%0A-%20Double-check%20STORAGE_TYPE%20and%20related%20credentials%20in%20both%20containers.%0A-%20If%20using%20S3/Azure/etc.%2C%20verify%20the%20worker%20can%20access%20the%20storage%20service.%0A%0AIf%20all%20storage%20settings%20are%20correct%20and%20the%20problem%20persists%2C%20enable%20debug%20logging%20in%20the%20worker%20and%20check%20for%20silent%20extraction%20or%20storage%20errors%20during%20document%20indexing.)&nbsp;[](https://go.dosu.dev/discord-bot)&nbsp;[](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28763)

yindo commented 2026-02-21 20:08:07 -05:00

Author

Owner

Copy Link

Copy Source

@talenhon commented on GitHub (Nov 30, 2025):

@dosu

1. API and worker containers have access to the same storage backend.

2. Tried both opendl and s3, both storage shows upload complete with a upload file exist in the storage. (1.9kb in size)

3. Enabled logging on worker

2025-12-01 01:15:34.440 INFO [MainThread] [strategy.py:161] - Task tasks.document_indexing_task.priority_document_indexing_task[a5c4b0d7-783f-47d5-9369-543606cdd1d9] received
2025-12-01 01:15:34.440 DEBUG [MainThread] [base.py:149] - TaskPool: Apply <function fast_trace_task at 0xffff7ff0fa60> (args:('tasks.document_indexing_task.priority_document_indexing_task', 'a5c4b0d7-783f-47d5-9369-543606cdd1d9', {'lang': 'py', 'task': 'tasks.document_indexing_task.priority_document_indexing_task', 'id': 'a5c4b0d7-783f-47d5-9369-543606cdd1d9', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a5c4b0d7-783f-47d5-9369-543606cdd1d9', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'tenant_id': '12763198-a92e-4b38-b350-9555e78f8aba', 'dataset_id': 'd55317b7-627a-4442-a09b-b1a74b9dd4c0', 'document_ids': ['bce58a44-a0d7-4e5a-9c93-ba5f31f84cc5']}", 'origin': 'gen18@019b8756b420', 'ignore_result': True, 'replaced_task_nesting': 0, 'stamped_headers': None, 'stamps': {}, 'properties': {'correlation_id': 'a5c4b0d7-783f-47d5-9369-543606cdd1d9', 'reply_to': '11bcfd7f-07ee-3a49-bfb5-0834cc22010b', 'delivery_mode': 2, 'delivery_info': {'exchange': '', 'routing_key': 'priority_dataset'}, 'priority': 0, 'body_encoding': 'base64',... kwargs:{})
2025-12-01 01:15:34.440 INFO [Dummy-14] [document_indexing_task.py:167] - priority document indexing task received: 12763198-a92e-4b38-b350-9555e78f8aba - d55317b7-627a-4442-a09b-b1a74b9dd4c0 - ['bce58a44-a0d7-4e5a-9c93-ba5f31f84cc5']
2025-12-01 01:15:34.443 INFO [Dummy-14] [document_indexing_task.py:85] - Start process document: bce58a44-a0d7-4e5a-9c93-ba5f31f84cc5
2025-12-01 01:15:34.453 DEBUG [Dummy-14] [opendal_storage.py:74] - file upload_files/12763198-a92e-4b38-b350-9555e78f8aba/bdbc6b13-c1c8-48c2-b045-0751840674ce.txt downloaded to /tmp/tmpliz9k4de/jpbhwmou.txt
2025-12-01 01:15:34.466 DEBUG [Dummy-14] [_config.py:82] - load_ssl_context verify=True cert=None trust_env=True http2=False
2025-12-01 01:15:34.467 DEBUG [Dummy-14] [_config.py:148] - load_verify_locations cafile='/app/api/.venv/lib/python3.12/site-packages/certifi/cacert.pem'
2025-12-01 01:15:34.505 DEBUG [Dummy-14] [_trace.py:47] - connect_tcp.started host='plugin_daemon' port=5002 local_address=None timeout=300.0 socket_options=None
2025-12-01 01:15:34.509 DEBUG [Dummy-14] [_trace.py:47] - connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0xffff601675f0>
2025-12-01 01:15:34.509 DEBUG [Dummy-14] [_trace.py:47] - send_request_headers.started request=<Request [b'GET']>
2025-12-01 01:15:34.509 DEBUG [Dummy-14] [_trace.py:47] - send_request_headers.complete
2025-12-01 01:15:34.510 DEBUG [Dummy-14] [_trace.py:47] - send_request_body.started request=<Request [b'GET']>
2025-12-01 01:15:34.510 DEBUG [Dummy-14] [_trace.py:47] - send_request_body.complete
2025-12-01 01:15:34.510 DEBUG [Dummy-14] [_trace.py:47] - receive_response_headers.started request=<Request [b'GET']>
2025-12-01 01:15:34.512 DEBUG [Dummy-14] [_trace.py:47] - receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Content-Type', b'application/json; charset=utf-8'), (b'Date', b'Mon, 01 Dec 2025 01:15:34 GMT'), (b'Transfer-Encoding', b'chunked')])
2025-12-01 01:15:34.513 INFO [Dummy-14] [_client.py:1038] - HTTP Request: GET http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/management/models?page=1&page_size=256 "HTTP/1.1 200 OK"
2025-12-01 01:15:34.513 DEBUG [Dummy-14] [_trace.py:47] - receive_response_body.started request=<Request [b'GET']>
2025-12-01 01:15:34.514 DEBUG [Dummy-14] [_trace.py:47] - receive_response_body.complete
2025-12-01 01:15:34.514 DEBUG [Dummy-14] [_trace.py:47] - response_closed.started
2025-12-01 01:15:34.514 DEBUG [Dummy-14] [_trace.py:47] - response_closed.complete
2025-12-01 01:15:34.515 DEBUG [Dummy-14] [_trace.py:47] - close.started
2025-12-01 01:15:34.515 DEBUG [Dummy-14] [_trace.py:47] - close.complete
2025-12-01 01:15:34.547 DEBUG [Dummy-14] [_config.py:82] - load_ssl_context verify=True cert=None trust_env=True http2=False
2025-12-01 01:15:34.548 DEBUG [Dummy-14] [_config.py:148] - load_verify_locations cafile='/app/api/.venv/lib/python3.12/site-packages/certifi/cacert.pem'
2025-12-01 01:15:34.585 DEBUG [Dummy-14] [_trace.py:47] - connect_tcp.started host='plugin_daemon' port=5002 local_address=None timeout=300.0 socket_options=None
2025-12-01 01:15:34.587 DEBUG [Dummy-14] [_trace.py:47] - connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0xffff60058860>
2025-12-01 01:15:34.587 DEBUG [Dummy-14] [_trace.py:47] - send_request_headers.started request=<Request [b'POST']>
2025-12-01 01:15:34.587 DEBUG [Dummy-14] [_trace.py:47] - send_request_headers.complete
2025-12-01 01:15:34.588 DEBUG [Dummy-14] [_trace.py:47] - send_request_body.started request=<Request [b'POST']>
2025-12-01 01:15:34.589 DEBUG [Dummy-14] [_trace.py:47] - send_request_body.complete
2025-12-01 01:15:34.589 DEBUG [Dummy-14] [_trace.py:47] - receive_response_headers.started request=<Request [b'POST']>
2025-12-01 01:15:34.591 DEBUG [Dummy-14] [_trace.py:47] - receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Content-Type', b'text/event-stream'), (b'Date', b'Mon, 01 Dec 2025 01:15:34 GMT'), (b'Transfer-Encoding', b'chunked')])
2025-12-01 01:15:34.592 INFO [Dummy-14] [_client.py:1038] - HTTP Request: POST http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/dispatch/text_embedding/num_tokens "HTTP/1.1 200 OK"
2025-12-01 01:15:34.592 DEBUG [Dummy-14] [_trace.py:47] - receive_response_body.started request=<Request [b'POST']>
2025-12-01 01:15:34.593 DEBUG [Dummy-14] [_trace.py:47] - receive_response_body.failed exception=GeneratorExit()
2025-12-01 01:15:34.594 DEBUG [Dummy-14] [_trace.py:47] - response_closed.started
2025-12-01 01:15:34.594 DEBUG [Dummy-14] [_trace.py:47] - response_closed.complete
2025-12-01 01:15:34.625 INFO [Dummy-14] [document_indexing_task.py:102] - Processed dataset: d55317b7-627a-4442-a09b-b1a74b9dd4c0 latency: 0.1841462339507416
2025-12-01 01:15:34.626 INFO [Dummy-14] [document_indexing_task.py:125] - document indexing tenant isolation queue next tasks: []
2025-12-01 01:15:34.658 INFO [Dummy-14] [trace.py:128] - Task tasks.document_indexing_task.priority_document_indexing_task[a5c4b0d7-783f-47d5-9369-543606cdd1d9] succeeded in 0.21797144296579063s: None

@talenhon commented on GitHub (Nov 30, 2025): @dosu 1. API and worker containers have access to the same storage backend. 2. Tried both opendl and s3, both storage shows upload complete with a upload file exist in the storage. (1.9kb in size) 3. Enabled logging on worker ``` 2025-12-01 01:15:34.440 INFO [MainThread] [strategy.py:161] - Task tasks.document_indexing_task.priority_document_indexing_task[a5c4b0d7-783f-47d5-9369-543606cdd1d9] received 2025-12-01 01:15:34.440 DEBUG [MainThread] [base.py:149] - TaskPool: Apply <function fast_trace_task at 0xffff7ff0fa60> (args:('tasks.document_indexing_task.priority_document_indexing_task', 'a5c4b0d7-783f-47d5-9369-543606cdd1d9', {'lang': 'py', 'task': 'tasks.document_indexing_task.priority_document_indexing_task', 'id': 'a5c4b0d7-783f-47d5-9369-543606cdd1d9', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a5c4b0d7-783f-47d5-9369-543606cdd1d9', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'tenant_id': '12763198-a92e-4b38-b350-9555e78f8aba', 'dataset_id': 'd55317b7-627a-4442-a09b-b1a74b9dd4c0', 'document_ids': ['bce58a44-a0d7-4e5a-9c93-ba5f31f84cc5']}", 'origin': 'gen18@019b8756b420', 'ignore_result': True, 'replaced_task_nesting': 0, 'stamped_headers': None, 'stamps': {}, 'properties': {'correlation_id': 'a5c4b0d7-783f-47d5-9369-543606cdd1d9', 'reply_to': '11bcfd7f-07ee-3a49-bfb5-0834cc22010b', 'delivery_mode': 2, 'delivery_info': {'exchange': '', 'routing_key': 'priority_dataset'}, 'priority': 0, 'body_encoding': 'base64',... kwargs:{}) 2025-12-01 01:15:34.440 INFO [Dummy-14] [document_indexing_task.py:167] - priority document indexing task received: 12763198-a92e-4b38-b350-9555e78f8aba - d55317b7-627a-4442-a09b-b1a74b9dd4c0 - ['bce58a44-a0d7-4e5a-9c93-ba5f31f84cc5'] 2025-12-01 01:15:34.443 INFO [Dummy-14] [document_indexing_task.py:85] - Start process document: bce58a44-a0d7-4e5a-9c93-ba5f31f84cc5 2025-12-01 01:15:34.453 DEBUG [Dummy-14] [opendal_storage.py:74] - file upload_files/12763198-a92e-4b38-b350-9555e78f8aba/bdbc6b13-c1c8-48c2-b045-0751840674ce.txt downloaded to /tmp/tmpliz9k4de/jpbhwmou.txt 2025-12-01 01:15:34.466 DEBUG [Dummy-14] [_config.py:82] - load_ssl_context verify=True cert=None trust_env=True http2=False 2025-12-01 01:15:34.467 DEBUG [Dummy-14] [_config.py:148] - load_verify_locations cafile='/app/api/.venv/lib/python3.12/site-packages/certifi/cacert.pem' 2025-12-01 01:15:34.505 DEBUG [Dummy-14] [_trace.py:47] - connect_tcp.started host='plugin_daemon' port=5002 local_address=None timeout=300.0 socket_options=None 2025-12-01 01:15:34.509 DEBUG [Dummy-14] [_trace.py:47] - connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0xffff601675f0> 2025-12-01 01:15:34.509 DEBUG [Dummy-14] [_trace.py:47] - send_request_headers.started request=<Request [b'GET']> 2025-12-01 01:15:34.509 DEBUG [Dummy-14] [_trace.py:47] - send_request_headers.complete 2025-12-01 01:15:34.510 DEBUG [Dummy-14] [_trace.py:47] - send_request_body.started request=<Request [b'GET']> 2025-12-01 01:15:34.510 DEBUG [Dummy-14] [_trace.py:47] - send_request_body.complete 2025-12-01 01:15:34.510 DEBUG [Dummy-14] [_trace.py:47] - receive_response_headers.started request=<Request [b'GET']> 2025-12-01 01:15:34.512 DEBUG [Dummy-14] [_trace.py:47] - receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Content-Type', b'application/json; charset=utf-8'), (b'Date', b'Mon, 01 Dec 2025 01:15:34 GMT'), (b'Transfer-Encoding', b'chunked')]) 2025-12-01 01:15:34.513 INFO [Dummy-14] [_client.py:1038] - HTTP Request: GET http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/management/models?page=1&page_size=256 "HTTP/1.1 200 OK" 2025-12-01 01:15:34.513 DEBUG [Dummy-14] [_trace.py:47] - receive_response_body.started request=<Request [b'GET']> 2025-12-01 01:15:34.514 DEBUG [Dummy-14] [_trace.py:47] - receive_response_body.complete 2025-12-01 01:15:34.514 DEBUG [Dummy-14] [_trace.py:47] - response_closed.started 2025-12-01 01:15:34.514 DEBUG [Dummy-14] [_trace.py:47] - response_closed.complete 2025-12-01 01:15:34.515 DEBUG [Dummy-14] [_trace.py:47] - close.started 2025-12-01 01:15:34.515 DEBUG [Dummy-14] [_trace.py:47] - close.complete 2025-12-01 01:15:34.547 DEBUG [Dummy-14] [_config.py:82] - load_ssl_context verify=True cert=None trust_env=True http2=False 2025-12-01 01:15:34.548 DEBUG [Dummy-14] [_config.py:148] - load_verify_locations cafile='/app/api/.venv/lib/python3.12/site-packages/certifi/cacert.pem' 2025-12-01 01:15:34.585 DEBUG [Dummy-14] [_trace.py:47] - connect_tcp.started host='plugin_daemon' port=5002 local_address=None timeout=300.0 socket_options=None 2025-12-01 01:15:34.587 DEBUG [Dummy-14] [_trace.py:47] - connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0xffff60058860> 2025-12-01 01:15:34.587 DEBUG [Dummy-14] [_trace.py:47] - send_request_headers.started request=<Request [b'POST']> 2025-12-01 01:15:34.587 DEBUG [Dummy-14] [_trace.py:47] - send_request_headers.complete 2025-12-01 01:15:34.588 DEBUG [Dummy-14] [_trace.py:47] - send_request_body.started request=<Request [b'POST']> 2025-12-01 01:15:34.589 DEBUG [Dummy-14] [_trace.py:47] - send_request_body.complete 2025-12-01 01:15:34.589 DEBUG [Dummy-14] [_trace.py:47] - receive_response_headers.started request=<Request [b'POST']> 2025-12-01 01:15:34.591 DEBUG [Dummy-14] [_trace.py:47] - receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Content-Type', b'text/event-stream'), (b'Date', b'Mon, 01 Dec 2025 01:15:34 GMT'), (b'Transfer-Encoding', b'chunked')]) 2025-12-01 01:15:34.592 INFO [Dummy-14] [_client.py:1038] - HTTP Request: POST http://plugin_daemon:5002/plugin/12763198-a92e-4b38-b350-9555e78f8aba/dispatch/text_embedding/num_tokens "HTTP/1.1 200 OK" 2025-12-01 01:15:34.592 DEBUG [Dummy-14] [_trace.py:47] - receive_response_body.started request=<Request [b'POST']> 2025-12-01 01:15:34.593 DEBUG [Dummy-14] [_trace.py:47] - receive_response_body.failed exception=GeneratorExit() 2025-12-01 01:15:34.594 DEBUG [Dummy-14] [_trace.py:47] - response_closed.started 2025-12-01 01:15:34.594 DEBUG [Dummy-14] [_trace.py:47] - response_closed.complete 2025-12-01 01:15:34.625 INFO [Dummy-14] [document_indexing_task.py:102] - Processed dataset: d55317b7-627a-4442-a09b-b1a74b9dd4c0 latency: 0.1841462339507416 2025-12-01 01:15:34.626 INFO [Dummy-14] [document_indexing_task.py:125] - document indexing tenant isolation queue next tasks: [] 2025-12-01 01:15:34.658 INFO [Dummy-14] [trace.py:128] - Task tasks.document_indexing_task.priority_document_indexing_task[a5c4b0d7-783f-47d5-9369-543606cdd1d9] succeeded in 0.21797144296579063s: None ```

yindo commented 2026-02-21 20:08:07 -05:00

Author

Owner

Copy Link

Copy Source

@dosubot[bot] commented on GitHub (Nov 30, 2025):

Your storage and worker setup look correct, and the logs confirm the file is downloaded and processed without errors. At this point, the most likely cause is that the document content is being dropped during cleaning or segmentation due to the process_rule or segmentation settings.

Dify's indexing pipeline will silently skip all segments if, after cleaning and splitting, the content is empty or doesn't meet the segmentation criteria (like chunk size or separator). This can happen if the process_rule is too aggressive, misconfigured, or if the payload structure for hierarchical segmentation is incorrect. For example, if you're using hierarchical mode, your payload must include both segmentation and subchunk_segmentation inside process_rule.rules, with parent_mode outside segmentation—otherwise, the API returns 200 OK but creates no segments, resulting in empty documents even for large files (details and example payload).

Try the following:

• Double-check your process_rule structure matches the working example for hierarchical segmentation.
• Switch from "automatic" to "custom" segmentation/cleaning in your API payload and reduce cleaning aggressiveness (e.g., disable remove_urls_emails).
• Test with a minimal, known-good hierarchical payload to rule out silent payload structure errors.
• If possible, add debug logging to capture the cleaned text and split segments during indexing to see if content is being dropped.

If you want, share your exact process_rule and segmentation settings, and I can help review them for subtle issues.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

@dosubot[bot] commented on GitHub (Nov 30, 2025): <!-- Answer --> Your storage and worker setup look correct, and the logs confirm the file is downloaded and processed without errors. At this point, the most likely cause is that the document content is being dropped during cleaning or segmentation due to the process_rule or segmentation settings. Dify's indexing pipeline will silently skip all segments if, after cleaning and splitting, the content is empty or doesn't meet the segmentation criteria (like chunk size or separator). This can happen if the process_rule is too aggressive, misconfigured, or if the payload structure for hierarchical segmentation is incorrect. For example, if you're using hierarchical mode, your payload must include both segmentation and subchunk_segmentation inside process_rule.rules, with parent_mode outside segmentation—otherwise, the API returns 200 OK but creates no segments, resulting in empty documents even for large files ([details and example payload](https://github.com/langgenius/dify/issues/23933)). Try the following: - Double-check your process_rule structure matches the working example for hierarchical segmentation. - Switch from "automatic" to "custom" segmentation/cleaning in your API payload and reduce cleaning aggressiveness (e.g., disable remove_urls_emails). - Test with a minimal, known-good hierarchical payload to rule out silent payload structure errors. - If possible, add debug logging to capture the cleaned text and split segments during indexing to see if content is being dropped. If you want, share your exact process_rule and segmentation settings, and I can help review them for subtle issues. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- ^{How did I do? [Good](https://app.dosu.dev/response-feedback/9807c69b-403b-48a6-92f2-cbb9a9709d34?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/9807c69b-403b-48a6-92f2-cbb9a9709d34?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/9807c69b-403b-48a6-92f2-cbb9a9709d34?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/9807c69b-403b-48a6-92f2-cbb9a9709d34?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/9807c69b-403b-48a6-92f2-cbb9a9709d34?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/9807c69b-403b-48a6-92f2-cbb9a9709d34?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/9807c69b-403b-48a6-92f2-cbb9a9709d34?feedback_type=other)} [](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[](https://cursor.com/link/prompt?text=Your%20storage%20and%20worker%20setup%20look%20correct%2C%20and%20the%20logs%20confirm%20the%20file%20is%20downloaded%20and%20processed%20without%20errors.%20At%20this%20point%2C%20the%20most%20likely%20cause%20is%20that%20the%20document%20content%20is%20being%20dropped%20during%20cleaning%20or%20segmentation%20due%20to%20the%20process_rule%20or%20segmentation%20settings.%0A%0ADify%27s%20indexing%20pipeline%20will%20silently%20skip%20all%20segments%20if%2C%20after%20cleaning%20and%20splitting%2C%20the%20content%20is%20empty%20or%20doesn%27t%20meet%20the%20segmentation%20criteria%20%28like%20chunk%20size%20or%20separator%29.%20This%20can%20happen%20if%20the%20process_rule%20is%20too%20aggressive%2C%20misconfigured%2C%20or%20if%20the%20payload%20structure%20for%20hierarchical%20segmentation%20is%20incorrect.%20For%20example%2C%20if%20you%27re%20using%20hierarchical%20mode%2C%20your%20payload%20must%20include%20both%20segmentation%20and%20subchunk_segmentation%20inside%20process_rule.rules%2C%20with%20parent_mode%20outside%20segmentation%E2%80%94otherwise%2C%20the%20API%20returns%20200%20OK%20but%20creates%20no%20segments%2C%20resulting%20in%20empty%20documents%20even%20for%20large%20files%20%28%5Bdetails%20and%20example%20payload%5D%28https%3A//github.com/langgenius/dify/issues/23933%29%29.%0A%0ATry%20the%20following%3A%0A-%20Double-check%20your%20process_rule%20structure%20matches%20the%20working%20example%20for%20hierarchical%20segmentation.%0A-%20Switch%20from%20%22automatic%22%20to%20%22custom%22%20segmentation/cleaning%20in%20your%20API%20payload%20and%20reduce%20cleaning%20aggressiveness%20%28e.g.%2C%20disable%20remove_urls_emails%29.%0A-%20Test%20with%20a%20minimal%2C%20known-good%20hierarchical%20payload%20to%20rule%20out%20silent%20payload%20structure%20errors.%0A-%20If%20possible%2C%20add%20debug%20logging%20to%20capture%20the%20cleaned%20text%20and%20split%20segments%20during%20indexing%20to%20see%20if%20content%20is%20being%20dropped.%0A%0AIf%20you%20want%2C%20share%20your%20exact%20process_rule%20and%20segmentation%20settings%2C%20and%20I%20can%20help%20review%20them%20for%20subtle%20issues.)&nbsp;[](https://go.dosu.dev/discord-bot)&nbsp;[](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28763)

yindo commented 2026-02-21 20:08:07 -05:00

Author

Owner

Copy Link

Copy Source

@talenhon commented on GitHub (Dec 1, 2025):

@dosu
Thanks. Key is inside the process_rule...
Suggest can provide a structure documentation or message in api return if the process_rule is missing items.

process_rule:
  mode: hierarchical
  rules:
    pre_processing_rules:
      - id: remove_extra_spaces
        enabled: true
      - id: remove_urls_emails
        enabled: true
    segmentation:
      separator: '###'
      max_tokens: 1024
    parent_mode: full-doc
    subchunk_segmentation:
      separator: \n
      max_tokens: 512

Tried below rules. So far it works for me.

{
    "name": "{{$guid}}",
    "text": "We are all explorers. Our desire to discover, and then share that new-found knowledge, is part of what makes us human – indeed, this has played an important part in our success as a species. ",
    "indexing_technique": "high_quality",
    "process_rule": {
        "mode": "hierarchical",
        "rules": {
            "pre_processing_rules": [
                {
                    "id": "remove_extra_spaces",
                    "enabled": true
                },
                {
                    "id": "remove_urls_emails",
                    "enabled": true
                }
            ],
            "segmentation": {
                "separator": "###",
                "max_tokens": 1024
            },
            "parent_mode": "full-doc",
            "subchunk_segmentation": {
                "separator": "\n",
                "max_tokens": 512
            }
        }
    },
    "doc_form": "hierarchical_model",
    "doc_language": "English"
}

@talenhon commented on GitHub (Dec 1, 2025): @dosu Thanks. Key is inside the process_rule... Suggest can provide a structure documentation or message in api return if the process_rule is missing items. ``` process_rule: mode: hierarchical rules: pre_processing_rules: - id: remove_extra_spaces enabled: true - id: remove_urls_emails enabled: true segmentation: separator: '###' max_tokens: 1024 parent_mode: full-doc subchunk_segmentation: separator: \n max_tokens: 512 ``` Tried below rules. So far it works for me. ``` { "name": "{{$guid}}", "text": "We are all explorers. Our desire to discover, and then share that new-found knowledge, is part of what makes us human – indeed, this has played an important part in our success as a species. ", "indexing_technique": "high_quality", "process_rule": { "mode": "hierarchical", "rules": { "pre_processing_rules": [ { "id": "remove_extra_spaces", "enabled": true }, { "id": "remove_urls_emails", "enabled": true } ], "segmentation": { "separator": "###", "max_tokens": 1024 }, "parent_mode": "full-doc", "subchunk_segmentation": { "separator": "\n", "max_tokens": 512 } } }, "doc_form": "hierarchical_model", "doc_language": "English" } ```

yindo commented 2026-02-21 20:08:07 -05:00

Author

Owner

Copy Link

Copy Source

@dosubot[bot] commented on GitHub (Dec 1, 2025):

Great catch—having the correct process_rule structure is absolutely key for hierarchical segmentation. Your working example matches the recommended format: mode set to "hierarchical", with both segmentation and subchunk_segmentation inside rules, and parent_mode outside segmentation. This structure avoids the silent failures that happen when fields are missing or mis-nested see details and examples.

Your suggestion for better documentation and API error messages is spot on. The good news: recent updates (see PR #28810) have added comprehensive validation and error handling for process_rule in document uploads. Now, if required fields are missing or invalid, the API will return clear error messages instead of silently accepting the payload. This covers all modes (automatic, custom, hierarchical) and checks for things like missing segmentation, missing parent_mode, and invalid types.

For reference, here’s a minimal working example for hierarchical segmentation:

{
  "process_rule": {
    "mode": "hierarchical",
    "rules": {
      "pre_processing_rules": [
        { "id": "remove_extra_spaces", "enabled": true },
        { "id": "remove_urls_emails", "enabled": true }
      ],
      "segmentation": {
        "separator": "###",
        "max_tokens": 1024
      },
      "parent_mode": "full-doc",
      "subchunk_segmentation": {
        "separator": "\n",
        "max_tokens": 512
      }
    }
  },
  "doc_form": "hierarchical_model",
  "indexing_technique": "high_quality"
}

There are still some documentation gaps, like missing details for the chunk_overlap property in segmentation (see docs issue), but the validation improvements should make troubleshooting much easier. If you’re running an older version, updating to the latest Dify release will give you these new error messages and validation checks.

Thanks for sharing your feedback—this helps make the API more robust for everyone!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

@dosubot[bot] commented on GitHub (Dec 1, 2025): <!-- Answer --> Great catch—having the correct process_rule structure is absolutely key for hierarchical segmentation. Your working example matches the recommended format: mode set to "hierarchical", with both segmentation and subchunk_segmentation inside rules, and parent_mode outside segmentation. This structure avoids the silent failures that happen when fields are missing or mis-nested [see details and examples](https://github.com/langgenius/dify/issues/23933). Your suggestion for better documentation and API error messages is spot on. The good news: recent updates (see [PR #28810](https://github.com/langgenius/dify/pull/28810)) have added comprehensive validation and error handling for process_rule in document uploads. Now, if required fields are missing or invalid, the API will return clear error messages instead of silently accepting the payload. This covers all modes (automatic, custom, hierarchical) and checks for things like missing segmentation, missing parent_mode, and invalid types. For reference, here’s a minimal working example for hierarchical segmentation: ```json { "process_rule": { "mode": "hierarchical", "rules": { "pre_processing_rules": [ { "id": "remove_extra_spaces", "enabled": true }, { "id": "remove_urls_emails", "enabled": true } ], "segmentation": { "separator": "###", "max_tokens": 1024 }, "parent_mode": "full-doc", "subchunk_segmentation": { "separator": "\n", "max_tokens": 512 } } }, "doc_form": "hierarchical_model", "indexing_technique": "high_quality" } ``` There are still some documentation gaps, like missing details for the chunk_overlap property in segmentation ([see docs issue](https://github.com/langgenius/dify-docs/issues/514)), but the validation improvements should make troubleshooting much easier. If you’re running an older version, updating to the latest Dify release will give you these new error messages and validation checks. Thanks for sharing your feedback—this helps make the API more robust for everyone! <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- ^{How did I do? [Good](https://app.dosu.dev/response-feedback/e2088aa7-39f3-4e30-b70d-2f238a14a9b5?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/e2088aa7-39f3-4e30-b70d-2f238a14a9b5?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/e2088aa7-39f3-4e30-b70d-2f238a14a9b5?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/e2088aa7-39f3-4e30-b70d-2f238a14a9b5?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/e2088aa7-39f3-4e30-b70d-2f238a14a9b5?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/e2088aa7-39f3-4e30-b70d-2f238a14a9b5?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/e2088aa7-39f3-4e30-b70d-2f238a14a9b5?feedback_type=other)} [](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[](https://cursor.com/link/prompt?text=Great%20catch%E2%80%94having%20the%20correct%20process_rule%20structure%20is%20absolutely%20key%20for%20hierarchical%20segmentation.%20Your%20working%20example%20matches%20the%20recommended%20format%3A%20mode%20set%20to%20%22hierarchical%22%2C%20with%20both%20segmentation%20and%20subchunk_segmentation%20inside%20rules%2C%20and%20parent_mode%20outside%20segmentation.%20This%20structure%20avoids%20the%20silent%20failures%20that%20happen%20when%20fields%20are%20missing%20or%20mis-nested%20%5Bsee%20details%20and%20examples%5D%28https%3A//github.com/langgenius/dify/issues/23933%29.%0A%0AYour%20suggestion%20for%20better%20documentation%20and%20API%20error%20messages%20is%20spot%20on.%20The%20good%20news%3A%20recent%20updates%20%28see%20%5BPR%20%2328810%5D%28https%3A//github.com/langgenius/dify/pull/28810%29%29%20have%20added%20comprehensive%20validation%20and%20error%20handling%20for%20process_rule%20in%20document%20uploads.%20Now%2C%20if%20required%20fields%20are%20missing%20or%20invalid%2C%20the%20API%20will%20return%20clear%20error%20messages%20instead%20of%20silently%20accepting%20the%20payload.%20This%20covers%20all%20modes%20%28automatic%2C%20custom%2C%20hierarchical%29%20and%20checks%20for%20things%20like%20missing%20segmentation%2C%20missing%20parent_mode%2C%20and%20invalid%20types.%0A%0AFor%20reference%2C%20here%E2%80%99s%20a%20minimal%20working%20example%20for%20hierarchical%20segmentation%3A%0A%0A%60%60%60json%0A%7B%0A%20%20%22process_rule%22%3A%20%7B%0A%20%20%20%20%22mode%22%3A%20%22hierarchical%22%2C%0A%20%20%20%20%22rules%22%3A%20%7B%0A%20%20%20%20%20%20%22pre_processing_rules%22%3A%20%5B%0A%20%20%20%20%20%20%20%20%7B%20%22id%22%3A%20%22remove_extra_spaces%22%2C%20%22enabled%22%3A%20true%20%7D%2C%0A%20%20%20%20%20%20%20%20%7B%20%22id%22%3A%20%22remove_urls_emails%22%2C%20%22enabled%22%3A%20true%20%7D%0A%20%20%20%20%20%20%5D%2C%0A%20%20%20%20%20%20%22segmentation%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22separator%22%3A%20%22%23%23%23%22%2C%0A%20%20%20%20%20%20%20%20%22max_tokens%22%3A%201024%0A%20%20%20%20%20%20%7D%2C%0A%20%20%20%20%20%20%22parent_mode%22%3A%20%22full-doc%22%2C%0A%20%20%20%20%20%20%22subchunk_segmentation%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22separator%22%3A%20%22%5Cn%22%2C%0A%20%20%20%20%20%20%20%20%22max_tokens%22%3A%20512%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%2C%0A%20%20%22doc_form%22%3A%20%22hierarchical_model%22%2C%0A%20%20%22indexing_technique%22%3A%20%22high_quality%22%0A%7D%0A%60%60%60%0A%0AThere%20are%20still%20some%20documentation%20gaps%2C%20like%20missing%20details%20for%20the%20chunk_overlap%20property%20in%20segmentation%20%28%5Bsee%20docs%20issue%5D%28https%3A//github.com/langgenius/dify-docs/issues/514%29%29%2C%20but%20the%20validation%20improvements%20should%20make%20troubleshooting%20much%20easier.%20If%20you%E2%80%99re%20running%20an%20older%20version%2C%20updating%20to%20the%20latest%20Dify%20release%20will%20give%20you%20these%20new%20error%20messages%20and%20validation%20checks.%0A%0AThanks%20for%20sharing%20your%20feedback%E2%80%94this%20helps%20make%20the%20API%20more%20robust%20for%20everyone%21)&nbsp;[](https://go.dosu.dev/discord-bot)&nbsp;[](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28763)

yindo referenced this issue 2026-02-21 20:45:34 -05:00

[PR #20599] raise error when process_rule is required but missing #29438

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

codex/strengthen-tool-provider-types

yanli/refactor-sandbox-and-e2b

0721-happy-path-smoke

feat/ws-proxy-support

chore/support-preview-mode-fe

gareth/esq1-170-rbac-for-agent

deploy/saas

linw/snp-549-reauth-after-accepting-invitation

0721-document-detail

feat/hitl-im-dev

deploy/enterprise

fix/empty-dataset-list-tip

feat/ce-telemetry-dev

0720-documents-tasks

deploy/agent

release/e-1.11.4

fix/cus-1422-enterprise-api-timeout

hotfix/cus-1422-enterprise-api-timeout

perf/optimize-markdown-renderer

feat/fix-undefined-answer-preview

hotfix/1.15.0-fix.15

dependabot/uv/dify-agent/pillow-12.3.0

0720-crawl-selection

0720-crawl-preview

0720-source-connections

0720-create-knowledge

0720-new-rag-contracts

laipz8200/human-input-loop-iteration

0720-new-rag-list

codex/migrate-knowledge-fs

hotfix/1.15.0-fix.14

lts/1.13.x

deploy/workflow-archive-catalog-discovery

dependabot/uv/api/pydantic-09001a97e5

laipz8200/completion-workflow-entry

dependabot/uv/api/lts/1.13.x/python-packages-6c2bf7110c

dependabot/uv/api/storage-4219cfbece

dependabot/uv/api/lts/1.13.x/dev-6fc4f8b90c

dependabot/uv/api/llm-46e0cee538

dependabot/uv/api/lts/1.13.x/vdb-72526f7860

dependabot/uv/api/opentelemetry-b8005594e1

dependabot/uv/api/lts/1.13.x/storage-4310e8b8d8

dependabot/uv/api/lts/1.13.x/database-c0fba9b420

dependabot/uv/api/lts/1.13.x/llm-5ce1d68736

dependabot/uv/api/lts/1.13.x/pydantic-09001a97e5

dependabot/uv/api/lts/1.13.x/opentelemetry-b8005594e1

dependabot/uv/api/google-68de7a6be5

dependabot/github_actions/lts/1.13.x/github-actions-dependencies-1a36b91cf1

dependabot/uv/api/lts/1.13.x/google-c93828e58f

dependabot/uv/api/lts/1.13.x/flask-1086fc46af

0717-knowledge-fs-contract

feat/secret-scrub-persistence

feat/llm-ft-timeout

feat/appdeploy-selected-workflow-dsl

0715-new-rag

laipz8200/llm-environment-variables

release/ee-1.15.0-hotfix

laipz8200/chore-remove-feature-preview-root-env

dependabot/go_modules/dify-agent-runtime/golang.org/x/net-0.55.0

hotfix/1.15.0-fix.12

hotfix/1.15.0-fix.12-cherrypick

hotfix/1.15.0-fix.13

build/invite-seats-precheck

hotfix/1.15.0-fix.11

laipz8200/workflow-persistence-mixed-mode

hotfix/1.15.0-fix.10

zhangxin/eng-558-licensed-seats-1.15

fix/wta-1307-effect-suspense-read

feat/cli-skills-agents

0713-vp-lint

deploy/dev

fix/agent-roster-answer-completion-main

feat/pay-by-invoice

codex/run-external-e2e-main-ci

agent/remove-sandbox-bind-mounts

feat/agent-v2-e2e-test

hotfix/1.15.0-fix.9

hotfix/1.15.0-fix.8

build/test-archive-export

linw/restore-agent-message-delta-feat

hotfix/1.15.0-fix.7

hotfix/1.15.0-fix.6

codex/pr-38227-session-types

codex/pr-38227-session-scope-split

fix/ui-fix

hotfix/1.15.0-fix.4

codex/review-pr-38480

hotfix/1.15.0-fix.2

copilot/fix-web-full-stack-e2e-job

fix/dify-2738-agent-node-dsl-export

hotfix/1.15.0-fix.3

fix/scope-segment-batch-delete

laipz8200/scope-segment-route-lookup

chore/i18n-sync-7b7956f3e7ca-e901a65a

codex/upgrade-orpc-v2-beta

hotfix/1.15.0-fix.1

codex/upgrade-openapi-ts-autofix

codex/upgrade-next-16-3-preview

hotfix/1.15.0-fix.0

hotfix/web-detail-adjustment-before-release

release/e-1.15.0

codex/replace-lamejs-with-mediabunny

copilot/add-more-tests

deploy/archive-db-resilience

fix/studio-nav-home-button

hotfix/1.14.2-fix.13

worktree-token-expired-error-code

hotfix-1.14.2-fix.12

codex/agent-v2-state-graph

codex/refactor-deployment-detail-components

hotfix/1.14.2-fix.12

build/fix/workflow-error-hanging

work/dify-2585-agent-drafts-pr

fix/dify-2574-soul-files-kv

fix/dify-2554-debug-conversation-per-user

hotfix/1.14.2-fix.11

feat/openapi-rbac

test/cli-e2e-error-body-contract

build/optimize-clean-messages-eligible-apps

zhaohao1004/optimize-clean-messages-cursor

gh-readonly-queue/main/pr-37592-c71f03f5902031b7c02e61af3aff5c2dee6f7d85

build/optimize-clean-messages-cursor

feat/country

codex/vp-fmt-migration

hotfix/1.14.2-fix.10

hotfix/1.14.2-fix.9

hotfix/1.14.2-fix.8

hotfix/1.14.2-fix.7

hotfix/1.14.2-fix.6

codex/page-title-updates

feat/agent-mention-stable-ids

fix/guard-workspace-with-defualt-roles

build/pr-37227-free-plan-workflow-run-cleanup

feat/cli-e2e-test-suite

feat/ui-start-node

feat/hitl-file-in-body

copilot/dify-team-question-agent-node

fix/wta-438-device-sso-redirect-origin

hotfix/1.14.2-fix.5

feat/dev-snippet-fronted

feat/mcp-token-transfer

chore/keep-info-icon-the-same-postion

codex/local-app-list-tag-filter

feat/snippet-new

hotfix/1.14.2-fix.4

codex/migrate-var-reference-combobox

hotfix/1.14.2-fix.3

feat/marketplace-template

chore/deploy-saas-workflow

hotfix/1.14.2-fix.2

build/cli

codex/snp-406-pointer-position-ref

hotfix/1.14.1-fix.3

build/feat/model-type-migration-script

fix/webapp-disabled-not-jump-to-webapp-login

fix/lb-model-type-dedup

song/eng-379-file-output-check-executor

chore/bump-authlib-1.6.12

fix/tag-trigger-preserve-case

chore/bump-sendgrid-lts-1.13.x

fix/model-type-enum-construction

refactor/add-missing-override-decorator

work/web-mcp-1.14.1

copilot/check-runtime-limits

laipz8200/workflow-response-stream-filter

copilot/query-issues-pr-images

laipz8200/fix-autofix-contract-generation

copilot/fix-autofix-job-failure

codex/deprecate-sys-files-compat

codex/fix-scrolling-issue-with-overflow-visible

copilot/research-retrieval-attachment-consumption

copilot/research-agent-node-implementation

chore/upgrade-xyflow-v12

hotfix/1.14.1-fix.1

build/1.14.1-fix.1-langfuse

hotfix/1.14.1-fix.2

feat/lts-lazy-dialogue-count

fix/can-not-create-knowledge

codex/fix-marketplace-hero-shake

codex/tag-name-url-filter

upgrade-graphon-to-0-3-0

codex/app-delete-enter-confirm

build/sample-tidb

codex/revert-pr-35708

chore/remove-dify-deploy-script

chore/simpler-deploy

lts/1.13.x-fix-35910

feat/storage-path-prefix

hotfix/1.13.3-fix10

hotfix/1.13.3-fix9

hotfix/1.14.0-fix.4

hotfix/1.14.0-fix.3

hotfix/1.14.0-fix.2

revert-35748-4-30-open-api

feat/upgrade-graphon

copilot/research-celery-task-ttl

hotfix/1.14.0-fix.1

copilot/fix-css-issue

feat/s3-public-base-url

hotfix/1.13.3-fix8

build/depot-speed-benchmark-20260426

build/depot-speed-benchmark-20260426-warm

hotfix/1.13.3-fix7

feat/evaluation

fix/workflow-line

chore/dependency-upgrade

deploy/zhaohao

feat/replayable-stream

feat/show-the-hierarchy-in-phonenix-trace

p363-reviewed

build/billing-v3

copilot/list-contributions-to-dify

cursor/migrate-tooltip-to-popover-infotips

refactor/quota-service

feat/new-biliing-quota

hotfix/1.13.3-fix6

cursor/dify-ui-infra

p363-hjlarry

cursor/refactor-select-trigger-api

chore/lts-13x-bump-deps

codex/tighten-button-tone-types

locate-queue-autocreate-deps

hotfix/1.13.3-fix5

feat/new-agent-node

hotfix/1.13.3-fix4

feat/per-db-credential

chore/i18n-sync-208604a3a819-9220dd74

chore/i18n-sync-1b369cede4b4-0e13ce3f

chore/i18n-sync-c93e40212e4c-0e13ce3f

codex/fix-vite-audit-in-sdk-test

copilot/fix-agent-dialogue-input-limit

codex/dify-ui-package-migration

4-2-no-global-loading

fix/ps-not-send

3-18-no-global-loading

feat/support-agent-sandbox

hotfix/1.13.3-fix3

hotfix/1.13.3-fix2

4-1-unocss

build/feat/saas-quota-v4

feat/remove-unused-is-deleted-from-conversation

codex-add-podman-compose-dev-scripts

fix-partnerstack-cookie-sync

hotfix/1.13.3-fix1

3-31-vite-task-cache

test/add-test-case-for-prompt-editor

build/feat/saas-quota

build/feat/saas-quota-v2

deploy/agent-dev

yanli/fix-iter-log

review-myscale-sqli

codex/force-node24-for-actions

ci/docker-build-narrowing

chore/dependabot-alert-to-bot

verify-email-reset-flow

3-26-cache-lint-for-autofix

gh-readonly-queue/main/pr-34120-2cbcf9518e40104286a3034f3c3c688cd4ff9fbf

gh-readonly-queue/main/pr-34123-1c72b8d9db03b93b3fc7f86bc522e53806f5a8f0

gh-readonly-queue/main/pr-33960-38285aa1ac6de98614456f49b4d834fe503f4e58

p383

yanli/phase3-code-scope

test/workflow-app

codex/migrate-old-toast-calls-and-update-translations

hotfix/1.13.2-fix2

focal-quokka

codex/fix-issue-33422-in-repository

hotfix/1.13.2-fix1

copilot/refactor-test-config-leak-fix

build/release-1.13.2

fix-kpa-issue

build/release-1.13.1-rc.3

3-18-dev-with-http2

build/release-1.13.1-rc.2

3-16-revert-changes

3-16-add-csp

build/release-1.13.1-rc.1

feat/source-for-plugin-tasks

3-16-next-lib

revert-33484-dependabot/pip/api/python-packages-2eba66ec58

move-trigger-nodes

copilot/refactor-test-config-leak-detection

fix/enterprise-api-error-handling

release/e-1.12.1

fix/vinext-deps

yanli/docx-image-render

copilot/analyze-dify-code

deploy/clean-task-trace

copilot/sub-pr-33044

3-6-type-check

feat/model-provider-refactor

copilot/refactor-unnecessary-cast-call

build/export-app-messages

1.12.1-otel-ee

hotfix/1.13.0-fix.4

update-checker

test/ui-primitive-wrapper-tests-pr

hotfix/1.13.0-fix.3

refactor/base-comp

deploy/memory-usage

hotfix/1.13.0-fix.2

hotfix/1.13.0-fix.1

build/sandbox-clean-job-260302-1

build/sandbox-clean-job

fix/otel-upgrade-e-1.12.1

build/optimize-workflow-run-cleanup

move-core-prompt-runtime

hotfix/1.12.1-fix.6

copilot/remove-pm2-from-docker

move-token-buffer-memory

32589-main-repo

fix/access-mode-sso-verified

refactor/remove-workspace-mutate-from-context

2-11-fix-lint

feat/plugins-marketplace-ssr-prefetch-loading

build/message-clean-performance-log

refactor/template-transform-node

hotfix/1.12.1-fix.5

2-9-stable-test-2

pre-align-hitl-frontend

fix/preserve-UNC-paths-in-conversation-opner

hotfix/1.12.1-fix.4

hotfix/1.12.1-fix.3

feat/add-update-all-plugin-button

refactor/share

hotfix/1.12.1-fix.2

test/inprove-quality

feat/add_templates

hotfix/1.12.1-fix.1

fix/ee-1511-set-default-provider-user-id-bug

copilot/fix-pnpm-test

hotfix/1.12.1-fix.0

feat/vibe-wf

fix/fix-test

fix/console_ns_import

release/1.12.1

fix/workflow-sync-draft

refactor/ssr-prefetch-zustand-removal

1-24-kubb-generate

refactor/migrate-react-window-to-tanstack-virtual

fix/input-ime-onpressenter

1-30-fix-failed-test

feat/structured-output-with-sandbox

yanli/init-ty

hotfix/1.11.4-fix.4

fix/trigger-plugin-output

fix/parallel-generation-render

refactor/configuration

yanli/add-opaque-body

build/billing-refactor-test-2026012701

refactor/debug-and-preview-store

refactor/tools-workflow

build/debug-billing-refactor

build/billing-service

build-billing-service

feat/add-agentation

refactor/app-components

build/collaboration

copilot/sub-pr-31424

refactor/local-storage

refactor/skill-bundler

hotfix/1.11.4-fix.3

build/archive-workflow-run

feat/pull-a-variable

hotfix/1.11.4-fix.2

copilot/sub-pr-30849

hotfix/1.11.4-fix.1

1-21-test-window-is-not-defined

revert-30727-feat/model-total-credits

copilot/sub-pr-31283

zhsama/multi-turn-popup

zhsama/panel-var-popup

12-19-demo-for-app-store

refactor/migrate-app-detail-to-tanstack-query

build/clear-message

feat/credential-sync-fix-ee

feat/llm-first-token-timeout

zhsama/assemble-var-input

release/e-1.11.2

feature/task-quadrant-view

zhsama/llm-warning-ui

build/clear-workflow-run

zhsama/agent-at-nodes

build/1.11.4

release/e-1.6.0-fix.1

release/e-0.15.8-fix.1

1-14-remove-web-app-access-token

refactor/plugins-detail

fix/reset-dev

feat/llm-support-tools

feat/knowledge-summary-index

fix/preact-security-vulnerability

copilot/fix-clear-query-params-test

refactor/query-params-nuqs

chore/upgrade-pnpm-10.28.0

8c92-update-api-core

detached

chore/i18n-sync-20260107-104636

chore/i18n-sync-20260107-102654

feat/grouping-branching

feat/agent-node-v2

hotfix/1.11.2-fix.3

feat/llm-node-support-tools

refactor/remove-axios-dependency

refactor/web-service-hooks

fix/pwa-plugin-migration

chore/refactor-text-generation

hotfix/1.11.2-fix.2

release/e-1.11.1

release/e-1.9.2

release/e-1.8.2

hotfix/1.11.2-hotfix-1

refactor/chat-messages-service-api

fix/knowledge-retrieval-node-affected-by-past-version

fix/esbuild-dependabot-145

chore/automated-i18n-updates-02e0fadef78f0c888b79e7a52ae6714ac73effdc

chore/automated-i18n-updates-eb73f9a9b9fc9eee6be786d2bf5f41fc77e240c4

hotfix/1.11.1-2

refactor-text-generate-item

feat/end-user-oauth

feat/add-qdrant-to-tidb-migration

codex/run-pnpm-lint-fix-and-commit-changes

feat/document-test

feat/e2e-testing

deploy/trigger-dev

hotfix/1.11.1-1

e-260

refactor/use-log-react-query

feat/fallback-interface-for-unauthorized-datasource-node

hotfix/1.10.1-fix.4

refactor/app/components/app/text-generate/item/index.tsx-and-add-test

release/e-1.6.0

hotfix/1.10.1-fix.3

hotfix/1.10.1-fix.2

feat/tracking-1203

deploy/end-user-oauth

hotfix/1.10.1-fix.1

refactor/marketplace-react-query

feat/detect-cognitive-complexity

fix/codeowners

fix/httpx_timeout_parameters

feat/end-user-auth-dummy-apis

chore/re-point-doc-urls

fix/share-markdown-xss

feat/data-tracking

feat/memory-orchestration-be

build/trigger-billing

chore/remove-docker-legacy-dir

copilot/reverse-engineer-dify-application

feat/amplitude

feat/trigger-saas

mysql-adapt

test/build-diff-edition

feat/no-root-image

feat/trigger

feat/ga-tracking

feat/collaboration

mysql-adaptation

feat/sqlalchemy-type-annotations

build/trigger-1

build/self-tenant-queue

mcp-conditional-capability

fix/mcp-capability

hotfix/1.9.2-fix.3

hotfix/1.9.2-fix.2

feat/iteration-node

hotfix/1.9.2-fix.1

hotfix/1.9.1-fix.1

e1ca7a9bdb0443b9f027b6de16

fix/login-status

feat/gtag-1016

test/monaco-editor-and-remix-icon-upgrade

copilot/fix-ce7a9189-f70c-4ea9-81ba-3f49303104d6

feat/memory-orchestration-be-dev-env

build/trigger

feat/add-knowledge-pipeline-template-command

chore/workflow-performance

copilot/fix-fc039c44-aa24-4844-82b2-e9ac237a13e9

test/log-refactor

chore/ssrf-config

feat/pinecone-vdb

copilot/fix-d8c5989d-5a30-418f-8dc4-099cf1f37218

deploy/rag-dev

hotfix/1.9.0-fix.2

pinecone

fix/document-query

feat/trigger_backup

feat/playwright-integration

feat/trigger_merging

feat/chunk-deduplication

feat/crendential-policy-default-check

feat/credential-policy

release/e-1.8.1

fix/template-list

feat/end_user

feat/trigger-fe

revert-25628-refactor-24421-app-datasets-part1

fix/plugin-tag-fallback

fix/unselect-credential

feat/limit-conv-var-desc-length

feat/rag-pipeline-service-api

fix/empty-plugin-list

feat/plugin-readme

feat/plugin-readme-fe

feat/container-node-extract-mapping

bug1

fix/model-credential

feat/big-var-value-show-fe

build/1.7.2-fix.2

release/e-1.7.2

build/boolean-type

feat/oauth-provider-fe

build/0.7.2-fix.1

feat/model-credentials

feat/model-auth

feat/rag-2-preview

release/e-1.7.1

fix/hotfix-tool-provider-deadlock

feat/knowledge-pipeline-access-control

config-tool-file-size

13604-when-using-redis-cluster-for-caching-add-global_prefix-to-all-keys-cached-by-redis

feat/model-credentils

build/oauth

feat/oauth

feat/human-input

build/plugin-auto-upgrade

test/performance

feat/tool-plugin-oauth

feat/tool-oauth

temp-feat-owner-transfer-enterprise-frontend

release/e-3.0.2

feat/plugin-auto-upgrade-fe

chore/offline-the-sys-files

feat/rag-pipeline

feat/r2

release/e-3.2.0

wtw/rag-pipeline

feat/add-retrival-field-in-dataset

feat/webapp-verified-sso-main

fix/app-not-published-error

feat/webapp-verified-sso-260

v141-hotfix

fix/20326-tool-invoke-error-instance-account

fix/19933-account-object-has-no-attribute-_current_tenant-when-workflow-uses-workflow-as-tool-node

hotfix/0519

fix/e-legacy-clean-up

fix/admin-permission

fix/e-admin-permission

feat/model-memory

feat/webapp-auth-api

1.0.0-fix

revert-17133-fix/i16990-text-to-language-settings

e-0156

hotfix/translation-fix

fix/slider-style-error

e-0154

release/0.15.6-alpha-1

cohere/cleanup-free-tenants-logs

chore/auto-dify

fix/rag-infinity-spaces-loop

fix/dataset-admin

fix/expose-debugging-port

fix/tool-info

fix/aws-s3-r2-compatible

feat/knowledge-metabase

fix/fail-branch-stream-output-error

release/0.15.3-fix1

feat/14009-feature-request-support-for-retrieving-historical-conversations-without-from-enduser-id

feat/upgrade-unstructured-version-10.2

feat/classnames-sort

feat/upgrade-unstructured-version

fix/version-check

fix/handle-agent-none-value

fix/agent-params-pasring

fix/agent-parallel

feat/knowledge-dark-mode

feat/custom-tool-input

release/0.15.2-fix1

0.15.1-admin-apis

feat/support-docx-image-view

feat/agent-stream-output

feat/workflow-node-dark-mode

fix/switch-strategy-clean-param

fix/chore-fix

opik-monitoring

provider-gpustack

feat/mecab-japanese-keywords

feat/retry-single-step-debug

revert-12086-feat/parent-child-retrieval

fix/remove-the-retry-index-field

fix/11839

feat/node-execution-retry

feat/support-extractor-tools-update-wip

feat/update-tidb-batchget-endpoint

fix/iteration-thread-pool-error

feat/support-multi-token-count

feat/parent-child-retrieval-api

fix/app-icon-is-missing

feat/support-extractor-tools-update

test/disable_site

build/add-dependabot

feat/support-extractor-tools

fix/docx-extract-image-ssrf

fix/redis-slow-in-gevent

fix/9772-internal-server-error-when-custom-disclaimer-longer-than-64-characters

alert-autofix-89

build/eslint-react-refresh-plugin

build/switch-to-pnpm

revert-9424-feat/update-dataset-clean-rule

fix/external-knowledge-retrieval-issues

feat/login-type-coontrol

feat/external-knowledge-api

feat/rag-external-knowledge-with-api

feat/external-knowledge

fix/notion-table-extract

fix/tooltip

feat/workflow-add-block-shortcut

feat/update-docs

fix/note-node-zoom-issue

fix/model-runtime-quato-issue

fix/db-lock-timeout

feat/new-tooltip

feat/web-app-sso

chore/optimize-app-workflow-deletion-slow-sql

fix/trace_app_config_app_id_idx

fix/extra-table-tracing-app-config

feat/update-beat-job-time-to-env

fix/index-estimate-error

chore/enterprise-license-status

fix/remove-tsne_position

feat/2p

feat/aws-iam-auth-check

feat/add-flashrank

feat/support-milvus-2.4

feat/add-resource-from-tools

bai

1.16.0

1.16.0-rc1

1.15.0

1.14.2

1.14.1

1.14.0

1.13.3

1.13.2

1.13.1

1.14.0-rc1

1.13.0

1.12.1

1.12.0

1.11.4

1.11.3

1.11.2

1.11.1

1.11.0

1.10.1-fix.1

1.10.1

1.10.0

1.10.0-rc1

1.9.2

1.9.1

1.9.0

2.0.0-beta.2

2.0.0-beta.1

1.8.1

1.8.0

1.7.2

1.7.1

1.7.0

1.6.0

1.5.1

1.5.0

1.4.3

1.4.2

0.15.8

1.4.1

1.4.0

1.3.1

0.15.7

1.3.0

0.15.6

1.2.0

0.15.6-alpha.1

1.1.3

0.15.5

1.1.2

0.15.4

1.1.1

1.1.0

1.0.1

1.0.0

0.15.3

0.15.2

0.15.1

1.0.0-beta.1

0.15.0

0.14.2

v0.8.3-fix1

0.14.1

0.14.0

0.13.2

0.13.1

0.13.0

0.12.1

0.12.0

0.11.2

0.11.1

0.10.2-fix1

0.11.0

0.10.2

0.10.1

0.10.0

0.10.0-beta3

0.9.2

0.9.1-fix1

0.10.0-beta2

0.10.0-beta1

0.9.1

0.9.0

0.8.3

0.8.2

0.8.1

0.8.0

0.8.0-beta1

0.7.3

0.7.2

0.7.1

0.7.0

0.6.16

0.6.15

0.6.14

0.6.13

0.6.12-fix1

0.6.12

0.6.11

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0-fix1

0.6.0

0.6.0-preview-workflow.2

0.5.11-fix1

0.5.11

0.6.0-preview-workflow.1

0.5.10

0.5.9

0.5.8

0.5.7

0.5.6

0.5.5

0.5.4

0.5.3

0.5.2

0.5.1

0.5.0

0.4.9

0.4.8

0.4.7

0.4.6

0.4.5

0.4.4

0.4.3

0.4.2

0.4.1

0.4.0

0.3.34

0.3.33

0.3.32

0.3.31-fix3

0.3.31-fix2

0.3.31-fix1

0.3.31

0.3.30

0.3.29

0.3.28

0.3.27

0.3.26

0.3.25

0.3.24

0.3.23

0.3.22

0.3.21

0.3.20

0.3.19

0.3.18

0.3.17

0.3.16

0.3.15

0.3.14

0.3.13

0.3.12

0.3.11

0.3.10

0.3.9

0.3.8

0.3.7

0.3.6

0.3.5

0.3.4

0.3.3

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

Labels

Clear labels

Knowledge Pipeline

Proposal

WIP

cloud

dependencies

feat:webapp

good first issue

outdated

performance

pull-request

Mirrored from GitHub Pull Request

python:uv

refactor

size:L

stale

status: accepting prs

status: blocked

status: in discussion

stt & tts

vulnerablity

web

wont-fix

🌊 feat:workflow

🌚 invalid

🐞 bug

👻 feat:rag

💪 enhancement

📚 documentation

🔨 feat:tools

🙋‍♂️ question

🙏 help wanted

🤔 cant-reproduce

🤖 feat:agent

No Label 🐞 bug

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

yindo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langgenius/dify#20598