langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

API uploaded documents don't show right chunking details like separators and chunk sizes at the interface #11954

New Issue
Closed
opened 2026-02-21 19:04:58 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 19:04:58 -05:00
Owner
Copy Link
Copy Source

Originally created by @oliverkaiser on GitHub (Mar 21, 2025).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.1.1

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

  1. upload document to knowledgebase via api incl. process_rules and special separators
  2. inspect document via dify
  3. it does show the right paragraph-child mode, but doesn't fill in the right settings in the form
  4. console inspection shows that the right values are stored on the document
Image

✔️ Expected Behavior

fill the right separator and chunk size in the form by default and don't show the default values.

❌ Actual Behavior

It does show the default configuration values.

Originally created by @oliverkaiser on GitHub (Mar 21, 2025). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.1.1 ### Cloud or Self Hosted Self Hosted (Source) ### Steps to reproduce 1. upload document to knowledgebase via api incl. process_rules and special separators 2. inspect document via dify 3. it does show the right paragraph-child mode, but doesn't fill in the right settings in the form 4. console inspection shows that the right values are stored on the document <img width="962" alt="Image" src="https://github.com/user-attachments/assets/f1d74e05-8576-40c0-8b5f-692551ec6358" /> ### ✔️ Expected Behavior fill the right separator and chunk size in the form by default and don't show the default values. ### ❌ Actual Behavior It does show the default configuration values.
yindo added the 🐞 bug👻 feat:rag labels 2026-02-21 19:04:58 -05:00
yindo closed this issue 2026-02-21 19:04:58 -05:00
yindo commented 2026-02-21 19:04:58 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 21, 2025):

Hello @oliverkaiser! I'm here to help you with any bugs, questions, or contributions you have for the project. Let me know how I can assist you!

To ensure that the API uploaded documents display the correct chunking details, such as separators and chunk sizes, in the interface instead of default values, you need to specify the process_rule object in the request body when updating a document. This object allows you to define custom processing rules, including segmentation and chunking details.

Here's an example of how you can structure the process_rule object in your request:

{
  "process_rule": {
    "mode": "custom",
    "rules": {
      "segmentation": {
        "separator": "<your_custom_separator>",
        "max_tokens": <your_custom_max_tokens>
      },
      "subchunk_segmentation": {
        "separator": "<your_custom_subchunk_separator>",
        "max_tokens": <your_custom_subchunk_max_tokens>,
        "chunk_overlap": <your_custom_chunk_overlap>
      }
    }
  }
}

Replace <your_custom_separator>, <your_custom_max_tokens>, <your_custom_subchunk_separator>, <your_custom_subchunk_max_tokens>, and <your_custom_chunk_overlap> with your desired values to override the default settings. This will ensure that the documents reflect the specified chunking details in the interface [1][2][3].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 21, 2025): <!-- Greeting --> Hello @oliverkaiser! I'm here to help you with any bugs, questions, or contributions you have for the project. Let me know how I can assist you! <!-- Answer --> To ensure that the API uploaded documents display the correct chunking details, such as separators and chunk sizes, in the interface instead of default values, you need to specify the `process_rule` object in the request body when updating a document. This object allows you to define custom processing rules, including segmentation and chunking details. Here's an example of how you can structure the `process_rule` object in your request: ```json { "process_rule": { "mode": "custom", "rules": { "segmentation": { "separator": "<your_custom_separator>", "max_tokens": <your_custom_max_tokens> }, "subchunk_segmentation": { "separator": "<your_custom_subchunk_separator>", "max_tokens": <your_custom_subchunk_max_tokens>, "chunk_overlap": <your_custom_chunk_overlap> } } } } ``` Replace `<your_custom_separator>`, `<your_custom_max_tokens>`, `<your_custom_subchunk_separator>`, `<your_custom_subchunk_max_tokens>`, and `<your_custom_chunk_overlap>` with your desired values to override the default settings. This will ensure that the documents reflect the specified chunking details in the interface <sup>[[1]](https://github.com/langgenius/dify/blob/main/web/app/(commonLayout)/datasets/template/template.en.mdx)</sup><sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents/2.-choose-a-chunk-mode.md)</sup><sup>[[3]](https://github.com/langgenius/dify/blob/main/web/app/components/datasets/create/step-two/index.tsx)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/0802e8ef-dba0-4f64-a04c-6d074a0381af?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/0802e8ef-dba0-4f64-a04c-6d074a0381af?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/0802e8ef-dba0-4f64-a04c-6d074a0381af?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/0802e8ef-dba0-4f64-a04c-6d074a0381af?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/0802e8ef-dba0-4f64-a04c-6d074a0381af?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/0802e8ef-dba0-4f64-a04c-6d074a0381af?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/0802e8ef-dba0-4f64-a04c-6d074a0381af?feedback_type=other)</sup>
yindo commented 2026-02-21 19:04:58 -05:00
Author
Owner
Copy Link
Copy Source

@oliverkaiser commented on GitHub (Apr 9, 2025):

Actually doesn't only affect api uploaded documents. Also when having parent-child stored with fulldoc retrieval configured wit custom chunk sizes and separators via the interface, when editing the form is pre-filled with the correct values provided from api and only include its defaults.

@oliverkaiser commented on GitHub (Apr 9, 2025): Actually doesn't only affect api uploaded documents. Also when having parent-child stored with fulldoc retrieval configured wit custom chunk sizes and separators via the interface, when editing the form is pre-filled with the correct values provided from api and only include its defaults.
yindo commented 2026-02-21 19:04:58 -05:00
Author
Owner
Copy Link
Copy Source

@HojaMuerta commented on GitHub (Apr 9, 2025):

I got the same issue.

@HojaMuerta commented on GitHub (Apr 9, 2025): I got the same issue.
yindo commented 2026-02-21 19:04:58 -05:00
Author
Owner
Copy Link
Copy Source

@WaterYue commented on GitHub (Aug 11, 2025):

for segmentation, why no segment overlap setting for the api, only have chunk overlap?
this is chunk setting or segment setting ?
Image
I'm going to call create-by-text, and the segment separator is "------------", I can call api like this:
curl --location --request POST 'https://api.dify.ai/v1/datasets/{dataset_id}/document/create-by-text' \ --header 'Authorization: Bearer {api_key}' \ --header 'Content-Type: application/json' \ --data-raw '{"name": "text","text": "text","indexing_technique": "high_quality","process_rule": {"rules": {"pre_processing_rules": [{"id": "remove_extra_spaces","enabled": true},{"id": "remove_urls_emails","enabled": true}],"segmentation": {"separator": "----****----****----","max_tokens": 4000}},"mode": "custom"}}'

but how to set segment overlap for the api ? after call complete, the "Chunk overlap" is 0. I do not want to use subchunk_segmentation.

@WaterYue commented on GitHub (Aug 11, 2025): for segmentation, why no segment overlap setting for the api, only have chunk overlap? this is chunk setting or segment setting ? <img width="498" height="298" alt="Image" src="https://github.com/user-attachments/assets/a94fcb7d-358c-4931-97fe-fc00629fee2a" /> I'm going to call create-by-text, and the segment separator is "----****----****----", I can call api like this: `curl --location --request POST 'https://api.dify.ai/v1/datasets/{dataset_id}/document/create-by-text' \ --header 'Authorization: Bearer {api_key}' \ --header 'Content-Type: application/json' \ --data-raw '{"name": "text","text": "text","indexing_technique": "high_quality","process_rule": {"rules": {"pre_processing_rules": [{"id": "remove_extra_spaces","enabled": true},{"id": "remove_urls_emails","enabled": true}],"segmentation": {"separator": "----****----****----","max_tokens": 4000}},"mode": "custom"}}' ` but how to set segment overlap for the api ? after call complete, the "Chunk overlap" is 0. I do not want to use subchunk_segmentation.
yindo commented 2026-02-21 19:04:58 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 12, 2025):

Hi, @oliverkaiser. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported that documents uploaded via API with custom process rules show correct paragraph-child mode, but the form interface displays default separator and chunk size values instead of the stored settings.
  • You later clarified that this issue also occurs when editing documents created via the interface, where the form pre-fills only default values rather than the API-provided ones.
  • Another user, HojaMuerta, confirmed experiencing the same problem.
  • WaterYue raised a related question about the absence of a segment overlap setting in the API for segmentation, noting only chunk overlap is available and seeking guidance on how to set segment overlap without using subchunk_segmentation.

Next Steps:

  • Please let me know if this issue is still relevant with the latest version of Dify by commenting here to keep the discussion open.
  • Otherwise, I will automatically close this issue in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Sep 12, 2025): Hi, @oliverkaiser. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported that documents uploaded via API with custom process rules show correct paragraph-child mode, but the form interface displays default separator and chunk size values instead of the stored settings. - You later clarified that this issue also occurs when editing documents created via the interface, where the form pre-fills only default values rather than the API-provided ones. - Another user, HojaMuerta, confirmed experiencing the same problem. - WaterYue raised a related question about the absence of a segment overlap setting in the API for segmentation, noting only chunk overlap is available and seeking guidance on how to set segment overlap without using subchunk_segmentation. **Next Steps:** - Please let me know if this issue is still relevant with the latest version of Dify by commenting here to keep the discussion open. - Otherwise, I will automatically close this issue in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug 👻 feat:rag
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#11954
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?