langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 20:45:26 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

segmentation and subchunk_segmentation not properly set when using create-by-text API #13164

New Issue
Closed
opened 2026-02-21 19:10:51 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 19:10:51 -05:00
Owner
Copy Link
Copy Source

Originally created by @hbprotoss on GitHub (Apr 21, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.2.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

call API /datasets/{dataset_id}/document/create-by-text with arguments

{
  "doc_form": "hierarchical_model",
  "doc_language": "Chinese",
  "indexing_technique": "high_quality",
  "name": "2.txt",
  "process_rule": {
    "mode": "custom",
    "rules": {
      "parent_mode": "full-doc",
      "pre_processing_rules": [
        {
          "enabled": true,
          "id": "remove_extra_spaces"
        },
        {
          "enabled": true,
          "id": "remove_urls_emails"
        }
      ],
      "segmentation": {
        "max_tokens": 4000,
        "separator": "\n\n"
      },
      "subchunk_segmentation": {
        "max_tokens": 2000,
        "separator": "\n"
      }
    }
  },
  "retrieval_model": {
    "reranking_enable": false,
    "reranking_mode": "",
    "score_threshold": -1,
    "score_threshold_enabled": false,
    "search_method": "hybrid_search",
    "top_k": 20
  },
  "text": "联即路。相构般得。林历低交导心立结该。\n非化己。因他参林。算便了回院许方入直。\n学深飞可始更价十任。养义当动。构着况规史铁素市保等。\n步即列支原。时经民主年五却北争据。精改花交交公认上马。\n部金打头南期。干转花状究动众满工第。例你作属周之清她质。\n百收统称完。快支层况。因油利声王重向明深。\n改我空。连声直支清。真使却时团省接而光。"
}

max_tokens in argument segmentation and subchunk_segmentation not working. The created document still using the default value

Image

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @hbprotoss on GitHub (Apr 21, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.2.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce call API `/datasets/{dataset_id}/document/create-by-text` with arguments ``` { "doc_form": "hierarchical_model", "doc_language": "Chinese", "indexing_technique": "high_quality", "name": "2.txt", "process_rule": { "mode": "custom", "rules": { "parent_mode": "full-doc", "pre_processing_rules": [ { "enabled": true, "id": "remove_extra_spaces" }, { "enabled": true, "id": "remove_urls_emails" } ], "segmentation": { "max_tokens": 4000, "separator": "\n\n" }, "subchunk_segmentation": { "max_tokens": 2000, "separator": "\n" } } }, "retrieval_model": { "reranking_enable": false, "reranking_mode": "", "score_threshold": -1, "score_threshold_enabled": false, "search_method": "hybrid_search", "top_k": 20 }, "text": "联即路。相构般得。林历低交导心立结该。\n非化己。因他参林。算便了回院许方入直。\n学深飞可始更价十任。养义当动。构着况规史铁素市保等。\n步即列支原。时经民主年五却北争据。精改花交交公认上马。\n部金打头南期。干转花状究动众满工第。例你作属周之清她质。\n百收统称完。快支层况。因油利声王重向明深。\n改我空。连声直支清。真使却时团省接而光。" } ``` max_tokens in argument `segmentation` and `subchunk_segmentation` not working. The created document still using the default value ![Image](https://github.com/user-attachments/assets/4b93a6cb-74bf-4e24-9b80-75ab98d7c61c) ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the good first issue label 2026-02-21 19:10:51 -05:00
yindo closed this issue 2026-02-21 19:10:51 -05:00
yindo commented 2026-02-21 19:10:52 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 21, 2025):

To resolve the issue with max_tokens settings in segmentation and subchunk_segmentation not working when using the create-by-text API, ensure that you are using the correct JSON payload structure. The parent_mode and subchunk_segmentation parameters should be placed under the rules object, and process_rule.mode should be set to 'hierarchical' instead of 'custom'. This will prevent the custom segmentation settings from reverting to default values. Here is the correct structure:

{
    "name": "2.txt",
    "text": "Your text here",
    "indexing_technique": "high_quality",
    "doc_form": "hierarchical_model",
    "process_rule": {
        "mode": "hierarchical",
        "rules": {
            "pre_processing_rules": [
                {"id": "remove_extra_spaces", "enabled": true},
                {"id": "remove_urls_emails", "enabled": true}
            ],
            "segmentation": {
                "separator": "\n\n",
                "max_tokens": 4000
            },
            "parent_mode": "full-doc",
            "subchunk_segmentation": {
                "separator": "\n",
                "max_tokens": 2000
            }
        }
    },
    "retrieval_model": {
        "reranking_enable": false,
        "reranking_mode": "",
        "score_threshold": -1,
        "score_threshold_enabled": false,
        "search_method": "hybrid_search",
        "top_k": 20
    }
}

Make sure you are using Dify version v0.15.2 or later, as this version addresses the segmentation recognition issue when using create_by_text with parent-child mode [1][2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 21, 2025): <!-- Answer --> To resolve the issue with `max_tokens` settings in `segmentation` and `subchunk_segmentation` not working when using the `create-by-text` API, ensure that you are using the correct JSON payload structure. The `parent_mode` and `subchunk_segmentation` parameters should be placed under the `rules` object, and `process_rule.mode` should be set to 'hierarchical' instead of 'custom'. This will prevent the custom segmentation settings from reverting to default values. Here is the correct structure: ```json { "name": "2.txt", "text": "Your text here", "indexing_technique": "high_quality", "doc_form": "hierarchical_model", "process_rule": { "mode": "hierarchical", "rules": { "pre_processing_rules": [ {"id": "remove_extra_spaces", "enabled": true}, {"id": "remove_urls_emails", "enabled": true} ], "segmentation": { "separator": "\n\n", "max_tokens": 4000 }, "parent_mode": "full-doc", "subchunk_segmentation": { "separator": "\n", "max_tokens": 2000 } } }, "retrieval_model": { "reranking_enable": false, "reranking_mode": "", "score_threshold": -1, "score_threshold_enabled": false, "search_method": "hybrid_search", "top_k": 20 } } ``` Make sure you are using Dify version v0.15.2 or later, as this version addresses the segmentation recognition issue when using `create_by_text` with parent-child mode <sup>[[1]](https://github.com/langgenius/dify/issues/13007)</sup><sup>[[2]](https://github.com/langgenius/dify/issues/13013)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=other)</sup>
yindo commented 2026-02-21 19:10:52 -05:00
Author
Owner
Copy Link
Copy Source

@hbprotoss commented on GitHub (Apr 21, 2025):

To resolve the issue with max_tokens settings in segmentation and subchunk_segmentation not working when using the create-by-text API, ensure that you are using the correct JSON payload structure. The parent_mode and subchunk_segmentation parameters should be placed under the rules object, and process_rule.mode should be set to 'hierarchical' instead of 'custom'. This will prevent the custom segmentation settings from reverting to default values. Here is the correct structure:

{
"name": "2.txt",
"text": "Your text here",
"indexing_technique": "high_quality",
"doc_form": "hierarchical_model",
"process_rule": {
"mode": "hierarchical",
"rules": {
"pre_processing_rules": [
{"id": "remove_extra_spaces", "enabled": true},
{"id": "remove_urls_emails", "enabled": true}
],
"segmentation": {
"separator": "\n\n",
"max_tokens": 4000
},
"parent_mode": "full-doc",
"subchunk_segmentation": {
"separator": "\n",
"max_tokens": 2000
}
}
},
"retrieval_model": {
"reranking_enable": false,
"reranking_mode": "",
"score_threshold": -1,
"score_threshold_enabled": false,
"search_method": "hybrid_search",
"top_k": 20
}
}
Make sure you are using Dify version v0.15.2 or later, as this version addresses the segmentation recognition issue when using create_by_text with parent-child mode [1][2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

not working

@hbprotoss commented on GitHub (Apr 21, 2025): > To resolve the issue with `max_tokens` settings in `segmentation` and `subchunk_segmentation` not working when using the `create-by-text` API, ensure that you are using the correct JSON payload structure. The `parent_mode` and `subchunk_segmentation` parameters should be placed under the `rules` object, and `process_rule.mode` should be set to 'hierarchical' instead of 'custom'. This will prevent the custom segmentation settings from reverting to default values. Here is the correct structure: > > { > "name": "2.txt", > "text": "Your text here", > "indexing_technique": "high_quality", > "doc_form": "hierarchical_model", > "process_rule": { > "mode": "hierarchical", > "rules": { > "pre_processing_rules": [ > {"id": "remove_extra_spaces", "enabled": true}, > {"id": "remove_urls_emails", "enabled": true} > ], > "segmentation": { > "separator": "\n\n", > "max_tokens": 4000 > }, > "parent_mode": "full-doc", > "subchunk_segmentation": { > "separator": "\n", > "max_tokens": 2000 > } > } > }, > "retrieval_model": { > "reranking_enable": false, > "reranking_mode": "", > "score_threshold": -1, > "score_threshold_enabled": false, > "search_method": "hybrid_search", > "top_k": 20 > } > } > Make sure you are using Dify version v0.15.2 or later, as this version addresses the segmentation recognition issue when using `create_by_text` with parent-child mode [[1]](https://github.com/langgenius/dify/issues/13007)[[2]](https://github.com/langgenius/dify/issues/13013). > > _To continue talking to [Dosu](https://dosu.dev), mention [@dosu](https://github.com/dosu)._ > > Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/f1267d21-5ee4-45bc-a069-027a3a300357?feedback_type=other) not working
yindo commented 2026-02-21 19:10:52 -05:00
Author
Owner
Copy Link
Copy Source

@hbprotoss commented on GitHub (Apr 21, 2025):

it seems an UI issue?

Image

@hbprotoss commented on GitHub (Apr 21, 2025): it seems an UI issue? ![Image](https://github.com/user-attachments/assets/5fc2b97b-0d71-407e-963f-69108062848a)
yindo commented 2026-02-21 19:10:52 -05:00
Author
Owner
Copy Link
Copy Source

@zouao0821 commented on GitHub (Apr 23, 2025):

curl --location --request POST 'https://****/base-api/v1/datasets/****96-a1ed-79546e66b357/document/create-by-file'
--header 'Authorization: Bearer ****'
--header 'Accept: /'
--header 'Host: '
--header 'Connection: keep-alive'
--header 'Content-Type: multipart/form-data; boundary=--------------------------170905425617371057115838'
--form 'data="{"indexing_technique":"high_quality","process_rule":{"rules":{"pre_processing_rules":[{"id":"remove_extra_spaces","enabled":true},{"id":"remove_urls_emails","enabled":false}],"segmentation":{"separator":"\n\n","max_tokens":4000},"parent_mode":"paragraph","subchunk_segmentation":{"separator":";","max_tokens":4000}},"mode":"hierarchical"}}"'
--form 'file=@"//chunk_3.xlsx"'

Why is it that after I upload with this, there is only the father-parent segment and no child segment

@zouao0821 commented on GitHub (Apr 23, 2025): curl --location --request POST 'https://****/base-api/v1/datasets/****96-a1ed-79546e66b357/document/create-by-file' \ --header 'Authorization: Bearer ****' \ --header 'Accept: */*' \ --header 'Host: ****' \ --header 'Connection: keep-alive' \ --header 'Content-Type: multipart/form-data; boundary=--------------------------170905425617371057115838' \ --form 'data="{\"indexing_technique\":\"high_quality\",\"process_rule\":{\"rules\":{\"pre_processing_rules\":[{\"id\":\"remove_extra_spaces\",\"enabled\":true},{\"id\":\"remove_urls_emails\",\"enabled\":false}],\"segmentation\":{\"separator\":\"\\n\\n\",\"max_tokens\":4000},\"parent_mode\":\"paragraph\",\"subchunk_segmentation\":{\"separator\":\";\",\"max_tokens\":4000}},\"mode\":\"hierarchical\"}}"' \ --form 'file=@"/****/chunk_3.xlsx"' Why is it that after I upload with this, there is only the father-parent segment and no child segment
yindo commented 2026-02-21 19:10:52 -05:00
Author
Owner
Copy Link
Copy Source

@hbprotoss commented on GitHub (Apr 24, 2025):

@zouao0821 not related, please create a new issue

@hbprotoss commented on GitHub (Apr 24, 2025): @zouao0821 not related, please create a new issue
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label good first issue
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#13164
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?