langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-28 05:20:07 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[Feature Request] Improve Parent-Child Chunking to Preserve Delimiter Lines (crucial for legal documents) #11888

New Issue
Closed
opened 2026-02-21 19:04:35 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 19:04:35 -05:00
Owner
Copy Link
Copy Source

Originally created by @jagga99 on GitHub (Mar 20, 2025).

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

  1. Description

2.1. The Problem
When using Parent-Child Mode in Dify to split legal documents by article headers (e.g., "Article 1.", "Article 2."), the system treats these headers as simple delimiters.
As a result, one or more of the following issues occur:
        - An introductory or “zero” chunk may be created before the first article header.
        - The article header (e.g., "Article 1.") is either attached to the end of the previous chunk or removed entirely.
        - Individual parent chunks do not begin with the article header, making navigation and reference difficult.

2.2. What Is Needed
A change in the chunking logic to preserve the delimiter line at the start of the new parent chunk.
Specifically, implement one of the following approaches:
        - Include Delimiter in Next Chunk: Add an option so that the entire delimiter line (e.g., "Article 1.") is retained as the first line of the new chunk.
        - Lookahead Splitting: Allow a lookahead-based split (e.g., using a pattern like (?=Article\s+\d+.)) so that the article header is not consumed and appears in the new chunk.
Provide a user configuration option to choose whether to retain or discard the delimiter in the output chunks.

2.3. Why It Matters
Legal Document Structure: Each article header (e.g., "Article 1.") is a key structural marker in legal texts.
Readability and Searchability: Users expect each chunk to start with the article header, which aids in reading, referencing, and precise searching.
Consistency: Ensuring that every parent chunk begins with the article header provides a predictable structure, crucial for effective retrieval.

2.4. Proposed Changes
Modify the parent chunking logic so that the delimiter (e.g., "Article X.") is preserved at the beginning of the new chunk.
Implement a configuration option (e.g., "Include delimiter in next chunk") for users to control this behavior.
Alternatively, support a lookahead-based splitting method that retains the delimiter as part of the resulting chunk.

2.5. Example of Desired Behavior
Current Behavior:
        - Chunk #1: Contains introductory text with the delimiter line (e.g., "Article 1.") at the end.
        - Chunk #2: Contains the text of Article 1 without the header at the beginning.
Desired Behavior:
        - Chunk #1: (Optional introductory text, if any)
        - Chunk #2: Begins exactly with "Article 1." and includes all the text for Article 1 up to "Article 2."
        - Chunk #3: Begins with "Article 2." and so forth.

  1. Additional Context
    This enhancement is particularly important for legal documents (e.g., tax codes, civil codes) where each article is defined by a header (e.g., "Article X."). Preserving these headers in the parent chunks greatly improves document navigation and the overall user experience.

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @jagga99 on GitHub (Mar 20, 2025). ### Self Checks - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. 2. Description 2.1. The Problem When using Parent-Child Mode in Dify to split legal documents by article headers (e.g., "Article 1.", "Article 2."), the system treats these headers as simple delimiters. As a result, one or more of the following issues occur:         - An introductory or “zero” chunk may be created before the first article header.         - The article header (e.g., "Article 1.") is either attached to the end of the previous chunk or removed entirely.         - Individual parent chunks do not begin with the article header, making navigation and reference difficult. 2.2. What Is Needed A change in the chunking logic to preserve the delimiter line at the start of the new parent chunk. Specifically, implement one of the following approaches:         - Include Delimiter in Next Chunk: Add an option so that the entire delimiter line (e.g., "Article 1.") is retained as the first line of the new chunk.         - Lookahead Splitting: Allow a lookahead-based split (e.g., using a pattern like (?=Article\s+\d+\.)) so that the article header is not consumed and appears in the new chunk. Provide a user configuration option to choose whether to retain or discard the delimiter in the output chunks. 2.3. Why It Matters Legal Document Structure: Each article header (e.g., "Article 1.") is a key structural marker in legal texts. Readability and Searchability: Users expect each chunk to start with the article header, which aids in reading, referencing, and precise searching. Consistency: Ensuring that every parent chunk begins with the article header provides a predictable structure, crucial for effective retrieval. 2.4. Proposed Changes Modify the parent chunking logic so that the delimiter (e.g., "Article X.") is preserved at the beginning of the new chunk. Implement a configuration option (e.g., "Include delimiter in next chunk") for users to control this behavior. Alternatively, support a lookahead-based splitting method that retains the delimiter as part of the resulting chunk. 2.5. Example of Desired Behavior Current Behavior:         - Chunk #1: Contains introductory text with the delimiter line (e.g., "Article 1.") at the end.         - Chunk #2: Contains the text of Article 1 without the header at the beginning. Desired Behavior:         - Chunk #1: (Optional introductory text, if any)         - Chunk #2: Begins exactly with "Article 1." and includes all the text for Article 1 up to "Article 2."         - Chunk #3: Begins with "Article 2." and so forth. 3. Additional Context This enhancement is particularly important for legal documents (e.g., tax codes, civil codes) where each article is defined by a header (e.g., "Article X."). Preserving these headers in the parent chunks greatly improves document navigation and the overall user experience. ### 2. Additional context or comments _No response_ ### 3. Can you help us with this feature? - [ ] I am interested in contributing to this feature.
yindo added the 💪 enhancement label 2026-02-21 19:04:35 -05:00
yindo closed this issue 2026-02-21 19:04:35 -05:00
yindo commented 2026-02-21 19:04:35 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 20, 2025):

Hi, @jagga99. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary:

  • Enhancement proposed for Parent-Child Mode to better handle legal documents.
  • Current chunking logic may misplace or remove article headers.
  • Suggestion to modify chunking logic to retain headers at the start of new chunks.
  • No comments or activity on the issue since it was opened.

Next Steps:

  • Please confirm if this issue is still relevant to the latest version of the Dify repository by commenting here.
  • If no updates are provided, the issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Apr 20, 2025): Hi, @jagga99. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog. I'm marking this issue as stale. **Issue Summary:** - Enhancement proposed for Parent-Child Mode to better handle legal documents. - Current chunking logic may misplace or remove article headers. - Suggestion to modify chunking logic to retain headers at the start of new chunks. - No comments or activity on the issue since it was opened. **Next Steps:** - Please confirm if this issue is still relevant to the latest version of the Dify repository by commenting here. - If no updates are provided, the issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#11888
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?