langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-20 09:10:05 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Parent-child chunking removes all whitespaces in child chunks #14476

New Issue
Closed
opened 2026-02-21 19:17:24 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 19:17:24 -05:00
Owner
Copy Link
Copy Source

Originally created by @tjroamer on GitHub (Jun 6, 2025).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.4.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

When using parent-child chunking method with the attached document, all whitespaces are removed so that the chunks are not usable.

  1. Create Knowledge
  2. Upload the attached document, click "Next"
  3. In "Chunk Settings", select "Parent-child"
  4. Click "Preview Chunk"

test-chunking-bug.md

✔️ Expected Behavior

The whitespaces should be kept.

❌ Actual Behavior

I got the following, note that all whitespace have been removed in the two child chunks of the first chunk:

Image

Originally created by @tjroamer on GitHub (Jun 6, 2025). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.4.1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce When using parent-child chunking method with the attached document, all whitespaces are removed so that the chunks are not usable. 1. Create Knowledge 2. Upload the attached document, click "Next" 3. In "Chunk Settings", select "Parent-child" 4. Click "Preview Chunk" [test-chunking-bug.md](https://github.com/user-attachments/files/20631698/test-chunking-bug.md) ### ✔️ Expected Behavior The whitespaces should be kept. ### ❌ Actual Behavior I got the following, note that all whitespace have been removed in the two child chunks of the first chunk: ![Image](https://github.com/user-attachments/assets/ded60b9d-b717-47bd-ab21-baa0d9a31c9f)
yindo added the 🐞 bugstale labels 2026-02-21 19:17:24 -05:00
yindo closed this issue 2026-02-21 19:17:24 -05:00
yindo commented 2026-02-21 19:17:24 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jun 6, 2025):

This whitespace removal is caused by the current implementation of the text splitting logic in Dify. Specifically, the TextSplitter class (and all its derivatives, including those used for parent-child chunking) strips leading and trailing whitespace from each chunk by default when joining sub-chunks, due to a hardcoded text.strip() call in the _join_docs method. There is currently no configuration or parameter to disable this behavior, and all subclasses inherit this logic unchanged. This means whitespace will always be removed from the start and end of child chunks, regardless of chunking settings or the keep_separator parameter.

A straightforward fix would be to add a strip_whitespace parameter to the TextSplitter class constructor, defaulting to True for backward compatibility, and then use this parameter to control whether text.strip() is called in _join_docs. This would allow you to preserve whitespace in child chunks when needed. All subclasses already support extra keyword arguments, so this change would be clean and backward compatible across the codebase.

There are no existing workarounds or configuration options in the current release or documentation to retain whitespace in chunks—this would require a code change in the splitter logic itself.
Reference

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Join Discord Share on X

@dosubot[bot] commented on GitHub (Jun 6, 2025): <!-- Answer --> This whitespace removal is caused by the current implementation of the text splitting logic in Dify. Specifically, the TextSplitter class (and all its derivatives, including those used for parent-child chunking) strips leading and trailing whitespace from each chunk by default when joining sub-chunks, due to a hardcoded text.strip() call in the _join_docs method. There is currently no configuration or parameter to disable this behavior, and all subclasses inherit this logic unchanged. This means whitespace will always be removed from the start and end of child chunks, regardless of chunking settings or the keep_separator parameter. A straightforward fix would be to add a strip_whitespace parameter to the TextSplitter class constructor, defaulting to True for backward compatibility, and then use this parameter to control whether text.strip() is called in _join_docs. This would allow you to preserve whitespace in child chunks when needed. All subclasses already support extra keyword arguments, so this change would be clean and backward compatible across the codebase. There are no existing workarounds or configuration options in the current release or documentation to retain whitespace in chunks—this would require a code change in the splitter logic itself. [Reference](https://github.com/langgenius/dify/blob/930c4cb60904f29b21f0cf5abf8da8063ed4ec7a/api/core/rag/splitter/text_splitter.py) <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/7db52699-a30b-4713-8a4f-22eebccc57c9?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/7db52699-a30b-4713-8a4f-22eebccc57c9?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/7db52699-a30b-4713-8a4f-22eebccc57c9?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/7db52699-a30b-4713-8a4f-22eebccc57c9?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/7db52699-a30b-4713-8a4f-22eebccc57c9?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/7db52699-a30b-4713-8a4f-22eebccc57c9?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/7db52699-a30b-4713-8a4f-22eebccc57c9?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/20768)
yindo commented 2026-02-21 19:17:25 -05:00
Author
Owner
Copy Link
Copy Source

@TedaLIEz commented on GitHub (Jun 9, 2025):

Have you tried to toggle off this option?

Image

@TedaLIEz commented on GitHub (Jun 9, 2025): Have you tried to toggle off this option? ![Image](https://github.com/user-attachments/assets/18d5b2a8-d2a4-4641-aef9-7c44f006e63c)
yindo commented 2026-02-21 19:17:25 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 31, 2025):

Hi, @tjroamer. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported that in Dify v1.4.1, the parent-child chunking method removes all whitespaces in child chunks, making them unusable.
  • The root cause is a hardcoded text.strip() call in the TextSplitter class that strips whitespace by default.
  • There is currently no option to disable this behavior, but a contributor suggested adding a strip_whitespace parameter for backward compatibility.
  • Another user inquired about toggling an option off, but no confirmed resolution or update has been provided yet.

Next Steps:

  • Please let me know if this issue is still relevant with the latest version of Dify by commenting here to keep the discussion open.
  • Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Aug 31, 2025): Hi, @tjroamer. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported that in Dify v1.4.1, the parent-child chunking method removes all whitespaces in child chunks, making them unusable. - The root cause is a hardcoded `text.strip()` call in the `TextSplitter` class that strips whitespace by default. - There is currently no option to disable this behavior, but a contributor suggested adding a `strip_whitespace` parameter for backward compatibility. - Another user inquired about toggling an option off, but no confirmed resolution or update has been provided yet. **Next Steps:** - Please let me know if this issue is still relevant with the latest version of Dify by commenting here to keep the discussion open. - Otherwise, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label stale 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#14476
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?