langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Chunks get spaces removed #16305

New Issue
Closed
opened 2026-02-21 19:25:33 -05:00 by yindo · 9 comments
yindo commented 2026-02-21 19:25:33 -05:00
Owner
Copy Link
Copy Source

Originally created by @DavideDelbianco on GitHub (Aug 20, 2025).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.7.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Have a document indexed as Parent-Child (full-doc)
I don't understand why sometimes spaces are removed and sometimes not.
The option to remove consecutive spaces, newlines and tabs is active.
But the spaces being removed are not consecutive, and it does not happen all the time.

Using Dify ETL

✔️ Expected Behavior

Should not remove spaces without reason

❌ Actual Behavior

Child chunks have spaces removed

Image Image
Originally created by @DavideDelbianco on GitHub (Aug 20, 2025). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.7.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Have a document indexed as Parent-Child (full-doc) I don't understand why sometimes spaces are removed and sometimes not. The option to remove consecutive spaces, newlines and tabs is active. But the spaces being removed are not consecutive, and it does not happen all the time. Using Dify ETL ### ✔️ Expected Behavior Should not remove spaces without reason ### ❌ Actual Behavior Child chunks have spaces removed <img width="1111" height="374" alt="Image" src="https://github.com/user-attachments/assets/fcb3088e-30b6-44a0-a373-c56837e3f239" /> <img width="898" height="470" alt="Image" src="https://github.com/user-attachments/assets/71f3cfe4-4c59-4478-a84c-3a2c3e18e8ec" />
yindo added the 🐞 buggood first issue labels 2026-02-21 19:25:33 -05:00
yindo closed this issue 2026-02-21 19:25:33 -05:00
yindo commented 2026-02-21 19:25:33 -05:00
Author
Owner
Copy Link
Copy Source

@DavideDelbianco commented on GitHub (Aug 20, 2025):

https://github.com/user-attachments/assets/9d473982-6920-4a34-b486-6bb37285d50f

I've just figured out how to replicate it.
In my case i have a single long phrase:

Matterhorn Museum (0.6 km), Gornerschlucht (0.8 km), Zermatt-Matterhorn Ski Resort (1.1 km), Zermatt Railway Station (1.2 km), Sunnegga Ski Resort (1.8 km), Riffelsee (3.7 km), Riffelhorn (3.8 km), Unter Gabelhorn (4 km), Schwarzsee (4..1km), Gornergrat (4..5km), KOSMA (4..5km), Hohtalli (5..2km).

If my chunking settings are parent-child full-doc with child chunk on ## (notice that char combination is not present)
and the whole phrase is longer than my child chunk settings (298 characters vs 256 chunk lenght)
the spaces between every word are removed.

if i choose a bigger chunk setting for example 512 the spaces are not removed

@DavideDelbianco commented on GitHub (Aug 20, 2025): https://github.com/user-attachments/assets/9d473982-6920-4a34-b486-6bb37285d50f I've just figured out how to replicate it. In my case i have a single long phrase: ```txt Matterhorn Museum (0.6 km), Gornerschlucht (0.8 km), Zermatt-Matterhorn Ski Resort (1.1 km), Zermatt Railway Station (1.2 km), Sunnegga Ski Resort (1.8 km), Riffelsee (3.7 km), Riffelhorn (3.8 km), Unter Gabelhorn (4 km), Schwarzsee (4..1km), Gornergrat (4..5km), KOSMA (4..5km), Hohtalli (5..2km). ``` If my chunking settings are parent-child full-doc with child chunk on ## (notice that char combination is not present) and the whole phrase is longer than my child chunk settings (298 characters vs 256 chunk lenght) the spaces between every word are removed. if i choose a bigger chunk setting for example 512 the spaces are not removed
yindo commented 2026-02-21 19:25:33 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 20, 2025):

This is an edge case in Dify's chunking logic: when your chosen delimiter ("##") isn't present and the chunk size is smaller than the text, the splitter falls back to splitting the text into individual characters. When these are merged back into chunks, spaces between words are lost, resulting in child chunks with no spaces. This only happens when the chunk size forces a split; if the chunk size is large enough (like 512), the whole phrase stays intact and spaces are preserved. The preprocessing option only collapses multiple spaces into one—it doesn't remove all spaces between words, so it's not the cause here.

To avoid this, use a delimiter that actually appears in your text, or set a chunk size large enough to fit your longest phrase. More details on the chunking logic can be found in the code here and here.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Aug 20, 2025): <!-- Answer --> This is an edge case in Dify's chunking logic: when your chosen delimiter ("##") isn't present and the chunk size is smaller than the text, the splitter falls back to splitting the text into individual characters. When these are merged back into chunks, spaces between words are lost, resulting in child chunks with no spaces. This only happens when the chunk size forces a split; if the chunk size is large enough (like 512), the whole phrase stays intact and spaces are preserved. The preprocessing option only collapses multiple spaces into one—it doesn't remove all spaces between words, so it's not the cause here. To avoid this, use a delimiter that actually appears in your text, or set a chunk size large enough to fit your longest phrase. More details on the chunking logic can be found in the code [here](https://github.com/langgenius/dify/blob/1caeac56f291a393fa2c418baa1e6c56d31f6655/api/core/rag/splitter/text_splitter.py) and [here](https://github.com/langgenius/dify/blob/1caeac56f291a393fa2c418baa1e6c56d31f6655/api/core/rag/splitter/fixed_text_splitter.py). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/988a742b-e959-4268-a486-abaa8e6dbaf9?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/988a742b-e959-4268-a486-abaa8e6dbaf9?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/988a742b-e959-4268-a486-abaa8e6dbaf9?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/988a742b-e959-4268-a486-abaa8e6dbaf9?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/988a742b-e959-4268-a486-abaa8e6dbaf9?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/988a742b-e959-4268-a486-abaa8e6dbaf9?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/988a742b-e959-4268-a486-abaa8e6dbaf9?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/24251)
yindo commented 2026-02-21 19:25:33 -05:00
Author
Owner
Copy Link
Copy Source

@DavideDelbianco commented on GitHub (Aug 20, 2025):

@crazywoola
Please can you have someone fix this ASAP?
It is causing major issues to me, because the semantic search will fail with the phrases transformed into a unique giant word...

The logic should be:
-> Split text by separator
-> Evaluate every chunk content obtained
-> Is the Chunk longer than the chosen limit?
-> NO -> Push the chunk into the list of chunks to save
-> YES -> Split current chunk on space character

** In case of YES do the following to recreate the chunk up to its limit:**
-> Cycle through split elements keeping a chunk length counter
-> For every "word" do the following:
--> determine the piece of text to concatenate (we call it concatenee)
--> Is the chunk NOT empty?
----> Then the concatenee = " word" (please notice the space) otherwise it is "word"
--> Check if chunk length + concatenee length is over the max length
--> YES -> push the chunk in the list of chunks to save and start a new one
--> Append concatenee to chunk (it will be either concatenated to previous phrase or the start of a new chunk)

@DavideDelbianco commented on GitHub (Aug 20, 2025): @crazywoola Please can you have someone fix this ASAP? It is causing major issues to me, because the semantic search will fail with the phrases transformed into a unique giant word... **The logic should be:** -> Split text by separator -> Evaluate every chunk content obtained -> Is the Chunk longer than the chosen limit? -> NO -> Push the chunk into the list of chunks to save -> YES -> Split current chunk on space character ** In case of YES do the following to recreate the chunk up to its limit:** -> Cycle through split elements keeping a chunk length counter -> For every "word" do the following: --> determine the piece of text to concatenate (we call it concatenee) --> Is the chunk NOT empty? ----> Then the concatenee = " word" (please notice the space) otherwise it is "word" --> Check if chunk length + concatenee length is over the max length --> YES -> push the chunk in the list of chunks to save and start a new one --> Append concatenee to chunk (it will be either concatenated to previous phrase or the start of a new chunk)
yindo commented 2026-02-21 19:25:33 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Aug 20, 2025):

@crazywoola Please can you have someone fix this ASAP? It is causing major issues to me, because the semantic search will fail with the phrases transformed into a unique giant word...

The logic should be: -> Split text by separator -> Evaluate every chunk content obtained -> Is the Chunk longer than the chosen limit? -> NO -> Push the chunk into the list of chunks to save -> YES -> Split current chunk on space character

** In case of YES do the following to recreate the chunk up to its limit:** -> Cycle through split elements keeping a chunk length counter -> For every "word" do the following: --> determine the piece of text to concatenate (we call it concatenee) --> Is the chunk NOT empty? ----> Then the concatenee = " word" (please notice the space) otherwise it is "word" --> Check if chunk length + concatenee length is over the max length --> YES -> push the chunk in the list of chunks to save and start a new one --> Append concatenee to chunk (it will be either concatenated to previous phrase or the start of a new chunk)

Thanks for the detailed information, these are very helpful, I will try to find a solution to it.

@crazywoola commented on GitHub (Aug 20, 2025): > [@crazywoola](https://github.com/crazywoola) Please can you have someone fix this ASAP? It is causing major issues to me, because the semantic search will fail with the phrases transformed into a unique giant word... > > **The logic should be:** -> Split text by separator -> Evaluate every chunk content obtained -> Is the Chunk longer than the chosen limit? -> NO -> Push the chunk into the list of chunks to save -> YES -> Split current chunk on space character > > ** In case of YES do the following to recreate the chunk up to its limit:** -> Cycle through split elements keeping a chunk length counter -> For every "word" do the following: --> determine the piece of text to concatenate (we call it concatenee) --> Is the chunk NOT empty? ----> Then the concatenee = " word" (please notice the space) otherwise it is "word" --> Check if chunk length + concatenee length is over the max length --> YES -> push the chunk in the list of chunks to save and start a new one --> Append concatenee to chunk (it will be either concatenated to previous phrase or the start of a new chunk) Thanks for the detailed information, these are very helpful, I will try to find a solution to it.
yindo commented 2026-02-21 19:25:34 -05:00
Author
Owner
Copy Link
Copy Source

@jubinsoni commented on GitHub (Aug 22, 2025):

pr to fix this
https://github.com/langgenius/dify/pull/24366

@crazywoola pls review

@jubinsoni commented on GitHub (Aug 22, 2025): pr to fix this https://github.com/langgenius/dify/pull/24366 @crazywoola pls review
yindo commented 2026-02-21 19:25:34 -05:00
Author
Owner
Copy Link
Copy Source

@DavideDelbianco commented on GitHub (Aug 22, 2025):

The fix may preserve spaces, but I think the current algorithm is faulty.
It logs a warning for phrases longer than chunk length.
If I specify a chunk length it may be dictated by the LLM context length.
The algorithm should try to respect that length and as per my suggestion, split the text in length compliant pieces, while also preserving spaces.

@DavideDelbianco commented on GitHub (Aug 22, 2025): The fix may preserve spaces, but I think the current algorithm is faulty. It logs a warning for phrases longer than chunk length. If I specify a chunk length it may be dictated by the LLM context length. The algorithm should try to respect that length and as per my suggestion, split the text in length compliant pieces, while also preserving spaces.
yindo commented 2026-02-21 19:25:34 -05:00
Author
Owner
Copy Link
Copy Source

@DavideDelbianco commented on GitHub (Aug 31, 2025):

@crazywoola is there a release date for RAG 2.0 ?
It seems like it is a major blocker for anything RAG related....

@DavideDelbianco commented on GitHub (Aug 31, 2025): @crazywoola is there a release date for RAG 2.0 ? It seems like it is a major blocker for anything RAG related....
yindo commented 2026-02-21 19:25:34 -05:00
Author
Owner
Copy Link
Copy Source

@maxliaops commented on GitHub (Sep 30, 2025):

  • Summary

    • Single newlines are dropped during chunking because the active separator list omits "\n", fallback splitting uses text.split() (which strips all whitespace), and the merge phase doesn’t reinsert separators due to an inverted keep_separator check.

  • Details

    1. Separator list in hierarchical/custom mode excludes "\n"

      • The splitter is created with:
        • separators=["\n\n", "。", ". ", " ", ""]
      • Because "\n" is missing, single newlines aren’t treated as boundaries.

    2. Fallback to space uses text.split()

      • When the chosen separator isn’t present or a piece still exceeds chunk_size, the splitter falls back to lower-priority separators.
      • For the space (" ") case it calls text.split(), which treats any whitespace as a delimiter and removes newline characters.

    3. Merge phase doesn’t preserve separators (logic inverted)

      • Current code uses:
        • _separator = "" if self._keep_separator else separator
      • This means even when keep_separator=True, the join uses an empty string, so line breaks are not reinserted.

    4. Filtering further removes newline-only tokens

      • The code drops empty strings and "\n" pieces:
        • splits = [s for s in splits if (s not in {"", "\n"})]

  • Impact

    • Markdown tables and long sentences can collapse into a single line, harming retrieval quality.

  • Fix proposals

    • Include "\n" in the separator list with higher priority:
      • separators=["\n\n", "\n", "。", ". ", " ", ""]
    • Correct the merge logic:
      • _separator = separator if self._keep_separator else ""
    • Optionally, allow configuring keep_separator and separators via processing rules; for line-aware chunking set separator to "\n".
@maxliaops commented on GitHub (Sep 30, 2025): - Summary - Single newlines are dropped during chunking because the active separator list omits "\n", fallback splitting uses text.split() (which strips all whitespace), and the merge phase doesn’t reinsert separators due to an inverted keep_separator check. - Details 1) Separator list in hierarchical/custom mode excludes "\n" - The splitter is created with: - separators=["\n\n", "。", ". ", " ", ""] - Because "\n" is missing, single newlines aren’t treated as boundaries. 2) Fallback to space uses text.split() - When the chosen separator isn’t present or a piece still exceeds chunk_size, the splitter falls back to lower-priority separators. - For the space (" ") case it calls text.split(), which treats any whitespace as a delimiter and removes newline characters. 3) Merge phase doesn’t preserve separators (logic inverted) - Current code uses: - _separator = "" if self._keep_separator else separator - This means even when keep_separator=True, the join uses an empty string, so line breaks are not reinserted. 4) Filtering further removes newline-only tokens - The code drops empty strings and "\n" pieces: - splits = [s for s in splits if (s not in {"", "\n"})] - Impact - Markdown tables and long sentences can collapse into a single line, harming retrieval quality. - Fix proposals - Include "\n" in the separator list with higher priority: - separators=["\n\n", "\n", "。", ". ", " ", ""] - Correct the merge logic: - _separator = separator if self._keep_separator else "" - Optionally, allow configuring keep_separator and separators via processing rules; for line-aware chunking set separator to "\n".
yindo commented 2026-02-21 19:25:34 -05:00
Author
Owner
Copy Link
Copy Source

@jubinsoni commented on GitHub (Oct 6, 2025):

closing the other pr and sharing a new one
implemented new logic suggested by maxliaops and added a test screenshot
pr : https://github.com/langgenius/dify/pull/26623

pls review @crazywoola

@jubinsoni commented on GitHub (Oct 6, 2025): closing the other pr and sharing a new one implemented new logic suggested by maxliaops and added a test screenshot pr : https://github.com/langgenius/dify/pull/26623 pls review @crazywoola
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label good first issue 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#16305
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?