langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 12:36:14 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Implement Overlapping Chunks in Embedding Configuration #939

New Issue
Closed
opened 2026-02-21 17:29:07 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 17:29:07 -05:00
Owner
Copy Link
Copy Source

Originally created by @Matraque on GitHub (Jan 20, 2024).

Originally assigned to: @JohnJyong, @VincePotato on GitHub.

Self Checks

1. Is this request related to a challenge you're experiencing?

Yes, semantic retrieval can pose a challenge when a selected chunk has meaningful context in adjacent chunks that could be missed. This is particularly noticeable when working with PDF files where the separation of chunks is often unclear.

2. Describe the feature you'd like to see

I propose the addition of an overlap configuration for chunks in the embedding configuration. This feature would allow users to choose if they want a chunk overlap and, if enabled, select the character amount of overlap. By default, it can be set to 100 characters.

3. How will this feature improve your workflow or experience?

This feature will significantly enhance the performance of the RAG, especially when dealing with PDF files. It will provide more flexibility and control over the chunking process, thus improving the overall user experience.

4. Additional context or comments

Let me know if you need any more clarification. I believe this feature is very important and is already proposed by most LLM Ops plateforms

5. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @Matraque on GitHub (Jan 20, 2024). Originally assigned to: @JohnJyong, @VincePotato on GitHub. ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to file this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). ### 1. Is this request related to a challenge you're experiencing? Yes, semantic retrieval can pose a challenge when a selected chunk has meaningful context in adjacent chunks that could be missed. This is particularly noticeable when working with PDF files where the separation of chunks is often unclear. ### 2. Describe the feature you'd like to see I propose the addition of an overlap configuration for chunks in the embedding configuration. This feature would allow users to choose if they want a chunk overlap and, if enabled, select the character amount of overlap. By default, it can be set to 100 characters. ### 3. How will this feature improve your workflow or experience? This feature will significantly enhance the performance of the RAG, especially when dealing with PDF files. It will provide more flexibility and control over the chunking process, thus improving the overall user experience. ### 4. Additional context or comments Let me know if you need any more clarification. I believe this feature is very important and is already proposed by most LLM Ops plateforms ### 5. Can you help us with this feature? - [X] I am interested in contributing to this feature.
yindo added the 💪 enhancement👻 feat:rag labels 2026-02-21 17:29:08 -05:00
yindo closed this issue 2026-02-21 17:29:08 -05:00
yindo commented 2026-02-21 17:29:08 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 20, 2024):

🤖

That's a fantastic idea! Adding an overlap configuration for chunks in the embedding configuration would indeed be a valuable addition to dify. It's great to see your interest in contributing to this feature. Your proposed solution could greatly enhance the performance of the RAG and improve the overall user experience, especially when working with PDF files.

I encourage you to go ahead and create a pull request with your proposed changes. Your contribution will be highly appreciated by the dify community. If you need any assistance during the process, feel free to reach out to the dify maintainers or the community for support.

Thank you for your willingness to contribute to dify!

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@dosubot[bot] commented on GitHub (Jan 20, 2024): _🤖_ <!-- Answer --> That's a fantastic idea! Adding an overlap configuration for chunks in the embedding configuration would indeed be a valuable addition to dify. It's great to see your interest in contributing to this feature. Your proposed solution could greatly enhance the performance of the RAG and improve the overall user experience, especially when working with PDF files. I encourage you to go ahead and create a pull request with your proposed changes. Your contribution will be highly appreciated by the dify community. If you need any assistance during the process, feel free to reach out to the dify maintainers or the community for support. Thank you for your willingness to contribute to dify! --- This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
yindo commented 2026-02-21 17:29:08 -05:00
Author
Owner
Copy Link
Copy Source

@canny[bot] commented on GitHub (Jan 23, 2024):

This issue has been linked to a Canny post: Implement Chunks Overlap in Embedding Configuration 🎉

@canny[bot] commented on GitHub (Jan 23, 2024): This issue has been linked to a Canny post: [Implement Chunks Overlap in Embedding Configuration](https://dify.canny.io/admin/board/user-feedback/p/implement-chunks-overlap-in-embedding-configuration) :tada:
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 👻 feat:rag 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#939
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?