langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-27 21:10:04 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #7930] fix: split text keep separator #25869

New Issue
Closed
opened 2026-02-21 20:28:16 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:28:16 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/7930

State: closed
Merged: Yes

Checklist:

Important

Please review the checklist below before submitting your pull request.

  • Please open an issue before creating a PR or link to an existing issue
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Description

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. Close issue syntax: Fixes #<issue number>, see documentation for more details.

Fixes #7929

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update, included: Dify Document
  • Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
  • Dependency upgrade

Testing Instructions

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

from core.rag.splitter.fixed_text_splitter import EnhanceRecursiveCharacterTextSplitter

character_splitter = EnhanceRecursiveCharacterTextSplitter.from_encoder(
    chunk_size=512,
    chunk_overlap=50,
    separators=None,
    embedding_model_instance=None
)

text = '''## 程序员和其他职业的区别

如果从纯粹外行的角度去简单的分辨，程序员最大的特征就是薪资很高。  
  
比起其他行业，动辄月薪一两万，年薪二三十万的程序员确实太香了。在薪资较高的那些行业中，比起对身份和背景有要求的金融，对学历和资历有要求医学，程序员既不需要你有不俗的家世背景去拉资源，也不需要你付出动辄十几年的努力去苦苦熬经验，几乎是普通家庭的孩子想要追高薪资的最佳选择。  
  
从某种意义上说，程序员的“前途”，是一种“钱途“。  
  
很多人会有这样的疑惑：

凭什么程序员这么有钱途？

如果从入行的角度简单的观察一下，程序员和其他非技术部门的工作日常，你就会发现，程序员的工作难度是最高的。  
  
突发的 bug 需要立即修复，难以用现有技术和知识去解决的复杂的业务场景，永远在学习，永远在更迭最新的工具和技能树是程序员的常态。  
  
比起文职类的简单工作，或是流水线工人的按部就班，作为程序员，花上一天时间去解决一个莫名其妙的报错，你此前学过的所有可能都无法派上用场，面对这些情况也许是家常便饭。  
  
修改不起作用的挫败，业务必须要完成的压力，不停 battle 强人所难的需求，处理前人留下的”屎山“代码，所有的这些对身心的双重折磨，都造就了程序员为什么薪资这么高的原因。  
  
本质上就是，难度越高的，赏金就越多。  
  
如果你打算要成为程序员，以上这些你都要做好准备。'''


list = character_splitter._split_text(text, ["\n\n", "。", ". ", " ", ""])

for i in enumerate(list):
    print(i)
**Original Pull Request:** https://github.com/langgenius/dify/pull/7930 **State:** closed **Merged:** Yes --- # Checklist: > [!IMPORTANT] > Please review the checklist below before submitting your pull request. - [x] Please open an issue before creating a PR or link to an existing issue - [ ] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods # Description Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. Close issue syntax: `Fixes #<issue number>`, see [documentation](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) for more details. Fixes #7929 ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [ ] Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement - [ ] Dependency upgrade # Testing Instructions Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration ```python from core.rag.splitter.fixed_text_splitter import EnhanceRecursiveCharacterTextSplitter character_splitter = EnhanceRecursiveCharacterTextSplitter.from_encoder( chunk_size=512, chunk_overlap=50, separators=None, embedding_model_instance=None ) text = '''## 程序员和其他职业的区别 如果从纯粹外行的角度去简单的分辨，程序员最大的特征就是薪资很高。 比起其他行业，动辄月薪一两万，年薪二三十万的程序员确实太香了。在薪资较高的那些行业中，比起对身份和背景有要求的金融，对学历和资历有要求医学，程序员既不需要你有不俗的家世背景去拉资源，也不需要你付出动辄十几年的努力去苦苦熬经验，几乎是普通家庭的孩子想要追高薪资的最佳选择。 从某种意义上说，程序员的“前途”，是一种“钱途“。 很多人会有这样的疑惑： 凭什么程序员这么有钱途？ 如果从入行的角度简单的观察一下，程序员和其他非技术部门的工作日常，你就会发现，程序员的工作难度是最高的。 突发的 bug 需要立即修复，难以用现有技术和知识去解决的复杂的业务场景，永远在学习，永远在更迭最新的工具和技能树是程序员的常态。 比起文职类的简单工作，或是流水线工人的按部就班，作为程序员，花上一天时间去解决一个莫名其妙的报错，你此前学过的所有可能都无法派上用场，面对这些情况也许是家常便饭。 修改不起作用的挫败，业务必须要完成的压力，不停 battle 强人所难的需求，处理前人留下的”屎山“代码，所有的这些对身心的双重折磨，都造就了程序员为什么薪资这么高的原因。 本质上就是，难度越高的，赏金就越多。 如果你打算要成为程序员，以上这些你都要做好准备。''' list = character_splitter._split_text(text, ["\n\n", "。", ". ", " ", ""]) for i in enumerate(list): print(i) ```
yindo added the pull-request label 2026-02-21 20:28:16 -05:00
yindo closed this issue 2026-02-21 20:28:19 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#25869
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?