langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Regular Expression in dify/api/core/tools/utils/text_processing_utils.py Removes Leading 《 Character from Document Segments #14738

New Issue
Closed
opened 2026-02-21 19:18:34 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 19:18:34 -05:00
Owner
Copy Link
Copy Source

Originally created by @hizhujianfeng on GitHub (Jun 19, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

latest

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Bug Report

Title: Regular Expression in dify/api/core/tools/utils/text_processing_utils.py Removes Leading 《 Character from Document Segments

Description:
The regular expression defined in dify/api/core/tools/utils/text_processing_utils.py (specifically, the remove_leading_symbols function) incorrectly removes the leading 《 character (U+300A, LEFT DOUBLE ANGLE BRACKET) from document segments. This behavior is unintended, as 《 is commonly used in CJK texts (e.g., as a book title marker) and should not be stripped from the start of text segments.

Steps to Reproduce:

  1. Use the remove_leading_symbols function from dify/api/core/tools/utils/text_processing_utils.py.
  2. Pass a string starting with 《, such as 《Example Text.
  3. Observe the output.

Expected Behavior:

  • The function should preserve the 《 character, as it is not a punctuation or symbol that should be removed.
  • Output for 《Example Text should remain 《Example Text.

Actual Behavior:

  • The function removes the leading 《 character.
  • Output for 《Example Text is Example Text.

Cause:

  • The regular expression r"^[\u2000-\u206F\u2E00-\u2E7F\u3000-\u303F!\"#$%&'()*+,./:;<=>?@^_~]+"includes the Unicode range\u3000-\u303F, which covers CJK symbols and punctuation, including 《` (U+300A).
  • This range is too broad, causing the function to strip valid characters like 《.

Suggested Fix:

  • Refactor the regular expression to exclude specific CJK characters like 《 (U+300A) and 》 (U+300B) from the removal logic.
  • Alternatively, use a more precise Unicode property (e.g., \p{P} for punctuation) and explicitly exclude book title markers.
  • Add test cases to ensure 《 and similar characters are preserved.

Additional Notes:

  • The function includes a comment: # FIXME this pattern is confused quick fix for #11868 maybe refactor it later, indicating the regular expression is a temporary solution and may need broader refactoring.
  • This issue may affect other CJK texts where 《 or similar characters are used meaningfully.

Environment:

  • File: dify/api/core/tools/utils/text_processing_utils.py
  • Function: remove_leading_symbols
  • Related Issue: Possibly linked to #11868 (as noted in the code comment)

Priority: Medium (affects CJK text processing accuracy)
Component: Text Processing Utilities

✔️ Expected Behavior

Refactor the regular expression to exclude specific CJK characters like 《 (U+300A) and 》 (U+300B) from the removal logic.

❌ Actual Behavior

No response

Originally created by @hizhujianfeng on GitHub (Jun 19, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version latest ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce **Bug Report** **Title**: Regular Expression in `dify/api/core/tools/utils/text_processing_utils.py` Removes Leading `《` Character from Document Segments **Description**: The regular expression defined in `dify/api/core/tools/utils/text_processing_utils.py` (specifically, the `remove_leading_symbols` function) incorrectly removes the leading `《` character (U+300A, LEFT DOUBLE ANGLE BRACKET) from document segments. This behavior is unintended, as `《` is commonly used in CJK texts (e.g., as a book title marker) and should not be stripped from the start of text segments. **Steps to Reproduce**: 1. Use the `remove_leading_symbols` function from `dify/api/core/tools/utils/text_processing_utils.py`. 2. Pass a string starting with `《`, such as `《Example Text`. 3. Observe the output. **Expected Behavior**: - The function should preserve the `《` character, as it is not a punctuation or symbol that should be removed. - Output for `《Example Text` should remain `《Example Text`. **Actual Behavior**: - The function removes the leading `《` character. - Output for `《Example Text` is `Example Text`. **Cause**: - The regular expression `r"^[\u2000-\u206F\u2E00-\u2E7F\u3000-\u303F!\"#$%&'()*+,./:;<=>?@^_`~]+"` includes the Unicode range `\u3000-\u303F`, which covers CJK symbols and punctuation, including `《` (U+300A). - This range is too broad, causing the function to strip valid characters like `《`. **Suggested Fix**: - Refactor the regular expression to exclude specific CJK characters like `《` (U+300A) and `》` (U+300B) from the removal logic. - Alternatively, use a more precise Unicode property (e.g., `\p{P}` for punctuation) and explicitly exclude book title markers. - Add test cases to ensure `《` and similar characters are preserved. **Additional Notes**: - The function includes a comment: `# FIXME this pattern is confused quick fix for #11868 maybe refactor it later`, indicating the regular expression is a temporary solution and may need broader refactoring. - This issue may affect other CJK texts where `《` or similar characters are used meaningfully. **Environment**: - File: `dify/api/core/tools/utils/text_processing_utils.py` - Function: `remove_leading_symbols` - Related Issue: Possibly linked to #11868 (as noted in the code comment) **Priority**: Medium (affects CJK text processing accuracy) **Component**: Text Processing Utilities ### ✔️ Expected Behavior Refactor the regular expression to exclude specific CJK characters like 《 (U+300A) and 》 (U+300B) from the removal logic. ### ❌ Actual Behavior _No response_
yindo added the 🐞 bugstale labels 2026-02-21 19:18:34 -05:00
yindo closed this issue 2026-02-21 19:18:34 -05:00
yindo commented 2026-02-21 19:18:34 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 22, 2025):

Hi, @hizhujianfeng. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported that the remove_leading_symbols function in dify/api/core/tools/utils/text_processing_utils.py incorrectly removes the leading 《 character.
  • This character is important in CJK texts and should not be stripped.
  • The problem is caused by the regex pattern including the Unicode range that covers 《.
  • There has been no further discussion or updates on this issue so far.

Next Steps:

  • Please let me know if this issue is still relevant with the latest version of Dify by commenting here.
  • If I don’t hear back within 15 days, I will automatically close this issue.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Aug 22, 2025): Hi, @hizhujianfeng. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported that the `remove_leading_symbols` function in `dify/api/core/tools/utils/text_processing_utils.py` incorrectly removes the leading `《` character. - This character is important in CJK texts and should not be stripped. - The problem is caused by the regex pattern including the Unicode range that covers `《`. - There has been no further discussion or updates on this issue so far. **Next Steps:** - Please let me know if this issue is still relevant with the latest version of Dify by commenting here. - If I don’t hear back within 15 days, I will automatically close this issue. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label stale 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#14738
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?