langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 12:36:14 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #28813] feat: complete test script of text splitter #32188

New Issue
Closed
opened 2026-02-21 20:50:55 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:50:55 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/28813

State: closed
Merged: Yes

feat: Add comprehensive test suite for text splitter functionality

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

This PR adds a comprehensive test suite for text splitter functionality in the RAG module. The test suite provides extensive coverage for all text splitting strategies used in the system.

Fixes

Closes https://github.com/langgenius/dify/issues/28812

What's Added

New Test File: api/tests/unit_tests/core/rag/splitter/test_text_splitter.py

The test suite includes 93 passing tests covering:

Core Splitter Types

  • RecursiveCharacterTextSplitter: Main splitter with recursive separator hierarchy
  • TokenTextSplitter: Token-based splitting using tiktoken
  • EnhanceRecursiveCharacterTextSplitter: Enhanced version with custom token counting
  • FixedRecursiveCharacterTextSplitter: Fixed separator with recursive fallback

Test Categories

  1. Helper Functions - Low-level splitting utilities and regex handling
  2. Core Functionality - Initialization, basic operations, separator behavior
  3. Overlap Handling - Context preservation between chunks
  4. Metadata Preservation - Source tracking and metadata inheritance
  5. Edge Cases - Empty text, unicode, extreme parameters
  6. Advanced Scenarios - Markdown, HTML, JSON, code, emails, tables
  7. Configuration Testing - Custom length functions, separator orders, overlap ratios
  8. Error Handling - Invalid inputs, special characters, robustness
  9. Performance - Consistency, determinism, information preservation

Key Features

  • ✅ 93 comprehensive test cases
  • ✅ 9 reusable pytest fixtures for different content types
  • ✅ Extensive documentation with docstrings and inline comments
  • ✅ Real-world scenarios (markdown, HTML, JSON, technical docs)
  • ✅ Edge case coverage (unicode, emojis, control characters)
  • ✅ Performance and determinism verification
  • ✅ All tests pass with proper error handling

Test Coverage

  • Recursive character splitting with multiple separator levels
  • Token-based splitting with overlap
  • Semantic-like splitting for structured documents
  • Overlap handling for context preservation
  • Metadata preservation across all splitter types
  • Unicode and multilingual text support
  • Various document formats (markdown, HTML, JSON, code, emails)

Documentation

Each test includes:

  • Clear docstrings explaining purpose and importance
  • Inline comments for complex assertions
  • Usage examples in module documentation
  • Test execution instructions

Screenshots

Not applicable - this is a test suite addition with no UI changes.

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Additional Notes

  • All 93 tests pass successfully
  • 4 tests are skipped (require tiktoken library)
  • Code follows project conventions and passes Ruff linter
  • Tests are well-organized in 11 test classes for maintainability
  • Comprehensive docstrings and comments for better understanding

Contribution by Gittensor, learn more at https://gittensor.io/

**Original Pull Request:** https://github.com/langgenius/dify/pull/28813 **State:** closed **Merged:** Yes --- # feat: Add comprehensive test suite for text splitter functionality > [!IMPORTANT] > > 1. Make sure you have read our [contribution guidelines](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) > 1. Ensure there is an associated issue and you have been assigned to it > 1. Use the correct syntax to link this PR: `Fixes #<issue number>`. ## Summary This PR adds a comprehensive test suite for text splitter functionality in the RAG module. The test suite provides extensive coverage for all text splitting strategies used in the system. ### Fixes Closes https://github.com/langgenius/dify/issues/28812 ### What's Added **New Test File**: `api/tests/unit_tests/core/rag/splitter/test_text_splitter.py` The test suite includes **93 passing tests** covering: #### Core Splitter Types - **RecursiveCharacterTextSplitter**: Main splitter with recursive separator hierarchy - **TokenTextSplitter**: Token-based splitting using tiktoken - **EnhanceRecursiveCharacterTextSplitter**: Enhanced version with custom token counting - **FixedRecursiveCharacterTextSplitter**: Fixed separator with recursive fallback #### Test Categories 1. **Helper Functions** - Low-level splitting utilities and regex handling 2. **Core Functionality** - Initialization, basic operations, separator behavior 3. **Overlap Handling** - Context preservation between chunks 4. **Metadata Preservation** - Source tracking and metadata inheritance 5. **Edge Cases** - Empty text, unicode, extreme parameters 6. **Advanced Scenarios** - Markdown, HTML, JSON, code, emails, tables 7. **Configuration Testing** - Custom length functions, separator orders, overlap ratios 8. **Error Handling** - Invalid inputs, special characters, robustness 9. **Performance** - Consistency, determinism, information preservation #### Key Features - ✅ 93 comprehensive test cases - ✅ 9 reusable pytest fixtures for different content types - ✅ Extensive documentation with docstrings and inline comments - ✅ Real-world scenarios (markdown, HTML, JSON, technical docs) - ✅ Edge case coverage (unicode, emojis, control characters) - ✅ Performance and determinism verification - ✅ All tests pass with proper error handling ### Test Coverage - Recursive character splitting with multiple separator levels - Token-based splitting with overlap - Semantic-like splitting for structured documents - Overlap handling for context preservation - Metadata preservation across all splitter types - Unicode and multilingual text support - Various document formats (markdown, HTML, JSON, code, emails) ### Documentation Each test includes: - Clear docstrings explaining purpose and importance - Inline comments for complex assertions - Usage examples in module documentation - Test execution instructions ## Screenshots Not applicable - this is a test suite addition with no UI changes. ## Checklist - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [x] I've updated the documentation accordingly. - [x] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods ## Additional Notes - All 93 tests pass successfully - 4 tests are skipped (require tiktoken library) - Code follows project conventions and passes Ruff linter - Tests are well-organized in 11 test classes for maintainability - Comprehensive docstrings and comments for better understanding Contribution by Gittensor, learn more at https://gittensor.io/
yindo added the pull-request label 2026-02-21 20:50:55 -05:00
yindo closed this issue 2026-02-21 20:50:55 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#32188
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?