langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-20 19:50:06 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #28879] feat: complete test script of sensitive word filter #32220

New Issue
Closed
opened 2026-02-21 20:50:59 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:50:59 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/28879

State: closed
Merged: Yes

Add Comprehensive Unit Tests for Sensitive Word Filter

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

This PR implements a comprehensive test suite for the sensitive word filter (KeywordsModeration) functionality. The test suite includes 100 test cases covering all aspects of the moderation system with detailed inline documentation following the Arrange-Act-Assert pattern.

Fixes

Closes https://github.com/langgenius/dify/issues/28878

What's Added

  • New test file: api/tests/unit_tests/core/moderation/test_sensitive_word_filter.py
  • 100 test cases organized into 15 test classes
  • Comprehensive coverage including:
    • Configuration validation (10 tests)
    • Word list matching (12 tests)
    • Case-insensitive matching (6 tests)
    • Input/output moderation (11 tests)
    • Performance testing with large datasets (6 tests)
    • Edge cases and boundary conditions (18 tests)
    • Multilingual support (6 tests)
    • Complex input types (7 tests)
    • Real-world scenarios (10 tests)
    • Error handling and recovery (5 tests)
    • Wildcard pattern behavior (4 tests)
    • Concurrent moderation (4 tests)

Key Features

✅ All tests pass (100/100)
✅ Ruff lint compliant - No linting issues
✅ Well-documented - Every test has detailed inline comments explaining Arrange-Act-Assert steps
✅ Performance validated - All tests complete within acceptable time limits (< 100-200ms)
✅ Follows TDD principles - Clear test structure with proper assertions

Test Coverage Highlights

  • Configuration validation: Tests all validation rules, limits, and error cases
  • Word matching: Substring matching, position independence, multi-field checking
  • Case sensitivity: Validates case-insensitive matching across all scenarios
  • Performance: Tests with 100 keywords and 10,000 character inputs
  • Internationalization: Tests Chinese, Japanese, Arabic, Cyrillic, and accented characters
  • Edge cases: Unicode emoji, special characters, empty inputs, unusual data types
  • Real-world scenarios: Profanity filtering, spam detection, PII protection, XSS prevention

Documentation

Each test function includes:

  • Descriptive docstrings
  • Inline comments following Arrange-Act-Assert pattern
  • Explanations of expected behavior
  • Context about why each test is important

Screenshots

Not applicable - this is a backend testing change with no UI impact.

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Testing

All tests pass successfully:

pytest tests/unit_tests/core/moderation/test_sensitive_word_filter.py -v
# Result: 100 passed, 1 warning in 0.51s

Linting verification:

ruff check tests/unit_tests/core/moderation/test_sensitive_word_filter.py
# Result: All checks passed!

Additional Notes

  • Tests are designed to be maintainable and easy to understand for new contributors
  • Performance benchmarks ensure the moderation system scales well
  • Comprehensive edge case coverage prevents regression bugs
  • Multilingual support testing ensures global usability

Contribution by Gittensor, learn more at https://gittensor.io/

**Original Pull Request:** https://github.com/langgenius/dify/pull/28879 **State:** closed **Merged:** Yes --- # Add Comprehensive Unit Tests for Sensitive Word Filter > [!IMPORTANT] > > 1. Make sure you have read our [contribution guidelines](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) > 1. Ensure there is an associated issue and you have been assigned to it > 1. Use the correct syntax to link this PR: `Fixes #<issue number>`. ## Summary This PR implements a comprehensive test suite for the sensitive word filter (`KeywordsModeration`) functionality. The test suite includes 100 test cases covering all aspects of the moderation system with detailed inline documentation following the Arrange-Act-Assert pattern. ### Fixes Closes https://github.com/langgenius/dify/issues/28878 ### What's Added - **New test file**: `api/tests/unit_tests/core/moderation/test_sensitive_word_filter.py` - **100 test cases** organized into 15 test classes - **Comprehensive coverage** including: - Configuration validation (10 tests) - Word list matching (12 tests) - Case-insensitive matching (6 tests) - Input/output moderation (11 tests) - Performance testing with large datasets (6 tests) - Edge cases and boundary conditions (18 tests) - Multilingual support (6 tests) - Complex input types (7 tests) - Real-world scenarios (10 tests) - Error handling and recovery (5 tests) - Wildcard pattern behavior (4 tests) - Concurrent moderation (4 tests) ### Key Features ✅ **All tests pass** (100/100) ✅ **Ruff lint compliant** - No linting issues ✅ **Well-documented** - Every test has detailed inline comments explaining Arrange-Act-Assert steps ✅ **Performance validated** - All tests complete within acceptable time limits (< 100-200ms) ✅ **Follows TDD principles** - Clear test structure with proper assertions ### Test Coverage Highlights - **Configuration validation**: Tests all validation rules, limits, and error cases - **Word matching**: Substring matching, position independence, multi-field checking - **Case sensitivity**: Validates case-insensitive matching across all scenarios - **Performance**: Tests with 100 keywords and 10,000 character inputs - **Internationalization**: Tests Chinese, Japanese, Arabic, Cyrillic, and accented characters - **Edge cases**: Unicode emoji, special characters, empty inputs, unusual data types - **Real-world scenarios**: Profanity filtering, spam detection, PII protection, XSS prevention ### Documentation Each test function includes: - Descriptive docstrings - Inline comments following Arrange-Act-Assert pattern - Explanations of expected behavior - Context about why each test is important ## Screenshots Not applicable - this is a backend testing change with no UI impact. ## Checklist - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [x] I've updated the documentation accordingly. - [x] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods ## Testing All tests pass successfully: ```bash pytest tests/unit_tests/core/moderation/test_sensitive_word_filter.py -v # Result: 100 passed, 1 warning in 0.51s ``` Linting verification: ```bash ruff check tests/unit_tests/core/moderation/test_sensitive_word_filter.py # Result: All checks passed! ``` ## Additional Notes - Tests are designed to be maintainable and easy to understand for new contributors - Performance benchmarks ensure the moderation system scales well - Comprehensive edge case coverage prevents regression bugs - Multilingual support testing ensures global usability Contribution by Gittensor, learn more at https://gittensor.io/
yindo added the pull-request label 2026-02-21 20:50:59 -05:00
yindo closed this issue 2026-02-21 20:50:59 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#32220
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?