langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #28900] Feat/Test Suite for Document Processing Task #32230

New Issue
Closed
opened 2026-02-21 20:51:00 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:51:00 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/28900

State: closed
Merged: No

Add Comprehensive Test Suite for Document Processing Tasks

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

This PR implements a comprehensive unit test suite for document processing tasks, covering all critical aspects of the document indexing workflow. The test suite ensures robust testing of document parsing, chunking strategies, embedding generation, status updates, and failure recovery mechanisms.

Fixes

Closes https://github.com/langgenius/dify/issues/28899

What's Changed

  • Added: tests/unit_tests/tasks/test_document_processing_task.py with 53 comprehensive test cases
  • Coverage: Document parsing, chunking strategy application, embedding generation, status updates, failure recovery, concurrent processing, document cleaning, and task integration

Key Features

  1. Document Parsing Tests (8 tests)

    • Multiple document batch processing
    • Upload file, Notion import, and website crawl extraction
    • Missing data validation and error handling

  2. Chunking Strategy Tests (11 tests)

    • Automatic, custom, and hierarchical chunking modes
    • Token limit validation
    • Multi-language support
    • Custom separator handling

  3. Embedding Generation Tests (7 tests)

    • High-quality vector indexing
    • Economy keyword-based indexing
    • Large batch processing (50+ documents)
    • Parent-child document structures

  4. Status Updates Tests (5 tests)

    • Complete lifecycle: parsing → splitting → indexing → completed
    • Segment status tracking

  5. Failure Recovery Tests (7 tests)

    • Provider token errors
    • Document pause detection
    • Vector space quota limits
    • Batch upload restrictions
    • Sandbox plan limitations

  6. Concurrent Processing Tests (3 tests)

    • Parallel document grouping
    • Tenant-isolated task queues
    • FIFO queue processing

  7. Document Cleaning Tests (5 tests)

    • Special character removal
    • Unicode text preservation
    • Control character filtering

  8. Task Integration Tests (7 tests)

    • End-to-end workflows for normal and priority tasks
    • Edge case handling (empty lists, missing data)
    • Different document forms (paragraph, QA, parent-child)

Test Statistics

  • Total Tests: 53
  • Pass Rate: 100%
  • Test Structure: Organized into 8 test classes using Arrange-Act-Assert pattern
  • Documentation: Comprehensive docstrings for each test explaining purpose and verification

Dependencies

No new dependencies required. Uses existing testing infrastructure:

  • pytest for test framework
  • unittest.mock for mocking
  • Existing Dify models and services

Screenshots

Not applicable - this is a backend testing enhancement with no UI changes.

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Testing

All tests pass successfully:

uv run --project api pytest api/tests/unit_tests/tasks/test_document_processing_task.py -v
# Result: 53 passed in 33.74s

The test suite follows TDD best practices with:

  • Clear test organization using factory pattern for test data
  • Proper mocking of external dependencies (DB, Redis, services)
  • Comprehensive edge case coverage
  • Self-documenting code with detailed docstrings

Contribution by Gittensor, learn more at https://gittensor.io/

**Original Pull Request:** https://github.com/langgenius/dify/pull/28900 **State:** closed **Merged:** No --- # Add Comprehensive Test Suite for Document Processing Tasks > [!IMPORTANT] > > 1. Make sure you have read our [contribution guidelines](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) > 1. Ensure there is an associated issue and you have been assigned to it > 1. Use the correct syntax to link this PR: `Fixes #<issue number>`. ## Summary This PR implements a comprehensive unit test suite for document processing tasks, covering all critical aspects of the document indexing workflow. The test suite ensures robust testing of document parsing, chunking strategies, embedding generation, status updates, and failure recovery mechanisms. ### Fixes Closes https://github.com/langgenius/dify/issues/28899 ### What's Changed - **Added**: `tests/unit_tests/tasks/test_document_processing_task.py` with 53 comprehensive test cases - **Coverage**: Document parsing, chunking strategy application, embedding generation, status updates, failure recovery, concurrent processing, document cleaning, and task integration ### Key Features 1. **Document Parsing Tests (8 tests)** - Multiple document batch processing - Upload file, Notion import, and website crawl extraction - Missing data validation and error handling 2. **Chunking Strategy Tests (11 tests)** - Automatic, custom, and hierarchical chunking modes - Token limit validation - Multi-language support - Custom separator handling 3. **Embedding Generation Tests (7 tests)** - High-quality vector indexing - Economy keyword-based indexing - Large batch processing (50+ documents) - Parent-child document structures 4. **Status Updates Tests (5 tests)** - Complete lifecycle: parsing → splitting → indexing → completed - Segment status tracking 5. **Failure Recovery Tests (7 tests)** - Provider token errors - Document pause detection - Vector space quota limits - Batch upload restrictions - Sandbox plan limitations 6. **Concurrent Processing Tests (3 tests)** - Parallel document grouping - Tenant-isolated task queues - FIFO queue processing 7. **Document Cleaning Tests (5 tests)** - Special character removal - Unicode text preservation - Control character filtering 8. **Task Integration Tests (7 tests)** - End-to-end workflows for normal and priority tasks - Edge case handling (empty lists, missing data) - Different document forms (paragraph, QA, parent-child) ### Test Statistics - **Total Tests**: 53 - **Pass Rate**: 100% - **Test Structure**: Organized into 8 test classes using Arrange-Act-Assert pattern - **Documentation**: Comprehensive docstrings for each test explaining purpose and verification ### Dependencies No new dependencies required. Uses existing testing infrastructure: - `pytest` for test framework - `unittest.mock` for mocking - Existing Dify models and services ## Screenshots Not applicable - this is a backend testing enhancement with no UI changes. ## Checklist - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [x] I've updated the documentation accordingly. - [x] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods ## Testing All tests pass successfully: ```bash uv run --project api pytest api/tests/unit_tests/tasks/test_document_processing_task.py -v # Result: 53 passed in 33.74s ``` The test suite follows TDD best practices with: - Clear test organization using factory pattern for test data - Proper mocking of external dependencies (DB, Redis, services) - Comprehensive edge case coverage - Self-documenting code with detailed docstrings Contribution by Gittensor, learn more at https://gittensor.io/
yindo added the pull-request label 2026-02-21 20:51:00 -05:00
yindo closed this issue 2026-02-21 20:51:00 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#32230
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?