langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #28828] feat: complete test script of indexing runner #32196

New Issue
Closed
opened 2026-02-21 20:50:56 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:50:56 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/28828

State: closed
Merged: Yes

Add Comprehensive Unit Tests for IndexingRunner

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

This PR adds comprehensive unit tests for the IndexingRunner class, which orchestrates the document indexing pipeline in the Dify RAG system. The test suite provides complete coverage of all major indexing functionality.

Fixes

Closes https://github.com/langgenius/dify/issues/28827

What's Added

  • New test file: api/tests/unit_tests/core/rag/indexing/test_indexing_runner.py
  • 35 passing tests covering all critical indexing operations
  • 4 helper functions to reduce boilerplate and improve test maintainability
  • Comprehensive documentation with detailed docstrings and inline comments

Test Coverage

The test suite covers:

  1. Document Parsing Pipeline (Extract Phase) - 5 tests

    • Upload file extraction
    • Notion import extraction
    • Website crawl extraction
    • Error handling for missing/invalid sources

  2. Chunk Creation Logic (Transform Phase) - 3 tests

    • High-quality indexing with embeddings
    • Economy indexing without embeddings
    • Custom segmentation rules

  3. Embedding Generation Orchestration - 3 tests

    • Vector index creation with parallel processing
    • Keyword index creation with threading
    • Parent-child index structure handling

  4. Vector Storage Integration (Load Phase) - 3 tests

    • Paragraph index segment storage
    • Parent-child index with child chunks
    • Word count calculation and updates

  5. Retry Logic & Error Handling - 10 tests

    • Complete end-to-end indexing flow
    • DocumentIsPausedError handling
    • ProviderTokenNotInitError handling
    • ObjectDeletedError handling
    • Document status management

  6. Additional Coverage - 11 tests

    • Document cleaning (automatic & custom modes)
    • Text splitter configuration
    • Batch upload limit enforcement
    • Token counting in parallel processing

Key Features

  • ✅ All tests use mocking to avoid external dependencies
  • ✅ Tests follow the Arrange-Act-Assert (AAA) pattern
  • ✅ Helper functions reduce code duplication
  • ✅ Comprehensive docstrings explain test purpose and assertions
  • ✅ Inline comments provide context for complex mocking scenarios
  • ✅ All tests pass locally and comply with project linting standards

Screenshots

Not applicable - this PR adds backend unit tests only.

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Testing

All tests pass successfully:

pytest api/tests/unit_tests/core/rag/indexing/test_indexing_runner.py -v
# Result: 35 passed, 1 warning in ~33s

Linting checks pass:

ruff check api/tests/unit_tests/core/rag/indexing/test_indexing_runner.py
# Result: All checks passed!

Contribution by Gittensor, learn more at https://gittensor.io/

**Original Pull Request:** https://github.com/langgenius/dify/pull/28828 **State:** closed **Merged:** Yes --- # Add Comprehensive Unit Tests for IndexingRunner > [!IMPORTANT] > > 1. Make sure you have read our [contribution guidelines](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) > 1. Ensure there is an associated issue and you have been assigned to it > 1. Use the correct syntax to link this PR: `Fixes #<issue number>`. ## Summary This PR adds comprehensive unit tests for the `IndexingRunner` class, which orchestrates the document indexing pipeline in the Dify RAG system. The test suite provides complete coverage of all major indexing functionality. ### Fixes Closes https://github.com/langgenius/dify/issues/28827 ### What's Added - **New test file**: `api/tests/unit_tests/core/rag/indexing/test_indexing_runner.py` - **35 passing tests** covering all critical indexing operations - **4 helper functions** to reduce boilerplate and improve test maintainability - **Comprehensive documentation** with detailed docstrings and inline comments ### Test Coverage The test suite covers: 1. **Document Parsing Pipeline (Extract Phase)** - 5 tests - Upload file extraction - Notion import extraction - Website crawl extraction - Error handling for missing/invalid sources 2. **Chunk Creation Logic (Transform Phase)** - 3 tests - High-quality indexing with embeddings - Economy indexing without embeddings - Custom segmentation rules 3. **Embedding Generation Orchestration** - 3 tests - Vector index creation with parallel processing - Keyword index creation with threading - Parent-child index structure handling 4. **Vector Storage Integration (Load Phase)** - 3 tests - Paragraph index segment storage - Parent-child index with child chunks - Word count calculation and updates 5. **Retry Logic & Error Handling** - 10 tests - Complete end-to-end indexing flow - DocumentIsPausedError handling - ProviderTokenNotInitError handling - ObjectDeletedError handling - Document status management 6. **Additional Coverage** - 11 tests - Document cleaning (automatic & custom modes) - Text splitter configuration - Batch upload limit enforcement - Token counting in parallel processing ### Key Features - ✅ All tests use mocking to avoid external dependencies - ✅ Tests follow the Arrange-Act-Assert (AAA) pattern - ✅ Helper functions reduce code duplication - ✅ Comprehensive docstrings explain test purpose and assertions - ✅ Inline comments provide context for complex mocking scenarios - ✅ All tests pass locally and comply with project linting standards ## Screenshots Not applicable - this PR adds backend unit tests only. ## Checklist - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [x] I've updated the documentation accordingly. - [x] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods ## Testing All tests pass successfully: ```bash pytest api/tests/unit_tests/core/rag/indexing/test_indexing_runner.py -v # Result: 35 passed, 1 warning in ~33s ``` Linting checks pass: ```bash ruff check api/tests/unit_tests/core/rag/indexing/test_indexing_runner.py # Result: All checks passed! ``` Contribution by Gittensor, learn more at https://gittensor.io/
yindo added the pull-request label 2026-02-21 20:50:56 -05:00
yindo closed this issue 2026-02-21 20:50:56 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#32196
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?