langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 12:36:14 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #28830] test: add comprehensive unit tests for DocumentIndexingTaskProxy #32200

New Issue
Closed
opened 2026-02-21 20:50:56 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:50:56 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/28830

State: closed
Merged: Yes

Summary

This PR adds comprehensive unit tests for the DocumentIndexingTaskProxy service, which handles critical document indexing task scheduling, queuing, and tenant isolation logic. The test suite ensures proper task routing based on billing configuration, tenant isolation queue management, batch operations, and error handling.

Fix: #28829

Test Coverage

1. Initialization and Configuration

  • ✅ Proxy initialization with tenant_id, dataset_id, and document_ids
  • ✅ TenantIsolatedTaskQueue initialization with correct parameters
  • ✅ Features property caching via @cached_property
  • ✅ Edge cases: empty document_ids, single document_id, large batches
  • ✅ Unicode and special character handling in IDs

2. Task Queue Routing

_send_to_direct_queue

  • ✅ Direct queue routing (bypasses tenant isolation)
  • ✅ Task function delay() call with correct parameters
  • ✅ Single document, multiple documents, empty documents
  • ✅ Unicode document IDs handling

_send_to_tenant_queue

  • ✅ Tenant queue routing with existing task key (pushes to waiting queue)
  • ✅ Tenant queue routing without task key (sets flag and executes immediately)
  • ✅ DocumentTask serialization and deserialization
  • ✅ Task key state transitions
  • ✅ Concurrent task queuing

3. Queue Type Selection

  • ✅ _send_to_default_tenant_queue - normal priority with tenant isolation
  • ✅ _send_to_priority_tenant_queue - high priority with tenant isolation
  • ✅ _send_to_priority_direct_queue - high priority without tenant isolation

4. Dispatch Logic (_dispatch, delay)

  • ✅ Billing enabled + SANDBOX plan → default tenant queue
  • ✅ Billing enabled + TEAM plan → priority tenant queue
  • ✅ Billing enabled + PROFESSIONAL plan → priority tenant queue
  • ✅ Billing disabled (self-hosted/enterprise) → priority direct queue
  • ✅ Edge cases: None plan, empty plan string
  • ✅ All CloudPlan enum values handling

5. Tenant Isolation and Queue Management

  • ✅ Task key existence checking (get_task_key)
  • ✅ Task waiting time setting (set_task_waiting_time)
  • ✅ Task pushing to queue (push_tasks)
  • ✅ Queue state transitions (idle → active → idle)
  • ✅ Multiple concurrent task handling
  • ✅ Task key lifecycle management

6. Batch Operations

  • ✅ Single document indexing
  • ✅ Multiple document batch indexing
  • ✅ Large batch handling (100+ documents)
  • ✅ Very large batch handling (1000+ documents)
  • ✅ Empty batch handling (edge case)

7. Error Handling and Retry Logic

  • ✅ Task function delay() failure handling
  • ✅ Queue operation failures (push_tasks, set_task_waiting_time)
  • ✅ Feature service failures
  • ✅ Invalid task function handling
  • ✅ Missing billing subscription handling

8. Integration Points

  • ✅ FeatureService integration (billing features, subscription plans)
  • ✅ TenantIsolatedTaskQueue integration (Redis operations)
  • ✅ Celery task integration (normal_document_indexing_task, priority_document_indexing_task)
  • ✅ DocumentTask entity serialization

9. Additional Test Scenarios

  • ✅ Full integration flows for all billing plans
  • ✅ Task queuing when another task is running
  • ✅ Performance testing with caching verification
  • ✅ Data integrity and immutability verification
  • ✅ Unicode and special character support

Test Structure

  • Test Data Factory: Centralized mock object creation for consistency
  • Comprehensive Documentation: Extensive comments explaining test scenarios, architecture, and queue mechanics
  • Error Path Coverage: All error conditions and edge cases are tested
  • Mock Strategy: Proper isolation using unittest.mock for dependencies (FeatureService, TenantIsolatedTaskQueue, Celery tasks)
  • Validation Testing: Ensures all routing rules are correctly enforced

Files Changed

  • api/tests/unit_tests/services/document_indexing_task_proxy.py (1,289+ lines)

Testing Approach

  • Follows existing test patterns from other service test files
  • Uses Arrange-Act-Assert structure
  • Comprehensive mocking of dependencies (FeatureService, TenantIsolatedTaskQueue, Celery tasks)
  • Tests both success and error paths
  • Validates queue state transitions and task routing decisions
  • Ensures proper tenant isolation behavior

Integration Points

These tests ensure proper integration with:

  • FeatureService: Billing configuration and subscription plan retrieval
  • TenantIsolatedTaskQueue: Redis-based queue operations for tenant isolation
  • Celery Tasks: Task scheduling via normal_document_indexing_task and priority_document_indexing_task
  • DocumentTask Entity: Task serialization for queue storage

Key Features Tested

  1. Task Routing Logic: Ensures tasks are routed to the correct queue based on billing configuration
  2. Tenant Isolation: Verifies that only one indexing task runs per tenant at a time
  3. Queue Management: Tests task queuing, state transitions, and concurrent request handling
  4. Batch Processing: Validates handling of single and multiple document indexing requests
  5. Error Resilience: Comprehensive error handling for all failure scenarios

Related Issues

This test suite addresses the need for comprehensive coverage of document indexing task proxy functionality, ensuring safe refactoring and preventing regressions in critical task scheduling logic that affects document indexing performance and tenant resource management.

Contribution by Gittensor, learn more at https://gittensor.io/

**Original Pull Request:** https://github.com/langgenius/dify/pull/28830 **State:** closed **Merged:** Yes --- ## Summary This PR adds comprehensive unit tests for the `DocumentIndexingTaskProxy` service, which handles critical document indexing task scheduling, queuing, and tenant isolation logic. The test suite ensures proper task routing based on billing configuration, tenant isolation queue management, batch operations, and error handling. Fix: #28829 ## Test Coverage ### 1. Initialization and Configuration - ✅ Proxy initialization with tenant_id, dataset_id, and document_ids - ✅ TenantIsolatedTaskQueue initialization with correct parameters - ✅ Features property caching via `@cached_property` - ✅ Edge cases: empty document_ids, single document_id, large batches - ✅ Unicode and special character handling in IDs ### 2. Task Queue Routing #### `_send_to_direct_queue` - ✅ Direct queue routing (bypasses tenant isolation) - ✅ Task function delay() call with correct parameters - ✅ Single document, multiple documents, empty documents - ✅ Unicode document IDs handling #### `_send_to_tenant_queue` - ✅ Tenant queue routing with existing task key (pushes to waiting queue) - ✅ Tenant queue routing without task key (sets flag and executes immediately) - ✅ DocumentTask serialization and deserialization - ✅ Task key state transitions - ✅ Concurrent task queuing ### 3. Queue Type Selection - ✅ `_send_to_default_tenant_queue` - normal priority with tenant isolation - ✅ `_send_to_priority_tenant_queue` - high priority with tenant isolation - ✅ `_send_to_priority_direct_queue` - high priority without tenant isolation ### 4. Dispatch Logic (`_dispatch`, `delay`) - ✅ Billing enabled + SANDBOX plan → default tenant queue - ✅ Billing enabled + TEAM plan → priority tenant queue - ✅ Billing enabled + PROFESSIONAL plan → priority tenant queue - ✅ Billing disabled (self-hosted/enterprise) → priority direct queue - ✅ Edge cases: None plan, empty plan string - ✅ All CloudPlan enum values handling ### 5. Tenant Isolation and Queue Management - ✅ Task key existence checking (`get_task_key`) - ✅ Task waiting time setting (`set_task_waiting_time`) - ✅ Task pushing to queue (`push_tasks`) - ✅ Queue state transitions (idle → active → idle) - ✅ Multiple concurrent task handling - ✅ Task key lifecycle management ### 6. Batch Operations - ✅ Single document indexing - ✅ Multiple document batch indexing - ✅ Large batch handling (100+ documents) - ✅ Very large batch handling (1000+ documents) - ✅ Empty batch handling (edge case) ### 7. Error Handling and Retry Logic - ✅ Task function delay() failure handling - ✅ Queue operation failures (push_tasks, set_task_waiting_time) - ✅ Feature service failures - ✅ Invalid task function handling - ✅ Missing billing subscription handling ### 8. Integration Points - ✅ FeatureService integration (billing features, subscription plans) - ✅ TenantIsolatedTaskQueue integration (Redis operations) - ✅ Celery task integration (normal_document_indexing_task, priority_document_indexing_task) - ✅ DocumentTask entity serialization ### 9. Additional Test Scenarios - ✅ Full integration flows for all billing plans - ✅ Task queuing when another task is running - ✅ Performance testing with caching verification - ✅ Data integrity and immutability verification - ✅ Unicode and special character support ## Test Structure - **Test Data Factory**: Centralized mock object creation for consistency - **Comprehensive Documentation**: Extensive comments explaining test scenarios, architecture, and queue mechanics - **Error Path Coverage**: All error conditions and edge cases are tested - **Mock Strategy**: Proper isolation using `unittest.mock` for dependencies (FeatureService, TenantIsolatedTaskQueue, Celery tasks) - **Validation Testing**: Ensures all routing rules are correctly enforced ## Files Changed - `api/tests/unit_tests/services/document_indexing_task_proxy.py` (1,289+ lines) ## Testing Approach - Follows existing test patterns from other service test files - Uses Arrange-Act-Assert structure - Comprehensive mocking of dependencies (FeatureService, TenantIsolatedTaskQueue, Celery tasks) - Tests both success and error paths - Validates queue state transitions and task routing decisions - Ensures proper tenant isolation behavior ## Integration Points These tests ensure proper integration with: - **FeatureService**: Billing configuration and subscription plan retrieval - **TenantIsolatedTaskQueue**: Redis-based queue operations for tenant isolation - **Celery Tasks**: Task scheduling via `normal_document_indexing_task` and `priority_document_indexing_task` - **DocumentTask Entity**: Task serialization for queue storage ## Key Features Tested 1. **Task Routing Logic**: Ensures tasks are routed to the correct queue based on billing configuration 2. **Tenant Isolation**: Verifies that only one indexing task runs per tenant at a time 3. **Queue Management**: Tests task queuing, state transitions, and concurrent request handling 4. **Batch Processing**: Validates handling of single and multiple document indexing requests 5. **Error Resilience**: Comprehensive error handling for all failure scenarios ## Related Issues This test suite addresses the need for comprehensive coverage of document indexing task proxy functionality, ensuring safe refactoring and preventing regressions in critical task scheduling logic that affects document indexing performance and tenant resource management. Contribution by Gittensor, learn more at https://gittensor.io/
yindo added the pull-request label 2026-02-21 20:50:56 -05:00
yindo closed this issue 2026-02-21 20:50:57 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#32200
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?