langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #28748] Add comprehensive unit tests for MetadataService (dataset metadata CRUD operations and filtering) #32165

New Issue
Closed
opened 2026-02-21 20:50:52 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:50:52 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/28748

State: closed
Merged: Yes

Summary

Adds comprehensive unit test coverage for MetadataService to ensure reliability and correctness of dataset metadata CRUD operations and filtering functionality.

Fix: #28744

Test Coverage

Metadata CRUD Operations

  • ✅ create_metadata - Create metadata fields

    • Successful creation with valid data
    • Name length validation (max 255 characters)
    • Duplicate name detection within dataset
    • Built-in field name conflict detection
    • Database transaction handling

  • ✅ update_metadata_name - Update metadata field names

    • Successful name update
    • Name length validation
    • Duplicate name detection
    • Built-in field conflict detection
    • Lock management (acquire and release)
    • Related document metadata updates
    • Not found error handling

  • ✅ delete_metadata - Delete metadata fields

    • Successful deletion
    • Document metadata cleanup
    • Lock management
    • Not found error handling
    • Transaction rollback on errors

Built-in Field Management

  • ✅ get_built_in_fields - Retrieve built-in field definitions
    • Successful retrieval of all built-in fields
    • Correct field structure (name, type)
    • Validation of specific built-in fields (document_name, uploader, etc.)

Lock Management

  • ✅ knowledge_base_metadata_lock_check - Concurrent operation prevention
    • Successful lock acquisition for dataset operations
    • Successful lock acquisition for document operations
    • Lock conflict detection and error handling
    • Proper lock expiration (3600 seconds)
    • Lock cleanup on errors

Metadata Retrieval

  • ✅ get_dataset_metadatas - Retrieve dataset metadata fields
    • Successful retrieval with metadata fields
    • Built-in field filtering (excludes built-in fields from list)
    • Usage count calculation for each metadata field
    • Built-in field enabled status

Test Structure

  • Test Data Factory: MetadataTestDataFactory provides reusable mock objects for metadata, bindings, datasets, documents, and entities
  • Comprehensive Coverage: Each method tested with success scenarios, error conditions, and edge cases
  • Proper Mocking: All dependencies properly mocked (db.session, redis_client, current_account_with_tenant, DocumentService, BuiltInField)
  • Follows Patterns: Test structure follows existing patterns from other service test files
  • Detailed Documentation: Extensive docstrings and comments (1086+ lines) for clarity and maintainability

Implementation Details

Metadata CRUD Flow

  1. Create: Validates name length, checks for duplicates and built-in conflicts, creates and commits to database
  2. Update: Validates new name, acquires lock, updates metadata and related documents, releases lock
  3. Delete: Acquires lock, deletes metadata and cleans up document metadata, releases lock

Lock Management

  • Redis-based locking prevents concurrent metadata operations
  • Separate locks for dataset-level and document-level operations
  • 3600-second expiration time
  • Automatic cleanup in finally blocks

Built-in Fields

  • System-defined metadata fields automatically available when enabled
  • Includes: document_name, uploader, upload_date, last_update_date, source
  • Can be enabled/disabled per dataset
  • Automatically added to document metadata when enabled

Files Changed

  • api/tests/unit_tests/services/dataset_metadata.py - New test file (1086 lines)

Testing

  • All tests follow Arrange-Act-Assert pattern
  • Proper use of pytest fixtures for test setup
  • Mock objects properly configured to simulate real behavior
  • Error conditions and edge cases thoroughly tested
  • Lock management and database transaction handling verified

Related

Part of ongoing effort to improve test coverage for dataset service methods.

Contribution by Gittensor, learn more at https://gittensor.io/

**Original Pull Request:** https://github.com/langgenius/dify/pull/28748 **State:** closed **Merged:** Yes --- ## Summary Adds comprehensive unit test coverage for `MetadataService` to ensure reliability and correctness of dataset metadata CRUD operations and filtering functionality. Fix: #28744 ## Test Coverage ### Metadata CRUD Operations - ✅ **create_metadata** - Create metadata fields - Successful creation with valid data - Name length validation (max 255 characters) - Duplicate name detection within dataset - Built-in field name conflict detection - Database transaction handling - ✅ **update_metadata_name** - Update metadata field names - Successful name update - Name length validation - Duplicate name detection - Built-in field conflict detection - Lock management (acquire and release) - Related document metadata updates - Not found error handling - ✅ **delete_metadata** - Delete metadata fields - Successful deletion - Document metadata cleanup - Lock management - Not found error handling - Transaction rollback on errors ### Built-in Field Management - ✅ **get_built_in_fields** - Retrieve built-in field definitions - Successful retrieval of all built-in fields - Correct field structure (name, type) - Validation of specific built-in fields (document_name, uploader, etc.) ### Lock Management - ✅ **knowledge_base_metadata_lock_check** - Concurrent operation prevention - Successful lock acquisition for dataset operations - Successful lock acquisition for document operations - Lock conflict detection and error handling - Proper lock expiration (3600 seconds) - Lock cleanup on errors ### Metadata Retrieval - ✅ **get_dataset_metadatas** - Retrieve dataset metadata fields - Successful retrieval with metadata fields - Built-in field filtering (excludes built-in fields from list) - Usage count calculation for each metadata field - Built-in field enabled status ## Test Structure - **Test Data Factory**: `MetadataTestDataFactory` provides reusable mock objects for metadata, bindings, datasets, documents, and entities - **Comprehensive Coverage**: Each method tested with success scenarios, error conditions, and edge cases - **Proper Mocking**: All dependencies properly mocked (`db.session`, `redis_client`, `current_account_with_tenant`, `DocumentService`, `BuiltInField`) - **Follows Patterns**: Test structure follows existing patterns from other service test files - **Detailed Documentation**: Extensive docstrings and comments (1086+ lines) for clarity and maintainability ## Implementation Details ### Metadata CRUD Flow 1. **Create**: Validates name length, checks for duplicates and built-in conflicts, creates and commits to database 2. **Update**: Validates new name, acquires lock, updates metadata and related documents, releases lock 3. **Delete**: Acquires lock, deletes metadata and cleans up document metadata, releases lock ### Lock Management - Redis-based locking prevents concurrent metadata operations - Separate locks for dataset-level and document-level operations - 3600-second expiration time - Automatic cleanup in finally blocks ### Built-in Fields - System-defined metadata fields automatically available when enabled - Includes: document_name, uploader, upload_date, last_update_date, source - Can be enabled/disabled per dataset - Automatically added to document metadata when enabled ## Files Changed - `api/tests/unit_tests/services/dataset_metadata.py` - New test file (1086 lines) ## Testing - All tests follow Arrange-Act-Assert pattern - Proper use of pytest fixtures for test setup - Mock objects properly configured to simulate real behavior - Error conditions and edge cases thoroughly tested - Lock management and database transaction handling verified ## Related Part of ongoing effort to improve test coverage for dataset service methods. Contribution by Gittensor, learn more at https://gittensor.io/
yindo added the pull-request label 2026-02-21 20:50:52 -05:00
yindo closed this issue 2026-02-21 20:50:52 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#32165
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?