langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #23574] fix: ensure vector database cleanup on dataset deletion regardless of document presence (affects all 33 vector databases) #30311

New Issue
Closed
opened 2026-02-21 20:47:15 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:47:15 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/23574

State: closed
Merged: Yes

Critical Bug Fix - Universal Vector Database Resource Leak

Fix https://github.com/langgenius/dify/issues/23577

Problem

All 33 supported vector databases were not properly cleaning up resources when deleting datasets that had no documents, causing systematic resource leakage
across the entire vector database ecosystem supported by Dify.

Scope of Impact

This bug affects every vector database implementation in Dify:

SQL-based Vector Stores (Table/Schema Cleanup):

  • ClickZetta - DROP TABLE schema.table_name
  • PGVector, TiDB Vector, Oracle, MySQL - DROP TABLE table_name
  • AnalyticDB, OceanBase, MatrixOne - Table deletion operations
  • Relyt, PGVecto-RS, OpenGauss, Vastbase - Schema cleanup

Cloud/Managed Vector Services (Collection Cleanup):

  • Milvus - drop_collection(collection_name)
  • Qdrant - Delete by filter with group_id
  • Chroma - delete_collection(collection_name)
  • Weaviate - delete_class(collection_name)
  • Elasticsearch/OpenSearch - indices.delete(index=collection_name)
  • Pinecone, Upstash, VikingDB - Index/collection deletion

Regional Cloud Providers:

  • Tencent Vector, Baidu Vector - drop_collection/drop_table
  • Huawei Cloud - Index deletion operations
  • And 15+ additional vector database implementations

Root Cause Analysis

The issue exists at the architectural level affecting all vector databases:

# ALL vector databases follow this pattern in IndexProcessor
def clean(self, dataset: Dataset, node_ids: Optional[list[str]], **kwargs):
    if documents is None or len(documents) == 0:
        logging.info("No documents found")  # ❌ SKIP CLEANUP
    else:
        # ✅ Only called when documents exist
        index_processor = IndexProcessorFactory(doc_form).init_index_processor()
        index_processor.clean(dataset, None, ...)  # This calls vector.delete()

Universal failure pattern:
1. Normal case: Dataset with documents → vector.delete() called → resources cleaned
2. Bug case: Empty dataset → cleanup skipped → resources leaked

Technical Impact by Database Type

| Database Type         | Resource Leaked     | Cleanup Method            | Impact                           |
|-----------------------|---------------------|---------------------------|----------------------------------|
| SQL Databases         | Tables/Schemas      | DROP TABLE                | Storage space, connection limits |
| Cloud Services        | Collections/Indices | API deletion calls        | Storage costs, quota limits      |
| Specialized Platforms | Custom resources    | Platform-specific cleanup | Service limits, billing          |

Solution

Move the vector database cleanup logic outside the document existence check to ensure universal resource cleanup:

# Fix: Always clean vector database resources regardless of document existence
# This ensures ALL 33 vector databases properly clean up resources
if doc_form is None:
    raise ValueError("Index type must be specified.")
index_processor = IndexProcessorFactory(doc_form).init_index_processor()
index_processor.clean(dataset, None, with_keywords=True, delete_child_chunks=True)

# Then handle document-specific cleanup
if documents is None or len(documents) == 0:
    logging.info("No documents found")
else:
    logging.info("Cleaning documents")
    # Document cleanup continues...

Universal Benefits

Before Fix (All Databases Affected):
- ❌ Resource leak: Empty datasets leave database resources behind
- ❌ Storage waste: Accumulated tables/collections/indices
- ❌ Cost impact: Unnecessary cloud storage/compute charges
- ❌ Scale issues: Resource exhaustion over time

After Fix (All Databases Benefit):
- ✅ Universal consistency: All 33 vector databases now properly clean resources
- ✅ No resource leaks: Tables, collections, and indices always removed
- ✅ Cost optimization: Eliminates unnecessary storage charges
- ✅ Backward compatible: Normal deletion paths unchanged
- ✅ Future-proof: New vector database integrations automatically benefit

Testing Verification

Multi-Database Testing Scenarios:

- SQL Databases (ClickZetta, PGVector) → Table cleanup verified
- Cloud Services (Milvus, Elasticsearch) → Collection deletion verified
- Edge Cases → Empty datasets now properly cleaned across all implementations
- Regression Testing → Normal deletion workflows unchanged

Test Coverage:

1. Create dataset + add documents → Delete dataset ✅
2. Create dataset → Delete all documents → Delete dataset ✅ FIXED
3. Create empty dataset → Delete dataset ✅ FIXED

Files Changed

- api/tasks/clean_dataset_task.py - Universal fix at IndexProcessor level

Impact Assessment

- Severity: High (affects all vector database implementations)
- Scope: Universal (all 33 supported vector databases)
- Risk: Low (backward compatible, single logical change)
- Benefits: Eliminates systematic resource leaks across entire vector ecosystem

---
🔍 Reviewers: This fix resolves a fundamental architectural issue affecting every vector database supported by Dify. Please verify the universal cleanup logic
maintains compatibility across all database implementations.

💡 Note: While discovered through ClickZetta usage, this bug has been silently affecting all vector database users, potentially causing significant resource
waste and scaling issues.
**Original Pull Request:** https://github.com/langgenius/dify/pull/23574 **State:** closed **Merged:** Yes --- ## Critical Bug Fix - Universal Vector Database Resource Leak Fix https://github.com/langgenius/dify/issues/23577 ### Problem **All 33 supported vector databases** were not properly cleaning up resources when deleting datasets that had no documents, causing systematic resource leakage across the entire vector database ecosystem supported by Dify. ### Scope of Impact This bug affects **every vector database implementation** in Dify: #### SQL-based Vector Stores (Table/Schema Cleanup): - **ClickZetta** - `DROP TABLE schema.table_name` - **PGVector, TiDB Vector, Oracle, MySQL** - `DROP TABLE table_name` - **AnalyticDB, OceanBase, MatrixOne** - Table deletion operations - **Relyt, PGVecto-RS, OpenGauss, Vastbase** - Schema cleanup #### Cloud/Managed Vector Services (Collection Cleanup): - **Milvus** - `drop_collection(collection_name)` - **Qdrant** - Delete by filter with `group_id` - **Chroma** - `delete_collection(collection_name)` - **Weaviate** - `delete_class(collection_name)` - **Elasticsearch/OpenSearch** - `indices.delete(index=collection_name)` - **Pinecone, Upstash, VikingDB** - Index/collection deletion #### Regional Cloud Providers: - **Tencent Vector, Baidu Vector** - `drop_collection/drop_table` - **Huawei Cloud** - Index deletion operations - **And 15+ additional vector database implementations** ### Root Cause Analysis The issue exists at the **architectural level** affecting all vector databases: ```python # ALL vector databases follow this pattern in IndexProcessor def clean(self, dataset: Dataset, node_ids: Optional[list[str]], **kwargs): if documents is None or len(documents) == 0: logging.info("No documents found") # ❌ SKIP CLEANUP else: # ✅ Only called when documents exist index_processor = IndexProcessorFactory(doc_form).init_index_processor() index_processor.clean(dataset, None, ...) # This calls vector.delete() Universal failure pattern: 1. Normal case: Dataset with documents → vector.delete() called → resources cleaned 2. Bug case: Empty dataset → cleanup skipped → resources leaked Technical Impact by Database Type | Database Type | Resource Leaked | Cleanup Method | Impact | |-----------------------|---------------------|---------------------------|----------------------------------| | SQL Databases | Tables/Schemas | DROP TABLE | Storage space, connection limits | | Cloud Services | Collections/Indices | API deletion calls | Storage costs, quota limits | | Specialized Platforms | Custom resources | Platform-specific cleanup | Service limits, billing | Solution Move the vector database cleanup logic outside the document existence check to ensure universal resource cleanup: # Fix: Always clean vector database resources regardless of document existence # This ensures ALL 33 vector databases properly clean up resources if doc_form is None: raise ValueError("Index type must be specified.") index_processor = IndexProcessorFactory(doc_form).init_index_processor() index_processor.clean(dataset, None, with_keywords=True, delete_child_chunks=True) # Then handle document-specific cleanup if documents is None or len(documents) == 0: logging.info("No documents found") else: logging.info("Cleaning documents") # Document cleanup continues... Universal Benefits Before Fix (All Databases Affected): - ❌ Resource leak: Empty datasets leave database resources behind - ❌ Storage waste: Accumulated tables/collections/indices - ❌ Cost impact: Unnecessary cloud storage/compute charges - ❌ Scale issues: Resource exhaustion over time After Fix (All Databases Benefit): - ✅ Universal consistency: All 33 vector databases now properly clean resources - ✅ No resource leaks: Tables, collections, and indices always removed - ✅ Cost optimization: Eliminates unnecessary storage charges - ✅ Backward compatible: Normal deletion paths unchanged - ✅ Future-proof: New vector database integrations automatically benefit Testing Verification Multi-Database Testing Scenarios: - SQL Databases (ClickZetta, PGVector) → Table cleanup verified - Cloud Services (Milvus, Elasticsearch) → Collection deletion verified - Edge Cases → Empty datasets now properly cleaned across all implementations - Regression Testing → Normal deletion workflows unchanged Test Coverage: 1. Create dataset + add documents → Delete dataset ✅ 2. Create dataset → Delete all documents → Delete dataset ✅ FIXED 3. Create empty dataset → Delete dataset ✅ FIXED Files Changed - api/tasks/clean_dataset_task.py - Universal fix at IndexProcessor level Impact Assessment - Severity: High (affects all vector database implementations) - Scope: Universal (all 33 supported vector databases) - Risk: Low (backward compatible, single logical change) - Benefits: Eliminates systematic resource leaks across entire vector ecosystem --- 🔍 Reviewers: This fix resolves a fundamental architectural issue affecting every vector database supported by Dify. Please verify the universal cleanup logic maintains compatibility across all database implementations. 💡 Note: While discovered through ClickZetta usage, this bug has been silently affecting all vector database users, potentially causing significant resource waste and scaling issues.
yindo added the pull-request label 2026-02-21 20:47:15 -05:00
yindo closed this issue 2026-02-21 20:47:15 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#30311
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?