langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #24528] feat: Add compressed file support to RAG document extraction system #30676

New Issue
Closed
opened 2026-02-21 20:48:00 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:48:00 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/24528

State: closed
Merged: No

fix #24531

Overview

This PR introduces comprehensive support for extracting documents from compressed archive files in the Dify RAG system. Users can now upload and process various archive formats including ZIP, TAR, 7Z, and RAR files, with automatic extraction and processing of contained documents.

Key Features

🗜️ Multi-Format Archive Support

  • ZIP files (.zip)
  • TAR archives (.tar, .tar.gz, .tar.bz2, .tgz, .tbz2)
  • 7-Zip archives (.7z)
  • RAR archives (.rar)

🔒 Security & Safety

  • File size limits : Individual files capped at 100MB, total extraction size at 1GB
  • Path traversal protection : Prevents malicious archives from writing outside extraction directory
  • Timeout protection : 5-minute extraction timeout to prevent resource exhaustion
  • Safe path validation : Blocks attempts to extract files with dangerous paths

🔄 Intelligent Processing

  • Recursive extraction : Automatically processes all supported file types within archives
  • Metadata enhancement : Preserves original archive information and file paths
  • Error handling : Graceful handling of corrupted files, unsupported formats, and extraction failures
  • Integration : Seamlessly works with existing ExtractProcessor pipeline

Files Changed

New Files

  • api/core/rag/extractor/archive_extractor.py - Core archive extraction implementation
  • api/tests/unit_tests/core/rag/extractor/test_archive_extractor.py - Comprehensive test suite

Modified Files

  • api/core/rag/extractor/extract_processor.py - Added archive format detection and routing

Technical Implementation

ArchiveExtractor Class

class ArchiveExtractor(BaseExtractor):
    """Extract documents from compressed 
    archive files.
    
    Supports: .zip, .tar, .tar.gz, .tar.
    bz2, .7z, .rar
    """

Key Methods

  • _detect_archive_format() - Automatic format detection based on file extensions
  • _extract_zip() , _extract_tar() , _extract_7z() , _extract_rar() - Format-specific extraction
  • _is_safe_path() - Security validation for extracted file paths
  • _process_extracted_file() - Recursive processing using existing extractors

Enhanced Metadata

Each extracted document includes:

  • archive_source : Original archive file path
  • archive_path : File's relative path within the archive
  • archive_format : Type of archive (zip, tar, etc.)
  • extracted_file_size : Size of the individual file
  • original_source : Maintains original file reference

Testing Coverage

Comprehensive test suite covering:

  • ✅ Archive format detection (ZIP, TAR, unsupported formats)
  • ✅ Security path validation
  • ✅ File processing for different content types
  • ✅ Successful extraction workflows
  • ✅ Error handling for corrupted files
  • ✅ File size limit enforcement
  • ✅ Integration with ExtractProcessor

Benefits

    Enhanced User Experience : Users can now upload compressed files directly without manual extraction
    Batch Processing : Process multiple documents in a single archive upload
    Security : Robust protection against malicious archives and resource exhaustion
    Maintainability : Clean integration with existing codebase and patterns
    Extensibility : Easy to add support for additional archive formats

Usage Example

# Archive files are automatically detected 
and processed
extractor = ArchiveExtractor
(file_path="documents.zip")
documents = extractor.extract()

# Each document contains enhanced metadata
for doc in documents:
    print(f"From archive: {doc.metadata
    ['archive_source']}")
    print(f"Original path: {doc.metadata
    ['archive_path']}")
    print(f"Content: {doc.page_content}")

Dependencies

Leverages existing Python libraries:

  • zipfile (built-in)
  • tarfile (built-in)
  • py7zr (for 7Z support)
  • rarfile (for RAR support)
    Total Changes : 585 lines added across 2 new files and 1 modified file

Testing : All tests pass with comprehensive coverage of archive extraction scenarios

**Original Pull Request:** https://github.com/langgenius/dify/pull/24528 **State:** closed **Merged:** No --- fix #24531 ## Overview This PR introduces comprehensive support for extracting documents from compressed archive files in the Dify RAG system. Users can now upload and process various archive formats including ZIP, TAR, 7Z, and RAR files, with automatic extraction and processing of contained documents. ## Key Features ### 🗜️ Multi-Format Archive Support - ZIP files (.zip) - TAR archives (.tar, .tar.gz, .tar.bz2, .tgz, .tbz2) - 7-Zip archives (.7z) - RAR archives (.rar) ### 🔒 Security & Safety - File size limits : Individual files capped at 100MB, total extraction size at 1GB - Path traversal protection : Prevents malicious archives from writing outside extraction directory - Timeout protection : 5-minute extraction timeout to prevent resource exhaustion - Safe path validation : Blocks attempts to extract files with dangerous paths ### 🔄 Intelligent Processing - Recursive extraction : Automatically processes all supported file types within archives - Metadata enhancement : Preserves original archive information and file paths - Error handling : Graceful handling of corrupted files, unsupported formats, and extraction failures - Integration : Seamlessly works with existing ExtractProcessor pipeline ## Files Changed ### New Files - api/core/rag/extractor/archive_extractor.py - Core archive extraction implementation - api/tests/unit_tests/core/rag/extractor/test_archive_extractor.py - Comprehensive test suite ### Modified Files - api/core/rag/extractor/extract_processor.py - Added archive format detection and routing ## Technical Implementation ### ArchiveExtractor Class ``` class ArchiveExtractor(BaseExtractor):     """Extract documents from compressed      archive files.          Supports: .zip, .tar, .tar.gz, .tar.     bz2, .7z, .rar     """ ``` ### Key Methods - _detect_archive_format() - Automatic format detection based on file extensions - _extract_zip() , _extract_tar() , _extract_7z() , _extract_rar() - Format-specific extraction - _is_safe_path() - Security validation for extracted file paths - _process_extracted_file() - Recursive processing using existing extractors ### Enhanced Metadata Each extracted document includes: - archive_source : Original archive file path - archive_path : File's relative path within the archive - archive_format : Type of archive (zip, tar, etc.) - extracted_file_size : Size of the individual file - original_source : Maintains original file reference ## Testing Coverage Comprehensive test suite covering: - ✅ Archive format detection (ZIP, TAR, unsupported formats) - ✅ Security path validation - ✅ File processing for different content types - ✅ Successful extraction workflows - ✅ Error handling for corrupted files - ✅ File size limit enforcement - ✅ Integration with ExtractProcessor ## Benefits 1. 1. Enhanced User Experience : Users can now upload compressed files directly without manual extraction 2. 2. Batch Processing : Process multiple documents in a single archive upload 3. 3. Security : Robust protection against malicious archives and resource exhaustion 4. 4. Maintainability : Clean integration with existing codebase and patterns 5. 5. Extensibility : Easy to add support for additional archive formats ## Usage Example ``` # Archive files are automatically detected  and processed extractor = ArchiveExtractor (file_path="documents.zip") documents = extractor.extract() # Each document contains enhanced metadata for doc in documents:     print(f"From archive: {doc.metadata     ['archive_source']}")     print(f"Original path: {doc.metadata     ['archive_path']}")     print(f"Content: {doc.page_content}") ``` ## Dependencies Leverages existing Python libraries: - zipfile (built-in) - tarfile (built-in) - py7zr (for 7Z support) - rarfile (for RAR support) Total Changes : 585 lines added across 2 new files and 1 modified file Testing : All tests pass with comprehensive coverage of archive extraction scenarios
yindo added the pull-request label 2026-02-21 20:48:00 -05:00
yindo closed this issue 2026-02-21 20:48:00 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#30676
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?