langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-26 18:36:52 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Add Archive/Compressed File Support to RAG Document Extractor #16440

New Issue
Closed
opened 2026-02-21 19:26:07 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 19:26:07 -05:00
Owner
Copy Link
Copy Source

Originally created by @brianxiadong on GitHub (Aug 25, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Feature Request: Add Archive/Compressed File Support to RAG Document Extractor

Feature Description

Add comprehensive support for archive and compressed file extraction in Dify's RAG document processing system. This feature would enable users to upload compressed archives containing multiple documents and have them automatically extracted and processed for RAG applications, significantly improving workflow efficiency when dealing with document collections.

Use Case

Primary Use Cases:

  • Document Collections: Process ZIP files containing multiple PDFs, Word documents, or text files
  • Backup Archives: Extract and index archived document collections
  • Bulk Upload: Enable users to upload entire document libraries in compressed format
  • Legacy Data: Process archived data from legacy systems stored in various compressed formats
  • Distribution Packages: Handle documentation packages distributed as compressed archives
  • Email Attachments: Process compressed email attachments containing multiple documents

Example Scenario:
"I was trying to upload a ZIP file containing 50+ PDF documents from our company's policy manual for my internal knowledge base, and I expected the system to automatically extract all the PDFs and process them individually. However, the current system doesn't support compressed files, so I had to manually extract the ZIP file and upload each PDF separately, which was time-consuming and inefficient."

Detailed Feature Requirements

Supported Archive Formats

  • ZIP: Standard ZIP archives (.zip)
  • TAR: TAR archives (.tar, .tar.gz, .tar.bz2, .tgz, .tbz2)
  • 7-Zip: 7-Zip archives (.7z)
  • RAR: RAR archives (.rar) - if licensing permits
  • GZIP: Individual GZIP files (.gz)
  • BZIP2: Individual BZIP2 files (.bz2)

Core Functionality

  • Recursive Extraction: Support nested archives (archives within archives)
  • Selective Processing: Extract and process only supported document types
  • Path Preservation: Maintain original folder structure in metadata
  • Size Limits: Implement extraction size limits to prevent abuse
  • Security Checks: Prevent zip bomb attacks and path traversal vulnerabilities
  • Progress Tracking: Provide extraction progress feedback for large archives

Technical Specifications

  • File Size Limits: Configurable limits for both archive size and extracted content
  • Timeout Protection: Prevent indefinite extraction operations
  • Memory Management: Efficient handling of large archives without excessive memory usage
  • Error Handling: Graceful handling of corrupted archives and unsupported files
  • Temporary Storage: Secure temporary extraction with automatic cleanup
  • Concurrent Processing: Support for processing multiple files from archives simultaneously

Security Features

  • Path Traversal Protection: Prevent extraction outside designated directories
  • Zip Bomb Detection: Detect and prevent decompression bombs
  • File Type Validation: Validate extracted files before processing
  • Size Ratio Checks: Monitor compression ratios to detect malicious archives
  • Timeout Controls: Prevent long-running extraction operations

Integration Requirements

  • ExtractProcessor Integration: Seamlessly integrate with existing document processing pipeline
  • Metadata Enhancement: Add archive-specific metadata (original path, archive type, extraction info)
  • Document Chunking: Process extracted documents using existing chunking strategies
  • Testing Coverage: Comprehensive unit and integration tests
  • Error Reporting: Detailed error reporting for failed extractions

Expected Behavior

  1. Archive Detection: Automatically detect supported archive formats by file extension
  2. Safe Extraction: Extract archives to secure temporary locations
  3. Recursive Processing: Process extracted documents using existing extractors
  4. Metadata Preservation: Maintain original file paths and archive information
  5. Error Handling: Handle corrupted archives and unsupported files gracefully
  6. Cleanup: Automatic cleanup of temporary extracted files
  7. Progress Feedback: Provide user feedback during extraction and processing

Technical Implementation Details

Proposed Architecture

ArchiveExtractor
├── _detect_archive_type()
├── _extract_zip()
├── _extract_tar()
├── _extract_7z()
├── _extract_rar()
├── _validate_extraction_path()
├── _check_security_limits()
└── _process_extracted_files()

Dependencies

  • zipfile: Built-in Python library for ZIP support
  • tarfile: Built-in Python library for TAR support
  • py7zr: Third-party library for 7-Zip support
  • rarfile: Third-party library for RAR support (optional)

Configuration Options

  • Maximum archive size
  • Maximum extracted size
  • Extraction timeout
  • Supported archive types
  • Temporary directory location

Additional Context

  • This feature addresses a common user pain point when dealing with document collections
  • Archive support is essential for enterprise users who often work with compressed document packages
  • Security considerations are paramount due to potential attack vectors
  • The implementation should follow existing extractor patterns for consistency
  • Consider integration with existing file upload limits and storage quotas

Performance Considerations

  • Implement streaming extraction for large archives
  • Use efficient temporary storage management
  • Consider parallel processing of extracted files
  • Optimize memory usage for large archive processing

Priority

High Priority - This feature would significantly improve user experience and workflow efficiency, especially for enterprise users dealing with large document collections.

Related Issues

  • Related to overall document extraction improvements
  • Part of expanding supported file formats for RAG applications
  • Addresses bulk document upload requirements
  • Enhances enterprise workflow capabilities

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @brianxiadong on GitHub (Aug 25, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. # Feature Request: Add Archive/Compressed File Support to RAG Document Extractor ## Feature Description Add comprehensive support for archive and compressed file extraction in Dify's RAG document processing system. This feature would enable users to upload compressed archives containing multiple documents and have them automatically extracted and processed for RAG applications, significantly improving workflow efficiency when dealing with document collections. ## Use Case **Primary Use Cases:** - **Document Collections**: Process ZIP files containing multiple PDFs, Word documents, or text files - **Backup Archives**: Extract and index archived document collections - **Bulk Upload**: Enable users to upload entire document libraries in compressed format - **Legacy Data**: Process archived data from legacy systems stored in various compressed formats - **Distribution Packages**: Handle documentation packages distributed as compressed archives - **Email Attachments**: Process compressed email attachments containing multiple documents **Example Scenario:** "I was trying to upload a ZIP file containing 50+ PDF documents from our company's policy manual for my internal knowledge base, and I expected the system to automatically extract all the PDFs and process them individually. However, the current system doesn't support compressed files, so I had to manually extract the ZIP file and upload each PDF separately, which was time-consuming and inefficient." ## Detailed Feature Requirements ### Supported Archive Formats - **ZIP**: Standard ZIP archives (.zip) - **TAR**: TAR archives (.tar, .tar.gz, .tar.bz2, .tgz, .tbz2) - **7-Zip**: 7-Zip archives (.7z) - **RAR**: RAR archives (.rar) - if licensing permits - **GZIP**: Individual GZIP files (.gz) - **BZIP2**: Individual BZIP2 files (.bz2) ### Core Functionality - **Recursive Extraction**: Support nested archives (archives within archives) - **Selective Processing**: Extract and process only supported document types - **Path Preservation**: Maintain original folder structure in metadata - **Size Limits**: Implement extraction size limits to prevent abuse - **Security Checks**: Prevent zip bomb attacks and path traversal vulnerabilities - **Progress Tracking**: Provide extraction progress feedback for large archives ### Technical Specifications - **File Size Limits**: Configurable limits for both archive size and extracted content - **Timeout Protection**: Prevent indefinite extraction operations - **Memory Management**: Efficient handling of large archives without excessive memory usage - **Error Handling**: Graceful handling of corrupted archives and unsupported files - **Temporary Storage**: Secure temporary extraction with automatic cleanup - **Concurrent Processing**: Support for processing multiple files from archives simultaneously ### Security Features - **Path Traversal Protection**: Prevent extraction outside designated directories - **Zip Bomb Detection**: Detect and prevent decompression bombs - **File Type Validation**: Validate extracted files before processing - **Size Ratio Checks**: Monitor compression ratios to detect malicious archives - **Timeout Controls**: Prevent long-running extraction operations ### Integration Requirements - **ExtractProcessor Integration**: Seamlessly integrate with existing document processing pipeline - **Metadata Enhancement**: Add archive-specific metadata (original path, archive type, extraction info) - **Document Chunking**: Process extracted documents using existing chunking strategies - **Testing Coverage**: Comprehensive unit and integration tests - **Error Reporting**: Detailed error reporting for failed extractions ## Expected Behavior 1. **Archive Detection**: Automatically detect supported archive formats by file extension 2. **Safe Extraction**: Extract archives to secure temporary locations 3. **Recursive Processing**: Process extracted documents using existing extractors 4. **Metadata Preservation**: Maintain original file paths and archive information 5. **Error Handling**: Handle corrupted archives and unsupported files gracefully 6. **Cleanup**: Automatic cleanup of temporary extracted files 7. **Progress Feedback**: Provide user feedback during extraction and processing ## Technical Implementation Details ### Proposed Architecture ``` ArchiveExtractor ├── _detect_archive_type() ├── _extract_zip() ├── _extract_tar() ├── _extract_7z() ├── _extract_rar() ├── _validate_extraction_path() ├── _check_security_limits() └── _process_extracted_files() ``` ### Dependencies - **zipfile**: Built-in Python library for ZIP support - **tarfile**: Built-in Python library for TAR support - **py7zr**: Third-party library for 7-Zip support - **rarfile**: Third-party library for RAR support (optional) ### Configuration Options - Maximum archive size - Maximum extracted size - Extraction timeout - Supported archive types - Temporary directory location ## Additional Context - This feature addresses a common user pain point when dealing with document collections - Archive support is essential for enterprise users who often work with compressed document packages - Security considerations are paramount due to potential attack vectors - The implementation should follow existing extractor patterns for consistency - Consider integration with existing file upload limits and storage quotas ## Performance Considerations - Implement streaming extraction for large archives - Use efficient temporary storage management - Consider parallel processing of extracted files - Optimize memory usage for large archive processing ## Priority High Priority - This feature would significantly improve user experience and workflow efficiency, especially for enterprise users dealing with large document collections. ## Related Issues - Related to overall document extraction improvements - Part of expanding supported file formats for RAG applications - Addresses bulk document upload requirements - Enhances enterprise workflow capabilities ### 2. Additional context or comments _No response_ ### 3. Can you help us with this feature? - [x] I am interested in contributing to this feature.
yindo added the 👻 feat:rag label 2026-02-21 19:26:07 -05:00
yindo closed this issue 2026-02-21 19:26:07 -05:00
yindo commented 2026-02-21 19:26:07 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 26, 2025):

Hi, @brianxiadong. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You requested support for extracting and processing archive/compressed files (ZIP, TAR, 7-Zip, RAR) in Dify's RAG document extractor.
  • The goal is to enable automatic extraction and indexing of documents within archives to improve efficiency.
  • This would facilitate bulk uploads and better handling of legacy or distribution package data.
  • There has been no further activity or comments on this issue since it was opened.
  • The issue remains unresolved at this time.

Next Steps:

  • Please let me know if this feature request is still relevant to the latest version of Dify by commenting on this issue.
  • If I do not hear back within 15 days, the issue will be automatically closed.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Sep 26, 2025): Hi, @brianxiadong. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You requested support for extracting and processing archive/compressed files (ZIP, TAR, 7-Zip, RAR) in Dify's RAG document extractor. - The goal is to enable automatic extraction and indexing of documents within archives to improve efficiency. - This would facilitate bulk uploads and better handling of legacy or distribution package data. - There has been no further activity or comments on this issue since it was opened. - The issue remains unresolved at this time. **Next Steps:** - Please let me know if this feature request is still relevant to the latest version of Dify by commenting on this issue. - If I do not hear back within 15 days, the issue will be automatically closed. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 👻 feat:rag
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#16440
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?