Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4052] Storage: Cloud Storage Integration (S3/GCS/Azure) #2580

New Issue
Closed
opened 2026-02-22 18:30:20 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:30:20 -05:00
Owner
Copy Link
Copy Source

Originally created by @jezweb on GitHub (Jun 26, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4052

Overview

Add support for cloud storage providers to enable better scalability, reliability, and deployment flexibility.

Related to upstream discussion: https://github.com/Mintplex-Labs/anything-llm/issues/2797

Current State

  • Local file system storage only
  • Limited by disk space
  • No redundancy or backups
  • Difficult to scale horizontally
  • No CDN support

Proposed Solution

1. Storage Abstraction Layer

Interface Design

interface StorageProvider {
  upload(file: Buffer, path: string): Promise<string>
  download(path: string): Promise<Buffer>
  delete(path: string): Promise<void>
  exists(path: string): Promise<boolean>
  getUrl(path: string): Promise<string>
  listFiles(prefix: string): Promise<string[]>
}

Supported Providers

  • Local filesystem (current)
  • Amazon S3
  • Google Cloud Storage
  • Azure Blob Storage
  • MinIO (S3-compatible)

2. Implementation Features

Core Features

  • Streaming uploads/downloads
  • Multipart uploads for large files
  • Signed URLs for direct access
  • Automatic retry logic
  • Progress tracking

Advanced Features

  • File deduplication via hashing
  • Automatic compression
  • Lifecycle policies
  • Cross-region replication
  • CDN integration

3. Migration Strategy

Backward Compatibility

  • Keep local storage as default
  • Gradual migration tools
  • Dual-write during transition
  • Verification and rollback

Data Organization

/workspaces/{workspace-id}/
  /documents/{doc-id}/
    original/
    processed/
    embeddings/
  /uploads/
  /exports/
/system/
  /models/
  /cache/
  /temp/

4. Configuration

# Storage Provider
STORAGE_PROVIDER=s3 # local < /dev/null | s3|gcs|azure

# S3 Configuration
S3_BUCKET=anythingllm-storage
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=xxx
S3_SECRET_ACCESS_KEY=xxx
S3_ENDPOINT=https://s3.amazonaws.com

# Storage Options
STORAGE_ENABLE_CDN=true
STORAGE_CDN_URL=https://cdn.example.com
STORAGE_COMPRESSION=true
STORAGE_DEDUPLICATION=true

5. Benefits

Operational

  • Unlimited storage capacity
  • Built-in redundancy
  • Automatic backups
  • Global availability

Performance

  • CDN for fast content delivery
  • Parallel uploads/downloads
  • Reduced server load
  • Better caching strategies

Cost

  • Pay-per-use pricing
  • Storage tiering options
  • Lifecycle management
  • Bandwidth optimization

Implementation Phases

Phase 1: Abstraction Layer

  1. Create storage provider interface
  2. Implement local filesystem provider
  3. Add S3 provider
  4. Unit tests for all providers

Phase 2: Integration

  1. Update file upload endpoints
  2. Migrate document processing
  3. Update file serving logic
  4. Add migration tools

Phase 3: Optimization

  1. Implement deduplication
  2. Add compression
  3. CDN integration
  4. Performance tuning

Expected Impact

  • 99.99% storage availability
  • 50% reduction in storage costs
  • 3x faster file delivery with CDN
  • Unlimited scalability

Labels: enhancement, storage, scalability, infrastructure

Originally created by @jezweb on GitHub (Jun 26, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4052 ## Overview Add support for cloud storage providers to enable better scalability, reliability, and deployment flexibility. Related to upstream discussion: https://github.com/Mintplex-Labs/anything-llm/issues/2797 ## Current State - Local file system storage only - Limited by disk space - No redundancy or backups - Difficult to scale horizontally - No CDN support ## Proposed Solution ### 1. Storage Abstraction Layer #### Interface Design ```typescript interface StorageProvider { upload(file: Buffer, path: string): Promise<string> download(path: string): Promise<Buffer> delete(path: string): Promise<void> exists(path: string): Promise<boolean> getUrl(path: string): Promise<string> listFiles(prefix: string): Promise<string[]> } ``` #### Supported Providers - Local filesystem (current) - Amazon S3 - Google Cloud Storage - Azure Blob Storage - MinIO (S3-compatible) ### 2. Implementation Features #### Core Features - Streaming uploads/downloads - Multipart uploads for large files - Signed URLs for direct access - Automatic retry logic - Progress tracking #### Advanced Features - File deduplication via hashing - Automatic compression - Lifecycle policies - Cross-region replication - CDN integration ### 3. Migration Strategy #### Backward Compatibility - Keep local storage as default - Gradual migration tools - Dual-write during transition - Verification and rollback #### Data Organization ``` /workspaces/{workspace-id}/ /documents/{doc-id}/ original/ processed/ embeddings/ /uploads/ /exports/ /system/ /models/ /cache/ /temp/ ``` ### 4. Configuration ```env # Storage Provider STORAGE_PROVIDER=s3 # local < /dev/null | s3|gcs|azure # S3 Configuration S3_BUCKET=anythingllm-storage S3_REGION=us-east-1 S3_ACCESS_KEY_ID=xxx S3_SECRET_ACCESS_KEY=xxx S3_ENDPOINT=https://s3.amazonaws.com # Storage Options STORAGE_ENABLE_CDN=true STORAGE_CDN_URL=https://cdn.example.com STORAGE_COMPRESSION=true STORAGE_DEDUPLICATION=true ``` ### 5. Benefits #### Operational - Unlimited storage capacity - Built-in redundancy - Automatic backups - Global availability #### Performance - CDN for fast content delivery - Parallel uploads/downloads - Reduced server load - Better caching strategies #### Cost - Pay-per-use pricing - Storage tiering options - Lifecycle management - Bandwidth optimization ## Implementation Phases ### Phase 1: Abstraction Layer 1. Create storage provider interface 2. Implement local filesystem provider 3. Add S3 provider 4. Unit tests for all providers ### Phase 2: Integration 1. Update file upload endpoints 2. Migrate document processing 3. Update file serving logic 4. Add migration tools ### Phase 3: Optimization 1. Implement deduplication 2. Add compression 3. CDN integration 4. Performance tuning ## Expected Impact - 99.99% storage availability - 50% reduction in storage costs - 3x faster file delivery with CDN - Unlimited scalability Labels: enhancement, storage, scalability, infrastructure
yindo closed this issue 2026-02-22 18:30:20 -05:00
yindo changed title from Storage: Cloud Storage Integration (S3/GCS/Azure) to [GH-ISSUE #4052] Storage: Cloud Storage Integration (S3/GCS/Azure) 2026-06-05 14:47:22 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2580
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?