Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4050] Backend Architecture: Implement Job Queue System with Bull/BullMQ #2579

New Issue
Closed
opened 2026-02-22 18:30:20 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:30:20 -05:00
Owner
Copy Link
Copy Source

Originally created by @jezweb on GitHub (Jun 26, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4050

Overview

Implement a robust job queue system for handling background tasks, improving system reliability and user experience.

Related to upstream discussion: https://github.com/Mintplex-Labs/anything-llm/issues/2797

Current State

  • Synchronous processing blocks requests
  • Limited to single job for document syncing
  • No retry logic or failure handling
  • No progress tracking for long operations

Proposed Job Queue Architecture

1. Queue Categories

Document Processing Queue

  • PDF parsing and OCR
  • Text extraction
  • Metadata extraction
  • Format conversions
    Priority: High, Concurrency: 2

Vector Embedding Queue

  • Text chunking
  • Embedding generation
  • Vector database insertion
  • Index optimization
    Priority: Medium, Concurrency: 4

LLM Request Queue

  • Rate limiting per provider
  • Cost tracking
  • Retry with backoff
  • Response caching
    Priority: High, Concurrency: Based on provider limits

Notification Queue

  • Email notifications
  • Webhook calls
  • System alerts
  • User notifications
    Priority: Low, Concurrency: 10

2. Job Features

Core Features

  • Priority levels (critical/high/medium/low)
  • Retry logic with exponential backoff
  • Dead letter queue for failed jobs
  • Progress tracking and events
  • Graceful shutdown handling

Advanced Features

  • Job dependencies and chains
  • Scheduled/recurring jobs
  • Rate limiting per queue
  • Job result storage
  • Distributed locking

3. Implementation Details

Dependencies

{
  "bull": "^4.x",
  "bullmq": "^4.x",
  "bull-board": "^2.x"
}

Worker Architecture

// Separate worker processes
- main-worker.js (orchestration)
- document-worker.js (CPU intensive)
- vector-worker.js (parallelizable)
- llm-worker.js (rate limited)

Configuration

ENABLE_JOB_QUEUE=true
REDIS_URL=redis://localhost:6379
JOB_QUEUE_CONCURRENCY=4
MAX_JOB_RETRIES=3
JOB_TIMEOUT=300000

4. User Benefits

Immediate

  • Non-blocking document uploads
  • Progress indicators for long operations
  • Automatic retry on failures
  • Better error handling

Long-term

  • Improved scalability
  • Resource optimization
  • Priority processing for paid users
  • Batch processing capabilities

Implementation Phases

Phase 1: Core Infrastructure

  1. Set up Bull/BullMQ with Redis
  2. Create base worker classes
  3. Implement job monitoring UI
  4. Add basic job types

Phase 2: Migration

  1. Convert document processing to jobs
  2. Move vector operations to queue
  3. Implement LLM request queuing
  4. Add progress tracking

Phase 3: Optimization

  1. Fine-tune concurrency settings
  2. Implement priority queues
  3. Add job chaining
  4. Performance monitoring

Expected Impact

  • 90% reduction in request timeout errors
  • 3-5x improvement in document processing throughput
  • Better resource utilization
  • Improved user experience with progress tracking

Labels: enhancement, architecture, backend, scalability

Originally created by @jezweb on GitHub (Jun 26, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4050 ## Overview Implement a robust job queue system for handling background tasks, improving system reliability and user experience. Related to upstream discussion: https://github.com/Mintplex-Labs/anything-llm/issues/2797 ## Current State - Synchronous processing blocks requests - Limited to single job for document syncing - No retry logic or failure handling - No progress tracking for long operations ## Proposed Job Queue Architecture ### 1. Queue Categories #### Document Processing Queue - PDF parsing and OCR - Text extraction - Metadata extraction - Format conversions Priority: High, Concurrency: 2 #### Vector Embedding Queue - Text chunking - Embedding generation - Vector database insertion - Index optimization Priority: Medium, Concurrency: 4 #### LLM Request Queue - Rate limiting per provider - Cost tracking - Retry with backoff - Response caching Priority: High, Concurrency: Based on provider limits #### Notification Queue - Email notifications - Webhook calls - System alerts - User notifications Priority: Low, Concurrency: 10 ### 2. Job Features #### Core Features - Priority levels (critical/high/medium/low) - Retry logic with exponential backoff - Dead letter queue for failed jobs - Progress tracking and events - Graceful shutdown handling #### Advanced Features - Job dependencies and chains - Scheduled/recurring jobs - Rate limiting per queue - Job result storage - Distributed locking ### 3. Implementation Details #### Dependencies ```json { "bull": "^4.x", "bullmq": "^4.x", "bull-board": "^2.x" } ``` #### Worker Architecture ```javascript // Separate worker processes - main-worker.js (orchestration) - document-worker.js (CPU intensive) - vector-worker.js (parallelizable) - llm-worker.js (rate limited) ``` #### Configuration ```env ENABLE_JOB_QUEUE=true REDIS_URL=redis://localhost:6379 JOB_QUEUE_CONCURRENCY=4 MAX_JOB_RETRIES=3 JOB_TIMEOUT=300000 ``` ### 4. User Benefits #### Immediate - Non-blocking document uploads - Progress indicators for long operations - Automatic retry on failures - Better error handling #### Long-term - Improved scalability - Resource optimization - Priority processing for paid users - Batch processing capabilities ## Implementation Phases ### Phase 1: Core Infrastructure 1. Set up Bull/BullMQ with Redis 2. Create base worker classes 3. Implement job monitoring UI 4. Add basic job types ### Phase 2: Migration 1. Convert document processing to jobs 2. Move vector operations to queue 3. Implement LLM request queuing 4. Add progress tracking ### Phase 3: Optimization 1. Fine-tune concurrency settings 2. Implement priority queues 3. Add job chaining 4. Performance monitoring ## Expected Impact - 90% reduction in request timeout errors - 3-5x improvement in document processing throughput - Better resource utilization - Improved user experience with progress tracking Labels: enhancement, architecture, backend, scalability
yindo closed this issue 2026-02-22 18:30:20 -05:00
yindo changed title from Backend Architecture: Implement Job Queue System with Bull/BullMQ to [GH-ISSUE #4050] Backend Architecture: Implement Job Queue System with Bull/BullMQ 2026-06-05 14:47:21 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2579
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?