Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #5192] [CLOSED] feat: Move Native Embedder and Reranker Into Isolated Workers w/ Job Queues | Add Document Embedding Status Events #5342

New Issue
Closed
opened 2026-06-05 15:21:02 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 15:21:02 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/5192
Author: @angelplusultra
Created: 3/11/2026
Status: ❌ Closed

Base: master ← Head: feat-native-embedder-job-queue

📝 Commits (10+)

  • d2540a2 implement native embedder job queue
  • e941b17 persist embedding progress across renders
  • a0cd78b add development worker timeouts
  • 573856a change to static method
  • c80107e native reranker
  • 0149020 remove useless return
  • acd1e3d lint
  • f4f6b78 simplify
  • af3c6e0 make embedding worker timeout value configurable by admin
  • 592905c add event emission for missing data

📊 Changes

24 files changed (+1164 additions, -99 deletions)

View changed files

➕ frontend/src/EmbeddingProgressContext.jsx (+165 -0)
📝 frontend/src/components/Modals/ManageWorkspace/Documents/WorkspaceDirectory/index.jsx (+142 -13)
📝 frontend/src/components/Modals/ManageWorkspace/Documents/index.jsx (+46 -17)
📝 frontend/src/components/Modals/ManageWorkspace/index.jsx (+32 -26)
📝 server/.env.example (+8 -1)
📝 server/endpoints/workspaces.js (+42 -1)
📝 server/models/documents.js (+49 -3)
📝 server/utils/EmbeddingEngines/native/index.js (+66 -16)
📝 server/utils/EmbeddingRerankers/native/index.js (+18 -2)
➕ server/utils/WorkerQueue/EmbeddingProgressBus.js (+62 -0)
➕ server/utils/WorkerQueue/WorkerQueue.js (+252 -0)
➕ server/utils/WorkerQueue/index.js (+81 -0)
📝 server/utils/helpers/updateENV.js (+4 -0)
📝 server/utils/vectorDbProviders/astra/index.js (+6 -2)
📝 server/utils/vectorDbProviders/base.js (+3 -1)
📝 server/utils/vectorDbProviders/chroma/index.js (+6 -2)
📝 server/utils/vectorDbProviders/lance/index.js (+9 -5)
📝 server/utils/vectorDbProviders/milvus/index.js (+6 -2)
📝 server/utils/vectorDbProviders/pgvector/index.js (+6 -2)
📝 server/utils/vectorDbProviders/pinecone/index.js (+6 -2)

...and 4 more files

📄 Description

Pull Request Type

  • ✨ feat (New feature)
  • 🐛 fix (Bug fix)
  • ♻️ refactor (Code refactoring without changing behavior)
  • 💄 style (UI style changes)
  • 🔨 chore (Build, CI, maintenance)
  • 📝 docs (Documentation updates)

Relevant Issues

resolves #

Description

Moves the native embedder and reranker from the main process into isolated child processes using child_process.fork() with a serial job queue. This prevents OOM crashes from large document batches from taking down the entire server, while adding real-time document and chunk-level progress reporting to the UI via SSE.

Architecture:

  • WorkerQueue class manages a forked child process with serial FIFO job processing, auto-fork on first job, and configurable TTL that kills the worker when idle
  • EmbeddingProgressBus (singleton EventEmitter) acts as the central hub for progress events between Document.addDocuments and SSE endpoint listeners, with event buffering for late-joining clients after page reloads
  • Separate queue instances for embedding and reranking workers — each gets its own forked process
  • Query embedding (lightweight, single text) still runs in-process; only bulk document embedding is routed through the worker queue

Progress Reporting:

  • SSE endpoint at /workspace/:slug/embed-progress streams document-level events (batch_starting, doc_starting, doc_complete, doc_failed, all_complete) and chunk-level events (chunk_progress)
  • EmbeddingProgressContext (React Context) manages progress state globally and connects SSE on component mount — server-side event replay catches up on any in-progress jobs without needing client-side persistence
  • Progress is scoped per-user — each user only sees their own embedding jobs, even when multiple users embed into the same workspace concurrently
  • Progress UI shows in the document management modal with per-file status (Queued → Progress Bar → Complete/Failed) and auto-clears 5 seconds after completion
  • When no embedding is in progress, the SSE connection stays open silently and waits — no premature signals are sent, avoiding race conditions between the SSE connection and the embed API call

Chunk-Level Progress:

  • embedChunksInProcess reports progress after each chunk group via an onProgress callback
  • The embedding worker bridges this to the parent process via IPC (process.send)
  • WorkerQueue receives the IPC messages and forwards them via its onProgress callback
  • Document.addDocuments sets the current document context (setEmbeddingContext) before each vectorization call so that chunk progress events can be attributed to the correct document, workspace, and user — this is safe because the embedding queue is serial (one job at a time)
  • The frontend renders a progress bar with percentage for each document being embedded

Worker TTL:

  • Configurable via environment variables (NATIVE_EMBEDDING_WORKER_TTL, NATIVE_RERANKING_WORKER_TTL) with defaults of 300s and 900s respectively
  • TTL controls how long the worker stays alive after finishing work to keep it hot for subsequent jobs
  • TTL values are re-read from env before each job so changes take effect without server restart
  • Not exposed in the UI — most users won't need to change these since Node worker startup time is negligible

Other Changes:

  • NativeEmbedder.embedChunks() routes through the worker queue — no changes needed in vector DB providers or non-native embedding engines
  • NativeEmbeddingReranker.rerankViaWorker() added as static method to route through the queue from LanceDB
  • batch_starting event emitted at the start of a batch with the full file list, so SSE history replay can seed all files as "pending" for late-joining clients
  • doc_failed event emitted when fileData() returns null or when the database write fails (previously files were silently skipped, stuck as "pending" in UI)

Visuals (if applicable)

https://github.com/user-attachments/assets/01e5d8fd-b27c-4e13-892d-464d6cf6505a

The document management modal now shows real-time embedding progress when documents are being embedded into a workspace. Each file shows a progress bar with percentage as its chunks are embedded, then transitions to Complete or Failed.

Additional Information

  • Worker TTL can be set to 0 for immediate shutdown after work completes
  • The reranking worker has a longer default TTL (900s) since it runs on every chat query when accuracy-optimized search is enabled with LanceDB, and frequent cold starts would add overhead
  • Event replay on SSE reconnect ensures no stale "Queued" states after page refresh — the server buffers all document-level and chunk-level events and replays them to new subscribers
  • Multi-user: progress is scoped per-user via SSE userId filtering. No cross-user visibility or document locking.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated (if applicable)
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/5192 **Author:** [@angelplusultra](https://github.com/angelplusultra) **Created:** 3/11/2026 **Status:** ❌ Closed **Base:** `master` ← **Head:** `feat-native-embedder-job-queue` --- ### 📝 Commits (10+) - [`d2540a2`](https://github.com/Mintplex-Labs/anything-llm/commit/d2540a280e6155f84810e22a61263c99429b3a8f) implement native embedder job queue - [`e941b17`](https://github.com/Mintplex-Labs/anything-llm/commit/e941b1789a6b4f75d95311d821a69081836165e8) persist embedding progress across renders - [`a0cd78b`](https://github.com/Mintplex-Labs/anything-llm/commit/a0cd78b7715a936acfbd0480a2f455e3e36f1aea) add development worker timeouts - [`573856a`](https://github.com/Mintplex-Labs/anything-llm/commit/573856ae70016b5e8a216156d95cbd4cfd3519b5) change to static method - [`c80107e`](https://github.com/Mintplex-Labs/anything-llm/commit/c80107e0aaba10bd956154bf5cd6bee3fe18335c) native reranker - [`0149020`](https://github.com/Mintplex-Labs/anything-llm/commit/0149020c7afa9a91a3f8aa0b932761f1b58eb57f) remove useless return - [`acd1e3d`](https://github.com/Mintplex-Labs/anything-llm/commit/acd1e3d1001f7d80c434dcda1bace132b037f185) lint - [`f4f6b78`](https://github.com/Mintplex-Labs/anything-llm/commit/f4f6b78e89d0169fafd119c181ef0825884975e1) simplify - [`af3c6e0`](https://github.com/Mintplex-Labs/anything-llm/commit/af3c6e0b252dd3bafed6d561f31ed3f5fd75e8af) make embedding worker timeout value configurable by admin - [`592905c`](https://github.com/Mintplex-Labs/anything-llm/commit/592905c7a344444770584517c3afb98fcbb5cb53) add event emission for missing data ### 📊 Changes **24 files changed** (+1164 additions, -99 deletions) <details> <summary>View changed files</summary> ➕ `frontend/src/EmbeddingProgressContext.jsx` (+165 -0) 📝 `frontend/src/components/Modals/ManageWorkspace/Documents/WorkspaceDirectory/index.jsx` (+142 -13) 📝 `frontend/src/components/Modals/ManageWorkspace/Documents/index.jsx` (+46 -17) 📝 `frontend/src/components/Modals/ManageWorkspace/index.jsx` (+32 -26) 📝 `server/.env.example` (+8 -1) 📝 `server/endpoints/workspaces.js` (+42 -1) 📝 `server/models/documents.js` (+49 -3) 📝 `server/utils/EmbeddingEngines/native/index.js` (+66 -16) 📝 `server/utils/EmbeddingRerankers/native/index.js` (+18 -2) ➕ `server/utils/WorkerQueue/EmbeddingProgressBus.js` (+62 -0) ➕ `server/utils/WorkerQueue/WorkerQueue.js` (+252 -0) ➕ `server/utils/WorkerQueue/index.js` (+81 -0) 📝 `server/utils/helpers/updateENV.js` (+4 -0) 📝 `server/utils/vectorDbProviders/astra/index.js` (+6 -2) 📝 `server/utils/vectorDbProviders/base.js` (+3 -1) 📝 `server/utils/vectorDbProviders/chroma/index.js` (+6 -2) 📝 `server/utils/vectorDbProviders/lance/index.js` (+9 -5) 📝 `server/utils/vectorDbProviders/milvus/index.js` (+6 -2) 📝 `server/utils/vectorDbProviders/pgvector/index.js` (+6 -2) 📝 `server/utils/vectorDbProviders/pinecone/index.js` (+6 -2) _...and 4 more files_ </details> ### 📄 Description ### Pull Request Type - [x] ✨ feat (New feature) - [ ] 🐛 fix (Bug fix) - [ ] ♻️ refactor (Code refactoring without changing behavior) - [ ] 💄 style (UI style changes) - [ ] 🔨 chore (Build, CI, maintenance) - [ ] 📝 docs (Documentation updates) ### Relevant Issues resolves # ### Description Moves the native embedder and reranker from the main process into isolated child processes using `child_process.fork()` with a serial job queue. This prevents OOM crashes from large document batches from taking down the entire server, while adding real-time document and chunk-level progress reporting to the UI via SSE. **Architecture:** - `WorkerQueue` class manages a forked child process with serial FIFO job processing, auto-fork on first job, and configurable TTL that kills the worker when idle - `EmbeddingProgressBus` (singleton EventEmitter) acts as the central hub for progress events between `Document.addDocuments` and SSE endpoint listeners, with event buffering for late-joining clients after page reloads - Separate queue instances for embedding and reranking workers — each gets its own forked process - Query embedding (lightweight, single text) still runs in-process; only bulk document embedding is routed through the worker queue **Progress Reporting:** - SSE endpoint at `/workspace/:slug/embed-progress` streams document-level events (`batch_starting`, `doc_starting`, `doc_complete`, `doc_failed`, `all_complete`) and chunk-level events (`chunk_progress`) - `EmbeddingProgressContext` (React Context) manages progress state globally and connects SSE on component mount — server-side event replay catches up on any in-progress jobs without needing client-side persistence - Progress is scoped per-user — each user only sees their own embedding jobs, even when multiple users embed into the same workspace concurrently - Progress UI shows in the document management modal with per-file status (Queued → Progress Bar → Complete/Failed) and auto-clears 5 seconds after completion - When no embedding is in progress, the SSE connection stays open silently and waits — no premature signals are sent, avoiding race conditions between the SSE connection and the embed API call **Chunk-Level Progress:** - `embedChunksInProcess` reports progress after each chunk group via an `onProgress` callback - The embedding worker bridges this to the parent process via IPC (`process.send`) - `WorkerQueue` receives the IPC messages and forwards them via its `onProgress` callback - `Document.addDocuments` sets the current document context (`setEmbeddingContext`) before each vectorization call so that chunk progress events can be attributed to the correct document, workspace, and user — this is safe because the embedding queue is serial (one job at a time) - The frontend renders a progress bar with percentage for each document being embedded **Worker TTL:** - Configurable via environment variables (`NATIVE_EMBEDDING_WORKER_TTL`, `NATIVE_RERANKING_WORKER_TTL`) with defaults of 300s and 900s respectively - TTL controls how long the worker stays alive after finishing work to keep it hot for subsequent jobs - TTL values are re-read from env before each job so changes take effect without server restart - Not exposed in the UI — most users won't need to change these since Node worker startup time is negligible **Other Changes:** - `NativeEmbedder.embedChunks()` routes through the worker queue — no changes needed in vector DB providers or non-native embedding engines - `NativeEmbeddingReranker.rerankViaWorker()` added as static method to route through the queue from LanceDB - `batch_starting` event emitted at the start of a batch with the full file list, so SSE history replay can seed all files as "pending" for late-joining clients - `doc_failed` event emitted when `fileData()` returns null or when the database write fails (previously files were silently skipped, stuck as "pending" in UI) ### Visuals (if applicable) https://github.com/user-attachments/assets/01e5d8fd-b27c-4e13-892d-464d6cf6505a The document management modal now shows real-time embedding progress when documents are being embedded into a workspace. Each file shows a progress bar with percentage as its chunks are embedded, then transitions to Complete or Failed. ### Additional Information - Worker TTL can be set to 0 for immediate shutdown after work completes - The reranking worker has a longer default TTL (900s) since it runs on every chat query when accuracy-optimized search is enabled with LanceDB, and frequent cold starts would add overhead - Event replay on SSE reconnect ensures no stale "Queued" states after page refresh — the server buffers all document-level and chunk-level events and replays them to new subscribers - Multi-user: progress is scoped per-user via SSE userId filtering. No cross-user visibility or document locking. ### Developer Validations - [x] I ran `yarn lint` from the root of the repo & committed changes - [x] Relevant documentation has been updated (if applicable) - [x] I have tested my code functionality - [x] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 15:21:02 -04:00
yindo closed this issue 2026-06-05 15:21:02 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5342
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?