Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #5254] [MERGED] feat: Document Embedding Status Events | Refactor Document Embedding to Job Queue and Forked Process #5374

New Issue
Closed
opened 2026-06-05 15:21:09 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 15:21:09 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/5254
Author: @angelplusultra
Created: 3/23/2026
Status: ✅ Merged
Merged: 4/6/2026
Merged by: @timothycarambat

Base: master ← Head: refactor-standardize-background-job-system

📝 Commits (10+)

  • d2540a2 implement native embedder job queue
  • e941b17 persist embedding progress across renders
  • a0cd78b add development worker timeouts
  • 573856a change to static method
  • c80107e native reranker
  • 0149020 remove useless return
  • acd1e3d lint
  • f4f6b78 simplify
  • af3c6e0 make embedding worker timeout value configurable by admin
  • 592905c add event emission for missing data

📊 Changes

23 files changed (+1221 additions, -90 deletions)

View changed files

➕ frontend/src/EmbeddingProgressContext.jsx (+240 -0)
📝 frontend/src/components/Modals/ManageWorkspace/Documents/WorkspaceDirectory/index.jsx (+180 -7)
📝 frontend/src/components/Modals/ManageWorkspace/Documents/index.jsx (+53 -23)
📝 frontend/src/components/Modals/ManageWorkspace/index.jsx (+4 -1)
📝 frontend/src/models/workspace.js (+9 -0)
📝 server/endpoints/workspaces.js (+82 -0)
➕ server/jobs/embedding-worker.js (+199 -0)
📝 server/models/documents.js (+59 -2)
📝 server/utils/BackgroundWorkers/index.js (+56 -33)
📝 server/utils/EmbeddingEngines/azureOpenAi/index.js (+6 -1)
📝 server/utils/EmbeddingEngines/cohere/index.js (+6 -1)
📝 server/utils/EmbeddingEngines/gemini/index.js (+6 -1)
📝 server/utils/EmbeddingEngines/genericOpenAi/index.js (+6 -1)
📝 server/utils/EmbeddingEngines/lemonade/index.js (+26 -10)
📝 server/utils/EmbeddingEngines/liteLLM/index.js (+10 -1)
📝 server/utils/EmbeddingEngines/lmstudio/index.js (+7 -4)
📝 server/utils/EmbeddingEngines/localAi/index.js (+10 -1)
📝 server/utils/EmbeddingEngines/native/index.js (+7 -1)
📝 server/utils/EmbeddingEngines/ollama/index.js (+5 -1)
📝 server/utils/EmbeddingEngines/openAi/index.js (+6 -1)

...and 3 more files

📄 Description

Pull Request Type

  • ✨ feat (New feature)
  • 🐛 fix (Bug fix)
  • ♻️ refactor (Code refactoring without changing behavior)
  • 💄 style (UI style changes)
  • 🔨 chore (Build, CI, maintenance)
  • 📝 docs (Documentation updates)

Relevant Issues

Description

Moves native embedding out of the main server process and into an isolated child process with a FIFO job queue, preventing OOM crashes from large document batches. Adds real-time per-file embedding progress in the UI via SSE.

Isolated embedding worker (server/jobs/embedding-worker.js)

  • Embedding now runs in a forked child process spawned through BackgroundService/Bree — the same infrastructure used for all other background jobs (cleanup, sync).
  • The worker stays alive between jobs to keep the ML model loaded in memory. After a configurable TTL (default 5 min, via NATIVE_EMBEDDING_WORKER_TTL), the worker is killed to free resources.
  • If the worker OOMs on a large document, only the child process dies — the main server stays up.

WorkerQueue (server/utils/WorkerQueue/)

  • WorkerQueue.js — generic serial FIFO job queue. Manages the worker lifecycle: spawns via Bree, waits for a ready handshake, sends jobs one at a time via IPC, handles results/errors, and manages the TTL idle timer.
  • EmbeddingProgressBus.js — singleton event emitter that buffers progress events per-workspace so late-joining SSE clients can replay history.
  • index.js — creates the embedding queue instance, wires the chunk progress callback to the bus, and registers graceful shutdown handlers.

NativeEmbedder changes (server/utils/EmbeddingEngines/native/index.js)

  • embedChunks() now routes through the worker queue instead of running in-process.
  • The original in-process logic is renamed to embedChunksInProcess() — only called by the worker.
  • embedTextInput() runs directly in-process (single query embeddings don't need isolation).
  • embedChunksInProcess() accepts an onProgress callback for chunk-level progress reporting.

Document model progress events (server/models/documents.js)

  • addDocuments() emits progress events on the bus: batch_starting, doc_starting, doc_complete, doc_failed, chunk_progress, and all_complete.

SSE endpoint (server/endpoints/workspaces.js)

  • New GET /workspace/:slug/embed-progress endpoint. Subscribes to the progress bus filtered by workspace and user, replays buffered history for reconnecting clients, and streams live events.

Vector DB providers

  • All 8 providers (Astra, Chroma, Lance, Milvus, PGVector, Pinecone, Qdrant, Weaviate) + the base class pass an embeddingContext parameter through addDocumentToNamespace() → embedChunks() so the worker queue can attach document context to progress events.

Frontend progress UI

  • EmbeddingProgressContext — React context that manages SSE connections via @microsoft/fetch-event-source. Tracks per-workspace, per-file status (pending → embedding → complete/failed) with chunk-level progress percentages. Auto-clears 5s after completion.
  • WorkspaceDirectory — replaces the generic loading spinner with per-file progress rows showing status icons and progress bars during embedding.
  • DocumentSettings — connects SSE on mount (catches in-progress jobs via history replay), triggers startEmbedding() before the API call, and auto-refreshes file lists when embedding completes.

Visuals (if applicable)

Additional Information

  • Worker TTL is configurable via NATIVE_EMBEDDING_WORKER_TTL env var (seconds, default 300).
  • This PR only covers embedding. Reranking is unchanged from master.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated (if applicable)
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/5254 **Author:** [@angelplusultra](https://github.com/angelplusultra) **Created:** 3/23/2026 **Status:** ✅ Merged **Merged:** 4/6/2026 **Merged by:** [@timothycarambat](https://github.com/timothycarambat) **Base:** `master` ← **Head:** `refactor-standardize-background-job-system` --- ### 📝 Commits (10+) - [`d2540a2`](https://github.com/Mintplex-Labs/anything-llm/commit/d2540a280e6155f84810e22a61263c99429b3a8f) implement native embedder job queue - [`e941b17`](https://github.com/Mintplex-Labs/anything-llm/commit/e941b1789a6b4f75d95311d821a69081836165e8) persist embedding progress across renders - [`a0cd78b`](https://github.com/Mintplex-Labs/anything-llm/commit/a0cd78b7715a936acfbd0480a2f455e3e36f1aea) add development worker timeouts - [`573856a`](https://github.com/Mintplex-Labs/anything-llm/commit/573856ae70016b5e8a216156d95cbd4cfd3519b5) change to static method - [`c80107e`](https://github.com/Mintplex-Labs/anything-llm/commit/c80107e0aaba10bd956154bf5cd6bee3fe18335c) native reranker - [`0149020`](https://github.com/Mintplex-Labs/anything-llm/commit/0149020c7afa9a91a3f8aa0b932761f1b58eb57f) remove useless return - [`acd1e3d`](https://github.com/Mintplex-Labs/anything-llm/commit/acd1e3d1001f7d80c434dcda1bace132b037f185) lint - [`f4f6b78`](https://github.com/Mintplex-Labs/anything-llm/commit/f4f6b78e89d0169fafd119c181ef0825884975e1) simplify - [`af3c6e0`](https://github.com/Mintplex-Labs/anything-llm/commit/af3c6e0b252dd3bafed6d561f31ed3f5fd75e8af) make embedding worker timeout value configurable by admin - [`592905c`](https://github.com/Mintplex-Labs/anything-llm/commit/592905c7a344444770584517c3afb98fcbb5cb53) add event emission for missing data ### 📊 Changes **23 files changed** (+1221 additions, -90 deletions) <details> <summary>View changed files</summary> ➕ `frontend/src/EmbeddingProgressContext.jsx` (+240 -0) 📝 `frontend/src/components/Modals/ManageWorkspace/Documents/WorkspaceDirectory/index.jsx` (+180 -7) 📝 `frontend/src/components/Modals/ManageWorkspace/Documents/index.jsx` (+53 -23) 📝 `frontend/src/components/Modals/ManageWorkspace/index.jsx` (+4 -1) 📝 `frontend/src/models/workspace.js` (+9 -0) 📝 `server/endpoints/workspaces.js` (+82 -0) ➕ `server/jobs/embedding-worker.js` (+199 -0) 📝 `server/models/documents.js` (+59 -2) 📝 `server/utils/BackgroundWorkers/index.js` (+56 -33) 📝 `server/utils/EmbeddingEngines/azureOpenAi/index.js` (+6 -1) 📝 `server/utils/EmbeddingEngines/cohere/index.js` (+6 -1) 📝 `server/utils/EmbeddingEngines/gemini/index.js` (+6 -1) 📝 `server/utils/EmbeddingEngines/genericOpenAi/index.js` (+6 -1) 📝 `server/utils/EmbeddingEngines/lemonade/index.js` (+26 -10) 📝 `server/utils/EmbeddingEngines/liteLLM/index.js` (+10 -1) 📝 `server/utils/EmbeddingEngines/lmstudio/index.js` (+7 -4) 📝 `server/utils/EmbeddingEngines/localAi/index.js` (+10 -1) 📝 `server/utils/EmbeddingEngines/native/index.js` (+7 -1) 📝 `server/utils/EmbeddingEngines/ollama/index.js` (+5 -1) 📝 `server/utils/EmbeddingEngines/openAi/index.js` (+6 -1) _...and 3 more files_ </details> ### 📄 Description ### Pull Request Type <!-- For change type, change [ ] to [x]. --> - [x] ✨ feat (New feature) - [ ] 🐛 fix (Bug fix) - [x] ♻️ refactor (Code refactoring without changing behavior) - [ ] 💄 style (UI style changes) - [ ] 🔨 chore (Build, CI, maintenance) - [ ] 📝 docs (Documentation updates) ### Relevant Issues <!-- Use "resolves #xxx" to auto resolve on merge. Otherwise, please use "connect #xxx" --> ### Description Moves native embedding out of the main server process and into an isolated child process with a FIFO job queue, preventing OOM crashes from large document batches. Adds real-time per-file embedding progress in the UI via SSE. #### Isolated embedding worker (`server/jobs/embedding-worker.js`) - Embedding now runs in a forked child process spawned through `BackgroundService/Bree` — the same infrastructure used for all other background jobs (cleanup, sync). - The worker stays alive between jobs to keep the ML model loaded in memory. After a configurable TTL (default 5 min, via `NATIVE_EMBEDDING_WORKER_TTL`), the worker is killed to free resources. - If the worker OOMs on a large document, only the child process dies — the main server stays up. #### WorkerQueue (`server/utils/WorkerQueue/`) - `WorkerQueue.js` — generic serial FIFO job queue. Manages the worker lifecycle: spawns via Bree, waits for a ready handshake, sends jobs one at a time via IPC, handles results/errors, and manages the TTL idle timer. - `EmbeddingProgressBus.js` — singleton event emitter that buffers progress events per-workspace so late-joining SSE clients can replay history. - `index.js` — creates the embedding queue instance, wires the chunk progress callback to the bus, and registers graceful shutdown handlers. #### NativeEmbedder changes (`server/utils/EmbeddingEngines/native/index.js`) - `embedChunks()` now routes through the worker queue instead of running in-process. - The original in-process logic is renamed to `embedChunksInProcess()` — only called by the worker. - `embedTextInput()` runs directly in-process (single query embeddings don't need isolation). - `embedChunksInProcess()` accepts an `onProgress` callback for chunk-level progress reporting. #### Document model progress events (`server/models/documents.js`) - `addDocuments()` emits progress events on the bus: `batch_starting`, `doc_starting`, `doc_complete`, `doc_failed`, `chunk_progress`, and `all_complete`. #### SSE endpoint (`server/endpoints/workspaces.js`) - New `GET /workspace/:slug/embed-progress` endpoint. Subscribes to the progress bus filtered by workspace and user, replays buffered history for reconnecting clients, and streams live events. #### Vector DB providers - All 8 providers (Astra, Chroma, Lance, Milvus, PGVector, Pinecone, Qdrant, Weaviate) + the base class pass an `embeddingContext` parameter through `addDocumentToNamespace()` → `embedChunks()` so the worker queue can attach document context to progress events. #### Frontend progress UI - `EmbeddingProgressContext` — React context that manages SSE connections via `@microsoft/fetch-event-source`. Tracks per-workspace, per-file status (`pending` → `embedding` → `complete`/`failed`) with chunk-level progress percentages. Auto-clears 5s after completion. - `WorkspaceDirectory` — replaces the generic loading spinner with per-file progress rows showing status icons and progress bars during embedding. - `DocumentSettings` — connects SSE on mount (catches in-progress jobs via history replay), triggers `startEmbedding()` before the API call, and auto-refreshes file lists when embedding completes. ### Visuals (if applicable) <!-- Add screenshots or screen recordings to demonstrate the changes, especially for UI updates. --> ### Additional Information - Worker TTL is configurable via `NATIVE_EMBEDDING_WORKER_TTL` env var (seconds, default 300). - This PR only covers embedding. Reranking is unchanged from master. ### Developer Validations <!-- All of the applicable items should be checked. --> - [x] I ran `yarn lint` from the root of the repo & committed changes - [x] Relevant documentation has been updated (if applicable) - [x] I have tested my code functionality - [x] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 15:21:09 -04:00
yindo closed this issue 2026-06-05 15:21:09 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5374
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?