Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #3972] [CLOSED] Feat/hybrid search bm25 milvus #4483

New Issue
Closed
opened 2026-02-22 18:35:55 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:35:55 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/3972
Author: @JoelMartinezSalesforce
Created: 6/9/2025
Status: ❌ Closed

Base: master ← Head: feat/hybrid-search-bm25-milvus

📝 Commits (3)

  • 7c22765 feat(search): introduce hybrid-ready components
  • 2da7894 Revert "feat(search): introduce hybrid-ready components"
  • a970e82 feat(search): introduce hybrid-ready components

📊 Changes

7 files changed (+1004218 additions, -6 deletions)

View changed files

➕ package-lock.json (+3832 -0)
📝 package.json (+4 -1)
➕ server/storage/models/Snowflake/snowflake-arctic-embed-m-v2.0/config.json (+36 -0)
➕ server/storage/models/Snowflake/snowflake-arctic-embed-m-v2.0/tokenizer.json (+1000184 -0)
➕ server/storage/models/Snowflake/snowflake-arctic-embed-m-v2.0/tokenizer_config.json (+61 -0)
➕ server/utils/EmbeddingEngines/bm25/BM25Index.js (+92 -0)
📝 server/utils/EmbeddingEngines/native/index.js (+9 -5)

📄 Description

This PR introduces the foundation for hybrid retrieval (dense + lexical) by delivering two key components:
1. Embedder upgrade
• Swapped Xenova/all-MiniLM-L6-v2 for Snowflake/snowflake-arctic-embed-m-v2.0.
• Updated cache path, switched to CLS pooling, and refined progress logging.
2. BM25 adapter
• Added BM25Index.js (wink-bm25-text-search + wink-nlp).
• Provides indexChunks, search, and deleteNamespace.
• Persists per-namespace indices under storage/bm25.

These changes do not yet alter runtime behaviour; they simply make BM25 indexing available so the upcoming hybrid orchestrator can fuse dense and lexical scores.

Additional Information
• BM25 indices are saved as plain JSON. Future work may introduce compression and incremental updates.
• No API routes were modified—existing Milvus search remains fully functional.
• Follow-up PRs will integrate the fusion layer, endpoint wiring, and documentation updates.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/3972 **Author:** [@JoelMartinezSalesforce](https://github.com/JoelMartinezSalesforce) **Created:** 6/9/2025 **Status:** ❌ Closed **Base:** `master` ← **Head:** `feat/hybrid-search-bm25-milvus` --- ### 📝 Commits (3) - [`7c22765`](https://github.com/Mintplex-Labs/anything-llm/commit/7c22765feee80ebec73b46c5042cad23f936361b) feat(search): introduce hybrid-ready components - [`2da7894`](https://github.com/Mintplex-Labs/anything-llm/commit/2da789439ab347356c175be029914b43eb0213b4) Revert "feat(search): introduce hybrid-ready components" - [`a970e82`](https://github.com/Mintplex-Labs/anything-llm/commit/a970e82bad597c881920fe33a1d8b5147bee16e4) feat(search): introduce hybrid-ready components ### 📊 Changes **7 files changed** (+1004218 additions, -6 deletions) <details> <summary>View changed files</summary> ➕ `package-lock.json` (+3832 -0) 📝 `package.json` (+4 -1) ➕ `server/storage/models/Snowflake/snowflake-arctic-embed-m-v2.0/config.json` (+36 -0) ➕ `server/storage/models/Snowflake/snowflake-arctic-embed-m-v2.0/tokenizer.json` (+1000184 -0) ➕ `server/storage/models/Snowflake/snowflake-arctic-embed-m-v2.0/tokenizer_config.json` (+61 -0) ➕ `server/utils/EmbeddingEngines/bm25/BM25Index.js` (+92 -0) 📝 `server/utils/EmbeddingEngines/native/index.js` (+9 -5) </details> ### 📄 Description This PR introduces the foundation for hybrid retrieval (dense + lexical) by delivering two key components: 1. Embedder upgrade • Swapped Xenova/all-MiniLM-L6-v2 for Snowflake/snowflake-arctic-embed-m-v2.0. • Updated cache path, switched to CLS pooling, and refined progress logging. 2. BM25 adapter • Added BM25Index.js (wink-bm25-text-search + wink-nlp). • Provides indexChunks, search, and deleteNamespace. • Persists per-namespace indices under storage/bm25. These changes do not yet alter runtime behaviour; they simply make BM25 indexing available so the upcoming hybrid orchestrator can fuse dense and lexical scores. Additional Information • BM25 indices are saved as plain JSON. Future work may introduce compression and incremental updates. • No API routes were modified—existing Milvus search remains fully functional. • Follow-up PRs will integrate the fusion layer, endpoint wiring, and documentation updates. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 18:35:55 -05:00
yindo closed this issue 2026-02-22 18:35:55 -05:00
yindo changed title from [PR #3972] Feat/hybrid search bm25 milvus to [PR #3972] [CLOSED] Feat/hybrid search bm25 milvus 2026-06-05 15:18:48 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#4483
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?