Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #4688] [CLOSED] Refactor vector db providers to use base VectorDatabase class #4716

New Issue
Closed
opened 2026-02-22 18:36:23 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:36:23 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/4688
Author: @shatfield4
Created: 11/25/2025
Status: ❌ Closed

Base: master ← Head: 4687-feat-refactor-vector-db-providers

📝 Commits (10+)

  • a05da2b refactor lancedb to use base class
  • 5c7118b refactor all vector db providers to use base class
  • f1aa9e4 fix pgvector sanitizeForJsonb to be static
  • c4baa9a remove unnecessary duplicated code for namespace-stats and delete-namespace
  • 25eb331 remove unnecessary duplicated code for performSimilaritySearch and curateSources
  • 1de1a15 remove unnecessary duplicated code for hasNamespace and deleteDocumentFromNamespace
  • 4a83041 remove more duplicate code
  • 1f3fe7b patch astra, qdrant, weaviate, zilliz for invalid citations
  • d65c777 Merge branch 'master' into 4687-feat-refactor-vector-db-providers
  • b3c4a94 final checks on all providers/reorder base class for consistency

📊 Changes

12 files changed (+963 additions, -984 deletions)

View changed files

📝 server/utils/helpers/index.js (+11 -11)
📝 server/utils/vectorDbProviders/astra/index.js (+82 -161)
➕ server/utils/vectorDbProviders/base.js (+311 -0)
📝 server/utils/vectorDbProviders/chroma/index.js (+64 -79)
📝 server/utils/vectorDbProviders/chromacloud/index.js (+28 -20)
📝 server/utils/vectorDbProviders/lance/index.js (+66 -83)
📝 server/utils/vectorDbProviders/milvus/index.js (+52 -129)
📝 server/utils/vectorDbProviders/pgvector/index.js (+124 -88)
📝 server/utils/vectorDbProviders/pinecone/index.js (+59 -31)
📝 server/utils/vectorDbProviders/qdrant/index.js (+50 -124)
📝 server/utils/vectorDbProviders/weaviate/index.js (+61 -128)
📝 server/utils/vectorDbProviders/zilliz/index.js (+55 -130)

📄 Description

Pull Request Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 🔨 chore
  • 📝 docs

Relevant Issues

resolves #4687

What is in this change?

  • Refactors vector db providers to use new VectorDatabase base class to normalize common vector db functions
  • Fixes bug with Astra, Qdrant, Weaviate where result.sourceDocument was not storing score and would not show score value in citation UI

Additional Information

  • LanceDB
  • Astra
  • Chroma
  • ChromaCloud
  • Milvus
  • PGVector
  • Pinecone
  • Qdrant
  • Weaviate
  • Zilliz

Notes

  • Refactor and remove duplicate code from Zilliz Cloud (use Milvus as base) in future PR (#4708)
  • Revisit (#3941)

Testing steps used for every provider

  • Add/remove documents using document picker
  • Check vector count updates accordingly (Milvus/Zilliz lag behind since flush does not happen immediately)
  • Run test query chat to ensure we receive citations correctly and showing score
  • Test reset vector db functionality -> shows 0 vectors after resetting
  • Check agent memory RAG works properly with storing and searching for chunks
  • Check cache clears and resets vector db on change of vector db provider

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/4688 **Author:** [@shatfield4](https://github.com/shatfield4) **Created:** 11/25/2025 **Status:** ❌ Closed **Base:** `master` ← **Head:** `4687-feat-refactor-vector-db-providers` --- ### 📝 Commits (10+) - [`a05da2b`](https://github.com/Mintplex-Labs/anything-llm/commit/a05da2ba338b186e36f386da6cf697e961190dd1) refactor lancedb to use base class - [`5c7118b`](https://github.com/Mintplex-Labs/anything-llm/commit/5c7118bf79511f5066737197a4db59e71a3c0e71) refactor all vector db providers to use base class - [`f1aa9e4`](https://github.com/Mintplex-Labs/anything-llm/commit/f1aa9e49b3ded4015abe88bbd62d7c2f362d3902) fix pgvector sanitizeForJsonb to be static - [`c4baa9a`](https://github.com/Mintplex-Labs/anything-llm/commit/c4baa9a92d5d0b85d0465bf72ad9022f375c0f1a) remove unnecessary duplicated code for namespace-stats and delete-namespace - [`25eb331`](https://github.com/Mintplex-Labs/anything-llm/commit/25eb33111c0649cd16684f2135bd58e9d8bccb07) remove unnecessary duplicated code for performSimilaritySearch and curateSources - [`1de1a15`](https://github.com/Mintplex-Labs/anything-llm/commit/1de1a15e95f6da55e1976c447122e4bd75cf860e) remove unnecessary duplicated code for hasNamespace and deleteDocumentFromNamespace - [`4a83041`](https://github.com/Mintplex-Labs/anything-llm/commit/4a830410cd89ea19d4b76fd286f846eb8aa20ff1) remove more duplicate code - [`1f3fe7b`](https://github.com/Mintplex-Labs/anything-llm/commit/1f3fe7bda1d0372ad69be54d12eebcb81cdeec17) patch astra, qdrant, weaviate, zilliz for invalid citations - [`d65c777`](https://github.com/Mintplex-Labs/anything-llm/commit/d65c7776bc635d3fa9011f3bccb3b347695d0904) Merge branch 'master' into 4687-feat-refactor-vector-db-providers - [`b3c4a94`](https://github.com/Mintplex-Labs/anything-llm/commit/b3c4a9448f7a774db1ca5af6dedbaa08e86791f5) final checks on all providers/reorder base class for consistency ### 📊 Changes **12 files changed** (+963 additions, -984 deletions) <details> <summary>View changed files</summary> 📝 `server/utils/helpers/index.js` (+11 -11) 📝 `server/utils/vectorDbProviders/astra/index.js` (+82 -161) ➕ `server/utils/vectorDbProviders/base.js` (+311 -0) 📝 `server/utils/vectorDbProviders/chroma/index.js` (+64 -79) 📝 `server/utils/vectorDbProviders/chromacloud/index.js` (+28 -20) 📝 `server/utils/vectorDbProviders/lance/index.js` (+66 -83) 📝 `server/utils/vectorDbProviders/milvus/index.js` (+52 -129) 📝 `server/utils/vectorDbProviders/pgvector/index.js` (+124 -88) 📝 `server/utils/vectorDbProviders/pinecone/index.js` (+59 -31) 📝 `server/utils/vectorDbProviders/qdrant/index.js` (+50 -124) 📝 `server/utils/vectorDbProviders/weaviate/index.js` (+61 -128) 📝 `server/utils/vectorDbProviders/zilliz/index.js` (+55 -130) </details> ### 📄 Description ### Pull Request Type <!-- For change type, change [ ] to [x]. --> - [ ] ✨ feat - [ ] 🐛 fix - [x] ♻️ refactor - [ ] 💄 style - [ ] 🔨 chore - [ ] 📝 docs ### Relevant Issues <!-- Use "resolves #xxx" to auto resolve on merge. Otherwise, please use "connect #xxx" --> resolves #4687 ### What is in this change? <!-- Describe the changes in this PR that are impactful to the repo. --> - Refactors vector db providers to use new `VectorDatabase` base class to normalize common vector db functions - Fixes bug with Astra, Qdrant, Weaviate where `result.sourceDocument` was not storing `score` and would not show score value in citation UI ### Additional Information <!-- Add any other context about the Pull Request here that was not captured above. --> - [x] LanceDB - [x] Astra - [x] Chroma - [x] ChromaCloud - [x] Milvus - [x] PGVector - [x] Pinecone - [x] Qdrant - [x] Weaviate - [x] Zilliz ### Notes - Refactor and remove duplicate code from Zilliz Cloud (use Milvus as base) in future PR (#4708) - Revisit (#3941) ### Testing steps used for every provider - Add/remove documents using document picker - Check vector count updates accordingly (Milvus/Zilliz lag behind since flush does not happen immediately) - Run test query chat to ensure we receive citations correctly and showing score - Test reset vector db functionality -> shows 0 vectors after resetting - Check agent memory RAG works properly with storing and searching for chunks - Check cache clears and resets vector db on change of vector db provider ### Developer Validations <!-- All of the applicable items should be checked. --> - [x] I ran `yarn lint` from the root of the repo & committed changes - [x] Relevant documentation has been updated - [x] I have tested my code functionality - [x] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 18:36:23 -05:00
yindo closed this issue 2026-02-22 18:36:23 -05:00
yindo changed title from [PR #4688] Refactor vector db providers to use base VectorDatabase class to [PR #4688] [CLOSED] Refactor vector db providers to use base VectorDatabase class 2026-06-05 15:20:00 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#4716
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?