Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3668] [FEAT]: Add an option to replace the internal embedding model with other sentence-transformers compatible models #2365

New Issue
Closed
opened 2026-02-22 18:29:22 -05:00 by yindo · 6 comments
yindo commented 2026-02-22 18:29:22 -05:00
Owner
Copy Link
Copy Source

Originally created by @JosefAschauer on GitHub (Apr 17, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3668

What would you like to see?

With my own RAG build, I tested with Snowflake/snowflake-arctic-embed-l-v2.0 which seems far superior to the current embedder and it supports multilangual documents
Any chance to see this coming? Or is it already possible (didn't find an option)?

Originally created by @JosefAschauer on GitHub (Apr 17, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3668 ### What would you like to see? With my own RAG build, I tested with Snowflake/snowflake-arctic-embed-l-v2.0 which seems far superior to the current embedder and it supports multilangual documents Any chance to see this coming? Or is it already possible (didn't find an option)?
yindo added the enhancementfeature request labels 2026-02-22 18:29:22 -05:00
yindo closed this issue 2026-02-22 18:29:22 -05:00
yindo commented 2026-02-22 18:29:22 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 17, 2025):

We likely won't replace the default embedder totally, but this issue to expand the ONNX model selection for it is still pending.

if you want a better model today you can simply use ollama or lmstudio to load that model in and then use that provider as your embedding model and youll have it running with very minimal effort.

@timothycarambat commented on GitHub (Apr 17, 2025): We likely won't replace the _default_ embedder totally, [but this issue to expand the ONNX model](https://github.com/Mintplex-Labs/anything-llm/issues/658) selection for it is still pending. if you want a better model _today_ you can simply use [ollama](https://ollama.com/library/snowflake-arctic-embed2) or lmstudio to load that model in and then use that provider as your embedding model and youll have it running with very minimal effort.
yindo commented 2026-02-22 18:29:22 -05:00
Author
Owner
Copy Link
Copy Source

@JosefAschauer commented on GitHub (Apr 17, 2025):

I tried with ollama and localai but compared to sentence transformers using arctic snowflake l v2, where it is meant to run in and got developed for, the outcome seems rather not-so-great. Didnt try lmstudio yet. Is there a specific reason you stick with your current internal embedder?

@JosefAschauer commented on GitHub (Apr 17, 2025): I tried with ollama and localai but compared to sentence transformers using arctic snowflake l v2, where it is meant to run in and got developed for, the outcome seems rather not-so-great. Didnt try lmstudio yet. Is there a specific reason you stick with your current internal embedder?
yindo commented 2026-02-22 18:29:22 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 17, 2025):

It is just a good catch-all and is <20mb to download, keep in mind it installs on its first use. Snowflake v2 is ~3GB in ONNX format, and people on really slow internet connections, that is just not tenable or reasonable to do on the fly.

This also does not mention the resource demand from loading a model that large on the fly to do embedding. We support runtimes on Raspberry Pi to massive servers.

It's a better model, sure, but that comes with tradeoffs as would any model, and the current default is just a good mix of size and accuracy even with its flaws, since users can opt in to replace it with something they know they can run or is better for their use case.

@timothycarambat commented on GitHub (Apr 17, 2025): It is just a good catch-all and is <20mb to download, keep in mind it installs on its first use. Snowflake v2 is ~3GB in ONNX format, and people on really slow internet connections, that is just not tenable or reasonable to do on the fly. This also does not mention the resource demand from loading a model that large on the fly to do embedding. We support runtimes on Raspberry Pi to massive servers. It's a better model, sure, but that comes with tradeoffs as would any model, and the current default is just a good mix of size and accuracy even with its flaws, since users can opt in to replace it with something they know they can run or is better for their use case.
yindo commented 2026-02-22 18:29:22 -05:00
Author
Owner
Copy Link
Copy Source

@JosefAschauer commented on GitHub (Apr 18, 2025):

Thank you and you are right to take care of people with slow internet but in this case, can't we just have an option in settings to replace the internal model with a different internal model? My main problem is multilingual embeddings where all-MiniLM-L6-v2 simply fails and a demand on multilingual embeddings is at least as widespread - I believe even more - as the demand to serve users with a limited internet connection and I even believe there are more guys out there where english isn't their main language compared to people with slow internet, especially compared to users with slow internet and an interest into running a RAG system:)

I assume you use the Sentence Transformers' ONNX integration since even all-MiniLM-L6-v2 was exported from sentence-transfomers and without, you would loose pretty much of the models base capabilities. Since my current favourite open source multilingual and high-ranking Snowflake L v2 was also developed for / in sentence trannsformer, would an option to download snowflake L v2 in ONNX format be a too massive effort since it would need too many other adjustments as well?

@JosefAschauer commented on GitHub (Apr 18, 2025): Thank you and you are right to take care of people with slow internet but in this case, can't we just have an option in settings to replace the internal model with a different internal model? My main problem is multilingual embeddings where all-MiniLM-L6-v2 simply fails and a demand on multilingual embeddings is at least as widespread - I believe even more - as the demand to serve users with a limited internet connection and I even believe there are more guys out there where english isn't their main language compared to people with slow internet, especially compared to users with slow internet and an interest into running a RAG system:) I assume you use the Sentence Transformers' ONNX integration since even all-MiniLM-L6-v2 was exported from sentence-transfomers and without, you would loose pretty much of the models base capabilities. Since my current favourite open source multilingual and high-ranking Snowflake L v2 was also developed for / in sentence trannsformer, would an option to download snowflake L v2 in ONNX format be a too massive effort since it would need too many other adjustments as well?
yindo commented 2026-02-22 18:29:23 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 18, 2025):

Correct, and that is what #658 is aimed at solving. We would just not mirror the alternative embedders and you would need to select that model explictly for us to download and set as your embedder.

@timothycarambat commented on GitHub (Apr 18, 2025): Correct, and that is what #658 is aimed at solving. We would just not mirror the alternative embedders and you would need to select that model explictly for us to download and set as your embedder.
yindo commented 2026-02-22 18:29:23 -05:00
Author
Owner
Copy Link
Copy Source

@JosefAschauer commented on GitHub (Apr 18, 2025):

This would be perfect! Thanks!

@JosefAschauer commented on GitHub (Apr 18, 2025): This would be perfect! Thanks!
yindo changed title from [FEAT]: Add an option to replace the internal embedding model with other sentence-transformers compatible models to [GH-ISSUE #3668] [FEAT]: Add an option to replace the internal embedding model with other sentence-transformers compatible models 2026-06-05 14:46:09 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2365
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?