Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1154] [FEAT]: Integration of Litellm as model proxy such as Ollama #709

New Issue
Closed
opened 2026-02-22 18:20:56 -05:00 by yindo · 6 comments
yindo commented 2026-02-22 18:20:56 -05:00
Owner
Copy Link
Copy Source

Originally created by @flefevre on GitHub (Apr 20, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1154

Originally assigned to: @shatfield4 on GitHub.

What would you like to see?

it could be good to integrate Litellm proxy https://github.com/BerriAI/litellm in addition to ollama
so AnythingLLM could be comaptible with a multiple api model serving

I have seen the post https://github.com/Mintplex-Labs/anything-llm/issues/271 but perhaps it has evolved in the right direction?

Originally created by @flefevre on GitHub (Apr 20, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1154 Originally assigned to: @shatfield4 on GitHub. ### What would you like to see? it could be good to integrate Litellm proxy https://github.com/BerriAI/litellm in addition to ollama so AnythingLLM could be comaptible with a multiple api model serving I have seen the post https://github.com/Mintplex-Labs/anything-llm/issues/271 but perhaps it has evolved in the right direction?
yindo added the enhancementIntegration Requestfeature request labels 2026-02-22 18:20:56 -05:00
yindo closed this issue 2026-02-22 18:20:56 -05:00
yindo commented 2026-02-22 18:20:57 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 21, 2024):

Do you mean using LiteLLM as your LLM provider or adding LiteLLM within AnythingLLM as the main backend? That other issue was the latter, which is not possible because that is a python service and would be more set up and configuration and overhead than it is worth to undertake.

Adding it as an LLM provider where you have to run LiteLLM on your own and we simply connect to it is very doable

@timothycarambat commented on GitHub (Apr 21, 2024): Do you mean using LiteLLM as your LLM provider or adding LiteLLM within AnythingLLM as the main backend? That other issue was the latter, which is not possible because that is a python service and would be more set up and configuration and overhead than it is worth to undertake. Adding it as an LLM provider where you have to run LiteLLM on your own and we simply connect to it is very doable
yindo commented 2026-02-22 18:20:57 -05:00
Author
Owner
Copy Link
Copy Source

@flefevre commented on GitHub (Apr 21, 2024):

Hello
I mean "running Littellm on my own" and "siimply connect anythingllm" to it.
This is a good pattern since Litellm can proxy model provider such as vllm
which can serve and multiplex several llm models.

I have setup a full configuration with ollama / litellm / vllm
and i have tested anythingllm server with ollama: it works like a charm
so i will be able to test anythingllm to litellm to serve llms models from
vllm and so test the speed increase in anythingllm

Let me if you want i make something.

I am sorry for my english.

François from France

On Sun, Apr 21, 2024 at 8:33 PM Timothy Carambat @.***>
wrote:

Do you mean using LiteLLM as your LLM provider or adding LiteLLM within
AnythingLLM as the main backend? That other issue was the latter, which is
not possible because that is a python service and would be more set up and
configuration and overhead than it is worth to undertake.

Adding it as an LLM provider where you have to run LiteLLM on your own and
we simply connect to it is very doable

—
Reply to this email directly, view it on GitHub
https://github.com/Mintplex-Labs/anything-llm/issues/1154#issuecomment-2068158265,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKZRFBBISI4HSFTVLY4LZ3Y6QA55AVCNFSM6AAAAABGQZ62Q2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRYGE2TQMRWGU
.
You are receiving this because you authored the thread.Message ID:
@.***>

--

François Le Fèvre
36 rue Jean Poulmarch
91190 Gif-sur-Yvette
0665604928

@flefevre commented on GitHub (Apr 21, 2024): Hello I mean "running Littellm on my own" and "siimply connect anythingllm" to it. This is a good pattern since Litellm can proxy model provider such as vllm which can serve and multiplex several llm models. I have setup a full configuration with ollama / litellm / vllm and i have tested anythingllm server with ollama: it works like a charm so i will be able to test anythingllm to litellm to serve llms models from vllm and so test the speed increase in anythingllm Let me if you want i make something. I am sorry for my english. François from France On Sun, Apr 21, 2024 at 8:33 PM Timothy Carambat ***@***.***> wrote: > Do you mean using LiteLLM as your LLM provider or adding LiteLLM within > AnythingLLM as the main backend? That other issue was the latter, which is > not possible because that is a python service and would be more set up and > configuration and overhead than it is worth to undertake. > > Adding it as an LLM provider where you have to run LiteLLM on your own and > we simply connect to it is very doable > > — > Reply to this email directly, view it on GitHub > <https://github.com/Mintplex-Labs/anything-llm/issues/1154#issuecomment-2068158265>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABKZRFBBISI4HSFTVLY4LZ3Y6QA55AVCNFSM6AAAAABGQZ62Q2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRYGE2TQMRWGU> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- ----------------------------------------- François Le Fèvre 36 rue Jean Poulmarch 91190 Gif-sur-Yvette 0665604928
yindo commented 2026-02-22 18:20:57 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 21, 2024):

Understand completely and that makes plenty of sense! I just wanted to ensure we were talking about the same feature.

@timothycarambat commented on GitHub (Apr 21, 2024): Understand completely and that makes plenty of sense! I just wanted to ensure we were talking about the same feature.
yindo commented 2026-02-22 18:20:57 -05:00
Author
Owner
Copy Link
Copy Source

@flefevre commented on GitHub (May 1, 2024):

Hello,
Implementing the Litellm binding in Anythingllm will help to solve such problem #1153 where AnythingLlm is able to connect to the vllm, but not to use the model served by vllm due to the specificity of each model, such as in my case mixtral8x7b.

do you know if this feature of Anythingllm Litellm compatibility will be put in the near future feature?
Thanks again. François

@flefevre commented on GitHub (May 1, 2024): Hello, Implementing the Litellm binding in Anythingllm will help to solve such problem #1153 where AnythingLlm is able to connect to the vllm, but not to use the model served by vllm due to the specificity of each model, such as in my case mixtral8x7b. do you know if this feature of Anythingllm Litellm compatibility will be put in the near future feature? Thanks again. François
yindo commented 2026-02-22 18:20:57 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 1, 2024):

@flefevre it is certainly not out of the question and would alleviate many of such cases of unsupported LLM runners not being able to use AnythingLLM while we later build more dedicated support for specific runners over time.

@timothycarambat commented on GitHub (May 1, 2024): @flefevre it is certainly not out of the question and would alleviate many of such cases of unsupported LLM runners not being able to use AnythingLLM while we later build more dedicated support for specific runners over time.
yindo commented 2026-02-22 18:20:57 -05:00
Author
Owner
Copy Link
Copy Source

@avinashkurup commented on GitHub (May 15, 2024):

Hi @timothycarambat may I know if this feature is being planned to be integrated in AnythingLLM, the binding of LiteLLM is something which we would like to see in the AnythingLLM.

@avinashkurup commented on GitHub (May 15, 2024): Hi @timothycarambat may I know if this feature is being planned to be integrated in AnythingLLM, the binding of LiteLLM is something which we would like to see in the AnythingLLM.
yindo changed title from [FEAT]: Integration of Litellm as model proxy such as Ollama to [GH-ISSUE #1154] [FEAT]: Integration of Litellm as model proxy such as Ollama 2026-06-05 14:36:47 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Integration Request enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#709
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?