Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-20 14:43:37 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #83] Swapping OpenAI with local LLM? #51

New Issue
Closed
opened 2026-02-22 18:17:40 -05:00 by yindo · 13 comments
yindo commented 2026-02-22 18:17:40 -05:00
Owner
Copy Link
Copy Source

Originally created by @kfeeeeee on GitHub (Jun 18, 2023).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/83

Hi,

Basically title. THe intro suggests that openai-access can be replaced with locally running models (maybe with oobabooga-openai-api?) Anyway, can't seem to find instructions / env settings for it. Could you tell me if it has bee implented already?

Originally created by @kfeeeeee on GitHub (Jun 18, 2023). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/83 Hi, Basically title. THe intro suggests that openai-access can be replaced with locally running models (maybe with oobabooga-openai-api?) Anyway, can't seem to find instructions / env settings for it. Could you tell me if it has bee implented already?
yindo added the enhancementstage: specifications labels 2026-02-22 18:17:40 -05:00
yindo closed this issue 2026-02-22 18:17:40 -05:00
yindo commented 2026-02-22 18:17:40 -05:00
Author
Owner
Copy Link
Copy Source

@Nasnl commented on GitHub (Jun 19, 2023):

Not yet as far as I can see...

@Nasnl commented on GitHub (Jun 19, 2023): Not yet as far as I can see...
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 19, 2023):

Not currently, the issue with localLLM or other local LLM programs is you need an API accessible endpoint much like what GPT4ALL provides. There are some API wrappers for LocalLLM that some people have built that would work in this instance.

More work needs to be done around this - open to PRs, embeddings is another issue altogether

@timothycarambat commented on GitHub (Jun 19, 2023): Not currently, the issue with localLLM or other local LLM programs is you need an API accessible endpoint much like what GPT4ALL provides. There are some API wrappers for [LocalLLM](https://github.com/QuangBK/localLLM_langchain/blob/main/llm_client.py) that some people have built that would work in this instance. More work needs to be done around this - open to PRs, embeddings is another issue altogether
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@kfeeeeee commented on GitHub (Jun 19, 2023):

Oka, thank you. Maybe a good starting point would be oobabooga's text-webui since it's capable of mimicking an openai API on port 5001.

@kfeeeeee commented on GitHub (Jun 19, 2023): Oka, thank you. Maybe a good starting point would be oobabooga's text-webui since it's capable of mimicking an openai API on port 5001.
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@danglingptr0x0 commented on GitHub (Jun 20, 2023):

Not currently, the issue with localLLM or other local LLM programs is you need an API accessible endpoint much like what GPT4ALL provides. There are some API wrappers for LocalLLM that some people have built that would work in this instance.

More work needs to be done around this - open to PRs, embeddings is another issue altogether

Gradio usually comes with a working API

@danglingptr0x0 commented on GitHub (Jun 20, 2023): > Not currently, the issue with localLLM or other local LLM programs is you need an API accessible endpoint much like what GPT4ALL provides. There are some API wrappers for [LocalLLM](https://github.com/QuangBK/localLLM_langchain/blob/main/llm_client.py) that some people have built that would work in this instance. > > More work needs to be done around this - open to PRs, embeddings is another issue altogether Gradio usually comes with a working API
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@AntonioCiolino commented on GitHub (Jun 20, 2023):

i could suggest chromadb for local embeddings - it's already set up and I've gotten it to work with two docker instances (localhost changes to host.docker.internal). getting a wrapper to work with LocalLLM is something I haven't tried yet though. If that works, this could become a fully autonomous solution for document search and chat, though possibly slow.

@AntonioCiolino commented on GitHub (Jun 20, 2023): i could suggest chromadb for local embeddings - it's already set up and I've gotten it to work with two docker instances (localhost changes to host.docker.internal). getting a wrapper to work with LocalLLM is something I haven't tried yet though. If that works, this could become a fully autonomous solution for document search and chat, though possibly slow.
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@AntonioCiolino commented on GitHub (Jun 26, 2023):

I got close, but the chromadb code has code in it that force calls openai currently.

@AntonioCiolino commented on GitHub (Jun 26, 2023): I got close, but the chromadb code has code in it that force calls openai currently.
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@AntonioCiolino commented on GitHub (Jun 26, 2023):

I should be more clear - I was using LocalAI to read through to a local LLM with the OpenAI API calls. After commenting out the /moderations endpoint (which LocalAI doesn't handle), I was eventually able to call LocalAI and get it to return. However, I discovered that the embeddings - which I also overrode to use BERT, don't work in AnythingLLM, as it's expecting embeddings to be OpenAI only. In the process I've managed to mess some of the indexing up; new files aren't getting properly found. I'm probably going to have to wipe this all clean to purge out the BERTs :) LocalAI can call OpenAI and get the OpenAI embeddings, so it's not a complete failure; I was able to get GPT4ALL to connect and respond to the questions.

To summarize: It does work, with some tweaking, but unless this project supports other types of embeddings (which I don't think is a calling), a fully non-connected Local LLM model isn't likely.

Too bad; I really want to do the stuff completely offline.

@AntonioCiolino commented on GitHub (Jun 26, 2023): I should be more clear - I was using LocalAI to read through to a local LLM with the OpenAI API calls. After commenting out the /moderations endpoint (which LocalAI doesn't handle), I was eventually able to call LocalAI and get it to return. However, I discovered that the embeddings - which I also overrode to use BERT, don't work in AnythingLLM, as it's expecting embeddings to be OpenAI only. In the process I've managed to mess some of the indexing up; new files aren't getting properly found. I'm probably going to have to wipe this all clean to purge out the BERTs :) LocalAI can call OpenAI and get the OpenAI embeddings, so it's not a complete failure; I was able to get GPT4ALL to connect and respond to the questions. To summarize: It does work, with some tweaking, but unless this project supports other types of embeddings (which I don't think is a calling), a fully non-connected Local LLM model isn't likely. Too bad; I really want to do the stuff completely offline.
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@ishaan-jaff commented on GitHub (Sep 28, 2023):

Hi @AntonioCiolino @simonSlamka @kfeeeeee @timothycarambat I’m the maintainer of LiteLLM - we allow you to create a proxy server to call 100+ LLMs, and I think it can solve your problem (I'd love your feedback if it does not)

Try it here: https://docs.litellm.ai/docs/proxy_server

Using LiteLLM Proxy Server

import openai
openai.api_base = "http://0.0.0.0:8000/" # proxy url
print(openai.ChatCompletion.create(model="test", messages=[{"role":"user", "content":"Hey!"}]))

Creating a proxy server

Ollama models

$ litellm --model ollama/llama2 --api_base http://localhost:11434

Hugging Face Models

$ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL]
$ litellm --model claude-instant-1

Anthropic

$ export ANTHROPIC_API_KEY=my-api-key
$ litellm --model claude-instant-1

Palm

$ export PALM_API_KEY=my-palm-key
$ litellm --model palm/chat-bison
@ishaan-jaff commented on GitHub (Sep 28, 2023): Hi @AntonioCiolino @simonSlamka @kfeeeeee @timothycarambat I’m the maintainer of LiteLLM - we allow you to create a proxy server to call 100+ LLMs, and I think it can solve your problem (I'd love your feedback if it does not) Try it here: https://docs.litellm.ai/docs/proxy_server ## Using LiteLLM Proxy Server ```python import openai openai.api_base = "http://0.0.0.0:8000/" # proxy url print(openai.ChatCompletion.create(model="test", messages=[{"role":"user", "content":"Hey!"}])) ``` ## Creating a proxy server Ollama models ```shell $ litellm --model ollama/llama2 --api_base http://localhost:11434 ``` Hugging Face Models ```shell $ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL] $ litellm --model claude-instant-1 ``` Anthropic ```shell $ export ANTHROPIC_API_KEY=my-api-key $ litellm --model claude-instant-1 ``` Palm ```shell $ export PALM_API_KEY=my-palm-key $ litellm --model palm/chat-bison ```
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@danielnbalasoiu commented on GitHub (Oct 12, 2023):

@ishaan-jaff have you managed to get anything-llm running using the proxy workaround you suggested?
If so, can you describe the steps?

@danielnbalasoiu commented on GitHub (Oct 12, 2023): @ishaan-jaff have you managed to get anything-llm running using the proxy workaround you suggested? If so, can you describe the steps?
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 28, 2023):

@ishaan-jaff Would really like to see NodeJS support for all the models the python client supports

@timothycarambat commented on GitHub (Oct 28, 2023): @ishaan-jaff Would really like to see NodeJS support for all the models the python client supports
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@franzbischoff commented on GitHub (Nov 6, 2023):

PR #335

@franzbischoff commented on GitHub (Nov 6, 2023): PR #335
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Nov 9, 2023):

LMStudio integration is now live: f499f1ba59

@timothycarambat commented on GitHub (Nov 9, 2023): LMStudio integration is now live: f499f1ba59f2e9f8be5e44c89a951e859382e005
yindo commented 2026-02-22 18:17:41 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Nov 9, 2023):

Moving conversation to #118

@timothycarambat commented on GitHub (Nov 9, 2023): Moving conversation to #118
yindo changed title from Swapping OpenAI with local LLM? to [GH-ISSUE #83] Swapping OpenAI with local LLM? 2026-06-05 14:33:12 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement stage: specifications
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#51
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?