Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #913] [BUG]: Ollama LLM + Embedder #547

New Issue
Closed
opened 2026-02-22 18:20:06 -05:00 by yindo · 8 comments
yindo commented 2026-02-22 18:20:06 -05:00
Owner
Copy Link
Copy Source

Originally created by @timothycarambat on GitHub (Mar 15, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/913

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

When I have Ollama set as both my LLM and embedder model it seems that sending chats results in a bug where Ollama cannot be used for both services.

This is each preference setting pointing to the same Ollama instance.

IMG_0964

Are there known steps to reproduce?

Set Ollama as LLM and embedder. Embed documents. Send Chat

Originally created by @timothycarambat on GitHub (Mar 15, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/913 ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? When I have Ollama set as both my LLM and embedder model it seems that sending chats results in a bug where Ollama cannot be used for both services. This is each preference setting pointing to the same Ollama instance. ![IMG_0964](https://github.com/Mintplex-Labs/anything-llm/assets/16845892/b361c191-9611-47e1-90f5-58acb4ec1718) ### Are there known steps to reproduce? Set Ollama as LLM and embedder. Embed documents. Send Chat
yindo added the possible bug label 2026-02-22 18:20:06 -05:00
yindo closed this issue 2026-02-22 18:20:07 -05:00
yindo commented 2026-02-22 18:20:07 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 15, 2024):

Currently unable to replicate using llama2 and nomic-embed as the LLM and embedder respectively.

@timothycarambat commented on GitHub (Mar 15, 2024): Currently unable to replicate using `llama2` and `nomic-embed` as the LLM and embedder respectively.
yindo commented 2026-02-22 18:20:07 -05:00
Author
Owner
Copy Link
Copy Source

@franva commented on GitHub (Mar 16, 2024):

Hi @timothycarambat I use Anything LLM + Ollama as the LLM Model and Embedding model but then I got an error say:
No embedding model was set.

Did you get this error? How did you fix it?

@franva commented on GitHub (Mar 16, 2024): Hi @timothycarambat I use Anything LLM + Ollama as the LLM Model and Embedding model but then I got an error say: `No embedding model was set.` Did you get this error? How did you fix it?
yindo commented 2026-02-22 18:20:07 -05:00
Author
Owner
Copy Link
Copy Source

@FarVision2 commented on GitHub (Mar 16, 2024):

Good news bad news

The ollama auto loader doesn't seem to kick occasionally when called first through an embedding before a language model is called

If you manually run Serve through ollama terminal or through a text engagement through web UI then it will process normally

Bad news is that the normic embedding is so fast that it blue screens my machine with DPC watchdog error for PDFs over 100mb with 5k pages. This is upserting into pinecone

I don't want to start manually splitting if there's a way to put in a delay buffer slider or slow it down somehow that would be great

@FarVision2 commented on GitHub (Mar 16, 2024): Good news bad news The ollama auto loader doesn't seem to kick occasionally when called first through an embedding before a language model is called If you manually run Serve through ollama terminal or through a text engagement through web UI then it will process normally Bad news is that the normic embedding is so fast that it blue screens my machine with DPC watchdog error for PDFs over 100mb with 5k pages. This is upserting into pinecone I don't want to start manually splitting if there's a way to put in a delay buffer slider or slow it down somehow that would be great
yindo commented 2026-02-22 18:20:07 -05:00
Author
Owner
Copy Link
Copy Source

@FarVision2 commented on GitHub (Mar 16, 2024):

You can simply watch the logs in each docker container to see what's going on

Anything hits the embedding API for services just fine it just doesn't actually trigger loading of the model when used

@FarVision2 commented on GitHub (Mar 16, 2024): You can simply watch the logs in each docker container to see what's going on Anything hits the embedding API for services just fine it just doesn't actually trigger loading of the model when used
yindo commented 2026-02-22 18:20:07 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 18, 2024):

We just use the Ollama API, i dont think there is even an API to "ask" Ollama to swap over. It is supposed to just do that automatically to load the model. I wonder is mlock is enabled and it cannot load the next model because it has once already loaded.

You are not serving ollama with any special ENVs or anything, right?

@timothycarambat commented on GitHub (Mar 18, 2024): We just use the Ollama API, i dont think there is even an API to "ask" Ollama to swap over. It is supposed to just do that automatically to load the model. I wonder is `mlock` is enabled and it cannot load the next model because it has once already loaded. You are not serving ollama with any special ENVs or anything, right?
yindo commented 2026-02-22 18:20:07 -05:00
Author
Owner
Copy Link
Copy Source

@FarVision2 commented on GitHub (Mar 18, 2024):

No nothing different at all. I tap the same API with DifyCE and Open-WebUI. Different models swap in and out as expected. I used the llama2 as language and embedding just for grims but of course it's not so great for embedding. I bounced the Docker a few times and everything kicks on and off as expected every single time except for Anything. :( I was trying to shake it out of the logs but after the GET "/api/tags" it didn't actually process the workflow. I will poke around some more today.

@FarVision2 commented on GitHub (Mar 18, 2024): No nothing different at all. I tap the same API with DifyCE and Open-WebUI. Different models swap in and out as expected. I used the llama2 as language and embedding just for grims but of course it's not so great for embedding. I bounced the Docker a few times and everything kicks on and off as expected every single time except for Anything. :( I was trying to shake it out of the logs but after the GET "/api/tags" it didn't actually process the workflow. I will poke around some more today.
yindo commented 2026-02-22 18:20:07 -05:00
Author
Owner
Copy Link
Copy Source

@zianttt commented on GitHub (May 29, 2024):

Hi, I am able to replicate this issue with the nomic-embed model. Is this issue already resolved?

image

@zianttt commented on GitHub (May 29, 2024): Hi, I am able to replicate this issue with the ```nomic-embed``` model. Is this issue already resolved? ![image](https://github.com/Mintplex-Labs/anything-llm/assets/92505929/1e353c1c-7d18-4832-9591-b23800518d6f)
yindo changed title from [BUG]: Ollama LLM + Embedder to [GH-ISSUE #913] [BUG]: Ollama LLM + Embedder 2026-06-05 14:35:55 -04:00
yindo commented 2026-06-05 14:35:56 -04:00
Author
Owner
Copy Link
Copy Source

@sunshineinsandiego commented on GitHub (Mar 25, 2026):

Same issue here. I am using Ollama for embedding (artic) and chat LLM, and the service is unreachable when I try to chat. Any recommendations?

[backend] info: [Ollama] Context windows cached for all models!
[backend] error: Error: Your Ollama instance could not be reached or is not responding. Please make sure it is running the API server and your connection information is correct in AnythingLLM.
    at #errorHandler (/app/server/utils/AiProviders/ollama/index.js:236:15)
    at /app/server/utils/AiProviders/ollama/index.js:340:31
    at async OllamaAILLM.streamGetChatCompletion (/app/server/utils/AiProviders/ollama/index.js:324:35)
    at async streamChatWithWorkspace (/app/server/utils/chats/stream.js:267:20)
    at async /app/server/endpoints/chat.js:150:9
<!-- gh-comment-id:4127365892 --> @sunshineinsandiego commented on GitHub (Mar 25, 2026): Same issue here. I am using Ollama for embedding (artic) and chat LLM, and the service is unreachable when I try to chat. Any recommendations? ``` [backend] info: [Ollama] Context windows cached for all models! [backend] error: Error: Your Ollama instance could not be reached or is not responding. Please make sure it is running the API server and your connection information is correct in AnythingLLM. at #errorHandler (/app/server/utils/AiProviders/ollama/index.js:236:15) at /app/server/utils/AiProviders/ollama/index.js:340:31 at async OllamaAILLM.streamGetChatCompletion (/app/server/utils/AiProviders/ollama/index.js:324:35) at async streamChatWithWorkspace (/app/server/utils/chats/stream.js:267:20) at async /app/server/endpoints/chat.js:150:9 ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#547
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?