Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1717] AnythingLLM with RAG and Vector Database Provider (Croma) Reloads LLM in VRAM #1124

New Issue
Closed
opened 2026-02-22 18:23:12 -05:00 by yindo · 7 comments
yindo commented 2026-02-22 18:23:12 -05:00
Owner
Copy Link
Copy Source

Originally created by @Smocvin on GitHub (Jun 18, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1717

How are you running AnythingLLM?

Docker (local)

What happened?

I want to report an issue with AnythingLLM when using Retrieval-Augmented Generation (RAG) and the vector database provider, Croma. When I have some vectors for RAG, the chat process involves uploading and downloading the LLM model into the VRAM every time I input a question. This results in a noticeable delay due to the repeated model loading process.

However, when I use Ollama directly, there are no such delays related to uploading and downloading the LLM into the VRAM. Additionally, I do not experience any delays when using AnythingLLM without the vector database (Vector Database set to 0).

This indicates that the issue is specifically related to the interaction between AnythingLLM, RAG, and the Croma vector database.

2X Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
RAM 256Gb DDR3
2X RTX-2080TI
Centos7
AnythingLLM - Docker (Local)

Are there known steps to reproduce?

No response

Originally created by @Smocvin on GitHub (Jun 18, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1717 ### How are you running AnythingLLM? Docker (local) ### What happened? I want to report an issue with AnythingLLM when using Retrieval-Augmented Generation (RAG) and the vector database provider, Croma. When I have some vectors for RAG, the chat process involves uploading and downloading the LLM model into the VRAM every time I input a question. This results in a noticeable delay due to the repeated model loading process. However, when I use Ollama directly, there are no such delays related to uploading and downloading the LLM into the VRAM. Additionally, I do not experience any delays when using AnythingLLM without the vector database (Vector Database set to 0). This indicates that the issue is specifically related to the interaction between AnythingLLM, RAG, and the Croma vector database. -------------------------------------------------------- 2X Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz RAM 256Gb DDR3 2X RTX-2080TI Centos7 AnythingLLM - Docker (Local) ### Are there known steps to reproduce? _No response_
yindo added the question label 2026-02-22 18:23:12 -05:00
yindo closed this issue 2026-02-22 18:23:13 -05:00
yindo commented 2026-02-22 18:23:13 -05:00
Author
Owner
Copy Link
Copy Source

@Smocvin commented on GitHub (Jun 18, 2024):

An update: The LLM reloads in VRAM every time only when using a third-party embedding provider (I use Ollama snow-flake-arctic-22m). There are no issues with the original AnythingLLM embedder.

@Smocvin commented on GitHub (Jun 18, 2024): An update: The LLM reloads in VRAM every time only when using a third-party embedding provider (I use Ollama snow-flake-arctic-22m). There are no issues with the original AnythingLLM embedder.
yindo commented 2026-02-22 18:23:13 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 18, 2024):

@Smocvin Are you using Ollama for your embedder or LLM or both? The issue only mentions you using Ollama as your LLM

@timothycarambat commented on GitHub (Jun 18, 2024): @Smocvin Are you using Ollama for your embedder or LLM or both? The issue only mentions you using Ollama as your LLM
yindo commented 2026-02-22 18:23:13 -05:00
Author
Owner
Copy Link
Copy Source

@Smocvin commented on GitHub (Jun 18, 2024):

When I use LLM -> Ollama and embedder "snow-flake-arctic-22m" -> Ollama = LLM in VRAM is uploading and dowloading
When I use LLM -> Ollama and embedder Original AnythingLLM = no issues with LLM uploading and dowloading in VRAM

@Smocvin commented on GitHub (Jun 18, 2024): When I use LLM -> Ollama and embedder "snow-flake-arctic-22m" -> Ollama = LLM in VRAM is uploading and dowloading When I use LLM -> Ollama and embedder Original AnythingLLM = no issues with LLM uploading and dowloading in VRAM
yindo commented 2026-02-22 18:23:13 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 18, 2024):

It is related to this issue and would be solved by this PR:

  • Ollama model unloads after 5 minutes #1585

The loading into vRAM is because we mlock it for future call performance. The timeout will resolve that matter however if you use Ollama for embedding and LLM and the system cannot hold both models in VRAM it will kick one out to make room for the one it needs it both cannot fit.

That is what is going on here - linked PR will fix that so moving discussion there

@timothycarambat commented on GitHub (Jun 18, 2024): It is related to this issue and would be solved by this PR: - Ollama model unloads after 5 minutes #1585 The loading into vRAM is because we `mlock` it for future call performance. The timeout will resolve that matter _however_ if you use Ollama for embedding and LLM **and** the system cannot hold both models in VRAM it will kick one out to make room for the one it needs it both cannot fit. That is what is going on here - linked PR will fix that so moving discussion there
yindo commented 2026-02-22 18:23:13 -05:00
Author
Owner
Copy Link
Copy Source

@Smocvin commented on GitHub (Jun 18, 2024):

I see the problem. The issue is not with AnythingLLM; it's with Ollama. Ollama cannot hold two models simultaneously, even though I am using small models that should fit in the VRAM.

@Smocvin commented on GitHub (Jun 18, 2024): I see the problem. The issue is not with AnythingLLM; it's with Ollama. Ollama cannot hold two models simultaneously, even though I am using small models that should fit in the VRAM.
yindo commented 2026-02-22 18:23:13 -05:00
Author
Owner
Copy Link
Copy Source

@Smocvin commented on GitHub (Jun 18, 2024):

Solution to Ollama VRAM Issue
I found the solution to the issue:

Open the file /etc/systemd/system/ollama.service.
In the [Service] section, add the following line:

Environment="OLLAMA_MAX_LOADED_MODELS=3"

Save the file and close it.
Next, execute the following commands in the terminal:

systemctl daemon-reload
systemctl restart ollama

This configuration change allows Ollama to hold up to three models simultaneously, resolving the VRAM reloading issue.

@Smocvin commented on GitHub (Jun 18, 2024): **Solution to Ollama VRAM Issue** I found the solution to the issue: Open the file /etc/systemd/system/ollama.service. In the [Service] section, add the following line: ``` Environment="OLLAMA_MAX_LOADED_MODELS=3" ``` Save the file and close it. Next, execute the following commands in the terminal: ``` systemctl daemon-reload systemctl restart ollama ``` This configuration change allows Ollama to hold up to three models simultaneously, resolving the VRAM reloading issue.
yindo commented 2026-02-22 18:23:13 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 18, 2024):

@Smocvin TIL! I did not know that was even a ENV key for ollama 😆

@timothycarambat commented on GitHub (Jun 18, 2024): @Smocvin TIL! I did not know that was even a ENV key for ollama 😆
yindo changed title from AnythingLLM with RAG and Vector Database Provider (Croma) Reloads LLM in VRAM to [GH-ISSUE #1717] AnythingLLM with RAG and Vector Database Provider (Croma) Reloads LLM in VRAM 2026-06-05 14:39:05 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1124
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?