Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3646] Performance issue #2353

New Issue
Closed
opened 2026-02-22 18:29:18 -05:00 by yindo · 1 comment
yindo commented 2026-02-22 18:29:18 -05:00
Owner
Copy Link
Copy Source

Originally created by @shravanveladi on GitHub (Apr 14, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3646

our stack: ollama+llama3.2:3b+qdrant+nomic+anythingllm(1.7.8) on c6i.2xlarge.
while using my anythingllm, for every prompt ollama intializing . Due to this issue, it takes 1min to get response.

how to fix it ? i have already added parameter "OLLAMA_KEEP_ALIVE_TIMEOUT=-1" in .env file, but still the issue not resolved.

I think this is basic issue because why model initializing for every prompt ? is it lack resource ? how can i confirm is it lack of resources ? We are missing some basic thing to fix it. please guide us.

Is there any solution expect going with GPU system because its too costly for now to us.

From chatgpt statement:
OLLAMA_KEEP_ALIVE_TIMEOUT=-1 is working correctly — the model is staying loaded.

🧠 OllamaAILLM initialized appears on every question — ⚠️ this means the model is being re-initialized repeatedly by AnythingLLM.

⏱️ Still takes ~1 minute per response → this is not normal and suggests orchestration lag.

Originally created by @shravanveladi on GitHub (Apr 14, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3646 our stack: ollama+llama3.2:3b+qdrant+nomic+anythingllm(1.7.8) on c6i.2xlarge. while using my anythingllm, for every prompt ollama intializing . Due to this issue, it takes 1min to get response. how to fix it ? i have already added parameter "OLLAMA_KEEP_ALIVE_TIMEOUT=-1" in .env file, but still the issue not resolved. I think this is basic issue because why model initializing for every prompt ? is it lack resource ? how can i confirm is it lack of resources ? We are missing some basic thing to fix it. please guide us. Is there any solution expect going with GPU system because its too costly for now to us. From chatgpt statement: OLLAMA_KEEP_ALIVE_TIMEOUT=-1 is working correctly — the model is staying loaded. 🧠 OllamaAILLM initialized appears on every question — ⚠️ this means the model is being re-initialized repeatedly by AnythingLLM. ⏱️ Still takes ~1 minute per response → this is not normal and suggests orchestration lag.
yindo closed this issue 2026-02-22 18:29:18 -05:00
yindo commented 2026-02-22 18:29:19 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 14, 2025):

As I replied to your other issue, you are using a moderately underpowered machine for this specific stack of models - if you are using RAG + this setup, you will see more tokens equal longer response times which getting 7-13 tok/s. There is no amount of software that AnythingLLM can write to make this faster. You can reduce the amount of context responses from 4->2, reduce the chat history window, swap to another quant of llama3.2, but end of the day you are limited by resources.

🧠 OllamaAILLM initialized appears on every question — ⚠️ this means the model is being re-initialized repeatedly by AnythingLLM.

This does not mean we are re-loading the model every request; this is just our wrapper. We run api requests to ollama so in this case when we ping the API the model is already loaded and inference begins immediately. You can also check this by the ollama logs, seeing whether reloading of the model into memory is occurring or not each request.

Also, I notice you say you are using nomic for embedding - through ollama I assume. If that is the case and you are using Ollama for both embedding and inference then what is likely occurring is that when we embed your prompt to do RAG, your limited RAM likely unloads llama, then when we try to run inference, we ask Ollama for LLM inference right after RAG search and it needs to reload the llama model again - this is all handled by Ollama automatically even if you specify OLLAMA_KEEP_ALIVE_TIMEOUT. Ollama cannot force a model to stay loaded if it needs to unload it to service a different request - like for nomic embedding.

Ollama manages model loading, not AnythingLLM, in this case. With only 16GB of memory, I am willing to bet that is not enough to run AnythingLLM + ollama with two models in mlock and it not be forced to dump at least one to keep the memory from going to swap - which will be very slow if that even is possible.

You need a better instance with more RAM or you can use the default embedding model, which runs on CPU and is more memory optimized, and from there you should be able to run Ollama and keep an LLM model in memory. Keep in mind you have 16GB across the whole instance - so your entire system and whatever it might be running all share that memory, and Ollama has access only to free memory.

Running a local model on CPU will be slow compared to GPU, but it can be done depending on your workload. However there are still perfomance considerations you need to consider during instance sizing to ensure everything in your stack can run in parallel with JIT unloading/reloading of models into memory.

32GB would be plenty here for such small models to be loaded in concurrency:
https://instances.vantage.sh/aws/ec2/c6i.4xlarge

Prior to resize, I would check ollama's logs to see if on each request if it is unloading and reloading models every request.
We already mlock models - https://github.com/Mintplex-Labs/anything-llm/blob/010eb6b124050399fb7eface3a2d3dee63ea4070/server/utils/AiProviders/ollama/index.js#L146 so the above is likely the case here

@timothycarambat commented on GitHub (Apr 14, 2025): As I replied to your other issue, you are using a moderately underpowered machine for this specific stack of models - if you are using RAG + this setup, you will see more tokens equal longer response times which getting 7-13 tok/s. There is no amount of software that AnythingLLM can write to make this faster. You can reduce the amount of context responses from 4->2, reduce the chat history window, swap to another quant of llama3.2, but end of the day you are limited by resources. > 🧠 OllamaAILLM initialized appears on every question — ⚠️ this means the model is being re-initialized repeatedly by AnythingLLM. This does not mean we are re-loading the model every request; this is just our wrapper. We run api requests to ollama so in this case when we ping the API the model is already loaded and inference begins immediately. You can also check this by the ollama logs, seeing whether reloading of the model into memory is occurring or not each request. Also, I notice you say you are using nomic for embedding - through ollama I assume. If that is the case and _you are using Ollama for both embedding and inference_ then what is likely occurring is that when we embed your prompt to do RAG, your limited RAM likely unloads llama, then when we try to run inference, we ask Ollama for LLM inference right after RAG search and it needs to reload the llama model again - this is all handled by Ollama automatically even if you specify `OLLAMA_KEEP_ALIVE_TIMEOUT`. Ollama cannot force a model to stay loaded if it needs to unload it to service a different request - like for nomic embedding. Ollama manages model loading, not AnythingLLM, in this case. With only 16GB of memory, I am willing to bet that is not enough to run AnythingLLM + ollama with two models in `mlock` and it not be forced to dump at least one to keep the memory from going to `swap` - which will be _very slow_ if that even is possible. You need a better instance with more RAM _or_ you can use the default embedding model, which runs on CPU and is more memory optimized, and from there you should be able to run Ollama and keep an LLM model in memory. Keep in mind you have 16GB _across the whole instance_ - so your entire system and whatever it might be running all share that memory, and Ollama has access only to free memory. Running a local model on CPU will be slow compared to GPU, but it can be done depending on your workload. However there are still perfomance considerations you need to consider during instance sizing to ensure everything in your stack can run in parallel with JIT unloading/reloading of models into memory. 32GB would be plenty here for such small models to be loaded in concurrency: https://instances.vantage.sh/aws/ec2/c6i.4xlarge Prior to resize, I would check ollama's logs to see if on each request if it is unloading and reloading models every request. We already `mlock` models - https://github.com/Mintplex-Labs/anything-llm/blob/010eb6b124050399fb7eface3a2d3dee63ea4070/server/utils/AiProviders/ollama/index.js#L146 so the above is likely the case here
yindo changed title from Performance issue to [GH-ISSUE #3646] Performance issue 2026-06-05 14:46:05 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2353
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?