Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1928] [BUG]: No GPU used for Native Bare Metal AnythingLLM Instance #1251

New Issue
Closed
opened 2026-02-22 18:23:55 -05:00 by yindo · 6 comments
yindo commented 2026-02-22 18:23:55 -05:00
Owner
Copy Link
Copy Source

Originally created by @omaralagouz on GitHub (Jul 23, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1928

How are you running AnythingLLM?

Local development

What happened?

I am using the Native option of a downloaded custom LLM (Mistral-7b-instruct-v0.2.Q5_K_M.gguf & Llama-2-7b-chat.Q5_K_M.gguf) but when i ask any question it takes about 1.5 minutes to get a response with RAG. I am running this in development on:

  • MS Windows 10 Pro
  • Processor: 12th Gen Intel(R) Core(TM) i9-12900T, 1400 MHz, 16 Core(s), 24 Logical Processor(s)
  • 64 GB RAM
  • GPU: NVIDIA T1000 8GB

Is there a way to use the GPU for inference, offloading a specific number of layers? I tried looking through the code but couldn't find where i can adjust this knowing that Llama.cpp is used in the background.

Are there known steps to reproduce?

No response

Originally created by @omaralagouz on GitHub (Jul 23, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1928 ### How are you running AnythingLLM? Local development ### What happened? I am using the Native option of a downloaded custom LLM (Mistral-7b-instruct-v0.2.Q5_K_M.gguf & Llama-2-7b-chat.Q5_K_M.gguf) but when i ask any question it takes about 1.5 minutes to get a response with RAG. I am running this in development on: - MS Windows 10 Pro - Processor: 12th Gen Intel(R) Core(TM) i9-12900T, 1400 MHz, 16 Core(s), 24 Logical Processor(s) - 64 GB RAM - GPU: NVIDIA T1000 8GB Is there a way to use the GPU for inference, offloading a specific number of layers? I tried looking through the code but couldn't find where i can adjust this knowing that Llama.cpp is used in the background. ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:23:55 -05:00
yindo closed this issue 2026-02-22 18:23:55 -05:00
yindo commented 2026-02-22 18:23:55 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 23, 2024):

Yeah, this is one of the pain points with the "native" llama.cpp module. Its becomes a lot more trouble to use directly vs just relying on the work of other local runners like Ollama or LMStudio so we will be deprecating the native runner in upcoming changes - unfortunately because of how the llama.cpp is built in-image it makes rebuilding so hard to do in-container to even hope to bind to the machines GPUs, plus it has no CUDA in the image so I imagine that wont even help :/

I highly recommend to swap out to another local LLM runner as we are going to remove that LLM provider soon because of issues like this

@timothycarambat commented on GitHub (Jul 23, 2024): Yeah, this is one of the pain points with the "native" llama.cpp module. Its becomes a lot more trouble to use directly vs just relying on the work of other local runners like Ollama or LMStudio so we will be deprecating the native runner in upcoming changes - unfortunately because of how the llama.cpp is built in-image it makes rebuilding so hard to do in-container to even _hope_ to bind to the machines GPUs, plus it has no CUDA in the image so I imagine that wont even help :/ I highly recommend to swap out to another local LLM runner as we are going to remove that LLM provider soon because of issues like this
yindo commented 2026-02-22 18:23:55 -05:00
Author
Owner
Copy Link
Copy Source

@omaralagouz commented on GitHub (Jul 23, 2024):

Yeah, this is one of the pain points with the "native" llama.cpp module. Its becomes a lot more trouble to use directly vs just relying on the work of other local runners like Ollama or LMStudio so we will be deprecating the native runner in upcoming changes - unfortunately because of how the llama.cpp is built in-image it makes rebuilding so hard to do in-container to even hope to bind to the machines GPUs, plus it has no CUDA in the image so I imagine that wont even help :/

I highly recommend to swap out to another local LLM runner as we are going to remove that LLM provider soon because of issues like this

The issue with switching to ollama or lmstudio is that the their server doesn’t allow for parallel API calls, which makes it so that it can’t be used as an application deployed somewhere for many users to log into and use simultaneously. If I’m wrong please correct me

@omaralagouz commented on GitHub (Jul 23, 2024): > Yeah, this is one of the pain points with the "native" llama.cpp module. Its becomes a lot more trouble to use directly vs just relying on the work of other local runners like Ollama or LMStudio so we will be deprecating the native runner in upcoming changes - unfortunately because of how the llama.cpp is built in-image it makes rebuilding so hard to do in-container to even _hope_ to bind to the machines GPUs, plus it has no CUDA in the image so I imagine that wont even help :/ > > I highly recommend to swap out to another local LLM runner as we are going to remove that LLM provider soon because of issues like this The issue with switching to ollama or lmstudio is that the their server doesn’t allow for parallel API calls, which makes it so that it can’t be used as an application deployed somewhere for many users to log into and use simultaneously. If I’m wrong please correct me
yindo commented 2026-02-22 18:23:56 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 23, 2024):

Ollama does now support parallelization/concurrency now as of 0.2.0

It previously did not, and LMStudio still does not

@timothycarambat commented on GitHub (Jul 23, 2024): Ollama does now support parallelization/concurrency [now as of 0.2.0](https://github.com/ollama/ollama/releases/tag/v0.2.0) It previously did not, and LMStudio still does not
yindo commented 2026-02-22 18:23:56 -05:00
Author
Owner
Copy Link
Copy Source

@omaralagouz commented on GitHub (Jul 24, 2024):

awesome! I downloaded ollama for windows and it's working great! It is in fact though using minimal GPU (4%), may you guide me as to how to increase the GPU usage? If im not mistaken it's to do with offloading more layers to the GPU? It was easier to do on LMStudio.

@omaralagouz commented on GitHub (Jul 24, 2024): awesome! I downloaded ollama for windows and it's working great! It is in fact though using minimal GPU (4%), may you guide me as to how to increase the GPU usage? If im not mistaken it's to do with offloading more layers to the GPU? It was easier to do on LMStudio.
yindo commented 2026-02-22 18:23:56 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 24, 2024):

Ollama should auto-bind to your GPU should the GPU support CUDA - they ship with 11.3 i think. What GPU and OS are you on? Closing this for now but we can keep the convo going.

https://github.com/ollama/ollama/blob/main/docs/gpu.md

@timothycarambat commented on GitHub (Jul 24, 2024): Ollama should auto-bind to your GPU should the GPU support CUDA - they ship with 11.3 i think. What GPU and OS are you on? Closing this for now but we can keep the convo going. https://github.com/ollama/ollama/blob/main/docs/gpu.md
yindo commented 2026-02-22 18:23:56 -05:00
Author
Owner
Copy Link
Copy Source

@omaralagouz commented on GitHub (Jul 26, 2024):

I'm on Windows 10, Lenovo Thinkstation with an Nvidia T1000 8GB GPU, The entire model is loaded onto the GPU memory (taking up 5-6 GB), and when I ask a question to the LLM the GPU usage % spikes to 100% then back to around 0%. I've tried setting the num_gpu parameter to -1, 10, 20, and 32 for Mistral, and all seem to be about the same.

@omaralagouz commented on GitHub (Jul 26, 2024): I'm on Windows 10, Lenovo Thinkstation with an Nvidia T1000 8GB GPU, The entire model is loaded onto the GPU memory (taking up 5-6 GB), and when I ask a question to the LLM the GPU usage % spikes to 100% then back to around 0%. I've tried setting the num_gpu parameter to -1, 10, 20, and 32 for Mistral, and all seem to be about the same.
yindo changed title from [BUG]: No GPU used for Native Bare Metal AnythingLLM Instance to [GH-ISSUE #1928] [BUG]: No GPU used for Native Bare Metal AnythingLLM Instance 2026-06-05 14:39:45 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1251
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?