Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1398] [BUG]: LLAMA3 not using GPU #892

New Issue
Closed
opened 2026-02-22 18:22:02 -05:00 by yindo · 12 comments
yindo commented 2026-02-22 18:22:02 -05:00
Owner
Copy Link
Copy Source

Originally created by @kalilfagundes on GitHub (May 14, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1398

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

Hello, everyone!
I installed the desktop app today (Windows) and runned it using llama3 7b. The question is that the model isn't using my GPU, but my CPU and I can't find the reason why. I already setted the executable to run on GPU in Nvidia Control Panel, but it didn't worked.
My GPU is Nvidia GeForce RTX 3050.

Are there known steps to reproduce?

No response

Originally created by @kalilfagundes on GitHub (May 14, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1398 ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? Hello, everyone! I installed the desktop app today (Windows) and runned it using llama3 7b. The question is that the model isn't using my GPU, but my CPU and I can't find the reason why. I already setted the executable to run on GPU in Nvidia Control Panel, but it didn't worked. My GPU is Nvidia GeForce RTX 3050. ### Are there known steps to reproduce? _No response_
yindo added the needs info / can't replicateDesktop labels 2026-02-22 18:22:02 -05:00
yindo closed this issue 2026-02-22 18:22:03 -05:00
yindo commented 2026-02-22 18:22:03 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 14, 2024):

The built in LLM engine for AnythingLLM is actually Ollama. Do you have Ollama on the same machine and does ollama outside of AnythingLLM use GPU?

If not, I'm thinking you may not have CUDA installed and the bindings are not being found to load and run models with GPU-layers

@timothycarambat commented on GitHub (May 14, 2024): The built in LLM engine for AnythingLLM is [actually Ollama](https://ollama.com). Do you have Ollama on the same machine and does ollama outside of AnythingLLM use GPU? If not, I'm thinking you may not have CUDA installed and the bindings are not being found to load and run models with GPU-layers
yindo commented 2026-02-22 18:22:03 -05:00
Author
Owner
Copy Link
Copy Source

@kalilfagundes commented on GitHub (May 14, 2024):

Title

I have no other Ollama instances and I have CUDA installed!

@kalilfagundes commented on GitHub (May 14, 2024): > Title I have no other Ollama instances and I have CUDA installed!
yindo commented 2026-02-22 18:22:03 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 14, 2024):

This seems like something Ollama needs to work on and not something we can manipulate directly via the built-in
https://github.com/ollama/ollama/issues/3201

It may be worth installing Ollama separately and using that as your LLM to fully leverage the GPU since it seems there is some kind of issues with that card/CUDA combination for native pickup. I'm on an NVIDIA 4090 and it finds my GPU and offloads accordingly

@timothycarambat commented on GitHub (May 14, 2024): This seems like something Ollama needs to work on and not something we can manipulate directly via the built-in https://github.com/ollama/ollama/issues/3201 It may be worth installing Ollama separately and using that as your LLM to fully leverage the GPU since it seems there is some kind of issues with that card/CUDA combination for native pickup. I'm on an NVIDIA 4090 and it finds my GPU and offloads accordingly
yindo commented 2026-02-22 18:22:03 -05:00
Author
Owner
Copy Link
Copy Source

@kalilfagundes commented on GitHub (May 14, 2024):

This seems like something Ollama needs to work on and not something we can manipulate directly via the built-in ollama/ollama#3201

It may be worth installing Ollama separately and using that as your LLM to fully leverage the GPU since it seems there is some kind of issues with that card/CUDA combination for native pickup. I'm on an NVIDIA 4090 and it finds my GPU and offloads accordingly

I did installed it separately and it seems that it's using my GPU now and acting faster. Good thing also since I will also integrate the Ollama with other softwares, it was the right decision.

Thanks, timothy.

@kalilfagundes commented on GitHub (May 14, 2024): > This seems like something Ollama needs to work on and not something we can manipulate directly via the built-in [ollama/ollama#3201](https://github.com/ollama/ollama/issues/3201) > > It may be worth installing Ollama separately and using that as your LLM to fully leverage the GPU since it seems there is some kind of issues with that card/CUDA combination for native pickup. I'm on an NVIDIA 4090 and it finds my GPU and offloads accordingly I did installed it separately and it seems that it's using my GPU now and acting faster. Good thing also since I will also integrate the Ollama with other softwares, it was the right decision. Thanks, timothy.
yindo commented 2026-02-22 18:22:03 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 14, 2024):

@kalilfagundes No problem, im sorry you came across that. The native GPU stuff can be tricky and our built-in makes things easy to get going but is certainly not foolproof, but there is always a way forward while staying local

@timothycarambat commented on GitHub (May 14, 2024): @kalilfagundes No problem, im sorry you came across that. The native GPU stuff can be tricky and our built-in makes things easy to get going but is certainly not foolproof, but there is always a way forward while staying local
yindo commented 2026-02-22 18:22:03 -05:00
Author
Owner
Copy Link
Copy Source

@Nododot commented on GitHub (May 28, 2024):

Same problem here, on Windows 11.
Ollama installed on WSL2 uses my Nvidia GPU, but AnythingLLM desktop app doesn't.
Tried llama3 and mistral, same outcome.
All software is up-to-date.
Nvidia GTX 1070, Ryzen 5600x.

@Nododot commented on GitHub (May 28, 2024): Same problem here, on Windows 11. Ollama installed on WSL2 uses my Nvidia GPU, but AnythingLLM desktop app doesn't. Tried llama3 and mistral, same outcome. All software is up-to-date. Nvidia GTX 1070, Ryzen 5600x.
yindo commented 2026-02-22 18:22:04 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 28, 2024):

@Nododot - the issue here is almost certainly being on WSL. If you use the Ollama windows app on the host machine outside WSL it will bind to GPUs, otherwise raise an issue on Ollama's repo as that would be out of scope for us anyway on this repo

@timothycarambat commented on GitHub (May 28, 2024): @Nododot - the issue here is almost certainly being on WSL. If you use the Ollama windows app on the host machine outside WSL it will bind to GPUs, otherwise raise an issue on [Ollama's repo](https://github.com/ollama/ollama) as that would be out of scope for us anyway on this repo
yindo commented 2026-02-22 18:22:04 -05:00
Author
Owner
Copy Link
Copy Source

@byzheng commented on GitHub (Jul 15, 2024):

anything-llm will use GPU for ollama just adding C:\Users\<user-name>\AppData\Local\Programs\AnythingLLM\resources\ollama to the TOP of environment variable PATH.

@byzheng commented on GitHub (Jul 15, 2024): anything-llm will use GPU for ollama just adding `C:\Users\<user-name>\AppData\Local\Programs\AnythingLLM\resources\ollama` to the TOP of environment variable `PATH`.
yindo commented 2026-02-22 18:22:04 -05:00
Author
Owner
Copy Link
Copy Source

@GF-Huang commented on GitHub (Sep 3, 2024):

Is this work as expected? High CPU high RAM high VRAM usage, but low GPU usage. Is it working on GPU?

image

I use this model.

image

And had add C:\Users\<user-name>\AppData\Local\Programs\AnythingLLM\resources\ollama to PATH.

@GF-Huang commented on GitHub (Sep 3, 2024): Is this work as expected? High CPU high RAM high VRAM usage, but low GPU usage. Is it working on GPU? ![image](https://github.com/user-attachments/assets/4748b174-13c2-493d-89b1-e869ba0ccde3) I use this model. ![image](https://github.com/user-attachments/assets/436fc48d-ccdf-4cc7-95e5-e67bdfe6bee3) And had add `C:\Users\<user-name>\AppData\Local\Programs\AnythingLLM\resources\ollama` to `PATH`.
yindo commented 2026-02-22 18:22:04 -05:00
Author
Owner
Copy Link
Copy Source

@1314XIAN commented on GitHub (Nov 27, 2024):

I also have the same problem, so I want to know how to using the GPU.
Because this I buy a new card, but it's using the cpu right now.

@1314XIAN commented on GitHub (Nov 27, 2024): I also have the same problem, so I want to know how to using the GPU. Because this I buy a new card, but it's using the cpu right now.
yindo commented 2026-02-22 18:22:04 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Nov 27, 2024):

We use Ollama to run on the GPU - did the ollama_lib files during installation install? You should have seen them download - this enables NVIDIA/AMD GPUs to work. Otherwise it will be stuck on CPU

@timothycarambat commented on GitHub (Nov 27, 2024): We use Ollama to run on the GPU - did the ollama_lib files during installation install? You should have seen them download - this enables NVIDIA/AMD GPUs to work. Otherwise it will be stuck on CPU
yindo commented 2026-02-22 18:22:04 -05:00
Author
Owner
Copy Link
Copy Source

@Aleksahek commented on GitHub (Dec 19, 2024):

how to use 100% gpu and 100% memory?

     
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC       |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+============|
|   0  NVIDIA GeForce RTX 2060        On  |   00000000:03:00.0  On |                  N/A |
| 29%   45C    P2             44W /  172W |    3992MiB /   6144MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=====================================================|
|    0   N/A  N/A     42153      C   ...rs/cuda_v12_avx/ollama_llama_server       3980MiB |
+-----------------------------------------------------------------------------------------+

llama3.3:70b

debian 12, cpu e5-2600, i do not use docker

@Aleksahek commented on GitHub (Dec 19, 2024): how to use 100% gpu and 100% memory? ``` +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 565.57.01 Driver Version: 565.57.01 CUDA Version: 12.7 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+============| | 0 NVIDIA GeForce RTX 2060 On | 00000000:03:00.0 On | N/A | | 29% 45C P2 44W / 172W | 3992MiB / 6144MiB | 1% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=====================================================| | 0 N/A N/A 42153 C ...rs/cuda_v12_avx/ollama_llama_server 3980MiB | +-----------------------------------------------------------------------------------------+ llama3.3:70b ``` debian 12, cpu e5-2600, i do not use docker
yindo changed title from [BUG]: LLAMA3 not using GPU to [GH-ISSUE #1398] [BUG]: LLAMA3 not using GPU 2026-06-05 14:37:47 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Desktop needs info / can't replicate
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#892
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?