Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3869] [FEAT]: NVIDIA NIM Installer support for RTX 5070 Ti and other Blackwell GPUs #2471

New Issue
Closed
opened 2026-02-22 18:29:49 -05:00 by yindo · 8 comments
yindo commented 2026-02-22 18:29:49 -05:00
Owner
Copy Link
Copy Source

Originally created by @jhsmith409 on GitHub (May 23, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3869

What would you like to see?

When selecting the "Run NVIDIA NIM Installer" option under LLM Preference / LLM Provider / NVIDIA NIM in the windows desktop version v1.8.1-r2, the NVIDIA Package Launcher gives an error "NVIDIA Installer cannot continue" "Your current GPU is not compatible with NIM functionality." When searching for more detail on this error message, it seems that this NIM Setup Version 0.1.9 requires an NVIDIA 4080 or higher. Although the processing power of a 5070 Ti is close and the memory is the same, the 5070 Ti fails a check here. It seems the anythingllm docs also say that the 4080 Super fails as well. Issue is in the NVIDIA installer not anythingLLM but, is there a later version of the NIM setup that recognizes newer cards? I see reports that even the RTX 6000 Pro fails in 0.1.9 of the NVIDIA NIM installer. Of course all the Blackwell GPUs require CUDA 12.8 or later and pytorch 2.7.0 or later which may not be in that NIM... Looks like the April 2025 (25.04) and later releases from NVIDIA NIMs include compatibility with Blackwell.

For now, I would suggest adding a message within the anything-llm desktop app on compatibility. I'd be happy to add that code snippet but I'm not seeing the Desktop Client repository for the Electron app in this GitHub repository. Can anyone tell me where to look?

Originally created by @jhsmith409 on GitHub (May 23, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3869 ### What would you like to see? When selecting the "Run NVIDIA NIM Installer" option under LLM Preference / LLM Provider / NVIDIA NIM in the windows desktop version v1.8.1-r2, the NVIDIA Package Launcher gives an error "NVIDIA Installer cannot continue" "Your current GPU is not compatible with NIM functionality." When searching for more detail on this error message, it seems that this NIM Setup Version 0.1.9 requires an NVIDIA 4080 or higher. Although the processing power of a 5070 Ti is close and the memory is the same, the 5070 Ti fails a check here. It seems the anythingllm docs also say that the 4080 Super fails as well. Issue is in the NVIDIA installer not anythingLLM but, is there a later version of the NIM setup that recognizes newer cards? I see reports that even the RTX 6000 Pro fails in 0.1.9 of the NVIDIA NIM installer. Of course all the Blackwell GPUs require CUDA 12.8 or later and pytorch 2.7.0 or later which may not be in that NIM... Looks like the April 2025 (25.04) and later releases from NVIDIA NIMs include compatibility with Blackwell. For now, I would suggest adding a message within the anything-llm desktop app on compatibility. I'd be happy to add that code snippet but I'm not seeing the Desktop Client repository for the Electron app in this GitHub repository. Can anyone tell me where to look?
yindo added the enhancementfeature request labels 2026-02-22 18:29:49 -05:00
yindo closed this issue 2026-02-22 18:29:49 -05:00
yindo commented 2026-02-22 18:29:49 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 25, 2025):

Issue is in the NVIDIA installer not anythingLLM but, is there a later version of the NIM setup that recognizes newer cards? I

NVIDIA would have to provide this to us. Obviously, the newest cards should be supported and IIRC they are. I will send this directly to them as it should just work. Ill see what information I get from there.

The installer runs and just says card incompatible, correct?

@timothycarambat commented on GitHub (May 25, 2025): > Issue is in the NVIDIA installer not anythingLLM but, is there a later version of the NIM setup that recognizes newer cards? I NVIDIA would have to provide this to us. Obviously, the newest cards should be supported and IIRC they are. I will send this directly to them as it should just work. Ill see what information I get from there. The installer runs and just says card incompatible, correct?
yindo commented 2026-02-22 18:29:49 -05:00
Author
Owner
Copy Link
Copy Source

@jhsmith409 commented on GitHub (May 25, 2025):

Correct.  Issue is in NVIDIA NIM.  NVIDIA installer gives the message that GPU card is not supported.On May 25, 2025, at 12:23 PM, Timothy Carambat @.***> wrote:timothycarambat left a comment (Mintplex-Labs/anything-llm#3869)

Issue is in the NVIDIA installer not anythingLLM but, is there a later version of the NIM setup that recognizes newer cards? I

NVIDIA would have to provide this to us. Obviously, the newest cards should be supported and IIRC they are. I will send this directly to them as it should just work. Ill see what information I get from there.
The installer runs and just says card incompatible, correct?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

@jhsmith409 commented on GitHub (May 25, 2025): Correct.  Issue is in NVIDIA NIM.  NVIDIA installer gives the message that GPU card is not supported.On May 25, 2025, at 12:23 PM, Timothy Carambat ***@***.***> wrote:timothycarambat left a comment (Mintplex-Labs/anything-llm#3869) Issue is in the NVIDIA installer not anythingLLM but, is there a later version of the NIM setup that recognizes newer cards? I NVIDIA would have to provide this to us. Obviously, the newest cards should be supported and IIRC they are. I will send this directly to them as it should just work. Ill see what information I get from there. The installer runs and just says card incompatible, correct? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>
yindo commented 2026-02-22 18:29:49 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 26, 2025):

Correct. Issue is in NVIDIA NIM. NVIDIA installer gives the message that GPU card is not supported.

Gotcha, then yes they need to provide us an installer. I sent this to them already so hopefully they can give me some guidance here

@timothycarambat commented on GitHub (May 26, 2025): > Correct. Issue is in NVIDIA NIM. NVIDIA installer gives the message that GPU card is not supported. Gotcha, then yes they need to provide us an installer. I sent this to them already so hopefully they can give me some guidance here
yindo commented 2026-02-22 18:29:49 -05:00
Author
Owner
Copy Link
Copy Source

@jhsmith409 commented on GitHub (May 26, 2025):

In general I have found that any releases from NVIDIA prior to April 2025 do not support the Blackwell chipsets (SM-120). If the NIM is numbered 25-04 or later, it generally works with Blackwell. We are all waiting on NVIDIA…. I am so tired of trying to force pytorch 2.7.0 and cuda 12.8 (or 12.9) dependencies on every compile (these are both required for Blackwell).

On May 26, 2025, at 7:19 PM, Timothy Carambat @.***> wrote:

timothycarambat
left a comment
(Mintplex-Labs/anything-llm#3869)
https://github.com/Mintplex-Labs/anything-llm/issues/3869#issuecomment-2910752172
Correct. Issue is in NVIDIA NIM. NVIDIA installer gives the message that GPU card is not supported.

Gotcha, then yes they need to provide us an installer. I sent this to them already so hopefully they can give me some guidance here

—
Reply to this email directly, view it on GitHub https://github.com/Mintplex-Labs/anything-llm/issues/3869#issuecomment-2910752172, or unsubscribe https://github.com/notifications/unsubscribe-auth/BPIRS5ZYFG7RD4LVPPBBWSL3AOOPFAVCNFSM6AAAAAB5YUVJ72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMJQG42TEMJXGI.
You are receiving this because you authored the thread.

@jhsmith409 commented on GitHub (May 26, 2025): In general I have found that any releases from NVIDIA prior to April 2025 do not support the Blackwell chipsets (SM-120). If the NIM is numbered 25-04 or later, it generally works with Blackwell. We are all waiting on NVIDIA…. I am so tired of trying to force pytorch 2.7.0 and cuda 12.8 (or 12.9) dependencies on every compile (these are both required for Blackwell). > On May 26, 2025, at 7:19 PM, Timothy Carambat ***@***.***> wrote: > > > timothycarambat > left a comment > (Mintplex-Labs/anything-llm#3869) > <https://github.com/Mintplex-Labs/anything-llm/issues/3869#issuecomment-2910752172> > Correct. Issue is in NVIDIA NIM. NVIDIA installer gives the message that GPU card is not supported. > > Gotcha, then yes they need to provide us an installer. I sent this to them already so hopefully they can give me some guidance here > > — > Reply to this email directly, view it on GitHub <https://github.com/Mintplex-Labs/anything-llm/issues/3869#issuecomment-2910752172>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPIRS5ZYFG7RD4LVPPBBWSL3AOOPFAVCNFSM6AAAAAB5YUVJ72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMJQG42TEMJXGI>. > You are receiving this because you authored the thread. >
yindo commented 2026-02-22 18:29:50 -05:00
Author
Owner
Copy Link
Copy Source

@joshin42 commented on GitHub (Jun 9, 2025):

Any word back from them?

Correct. Issue is in NVIDIA NIM. NVIDIA installer gives the message that GPU card is not supported.

Gotcha, then yes they need to provide us an installer. I sent this to them already so hopefully they can give me some guidance here

@joshin42 commented on GitHub (Jun 9, 2025): Any word back from them? > > Correct. Issue is in NVIDIA NIM. NVIDIA installer gives the message that GPU card is not supported. > > Gotcha, then yes they need to provide us an installer. I sent this to them already so hopefully they can give me some guidance here
yindo commented 2026-02-22 18:29:50 -05:00
Author
Owner
Copy Link
Copy Source

@jhsmith409 commented on GitHub (Jun 9, 2025):

I'm no longer waiting on NVIDIA... The latest releases of Ollama, llama.cpp, and vLLM now all support Blackwell/SM-120 through CUDA 12.8 or later. NVIDIA releases with version numbes 25.04 seem, in general, to support it but there are not many of them. So for now, run ollama, vLLM, llama.cpp (or LM Studio which uses llama.cpp).

@jhsmith409 commented on GitHub (Jun 9, 2025): I'm no longer waiting on NVIDIA... The latest releases of Ollama, llama.cpp, and vLLM now all support Blackwell/SM-120 through CUDA 12.8 or later. NVIDIA releases with version numbes 25.04 seem, in general, to support it but there are not many of them. So for now, run ollama, vLLM, llama.cpp (or LM Studio which uses llama.cpp).
yindo commented 2026-02-22 18:29:50 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 10, 2025):

FWIW, I did not get a direct response back on the matter from them to date

@timothycarambat commented on GitHub (Jun 10, 2025): FWIW, I did not get a direct response back on the matter from them to date
yindo commented 2026-02-22 18:29:50 -05:00
Author
Owner
Copy Link
Copy Source

@CryptoRola commented on GitHub (Jul 20, 2025):

just wondering is there any news or updates regarding this matter?
I just bought a rtx5060ti and have fallen into to the pickle, even though I was assured that the NIM would run on my newly purchased gpu.
Yes, the RTX 5060 Ti can run NIMs (NVIDIA Inference Microservices) because it's built with the Blackwell architecture, which includes specialized hardware like faster Tensor Cores and dedicated AI processing units that are optimized for AI workloads. These features, combined with the overall performance improvements of the Blackwell architecture, allow the 5060 Ti to handle the computational demands of NIMs effectively.

@CryptoRola commented on GitHub (Jul 20, 2025): just wondering is there any news or updates regarding this matter? I just bought a rtx5060ti and have fallen into to the pickle, even though I was assured that the NIM would run on my newly purchased gpu. Yes, the RTX 5060 Ti can run NIMs (NVIDIA Inference Microservices) because it's built with the Blackwell architecture, which includes specialized hardware like faster Tensor Cores and dedicated AI processing units that are optimized for AI workloads. These features, combined with the overall performance improvements of the Blackwell architecture, allow the 5060 Ti to handle the computational demands of NIMs effectively.
yindo changed title from [FEAT]: NVIDIA NIM Installer support for RTX 5070 Ti and other Blackwell GPUs to [GH-ISSUE #3869] [FEAT]: NVIDIA NIM Installer support for RTX 5070 Ti and other Blackwell GPUs 2026-06-05 14:46:44 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2471
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?