Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3929] [FEAT]: Add Intel NPU Support & Optimize GPU Utilization (Hardware: Intel Ultra 7 255H + Arc 140T) #2500

New Issue
Closed
opened 2026-02-22 18:29:57 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:29:57 -05:00
Owner
Copy Link
Copy Source

Originally created by @BarneyRoos on GitHub (Jun 1, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3929

  1. System Environment
- Device: Redmi Book Pro 16 2025  
- OS: Windows 11 Home Chinese Edition (Build 23H2 or higher)  
- CPU: Intel® Core™ Ultra 7 255H (with integrated NPU)  
- GPU: Intel® Arc™ 140T (16GB VRAM)  
- RAM: 32GB LPDDR5X 8400MT/s  
- Storage: 1TB SSD  
- Installed Drivers:  
  - Intel® Graphics Driver 31.0.101.5599  
  - Intel® NPU Driver 1.0.11412.42501 (via OpenVINO™)

  1. Issue Description
    When running Deepseek-R1-8b model:
    NPU utilization remains 0% (No activity in Task Manager)
    GPU utilization peaks at 13% (Averages 5-8%, VRAM usage < 2GB)
    Excessive CPU/RAM pressure:
    CPU usage: 70-90%
    RAM usage: 22GB/32GB
    Low inference speed (~15 tokens/s)

  2. Feature Requests
    Please prioritize:

  • NPU Integration:
    Add support for Intel® OpenVINO™ NPU inference backend (Docs)
    OR implement BigDL-LLM NPU acceleration (see code sample below)
  • GPU Optimization:
    Improve DirectML/DX12 backend workload distribution
    Add explicit GPU layer offloading control (e.g., -ngl 40 parameter)
  • Hardware Auto-Detection:
    Enable hybrid acceleration (NPU+GPU) for Intel® Ultra platforms

Thanks very much!

Image

Originally created by @BarneyRoos on GitHub (Jun 1, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3929 1. System Environment ``` - Device: Redmi Book Pro 16 2025 - OS: Windows 11 Home Chinese Edition (Build 23H2 or higher) - CPU: Intel® Core™ Ultra 7 255H (with integrated NPU) - GPU: Intel® Arc™ 140T (16GB VRAM) - RAM: 32GB LPDDR5X 8400MT/s - Storage: 1TB SSD - Installed Drivers: - Intel® Graphics Driver 31.0.101.5599 - Intel® NPU Driver 1.0.11412.42501 (via OpenVINO™) ``` 2. Issue Description When running Deepseek-R1-8b model: NPU utilization remains 0% (No activity in Task Manager) GPU utilization peaks at 13% (Averages 5-8%, VRAM usage < 2GB) Excessive CPU/RAM pressure: CPU usage: 70-90% RAM usage: 22GB/32GB Low inference speed (~15 tokens/s) 3. Feature Requests Please prioritize: * NPU Integration: Add support for Intel® OpenVINO™ NPU inference backend ([Docs](https://docs.openvino.ai/2024/learn-openvino/configure-npu-device.html)) OR implement BigDL-LLM NPU acceleration (see code sample below) * GPU Optimization: Improve DirectML/DX12 backend workload distribution Add explicit GPU layer offloading control (e.g., -ngl 40 parameter) * Hardware Auto-Detection: Enable hybrid acceleration (NPU+GPU) for Intel® Ultra platforms Thanks very much! ![Image](https://github.com/user-attachments/assets/e2b74cba-275a-47ee-9a2a-d4ba327db7e3)
yindo added the enhancementfeature request labels 2026-02-22 18:29:57 -05:00
yindo closed this issue 2026-02-22 18:29:57 -05:00
yindo commented 2026-02-22 18:29:58 -05:00
Author
Owner
Copy Link
Copy Source

@digitalassassins commented on GitHub (Jun 2, 2025):

I have the Intel Ultra 9 285K, The NPU can push 18 AI TOPS, which is rubbish. The Intel Ultra 7 255H can only push 13 AI TOPS.

To put it into perspective, a £299 Nvidia RTX5060 can push 759 AI TOPS.

So the cheapest NVIDIA card on the market is 260x faster.

You probably wouldn't see any benefit from using the onboard NPU. It's that low, it's lower than the recommended minimum specifications for Windows Co-Pilot AI modifications built into Windows. You can't switch those features on in Windows because the NPU isn't powerful enough. You need an NPU with a minimum 40+ TOPS to switch on the features in Windows.

@digitalassassins commented on GitHub (Jun 2, 2025): I have the Intel Ultra 9 285K, The NPU can push 18 AI TOPS, which is rubbish. The Intel Ultra 7 255H can only push 13 AI TOPS. To put it into perspective, a £299 Nvidia RTX5060 can push 759 AI TOPS. So the cheapest NVIDIA card on the market is 260x faster. You probably wouldn't see any benefit from using the onboard NPU. It's that low, it's lower than the recommended minimum specifications for Windows Co-Pilot AI modifications built into Windows. You can't switch those features on in Windows because the NPU isn't powerful enough. You need an NPU with a minimum 40+ TOPS to switch on the features in Windows.
yindo commented 2026-02-22 18:29:58 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 3, 2025):

We only support dedicated NPU support on the Snapdragon X Elite. We do not have a dedicated engine for NPU inference with Intel. The only reason we even have the Snapdragon support is because they worked with us on it, but we are not going to build an engine for every NPU provider, especially since the effort is quite substantial and the nuance between silicon providers is enough nightmares for a single provider.

We are not an LLM engine and are an LLM tool suite, we will not be expanding NPU support to different silicon providers unless there is an incentive to do so.

However, you can install something like Foundry Local which will likely have Intel support (if not already) and can easily plug into AnythingLLM with ease. Then all of that gets paved over, and you get the best of hardware and tooling.

Also, @digitalassassins is 100% correct. Anything <40 TOPS will be horrible - even at the lowest quant, and other details you'll barely be working at 3 tok/sec with that level of tops. I can run Deepseek R1 (Qwen 1.5B distill) via ONNX on Snapdragon X Elite at 46 tok/sec. LLama 3.1 8B (via QNN) at like 20 tok/s.

Basically, yeah, you need better hardware, but also, that is not our core focus as well at AnythingLLM. For now, you are stuck with CPU inferencing on that build and even then you will need to stick to lower param models or high quantization to get decent speeds

@timothycarambat commented on GitHub (Jun 3, 2025): We only support dedicated NPU support on the Snapdragon X Elite. We do not have a dedicated engine for NPU inference with Intel. The only reason we even have the Snapdragon support is because they worked with us on it, but we are not going to build an engine for every NPU provider, especially since the effort is _quite substantial_ and the nuance between silicon providers is enough nightmares for a single provider. We are not an LLM engine and are an LLM tool suite, we will not be expanding NPU support to different silicon providers unless there is an incentive to do so. **However**, you can install something like [Foundry Local](https://github.com/microsoft/Foundry-Local/tree/v0.3.9267) which will likely have Intel support (if not already) and can easily plug into AnythingLLM with ease. Then all of that gets paved over, and you get the best of hardware and tooling. Also, @digitalassassins is 100% correct. Anything <40 TOPS will be _horrible_ - even at the lowest quant, and other details you'll barely be working at 3 tok/sec with that level of tops. I can run Deepseek R1 (Qwen 1.5B distill) via ONNX on Snapdragon X Elite at 46 tok/sec. LLama 3.1 8B (via QNN) at like 20 tok/s. Basically, yeah, you need better hardware, but also, that is not our core focus as well at AnythingLLM. For now, you are stuck with CPU inferencing on that build and even then you will need to stick to lower param models or high quantization to get decent speeds
yindo changed title from [FEAT]: Add Intel NPU Support & Optimize GPU Utilization (Hardware: Intel Ultra 7 255H + Arc 140T) to [GH-ISSUE #3929] [FEAT]: Add Intel NPU Support & Optimize GPU Utilization (Hardware: Intel Ultra 7 255H + Arc 140T) 2026-06-05 14:46:56 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2500
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?