Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3061] [FEAT]: Support for Intel NPU and include DeepSeek-R1, Janus-Pro & Qwen2.5-VL series of models for Qualcomm QNN LLM Provider #1965

New Issue
Closed
opened 2026-02-22 18:27:27 -05:00 by yindo · 6 comments
yindo commented 2026-02-22 18:27:27 -05:00
Owner
Copy Link
Copy Source

Originally created by @zytoh0 on GitHub (Jan 30, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3061

What would you like to see?

Support for Intel NPU and include DeepSeek-R1, Janus-Pro & Qwen2.5-VL series of models for Qualcomm QNN LLM Provider:

DeepSeek-R1 series: https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d

Janus-Pro series:
deepseek-ai/Janus-Pro-1B: https://huggingface.co/deepseek-ai/Janus-Pro-1B
deepseek-ai/Janus-Pro-7B: https://huggingface.co/deepseek-ai/Janus-Pro-7B

Qwen2.5-VL Series: https://huggingface.co/collections/Qwen/qwen25-vl-6795ffac22b334a837c0f9a5

Originally created by @zytoh0 on GitHub (Jan 30, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3061 ### What would you like to see? Support for Intel NPU and include DeepSeek-R1, Janus-Pro & Qwen2.5-VL series of models for Qualcomm QNN LLM Provider: DeepSeek-R1 series: https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d Janus-Pro series: deepseek-ai/Janus-Pro-1B: https://huggingface.co/deepseek-ai/Janus-Pro-1B deepseek-ai/Janus-Pro-7B: https://huggingface.co/deepseek-ai/Janus-Pro-7B Qwen2.5-VL Series: https://huggingface.co/collections/Qwen/qwen25-vl-6795ffac22b334a837c0f9a5
yindo added the enhancementfeature request labels 2026-02-22 18:27:27 -05:00
yindo closed this issue 2026-02-22 18:27:27 -05:00
yindo commented 2026-02-22 18:27:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 30, 2025):

While I am all for porting as many models as possible to NPU-compatibility, that is not what AnythingLLM does at its core. We will support them when they are converted by Qualcomm (or otherwise) so we can use them on NPU. It is a very intensive and tedious process to port models to run on NPU instruction set. I have yet to see a vision model running on NPU anywhere - not sure if you have sources for that.

Until then, they would have to run on CPU/RAM for ARM devices via tools like Ollama or LMStudio which can run on windows ARM64 as well as support GGUF runtimes, which is the only way to run these locally. Not even llama.cpp supports NPU natively, so we had to build our own inference engine to support this runtime. That is blocking easier adoption and mass conversion of models like what happens with model -> GGUF today.

Its early days for NPU compatibility - especially for new models that still need to be ported.

@timothycarambat commented on GitHub (Jan 30, 2025): While I am all for porting as many models as possible to NPU-compatibility, that is not what AnythingLLM does at its core. We will support them when they are converted by Qualcomm (or otherwise) so we can use them on NPU. It is a very intensive and tedious process to port models to run on NPU instruction set. I have yet to see a vision model running on NPU anywhere - not sure if you have sources for that. Until then, they would have to run on CPU/RAM for ARM devices via tools like Ollama or LMStudio which can run on windows ARM64 as well as support GGUF runtimes, which is the only way to run these locally. Not even llama.cpp supports NPU natively, so we had to build our own inference engine to support this runtime. That is blocking easier adoption and mass conversion of models like what happens with model -> GGUF today. Its early days for NPU compatibility - especially for new models that still need to be ported.
yindo commented 2026-02-22 18:27:27 -05:00
Author
Owner
Copy Link
Copy Source

@zytoh0 commented on GitHub (Jan 30, 2025):

While I am all for porting as many models as possible to NPU-compatibility, that is not what AnythingLLM does at its core. We will support them when they are converted by Qualcomm (or otherwise) so we can use them on NPU. It is a very intensive and tedious process to port models to run on NPU instruction set. I have yet to see a vision model running on NPU anywhere - not sure if you have sources for that.

Until then, they would have to run on CPU/RAM for ARM devices via tools like Ollama or LMStudio which can run on windows ARM64 as well as support GGUF runtimes, which is the only way to run these locally. Not even llama.cpp supports NPU natively, so we had to build our own inference engine to support this runtime. That is blocking easier adoption and mass conversion of models like what happens with model -> GGUF today.

Its early days for NPU compatibility - especially for new models that still need to be ported.

Hi @timothycarambat

I completely understand that AnythingLLM does not handle direct porting of models to NPUs and that it is a tedious process. However, since there is growing interest in running models on NPUs—especially with Qualcomm’s AI stack—I wanted to ask whether you could:

  1. Provide any documentation or insights into how models are ported to run on NPUs. If Qualcomm (or others) have any guidance, sharing it could help the broader community contribute.

  2. Open-source part of your inference engine or provide an API to extend it so the community can contribute to improving NPU compatibility. If that is not possible, perhaps outlining how your team approached NPU inference could help others build support.

  3. Clarify what specific roadblocks exist for models like DeepSeek-R1, Janus-Pro, or Qwen2.5-VL to run on NPUs. Are there specific architecture constraints, lack of toolchains, or other dependencies blocking this?

I believe that enabling community contributions could accelerate model porting efforts similar to how GGUF adoption has grown. Even if AnythingLLM doesn't do the model porting, giving the community a pathway to assist could be valuable.

Would love to hear your thoughts!

Thanks.

@zytoh0 commented on GitHub (Jan 30, 2025): > While I am all for porting as many models as possible to NPU-compatibility, that is not what AnythingLLM does at its core. We will support them when they are converted by Qualcomm (or otherwise) so we can use them on NPU. It is a very intensive and tedious process to port models to run on NPU instruction set. I have yet to see a vision model running on NPU anywhere - not sure if you have sources for that. > > Until then, they would have to run on CPU/RAM for ARM devices via tools like Ollama or LMStudio which can run on windows ARM64 as well as support GGUF runtimes, which is the only way to run these locally. Not even llama.cpp supports NPU natively, so we had to build our own inference engine to support this runtime. That is blocking easier adoption and mass conversion of models like what happens with model -> GGUF today. > > Its early days for NPU compatibility - especially for new models that still need to be ported. Hi @timothycarambat I completely understand that AnythingLLM does not handle direct porting of models to NPUs and that it is a tedious process. However, since there is growing interest in running models on NPUs—especially with Qualcomm’s AI stack—I wanted to ask whether you could: 1. Provide any documentation or insights into how models are ported to run on NPUs. If Qualcomm (or others) have any guidance, sharing it could help the broader community contribute. 2. Open-source part of your inference engine or provide an API to extend it so the community can contribute to improving NPU compatibility. If that is not possible, perhaps outlining how your team approached NPU inference could help others build support. 3. Clarify what specific roadblocks exist for models like DeepSeek-R1, Janus-Pro, or Qwen2.5-VL to run on NPUs. Are there specific architecture constraints, lack of toolchains, or other dependencies blocking this? I believe that enabling community contributions could accelerate model porting efforts similar to how GGUF adoption has grown. Even if AnythingLLM doesn't do the model porting, giving the community a pathway to assist could be valuable. Would love to hear your thoughts! Thanks.
yindo commented 2026-02-22 18:27:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 1, 2025):

Provide any documentation or insights into how models are ported to run on NPUs. If Qualcomm (or others) have any guidance, sharing it could help the broader community contribute.

Candidly, our service we have for getting models "ported" is all direct from Qualcomm themselves. That being said here are the resources we specifically leverage:

Open-source part of your inference engine or provide an API to extend it so the community can contribute to improving NPU compatibility. If that is not possible, perhaps outlining how your team approached NPU inference could help others build support.

This is exactly where we started - ChatApp (C++)
We rely on the Genie SDK to do inference - makes it super easy (also in c++)

Clarify what specific roadblocks exist for models like DeepSeek-R1, Janus-Pro, or Qwen2.5-VL to run on NPUs. Are there specific architecture constraints, lack of toolchains, or other dependencies blocking this?

This is a Qualcomm question. Like I know the NPU has a very specific way of "transforming" a pytorch or ONNX model to be able to run on NPU. This requires basically know the input and output shapes of the specific model you are trying to run. If a model has dynamic shape input - that is the thing that needs to change, among others.

This basically means each model you want to support needs it's own code to be loaded and run in a common interface. There is no transformers library like there is for python in C++. So you need to write your own tokenizer and any other nuanced parts of transforming inputs and outputs to make a coherent response.

LIke I mentioned, very tedious. I wish we could just load in a GGUF or ONNX and call it day, but that just simply isn't a reality. It is something the Dell AI Studio is promising.

@timothycarambat commented on GitHub (Feb 1, 2025): > Provide any documentation or insights into how models are ported to run on NPUs. If Qualcomm (or others) have any guidance, sharing it could help the broader community contribute. Candidly, our service we have for getting models "ported" is all direct from Qualcomm themselves. That being said here are the resources we specifically leverage: - [ONNX Execution provider](https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html) - [Qualcomm AI HUB](https://aihub.qualcomm.com/compute/models) (prebuild models) - [Deploying on device](https://github.com/quic/ai-hub-apps/tree/main/tutorials/llm_on_genie) > Open-source part of your inference engine or provide an API to extend it so the community can contribute to improving NPU compatibility. If that is not possible, perhaps outlining how your team approached NPU inference could help others build support. This is exactly where we started - [ChatApp](https://github.com/quic/ai-hub-apps/blob/main/apps/windows/cpp/ChatApp/README.md) (C++) We rely on the [Genie SDK](http://qualcomm.com/developer/software/gen-ai-inference-extensions) to do inference - makes it super easy (also in c++) > Clarify what specific roadblocks exist for models like DeepSeek-R1, Janus-Pro, or Qwen2.5-VL to run on NPUs. Are there specific architecture constraints, lack of toolchains, or other dependencies blocking this? This is a Qualcomm question. Like I know the NPU has a very specific way of "transforming" a pytorch or ONNX model to be able to run on NPU. This requires basically know the input and output shapes of the _specific_ model you are trying to run. If a model has dynamic shape input - that is the thing that needs to change, among others. This basically means each model you want to support needs it's own code to be loaded and run in a common interface. There is _no transformers_ library like there is for python in C++. So you need to write your own tokenizer and any other nuanced parts of transforming inputs and outputs to make a coherent response. LIke I mentioned, very tedious. I wish we could just load in a GGUF or ONNX and call it day, but that just simply isn't a reality. It is something the [Dell AI Studio is promising.](https://www.dell.com/en-us/dt/corporate/newsroom/announcements/detailpage.press-releases~usa~2025~01~dell-technologies-leads-ai-pc-movement-with-new-redesigned-pc-portfolio.htm#/filter-on/Country:en-us)
yindo commented 2026-02-22 18:27:27 -05:00
Author
Owner
Copy Link
Copy Source

@talynone commented on GitHub (Feb 2, 2025):

Looks like Microsoft has done work on it already:

https://blogs.windows.com/windowsdeveloper/2025/01/29/running-distilled-deepseek-r1-models-locally-on-copilot-pcs-powered-by-windows-copilot-runtime/

@talynone commented on GitHub (Feb 2, 2025): Looks like Microsoft has done work on it already: https://blogs.windows.com/windowsdeveloper/2025/01/29/running-distilled-deepseek-r1-models-locally-on-copilot-pcs-powered-by-windows-copilot-runtime/
yindo commented 2026-02-22 18:27:27 -05:00
Author
Owner
Copy Link
Copy Source

@fobrs commented on GitHub (Feb 2, 2025):

I have yet to see a vision model running on NPU anywhere - not sure if you have sources for that.

When YOLO is a vision model, then you can see it in action on a hexagon NPU by running this example

@fobrs commented on GitHub (Feb 2, 2025): > I have yet to see a vision model running on NPU anywhere - not sure if you have sources for that. When YOLO is a vision model, then you can see it in action on a hexagon NPU by running this [example](https://github.com/fobrs/yolov9_npu)
yindo commented 2026-02-22 18:27:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 3, 2025):

@fobrs Sorry, i wasnt being exact in my speech. I meant a multi-modal LLM. Not a model that can use image or do image detection.

@timothycarambat commented on GitHub (Feb 3, 2025): @fobrs Sorry, i wasnt being exact in my speech. I meant a multi-modal _LLM_. Not a model that can use image or do image detection.
yindo changed title from [FEAT]: Support for Intel NPU and include DeepSeek-R1, Janus-Pro & Qwen2.5-VL series of models for Qualcomm QNN LLM Provider to [GH-ISSUE #3061] [FEAT]: Support for Intel NPU and include DeepSeek-R1, Janus-Pro & Qwen2.5-VL series of models for Qualcomm QNN LLM Provider 2026-06-05 14:43:42 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1965
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?