Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4297] [BUG]: Error: 503 status code (no body) at APIError.generate #2735

New Issue
Closed
opened 2026-02-22 18:31:00 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:31:00 -05:00
Owner
Copy Link
Copy Source

Originally created by @scara1701 on GitHub (Aug 15, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4297

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

Hello,

I am trying to use Anything LLM on my snapdragon machine. Specifically I want to try using the NPU in the laptop. Unfortunately I am unable to send a message to the LLM.

The LLM is downloaded and seems to be extracted into the QNN folder.
I have tried:

  • llama_v3_2_3b_chat_16k
  • phi_3_5_mini_instruct_4k

{"level":"info","message":"\u001b[36m[QnnNativeEmbedder]\u001b[0m Checking for valid NPU cores.","service":"backend"}
{"level":"info","message":"\u001b[36m[QnnNativeEmbedder]\u001b[0m Checking core Snapdragon(R) X Elite - X1E78100 - Qualcomm(R) Oryon(TM) CPU - MATCH","service":"backend"}
{"level":"info","message":"\u001b[36m[QnnNativeEmbedder]\u001b[0m Initialized all-minilm-l6-v2 model","service":"backend"}
{"level":"info","message":"\u001b[36m[QNN Engine]\u001b[0m Initialized with model: llama_v3_2_3b_chat_16k. Context window: 16384","service":"backend"}
{"level":"info","message":"\u001b[35m[TokenManager]\u001b[0m Returning existing instance for model: C:\Users\gwend\AppData\Roaming\anythingllm-desktop\storage\models\QNN\llama_v3_2_3b_chat_16k","service":"backend"}
{"level":"error","message":"Error: 503 status code (no body)\n at APIError.generate (C:\Users\gwend\AppData\Local\Programs\AnythingLLM\resources\backend\node_modules\openai\error.js:66:20)\n at OpenAI.makeStatusError (C:\Users\gwend\AppData\Local\Programs\AnythingLLM\resources\backend\node_modules\openai\core.js:302:33)\n at OpenAI.makeRequest (C:\Users\gwend\AppData\Local\Programs\AnythingLLM\resources\backend\node_modules\openai\core.js:346:30)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Na.measureStream (C:\Users\gwend\AppData\Local\Programs\AnythingLLM\resources\backend\server.js:21:8903)\n at async r.streamGetChatCompletion (C:\Users\gwend\AppData\Local\Programs\AnythingLLM\resources\backend\server.js:43:2967)\n at async gO (C:\Users\gwend\AppData\Local\Programs\AnythingLLM\resources\backend\server.js:288:3125)\n at async C:\Users\gwend\AppData\Local\Programs\AnythingLLM\resources\backend\server.js:288:4762","service":"backend"}

Are there known steps to reproduce?

No response

Originally created by @scara1701 on GitHub (Aug 15, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4297 ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? Hello, I am trying to use Anything LLM on my snapdragon machine. Specifically I want to try using the NPU in the laptop. Unfortunately I am unable to send a message to the LLM. The LLM is downloaded and seems to be extracted into the QNN folder. I have tried: - llama_v3_2_3b_chat_16k - phi_3_5_mini_instruct_4k > {"level":"info","message":"\u001b[36m[QnnNativeEmbedder]\u001b[0m Checking for valid NPU cores.","service":"backend"} {"level":"info","message":"\u001b[36m[QnnNativeEmbedder]\u001b[0m Checking core Snapdragon(R) X Elite - X1E78100 - Qualcomm(R) Oryon(TM) CPU - MATCH","service":"backend"} {"level":"info","message":"\u001b[36m[QnnNativeEmbedder]\u001b[0m Initialized all-minilm-l6-v2 model","service":"backend"} {"level":"info","message":"\u001b[36m[QNN Engine]\u001b[0m Initialized with model: llama_v3_2_3b_chat_16k. Context window: 16384","service":"backend"} {"level":"info","message":"\u001b[35m[TokenManager]\u001b[0m Returning existing instance for model: C:\\Users\\gwend\\AppData\\Roaming\\anythingllm-desktop\\storage\\models\\QNN\\llama_v3_2_3b_chat_16k","service":"backend"} {"level":"error","message":"Error: 503 status code (no body)\n at APIError.generate (C:\\Users\\gwend\\AppData\\Local\\Programs\\AnythingLLM\\resources\\backend\\node_modules\\openai\\error.js:66:20)\n at OpenAI.makeStatusError (C:\\Users\\gwend\\AppData\\Local\\Programs\\AnythingLLM\\resources\\backend\\node_modules\\openai\\core.js:302:33)\n at OpenAI.makeRequest (C:\\Users\\gwend\\AppData\\Local\\Programs\\AnythingLLM\\resources\\backend\\node_modules\\openai\\core.js:346:30)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Na.measureStream (C:\\Users\\gwend\\AppData\\Local\\Programs\\AnythingLLM\\resources\\backend\\server.js:21:8903)\n at async r.streamGetChatCompletion (C:\\Users\\gwend\\AppData\\Local\\Programs\\AnythingLLM\\resources\\backend\\server.js:43:2967)\n at async gO (C:\\Users\\gwend\\AppData\\Local\\Programs\\AnythingLLM\\resources\\backend\\server.js:288:3125)\n at async C:\\Users\\gwend\\AppData\\Local\\Programs\\AnythingLLM\\resources\\backend\\server.js:288:4762","service":"backend"} ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:31:00 -05:00
yindo closed this issue 2026-02-22 18:31:00 -05:00
yindo commented 2026-02-22 18:31:01 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 15, 2025):

llama_v3_2_3b_chat_16k

This is the main issue. If you swap to a smaller model (like the 8k) you should be fine. When the system boots the NPU engine we have to load the entire model into the NPU share memory in order to get inference. On machines with ~15gb RAM this can fail silently or just take honestly forever (resulting in 503's).

We will be removing the 16k context from the app because it takes forever to load in and loading models into the NPU memory takes a lot of time even as-is.

@timothycarambat commented on GitHub (Aug 15, 2025): > llama_v3_2_3b_chat_16k This is the main issue. If you swap to a smaller model (like the 8k) you should be fine. When the system boots the NPU engine we have to load the _entire_ model into the NPU share memory in order to get inference. On machines with ~15gb RAM this can fail silently or just take honestly forever (resulting in 503's). We will be removing the 16k context from the app because it takes forever to load in and loading models into the NPU memory takes a lot of time even as-is.
yindo commented 2026-02-22 18:31:01 -05:00
Author
Owner
Copy Link
Copy Source

@scara1701 commented on GitHub (Aug 16, 2025):

I don't know why, but today it seems to work. Perhaps something just needed a reboot.. :)
If you need any testing, my Snapdragon machine has 32GB of system memory.

Image
@scara1701 commented on GitHub (Aug 16, 2025): I don't know why, but today it seems to work. Perhaps something just needed a reboot.. :) If you need any testing, my Snapdragon machine has 32GB of system memory. <img width="1472" height="920" alt="Image" src="https://github.com/user-attachments/assets/2e5db2de-5450-442b-bb84-e4ffa4454652" />
yindo commented 2026-02-22 18:31:01 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 17, 2025):

I don't know why, but today it seems to work. Perhaps something just needed a reboot.. :)

For what it is worth, we have seen the backend server for the QNN engine do two things on various devices:

  • Take up 20% CPU when at idle
  • Take longer to bootup after device has been "deep sleeping" or Hibernating

Usually, a reboot resolves everything. I once had a machine that sat for a few days, started the app + inference the NPU would not go past 30%. Nothing else was running. Rebooted the machine and have been totally unable to replicate it since.

My hunch is random OS/OEM background jobs or energy management processes or something but its basically a black box. In general, restarting the app or even a log out/log in fixes any issue. Very frustrating as things like this are what keep the ARM64 build in beta.

@timothycarambat commented on GitHub (Aug 17, 2025): > I don't know why, but today it seems to work. Perhaps something just needed a reboot.. :) For what it is worth, we have seen the backend server for the QNN engine do two things on various devices: - Take up 20% CPU when at idle - Take longer to bootup after device has been "deep sleeping" or Hibernating _Usually,_ a reboot resolves everything. I once had a machine that sat for a few days, started the app + inference the NPU would not go past 30%. Nothing else was running. Rebooted the machine and have been totally unable to replicate it since. My hunch is random OS/OEM background jobs or energy management processes or something but its basically a black box. In general, restarting the app or even a log out/log in fixes any issue. Very frustrating as things like this are what keep the ARM64 build in beta.
yindo changed title from [BUG]: Error: 503 status code (no body) at APIError.generate to [GH-ISSUE #4297] [BUG]: Error: 503 status code (no body) at APIError.generate 2026-06-05 14:48:14 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2735
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?