Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2464] [BUG]: No response is returned, no errors in logs #1601

New Issue
Closed
opened 2026-02-22 18:25:36 -05:00 by yindo · 7 comments
yindo commented 2026-02-22 18:25:36 -05:00
Owner
Copy Link
Copy Source

Originally created by @JackTheTripperr on GitHub (Oct 13, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2464

Originally assigned to: @timothycarambat on GitHub.

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

I'm using a local installation of AnythingLLM on Windows 11 with APIpie via the generic OpenAI provider option. When I submit my prompt, I don't get a response. I'm not sure where things are getting hung up because I'm not seeing any errors when running debug or checking the logs. Additionally, I can see through my API usage that the API provider charged me for the completion, so I'm not entirely sure why the response isn't showing up.

I've tried various models (admittedly, all from APIpie), checked for firewall issues, you name it. Everything appears to be working correctly, I just don't see the response. Any help would be greatly appreciated!

{"level":"info","message":"\u001b[36m[EncryptionManager]\u001b[0m Loaded existing key & salt for encrypting arbitrary data.","service":"backend"}
{"level":"info","message":"\u001b[32m[TELEMETRY ENABLED]\u001b[0m Anonymous Telemetry enabled. Telemetry helps Mintplex Labs Inc improve AnythingLLM.","service":"backend"}
{"level":"info","message":"prisma:info Starting a sqlite pool with 17 connections.","service":"backend"}
{"level":"info","message":"\u001b[32m[TELEMETRY SENT]\u001b[0m {\"event\":\"server_boot\",\"distinctId\":\"f679be1b-d29e-437f-8524-49101d68923d\",\"properties\":{\"runtime\":\"desktop\"}}","service":"backend"}
{"level":"info","message":"Skipping preloading of AnythingLLMOllama - LLM_PROVIDER is generic-openai.","service":"backend"}
{"level":"info","message":"\u001b[36m[CommunicationKey]\u001b[0m RSA key pair generated for signed payloads within AnythingLLM services.","service":"backend"}
{"level":"info","message":"\u001b[36m[EncryptionManager]\u001b[0m Loaded existing key & salt for encrypting arbitrary data.","service":"backend"}
{"level":"info","message":"[production] AnythingLLM Standalone Backend listening on port 3001. Network discovery is enabled.","service":"backend"}
{"level":"info","message":"\u001b[36m[BackgroundWorkerService]\u001b[0m Feature is not enabled and will not be started.","service":"backend"}
{"level":"info","message":"\u001b[32m[Event Logged]\u001b[0m - update_llm_provider","service":"backend"}
{"level":"info","message":"\u001b[36m[NativeEmbedder]\u001b[0m Initialized","service":"backend"}
{"level":"info","message":"\u001b[32m[Event Logged]\u001b[0m - update_llm_provider","service":"backend"}
{"level":"info","message":"\u001b[36m[EncryptionManager]\u001b[0m Loaded existing key & salt for encrypting arbitrary data.","service":"backend"}
{"level":"info","message":"\u001b[32m[TELEMETRY ENABLED]\u001b[0m Anonymous Telemetry enabled. Telemetry helps Mintplex Labs Inc improve AnythingLLM.","service":"backend"}
{"level":"info","message":"prisma:info Starting a sqlite pool with 17 connections.","service":"backend"}
{"level":"info","message":"\u001b[32m[TELEMETRY SENT]\u001b[0m {\"event\":\"server_boot\",\"distinctId\":\"f679be1b-d29e-437f-8524-49101d68923d\",\"properties\":{\"runtime\":\"desktop\"}}","service":"backend"}
{"level":"info","message":"Skipping preloading of AnythingLLMOllama - LLM_PROVIDER is generic-openai.","service":"backend"}
{"level":"info","message":"\u001b[36m[CommunicationKey]\u001b[0m RSA key pair generated for signed payloads within AnythingLLM services.","service":"backend"}
{"level":"info","message":"\u001b[36m[EncryptionManager]\u001b[0m Loaded existing key & salt for encrypting arbitrary data.","service":"backend"}
{"level":"info","message":"[production] AnythingLLM Standalone Backend listening on port 3001. Network discovery is enabled.","service":"backend"}
{"level":"info","message":"\u001b[36m[BackgroundWorkerService]\u001b[0m Feature is not enabled and will not be started.","service":"backend"}
{"level":"info","message":"\u001b[36m[NativeEmbedder]\u001b[0m Initialized","service":"backend"}
{"level":"info","message":"\u001b[36m[uo]\u001b[0m Inference API: https://apipie.ai/v1/ Model: hermes-3-llama-3.1-405b","service":"backend"}
{"level":"info","message":"\u001b[32m[TELEMETRY SENT]\u001b[0m {\"event\":\"sent_chat\",\"distinctId\":\"f679be1b-d29e-437f-8524-49101d68923d\",\"properties\":{\"multiUserMode\":false,\"LLMSelection\":\"generic-openai\",\"Embedder\":\"native\",\"VectorDbSelection\":\"lancedb\",\"multiModal\":false,\"TTSSelection\":\"elevenlabs\",\"runtime\":\"desktop\"}}","service":"backend"}
{"level":"info","message":"\u001b[32m[Event Logged]\u001b[0m - sent_chat","service":"backend"}

screen01
screen02
screen03
screen04

Are there known steps to reproduce?

No response

Originally created by @JackTheTripperr on GitHub (Oct 13, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2464 Originally assigned to: @timothycarambat on GitHub. ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? I'm using a local installation of AnythingLLM on Windows 11 with APIpie via the generic OpenAI provider option. When I submit my prompt, I don't get a response. I'm not sure where things are getting hung up because I'm not seeing any errors when running debug or checking the logs. Additionally, I can see through my API usage that the API provider charged me for the completion, so I'm not entirely sure why the response isn't showing up. I've tried various models (admittedly, all from APIpie), checked for firewall issues, you name it. Everything _appears_ to be working correctly, I just don't _see_ the response. Any help would be greatly appreciated! ``` {"level":"info","message":"\u001b[36m[EncryptionManager]\u001b[0m Loaded existing key & salt for encrypting arbitrary data.","service":"backend"} {"level":"info","message":"\u001b[32m[TELEMETRY ENABLED]\u001b[0m Anonymous Telemetry enabled. Telemetry helps Mintplex Labs Inc improve AnythingLLM.","service":"backend"} {"level":"info","message":"prisma:info Starting a sqlite pool with 17 connections.","service":"backend"} {"level":"info","message":"\u001b[32m[TELEMETRY SENT]\u001b[0m {\"event\":\"server_boot\",\"distinctId\":\"f679be1b-d29e-437f-8524-49101d68923d\",\"properties\":{\"runtime\":\"desktop\"}}","service":"backend"} {"level":"info","message":"Skipping preloading of AnythingLLMOllama - LLM_PROVIDER is generic-openai.","service":"backend"} {"level":"info","message":"\u001b[36m[CommunicationKey]\u001b[0m RSA key pair generated for signed payloads within AnythingLLM services.","service":"backend"} {"level":"info","message":"\u001b[36m[EncryptionManager]\u001b[0m Loaded existing key & salt for encrypting arbitrary data.","service":"backend"} {"level":"info","message":"[production] AnythingLLM Standalone Backend listening on port 3001. Network discovery is enabled.","service":"backend"} {"level":"info","message":"\u001b[36m[BackgroundWorkerService]\u001b[0m Feature is not enabled and will not be started.","service":"backend"} {"level":"info","message":"\u001b[32m[Event Logged]\u001b[0m - update_llm_provider","service":"backend"} {"level":"info","message":"\u001b[36m[NativeEmbedder]\u001b[0m Initialized","service":"backend"} {"level":"info","message":"\u001b[32m[Event Logged]\u001b[0m - update_llm_provider","service":"backend"} {"level":"info","message":"\u001b[36m[EncryptionManager]\u001b[0m Loaded existing key & salt for encrypting arbitrary data.","service":"backend"} {"level":"info","message":"\u001b[32m[TELEMETRY ENABLED]\u001b[0m Anonymous Telemetry enabled. Telemetry helps Mintplex Labs Inc improve AnythingLLM.","service":"backend"} {"level":"info","message":"prisma:info Starting a sqlite pool with 17 connections.","service":"backend"} {"level":"info","message":"\u001b[32m[TELEMETRY SENT]\u001b[0m {\"event\":\"server_boot\",\"distinctId\":\"f679be1b-d29e-437f-8524-49101d68923d\",\"properties\":{\"runtime\":\"desktop\"}}","service":"backend"} {"level":"info","message":"Skipping preloading of AnythingLLMOllama - LLM_PROVIDER is generic-openai.","service":"backend"} {"level":"info","message":"\u001b[36m[CommunicationKey]\u001b[0m RSA key pair generated for signed payloads within AnythingLLM services.","service":"backend"} {"level":"info","message":"\u001b[36m[EncryptionManager]\u001b[0m Loaded existing key & salt for encrypting arbitrary data.","service":"backend"} {"level":"info","message":"[production] AnythingLLM Standalone Backend listening on port 3001. Network discovery is enabled.","service":"backend"} {"level":"info","message":"\u001b[36m[BackgroundWorkerService]\u001b[0m Feature is not enabled and will not be started.","service":"backend"} {"level":"info","message":"\u001b[36m[NativeEmbedder]\u001b[0m Initialized","service":"backend"} {"level":"info","message":"\u001b[36m[uo]\u001b[0m Inference API: https://apipie.ai/v1/ Model: hermes-3-llama-3.1-405b","service":"backend"} {"level":"info","message":"\u001b[32m[TELEMETRY SENT]\u001b[0m {\"event\":\"sent_chat\",\"distinctId\":\"f679be1b-d29e-437f-8524-49101d68923d\",\"properties\":{\"multiUserMode\":false,\"LLMSelection\":\"generic-openai\",\"Embedder\":\"native\",\"VectorDbSelection\":\"lancedb\",\"multiModal\":false,\"TTSSelection\":\"elevenlabs\",\"runtime\":\"desktop\"}}","service":"backend"} {"level":"info","message":"\u001b[32m[Event Logged]\u001b[0m - sent_chat","service":"backend"} ``` ![screen01](https://github.com/user-attachments/assets/966f3416-94eb-4291-bcf2-10094112aa16) ![screen02](https://github.com/user-attachments/assets/9be86cd2-a610-4813-a205-1fd0a970088d) ![screen03](https://github.com/user-attachments/assets/998cbc19-a6d0-4653-b6a4-69942f48d0a9) ![screen04](https://github.com/user-attachments/assets/d43fc7bc-ee8b-41ca-b30e-9dca82bfcf22) ### Are there known steps to reproduce? _No response_
yindo added the enhancementIntegration Requestfeature request labels 2026-02-22 18:25:36 -05:00
yindo closed this issue 2026-02-22 18:25:36 -05:00
yindo commented 2026-02-22 18:25:37 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 14, 2024):

I think this may be related to this feat request: https://github.com/Mintplex-Labs/anything-llm/issues/989

in general if its OpenAI compliant it should work with the generic connector as you just did in this issue, however its not always 100% handled and foolproof. I think its going to take us specifically handle that integration.

The root cause almost always seems to be stop token in the stop_reason of the response or delta missing in streaming responses. That is often why we then need to go and build a special handler. I see nothing you did wrong here, clearly just needs support for the LLM provider to be handled appropriately

@timothycarambat commented on GitHub (Oct 14, 2024): I think this may be related to this feat request: https://github.com/Mintplex-Labs/anything-llm/issues/989 _in general_ if its OpenAI compliant it should work with the generic connector as you just did in this issue, however its not always 100% handled and foolproof. I think its going to take us specifically handle that integration. The root cause almost _always_ seems to be stop token in the `stop_reason` of the response or `delta` missing in streaming responses. That is often why we then need to go and build a special handler. I see nothing you did wrong here, clearly just needs support for the LLM provider to be handled appropriately
yindo commented 2026-02-22 18:25:37 -05:00
Author
Owner
Copy Link
Copy Source

@JackTheTripperr commented on GitHub (Oct 14, 2024):

I think this may be related to this feat request: #989

in general if its OpenAI compliant it should work with the generic connector as you just did in this issue, however its not always 100% handled and foolproof. I think its going to take us specifically handle that integration.

The root cause almost always seems to be stop token in the stop_reason of the response or delta missing in streaming responses. That is often why we then need to go and build a special handler. I see nothing you did wrong here, clearly just needs support for the LLM provider to be handled appropriately

Thanks for taking the time to respond, I really appreciate it. I'll keep my eyes open for support in the future!

@JackTheTripperr commented on GitHub (Oct 14, 2024): > I think this may be related to this feat request: #989 > > _in general_ if its OpenAI compliant it should work with the generic connector as you just did in this issue, however its not always 100% handled and foolproof. I think its going to take us specifically handle that integration. > > The root cause almost _always_ seems to be stop token in the `stop_reason` of the response or `delta` missing in streaming responses. That is often why we then need to go and build a special handler. I see nothing you did wrong here, clearly just needs support for the LLM provider to be handled appropriately Thanks for taking the time to respond, I really appreciate it. I'll keep my eyes open for support in the future!
yindo commented 2026-02-22 18:25:37 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 14, 2024):

@JackTheTripperr - looking into this it seems to specifically impact openrouter models only and further, only when stream:true. I wouldn't want to merge a full integration unless streaming is supported. Oddly enough other providers work fine - its only openrouter models doing this.

@timothycarambat commented on GitHub (Oct 14, 2024): @JackTheTripperr - looking into this it seems to _specifically_ impact `openrouter` models only and further, only when `stream:true`. I wouldn't want to merge a full integration unless `streaming` is supported. Oddly enough other providers work fine - its only openrouter models doing this.
yindo commented 2026-02-22 18:25:37 -05:00
Author
Owner
Copy Link
Copy Source

@JackTheTripperr commented on GitHub (Oct 24, 2024):

Thanks for officially implementing this provider, I really appreciate it.

@JackTheTripperr commented on GitHub (Oct 24, 2024): Thanks for officially implementing this provider, I really appreciate it.
yindo commented 2026-02-22 18:25:37 -05:00
Author
Owner
Copy Link
Copy Source

@EncryptShawn commented on GitHub (Nov 5, 2024):

Hey guys, sorry we just saw this. We had some patches we pushed not long ago which likely resolved whatever the issue was. I pulled the latest AnythingLLM code and re-enabled the streaming support for APIpie and built and tested it and verified proper streaming as well as seeing the different providers being hit (to include openrouter) in the APIpie accounting. I think you guys should be good to enabled streaming support natively again.

Also btw, we make any provider/model that does not support streaming natively, stream anyway. We just chunk them up and send the chunks all at once at the end in a standard OpenAI format along with our extended usage data. To use only models that have been verified to provide proper streaming support (near immediate response) and support chat prompt structure for chat memory (rotating user/assistant support) you can call models route with ?xchat but again we make all providers/models stream compliant so should be no problem to have stream: true anywhere, its just a matter of if its streamed properly or all at once.

Also I have to say i really like what you guys are doing here. Your installer made this one of the most easiest local agent installs I have seen to date, that has to be beneficial for adoption. Its also a great locally run replacement for ChatGPT but with more capability. Thanks for what you guys are doing!

@EncryptShawn commented on GitHub (Nov 5, 2024): Hey guys, sorry we just saw this. We had some patches we pushed not long ago which likely resolved whatever the issue was. I pulled the latest AnythingLLM code and re-enabled the streaming support for APIpie and built and tested it and verified proper streaming as well as seeing the different providers being hit (to include openrouter) in the APIpie accounting. I think you guys should be good to enabled streaming support natively again. Also btw, we make any provider/model that does not support streaming natively, stream anyway. We just chunk them up and send the chunks all at once at the end in a standard OpenAI format along with our extended usage data. To use only models that have been verified to provide proper streaming support (near immediate response) and support chat prompt structure for chat memory (rotating user/assistant support) you can call models route with ?xchat but again we make all providers/models stream compliant so should be no problem to have stream: true anywhere, its just a matter of if its streamed properly or all at once. Also I have to say i really like what you guys are doing here. Your installer made this one of the most easiest local agent installs I have seen to date, that has to be beneficial for adoption. Its also a great locally run replacement for ChatGPT but with more capability. Thanks for what you guys are doing!
yindo commented 2026-02-22 18:25:37 -05:00
Author
Owner
Copy Link
Copy Source

@Toocky commented on GitHub (Nov 6, 2024):

@timothycarambat

Just want to reiterate what Shawn said Thanks for picking that up.

While we have you, I just wanted to touch more on what Shawn mentioned with the models route, as we have a wide range of offerings our models route is extremely large and without filtering is probably going to impact user experience.

Currently it looks like you are pulling the full models route including all Voice and Image models, and when you select one of these AnythingLLM wont work.

We have a number of different types & subtypes we have marked models as to help improve this and if you pull "https://apipie.ai/models?subtype=chatx" for the LLM configuration then you will get the smaller list of models that should give the best chat experience.

Likewise if you were to enable us as a Embedding Provider or a Transcription or Voice provider you could also pull the models specifically for those types

Also if you were interested in more adoption we have users actively seeking a application that uses our full offering including our image models.

You can read more about the models route here.
https://apipie.ai/docs/Features/Models

@Toocky commented on GitHub (Nov 6, 2024): @timothycarambat Just want to reiterate what Shawn said Thanks for picking that up. While we have you, I just wanted to touch more on what Shawn mentioned with the models route, as we have a wide range of offerings our models route is extremely large and without filtering is probably going to impact user experience. Currently it looks like you are pulling the full models route including all Voice and Image models, and when you select one of these AnythingLLM wont work. We have a number of different types & subtypes we have marked models as to help improve this and if you pull "https://apipie.ai/models?subtype=chatx" for the LLM configuration then you will get the smaller list of models that should give the best chat experience. Likewise if you were to enable us as a Embedding Provider or a Transcription or Voice provider you could also pull the models specifically for those types Also if you were interested in more adoption we have users actively seeking a application that uses our full offering including our image models. You can read more about the models route here. https://apipie.ai/docs/Features/Models
yindo commented 2026-02-22 18:25:37 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Nov 6, 2024):

@Toocky Ah, thank you for that link! I for sure missed it and was just hoping users would select the appropriate model for their use case and know better, which is for sure asking for random issues on Github 😆

I will break your comment into a new Issue(s) so we can improve the functionality.

Also @EncryptShawn thanks for looking into the streaming thing. We were really puzzled by it but didn't want to block it totally from being shipped

@timothycarambat commented on GitHub (Nov 6, 2024): @Toocky Ah, thank you for that link! I for sure missed it and was just hoping users would select the appropriate model for their use case and know better, which is for sure asking for random issues on Github 😆 I will break your comment into a new Issue(s) so we can improve the functionality. Also @EncryptShawn thanks for looking into the streaming thing. We were really puzzled by it but didn't want to block it totally from being shipped
yindo changed title from [BUG]: No response is returned, no errors in logs to [GH-ISSUE #2464] [BUG]: No response is returned, no errors in logs 2026-06-05 14:41:40 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Integration Request enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1601
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?