Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #485] Incomplete response from LM studio endpoint #278

New Issue
Closed
opened 2026-02-22 18:18:41 -05:00 by yindo · 16 comments
yindo commented 2026-02-22 18:18:41 -05:00
Owner
Copy Link
Copy Source

Originally created by @unseenscholar on GitHub (Dec 21, 2023).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/485

I am getting incomplete responses while using the LM studio endpoint. The response cuts off midway while streaming, sometimes after the first word or after half a sentence. I am running on docker.

Originally created by @unseenscholar on GitHub (Dec 21, 2023). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/485 I am getting incomplete responses while using the LM studio endpoint. The response cuts off midway while streaming, sometimes after the first word or after half a sentence. I am running on docker.
yindo added the questionIntegration Request labels 2026-02-22 18:18:41 -05:00
yindo closed this issue 2026-02-22 18:18:41 -05:00
yindo commented 2026-02-22 18:18:42 -05:00
Author
Owner
Copy Link
Copy Source

@shatfield4 commented on GitHub (Dec 21, 2023):

Hi @Daniel-Dan-Espinoza, which model are you using in LM Studio? This typically happens when models that are less optimized for chatting are being used. Also, are you running LM Studio locally on the same machine as your AnythingLLM docker container?

@shatfield4 commented on GitHub (Dec 21, 2023): Hi @Daniel-Dan-Espinoza, which model are you using in LM Studio? This typically happens when models that are less optimized for chatting are being used. Also, are you running LM Studio locally on the same machine as your AnythingLLM docker container?
yindo commented 2026-02-22 18:18:42 -05:00
Author
Owner
Copy Link
Copy Source

@unseenscholar commented on GitHub (Dec 22, 2023):

I was using Starling model. I am running both LM Studio and AnythingLLM docker container on the same machine.

@unseenscholar commented on GitHub (Dec 22, 2023): I was using Starling model. I am running both LM Studio and AnythingLLM docker container on the same machine.
yindo commented 2026-02-22 18:18:42 -05:00
Author
Owner
Copy Link
Copy Source

@unseenscholar commented on GitHub (Dec 22, 2023):

I tried Local AI but have the same issue

@unseenscholar commented on GitHub (Dec 22, 2023): I tried Local AI but have the same issue
yindo commented 2026-02-22 18:18:42 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Dec 26, 2023):

When interacting with LM Studio we leave the entire run of inferencing on the LM Studio side. We simply pass along the input and wait for LM Studio to be done with output.

When an inferencing is running Its likely the output being sent to AnythingLLM is not being dropped, but that LM Studio stops generating output and AnythingLLM assumes the response to be done.

Can you ensure that the model is not continuing to generate response when AnythingLLM says the response is complete? This would help determine if the issue is with the model/config on LM Studio or AnythingLLM

@timothycarambat commented on GitHub (Dec 26, 2023): When interacting with LM Studio we leave the entire run of inferencing on the LM Studio side. We simply pass along the input and wait for LM Studio to be done with output. When an inferencing is running Its likely the output being sent to AnythingLLM is not being dropped, but that LM Studio stops generating output and AnythingLLM assumes the response to be done. Can you ensure that the model is not continuing to generate response when AnythingLLM says the response is complete? This would help determine if the issue is with the model/config on LM Studio or AnythingLLM
yindo commented 2026-02-22 18:18:42 -05:00
Author
Owner
Copy Link
Copy Source

@lunamidori5 commented on GitHub (Dec 29, 2023):

@timothycarambat this is the streaming bug fix for localai we added. This is the fix working but we need to learn why its dropping the packets

@lunamidori5 commented on GitHub (Dec 29, 2023): @timothycarambat this is the streaming bug fix for localai we added. This is the fix working but we need to learn why its dropping the packets
yindo commented 2026-02-22 18:18:42 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Dec 29, 2023):

@lunamidori5 We would need to confirm that the user is running the patched version, and if so then yes for sure need to see why. To be fair, I have yet to replicate this issue with LocalAi (or LM Studio for that matter)

@timothycarambat commented on GitHub (Dec 29, 2023): @lunamidori5 We would need to confirm that the user is running the patched version, and if so then yes for sure need to see why. To be fair, I have yet to replicate this issue with LocalAi (or LM Studio for that matter)
yindo commented 2026-02-22 18:18:42 -05:00
Author
Owner
Copy Link
Copy Source

@lunamidori5 commented on GitHub (Dec 29, 2023):

@timothycarambat at least im not the only one with this bug (I am starting to think it maybe the way some routers work...)

@lunamidori5 commented on GitHub (Dec 29, 2023): @timothycarambat at least im not the only one with this bug (I am starting to think it maybe the way some routers work...)
yindo commented 2026-02-22 18:18:42 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 16, 2024):

Closing as stale

@timothycarambat commented on GitHub (Jan 16, 2024): Closing as stale
yindo commented 2026-02-22 18:18:43 -05:00
Author
Owner
Copy Link
Copy Source

@dlaliberte commented on GitHub (Feb 11, 2024):

I am seeing this problem, using the latest version of AnythingLLM (0.2.0?). I saw it when using LM Studio, but then it seemed to clear up on its own, or maybe it was after I reset the chat in AnythingLLM. Then I got the empty content complaint from LM Studio, and I decided enough is enough, and I switched to Kobold. Now I am seeing the one token problem using Kobold, via the Local AI LLM setting in AnythingLLM (chat model selection: (which I can't seem to copy from the form, sigh...) koboldcpp/dolphin-2.2.1-mistral-7b.Q5_K_S). Resetting the chat history doesn't help.

I'm fine with disabling the streaming mode, for now. I don't see any way to do that, either in AnythingLLM or Kobold.

Looks like this might be a relevant issue: https://github.com/LostRuins/koboldcpp/issues/669 So it may be a bug on the Kobold side.

@dlaliberte commented on GitHub (Feb 11, 2024): I am seeing this problem, using the latest version of AnythingLLM (0.2.0?). I saw it when using LM Studio, but then it seemed to clear up on its own, or maybe it was after I reset the chat in AnythingLLM. Then I got the empty content complaint from LM Studio, and I decided enough is enough, and I switched to Kobold. Now I am seeing the one token problem using Kobold, via the Local AI LLM setting in AnythingLLM (chat model selection: (which I can't seem to copy from the form, sigh...) koboldcpp/dolphin-2.2.1-mistral-7b.Q5_K_S). Resetting the chat history doesn't help. I'm fine with disabling the streaming mode, for now. I don't see any way to do that, either in AnythingLLM or Kobold. Looks like this might be a relevant issue: https://github.com/LostRuins/koboldcpp/issues/669 So it may be a bug on the Kobold side.
yindo commented 2026-02-22 18:18:43 -05:00
Author
Owner
Copy Link
Copy Source

@lunamidori5 commented on GitHub (Feb 12, 2024):

@timothycarambat could you add a "no streaming" check mark to the llm screen?

@lunamidori5 commented on GitHub (Feb 12, 2024): @timothycarambat could you add a "no streaming" check mark to the llm screen?
yindo commented 2026-02-22 18:18:43 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 12, 2024):

Is it because there is an issue with streaming or because certain models do not support it?

@timothycarambat commented on GitHub (Feb 12, 2024): Is it because there is an issue with streaming or because certain models do not support it?
yindo commented 2026-02-22 18:18:43 -05:00
Author
Owner
Copy Link
Copy Source

@dlaliberte commented on GitHub (Feb 13, 2024):

Is it because there is an issue with streaming or because certain models do not support it?

With Kobold, I was seeing the whole stream of tokens being generated, so clearly the model supports streaming, and Kobold supports streaming, but from the AnythingLLM side, it was already done after the first token came in. So it seems like a timeout issue, not waiting long enough for the next token? It seems there should be a special signal that the stream is finished because, otherwise, how would anyone know?

@dlaliberte commented on GitHub (Feb 13, 2024): > Is it because there is an issue with streaming or because certain models do not support it? With Kobold, I was seeing the whole stream of tokens being generated, so clearly the model supports streaming, and Kobold supports streaming, but from the AnythingLLM side, it was already done after the first token came in. So it seems like a timeout issue, not waiting long enough for the next token? It seems there should be a special signal that the stream is finished because, otherwise, how would anyone know?
yindo commented 2026-02-22 18:18:43 -05:00
Author
Owner
Copy Link
Copy Source

@lunamidori5 commented on GitHub (Feb 14, 2024):

Is it because there is an issue with streaming or because certain models do not support it?

LocalAI and Google Gem Still have that streaming bug from before...

@lunamidori5 commented on GitHub (Feb 14, 2024): > Is it because there is an issue with streaming or because certain models do not support it? LocalAI and Google Gem Still have that streaming bug from before...
yindo commented 2026-02-22 18:18:43 -05:00
Author
Owner
Copy Link
Copy Source

@yoonghm commented on GitHub (Feb 5, 2025):

I encountered the same issue. My LM Studio version is 0.3.9, and the model is DeepSeek-Coder-V2-Lite-nstruct-GGUF.

@yoonghm commented on GitHub (Feb 5, 2025): I encountered the same issue. My LM Studio version is 0.3.9, and the model is DeepSeek-Coder-V2-Lite-nstruct-GGUF.
yindo commented 2026-02-22 18:18:43 -05:00
Author
Owner
Copy Link
Copy Source

@jwiegley commented on GitHub (Apr 8, 2025):

For what it's worth, I'm having the same problem, using curl as the query tool:

curl \
--location \
-v \
-D- \
-d\{\"model\"\:\"deepseek-v3-0324\"\,\"messages\"\:\[\{\"role\"\:\"system\"\,\"content\"\:\"You\ are\ a\ large\ language\ model\ living\ in\ Emacs\ and\ a\ helpful\ assistant.\ Respond\ concisely.\"\}\,\{\"role\"\:\"user\"\,\"content\"\:\"What\ sort\ of\ questions\ are\ you\ best\ suited\ to\ answer\?\ Please\ give\ a\ concrete\ answer.\"\}\]\,\"temperature\"\:1.0,\"stream\":true,\"stream_options\":\{\"include_usage\":true\},\"max_tokens\":-1\} \
-HContent-Type\:\ application/json \
http://127.0.0.1:1234/v1/chat/completions

This will stream output for a little while, and then just stop. I'm using LM Studio 0.0.39, and the model is Qwen/QwQ-32B-GGUF/qwq-32b-q8_0.gguf, although it doesn't matter what model I try, it happens with several. I can load the same model with llama-server from llama.cpp and everything works fine.

@jwiegley commented on GitHub (Apr 8, 2025): For what it's worth, I'm having the same problem, using `curl` as the query tool: ``` curl \ --location \ -v \ -D- \ -d\{\"model\"\:\"deepseek-v3-0324\"\,\"messages\"\:\[\{\"role\"\:\"system\"\,\"content\"\:\"You\ are\ a\ large\ language\ model\ living\ in\ Emacs\ and\ a\ helpful\ assistant.\ Respond\ concisely.\"\}\,\{\"role\"\:\"user\"\,\"content\"\:\"What\ sort\ of\ questions\ are\ you\ best\ suited\ to\ answer\?\ Please\ give\ a\ concrete\ answer.\"\}\]\,\"temperature\"\:1.0,\"stream\":true,\"stream_options\":\{\"include_usage\":true\},\"max_tokens\":-1\} \ -HContent-Type\:\ application/json \ http://127.0.0.1:1234/v1/chat/completions ``` This will stream output for a little while, and then just stop. I'm using LM Studio 0.0.39, and the model is `Qwen/QwQ-32B-GGUF/qwq-32b-q8_0.gguf`, although it doesn't matter what model I try, it happens with several. I can load the same model with `llama-server` from llama.cpp and everything works fine.
yindo changed title from Incomplete response from LM studio endpoint to [GH-ISSUE #485] Incomplete response from LM studio endpoint 2026-06-05 14:34:26 -04:00
yindo commented 2026-06-05 14:34:27 -04:00
Author
Owner
Copy Link
Copy Source

@brainz80 commented on GitHub (May 25, 2026):

Having same issue with LM Studio 0.4.14 (Build 4).

I'm running qwen/qwen3.6-35b-a3b.

  • Context length: 128000
  • GPU Offload: 40
  • Number of layers for which to force MoE weights onto CPU: 25

My GPU is a RTX 5080 OC (with 15.92 GB VRAM)

<!-- gh-comment-id:4536654991 --> @brainz80 commented on GitHub (May 25, 2026): Having same issue with LM Studio 0.4.14 (Build 4). I'm running `qwen/qwen3.6-35b-a3b`. - Context length: `128000` - GPU Offload: `40` - Number of layers for which to force MoE weights onto CPU: `25` My GPU is a `RTX 5080 OC` (with `15.92` GB VRAM)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Integration Request question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#278
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?