Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5219] [FEAT]: Use Native LM Studio /api/v1/chat for chat history optimization #4975

New Issue
Closed
opened 2026-06-05 14:51:14 -04:00 by yindo · 3 comments
yindo commented 2026-06-05 14:51:14 -04:00
Owner
Copy Link
Copy Source

Originally created by @raucodes on GitHub (Mar 16, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5219

How are you running AnythingLLM?

Docker (local)

What happened?

Hi team,

I believe there is an issue — or at least a major missed optimization — in the native LM Studio connector.

Summary

When using the native LM Studio provider in AnythingLLM (Docker version), the app appears to use:

  • GET /api/v0/models for model discovery
  • but POST /v1/chat/completions for inference

instead of LM Studio’s stateful chat endpoint:

  • POST /api/v1/chat

This means chat requests are effectively handled in a stateless OpenAI-compatible mode, so the full conversation history is sent again on every turn instead of reusing LM Studio’s server-side conversation state.

For short chats this is fine, but for long conversations this becomes a major performance problem.

Environment

  • AnythingLLM: Docker version
  • Backend: LM Studio server
  • Observed via: lms log stream --source server
  • Model used: Thinking3.5:latest

What I observed

From LM Studio logs, AnythingLLM does the following:

  1. Fetches models via LM Studio native API:
GET /api/v0/models
  1. But then sends chat inference requests to:
POST /v1/chat/completions

Example from the LM Studio server log:

[DEBUG] Received request: GET to /api/v0/models
...
[DEBUG] Received request: POST to /v1/chat/completions with body {
  "model": "Thinking3.5:latest",
  "stream": true,
  "messages": [
    { "role": "system", ... },
    { "role": "user", ... },
    { "role": "assistant", ... },
    { "role": "user", ... },
    { "role": "assistant", ... },
    { "role": "user", ... }
  ],
  "temperature": 0.7
}

LM Studio then logs:

[INFO][Thinking3.5:latest] Running chat completion on conversation with 6 messages.

So the native LM Studio connector is not actually using LM Studio’s stateful chat flow for inference.

Why this matters

LM Studio provides a stateful chat API (/api/v1/chat) specifically to avoid resending the entire message history every turn.

Using /v1/chat/completions instead means:

  • the entire chat history is resent every turn
  • LM Studio has to rebuild prompt state repeatedly
  • long chats become increasingly slow
  • KV/cache reuse benefits from LM Studio’s stateful chat API are not used

This is especially noticeable with large local models and long technical conversations.

Additional concern

In my logs, the assistant messages being resent also appear to include internal reasoning content such as:

<think>Thinking Process: ...

If AnythingLLM is resending hidden reasoning / chain-of-thought-style content back into history, this makes the problem much worse because the effective prompt size grows far faster than the visible conversation would suggest.

Even a relatively small number of turns can become very expensive if each assistant turn contains large hidden reasoning blocks.

Expected behavior

When the user selects the native LM Studio connector, I would expect AnythingLLM to use LM Studio’s native stateful chat API for inference as well, not only for model discovery.

At minimum, I would expect this to be available as an optional mode.

Actual behavior

  • LM Studio native API is used for model discovery
  • inference still goes through /v1/chat/completions
  • full history is resent every turn
  • long chats degrade heavily in performance
  • possible hidden reasoning content may also be resent

Suggested improvement

Option 1: Support LM Studio stateful chat API

When using the LM Studio provider, add support for:

POST /api/v1/chat

and store the LM Studio conversation reference (response_id / previous_response_id) per AnythingLLM chat session.

This would allow:

  • server-side conversation continuation
  • less prompt rebuilding
  • better long-chat performance
  • more effective cache reuse

Option 2: Add a provider setting / toggle

Example:

  • Use LM Studio native stateful chat API
  • fallback to /v1/chat/completions if disabled or unsupported

Option 3: Strip hidden reasoning before resending history

If AnythingLLM must continue using stateless chat completions, it would still help significantly to:

  • remove <think>...</think> or hidden reasoning blocks from assistant history
  • only resend user-visible assistant output

That alone could reduce prompt bloat a lot.

Reproduction steps

  1. Run LM Studio server
  2. Run AnythingLLM Docker
  3. Configure the native LM Studio connector
  4. Start log streaming in LM Studio:
lms log stream --source server

  1. Send a few messages in AnythingLLM

  2. Observe that:

    • model discovery uses /api/v0/models
    • chat inference uses /v1/chat/completions
    • the full message history is resent each turn

Closing note

The current behavior makes the native LM Studio connector feel only partially native:

  • native for model discovery
  • but effectively OpenAI-compatible/stateless for inference

For local large-model users, especially those running long chats, proper support for LM Studio’s stateful chat API would be a very meaningful improvement.

Thanks.

Are there known steps to reproduce?

No response

Originally created by @raucodes on GitHub (Mar 16, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5219 ### How are you running AnythingLLM? Docker (local) ### What happened? Hi team, I believe there is an issue — or at least a major missed optimization — in the native LM Studio connector. ## Summary When using the **native LM Studio provider** in AnythingLLM (Docker version), the app appears to use: * `GET /api/v0/models` for model discovery * but **`POST /v1/chat/completions` for inference** instead of LM Studio’s **stateful** chat endpoint: * `POST /api/v1/chat` This means chat requests are effectively handled in a **stateless OpenAI-compatible mode**, so the full conversation history is sent again on every turn instead of reusing LM Studio’s server-side conversation state. For short chats this is fine, but for long conversations this becomes a major performance problem. --- ## Environment * **AnythingLLM**: Docker version * **Backend**: LM Studio server * **Observed via**: `lms log stream --source server` * **Model used**: `Thinking3.5:latest` --- ## What I observed From LM Studio logs, AnythingLLM does the following: 1. Fetches models via LM Studio native API: ```text GET /api/v0/models ``` 2. But then sends chat inference requests to: ```text POST /v1/chat/completions ``` Example from the LM Studio server log: ```text [DEBUG] Received request: GET to /api/v0/models ... [DEBUG] Received request: POST to /v1/chat/completions with body { "model": "Thinking3.5:latest", "stream": true, "messages": [ { "role": "system", ... }, { "role": "user", ... }, { "role": "assistant", ... }, { "role": "user", ... }, { "role": "assistant", ... }, { "role": "user", ... } ], "temperature": 0.7 } ``` LM Studio then logs: ```text [INFO][Thinking3.5:latest] Running chat completion on conversation with 6 messages. ``` So the native LM Studio connector is not actually using LM Studio’s stateful chat flow for inference. --- ## Why this matters LM Studio provides a stateful chat API (`/api/v1/chat`) specifically to avoid resending the entire message history every turn. Using `/v1/chat/completions` instead means: * the entire chat history is resent every turn * LM Studio has to rebuild prompt state repeatedly * long chats become increasingly slow * KV/cache reuse benefits from LM Studio’s stateful chat API are not used This is especially noticeable with large local models and long technical conversations. --- ## Additional concern In my logs, the `assistant` messages being resent also appear to include internal reasoning content such as: ```text <think>Thinking Process: ... ``` If AnythingLLM is resending hidden reasoning / chain-of-thought-style content back into history, this makes the problem much worse because the effective prompt size grows far faster than the visible conversation would suggest. Even a relatively small number of turns can become very expensive if each assistant turn contains large hidden reasoning blocks. --- ## Expected behavior When the user selects the **native LM Studio connector**, I would expect AnythingLLM to use LM Studio’s native stateful chat API for inference as well, not only for model discovery. At minimum, I would expect this to be available as an optional mode. --- ## Actual behavior * LM Studio native API is used for model discovery * inference still goes through `/v1/chat/completions` * full history is resent every turn * long chats degrade heavily in performance * possible hidden reasoning content may also be resent --- ## Suggested improvement ### Option 1: Support LM Studio stateful chat API When using the LM Studio provider, add support for: ```text POST /api/v1/chat ``` and store the LM Studio conversation reference (`response_id` / `previous_response_id`) per AnythingLLM chat session. This would allow: * server-side conversation continuation * less prompt rebuilding * better long-chat performance * more effective cache reuse ### Option 2: Add a provider setting / toggle Example: * **Use LM Studio native stateful chat API** * fallback to `/v1/chat/completions` if disabled or unsupported ### Option 3: Strip hidden reasoning before resending history If AnythingLLM must continue using stateless chat completions, it would still help significantly to: * remove `<think>...</think>` or hidden reasoning blocks from assistant history * only resend user-visible assistant output That alone could reduce prompt bloat a lot. --- ## Reproduction steps 1. Run LM Studio server 2. Run AnythingLLM Docker 3. Configure the **native LM Studio connector** 4. Start log streaming in LM Studio: ```bash lms log stream --source server ``` 5. Send a few messages in AnythingLLM 6. Observe that: * model discovery uses `/api/v0/models` * chat inference uses `/v1/chat/completions` * the full message history is resent each turn --- ## Closing note The current behavior makes the native LM Studio connector feel only partially native: * native for model discovery * but effectively OpenAI-compatible/stateless for inference For local large-model users, especially those running long chats, proper support for LM Studio’s stateful chat API would be a very meaningful improvement. Thanks. ### Are there known steps to reproduce? _No response_
yindo added the enhancementfeature request labels 2026-06-05 14:51:14 -04:00
yindo closed this issue 2026-06-05 14:51:14 -04:00
yindo commented 2026-06-05 14:51:14 -04:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 17, 2026):

The /v1/chat API does not support passed in tool calling only its built in MCP managed tools. This would make every tool you add or use in AnythingLLM unusable in LM Studio - this would functionally make LM Studio with AnythingLLM much less useful?

<!-- gh-comment-id:4076114807 --> @timothycarambat commented on GitHub (Mar 17, 2026): The /v1/chat API [does not support passed in tool calling](https://lmstudio.ai/docs/developer/rest/chat) only its built in MCP managed tools. This would make every tool you add or use in AnythingLLM unusable in LM Studio - this would functionally make LM Studio with AnythingLLM much less useful?
yindo commented 2026-06-05 14:51:15 -04:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 17, 2026):

I also checked all over the LM Studio docs. I dont see any information about chat being more performant than the OpenAI compatible API. This would make sense because the KV would already be present from the previous inference since the model is loaded - using /chat just seems to have LM Studio store the chat, which they then re-inject for your on your behalf.

Since we are sending the same history over and over + the new message the KV should remain. I know llama.cpp does have some form of prompt caching but I didnt find anything about KV cache or prompt-caching via that endpoint besides the store param which just keeps history in LMstudio so you dont need to do it elsewhere

<!-- gh-comment-id:4076146302 --> @timothycarambat commented on GitHub (Mar 17, 2026): I also checked all over the LM Studio docs. I dont see any information about chat being more performant than the OpenAI compatible API. This would make sense because the KV would already be present from the previous inference since the model is loaded - using /chat just seems to have LM Studio store the chat, which they then re-inject for your on your behalf. Since we are sending the same history over and over + the new message the KV should remain. I know llama.cpp does have some form of prompt caching but I didnt find anything about KV cache or prompt-caching via that endpoint besides the `store` param which just keeps history in LMstudio so you dont need to do it elsewhere
yindo commented 2026-06-05 14:51:15 -04:00
Author
Owner
Copy Link
Copy Source

@raucodes commented on GitHub (Mar 17, 2026):

Thanks for the clarification — the tool-calling concern makes sense, and I agree that replacing the current inference path globally with LM Studio’s native /api/v1/chat endpoint would likely break important AnythingLLM functionality.

That said, my main point was not necessarily that /api/v1/chat must replace the current default behavior for all LM Studio users.

My concern is more about long-chat efficiency and state handling.

Right now, the LM Studio provider feels only partially native:
• native for model discovery
• but effectively stateless/OpenAI-compatible for inference

In practice, that means the full visible history is resent every turn, and with some model/backend combinations this does not appear to result in stable or efficient reuse in real-world long chats.

From my testing and LM Studio server logs, “sending the same history again” does not always translate into consistently cheap follow-up turns. In some cases prompt processing still grows significantly over time, especially with large local models and long technical conversations.

I think there are still a few worthwhile improvements here, even if /api/v1/chat cannot be used as the universal default:
1. Optional stateful mode where possible
Even if it cannot support all AnythingLLM tooling features, an optional LM Studio native/stateful mode could still be useful for users who prioritize long-chat performance over tool compatibility.
2. Use response IDs / continuation semantics when supported
Serving conversation state via response IDs or continuation references is cleaner than resending the full transcript on every turn. Even if this cannot be done in every path, it would be a more native and more efficient model for providers that support it.
3. Do not resend hidden reasoning / content
If AnythingLLM continues using stateless history replay, it would still help a lot to strip hidden reasoning blocks before re-injecting assistant history. Re-sending internal thinking content makes prompt growth much worse without improving the visible conversation.

So I completely understand the tooling limitation, but I still think there is room for improvement here:
• either through an optional mode
• or through better history compaction / reasoning stripping
• or through cleaner continuation handling where LM Studio supports it

At minimum, documenting that the current LM Studio provider uses native discovery but stateless OpenAI-style inference would already help set expectations correctly for users running long local chats.

<!-- gh-comment-id:4077620842 --> @raucodes commented on GitHub (Mar 17, 2026): Thanks for the clarification — the tool-calling concern makes sense, and I agree that replacing the current inference path globally with LM Studio’s native /api/v1/chat endpoint would likely break important AnythingLLM functionality. That said, my main point was not necessarily that /api/v1/chat must replace the current default behavior for all LM Studio users. My concern is more about long-chat efficiency and state handling. Right now, the LM Studio provider feels only partially native: • native for model discovery • but effectively stateless/OpenAI-compatible for inference In practice, that means the full visible history is resent every turn, and with some model/backend combinations this does not appear to result in stable or efficient reuse in real-world long chats. From my testing and LM Studio server logs, “sending the same history again” does not always translate into consistently cheap follow-up turns. In some cases prompt processing still grows significantly over time, especially with large local models and long technical conversations. I think there are still a few worthwhile improvements here, even if /api/v1/chat cannot be used as the universal default: 1. Optional stateful mode where possible Even if it cannot support all AnythingLLM tooling features, an optional LM Studio native/stateful mode could still be useful for users who prioritize long-chat performance over tool compatibility. 2. Use response IDs / continuation semantics when supported Serving conversation state via response IDs or continuation references is cleaner than resending the full transcript on every turn. Even if this cannot be done in every path, it would be a more native and more efficient model for providers that support it. 3. Do not resend hidden reasoning / <think> content If AnythingLLM continues using stateless history replay, it would still help a lot to strip hidden reasoning blocks before re-injecting assistant history. Re-sending internal thinking content makes prompt growth much worse without improving the visible conversation. So I completely understand the tooling limitation, but I still think there is room for improvement here: • either through an optional mode • or through better history compaction / reasoning stripping • or through cleaner continuation handling where LM Studio supports it At minimum, documenting that the current LM Studio provider uses native discovery but stateless OpenAI-style inference would already help set expectations correctly for users running long local chats.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#4975
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?