Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1569] [FEAT]: LM Studio Multi-model inference server support (Workspace/Agent) #1017

New Issue
Closed
opened 2026-02-22 18:22:41 -05:00 by yindo · 5 comments
yindo commented 2026-02-22 18:22:41 -05:00
Owner
Copy Link
Copy Source

Originally created by @wallartup on GitHub (May 30, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1569

How are you running AnythingLLM?

Docker (local)

What happened?

Going into workspace settings and can choose every LLM provider except LM studio. I have added 2 models to LM studio playground to be able to load them directly into memory and keep them there, however I can't seem to swap between them.

The reason I am using LM studio instead of Ollama is that LM studio is much faster than Ollama as Ollama offloads the model from memory and have to reload the model to memory.

Are there known steps to reproduce?

No response

Originally created by @wallartup on GitHub (May 30, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1569 ### How are you running AnythingLLM? Docker (local) ### What happened? Going into workspace settings and can choose every LLM provider except LM studio. I have added 2 models to LM studio playground to be able to load them directly into memory and keep them there, however I can't seem to swap between them. The reason I am using LM studio instead of Ollama is that LM studio is much faster than Ollama as Ollama offloads the model from memory and have to reload the model to memory. ### Are there known steps to reproduce? _No response_
yindo added the enhancement label 2026-02-22 18:22:41 -05:00
yindo closed this issue 2026-02-22 18:22:41 -05:00
yindo commented 2026-02-22 18:22:41 -05:00
Author
Owner
Copy Link
Copy Source

@shatfield4 commented on GitHub (May 30, 2024):

This is actually done intentionally, LMStudio does not support switching between multiple models and being run at the same time so that is why we block LMStudio from being configured at the workspace level.

@shatfield4 commented on GitHub (May 30, 2024): This is actually done intentionally, LMStudio does not support switching between multiple models and being run at the same time so that is why we block LMStudio from being configured at the workspace level.
yindo commented 2026-02-22 18:22:41 -05:00
Author
Owner
Copy Link
Copy Source

@wallartup commented on GitHub (May 31, 2024):

This is actually done intentionally, LMStudio does not support switching between multiple models and being run at the same time so that is why we block LMStudio from being configured at the workspace level.

I have loaded 2 models at the same time with playground and using other apps to inference and it seems like it's working. Could it be worth testing now that they have updated their solution?

@wallartup commented on GitHub (May 31, 2024): > This is actually done intentionally, LMStudio does not support switching between multiple models and being run at the same time so that is why we block LMStudio from being configured at the workspace level. I have loaded 2 models at the same time with playground and using other apps to inference and it seems like it's working. Could it be worth testing now that they have updated their solution?
yindo commented 2026-02-22 18:22:41 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 31, 2024):

@wallartup this is using the multi-model chat server right? You can use this as your LLM, but it will not still appear in the UI for a workspace specific LLM. The multi-model endpoint support from LMStudio is somewhat new and many people still do not use that as the "main" inference server and it winds up creating more issues for us helping people understand LMStudio has two inference servers.

We can add LMStudio support for workspace models and as long as you are using the multi-model, we can populate the dropdown

@timothycarambat commented on GitHub (May 31, 2024): @wallartup this is using the multi-model chat server right? You can use this as your LLM, but it will not still appear in the UI for a workspace specific LLM. The multi-model endpoint support from LMStudio is somewhat new and many people still do not use that as the "main" inference server and it winds up creating more issues for us helping people understand LMStudio has two inference servers. We _can_ add LMStudio support for workspace models and as long as you are using the multi-model, we can populate the dropdown
yindo commented 2026-02-22 18:22:42 -05:00
Author
Owner
Copy Link
Copy Source

@wallartup commented on GitHub (Jun 1, 2024):

@wallartup this is using the multi-model chat server right? You can use this as your LLM, but it will not still appear in the UI for a workspace specific LLM. The multi-model endpoint support from LMStudio is somewhat new and many people still do not use that as the "main" inference server and it winds up creating more issues for us helping people understand LMStudio has two inference servers.

We can add LMStudio support for workspace models and as long as you are using the multi-model, we can populate the dropdown

@timothycarambat
That's correct. If you use playground (multi-model chat server) it works, the pro over Ollama is that it really loads into the memory and does not "ever" unload. The inferencing is therefore lightning fast in comparison to Ollama which at times is actually quite slow (even with Mlock), this is why I want it added in the workspace menu. I might be able to figure it out myself but I think giving people the option and calling it (LM studio PLAYGROUND ONLY) you are also mitigating that risk.

@wallartup commented on GitHub (Jun 1, 2024): > @wallartup this is using the multi-model chat server right? You can use this as your LLM, but it will not still appear in the UI for a workspace specific LLM. The multi-model endpoint support from LMStudio is somewhat new and many people still do not use that as the "main" inference server and it winds up creating more issues for us helping people understand LMStudio has two inference servers. > > We _can_ add LMStudio support for workspace models and as long as you are using the multi-model, we can populate the dropdown @timothycarambat That's correct. If you use playground (multi-model chat server) it works, the pro over Ollama is that it really loads into the memory and does not "ever" unload. The inferencing is therefore lightning fast in comparison to Ollama which at times is actually quite slow (even with Mlock), this is why I want it added in the workspace menu. I might be able to figure it out myself but I think giving people the option and calling it (LM studio PLAYGROUND ONLY) you are also mitigating that risk.
yindo commented 2026-02-22 18:22:42 -05:00
Author
Owner
Copy Link
Copy Source

@mrdjohnson commented on GitHub (Jun 1, 2024):

If you use playground (multi-model chat server) it works

Super fun fact on that: Loading Models with lmstudio sdk; if Anything LLM used the lmstudio sdk it could allow users to load and use models without needing the playground at all!

I think giving people the option and calling it (LM studio PLAYGROUND ONLY)
@wallartup 1: based on the information included above maybe we should not call it that? 2; I have been working with the LM Studio team lately on getting loading through the API as well. My current suggestion is that the server options include an "auto-load" option, this will allow users to see ALL their models, and whichever they pick get loaded and used; Some variation of this will be coming soon so calling it PLAYGROUND ONLY mode might confuse users (I can also just create another github issue when that time comes)

@mrdjohnson commented on GitHub (Jun 1, 2024): > If you use playground (multi-model chat server) it works Super fun fact on that: [Loading Models with lmstudio sdk](https://github.com/lmstudio-ai/lmstudio.js?tab=readme-ov-file#loading-an-llm-and-predicting-with-it); if Anything LLM used the lmstudio sdk it could allow users to load and use models without needing the playground at all! > I think giving people the option and calling it (LM studio PLAYGROUND ONLY) @wallartup 1: based on the information included above maybe we should not call it that? 2; I have been working with the LM Studio team lately on getting loading through the API as well. My current suggestion is that the server options include an "auto-load" option, this will allow users to see ALL their models, and whichever they pick get loaded and used; Some variation of this will be coming soon so calling it PLAYGROUND ONLY mode might confuse users (I can also just create another github issue when that time comes)
yindo changed title from [FEAT]: LM Studio Multi-model inference server support (Workspace/Agent) to [GH-ISSUE #1569] [FEAT]: LM Studio Multi-model inference server support (Workspace/Agent) 2026-06-05 14:38:29 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1017
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?