Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2998] AnythingLLM workspaces as endpoints for litellm - How to #1919

New Issue
Closed
opened 2026-02-22 18:27:14 -05:00 by yindo · 5 comments
yindo commented 2026-02-22 18:27:14 -05:00
Owner
Copy Link
Copy Source

Originally created by @JosefAschauer on GitHub (Jan 21, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2998

Originally assigned to: @shatfield4 on GitHub.

Hi, first of all: WELL DONE AND THANK YOU FOR THIS FANTASTIC TOOL!

At the moment, I'm tring to setup models in LiteLLM to use specific workspaces as api (chat) endpoints. After trying 2 full days, I dare to ask my question in this forum.
Since I was assuming, the AnythingLLM api might be compatible with openai's api, my last attempt was to set the model in the litellm-config.yaml like that:

The model loads but prompts are not answered, in the litellm UI playground I do not see any errors, it just never comes back with a response.

Tried in PS and get a 500:

Full Error Details: {"error":{"message":"litellm.APIError: APIError: OpenAIException - 'str' object has no attribute 'model_dump'\nReceived Model Group=anythingllm-history\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"500"}}
Error Details: {
"Message": "{"error":{"message":"litellm.APIError: APIError: OpenAIException - \u0027str\u0027 object has no attribute \u0027model_dump\u0027\nReceived Model Group=anythingllm-history\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"500"}}",
"RecommendedAction": ""
}
Response body: {"error":{"message":"litellm.APIError: APIError: OpenAIException - 'str' object has no attribute 'model_dump'\nReceived Model Group=anythingllm-history\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"500"}}

Maybe someone is able to help with this. I just don't know what model I need to use for anythingLLM in litellm_params

Originally created by @JosefAschauer on GitHub (Jan 21, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2998 Originally assigned to: @shatfield4 on GitHub. Hi, first of all: WELL DONE AND THANK YOU FOR THIS FANTASTIC TOOL! At the moment, I'm tring to setup models in LiteLLM to use specific workspaces as api (chat) endpoints. After trying 2 full days, I dare to ask my question in this forum. Since I was assuming, the AnythingLLM api might be compatible with openai's api, my last attempt was to set the model in the litellm-config.yaml like that: - model_name: anythingllm-history litellm_params: model: "openai/anythingllm" api_base: http://192.168.xxx.xx:3001/api/v1/workspace/history/chat api_key: XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX model_info: mode: "chat" The model loads but prompts are not answered, in the litellm UI playground I do not see any errors, it just never comes back with a response. Tried in PS and get a 500: Full Error Details: {"error":{"message":"litellm.APIError: APIError: OpenAIException - 'str' object has no attribute 'model_dump'\nReceived Model Group=anythingllm-history\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"500"}} Error Details: { "Message": "{\"error\":{\"message\":\"litellm.APIError: APIError: OpenAIException - \u0027str\u0027 object has no attribute \u0027model_dump\u0027\\nReceived Model Group=anythingllm-history\\nAvailable Model Group Fallbacks=None\",\"type\":null,\"param\":null,\"code\":\"500\"}}", "RecommendedAction": "" } Response body: {"error":{"message":"litellm.APIError: APIError: OpenAIException - 'str' object has no attribute 'model_dump'\nReceived Model Group=anythingllm-history\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"500"}} Maybe someone is able to help with this. I just don't know what model I need to use for anythingLLM in litellm_params
yindo added the investigating label 2026-02-22 18:27:14 -05:00
yindo closed this issue 2026-02-22 18:27:14 -05:00
yindo commented 2026-02-22 18:27:14 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 21, 2025):

At the moment, I'm tring to setup models in LiteLLM to use specific workspaces as api (chat) endpoints. After trying 2 full days, I dare to ask my question in this forum.
Since I was assuming, the AnythingLLM api might be compatible with openai's api, my last attempt was to set the model in the litellm-config.yaml like that:

model_name: anythingllm-history
litellm_params:
model: "openai/anythingllm"
api_base: http://192.168.xxx.xx:3001/api/v1/workspace/history/chat
api_key: XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX
model_info:
mode: "chat"

You just want to you a specific workspace as a model to chat with - correct? The reason you get nothing back is because that API base during inference will be built out to http://192.168.xxx.xx:3001/api/v1/workspace/history/chat/v1/chat/completion which does not exist.

If you want to chat with a specific workspace you should just be able to do this (I am not a LiteLLM expert)

 model_name: anythingllm-history
 litellm_params:
 model: "openai/anythingllm"
 api_base: http://192.168.xxx.xx:3001/api/v1/openai/chat/completions
 api_key: XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX
 model_info:
 mode: "chat"

That endpoint is how you get chat responses, what you have is just history. Also, the model name for this API is the slug of a workspace. You can see all available "models" from the openAI compatible http://192.168.xxx.xx:3001/api/v1/openai/chat/completions

With AnythingLLM the model for OpenAI can be a workspace, so that we can automatically handle RAG, reranking and all this in a single OpenAI-like call.

@timothycarambat commented on GitHub (Jan 21, 2025): > At the moment, I'm tring to setup models in LiteLLM to use specific workspaces as api (chat) endpoints. After trying 2 full days, I dare to ask my question in this forum. > Since I was assuming, the AnythingLLM api might be compatible with openai's api, my last attempt was to set the model in the litellm-config.yaml like that: > > model_name: anythingllm-history > litellm_params: > model: "openai/anythingllm" > api_base: http://192.168.xxx.xx:3001/api/v1/workspace/history/chat > api_key: XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX > model_info: > mode: "chat" You just want to you a specific workspace as a model to chat with - correct? The reason you get nothing back is because that API base during inference will be built out to `http://192.168.xxx.xx:3001/api/v1/workspace/history/chat/v1/chat/completion` which does not exist. If you want to chat with a specific workspace you should just be able to do this (I am not a LiteLLM expert) ```yaml model_name: anythingllm-history litellm_params: model: "openai/anythingllm" api_base: http://192.168.xxx.xx:3001/api/v1/openai/chat/completions api_key: XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX model_info: mode: "chat" ``` That endpoint is how you get chat responses, what you have is just history. Also, the model name for this API is the `slug` of a workspace. You can see all available "models" from the openAI compatible `http://192.168.xxx.xx:3001/api/v1/openai/chat/completions` With AnythingLLM the model for OpenAI can be a workspace, so that we can automatically handle RAG, reranking and all this in a single OpenAI-like call.
yindo commented 2026-02-22 18:27:14 -05:00
Author
Owner
Copy Link
Copy Source

@shatfield4 commented on GitHub (Jan 21, 2025):

I did some testing and was able to get it working with this config.

The biggest thing here is the api_base and ensuring it is correct. LiteLLM automatically adds /chat/completions to the end of the api_base.

- model_name: anythingllm-history
   litellm_params:
      model: openai/testhistory
      api_base: http://127.0.0.1:3001/api/v1/openai
      api_key: XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX
      temperature: 0.7
@shatfield4 commented on GitHub (Jan 21, 2025): I did some testing and was able to get it working with this config. The biggest thing here is the api_base and ensuring it is correct. LiteLLM automatically adds `/chat/completions` to the end of the api_base. ``` - model_name: anythingllm-history litellm_params: model: openai/testhistory api_base: http://127.0.0.1:3001/api/v1/openai api_key: XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX temperature: 0.7 ```
yindo commented 2026-02-22 18:27:14 -05:00
Author
Owner
Copy Link
Copy Source

@JosefAschauer commented on GitHub (Jan 21, 2025):

Thanks for you answer! My workspace has the name 'history' since it contains certain texts about historyical facts - which might be a bit misleading. My goal is to chat with this workspace.
In the model definition, i put the slug Into the model name? So in my case: openai/history?

@JosefAschauer commented on GitHub (Jan 21, 2025): Thanks for you answer! My workspace has the name 'history' since it contains certain texts about historyical facts - which might be a bit misleading. My goal is to chat with this workspace. In the model definition, i put the slug Into the model name? So in my case: openai/history?
yindo commented 2026-02-22 18:27:14 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 21, 2025):

@JosefAschauer for the model parameter
model: "anythingllm-workspace-slug"

@timothycarambat commented on GitHub (Jan 21, 2025): @JosefAschauer for the `model` parameter `model: "anythingllm-workspace-slug"`
yindo commented 2026-02-22 18:27:15 -05:00
Author
Owner
Copy Link
Copy Source

@JosefAschauer commented on GitHub (Jan 21, 2025):

I just tested and it works as intended - perfect!
Thank you!

@JosefAschauer commented on GitHub (Jan 21, 2025): I just tested and it works as intended - perfect! Thank you!
yindo changed title from AnythingLLM workspaces as endpoints for litellm - How to to [GH-ISSUE #2998] AnythingLLM workspaces as endpoints for litellm - How to 2026-06-05 14:43:28 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label investigating
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1919
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?