Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3421] [BUG]: Recognize LiteLLM model's max context and remove max_tokens #2206

New Issue
Closed
opened 2026-02-22 18:28:38 -05:00 by yindo · 10 comments
yindo commented 2026-02-22 18:28:38 -05:00
Owner
Copy Link
Copy Source

Originally created by @ringge on GitHub (Mar 8, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3421

Originally assigned to: @angelplusultra on GitHub.

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

I'm using document pinning for my workspace.
If using the model directly, for example I used Gemini 2.0, or any other model, the document pinning works as expected.
However, if using the same model (like Gemini 2.0) through LiteLLM, anythingLLM seems not being able to detect the max context through LiteLLM so it automatically truncate the document

Are there known steps to reproduce?

  1. Upload a document, enable document pinning
  2. Set model used as Gemini 2.0, it works
  3. Switch model to LiteLLM, still using Gemini 2.0, the prompt is truncated
Originally created by @ringge on GitHub (Mar 8, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3421 Originally assigned to: @angelplusultra on GitHub. ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? I'm using document pinning for my workspace. If using the model directly, for example I used Gemini 2.0, or any other model, the document pinning works as expected. However, if using the same model (like Gemini 2.0) through LiteLLM, anythingLLM seems not being able to detect the max context through LiteLLM so it automatically truncate the document ### Are there known steps to reproduce? 1. Upload a document, enable document pinning 2. Set model used as Gemini 2.0, it works 3. Switch model to LiteLLM, still using Gemini 2.0, the prompt is truncated
yindo added the enhancementbugcore-team-onlyinvestigating labels 2026-02-22 18:28:38 -05:00
yindo closed this issue 2026-02-22 18:28:38 -05:00
yindo commented 2026-02-22 18:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 9, 2025):

Since LiteLLM's /models endpoint does not return a context_length for each model, it is up to you to define it properly for a model. Then the max window is applied on that value.

Image
@timothycarambat commented on GitHub (Mar 9, 2025): Since LiteLLM's `/models` endpoint does not return a `context_length` for each model, it is up to you to define it properly for a model. Then the max window is applied on that value. <img width="975" alt="Image" src="https://github.com/user-attachments/assets/2f92efe6-a6fe-42f0-8394-9030ff3ffb42" />
yindo commented 2026-02-22 18:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@ringge commented on GitHub (Mar 9, 2025):

hi, many thanks for your response
I tried to set the "Token context window" per your guide as 128000, but it seems anything LLM takes it as maxOutputTokens

litellm.BadRequestError: VertexAIException BadRequestError - b'{\n "error": {\n "code": 400,\n "message": "Unable to submit request because it has a maxOutputTokens value of 128000 but the supported range is from 1 (inclusive) to 8193 (exclusive). Update the value and try again.",\n "status": "INVALID_ARGUMENT"\n }\n}\n'

it seems LiteLLM returns all those max_input_tokens in the model/info endpoint
Reference: https://docs.litellm.ai/docs/proxy/model_management#get-model-information---modelinfo

Image
@ringge commented on GitHub (Mar 9, 2025): hi, many thanks for your response I tried to set the "Token context window" per your guide as 128000, but it seems anything LLM takes it as maxOutputTokens `litellm.BadRequestError: VertexAIException BadRequestError - b'{\n "error": {\n "code": 400,\n "message": "Unable to submit request because it has a maxOutputTokens value of 128000 but the supported range is from 1 (inclusive) to 8193 (exclusive). Update the value and try again.",\n "status": "INVALID_ARGUMENT"\n }\n}\n'` it seems LiteLLM returns all those max_input_tokens in the model/info endpoint Reference: https://docs.litellm.ai/docs/proxy/model_management#get-model-information---modelinfo <img width="875" alt="Image" src="https://github.com/user-attachments/assets/942b5935-6a3f-4e89-b405-8db869bf09d1" />
yindo commented 2026-02-22 18:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 10, 2025):

We use LiteLLM via its OpenAI interface, which allows us to pass a max_tokens
https://github.com/Mintplex-Labs/anything-llm/blob/0b1a0e0a1f169b4e4cadc8403f6855e6eacb39e8/server/utils/AiProviders/liteLLM/index.js#L134

While i dont think there is another parameter for input, as that would make sense to have, LiteLLM is throwing an error when the output is set to a higher value. That being said, the values should be decoupled since the max context != max output

@timothycarambat commented on GitHub (Mar 10, 2025): We use LiteLLM via its OpenAI interface, which allows us to pass a `max_tokens` https://github.com/Mintplex-Labs/anything-llm/blob/0b1a0e0a1f169b4e4cadc8403f6855e6eacb39e8/server/utils/AiProviders/liteLLM/index.js#L134 While i dont think there is another parameter for input, as that would make sense to have, LiteLLM is throwing an error when the output is set to a higher value. That being said, the values should be decoupled since the max context != max output
yindo commented 2026-02-22 18:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@DarthMuzammil commented on GitHub (Mar 17, 2025):

hi @timothycarambat can i work on this issue?

@DarthMuzammil commented on GitHub (Mar 17, 2025): hi @timothycarambat can i work on this issue?
yindo commented 2026-02-22 18:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@rradnet commented on GitHub (Apr 16, 2025):

I find that OpenAI models api call doesn't return the context size. If I set the LMSTUDIO_MODEL_TOKEN_LIMIT = '', I can run open ai models with a larger context window and chat history.
It would be 'Great' if we could set the outbound context window per workspace. Bonus, to set the allowed return context per workspace. Ideally, this data would be indexed by provider and model, so enter it once or retrieve from API it can be stored locally.

@rradnet commented on GitHub (Apr 16, 2025): I find that OpenAI models api call doesn't return the context size. If I set the LMSTUDIO_MODEL_TOKEN_LIMIT = '<largevalue>', I can run open ai models with a larger context window and chat history. It would be 'Great' if we could set the outbound context window per workspace. Bonus, to set the allowed return context per workspace. Ideally, this data would be indexed by provider and model, so enter it once or retrieve from API it can be stored locally.
yindo commented 2026-02-22 18:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@rradnet commented on GitHub (Apr 16, 2025):

The post remove a very important fact. LMSTUDIO_MODEL_TOKEN_LIMIT = '100000' large size.

@rradnet commented on GitHub (Apr 16, 2025): The post remove a very important fact. LMSTUDIO_MODEL_TOKEN_LIMIT = '100000' large size.
yindo commented 2026-02-22 18:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@olandir commented on GitHub (Feb 11, 2026):

This issue appears to still be open. Has there been any resolution on this. It appears that if I put the actual context window into the field, it complains because it's bigger than the max output tokens. So for Claude Sonnet 4.5 I have a Context (or Max Input Tokens) of 200000 and a max output of 64000 but if i put 200000 into the field I get this error

400 litellm.BadRequestError: BedrockException - {"message":"The maximum tokens you requested exceeds the model limit of 64000. Try again with a maximum tokens value that is lower than 64000."}. Received Model Group=claude-sonnet-4.5 Available Model Group Fallbacks=None

I even manually went into LiteLLM and set the Max_Tokens field to 200000 but that did not fix it. Apparently AnythingLLM's Max Context field is set to Max Output Tokens so either the label is wrong or there's something wrong in the code.

I get around this by using OpenAI-Generic since LiteLLM is basically making that endpoint for me and it works perfectly there and has both the context and the max tokens fields which I assume "Max Tokens" for AnythingLLM is Max Output tokens

@olandir commented on GitHub (Feb 11, 2026): This issue appears to still be open. Has there been any resolution on this. It appears that if I put the actual context window into the field, it complains because it's bigger than the max output tokens. So for Claude Sonnet 4.5 I have a Context (or Max Input Tokens) of 200000 and a max output of 64000 but if i put 200000 into the field I get this error 400 litellm.BadRequestError: BedrockException - {"message":"The maximum tokens you requested exceeds the model limit of 64000. Try again with a maximum tokens value that is lower than 64000."}. Received Model Group=claude-sonnet-4.5 Available Model Group Fallbacks=None I even manually went into LiteLLM and set the Max_Tokens field to 200000 but that did not fix it. Apparently AnythingLLM's Max Context field is set to Max Output Tokens so either the label is wrong or there's something wrong in the code. I get around this by using OpenAI-Generic since LiteLLM is basically making that endpoint for me and it works perfectly there and has both the context and the max tokens fields which I assume "Max Tokens" for AnythingLLM is Max Output tokens
yindo commented 2026-02-22 18:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 12, 2026):

"Max Tokens" for AnythingLLM is Max Output tokens

Max tokens is actually the total context window for a given message request (or it should be) however the code shows
https://github.com/Mintplex-Labs/anything-llm/blob/6f443ce11a5bd675efb210d757f057d0e7cb3c3c/server/utils/AiProviders/liteLLM/index.js#L135

Which controls the amount of output tokens, when in reality this is the figure that drives/controls the actual need to do message compression by checking if the context is going to overflow.

@timothycarambat commented on GitHub (Feb 12, 2026): > "Max Tokens" for AnythingLLM is Max Output tokens Max tokens is actually the total context window for a given message request (or it should be) however the code shows https://github.com/Mintplex-Labs/anything-llm/blob/6f443ce11a5bd675efb210d757f057d0e7cb3c3c/server/utils/AiProviders/liteLLM/index.js#L135 Which controls the amount of output tokens, when in reality this is the figure that drives/controls the actual need to do message compression by checking if the context is going to overflow.
yindo commented 2026-02-22 18:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@olandir commented on GitHub (Feb 12, 2026):

@timothycarambat so what would fix this issue.=? I can't quite tell form this section of code. If I set the Max Tokens field to the max output tokens then there is no error thrown, the problem is that I don't know if it's restricting the entire context to the output tokens (which for claude is significantly less) as that would impact what's being sent to LLM (could end up truncating a conversation.

Thanks!

@olandir commented on GitHub (Feb 12, 2026): @timothycarambat so what would fix this issue.=? I can't quite tell form this section of code. If I set the Max Tokens field to the max output tokens then there is no error thrown, the problem is that I don't know if it's restricting the entire context to the output tokens (which for claude is significantly less) as that would impact what's being sent to LLM (could end up truncating a conversation. Thanks!
yindo commented 2026-02-22 18:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 12, 2026):

Just patched it with 5fb1281891 to prevent that field from controlling the unrelated max_output tokens field - it should purely be used for context window assessment.

That being said, from what i can see LiteLLM does not a consistent setting for grabbing the active model context window value and so you should still set it or else it can default to a much lower context window.

@timothycarambat commented on GitHub (Feb 12, 2026): Just patched it with 5fb12818910746f8846cdbf3519ac4f25e7923f6 to prevent that field from controlling the unrelated max_output tokens field - it should purely be used for context window assessment. That being said, from what i can see LiteLLM does not a consistent setting for grabbing the active model context window value and so you should still set it or else it can default to a much lower context window.
yindo changed title from [BUG]: Recognize LiteLLM model's max context and remove `max_tokens` to [GH-ISSUE #3421] [BUG]: Recognize LiteLLM model's max context and remove `max_tokens` 2026-06-05 14:45:10 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label bug core-team-only enhancement investigating
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2206
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?