Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1183] [QUESTION] Token context window settings #727

New Issue
Closed
opened 2026-02-22 18:21:03 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:21:03 -05:00
Owner
Copy Link
Copy Source

Originally created by @ShadowArcanist on GitHub (Apr 24, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1183

All models have different context window size and it is quite hard to find them online like I have to spend few minutes on google to find the context window size of certain models.

So is it possible to auto-generate this value based on the LLM model user selects?
image

Also I tried to put 999999 as the value for Token context window and it didn't affected the response I get from model yet.. So Can I just always put 999999 ?
image

Someone on our discord server also asked how they can know which number to enter on the Token context window, so I think auto-generating the token context value based on model they choose would be helpful for beginners.
image

Originally created by @ShadowArcanist on GitHub (Apr 24, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1183 All models have different context window size and it is quite hard to find them online like I have to spend few minutes on google to find the context window size of certain models. So is it possible to auto-generate this value based on the LLM model user selects? ![image](https://github.com/Mintplex-Labs/anything-llm/assets/162910371/93339985-52ac-4219-aeb4-f79771eb9ff1) Also I tried to put 999999 as the value for Token context window and it didn't affected the response I get from model yet.. So Can I just always put 999999 ? ![image](https://github.com/Mintplex-Labs/anything-llm/assets/162910371/e2107bd2-831d-46dc-8a6e-b721c4e9e3e7) Someone on our discord server also asked how they can know which number to enter on the Token context window, so I think auto-generating the token context value based on model they choose would be helpful for beginners. ![image](https://github.com/Mintplex-Labs/anything-llm/assets/162910371/7ff7256a-3375-4c15-840b-a489dd763c33)
yindo closed this issue 2026-02-22 18:21:03 -05:00
yindo commented 2026-02-22 18:21:03 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 24, 2024):

Its actually a really frustrating problem. OpenRouter is the only provider that tells you the context window per model. Not even OpenAI tells you - you have to go to their docs.

So to get the proper context length you should go to the models HuggingFace repo and hope it is in the model card or you can google it.

We cannot assume it because if we say 'everything unknown is 8192' then when you use a smaller model like phi-2 which has a context of 2048, you will have an issue.

Llama-3 8B has a context length of 8K. I just googled "Llama3 8B context length".

If you put 999999 you wont have problems until your chat is 8001 tokens long. Then ollama will crash because you tried to load too many tokens into the model. Some model runners will auto-prune the context and chop off the top, but there is no way for us to get the context lengths easily.

Its something i scream about daily.

@timothycarambat commented on GitHub (Apr 24, 2024): Its actually a really frustrating problem. OpenRouter is the **only** provider that tells you the context window per model. Not even OpenAI tells you - you have to go to their docs. So to get the proper context length you should go to the models HuggingFace repo and hope it is in the model card or you can google it. We cannot assume it because if we say 'everything unknown is 8192' then when you use a smaller model like phi-2 which has a context of 2048, you will have an issue. Llama-3 8B has a context length of 8K. I just googled "Llama3 8B context length". If you put 999999 you wont have problems until your chat is 8001 tokens long. Then ollama will crash because you tried to load too many tokens into the model. Some model runners will auto-prune the context and chop off the top, but there is no way for us to get the context lengths easily. Its something i scream about daily.
yindo commented 2026-02-22 18:21:03 -05:00
Author
Owner
Copy Link
Copy Source

@ShadowArcanist commented on GitHub (Apr 24, 2024):

Ahh okay got it! Thanks for taking time to explain it in detail 😊

@ShadowArcanist commented on GitHub (Apr 24, 2024): Ahh okay got it! Thanks for taking time to explain it in detail 😊
yindo commented 2026-02-22 18:21:03 -05:00
Author
Owner
Copy Link
Copy Source

@CallMeLaNN commented on GitHub (Jun 13, 2024):

I understand that this is manual way to look into the doc but most doc state in the format of 8K instead of 8192. I think the textbox need to be more friendly or at least to tell to times 1024 first.

@CallMeLaNN commented on GitHub (Jun 13, 2024): I understand that this is manual way to look into the doc but most doc state in the format of 8K instead of 8192. I think the textbox need to be more friendly or at least to tell to times 1024 first.
yindo changed title from [QUESTION] Token context window settings to [GH-ISSUE #1183] [QUESTION] Token context window settings 2026-06-05 14:36:53 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#727
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?