Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1954] [FEAT]: "max tokens" parameter override for Ollama #1267

New Issue
Closed
opened 2026-02-22 18:23:59 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:23:59 -05:00
Owner
Copy Link
Copy Source

Originally created by @MrSimonC on GitHub (Jul 24, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1954

How are you running AnythingLLM?

Docker (local)

What happened?

As I've discovered while using Ollama, the default context window size of 2000 tokens can be too small for modern use cases. While researching possible solutions, I found that the num_ctx parameter is required when making API calls to Ollama to specify a different context window size.

I've looked into the "max tokens" feature in AnythingLLM, and it appears that we do have this option available. However, upon further inspection, I realized that this parameter is not being utilized by Ollama. The max_tokens parameter is present in the AnythingLLM codebase, but it's never sent to Ollama or used in the API call.

I've also looked at the LangChain tool, which supports the needed num_ctx parameter through their integration with Ollama. This further highlights the need for us to implement this feature in AnythingLLM.

I propose adding an option to AnythingLLM that allows advanced users to select from a drop-down list of common context window sizes (e.g., 2K, 4K, 8K, 16K, 32K, 64K, 128K etc.) for Ollama models. This feature would enable users to easily adjust the context window size without requiring custom model files.

Example Use Case:

Users who frequently interact with large datasets or need more accurate results from their AI models can benefit from this feature. By offering a range of context window sizes, we can cater to different use cases and improve overall user experience.

Implementation:

The implementation would involve modifying the AnythingLLM API call to langchain to include a parameter for selecting the context window size - ending up being bound to the num_ctx param in langchain. This parameter would then be passed through to Ollama's API, ensuring that the Ollama API behaves correctly when trying to use larger context windows.

Are there known steps to reproduce?

Reproducible steps are easy, simply connecting any Ollama model with any max token size and then stream the Ollama server logs.

If the line appears like so:
Jul 24 11:10:51 PCHOSTNAME ollama[774]: llama_new_context_with_model: n_ctx = 2048 then this is incorrect, and is defaulting to the 2K token size. this last number should update in the logs if successfully implemented.

Originally created by @MrSimonC on GitHub (Jul 24, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1954 ### How are you running AnythingLLM? Docker (local) ### What happened? As I've discovered while using Ollama, the default context window size of 2000 tokens can be too small for modern use cases. While researching possible solutions, I found that the `num_ctx` [parameter is required when making API calls to Ollama](https://github.com/ollama/ollama/issues/5862#issuecomment-2246392867) to specify a different context window size. I've looked into the "max tokens" feature in AnythingLLM, and it appears that we do have this option available. However, upon further inspection, I realized that this parameter is not being utilized by Ollama. The `max_tokens` parameter is present in the AnythingLLM codebase, but it's never sent to Ollama or used in the API call. I've also looked at the LangChain tool, [which supports the needed `num_ctx` parameter](https://github.com/langchain-ai/langchain/blob/379803751e5ae40a2aadcb4072dbb2525187dd1f/libs/partners/ollama/langchain_ollama/llms.py#L55) through their integration with Ollama. This further highlights the need for us to implement this feature in AnythingLLM. I propose adding an option to AnythingLLM that allows advanced users to select from a drop-down list of common context window sizes (e.g., 2K, 4K, 8K, 16K, 32K, 64K, 128K etc.) for Ollama models. This feature would enable users to easily adjust the context window size without requiring custom model files. **Example Use Case:** Users who frequently interact with large datasets or need more accurate results from their AI models can benefit from this feature. By offering a range of context window sizes, we can cater to different use cases and improve overall user experience. **Implementation:** The implementation would involve modifying the AnythingLLM API call to langchain to include a parameter for selecting the context window size - ending up being bound to the `num_ctx` param in langchain. This parameter would then be passed through to Ollama's API, ensuring that the Ollama API behaves correctly when trying to use larger context windows. ### Are there known steps to reproduce? Reproducible steps are easy, simply connecting any Ollama model with any max token size and then stream the [Ollama server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md). If the line appears like so: `Jul 24 11:10:51 PCHOSTNAME ollama[774]: llama_new_context_with_model: n_ctx = 2048` then this is incorrect, and is defaulting to the 2K token size. this last number should update in the logs if successfully implemented.
yindo added the enhancement label 2026-02-22 18:23:59 -05:00
yindo closed this issue 2026-02-22 18:24:00 -05:00
yindo commented 2026-02-22 18:24:00 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 24, 2024):

Again, great write-up and super clean - you make following these issues easy haha. I would slot this as an "improvement" over a bug per-se as it falls under the same scope as #1902 - either way will see how we can slot this in.

@timothycarambat commented on GitHub (Jul 24, 2024): Again, great write-up and super clean - you make following these issues easy haha. I would slot this as an "improvement" over a bug per-se as it falls under the same scope as #1902 - either way will see how we can slot this in.
yindo commented 2026-02-22 18:24:00 -05:00
Author
Owner
Copy Link
Copy Source

@MrSimonC commented on GitHub (Aug 8, 2024):

Just to confirm #2014 has completely solved this issue. Thanks so much. Tested on MacOS desktop - confirmed working.

@MrSimonC commented on GitHub (Aug 8, 2024): Just to confirm #2014 has completely solved this issue. Thanks so much. Tested on MacOS desktop - confirmed working.
yindo changed title from [FEAT]: "max tokens" parameter override for Ollama to [GH-ISSUE #1954] [FEAT]: "max tokens" parameter override for Ollama 2026-06-05 14:39:51 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1267
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?