Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1991] [FEAT]: Ollama n_ctx for VRAM allocation and performance on responses #1294

New Issue
Closed
opened 2026-02-22 18:24:08 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:24:08 -05:00
Owner
Copy Link
Copy Source

Originally created by @coniuc2d on GitHub (Jul 28, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1991

Originally assigned to: @shatfield4 on GitHub.

How are you running AnythingLLM?

Docker (remote machine)

What happened?

Install Ollama using provided script (linux version, ubuntu 22.04), install AnythingLLM using provided easy script amd docker. Everything runs great however i noticed, that out of 8gb vram on my 5700xt only 74% is reserved no matter what i set in AnythingLLM.
Before you shout at me, im retired plumber. It took me two days to check this out. Give me a brake if I made mistake with config:)

Are there known steps to reproduce?

In ollama serve, using /set parameter num_ctx 128000 ollama takes all my Vram and close to 22gb ram.
In ollama serve, using /set parameter num_ctx 11200 ollama takes 99% Vram and responses are much much better.
In ollama serve, using default settings (for newbies, like me:)) only 74% of Vram is reserved. Responses are worse than above.

Looks like AnythingLLM is not forwarding changes of context to ollama. Whatever you set, default llama3.1:latest stays at 1024.

Originally created by @coniuc2d on GitHub (Jul 28, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1991 Originally assigned to: @shatfield4 on GitHub. ### How are you running AnythingLLM? Docker (remote machine) ### What happened? Install Ollama using provided script (linux version, ubuntu 22.04), install AnythingLLM using provided easy script amd docker. Everything runs great however i noticed, that out of 8gb vram on my 5700xt only 74% is reserved no matter what i set in AnythingLLM. Before you shout at me, im retired plumber. It took me two days to check this out. Give me a brake if I made mistake with config:) ### Are there known steps to reproduce? In ollama serve, using /set parameter num_ctx 128000 ollama takes all my Vram and close to 22gb ram. In ollama serve, using /set parameter num_ctx 11200 ollama takes 99% Vram and responses are much much better. In ollama serve, using default settings (for newbies, like me:)) only 74% of Vram is reserved. Responses are worse than above. Looks like AnythingLLM is not forwarding changes of context to ollama. Whatever you set, default llama3.1:latest stays at 1024.
yindo added the enhancementfeature request labels 2026-02-22 18:24:09 -05:00
yindo closed this issue 2026-02-22 18:24:09 -05:00
yindo commented 2026-02-22 18:24:09 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 28, 2024):

Ollama serve does not overwrite API connection request defaults and information, which is how AnythingLLM communicates with Ollama.

As you can see here, we do not set numCTX https://github.com/Mintplex-Labs/anything-llm/blob/ae58a2cb0db7c74d9c43c3f8ac748725e28ab31c/server/utils/AiProviders/ollama/index.js#L31

This comment below though is probably more helpful by editing the model's Modelfile for Ollama and it will be persistent and is probably better handled there.
https://github.com/ollama/ollama/issues/5965#issuecomment-2252354726

The main reason I hesitate to auto-set n_ctx to the model's max token size is that the 100% VRAM reservation by default with AnythingLLM may be not what people want. intuitively, I have to assume the Ollama team set that limit for some reason - although it is not outright stated why.

Either way, this for sure falls under a "Feature/improvement" as opposed to a bug (like https://github.com/Mintplex-Labs/anything-llm/pull/1920), but that comment above should help keep your persistent settings 👍

@timothycarambat commented on GitHub (Jul 28, 2024): Ollama serve does not overwrite API connection request defaults and information, which is how AnythingLLM communicates with Ollama. As you can see here, we do not set numCTX https://github.com/Mintplex-Labs/anything-llm/blob/ae58a2cb0db7c74d9c43c3f8ac748725e28ab31c/server/utils/AiProviders/ollama/index.js#L31 This comment below though is probably more helpful by editing the model's `Modelfile` for Ollama and it will be persistent and is _probably_ better handled there. https://github.com/ollama/ollama/issues/5965#issuecomment-2252354726 The main reason I hesitate to auto-set `n_ctx` to the model's max token size is that the 100% VRAM reservation by default with AnythingLLM may be _not_ what people want. intuitively, I have to assume the Ollama team set that limit for **some** reason - although it is not outright stated why. Either way, this for sure falls under a "Feature/improvement" as opposed to a bug (like https://github.com/Mintplex-Labs/anything-llm/pull/1920), but that comment above should help keep your persistent settings 👍
yindo commented 2026-02-22 18:24:09 -05:00
Author
Owner
Copy Link
Copy Source

@coniuc2d commented on GitHub (Jul 28, 2024):

Thanks for patience!
For those that (like me) are learning, llama3.1 q8_0 with num_gpu 10 and num_ctx 10240 uses 95% of 8GB Vram, about 4,5GB Ram and gives 2.5t/s. Much better responses. I will test embedding tomorrow but i expect also much better results than on stock settings.
All This on xeon 12/24 and radeon 5700xt using ubuntu 22.04 and docker version of fantastic AnythingLLM.

@coniuc2d commented on GitHub (Jul 28, 2024): Thanks for patience! For those that (like me) are learning, llama3.1 q8_0 with num_gpu 10 and num_ctx 10240 uses 95% of 8GB Vram, about 4,5GB Ram and gives 2.5t/s. Much better responses. I will test embedding tomorrow but i expect also much better results than on stock settings. All This on xeon 12/24 and radeon 5700xt using ubuntu 22.04 and docker version of fantastic AnythingLLM.
yindo commented 2026-02-22 18:24:09 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 31, 2024):

https://github.com/ollama/ollama/issues/1005#issue-1977548240

@timothycarambat commented on GitHub (Jul 31, 2024): https://github.com/ollama/ollama/issues/1005#issue-1977548240
yindo changed title from [FEAT]: Ollama `n_ctx` for VRAM allocation and performance on responses to [GH-ISSUE #1991] [FEAT]: Ollama `n_ctx` for VRAM allocation and performance on responses 2026-06-05 14:39:59 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1294
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?