Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4443] [CHORE]: API disconnect on slow responses #2831

New Issue
Closed
opened 2026-02-22 18:31:27 -05:00 by yindo · 9 comments
yindo commented 2026-02-22 18:31:27 -05:00
Owner
Copy Link
Copy Source

Originally created by @AnalogKnight on GitHub (Sep 28, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4443

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

I have deployed Ollama on a server within my local network, running very large models on the CPU. Its response is extremely slow, taking almost 10–30 minutes just to successfully load a model.

When I try to chat with the model through AnythingLLM, it always returns the following message after about 5 minutes: “Your Ollama instance could not be reached or is not responding. Please make sure it is running the API server and your connection information is correct in AnythingLLM.”
I assume that AnythingLLM has a fixed 5‑minute timeout when connecting to the Ollama server. Is there any way to change this timeout? My Ollama instance is just very slow to respond, not unresponsive.

I tried setting the Ollama environment variable OLLAMA_REQUEST_TIMEOUT, but it doesn’t seem to work, so I believe the issue lies in the API request sent by AnythingLLM. If I am mistaken, I apologize in advance.

Any help would be appreciated.

Are there known steps to reproduce?

No response

Originally created by @AnalogKnight on GitHub (Sep 28, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4443 ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? I have deployed Ollama on a server within my local network, running very large models on the CPU. Its response is extremely slow, taking almost 10–30 minutes just to successfully load a model. When I try to chat with the model through AnythingLLM, it always returns the following message after about 5 minutes: “Your Ollama instance could not be reached or is not responding. Please make sure it is running the API server and your connection information is correct in AnythingLLM.” I assume that AnythingLLM has a fixed 5‑minute timeout when connecting to the Ollama server. Is there any way to change this timeout? My Ollama instance is just very slow to respond, not unresponsive. I tried setting the Ollama environment variable OLLAMA_REQUEST_TIMEOUT, but it doesn’t seem to work, so I believe the issue lies in the API request sent by AnythingLLM. If I am mistaken, I apologize in advance. Any help would be appreciated. ### Are there known steps to reproduce? _No response_
yindo closed this issue 2026-02-22 18:31:27 -05:00
yindo commented 2026-02-22 18:31:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 29, 2025):

While we can investigate why that timeout error occurs when interacting with the Ollama api, is this really the ideal case for use with a tool like AnythingLLM? I understand you may wish to run larger models that overwhelm the current resources on the machine but this will likely lead to not only horribly bad reply speeds, but just a bad experience in general :/

I can understand the idea of "its my machine, i dont care and I will wait", but just wondering if that is something we should even work around!

@timothycarambat commented on GitHub (Sep 29, 2025): While we can investigate why that timeout error occurs when interacting with the Ollama api, is this really the ideal case for use with a tool like AnythingLLM? I understand you may wish to run larger models that overwhelm the current resources on the machine but this will likely lead to not only horribly bad reply speeds, but just a bad experience in general :/ I can understand the idea of "its my machine, i dont care and I will wait", but just wondering if that is something we should even work around!
yindo commented 2026-02-22 18:31:27 -05:00
Author
Owner
Copy Link
Copy Source

@AnalogKnight commented on GitHub (Sep 29, 2025):

Thank you for your kind response. For me personally, I think this is important. My hardware specs aren’t too bad—I have 2×RTX 3090s and over a hundred gigabytes of memory. In most of my use cases, deploying a model like GPT-OSS:20b is more than sufficient.

However, when I do something like asking the AI to continue writing a very long article, smaller models can never maintain contextual coherence between the new content and the original text. I agree that simply running the largest possible model most of the time indeed just a bad experience in general. But if the real issue is whether the work can be done or not, then perhaps the quality of the experience becomes less critical.

Although I may rarely need the AI to handle such demanding tasks, if I do, I can let a larger model take care of it while I go to sleep. With a smaller model, though, I might have to make it repeat the task dozens or even hundreds of times, and still end up without a satisfactory result.

In addition, sometimes Ollama isn’t actually that slow to respond. For example, when I run DeepSeek 70b, it may take 5–10 minutes to load the model, during which both my CPU and GPU are fully engaged. That’s really just about the time it takes to have a cup of tea, and I consider that an acceptable wait.

@AnalogKnight commented on GitHub (Sep 29, 2025): Thank you for your kind response. For me personally, I think this is important. My hardware specs aren’t too bad—I have 2×RTX 3090s and over a hundred gigabytes of memory. In most of my use cases, deploying a model like GPT-OSS:20b is more than sufficient. However, when I do something like asking the AI to continue writing a very long article, smaller models can never maintain contextual coherence between the new content and the original text. I agree that simply running the largest possible model most of the time indeed just a bad experience in general. But if the real issue is whether the work can be done or not, then perhaps the quality of the experience becomes less critical. Although I may rarely need the AI to handle such demanding tasks, if I do, I can let a larger model take care of it while I go to sleep. With a smaller model, though, I might have to make it repeat the task dozens or even hundreds of times, and still end up without a satisfactory result. In addition, sometimes Ollama isn’t actually that slow to respond. For example, when I run DeepSeek 70b, it may take 5–10 minutes to load the model, during which both my CPU and GPU are fully engaged. That’s really just about the time it takes to have a cup of tea, and I consider that an acceptable wait.
yindo commented 2026-02-22 18:31:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 29, 2025):

Looking at docs, there is OLLAMA_LOAD_TIMEOUT which explicitly allows you to override the timeout applied for Ollama to load a large model, however, this isnt exactly the issu,e just thought it was worth bringing up.

For the long lived request we can apply something like this to override the 5m fetch default, which isnt a configurable option normally.

@timothycarambat commented on GitHub (Sep 29, 2025): Looking at docs, there is [`OLLAMA_LOAD_TIMEOUT`](https://github.com/ollama/ollama/issues/5081#issuecomment-2458513769) which explicitly allows you to override the timeout applied for Ollama to load a large model, however, this isnt exactly the issu,e just thought it was worth bringing up. For the long lived request we can apply [something like this](https://github.com/karakeep-app/karakeep/issues/628#issuecomment-2469683894) to override the 5m `fetch` default, which isnt a configurable option normally.
yindo commented 2026-02-22 18:31:27 -05:00
Author
Owner
Copy Link
Copy Source

@AnalogKnight commented on GitHub (Sep 30, 2025):

Thanks so much for the quick update! I was wondering—if I’d like to try out this change, does AnythingLLM provide a nightly build or something similar, or would I need to build it myself?

@AnalogKnight commented on GitHub (Sep 30, 2025): Thanks so much for the quick update! I was wondering—if I’d like to try out this change, does AnythingLLM provide a nightly build or something similar, or would I need to build it myself?
yindo commented 2026-02-22 18:31:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 1, 2025):

That is always the latest tag on Docker. You can rebuild it yourself if you really would like to, but every merged commit to master starts a new latest build. Because we build for amd64 and arm64 via QEMU that new build takes ~1hr.

Build: https://github.com/Mintplex-Labs/anything-llm/actions/runs/18108443363

If we do a merge into master while a build is occurring, the current build is killed and restarted with the past + new changes - so latest is always the newest code but has the tradeoff of potentially being unstable.

@timothycarambat commented on GitHub (Oct 1, 2025): That is always the `latest` tag on Docker. You can rebuild it yourself if you really would like to, but every merged commit to `master` starts a new `latest` build. Because we build for amd64 and arm64 via QEMU that new build takes ~1hr. Build: https://github.com/Mintplex-Labs/anything-llm/actions/runs/18108443363 If we do a merge into master while a build is occurring, the current build is killed and restarted with the past + new changes - so `latest` is always the newest code but has the tradeoff of potentially being unstable.
yindo commented 2026-02-22 18:31:27 -05:00
Author
Owner
Copy Link
Copy Source

@AnalogKnight commented on GitHub (Oct 2, 2025):

Image

Hi, I tried this, but it doesn't seem to work. I set both OLLAMA_REQUEST_TIMEOUT and OLLAMA_RESPONSE_TIMEOUT to 99999999, but the problem persists.
I also tried setting them both to 1, but AnythingLLM didn't stop responding immediately either; it still waited for 5 minutes.

Image
@AnalogKnight commented on GitHub (Oct 2, 2025): <img width="3384" height="216" alt="Image" src="https://github.com/user-attachments/assets/265f66bb-b1aa-4966-bd43-89621c94001c" /> Hi, I tried this, but it doesn't seem to work. I set both OLLAMA_REQUEST_TIMEOUT and OLLAMA_RESPONSE_TIMEOUT to 99999999, but the problem persists. I also tried setting them both to 1, but AnythingLLM didn't stop responding immediately either; it still waited for 5 minutes. <img width="1052" height="641" alt="Image" src="https://github.com/user-attachments/assets/eef67c00-8502-4f56-ad0d-654c4bc96e00" />
yindo commented 2026-02-22 18:31:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 2, 2025):

Are you setting this as a container ENV (eg: -e ....)? That is not how AnythingLLM config vars are set.
Open the folder you are binding the storage to when booting the container in file explorer/terminal.

Example given this start command

export STORAGE_LOCATION=$HOME/anythingllm && \
mkdir -p $STORAGE_LOCATION && \
touch "$STORAGE_LOCATION/.env" && \
docker run -d -p 3001:3001 \
--cap-add SYS_ADMIN \
-v ${STORAGE_LOCATION}:/app/server/storage \
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
-e STORAGE_DIR="/app/server/storage" \
mintplexlabs/anythingllm:latest

My storage should be $HOME/anythingllm. Open $HOME/anythingllm/.env and add this to the bottom of the config file.
OLLAMA_RESPONSE_TIMEOUT=999999

Restart the container and now the setting will apply 👍

If you set the env in the container during runtime, it will not be applied

@timothycarambat commented on GitHub (Oct 2, 2025): Are you setting this as a container ENV (eg: `-e ....`)? That is not how AnythingLLM config vars are set. Open the folder you are binding the storage to when booting the container in file explorer/terminal. Example given this start command ```bash export STORAGE_LOCATION=$HOME/anythingllm && \ mkdir -p $STORAGE_LOCATION && \ touch "$STORAGE_LOCATION/.env" && \ docker run -d -p 3001:3001 \ --cap-add SYS_ADMIN \ -v ${STORAGE_LOCATION}:/app/server/storage \ -v ${STORAGE_LOCATION}/.env:/app/server/.env \ -e STORAGE_DIR="/app/server/storage" \ mintplexlabs/anythingllm:latest ``` My storage should be `$HOME/anythingllm`. Open `$HOME/anythingllm/.env` and add this to the bottom of the config file. `OLLAMA_RESPONSE_TIMEOUT=999999` Restart the container and now the setting will apply 👍 If you set the env in the container during runtime, it will not be applied
yindo commented 2026-02-22 18:31:28 -05:00
Author
Owner
Copy Link
Copy Source

@AnalogKnight commented on GitHub (Oct 3, 2025):

Thank you, that worked. I'm sorry I had no knowledge in this area and misunderstood it. I really appreciate your patient help.

@AnalogKnight commented on GitHub (Oct 3, 2025): Thank you, that worked. I'm sorry I had no knowledge in this area and misunderstood it. I really appreciate your patient help.
yindo commented 2026-02-22 18:31:28 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 3, 2025):

No problem, in fact, what you did would make sense to 99% of devs, so if anything, this is an improvement among many I can now store in my brain 😄

@timothycarambat commented on GitHub (Oct 3, 2025): No problem, in fact, what you did would make sense to 99% of devs, so if anything, this is an improvement among many I can now store in my brain 😄
yindo changed title from [CHORE]: API disconnect on slow responses to [GH-ISSUE #4443] [CHORE]: API disconnect on slow responses 2026-06-05 14:48:48 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2831
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?