[GH-ISSUE #1745] [BUG]: Ollama models not loading with http://172.17.0.1:11434 on Debian 12 #1137

Closed
opened 2026-02-22 18:23:18 -05:00 by yindo · 11 comments
Owner

Originally created by @Mattias-Code on GitHub (Jun 22, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1745

How are you running AnythingLLM?

Docker (local)

What happened?

Stuck at loading Ollama models, verified that Ollama is running on 127.0.0.1:11434

and used 172.17.0.1:11434 as url according to documentation.

Screenshot from 2024-06-22 17-35-40

AnythingLLM logs:

[NativeEmbedder] Initialized
TypeError: fetch failed
at Object.fetch (node:internal/deps/undici/undici:11731:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async ollamaAIModels (/app/server/utils/helpers/customModels.js:245:18)
at async getCustomModels (/app/server/utils/helpers/customModels.js:33:14)
at async /app/server/endpoints/system.js:925:35 {
cause: Error: connect ECONNREFUSED 172.17.0.1:11434
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '172.17.0.1',
port: 11434
}
}

Really appreciate the job that you guys put down to make this software a reality. I've had a flawless time using AnythingLLM with Anthropic API!

Are there known steps to reproduce?

No response

Originally created by @Mattias-Code on GitHub (Jun 22, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1745 ### How are you running AnythingLLM? Docker (local) ### What happened? Stuck at loading Ollama models, verified that Ollama is running on 127.0.0.1:11434 and used 172.17.0.1:11434 as url according to documentation. ![Screenshot from 2024-06-22 17-35-40](https://github.com/Mintplex-Labs/anything-llm/assets/172132794/cbe72700-ca51-49d2-8ef6-e773b7b2849e) AnythingLLM logs: [NativeEmbedder] Initialized TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11731:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async ollamaAIModels (/app/server/utils/helpers/customModels.js:245:18) at async getCustomModels (/app/server/utils/helpers/customModels.js:33:14) at async /app/server/endpoints/system.js:925:35 { cause: Error: connect ECONNREFUSED 172.17.0.1:11434 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16) { errno: -111, code: 'ECONNREFUSED', syscall: 'connect', address: '172.17.0.1', port: 11434 } } Really appreciate the job that you guys put down to make this software a reality. I've had a flawless time using AnythingLLM with Anthropic API! ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:23:18 -05:00
yindo closed this issue 2026-02-22 18:23:18 -05:00
Author
Owner

@alex-dna-tech commented on GitHub (Jun 23, 2024):

The same problem on MacOS Desktop version v1.5.7
Screenshot 2024-06-23 at 19 14 38

Screenshot 2024-06-23 at 19 17 20
@alex-dna-tech commented on GitHub (Jun 23, 2024): The same problem on MacOS Desktop version v1.5.7 <img width="1137" alt="Screenshot 2024-06-23 at 19 14 38" src="https://github.com/Mintplex-Labs/anything-llm/assets/74361701/f8763893-88da-4a32-b291-1be09dc4cd21"> <img width="1126" alt="Screenshot 2024-06-23 at 19 17 20" src="https://github.com/Mintplex-Labs/anything-llm/assets/74361701/a8845277-4baa-4ba2-8242-a480268a678f">
Author
Owner

@timothycarambat commented on GitHub (Jun 23, 2024):

The connector does work, you just are using the wrong URL or Ollama is actually not running on the host.
assuming both are on the same machine and you _confirmed that ollama serve is running.

If on Docker:

  • you need to specific the host machine IP or you can use HTTP://host.docker.internal to access the host machine
    However, on Ubuntu/Linux docker the above URL does not work and you need to use 172.17.0.1:11434. If this is not your situation you dont need to use this URL. This also presumes you have not already tried to boot Ollama with the proper ENVs for ollama to be able to listen on the correct loopback. 172.17.0.1 = 0.0.0.0

On desktop:
HTTP://127.0.01:11434

Things to read about this:
https://docs.useanything.com/ollama-models-not-loading
https://github.com/Mintplex-Labs/anything-llm/blob/master/server/utils/AiProviders/ollama/README.md

@timothycarambat commented on GitHub (Jun 23, 2024): The connector does work, you just are using the wrong URL or Ollama is actually not running on the host. _assuming both are on the same machine_ and you _confirmed that `ollama serve` is running. If on Docker: - you need to specific the host machine IP or you can use HTTP://host.docker.internal to access the host machine **However**, on Ubuntu/Linux docker the above URL does not work and you need to use 172.17.0.1:11434. If this is not your situation you dont need to use this URL. This also presumes you have not already tried to boot Ollama with the proper ENVs for ollama to be able to listen on the correct loopback. 172.17.0.1 = 0.0.0.0 On desktop: HTTP://127.0.01:11434 Things to read about this: https://docs.useanything.com/ollama-models-not-loading https://github.com/Mintplex-Labs/anything-llm/blob/master/server/utils/AiProviders/ollama/README.md
Author
Owner

@Mattias-Code commented on GitHub (Jun 24, 2024):

After following the documentation at https://github.com/Mintplex-Labs/anything-llm/blob/master/server/utils/AiProviders/ollama/README.md I managed to resolve the issue by binding Ollama to 0.0.0.0 and using 172.17.0.1:11434 as base URL.

Thank you. For future problems I'll take a closer look at the github before creating a new issue!

@Mattias-Code commented on GitHub (Jun 24, 2024): After following the documentation at https://github.com/Mintplex-Labs/anything-llm/blob/master/server/utils/AiProviders/ollama/README.md I managed to resolve the issue by binding Ollama to 0.0.0.0 and using 172.17.0.1:11434 as base URL. Thank you. For future problems I'll take a closer look at the github before creating a new issue!
Author
Owner

@timothycarambat commented on GitHub (Jun 24, 2024):

@Mattias-Code Thanks for the reply + resolution. Also no big deal, we get Ollama connection issues about 5-10 times a week because of that caveat on address binding Ollama has - you are not alone!

@timothycarambat commented on GitHub (Jun 24, 2024): @Mattias-Code Thanks for the reply + resolution. Also no big deal, we get Ollama connection issues about 5-10 times a week because of that caveat on address binding Ollama has - you are not alone!
Author
Owner

@QRMarketing commented on GitHub (Aug 17, 2024):

Not sure whether this is an issue or not - since this is closed. But ... I read all the links the env variables are all ok. Every time I restart the server the only way I have been able to wake up ollama is to go and get it to auto detect the end point and save it. Quite annoying ... but it is what it is.

I notice a JavaScript error on the save - so maybe it doesn't really save it - so the server restart doesn't remember. Though the toast say it was saved.

index-a13552ba.js:203 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'preventDefault')
at handleSubmit (index-a13552ba.js:203:7)

function GeneralLLMPreference() {
var _a, _b;
const [saving, setSaving] = reactExports.useState(false);
const [hasChanges, setHasChanges] = reactExports.useState(false);
const [settings, setSettings] = reactExports.useState(null);
const [loading, setLoading] = reactExports.useState(true);
const [searchQuery, setSearchQuery] = reactExports.useState("");
const [filteredLLMs, setFilteredLLMs] = reactExports.useState([]);
const [selectedLLM, setSelectedLLM] = reactExports.useState(null);
const [searchMenuOpen, setSearchMenuOpen] = reactExports.useState(false);
const searchInputRef = reactExports.useRef(null);
const isHosted = window.location.hostname.includes("useanything.com");
const { t } = useTranslation();
const handleSubmit = async (e) => { // e is undefined
e.preventDefault();
const form = e.target;
const data = { LLMProvider: selectedLLM };
const formData = new FormData(form);
for (var [key, value] of formData.entries())
data[key] = value;
const { error } = await System.updateSystem(data);
setSaving(true);
if (error) {
showToast(Failed to save LLM settings: ${error}, "error");
} else {
showToast("LLM preferences saved successfully.", "success");
}
setSaving(false);
setHasChanges(!!error);
};

@QRMarketing commented on GitHub (Aug 17, 2024): Not sure whether this is an issue or not - since this is closed. But ... I read all the links the env variables are all ok. Every time I restart the server the only way I have been able to wake up ollama is to go and get it to auto detect the end point and save it. Quite annoying ... but it is what it is. I notice a JavaScript error on the save - so maybe it doesn't really save it - so the server restart doesn't remember. Though the toast say it was saved. index-a13552ba.js:203 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'preventDefault') at handleSubmit (index-a13552ba.js:203:7) function GeneralLLMPreference() { var _a, _b; const [saving, setSaving] = reactExports.useState(false); const [hasChanges, setHasChanges] = reactExports.useState(false); const [settings, setSettings] = reactExports.useState(null); const [loading, setLoading] = reactExports.useState(true); const [searchQuery, setSearchQuery] = reactExports.useState(""); const [filteredLLMs, setFilteredLLMs] = reactExports.useState([]); const [selectedLLM, setSelectedLLM] = reactExports.useState(null); const [searchMenuOpen, setSearchMenuOpen] = reactExports.useState(false); const searchInputRef = reactExports.useRef(null); const isHosted = window.location.hostname.includes("useanything.com"); const { t } = useTranslation(); const handleSubmit = async (e) => { // e is undefined e.preventDefault(); const form = e.target; const data = { LLMProvider: selectedLLM }; const formData = new FormData(form); for (var [key, value] of formData.entries()) data[key] = value; const { error } = await System.updateSystem(data); setSaving(true); if (error) { showToast(`Failed to save LLM settings: ${error}`, "error"); } else { showToast("LLM preferences saved successfully.", "success"); } setSaving(false); setHasChanges(!!error); };
Author
Owner

@QRMarketing commented on GitHub (Aug 17, 2024):

For those that care. I did fix this. Yes, I set up the environment variables and the .env to no avail. I suppose there is a real fix either in the code or my environment. But I did a hack in the server at startup to get it to work:

In server.js near start of the script:
const { ResetOllama } = require('./ollamareset.js'); /* kluge wake up ollama */ setTimeout(async () => { await ResetOllama(); }, 1000);
And ollamareset.js':

`
const { updateENV } = require("./utils/helpers/updateENV");

async function ResetOllama() {

if(process.env.LLM_PROVIDER !== "ollama") return;

console.log("Wake up Ollama.");

try {
    
    const body = {
        LLMProvider: 'ollama',
        OllamaLLMBasePath: 'http://127.0.0.1:11434',
        OllamaLLMKeepAliveSeconds: '300',
        OllamaLLMModelPref: 'llama3.1:latest',
        OllamaLLMTokenLimit: '4096',
    };
    
    await updateENV(
      body,
      true
    );
  } catch (e) {
    console.error(e.message, e);
  }

}

module.exports = {
ResetOllama
};
`

@QRMarketing commented on GitHub (Aug 17, 2024): For those that care. I did fix this. Yes, I set up the environment variables and the .env to no avail. I suppose there is a real fix either in the code or my environment. But I did a hack in the server at startup to get it to work: In server.js near start of the script: ` const { ResetOllama } = require('./ollamareset.js'); /* kluge wake up ollama */ setTimeout(async () => { await ResetOllama(); }, 1000); ` And ollamareset.js': ` const { updateENV } = require("./utils/helpers/updateENV"); async function ResetOllama() { if(process.env.LLM_PROVIDER !== "ollama") return; console.log("Wake up Ollama."); try { const body = { LLMProvider: 'ollama', OllamaLLMBasePath: 'http://127.0.0.1:11434', OllamaLLMKeepAliveSeconds: '300', OllamaLLMModelPref: 'llama3.1:latest', OllamaLLMTokenLimit: '4096', }; await updateENV( body, true ); } catch (e) { console.error(e.message, e); } } module.exports = { ResetOllama }; `
Author
Owner

@timothycarambat commented on GitHub (Aug 18, 2024):

If you have to do that everytime you start the app then the app is not saving information to the .env file? Are you in Docker?

The error with the no preventDefault is especially off, since that is just a handler for the form, where e would for sure exists so we need to check that out as well. What is your environment?

@timothycarambat commented on GitHub (Aug 18, 2024): If you have to do that everytime you start the app then the app is not saving information to the .env file? Are you in Docker? The error with the no preventDefault is especially off, since that is just a handler for the form, where `e` would for sure exists so we need to check that out as well. What is your environment?
Author
Owner
@timothycarambat commented on GitHub (Aug 18, 2024): https://github.com/Mintplex-Labs/anything-llm/blob/a8d25c7dd31e90b06c9ea8575ada2321928dcfa4/frontend/src/pages/GeneralSettings/LLMPreference/index.jsx#L341
Author
Owner

@QRMarketing commented on GitHub (Aug 18, 2024):

STOP wasting your time researching my Agent Configuration problem - it was my stupidity in not setting the Agent Configuration correctly. The issues of the Javascript errors and my kluge fix is another matter. With all this it is now functioning.

Below is my .env same as well as my Windows environment variables. Yes stand alone on Windows. Everything works, Embed widget and workspace chats, etc ... (with the kluge fix - but i removed it and it works as well, you just need to resave the LLM preference each server restart to wake it up.). The only thing that does not work, which is all related to this point; I discovered is Agents (yes refreshed source - I see you have made recent changes) see below. Why the other functions see ollama but not Agents?

There seems to be a discussion over this OLLAMA_BASE_PATH='http://127.17.0.1:11434' or OLLAMA_BASE_PATH='http://127.0.0.1:11434'. http://127.17.0.1:11434 nothing works....

As far as the Javascript error you mention. I put a break point on it. It seems to get there twice. First time with e undefined and the second time ok. So the function actually works (i.e. saves the LLM selection.)

Any help would be great.

Adrian

The error log:

PS C:\Users\User\Desktop\Any\anything-llm\server> [TELEMETRY SENT] sent_chat
sent_chat { workspaceName: 'ecologene', chatModel: 'llama3.1:latest' } User: 1
[AgentHandler] Start 88a7d05f-baab-49de-aacb-d7d554ff9fba::ollama:llama3:latest
[TELEMETRY SENT] agent_chat_started
[AgentHandler] Attached websocket plugin to Agent cluster
[AgentHandler] Attached chat-history plugin to Agent cluster
[AgentHandler] Attaching user and default agent to Agent cluster.
[AgentHandler] Attached rag-memory plugin to Agent cluster
[AgentHandler] Attached document-summarizer plugin to Agent cluster
[AgentHandler] Attached web-scraping plugin to Agent cluster
[AgentHandler] Attached create-chart plugin to Agent cluster
model "llama3:latest" not found, try pulling it first ResponseError: model "llama3:latest" not found, try pulling it first
at checkOk (C:\Users\User\Desktop\Any\anything-llm\server\node_modules\ollama\dist\shared\ollama.a247cdd6.cjs:38:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async post (C:\Users\User\Desktop\Any\anything-llm\server\node_modules\ollama\dist\shared\ollama.a247cdd6.cjs:86:3)
at async Ollama.processStreamableRequest (C:\Users\User\Desktop\Any\anything-llm\server\node_modules\ollama\dist\shared\ollama.a247cdd6.cjs:183:22)
at async #handleFunctionCallChat (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\providers\ollama.js:29:22)
at async OllamaProvider.functionCall (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\providers\helpers\untooled.js:110:22)
at async OllamaProvider.complete (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\providers\ollama.js:50:36)
at async AIbitat.handleExecution (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\index.js:592:24)
at async AIbitat.reply (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\index.js:574:21)
at async AIbitat.chat (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\index.js:374:15) {
error: 'model "llama3:latest" not found, try pulling it first',
status_code: 404
}
[AgentHandler] End 88a7d05f-baab-49de-aacb-d7d554ff9fba::ollama:llama3:latest

My .env
######## ANYTHINGLLM SETTINGS ############
LLM_PROVIDER='ollama'
OLLAMA_HOST='0.0.0.0'
OLLAMA_BASE_PATH='http://127.0.0.1:11434'
OLLAMA_MODEL_PREF='llama3:latest'
OLLAMA_MODEL_TOKEN_LIMIT='4096'
OLLAMA_KEEP_ALIVE_TIMEOUT='300'

@QRMarketing commented on GitHub (Aug 18, 2024): **STOP wasting your time researching my Agent Configuration problem** - it was my stupidity in not setting the Agent Configuration correctly. The issues of the Javascript errors and my kluge fix is another matter. With all this it is now functioning. Below is my .env same as well as my Windows environment variables. Yes stand alone on Windows. Everything works, Embed widget and workspace chats, etc ... (with the kluge fix - but i removed it and it works as well, you just need to resave the LLM preference each server restart to wake it up.). **The only thing that does not work, which is all related to this point; I discovered is Agents (yes refreshed source - I see you have made recent changes) see below.** Why the other functions see ollama but not Agents? There seems to be a discussion over this OLLAMA_BASE_PATH='http://127.17.0.1:11434' or OLLAMA_BASE_PATH='http://127.0.0.1:11434'. http://127.17.0.1:11434 nothing works.... As far as the Javascript error you mention. I put a break point on it. It seems to get there twice. First time with e undefined and the second time ok. So the function actually works (i.e. saves the LLM selection.) Any help would be great. ![Adrian](https://github.com/user-attachments/assets/5be7b7a6-aefc-49a3-9118-46311a934204) The error log: PS C:\Users\User\Desktop\Any\anything-llm\server> [TELEMETRY SENT] sent_chat sent_chat { workspaceName: 'ecologene', chatModel: 'llama3.1:latest' } User: 1 [AgentHandler] Start 88a7d05f-baab-49de-aacb-d7d554ff9fba::ollama:llama3:latest [TELEMETRY SENT] agent_chat_started [AgentHandler] Attached websocket plugin to Agent cluster [AgentHandler] Attached chat-history plugin to Agent cluster [AgentHandler] Attaching user and default agent to Agent cluster. [AgentHandler] Attached rag-memory plugin to Agent cluster [AgentHandler] Attached document-summarizer plugin to Agent cluster [AgentHandler] Attached web-scraping plugin to Agent cluster [AgentHandler] Attached create-chart plugin to Agent cluster model "llama3:latest" not found, try pulling it first ResponseError: model "llama3:latest" not found, try pulling it first at checkOk (C:\Users\User\Desktop\Any\anything-llm\server\node_modules\ollama\dist\shared\ollama.a247cdd6.cjs:38:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async post (C:\Users\User\Desktop\Any\anything-llm\server\node_modules\ollama\dist\shared\ollama.a247cdd6.cjs:86:3) at async Ollama.processStreamableRequest (C:\Users\User\Desktop\Any\anything-llm\server\node_modules\ollama\dist\shared\ollama.a247cdd6.cjs:183:22) at async #handleFunctionCallChat (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\providers\ollama.js:29:22) at async OllamaProvider.functionCall (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\providers\helpers\untooled.js:110:22) at async OllamaProvider.complete (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\providers\ollama.js:50:36) at async AIbitat.handleExecution (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\index.js:592:24) at async AIbitat.reply (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\index.js:574:21) at async AIbitat.chat (C:\Users\User\Desktop\Any\anything-llm\server\utils\agents\aibitat\index.js:374:15) { error: 'model "llama3:latest" not found, try pulling it first', status_code: 404 } [AgentHandler] End 88a7d05f-baab-49de-aacb-d7d554ff9fba::ollama:llama3:latest My .env ######## ANYTHINGLLM SETTINGS ############ LLM_PROVIDER='ollama' OLLAMA_HOST='0.0.0.0' OLLAMA_BASE_PATH='http://127.0.0.1:11434' OLLAMA_MODEL_PREF='llama3:latest' OLLAMA_MODEL_TOKEN_LIMIT='4096' OLLAMA_KEEP_ALIVE_TIMEOUT='300'
Author
Owner

@timothycarambat commented on GitHub (Aug 19, 2024):

Seems like the connection works fine.

error: 'model "llama3:latest" not found, try pulling it first',

The model you have set for agents (if set at all) is not installed in that Ollama service so it can't invoke llama 3. ollama pull llama3:latest if that is what you are tying to run.

OLLAMA_MODEL_PREF='llama3:latest'

This sets the model you want to use, it does not download it for you in Ollama. You still need to run ollama pull llama3:latest

@timothycarambat commented on GitHub (Aug 19, 2024): Seems like the connection works fine. > error: 'model "llama3:latest" not found, try pulling it first', The model you have set for agents (if set at all) is not installed in that Ollama service so it can't invoke llama 3. `ollama pull llama3:latest` if that is what you are tying to run. > OLLAMA_MODEL_PREF='llama3:latest' This sets the model you want to use, it does not download it for you in Ollama. You still need to run `ollama pull llama3:latest`
Author
Owner

@Kentauros10710 commented on GitHub (Dec 9, 2024):

Based on my own testing, I ultimately managed it by downloading Ollama using Docker as well. Since my Docker environment is running both Anything LLM and Ollama simultaneously, they are able to interact and integrate with each other.

@Kentauros10710 commented on GitHub (Dec 9, 2024): Based on my own testing, I ultimately managed it by downloading Ollama using Docker as well. Since my Docker environment is running both Anything LLM and Ollama simultaneously, they are able to interact and integrate with each other.
yindo changed title from [BUG]: Ollama models not loading with http://172.17.0.1:11434 on Debian 12 to [GH-ISSUE #1745] [BUG]: Ollama models not loading with http://172.17.0.1:11434 on Debian 12 2026-06-05 14:39:09 -04:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1137