Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1644] [BUG]: LLMs not loading #1070

New Issue
Closed
opened 2026-02-22 18:22:59 -05:00 by yindo · 6 comments
yindo commented 2026-02-22 18:22:59 -05:00
Owner
Copy Link
Copy Source

Originally created by @HerroHK on GitHub (Jun 9, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1644

How are you running AnythingLLM?

Docker (remote machine)

What happened?

My setup and issue:

  1. Ubuntu 22.04 server with Ollama, WebUI, ChromaDB, and AnythingLLM in office environment
  2. AnythingLLM and WebUI are running in Docker container
  3. Ollama w/ WebUI is running properly, I have 2 Llama3 manifests/models and both are available via WebUI
  4. Ollama via terminal is properly running
  5. Just a notification that WebUI is running on host network (127.17.01 / 127.0.0.1) and AnythingLLM on bridge network (127.17.0.5) - I can manage this via Portainer.
  6. Yet, I have issues (still) loading the models/manifests to AnythingLLM (I know, reported before) - no models load!

Resolutions tried:

  1. I have tried to use http://host.docker.internal:11434
  2. I have tried http://localhost:11434
  3. I have tried http://127.17.0.1:11434
  4. I have tried the subnet IP 192.168.166.19:11434 for this device
  5. I have tried changing Ollama port (as it was a suggested this may fix it)
  6. I have uninstalled and reinstalled my whole setup
  7. I have tried the "Native" model option but it doesn't load any manifests

Seems I am unable to load manifests/models. What other possible solutions do I have? Just thinking what the issue could be, e.g. bridge networks not having permission to see the models on host? Or a routing issue (e.g. 127.17.0.1:11434 not being forwarded to 127.0.0.1:11434?)?

Any input welcomed. And this seems to be a real bug as I have seen several reports on this.

Are there known steps to reproduce?

No response

Originally created by @HerroHK on GitHub (Jun 9, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1644 ### How are you running AnythingLLM? Docker (remote machine) ### What happened? My setup and issue: 1. Ubuntu 22.04 server with Ollama, WebUI, ChromaDB, and AnythingLLM in office environment 2. AnythingLLM and WebUI are running in Docker container 3. Ollama w/ WebUI is running properly, I have 2 Llama3 manifests/models and both are available via WebUI 4. Ollama via terminal is properly running 5. Just a notification that WebUI is running on host network (127.17.01 / 127.0.0.1) and AnythingLLM on bridge network (127.17.0.5) - I can manage this via Portainer. 6. Yet, I have issues (still) loading the models/manifests to AnythingLLM (I know, reported before) - no models load! Resolutions tried: 1. I have tried to use http://host.docker.internal:11434 2. I have tried http://localhost:11434 3. I have tried http://127.17.0.1:11434 4. I have tried the subnet IP 192.168.166.19:11434 for this device 5. I have tried changing Ollama port (as it was a suggested this may fix it) 7. I have uninstalled and reinstalled my whole setup 8. I have tried the "Native" model option but it doesn't load any manifests Seems I am unable to load manifests/models. What other possible solutions do I have? Just thinking what the issue could be, e.g. bridge networks not having permission to see the models on host? Or a routing issue (e.g. 127.17.0.1:11434 not being forwarded to 127.0.0.1:11434?)? Any input welcomed. And this seems to be a real bug as I have seen several reports on this. ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:22:59 -05:00
yindo closed this issue 2026-02-22 18:22:59 -05:00
yindo commented 2026-02-22 18:22:59 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 10, 2024):

Since you are on Ubuntu the host.docker.internal gateway does not exist (frustrating hidden feature of Docker on Ubuntu)
Try http://172.17.0.1:11434

Backup reference if even that does not work:
https://github.com/Mintplex-Labs/anything-llm/blob/master/server/utils/AiProviders/ollama/README.md

@timothycarambat commented on GitHub (Jun 10, 2024): Since you are on Ubuntu the `host.docker.internal` gateway does not exist (frustrating hidden feature of Docker on Ubuntu) Try `http://172.17.0.1:11434` Backup reference if even that does not work: https://github.com/Mintplex-Labs/anything-llm/blob/master/server/utils/AiProviders/ollama/README.md
yindo commented 2026-02-22 18:22:59 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 10, 2024):

172.17.0.1 will map to 127.0.0.1 by the way, so you may need to do some additional routing if that ip/port is occupied

@timothycarambat commented on GitHub (Jun 10, 2024): 172.17.0.1 will map to 127.0.0.1 by the way, so you may need to do some additional routing if that ip/port is occupied
yindo commented 2026-02-22 18:22:59 -05:00
Author
Owner
Copy Link
Copy Source

@HerroHK commented on GitHub (Jun 10, 2024):

Thanks. Initially I received some sort of error as per screenshot below.

But after changing to 172.17.0.5 and back to 172.17.0.1 the error disappeared. but I am once again back to being stuck where I was.

Not sure about the routing part though, I came this far but for the routing it seems I may break more than I can fix when I do.

Screenshot 2024-06-11 at 08 46 46
@HerroHK commented on GitHub (Jun 10, 2024): Thanks. Initially I received some sort of error as per screenshot below. But after changing to 172.17.0.5 and back to 172.17.0.1 the error disappeared. but I am once again back to being stuck where I was. Not sure about the routing part though, I came this far but for the routing it seems I may break more than I can fix when I do. <img width="422" alt="Screenshot 2024-06-11 at 08 46 46" src="https://github.com/Mintplex-Labs/anything-llm/assets/170845944/443881c4-e0e3-413b-985a-509be5d39889">
yindo commented 2026-02-22 18:23:00 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 11, 2024):

@HerroHK are you running AnythingLLM in docker or are you running a local dev environment? I dont see how it would be possible in docker to see the frontend but it fail to communicate with the backend because the frontend is pre-bundled to the backend in the docker image.

@timothycarambat commented on GitHub (Jun 11, 2024): @HerroHK are you running AnythingLLM in docker or are you running a local dev environment? I dont see how it would be possible in docker to _see_ the frontend but it fail to communicate with the backend because the frontend is pre-bundled to the backend in the docker image.
yindo commented 2026-02-22 18:23:00 -05:00
Author
Owner
Copy Link
Copy Source

@HerroHK commented on GitHub (Jun 12, 2024):

I tried several things, including following the instructions here , but all to no avail.

Eventually, I reinstalled using the --network=host flag:
docker run -d --network=host -p 3001:3001 --name AnythingLLM mintplexlabs/anythingllm --restart always

I added ENV for PORT=3001.

This moved the installation to host, where my ollama lives. It is working now.
Thanks for the support.

@HerroHK commented on GitHub (Jun 12, 2024): I tried several things, including [following the instructions here ](https://www.howtogeek.com/devops/how-to-connect-to-localhost-within-a-docker-container/), but all to no avail. Eventually, I reinstalled using the --network=host flag: docker run -d --network=host -p 3001:3001 --name AnythingLLM mintplexlabs/anythingllm --restart always I added ENV for PORT=3001. This moved the installation to host, where my ollama lives. It is working now. Thanks for the support.
yindo commented 2026-02-22 18:23:00 -05:00
Author
Owner
Copy Link
Copy Source

@dadmaan commented on GitHub (Nov 21, 2024):

Adding network_mode: host to the docker-compose.yml file also fixed this for me. See the complete example below:

`
services:
anythingllm:
image: mintplexlabs/anythingllm
container_name: anythingllm
network_mode: host
ports:
- "3001:3001"
cap_add:
- SYS_ADMIN
environment:
# Adjust for your environment
- STORAGE_DIR=/app/server/storage
- JWT_SECRET="make this a large list of random numbers and letters 20+"
- LLM_PROVIDER=ollama
- OLLAMA_BASE_PATH=http://127.0.0.1:11434
- OLLAMA_MODEL_PREF=llama3.1:latest
- OLLAMA_MODEL_TOKEN_LIMIT=4096
- EMBEDDING_ENGINE=ollama
- EMBEDDING_BASE_PATH=http://127.0.0.1:11434
- EMBEDDING_MODEL_PREF=nomic-embed-text:latest
- EMBEDDING_MODEL_MAX_CHUNK_LENGTH=8192
- VECTOR_DB=lancedb
- WHISPER_PROVIDER=local
- TTS_PROVIDER=native
- PASSWORDMINCHAR=8
# Add any other keys here for services or settings
# you can find in the docker/.env.example file
volumes:
- anythingllm_storage:/app/server/storage
restart: always

volumes:
anythingllm_storage:
driver: local
driver_opts:
type: none
o: bind
device: /path/to/storage
`

@dadmaan commented on GitHub (Nov 21, 2024): Adding `network_mode: host` to the `docker-compose.yml` file also fixed this for me. See the complete example below: ` services: anythingllm: image: mintplexlabs/anythingllm container_name: anythingllm network_mode: host ports: - "3001:3001" cap_add: - SYS_ADMIN environment: # Adjust for your environment - STORAGE_DIR=/app/server/storage - JWT_SECRET="make this a large list of random numbers and letters 20+" - LLM_PROVIDER=ollama - OLLAMA_BASE_PATH=http://127.0.0.1:11434 - OLLAMA_MODEL_PREF=llama3.1:latest - OLLAMA_MODEL_TOKEN_LIMIT=4096 - EMBEDDING_ENGINE=ollama - EMBEDDING_BASE_PATH=http://127.0.0.1:11434 - EMBEDDING_MODEL_PREF=nomic-embed-text:latest - EMBEDDING_MODEL_MAX_CHUNK_LENGTH=8192 - VECTOR_DB=lancedb - WHISPER_PROVIDER=local - TTS_PROVIDER=native - PASSWORDMINCHAR=8 # Add any other keys here for services or settings # you can find in the docker/.env.example file volumes: - anythingllm_storage:/app/server/storage restart: always volumes: anythingllm_storage: driver: local driver_opts: type: none o: bind device: /path/to/storage `
yindo changed title from [BUG]: LLMs not loading to [GH-ISSUE #1644] [BUG]: LLMs not loading 2026-06-05 14:38:49 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1070
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?