Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4180] [Ops Guidance] - Docker container fails to start due to local storage exhaustion and EFS permission issues #2660

New Issue
Closed
opened 2026-02-22 18:30:39 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:30:39 -05:00
Owner
Copy Link
Copy Source

Originally created by @roller100 on GitHub (Jul 21, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4180

How are you running AnythingLLM?

Docker on a remote AWS EC2 instance with an EFS volume for persistent storage.

Our Experience

We are sharing our experience deploying AnythingLLM on an AWS EC2 instance where the application would become unresponsive over time. Our investigation revealed that the instance's local storage was filling up, causing instability and preventing the application and its MCP servers from starting correctly.

What We Observed

  1. Disk Space Exhaustion: The EC2 instance's root volume (/dev/nvme0n1p1) reached 80% capacity. A detailed investigation showed that the /var/lib/docker/overlay2 directory was consuming nearly 6GB of space. This was caused by the running container's writable layer growing over time.
  2. MCP Server Failure: Our Neo4j MCP server was failing to start. The application logs showed a "Connection closed" error, which we later found was caused by a permissions issue.
  3. Permission Errors: After attempting to fix the storage issue by mounting a host directory to /app/.cache, the container would fail to start. The logs showed a clear error: error: failed to create directory /app/.cache/uv: Permission denied (os error 13).

Our Environment

  • AWS EC2 instance with limited local storage
  • Docker container running mintplexplexlabs/anythingllm:latest
  • AWS EFS for persistent storage, mounted at /mnt/efs on the host.
  • MCP server configured via anythingllm_mcp_servers.json.

Relevant Logs

The key error that led us to the solution was the permission issue:

error: failed to create directory /app/.cache/uv: Permission denied (os error 13)

This was followed by the MCP server failure:

[backend] info: [MCPCompatibilityLayer] Failed to start MCP server: {"error":"MCP error -32000: Connection closed",...}

How We Addressed It

We identified two distinct problems and implemented the following solutions:

  1. Redirecting Cache to EFS: The primary issue was that the application's cache (/app/.cache) was being written to the container's temporary local storage. We redirected this to the scalable EFS volume by adding a volume mount to our docker run command.
  2. Correcting EFS Permissions: The "Permission denied" error occurred because the directory we created on the EFS volume was owned by root. The application, running as a non-root user, could not write to it. We resolved this by setting open permissions on the EFS directory: sudo chmod 777 /mnt/efs/cache.

Our final, stable docker run command is:

sudo docker run -d -p XXXX:XXXX --cap-add SYS_ADMIN \
  --memory=3g --memory-swap=3g \
  -v /mnt/efs:/app/server/storage \
  -v /mnt/efs/.env:/app/server/.env \
  -v /mnt/efs/cache:/app/.cache \
  -e STORAGE_DIR=/app/server/storage \
  mintplexlabs/anythingllm:latest

Ideas for Discussion

  1. Documentation: Could the official Docker deployment documentation be updated to recommend mounting a volume for /app/.cache? This seems like a critical step for long-term stability.
  2. Permissions: It would be helpful to add a note in the documentation about ensuring correct file permissions when mounting host volumes, as this was a tricky issue to diagnose.
  3. Error Handling: Could the application provide a clearer, more direct error message on startup if it detects that its essential directories (like the cache) are not writable?

Community Benefit

We hope sharing this detailed experience helps others who are deploying AnythingLLM in a similar AWS environment. By correctly managing storage and permissions, the platform can run in a much more stable and scalable manner.

Originally created by @roller100 on GitHub (Jul 21, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4180 **How are you running AnythingLLM?** Docker on a remote AWS EC2 instance with an EFS volume for persistent storage. **Our Experience** We are sharing our experience deploying AnythingLLM on an AWS EC2 instance where the application would become unresponsive over time. Our investigation revealed that the instance's local storage was filling up, causing instability and preventing the application and its MCP servers from starting correctly. **What We Observed** 1. **Disk Space Exhaustion**: The EC2 instance's root volume (`/dev/nvme0n1p1`) reached 80% capacity. A detailed investigation showed that the `/var/lib/docker/overlay2` directory was consuming nearly 6GB of space. This was caused by the running container's writable layer growing over time. 2. **MCP Server Failure**: Our Neo4j MCP server was failing to start. The application logs showed a "Connection closed" error, which we later found was caused by a permissions issue. 3. **Permission Errors**: After attempting to fix the storage issue by mounting a host directory to `/app/.cache`, the container would fail to start. The logs showed a clear error: `error: failed to create directory /app/.cache/uv: Permission denied (os error 13)`. **Our Environment** * AWS EC2 instance with limited local storage * Docker container running `mintplexplexlabs/anythingllm:latest` * AWS EFS for persistent storage, mounted at `/mnt/efs` on the host. * MCP server configured via `anythingllm_mcp_servers.json`. **Relevant Logs** The key error that led us to the solution was the permission issue: ``` error: failed to create directory /app/.cache/uv: Permission denied (os error 13) ``` This was followed by the MCP server failure: ``` [backend] info: [MCPCompatibilityLayer] Failed to start MCP server: {"error":"MCP error -32000: Connection closed",...} ``` **How We Addressed It** We identified two distinct problems and implemented the following solutions: 1. **Redirecting Cache to EFS**: The primary issue was that the application's cache (`/app/.cache`) was being written to the container's temporary local storage. We redirected this to the scalable EFS volume by adding a volume mount to our `docker run` command. 2. **Correcting EFS Permissions**: The "Permission denied" error occurred because the directory we created on the EFS volume was owned by `root`. The application, running as a non-root user, could not write to it. We resolved this by setting open permissions on the EFS directory: `sudo chmod 777 /mnt/efs/cache`. Our final, stable `docker run` command is: ``` sudo docker run -d -p XXXX:XXXX --cap-add SYS_ADMIN \ --memory=3g --memory-swap=3g \ -v /mnt/efs:/app/server/storage \ -v /mnt/efs/.env:/app/server/.env \ -v /mnt/efs/cache:/app/.cache \ -e STORAGE_DIR=/app/server/storage \ mintplexlabs/anythingllm:latest ``` **Ideas for Discussion** 1. **Documentation**: Could the official Docker deployment documentation be updated to recommend mounting a volume for `/app/.cache`? This seems like a critical step for long-term stability. 2. **Permissions**: It would be helpful to add a note in the documentation about ensuring correct file permissions when mounting host volumes, as this was a tricky issue to diagnose. 3. **Error Handling**: Could the application provide a clearer, more direct error message on startup if it detects that its essential directories (like the cache) are not writable? **Community Benefit** We hope sharing this detailed experience helps others who are deploying AnythingLLM in a similar AWS environment. By correctly managing storage and permissions, the platform can run in a much more stable and scalable manner.
yindo closed this issue 2026-02-22 18:30:39 -05:00
yindo commented 2026-02-22 18:30:40 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 21, 2025):

All of this is user-side implementation and configuration help. It's worth a doc, but not really a GitHub issue - there is nothing for us to action on here.

@timothycarambat commented on GitHub (Jul 21, 2025): All of this is user-side implementation and configuration help. It's worth a doc, but not really a GitHub issue - there is nothing for us to action on here.
yindo commented 2026-02-22 18:30:40 -05:00
Author
Owner
Copy Link
Copy Source

@roller100 commented on GitHub (Jul 21, 2025):

Agreed @timothycarambat - more sharing for the community to share learning and Op Guidance.

@roller100 commented on GitHub (Jul 21, 2025): Agreed @timothycarambat - more sharing for the community to share learning and Op Guidance.
yindo changed title from [Ops Guidance] - Docker container fails to start due to local storage exhaustion and EFS permission issues to [GH-ISSUE #4180] [Ops Guidance] - Docker container fails to start due to local storage exhaustion and EFS permission issues 2026-06-05 14:47:47 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2660
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?