langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[bug] Memory leak in dify-sandbox 1.3.1 (K8s offline deployment) #15327

New Issue
Closed
opened 2026-02-21 19:21:13 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 19:21:13 -05:00
Owner
Copy Link
Copy Source

Originally created by @Jy1126 on GitHub (Jul 13, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

v1.3.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Deployed with the official Docker images in an offline Kubernetes cluster
    (containerd runtime, no docker-compose).
  2. Deploy Dify v1.3.1 in an offline Kubernetes cluster * K8s v1.28.15 • containerd 1.7.22 • RHEL 8.6 (kernel 4.18)
  3. Run 2 replicas of dify-sandbox with resources 4 CPU / 4 Gi each.
  4. Keep the system under light load (~50 requests/day) for ~60 days.
  5. Monitor container_memory_working_set_bytes (Prometheus)
    or Grafana panel “WorkingSet内存使用量”.
  6. Observe a linear increase until WorkingSet ≈ 3.8 Gi (≈ 95 % of limit);
    at that point dify-api starts throwing time-out errors.

✔️ Expected Behavior

  • WorkingSet memory of dify-sandbox remains stable (or oscillates within a small range) under light traffic.
  • No OOM-kill, no upstream time-outs in dify-api.

❌ Actual Behavior

  • WorkingSet grows linearly from ~0.5 Gi to ~3.8 Gi in ~60 days.
  • When usage > 95 % of the 4 Gi limit, dify-api returns:

Restarting the sandbox pods resets memory to 8Gi; the growth pattern repeats.

  • Sandbox logs constantly print
Originally created by @Jy1126 on GitHub (Jul 13, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version v1.3.1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Deployed with the official Docker images in an **offline Kubernetes cluster** (containerd runtime, no docker-compose). 2. Deploy **Dify v1.3.1** in an **offline** Kubernetes cluster * K8s v1.28.15 • containerd 1.7.22 • RHEL 8.6 (kernel 4.18) 3. Run **2 replicas** of `dify-sandbox` with resources `4 CPU / 4 Gi` each. 4. Keep the system under light load (~50 requests/day) for ~60 days. 5. Monitor `container_memory_working_set_bytes` (Prometheus) or Grafana panel “WorkingSet内存使用量”. 6. Observe a linear increase until WorkingSet ≈ 3.8 Gi (≈ 95 % of limit); at that point `dify-api` starts throwing time-out errors. ### ✔️ Expected Behavior * WorkingSet memory of `dify-sandbox` remains stable (or oscillates within a small range) under light traffic. * No OOM-kill, no upstream time-outs in `dify-api`. ### ❌ Actual Behavior * WorkingSet grows **linearly** from ~0.5 Gi to ~3.8 Gi in ~60 days. * When usage > 95 % of the 4 Gi limit, `dify-api` returns: Restarting the sandbox pods resets memory to 8Gi; the growth pattern repeats. * Sandbox logs constantly print
yindo added the 🐞 bug label 2026-02-21 19:21:13 -05:00
yindo closed this issue 2026-02-21 19:21:13 -05:00
yindo commented 2026-02-21 19:21:14 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 13, 2025):

Hi @Jy1126! I'm Dosu and I’m helping the dify team.

I couldn't find any documented or previously fixed memory leaks in dify-sandbox v1.3.1 or later, and there are no official troubleshooting steps for this specific issue. The architecture of dify-sandbox is designed for secure, isolated code execution and doesn't expose memory management settings beyond standard environment variables like WORKER_TIMEOUT, but nothing directly related to memory leaks is documented in the docker-compose or docs [1] [2].

To help diagnose further, could you share a sample of the logs that are "constantly printing" during the memory growth? This might reveal if there are orphaned processes, unreleased resources, or abnormal subprocess behavior. You might also try increasing the logging verbosity or enabling debug mode in dify-sandbox to capture more details.

If you haven't already, check if there's a newer dify-sandbox Docker image available, as fixes may have been released after v1.3.1. Monitoring inside the container with tools like top, ps, or Go's pprof (if enabled) could help pinpoint where memory is accumulating.

Let me know if you can provide more log details or if you notice any patterns in the memory growth after certain types of requests or code executions.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Join Discord Share on X

@dosubot[bot] commented on GitHub (Jul 13, 2025): <!-- Greeting --> Hi @Jy1126! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the dify team. <!-- Answer --> I couldn't find any documented or previously fixed memory leaks in dify-sandbox v1.3.1 or later, and there are no official troubleshooting steps for this specific issue. The architecture of dify-sandbox is designed for secure, isolated code execution and doesn't expose memory management settings beyond standard environment variables like WORKER_TIMEOUT, but nothing directly related to memory leaks is documented in the docker-compose or docs [[1]](https://github.com/crazywoola/dify-docs-mintlify/blob/ef2d56c19222d395f51c076e3d934af9c143b841/en/development/backend/sandbox/README.mdx) [[2]](https://github.com/langgenius/dify/blob/e7388779a14f45ff1bf50a776086459f4f08b153/docker/docker-compose.yaml). To help diagnose further, could you share a sample of the logs that are "constantly printing" during the memory growth? This might reveal if there are orphaned processes, unreleased resources, or abnormal subprocess behavior. You might also try increasing the logging verbosity or enabling debug mode in dify-sandbox to capture more details. If you haven't already, check if there's a newer dify-sandbox Docker image available, as fixes may have been released after v1.3.1. Monitoring inside the container with tools like `top`, `ps`, or Go's `pprof` (if enabled) could help pinpoint where memory is accumulating. Let me know if you can provide more log details or if you notice any patterns in the memory growth after certain types of requests or code executions. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/1db9e9b1-8ff2-4620-be27-ab1903166cd0?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/1db9e9b1-8ff2-4620-be27-ab1903166cd0?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/1db9e9b1-8ff2-4620-be27-ab1903166cd0?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/1db9e9b1-8ff2-4620-be27-ab1903166cd0?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/1db9e9b1-8ff2-4620-be27-ab1903166cd0?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/1db9e9b1-8ff2-4620-be27-ab1903166cd0?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/1db9e9b1-8ff2-4620-be27-ab1903166cd0?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/22319)
yindo commented 2026-02-21 19:21:14 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Jul 14, 2025):

Sorry, we do not provide support for deploying using k8s. :)

@crazywoola commented on GitHub (Jul 14, 2025): Sorry, we do not provide support for deploying using k8s. :)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#15327
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?