langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Cann't connect to Sandbox randomly at 10 concurrent processes #7950

New Issue
Closed
opened 2026-02-21 18:23:11 -05:00 by yindo · 15 comments
yindo commented 2026-02-21 18:23:11 -05:00
Owner
Copy Link
Copy Source

Originally created by @yuhua6379 on GitHub (Feb 2, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Image

Desc

I used concurrent mode for a workflow. It seems like there is something wrong with the sandbox. But I checkout all the logs of all containers(docker).

MORE details:

I am invoking 10 workflows in parallel, and each workflow has a concurrency level of 10 in its concurrent mode. This means the total concurrency is 10*10=100 concurrent executions

The error rate/failure rate is approximately 2/330 (roughly 0.6% or 0.006)

✔️ Expected Behavior

work successfully

❌ Actual Behavior

it fails randomly

Originally created by @yuhua6379 on GitHub (Feb 2, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.15.1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce ![Image](https://github.com/user-attachments/assets/da1ef40d-9a5e-4098-9f0c-b56133ce66fe) # Desc I used concurrent mode for a workflow. It seems like there is something wrong with the sandbox. But I checkout all the logs of all containers(docker). # MORE details: I am invoking 10 workflows in parallel, and each workflow has a concurrency level of 10 in its concurrent mode. This means the total concurrency is 10*10=100 concurrent executions The error rate/failure rate is approximately 2/330 (roughly 0.6% or 0.006) ### ✔️ Expected Behavior work successfully ### ❌ Actual Behavior it fails randomly
yindo added the 🐞 bugstale labels 2026-02-21 18:23:11 -05:00
yindo closed this issue 2026-02-21 18:23:11 -05:00
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 2, 2025):

I found a similar discussion titled "Sandbox service start" that might be relevant to your issue. It was last updated on September 18, 2024, and is currently unsolved. You can view the discussion here [1]. This might provide some insights or shared experiences that could help address the random connection failures you're encountering with the sandbox service.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Feb 2, 2025): <!-- Answer --> I found a similar discussion titled "Sandbox service start" that might be relevant to your issue. It was last updated on September 18, 2024, and is currently unsolved. You can view the discussion [here](https://github.com/langgenius/dify/discussions/8520) <sup>[[1]](https://github.com/langgenius/dify/discussions/8520)</sup>. This might provide some insights or shared experiences that could help address the random connection failures you're encountering with the sandbox service. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/3cde69f5-d8bd-4dc6-84f9-9c6bd721881d?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/3cde69f5-d8bd-4dc6-84f9-9c6bd721881d?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/3cde69f5-d8bd-4dc6-84f9-9c6bd721881d?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/3cde69f5-d8bd-4dc6-84f9-9c6bd721881d?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/3cde69f5-d8bd-4dc6-84f9-9c6bd721881d?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/3cde69f5-d8bd-4dc6-84f9-9c6bd721881d?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@yuhua6379 commented on GitHub (Feb 2, 2025):

@dosu Not this problem. It might be the limitaion of sanbox task instance.

@yuhua6379 commented on GitHub (Feb 2, 2025): @dosu Not this problem. It might be the limitaion of sanbox task instance.
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 2, 2025):

The random connection failures could indeed be due to limitations of the sandbox task instance. The sandbox environment in Dify has several constraints, such as maximum string length, array lengths, and worker timeout settings, which could impact performance under high concurrency. Specifically, the sandbox worker timeout is set to 15 seconds, and network access is enabled by default, which might contribute to connection issues if the system is under heavy load [1][2][3].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Feb 2, 2025): <!-- Answer --> The random connection failures could indeed be due to limitations of the sandbox task instance. The sandbox environment in Dify has several constraints, such as maximum string length, array lengths, and worker timeout settings, which could impact performance under high concurrency. Specifically, the sandbox worker timeout is set to 15 seconds, and network access is enabled by default, which might contribute to connection issues if the system is under heavy load <sup>[[1]](https://github.com/langgenius/dify/blob/main/docker/docker-compose.yaml)</sup><sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/workflow/node/code.md)</sup><sup>[[3]](https://github.com/langgenius/dify-docs/blob/main/zh_CN/development/backend/sandbox/README.md)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/69bd5fb0-58a8-484a-bd16-dfa4031ab2c8?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/69bd5fb0-58a8-484a-bd16-dfa4031ab2c8?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/69bd5fb0-58a8-484a-bd16-dfa4031ab2c8?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/69bd5fb0-58a8-484a-bd16-dfa4031ab2c8?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/69bd5fb0-58a8-484a-bd16-dfa4031ab2c8?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/69bd5fb0-58a8-484a-bd16-dfa4031ab2c8?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@yuhua6379 commented on GitHub (Feb 2, 2025):

@dosu Are there any limitaions about number of workers of sandbox?

@yuhua6379 commented on GitHub (Feb 2, 2025): @dosu Are there any limitaions about number of workers of sandbox?
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 2, 2025):

Yes, there is a limitation on the number of workers in the sandbox. The max_workers parameter is set to 4 [1].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Feb 2, 2025): <!-- Answer --> Yes, there is a limitation on the number of workers in the sandbox. The `max_workers` parameter is set to 4 <sup>[[1]](https://github.com/langgenius/dify/blob/main/docker/volumes/sandbox/conf/config.yaml)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/432caa36-9dfb-4eaa-bf14-dfdeb8c7088f?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/432caa36-9dfb-4eaa-bf14-dfdeb8c7088f?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/432caa36-9dfb-4eaa-bf14-dfdeb8c7088f?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/432caa36-9dfb-4eaa-bf14-dfdeb8c7088f?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/432caa36-9dfb-4eaa-bf14-dfdeb8c7088f?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/432caa36-9dfb-4eaa-bf14-dfdeb8c7088f?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@yuhua6379 commented on GitHub (Feb 2, 2025):

@dosu Can I access to this parameter thorught docker-compose.yaml or environment vars?

@yuhua6379 commented on GitHub (Feb 2, 2025): @dosu Can I access to this parameter thorught docker-compose.yaml or environment vars?
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 2, 2025):

You can adjust the max_workers parameter by modifying the config.yaml file directly within the sandbox configuration. This parameter is not exposed through docker-compose.yaml or environment variables by default. To change it, you would need to access the configuration file in the Docker volume where the sandbox configuration is stored and update the max_workers value accordingly. After making changes, restart the sandbox service to apply the new settings.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Feb 2, 2025): <!-- Answer --> You can adjust the `max_workers` parameter by modifying the `config.yaml` file directly within the sandbox configuration. This parameter is not exposed through `docker-compose.yaml` or environment variables by default. To change it, you would need to access the configuration file in the Docker volume where the sandbox configuration is stored and update the `max_workers` value accordingly. After making changes, restart the sandbox service to apply the new settings. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/c66e6c82-adea-40da-93a7-2085f95980d0?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/c66e6c82-adea-40da-93a7-2085f95980d0?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/c66e6c82-adea-40da-93a7-2085f95980d0?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/c66e6c82-adea-40da-93a7-2085f95980d0?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/c66e6c82-adea-40da-93a7-2085f95980d0?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/c66e6c82-adea-40da-93a7-2085f95980d0?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@yuhua6379 commented on GitHub (Feb 2, 2025):

@dosu What's the meaning of max_requests of this config.yaml? Same as a parameter of sandbox.

@yuhua6379 commented on GitHub (Feb 2, 2025): @dosu What's the meaning of max_requests of this config.yaml? Same as a parameter of sandbox.
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 2, 2025):

The max_requests parameter in the config.yaml file specifies the maximum number of requests a worker can handle before it is restarted. This helps manage resource usage and ensures that any potential memory leaks or resource exhaustion issues are mitigated by periodically refreshing the worker processes. Adjusting this value can help balance performance and resource management based on your specific workload requirements.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Feb 2, 2025): <!-- Answer --> The `max_requests` parameter in the `config.yaml` file specifies the maximum number of requests a worker can handle before it is restarted. This helps manage resource usage and ensures that any potential memory leaks or resource exhaustion issues are mitigated by periodically refreshing the worker processes. Adjusting this value can help balance performance and resource management based on your specific workload requirements. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/4b963dfb-1bf8-4810-8cdd-ffe02f19c52d?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/4b963dfb-1bf8-4810-8cdd-ffe02f19c52d?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/4b963dfb-1bf8-4810-8cdd-ffe02f19c52d?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/4b963dfb-1bf8-4810-8cdd-ffe02f19c52d?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/4b963dfb-1bf8-4810-8cdd-ffe02f19c52d?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/4b963dfb-1bf8-4810-8cdd-ffe02f19c52d?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@yuhua6379 commented on GitHub (Feb 2, 2025):

@dosu Thank you. I would try. Maybe this limitaion is the key issue.

@yuhua6379 commented on GitHub (Feb 2, 2025): @dosu Thank you. I would try. Maybe this limitaion is the key issue.
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 2, 2025):

You're welcome! If you have any more questions or need further assistance while adjusting the settings, feel free to reach out. Good luck with your testing, and I hope this resolves the issue!

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Feb 2, 2025): <!-- Answer --> You're welcome! If you have any more questions or need further assistance while adjusting the settings, feel free to reach out. Good luck with your testing, and I hope this resolves the issue! <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/e5034d9f-9b3b-42d3-a40e-0293861b679f?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/e5034d9f-9b3b-42d3-a40e-0293861b679f?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/e5034d9f-9b3b-42d3-a40e-0293861b679f?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/e5034d9f-9b3b-42d3-a40e-0293861b679f?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/e5034d9f-9b3b-42d3-a40e-0293861b679f?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/e5034d9f-9b3b-42d3-a40e-0293861b679f?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@yuhua6379 commented on GitHub (Feb 2, 2025):

Still have this same problem after I changed max_workers to 32.

Image

@yuhua6379 commented on GitHub (Feb 2, 2025): Still have this same problem after I changed max_workers to 32. ![Image](https://github.com/user-attachments/assets/d361b328-e2cc-44c2-b9d7-678f5d64222e)
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@yuhua6379 commented on GitHub (Feb 3, 2025):

@laipz8200 Please help. Is this a bug or sth?

@yuhua6379 commented on GitHub (Feb 3, 2025): @laipz8200 Please help. Is this a bug or sth?
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Feb 7, 2025):

Please check if #13108 resolves this issue.

@laipz8200 commented on GitHub (Feb 7, 2025): Please check if #13108 resolves this issue.
yindo commented 2026-02-21 18:23:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 10, 2025):

Hi, @yuhua6379. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

  • Random connection failures to the Sandbox occur during high concurrency in Dify version 0.15.1.
  • The error rate is low at 0.6% and happens in a Docker environment.
  • Increasing max_workers from 4 to 32 did not resolve the issue.
  • @laipz8200 suggested checking if issue #13108 might resolve the problem.

Next Steps

  • Is this issue still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open.
  • If there are no further updates, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Mar 10, 2025): Hi, @yuhua6379. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog. I'm marking this issue as stale. **Issue Summary** - Random connection failures to the Sandbox occur during high concurrency in Dify version 0.15.1. - The error rate is low at 0.6% and happens in a Docker environment. - Increasing `max_workers` from 4 to 32 did not resolve the issue. - @laipz8200 suggested checking if issue #13108 might resolve the problem. **Next Steps** - Is this issue still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open. - If there are no further updates, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label stale 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7950
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?