langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 20:45:26 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Max submit count 100 of workflow thread pool reached. #5742

New Issue
Closed
opened 2026-02-21 18:12:23 -05:00 by yindo · 6 comments
yindo commented 2026-02-21 18:12:23 -05:00
Owner
Copy Link
Copy Source

Originally created by @hwb96 on GitHub (Sep 23, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.8.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

After updating to version 0.8.3, the previous workflow no longer works properly and displays: "Run failed: Failed to invoke tool: Max submit count 100 of workflow thread pool reached."
image

This used to work fine before.

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @hwb96 on GitHub (Sep 23, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.8.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce After updating to version 0.8.3, the previous workflow no longer works properly and displays: "Run failed: Failed to invoke tool: Max submit count 100 of workflow thread pool reached." ![image](https://github.com/user-attachments/assets/6b80e28d-73ca-4a73-a0d4-8ad27cc58191) This used to work fine before. ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🌊 feat:workflow label 2026-02-21 18:12:23 -05:00
yindo closed this issue 2026-02-21 18:12:23 -05:00
yindo commented 2026-02-21 18:12:24 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 23, 2024):

To resolve the "Max submit count 100 of workflow thread pool reached" error after updating to Dify version 0.8.3 in a self-hosted Docker environment, you need to increase the max_submit_count parameter in the GraphEngineThreadPool initialization.

Here's how you can modify the GraphEngine class to increase the max_submit_count:

  1. Locate the GraphEngine class in the api/core/workflow/graph_engine/graph_engine.py file.
  2. Update the thread_pool_max_submit_count to a higher value.

For example, if you want to increase the max_submit_count to 200, you can modify the code as follows:

class GraphEngine:
    workflow_thread_pool_mapping: dict[str, GraphEngineThreadPool] = {}

    def __init__(
        self,
        tenant_id: str,
        app_id: str,
        workflow_type: WorkflowType,
        workflow_id: str,
        user_id: str,
        user_from: UserFrom,
        invoke_from: InvokeFrom,
        call_depth: int,
        graph: Graph,
        graph_config: Mapping[str, Any],
        variable_pool: VariablePool,
        max_execution_steps: int,
        max_execution_time: int,
        thread_pool_id: Optional[str] = None,
    ) -> None:
        thread_pool_max_submit_count = 200  # Increase this value
        thread_pool_max_workers = 10

        # init thread pool
        if thread_pool_id:
            if thread_pool_id not in GraphEngine.workflow_thread_pool_mapping:
                raise ValueError(f"Max submit count {thread_pool_max_submit_count} of workflow thread pool reached.")

            self.thread_pool_id = thread_pool_id
            self.thread_pool = GraphEngine.workflow_thread_pool_mapping[thread_pool_id]
            self.is_main_thread_pool = False
        else:
            self.thread_pool = GraphEngineThreadPool(
                max_workers=thread_pool_max_workers, max_submit_count=thread_pool_max_submit_count
            )
            self.thread_pool_id = str(uuid.uuid4())
            self.is_main_thread_pool = True
            GraphEngine.workflow_thread_pool_mapping[self.thread_pool_id] = self.thread_pool

        self.graph = graph
        self.init_params = GraphInitParams(
            tenant_id=tenant_id,
            app_id=app_id,
            workflow_type=workflow_type,
            workflow_id=workflow_id,
            graph_config=graph_config,
            user_id=user_id,
            user_from=user_from,
            invoke_from=invoke_from,
            call_depth=call_depth,
        )

        self.graph_runtime_state = GraphRuntimeState(variable_pool=variable_pool, start_at=time.perf_counter())
        self.max_execution_steps = max_execution_steps
        self.max_execution_time = max_execution_time

By increasing the thread_pool_max_submit_count, you allow more tasks to be submitted to the thread pool before reaching the limit [1].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 23, 2024): <!-- Answer --> To resolve the "Max submit count 100 of workflow thread pool reached" error after updating to Dify version 0.8.3 in a self-hosted Docker environment, you need to increase the `max_submit_count` parameter in the `GraphEngineThreadPool` initialization. Here's how you can modify the `GraphEngine` class to increase the `max_submit_count`: 1. Locate the `GraphEngine` class in the `api/core/workflow/graph_engine/graph_engine.py` file. 2. Update the `thread_pool_max_submit_count` to a higher value. For example, if you want to increase the `max_submit_count` to 200, you can modify the code as follows: ```python class GraphEngine: workflow_thread_pool_mapping: dict[str, GraphEngineThreadPool] = {} def __init__( self, tenant_id: str, app_id: str, workflow_type: WorkflowType, workflow_id: str, user_id: str, user_from: UserFrom, invoke_from: InvokeFrom, call_depth: int, graph: Graph, graph_config: Mapping[str, Any], variable_pool: VariablePool, max_execution_steps: int, max_execution_time: int, thread_pool_id: Optional[str] = None, ) -> None: thread_pool_max_submit_count = 200 # Increase this value thread_pool_max_workers = 10 # init thread pool if thread_pool_id: if thread_pool_id not in GraphEngine.workflow_thread_pool_mapping: raise ValueError(f"Max submit count {thread_pool_max_submit_count} of workflow thread pool reached.") self.thread_pool_id = thread_pool_id self.thread_pool = GraphEngine.workflow_thread_pool_mapping[thread_pool_id] self.is_main_thread_pool = False else: self.thread_pool = GraphEngineThreadPool( max_workers=thread_pool_max_workers, max_submit_count=thread_pool_max_submit_count ) self.thread_pool_id = str(uuid.uuid4()) self.is_main_thread_pool = True GraphEngine.workflow_thread_pool_mapping[self.thread_pool_id] = self.thread_pool self.graph = graph self.init_params = GraphInitParams( tenant_id=tenant_id, app_id=app_id, workflow_type=workflow_type, workflow_id=workflow_id, graph_config=graph_config, user_id=user_id, user_from=user_from, invoke_from=invoke_from, call_depth=call_depth, ) self.graph_runtime_state = GraphRuntimeState(variable_pool=variable_pool, start_at=time.perf_counter()) self.max_execution_steps = max_execution_steps self.max_execution_time = max_execution_time ``` By increasing the `thread_pool_max_submit_count`, you allow more tasks to be submitted to the thread pool before reaching the limit <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/workflow/graph_engine/graph_engine.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:12:24 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Sep 23, 2024):

Because of the introduction of parallelism, we have put some restrictions on the number of nodes that can be processed

@crazywoola commented on GitHub (Sep 23, 2024): Because of the introduction of parallelism, we have put some restrictions on the number of nodes that can be processed
yindo commented 2026-02-21 18:12:24 -05:00
Author
Owner
Copy Link
Copy Source

@fishisnow commented on GitHub (Oct 16, 2024):

I encountered the same bug. In version 0.8.3, the iteration node does not work.

@fishisnow commented on GitHub (Oct 16, 2024): I encountered the same bug. In version 0.8.3, the iteration node does not work.
yindo commented 2026-02-21 18:12:24 -05:00
Author
Owner
Copy Link
Copy Source

@johnpccd commented on GitHub (Nov 6, 2024):

Because of the introduction of parallelism, we have put some restrictions on the number of nodes that can be processed

Hi @crazywoola, can we make this configurable through an env var?

I am having this issue on 0.11.0.. using nested parallel iterations (the nesting still doesn't add up to a 100 yet am getting the error)

@johnpccd commented on GitHub (Nov 6, 2024): > Because of the introduction of parallelism, we have put some restrictions on the number of nodes that can be processed Hi @crazywoola, can we make this configurable through an env var? I am having this issue on 0.11.0.. using nested parallel iterations (the nesting still doesn't add up to a 100 yet am getting the error)
yindo commented 2026-02-21 18:12:24 -05:00
Author
Owner
Copy Link
Copy Source

@detongz commented on GitHub (Dec 11, 2024):

same error here:

api-1         | 2024-12-11 23:49:45,878.878 INFO [ThreadPoolExecutor-7_0] [_client.py:1038] - HTTP Request: POST http://sandbox:8194/v1/sandbox/run "HTTP/1.1 200 OK"
api-1         | 2024-12-11 23:49:45,881.881 ERROR [ThreadPoolExecutor-7_0] [graph_engine.py:186] - Unknown Error when graph running
api-1         | Traceback (most recent call last):
api-1         |   File "/app/api/core/workflow/graph_engine/graph_engine.py", line 145, in run
api-1         |     for item in generator:
api-1         |                 ^^^^^^^^^
api-1         |   File "/app/api/core/workflow/nodes/end/end_stream_processor.py", line 29, in process
api-1         |     for event in generator:
api-1         |                  ^^^^^^^^^
api-1         |   File "/app/api/core/workflow/graph_engine/graph_engine.py", line 379, in _run
api-1         |     for item in parallel_generator:
api-1         |                 ^^^^^^^^^^^^^^^^^^
api-1         |   File "/app/api/core/workflow/graph_engine/graph_engine.py", line 432, in _run_parallel_branches
api-1         |     future = self.thread_pool.submit(
api-1         |              ^^^^^^^^^^^^^^^^^^^^^^^^
api-1         |   File "/app/api/core/workflow/graph_engine/graph_engine.py", line 59, in submit
api-1         |     self.check_is_full()
api-1         |   File "/app/api/core/workflow/graph_engine/graph_engine.py", line 68, in check_is_full
api-1         |     raise ValueError(f"Max submit count {self.max_submit_count} of workflow thread pool reached.")
api-1         | ValueError: Max submit count 100 of workflow thread pool reached.
@detongz commented on GitHub (Dec 11, 2024): same error here: ``` api-1 | 2024-12-11 23:49:45,878.878 INFO [ThreadPoolExecutor-7_0] [_client.py:1038] - HTTP Request: POST http://sandbox:8194/v1/sandbox/run "HTTP/1.1 200 OK" api-1 | 2024-12-11 23:49:45,881.881 ERROR [ThreadPoolExecutor-7_0] [graph_engine.py:186] - Unknown Error when graph running api-1 | Traceback (most recent call last): api-1 | File "/app/api/core/workflow/graph_engine/graph_engine.py", line 145, in run api-1 | for item in generator: api-1 | ^^^^^^^^^ api-1 | File "/app/api/core/workflow/nodes/end/end_stream_processor.py", line 29, in process api-1 | for event in generator: api-1 | ^^^^^^^^^ api-1 | File "/app/api/core/workflow/graph_engine/graph_engine.py", line 379, in _run api-1 | for item in parallel_generator: api-1 | ^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/core/workflow/graph_engine/graph_engine.py", line 432, in _run_parallel_branches api-1 | future = self.thread_pool.submit( api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/core/workflow/graph_engine/graph_engine.py", line 59, in submit api-1 | self.check_is_full() api-1 | File "/app/api/core/workflow/graph_engine/graph_engine.py", line 68, in check_is_full api-1 | raise ValueError(f"Max submit count {self.max_submit_count} of workflow thread pool reached.") api-1 | ValueError: Max submit count 100 of workflow thread pool reached. ```
yindo commented 2026-02-21 18:12:24 -05:00
Author
Owner
Copy Link
Copy Source

@billwj2023-prog commented on GitHub (Aug 26, 2025):

由于引入了并行性，我们对可处理的节点数量进行了一些限制

你好@crazywoola，我们可以通过环境变量来实现这一点吗？

我在 0.11.0 上遇到了这个问题..使用嵌套并行迭代（嵌套仍然没有达到 100 但出现了错误）

Can the same issue be relaxed setting the envrionment variable(.env)file?

@billwj2023-prog commented on GitHub (Aug 26, 2025): > > 由于引入了并行性，我们对可处理的节点数量进行了一些限制 > > 你好[@crazywoola](https://github.com/crazywoola)，我们可以通过环境变量来实现这一点吗？ > > 我在 0.11.0 上遇到了这个问题..使用嵌套并行迭代（嵌套仍然没有达到 100 但出现了错误） Can the same issue be relaxed setting the envrionment variable(.env)file?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🌊 feat:workflow
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#5742
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?