langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Why does 1.1.3 run the CPU to 100%. Even run the machine to death? How to set the LLM/invoice timeout? #12502

New Issue
Closed
opened 2026-02-21 19:07:43 -05:00 by yindo · 11 comments
yindo commented 2026-02-21 19:07:43 -05:00
Owner
Copy Link
Copy Source

Originally created by @lichao4Java on GitHub (Apr 3, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.1.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Why does 1.1.3 run the CPU to 100%. Even run the machine to death?

I found this information in the plugins log, it seems that the plugin was killed and there were many POST requests that took more than 6 minutes/ plugin/951cb2d7-6e44-4007-9206-dc7bd0c85402/dispatch/llm/invoke
I suspect that the timeout period is too long and a large number of requests are piling up, causing the machine resources to be exhausted. May I ask what the problem is and how to configure the timeout period to avoid depletion of dify resources due to the llm/invoice interface not responding?

plugins version:
OpenAI
0.0.14

OpenRouter
0.0.6

Image Image

✔️ Expected Behavior

CPU to 100% and crashed

❌ Actual Behavior

configure the timeout period to avoid depletion of dify resources due to the llm/invoice interface not responding.

Originally created by @lichao4Java on GitHub (Apr 3, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.1.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Why does 1.1.3 run the CPU to 100%. Even run the machine to death? I found this information in the plugins log, it seems that the plugin was killed and there were many POST requests that took more than 6 minutes/ plugin/951cb2d7-6e44-4007-9206-dc7bd0c85402/dispatch/llm/invoke I suspect that the timeout period is too long and a large number of requests are piling up, causing the machine resources to be exhausted. May I ask what the problem is and how to configure the timeout period to avoid depletion of dify resources due to the llm/invoice interface not responding? plugins version: OpenAI 0.0.14 OpenRouter 0.0.6 <img width="649" alt="Image" src="https://github.com/user-attachments/assets/c1869643-970d-4360-92e8-e576d2613dc4" /> <img width="1253" alt="Image" src="https://github.com/user-attachments/assets/588cd213-092d-4eb8-81ca-b14ad2efc6e7" /> ### ✔️ Expected Behavior CPU to 100% and crashed ### ❌ Actual Behavior configure the timeout period to avoid depletion of dify resources due to the llm/invoice interface not responding.
yindo closed this issue 2026-02-21 19:07:43 -05:00
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 3, 2025):

To configure the timeout period in Dify version 1.1.3 and prevent CPU overuse, follow these steps:

  1. Modify the Source Code:

    • Locate the file api/core/model_runtime/model_providers/ollama/llm/llm.py.
    • Change line 60 from timeout=(10, 300) to a longer timeout, such as timeout=(10, 3600) for 60 minutes.

  2. Update Environment Variables:

    • Create a .env file by copying from .env.example: 
      cp .env.example .env
    • Set the following environment variables in the .env file with appropriate timeout values (in seconds):
      • APP_MAX_EXECUTION_TIME=3600
      • API_TOOL_DEFAULT_READ_TIMEOUT=3600
      • WORKFLOW_MAX_EXECUTION_TIME=3600
      • HTTP_REQUEST_MAX_CONNECT_TIMEOUT=3600
      • HTTP_REQUEST_MAX_READ_TIMEOUT=3600
      • HTTP_REQUEST_MAX_WRITE_TIMEOUT=3600
    • Set the GUNICORN_TIMEOUT environment variable to a higher value, such as 360, to support longer SSE connection times.

  3. Rebuild the API Service:

    • Use docker-compose to rebuild the API service from source to apply the changes.

Additionally, ensure that the HTTP_REQUEST_DEFAULT_TIMEOUT in api/core/workflow/nodes/http_request/node.py is set correctly using the environment variables:

  • connect=dify_config.HTTP_REQUEST_MAX_CONNECT_TIMEOUT
  • read=dify_config.HTTP_REQUEST_MAX_READ_TIMEOUT
  • write=dify_config.HTTP_REQUEST_MAX_WRITE_TIMEOUT

These adjustments should help manage the CPU usage by ensuring that the timeout settings are properly configured [1][2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 3, 2025): <!-- Answer --> To configure the timeout period in Dify version 1.1.3 and prevent CPU overuse, follow these steps: 1. **Modify the Source Code:** - Locate the file `api/core/model_runtime/model_providers/ollama/llm/llm.py`. - Change line 60 from `timeout=(10, 300)` to a longer timeout, such as `timeout=(10, 3600)` for 60 minutes. 2. **Update Environment Variables:** - Create a `.env` file by copying from `.env.example`: ```bash cp .env.example .env ``` - Set the following environment variables in the `.env` file with appropriate timeout values (in seconds): - `APP_MAX_EXECUTION_TIME=3600` - `API_TOOL_DEFAULT_READ_TIMEOUT=3600` - `WORKFLOW_MAX_EXECUTION_TIME=3600` - `HTTP_REQUEST_MAX_CONNECT_TIMEOUT=3600` - `HTTP_REQUEST_MAX_READ_TIMEOUT=3600` - `HTTP_REQUEST_MAX_WRITE_TIMEOUT=3600` - Set the `GUNICORN_TIMEOUT` environment variable to a higher value, such as 360, to support longer SSE connection times. 3. **Rebuild the API Service:** - Use `docker-compose` to rebuild the API service from source to apply the changes. Additionally, ensure that the `HTTP_REQUEST_DEFAULT_TIMEOUT` in `api/core/workflow/nodes/http_request/node.py` is set correctly using the environment variables: - `connect=dify_config.HTTP_REQUEST_MAX_CONNECT_TIMEOUT` - `read=dify_config.HTTP_REQUEST_MAX_READ_TIMEOUT` - `write=dify_config.HTTP_REQUEST_MAX_WRITE_TIMEOUT` These adjustments should help manage the CPU usage by ensuring that the timeout settings are properly configured <sup>[[1]](https://github.com/langgenius/dify/issues/12880)</sup><sup>[[2]](https://github.com/langgenius/dify/issues/12987)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/055fefb8-74aa-40d1-b6ed-c07a04288575?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/055fefb8-74aa-40d1-b6ed-c07a04288575?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/055fefb8-74aa-40d1-b6ed-c07a04288575?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/055fefb8-74aa-40d1-b6ed-c07a04288575?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/055fefb8-74aa-40d1-b6ed-c07a04288575?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/055fefb8-74aa-40d1-b6ed-c07a04288575?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/055fefb8-74aa-40d1-b6ed-c07a04288575?feedback_type=other)</sup>
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Apr 3, 2025):

I upgraded from a lower version to 1.1.3. If I update the. env file, do I just need to restart Docker?

@lichao4Java commented on GitHub (Apr 3, 2025): I upgraded from a lower version to 1.1.3. If I update the. env file, do I just need to restart Docker?
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Apr 3, 2025):

I upgraded from a lower version to 1.1.3. If I update the. env file, do I just need to restart Docker?

Yes

Actually the problem is not in the plugin_daemon itself, for some plugins, if it uses legacy sdk.

For example, it will have the memory leak issue.

https://github.com/langgenius/dify-official-plugins/blob/0abda652c5912da00d68544febf1334259eab127/models/openai/requirements.txt#L1

@crazywoola commented on GitHub (Apr 3, 2025): > I upgraded from a lower version to 1.1.3. If I update the. env file, do I just need to restart Docker? Yes Actually the problem is not in the plugin_daemon itself, for some plugins, if it uses legacy sdk. For example, it will have the memory leak issue. https://github.com/langgenius/dify-official-plugins/blob/0abda652c5912da00d68544febf1334259eab127/models/openai/requirements.txt#L1
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Apr 3, 2025):

I upgraded from a lower version to 1.1.3. If I update the. env file, do I just need to restart Docker?

Yes

Actually the problem is not in the plugin_daemon itself, for some plugins, if it uses legacy sdk.

For example, it will have the memory leak issue.

https://github.com/langgenius/dify-official-plugins/blob/0abda652c5912da00d68544febf1334259eab127/models/openai/requirements.txt#L1

I am using official DIY plugins, and they are all the latest versions. Is there any other known official plugin that has not been upgraded and has this issue?

@lichao4Java commented on GitHub (Apr 3, 2025): > > I upgraded from a lower version to 1.1.3. If I update the. env file, do I just need to restart Docker? > > Yes > > Actually the problem is not in the plugin_daemon itself, for some plugins, if it uses legacy sdk. > > For example, it will have the memory leak issue. > > https://github.com/langgenius/dify-official-plugins/blob/0abda652c5912da00d68544febf1334259eab127/models/openai/requirements.txt#L1 I am using official DIY plugins, and they are all the latest versions. Is there any other known official plugin that has not been upgraded and has this issue?
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@DavideDelbianco commented on GitHub (Apr 3, 2025):

@crazywoola

I upgraded from a lower version to 1.1.3. If I update the. env file, do I just need to restart Docker?

Yes

Actually the problem is not in the plugin_daemon itself, for some plugins, if it uses legacy sdk.

For example, it will have the memory leak issue.

https://github.com/langgenius/dify-official-plugins/blob/0abda652c5912da00d68544febf1334259eab127/models/openai/requirements.txt#L1

Are you telling us that OpenAI v 0.0.14 and also v0.0.15 have a known memory leak issue?

What SDK is causing the memory leak? Because the requirements seems not changed in the latest 2 versions after the one you linked

@DavideDelbianco commented on GitHub (Apr 3, 2025): @crazywoola > > I upgraded from a lower version to 1.1.3. If I update the. env file, do I just need to restart Docker? > > Yes > > Actually the problem is not in the plugin_daemon itself, for some plugins, if it uses legacy sdk. > > For example, it will have the memory leak issue. > > https://github.com/langgenius/dify-official-plugins/blob/0abda652c5912da00d68544febf1334259eab127/models/openai/requirements.txt#L1 Are you telling us that OpenAI v 0.0.14 and also v0.0.15 have a known memory leak issue? What SDK is causing the memory leak? Because the requirements seems not changed in the latest 2 versions after the one you linked
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Apr 3, 2025):

The situation where CPU 100% dify cannot be opened has occurred again. The log is as follows:

TOP:

Image

docker-api-1 logs:

Image

docker-plugin_daemon-1 logs:

Is this the reason for some time-consuming/dispatch/lm/invike calls? Can the connection fail to release the heap and cause system downtime?

Image

@crazywoola

@lichao4Java commented on GitHub (Apr 3, 2025): The situation where CPU 100% dify cannot be opened has occurred again. The log is as follows: TOP: <img width="625" alt="Image" src="https://github.com/user-attachments/assets/52e313e4-d262-438b-bffc-5d41bd59d86c" /> docker-api-1 logs: <img width="1501" alt="Image" src="https://github.com/user-attachments/assets/67275860-f029-4cdb-bde9-db11a040c355" /> docker-plugin_daemon-1 logs: **Is this the reason for some time-consuming/dispatch/lm/invike calls? Can the connection fail to release the heap and cause system downtime?** ![Image](https://github.com/user-attachments/assets/a7e3e005-ff99-4c50-adef-889896c554da) @crazywoola
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Apr 3, 2025):

root 322700 17.6 24.4 2677920 1930952 ? Rl 05:21 57:58 /app/api/.venv/bin/python /app/api/.venv/bin/gunicorn --bind 0.0.0.0:5001 --workers 1 --worker-class gevent --worker-connections 10 --timeout 360 app:app

how can fix this promble?

What timeout parameters are adjusted for dify?

@lichao4Java commented on GitHub (Apr 3, 2025): root 322700 17.6 24.4 2677920 1930952 ? Rl 05:21 57:58 /app/api/.venv/bin/python /app/api/.venv/bin/gunicorn --bind 0.0.0.0:5001 --workers 1 --worker-class gevent --worker-connections 10 --timeout 360 app:app how can fix this promble? What timeout parameters are adjusted for dify?
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Apr 3, 2025):

when i upload a image , chat to gpt-4o and the error is :

/app/api/.venv/lib/python3.12/site-packages/pydantic/main.py:390: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue: Expected str but got list with value [TextPromptMessageContent...il=<DETAIL.LOW: 'low'>)] - serialized value may not be as expected
PydanticSerializationUnexpectedValue: Expected PromptMessageContent but got TextPromptMessageContent with value TextPromptMessageContent(... data='这是什么？') - serialized value may not be as expected
return self.pydantic_serializer.to_python(
/app/api/.venv/lib/python3.12/site-packages/pydantic/main.py:390: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue: Expected str but got list with value [TextPromptMessageContent...=<DETAIL.HIGH: 'high'>)] - serialized value may not be as expected
PydanticSerializationUnexpectedValue: Expected PromptMessageContent but got TextPromptMessageContent with value TextPromptMessageContent(... data='这是什么？') - serialized value may not be as expected
return self.pydantic_serializer.to_python(

Is this the problem causing the memory leak? Do we need to roll back the OpenAI plugin to 0.0.13? This version cannot use multimodality correctly

@crazywoola

@lichao4Java commented on GitHub (Apr 3, 2025): when i upload a image , chat to gpt-4o and the error is : /app/api/.venv/lib/python3.12/site-packages/pydantic/main.py:390: UserWarning: Pydantic serializer warnings: PydanticSerializationUnexpectedValue: Expected `str` but got `list` with value `[TextPromptMessageContent...il=<DETAIL.LOW: 'low'>)]` - serialized value may not be as expected PydanticSerializationUnexpectedValue: Expected `PromptMessageContent` but got `TextPromptMessageContent` with value `TextPromptMessageContent(... data='这是什么？')` - serialized value may not be as expected return self.__pydantic_serializer__.to_python( /app/api/.venv/lib/python3.12/site-packages/pydantic/main.py:390: UserWarning: Pydantic serializer warnings: PydanticSerializationUnexpectedValue: Expected `str` but got `list` with value `[TextPromptMessageContent...=<DETAIL.HIGH: 'high'>)]` - serialized value may not be as expected PydanticSerializationUnexpectedValue: Expected `PromptMessageContent` but got `TextPromptMessageContent` with value `TextPromptMessageContent(... data='这是什么？')` - serialized value may not be as expected return self.__pydantic_serializer__.to_python( Is this the problem causing the memory leak? Do we need to roll back the OpenAI plugin to 0.0.13? This version cannot use multimodality correctly @crazywoola
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Apr 3, 2025):

Image

root 906076 6.2 1.7 362804 141204 ? Rl 11:20 11:37 /app/storage/cwd/langgenius/openrouter-0.0.6@4dd8884301ac440b89ae04ade29bf27ad8adc2ee9c238d6e0a0c1201c5915e67/.venv/bin/python -m main

openrouter-0.0.6 CPU100%

@lichao4Java commented on GitHub (Apr 3, 2025): ![Image](https://github.com/user-attachments/assets/461e583f-2f3d-4329-825a-f99fa50f6dba) root 906076 6.2 1.7 362804 141204 ? Rl 11:20 11:37 /app/storage/cwd/langgenius/openrouter-0.0.6@4dd8884301ac440b89ae04ade29bf27ad8adc2ee9c238d6e0a0c1201c5915e67/.venv/bin/python -m main openrouter-0.0.6 CPU100%
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@hejuntt1014 commented on GitHub (Apr 5, 2025):

I’ve encountered the same issue. Under high concurrency or after the request has completed, as long as the CPU usage reaches 100%, it never goes down—it just freezes completely. I’ve tested both the Tongyi(通义千问) and OpenAI-API-compatible plugins, and they both have this problem.

@hejuntt1014 commented on GitHub (Apr 5, 2025): I’ve encountered the same issue. Under high concurrency or after the request has completed, as long as the CPU usage reaches 100%, it never goes down—it just freezes completely. I’ve tested both the Tongyi(通义千问) and OpenAI-API-compatible plugins, and they both have this problem.
yindo commented 2026-02-21 19:07:43 -05:00
Author
Owner
Copy Link
Copy Source

@weimengduo commented on GitHub (Jun 10, 2025):

I am having the same issue! Still have no idea, but I am using BT panel to install dify

@weimengduo commented on GitHub (Jun 10, 2025): I am having the same issue! Still have no idea, but I am using BT panel to install dify
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#12502
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?