[ERROR]PluginDaemonInternalServerError: killed by timeout #9486

Closed
opened 2026-02-21 18:35:17 -05:00 by yindo · 0 comments
Owner

Originally created by @liudonghua123 on GitHub (Mar 6, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.0.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I encountered some service failures sometimes, and I checked the log and found the following errors.

Logs
nginx-1          | 192.168.144.1 - - [06/Mar/2025:06:32:17 +0000] "GET /console/api/apps/28669fd5-16f0-45b3-831d-8a7231d0a3ca/workflows/draft HTTP/1.1" 200 3291 "https://dify.app.ynu.edu.cn/app/28669fd5-16f0-45b3-831d-8a7231d0a3ca/workflow" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36 Edg/133.0.0.0" "113.55.127.140"
plugin_daemon-1  | [GIN] 2025/03/06 - 06:32:18 | 200 |  439.616271ms |  192.168.144.11 | POST     "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/text_embedding/invoke"
plugin_daemon-1  | [GIN] 2025/03/06 - 06:32:18 | 200 |  704.057177ms |  192.168.144.11 | POST     "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/text_embedding/invoke"
api-1            | 2025-03-06 06:32:18.985 INFO [Thread-1107 (_generate_worker)] [_client.py:1038] - HTTP Request: POST http://sandbox:8194/v1/sandbox/run "HTTP/1.1 200 OK"
sandbox-1        | [GIN] 2025/03/06 - 06:32:18 | 200 |  182.220057ms |    192.168.96.4 | POST     "/v1/sandbox/run"
plugin_daemon-1  | [GIN] 2025/03/06 - 06:32:19 | 200 |   22.474066ms |  192.168.144.11 | GET      "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/management/models?page=1&page_size=256"
plugin_daemon-1  | [GIN] 2025/03/06 - 06:32:19 | 200 |    3.918187ms |  192.168.144.11 | POST     "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema"
plugin_daemon-1  | [GIN] 2025/03/06 - 06:32:19 | 200 |    4.059227ms |  192.168.144.11 | POST     "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema"
plugin_daemon-1  | [GIN] 2025/03/06 - 06:32:19 | 200 |    4.244066ms |  192.168.144.11 | POST     "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema"
plugin_daemon-1  | [GIN] 2025/03/06 - 06:32:19 | 200 |    6.569322ms |  192.168.144.11 | POST     "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema"
plugin_daemon-1  | [GIN] 2025/03/06 - 06:32:19 | 200 |    3.514292ms |  192.168.144.11 | POST     "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema"
plugin_daemon-1  | [GIN] 2025/03/06 - 06:32:19 | 200 |    2.260335ms |  192.168.144.11 | POST     "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/llm/num_tokens"
plugin_daemon-1  | 2025/03/06 06:32:22 factory.go:28: [ERROR]PluginDaemonInternalServerError: killed by timeout
plugin_daemon-1  | goroutine 48360 [running]:
plugin_daemon-1  | runtime/debug.Stack()
plugin_daemon-1  | 	/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/types/exception.InternalServerError({0x180b640, 0xc0006c48c0})
plugin_daemon-1  | 	/app/internal/types/exception/factory.go:27 +0x27
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/service.baseSSEService[...](0xc000862ff0, 0x15f6c0d, 0xf0?)
plugin_daemon-1  | 	/app/internal/service/base_sse.go:81 +0x4e6
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/service.InvokeLLM(0xc0001ec780, 0xc000028600, 0xf0)
plugin_daemon-1  | 	/app/internal/service/invoke_model.go:35 +0x198
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.InvokeLLM.func8.1({{{0xc0005b27dd, 0x24}, {0xc0017044d0, 0x7}}, {{0x0, 0x0}}, {0xc000f06240, 0x58}, 0x0, 0x0, ...})
plugin_daemon-1  | 	/app/internal/server/controllers/model.go:20 +0x89
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server/controllers.BindPluginDispatchRequest[...].func1()
plugin_daemon-1  | 	/app/internal/server/controllers/base.go:53 +0x9c
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server/controllers.BindRequest[...](0xc000028600, 0xc000863328)
plugin_daemon-1  | 	/app/internal/server/controllers/base.go:31 +0x275
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server/controllers.BindPluginDispatchRequest[...](0x1603a20?, 0xa?)
plugin_daemon-1  | 	/app/internal/server/controllers/base.go:37 +0x45
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.InvokeLLM.func8(0xc000028600)
plugin_daemon-1  | 	/app/internal/server/controllers/model.go:17 +0x74
plugin_daemon-1  | github.com/gin-gonic/gin.(*Context).Next(0xc000028600)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).InitClusterID.func3(0xc000028600)
plugin_daemon-1  | 	/app/internal/server/middleware.go:170 +0x54
plugin_daemon-1  | github.com/gin-gonic/gin.(*Context).Next(0xc000028600)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).RedirectPluginInvoke.func2(0xc000028600)
plugin_daemon-1  | 	/app/internal/server/middleware.go:97 +0x74
plugin_daemon-1  | github.com/gin-gonic/gin.(*Context).Next(0xc000028600)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).FetchPluginInstallation.func1(0xc000028600)
plugin_daemon-1  | 	/app/internal/server/middleware.go:66 +0x2f2
plugin_daemon-1  | github.com/gin-gonic/gin.(*Context).Next(0xc000028600)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginGroup.CheckingKey.func1(0xc000028600)
plugin_daemon-1  | 	/app/internal/server/middleware.go:24 +0x95
plugin_daemon-1  | github.com/gin-gonic/gin.(*Context).Next(...)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
plugin_daemon-1  | github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000028600)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/recovery.go:102 +0x7a
plugin_daemon-1  | github.com/gin-gonic/gin.(*Context).Next(...)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
plugin_daemon-1  | github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000028600)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/logger.go:249 +0xe5
plugin_daemon-1  | github.com/gin-gonic/gin.(*Context).Next(...)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185
plugin_daemon-1  | github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc00040a9c0, 0xc000028600)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:633 +0x892
plugin_daemon-1  | github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc00040a9c0, {0x181dac0, 0xc000216620}, 0xc00077db00)
plugin_daemon-1  | 	/go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:589 +0x1b2
plugin_daemon-1  | net/http.serverHandler.ServeHTTP({0x1813868?}, {0x181dac0?, 0xc000216620?}, 0x6?)
plugin_daemon-1  | 	/usr/local/go/src/net/http/server.go:3142 +0x8e
plugin_daemon-1  | net/http.(*conn).serve(0xc001527b00, {0x1820270, 0xc0005b9ce0})
plugin_daemon-1  | 	/usr/local/go/src/net/http/server.go:2044 +0x5e8
plugin_daemon-1  | created by net/http.(*Server).Serve in goroutine 89
plugin_daemon-1  | 	/usr/local/go/src/net/http/server.go:3290 +0x4b4
plugin_daemon-1  | 
plugin_daemon-1  | [GIN] 2025/03/06 - 06:32:22 | 200 |          4m0s |  192.168.144.11 | POST     "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/llm/invoke"
nginx-1          | 192.168.144.1 - - [06/Mar/2025:06:32:23 +0000] "GET /api/parameters HTTP/1.1" 200 1220 "https://dify.app.ynu.edu.cn/chat/gH9Gkwu0kmaIEloS" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 NetType/WIFI MicroMessenger/7.0.20.1781(0x6700143B) WindowsWechat(0x63090cbd) XWEB/11581 Flue" "113.55.124.122"

✔️ Expected Behavior

No errors.

Actual Behavior

The dify web services down sometimes.

Originally created by @liudonghua123 on GitHub (Mar 6, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.0.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce I encountered some service failures sometimes, and I checked the log and found the following errors. <details><summary>Logs</summary> ``` nginx-1 | 192.168.144.1 - - [06/Mar/2025:06:32:17 +0000] "GET /console/api/apps/28669fd5-16f0-45b3-831d-8a7231d0a3ca/workflows/draft HTTP/1.1" 200 3291 "https://dify.app.ynu.edu.cn/app/28669fd5-16f0-45b3-831d-8a7231d0a3ca/workflow" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36 Edg/133.0.0.0" "113.55.127.140" plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:18 | 200 | 439.616271ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/text_embedding/invoke" plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:18 | 200 | 704.057177ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/text_embedding/invoke" api-1 | 2025-03-06 06:32:18.985 INFO [Thread-1107 (_generate_worker)] [_client.py:1038] - HTTP Request: POST http://sandbox:8194/v1/sandbox/run "HTTP/1.1 200 OK" sandbox-1 | [GIN] 2025/03/06 - 06:32:18 | 200 | 182.220057ms | 192.168.96.4 | POST "/v1/sandbox/run" plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 22.474066ms | 192.168.144.11 | GET "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/management/models?page=1&page_size=256" plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 3.918187ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema" plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 4.059227ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema" plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 4.244066ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema" plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 6.569322ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema" plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 3.514292ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/model/schema" plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:19 | 200 | 2.260335ms | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/llm/num_tokens" plugin_daemon-1 | 2025/03/06 06:32:22 factory.go:28: [ERROR]PluginDaemonInternalServerError: killed by timeout plugin_daemon-1 | goroutine 48360 [running]: plugin_daemon-1 | runtime/debug.Stack() plugin_daemon-1 | /usr/local/go/src/runtime/debug/stack.go:24 +0x5e plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/types/exception.InternalServerError({0x180b640, 0xc0006c48c0}) plugin_daemon-1 | /app/internal/types/exception/factory.go:27 +0x27 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/service.baseSSEService[...](0xc000862ff0, 0x15f6c0d, 0xf0?) plugin_daemon-1 | /app/internal/service/base_sse.go:81 +0x4e6 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/service.InvokeLLM(0xc0001ec780, 0xc000028600, 0xf0) plugin_daemon-1 | /app/internal/service/invoke_model.go:35 +0x198 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.InvokeLLM.func8.1({{{0xc0005b27dd, 0x24}, {0xc0017044d0, 0x7}}, {{0x0, 0x0}}, {0xc000f06240, 0x58}, 0x0, 0x0, ...}) plugin_daemon-1 | /app/internal/server/controllers/model.go:20 +0x89 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server/controllers.BindPluginDispatchRequest[...].func1() plugin_daemon-1 | /app/internal/server/controllers/base.go:53 +0x9c plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server/controllers.BindRequest[...](0xc000028600, 0xc000863328) plugin_daemon-1 | /app/internal/server/controllers/base.go:31 +0x275 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server/controllers.BindPluginDispatchRequest[...](0x1603a20?, 0xa?) plugin_daemon-1 | /app/internal/server/controllers/base.go:37 +0x45 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.InvokeLLM.func8(0xc000028600) plugin_daemon-1 | /app/internal/server/controllers/model.go:17 +0x74 plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(0xc000028600) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).InitClusterID.func3(0xc000028600) plugin_daemon-1 | /app/internal/server/middleware.go:170 +0x54 plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(0xc000028600) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).RedirectPluginInvoke.func2(0xc000028600) plugin_daemon-1 | /app/internal/server/middleware.go:97 +0x74 plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(0xc000028600) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginDispatchGroup.(*App).FetchPluginInstallation.func1(0xc000028600) plugin_daemon-1 | /app/internal/server/middleware.go:66 +0x2f2 plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(0xc000028600) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 +0x2b plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).pluginGroup.CheckingKey.func1(0xc000028600) plugin_daemon-1 | /app/internal/server/middleware.go:24 +0x95 plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(...) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 plugin_daemon-1 | github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000028600) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/recovery.go:102 +0x7a plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(...) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 plugin_daemon-1 | github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000028600) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/logger.go:249 +0xe5 plugin_daemon-1 | github.com/gin-gonic/gin.(*Context).Next(...) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/context.go:185 plugin_daemon-1 | github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc00040a9c0, 0xc000028600) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:633 +0x892 plugin_daemon-1 | github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc00040a9c0, {0x181dac0, 0xc000216620}, 0xc00077db00) plugin_daemon-1 | /go/pkg/mod/github.com/gin-gonic/gin@v1.10.0/gin.go:589 +0x1b2 plugin_daemon-1 | net/http.serverHandler.ServeHTTP({0x1813868?}, {0x181dac0?, 0xc000216620?}, 0x6?) plugin_daemon-1 | /usr/local/go/src/net/http/server.go:3142 +0x8e plugin_daemon-1 | net/http.(*conn).serve(0xc001527b00, {0x1820270, 0xc0005b9ce0}) plugin_daemon-1 | /usr/local/go/src/net/http/server.go:2044 +0x5e8 plugin_daemon-1 | created by net/http.(*Server).Serve in goroutine 89 plugin_daemon-1 | /usr/local/go/src/net/http/server.go:3290 +0x4b4 plugin_daemon-1 | plugin_daemon-1 | [GIN] 2025/03/06 - 06:32:22 | 200 | 4m0s | 192.168.144.11 | POST "/plugin/ffbb405d-3e9f-4a58-868c-ce210c7c1864/dispatch/llm/invoke" nginx-1 | 192.168.144.1 - - [06/Mar/2025:06:32:23 +0000] "GET /api/parameters HTTP/1.1" 200 1220 "https://dify.app.ynu.edu.cn/chat/gH9Gkwu0kmaIEloS" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 NetType/WIFI MicroMessenger/7.0.20.1781(0x6700143B) WindowsWechat(0x63090cbd) XWEB/11581 Flue" "113.55.124.122" ``` </details> ### ✔️ Expected Behavior No errors. ### ❌ Actual Behavior The dify web services down sometimes.
yindo added the 🐞 bug label 2026-02-21 18:35:17 -05:00
yindo closed this issue 2026-02-21 18:35:18 -05:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#9486