langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #17928] fix: start the plugin daemon after the database has become healthy #28801

New Issue
Closed
opened 2026-02-21 20:44:09 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:44:09 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/17928

State: closed
Merged: Yes

Summary

This PR Changes following:

  • Update plugin_daemon.depends_on to have db.condition.service_healthy to wait for the database to be healthy before plugin_daemon is started
  • Update db.healthcheck to ensure that external connections are possible using the hostname (-h), username (-U), and database name (-d).
  • Increase retries from 30 to 60 since it will take a little longer than before to be judged as healthy.

Closes #17927

Screenshots

Before

$ docker inspect docker-plugin_daemon-1 | jq '.[0].RestartCount'

✨ Restarts 7 times before it works normally
7

$ docker compose logs plugin_daemon

✨ The 1st attempt
plugin_daemon-1  | 2025/04/12 08:36:03 pool.go:32: [INFO]init routine pool, size: 10000
plugin_daemon-1  | 
plugin_daemon-1  | 2025/04/12 08:36:03 /app/internal/db/pg/pg.go:14
plugin_daemon-1  | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=dify_plugin`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  |
plugin_daemon-1  | 2025/04/12 08:36:03 /app/internal/db/pg/pg.go:18
plugin_daemon-1  | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  | 2025/04/12 08:36:03 init.go:85: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  | panic: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  |
plugin_daemon-1  | goroutine 1 [running]:
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/utils/log.writeLog({0x1857285, 0x5}, {0x1888432?, 0x7?}, 0x1, {0xc00012fce0, 0x1, 0x1})
plugin_daemon-1  |      /app/internal/utils/log/log.go:40 +0x305
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/utils/log.Panic(...)
plugin_daemon-1  |      /app/internal/utils/log/log.go:66
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/db.Init(0x2710?)
plugin_daemon-1  |      /app/internal/db/init.go:85 +0x285
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).Run(0xc0001b4228, 0xc0002e7008)
plugin_daemon-1  |      /app/internal/server/server.go:74 +0xd2
plugin_daemon-1  | main.main()
plugin_daemon-1  |      /app/cmd/server/main.go:28 +0x125

✨ The 2nd attempt
plugin_daemon-1  | 2025/04/12 08:36:05 pool.go:32: [INFO]init routine pool, size: 10000
plugin_daemon-1  |
plugin_daemon-1  | 2025/04/12 08:36:05 /app/internal/db/pg/pg.go:14
plugin_daemon-1  | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=dify_plugin`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  |
plugin_daemon-1  | 2025/04/12 08:36:05 /app/internal/db/pg/pg.go:18
plugin_daemon-1  | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  | 2025/04/12 08:36:05 init.go:85: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  | panic: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  |
plugin_daemon-1  | goroutine 1 [running]:
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/utils/log.writeLog({0x1857285, 0x5}, {0x1888432?, 0x7?}, 0x1, {0xc00041fce0, 0x1, 0x1})
plugin_daemon-1  |      /app/internal/utils/log/log.go:40 +0x305
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/utils/log.Panic(...)
plugin_daemon-1  |      /app/internal/utils/log/log.go:66
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/db.Init(0x2710?)
plugin_daemon-1  |      /app/internal/db/init.go:85 +0x285
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).Run(0xc0001b6378, 0xc00026f008)
plugin_daemon-1  |      /app/internal/server/server.go:74 +0xd2
plugin_daemon-1  | main.main()
plugin_daemon-1  |      /app/cmd/server/main.go:28 +0x125

✨ The 3rd attempt
plugin_daemon-1  | 2025/04/12 08:36:06 pool.go:32: [INFO]init routine pool, size: 10000
plugin_daemon-1  |
plugin_daemon-1  | 2025/04/12 08:36:06 /app/internal/db/pg/pg.go:14
plugin_daemon-1  | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=dify_plugin`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  |
plugin_daemon-1  | 2025/04/12 08:36:06 /app/internal/db/pg/pg.go:18
plugin_daemon-1  | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  | 2025/04/12 08:36:06 init.go:85: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  | panic: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused)
plugin_daemon-1  |
plugin_daemon-1  | goroutine 1 [running]:
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/utils/log.writeLog({0x1857285, 0x5}, {0x1888432?, 0x7?}, 0x1, {0xc000513ce0, 0x1, 0x1})
plugin_daemon-1  |      /app/internal/utils/log/log.go:40 +0x305
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/utils/log.Panic(...)
plugin_daemon-1  |      /app/internal/utils/log/log.go:66
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/db.Init(0x2710?)
plugin_daemon-1  |      /app/internal/db/init.go:85 +0x285
plugin_daemon-1  | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).Run(0xc000114b70, 0xc0001df008)
plugin_daemon-1  |      /app/internal/server/server.go:74 +0xd2
plugin_daemon-1  | main.main()
plugin_daemon-1  |      /app/cmd/server/main.go:28 +0x125

...
✨ Finally started normally
plugin_daemon-1  | 2025/04/12 08:36:20 pool.go:32: [INFO]init routine pool, size: 10000
plugin_daemon-1  |
plugin_daemon-1  | 2025/04/12 08:36:20 /app/internal/db/pg/pg.go:14
plugin_daemon-1  | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=dify_plugin`: server error (FATAL: database "dify_plugin" does not exist (SQLSTATE 3D000))     
plugin_daemon-1  | 2025/04/12 08:36:22 init.go:93: [INFO]dify plugin db initialized
plugin_daemon-1  | 2025/04/12 08:36:22 manager.go:159: [INFO]start plugin manager daemon...
plugin_daemon-1  | 2025/04/12 08:36:22 watcher.go:16: [INFO]start to handle new plugins in path: plugin
plugin_daemon-1  | 2025/04/12 08:36:22 init.go:19: [INFO]Persistence initialized
plugin_daemon-1  | [gnet] 2025-04-12T08:36:22.73824398Z INFO    logging/logger.go:256   Launching gnet with 8 event-loops, listening on: tcp://0.0.0.0:5003
plugin_daemon-1  | 2025/04/12 08:36:23 cluster_lifetime.go:113: [INFO]current node has become the master of the cluster

After

$ docker inspect docker-plugin_daemon-1 | jq '.[0].RestartCount'

✨ Never be restarted
0

$ docker compose logs plugin_daemon

✨ Started nomally on the 1st attempt, without any panics
plugin_daemon-1  | 2025/04/12 09:02:39 pool.go:32: [INFO]init routine pool, size: 10000
plugin_daemon-1  | 
plugin_daemon-1  | 2025/04/12 09:02:39 /app/internal/db/pg/pg.go:14
plugin_daemon-1  | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=dify_plugin`: server error (FATAL: database "dify_plugin" does not exist (SQLSTATE 3D000))     
plugin_daemon-1  | 2025/04/12 09:02:41 init.go:93: [INFO]dify plugin db initialized
plugin_daemon-1  | 2025/04/12 09:02:41 manager.go:159: [INFO]start plugin manager daemon...
plugin_daemon-1  | 2025/04/12 09:02:41 init.go:19: [INFO]Persistence initialized
plugin_daemon-1  | 2025/04/12 09:02:41 watcher.go:16: [INFO]start to handle new plugins in path: plugin
plugin_daemon-1  | [gnet] 2025-04-12T09:02:41.279542923Z        INFO    logging/logger.go:256   Launching gnet with 8 event-loops, listening on: tcp://0.0.0.0:5003
plugin_daemon-1  | 2025/04/12 09:02:41 cluster_lifetime.go:113: [INFO]current node has become the master of the cluster

Checklist

Important

Please review the checklist below before submitting your pull request.

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
**Original Pull Request:** https://github.com/langgenius/dify/pull/17928 **State:** closed **Merged:** Yes --- # Summary This PR Changes following: - Update `plugin_daemon.depends_on` to have `db.condition.service_healthy` to wait for the database to be healthy before `plugin_daemon` is started - Update `db.healthcheck` to ensure that external connections are possible using the hostname (-h), username (-U), and database name (-d). - Increase `retries` from 30 to 60 since it will take a little longer than before to be judged as healthy. Closes #17927 # Screenshots ## Before ```bash $ docker inspect docker-plugin_daemon-1 | jq '.[0].RestartCount' ✨ Restarts 7 times before it works normally 7 ``` ```bash $ docker compose logs plugin_daemon ✨ The 1st attempt plugin_daemon-1 | 2025/04/12 08:36:03 pool.go:32: [INFO]init routine pool, size: 10000 plugin_daemon-1 | plugin_daemon-1 | 2025/04/12 08:36:03 /app/internal/db/pg/pg.go:14 plugin_daemon-1 | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=dify_plugin`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | plugin_daemon-1 | 2025/04/12 08:36:03 /app/internal/db/pg/pg.go:18 plugin_daemon-1 | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | 2025/04/12 08:36:03 init.go:85: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | panic: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | plugin_daemon-1 | goroutine 1 [running]: plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/utils/log.writeLog({0x1857285, 0x5}, {0x1888432?, 0x7?}, 0x1, {0xc00012fce0, 0x1, 0x1}) plugin_daemon-1 | /app/internal/utils/log/log.go:40 +0x305 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/utils/log.Panic(...) plugin_daemon-1 | /app/internal/utils/log/log.go:66 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/db.Init(0x2710?) plugin_daemon-1 | /app/internal/db/init.go:85 +0x285 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).Run(0xc0001b4228, 0xc0002e7008) plugin_daemon-1 | /app/internal/server/server.go:74 +0xd2 plugin_daemon-1 | main.main() plugin_daemon-1 | /app/cmd/server/main.go:28 +0x125 ✨ The 2nd attempt plugin_daemon-1 | 2025/04/12 08:36:05 pool.go:32: [INFO]init routine pool, size: 10000 plugin_daemon-1 | plugin_daemon-1 | 2025/04/12 08:36:05 /app/internal/db/pg/pg.go:14 plugin_daemon-1 | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=dify_plugin`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | plugin_daemon-1 | 2025/04/12 08:36:05 /app/internal/db/pg/pg.go:18 plugin_daemon-1 | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | 2025/04/12 08:36:05 init.go:85: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | panic: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | plugin_daemon-1 | goroutine 1 [running]: plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/utils/log.writeLog({0x1857285, 0x5}, {0x1888432?, 0x7?}, 0x1, {0xc00041fce0, 0x1, 0x1}) plugin_daemon-1 | /app/internal/utils/log/log.go:40 +0x305 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/utils/log.Panic(...) plugin_daemon-1 | /app/internal/utils/log/log.go:66 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/db.Init(0x2710?) plugin_daemon-1 | /app/internal/db/init.go:85 +0x285 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).Run(0xc0001b6378, 0xc00026f008) plugin_daemon-1 | /app/internal/server/server.go:74 +0xd2 plugin_daemon-1 | main.main() plugin_daemon-1 | /app/cmd/server/main.go:28 +0x125 ✨ The 3rd attempt plugin_daemon-1 | 2025/04/12 08:36:06 pool.go:32: [INFO]init routine pool, size: 10000 plugin_daemon-1 | plugin_daemon-1 | 2025/04/12 08:36:06 /app/internal/db/pg/pg.go:14 plugin_daemon-1 | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=dify_plugin`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | plugin_daemon-1 | 2025/04/12 08:36:06 /app/internal/db/pg/pg.go:18 plugin_daemon-1 | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | 2025/04/12 08:36:06 init.go:85: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | panic: [PANIC]failed to init dify plugin db: failed to connect to `host=db user=postgres database=postgres`: dial error (dial tcp 172.18.0.4:5432: connect: connection refused) plugin_daemon-1 | plugin_daemon-1 | goroutine 1 [running]: plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/utils/log.writeLog({0x1857285, 0x5}, {0x1888432?, 0x7?}, 0x1, {0xc000513ce0, 0x1, 0x1}) plugin_daemon-1 | /app/internal/utils/log/log.go:40 +0x305 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/utils/log.Panic(...) plugin_daemon-1 | /app/internal/utils/log/log.go:66 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/db.Init(0x2710?) plugin_daemon-1 | /app/internal/db/init.go:85 +0x285 plugin_daemon-1 | github.com/langgenius/dify-plugin-daemon/internal/server.(*App).Run(0xc000114b70, 0xc0001df008) plugin_daemon-1 | /app/internal/server/server.go:74 +0xd2 plugin_daemon-1 | main.main() plugin_daemon-1 | /app/cmd/server/main.go:28 +0x125 ... ✨ Finally started normally plugin_daemon-1 | 2025/04/12 08:36:20 pool.go:32: [INFO]init routine pool, size: 10000 plugin_daemon-1 | plugin_daemon-1 | 2025/04/12 08:36:20 /app/internal/db/pg/pg.go:14 plugin_daemon-1 | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=dify_plugin`: server error (FATAL: database "dify_plugin" does not exist (SQLSTATE 3D000)) plugin_daemon-1 | 2025/04/12 08:36:22 init.go:93: [INFO]dify plugin db initialized plugin_daemon-1 | 2025/04/12 08:36:22 manager.go:159: [INFO]start plugin manager daemon... plugin_daemon-1 | 2025/04/12 08:36:22 watcher.go:16: [INFO]start to handle new plugins in path: plugin plugin_daemon-1 | 2025/04/12 08:36:22 init.go:19: [INFO]Persistence initialized plugin_daemon-1 | [gnet] 2025-04-12T08:36:22.73824398Z INFO logging/logger.go:256 Launching gnet with 8 event-loops, listening on: tcp://0.0.0.0:5003 plugin_daemon-1 | 2025/04/12 08:36:23 cluster_lifetime.go:113: [INFO]current node has become the master of the cluster ``` ## After ```bash $ docker inspect docker-plugin_daemon-1 | jq '.[0].RestartCount' ✨ Never be restarted 0 ``` ```bash $ docker compose logs plugin_daemon ✨ Started nomally on the 1st attempt, without any panics plugin_daemon-1 | 2025/04/12 09:02:39 pool.go:32: [INFO]init routine pool, size: 10000 plugin_daemon-1 | plugin_daemon-1 | 2025/04/12 09:02:39 /app/internal/db/pg/pg.go:14 plugin_daemon-1 | [error] failed to initialize database, got error failed to connect to `host=db user=postgres database=dify_plugin`: server error (FATAL: database "dify_plugin" does not exist (SQLSTATE 3D000)) plugin_daemon-1 | 2025/04/12 09:02:41 init.go:93: [INFO]dify plugin db initialized plugin_daemon-1 | 2025/04/12 09:02:41 manager.go:159: [INFO]start plugin manager daemon... plugin_daemon-1 | 2025/04/12 09:02:41 init.go:19: [INFO]Persistence initialized plugin_daemon-1 | 2025/04/12 09:02:41 watcher.go:16: [INFO]start to handle new plugins in path: plugin plugin_daemon-1 | [gnet] 2025-04-12T09:02:41.279542923Z INFO logging/logger.go:256 Launching gnet with 8 event-loops, listening on: tcp://0.0.0.0:5003 plugin_daemon-1 | 2025/04/12 09:02:41 cluster_lifetime.go:113: [INFO]current node has become the master of the cluster ``` # Checklist > [!IMPORTANT] > Please review the checklist below before submitting your pull request. - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [ ] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [ ] I've updated the documentation accordingly. - [ ] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods
yindo added the pull-request label 2026-02-21 20:44:09 -05:00
yindo closed this issue 2026-02-21 20:44:09 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#28801
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?