langgenius/dify-plugin-daemon
1
0
Fork 0
mirror of https://github.com/langgenius/dify-plugin-daemon.git synced 2026-07-21 17:25:23 -04:00
Code Issues 111 Packages Projects Releases 10 Wiki Activity

Plugin daemon doesn't get the correct redis master under sentinel mode when master changes #196

New Issue
Closed
opened 2026-02-16 00:20:27 -05:00 by yindo · 1 comment
yindo commented 2026-02-16 00:20:27 -05:00
Owner
Copy Link
Copy Source

Originally created by @BorisPolonsky on GitHub (Oct 14, 2025).

Self Checks

To make sure we get to you in time, please check the following :)

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • "Please do not modify this template :) and fill in all the required fields."

Versions

  1. dify-plugin-daemon Version 0.3.0-local
  2. dify-api Version 1.9.1

Describe the bug
PluginDaemon doesn't update master info under sentinel mode after were re-election of Redis.

To Reproduce
Steps to reproduce the behavior:

  1. Setup Redis
kubectl get po -n test | grep redis
redis-sentinel-node-0                      2/2     Running     2 (3h8m ago)   18h
redis-sentinel-node-1                      2/2     Running     2 (3h8m ago)   18h
redis-sentinel-node-2                      2/2     Running     2 (3h8m ago)   18h
  1. Deploy dify 1.9.1, validate envs of plugin Daemon, and make sure it's working initially.
kubectl exec -it -n test -n test test-dify-plugin-daemon-6f975487c-nxm5j -- env | grep REDIS
REDIS_SENTINEL_PASSWORD=difyai123456
REDIS_SENTINEL_USERNAME=
REDIS_PASSWORD=difyai123456
REDIS_SENTINEL_SERVICE_NAME=mymaster
REDIS_USE_SENTINEL=true
REDIS_DB=0
REDIS_SENTINEL_SOCKET_TIMEOUT=0.1
REDIS_SENTINELS=redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local:26379,redis-sentinel-node-1.redis-sentinel-headless.test.svc.cluster.local:26379,redis-sentinel-node-2.redis-sentinel-headless.test.svc.cluster.local:26379
REDIS_USERNAME=
  1. Trigger a master switch in Redis and validate connection
kubectl exec -it -n test redis-sentinel-node-2 -- bash
Defaulted container "redis" out of: redis, sentinel
I have no name!@redis-sentinel-node-2:/$ redis-cli -h redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local -p 26379 -a difyai123456 sentinel get-master-addr-by-name mymaster
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1) "redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local"
2) "6379"
I have no name!@redis-sentinel-node-2:/$

Master node: redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local:6379
4. restart pluginDaemon and it fails to get correct master (redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local:6379) and tries to write to a READONLY node (redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local:6379)

kubectl logs -n test test-dify-plugin-daemon-6f975487c-nxm5j
2025/10/15 03:18:47 pool.go:32: [INFO]init routine pool, size: 10000
2025/10/15 03:18:48 init.go:104: [INFO]dify plugin db initialized
2025/10/15 03:18:48 manager.go:114: [INFO]start plugin manager daemon...
redis: 2025/10/15 03:18:48 sentinel.go:720: sentinel: new master="mymaster" addr="redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local:6379"
2025/10/15 03:18:48 watcher.go:17: [INFO]start to handle new plugins in path: plugin
2025/10/15 03:18:48 watcher.go:18: [INFO]Launching plugins with max concurrency: 2
2025/10/15 03:18:48 init.go:20: [INFO]Persistence initialized
SDK 2025/10/15 03:18:48 WARN Response has no supported checksum. Not validating response payload.
SDK 2025/10/15 03:18:48 WARN Response has no supported checksum. Not validating response payload.
2025/10/15 03:18:48 cluster_lifetime.go:87: [ERROR]failed to update the status of the node: READONLY You can't write against a read only replica.
[gnet] 2025-10-15T03:18:48.187048079Z   INFO    logging/logger.go:256   Launching gnet with 8 event-loops, listening on: tcp://0.0.0.0:5003
2025/10/15 03:18:48 runtime_lifetime.go:31: [ERROR]add plugin to cluster failed: READONLY You can't write against a read only replica.
2025/10/15 03:18:48 runtime_lifetime.go:31: [ERROR]add plugin to cluster failed: READONLY You can't write against a read only replica.
2025/10/15 03:18:51 cluster_lifetime.go:150: [ERROR]failed to vote the ips of the nodes: READONLY You can't write against a read only replica.
2025/10/15 03:18:51 cluster_lifetime.go:97: [ERROR]failed to vote the ips of the nodes: READONLY You can't write against a read only replica.
2025/10/15 03:18:51 plugin.go:109: [ERROR]failed to update plugin state: READONLY You can't write against a read only replica.
2025/10/15 03:18:51 plugin.go:109: [ERROR]failed to update plugin state: READONLY You can't write against a read only replica.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Originally created by @BorisPolonsky on GitHub (Oct 14, 2025). **Self Checks** To make sure we get to you in time, please check the following :) - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify-plugin-daemon/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] "Please do not modify this template :) and fill in all the required fields." **Versions** 1. dify-plugin-daemon Version 0.3.0-local 2. dify-api Version 1.9.1 **Describe the bug** PluginDaemon doesn't update master info under sentinel mode after were re-election of Redis. **To Reproduce** Steps to reproduce the behavior: 1. Setup Redis ``` kubectl get po -n test | grep redis redis-sentinel-node-0 2/2 Running 2 (3h8m ago) 18h redis-sentinel-node-1 2/2 Running 2 (3h8m ago) 18h redis-sentinel-node-2 2/2 Running 2 (3h8m ago) 18h ``` 2. Deploy dify 1.9.1, validate envs of plugin Daemon, and make sure it's working initially. ```bash kubectl exec -it -n test -n test test-dify-plugin-daemon-6f975487c-nxm5j -- env | grep REDIS REDIS_SENTINEL_PASSWORD=difyai123456 REDIS_SENTINEL_USERNAME= REDIS_PASSWORD=difyai123456 REDIS_SENTINEL_SERVICE_NAME=mymaster REDIS_USE_SENTINEL=true REDIS_DB=0 REDIS_SENTINEL_SOCKET_TIMEOUT=0.1 REDIS_SENTINELS=redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local:26379,redis-sentinel-node-1.redis-sentinel-headless.test.svc.cluster.local:26379,redis-sentinel-node-2.redis-sentinel-headless.test.svc.cluster.local:26379 REDIS_USERNAME= ``` 3. Trigger a master switch in Redis and validate connection ```bash kubectl exec -it -n test redis-sentinel-node-2 -- bash Defaulted container "redis" out of: redis, sentinel I have no name!@redis-sentinel-node-2:/$ redis-cli -h redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local -p 26379 -a difyai123456 sentinel get-master-addr-by-name mymaster Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 1) "redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local" 2) "6379" I have no name!@redis-sentinel-node-2:/$ ``` Master node: `redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local:6379` 4. restart `pluginDaemon` and it fails to get correct master (`redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local:6379`) and tries to write to a READONLY node (`redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local:6379`) ``` kubectl logs -n test test-dify-plugin-daemon-6f975487c-nxm5j 2025/10/15 03:18:47 pool.go:32: [INFO]init routine pool, size: 10000 2025/10/15 03:18:48 init.go:104: [INFO]dify plugin db initialized 2025/10/15 03:18:48 manager.go:114: [INFO]start plugin manager daemon... redis: 2025/10/15 03:18:48 sentinel.go:720: sentinel: new master="mymaster" addr="redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local:6379" 2025/10/15 03:18:48 watcher.go:17: [INFO]start to handle new plugins in path: plugin 2025/10/15 03:18:48 watcher.go:18: [INFO]Launching plugins with max concurrency: 2 2025/10/15 03:18:48 init.go:20: [INFO]Persistence initialized SDK 2025/10/15 03:18:48 WARN Response has no supported checksum. Not validating response payload. SDK 2025/10/15 03:18:48 WARN Response has no supported checksum. Not validating response payload. 2025/10/15 03:18:48 cluster_lifetime.go:87: [ERROR]failed to update the status of the node: READONLY You can't write against a read only replica. [gnet] 2025-10-15T03:18:48.187048079Z INFO logging/logger.go:256 Launching gnet with 8 event-loops, listening on: tcp://0.0.0.0:5003 2025/10/15 03:18:48 runtime_lifetime.go:31: [ERROR]add plugin to cluster failed: READONLY You can't write against a read only replica. 2025/10/15 03:18:48 runtime_lifetime.go:31: [ERROR]add plugin to cluster failed: READONLY You can't write against a read only replica. 2025/10/15 03:18:51 cluster_lifetime.go:150: [ERROR]failed to vote the ips of the nodes: READONLY You can't write against a read only replica. 2025/10/15 03:18:51 cluster_lifetime.go:97: [ERROR]failed to vote the ips of the nodes: READONLY You can't write against a read only replica. 2025/10/15 03:18:51 plugin.go:109: [ERROR]failed to update plugin state: READONLY You can't write against a read only replica. 2025/10/15 03:18:51 plugin.go:109: [ERROR]failed to update plugin state: READONLY You can't write against a read only replica. ``` **Expected behavior** A clear and concise description of what you expected to happen. **Screenshots** If applicable, add screenshots to help explain your problem. **Additional context** Add any other context about the problem here.
yindo closed this issue 2026-02-16 00:20:27 -05:00
yindo commented 2026-02-16 00:20:28 -05:00
Author
Owner
Copy Link
Copy Source

@BorisPolonsky commented on GitHub (Oct 14, 2025):

It appears that sentinel didn't return the correct master

kubectl exec -it -n test redis-sentinel-node-2 -- redis-cli -h redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local -p 6379 -a difyai123456 info replication | grep -E "role:|replica-read-only"
Defaulted container "redis" out of: redis, sentinel
role:slave

Closing for now.

@BorisPolonsky commented on GitHub (Oct 14, 2025): It appears that sentinel didn't return the correct master ```bash kubectl exec -it -n test redis-sentinel-node-2 -- redis-cli -h redis-sentinel-node-0.redis-sentinel-headless.test.svc.cluster.local -p 6379 -a difyai123456 info replication | grep -E "role:|replica-read-only" Defaulted container "redis" out of: redis, sentinel role:slave ``` Closing for now.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Windows
bug
cmd
documentation
enhancement
go
good first issue
plugin-daemon
pull-request
Mirrored from GitHub Pull Request
 question
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-plugin-daemon#196
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?