langgenius/dify-plugin-daemon
1
0
Fork 0
mirror of https://github.com/langgenius/dify-plugin-daemon.git synced 2026-07-21 17:25:23 -04:00
Code Issues 111 Packages Projects Releases 10 Wiki Activity

BUG: Plugin Daemon Signals Readiness Before Plugins Complete Startup in Kubernetes Deployments #244

New Issue
Open
opened 2026-02-16 01:15:07 -05:00 by yindo · 1 comment
yindo commented 2026-02-16 01:15:07 -05:00
Owner
Copy Link
Copy Source

Originally created by @NieRonghua on GitHub (Feb 6, 2026).

Self Checks

To make sure we get to you in time, please check the following :)

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • "Please do not modify this template :) and fill in all the required fields."

Versions

  1. dify-plugin-daemon Version: 0.5.1
  2. dify-api Version: 1.11.1

Describe the bug

In Kubernetes environments, the plugin daemon has a race condition during startup:

  1. HTTP service starts immediately (0s) → readiness probe gets 200 ✅
  2. Plugins start asynchronously in background (1-600+ seconds) ⏳
  3. K8s Gateway detects ready status (~5s) → starts forwarding traffic 🚨
  4. Plugins still initializing or startup fails (~30-600s) → requests return errors ❌

Root Causes

# Issue Impact
1 No plugin state awareness readiness probe only checks HTTP service, not plugin readiness
2 Time desynchronization HTTP fast (1-5s), plugins slow (30-600s), async startup
3 Poor observability Cannot see plugin startup progress or failure reasons
4 Hardcoded retry limit 15 retries fixed in code, startup takes 600+ seconds

Current Behavior

0s      : HTTP server running ✅
         readiness probe → 200 OK (but plugins not ready!)

~5s     : K8s Gateway traffic begins flowing
         Requests fail if plugins still loading

~30-600s: Plugins finally ready (or failed)
         Too late! Traffic already flowing

Business Constraints

  • Private cloud deployment - no manual intervention possible
  • Partial availability acceptable - service should start even if some plugins fail
  • Fast startup required - K8s gateway needs sufficient time to detect true ready state

To Reproduce

Steps to reproduce the behavior in Kubernetes:

  1. Deploy dify-plugin-daemon to Kubernetes with current readiness probe pointing to /health/check
  2. Observe initial readiness probe response time (returns 200 immediately)
  3. Wait for plugins to fully load (10-600+ seconds depending on plugin count/type)
  4. Monitor traffic during this window:
    • Traffic arrives before plugins fully loaded
    • Requests fail with "plugin not ready" or timeout errors
  5. Check logs to see retry attempts are hardcoded at 15 times

Expected behavior

  • ✅ readiness probe returns 503 while plugins are still starting
  • ✅ readiness probe returns 200 only after all plugin startup attempts complete (success or failure after max retries)
  • ✅ Configurable retry limit instead of hardcoded 15
  • ✅ Visible plugin startup state and error information
  • ✅ Startup time reduced from 600s to ~225s (63% improvement)

Screenshots
The Kubernetes pod health check has passed, and the pod has started accepting API requests forwarded by the gateway.
Image

However, the plugin in local mode is still starting up, causing the plugin to be called.

Image

Additional context
Dify is deployed via Kubernetes.

Originally created by @NieRonghua on GitHub (Feb 6, 2026). **Self Checks** To make sure we get to you in time, please check the following :) - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify-plugin-daemon/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] "Please do not modify this template :) and fill in all the required fields." **Versions** 1. **dify-plugin-daemon Version**: 0.5.1 2. **dify-api Version**: 1.11.1 ## Describe the bug In Kubernetes environments, the plugin daemon has a **race condition during startup**: 1. **HTTP service starts immediately** (0s) → readiness probe gets 200 ✅ 2. **Plugins start asynchronously** in background (1-600+ seconds) ⏳ 3. **K8s Gateway detects ready status** (~5s) → starts forwarding traffic 🚨 4. **Plugins still initializing** or startup fails (~30-600s) → requests return errors ❌ ### Root Causes | # | Issue | Impact | |---|-------|--------| | 1 | **No plugin state awareness** | readiness probe only checks HTTP service, not plugin readiness | | 2 | **Time desynchronization** | HTTP fast (1-5s), plugins slow (30-600s), async startup | | 3 | **Poor observability** | Cannot see plugin startup progress or failure reasons | | 4 | **Hardcoded retry limit** | 15 retries fixed in code, startup takes 600+ seconds | ### Current Behavior ``` 0s : HTTP server running ✅ readiness probe → 200 OK (but plugins not ready!) ~5s : K8s Gateway traffic begins flowing Requests fail if plugins still loading ~30-600s: Plugins finally ready (or failed) Too late! Traffic already flowing ``` ### Business Constraints - **Private cloud deployment** - no manual intervention possible - **Partial availability acceptable** - service should start even if some plugins fail - **Fast startup required** - K8s gateway needs sufficient time to detect true ready state --- ## To Reproduce Steps to reproduce the behavior in Kubernetes: 1. Deploy dify-plugin-daemon to Kubernetes with current readiness probe pointing to `/health/check` 2. Observe initial readiness probe response time (returns 200 immediately) 3. Wait for plugins to fully load (10-600+ seconds depending on plugin count/type) 4. Monitor traffic during this window: - Traffic arrives before plugins fully loaded - Requests fail with "plugin not ready" or timeout errors 5. Check logs to see retry attempts are hardcoded at 15 times --- ## Expected behavior - ✅ readiness probe returns **503** while plugins are still starting - ✅ readiness probe returns **200** only after **all plugin startup attempts complete** (success or failure after max retries) - ✅ Configurable retry limit instead of hardcoded 15 - ✅ Visible plugin startup state and error information - ✅ Startup time reduced from 600s to ~225s (63% improvement) --- **Screenshots** The Kubernetes pod health check has passed, and the pod has started accepting API requests forwarded by the gateway. <img width="1668" height="189" alt="Image" src="https://github.com/user-attachments/assets/b5b8fc69-885f-41e0-9a8d-c4ca7a577970" /> However, the plugin in local mode is still starting up, causing the plugin to be called. <img width="465" height="164" alt="Image" src="https://github.com/user-attachments/assets/c5ef0164-af86-47bb-a5e1-0234c22603f1" /> **Additional context** Dify is deployed via Kubernetes.
yindo added the bug label 2026-02-16 01:15:07 -05:00
yindo commented 2026-02-16 01:15:07 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 6, 2026):

You've correctly identified a real issue: the current readiness probe at /health/check always returns 200 OK as soon as the HTTP server starts, without checking if plugins have finished initializing. This means Kubernetes routes traffic before plugins are ready, causing errors like "plugin not ready" or "plugin runtime not found" during startup windows. The readiness endpoint is implemented in internal/server/controllers/health_check.go and does not consult plugin state or lifecycle at all—it just reports basic service status and metrics, regardless of plugin readiness [source].

Internally, plugin startup state is tracked using the PluginRuntime and PluginRuntimeState structs, which record status (active, launching, pending, etc.), restarts, timestamps, and logs [source]. However, this state is not surfaced in the readiness probe or health endpoint.

There are currently no configuration options or environment variables to control readiness probe behavior or plugin startup retry limits. The retry limit is hardcoded, and the only relevant plugin startup config is PLUGIN_LOCAL_LAUNCHING_CONCURRENT, which controls parallel launches but not readiness or retries [source].

To address this, you would need to modify the /health/check handler to check the state of all plugin runtimes and return 503 until all plugins have either started successfully or failed after their maximum retry attempts. You can access plugin states via the plugin manager and aggregate their statuses. Once all plugins are in a terminal state (active or stopped), return 200; otherwise, return 503 and include plugin startup progress or error info in the response for better observability.

There are no merged PRs or existing config flags for this, so this would require a code change. For distributed/multi-pod setups, be aware that plugin state is not synchronized across pods, which can cause additional "plugin not found" errors unless you implement a distributed lock or run a single replica [source].

If you need a starting point, focus on updating internal/server/controllers/health_check.go to consult plugin states and expose them in the readiness response. This will make the readiness probe accurately reflect true service readiness and help avoid traffic routing before plugins are ready.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Feb 6, 2026): <!-- Answer --> You've correctly identified a real issue: the current readiness probe at `/health/check` always returns 200 OK as soon as the HTTP server starts, without checking if plugins have finished initializing. This means Kubernetes routes traffic before plugins are ready, causing errors like "plugin not ready" or "plugin runtime not found" during startup windows. The readiness endpoint is implemented in `internal/server/controllers/health_check.go` and does not consult plugin state or lifecycle at all—it just reports basic service status and metrics, regardless of plugin readiness [[source]](https://github.com/langgenius/dify-plugin-daemon/blob/f272e6c3fcd8a18c6ff571d7fcbfde857f63ef99/internal/server/controllers/health_check.go#L33-L45). Internally, plugin startup state is tracked using the `PluginRuntime` and `PluginRuntimeState` structs, which record status (`active`, `launching`, `pending`, etc.), restarts, timestamps, and logs [[source]](https://github.com/langgenius/dify-plugin-daemon/blob/f272e6c3fcd8a18c6ff571d7fcbfde857f63ef99/pkg/entities/plugin_entities/runtime.go#L15-L191). However, this state is not surfaced in the readiness probe or health endpoint. There are currently no configuration options or environment variables to control readiness probe behavior or plugin startup retry limits. The retry limit is hardcoded, and the only relevant plugin startup config is `PLUGIN_LOCAL_LAUNCHING_CONCURRENT`, which controls parallel launches but not readiness or retries [[source]](https://github.com/langgenius/dify-plugin-daemon/blob/f272e6c3fcd8a18c6ff571d7fcbfde857f63ef99/internal/types/app/config.go#L22-L241). To address this, you would need to modify the `/health/check` handler to check the state of all plugin runtimes and return 503 until all plugins have either started successfully or failed after their maximum retry attempts. You can access plugin states via the plugin manager and aggregate their statuses. Once all plugins are in a terminal state (`active` or `stopped`), return 200; otherwise, return 503 and include plugin startup progress or error info in the response for better observability. There are no merged PRs or existing config flags for this, so this would require a code change. For distributed/multi-pod setups, be aware that plugin state is not synchronized across pods, which can cause additional "plugin not found" errors unless you implement a distributed lock or run a single replica [[source]](https://github.com/langgenius/dify-plugin-daemon/issues/577). If you need a starting point, focus on updating `internal/server/controllers/health_check.go` to consult plugin states and expose them in the readiness response. This will make the readiness probe accurately reflect true service readiness and help avoid traffic routing before plugins are ready. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/f5f7223b-b7f0-4cf8-9a8c-fada5b2fa6f4?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/f5f7223b-b7f0-4cf8-9a8c-fada5b2fa6f4?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/f5f7223b-b7f0-4cf8-9a8c-fada5b2fa6f4?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/f5f7223b-b7f0-4cf8-9a8c-fada5b2fa6f4?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/f5f7223b-b7f0-4cf8-9a8c-fada5b2fa6f4?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/f5f7223b-b7f0-4cf8-9a8c-fada5b2fa6f4?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/f5f7223b-b7f0-4cf8-9a8c-fada5b2fa6f4?feedback_type=other)</sup> [![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=You%27ve%20correctly%20identified%20a%20real%20issue%3A%20the%20current%20readiness%20probe%20at%20%60/health/check%60%20always%20returns%20200%20OK%20as%20soon%20as%20the%20HTTP%20server%20starts%2C%20without%20checking%20if%20plugins%20have%20finished%20initializing.%20This%20means%20Kubernetes%20routes%20traffic%20before%20plugins%20are%20ready%2C%20causing%20errors%20like%20%22plugin%20not%20ready%22%20or%20%22plugin%20runtime%20not%20found%22%20during%20startup%20windows.%20The%20readiness%20endpoint%20is%20implemented%20in%20%60internal/server/controllers/health_check.go%60%20and%20does%20not%20consult%20plugin%20state%20or%20lifecycle%20at%20all%E2%80%94it%20just%20reports%20basic%20service%20status%20and%20metrics%2C%20regardless%20of%20plugin%20readiness%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify-plugin-daemon/blob/f272e6c3fcd8a18c6ff571d7fcbfde857f63ef99/internal/server/controllers/health_check.go%23L33-L45%29.%0A%0AInternally%2C%20plugin%20startup%20state%20is%20tracked%20using%20the%20%60PluginRuntime%60%20and%20%60PluginRuntimeState%60%20structs%2C%20which%20record%20status%20%28%60active%60%2C%20%60launching%60%2C%20%60pending%60%2C%20etc.%29%2C%20restarts%2C%20timestamps%2C%20and%20logs%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify-plugin-daemon/blob/f272e6c3fcd8a18c6ff571d7fcbfde857f63ef99/pkg/entities/plugin_entities/runtime.go%23L15-L191%29.%20However%2C%20this%20state%20is%20not%20surfaced%20in%20the%20readiness%20probe%20or%20health%20endpoint.%0A%0AThere%20are%20currently%20no%20configuration%20options%20or%20environment%20variables%20to%20control%20readiness%20probe%20behavior%20or%20plugin%20startup%20retry%20limits.%20The%20retry%20limit%20is%20hardcoded%2C%20and%20the%20only%20relevant%20plugin%20startup%20config%20is%20%60PLUGIN_LOCAL_LAUNCHING_CONCURRENT%60%2C%20which%20controls%20parallel%20launches%20but%20not%20readiness%20or%20retries%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify-plugin-daemon/blob/f272e6c3fcd8a18c6ff571d7fcbfde857f63ef99/internal/types/app/config.go%23L22-L241%29.%0A%0ATo%20address%20this%2C%20you%20would%20need%20to%20modify%20the%20%60/health/check%60%20handler%20to%20check%20the%20state%20of%20all%20plugin%20runtimes%20and%20return%20503%20until%20all%20plugins%20have%20either%20started%20successfully%20or%20failed%20after%20their%20maximum%20retry%20attempts.%20You%20can%20access%20plugin%20states%20via%20the%20plugin%20manager%20and%20aggregate%20their%20statuses.%20Once%20all%20plugins%20are%20in%20a%20terminal%20state%20%28%60active%60%20or%20%60stopped%60%29%2C%20return%20200%3B%20otherwise%2C%20return%20503%20and%20include%20plugin%20startup%20progress%20or%20error%20info%20in%20the%20response%20for%20better%20observability.%0A%0AThere%20are%20no%20merged%20PRs%20or%20existing%20config%20flags%20for%20this%2C%20so%20this%20would%20require%20a%20code%20change.%20For%20distributed/multi-pod%20setups%2C%20be%20aware%20that%20plugin%20state%20is%20not%20synchronized%20across%20pods%2C%20which%20can%20cause%20additional%20%22plugin%20not%20found%22%20errors%20unless%20you%20implement%20a%20distributed%20lock%20or%20run%20a%20single%20replica%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify-plugin-daemon/issues/577%29.%0A%0AIf%20you%20need%20a%20starting%20point%2C%20focus%20on%20updating%20%60internal/server/controllers/health_check.go%60%20to%20consult%20plugin%20states%20and%20expose%20them%20in%20the%20readiness%20response.%20This%20will%20make%20the%20readiness%20probe%20accurately%20reflect%20true%20service%20readiness%20and%20help%20avoid%20traffic%20routing%20before%20plugins%20are%20ready.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify-plugin-daemon/issues/598)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Windows
bug
cmd
documentation
enhancement
go
good first issue
plugin-daemon
pull-request
Mirrored from GitHub Pull Request
 question
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-plugin-daemon#244
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?