mirror of
https://github.com/langgenius/dify-plugin-daemon.git
synced 2026-07-21 17:25:23 -04:00
install plugin in cluster env, InitPythonEnvironment will delete .venv dir causing race conditions and failures #221
Closed
opened 2026-02-16 00:20:33 -05:00 by yindo
·
6 comments
No Branch/Tag Specified
main
build/fix-serverless-runtime-error-propagation
gh-pages
build/test-document2
build/test-document
build/onboarding-ui
build/slim-extract
codex/depot-builds
codex/mac-runner-benchmark
feat/storage-path-prefix
feat/multi-db-user
feat/slim-action
deploy/dev
feat/dify-cli
feat/add-e2e
build/pg-bouncer
codex/add-multimodal-rerank-and-embedding-apis
refactor/local-runtime
build/multimodal-embeddings
codex/refactor-routine.submit-label-handling
codex/refactor-service-layer-based-on-provided-plan
feat/trigger-response
feat/trigger
build/trigger
deploy/trigger-dev
codex/remove-claude-code-reviewer-from-github-ci
codex/add-manifest-check-to-upload-endpoint
feat/no-root-dockerfile
fix/json-schema
fix/blocking-anthorized-langgenius
454-bump-cli-template
build/datasource
feat/datasource
bump-cloud-kit
feat/rag-tag
fix/change-session-not-found-to-400
feat/plugin-readme
add-claude-github-actions-1756274550417
docs/comprehensive-development-documentation
chore/remove-json-schema-validation-error
71cef04
fix/missing-parameter-type
fix/sessions-log
fix/template.env
bump/go-git
build/oauth
feat/oauth-refresh-token
feat/plugin-oauth
feat/tool-oauth-cli
build/plugin-oauth
feat/readme-i18n
fix/memory-leak
feat/icon-dark
feat/default-icon
plugin_launch_concurent
feat/collect-active-requests
feat/dark-icon
feat/support-structured-llm-output
feat/dynamic-selector
fix/reduce-logs
feat/decode-plugin-package
feat/db-extras
fix/backwards-invocation-overflow
feat/length-prefixed-chunking
fix/http-request-reader-header
chore/unify-configurations
fix/hardcoded-serverless-runtime-timeout
fix/cmd
fix/signature
refactor/implement-gen-routes
fix/redis-lock
refactor/codegen
feat/add-authorized-category
chore/style
feat/run-plugin-cli
reduce/run-once
feat/reinstall-serverless-runtime
feat/support-setup-process
fix/apply-stdio-buffer-size
chore/add-warning-messages-to-installed-bucket
feat/repo
enhance/stdio
feat/make-buffer-size-configurable
fix/moderation-init
fix/only-validate-profile-on-quick-mode
feat/support-quick-init-plugins
refactor/oauth-parameters
feat/oauth
refactor/simplify-plugin-invocation
test/integration-test-for-plugins
fix/backwards-compatible-to-llm-result-chunk
feat/auto-scale
enhance/reduce-ci-tests
fix/cli-ci
enhance/removes-llm-result-prompt-messages
fix/disable-benchmark-logs
benchmark/local-runtime
chore/remove-useless-benchmark
feat/benchmark
feat/fetch-app-info
fix/windows-remap-assets
fix/skip-hidden-file
feat/stream-tool-blob-message
fix/path-travel
feat/template-add-ci
enhance/version-compare
feat/sign-apple-os-cli
feat/support-minimal-dify-version-required
refactor/stdip
feat/add-serverless-connector-launching-timeout
fix/remove-prompt_messages-from-llm-result-chunk
feat/standardize-plugin-sdk-versions
chore/update-docs-and-refine-wording
update/readme-cli
fix/infinity-environment-setup
fix/cbor-unmarshaling
fix/use-aws-iam-baseendpoint
fix/lost-query-params-in-endpoint
fix/tiktoken
cohre/update-readme
fix/redis-tests
fix/friendly-identity
fix/marshal-any-map
chore/upgrade-ants
fix/graceful-precompile
enhance/tiktoken
feat/graceful-shutdown
fix/close-serverless-response
fix/correct-cli-guide
fix/plugin-active-log
fix/remove-proxy-args-from-uv
fix/endpoint-hook-url
fix/ci-credentials
feat/disable-gevent
fix/add-gcc
fix/hardcoded-endpoint-timeout
fix/bump-cli-sdk-version
fix/deadloop-when-redis-disconnect
fix/enhence/speed-up-environment-setup
enhance/introduce-uv
fix/change-default-db
fix/deadlock
readme
chore/env.example
feat/add-action-in-url
fix/add-more-pip-args
fix/optimize-local-heartbeat
optimize/db-init
fix/optimize-internal-server-error
fix/increase-default-plugin-max-execution-timeout
fix/force-patch-older-version
enhance/increase-installing-process
LICENSE
fix/set-user-id-to-unrequired
fix/max-launching-concurrent
improve/error-handing-in-serverless
refactor/json-unmarshaler-enhancement
fix/add-pip-mirror-url
enhance/serverless-connector
0.6.5
0.6.4
0.6.3
0.6.2
0.6.1
0.6.0
0.5.9
0.5.8
0.5.7
0.5.6
0.5.5
0.5.4
0.5.3
0.5.2
0.5.1
0.5.0
0.4.1
0.4.0
0.3.3
0.3.2
0.3.1
0.3.0
0.3.0b1
0.2.0
0.1.3
0.1.2
0.1.1
0.1.0
0.0.10
0.0.9
0.0.8
0.0.7
0.0.6
0.0.5
0.0.4
0.0.3
0.0.2
0.0.1
0.0.1-beta.23
0.0.1-beta.22
0.0.1-beta.21
0.0.1-beta.20
0.0.1-beta.19
0.0.1-beta.18
0.0.1-beta.17
0.0.1-beta.16
0.0.1-beta.15
0.0.1-beta.14
0.0.1-beta.13
0.0.1-beta.12
0.0.1-beta.11
0.0.1-beta.10
0.0.1-beta.9
0.0.1-beta.8
0.0.1-beta.7
0.0.1-beta.6
0.0.1-beta.5
0.0.1-beta.4
0.0.1-beta.3
0.0.1-beta.2
0.0.1-beta.1
Milestone
No items
No Milestone
Projects
Clear projects
No project
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: langgenius/dify-plugin-daemon#221
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @avtion on GitHub (Dec 17, 2025).
Originally assigned to: @fatelei on GitHub.
Self Checks
To make sure we get to you in time, please check the following :)
Versions
Describe the bug
When running multiple pods of dify-plugin-daemon in k8s, the
InitPythonEnvironment()function can be executed concurrently by different pods for the same plugin. This race condition may cause thedeleteVirtualEnvironment()function to be triggered while another pod is still initializing the Python virtual environment, resulting in the virtual environment being deleted during the initialization process. This leads to initialization failures and unpredictable behavior.Root Cause
The issue is triggered by the
handleNewLocalPlugins()function in the cluster mode:startLocalMonitor()which callshandleNewLocalPlugins()every 30 secondshandleNewLocalPlugins()lists all installed plugins from the sharedinstalledBucket(cross-pod shared storage)c.localPluginRuntimes.Exists())LaunchLocalPlugin()->InitEnvironment()->InitPythonEnvironment()LaunchLocalPlugin()has alocalPluginInstallationLock, this lock is in-memory and only prevents concurrent launches within the same poddeleteVirtualEnvironment()destroys the environment being built by another podTo Reproduce
Steps to reproduce the behavior:
Expected behavior
@dosubot[bot] commented on GitHub (Dec 17, 2025):
Hi @avtion! I'm Dosu and I’m helping the dify-plugin-daemon team.
This is a known issue in cluster environments: when multiple dify-plugin-daemon pods share storage, they can concurrently run
InitPythonEnvironment, which deletes and recreates the.venvdirectory. The in-memory lock (localPluginInstallationLock) only prevents concurrent initialization within a single pod, not across pods, so race conditions occur and environments get deleted mid-initialization, causing failures and "virtual environment is invalid" errors. There is currently no distributed locking for environment initialization—only a temporary distributed lock (using Redis) for plugin installation tasks, not for the Python environment setup itself (code reference, issue #527).The open-source version does not officially support multi-instance cluster mode; this is acknowledged by maintainers, and the enterprise edition may have different features (issue #132). There are no built-in workarounds or best practices documented for this scenario. To avoid these race conditions, you can: run only a single replica, avoid sharing plugin working directories across pods, or implement your own distributed lock (e.g., using Redis) around environment initialization.
If you need true multi-pod support with shared storage, a distributed locking mechanism is required to serialize environment setup for each plugin.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
@avtion commented on GitHub (Dec 17, 2025):
@dosu this is not about multi install or update plugin request, this is a new promble for 0.5.0 version ControlPanel watchdog functions.
@dosubot[bot] commented on GitHub (Dec 17, 2025):
You're right—this isn't about concurrent install/update requests, but a new issue introduced by the ControlPanel watchdog logic in 0.5.0. The watchdog (specifically, startLocalMonitor and handleNewLocalPlugins) now scans for and auto-launches plugins every 30 seconds across all pods. In a cluster, this means multiple pods can simultaneously try to initialize the same plugin environment, each running InitPythonEnvironment and deleteVirtualEnvironment without any distributed lock. This causes race conditions where one pod deletes the .venv directory while another is still initializing it, leading to failures and "virtual environment is invalid" errors.
There’s currently no built-in distributed locking or coordination for this watchdog-triggered environment setup—only for plugin install/upgrade tasks. The only mitigations are to run a single replica, avoid shared plugin directories, or implement your own distributed lock around environment initialization. No config flags or hooks exist to disable or coordinate the watchdog in multi-pod setups, and no recent PRs address this gap.
If you need true multi-pod support, a distributed lock (e.g., via Redis) is required to serialize environment setup for each plugin.
Relevant code reference
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
@Yeuoly commented on GitHub (Dec 17, 2025):
Hi, In a cluster environment, you should never share the
cwddirectory among pods, it definitely cases the race condition error, instead, you may separate them, only share directories likepluginsplugin_packagesetc.@avtion commented on GitHub (Dec 17, 2025):
@Yeuoly Thanks for your reply. Not sharing the
cwdfolder between pods is indeed a way to avoid concurrent operations by multiple pods, but it also introduces new issues.If the cwd folder is not shared, it means that each Pod needs to individually unzip the
difypkgfile, download dependencies, and precompile during startup or when installing new plugins. In cases with a large number of plugins, this could lead to significantly longer startup times for each Pod.At the same time, in the current Helm deployment method recommended by Dify,
PLUGIN_WORKING_PATHusually points to/app/storage/cwd, while the mounted Persistent Volume directory is/app/storage. Splitting thecwddirectory does not seem to be an easy task for previously deployed clusters.https://github.com/douban/charts/blob/master/charts/dify/templates/deployment.yaml#L503
https://github.com/douban/charts/blob/master/charts/dify/templates/deployment.yaml#L503
Perhaps could consider introducing support for distributed locks in the
InitPythonEnvironmentprocess to avoid issues that may arise from multiple pods coexisting and potentially triggeringdeleteVirtualEnvironmentwhen sharing thecwddirectory.@Yeuoly commented on GitHub (Dec 19, 2025):
@avtion , it's by design, adding locks here the there causes so much issues in distributed systems, in cases:
There were so much issues we need to handle, as a result, you need to use https://github.com/langgenius/dify-plugin-daemon/blob/main/docs/runtime/sri.md if you are hosting a huge system which requires many plugins or you can contact business service.
As the secondary solution, the local plugin runtime requires a long time startup, but it works, I guess you may increase the parallel startup counts here https://github.com/langgenius/dify-plugin-daemon/blob/main/internal/types/app/config.go#L103C2-L103C32, and use rolling updates on k8s