Commit Graph

42 Commits

Author SHA1 Message Date
wangxiaolei 4314b8f1ee fix: synchronize remote debugging plugins across cluster nodes (#589)
Remote debugging plugins were not being synchronized across cluster nodes,
causing "no plugin available nodes found" errors when trying to invoke
plugins from different nodes.

1. **Remote debugging plugins not registered to cluster** - The
   `ClusterTunnel` notifier was not being added to ControlPanel
2. **Plugin ID inconsistency** - Remote plugins used different plugin_id
   formats during installation vs. querying
3. **Non-idempotent registration** - `RegisterPlugin` failed on reconnection
   with "plugin has been registered" error

- **internal/types/models/curd/atomic.go**:
  - Unify plugin_id calculation for remote plugins (author/name without version)
  - Remove plugin_id from plugin query conditions
  - Clear old cache when plugin_id is updated

- **internal/cluster/plugin.go**:
  - Make `RegisterPlugin` idempotent by updating existing plugin instead
    of returning error

- **internal/core/control_panel/daemon.go**:
  - Add cluster field to ControlPanel
  - Add SetCluster() method for lazy cluster initialization

- **internal/core/control_panel/server_debugger.go**:
  - Register remote debugging plugins to cluster on connection
  - Unregister from cluster on disconnection

- **internal/core/plugin_manager/manager.go**:
  - Add SetCluster() method to set cluster after initialization

- **internal/server/server.go**:
  - Call SetCluster() instead of AddClusterTunnel()

Only remote debugging plugins are synchronized across cluster nodes.
Local plugins run only on the node where they are installed and are
not registered to the cluster.

- Error handling improvements using `errors.Is()` instead of `==`
- Handle 404 for missing plugin assets gracefully
- Handle already-installed debugging plugins gracefully

- Remote debugging plugin can be invoked from any node in the cluster
- Plugin reconnection works without errors
- Cache invalidation works correctly when plugin_id changes
2026-03-05 14:46:13 +08:00
Byron.wang ca3d00229e Feat/Implement structured logging and Trace ID propagation (#552)
* use slog instead of log package and format to new log schema

* update the environment name to LOG_OUTPUT_FORMAT

* add the env to .env.example

* fix log reference error

* change the order of milldlewares

* delete unused code

* fix the concurrently session potential race condition

* fix the log format in tests

* update the duplicate code

* refactor: convert log functions to slog structured format

- Change log.Error/Info/Warn/Debug/Panic to accept msg + key-value pairs
- Remove printf-style formatting from log functions
- Update log calls in internal/cluster, internal/db, internal/core/session_manager
- Remove unused 'initialized' variable from log package
- Remaining files will be updated in follow-up commits

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: update all log call sites to use slog structured format

Convert all log.Error, log.Info, log.Warn, log.Debug, and log.Panic
calls from printf-style formatting to slog key-value pairs.

Before: log.Error("failed to do something: %s", err.Error())
After:  log.Error("failed to do something", "error", err)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: update cmd/ log calls to use slog structured format

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: implement GnetLogger for structured logging in gnet

* refactor: remove deprecated log visibility functions and related calls

* feat: enhance session management with trace and identity context propagation

* feat: implement serverless transaction handler and writer for plugin runtime

* refactor: rename context field to traceCtx in RealBackwardsInvocation

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Yeuoly <admin@srmxy.cn>
2025-12-30 11:00:48 +08:00
Michael.Y.Ma 8ff1c2c9f2 Feat: Add OSS-specific parameters for HW and ALI private deployment (#539)
* Update internal/types/app/config.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update internal/server/server.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-17 16:36:56 +08:00
Yunlu Wen f954688930 enterprise related adjustment (#510)
* remove enterprise logics

* release lock if runtime already installed

* support setting serverless endpoint by api

* remove global tenant id totally

* scan timeout tasks

* adding comments

* use index in loop

* add log
2025-12-03 18:13:54 +08:00
Yeuoly 888ad788bc refactor: plugin lifecycle control panel (#499)
* refactor: introduce local plugin control panel and cleanup environment setup process

* fix: args

* refactor: new local runtime

* temp: stash work for refactor on RemotePluginServer

* refactor: unify local runtime lifetime and sperate init environment process

* chore: add missing files

* stash

* refactor: local plugin lifetime control

* refactor: complete installation process of control panel

* refactor: adapt service layer to new controlpanel

* refactor: pluginManager.Install

* fix: add routine wrap to InstallServerless, avoid blocking main thread

* feat: reinstall serverless runtime

* chore: add comments to Reinstall and update confusing naming

* refactor: unify install plugin service

* refactor: add labels to debugging runtime

* refactor: add getters to plugin manager

* refactor: split install service to decode/install_task/install service

* ???

* refactor: adapt controllers

* refactor: session write

* refactor: session runtime

* Refine install task orchestration (#501)

* refactor: installing task

* refactor cluster management, decouple lifetime management and cluster

* fix cli test command

* fix: cleanup TODO comments and implement GracefulStop for instance

* feat: add logger to control panel

* fix: multiple nil references

* refactor: better lifetime control

* refactor: better cycle interval

* fix(LocalPluginRuntime): prevent returning err when it's not  error

* fix: avoid adding empty PipExtraArgs

* fix: missing errors in Environment init

* fix: add truncateMessage to avoid db explosion

* cleanup: better lifecycle management

* fix: init status at the beginning of installation

* optimize: GracefulStop for pluginInstance

* refactor: tests

* refactor: centralize routine labels (#504)

* cleanup: RoutineKey

* fix: init routine pool

* fix: correctly handle cluster register error

* fix: memory leak

* fix: add \n to instance write

* fix(installer.go): set success to true after succeed for defer func

* refactor

* fix: missing cwd in testutils

* fix: scaleup default runtime nums to 1 when testing

* fix: localruntime appconfig in testing module

* Update internal/core/local_runtime/load_balancing.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* fix: more efficiency implement in installer_local.go

* fix: returns after failing in onDebuggingRuntimeDisconnected

* fix: returns after failing in onDebuggingRuntimeDisconnected

* fix: splits tests

* refactor: naming

* refactor: manifest.VersionX

* fix: adapt SetDefault to tests

* fix: enforce use constants in DBType

* fix: generate

* fix: linter

* cleanup tests

* refactor: change  package to

* cleanup: useless codes

* Update internal/cluster/plugin.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* cleanup

* refactor: decouple connection_key management from debugging_time

* refactor: confused naming

* feat: recycle resources to adapt to https://github.com/langgenius/dify-plugin-daemon/pull/500

* refactor: confusing redirecting

* fix: support get serverless runtime

* fix: race condition in Launching

* fix: avoid ManifestValidate in first step of debugging handshake

* fix: adding ReleaseAllLocks to finalizers

* wtf: what a beautiful code

* refactor: rename Stream.Async to Stream.Process

* fix: kill process if daed instance was detected

* fix: correctly handle failures

* fix: consistence of difference interfaces

* fix: add stacktrace to panic

* fix: only trigger once  event

* fix: ensure plugin runtime was shutdown

* feat: cleanup install tasks

* fix: add scale logs

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-11-18 17:28:02 +08:00
sHuXnHs ad4a7c4d73 tx cos support endpoint (#506) 2025-11-18 14:14:00 +08:00
Yunlu Wen 9e3081bd17 feat: support orphan plugin (#440)
* support orphan plugin in serverless mode

* add validation to UpgradePlugin

* log reinstalls
2025-09-12 20:34:38 +08:00
Byron.wang 18e91bbb37 fix s3 client path style not used (#344)
* bump cloud-kit version to fix #343

* change env name USE_AWS_S3 to S3_USE_AWS

* update s3_use_aws default value to true

* update readme about the upgrade notice
2025-06-13 15:10:21 +08:00
Byron.wang debb3744c0 add USE_AWS_S3 args avoid ambiguity. (#340) 2025-06-11 14:32:53 +08:00
Byron.wang 1c9e28bc75 Feat: Replace the internal/oss module with dify-cloud-kit (#317)
* replace internal oss with dify-cloud-kit

* remove validate

* fix tests

* fix tests
2025-05-30 16:44:59 +08:00
bravomark 7492a3d8cd feat: Support Alibaba Cloud OSS (#261)
* feat: support aliyun OSS

* feat: support aliyun OSS
2025-05-07 16:46:18 +08:00
Hironori Yamamoto 2406e0a09f feat: implement gcs storage (#237)
Co-authored-by: Hironori Yamamoto <hironori-yamamoto@m3.com>
2025-04-25 13:29:08 +08:00
te-chan 2f84673539 feat: Add Azure blob storage connectivity 2025-04-03 21:17:47 +09:00
Yeuoly d43de8b7b2 Merge pull request #157 from lengyhua/main
feat: simplify the process in db migrate & fix comment error
2025-03-29 00:00:16 +09:00
lengyhua 28f582a6a9 feat: add oss supported types 2025-03-28 01:56:37 -04:00
akisaya 038f152769 make pathStyle param configurable when using s3 compatible storage 2025-03-22 16:59:15 +08:00
Yeuoly ef41d22df4 Merge pull request #97 from quicksandznzn/main
feat: oss support tencent cos
2025-03-20 15:56:48 +08:00
yuhang2.zhang 261886167b update: Add Support for Custom S3 Endpoints to Enhance Compatibility with S3-Compatible Services. 2025-03-18 17:20:47 +08:00
quicksandzn d3db63ea68 feat: oss support tencent cos 2025-03-12 11:25:38 +08:00
Yeuoly 9ef9f16e6e feat: support for sentry 2024-12-17 20:51:14 +08:00
Yeuoly 84703ef1b4 refactor: uni storage 2024-11-08 19:17:19 +08:00
Yeuoly 69dfe0d8f8 enhancement: remove force reloading process of each time process launched 2024-11-06 13:58:53 +08:00
Yeuoly 79b6e0450c refactor: install plugin to locals 2024-10-29 17:04:03 +08:00
Yeuoly a55f6f507f refactor: install plugin from local 2024-10-18 20:29:02 +08:00
Yeuoly 0a23ebd3e1 release: aws handler 2024-09-12 23:18:38 +08:00
Yeuoly cda65741b1 feat: install 2024-09-10 18:26:22 +08:00
Yeuoly 046875513e refactor: env 2024-08-28 23:41:37 +08:00
Yeuoly e139f38c1f feat: service support persistence 2024-08-28 23:37:55 +08:00
Yeuoly b338eb77ea feat: support persistence 2024-08-28 13:45:58 +08:00
Yeuoly 86733a5837 chore: remove lambda 2024-08-12 16:16:08 +08:00
Yeuoly 6798809680 refactor: storage 2024-08-09 16:27:01 +08:00
Yeuoly 99d196c9bd tests: webhook 2024-08-05 16:30:22 +08:00
Yeuoly c86d0060c4 feat: redirect requests 2024-08-05 15:21:35 +08:00
Yeuoly 4073e2fa5e tests: cluster lifetime 2024-08-01 17:03:03 +08:00
Yeuoly 2b3529296d feat: schedule plugins 2024-08-01 00:57:23 +08:00
Yeuoly 1ea1134488 refactor 2024-07-30 18:30:21 +08:00
Yeuoly 367cad55da feat: cluster managment 2024-07-30 17:36:32 +08:00
Yeuoly ce4ce368e2 feat: add db 2024-07-29 16:02:24 +08:00
Yeuoly d35cf3cf4e fix: undead processes 2024-07-24 18:36:18 +08:00
Yeuoly f85bd95840 fix: subprocess hangs 2024-07-24 17:35:10 +08:00
Yeuoly bc3218ca8c feat: examples 2024-07-02 22:28:53 +08:00
Yeuoly 3fc489f4d3 feat: tcp 2024-07-01 21:21:09 +08:00