Commit Graph

64 Commits

Author SHA1 Message Date
wangxiaolei 4314b8f1ee fix: synchronize remote debugging plugins across cluster nodes (#589)
Remote debugging plugins were not being synchronized across cluster nodes,
causing "no plugin available nodes found" errors when trying to invoke
plugins from different nodes.

1. **Remote debugging plugins not registered to cluster** - The
   `ClusterTunnel` notifier was not being added to ControlPanel
2. **Plugin ID inconsistency** - Remote plugins used different plugin_id
   formats during installation vs. querying
3. **Non-idempotent registration** - `RegisterPlugin` failed on reconnection
   with "plugin has been registered" error

- **internal/types/models/curd/atomic.go**:
  - Unify plugin_id calculation for remote plugins (author/name without version)
  - Remove plugin_id from plugin query conditions
  - Clear old cache when plugin_id is updated

- **internal/cluster/plugin.go**:
  - Make `RegisterPlugin` idempotent by updating existing plugin instead
    of returning error

- **internal/core/control_panel/daemon.go**:
  - Add cluster field to ControlPanel
  - Add SetCluster() method for lazy cluster initialization

- **internal/core/control_panel/server_debugger.go**:
  - Register remote debugging plugins to cluster on connection
  - Unregister from cluster on disconnection

- **internal/core/plugin_manager/manager.go**:
  - Add SetCluster() method to set cluster after initialization

- **internal/server/server.go**:
  - Call SetCluster() instead of AddClusterTunnel()

Only remote debugging plugins are synchronized across cluster nodes.
Local plugins run only on the node where they are installed and are
not registered to the cluster.

- Error handling improvements using `errors.Is()` instead of `==`
- Handle 404 for missing plugin assets gracefully
- Handle already-installed debugging plugins gracefully

- Remote debugging plugin can be invoked from any node in the cluster
- Plugin reconnection works without errors
- Cache invalidation works correctly when plugin_id changes
2026-03-05 14:46:13 +08:00
wangxiaolei f272e6c3fc feat: install plugin should be idempotent (#576)
* feat: install plugin should be idempotent

* chore: add new test
2026-01-27 16:28:23 +08:00
Oscaner Miao 2be1f651a9 feat(#450): add Redis SSL/TLS configuration support (#558)
* feat(#450): add Redis SSL/TLS configuration support

Add comprehensive SSL/TLS support for Redis connections with configurable certificate verification modes. Introduces new environment variables for SSL configuration including REDIS_USE_SSL, REDIS_SSL_CERT_REQS (supporting CERT_NONE, CERT_OPTIONAL, CERT_REQUIRED), and REDIS_SSL_CA_CERTS for custom CA certificates.

Changes:
- Add Redis SSL configuration options to .env.example
- Implement RedisTLSConfig() method to build tls.Config based on environment settings
- Pass TLS config to both standard Redis and Sentinel mode initializers
- Support custom CA certificate loading and verification modes
- Set minimum TLS version to 1.2 for security
- Minor whitespace cleanup in existing config comments

This enables secure Redis connections in production environments with flexible certificate verification options.

* fix(#450): prevent reference cycle in TLS config and simplify SSL setup

- Capture only RootCAs in VerifyConnection closure to avoid retaining
  entire tlsConf and potential reference cycles
- Remove redundant nil checks for tlsConf in Redis client initialization
  since tlsConf is guaranteed to be non-nil when useSsl is true
- Update comments to reflect actual behavior and constraints

* fix(#450): improve Redis TLS certificate verification logic for optional certificates

* fix(#450): simplify Redis TLS certificate verification logic for optional and required certificates

* docs(#450): add note for CA certificate file path in Redis SSL configuration

* test(#450): add comprehensive tests for Redis TLS configuration

* fix(#450): enhance Redis SSL configuration documentation and enforce CA cert requirement

* fix(#450): add nil TLS parameter to InitRedisClient calls in tests

Update all InitRedisClient function calls across test files to include the new nil parameter for TLS configuration. This change maintains backward compatibility by explicitly passing nil for TLS settings in non-TLS test scenarios.

* fix(#450): add default TLS configuration for Redis client when no tlsConf is provided
2026-01-13 16:27:16 +08:00
Byron.wang ca3d00229e Feat/Implement structured logging and Trace ID propagation (#552)
* use slog instead of log package and format to new log schema

* update the environment name to LOG_OUTPUT_FORMAT

* add the env to .env.example

* fix log reference error

* change the order of milldlewares

* delete unused code

* fix the concurrently session potential race condition

* fix the log format in tests

* update the duplicate code

* refactor: convert log functions to slog structured format

- Change log.Error/Info/Warn/Debug/Panic to accept msg + key-value pairs
- Remove printf-style formatting from log functions
- Update log calls in internal/cluster, internal/db, internal/core/session_manager
- Remove unused 'initialized' variable from log package
- Remaining files will be updated in follow-up commits

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: update all log call sites to use slog structured format

Convert all log.Error, log.Info, log.Warn, log.Debug, and log.Panic
calls from printf-style formatting to slog key-value pairs.

Before: log.Error("failed to do something: %s", err.Error())
After:  log.Error("failed to do something", "error", err)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: update cmd/ log calls to use slog structured format

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: implement GnetLogger for structured logging in gnet

* refactor: remove deprecated log visibility functions and related calls

* feat: enhance session management with trace and identity context propagation

* feat: implement serverless transaction handler and writer for plugin runtime

* refactor: rename context field to traceCtx in RealBackwardsInvocation

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Yeuoly <admin@srmxy.cn>
2025-12-30 11:00:48 +08:00
wangxiaolei 68db8475c9 fix: fix ReadCloser is not closed (#518) 2025-12-03 18:18:30 +08:00
Yeuoly 888ad788bc refactor: plugin lifecycle control panel (#499)
* refactor: introduce local plugin control panel and cleanup environment setup process

* fix: args

* refactor: new local runtime

* temp: stash work for refactor on RemotePluginServer

* refactor: unify local runtime lifetime and sperate init environment process

* chore: add missing files

* stash

* refactor: local plugin lifetime control

* refactor: complete installation process of control panel

* refactor: adapt service layer to new controlpanel

* refactor: pluginManager.Install

* fix: add routine wrap to InstallServerless, avoid blocking main thread

* feat: reinstall serverless runtime

* chore: add comments to Reinstall and update confusing naming

* refactor: unify install plugin service

* refactor: add labels to debugging runtime

* refactor: add getters to plugin manager

* refactor: split install service to decode/install_task/install service

* ???

* refactor: adapt controllers

* refactor: session write

* refactor: session runtime

* Refine install task orchestration (#501)

* refactor: installing task

* refactor cluster management, decouple lifetime management and cluster

* fix cli test command

* fix: cleanup TODO comments and implement GracefulStop for instance

* feat: add logger to control panel

* fix: multiple nil references

* refactor: better lifetime control

* refactor: better cycle interval

* fix(LocalPluginRuntime): prevent returning err when it's not  error

* fix: avoid adding empty PipExtraArgs

* fix: missing errors in Environment init

* fix: add truncateMessage to avoid db explosion

* cleanup: better lifecycle management

* fix: init status at the beginning of installation

* optimize: GracefulStop for pluginInstance

* refactor: tests

* refactor: centralize routine labels (#504)

* cleanup: RoutineKey

* fix: init routine pool

* fix: correctly handle cluster register error

* fix: memory leak

* fix: add \n to instance write

* fix(installer.go): set success to true after succeed for defer func

* refactor

* fix: missing cwd in testutils

* fix: scaleup default runtime nums to 1 when testing

* fix: localruntime appconfig in testing module

* Update internal/core/local_runtime/load_balancing.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* fix: more efficiency implement in installer_local.go

* fix: returns after failing in onDebuggingRuntimeDisconnected

* fix: returns after failing in onDebuggingRuntimeDisconnected

* fix: splits tests

* refactor: naming

* refactor: manifest.VersionX

* fix: adapt SetDefault to tests

* fix: enforce use constants in DBType

* fix: generate

* fix: linter

* cleanup tests

* refactor: change  package to

* cleanup: useless codes

* Update internal/cluster/plugin.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* cleanup

* refactor: decouple connection_key management from debugging_time

* refactor: confused naming

* feat: recycle resources to adapt to https://github.com/langgenius/dify-plugin-daemon/pull/500

* refactor: confusing redirecting

* fix: support get serverless runtime

* fix: race condition in Launching

* fix: avoid ManifestValidate in first step of debugging handshake

* fix: adding ReleaseAllLocks to finalizers

* wtf: what a beautiful code

* refactor: rename Stream.Async to Stream.Process

* fix: kill process if daed instance was detected

* fix: correctly handle failures

* fix: consistence of difference interfaces

* fix: add stacktrace to panic

* fix: only trigger once  event

* fix: ensure plugin runtime was shutdown

* feat: cleanup install tasks

* fix: add scale logs

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-11-18 17:28:02 +08:00
Yeuoly 31d7c7417e feat[0.1.0]: introduce 'run' command for local plugin execution (#283)
* feat: introduce 'run' command for local plugin execution

- Added a new command `run` to launch plugins locally, allowing communication through stdin/stdout.
- Removed the previous `test` command and its associated functionality to streamline the plugin testing process.

* feat: enhance 'run' command with TCP support

- Introduced a new `RunPluginPayload` structure to encapsulate plugin execution parameters.
- Added TCP communication mode to the `run` command, allowing multiple client connections.
- Updated command flags to configure run mode and logging options.
- Implemented client handling and server creation for both stdin/stdout and TCP modes.

* docs: improve comments in RunPlugin function for clarity

- Enhanced comments to provide clearer explanations of the plugin decoding process and the creation of client streams for both stdin/stdout and TCP modes.
- Updated comments to reflect the functionality and behavior of the plugin execution flow.

* refactor: update command structure and enhance plugin invocation handling

- Changed the command structure to add `runPluginCommand` under `pluginCommand` for better organization.
- Introduced `InvokePluginPayload` type to encapsulate plugin invocation details.
- Enhanced `RunPlugin` function to handle responses and errors more effectively, including logging to stdout.
- Updated client handling to support session management and improved error handling during plugin invocation.
- Renamed TCP server creation function for consistency.

* feat: enhance plugin response handling and logging capabilities

- Added a new flag `--response-format` to specify the output format (text or json) for plugin responses.
- Introduced a `logger` to manage logging output to stdout with timestamps and file information.
- Updated `logResponse` and `systemLog` functions to handle different response formats.
- Enhanced `handleClient` and `RunPlugin` functions to utilize the new response format feature.
- Implemented signal handling to clean up temporary directories on shutdown.

* feat: enhance plugin response structure and logging

- Added `InvokeID` to `InvokePluginPayload` and `GenericResponse` for better tracking of plugin invocations.
- Updated `logResponse` to include `InvokeID` in error responses for improved debugging.
- Enhanced client handling in `handleClient` to log plugin readiness and received requests.
- Refactored client stream creation for better readability and consistency.

* feat: add plugin invoke end response type and logging

- Introduced `GENERIC_RESPONSE_TYPE_PLUGIN_INVOKE_END` to enhance response tracking for plugin invocations.
- Updated `handleClient` to log the end of plugin invocation, improving visibility into the plugin lifecycle.

* chore: remove fullfeature tags
2025-05-16 14:20:29 +08:00
Taeyoung Park 5d83258371 Add optional Redis username authentication (#218)
* You can include username for Redis Auth

* Redis auth with username - test code update

* fix: tests

---------

Co-authored-by: Yeuoly <admin@srmxy.cn>
2025-04-25 20:42:14 +08:00
非法操作 25e013618a fix self.session.delete alway success (#12)
* fix self.session.delete alway success

* fix save data always failed

* redis delete response deleted item num

* add deleted num to invocate response
2025-04-11 13:32:52 +08:00
Yeuoly 26b61c3517 refactor: use random port for simulation server in redirect traffic test and add server readiness wait 2025-04-08 13:58:13 +09:00
Yeuoly 36e7a10169 fix: cobr unmarshal map[string]any to map[any]any, feat: add tests for redirecting traffic with path style and refactor redirect request logic 2025-04-08 13:51:26 +09:00
Yeuoly cddbad21be feat: add test for redirecting traffic without query parameters 2025-04-07 13:33:41 +09:00
Yeuoly 9e676b7468 feat: add test for redirecting traffic with query parameters and refactor URL construction logic 2025-04-07 13:32:28 +09:00
Yeuoly eed91f2359 feat: add X-Original-Host header handling in endpoint requests and improve request redirection URL construction 2025-04-07 13:28:53 +09:00
quicksandzn 6e32f5c505 optimize: tests 2025-03-27 14:55:37 +08:00
Yeuoly 1944a829d2 feat: add plugin access action to session write method (#64)
Modify session write method to include a new plugin access action parameter across multiple runtime implementations. This change allows for more granular tracking and handling of plugin interactions by introducing an optional action context to the write method.
2025-03-06 14:51:32 +08:00
Masashi Tomooka c6c882e86a support REDIS_USE_SSL config parameter (#21)
* support REDIS_USE_SSL config parameter

* fix tests

* add tests
2025-01-30 00:32:38 +08:00
Yeuoly 4e8d9c4e5e fix: avoid multiple existing message 2025-01-07 22:29:09 +08:00
Yeuoly 15060ee312 refactor: move entities to pkg 2025-01-06 14:52:02 +08:00
Yeuoly 8796edad8c refactor: confused code structure 2025-01-06 14:16:29 +08:00
Yeuoly 407fb941ca feat: support for pprof 2024-12-04 14:15:16 +08:00
Yeuoly 535921c44a fix: add some comments to test file 2024-11-27 19:38:38 +08:00
Yeuoly f1dbcf1835 fix: ci 2024-11-27 19:37:31 +08:00
Yeuoly 350bfafe14 fix: cluster scheduler controller 2024-11-27 18:59:56 +08:00
Yeuoly 94554c0934 fix: tests 2024-11-27 18:11:13 +08:00
Yeuoly 0af6f6fcd2 fix: ci 2024-11-27 00:07:16 +08:00
Yeuoly 883e12c516 plugin state id mismatched 2024-11-27 00:02:48 +08:00
Yeuoly 06ab89550a feat: support detailed logs 2024-11-26 23:42:35 +08:00
Yeuoly 09e214f2b2 feat: add bundle to plugin 2024-11-13 20:29:49 +08:00
Yeuoly 26fd94ebcd refactor: convert snakecase to camelcase 2024-11-12 21:24:41 +08:00
Yeuoly 670096c977 feat: support detailed error messages 2024-11-06 20:29:28 +08:00
Yeuoly 864b104fcc refactor: plugin assets management 2024-09-24 18:04:27 +08:00
Yeuoly 62a2a5c906 refactor: manifest 2024-09-13 15:21:36 +08:00
Yeuoly 0a23ebd3e1 release: aws handler 2024-09-12 23:18:38 +08:00
Yeuoly 1df861f9fc refactor: checksum 2024-09-11 14:46:16 +08:00
Yeuoly 8dd58b32c4 refactor 2024-09-09 20:03:32 +08:00
GareArc caee1daded fix: typo for checking map item existence 2024-09-04 23:51:18 -04:00
Yeuoly 3603a9c5b9 refactor 2024-09-04 14:39:07 +08:00
Yeuoly c8f699c9ce chore: cleanup 2024-09-03 22:39:17 +08:00
Yeuoly dc402cd948 refactor: plugin id/identity and plugin installation 2024-09-03 22:38:19 +08:00
Yeuoly c8c023521d add label to manifest 2024-09-02 20:16:10 +08:00
Yeuoly 8294a61d4c refactor: session manager 2024-08-23 23:27:43 +08:00
Yeuoly 11989403be refactor: backwards invocation 2024-08-23 20:58:33 +08:00
Yeuoly c714d99250 refactor: load plugins 2024-08-15 19:02:07 +08:00
Yeuoly 4972f3011f feat: register remote plugins 2024-08-07 17:39:16 +08:00
Yeuoly 6b554a40f7 tests: redirects 2024-08-07 17:02:01 +08:00
Yeuoly e844f96b4a refactor: address from ip to ip:port 2024-08-07 13:03:43 +08:00
Yeuoly 99d196c9bd tests: webhook 2024-08-05 16:30:22 +08:00
Yeuoly 0d5c4f6b98 fix: deepcopy directed http request 2024-08-05 15:23:57 +08:00
Yeuoly c86d0060c4 feat: redirect requests 2024-08-05 15:21:35 +08:00