* remove enterprise logics
* release lock if runtime already installed
* support setting serverless endpoint by api
* remove global tenant id totally
* scan timeout tasks
* adding comments
* use index in loop
* add log
* feat(plugins): add FetchPluginReadme endpoint and update launch configurations
* feat: add PluginReadme database model
* feat: implement readme extracting and storage
* feat: implement readme endpoint
* feat: add plugin asset extraction endpoint with caching support
* Implement trigger functionality and clean up dynamic select code
- Added new trigger-related access types and actions in access.go.
- Introduced new HTTP routes for trigger operations in http_server.gen.go.
- Updated plugin declaration to include triggers in plugin_entities.
- Removed unused dynamic select service and controller files.
- Enhanced message handling in debugging_runtime to support trigger registration.
This update enhances the plugin system by integrating trigger capabilities while cleaning up legacy code.
* Refactor trigger-related types and enhance dynamic select functionality
- Updated TriggerProviderIdentity and TriggerProviderConfiguration to improve structure and validation.
- Renamed TriggerConfiguration to TriggerDeclaration for clarity.
- Added CredentialType to RequestDynamicParameterSelect for better request handling.
- Enhanced PluginDecoderHelper to read and unmarshal trigger files correctly.
These changes streamline the trigger system and improve the overall request handling in the plugin architecture.
* Add trigger functionality and enhance database integration
- Introduced TriggerInstallation model for managing trigger installations in the database.
- Updated autoMigrate function to include trigger installations in the migration process.
- Added new HTTP routes for listing and retrieving triggers in the HTTP server.
- Implemented ListTriggers and GetTrigger controller functions for handling trigger requests.
- Enhanced plugin management functions to create, update, and delete trigger installations during plugin lifecycle events.
These changes integrate trigger capabilities into the system, improving the overall plugin functionality and management.
* feat: add remapping for trigger icons in MediaBucket
- Enhanced the RemapAssets function to include remapping of trigger identity icons and dark icons.
- Added error handling for remapping failures to ensure robust asset management.
These changes improve the handling of trigger assets within the plugin system, ensuring icons are correctly remapped as needed.
* feat: add Multiple field to TriggerParameter for enhanced configuration
- Introduced a new Multiple field in the TriggerParameter struct to allow for multiple values in trigger configurations.
- This addition improves the flexibility of trigger parameters within the plugin system.
These changes enhance the capability of trigger parameters, enabling more complex configurations.
* feat: add Multiple field to ProviderConfig for enhanced configuration
- Introduced a new Multiple field in the ProviderConfig struct to allow for multiple values in provider configurations.
- This addition improves the flexibility of provider options within the plugin system.
These changes enhance the capability of provider configurations, enabling more complex setups.
* fix(plugin): update validation error messages in ManifestValidate method
- Enhanced error messages in the ManifestValidate function to include 'trigger' in the validation checks for plugin declarations.
- Updated logic to ensure that all relevant fields are considered when validating the presence of mutually exclusive parameters.
* feat(trigger): add CHECKBOX parameter type to plugin entities and refactor the trigger provider strcuture
- Introduced a new CHECKBOX parameter type in constant.go for plugin entities.
- Updated tool_declaration.go and trigger_declaration.go to include TOOL_PARAMETER_TYPE_CHECKBOX and TRIGGER_PARAMETER_TYPE_CHECKBOX respectively.
- Enhanced validation logic to accommodate the new CHECKBOX type in parameter checks.
* fix(trigger): update SubscriptionSchema validation in TriggerProviderDeclaration
- Changed SubscriptionSchema validation from 'omitempty' to 'required' in TriggerProviderDeclaration to ensure it is always provided.
- Updated SubscriptionConstructor field to be a pointer to allow for optional inclusion in the trigger provider configuration.
* fix(trigger): rename ParametersSchema to Parameters in SubscriptionConstructor
- Updated the SubscriptionConstructor struct to rename the ParametersSchema field to Parameters for consistency.
- Adjusted related JSON and YAML marshaling logic to reflect the new field name, ensuring proper handling of trigger parameters.
* refactor(trigger): enhance YAML unmarshalling for SubscriptionConstructor and SubscriptionSchema
- Introduced a new helper function to convert YAML nodes to ProviderConfig lists, improving the handling of subscription_schema and credentials_schema.
- Updated the UnmarshalYAML method to utilize the new function, simplifying the logic for parsing different YAML formats.
- Ensured proper initialization of SubscriptionConstructor fields to prevent nil pointer dereferences.
* fix(trigger): update SubscriptionConstructor validation in TriggerProviderDeclaration
- Changed the validation for SubscriptionConstructor in TriggerProviderDeclaration from 'omitempty,dive' to 'omitempty' to simplify the validation logic.
- Ensured that the SubscriptionConstructor field remains optional while maintaining its intended functionality.
* refactor(trigger): rename Trigger to Event in plugin entities and related structures
- Updated the naming conventions in trigger_declaration.go to replace 'Trigger' with 'Event' for better clarity and consistency.
- Adjusted related types, validation functions, and unmarshalling logic to reflect the new 'Event' terminology.
- Ensured that all references to triggers in the codebase are updated to events, including in the SubscriptionConstructor and response structures.
* refactor(trigger): rename TriggerInvoke to TriggerInvokeEvent and update related structures
- Renamed TriggerInvoke function and associated request/response types to TriggerInvokeEvent for improved clarity.
- Updated routing and controller methods to reflect the new naming convention.
- Ensured all references to the trigger invoke functionality are consistent with the new event terminology.
* refactor(trigger): remove Subscription struct from trigger_declaration.go and update TriggerDispatchEventRequest
- Removed the Subscription struct from trigger_declaration.go to streamline the codebase.
- Added Credentials field to TriggerDispatchEventRequest for enhanced functionality and clarity.
- Ensured that the changes maintain consistency with the existing naming conventions and structures.
* fix(trigger): improve nil checks for SubscriptionConstructor in TriggerProviderDeclaration
- Added nil checks for SubscriptionConstructor before accessing its fields to prevent potential nil pointer dereferences.
- Ensured that Parameters and CredentialsSchema are initialized only if SubscriptionConstructor is not nil, enhancing code robustness.
* fix(plugin): add recovery mechanism in OnTraffic to handle panics
- Introduced a deferred function in OnTraffic to recover from panics, logging the error and stack trace for better debugging.
- This enhancement improves the stability of the DifyServer by preventing crashes due to unexpected runtime errors.
* feat(trigger): add Subscription field to TriggerInvokeEventRequest
- Introduced a new Subscription field in the TriggerInvokeEventRequest struct to accommodate subscription data.
- Ensured the field is marked as required, enhancing the request's functionality and validation requirements.
* refactor(event): simplify EventDescription structure in EventDeclaration
- Removed the EventDescription struct and replaced it with a direct I18nObject field in EventDeclaration.
- This change streamlines the event configuration by reducing complexity while maintaining required validation for the description.
* feat(trigger): add UserID field to TriggerDispatchEventResponse
- Introduced a new UserID field in the TriggerDispatchEventResponse struct to include user identification in the response.
- The field is marked as optional, enhancing the response's flexibility while maintaining existing functionality.
* feat: add payload to TriggerDispatchEventResponse
* fix
* feat(trigger): update TriggerDispatchEventResponse structure
* fix: avoid path collusion
* fix: missing )
* fix: query param
* fix: form param
* fix: remove redundant dynamic parameter access type
* fix: remove dynamic parameter access type from validation
* Update internal/server/controllers/plugins.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Stream <Stream_2@qq.com>
Co-authored-by: Yeuoly <admin@srmxy.cn>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* feat(datasource): Implement datasource validation and invocation steps
- Added new functionality for validating datasource credentials and invoking the first and second steps of the datasource process.
- Introduced new API endpoints for datasource operations in the HTTP server.
- Created corresponding service and controller methods to handle datasource requests.
- Defined new request and response structures for datasource operations in the entities package.
* feat: add routes
* feat(datasource): Add initial datasource models and validation
- Introduced new `DatasourceInstallation` model to represent datasource installations with relevant fields.
- Created `datasource.go` file for future datasource service implementation.
- Enhanced `datasource_declaration.go` with new types and validation functions for datasource provider and parameters.
- Updated `plugin_declaration.go` to include datasource support in plugin structures.
- Refactored `agent_declaration.go` and `tool_declaration.go` to use a unified `ParameterOption` type for options.
* refactor(datasource): Update request and response types for online document content retrieval
- Renamed and updated request and response types for the `DatasourceGetOnlineDocumentPageContent` function to improve clarity and consistency.
- Introduced `RequestInvokeOnlineDocumentDatasourceGetContent` and `DatasourceInvokeOnlineDocumentGetContentResponse` types.
- Adjusted related function signatures and dispatchers to reflect the new types across the datasource service and controller implementations.
* feat(datasource): Implement datasource installation handling
- Added functionality to create and update `DatasourceInstallation` records during plugin installation and upgrade processes.
- Enhanced the `InstallPlugin` function to create a new datasource installation if a datasource declaration is present.
- Updated the `UpgradePlugin` function to handle the deletion of the original datasource installation and creation of a new one if the datasource declaration changes.
* feat(datasource): Add datasource registration handling in plugin runtime
- Introduced handling for datasource declarations in the plugin runtime.
- Updated `RemotePluginRuntime` to track datasource registration status.
- Enhanced message processing to include validation and assignment of datasource declarations.
* feat(datasource): Implement datasource listing and retrieval endpoints
- Added `ListDatasources` and `GetDatasource` functions to the service layer for handling datasource queries.
- Created corresponding controller methods to process HTTP requests for listing and retrieving datasources.
- Implemented request validation for tenant ID, page, page size, plugin ID, and provider parameters.
* add datasource routes
* feat(datasource): Add icon remapping for datasource declarations
- Implemented functionality to remap icons for both the main datasource and its individual datasources within the plugin declaration.
- Enhanced error handling to provide clearer feedback when remapping fails.
* fix(datasource): update OAuthSchema validation to remove unnecessary 'dive' tag
- Modified the validation tag for OAuthSchema in DatasourceProviderDeclaration to simplify the validation process by removing the 'dive' requirement.
* feat(tests): add datasource declaration parsing and output
- Introduced a new main.go file for testing datasource declaration parsing.
- Implemented JSON unmarshalling for RemotePluginRegisterPayload and DatasourceProviderDeclaration.
- Added a sample datasource declaration for testing purposes.
* fix(plugin): improve error handling in UninstallPlugin and add datasource deletion
- Enhanced error handling to return a specific message when a plugin is not installed.
- Added functionality to delete the associated datasource installation during the uninstallation process.
* feat(datasource): add support for decoding datasource provider declaration
- Added custom JSON marshalling and unmarshalling methods to handle CredentialsSchema and Datasources more effectively.
- Improved error handling during YAML unmarshalling to support both object and array formats for CredentialsSchema.
- Ensured proper initialization of DatasourceFiles and Tags to prevent nil references.
* fix: provider type
* refactor(datasource): simplify JSON unmarshalling for CredentialsSchema
- Removed complex handling of CredentialsSchema in the UnmarshalJSON method, focusing on the Datasources field.
- Streamlined the code to improve readability and maintainability by eliminating unnecessary checks and logic related to CredentialsSchema.
* feat: streaming datasource
* feat: datasource
* feat: datasource
* feat:datasource
* feat:datasource
* feat:datasource
* feat: add redirect_uri field to OAuth request structs
* feat: add online driver file request and response structures
* feat: add online driver file request and response structures
* feat: add online_driver datasource type to validation
* feat: rename online driver to online drive and update related classes and methods :)
* refactor: rename OnlineDocumentPageChunk to DatasourceGetPagesResponse and update related references
* feat: update request types for online drive browsing and downloading
* feat: add metadata field to OAuthGetCredentialsResult
* feat: built-in json schema definations
* feat(plugin_entities): add built-in schema definitions and processing for datasource YAML
* test(plugin_entities): add unit tests for schema definitions and YAML processing
* refactor(plugin_entities): centralize built-in schema definitions and processing
* refactor(plugin_entities): remove unused properties from built-in schema definitions
* refactor(plugin_entities): built-in schema & new datasource structure
* refactor(plugin_entities): enhance schema processing with checks and error handling
* refactor(plugin_entities): update validation for OnlineDriveBrowseFilesRequest prefix field to be optional
* refactor(json_schema): remove json schema definitions and validation
* fix(plugin_entities): remove output_schema validation tests
* feat: add rag tag
---------
Co-authored-by: Harry <xh001x@hotmail.com>
Co-authored-by: Dongyu Li <544104925@qq.com>
Co-authored-by: Novice <novice12185727@gmail.com>
- Updated AWS-related types and functions to reflect Serverless terminology, including renaming `AWSTransactionHandler` to `ServerlessTransactionHandler` and `AWSTransactionWriter` to `ServerlessTransactionWriter`.
- Adjusted error messages and comments to align with the new naming conventions.
- Modified installation functions to use Serverless terminology for clarity and consistency across the codebase.
* feat(oauth): implement refresh credentials functionality
- Added RefreshCredentials endpoint to handle OAuth credential refresh requests.
- Introduced RequestOAuthRefreshCredentials structure for request validation.
- Updated access types and actions to include refresh credentials.
- Enhanced server routing and controller logic to support the new functionality.
- Updated OAuth entities to include expiration handling for refreshed credentials.
* feat(oauth): add metadata field to OAuthGetCredentialsResult
---------
Co-authored-by: Harry <xh001x@hotmail.com>
- Implemented middleware to track active requests and active dispatch requests using atomic counters.
- Updated health check response to include counts of active requests and active dispatch requests.
- Integrated the new middleware into the HTTP server and plugin dispatch group for improved monitoring.
* feat(dynamic_select): implement dynamic parameter fetching functionality
- Added FetchDynamicParameterOptions function to handle dynamic parameter selection.
- Introduced new access type and action for dynamic select in access_types.
- Updated HTTP server routes to include the new endpoint for fetching dynamic parameters.
- Created necessary service and controller files for dynamic select operations.
* refactor(access_types): rename dynamic select access type to dynamic parameter
- Updated access type constants to reflect the change from PLUGIN_ACCESS_TYPE_DYNAMIC_SELECT to PLUGIN_ACCESS_TYPE_DYNAMIC_PARAMETER.
- Adjusted related references in the PluginDispatchers and FetchDynamicParameterOptions function to maintain consistency.
* feat: add decode plugin from identifier endpoint
- Introduced a new endpoint to decode a plugin from a unique identifier.
- Implemented the DecodePluginFromIdentifier function to handle decoding and verification of plugin signatures.
- Updated the HTTP server routes to include the new decode endpoint.
* refactor: update decode plugin endpoint path
- Moved the decode plugin from identifier endpoint to a new path for better organization.
- Updated the HTTP server routes accordingly to reflect the new endpoint structure.
* bump cloud-kit version to fix#343
* change env name USE_AWS_S3 to S3_USE_AWS
* update s3_use_aws default value to true
* update readme about the upgrade notice
* feat: Generate HTTP server routes from template
- Added a new file `http_server.gen.go` to automatically generate HTTP server routes based on defined dispatchers.
- Refactored existing route definitions in `http_server.go` to utilize the generated routes, improving maintainability.
- Introduced a code generation function in `generator.go` to create the HTTP server file, enhancing the plugin development workflow.
- Updated the template for HTTP server generation to streamline route creation for various controllers.
* fix: Update OAuth paths in PluginDispatchers for consistency
- Changed the path for authorization URL from `/oauth/authorization_url` to `/oauth/get_authorization_url`.
- Updated the path for credentials from `/oauth/credentials` to `/oauth/get_credentials` to align with naming conventions.
* feat: Add code generation for plugin controllers and services
- Introduced a code generation mechanism for plugin controllers and services, allowing for automatic generation based on defined dispatchers.
- Created new files for generated controllers, services, and templates to streamline the plugin invocation process.
- Removed outdated functions related to tool validation and runtime parameters, consolidating functionality into generated files.
- Updated dependencies in go.mod and go.sum to include necessary packages for the new code generation features.
* fix
* feat: add plugin reinstallation functionality and admin API support
- Implemented ReinstallToAWSFromPkg method to allow reinstallation of plugins on AWS Lambda, updating function URL and name.
- Added clearServerlessRuntimeCache method to manage serverless runtime cache.
- Enhanced LaunchPlugin to support an ignoreIdempotent flag for forced reinstallation.
- Introduced admin API endpoints for plugin reinstallation, secured with an API key validation middleware.
- Updated configuration to include AdminApiEnabled and AdminApiKey settings.
* refactor: update plugin reinstallation endpoint and improve unauthorized response
- Changed the plugin reinstallation endpoint from "/plugins/reinstall" to "/plugin/serverless/reinstall" for better clarity.
- Modified the unauthorized response in the AdminAPIKey middleware to return a more descriptive JSON message.
* feat: implement OAuth functionality in plugin daemon
- Added OAuth service methods for getting authorization URLs and credentials.
- Updated access types to include OAuth-related actions.
- Created new controller for handling OAuth requests.
- Introduced entities for OAuth results and requests.
- Enhanced plugin entities to support OAuth schema in tool declarations.
* feat: add OAuth endpoints for authorization and credentials retrieval
- Introduced new POST endpoints for obtaining authorization URLs and credentials in the OAuth controller.
- Enhanced the plugin dispatch group to include these new OAuth routes, improving integration with OAuth services.
Modify endpoint-related functions to support configurable maximum execution time:
- Update EndpointHandler signature to include maxExecutionTime
- Pass max execution timeout from config to endpoint service
- Modify timeout mechanism to use configurable duration instead of hardcoded 240 seconds
Michael.Y.Ma