- Introduced a deferred function in OnTraffic to recover from panics, logging the error and stack trace for better debugging.
- This enhancement improves the stability of the DifyServer by preventing crashes due to unexpected runtime errors.
- Renamed TriggerInvoke function and associated request/response types to TriggerInvokeEvent for improved clarity.
- Updated routing and controller methods to reflect the new naming convention.
- Ensured all references to the trigger invoke functionality are consistent with the new event terminology.
* feat(plugin): implement unauthorized Langgenius plugin blocking
- Added configuration option to disable installation of plugins falsely claiming Langgenius authorship.
- Introduced new error handling for unauthorized Langgenius claims during plugin installation.
- Implemented tests to validate unauthorized Langgenius detection logic.
- Updated environment configuration and service files to support the new feature.
* fix: typo
* refactor(plugin): update Langgenius plugin signature enforcement
- Renamed configuration option from DISABLE_UNAUTHORIZED_LANGGENIUS_PACKAGE to ENFORCE_LANGGENIUS_PLUGIN_SIGNATURES for clarity.
- Updated error messages and logic in the plugin installation process to reflect the new configuration.
- Enhanced tests to validate the behavior of unauthorized Langgenius plugin detection with the new enforcement setting.
* feat(datasource): Implement datasource validation and invocation steps
- Added new functionality for validating datasource credentials and invoking the first and second steps of the datasource process.
- Introduced new API endpoints for datasource operations in the HTTP server.
- Created corresponding service and controller methods to handle datasource requests.
- Defined new request and response structures for datasource operations in the entities package.
* feat: add routes
* feat(datasource): Add initial datasource models and validation
- Introduced new `DatasourceInstallation` model to represent datasource installations with relevant fields.
- Created `datasource.go` file for future datasource service implementation.
- Enhanced `datasource_declaration.go` with new types and validation functions for datasource provider and parameters.
- Updated `plugin_declaration.go` to include datasource support in plugin structures.
- Refactored `agent_declaration.go` and `tool_declaration.go` to use a unified `ParameterOption` type for options.
* refactor(datasource): Update request and response types for online document content retrieval
- Renamed and updated request and response types for the `DatasourceGetOnlineDocumentPageContent` function to improve clarity and consistency.
- Introduced `RequestInvokeOnlineDocumentDatasourceGetContent` and `DatasourceInvokeOnlineDocumentGetContentResponse` types.
- Adjusted related function signatures and dispatchers to reflect the new types across the datasource service and controller implementations.
* feat(datasource): Implement datasource installation handling
- Added functionality to create and update `DatasourceInstallation` records during plugin installation and upgrade processes.
- Enhanced the `InstallPlugin` function to create a new datasource installation if a datasource declaration is present.
- Updated the `UpgradePlugin` function to handle the deletion of the original datasource installation and creation of a new one if the datasource declaration changes.
* feat(datasource): Add datasource registration handling in plugin runtime
- Introduced handling for datasource declarations in the plugin runtime.
- Updated `RemotePluginRuntime` to track datasource registration status.
- Enhanced message processing to include validation and assignment of datasource declarations.
* feat(datasource): Implement datasource listing and retrieval endpoints
- Added `ListDatasources` and `GetDatasource` functions to the service layer for handling datasource queries.
- Created corresponding controller methods to process HTTP requests for listing and retrieving datasources.
- Implemented request validation for tenant ID, page, page size, plugin ID, and provider parameters.
* add datasource routes
* feat(datasource): Add icon remapping for datasource declarations
- Implemented functionality to remap icons for both the main datasource and its individual datasources within the plugin declaration.
- Enhanced error handling to provide clearer feedback when remapping fails.
* fix(datasource): update OAuthSchema validation to remove unnecessary 'dive' tag
- Modified the validation tag for OAuthSchema in DatasourceProviderDeclaration to simplify the validation process by removing the 'dive' requirement.
* feat(tests): add datasource declaration parsing and output
- Introduced a new main.go file for testing datasource declaration parsing.
- Implemented JSON unmarshalling for RemotePluginRegisterPayload and DatasourceProviderDeclaration.
- Added a sample datasource declaration for testing purposes.
* fix(plugin): improve error handling in UninstallPlugin and add datasource deletion
- Enhanced error handling to return a specific message when a plugin is not installed.
- Added functionality to delete the associated datasource installation during the uninstallation process.
* feat(datasource): add support for decoding datasource provider declaration
- Added custom JSON marshalling and unmarshalling methods to handle CredentialsSchema and Datasources more effectively.
- Improved error handling during YAML unmarshalling to support both object and array formats for CredentialsSchema.
- Ensured proper initialization of DatasourceFiles and Tags to prevent nil references.
* fix: provider type
* refactor(datasource): simplify JSON unmarshalling for CredentialsSchema
- Removed complex handling of CredentialsSchema in the UnmarshalJSON method, focusing on the Datasources field.
- Streamlined the code to improve readability and maintainability by eliminating unnecessary checks and logic related to CredentialsSchema.
* feat: streaming datasource
* feat: datasource
* feat: datasource
* feat:datasource
* feat:datasource
* feat:datasource
* feat: add redirect_uri field to OAuth request structs
* feat: add online driver file request and response structures
* feat: add online driver file request and response structures
* feat: add online_driver datasource type to validation
* feat: rename online driver to online drive and update related classes and methods :)
* refactor: rename OnlineDocumentPageChunk to DatasourceGetPagesResponse and update related references
* feat: update request types for online drive browsing and downloading
* feat: add metadata field to OAuthGetCredentialsResult
* feat: built-in json schema definations
* feat(plugin_entities): add built-in schema definitions and processing for datasource YAML
* test(plugin_entities): add unit tests for schema definitions and YAML processing
* refactor(plugin_entities): centralize built-in schema definitions and processing
* refactor(plugin_entities): remove unused properties from built-in schema definitions
* refactor(plugin_entities): built-in schema & new datasource structure
* refactor(plugin_entities): enhance schema processing with checks and error handling
* refactor(plugin_entities): update validation for OnlineDriveBrowseFilesRequest prefix field to be optional
* refactor(json_schema): remove json schema definitions and validation
* fix(plugin_entities): remove output_schema validation tests
* feat: add rag tag
---------
Co-authored-by: Harry <xh001x@hotmail.com>
Co-authored-by: Dongyu Li <544104925@qq.com>
Co-authored-by: Novice <novice12185727@gmail.com>
- Enhanced the RemapAssets function to include remapping of trigger identity icons and dark icons.
- Added error handling for remapping failures to ensure robust asset management.
These changes improve the handling of trigger assets within the plugin system, ensuring icons are correctly remapped as needed.
- Introduced TriggerInstallation model for managing trigger installations in the database.
- Updated autoMigrate function to include trigger installations in the migration process.
- Added new HTTP routes for listing and retrieving triggers in the HTTP server.
- Implemented ListTriggers and GetTrigger controller functions for handling trigger requests.
- Enhanced plugin management functions to create, update, and delete trigger installations during plugin lifecycle events.
These changes integrate trigger capabilities into the system, improving the overall plugin functionality and management.
- Added new trigger-related access types and actions in access.go.
- Introduced new HTTP routes for trigger operations in http_server.gen.go.
- Updated plugin declaration to include triggers in plugin_entities.
- Removed unused dynamic select service and controller files.
- Enhanced message handling in debugging_runtime to support trigger registration.
This update enhances the plugin system by integrating trigger capabilities while cleaning up legacy code.
- Updated AWS-related types and functions to reflect Serverless terminology, including renaming `AWSTransactionHandler` to `ServerlessTransactionHandler` and `AWSTransactionWriter` to `ServerlessTransactionWriter`.
- Adjusted error messages and comments to align with the new naming conventions.
- Modified installation functions to use Serverless terminology for clarity and consistency across the codebase.
- Modified GetSession to return an error alongside the session, improving error handling when a session is not found.
- Updated invocation of GetSession in AWS event handler to handle the new error return value.
- Removed logging statements in favor of returning errors directly for better error propagation.
* feat(oauth): implement refresh credentials functionality
- Added RefreshCredentials endpoint to handle OAuth credential refresh requests.
- Introduced RequestOAuthRefreshCredentials structure for request validation.
- Updated access types and actions to include refresh credentials.
- Enhanced server routing and controller logic to support the new functionality.
- Updated OAuth entities to include expiration handling for refreshed credentials.
* feat(oauth): add metadata field to OAuthGetCredentialsResult
---------
Co-authored-by: Harry <xh001x@hotmail.com>
* feat(oauth): add RedirectURI field to OAuth request structures
* feat(oauth): update OAuthSchema validation
* feat: add Context field to request and session structures
* feat: add CredentialType field to Credentials and InvokeToolRequest structures
* fix: handle unhandled default case in basic_type.go
* feat: add support for build branches in build-push.yml
* refactor(plugin_manager): enhance HTTP client timeout handling in serverless runtime with DialContext
- Updated the HTTP client in the ServerlessPluginRuntime to use a context-aware DialContext for better timeout management.
- Removed the write timeout option from the HTTP request builder, streamlining the request handling process.
- Improved connection handling by setting write deadlines based on the PluginMaxExecutionTimeout.
This change enhances the reliability of network operations within the serverless runtime environment.
* refactor(plugin_manager): remove write deadline setting in serverless runtime connection initialization
- Eliminated the write deadline setting from the connection initialization in the ServerlessPluginRuntime.
- This change simplifies the connection handling process and aligns with the recent enhancements to timeout management.
This update contributes to a more streamlined and efficient network operation within the serverless environment.
* refactor(plugin_manager): adjust HTTP client timeout settings in serverless runtime
- Modified the HTTP client configuration in the ServerlessPluginRuntime to set the TLS handshake timeout based on PluginMaxExecutionTimeout.
- Retained the IdleConnTimeout setting to ensure consistent connection management.
This update improves the timeout handling for secure connections, enhancing overall network reliability in the serverless environment.
- Refactored the RemapAssets function to streamline the remapping of icon fields for both models and tools, including support for dark mode icons.
- Introduced new fields IconSmallDark and IconLargeDark in the ModelProviderDeclaration and added IconDark in ToolProviderIdentity to accommodate dark mode assets.
- Improved error handling during the remapping process for better clarity and maintainability.
* feat(plugin_manager): optimize local plugin startup with concurrent control
- Add semaphore-based concurrency control for plugin launches
- Implement parallel plugin startup using goroutines
- Optimize error handling to prevent goroutine blocking
- Add concurrency metrics logging
Note: handleNewLocalPlugins now accepts config parameter with default concurrency limit
* feat(plugin_manager): make local plugin launching concurrency configurable
* fix(plugin_manager): optimize comment and error handling
- Updated comments to clarify the concurrent plugin launching configuration.
- Added a nil check for the error channel during plugin startup to improve code robustness.
* refactor(plugin_manager): refactor plugin startup logic
- Remove the semaphore mechanism and switch to using routine.Submit for concurrency management
* fix(plugin_manager): Optimize plugin startup logs and concurrency control
- Added log output for maximum concurrency when starting local plugins
- Implemented a channel-based concurrency control mechanism to ensure limits are not exceeded
- Fixed closure variable capture issue to prevent incorrect plugin information
- Improved error handling to avoid deadlocks during startup
* fix(plugin_manager): simplify error channel handling and semaphore release logic
---------
Co-authored-by: jim02.he <jim02.he@vipshop.com>
* feat: add support for dark icon
- Introduced IconDark field in PluginDeclaration and related structures to support dark mode icons.
- Updated the installation process to handle dark icons.
- Enhanced asset validation to check for the presence of dark icons.
This change improves the visual consistency of plugins in dark mode environments.
* enhance(cli/icon): add plugin icon support with multiple categories
- Added support for light and dark icons for various plugin categories including agent, datasource, extension, model, tool, and trigger.
- Replaced the previous single icon implementation with a structured map for better organization and retrieval of icons based on category and theme.
- Removed the old Python icon file to streamline asset management.
This update improves the visual representation of plugins across different themes, enhancing user experience.
* change icons
* fix
* fix
* comments
- Implemented middleware to track active requests and active dispatch requests using atomic counters.
- Updated health check response to include counts of active requests and active dispatch requests.
- Integrated the new middleware into the HTTP server and plugin dispatch group for improved monitoring.
* feat: add InvokeLLMWithStructuredOutput functionality
- Introduced a new method InvokeLLMWithStructuredOutput to the BackwardsInvocation interface for handling structured output requests.
- Added corresponding request and response types to support structured output.
- Implemented the method in both RealBackwardsInvocation and MockedDifyInvocation for testing purposes.
- Updated permission handling and task execution for the new structured output invocation type.
This enhancement allows for more flexible and detailed responses from the LLM, improving the overall functionality of the invocation system.
* refactor: enhance LLMResultChunkWithStructuredOutput structure
- Updated the LLMResultChunkWithStructuredOutput type to include additional fields: Model, SystemFingerprint, and Delta.
- Added comments to clarify the reasoning behind the structure and the use of type embedding for JSON marshaling.
This change improves the clarity and functionality of the LLMResultChunkWithStructuredOutput type, ensuring proper JSON serialization.
* refactor: streamline LLMResultChunk construction in InvokeLLMWithStructuredOutput
- Simplified the construction of LLMResultChunk and LLMResultChunkWithStructuredOutput by removing unnecessary type embedding.
- Enhanced readability and maintainability of the code while preserving functionality.
This change contributes to cleaner code and improved clarity in the handling of structured output responses.
* feat(dynamic_select): implement dynamic parameter fetching functionality
- Added FetchDynamicParameterOptions function to handle dynamic parameter selection.
- Introduced new access type and action for dynamic select in access_types.
- Updated HTTP server routes to include the new endpoint for fetching dynamic parameters.
- Created necessary service and controller files for dynamic select operations.
* refactor(access_types): rename dynamic select access type to dynamic parameter
- Updated access type constants to reflect the change from PLUGIN_ACCESS_TYPE_DYNAMIC_SELECT to PLUGIN_ACCESS_TYPE_DYNAMIC_PARAMETER.
- Adjusted related references in the PluginDispatchers and FetchDynamicParameterOptions function to maintain consistency.
* feat: add decode plugin from identifier endpoint
- Introduced a new endpoint to decode a plugin from a unique identifier.
- Implemented the DecodePluginFromIdentifier function to handle decoding and verification of plugin signatures.
- Updated the HTTP server routes to include the new decode endpoint.
* refactor: update decode plugin endpoint path
- Moved the decode plugin from identifier endpoint to a new path for better organization.
- Updated the HTTP server routes accordingly to reflect the new endpoint structure.
- Updated .env.example to include DB_EXTRAS and DB_CHARSET variables.
- Refactored InitPluginDB functions for PostgreSQL and MySQL to accept a configuration struct, allowing for more flexible database connection settings.
- Adjusted connection pool settings to utilize new configuration options for charset and extras.
This change improves the configurability of database connections and prepares the codebase for future enhancements.
Directly fetch and invoke the listener for a given session_id instead of iterating over the entire listener map.
Signed-off-by: guanz42 <guanz42@163.com>
* bump cloud-kit version to fix#343
* change env name USE_AWS_S3 to S3_USE_AWS
* update s3_use_aws default value to true
* update readme about the upgrade notice
Stream