- Updated the .dify_cli.json configuration to include cli_api_url, cli_api_session_id, and cli_api_secret for improved API interaction.
- Refactored the callDifyAPI and fetchProviders functions to utilize the new CLI API structure, including HMAC signature generation for enhanced security.
- Modified EnvConfig to reflect the new CLI API parameters, ensuring required fields are validated.
This update enhances the security and functionality of the CLI by transitioning to a new API structure and improving request handling.
* remove enterprise logics
* release lock if runtime already installed
* support setting serverless endpoint by api
* remove global tenant id totally
* scan timeout tasks
* adding comments
* use index in loop
* add log
* feat(plugins): add FetchPluginReadme endpoint and update launch configurations
* feat: add PluginReadme database model
* feat: implement readme extracting and storage
* feat: implement readme endpoint
* feat: add plugin asset extraction endpoint with caching support
* Implement trigger functionality and clean up dynamic select code
- Added new trigger-related access types and actions in access.go.
- Introduced new HTTP routes for trigger operations in http_server.gen.go.
- Updated plugin declaration to include triggers in plugin_entities.
- Removed unused dynamic select service and controller files.
- Enhanced message handling in debugging_runtime to support trigger registration.
This update enhances the plugin system by integrating trigger capabilities while cleaning up legacy code.
* Refactor trigger-related types and enhance dynamic select functionality
- Updated TriggerProviderIdentity and TriggerProviderConfiguration to improve structure and validation.
- Renamed TriggerConfiguration to TriggerDeclaration for clarity.
- Added CredentialType to RequestDynamicParameterSelect for better request handling.
- Enhanced PluginDecoderHelper to read and unmarshal trigger files correctly.
These changes streamline the trigger system and improve the overall request handling in the plugin architecture.
* Add trigger functionality and enhance database integration
- Introduced TriggerInstallation model for managing trigger installations in the database.
- Updated autoMigrate function to include trigger installations in the migration process.
- Added new HTTP routes for listing and retrieving triggers in the HTTP server.
- Implemented ListTriggers and GetTrigger controller functions for handling trigger requests.
- Enhanced plugin management functions to create, update, and delete trigger installations during plugin lifecycle events.
These changes integrate trigger capabilities into the system, improving the overall plugin functionality and management.
* feat: add remapping for trigger icons in MediaBucket
- Enhanced the RemapAssets function to include remapping of trigger identity icons and dark icons.
- Added error handling for remapping failures to ensure robust asset management.
These changes improve the handling of trigger assets within the plugin system, ensuring icons are correctly remapped as needed.
* feat: add Multiple field to TriggerParameter for enhanced configuration
- Introduced a new Multiple field in the TriggerParameter struct to allow for multiple values in trigger configurations.
- This addition improves the flexibility of trigger parameters within the plugin system.
These changes enhance the capability of trigger parameters, enabling more complex configurations.
* feat: add Multiple field to ProviderConfig for enhanced configuration
- Introduced a new Multiple field in the ProviderConfig struct to allow for multiple values in provider configurations.
- This addition improves the flexibility of provider options within the plugin system.
These changes enhance the capability of provider configurations, enabling more complex setups.
* fix(plugin): update validation error messages in ManifestValidate method
- Enhanced error messages in the ManifestValidate function to include 'trigger' in the validation checks for plugin declarations.
- Updated logic to ensure that all relevant fields are considered when validating the presence of mutually exclusive parameters.
* feat(trigger): add CHECKBOX parameter type to plugin entities and refactor the trigger provider strcuture
- Introduced a new CHECKBOX parameter type in constant.go for plugin entities.
- Updated tool_declaration.go and trigger_declaration.go to include TOOL_PARAMETER_TYPE_CHECKBOX and TRIGGER_PARAMETER_TYPE_CHECKBOX respectively.
- Enhanced validation logic to accommodate the new CHECKBOX type in parameter checks.
* fix(trigger): update SubscriptionSchema validation in TriggerProviderDeclaration
- Changed SubscriptionSchema validation from 'omitempty' to 'required' in TriggerProviderDeclaration to ensure it is always provided.
- Updated SubscriptionConstructor field to be a pointer to allow for optional inclusion in the trigger provider configuration.
* fix(trigger): rename ParametersSchema to Parameters in SubscriptionConstructor
- Updated the SubscriptionConstructor struct to rename the ParametersSchema field to Parameters for consistency.
- Adjusted related JSON and YAML marshaling logic to reflect the new field name, ensuring proper handling of trigger parameters.
* refactor(trigger): enhance YAML unmarshalling for SubscriptionConstructor and SubscriptionSchema
- Introduced a new helper function to convert YAML nodes to ProviderConfig lists, improving the handling of subscription_schema and credentials_schema.
- Updated the UnmarshalYAML method to utilize the new function, simplifying the logic for parsing different YAML formats.
- Ensured proper initialization of SubscriptionConstructor fields to prevent nil pointer dereferences.
* fix(trigger): update SubscriptionConstructor validation in TriggerProviderDeclaration
- Changed the validation for SubscriptionConstructor in TriggerProviderDeclaration from 'omitempty,dive' to 'omitempty' to simplify the validation logic.
- Ensured that the SubscriptionConstructor field remains optional while maintaining its intended functionality.
* refactor(trigger): rename Trigger to Event in plugin entities and related structures
- Updated the naming conventions in trigger_declaration.go to replace 'Trigger' with 'Event' for better clarity and consistency.
- Adjusted related types, validation functions, and unmarshalling logic to reflect the new 'Event' terminology.
- Ensured that all references to triggers in the codebase are updated to events, including in the SubscriptionConstructor and response structures.
* refactor(trigger): rename TriggerInvoke to TriggerInvokeEvent and update related structures
- Renamed TriggerInvoke function and associated request/response types to TriggerInvokeEvent for improved clarity.
- Updated routing and controller methods to reflect the new naming convention.
- Ensured all references to the trigger invoke functionality are consistent with the new event terminology.
* refactor(trigger): remove Subscription struct from trigger_declaration.go and update TriggerDispatchEventRequest
- Removed the Subscription struct from trigger_declaration.go to streamline the codebase.
- Added Credentials field to TriggerDispatchEventRequest for enhanced functionality and clarity.
- Ensured that the changes maintain consistency with the existing naming conventions and structures.
* fix(trigger): improve nil checks for SubscriptionConstructor in TriggerProviderDeclaration
- Added nil checks for SubscriptionConstructor before accessing its fields to prevent potential nil pointer dereferences.
- Ensured that Parameters and CredentialsSchema are initialized only if SubscriptionConstructor is not nil, enhancing code robustness.
* fix(plugin): add recovery mechanism in OnTraffic to handle panics
- Introduced a deferred function in OnTraffic to recover from panics, logging the error and stack trace for better debugging.
- This enhancement improves the stability of the DifyServer by preventing crashes due to unexpected runtime errors.
* feat(trigger): add Subscription field to TriggerInvokeEventRequest
- Introduced a new Subscription field in the TriggerInvokeEventRequest struct to accommodate subscription data.
- Ensured the field is marked as required, enhancing the request's functionality and validation requirements.
* refactor(event): simplify EventDescription structure in EventDeclaration
- Removed the EventDescription struct and replaced it with a direct I18nObject field in EventDeclaration.
- This change streamlines the event configuration by reducing complexity while maintaining required validation for the description.
* feat(trigger): add UserID field to TriggerDispatchEventResponse
- Introduced a new UserID field in the TriggerDispatchEventResponse struct to include user identification in the response.
- The field is marked as optional, enhancing the response's flexibility while maintaining existing functionality.
* feat: add payload to TriggerDispatchEventResponse
* fix
* feat(trigger): update TriggerDispatchEventResponse structure
* fix: avoid path collusion
* fix: missing )
* fix: query param
* fix: form param
* fix: remove redundant dynamic parameter access type
* fix: remove dynamic parameter access type from validation
* Update internal/server/controllers/plugins.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Stream <Stream_2@qq.com>
Co-authored-by: Yeuoly <admin@srmxy.cn>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* feat(datasource): Implement datasource validation and invocation steps
- Added new functionality for validating datasource credentials and invoking the first and second steps of the datasource process.
- Introduced new API endpoints for datasource operations in the HTTP server.
- Created corresponding service and controller methods to handle datasource requests.
- Defined new request and response structures for datasource operations in the entities package.
* feat: add routes
* feat(datasource): Add initial datasource models and validation
- Introduced new `DatasourceInstallation` model to represent datasource installations with relevant fields.
- Created `datasource.go` file for future datasource service implementation.
- Enhanced `datasource_declaration.go` with new types and validation functions for datasource provider and parameters.
- Updated `plugin_declaration.go` to include datasource support in plugin structures.
- Refactored `agent_declaration.go` and `tool_declaration.go` to use a unified `ParameterOption` type for options.
* refactor(datasource): Update request and response types for online document content retrieval
- Renamed and updated request and response types for the `DatasourceGetOnlineDocumentPageContent` function to improve clarity and consistency.
- Introduced `RequestInvokeOnlineDocumentDatasourceGetContent` and `DatasourceInvokeOnlineDocumentGetContentResponse` types.
- Adjusted related function signatures and dispatchers to reflect the new types across the datasource service and controller implementations.
* feat(datasource): Implement datasource installation handling
- Added functionality to create and update `DatasourceInstallation` records during plugin installation and upgrade processes.
- Enhanced the `InstallPlugin` function to create a new datasource installation if a datasource declaration is present.
- Updated the `UpgradePlugin` function to handle the deletion of the original datasource installation and creation of a new one if the datasource declaration changes.
* feat(datasource): Add datasource registration handling in plugin runtime
- Introduced handling for datasource declarations in the plugin runtime.
- Updated `RemotePluginRuntime` to track datasource registration status.
- Enhanced message processing to include validation and assignment of datasource declarations.
* feat(datasource): Implement datasource listing and retrieval endpoints
- Added `ListDatasources` and `GetDatasource` functions to the service layer for handling datasource queries.
- Created corresponding controller methods to process HTTP requests for listing and retrieving datasources.
- Implemented request validation for tenant ID, page, page size, plugin ID, and provider parameters.
* add datasource routes
* feat(datasource): Add icon remapping for datasource declarations
- Implemented functionality to remap icons for both the main datasource and its individual datasources within the plugin declaration.
- Enhanced error handling to provide clearer feedback when remapping fails.
* fix(datasource): update OAuthSchema validation to remove unnecessary 'dive' tag
- Modified the validation tag for OAuthSchema in DatasourceProviderDeclaration to simplify the validation process by removing the 'dive' requirement.
* feat(tests): add datasource declaration parsing and output
- Introduced a new main.go file for testing datasource declaration parsing.
- Implemented JSON unmarshalling for RemotePluginRegisterPayload and DatasourceProviderDeclaration.
- Added a sample datasource declaration for testing purposes.
* fix(plugin): improve error handling in UninstallPlugin and add datasource deletion
- Enhanced error handling to return a specific message when a plugin is not installed.
- Added functionality to delete the associated datasource installation during the uninstallation process.
* feat(datasource): add support for decoding datasource provider declaration
- Added custom JSON marshalling and unmarshalling methods to handle CredentialsSchema and Datasources more effectively.
- Improved error handling during YAML unmarshalling to support both object and array formats for CredentialsSchema.
- Ensured proper initialization of DatasourceFiles and Tags to prevent nil references.
* fix: provider type
* refactor(datasource): simplify JSON unmarshalling for CredentialsSchema
- Removed complex handling of CredentialsSchema in the UnmarshalJSON method, focusing on the Datasources field.
- Streamlined the code to improve readability and maintainability by eliminating unnecessary checks and logic related to CredentialsSchema.
* feat: streaming datasource
* feat: datasource
* feat: datasource
* feat:datasource
* feat:datasource
* feat:datasource
* feat: add redirect_uri field to OAuth request structs
* feat: add online driver file request and response structures
* feat: add online driver file request and response structures
* feat: add online_driver datasource type to validation
* feat: rename online driver to online drive and update related classes and methods :)
* refactor: rename OnlineDocumentPageChunk to DatasourceGetPagesResponse and update related references
* feat: update request types for online drive browsing and downloading
* feat: add metadata field to OAuthGetCredentialsResult
* feat: built-in json schema definations
* feat(plugin_entities): add built-in schema definitions and processing for datasource YAML
* test(plugin_entities): add unit tests for schema definitions and YAML processing
* refactor(plugin_entities): centralize built-in schema definitions and processing
* refactor(plugin_entities): remove unused properties from built-in schema definitions
* refactor(plugin_entities): built-in schema & new datasource structure
* refactor(plugin_entities): enhance schema processing with checks and error handling
* refactor(plugin_entities): update validation for OnlineDriveBrowseFilesRequest prefix field to be optional
* refactor(json_schema): remove json schema definitions and validation
* fix(plugin_entities): remove output_schema validation tests
* feat: add rag tag
---------
Co-authored-by: Harry <xh001x@hotmail.com>
Co-authored-by: Dongyu Li <544104925@qq.com>
Co-authored-by: Novice <novice12185727@gmail.com>
- Updated github.com/go-git/go-git from v4.7.0 to v5.16.2.
- Bumped golang.org/x/tools from v0.22.0 to v0.35.0.
- Added indirect dependencies for github.com/go-git/gcfg and github.com/go-git/go-billy.
- Updated various indirect dependencies including golang.org/x/net, golang.org/x/sys, and golang.org/x/text.
* feat(oauth): implement refresh credentials functionality
- Added RefreshCredentials endpoint to handle OAuth credential refresh requests.
- Introduced RequestOAuthRefreshCredentials structure for request validation.
- Updated access types and actions to include refresh credentials.
- Enhanced server routing and controller logic to support the new functionality.
- Updated OAuth entities to include expiration handling for refreshed credentials.
* feat(oauth): add metadata field to OAuthGetCredentialsResult
---------
Co-authored-by: Harry <xh001x@hotmail.com>
* feat(plugin_decoder): add support for internationalized readme files
- Introduced the AvailableI18nReadme method in the PluginDecoder interface to retrieve available readme files in multiple languages.
- Implemented the method in FSPluginDecoder and ZipPluginDecoder to read localized readme files from the filesystem and zip archives.
- Enhanced UnixPluginDecoder to handle readme files in a structured manner, including support for reading from a dedicated "readme" directory.
- Added unit tests to verify the functionality of the AvailableI18nReadme method and ensure correct retrieval of localized readme content.
* feat(plugin): add support for multilingual README generation
- Introduced functionality to create README files in multiple languages (Simplified Chinese, Japanese, Portuguese) based on user selection.
- Enhanced the profile management to include options for enabling internationalized README and selecting languages.
- Added new language choice structure to manage language options and their selection state.
- Implemented rendering and writing of language-specific README files during plugin creation.
- Included new README template files for each supported language.
* feat(plugin): add README command and list functionality
- Introduced a new `readme` command to the plugin CLI for managing README files.
- Added `list` subcommand to display available README languages for a specified plugin path.
- Implemented functionality to read and list supported README languages in a tabular format.
- Enhanced error handling for plugin file reading and decoding processes.
* feat(oauth): add RedirectURI field to OAuth request structures
* feat(oauth): update OAuthSchema validation
* feat: add Context field to request and session structures
* feat: add CredentialType field to Credentials and InvokeToolRequest structures
* fix: handle unhandled default case in basic_type.go
* feat: add support for build branches in build-push.yml
- Refactored the RemapAssets function to streamline the remapping of icon fields for both models and tools, including support for dark mode icons.
- Introduced new fields IconSmallDark and IconLargeDark in the ModelProviderDeclaration and added IconDark in ToolProviderIdentity to accommodate dark mode assets.
- Improved error handling during the remapping process for better clarity and maintainability.
* feat: add support for dark icon
- Introduced IconDark field in PluginDeclaration and related structures to support dark mode icons.
- Updated the installation process to handle dark icons.
- Enhanced asset validation to check for the presence of dark icons.
This change improves the visual consistency of plugins in dark mode environments.
* enhance(cli/icon): add plugin icon support with multiple categories
- Added support for light and dark icons for various plugin categories including agent, datasource, extension, model, tool, and trigger.
- Replaced the previous single icon implementation with a structured map for better organization and retrieval of icons based on category and theme.
- Removed the old Python icon file to streamline asset management.
This update improves the visual representation of plugins across different themes, enhancing user experience.
* change icons
* fix
* fix
* comments
* feat: add InvokeLLMWithStructuredOutput functionality
- Introduced a new method InvokeLLMWithStructuredOutput to the BackwardsInvocation interface for handling structured output requests.
- Added corresponding request and response types to support structured output.
- Implemented the method in both RealBackwardsInvocation and MockedDifyInvocation for testing purposes.
- Updated permission handling and task execution for the new structured output invocation type.
This enhancement allows for more flexible and detailed responses from the LLM, improving the overall functionality of the invocation system.
* refactor: enhance LLMResultChunkWithStructuredOutput structure
- Updated the LLMResultChunkWithStructuredOutput type to include additional fields: Model, SystemFingerprint, and Delta.
- Added comments to clarify the reasoning behind the structure and the use of type embedding for JSON marshaling.
This change improves the clarity and functionality of the LLMResultChunkWithStructuredOutput type, ensuring proper JSON serialization.
* refactor: streamline LLMResultChunk construction in InvokeLLMWithStructuredOutput
- Simplified the construction of LLMResultChunk and LLMResultChunkWithStructuredOutput by removing unnecessary type embedding.
- Enhanced readability and maintainability of the code while preserving functionality.
This change contributes to cleaner code and improved clarity in the handling of structured output responses.
* feat(dynamic_select): implement dynamic parameter fetching functionality
- Added FetchDynamicParameterOptions function to handle dynamic parameter selection.
- Introduced new access type and action for dynamic select in access_types.
- Updated HTTP server routes to include the new endpoint for fetching dynamic parameters.
- Created necessary service and controller files for dynamic select operations.
* refactor(access_types): rename dynamic select access type to dynamic parameter
- Updated access type constants to reflect the change from PLUGIN_ACCESS_TYPE_DYNAMIC_SELECT to PLUGIN_ACCESS_TYPE_DYNAMIC_PARAMETER.
- Adjusted related references in the PluginDispatchers and FetchDynamicParameterOptions function to maintain consistency.
- Updated the method in the interface to remove the parameter, simplifying its usage.
- Introduced a new function to provide a default verification structure.
- Added a file to store verification data, improving the plugin signing process.
- Enhanced tests in to validate the verification process, ensuring proper handling of success and failure scenarios.
- Refactored related code to accommodate the new verification structure and improve overall maintainability.
* feat: Enhance plugin signing with authorized category verification
- Added support for an `authorized_category` flag in the signature command to validate the category before signing.
- Updated the `Sign` function to accept a verification parameter, allowing for category-based signing.
- Enhanced error handling for invalid categories during the signing process.
- Updated tests to cover new verification scenarios and ensure proper functionality with the authorized category.
* fix
* fix
* test
* test: Add unit test for plugin verification without verification field
- Introduced a new test case to verify the behavior of plugins that lack a verification field.
- Updated the signature_test.go file to include the test, ensuring proper functionality of the signing process.
- Removed the outdated verifier_test.go file and associated test data to streamline the codebase.
* feat: add repository URL support for plugin manifest
- Introduced a new optional flag for specifying the plugin repository URL during initialization.
- Updated the InitPluginWithFlags function to handle the new repository parameter.
- Enhanced profile management to include repository input.
- Modified related tests to validate the new repository functionality.
* fix: improve input validation in profile checkRule method
- Updated the checkRule method to ensure cursor is within valid range before checking for empty input values. This change prevents potential out-of-bounds errors and enhances the robustness of the input validation process.
* feat: implement OAuth functionality in plugin daemon
- Added OAuth service methods for getting authorization URLs and credentials.
- Updated access types to include OAuth-related actions.
- Created new controller for handling OAuth requests.
- Introduced entities for OAuth results and requests.
- Enhanced plugin entities to support OAuth schema in tool declarations.
* feat: add OAuth endpoints for authorization and credentials retrieval
- Introduced new POST endpoints for obtaining authorization URLs and credentials in the OAuth controller.
- Enhanced the plugin dispatch group to include these new OAuth routes, improving integration with OAuth services.
* refactor: clean up OAuth entity and request structures
- Removed unused validation logic and constants from the OAuth entity file.
- Added required provider field to OAuth request structures for better validation and clarity.
* feat: implement OAuth functionality in plugin daemon
- Added OAuth service methods for getting authorization URLs and credentials.
- Updated access types to include OAuth-related actions.
- Created new controller for handling OAuth requests.
- Introduced entities for OAuth results and requests.
- Enhanced plugin entities to support OAuth schema in tool declarations.
* feat: add OAuth endpoints for authorization and credentials retrieval
- Introduced new POST endpoints for obtaining authorization URLs and credentials in the OAuth controller.
- Enhanced the plugin dispatch group to include these new OAuth routes, improving integration with OAuth services.
- Implemented a compatibility layer in LLMResultChunk to ensure backward compatibility with the old format by adding a `PromptMessages` field.
- Added a new test case `TestLLMResultChunkCompatibility` to verify the JSON marshaling behavior of the updated LLMResultChunk structure.
* refactor(plugin_packager): update Assets method to use OS-independent path separator
- Modified the Assets method in FSPluginDecoder and PluginDecoderHelper to accept a separator argument, ensuring compatibility across different operating systems.
- Updated the Assets method in ZipPluginDecoder to use '/' as the separator for zip files, addressing issue #166.
- Improved code readability by adjusting formatting in the ZipPluginDecoder struct.
* fix: add tests
* feat(plugin): add version requirement functionality to plugin initialization
- Introduced a new `version_require` sub-menu to handle minimal Dify version input.
- Updated the plugin initialization process to include the new version requirement.
- Enhanced the `PluginMeta` struct to store the minimal Dify version.
- Added validation for the minimal Dify version input to ensure correctness.
* fix(plugin): rename MinimalDifyVersion to MinimumDifyVersion for consistency
- Updated the `PluginMeta` struct to change the field name from `MinimalDifyVersion` to `MinimumDifyVersion`.
- Adjusted the plugin initialization logic to reflect the new field name, ensuring proper handling of version requirements.
Harry