Files
dify-plugin-daemon/pkg/plugin_packager
Yeuoly 878edde455 feat(datasource): Implement datasource validation and invocation steps (#295)
* feat(datasource): Implement datasource validation and invocation steps

- Added new functionality for validating datasource credentials and invoking the first and second steps of the datasource process.
- Introduced new API endpoints for datasource operations in the HTTP server.
- Created corresponding service and controller methods to handle datasource requests.
- Defined new request and response structures for datasource operations in the entities package.

* feat: add routes

* feat(datasource): Add initial datasource models and validation

- Introduced new `DatasourceInstallation` model to represent datasource installations with relevant fields.
- Created `datasource.go` file for future datasource service implementation.
- Enhanced `datasource_declaration.go` with new types and validation functions for datasource provider and parameters.
- Updated `plugin_declaration.go` to include datasource support in plugin structures.
- Refactored `agent_declaration.go` and `tool_declaration.go` to use a unified `ParameterOption` type for options.

* refactor(datasource): Update request and response types for online document content retrieval

- Renamed and updated request and response types for the `DatasourceGetOnlineDocumentPageContent` function to improve clarity and consistency.
- Introduced `RequestInvokeOnlineDocumentDatasourceGetContent` and `DatasourceInvokeOnlineDocumentGetContentResponse` types.
- Adjusted related function signatures and dispatchers to reflect the new types across the datasource service and controller implementations.

* feat(datasource): Implement datasource installation handling

- Added functionality to create and update `DatasourceInstallation` records during plugin installation and upgrade processes.
- Enhanced the `InstallPlugin` function to create a new datasource installation if a datasource declaration is present.
- Updated the `UpgradePlugin` function to handle the deletion of the original datasource installation and creation of a new one if the datasource declaration changes.

* feat(datasource): Add datasource registration handling in plugin runtime

- Introduced handling for datasource declarations in the plugin runtime.
- Updated `RemotePluginRuntime` to track datasource registration status.
- Enhanced message processing to include validation and assignment of datasource declarations.

* feat(datasource): Implement datasource listing and retrieval endpoints

- Added `ListDatasources` and `GetDatasource` functions to the service layer for handling datasource queries.
- Created corresponding controller methods to process HTTP requests for listing and retrieving datasources.
- Implemented request validation for tenant ID, page, page size, plugin ID, and provider parameters.

* add datasource routes

* feat(datasource): Add icon remapping for datasource declarations

- Implemented functionality to remap icons for both the main datasource and its individual datasources within the plugin declaration.
- Enhanced error handling to provide clearer feedback when remapping fails.

* fix(datasource): update OAuthSchema validation to remove unnecessary 'dive' tag

- Modified the validation tag for OAuthSchema in DatasourceProviderDeclaration to simplify the validation process by removing the 'dive' requirement.

* feat(tests): add datasource declaration parsing and output

- Introduced a new main.go file for testing datasource declaration parsing.
- Implemented JSON unmarshalling for RemotePluginRegisterPayload and DatasourceProviderDeclaration.
- Added a sample datasource declaration for testing purposes.

* fix(plugin): improve error handling in UninstallPlugin and add datasource deletion

- Enhanced error handling to return a specific message when a plugin is not installed.
- Added functionality to delete the associated datasource installation during the uninstallation process.

* feat(datasource): add support for decoding datasource provider declaration

- Added custom JSON marshalling and unmarshalling methods to handle CredentialsSchema and Datasources more effectively.
- Improved error handling during YAML unmarshalling to support both object and array formats for CredentialsSchema.
- Ensured proper initialization of DatasourceFiles and Tags to prevent nil references.

* fix: provider type

* refactor(datasource): simplify JSON unmarshalling for CredentialsSchema

- Removed complex handling of CredentialsSchema in the UnmarshalJSON method, focusing on the Datasources field.
- Streamlined the code to improve readability and maintainability by eliminating unnecessary checks and logic related to CredentialsSchema.

* feat: streaming datasource

* feat: datasource

* feat: datasource

* feat:datasource

* feat:datasource

* feat:datasource

* feat: add redirect_uri field to OAuth request structs

* feat: add online driver file request and response structures

* feat: add online driver file request and response structures

* feat: add online_driver datasource type to validation

* feat: rename online driver to online drive and update related classes and methods :)

* refactor: rename OnlineDocumentPageChunk to DatasourceGetPagesResponse and update related references

* feat: update request types for online drive browsing and downloading

* feat: add metadata field to OAuthGetCredentialsResult

* feat: built-in json schema definations

* feat(plugin_entities): add built-in schema definitions and processing for datasource YAML

* test(plugin_entities): add unit tests for schema definitions and YAML processing

* refactor(plugin_entities): centralize built-in schema definitions and processing

* refactor(plugin_entities): remove unused properties from built-in schema definitions

* refactor(plugin_entities): built-in schema & new datasource structure

* refactor(plugin_entities): enhance schema processing with  checks and error handling

* refactor(plugin_entities): update validation for OnlineDriveBrowseFilesRequest prefix field to be optional

* refactor(json_schema): remove json schema definitions and validation

* fix(plugin_entities): remove output_schema validation tests

* feat: add rag tag

---------

Co-authored-by: Harry <xh001x@hotmail.com>
Co-authored-by: Dongyu Li <544104925@qq.com>
Co-authored-by: Novice <novice12185727@gmail.com>
2025-09-16 17:32:14 +08:00
..