diff --git a/README.md b/README.md index eaace18..141b836 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,127 @@ The following figure demonstrates the VPE architecture. ![VPE architecture](./figures/videoProcessingEngine_architecture_english.png) + +#### 各模块功能说明 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Level module Description of the function
Interface Video color space CAPI Provides interfaces for color space conversion in video scenes
Picture color space CAPI Provides APIs for color space conversion of image scenes
Video detail enhancement CAPI Provides APIs for video superresolution algorithms and sharpening algorithms
Picture detail enhancement CAPI Provides APIs related to image superresolution algorithm and sharpening algorithm
Video dynamic metadata CAPI APIs are provided for invoking dynamic metadata generation algorithms for video content
Image dynamic metadata CAPI APIs are provided for dynamic metadata generation algorithms for image content
The TS interface is enhanced for image details Provides TS interfaces for image super-resolution algorithm and sharpening algorithm
Atomically capable encapsulation layer Video color space atomic capabilities Realize the channel scheduling and context management of the video scene color space conversion software, and realize the process control of the video stream
Picture color space atomic capability Realize the color space conversion software channel scheduling of picture scenes
Video details enhance atomic capabilities Realize video scene clarity enhancement and scaling algorithm software channel scheduling and context management, and realize video stream process control
Picture details enhance atomic capabilities Realize the image scene clarity enhancement and scaling algorithm software channel scheduling
Dynamic metadata atomic capabilities for video Realize the dynamic metadata generation software channel scheduling of video scenes
Dynamic metadata atomic capabilities for images Realize the dynamic metadata generation software channel scheduling of image scenes
Algorithm plug-in layer Video color space processing algorithm plug-in Implement the function of video color space conversion algorithm, including SDR2SDR, HDR2SDR, and HDR2HDR
Plug-in for image color space algorithms Realize the function of image color space conversion algorithm, including SDR2SDR, single-layer to double-layer, double-layer to single-layer
Video Detail Enhancement Algorithm Plug-in Implement video scaling and image quality enhancement algorithms
Plug-in for image detail enhancement algorithms Implement image scaling and image quality enhancement algorithms
Video dynamic metadata algorithm plug-in Implement the video dynamic metadata production algorithm
Image dynamic metadata algorithm plug-in Implement the dynamic metadata generation algorithm for images
Plugin management Plugin registration Provide the function of system developer plug-in registration
Capability inquiry Application developers can use the capability query function to confirm whether a given plug-in is supported on that device or system
Plug-in calls Invoke specific plug-in capabilities to complete related algorithm functions
Service management Resource management Resource scheduling and algorithm context management, such as information about the frames before and after video content
Process management Complete functions such as cross-process communication
+ + +| Dependency modules | Description of the function | +| :-- | :-- | +| graphic_graphic_surface | Provide video surface support | +| graphic_graphic_2d | Provide image surfacebuffer support | +| multimedia_media_foundation | Pixelmap support is available | +| multimedia_image_framework | Format parameter setting is supported | +| third_party_skia | Provides a scaling algorithm | + ## Directory Structure The structure of the repository directory is as follows: @@ -20,6 +141,8 @@ The structure of the repository directory is as follows: │ ├── colorspace_converter # Image color space conversion algorithm framework │ ├── colorspace_converter_display # Image color space display algorithm framework │ ├── colorspace_converter_video # Video color space conversion algorithm framework +│ ├── common # Common +│ ├── contrast_enhancer # Contrast enhancement algorithm framework │ ├── detail_enhancer # Image detail enhancement algorithm framework │ ├── detail_enhancer_video # Video detail enhancement algorithm framework │ ├── extension_manager # Plugin management @@ -56,7 +179,7 @@ Run the following command to build the VPE for the 64-bit ARM system: ### How to Use As a component of OpenHarmony, the VPE provides video and image processing capabilities, including color space conversion, dynamic metadata generation, and detail enhancement. -#### Image Scale +#### The app developer calls the image scaling sample 1. Add a header file. ```cpp #include @@ -99,7 +222,7 @@ As a component of OpenHarmony, the VPE provides video and image processing capab ```cpp OH_ImageProcessing_DeinitializeEnvironment(); ``` -#### video zoom +#### The app developer calls the video zoom sample 1. Add a header file. ```cpp #include @@ -235,6 +358,150 @@ As a component of OpenHarmony, the VPE provides video and image processing capab {"ImageMetadataGeneratorExtensions", RegisterImageMetadataGeneratorExtensions} }; ``` + +#### The system developer implements a custom algorithm plug-in registration example +The video processing engine provides algorithm plug-in registration interfaces such as color space conversion, dynamic metadata generation, and detail enhancement, and system users can register their own algorithms to enrich algorithm plug-ins. The video processing engine defines the basic classes of various algorithms, as follows: +| Function | Base class | Header file | +| :-- | :-- | :-- | +| Video color space conversion | ColorSpaceConverterBase | colorspace_converter_base.h | +| Picture color space conversion | ColorSpaceConverterBase | colorspace_converter_base.h | +| Video detail enhancement | DetailEnhancerBase | detail_enhancer_base.h | +| Picture detail enhancement | DetailEnhancerBase | detail_enhancer_base.h | +| Video dynamic metadata | MetadataGeneratorBase | metadata_generator_base.h | +| Image dynamic metadata | MetadataGeneratorBase | metadata_generator_base.h | + +
+Take the scaling algorithm as an example, the plug-in algorithm has been implemented in framework/algorithm/extensions/skia, and the specific development steps are described below + +1. Create a new folder in the framework/algorithm/extensions directory, for example, skia, along with the corresponding cpp and h files + +2. The scaling algorithm header file is implemented, see framework/algorithm/extensions/skia/include/skia_impl.h + +```cpp +#ifndef SKIA_IMPL_H +#define SKIA_IMPL_H + +#include "surface_buffer.h" +#include "include/core/SkYUVAPixmaps.h" + +#include "algorithm_errors.h" +#include "detail_enhancer_base.h" +#include "detail_enhancer_capability.h" + +namespace OHOS { +namespace Media { +namespace VideoProcessingEngine { + +class Skia : public DetailEnhancerBase { +public: + Skia() = default; + virtual ~Skia() = default; + Skia(const Skia&) = delete; + Skia& operator=(const Skia&) = delete; + Skia(Skia&&) = delete; + Skia& operator=(Skia&&) = delete; + static std::unique_ptr Create(); + static DetailEnhancerCapability BuildCapabilities(); + VPEAlgoErrCode Init() override; + VPEAlgoErrCode Deinit() override; + VPEAlgoErrCode SetParameter(const DetailEnhancerParameters& parameter, int type, bool flag) override; + VPEAlgoErrCode Process(const sptr& input, const sptr& output) override; +}; +void RegisterSkiaExtensions(uintptr_t extensionListAddr); +} // VideoProcessingEngine +} // Media +} // OHOS +#endif // SKIA_IMPL_H +``` +3. For details on plug-in algorithm capability registration and plug-in algorithm implementation, see framework/algorithm/extensions/skia/skia_impl.cpp + +```cpp +#include "skia_impl.h" + +#include +#include +#include +#include "detail_enhancer_extension.h" +#include "utils.h" + +#include "vpe_log.h" + +namespace OHOS { +namespace Media { +namespace VideoProcessingEngine { + +namespace { + +constexpr Extension::Rank RANK = Extension::Rank::RANK_DEFAULT; +constexpr uint32_t VERSION = 0; +} // namespace + +std::unique_ptr Skia::Create() +{ + return std::make_unique(); +} + + +DetailEnhancerCapability Skia::BuildCapabilities() +{ + std::vector levels = { DETAIL_ENH_LEVEL_NONE, DETAIL_ENH_LEVEL_LOW, DETAIL_ENH_LEVEL_MEDIUM, + DETAIL_ENH_LEVEL_HIGH_EVE, DETAIL_ENH_LEVEL_HIGH_AISR, DETAIL_ENH_LEVEL_VIDEO}; + DetailEnhancerCapability capability = { levels, RANK, VERSION }; + return capability; +} + +VPEAlgoErrCode Skia::Init() +{ + return VPE_ALGO_ERR_OK; +} + +VPEAlgoErrCode Skia::Deinit() +{ + return VPE_ALGO_ERR_OK; +} + +VPEAlgoErrCode Skia::SetParameter([[maybe_unused]] const DetailEnhancerParameters& parameter, + [[maybe_unused]] int type, [[maybe_unused]] bool flag) +{ + return VPE_ALGO_ERR_OK; +} + +VPEAlgoErrCode Skia::Process(const sptr& input, const sptr& output) +{ + return errCode; +} + +static std::vector> RegisterExtensions() +{ + std::vector> extensions; + + auto extension = std::make_shared(); + CHECK_AND_RETURN_RET_LOG(extension != nullptr, extensions, "null pointer"); + extension->info = { Extension::ExtensionType::DETAIL_ENHANCER, "SKIA", "0.0.1" }; + extension->creator = Skia::Create; + extension->capabilitiesBuilder = Skia::BuildCapabilities; + extensions.push_back(std::static_pointer_cast(extension)); + + return extensions; +} + +void RegisterSkiaExtensions(uintptr_t extensionListAddr) +{ + Extension::DoRegisterExtensions(extensionListAddr, RegisterExtensions); +} +} // VideoProcessingEngine +} // Media +} // OHOS +``` + +4. Add a callback function for registering a plug-in, and the VPE plug-in management will iterate through all plug-in registration functions. + see framework/algorithm/extension_manager/include/static_extension_list.h +```cpp +const std::unordered_map staticExtensionsRegisterMap = { + { "Skia", RegisterSkiaExtensions }, +}; +``` + ## Repositories Involved - [graphic_graphic_2d](https://gitee.com/openharmony/graphic_graphic_2d) diff --git a/README_zh.md b/README_zh.md index 47f9ef6..6285bfb 100644 --- a/README_zh.md +++ b/README_zh.md @@ -41,7 +41,7 @@ VPE引擎的主要结构如下图所示： 提供接口可用于图片内容动态元数据生成算法调用 - 图片细节增强TS 接口 + 图片细节增强TS接口 提供图片超分算法、锐化算法的TS接口