VPE(multimedia_video_processing_engine)

Overview

The Video Processing Engine (VPE) is a media engine for processing video and image data. It offers a range of fundamental capabilities including enhancements to details, contrast, luminance, and dynamic ranges. It also supports essential algorithms for color space conversion, scaling and upscaling, and dynamic metadata generation for transcoding, sharing, and post-processing for display.

The following figure demonstrates the VPE architecture.

各模块功能说明

Level	module	Description of the function
Interface	Video color space CAPI	Provides interfaces for color space conversion in video scenes
	Picture color space CAPI	Provides APIs for color space conversion of image scenes
	Video detail enhancement CAPI	Provides APIs for video superresolution algorithms and sharpening algorithms
	Picture detail enhancement CAPI	Provides APIs related to image superresolution algorithm and sharpening algorithm
	Video dynamic metadata CAPI	APIs are provided for invoking dynamic metadata generation algorithms for video content
	Image dynamic metadata CAPI	APIs are provided for dynamic metadata generation algorithms for image content
	The TS interface is enhanced for image details	Provides TS interfaces for image super-resolution algorithm and sharpening algorithm
Atomically capable encapsulation layer	Video color space atomic capabilities	Realize the channel scheduling and context management of the video scene color space conversion software, and realize the process control of the video stream
	Picture color space atomic capability	Realize the color space conversion software channel scheduling of picture scenes
	Video details enhance atomic capabilities	Realize video scene clarity enhancement and scaling algorithm software channel scheduling and context management, and realize video stream process control
	Picture details enhance atomic capabilities	Realize the image scene clarity enhancement and scaling algorithm software channel scheduling
	Dynamic metadata atomic capabilities for video	Realize the dynamic metadata generation software channel scheduling of video scenes
	Dynamic metadata atomic capabilities for images	Realize the dynamic metadata generation software channel scheduling of image scenes
Algorithm plug-in layer	Video color space processing algorithm plug-in	Implement the function of video color space conversion algorithm, including SDR2SDR, HDR2SDR, and HDR2HDR
	Plug-in for image color space algorithms	Realize the function of image color space conversion algorithm, including SDR2SDR, single-layer to double-layer, double-layer to single-layer
	Video Detail Enhancement Algorithm Plug-in	Implement video scaling and image quality enhancement algorithms
	Plug-in for image detail enhancement algorithms	Implement image scaling and image quality enhancement algorithms
	Video dynamic metadata algorithm plug-in	Implement the video dynamic metadata production algorithm
	Image dynamic metadata algorithm plug-in	Implement the dynamic metadata generation algorithm for images
Plugin management	Plugin registration	Provide the function of system developer plug-in registration
	Capability inquiry	Application developers can use the capability query function to confirm whether a given plug-in is supported on that device or system
	Plug-in calls	Invoke specific plug-in capabilities to complete related algorithm functions
Service management	Resource management	Resource scheduling and algorithm context management, such as information about the frames before and after video content
	Process management	Complete functions such as cross-process communication

Dependency modules	Description of the function
graphic_graphic_surface	Provide video surface support
graphic_graphic_2d	Provide image surfacebuffer support
multimedia_media_foundation	Pixelmap support is available
multimedia_image_framework	Format parameter setting is supported
third_party_skia	Provides a scaling algorithm

Directory Structure

The structure of the repository directory is as follows:

/foundation/multimedia/video_processing_engine/
├── framework                                  # Framework code
│   ├── algorithm                              # Algorithm framework
│       ├── aihdr_enhancer                     # Image HDR enhancement algorithm framework
│       ├── aihdr_enhancer_video               # Video HDR enhancement algorithm framework
│       ├── colorspace_converter               # Image color space conversion algorithm framework
│       ├── colorspace_converter_display       # Image color space display algorithm framework
│       ├── colorspace_converter_video         # Video color space conversion algorithm framework
│       ├── common                             # Common
│       ├── contrast_enhancer                  # Contrast enhancement algorithm framework
│       ├── detail_enhancer                    # Image detail enhancement algorithm framework
│       ├── detail_enhancer_video              # Video detail enhancement algorithm framework
│       ├── extension_manager                  # Plugin management
│       ├── metadata_generator                 # Image metadata generation algorithm framework
│       ├── metadata_generator_video           # Video metadata generation algorithm framework
│       ├── video_variable_refresh_rate        # Video variable frame rate algorithm framework
│   ├── capi                                   # CAPI layer
│       ├── image_processing                   # Image CAPI
│       ├── video_processing                   # Video CAPI
│   ├── dfx                                    # DFX code
├── interfaces                                 # API layer
│   ├── inner_api                              # Internal APIs
│   ├── kits                                   # Application APIs
├── services                                   # Service code
├── sertestvices                               # Test code

Build

Run the following command to build the VPE for the 32-bit ARM system:

./build.sh --product-name {product_name} --ccache --build-target video_processing_engine

Run the following command to build the VPE for the 64-bit ARM system:

./build.sh --product-name {product_name} --ccache --target-cpu arm64 --build-target video_processing_engine

product_name indicates the product supported, for example, rk3568.

Description

How to Use

As a component of OpenHarmony, the VPE provides video and image processing capabilities, including color space conversion, dynamic metadata generation, and detail enhancement.

The app developer calls the image scaling sample

1. Add a header file.

   #include <hilog/log.h>
   #include <multimedia/image_framework/image_pixel_map_mdk.h>
   #include <multimedia/image_framework/image/pixelmap_native.h>
   #include <multimedia/video_processing_engine/image_processing.h>
   #include <multimedia/video_processing_engine/image_processing_types.h>
   #include <multimedia/player_framework/native_avformat.h>
   #include <napi/native_api.h>
   

2. Optionally, initialize the environment. Generally, it is called when it is used for the first time in the process, and some time-consuming operations can be completed in advance.

   ImageProcessing_ErrorCode ret =  OH_ImageProcessing_InitializeEnvironment();
   

3. Create a detail enhancement module. Applications can create image detail enhancement modules by image processing engine module types. The variables in the example are described as follows: imageProcessor：an instance of the Detail Enhancement module. IMAGE_PROCESSING_TYPE_DETAIL_ENHANCER：Detail enhancement type. Expected return value: IMAGE_PROCESSING_SUCCESS

   ImageProcessing_ErrorCode ret = OH_ImageProcessing_Create(&imageProcessor, IMAGE_PROCESSING_TYPE_DETAIL_ENHANCER);
   

4. (Optional) Configure the detail enhancement gear, there are currently three gears of high, medium and low and NONE optional, if not configured, the default gear is LOW.

   OH_AVFormat* parameter = OH_AVFormat_Create();
   OH_AVFormat_SetIntValue(parameter, IMAGE_DETAIL_ENHANCER_PARAMETER_KEY_QUALITY_LEVEL,
       IMAGE_DETAIL_ENHANCER_QUALITY_LEVEL_HIGH);
   ImageProcessing_ErrorCode ret = OH_ImageProcessing_SetParameter(imageProcessor,parameter);
   

5. Initiates detail enhancement processing.

   ImageProcessing_ErrorCode ret = OH_ImageProcessing_EnhanceDetail(imageProcessor, srcImage, dstImage);
   

6. Release the processing instance.

   ImageProcessing_ErrorCode ret = OH_ImageProcessing_Destroy(imageProcessor);
   

7. Free up processing resources.

   OH_ImageProcessing_DeinitializeEnvironment();
   

The app developer calls the video zoom sample

1. Add a header file.

   #include <ace/xcomponent/native_interface_xcomponent.h>
   #include <multimedia/player_framework/native_avformat.h>
   #include <multimedia/video_processing_engine/video_processing.h>
   #include <multimedia/video_processing_engine/video_processing_types.h>
   #include <native_window/external_window.h>
   #include <native_buffer/native_buffer.h>
   

2. Optionally, create a decode instance. The input of the detail enhancement module can be a video stream decoded by the system, or the application can fill the window with video data (for example, the application directly fills the data into the window after internal soft decoding). If you select a system decoder to process a video file or video stream, you can create a decoder instance as input to the Detail Enhancement module.

   OH_AVSource* source_ = OH_AVSource_CreateWithFD(inputFd, inputFileOffset, inputFileSize);
   OH_AVDemuxer* demuxer_ = OH_AVDemuxer_CreateWithSource(source_);
   auto sourceFormat = std::shared_ptr<OH_AVFormat>(OH_AVSource_GetSourceFormat(source_), OH_AVFormat_Destroy);
   OH_AVCodec * decoder_ = OH_VideoDecoder_CreateByMime(videoCodecMime.c_str());
   OH_AVFormat *format = OH_AVFormat_Create();
   OH_AVFormat_SetIntValue(format, OH_MD_KEY_WIDTH, videoWidth);
   OH_AVFormat_SetIntValue(format, OH_MD_KEY_HEIGHT, videoHeight);
   OH_AVFormat_SetDoubleValue(format, OH_MD_KEY_FRAME_RATE, frameRate);
   OH_AVFormat_SetIntValue(format, OH_MD_KEY_PIXEL_FORMAT, pixelFormat);
   OH_AVFormat_SetIntValue(format, OH_MD_KEY_ROTATION, rotation);
   int ret = OH_VideoDecoder_Configure(decoder_, format);
   OH_AVFormat_Destroy(format);
   OH_VideoDecoder_RegisterCallback(decoder_,
           {SampleCallback::OnCodecError, SampleCallback::OnCodecFormatChange,
           SampleCallback::OnNeedInputBuffer, SampleCallback::OnNewOutputBuffer}, videoDecContext_);
   
   int ret = OH_VideoDecoder_Prepare(decoder_);
   
   videoDecContext_ = new CodecUserData;
   

3. Optionally, initialize the environment. Generally, it is called when it is used for the first time in the process, and some time-consuming operations can be completed in advance.

   VideoProcessing_ErrorCode ret = OH_VideoProcessing_InitializeEnvironment();
   

4. Create a detail enhancement module. Applications can create detail enhancement modules by video processing engine module types. The variables in the example are described as follows: videoProcessor：an instance of the Detail Enhancement module. VIDEO_PROCESSING_TYPE_DETAIL_ENHANCER：Detail enhancement type. Expected return value: VIDEO_PROCESSING_SUCCESS

   VideoProcessing_ErrorCode ret = OH_VideoProcessing_Create(&videoProcessor， VIDEO_PROCESSING_TYPE_DETAIL_ENHANCER);
   

5. Configure an asynchronous callback function.

   ret = OH_VideoProcessingCallback_Create(&callback);
   OH_VideoProcessingCallback_BindOnError(callback, OnError);
   OH_VideoProcessingCallback_BindOnState(callback, OnState);
   OH_VideoProcessingCallback_BindOnNewOutputBuffer(callback, OnNewOutputBuffer);
   ret = OH_VideoProcessing_RegisterCallback(videoProcessor, callback, this);
   void OnError(OH_VideoProcessing* videoProcessor, VideoProcessing_ErrorCode error, void* userData);
   void OnState(OH_VideoProcessing* videoProcessor, VideoProcessing_State state, void* userData);
   void OnNewOutputBuffer(OH_VideoProcessing* videoProcessor, uint32_t index, void* userData);
   

6. (Optional) Configure the detail enhancement gear, there are currently three gears of high, medium and low and NONE optional, if not configured, the default gear is LOW.

   OH_AVFormat* parameter = OH_AVFormat_Create();
   OH_AVFormat_SetIntValue(parameter, VIDEO_DETAIL_ENHANCER_PARAMETER_KEY_QUALITY_LEVEL, VIDEO_DETAIL_ENHANCER_QUALITY_LEVEL_HIGH);
   OH_VideoProcessing_SetParameter(videoProcessor, parameter);
   

7. Get your Surface.

   ret = OH_VideoProcessing_GetSurface(videoProcessor, inputWindow);
   OH_VideoDecoder_SetSurface(decoder_,  inputWindow_);
   

8. Set Surface

   ret = OH_VideoProcessing_SetSurface(videoProcessor, outWindow);
   

9. Create a decoder input and output thread.

   std::unique_ptr<std::thread> videoDecInputThread_ = std::make_unique<std::thread>(&Player::VideoDecInputThread, this);
   std::unique_ptr<std::thread> videoDecOutputThread_ = std::make_unique<std::thread>(&Player::VideoDecOutputThread, this);
   

10. Initiates detail enhancement processing.

    int ret = OH_VideoDecoder_Start(decoder_);
    ret = OH_VideoProcessing_Start(videoProcessor);
    

11. Call OH_VideoProcessing_Stop()

    VideoProcessing_ErrorCode ret = OH_VideoProcessing_Stop(videoProcessor);
    

12. Release the processing instance.

    VideoProcessing_ErrorCode ret = OH_VideoProcessing_Destroy(videoProcessor)；
    VideoProcessing_ErrorCode ret = OH_VideoProcessingCallback_Destroy(callback);
    

13. Free up processing resources.

    VideoProcessing_ErrorCode ret = OH_VideoProcessing_DeinitializeEnvironment();
    

Custom algorithm plugin registration

1. Implement plugin registration function

   std::vector<MetadataGeneratorCapability> ImageMetadataGen::BuildCapabilities()
   {
       std::vector<MetadataGeneratorCapability> capabilities;
       for (const auto &inColorspace : inColorspaceList) {
           MetadataGeneratorCapability capability = { inColorspace, pixelFormatMap, RANK, IMAGEMETAGENVERSION };
           capabilities.emplace_back(capability);
       }
       return capabilities;
   }
   
   static std::vector<std::shared_ptr<OHOS::Media::VideoProcessingEngine::Extension::ExtensionBase>> RegisterExtensions()
   {
       std::vector<std::shared_ptr<OHOS::Media::VideoProcessingEngine::Extension::ExtensionBase>> extensions;
       auto extension = std::make_shared<OHOS::Media::VideoProcessingEngine::Extension::MetadataGeneratorExtension>();
       extension->info = { OHOS::Media::VideoProcessingEngine::Extension::ExtensionType::METADATA_GENERATOR,
       "ImageMetadataGen", "v1" };
       extension->capabilitiesBuilder = OHOS::Media::VideoProcessingEngine::ImageMetadataGen::BuildCapabilities;
       extensions.push_back(
           std::static_pointer_cast<OHOS::Media::VideoProcessingEngine::Extension::ExtensionBase>(extension));
       return extensions;
   }
   
   void RegisterImageMetadataGeneratorExtensions(uintptr_t extensionListAddr)
   {
       OHOS::Media::VideoProcessingEngine::Extension::DoRegisterExtensions(extensionListAddr, RegisterExtensions);
   }
   

2. Implement algorithm

   ImageMetadataGen::Process(const sptr<SurfaceBuffer> &input)
   

3. Add registered plugin callback function. Add at staticExtensionsRegisterMap

   const std::unordered_map<std::string, RegisterExtensionFunc> staticExtensionsRegisterMap = {
       {"ImageMetadataGeneratorExtensions", RegisterImageMetadataGeneratorExtensions}
   };
   

The system developer implements a custom algorithm plug-in registration example

The video processing engine provides algorithm plug-in registration interfaces such as color space conversion, dynamic metadata generation, and detail enhancement, and system users can register their own algorithms to enrich algorithm plug-ins. The video processing engine defines the basic classes of various algorithms, as follows:

Function	Base class	Header file
Video color space conversion	ColorSpaceConverterBase	colorspace_converter_base.h
Picture color space conversion	ColorSpaceConverterBase	colorspace_converter_base.h
Video detail enhancement	DetailEnhancerBase	detail_enhancer_base.h
Picture detail enhancement	DetailEnhancerBase	detail_enhancer_base.h
Video dynamic metadata	MetadataGeneratorBase	metadata_generator_base.h
Image dynamic metadata	MetadataGeneratorBase	metadata_generator_base.h

Take the scaling algorithm as an example, the plug-in algorithm has been implemented in framework/algorithm/extensions/skia, and the specific development steps are described below

1. Create a new folder in the framework/algorithm/extensions directory, for example, skia, along with the corresponding cpp and h files

2. The scaling algorithm header file is implemented, see framework/algorithm/extensions/skia/include/skia_impl.h

#ifndef SKIA_IMPL_H
#define SKIA_IMPL_H

#include "surface_buffer.h"
#include "include/core/SkYUVAPixmaps.h"

#include "algorithm_errors.h"
#include "detail_enhancer_base.h"
#include "detail_enhancer_capability.h"

namespace OHOS {
namespace Media {
namespace VideoProcessingEngine {

class Skia : public DetailEnhancerBase {
public:
    Skia() = default;
    virtual ~Skia() = default;
    Skia(const Skia&) = delete;
    Skia& operator=(const Skia&) = delete;
    Skia(Skia&&) = delete;
    Skia& operator=(Skia&&) = delete;
    static std::unique_ptr<DetailEnhancerBase> Create();
    static DetailEnhancerCapability BuildCapabilities();
    VPEAlgoErrCode Init() override;
    VPEAlgoErrCode Deinit() override;
    VPEAlgoErrCode SetParameter(const DetailEnhancerParameters& parameter, int type, bool flag) override;
    VPEAlgoErrCode Process(const sptr<SurfaceBuffer>& input, const sptr<SurfaceBuffer>& output) override;
};
void RegisterSkiaExtensions(uintptr_t extensionListAddr);
} // VideoProcessingEngine
} // Media
} // OHOS
#endif // SKIA_IMPL_H

3. For details on plug-in algorithm capability registration and plug-in algorithm implementation, see framework/algorithm/extensions/skia/skia_impl.cpp

#include "skia_impl.h"

#include <array>
#include <chrono>
#include <dlfcn.h>
#include "detail_enhancer_extension.h"
#include "utils.h"

#include "vpe_log.h"

namespace OHOS {
namespace Media {
namespace VideoProcessingEngine {

namespace {

constexpr Extension::Rank RANK = Extension::Rank::RANK_DEFAULT;
constexpr uint32_t VERSION = 0;
} // namespace

std::unique_ptr<DetailEnhancerBase> Skia::Create()
{
    return std::make_unique<Skia>();
}


DetailEnhancerCapability Skia::BuildCapabilities()
{
    std::vector<uint32_t> levels = { DETAIL_ENH_LEVEL_NONE, DETAIL_ENH_LEVEL_LOW, DETAIL_ENH_LEVEL_MEDIUM,
        DETAIL_ENH_LEVEL_HIGH_EVE, DETAIL_ENH_LEVEL_HIGH_AISR, DETAIL_ENH_LEVEL_VIDEO};
    DetailEnhancerCapability capability = { levels, RANK, VERSION };
    return capability;
}

VPEAlgoErrCode Skia::Init()
{
    return VPE_ALGO_ERR_OK;
}

VPEAlgoErrCode Skia::Deinit()
{
    return VPE_ALGO_ERR_OK;
}

VPEAlgoErrCode Skia::SetParameter([[maybe_unused]] const DetailEnhancerParameters& parameter,
    [[maybe_unused]] int type, [[maybe_unused]] bool flag)
{
    return VPE_ALGO_ERR_OK;
}

VPEAlgoErrCode Skia::Process(const sptr<SurfaceBuffer>& input, const sptr<SurfaceBuffer>& output)
{
    return errCode;
}

static std::vector<std::shared_ptr<Extension::ExtensionBase>> RegisterExtensions()
{
    std::vector<std::shared_ptr<Extension::ExtensionBase>> extensions;

    auto extension = std::make_shared<Extension::DetailEnhancerExtension>();
    CHECK_AND_RETURN_RET_LOG(extension != nullptr, extensions, "null pointer");
    extension->info = { Extension::ExtensionType::DETAIL_ENHANCER, "SKIA", "0.0.1" };
    extension->creator = Skia::Create;
    extension->capabilitiesBuilder = Skia::BuildCapabilities;
    extensions.push_back(std::static_pointer_cast<Extension::ExtensionBase>(extension));

    return extensions;
}

void RegisterSkiaExtensions(uintptr_t extensionListAddr)
{
    Extension::DoRegisterExtensions(extensionListAddr, RegisterExtensions);
}
} // VideoProcessingEngine
} // Media
} // OHOS

4. Add a callback function for registering a plug-in, and the VPE plug-in management will iterate through all plug-in registration functions. see framework/algorithm/extension_manager/include/static_extension_list.h

const std::unordered_map<std::string, RegisterExtensionFunc> staticExtensionsRegisterMap = {
    { "Skia", RegisterSkiaExtensions },
};

Repositories Involved

• graphic_graphic_2d
• graphic_graphic_surface
• multimedia_image_framework
• multimedia_media_foundation
• third_party_skia