diff --git a/README.md b/README.md index e2a3989..433bf0f 100644 --- a/README.md +++ b/README.md @@ -1,36 +1,199 @@ -# ai_intelligent_voice_framework +# Intelligent Voice Framework -#### Description -{**When you're done, you can delete the content in this README and update the file with details for others getting started with your repository**} +## Overview -#### Software Architecture -Software architecture description +### Introduction -#### Installation +The intelligent voice framework consists of the intelligent voice service framework and intelligent voice driver. It implements voice enrollment and voice wakeup. -1. xxxx -2. xxxx -3. xxxx +**Figure 1** Architecture of the intelligent voice framework -#### Instructions +![image.png](figures/en-intelligent-voice-framework.png) -1. xxxx -2. xxxx -3. xxxx +The intelligent voice service framework provides the following features: -#### Contribution +- System event monitoring: monitoring system events such as unlocking upon power-on and screen-on/off +- Concurrency policy: intelligent voice service concurrency management -1. Fork the repository -2. Create Feat_xxx branch -3. Commit your code -4. Create Pull Request +- Intelligent voice service: voice enrollment, voice wakeup, and more +- Sound trigger: Digital Signal Processor (DSP) model loading, DSP algorithm enabling/disabling, and DSP event processing + +The intelligent voice driver provides the following features: + +- Engine algorithm: intelligent voice algorithm engine and event reporting + +- Device driver: DSP model loading/unloading, DSP algorithm enabling/disabling, event reporting, and hardware-related channel configuration + +### Basic Concepts +- Voice enrollment: process of converting a wakeup word spoken by a user into an acoustic model and a voiceprint feature, which will be used for comparison during voice wakeup +- Voice wakeup: process of checking whether the current speaker is a registered user and if yes, waking up the system +- DSP chip: chip that implements digital signal processing + +### Directory Structure + +The structure of the repository directory is as follows: + +```shell +/foundation/ai/intelligent_voice_framework # Service code of the intelligent audio framework +├── frameworks # Framework code +│ ├── native # Internal API implementation +│ └── js # External API implementation +├── interfaces # API code +│ ├── inner_api # Internal APIs +│ └── kits # External APIs +├── sa_profile # Service configuration profile +├├── services # Service code +├── LICENSE # License file +├── tests # Developer test +└── utils # Public functions +``` + +### Constraints + +Currently, the intelligent voice framework supports the enrollment and wakeup of only one wakeup word. -#### Gitee Feature +## Available APIs +### APIs Used for Voice Enrollment -1. You can use Readme\_XXX.md to support different languages, such as Readme\_en.md, Readme\_zh.md -2. Gitee blog [blog.gitee.com](https://blog.gitee.com) -3. Explore open source project [https://gitee.com/explore](https://gitee.com/explore) -4. The most valuable open source project [GVP](https://gitee.com/gvp) -5. The manual of Gitee [https://gitee.com/help](https://gitee.com/help) -6. The most popular members [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/) +| API | Description | +| ------------------------------------------------------------ | ------------------ | +| createEnrollIntelligentVoiceEngine(descriptor: EnrollIntelligentVoiceEngineDescriptor): EnrollIntelligentVoiceEngine | Creates an enrollment engine. | +| init(config: EnrollEngineConfig): EnrollIntelligentVoiceEngineCallbackInfo | Initializes this enrollment engine. | +| start(isLast: boolean): EnrollIntelligentVoiceEngineCallbackInfo | Starts enrollment. | +| stop(): void | Stops enrollment. | +| commit(): EnrollIntelligentVoiceEngineCallbackInfo | Commits the enrollment data. | +| setWakeupHapInfo(info: WakeupHapInfo): void | Sets the wakeup application information.| +| setSensibility(sensibility: SensibilityType): void | Sets the sensitivity. | +| release(): void | Releases this enrollment engine. | + +### APIs Used for Voice Wakeup + +| API | Description | +| ------------------------------------------------------------ | ------------------ | +| createWakeupIntelligentVoiceEngine(descriptor: WakeupIntelligentVoiceEngineDescriptor): WakeupIntelligentVoiceEngine | Creates a wakeup engine. | +| setWakeupHapInfo(info: WakeupHapInfo): void | Sets the wakeup application information.| +| setSensibility(sensibility: SensibilityType): void | Sets the sensitivity. | +| on(type: 'wakeupIntelligentVoiceEvent', callback: Callback): void | Subscribes to wakeup events. | +| release(): void | Releases this wakeup engine. | + + +## How to Develop + +### Voice Enrollment + +The voice enrollment process is an interaction process initiated by a user through the enrollment page of an application. The main process is as follows: +1. A user starts enrollment (creating and initializing the enrollment engine), and the enrollment page is displayed. +2. The enrollment page asks the user to speak a wakeup word, and the user speaks the wakeup word (starting enrollment). The enrollment page asks the user to speak the wakeup word again several times. +3. After the enrollment data is committed, the enrollment process is complete. +The code snippet is as follows: + +```js +// Import the intelligentVoice module. +import intelligentVoice from '@ohos.ai.intelligentVoice'; + +// Obtain the intelligent audio management service. +var manager = intellVoice.getIntelligentVoiceManager(); +if (manager == null) { + console.error("Get IntelligentVoiceManager failed."); +} else { + console.info("Get IntelligentVoiceManager success."); + return; +} + +// Create an enrollment engine. +var engine = null; +let engineDescriptor = { + wakeupPhrase: '', // Set a wakeup word. +} +await intellVoice.createEnrollIntelligentVoiceEngine(engineDescriptor).then((data) => { + engine = data; + console.info('Create EnrollIntelligentVoice Engine finish'); +}).catch((err) => { + console.error('Create EnrollIntelligentVoice Engine failed, err: ' + err.message); +}); +if (engine == null) { + console.error('Create EnrollIntelligentVoice Engine failed'); + return; +} + +// Initialize the enrollment engine. +let config = { + language: "zh", // Chinese + area: "CN", // China +} +engine.init(config).then((data) => { + console.info('Init EnrollIntelligentVoice Engine finish'); +}).catch((err) => { + console.info('Init EnrollIntelligentVoice Engine failed, err: '+ err.message); +}); + +// Start enrollment. +let isLast = true; // The value true means that this is the last time to start enrollment, and false means the opposite. The value true is used here. +engine.start(isLast).then((data) => { + console.info('Start enrollment finish'); +}).catch((err) => { + console.info('Start enrollment failed, err: '+ err.message); +}); + +// Commit the enrollment data. +engine.commit().then((data) => { + console.info('Commit enroll result finish'); +}).catch((err) => { + console.info('Commit enroll result failed, err: '+ err.message); +}); + +// Deliver the voice wakeup application information. +let info = { + bundleName: "demo", // Bundle name of the application. demo here is for reference only. Set this parameter based on your application. + abilityName: "demo", // Ability name of the application. demo here is for reference only. Set this parameter based on your application. +} +engine.setWakeupHapInfo(info).then((data) => { + console.info('Set wakeup hap info finish'); +}).catch((err) => { + console.info('Set wakeup hap info failed, err: '+ err.message); +}); + +// Release the enrollment engine. +engine.release().then((data) => { + console.info('Release EnrollIntelligentVoice engine success.'); +}).catch((err) => { + console.info('Release EnrollIntelligentVoice engine failed, err: '+ err.message); +}); +``` + +### Voice Wakeup + +Voice wakeup is controlled by the intelligent voice framework. Upper-layer applications only need to create a wakeup engine by calling **createWakeupIntelligentVoiceEngine** and then subscribe to wakeup events. + +```js +// Obtain the wakeup engine. +var engine = null; +let engineDescriptor = { + needApAlgEngine: true, // Specify whether the framework needs to provide the AP algorithm engine. + wakeupPhrase: '', // Set a wakeup word. +} +await intellVoice.createWakeupIntelligentVoiceEngine(engineDescriptor).then((data) => { + engine = data; + console.info('Create WakeupIntelligentVoice Engine finish'); +}).catch((err) => { + console.error('Create WakeupIntelligentVoice Engine failed, err: ' + err.message); +}); +if (engine == null) { + console.error('Create WakeupIntelligentVoice Engine failed'); + return; +} + +// Subscribe to wakeup events. +engine.on('wakeupIntelligentVoiceEvent',(callback) => { + console.info('wakeupIntelligentVoiceEvent CallBackInfo:') + for (let prop in callback) { + console.info('wakeupIntelligentVoiceEvent prop: ' + prop); + console.info('wakeupIntelligentVoiceEvent value: ' + callback[prop]); + } +}); +``` + +## Repositories Involved + +intelligent_voice_framework diff --git a/README_zh.md b/README_zh.md index 3f18389..8ae3504 100755 --- a/README_zh.md +++ b/README_zh.md @@ -8,7 +8,7 @@ **图 1** 智能语音组件架构图 -![image.png](figures/intelligent-voice-framework.png) +![image.png](figures/zh-intelligent-voice-framework.png) 智能语音服务框架支持如下功能： 系统事件监测：开机解锁、亮灭屏等系统事件监测 diff --git a/figures/en-intelligent-voice-framework.png b/figures/en-intelligent-voice-framework.png new file mode 100644 index 0000000..0f415b1 Binary files /dev/null and b/figures/en-intelligent-voice-framework.png differ diff --git a/figures/intelligent-voice-framework.png b/figures/zh-intelligent-voice-framework.png similarity index 100% rename from figures/intelligent-voice-framework.png rename to figures/zh-intelligent-voice-framework.png diff --git a/patches/developtools_syscap_codec.patch b/patches/developtools_syscap_codec.patch index 5b7ae3b..51b4b9c 100644 --- a/patches/developtools_syscap_codec.patch +++ b/patches/developtools_syscap_codec.patch @@ -1,4 +1,4 @@ -From 6a0d56c42b6c1203f451e71ebd9941c2144534d0 Mon Sep 17 00:00:00 2001 +From 7ac57c5c7041b4b3482b4d3dddb10daf7e87321e Mon Sep 17 00:00:00 2001 From: lvqiang214 Date: Fri, 7 Jul 2023 16:19:34 +0800 Subject: [PATCH] add intelligent voice core @@ -9,21 +9,21 @@ Signed-off-by: lvqiang214 1 file changed, 2 insertions(+) diff --git a/include/codec_config/syscap_define.h b/include/codec_config/syscap_define.h -index b22af3a..742d27a 100644 +index 54cc6ad..39d46f2 100755 --- a/include/codec_config/syscap_define.h +++ b/include/codec_config/syscap_define.h -@@ -298,6 +298,7 @@ typedef enum SystemCapabilityNum { - STARTUP_SYSTEMINFO_LITE, +@@ -299,6 +299,7 @@ typedef enum SystemCapabilityNum { CLOUD_ADS, MULTIMEDIA_AUDIO_PLAYBACKCAPTURE, + MULTIMEDIA_MEDIA_AVSCREENCAPTURE, + AI_INTELLIGENTVOICE_CORE, // Add before here SYSCAP_BASIC_END = 500, } SyscapNum; -@@ -578,6 +579,7 @@ const static SyscapWithNum g_arraySyscap[] = { - {"SystemCapability.FileManagement.File.FileIO.Lite", FILEMANAGEMENT_FILE_FILEIO_LITE}, +@@ -580,6 +581,7 @@ const static SyscapWithNum g_arraySyscap[] = { {"SystemCapability.Cloud.Ads", CLOUD_ADS}, {"SystemCapability.Multimedia.Audio.PlaybackCapture", MULTIMEDIA_AUDIO_PLAYBACKCAPTURE}, + {"SystemCapability.Multimedia.Media.AVScreenCapture", MULTIMEDIA_MEDIA_AVSCREENCAPTURE}, + {"SystemCapability.AI.IntelligentVoice.Core", AI_INTELLIGENTVOICE_CORE}, };