diff --git a/README.md b/README.md index cabdcace6..d9b11b427 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ -[//]: # (
) +[//]: # '
' ([简体中文](./README_zh.md)|English) -[//]: # (# FunASR: A Fundamental End-to-End Speech Recognition Toolkit) +[//]: # "# FunASR: A Fundamental End-to-End Speech Recognition Toolkit" [![SVG Banners](https://svg-banners.vercel.app/api?type=origin&text1=FunASR🤠&text2=💖%20A%20Fundamental%20End-to-End%20Speech%20Recognition%20Toolkit&width=800&height=210)](https://github.com/Akshay090/svg-banners) @@ -15,7 +15,7 @@ FunASR hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！ [**Highlights**](#highlights) -| [**News**](https://github.com/alibaba-damo-academy/FunASR#whats-new) +| [**News**](https://github.com/alibaba-damo-academy/FunASR#whats-new) | [**Installation**](#installation) | [**Quick Start**](#quick-start) | [**Tutorial**](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/tutorial/README.md) @@ -23,17 +23,18 @@ | [**Model Zoo**](#model-zoo) | [**Contact**](#contact) - - - -## Highlights -- FunASR is a fundamental speech recognition toolkit that offers a variety of features, including speech recognition (ASR), Voice Activity Detection (VAD), Punctuation Restoration, Language Models, Speaker Verification, Speaker Diarization and multi-talker ASR. FunASR provides convenient scripts and tutorials, supporting inference and fine-tuning of pre-trained models. -- We have released a vast collection of academic and industrial pretrained models on the [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition) and [huggingface](https://huggingface.co/FunASR), which can be accessed through our [Model Zoo](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/model_zoo/modelscope_models.md). The representative [Paraformer-large](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), a non-autoregressive end-to-end speech recognition model, has the advantages of high accuracy, high efficiency, and convenient deployment, supporting the rapid construction of speech recognition services. For more details on service deployment, please refer to the [service deployment document](runtime/readme_cn.md). +## Highlights + +- FunASR is a fundamental speech recognition toolkit that offers a variety of features, including speech recognition (ASR), Voice Activity Detection (VAD), Punctuation Restoration, Language Models, Speaker Verification, Speaker Diarization and multi-talker ASR. FunASR provides convenient scripts and tutorials, supporting inference and fine-tuning of pre-trained models. +- We have released a vast collection of academic and industrial pretrained models on the [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition) and [huggingface](https://huggingface.co/FunASR), which can be accessed through our [Model Zoo](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/model_zoo/modelscope_models.md). The representative [Paraformer-large](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), a non-autoregressive end-to-end speech recognition model, has the advantages of high accuracy, high efficiency, and convenient deployment, supporting the rapid construction of speech recognition services. For more details on service deployment, please refer to the [service deployment document](runtime/readme_cn.md). + ## What's new: + +- 2025/12/15: [Fun-ASR-Nano-2512](https://modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) is an end-to-end speech recognition large model trained on tens of millions of hours real speech data. It supports low-latency real-time transcription and covers 31 languages. - 2024/10/29: Real-time Transcription Service 1.12 released, The 2pass-offline mode supports the SensevoiceSmal model；([docs](runtime/readme.md)); - 2024/10/10：Added support for the Whisper-large-v3-turbo model, a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. It can be downloaded from the [modelscope](examples/industrial_data_pretraining/whisper/demo.py), and [openai](examples/industrial_data_pretraining/whisper/demo_from_openai.py). - 2024/09/26: Offline File Transcription Service 4.6, Offline File Transcription Service of English 1.7, Real-time Transcription Service 1.11 released, fix memory leak & Support the SensevoiceSmall onnx model；File Transcription Service 2.0 GPU released, Fix GPU memory leak; ([docs](runtime/readme.md)); @@ -57,7 +58,7 @@ - 2024/01/03: Real-time Transcription Service 1.6 released，The 2pass-offline mode supports Ngram language model decoding and WFST hotwords, while also addressing known crash issues and memory leak problems, ([docs](runtime/readme.md#the-real-time-transcription-service-mandarin-cpu)) - 2024/01/03: Fixed known crash issues as well as memory leak problems, ([docs](runtime/readme.md#file-transcription-service-english-cpu)). - 2023/12/04: The Funasr SDK for Windows version 1.0 has been released, featuring support for The offline file transcription service (CPU) of Mandarin, The offline file transcription service (CPU) of English, The real-time transcription service (CPU) of Mandarin. For more details, please refer to the official documentation or release notes([FunASR-Runtime-Windows](https://www.modelscope.cn/models/damo/funasr-runtime-win-cpu-x64/summary)) -- 2023/11/08: The offline file transcription service 3.0 (CPU) of Mandarin has been released, adding punctuation large model, Ngram language model, and wfst hot words. For detailed information, please refer to [docs](runtime#file-transcription-service-mandarin-cpu). +- 2023/11/08: The offline file transcription service 3.0 (CPU) of Mandarin has been released, adding punctuation large model, Ngram language model, and wfst hot words. For detailed information, please refer to [docs](runtime#file-transcription-service-mandarin-cpu). - 2023/10/17: The offline file transcription service (CPU) of English has been released. For more details, please refer to ([docs](runtime#file-transcription-service-english-cpu)). - 2023/10/13: [SlideSpeech](https://slidespeech.github.io/): A large scale multi-modal audio-visual corpus with a significant amount of real-time synchronized slides. - 2023/10/10: The ASR-SpeakersDiarization combined pipeline [Paraformer-VAD-SPK](https://github.com/alibaba-damo-academy/FunASR/blob/main/egs_modelscope/asr_vad_spk/speech_paraformer-large-vad-punc-spk_asr_nat-zh-cn/demo.py) is now released. Experience the model to get recognition results with speaker information. @@ -70,9 +71,11 @@ + ## Installation - Requirements + ```text python>=3.8 torch>=1.13 @@ -80,14 +83,18 @@ torchaudio ``` - Install for pypi + ```shell pip3 install -U funasr ``` + - Or install from source code -``` sh + +```sh git clone https://github.com/alibaba/FunASR.git && cd FunASR pip3 install -e ./ ``` + - Install modelscope or huggingface_hub for the pretrained models (Optional) ```shell @@ -95,37 +102,34 @@ pip3 install -U modelscope huggingface_hub ``` ## Model Zoo + FunASR has open-sourced a large number of pre-trained models on industrial data. You are free to use, copy, modify, and share FunASR models under the [Model License Agreement](./MODEL_LICENSE). Below are some representative models, for more models please refer to the [Model Zoo](./model_zoo). (Note: ⭐ represents the ModelScope model zoo, 🤗 represents the Huggingface model zoo, 🍀 represents the OpenAI model zoo) +| Model Name | Task Details | Training Data | Parameters | +| :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------: | :------------------------------: | :--------: | +| SenseVoiceSmall
([⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) ) | multiple speech understanding capabilities, including ASR, ITN, LID, SER, and AED, support languages such as zh, yue, en, ja, ko | 300000 hours | 234M | +| paraformer-zh
([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) ) | speech recognition, with timestamps, non-streaming | 60000 hours, Mandarin | 220M | +| paraformer-zh-streaming
( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗](https://huggingface.co/funasr/paraformer-zh-streaming) ) | speech recognition, streaming | 60000 hours, Mandarin | 220M | +| paraformer-en
( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗](https://huggingface.co/funasr/paraformer-en) ) | speech recognition, without timestamps, non-streaming | 50000 hours, English | 220M | +| conformer-en
( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗](https://huggingface.co/funasr/conformer-en) ) | speech recognition, non-streaming | 50000 hours, English | 220M | +| ct-punc
( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗](https://huggingface.co/funasr/ct-punc) ) | punctuation restoration | 100M, Mandarin and English | 290M | +| fsmn-vad
( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) ) | voice activity detection | 5000 hours, Mandarin and English | 0.4M | +| fsmn-kws
( [⭐](https://modelscope.cn/models/iic/speech_charctc_kws_phone-xiaoyun/summary) ) | keyword spotting，streaming | 5000 hours, Mandarin | 0.7M | +| fa-zh
( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗](https://huggingface.co/funasr/fa-zh) ) | timestamp prediction | 5000 hours, Mandarin | 38M | +| cam++
( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) ) | speaker verification/diarization | 5000 hours | 7.2M | +| Whisper-large-v3
([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3/summary) [🍀](https://github.com/openai/whisper) ) | speech recognition, with timestamps, non-streaming | multilingual | 1550 M | +| Whisper-large-v3-turbo
([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3-turbo/summary) [🍀](https://github.com/openai/whisper) ) | speech recognition, with timestamps, non-streaming | multilingual | 809 M | +| Qwen-Audio
([⭐](examples/industrial_data_pretraining/qwen_audio/demo.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio) ) | audio-text multimodal models (pretraining) | multilingual | 8B | +| Qwen-Audio-Chat
([⭐](examples/industrial_data_pretraining/qwen_audio/demo_chat.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio-Chat) ) | audio-text multimodal models (chat) | multilingual | 8B | +| emotion2vec+large
([⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) ) | speech emotion recongintion | 40000 hours | 300M | -| Model Name | Task Details | Training Data | Parameters | -|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------:|:--------------------------------:|:----------:| -| SenseVoiceSmall
([⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) ) | multiple speech understanding capabilities, including ASR, ITN, LID, SER, and AED, support languages such as zh, yue, en, ja, ko | 300000 hours | 234M | -| paraformer-zh
([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) ) | speech recognition, with timestamps, non-streaming | 60000 hours, Mandarin | 220M | -| paraformer-zh-streaming
( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗](https://huggingface.co/funasr/paraformer-zh-streaming) ) | speech recognition, streaming | 60000 hours, Mandarin | 220M | -| paraformer-en
( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗](https://huggingface.co/funasr/paraformer-en) ) | speech recognition, without timestamps, non-streaming | 50000 hours, English | 220M | -| conformer-en
( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗](https://huggingface.co/funasr/conformer-en) ) | speech recognition, non-streaming | 50000 hours, English | 220M | -| ct-punc
( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗](https://huggingface.co/funasr/ct-punc) ) | punctuation restoration | 100M, Mandarin and English | 290M | -| fsmn-vad
( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) ) | voice activity detection | 5000 hours, Mandarin and English | 0.4M | -| fsmn-kws
( [⭐](https://modelscope.cn/models/iic/speech_charctc_kws_phone-xiaoyun/summary) ) | keyword spotting，streaming | 5000 hours, Mandarin | 0.7M | -| fa-zh
( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗](https://huggingface.co/funasr/fa-zh) ) | timestamp prediction | 5000 hours, Mandarin | 38M | -| cam++
( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) ) | speaker verification/diarization | 5000 hours | 7.2M | -| Whisper-large-v3
([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3/summary) [🍀](https://github.com/openai/whisper) ) | speech recognition, with timestamps, non-streaming | multilingual | 1550 M | -| Whisper-large-v3-turbo
([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3-turbo/summary) [🍀](https://github.com/openai/whisper) ) | speech recognition, with timestamps, non-streaming | multilingual | 809 M | -| Qwen-Audio
([⭐](examples/industrial_data_pretraining/qwen_audio/demo.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio) ) | audio-text multimodal models (pretraining) | multilingual | 8B | -| Qwen-Audio-Chat
([⭐](examples/industrial_data_pretraining/qwen_audio/demo_chat.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio-Chat) ) | audio-text multimodal models (chat) | multilingual | 8B | -| emotion2vec+large
([⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) ) | speech emotion recongintion | 40000 hours | 300M | - - - - -[//]: # () -[//]: # (FunASR supports pre-trained or further fine-tuned models for deployment as a service. The CPU version of the Chinese offline file conversion service has been released, details can be found in [docs](funasr/runtime/docs/SDK_tutorial.md). More detailed information about service deployment can be found in the [deployment roadmap](funasr/runtime/readme_cn.md).) - +[//]: # +[//]: # "FunASR supports pre-trained or further fine-tuned models for deployment as a service. The CPU version of the Chinese offline file conversion service has been released, details can be found in [docs](funasr/runtime/docs/SDK_tutorial.md). More detailed information about service deployment can be found in the [deployment roadmap](funasr/runtime/readme_cn.md)." + ## Quick Start Below is a quick start tutorial. Test audio files ([Mandarin](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav), [English](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav)). @@ -139,7 +143,9 @@ funasr ++model=paraformer-zh ++vad_model="fsmn-vad" ++punc_model="ct-punc" ++inp Notes: Support recognition of single audio file, as well as file list in Kaldi-style wav.scp format: `wav_id wav_pat` ### Speech Recognition (Non-streaming) + #### SenseVoice + ```python from funasr import AutoModel from funasr.utils.postprocess_utils import rich_transcription_postprocess @@ -166,7 +172,9 @@ res = model.generate( text = rich_transcription_postprocess(res[0]["text"]) print(text) ``` + Parameter Description: + - `model_dir`: The name of the model, or the path to the model on the local disk. - `vad_model`: This indicates the activation of VAD (Voice Activity Detection). The purpose of VAD is to split long audio into shorter clips. In this case, the inference time includes both VAD and SenseVoice total consumption, and represents the end-to-end latency. If you wish to test the SenseVoice model's inference time separately, the VAD model can be disabled. - `vad_kwargs`: Specifies the configurations for the VAD model. `max_single_segment_time`: denotes the maximum duration for audio segmentation by the `vad_model`, with the unit being milliseconds (ms). @@ -176,21 +184,24 @@ Parameter Description: - `ban_emo_unk`: Whether to ban the output of the `emo_unk` token. #### Paraformer + ```python from funasr import AutoModel # paraformer-zh is a multi-functional asr model # use vad, punc, spk or not as you need -model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc", - # spk_model="cam++", +model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc", + # spk_model="cam++", ) -res = model.generate(input=f"{model.model_path}/example/asr_example.wav", - batch_size_s=300, +res = model.generate(input=f"{model.model_path}/example/asr_example.wav", + batch_size_s=300, hotword='魔搭') print(res) ``` + Note: `hub`: represents the model repository, `ms` stands for selecting ModelScope download, `hf` stands for selecting Huggingface download. ### Speech Recognition (Streaming) + ```python from funasr import AutoModel @@ -215,11 +226,13 @@ for i in range(total_chunk_num): res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size, encoder_chunk_look_back=encoder_chunk_look_back, decoder_chunk_look_back=decoder_chunk_look_back) print(res) ``` + Note: `chunk_size` is the configuration for streaming latency.` [0,10,5]` indicates that the real-time display granularity is `10*60=600ms`, and the lookahead information is `5*60=300ms`. Each inference input is `600ms` (sample points are `16000*0.6=960`), and the output is the corresponding text. For the last speech segment input, `is_final=True` needs to be set to force the output of the last word.
More Examples ### Voice Activity Detection (Non-Streaming) + ```python from funasr import AutoModel @@ -228,9 +241,11 @@ wav_file = f"{model.model_path}/example/vad_example.wav" res = model.generate(input=wav_file) print(res) ``` + Note: The output format of the VAD model is: `[[beg1, end1], [beg2, end2], ..., [begN, endN]]`, where `begN/endN` indicates the starting/ending point of the `N-th` valid audio segment, measured in milliseconds. ### Voice Activity Detection (Streaming) + ```python from funasr import AutoModel @@ -252,14 +267,18 @@ for i in range(total_chunk_num): if len(res[0]["value"]): print(res) ``` + Note: The output format for the streaming VAD model can be one of four scenarios: + - `[[beg1, end1], [beg2, end2], .., [begN, endN]]`：The same as the offline VAD output result mentioned above. - `[[beg, -1]]`：Indicates that only a starting point has been detected. - `[[-1, end]]`：Indicates that only an ending point has been detected. -- `[]`：Indicates that neither a starting point nor an ending point has been detected. +- `[]`：Indicates that neither a starting point nor an ending point has been detected. The output is measured in milliseconds and represents the absolute time from the starting point. + ### Punctuation Restoration + ```python from funasr import AutoModel @@ -267,7 +286,9 @@ model = AutoModel(model="ct-punc") res = model.generate(input="那今天的会就到这里吧 happy new year 明年见") print(res) ``` + ### Timestamp Prediction + ```python from funasr import AutoModel @@ -278,8 +299,8 @@ res = model.generate(input=(wav_file, text_file), data_type=("sound", "text")) print(res) ``` - ### Speech Emotion Recognition + ```python from funasr import AutoModel @@ -291,7 +312,7 @@ res = model.generate(wav_file, output_dir="./outputs", granularity="utterance", print(res) ``` -More usages ref to [docs](docs/tutorial/README_zh.md), +More usages ref to [docs](docs/tutorial/README_zh.md), more examples ref to [demo](https://github.com/alibaba-damo-academy/FunASR/tree/main/examples/industrial_data_pretraining)
@@ -299,11 +320,13 @@ more examples ref to [demo](https://github.com/alibaba-damo-academy/FunASR/tree/ ## Export ONNX ### Command-line usage + ```shell funasr-export ++model=paraformer ++quantize=false ++device=cpu ``` ### Python + ```python from funasr import AutoModel @@ -313,6 +336,7 @@ res = model.export(quantize=False) ``` ### Test ONNX + ```python # pip3 install -U funasr-onnx from pathlib import Path @@ -333,7 +357,9 @@ print(result) More examples ref to [demo](runtime/python/onnxruntime) ## Deployment Service + FunASR supports deploying pre-trained or further fine-tuned models for service. Currently, it supports the following types of service deployment: + - File transcription service, Mandarin, CPU version, done - The real-time transcription service, Mandarin (CPU), done - File transcription service, English, CPU version, done @@ -342,31 +368,33 @@ FunASR supports deploying pre-trained or further fine-tuned models for service. For more detailed information, please refer to the [service deployment documentation](runtime/readme.md). - + ## Community Communication + If you encounter problems in use, you can directly raise Issues on the github page. You can also scan the following DingTalk group to join the community group for communication and discussion. | DingTalk group | -|:-------------------------------------------------------------------:| +| :-----------------------------------------------------------------: | |
| ## Contributors -|
|
|
|
|
| | -|:------------------------------------------------------------------:|:---------------------------------------------------------------:|:--------------------------------------------------------------:|:-------------------------------------------------------:|:-----------------------------------------------------------:|:------------------------------------------------------:| +|
|
|
|
| | | +| :----------------------------------------------------------------: | :-------------------------------------------------------------: | :-----------------------------------------------------------: | :-----------------------------------------------------: | :-------------------------------------------------------: | :----------------------------------------------------: | The contributors can be found in [contributors list](./Acknowledge.md) ## License + This project is licensed under [The MIT License](https://opensource.org/licenses/MIT). FunASR also contains various third-party components and some code modified from other repos under other open source licenses. The use of pretraining model is subject to [model license](./MODEL_LICENSE) - ## Citations -``` bibtex + +```bibtex @inproceedings{gao2023funasr, author={Zhifu Gao and Zerui Li and Jiaming Wang and Haoneng Luo and Xian Shi and Mengzhe Chen and Yabin Li and Lingyun Zuo and Zhihao Du and Zhangyu Xiao and Shiliang Zhang}, title={FunASR: A Fundamental End-to-End Speech Recognition Toolkit}, diff --git a/README_zh.md b/README_zh.md index b36700890..ecdba3ea0 100644 --- a/README_zh.md +++ b/README_zh.md @@ -1,21 +1,18 @@ -[//]: # (
) +[//]: # '
' (简体中文|[English](./README.md)) - - [![SVG Banners](https://svg-banners.vercel.app/api?type=origin&text1=FunASR🤠&text2=💖%20A%20Fundamental%20End-to-End%20Speech%20Recognition%20Toolkit&width=800&height=210)](https://github.com/Akshay090/svg-banners) -[//]: # (# FunASR: A Fundamental End-to-End Speech Recognition Toolkit) +[//]: # "# FunASR: A Fundamental End-to-End Speech Recognition Toolkit" [![PyPI](https://img.shields.io/pypi/v/funasr)](https://pypi.org/project/funasr/) +FunASR 希望在语音识别的学术研究和工业应用之间架起一座桥梁。通过发布工业级语音识别模型的训练和微调，研究人员和开发人员可以更方便地进行语音识别模型的研究和生产，并推动语音识别生态的发展。让语音识别更有趣！ -FunASR希望在语音识别的学术研究和工业应用之间架起一座桥梁。通过发布工业级语音识别模型的训练和微调，研究人员和开发人员可以更方便地进行语音识别模型的研究和生产，并推动语音识别生态的发展。让语音识别更有趣！ - -
+

- 核心功能 + 核心功能 ｜ 最新动态 ｜ 安装 ｜ 快速开始 @@ -27,69 +24,78 @@ FunASR希望在语音识别的学术研究和工业应用之间架起一座桥

+ ## 核心功能 -- FunASR是一个基础语音识别工具包，提供多种功能，包括语音识别（ASR）、语音端点检测（VAD）、标点恢复、语言模型、说话人验证、说话人分离和多人对话语音识别等。FunASR提供了便捷的脚本和教程，支持预训练好的模型的推理与微调。 + +- FunASR 是一个基础语音识别工具包，提供多种功能，包括语音识别（ASR）、语音端点检测（VAD）、标点恢复、语言模型、说话人验证、说话人分离和多人对话语音识别等。FunASR 提供了便捷的脚本和教程，支持预训练好的模型的推理与微调。 - 我们在[ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition)与[huggingface](https://huggingface.co/FunASR)上发布了大量开源数据集或者海量工业数据训练的模型，可以通过我们的[模型仓库](https://github.com/modelscope/FunASR/blob/main/model_zoo/readme_zh.md)了解模型的详细信息。代表性的[Paraformer](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary)非自回归端到端语音识别模型具有高精度、高效率、便捷部署的优点，支持快速构建语音识别服务，详细信息可以阅读([服务部署文档](runtime/readme_cn.md))。 + ## 最新动态 -- 2024/10/29: 中文实时语音听写服务 1.12 发布，2pass-offline模式支持SensevoiceSmall模型；详细信息参阅([部署文档](runtime/readme_cn.md)) -- 2024/10/10：新增加Whisper-large-v3-turbo模型支持，多语言语音识别/翻译/语种识别，支持从 [modelscope](examples/industrial_data_pretraining/whisper/demo.py)仓库下载，也支持从 [openai](examples/industrial_data_pretraining/whisper/demo_from_openai.py)仓库下载模型。 -- 2024/09/26: 中文离线文件转写服务 4.6、英文离线文件转写服务 1.7、中文实时语音听写服务 1.11 发布，修复ONNX内存泄漏、支持SensevoiceSmall onnx模型；中文离线文件转写服务GPU 2.0 发布，修复显存泄漏; 详细信息参阅([部署文档](runtime/readme_cn.md)) -- 2024/09/25：新增语音唤醒模型，支持[fsmn_kws](https://modelscope.cn/models/iic/speech_sanm_kws_phone-xiaoyun-commands-online), [fsmn_kws_mt](https://modelscope.cn/models/iic/speech_sanm_kws_phone-xiaoyun-commands-online), [sanm_kws](https://modelscope.cn/models/iic/speech_sanm_kws_phone-xiaoyun-commands-offline), [sanm_kws_streaming](https://modelscope.cn/models/iic/speech_sanm_kws_phone-xiaoyun-commands-online) 4个模型的微调和推理。 + +- 2025/12/15: [Fun-ASR-Nano-2512](https://modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) 是一款基于数千万小时真实语音数据训练的端到端语音识别大模型。它支持低延迟实时转写，并涵盖 31 种语言识别功能。 +- 2024/10/29: 中文实时语音听写服务 1.12 发布，2pass-offline 模式支持 SensevoiceSmall 模型；详细信息参阅([部署文档](runtime/readme_cn.md)) +- 2024/10/10：新增加 Whisper-large-v3-turbo 模型支持，多语言语音识别/翻译/语种识别，支持从 [modelscope](examples/industrial_data_pretraining/whisper/demo.py)仓库下载，也支持从 [openai](examples/industrial_data_pretraining/whisper/demo_from_openai.py)仓库下载模型。 +- 2024/09/26: 中文离线文件转写服务 4.6、英文离线文件转写服务 1.7、中文实时语音听写服务 1.11 发布，修复 ONNX 内存泄漏、支持 SensevoiceSmall onnx 模型；中文离线文件转写服务 GPU 2.0 发布，修复显存泄漏; 详细信息参阅([部署文档](runtime/readme_cn.md)) +- 2024/09/25：新增语音唤醒模型，支持[fsmn_kws](https://modelscope.cn/models/iic/speech_sanm_kws_phone-xiaoyun-commands-online), [fsmn_kws_mt](https://modelscope.cn/models/iic/speech_sanm_kws_phone-xiaoyun-commands-online), [sanm_kws](https://modelscope.cn/models/iic/speech_sanm_kws_phone-xiaoyun-commands-offline), [sanm_kws_streaming](https://modelscope.cn/models/iic/speech_sanm_kws_phone-xiaoyun-commands-online) 4 个模型的微调和推理。 - 2024/07/04：[SenseVoice](https://github.com/FunAudioLLM/SenseVoice) 是一个基础语音理解模型，具备多种语音理解能力，涵盖了自动语音识别（ASR）、语言识别（LID）、情感识别（SER）以及音频事件检测（AED）。 -- 2024/07/01：中文离线文件转写服务GPU版本 1.1发布，优化bladedisc模型兼容性问题；详细信息参阅([部署文档](runtime/readme_cn.md)) -- 2024/06/27：中文离线文件转写服务GPU版本 1.0发布，支持动态batch，支持多路并发，在长音频测试集上单线RTF为0.0076，多线加速比为1200+（CPU为330+）；详细信息参阅([部署文档](runtime/readme_cn.md)) +- 2024/07/01：中文离线文件转写服务 GPU 版本 1.1 发布，优化 bladedisc 模型兼容性问题；详细信息参阅([部署文档](runtime/readme_cn.md)) +- 2024/06/27：中文离线文件转写服务 GPU 版本 1.0 发布，支持动态 batch，支持多路并发，在长音频测试集上单线 RTF 为 0.0076，多线加速比为 1200+（CPU 为 330+）；详细信息参阅([部署文档](runtime/readme_cn.md)) - 2024/05/15：新增加情感识别模型，[emotion2vec+large](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary)，[emotion2vec+base](https://modelscope.cn/models/iic/emotion2vec_plus_base/summary)，[emotion2vec+seed](https://modelscope.cn/models/iic/emotion2vec_plus_seed/summary)，输出情感类别为：生气/angry，开心/happy，中立/neutral，难过/sad。 -- 2024/05/15: 中文离线文件转写服务 4.5、英文离线文件转写服务 1.6、中文实时语音听写服务 1.10 发布，适配FunASR 1.0模型结构；详细信息参阅([部署文档](runtime/readme_cn.md)) -- 2024/03/05：新增加Qwen-Audio与Qwen-Audio-Chat音频文本模态大模型，在多个音频领域测试榜单刷榜，中支持语音对话，详细用法见 [示例](examples/industrial_data_pretraining/qwen_audio)。 -- 2024/03/05：新增加Whisper-large-v3模型支持，多语言语音识别/翻译/语种识别，支持从 [modelscope](examples/industrial_data_pretraining/whisper/demo.py)仓库下载，也支持从 [openai](examples/industrial_data_pretraining/whisper/demo_from_openai.py)仓库下载模型。 -- 2024/03/05: 中文离线文件转写服务 4.4、英文离线文件转写服务 1.5、中文实时语音听写服务 1.9 发布，docker镜像支持arm64平台，升级modelscope版本；详细信息参阅([部署文档](runtime/readme_cn.md)) -- 2024/01/30：funasr-1.0发布，更新说明[文档](https://github.com/alibaba-damo-academy/FunASR/discussions/1319) +- 2024/05/15: 中文离线文件转写服务 4.5、英文离线文件转写服务 1.6、中文实时语音听写服务 1.10 发布，适配 FunASR 1.0 模型结构；详细信息参阅([部署文档](runtime/readme_cn.md)) +- 2024/03/05：新增加 Qwen-Audio 与 Qwen-Audio-Chat 音频文本模态大模型，在多个音频领域测试榜单刷榜，中支持语音对话，详细用法见 [示例](examples/industrial_data_pretraining/qwen_audio)。 +- 2024/03/05：新增加 Whisper-large-v3 模型支持，多语言语音识别/翻译/语种识别，支持从 [modelscope](examples/industrial_data_pretraining/whisper/demo.py)仓库下载，也支持从 [openai](examples/industrial_data_pretraining/whisper/demo_from_openai.py)仓库下载模型。 +- 2024/03/05: 中文离线文件转写服务 4.4、英文离线文件转写服务 1.5、中文实时语音听写服务 1.9 发布，docker 镜像支持 arm64 平台，升级 modelscope 版本；详细信息参阅([部署文档](runtime/readme_cn.md)) +- 2024/01/30：funasr-1.0 发布，更新说明[文档](https://github.com/alibaba-damo-academy/FunASR/discussions/1319)
展开日志 - 2024/01/30：新增加情感识别 [模型链接](https://www.modelscope.cn/models/iic/emotion2vec_base_finetuned/summary)，原始模型 [repo](https://github.com/ddlBoJack/emotion2vec). -- 2024/01/25: 中文离线文件转写服务 4.2、英文离线文件转写服务 1.3，优化vad数据处理方式，大幅降低峰值内存占用，内存泄漏优化；中文实时语音听写服务 1.7 发布，客户端优化；详细信息参阅([部署文档](runtime/readme_cn.md)) -- 2024/01/09: funasr社区软件包windows 2.0版本发布，支持软件包中文离线文件转写4.1、英文离线文件转写1.2、中文实时听写服务1.6的最新功能，详细信息参阅([FunASR社区软件包windows版本](https://www.modelscope.cn/models/damo/funasr-runtime-win-cpu-x64/summary)) -- 2024/01/03: 中文离线文件转写服务 4.0 发布，新增支持8k模型、优化时间戳不匹配问题及增加句子级别时间戳、优化英文单词fst热词效果、支持自动化配置线程参数，同时修复已知的crash问题及内存泄漏问题，详细信息参阅([部署文档](runtime/readme_cn.md#中文离线文件转写服务cpu版本)) -- 2024/01/03: 中文实时语音听写服务 1.6 发布，2pass-offline模式支持Ngram语言模型解码、wfst热词，同时修复已知的crash问题及内存泄漏问题，详细信息参阅([部署文档](runtime/readme_cn.md#中文实时语音听写服务cpu版本)) -- 2024/01/03: 英文离线文件转写服务 1.2 发布，修复已知的crash问题及内存泄漏问题，详细信息参阅([部署文档](runtime/readme_cn.md#英文离线文件转写服务cpu版本)) -- 2023/12/04: funasr社区软件包windows 1.0版本发布，支持中文离线文件转写、英文离线文件转写、中文实时听写服务，详细信息参阅([FunASR社区软件包windows版本](https://www.modelscope.cn/models/damo/funasr-runtime-win-cpu-x64/summary)) -- 2023/11/08：中文离线文件转写服务3.0 CPU版本发布，新增标点大模型、Ngram语言模型与wfst热词，详细信息参阅([部署文档](runtime/readme_cn.md#中文离线文件转写服务cpu版本)) -- 2023/10/17: 英文离线文件转写服务一键部署的CPU版本发布，详细信息参阅([部署文档](runtime/readme_cn.md#英文离线文件转写服务cpu版本)) +- 2024/01/25: 中文离线文件转写服务 4.2、英文离线文件转写服务 1.3，优化 vad 数据处理方式，大幅降低峰值内存占用，内存泄漏优化；中文实时语音听写服务 1.7 发布，客户端优化；详细信息参阅([部署文档](runtime/readme_cn.md)) +- 2024/01/09: funasr 社区软件包 windows 2.0 版本发布，支持软件包中文离线文件转写 4.1、英文离线文件转写 1.2、中文实时听写服务 1.6 的最新功能，详细信息参阅([FunASR 社区软件包 windows 版本](https://www.modelscope.cn/models/damo/funasr-runtime-win-cpu-x64/summary)) +- 2024/01/03: 中文离线文件转写服务 4.0 发布，新增支持 8k 模型、优化时间戳不匹配问题及增加句子级别时间戳、优化英文单词 fst 热词效果、支持自动化配置线程参数，同时修复已知的 crash 问题及内存泄漏问题，详细信息参阅([部署文档](runtime/readme_cn.md#中文离线文件转写服务cpu版本)) +- 2024/01/03: 中文实时语音听写服务 1.6 发布，2pass-offline 模式支持 Ngram 语言模型解码、wfst 热词，同时修复已知的 crash 问题及内存泄漏问题，详细信息参阅([部署文档](runtime/readme_cn.md#中文实时语音听写服务cpu版本)) +- 2024/01/03: 英文离线文件转写服务 1.2 发布，修复已知的 crash 问题及内存泄漏问题，详细信息参阅([部署文档](runtime/readme_cn.md#英文离线文件转写服务cpu版本)) +- 2023/12/04: funasr 社区软件包 windows 1.0 版本发布，支持中文离线文件转写、英文离线文件转写、中文实时听写服务，详细信息参阅([FunASR 社区软件包 windows 版本](https://www.modelscope.cn/models/damo/funasr-runtime-win-cpu-x64/summary)) +- 2023/11/08：中文离线文件转写服务 3.0 CPU 版本发布，新增标点大模型、Ngram 语言模型与 wfst 热词，详细信息参阅([部署文档](runtime/readme_cn.md#中文离线文件转写服务cpu版本)) +- 2023/10/17: 英文离线文件转写服务一键部署的 CPU 版本发布，详细信息参阅([部署文档](runtime/readme_cn.md#英文离线文件转写服务cpu版本)) - 2023/10/13: [SlideSpeech](https://slidespeech.github.io/): 一个大规模的多模态音视频语料库，主要是在线会议或者在线课程场景，包含了大量与发言人讲话实时同步的幻灯片。 - 2023.10.10: [Paraformer-long-Spk](https://github.com/alibaba-damo-academy/FunASR/blob/main/egs_modelscope/asr_vad_spk/speech_paraformer-large-vad-punc-spk_asr_nat-zh-cn/demo.py)模型发布，支持在长语音识别的基础上获取每句话的说话人标签。 -- 2023.10.07: [FunCodec](https://github.com/alibaba-damo-academy/FunCodec): FunCodec提供开源模型和训练工具，可以用于音频离散编码，以及基于离散编码的语音识别、语音合成等任务。 -- 2023.09.01: 中文离线文件转写服务2.0 CPU版本发布，新增ffmpeg、时间戳与热词模型支持，详细信息参阅([部署文档](runtime/readme_cn.md#中文离线文件转写服务cpu版本)) -- 2023.08.07: 中文实时语音听写服务一键部署的CPU版本发布，详细信息参阅([部署文档](runtime/readme_cn.md#中文实时语音听写服务cpu版本)) -- 2023.07.17: BAT一种低延迟低内存消耗的RNN-T模型发布，详细信息参阅（[BAT](egs/aishell/bat)） -- 2023.06.26: ASRU2023 多通道多方会议转录挑战赛2.0完成竞赛结果公布，详细信息参阅（[M2MeT2.0](https://alibaba-damo-academy.github.io/FunASR/m2met2_cn/index.html)） +- 2023.10.07: [FunCodec](https://github.com/alibaba-damo-academy/FunCodec): FunCodec 提供开源模型和训练工具，可以用于音频离散编码，以及基于离散编码的语音识别、语音合成等任务。 +- 2023.09.01: 中文离线文件转写服务 2.0 CPU 版本发布，新增 ffmpeg、时间戳与热词模型支持，详细信息参阅([部署文档](runtime/readme_cn.md#中文离线文件转写服务cpu版本)) +- 2023.08.07: 中文实时语音听写服务一键部署的 CPU 版本发布，详细信息参阅([部署文档](runtime/readme_cn.md#中文实时语音听写服务cpu版本)) +- 2023.07.17: BAT 一种低延迟低内存消耗的 RNN-T 模型发布，详细信息参阅（[BAT](egs/aishell/bat)） +- 2023.06.26: ASRU2023 多通道多方会议转录挑战赛 2.0 完成竞赛结果公布，详细信息参阅（[M2MeT2.0](https://alibaba-damo-academy.github.io/FunASR/m2met2_cn/index.html)）
+ ## 安装教程 -- 安装funasr之前，确保已经安装了下面依赖环境: +- 安装 funasr 之前，确保已经安装了下面依赖环境: + ```text python>=3.8 torch>=1.13 torchaudio ``` -- pip安装 +- pip 安装 + ```shell pip3 install -U funasr ``` - 或者从源代码安装 -``` sh + +```sh git clone https://github.com/alibaba/FunASR.git && cd FunASR pip3 install -e ./ ``` -如果需要使用工业预训练模型，安装modelscope与huggingface_hub（可选） +如果需要使用工业预训练模型，安装 modelscope 与 huggingface_hub（可选） ```shell pip3 install -U modelscope huggingface huggingface_hub @@ -97,30 +103,30 @@ pip3 install -U modelscope huggingface huggingface_hub ## 模型仓库 -FunASR开源了大量在工业数据上预训练模型，您可以在[模型许可协议](./MODEL_LICENSE)下自由使用、复制、修改和分享FunASR模型，下面列举代表性的模型，更多模型请参考 [模型仓库](./model_zoo)。 +FunASR 开源了大量在工业数据上预训练模型，您可以在[模型许可协议](./MODEL_LICENSE)下自由使用、复制、修改和分享 FunASR 模型，下面列举代表性的模型，更多模型请参考 [模型仓库](./model_zoo)。 -（注：⭐ 表示ModelScope模型仓库，🤗 表示Huggingface模型仓库，🍀表示OpenAI模型仓库） +（注：⭐ 表示 ModelScope 模型仓库，🤗 表示 Huggingface 模型仓库，🍀 表示 OpenAI 模型仓库） - -| 模型名字 | 任务详情 | 训练数据 | 参数量 | -|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:------------------:|:--------------:|:------:| -| SenseVoiceSmall
([⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) ) | 多种语音理解能力，涵盖了自动语音识别（ASR）、语言识别（LID）、情感识别（SER）以及音频事件检测（AED） | 400000小时，中文 | 330M | -| paraformer-zh
([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) ) | 语音识别，带时间戳输出，非实时 | 60000小时，中文 | 220M | -| paraformer-zh-streaming
( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗](https://huggingface.co/funasr/paraformer-zh-streaming) ) | 语音识别，实时 | 60000小时，中文 | 220M | -| paraformer-en
( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗](https://huggingface.co/funasr/paraformer-en) ) | 语音识别，非实时 | 50000小时，英文 | 220M | -| conformer-en
( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗](https://huggingface.co/funasr/conformer-en) ) | 语音识别，非实时 | 50000小时，英文 | 220M | -| ct-punc
( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗](https://huggingface.co/funasr/ct-punc) ) | 标点恢复 | 100M，中文与英文 | 290M | -| fsmn-vad
( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) ) | 语音端点检测，实时 | 5000小时，中文与英文 | 0.4M | -| fsmn-kws
( [⭐](https://modelscope.cn/models/iic/speech_charctc_kws_phone-xiaoyun/summary) ) | 语音唤醒，实时 | 5000小时，中文 | 0.7M | -| fa-zh
( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗](https://huggingface.co/funasr/fa-zh) ) | 字级别时间戳预测 | 50000小时，中文 | 38M | -| cam++
( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) ) | 说话人确认/分割 | 5000小时 | 7.2M | -| Whisper-large-v3
([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3/summary) [🍀](https://github.com/openai/whisper) ) | 语音识别，带时间戳输出，非实时 | 多语言 | 1550 M | -| Whisper-large-v3-turbo
([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3-turbo/summary) [🍀](https://github.com/openai/whisper) ) | 语音识别，带时间戳输出，非实时 | 多语言 | 809 M | -| Qwen-Audio
([⭐](examples/industrial_data_pretraining/qwen_audio/demo.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio) ) | 音频文本多模态大模型（预训练） | 多语言 | 8B | -| Qwen-Audio-Chat
([⭐](examples/industrial_data_pretraining/qwen_audio/demo_chat.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio-Chat) ) | 音频文本多模态大模型（chat版本） | 多语言 | 8B | -| emotion2vec+large
([⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) ) | 情感识别模型 | 40000小时，4种情感类别 | 300M | +| 模型名字 | 任务详情 | 训练数据 | 参数量 | +| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------: | :----------------------: | :----: | +| SenseVoiceSmall
([⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) ) | 多种语音理解能力，涵盖了自动语音识别（ASR）、语言识别（LID）、情感识别（SER）以及音频事件检测（AED） | 400000 小时，中文 | 330M | +| paraformer-zh
([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) ) | 语音识别，带时间戳输出，非实时 | 60000 小时，中文 | 220M | +| paraformer-zh-streaming
( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗](https://huggingface.co/funasr/paraformer-zh-streaming) ) | 语音识别，实时 | 60000 小时，中文 | 220M | +| paraformer-en
( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗](https://huggingface.co/funasr/paraformer-en) ) | 语音识别，非实时 | 50000 小时，英文 | 220M | +| conformer-en
( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗](https://huggingface.co/funasr/conformer-en) ) | 语音识别，非实时 | 50000 小时，英文 | 220M | +| ct-punc
( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗](https://huggingface.co/funasr/ct-punc) ) | 标点恢复 | 100M，中文与英文 | 290M | +| fsmn-vad
( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) ) | 语音端点检测，实时 | 5000 小时，中文与英文 | 0.4M | +| fsmn-kws
( [⭐](https://modelscope.cn/models/iic/speech_charctc_kws_phone-xiaoyun/summary) ) | 语音唤醒，实时 | 5000 小时，中文 | 0.7M | +| fa-zh
( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗](https://huggingface.co/funasr/fa-zh) ) | 字级别时间戳预测 | 50000 小时，中文 | 38M | +| cam++
( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) ) | 说话人确认/分割 | 5000 小时 | 7.2M | +| Whisper-large-v3
([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3/summary) [🍀](https://github.com/openai/whisper) ) | 语音识别，带时间戳输出，非实时 | 多语言 | 1550 M | +| Whisper-large-v3-turbo
([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3-turbo/summary) [🍀](https://github.com/openai/whisper) ) | 语音识别，带时间戳输出，非实时 | 多语言 | 809 M | +| Qwen-Audio
([⭐](examples/industrial_data_pretraining/qwen_audio/demo.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio) ) | 音频文本多模态大模型（预训练） | 多语言 | 8B | +| Qwen-Audio-Chat
([⭐](examples/industrial_data_pretraining/qwen_audio/demo_chat.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio-Chat) ) | 音频文本多模态大模型（chat 版本） | 多语言 | 8B | +| emotion2vec+large
([⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) ) | 情感识别模型 | 40000 小时，4 种情感类别 | 300M | + ## 快速开始 下面为快速上手教程，测试音频（[中文](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav)，[英文](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav)） @@ -131,10 +137,12 @@ FunASR开源了大量在工业数据上预训练模型，您可以在[模型许 funasr ++model=paraformer-zh ++vad_model="fsmn-vad" ++punc_model="ct-punc" ++input=asr_example_zh.wav ``` -注：支持单条音频文件识别，也支持文件列表，列表为kaldi风格wav.scp：`wav_id wav_path` +注：支持单条音频文件识别，也支持文件列表，列表为 kaldi 风格 wav.scp：`wav_id wav_path` ### 非实时语音识别 + #### SenseVoice + ```python from funasr import AutoModel from funasr.utils.postprocess_utils import rich_transcription_postprocess @@ -161,29 +169,33 @@ res = model.generate( text = rich_transcription_postprocess(res[0]["text"]) print(text) ``` + 参数说明： + - `model_dir`：模型名称，或本地磁盘中的模型路径。 -- `vad_model`：表示开启VAD，VAD的作用是将长音频切割成短音频，此时推理耗时包括了VAD与SenseVoice总耗时，为链路耗时，如果需要单独测试SenseVoice模型耗时，可以关闭VAD模型。 -- `vad_kwargs`：表示VAD模型配置,`max_single_segment_time`: 表示`vad_model`最大切割音频时长, 单位是毫秒ms。 +- `vad_model`：表示开启 VAD，VAD 的作用是将长音频切割成短音频，此时推理耗时包括了 VAD 与 SenseVoice 总耗时，为链路耗时，如果需要单独测试 SenseVoice 模型耗时，可以关闭 VAD 模型。 +- `vad_kwargs`：表示 VAD 模型配置,`max_single_segment_time`: 表示`vad_model`最大切割音频时长, 单位是毫秒 ms。 - `use_itn`：输出结果中是否包含标点与逆文本正则化。 -- `batch_size_s` 表示采用动态batch，batch中总音频时长，单位为秒s。 -- `merge_vad`：是否将 vad 模型切割的短音频碎片合成，合并后长度为`merge_length_s`，单位为秒s。 -- `ban_emo_unk`：禁用emo_unk标签，禁用后所有的句子都会被赋与情感标签。 +- `batch_size_s` 表示采用动态 batch，batch 中总音频时长，单位为秒 s。 +- `merge_vad`：是否将 vad 模型切割的短音频碎片合成，合并后长度为`merge_length_s`，单位为秒 s。 +- `ban_emo_unk`：禁用 emo_unk 标签，禁用后所有的句子都会被赋与情感标签。 #### Paraformer + ```python from funasr import AutoModel # paraformer-zh is a multi-functional asr model # use vad, punc, spk or not as you need -model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc", +model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc", # spk_model="cam++" ) -res = model.generate(input=f"{model.model_path}/example/asr_example.wav", - batch_size_s=300, +res = model.generate(input=f"{model.model_path}/example/asr_example.wav", + batch_size_s=300, hotword='魔搭') print(res) ``` -注：`hub`：表示模型仓库，`ms`为选择modelscope下载，`hf`为选择huggingface下载。 + +注：`hub`：表示模型仓库，`ms`为选择 modelscope 下载，`hf`为选择 huggingface 下载。 ### 实时语音识别 @@ -217,6 +229,7 @@ for i in range(total_chunk_num):
更多例子 ### 语音端点检测（非实时） + ```python from funasr import AutoModel @@ -226,10 +239,12 @@ wav_file = f"{model.model_path}/example/vad_example.wav" res = model.generate(input=wav_file) print(res) ``` -注：VAD模型输出格式为：`[[beg1, end1], [beg2, end2], .., [begN, endN]]`，其中`begN/endN`表示第`N`个有效音频片段的起始点/结束点， + +注：VAD 模型输出格式为：`[[beg1, end1], [beg2, end2], .., [begN, endN]]`，其中`begN/endN`表示第`N`个有效音频片段的起始点/结束点， 单位为毫秒。 ### 语音端点检测（实时） + ```python from funasr import AutoModel @@ -251,14 +266,17 @@ for i in range(total_chunk_num): if len(res[0]["value"]): print(res) ``` -注：流式VAD模型输出格式为4种情况： -- `[[beg1, end1], [beg2, end2], .., [begN, endN]]`：同上离线VAD输出结果。 + +注：流式 VAD 模型输出格式为 4 种情况： + +- `[[beg1, end1], [beg2, end2], .., [begN, endN]]`：同上离线 VAD 输出结果。 - `[[beg, -1]]`：表示只检测到起始点。 - `[[-1, end]]`：表示只检测到结束点。 - `[]`：表示既没有检测到起始点，也没有检测到结束点 -输出结果单位为毫秒，从起始点开始的绝对时间。 + 输出结果单位为毫秒，从起始点开始的绝对时间。 ### 标点恢复 + ```python from funasr import AutoModel @@ -269,6 +287,7 @@ print(res) ``` ### 时间戳预测 + ```python from funasr import AutoModel @@ -281,6 +300,7 @@ print(res) ``` ### 情感识别 + ```python from funasr import AutoModel @@ -297,13 +317,16 @@ print(res)
-## 导出ONNX +## 导出 ONNX + ### 从命令行导出 + ```shell funasr-export ++model=paraformer ++quantize=false ``` -### 从Python导出 +### 从 Python 导出 + ```python from funasr import AutoModel @@ -312,7 +335,8 @@ model = AutoModel(model="paraformer") res = model.export(quantize=False) ``` -### 测试ONNX +### 测试 ONNX + ```python # pip3 install -U funasr-onnx from pathlib import Path @@ -333,42 +357,43 @@ print(result) 更多例子请参考 [样例](runtime/python/onnxruntime) -## 服务部署 -FunASR支持预训练或者进一步微调的模型进行服务部署。目前支持以下几种服务部署： -- 中文离线文件转写服务（CPU版本），已完成 -- 中文流式语音识别服务（CPU版本），已完成 -- 英文离线文件转写服务（CPU版本），已完成 -- 中文离线文件转写服务（GPU版本），进行中 +## 服务部署 + +FunASR 支持预训练或者进一步微调的模型进行服务部署。目前支持以下几种服务部署： + +- 中文离线文件转写服务（CPU 版本），已完成 +- 中文流式语音识别服务（CPU 版本），已完成 +- 英文离线文件转写服务（CPU 版本），已完成 +- 中文离线文件转写服务（GPU 版本），进行中 - 更多支持中 详细信息可以参阅([服务部署文档](runtime/readme_cn.md))。 - + ## 联系我们 -如果您在使用中遇到问题，可以直接在github页面提Issues。欢迎语音兴趣爱好者扫描以下的钉钉群二维码加入社区群，进行交流和讨论。 +如果您在使用中遇到问题，可以直接在 github 页面提 Issues。欢迎语音兴趣爱好者扫描以下的钉钉群二维码加入社区群，进行交流和讨论。 -| 钉钉群 | -|:-------------------------------------------------------------------:| +| 钉钉群 | +| :-----------------------------------------------------------------: | |
| ## 社区贡献者 -|
|
|
|
|
|
| -|:------------------------------------------------------------------:|:---------------------------------------------------------------:|:--------------------------------------------------------------:|:-------------------------------------------------------:|:-----------------------------------------------------------:|:------------------------------------------------------:| +|
|
|
|
| | | +| :----------------------------------------------------------------: | :-------------------------------------------------------------: | :-----------------------------------------------------------: | :-----------------------------------------------------: | :-------------------------------------------------------: | :----------------------------------------------------: | 贡献者名单请参考（[致谢名单](./Acknowledge.md)） - ## 许可协议 -项目遵循[The MIT License](https://opensource.org/licenses/MIT)开源协议，模型许可协议请参考（[模型协议](./MODEL_LICENSE)） +项目遵循[The MIT License](https://opensource.org/licenses/MIT)开源协议，模型许可协议请参考（[模型协议](./MODEL_LICENSE)） ## 论文引用 -``` bibtex +```bibtex @inproceedings{gao2023funasr, author={Zhifu Gao and Zerui Li and Jiaming Wang and Haoneng Luo and Xian Shi and Mengzhe Chen and Yabin Li and Lingyun Zuo and Zhihao Du and Zhangyu Xiao and Shiliang Zhang}, title={FunASR: A Fundamental End-to-End Speech Recognition Toolkit}, diff --git a/funasr/models/fun_asr_nano/__init__.py b/funasr/models/fun_asr_nano/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/funasr/models/fun_asr_nano/model.py b/funasr/models/fun_asr_nano/model.py new file mode 100644 index 000000000..70f234910 --- /dev/null +++ b/funasr/models/fun_asr_nano/model.py @@ -0,0 +1,682 @@ +import json +import logging +import os +import random +import re +import string +import time +import traceback + +import torch +import torch.nn as nn +from funasr import AutoModel +from funasr.metrics.compute_acc import compute_accuracy +from funasr.register import tables +from funasr.train_utils.device_funcs import force_gatherable, to_device +from funasr.utils.datadir_writer import DatadirWriter +from funasr.utils.load_utils import extract_fbank, load_audio_text_image_video + +dtype_map = {"bf16": torch.bfloat16, "fp16": torch.float16, "fp32": torch.float32} + + +@tables.register("model_classes", "FunASRNano") +class FunASRNano(nn.Module): + def __init__( + self, + audio_encoder: str = None, + audio_encoder_conf: dict = None, + audio_adaptor: str = None, + audio_adaptor_conf: dict = None, + llm: str = None, + llm_conf: dict = None, + input_size: int = 80, + length_normalized_loss: bool = False, + **kwargs, + ): + super().__init__() + + # audio encoder + hub = audio_encoder_conf.get("hub", None) + self.audio_encoder_activation_checkpoint = audio_encoder_conf.get( + "activation_checkpoint", False + ) + if hub == "ms": + model = AutoModel(model=audio_encoder, model_revision="master") + audio_encoder_output_size = ( + model.model.encoder_output_size + if hasattr(model.model, "encoder_output_size") + else -1 + ) + audio_encoder = ( + model.model.model.encoder + if hasattr(model.model, "model") + else model.model.encoder + ) + else: + encoder_class = tables.encoder_classes.get(audio_encoder) + audio_encoder = encoder_class(input_size=input_size, **audio_encoder_conf) + audio_encoder_output_size = audio_encoder.output_size() + freeze = audio_encoder_conf.get("freeze", True) + freeze_layer_num = int(audio_encoder_conf.get("freeze_layer_num", -1)) + + if freeze: + for name, param in audio_encoder.named_parameters(): + param.requires_grad = False + audio_encoder.eval() + self.audio_encoder = audio_encoder + # llm + self.llm = None + init_param_path = llm_conf.get("init_param_path", None) + llm_dim = None + + from transformers import AutoModelForCausalLM + + llm_load_kwargs = llm_conf.get("load_kwargs", {}) + model = AutoModelForCausalLM.from_pretrained( + init_param_path, + load_in_8bit=None, + device_map=None, + use_cache=None, + **llm_load_kwargs, + ) + + freeze = llm_conf.get("freeze", True) + if freeze: + for name, param in model.named_parameters(): + param.requires_grad = False + model.eval() + logging.info(f"use_lora: {llm_conf.get('use_lora', False)}") + if llm_conf.get("use_lora", False): + from omegaconf import DictConfig, OmegaConf + + lora_conf = llm_conf.get("lora_conf", {}) + if isinstance(lora_conf, (OmegaConf, DictConfig)): + lora_conf = OmegaConf.to_container(lora_conf, resolve=True) + from peft import LoraConfig, PeftModel, get_peft_model + + lora_init_param_path = lora_conf.get("init_param_path", None) + if lora_init_param_path is not None: + logging.info(f"lora_init_param_path: {lora_init_param_path}") + model = PeftModel.from_pretrained(model, lora_init_param_path) + for name, param in model.named_parameters(): + if not lora_conf.get("freeze_lora", False): + if "lora_" in name: + param.requires_grad = True + else: + peft_config = LoraConfig(**lora_conf) + model = get_peft_model(model, peft_config) + model.print_trainable_parameters() + + if llm_conf.get("activation_checkpoint", False): + model.gradient_checkpointing_enable() + + self.llm_dtype = llm_conf.get("llm_dtype", "fp32") + self.llm = model.to(dtype_map[self.llm_dtype]) + llm_dim = model.get_input_embeddings().weight.shape[-1] + + # adaptor + adaptor_class = tables.adaptor_classes.get(audio_adaptor) + if audio_encoder_output_size > 0: + audio_adaptor_conf["encoder_dim"] = audio_encoder_output_size + audio_adaptor_conf["llm_dim"] = ( + llm_dim if llm_dim is not None else audio_adaptor_conf["llm_dim"] + ) + audio_adaptor = adaptor_class(**audio_adaptor_conf) + init_param_path = audio_adaptor_conf.get("init_param_path", None) + if init_param_path is not None: + src_state = torch.load(init_param_path, map_location="cpu") + flag = audio_adaptor.load_state_dict(src_state, strict=False) + logging.info( + f"Loading audio_adaptor ckpt: {init_param_path}, status: {flag}" + ) + freeze = audio_adaptor_conf.get("freeze", False) + if freeze: + for name, param in audio_adaptor.named_parameters(): + param.requires_grad = False + audio_adaptor.eval() + self.audio_adaptor = audio_adaptor + + self.length_normalized_loss = length_normalized_loss + self.feat_permute = audio_encoder_conf.get("feat_permute", True) + rank = int(os.environ.get("RANK", 0)) + logging.info(f"rank: {rank}, model is builded.") + + def forward( + self, + speech: torch.Tensor = None, + speech_lengths: torch.Tensor = None, + input_ids: torch.Tensor = None, + attention_mask: torch.Tensor = None, + labels_ids: torch.Tensor = None, + fbank_beg: torch.Tensor = None, + fbank_mask: torch.Tensor = None, + **kwargs, + ): + batch_size, token_num = input_ids.shape + stats = {} + input_ids[input_ids < 0] = 0 + inputs_embeds = self.llm.model.get_input_embeddings()(input_ids) + if speech is not None: + if len(speech_lengths.size()) > 1: + speech_lengths = speech_lengths[:, 0] + batch_size_speech, frames, _ = speech.shape + + # audio encoder + if self.audio_encoder_activation_checkpoint: + from torch.utils.checkpoint import checkpoint + + encoder_out, encoder_out_lens = checkpoint( + self.encode, speech, speech_lengths, use_reentrant=False + ) + else: + encoder_out, encoder_out_lens = self.encode(speech, speech_lengths) + + # audio_adaptor + encoder_out, encoder_out_lens = self.audio_adaptor( + encoder_out, encoder_out_lens + ) + + batch_size, token_num, dims = inputs_embeds.shape + fake_token_len = kwargs.get("fake_token_len") + fake_token_len[fake_token_len < 0] = 0 + fbank_beg[fbank_beg < 0] = 0 + + speech_idx = 0 + for batch_idx in range(batch_size): + for turn_id in range(fbank_beg.shape[1]): + fbank_beg_idx = fbank_beg[batch_idx, turn_id].item() + if fbank_beg_idx > 0: + speech_token_len = fake_token_len[batch_idx, turn_id] + speech_token = encoder_out[speech_idx, :speech_token_len, :] + + try: + inputs_embeds[ + batch_idx, + fbank_beg_idx : fbank_beg_idx + speech_token_len, + :, + ] = speech_token + except Exception as e: + logging.error(f"{str(e)}, {traceback.format_exc()}") + logging.info( + f"batch_idx: {batch_idx}, inputs_embeds: {inputs_embeds.shape}, fbank_beg_idx: {fbank_beg_idx}, speech_token_len: {speech_token_len}, encoder_out: {encoder_out.shape}, encoder_out_lens: {encoder_out_lens}, fake_token_len: {fake_token_len}, speech_lengths: {speech_lengths}" + ) + speech_token_len = encoder_out_lens[speech_idx].item() + speech_token = encoder_out[speech_idx, :speech_token_len, :] + inputs_embeds[ + batch_idx, + fbank_beg_idx : fbank_beg_idx + speech_token_len, + :, + ] = speech_token + + speech_idx += 1 + + stats["batch_size_speech"] = batch_size_speech + stats["batch_size_x_frames"] = frames * batch_size_speech + stats["batch_size_real_frames"] = speech_lengths.sum().item() + stats["padding_frames"] = ( + stats["batch_size_x_frames"] - stats["batch_size_real_frames"] + ) + + with torch.cuda.amp.autocast( + enabled=True if self.llm_dtype != "fp32" else False, + dtype=dtype_map[self.llm_dtype], + ): + labels_ids[labels_ids == -1] = -100 + attention_mask[attention_mask < 0] = 0 + model_outputs = self.llm( + inputs_embeds=inputs_embeds.to(dtype_map[self.llm_dtype]), + attention_mask=attention_mask, + labels=labels_ids, + ) + loss = model_outputs.loss + + with torch.no_grad(): + preds = torch.argmax(model_outputs.logits, -1) + acc_att = compute_accuracy( + preds[:, :-1], labels_ids[:, 1:], ignore_label=-100 + ) + stats["acc"] = acc_att + + stats["loss"] = torch.clone(loss.detach()) + stats["batch_size"] = batch_size + + stats["batch_size_x_tokens"] = token_num * batch_size + stats["batch_size_real_tokens"] = attention_mask.sum().item() + stats["padding_tokens"] = ( + stats["batch_size_x_tokens"] - stats["batch_size_real_tokens"] + ) + + dialog_turns = (fbank_beg > 0).sum(-1) + dialog_turns_max = torch.max(dialog_turns).int().item() + dialog_turns_avg = dialog_turns.sum().item() / batch_size + stats["dialog_turns_max"] = dialog_turns_max + stats["dialog_turns_avg"] = dialog_turns_avg + + # force_gatherable: to-device and to-tensor if scalar for DataParallel + if self.length_normalized_loss: + batch_size = int((labels_ids > 0 + 1).sum()) + loss, stats, weight = force_gatherable((loss, stats, batch_size), loss.device) + return loss, stats, weight + + def forward_export(self, speech, speech_lengths, **kwargs): + x, olens = self.audio_encoder(speech, speech_lengths) + encoder_out, encoder_out_lens = self.audio_adaptor(x, olens) + return encoder_out, encoder_out_lens + + def encode(self, speech, speech_lengths): + # audio encoder + if self.feat_permute: + encoder_out, encoder_out_lens = self.audio_encoder( + speech.permute(0, 2, 1), speech_lengths + ) + else: + encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths) + + return encoder_out, encoder_out_lens + + def data_template(self, data): + system, user, assistant = [], [], [] + for i, item in enumerate(data): + role = item["role"] + content = item["content"] + if role == "system": + system.append(content) + elif role == "user": + if "audio" in item: + audio = item["audio"] + content = [content, audio] + user.append(content) + elif role == "assistant": + assistant.append(content) + + system = system * len(user) + + contents = { + "system": system, + "user": user, + "assistant": assistant, + } + + return contents + + def data_load_speech( + self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs + ): + system = contents["system"] + user = contents["user"] + assistant = contents["assistant"] + pattern = re.compile(r"(<\|startofspeech\|>.*?<\|endofspeech\|>)") + do_think = True + sys_prompt = True + if "dataset_conf" in kwargs: + do_think = kwargs["dataset_conf"].get("do_think", True) + sys_prompt = kwargs["dataset_conf"].get("sys_prompt", True) + + input_ids, labels, fbank, fbank_lens, fbank_mask, fbank_beg, fake_token_len = ( + [], + [], + [], + [], + [], + [], + [], + ) + input_source_ids = [] + for i, (system_prompt, user_prompt, target_out) in enumerate( + zip(system, user, assistant) + ): + if i >= kwargs.get("multiturn_num_max", 5): + break + if len(input_ids) > kwargs.get("max_token_length", 1500): + break + if isinstance(user_prompt, (list, tuple)): + user_prompt, audio = user_prompt + if i == 0: + if kwargs.get("infer_with_assistant_input", False): + source_input = f"<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_prompt}" + if not sys_prompt: + source_input = f"<|im_start|>user\n{user_prompt}" + else: + source_input = f"<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n" + if not sys_prompt: + source_input = f"<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n" + else: + if kwargs.get("infer_with_assistant_input", False): + source_input = f"<|im_start|>user\n{user_prompt}" + else: + source_input = f"<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n" + if not do_think: + source_input += "\n\n\n\n" + + splits = pattern.split(source_input) + source_ids = [] + fbank_mask_i = [] + fake_token_len_i = 0 + fbank_beg_i = -1 + speech, speech_lengths = [], [] + for k, sub_str in enumerate(splits): + if not sub_str.startswith("<|startofspeech|>"): + sub_token = tokenizer.encode(sub_str) + source_ids += sub_token + fbank_mask_i += [0] * len(sub_token) + else: + sub_str = sub_str.replace("<|startofspeech|>", "").replace( + "<|endofspeech|>", "" + ) + if sub_str.startswith("!"): + sub_str = sub_str[1:] + if sub_str.startswith("!"): # !!: audio sample point + sub_str = audio + try: + time1 = time.perf_counter() + data_src = load_audio_text_image_video( + sub_str, fs=frontend.fs, **kwargs + ) + time2 = time.perf_counter() + meta_data["load_data"] = f"{time2 - time1:0.3f}" + except Exception as e: + logging.error( + f"Loading wav failed! {str(e)}, {traceback.format_exc()}" + ) + + speech, speech_lengths = extract_fbank( + data_src, + data_type=kwargs.get("data_type", "sound"), + frontend=frontend, + is_final=True, + ) # speech: [b, T, d] + + time3 = time.perf_counter() + meta_data["extract_feat"] = f"{time3 - time2:0.3f}" + meta_data["batch_data_time"] = ( + speech_lengths.sum().item() + * frontend.frame_shift + * frontend.lfr_n + / 1000 + ) + + if self.feat_permute: + speech = speech.permute(0, 2, 1) + + olens = 1 + (speech_lengths[0].item() - 3 + 2 * 1) // 2 + olens = 1 + (olens - 3 + 2 * 1) // 2 + fake_token_len_i = (olens - 1) // 2 + 1 + fake_token = [0] * fake_token_len_i + fbank_beg_i = len(source_ids) + source_ids += fake_token + fbank_mask_i += [1] * len(fake_token) + + fbank_beg += [fbank_beg_i + len(input_ids)] + fake_token_len += [fake_token_len_i] + source_mask = [-100] * len(source_ids) + target_out = f"{target_out}<|im_end|>" + target_ids = tokenizer.encode(target_out) + input_source_ids = input_ids + source_ids + input_ids += source_ids + target_ids + labels += source_mask + target_ids + fbank_mask += fbank_mask_i + if len(speech) > 0: + fbank.append(speech[0, :, :]) + fbank_lens.append(speech_lengths) + + input_ids = torch.tensor( + input_ids, dtype=torch.int64 + ) # [: self.max_token_length] + attention_mask = torch.tensor([1] * len(input_ids), dtype=torch.int32) + labels = torch.tensor(labels, dtype=torch.int64) # [: self.max_token_length] + + fbank_mask = torch.tensor(fbank_mask, dtype=torch.float32) + fbank_beg = torch.tensor(fbank_beg, dtype=torch.int32) + fake_token_len = torch.tensor(fake_token_len, dtype=torch.int32) + source_ids = torch.tensor(input_source_ids, dtype=torch.int64) + target_ids = torch.tensor(target_ids, dtype=torch.int64) + + if len(fbank) > 0: + speech = torch.nn.utils.rnn.pad_sequence( + fbank, batch_first=True, padding_value=0.0 + ) + speech_lengths = torch.nn.utils.rnn.pad_sequence( + fbank_lens, batch_first=True, padding_value=-1 + ) + else: + speech = [] + speech_lengths = [] + output = { + "speech": speech, + "speech_lengths": speech_lengths, + "fbank_mask": fbank_mask[None, :], + "fbank_beg": fbank_beg[None,], + "fake_token_len": fake_token_len[None, :], + "input_ids": input_ids[None,], + "attention_mask": attention_mask[None,], + "labels_ids": labels, + "source_ids": source_ids[None, :], + "target_ids": target_ids[None, :], + } + + return output + + def inference_prepare( + self, + data_in, + data_lengths=None, + key: list = None, + tokenizer=None, + frontend=None, + **kwargs, + ): + meta_data = {} + + if kwargs.get("batch_size", 1) > 1: + raise NotImplementedError("batch decoding is not implemented") + + contents = self.data_template(data_in[0]) + output = self.data_load_speech( + contents, tokenizer, frontend, meta_data=meta_data, **kwargs + ) + batch = to_device(output, kwargs["device"]) + + # audio encoder + speech = batch["speech"] + + if len(speech) > 0: + if "audio_embedding" in kwargs and "audio_embedding_lens" in kwargs: + encoder_out = kwargs["audio_embedding"] + encoder_out_lens = kwargs["audio_embedding_lens"] + else: + speech_lengths = batch["speech_lengths"][:, 0] + # fp16 + if kwargs.get("fp16", False): + speech = speech.to(torch.float16) + elif kwargs.get("bf16", False): + speech = speech.to(torch.bfloat16) + # audio encoder + encoder_out, encoder_out_lens = self.encode(speech, speech_lengths) + + # audio_adaptor + encoder_out, encoder_out_lens = self.audio_adaptor( + encoder_out, encoder_out_lens + ) + meta_data["audio_adaptor_out"] = encoder_out + meta_data["audio_adaptor_out_lens"] = encoder_out_lens + + input_ids = batch["input_ids"] + source_ids = batch["source_ids"] + fbank_beg = batch["fbank_beg"] + fake_token_len = batch["fake_token_len"] + + if not kwargs.get("tearchforing", False): + input_ids = source_ids + + input_ids[input_ids < 0] = 0 + inputs_embeds = self.llm.model.get_input_embeddings()(input_ids) + + batch_size, token_num, dims = inputs_embeds.shape + + fake_token_len[fake_token_len < 0] = 0 + fbank_beg[fbank_beg < 0] = 0 + + speech_idx = 0 + for batch_idx in range(batch_size): + for turn_id in range(fbank_beg.shape[1]): + fbank_beg_idx = fbank_beg[batch_idx, turn_id].item() + if fbank_beg_idx > 0: + speech_token_len = fake_token_len[batch_idx, turn_id] + speech_token = encoder_out[speech_idx, :speech_token_len, :] + + try: + inputs_embeds[ + batch_idx, + fbank_beg_idx : fbank_beg_idx + speech_token_len, + :, + ] = speech_token + except Exception as e: + # + logging.error(f"{str(e)}, {traceback.format_exc()}") + logging.info( + f"batch_idx: {batch_idx}, inputs_embeds: {inputs_embeds.shape}, fbank_beg_idx: {fbank_beg_idx}, speech_token_len: {speech_token_len}, encoder_out: {encoder_out.shape}, encoder_out_lens: {encoder_out_lens}, fake_token_len: {fake_token_len}, speech_lengths: {speech_lengths}" + ) + speech_token_len = encoder_out_lens[speech_idx].item() + speech_token = encoder_out[speech_idx, :speech_token_len, :] + inputs_embeds[ + batch_idx, + fbank_beg_idx : fbank_beg_idx + speech_token_len, + :, + ] = speech_token + + speech_idx += 1 + return inputs_embeds, contents, batch, source_ids, meta_data + + def inference( + self, + data_in, + data_lengths=None, + key: list = None, + tokenizer=None, + frontend=None, + **kwargs, + ): + new_data_in = [] + for data in data_in: + if isinstance(data, str): + new_data_in.append( + [ + {"role": "system", "content": "You are a helpful assistant."}, + { + "role": "user", + "content": f"语音转写：<|startofspeech|>!{data}<|endofspeech|>", + }, + {"role": "assistant", "content": "null"}, + ] + ) + elif isinstance(data, torch.Tensor): + new_data_in.append( + [ + {"role": "system", "content": "You are a helpful assistant."}, + { + "role": "user", + "content": f"语音转写：<|startofspeech|>!!<|endofspeech|>", + "audio": data, + }, + {"role": "assistant", "content": "null"}, + ] + ) + data_in = new_data_in + + if key is None: + key = [] + for _ in data_in: + chars = string.ascii_letters + string.digits + key.append( + "rand_key_" + "".join(random.choice(chars) for _ in range(13)) + ) + + return self.inference_llm( + data_in, + data_lengths=data_lengths, + key=key, + tokenizer=tokenizer, + frontend=frontend, + **kwargs, + ) + + def inference_llm( + self, + data_in, + data_lengths=None, + key: list = None, + tokenizer=None, + frontend=None, + **kwargs, + ): + inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare( + data_in, data_lengths, key, tokenizer, frontend, **kwargs + ) + llm_dtype = kwargs.get("llm_dtype", "fp32") + if llm_dtype == "fp32": + llm_dtype = "fp16" if kwargs.get("fp16", False) else llm_dtype + llm_dtype = "bf16" if kwargs.get("bf16", False) else llm_dtype + + with torch.cuda.amp.autocast( + enabled=True if llm_dtype != "fp32" else False, dtype=dtype_map[llm_dtype] + ): + label = contents["assistant"][-1] + self.llm = self.llm.to(dtype_map[llm_dtype]) + inputs_embeds = inputs_embeds.to(dtype_map[llm_dtype]) + llm_kwargs = kwargs.get("llm_kwargs", {}) + if not kwargs.get("teachforing", False): + generated_ids = self.llm.generate( + inputs_embeds=inputs_embeds, + max_new_tokens=kwargs.get("max_length", 512), + **llm_kwargs, + ) + + response = tokenizer.batch_decode( + generated_ids, + skip_special_tokens=kwargs.get("skip_special_tokens", True), + )[0] + + loss = None + else: + labels_ids = batch["labels_ids"] + labels_ids[labels_ids == -1] = -100 + attention_mask = batch.get("attention_mask", None) + model_outputs = self.llm( + inputs_embeds=inputs_embeds, + attention_mask=attention_mask, + labels=labels_ids, + **llm_kwargs, + ) + + preds = torch.argmax(model_outputs.logits, -1)[:, source_ids.shape[1] :] + response = tokenizer.batch_decode( + preds, + add_special_tokens=False, + skip_special_tokens=kwargs.get("skip_special_tokens", True), + )[0] + loss = model_outputs.loss.item() + + ibest_writer = None + if kwargs.get("output_dir") is not None: + if not hasattr(self, "writer"): + self.writer = DatadirWriter(kwargs.get("output_dir")) + ibest_writer = self.writer[f"{0 + 1}best_recog"] + + results = [] + response_clean = re.sub(r"[^\w\s\u3000\u4e00-\u9fff]+", "", response) + result_i = { + "key": key[0], + "text": re.sub(r'\s+', ' ', response.replace("/sil", " ")), + "text_tn": response_clean, + "label": label, + } + if loss is not None: + result_i["loss"] = loss + results.append(result_i) + + if ibest_writer is not None: + ibest_writer["text"][key[0]] = response.replace("\n", " ") + ibest_writer["label"][key[0]] = label.replace("\n", " ") + ibest_writer["text_tn"][key[0]] = response_clean + + return results, meta_data