* Remove hard torchaudio dependency for inference; add kaldi-native-fbank fbank backend
Make torchaudio an optional dependency for the inference path so FunASR
can run on Ascend NPU (aarch64) servers, where no matching torchaudio
wheel exists (Huawei's Ascend torch is version-decoupled from the PyTorch
upstream release line).
- Add funasr/utils/fbank.py: an fbank shim that delegates to
torchaudio.compliance.kaldi when torchaudio is installed, otherwise
falls back to kaldi-native-fbank (a torch-free C++ library with aarch64
wheels). Options and frame-shift API are mapped to torchaudio semantics
so outputs are equivalent.
- Switch funasr/frontends/wav_frontend.py, funasr/utils/speaker_utils.py
and funasr/models/campplus/utils.py to the shim.
- Make the torchaudio import optional in funasr/utils/load_utils.py
(resampler falls back to librosa).
No device-specific branching is introduced — a pure backend fallback that
also benefits CPU-only and other accelerator platforms.
Verified end-to-end on Ascend 910B (npu:0) with torchaudio absent:
paraformer-large loads and infers correctly, RTF ~0.016 (~60x realtime).
* Make torchaudio import optional in paraformer_v2, fun_asr_nano and dataset preprocessors
Follow-up to the inference-path change: make the remaining top-level
torchaudio imports optional so these modules import cleanly when
torchaudio is not installed (e.g. on Ascend NPU servers).
- funasr/models/paraformer_v2_community/model.py
- funasr/models/fun_asr_nano/tools/utils.py
- funasr/datasets/audio_datasets/preprocessor.py
- funasr/datasets/llm_datasets/preprocessor.py
Functions that still require torchaudio (forced_align, Resample) degrade
to a clear error only when called, rather than a hard import failure.
* Address review: declare knf extra, harden fbank fallback and torchaudio guards
Respond to maintainer review feedback.
- Declare kaldi-native-fbank as an optional 'knf' extra in setup.py; the
fbank module now raises an actionable ImportError only when fbank() is
actually called with neither torchaudio nor kaldi-native-fbank installed
(previously it failed at import time with a bare ModuleNotFoundError).
- Harden the fbank fallback contract: handle the channel argument explicitly
and reject torchaudio-only options (subtract_mean, min_duration, VTLN
warping, non-default blackman_coeff) with NotImplementedError instead of
silently dropping them via **kwargs.
- Add a shared guard funasr/utils/torchaudio_compat.py and use it for
resampling, forced alignment and speed perturbation so a missing torchaudio
fails loudly instead of returning a silently wrong result (empty alignment).
- Add regression tests (tests/test_torchaudio_optional.py) that import the
affected modules with torchaudio absent.
Decode encoded audio containers before inference while preserving the raw PCM fast path. Rewind file-like decoder fallbacks and support ffmpeg stdin decoding.
Signed-off-by: LauraGPT <[email protected]>
When an audio file path did not exist, load_audio_text_image_video silently
passed the raw string downstream, crashing later with a cryptic
"expected Tensor as element 1 in argument 0, but got str" deep inside the VAD.
New users following the README quickstart (placeholder meeting.wav) hit this
confusing internal error.
Now raise a clear FileNotFoundError naming the missing path. Scoped to
data_type=sound so text inputs are unaffected.
Tested: missing path -> FileNotFoundError with helpful message; valid path ->
unchanged.
Co-authored-by: LauraGPT <[email protected]>
1. auto_model.py: When ASR model (e.g. SenseVoice) doesn't produce
timestamps, automatically fall back to vad_segment mode for speaker
diarization instead of crashing. Fixes#2945.
2. load_utils.py: Add soundfile as fallback when torchaudio.load fails
(torchaudio>=2.11 requires torchcodec). Falls through to ffmpeg only
if soundfile also fails.
* Fix#2787: skip validate_frame_rate for raw PCM bytes in load_bytes
`validate_frame_rate` calls `pydub.AudioSegment.from_file` which takes
~200 ms. When the caller passes raw PCM bytes (no container header) the
call always fails and the exception is silently swallowed, wasting 200 ms
on every invocation.
Add a lightweight `_is_audio_container` helper that inspects the first
few magic bytes of the input. `load_bytes` now only calls
`validate_frame_rate` when the payload is a recognised container format
(WAV / MP3 / OGG / FLAC / MP4). Raw PCM buffers skip the validation
entirely, eliminating the latency hit.
Also remove a stray `import pdb` debug statement.
* Apply suggestions from code review
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Your Name <[email protected]>
Co-authored-by: zhifu gao <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
LauraGPT