Synchronize the bilingual current-public fixtures, evidence assertion, and valid English fixture with the GPT-SoVITS #2824 contract. Source commit was signed and carried a DCO trailer; focused and related contract tests passed.
* Remove hard torchaudio dependency for inference; add kaldi-native-fbank fbank backend
Make torchaudio an optional dependency for the inference path so FunASR
can run on Ascend NPU (aarch64) servers, where no matching torchaudio
wheel exists (Huawei's Ascend torch is version-decoupled from the PyTorch
upstream release line).
- Add funasr/utils/fbank.py: an fbank shim that delegates to
torchaudio.compliance.kaldi when torchaudio is installed, otherwise
falls back to kaldi-native-fbank (a torch-free C++ library with aarch64
wheels). Options and frame-shift API are mapped to torchaudio semantics
so outputs are equivalent.
- Switch funasr/frontends/wav_frontend.py, funasr/utils/speaker_utils.py
and funasr/models/campplus/utils.py to the shim.
- Make the torchaudio import optional in funasr/utils/load_utils.py
(resampler falls back to librosa).
No device-specific branching is introduced — a pure backend fallback that
also benefits CPU-only and other accelerator platforms.
Verified end-to-end on Ascend 910B (npu:0) with torchaudio absent:
paraformer-large loads and infers correctly, RTF ~0.016 (~60x realtime).
* Make torchaudio import optional in paraformer_v2, fun_asr_nano and dataset preprocessors
Follow-up to the inference-path change: make the remaining top-level
torchaudio imports optional so these modules import cleanly when
torchaudio is not installed (e.g. on Ascend NPU servers).
- funasr/models/paraformer_v2_community/model.py
- funasr/models/fun_asr_nano/tools/utils.py
- funasr/datasets/audio_datasets/preprocessor.py
- funasr/datasets/llm_datasets/preprocessor.py
Functions that still require torchaudio (forced_align, Resample) degrade
to a clear error only when called, rather than a hard import failure.
* Address review: declare knf extra, harden fbank fallback and torchaudio guards
Respond to maintainer review feedback.
- Declare kaldi-native-fbank as an optional 'knf' extra in setup.py; the
fbank module now raises an actionable ImportError only when fbank() is
actually called with neither torchaudio nor kaldi-native-fbank installed
(previously it failed at import time with a bare ModuleNotFoundError).
- Harden the fbank fallback contract: handle the channel argument explicitly
and reject torchaudio-only options (subtract_mean, min_duration, VTLN
warping, non-default blackman_coeff) with NotImplementedError instead of
silently dropping them via **kwargs.
- Add a shared guard funasr/utils/torchaudio_compat.py and use it for
resampling, forced alignment and speed perturbation so a missing torchaudio
fails loudly instead of returning a silently wrong result (empty alignment).
- Add regression tests (tests/test_torchaudio_optional.py) that import the
affected modules with torchaudio absent.
Use the non-deprecated torch.amp APIs when available while preserving the torch.cuda.amp fallback and legacy autocast argument semantics on older PyTorch releases.
Resolve distributed trainer settings once with top-level precedence and backwards-compatible train_conf fallbacks. Remove consumed mode keys before expanding trainer_conf, reject simultaneous FSDP and DeepSpeed, and cover the behavior with focused tests.
The tests merged in #3456 were flaky: the state-dict round-trip test draws from
the unseeded global RNG, and the merged vs unmerged fp32 forwards can differ by
~4e-5 (different op orderings), above torch.allclose's default 1e-5 rtol. The
test failed on roughly one in three full-file runs.
- Seed the RNG per test so every test is reproducible.
- Compare merge/unmerge at a fp32-appropriate rtol=1e-3. A genuinely broken
merge differs by the full adapter term (O(1-10)), not ~1e-5, so the tolerance
is still discriminating (verified: a 'mark merged but don't fold' bug yields
max_abs_diff ~12).
- Add test_mark_only_lora_as_trainable_keeps_adapters_trainable, which exercises
the lora_only=true flow (mark_only_lora_as_trainable in train.py/train_ds.py)
that the previous reply described but did not actually test.
Verified: 8/8 pass, stable across repeated runs; the full nano set is 35 passed.
Co-authored-by: Claude <[email protected]>
* feat(nano): support LoRA finetuning of the Qwen3 LLM
The FunASR-Nano model config already declares llm_conf.use_lora and
llm_conf.lora_conf (r/lora_alpha/lora_dropout/target_modules), but nothing
consumed them: use_lora was silently ignored and training fell back to the
full-parameter LLM path. Wire it up:
- When llm_conf.use_lora is true, replace the LLM target Linear layers
(default q_proj/v_proj) with lora.Linear adapters, sharing the frozen base
weight and adding trainable lora_A/lora_B. The adapter params are created in
the base weight's dtype (bf16), so the LoRA path does not depend on autocast.
- Pairs with the existing lora_only / mark_only_lora_as_trainable flow in
train.py / train_ds.py: lora_only=true freezes every non-LoRA parameter for
pure-LoRA training; lora_only=false + unfrozen encoder/adaptor conf keeps
them trainable while LoRA-tweaking only the LLM.
- Checkpoints save the full state dict (unchanged base + adapter params), so a
LoRA run can be resumed or decoded with the same use_lora=true config; for a
standalone deployment checkpoint, fold with W' = W + alpha/r * B @ A.
- Add lora_finetune.sh and document the recipe in finetune.md / finetune_zh.md.
Validated end-to-end with the real model: 56 adapters injected (28 Qwen3
layers x q_proj+v_proj), optimizer updates only the 112 LoRA params, and a
2-step smoke training run through funasr-train-ds completes and saves a
checkpoint whose lora keys are bf16 and match the base dtype.
Co-Authored-By: Claude <[email protected]>
* test(nano): add offline unit tests for LoRA injection
Drive FunASRNano._apply_lora_to_llm against a tiny fake LLM mirroring the
Qwen3 parameter layout (layers.<i>.self_attn.{q,k,v,o}_proj / mlp.{gate,up,down}_proj).
No model download needed.
Covers the points requested in review:
1. only configured target modules are replaced;
2. the initial output matches the original linear layers (lora_B init zero);
3. base weights stay frozen while lora_A/lora_B are trainable (llm_conf.freeze
applied before injection, as in the real init path);
4. a non-zero adapter survives a state-dict save/load round trip and train/eval
transitions (including the eval-merge / train-unmerge convention);
5. the no-matching-target configuration is an explicit no-op with a warning.
Verified honest: on upstream main (no _apply_lora_to_llm) all 7 fail with
AttributeError; with the fix 7/7 pass. Full nano set: 34 passed.
Co-Authored-By: Claude <[email protected]>
---------
Co-authored-by: Claude <[email protected]>
LauraGPT