mirror of
https://github.com/modelscope/FunASR.git
synced 2026-10-12 02:19:18 +00:00
docs: add migration benchmark example
Add a runnable migration benchmark example for representative FunASR audio evaluation and link it from README/docs.
This commit is contained in:
zhifu gao
@@ -66,7 +66,7 @@ That's it. **One model, one call** — VAD segmentation, speech recognition, pun
|
||||
| Cost | Free | Free | $0.006/min+ |
|
||||
| CPU viable | ✅ 17x realtime | ❌ Too slow | N/A |
|
||||
|
||||
Planning a switch from Whisper or a cloud ASR provider? Use the [migration guide](./docs/migration_from_whisper.md) to benchmark representative audio, map features, and roll out safely.
|
||||
Planning a switch from Whisper or a cloud ASR provider? Use the [migration guide](./docs/migration_from_whisper.md) and [benchmark example](./examples/migration/) to test representative audio, map features, and roll out safely.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+1
-1
@@ -65,7 +65,7 @@ result = model.generate(input="meeting.wav")
|
||||
| 费用 | 免费 | 免费 | ¥0.04/分钟起 |
|
||||
| CPU 可用 | ✅ 17 倍实时 | ❌ 太慢 | 不适用 |
|
||||
|
||||
计划从 Whisper 或云端 ASR 切换？请按 [迁移指南](./docs/migration_from_whisper_zh.md) 用代表性音频评测、映射功能并安全上线。
|
||||
计划从 Whisper 或云端 ASR 切换？请按 [迁移指南](./docs/migration_from_whisper_zh.md) 和 [评测示例](./examples/migration/) 用代表性音频评测、映射功能并安全上线。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ Stay on your current pipeline if you need a managed service with no operations w
|
||||
|
||||
1. Pick 20-50 representative audio files. Include short clips, long recordings, noisy samples, different speakers, and the languages or dialects you care about.
|
||||
2. Run your current Whisper or cloud ASR pipeline exactly as you use it in production. Save transcripts, latency, cost, and failure cases.
|
||||
3. Run FunASR locally with the README quick start, then choose a deployment path from the [deployment matrix](./deployment_matrix.md).
|
||||
3. Run FunASR locally with the README quick start, or use the [migration benchmark example](../examples/migration/) to measure a representative audio folder. Then choose a deployment path from the [deployment matrix](./deployment_matrix.md).
|
||||
4. Compare output with human review or your normal WER/CER process. Do not compare only one clean demo file.
|
||||
5. Run the OpenAI-compatible API smoke test if your application already uses OpenAI-style clients.
|
||||
6. Record warmup time, model download time, device, GPU/CPU type, batch size, and audio duration separately from steady-state throughput.
|
||||
@@ -69,7 +69,7 @@ curl http://localhost:8000/v1/audio/transcriptions \
|
||||
-F response_format=verbose_json
|
||||
```
|
||||
|
||||
If you want a repeatable container smoke test, start from `examples/openai_api/docker-compose.yml` and verify it with `BASE_URL=http://localhost:8000 bash examples/openai_api/smoke_test.sh`.
|
||||
If you want a repeatable folder-level benchmark, run [`examples/migration/benchmark_funasr.py`](../examples/migration/benchmark_funasr.py) to produce `results.jsonl` and `summary.md` for your own audio set. For a container smoke test, start from `examples/openai_api/docker-compose.yml` and verify it with `BASE_URL=http://localhost:8000 bash examples/openai_api/smoke_test.sh`.
|
||||
|
||||
## Quality and speed checklist
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
1. 选择 20-50 条有代表性的音频，覆盖短音频、长录音、噪声、多说话人、目标语言和方言。
|
||||
2. 按生产方式运行当前 Whisper 或云端 ASR，保存转写结果、延迟、成本和失败样例。
|
||||
3. 用 README 快速开始跑 FunASR，再根据 [部署选型表](./deployment_matrix_zh.md) 选择服务路径。
|
||||
3. 用 README 快速开始跑 FunASR，也可以用 [迁移评测示例](../examples/migration/) 测量一组代表性音频。然后根据 [部署选型表](./deployment_matrix_zh.md) 选择服务路径。
|
||||
4. 用人工审阅或现有 WER/CER 流程比较结果，不要只看一个干净 demo 音频。
|
||||
5. 如果应用已经使用 OpenAI 风格客户端，运行 OpenAI 兼容 API smoke test。
|
||||
6. 分开记录 warmup、模型下载、设备、GPU/CPU 型号、batch size、音频时长和稳定吞吐。
|
||||
@@ -69,7 +69,7 @@ curl http://localhost:8000/v1/audio/transcriptions \
|
||||
-F response_format=verbose_json
|
||||
```
|
||||
|
||||
如果需要可复现的容器 smoke test，可以从 `examples/openai_api/docker-compose.yml` 开始，并用 `BASE_URL=http://localhost:8000 bash examples/openai_api/smoke_test.sh` 验证。
|
||||
如果要对自己的音频目录做可复现评测，可以运行 [`examples/migration/benchmark_funasr.py`](../examples/migration/benchmark_funasr.py) 生成 `results.jsonl` 和 `summary.md`。如果需要容器 smoke test，可以从 `examples/openai_api/docker-compose.yml` 开始，并用 `BASE_URL=http://localhost:8000 bash examples/openai_api/smoke_test.sh` 验证。
|
||||
|
||||
## 质量与速度检查清单
|
||||
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# FunASR Migration Benchmark Example
|
||||
|
||||
Use this example when you are comparing FunASR with Whisper, OpenAI audio APIs, or a cloud ASR provider. It runs FunASR over your representative audio set and writes machine-readable results plus a Markdown summary.
|
||||
|
||||
The script does not claim accuracy by itself. Run your baseline on the same files, then compare transcripts with human review or your normal WER/CER workflow.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
python examples/migration/benchmark_funasr.py \
|
||||
--input /path/to/audio_samples \
|
||||
--recursive \
|
||||
--model iic/SenseVoiceSmall \
|
||||
--device cuda \
|
||||
--spk-model cam++ \
|
||||
--output-dir outputs/funasr_migration_eval \
|
||||
--metadata baseline=whisper-large-v3
|
||||
```
|
||||
|
||||
Outputs:
|
||||
|
||||
- `results.jsonl`: one JSON object per audio file with text, elapsed seconds, audio duration, realtime factor, model, device, and errors.
|
||||
- `summary.md`: run configuration, aggregate speed, per-file previews, and next comparison steps.
|
||||
|
||||
## CPU smoke test
|
||||
|
||||
For a portable first check, use CPU and a small audio folder:
|
||||
|
||||
```bash
|
||||
python examples/migration/benchmark_funasr.py \
|
||||
--input ./samples \
|
||||
--model iic/SenseVoiceSmall \
|
||||
--device cpu \
|
||||
--output-dir outputs/funasr_cpu_smoke
|
||||
```
|
||||
|
||||
|
||||
## 中文快速说明
|
||||
|
||||
这个示例用于从 Whisper、OpenAI 音频 API 或云端 ASR 迁移前的本地评测。请用同一批代表性音频分别跑旧方案和 FunASR，再用人工审阅或 WER/CER 流程比较质量。
|
||||
|
||||
```bash
|
||||
python examples/migration/benchmark_funasr.py \
|
||||
--input /path/to/audio_samples \
|
||||
--recursive \
|
||||
--model iic/SenseVoiceSmall \
|
||||
--device cuda \
|
||||
--spk-model cam++ \
|
||||
--output-dir outputs/funasr_migration_eval \
|
||||
--metadata baseline=whisper-large-v3
|
||||
```
|
||||
|
||||
输出文件：
|
||||
|
||||
- `results.jsonl`：每条音频的文本、耗时、音频时长、实时倍速和错误信息。
|
||||
- `summary.md`：运行配置、总体速度、逐文件预览和下一步对比建议。
|
||||
|
||||
## What to compare
|
||||
|
||||
Track the same fields for your old ASR stack and FunASR:
|
||||
|
||||
| Field | Why it matters |
|
||||
|---|---|
|
||||
| Audio duration, language, domain, sample rate, speaker count | Keeps the comparison representative. |
|
||||
| Model name, version, device, CUDA/PyTorch versions | Makes results reproducible. |
|
||||
| Model load time vs inference time | Separates cold start from steady-state throughput. |
|
||||
| WER/CER or human review notes | Captures quality beyond speed. |
|
||||
| Failed-file rate and error messages | Shows operational risk before rollout. |
|
||||
|
||||
See the [migration guide](../../docs/migration_from_whisper.md) for the full evaluation and rollout checklist.
|
||||
@@ -0,0 +1,278 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Benchmark FunASR on representative audio during ASR migration.
|
||||
|
||||
The script intentionally measures only the FunASR side of a migration test. Run
|
||||
Whisper or a cloud ASR baseline separately, then compare transcripts with your
|
||||
normal WER/CER or human-review process.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import wave
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Optional
|
||||
|
||||
try:
|
||||
from funasr import AutoModel
|
||||
from funasr.utils.postprocess_utils import rich_transcription_postprocess
|
||||
except Exception as exc: # pragma: no cover - import message is for users
|
||||
print(
|
||||
"Failed to import FunASR. Install it with `pip install -U funasr` "
|
||||
"or run this script from the repository root.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
raise
|
||||
|
||||
AUDIO_EXTENSIONS = (
|
||||
".wav",
|
||||
".mp3",
|
||||
".flac",
|
||||
".m4a",
|
||||
".aac",
|
||||
".ogg",
|
||||
".opus",
|
||||
".wma",
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run FunASR over a representative audio set and write migration benchmark results."
|
||||
)
|
||||
parser.add_argument("--input", "-i", type=Path, required=True, help="Audio file or folder to benchmark.")
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
"-o",
|
||||
type=Path,
|
||||
default=Path("outputs/migration_benchmark"),
|
||||
help="Directory for results.jsonl and summary.md.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
"-m",
|
||||
default="iic/SenseVoiceSmall",
|
||||
help="FunASR model name or ModelScope/Hugging Face id.",
|
||||
)
|
||||
parser.add_argument("--device", "-d", default="cpu", help="Inference device: cpu, cuda, or mps.")
|
||||
parser.add_argument("--vad-model", default="fsmn-vad", help="VAD model, or 'none' to disable.")
|
||||
parser.add_argument("--spk-model", default="", help="Optional speaker model such as cam++.")
|
||||
parser.add_argument("--language", default="auto", help="Language hint passed to model.generate.")
|
||||
parser.add_argument("--batch-size", type=int, default=1, help="batch_size passed to model.generate.")
|
||||
parser.add_argument("--recursive", "-r", action="store_true", help="Recursively scan input folders.")
|
||||
parser.add_argument(
|
||||
"--extensions",
|
||||
nargs="+",
|
||||
default=list(AUDIO_EXTENSIONS),
|
||||
help="Audio extensions to include when --input is a folder.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--metadata",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Free-form key=value metadata to copy into the summary, e.g. --metadata baseline=whisper-large-v3.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def normalize_extensions(values: Iterable[str]) -> List[str]:
|
||||
return sorted({value.lower() if value.startswith(".") else f".{value.lower()}" for value in values})
|
||||
|
||||
|
||||
def iter_audio_files(path: Path, extensions: List[str], recursive: bool) -> List[Path]:
|
||||
if path.is_file():
|
||||
return [path]
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"Input path does not exist: {path}")
|
||||
walker = path.rglob("*") if recursive else path.iterdir()
|
||||
return sorted(p for p in walker if p.is_file() and p.suffix.lower() in extensions)
|
||||
|
||||
|
||||
def audio_duration_seconds(path: Path) -> Optional[float]:
|
||||
try:
|
||||
import soundfile as sf
|
||||
|
||||
info = sf.info(str(path))
|
||||
if info.samplerate:
|
||||
return float(info.frames) / float(info.samplerate)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if path.suffix.lower() == ".wav":
|
||||
try:
|
||||
with wave.open(str(path), "rb") as wav_file:
|
||||
frames = wav_file.getnframes()
|
||||
rate = wav_file.getframerate()
|
||||
if rate:
|
||||
return float(frames) / float(rate)
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def metadata_dict(items: Iterable[str]) -> Dict[str, str]:
|
||||
parsed: Dict[str, str] = {}
|
||||
for item in items:
|
||||
if "=" not in item:
|
||||
parsed[item] = ""
|
||||
continue
|
||||
key, value = item.split("=", 1)
|
||||
parsed[key.strip()] = value.strip()
|
||||
return parsed
|
||||
|
||||
|
||||
def extract_text(result: Any) -> str:
|
||||
if not result:
|
||||
return ""
|
||||
first = result[0] if isinstance(result, list) else result
|
||||
if isinstance(first, dict):
|
||||
text = first.get("text", "")
|
||||
else:
|
||||
text = str(first)
|
||||
try:
|
||||
return rich_transcription_postprocess(text)
|
||||
except Exception:
|
||||
return text
|
||||
|
||||
|
||||
def write_jsonl(path: Path, rows: Iterable[Dict[str, Any]]) -> None:
|
||||
with path.open("w", encoding="utf-8") as handle:
|
||||
for row in rows:
|
||||
handle.write(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def markdown_summary(rows: List[Dict[str, Any]], args: argparse.Namespace, model_load_seconds: float) -> str:
|
||||
successful = [row for row in rows if not row.get("error")]
|
||||
failed = [row for row in rows if row.get("error")]
|
||||
known_audio = [row for row in successful if row.get("duration_seconds")]
|
||||
total_audio = sum(float(row["duration_seconds"]) for row in known_audio)
|
||||
total_elapsed = sum(float(row["elapsed_seconds"]) for row in successful)
|
||||
throughput = total_audio / total_elapsed if total_audio and total_elapsed else None
|
||||
meta = metadata_dict(args.metadata)
|
||||
|
||||
lines = [
|
||||
"# FunASR Migration Benchmark Summary",
|
||||
"",
|
||||
"## Run configuration",
|
||||
"",
|
||||
f"- Input: `{args.input}`",
|
||||
f"- Model: `{args.model}`",
|
||||
f"- Device: `{args.device}`",
|
||||
f"- VAD model: `{args.vad_model}`",
|
||||
f"- Speaker model: `{args.spk_model or 'none'}`",
|
||||
f"- Language: `{args.language}`",
|
||||
f"- Batch size: `{args.batch_size}`",
|
||||
f"- Model load seconds: `{model_load_seconds:.3f}`",
|
||||
]
|
||||
for key, value in meta.items():
|
||||
lines.append(f"- {key}: `{value}`")
|
||||
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Aggregate results",
|
||||
"",
|
||||
f"- Files: `{len(rows)}`",
|
||||
f"- Successful: `{len(successful)}`",
|
||||
f"- Failed: `{len(failed)}`",
|
||||
f"- Known audio seconds: `{total_audio:.3f}`" if known_audio else "- Known audio seconds: `unknown`",
|
||||
f"- Inference seconds: `{total_elapsed:.3f}`" if successful else "- Inference seconds: `0.000`",
|
||||
f"- Aggregate realtime factor: `{throughput:.3f}x`" if throughput else "- Aggregate realtime factor: `unknown`",
|
||||
"",
|
||||
"## Per-file results",
|
||||
"",
|
||||
"| File | Audio sec | Inference sec | RTF | Status | Text preview |",
|
||||
"|---|---:|---:|---:|---|---|",
|
||||
]
|
||||
)
|
||||
for row in rows:
|
||||
duration = row.get("duration_seconds")
|
||||
elapsed = row.get("elapsed_seconds")
|
||||
rtf = row.get("realtime_factor")
|
||||
status = "error" if row.get("error") else "ok"
|
||||
preview = (row.get("text") or row.get("error") or "").replace("|", "\\|").replace("\n", " ")[:120]
|
||||
lines.append(
|
||||
"| {file} | {duration} | {elapsed} | {rtf} | {status} | {preview} |".format(
|
||||
file=row["input"],
|
||||
duration=f"{duration:.3f}" if isinstance(duration, (int, float)) else "",
|
||||
elapsed=f"{elapsed:.3f}" if isinstance(elapsed, (int, float)) else "",
|
||||
rtf=f"{rtf:.3f}x" if isinstance(rtf, (int, float)) else "",
|
||||
status=status,
|
||||
preview=preview,
|
||||
)
|
||||
)
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Next comparison steps",
|
||||
"",
|
||||
"- Run your Whisper or cloud ASR baseline on the same files.",
|
||||
"- Compare transcripts with human review or your normal WER/CER workflow.",
|
||||
"- Keep model download and warmup time separate from steady-state throughput.",
|
||||
"- Share reproducible findings in a FunASR showcase issue when possible.",
|
||||
"",
|
||||
]
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
extensions = normalize_extensions(args.extensions)
|
||||
files = iter_audio_files(args.input, extensions, args.recursive)
|
||||
if not files:
|
||||
print(f"No audio files found under {args.input} for extensions {extensions}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
vad_model = None if args.vad_model.lower() == "none" else args.vad_model
|
||||
model_kwargs: Dict[str, Any] = {"model": args.model, "vad_model": vad_model, "device": args.device}
|
||||
if args.spk_model:
|
||||
model_kwargs["spk_model"] = args.spk_model
|
||||
|
||||
print(f"Loading FunASR model: {args.model} on {args.device}")
|
||||
load_start = time.perf_counter()
|
||||
model = AutoModel(**model_kwargs)
|
||||
model_load_seconds = time.perf_counter() - load_start
|
||||
print(f"Model loaded in {model_load_seconds:.3f}s")
|
||||
|
||||
rows: List[Dict[str, Any]] = []
|
||||
for index, audio_path in enumerate(files, start=1):
|
||||
display = str(audio_path if args.input.is_file() else audio_path.relative_to(args.input))
|
||||
print(f"[{index}/{len(files)}] {display}")
|
||||
duration = audio_duration_seconds(audio_path)
|
||||
start = time.perf_counter()
|
||||
row: Dict[str, Any] = {
|
||||
"input": display,
|
||||
"path": str(audio_path),
|
||||
"duration_seconds": duration,
|
||||
"model": args.model,
|
||||
"device": args.device,
|
||||
"language": args.language,
|
||||
}
|
||||
try:
|
||||
result = model.generate(input=str(audio_path), language=args.language, batch_size=args.batch_size)
|
||||
elapsed = time.perf_counter() - start
|
||||
row["elapsed_seconds"] = elapsed
|
||||
row["realtime_factor"] = (duration / elapsed) if duration and elapsed else None
|
||||
row["text"] = extract_text(result)
|
||||
print(f" ok: {elapsed:.3f}s" + (f", {row['realtime_factor']:.3f}x" if row["realtime_factor"] else ""))
|
||||
except Exception as exc: # keep benchmarking other files
|
||||
elapsed = time.perf_counter() - start
|
||||
row["elapsed_seconds"] = elapsed
|
||||
row["realtime_factor"] = None
|
||||
row["error"] = repr(exc)
|
||||
print(f" error: {exc}", file=sys.stderr)
|
||||
rows.append(row)
|
||||
|
||||
args.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
results_path = args.output_dir / "results.jsonl"
|
||||
summary_path = args.output_dir / "summary.md"
|
||||
write_jsonl(results_path, rows)
|
||||
summary_path.write_text(markdown_summary(rows, args, model_load_seconds), encoding="utf-8")
|
||||
print(f"\nWrote {results_path}")
|
||||
print(f"Wrote {summary_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user