mirror of
https://github.com/modelscope/FunASR.git
synced 2026-10-12 02:19:18 +00:00
docs: add migration benchmark issue template
Add a dedicated migration benchmark report template and link it from migration docs and examples.
This commit is contained in:
@@ -0,0 +1,71 @@
|
||||
---
|
||||
name: 📊 Migration Benchmark Report
|
||||
about: Share FunASR results when comparing against Whisper, OpenAI audio APIs, or cloud ASR
|
||||
labels: 'benchmark, showcase, needs triage'
|
||||
---
|
||||
|
||||
Thanks for benchmarking FunASR on your own audio. Migration reports help new users decide whether FunASR fits their language, domain, hardware, and deployment constraints.
|
||||
|
||||
If you need help debugging a failure, please use Bug Report or Deployment Help instead.
|
||||
|
||||
## Summary
|
||||
|
||||
<!-- One or two sentences: what did you compare, and what was the headline result? -->
|
||||
|
||||
## Baseline
|
||||
|
||||
- Baseline ASR (`Whisper`, `Whisper large-v3`, cloud provider, internal system, other):
|
||||
- Baseline runtime or API:
|
||||
- Baseline hardware or pricing tier:
|
||||
|
||||
## FunASR setup
|
||||
|
||||
- FunASR version:
|
||||
- Model(s):
|
||||
- Runtime path (`Python API`, `funasr-server`, `OpenAI API`, `Docker`, `WebSocket`, `vLLM`, other):
|
||||
- Device (`cuda`, `cpu`, `mps`):
|
||||
- GPU / CPU:
|
||||
- CUDA / PyTorch versions:
|
||||
- Command or script used:
|
||||
|
||||
```bash
|
||||
|
||||
```
|
||||
|
||||
## Audio set
|
||||
|
||||
- Number of files:
|
||||
- Total audio duration:
|
||||
- Language(s) / dialect(s):
|
||||
- Domain (`meeting`, `call-center`, `subtitle`, `lecture`, `media`, `noisy field audio`, other):
|
||||
- Speaker count range:
|
||||
- Sample rate / format:
|
||||
- Can any sample be shared publicly? yes/no
|
||||
|
||||
## Results
|
||||
|
||||
<!-- Paste the aggregate section from examples/migration/benchmark_funasr.py summary.md, or summarize your own measurement. -->
|
||||
|
||||
```text
|
||||
|
||||
```
|
||||
|
||||
## Quality notes
|
||||
|
||||
<!-- Share WER/CER if available, or human-review notes about names, numbers, punctuation, timestamps, speaker labels, and domain terms. -->
|
||||
|
||||
## Operational notes
|
||||
|
||||
- Model download / warmup time:
|
||||
- Steady-state throughput:
|
||||
- Memory usage:
|
||||
- Failed files or error rate:
|
||||
- Deployment blockers:
|
||||
|
||||
## Links or artifacts
|
||||
|
||||
<!-- Public repo, demo, blog, benchmark sheet, screenshot, anonymized transcript snippet, or architecture diagram. Do not include private audio, credentials, or customer data. -->
|
||||
|
||||
## What should FunASR improve next?
|
||||
|
||||
<!-- Missing docs, rough edges, model gaps, deployment friction, benchmark needs, etc. -->
|
||||
@@ -93,4 +93,4 @@ Track these fields for both the old pipeline and FunASR:
|
||||
|
||||
## Share the result
|
||||
|
||||
If FunASR replaces or complements your existing ASR stack, consider opening a [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md). Migration reports with hardware, speed, quality notes, and deployment details help new users choose the right path faster.
|
||||
If FunASR replaces or complements your existing ASR stack, consider opening a [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md) or [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md). Migration reports with hardware, speed, quality notes, and deployment details help new users choose the right path faster.
|
||||
|
||||
@@ -93,4 +93,4 @@ curl http://localhost:8000/v1/audio/transcriptions \
|
||||
|
||||
## 分享迁移结果
|
||||
|
||||
如果 FunASR 替代或补充了你的现有 ASR 栈,欢迎开一个 [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md)。包含硬件、速度、质量记录和部署细节的迁移报告,能帮助新用户更快选型。
|
||||
如果 FunASR 替代或补充了你的现有 ASR 栈,欢迎提交 [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md) 或 [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md)。包含硬件、速度、质量记录和部署细节的迁移报告,能帮助新用户更快选型。
|
||||
|
||||
@@ -78,7 +78,7 @@ Use this path when deciding whether FunASR is a good replacement for Whisper or
|
||||
|
||||
## Share your result
|
||||
|
||||
If FunASR works well in your project, consider opening a [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md) or GitHub Discussion with:
|
||||
If FunASR works well in your project, consider opening a [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md), [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md), or GitHub Discussion with:
|
||||
|
||||
- Use case and deployment mode.
|
||||
- Model, device, and processing speed.
|
||||
|
||||
@@ -78,7 +78,7 @@ curl http://localhost:8000/v1/audio/transcriptions \
|
||||
|
||||
## 分享你的结果
|
||||
|
||||
如果 FunASR 在你的项目里效果不错,欢迎通过 [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md) 或 GitHub Discussion 分享:
|
||||
如果 FunASR 在你的项目里效果不错,欢迎通过 [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md)、[Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md) 或 GitHub Discussion 分享:
|
||||
|
||||
- 使用场景和部署方式。
|
||||
- 模型、设备和处理速度。
|
||||
|
||||
@@ -55,6 +55,8 @@ python examples/migration/benchmark_funasr.py \
|
||||
- `results.jsonl`:每条音频的文本、耗时、音频时长、实时倍速和错误信息。
|
||||
- `summary.md`:运行配置、总体速度、逐文件预览和下一步对比建议。
|
||||
|
||||
如果结果可以公开,欢迎提交 [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md),帮助其他用户参考你的硬件、音频领域和质量记录。
|
||||
|
||||
## What to compare
|
||||
|
||||
Track the same fields for your old ASR stack and FunASR:
|
||||
@@ -67,4 +69,4 @@ Track the same fields for your old ASR stack and FunASR:
|
||||
| WER/CER or human review notes | Captures quality beyond speed. |
|
||||
| Failed-file rate and error messages | Shows operational risk before rollout. |
|
||||
|
||||
See the [migration guide](../../docs/migration_from_whisper.md) for the full evaluation and rollout checklist.
|
||||
See the [migration guide](../../docs/migration_from_whisper.md) for the full evaluation and rollout checklist. If you can share results publicly, open a [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md) so others can learn from your hardware, audio domain, and quality notes.
|
||||
|
||||
Reference in New Issue
Block a user