modelscope/FunASR
1
0
Fork 0
mirror of https://github.com/modelscope/FunASR.git synced 2026-10-12 02:19:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #3663 from modelscope/codex/historical-benchmark-docs-20260907

Browse Source 
docs: maintain bilingual historical benchmark sources and routes
This commit is contained in:
zhifu gao
2026-09-07 21:17:43 +08:00
committed by GitHub
parent 386f6f9106 e7adf031a2
commit dd79f9dd5a
13 changed files with 1147 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+1
View File
@@ -51,6 +51,7 @@ interfaces. Choose the matching protocol before selecting a client.
## Evaluate and Operate
- [Historical ASR benchmark](benchmark/historical_asr.md) / [中文](benchmark/historical_asr_zh.md): preserved results and provenance limits
- [Performance methodology](benchmark/rtf_reproducibility.md) and [realtime WebSocket benchmarks](benchmark/realtime_ws_benchmark.md)
- [Troubleshooting](troubleshooting.md) / [中文](troubleshooting_zh.md)
- [Use cases](use_case_showcase.md), [community integrations](community_projects.md), and [repository responsibilities](repository_roles.md)
docs/benchmark/historical_asr.md
+113
View File
@@ -0,0 +1,113 @@
# Historical ASR Benchmark
[中文](historical_asr_zh.md)
This is a historical record with incomplete provenance, retained for readers of
earlier FunASR comparisons. It is not a new measurement, a universal leaderboard,
or a guarantee for a current checkpoint, machine or deployment. For a new
evaluation, start with the [performance methodology](rtf_reproducibility.md).
## Historical Summary
The table below preserves the original report's wording and numbers. Its
"best" labels refer only to that report, not to all available models or hardware.
| Metric | Result |
| --- | --- |
| Dataset | 184 long-form Chinese audio files, 11,539 s total, 192.3 min. |
| GPU | NVIDIA H100 80GB HBM3. |
| Best GPU speed | SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run. |
| Best CPU speed | SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime. |
| Baseline | OpenAI Whisper-large-v3: 13.4x realtime on GPU. |
The **169.6x full run and 211.8x initial run are separate reported results**.
The original page does not disclose the measurement date. The source was
checked on **2026-09-07**, which is a snapshot audit date, not the measurement date.
## Historical Results
All values and notes in this table are archived claims from the original report.
The notes are not current API capability guarantees. In particular, a model's
raw tags do not imply that an HTTP endpoint returns those tags, and an old
timestamp limitation must not replace the current [model selection guide](../model_selection.md).
| Model | Device | RTF | Speed | CER | Notes |
| --- | --- | --- | --- | --- | --- |
| SenseVoice-Small | GPU | 0.005896 | 169.6x | 7.81% | ASR + language / emotion / event tags; CER after tag stripping. |
| Paraformer-Large | GPU | 0.008359 | 119.6x | 10.18% | Fast non-autoregressive Chinese ASR with VAD/punctuation pipeline. |
| Fun-ASR-Nano | GPU | 0.058803 | 17.0x | 8.06% | LLM-based ASR for Chinese, English, Japanese, seven Chinese dialect groups, and 26 regional accents; supports hotwords. Reliable checkpoint-native timestamps are not available ([#106](https://github.com/QwenAudio/Fun-ASR/issues/106)). |
| GLM-ASR-Nano | GPU | 0.026974 | 37.1x | 31.07% | LLM-based multilingual ASR. |
| Whisper-large-v3-turbo (OpenAI) | GPU | 0.021708 | 46.1x | 21.71% | OpenAI Whisper implementation. |
| Whisper-large-v3 (OpenAI) | GPU | 0.074694 | 13.4x | 20.02% | Baseline for large Whisper quality. |
| SenseVoice-Small | CPU | 0.057988 | 17.2x | 7.81% | CPU run from the remaining benchmark script. |
| Paraformer-Large | CPU | 0.064056 | 15.6x | 10.18% | CPU viable for batch jobs. |
| Fun-ASR-Nano | CPU | 0.274318 | 3.6x | 8.06% | LLM-based model is heavier but still above realtime. |
The repeated CPU/GPU CER values do not establish independent per-device scoring.
The raw predictions, references and scoring program are not available in the
audited record. The tag-stripping statement is preserved as a historical claim,
not as newly verified scorer output. Numerical precision and rounded speed/RTF
pairs are retained without recalculation.
## Provenance and Limitations
The [original English HTML](https://github.com/modelscope/FunASR/blob/67d63b80a246dc33749e43904c294e0409cd9183/benchmark.html)
is pinned to its historical GitHub Pages commit. Its bytes matched the archived
public page during the source audit. That establishes the source of this table,
not the correctness or reproducibility of the underlying measurements.
The old report describes RTF as total inference time divided by total audio
duration, and speed as its reciprocal. The latter is also called RTFx:
```text
RTF = total inference time / total audio duration
RTFx = total audio duration / total inference time = 1 / RTF
```
The following commands are **historical text and cannot be run directly from
the audited checkout**. All three referenced files were absent at FunASR source
revision `386f6f9106684ba5a114e796147db4396a09eab5`; no replacement scripts or
reproduction data are supplied by this document.
```text
python benchmark/run_full_benchmark.py
python benchmark/run_remaining.py
python benchmark/fix_sensevoice_cer.py
```
The audited report does not provide a CPU model/thread count, dataset membership
and reference manifest, exact checkpoint revisions, software/driver versions,
per-file predictions or timing logs, or a complete timing scope covering warmup,
I/O and preprocessing. Without these materials, the old table is not directly
reproducible and its CPU-versus-GPU headline is not a production-wide guarantee.
This record's **11,539 seconds** must remain separate from the **11,541 seconds**
reported in the [vLLM methodology](rtf_reproducibility.md). Both mention 184 files,
but that alone does not prove identical dataset membership. Do not merge their
rows or silently normalize the two-second discrepancy.
## Choosing a Current Path
The following is the **original recommendation table, retained as historical
context only**. It is not a newly validated recommendation or performance ranking.
| Need | Recommended model |
| --- | --- |
| Fastest production transcription | SenseVoice-Small or Paraformer-Large. |
| CPU batch transcription | SenseVoice-Small first; Paraformer-Large for Chinese production pipelines. |
| Chinese/English/Japanese LLM-style recognition with dialect and accent coverage | Fun-ASR-Nano; use the separate [Fun-ASR-MLT-Nano](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512) checkpoint for 31 languages, and use [vLLM](../vllm_guide.md) for higher LLM decoding throughput. |
| OpenAI-compatible local endpoint | [funasr-server](../agent_integration.md) with model alias `sensevoice`, `paraformer`, or `fun-asr-nano`. |
For current decisions, use the [model and capability guide](../model_selection.md),
[Agent integration contracts](../agent_integration.md), and
[vLLM deployment guide](../vllm_guide.md). The separate MLT checkpoint's language
coverage must not be attributed to Fun-ASR-Nano. Evaluate your own audio, runtime
and end-to-end latency before selecting a deployment.
Use the [performance methodology](rtf_reproducibility.md) for new measurements
and the [WebSocket benchmark guide](realtime_ws_benchmark.md) for concurrent
realtime services. The current
[migration timing helper](../../examples/migration/benchmark_funasr.py) measures
FunASR on your own audio; it does not compute CER/WER, run Whisper or reproduce
the missing historical scripts. Keep your timing scope, failed files and quality
evaluation separate and explicit.
docs/benchmark/historical_asr_zh.md
+98
View File
@@ -0,0 +1,98 @@
# 历史 ASR 评测
[English](historical_asr.md)
这是一份**来源不完整的历史记录**，用于帮助读者理解早期 FunASR 对比结果。
它不是新测量、通用排行榜，也不是对当前 checkpoint、机器或部署的保证。
开展新的评测请先阅读[性能评测方法](rtf_reproducibility.md)。
## 历史概览
下表保留原报告的措辞与数字。“最佳”等标签仅指该份报告，不能推广到所有模型或硬件。
| 指标 | 结果 |
| --- | --- |
| 数据集 | 184 条中文长音频，总时长 11,539 秒，约 192.3 分钟。 |
| GPU | NVIDIA H100 80GB HBM3. |
| 最佳 GPU 速度 | SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run. |
| 最佳 CPU 速度 | SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime. |
| 基线 | OpenAI Whisper-large-v3：GPU 上 13.4 倍实时。 |
**完整运行的 169.6x 与初次运行的 211.8x 是两个独立报告的结果**。
原页面没有披露测量日期。**2026-09-07 是本次来源快照核对日期，不是测量日期**。
## 历史结果
下表全部数值和说明都是原报告的历史表述，**不是当前 API 能力保证**。
模型原始输出包含标签，不代表 HTTP 接口一定返回这些标签；旧时间戳限制也不能代替
当前的[模型选型说明](../model_selection_zh.md)。
| 模型 | 设备 | RTF | 速度 | CER | 说明 |
| --- | --- | --- | --- | --- | --- |
| SenseVoice-Small | GPU | 0.005896 | 169.6x | 7.81% | ASR + 语种 / 情感 / 事件标签；CER 已去除标签后计算。 |
| Paraformer-Large | GPU | 0.008359 | 119.6x | 10.18% | 高速非自回归中文 ASR，适合 VAD/标点生产流水线。 |
| Fun-ASR-Nano | GPU | 0.058803 | 17.0x | 8.06% | LLM-based 中/英/日 ASR，另覆盖 7 类中文方言和 26 种地域口音，支持热词；不提供可靠的 checkpoint 原生时间戳（[#106](https://github.com/QwenAudio/Fun-ASR/issues/106)）。 |
| GLM-ASR-Nano | GPU | 0.026974 | 37.1x | 31.07% | LLM-based 多语种 ASR。 |
| Whisper-large-v3-turbo (OpenAI) | GPU | 0.021708 | 46.1x | 21.71% | OpenAI Whisper 实现。 |
| Whisper-large-v3 (OpenAI) | GPU | 0.074694 | 13.4x | 20.02% | 基线 for large Whisper quality. |
| SenseVoice-Small | CPU | 0.057988 | 17.2x | 7.81% | CPU 结果来自 remaining benchmark 脚本。 |
| Paraformer-Large | CPU | 0.064056 | 15.6x | 10.18% | CPU 上可用于批量任务。 |
| Fun-ASR-Nano | CPU | 0.274318 | 3.6x | 8.06% | LLM-based 模型更重，但仍高于实时。 |
CPU/GPU 行重复出现相同 CER，不能证明曾分别独立评分。审计材料中没有原始预测、
参考文本或评分程序。“去除标签后计算”保留为历史说法，不是本次验证过的评分输出。
原始精度以及经过舍入的速度/RTF 数值对均原样保留，不重新计算。
## 来源与限制
[原中文 HTML](https://github.com/modelscope/FunASR/blob/67d63b80a246dc33749e43904c294e0409cd9183/zh/benchmark.html)
固定到历史 GitHub Pages 提交；本次核对时，该文件与旧公网快照逐字节一致。
这能确认表格出处，但不能证明原始测量正确或可复现。
原报告将 RTF 定义为总推理时间除以总音频时长，速度为其倒数，后者也称 RTFx：
```text
RTF = total inference time / total audio duration
RTFx = total audio duration / total inference time = 1 / RTF
```
以下命令仅是**历史文本，不能直接执行**：在本次审计的 FunASR 源码版本
`386f6f9106684ba5a114e796147db4396a09eab5` 中，三个文件都不存在。
本页没有提供替代脚本或复现数据。
```text
python benchmark/run_full_benchmark.py
python benchmark/run_remaining.py
python benchmark/fix_sensevoice_cer.py
```
审计的原报告未披露 CPU 型号/线程数、数据成员与参考文本清单、精确 checkpoint
revision、软件/驱动版本、逐文件预测和计时日志，以及是否包含预热、I/O、预处理等
完整计时范围。缺少这些材料，旧表无法直接复现，“CPU 对比 GPU”的标题也不能视作
面向所有生产环境的保证。
这份记录的 **11,539 秒**必须与 [vLLM 评测方法](rtf_reproducibility.md) 中的
**11,541 秒**分开引用。两者都提到 184 个文件，并不能证明数据成员完全一致；
不要合并两份表格的结果，也不要自行消除两秒差异。
## 当前选型
下面是**原报告的建议表，仅保留为历史上下文**，不是新验证的推荐或性能排名。
| 需求 | 推荐模型 |
| --- | --- |
| 最快生产转写 | SenseVoice-Small 或 Paraformer-Large。 |
| CPU 批量转写 | 优先 SenseVoice-Small；中文生产流水线可选 Paraformer-Large。 |
| 中/英/日及中文方言、口音的 LLM-style 识别 | Fun-ASR-Nano；需要 31 语种时使用独立的 [Fun-ASR-MLT-Nano](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512) checkpoint，并使用 [vLLM](../vllm_guide_zh.md) 提升 LLM 解码吞吐。 |
| OpenAI 兼容本地服务 | 使用 [funasr-server](../agent_integration_zh.md)，模型别名为 `sensevoice`、`paraformer` 或 `fun-asr-nano`。 |
当前决策请参考[模型与能力说明](../model_selection_zh.md)、
[Agent 接口契约](../agent_integration_zh.md)和 [vLLM 部署说明](../vllm_guide_zh.md)。
独立 MLT checkpoint 的语种覆盖不能归到 Fun-ASR-Nano；选型前请使用自己的音频、
运行时与端到端延迟要求进行评估。
新的测量按[性能评测方法](rtf_reproducibility.md)记录；并发实时服务参考
[WebSocket 压测说明](realtime_ws_benchmark.md)。当前的
[迁移计时工具](../../examples/migration/benchmark_funasr.py)只测量 FunASR 在自有音频
上的表现，**不计算 CER/WER**、不运行 Whisper，也不复现缺失的历史脚本。
请分别记录计时范围、失败文件和质量评估。
docs/index.rst
+2
View File
@@ -60,6 +60,8 @@ entry to these guides. Source Markdown remains in this repository.
:maxdepth: 1
:caption: Evaluate and Operate
benchmark/historical_asr
benchmark/historical_asr_zh
benchmark/rtf_reproducibility
benchmark/realtime_ws_benchmark
troubleshooting
docs/use_case_showcase.md
+1 -1
View File
@@ -8,7 +8,7 @@ FunASR is useful far beyond a single offline transcription command. This page co
|---|---|---|
| Try FunASR in a browser | [Colab quickstart](../examples/colab/) | Run a public sample and upload your own audio before setting up a local environment. |
| Transcribe one file locally | [README quick start](../README.md#quick-start) and [model selection guide](./model_selection.md) | Verify install, model choice, and model download in minutes. |
| Compare accuracy and speed | [Benchmark report](https://modelscope.github.io/FunASR/benchmark.html) | Reproduce the 184-file long-audio benchmark before choosing a model. |
| Compare accuracy and speed | [Historical benchmark](./benchmark/historical_asr.md) and [current methodology](./benchmark/rtf_reproducibility.md) | Read the historical results and provenance limits, then benchmark representative local audio before choosing a model. |
| Migrate from Whisper/cloud ASR | [Migration guide](./migration_from_whisper.md) | Map existing pipelines to FunASR, benchmark representative audio, and plan a safe rollout. |
| Build a private speech API | [OpenAI-compatible API example](../examples/openai_api/), [Gradio browser demo](../examples/openai_api/GRADIO.md), [client recipes](../examples/openai_api/CLIENTS.md), [JavaScript/TypeScript recipes](../examples/openai_api/JAVASCRIPT.md), and [workflow recipes](../examples/openai_api/WORKFLOWS.md) | Reuse LangChain, Dify, n8n, AutoGen, and other OpenAI-style clients without sending audio to a cloud ASR provider. |
| Reuse an existing integration | [Community integrations](./community_projects.md) | Start from verified upstream paths for voice agents, local assistants, desktop subtitles, model serving, and Rust VAD. |
web-pages/product-site/assets/css/experience.css
+2
View File
@@ -158,6 +158,8 @@ td, th { padding: 14px; }
.docs-article pre code { background: none; color: inherit; white-space: pre; padding: 0; }
.docs-article :not(pre) > code { padding: 2px 5px; border-radius: 4px; background: #f0f4f1; font-size: .88em; }
.docs-article table { display: block; overflow-x: auto; max-width: 100%; width: 100%; overflow-wrap: normal; }
.docs-article #historical-results ~ table td:nth-child(6),
.docs-article #\5386\53f2\7ed3\679c ~ table td:nth-child(6) { min-width: 260px; }
.docs-article blockquote { border-left: 3px solid #7ea98e; background: #f3f8f4; margin: 20px 0; padding: 8px 20px; color: #52655a; }
.docs-toc { position: sticky; top: 104px; max-height: calc(100vh - 130px); overflow-y: auto; border-left: 1px solid var(--line); padding-left: 18px; font-size: 12px; }
.docs-toc strong { color: var(--ink); }
web-pages/product-site/data/documentation.json
+1
View File
@@ -33,6 +33,7 @@
{"slug": "troubleshooting", "group": "operate", "zh": "安装与推理故障排查", "en": "Troubleshooting", "source_zh": "docs/troubleshooting_zh.md", "source_en": "docs/troubleshooting.md"},
{"slug": "security", "group": "operate", "zh": "服务安全边界", "en": "Service security boundaries", "source_zh": "examples/openai_api/SECURITY_zh.md", "source_en": "examples/openai_api/SECURITY.md"},
{"slug": "benchmark-method", "group": "operate", "zh": "可复现性能评测", "en": "Reproducible performance tests", "source_zh": "docs/benchmark/rtf_reproducibility.md", "source_en": "docs/benchmark/rtf_reproducibility.md"},
{"slug": "historical-asr-benchmark", "group": "operate", "zh": "历史 ASR 评测", "en": "Historical ASR benchmark", "source_zh": "docs/benchmark/historical_asr_zh.md", "source_en": "docs/benchmark/historical_asr.md"},
{"slug": "realtime-benchmark", "group": "operate", "zh": "实时 WebSocket 压测", "en": "Realtime WebSocket benchmarking", "source_zh": "docs/benchmark/realtime_ws_benchmark.md", "source_en": "docs/benchmark/realtime_ws_benchmark.md"},
{"slug": "agent-integration", "group": "ecosystem", "zh": "Agent 集成", "en": "Agent integration", "source_zh": "docs/agent_integration_zh.md", "source_en": "docs/agent_integration.md"},
{"slug": "use-cases", "group": "ecosystem", "zh": "应用场景与示例", "en": "Use cases & examples", "source_zh": "docs/use_case_showcase_zh.md", "source_en": "docs/use_case_showcase.md"},
web-pages/product-site/data/legacy_doc_anchors.json
+16
View File
@@ -1,6 +1,10 @@
{
"source_ref": "origin/gh-pages",
"source_revision": "871ac766876c2f511550478d88c86aa3daab3172",
"additional_source_revisions": {
"benchmark.html": "67d63b80a246dc33749e43904c294e0409cd9183",
"zh/benchmark.html": "67d63b80a246dc33749e43904c294e0409cd9183"
},
"notes": "Published fragments map to the nearest current topic heading. Consolidated quickstart topics use workflow sections; removed standalone runtime diagnostics use the deployment readiness checklist. Source articles and API reference pages are not rewritten by this compatibility layer.",
"pages": {
"tutorial.html": {
@@ -148,6 +152,18 @@
"voice": "desktop-voice-input",
"subtitle": "subtitle-generation"
},
"benchmark.html": {
"summary": "historical-summary",
"table": "historical-results",
"method": "provenance-and-limitations",
"choose": "choosing-a-current-path"
},
"zh/benchmark.html": {
"summary": "历史概览",
"table": "历史结果",
"method": "来源与限制",
"choose": "当前选型"
},
"zh/agent.html": {
"server": "http-服务",
"sdk": "sdk-与-curl",
web-pages/product-site/export_docs.py
+1
View File
@@ -18,6 +18,7 @@ ALIASES = {
'vllm': 'vllm.html',
'deployment-matrix': 'deployment-matrix.html',
'agent-integration': 'agent.html',
'historical-asr-benchmark': 'benchmark.html',
}
web-pages/product-site/tests/browser/documentation.spec.ts
+40
View File
@@ -93,6 +93,46 @@ test('public speech sample is playable and waveform has actual pixels', async ({
expect(await page.locator('.hero-image').evaluate((image: HTMLImageElement) => image.naturalWidth)).toBeGreaterThan(1000);
});
for (const prefix of ['', '/en']) {
for (const width of [390, 1440]) {
test(`Historical ASR benchmark is discoverable and readable ${prefix || '/zh'} ${width}`, async ({ page }) => {
await page.setViewportSize({ width, height: 900 });
const errors: string[] = [];
page.on('pageerror', error => errors.push(error.message));
await page.goto(`${prefix}/docs/`);
await page.locator('[data-doc-search] input').fill(prefix ? 'Historical ASR' : '历史 ASR');
const route = `${prefix}/docs/historical-asr-benchmark.html`;
await page.locator(`.search-results a[href="${route}"]`).click();
await expect(page).toHaveURL(new RegExp(`${route}$`));
await expect(page.locator('.docs-article h2')).toHaveCount(4);
await expect(page.locator('.docs-article table')).toHaveCount(3);
const results = page.locator('.docs-article table').nth(1);
await expect(results.locator('tr')).toHaveCount(10);
const notes = await results.locator('td:nth-child(6)').evaluateAll(nodes => nodes.map(node => ({
width: node.getBoundingClientRect().width,
rowHeight: node.parentElement!.getBoundingClientRect().height,
})));
expect(notes.every(note => note.width >= 260 && note.rowHeight < 240)).toBeTruthy();
const cell = results.locator('td').filter({ hasText: /^0\.005896$/ });
await cell.scrollIntoViewIfNeeded();
const numberHeight = await cell.evaluate(node => {
const range = document.createRange();
range.selectNodeContents(node);
return range.getBoundingClientRect().height;
});
expect(numberHeight).toBeLessThan(32);
for (const table of await page.locator('.docs-article table').all()) {
await table.scrollIntoViewIfNeeded();
expect(await page.evaluate(() => document.documentElement.scrollWidth - innerWidth)).toBeLessThanOrEqual(1);
}
const peer = `${prefix ? '' : '/en'}/docs/historical-asr-benchmark.html`;
await page.locator(`.docs-article a[href="${peer}"]`).click();
await expect(page).toHaveURL(new RegExp(`${peer}$`));
expect(errors).toEqual([]);
});
}
}
for (const prefix of ['', '/en']) {
for (const width of [390, 1440]) {
test(`Agent integration is discoverable and readable ${prefix || '/zh'} ${width}`, async ({ page, context }) => {
web-pages/product-site/tests/fixtures/historical_asr_tables.json
Vendored
+807
View File
@@ -0,0 +1,807 @@
{
"schema_version": 1,
"source_commit": "67d63b80a246dc33749e43904c294e0409cd9183",
"status": "Frozen historical HTML cells, not new measurements",
"pages": [
{
"language": "en",
"requested_url": "https://modelscope.github.io/FunASR/benchmark.html",
"fetched_at_utc": "2026-09-07T11:03:49.943245+00:00",
"sha256": "e11c0d3842555f8f376d8ee658ee1dc4c45e92c08f2d420b3b60913fde6d8186",
"tables": [
{
"section_id": "summary",
"rows": [
[
{
"text": "Metric",
"links": []
},
{
"text": "Result",
"links": []
}
],
[
{
"text": "Dataset",
"links": []
},
{
"text": "184 long-form Chinese audio files, 11,539 s total, 192.3 min.",
"links": []
}
],
[
{
"text": "GPU",
"links": []
},
{
"text": "NVIDIA H100 80GB HBM3.",
"links": []
}
],
[
{
"text": "Best GPU speed",
"links": []
},
{
"text": "SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run.",
"links": []
}
],
[
{
"text": "Best CPU speed",
"links": []
},
{
"text": "SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime.",
"links": []
}
],
[
{
"text": "Baseline",
"links": []
},
{
"text": "OpenAI Whisper-large-v3: 13.4x realtime on GPU.",
"links": []
}
]
]
},
{
"section_id": "table",
"rows": [
[
{
"text": "Model",
"links": []
},
{
"text": "Device",
"links": []
},
{
"text": "RTF",
"links": []
},
{
"text": "Speed",
"links": []
},
{
"text": "CER",
"links": []
},
{
"text": "Notes",
"links": []
}
],
[
{
"text": "SenseVoice-Small",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.005896",
"links": []
},
{
"text": "169.6x",
"links": []
},
{
"text": "7.81%",
"links": []
},
{
"text": "ASR + language / emotion / event tags; CER after tag stripping.",
"links": []
}
],
[
{
"text": "Paraformer-Large",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.008359",
"links": []
},
{
"text": "119.6x",
"links": []
},
{
"text": "10.18%",
"links": []
},
{
"text": "Fast non-autoregressive Chinese ASR with VAD/punctuation pipeline.",
"links": []
}
],
[
{
"text": "Fun-ASR-Nano",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.058803",
"links": []
},
{
"text": "17.0x",
"links": []
},
{
"text": "8.06%",
"links": []
},
{
"text": "LLM-based ASR for Chinese, English, Japanese, seven Chinese dialect groups, and 26 regional accents; supports hotwords. Reliable checkpoint-native timestamps are not available (#106).",
"links": [
"https://github.com/QwenAudio/Fun-ASR/issues/106"
]
}
],
[
{
"text": "GLM-ASR-Nano",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.026974",
"links": []
},
{
"text": "37.1x",
"links": []
},
{
"text": "31.07%",
"links": []
},
{
"text": "LLM-based multilingual ASR.",
"links": []
}
],
[
{
"text": "Whisper-large-v3-turbo (OpenAI)",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.021708",
"links": []
},
{
"text": "46.1x",
"links": []
},
{
"text": "21.71%",
"links": []
},
{
"text": "OpenAI Whisper implementation.",
"links": []
}
],
[
{
"text": "Whisper-large-v3 (OpenAI)",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.074694",
"links": []
},
{
"text": "13.4x",
"links": []
},
{
"text": "20.02%",
"links": []
},
{
"text": "Baseline for large Whisper quality.",
"links": []
}
],
[
{
"text": "SenseVoice-Small",
"links": []
},
{
"text": "CPU",
"links": []
},
{
"text": "0.057988",
"links": []
},
{
"text": "17.2x",
"links": []
},
{
"text": "7.81%",
"links": []
},
{
"text": "CPU run from the remaining benchmark script.",
"links": []
}
],
[
{
"text": "Paraformer-Large",
"links": []
},
{
"text": "CPU",
"links": []
},
{
"text": "0.064056",
"links": []
},
{
"text": "15.6x",
"links": []
},
{
"text": "10.18%",
"links": []
},
{
"text": "CPU viable for batch jobs.",
"links": []
}
],
[
{
"text": "Fun-ASR-Nano",
"links": []
},
{
"text": "CPU",
"links": []
},
{
"text": "0.274318",
"links": []
},
{
"text": "3.6x",
"links": []
},
{
"text": "8.06%",
"links": []
},
{
"text": "LLM-based model is heavier but still above realtime.",
"links": []
}
]
]
},
{
"section_id": "choose",
"rows": [
[
{
"text": "Need",
"links": []
},
{
"text": "Recommended model",
"links": []
}
],
[
{
"text": "Fastest production transcription",
"links": []
},
{
"text": "SenseVoice-Small or Paraformer-Large.",
"links": []
}
],
[
{
"text": "CPU batch transcription",
"links": []
},
{
"text": "SenseVoice-Small first; Paraformer-Large for Chinese production pipelines.",
"links": []
}
],
[
{
"text": "Chinese/English/Japanese LLM-style recognition with dialect and accent coverage",
"links": []
},
{
"text": "Fun-ASR-Nano; use the separate Fun-ASR-MLT-Nano checkpoint for 31 languages, and use vLLM for higher LLM decoding throughput.",
"links": [
"https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512",
"vllm.html"
]
}
],
[
{
"text": "OpenAI-compatible local endpoint",
"links": []
},
{
"text": "funasr-server with model alias sensevoice, paraformer, or fun-asr-nano.",
"links": [
"agent.html"
]
}
]
]
}
]
},
{
"language": "zh",
"requested_url": "https://modelscope.github.io/FunASR/zh/benchmark.html",
"fetched_at_utc": "2026-09-07T11:03:50.243350+00:00",
"sha256": "af8cf07e9cf06c9ce5865b54f34581a2911ede699f750776634d8b6d22183462",
"tables": [
{
"section_id": "summary",
"rows": [
[
{
"text": "指标",
"links": []
},
{
"text": "结果",
"links": []
}
],
[
{
"text": "数据集",
"links": []
},
{
"text": "184 条中文长音频，总时长 11,539 秒，约 192.3 分钟。",
"links": []
}
],
[
{
"text": "GPU",
"links": []
},
{
"text": "NVIDIA H100 80GB HBM3.",
"links": []
}
],
[
{
"text": "最佳 GPU 速度",
"links": []
},
{
"text": "SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run.",
"links": []
}
],
[
{
"text": "最佳 CPU 速度",
"links": []
},
{
"text": "SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime.",
"links": []
}
],
[
{
"text": "基线",
"links": []
},
{
"text": "OpenAI Whisper-large-v3：GPU 上 13.4 倍实时。",
"links": []
}
]
]
},
{
"section_id": "table",
"rows": [
[
{
"text": "模型",
"links": []
},
{
"text": "设备",
"links": []
},
{
"text": "RTF",
"links": []
},
{
"text": "速度",
"links": []
},
{
"text": "CER",
"links": []
},
{
"text": "说明",
"links": []
}
],
[
{
"text": "SenseVoice-Small",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.005896",
"links": []
},
{
"text": "169.6x",
"links": []
},
{
"text": "7.81%",
"links": []
},
{
"text": "ASR + 语种 / 情感 / 事件标签；CER 已去除标签后计算。",
"links": []
}
],
[
{
"text": "Paraformer-Large",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.008359",
"links": []
},
{
"text": "119.6x",
"links": []
},
{
"text": "10.18%",
"links": []
},
{
"text": "高速非自回归中文 ASR，适合 VAD/标点生产流水线。",
"links": []
}
],
[
{
"text": "Fun-ASR-Nano",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.058803",
"links": []
},
{
"text": "17.0x",
"links": []
},
{
"text": "8.06%",
"links": []
},
{
"text": "LLM-based 中/英/日 ASR，另覆盖 7 类中文方言和 26 种地域口音，支持热词；不提供可靠的 checkpoint 原生时间戳（#106）。",
"links": [
"https://github.com/QwenAudio/Fun-ASR/issues/106"
]
}
],
[
{
"text": "GLM-ASR-Nano",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.026974",
"links": []
},
{
"text": "37.1x",
"links": []
},
{
"text": "31.07%",
"links": []
},
{
"text": "LLM-based 多语种 ASR。",
"links": []
}
],
[
{
"text": "Whisper-large-v3-turbo (OpenAI)",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.021708",
"links": []
},
{
"text": "46.1x",
"links": []
},
{
"text": "21.71%",
"links": []
},
{
"text": "OpenAI Whisper 实现。",
"links": []
}
],
[
{
"text": "Whisper-large-v3 (OpenAI)",
"links": []
},
{
"text": "GPU",
"links": []
},
{
"text": "0.074694",
"links": []
},
{
"text": "13.4x",
"links": []
},
{
"text": "20.02%",
"links": []
},
{
"text": "基线 for large Whisper quality.",
"links": []
}
],
[
{
"text": "SenseVoice-Small",
"links": []
},
{
"text": "CPU",
"links": []
},
{
"text": "0.057988",
"links": []
},
{
"text": "17.2x",
"links": []
},
{
"text": "7.81%",
"links": []
},
{
"text": "CPU 结果来自 remaining benchmark 脚本。",
"links": []
}
],
[
{
"text": "Paraformer-Large",
"links": []
},
{
"text": "CPU",
"links": []
},
{
"text": "0.064056",
"links": []
},
{
"text": "15.6x",
"links": []
},
{
"text": "10.18%",
"links": []
},
{
"text": "CPU 上可用于批量任务。",
"links": []
}
],
[
{
"text": "Fun-ASR-Nano",
"links": []
},
{
"text": "CPU",
"links": []
},
{
"text": "0.274318",
"links": []
},
{
"text": "3.6x",
"links": []
},
{
"text": "8.06%",
"links": []
},
{
"text": "LLM-based 模型更重，但仍高于实时。",
"links": []
}
]
]
},
{
"section_id": "choose",
"rows": [
[
{
"text": "需求",
"links": []
},
{
"text": "推荐模型",
"links": []
}
],
[
{
"text": "最快生产转写",
"links": []
},
{
"text": "SenseVoice-Small 或 Paraformer-Large。",
"links": []
}
],
[
{
"text": "CPU 批量转写",
"links": []
},
{
"text": "优先 SenseVoice-Small；中文生产流水线可选 Paraformer-Large。",
"links": []
}
],
[
{
"text": "中/英/日及中文方言、口音的 LLM-style 识别",
"links": []
},
{
"text": "Fun-ASR-Nano；需要 31 语种时使用独立的 Fun-ASR-MLT-Nano checkpoint，并使用 vLLM 提升 LLM 解码吞吐。",
"links": [
"https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512",
"vllm.html"
]
}
],
[
{
"text": "OpenAI 兼容本地服务",
"links": []
},
{
"text": "使用 funasr-server，模型别名为 sensevoice、paraformer 或 fun-asr-nano。",
"links": [
"agent.html"
]
}
]
]
}
]
}
]
}
web-pages/product-site/tests/test_documentation.py
+55
View File
@@ -209,3 +209,58 @@ def test_agent_recipes_are_bilingual_and_use_existing_script_options():
'examples/voice_input/funasr_input.py',
'examples/subtitle/generate_subtitle.py'}
assert recipes[0] == recipes[1]
@pytest.mark.parametrize('language,prefix,suffix', [('en', 'en/', ''), ('zh', '', '_zh')])
def test_historical_benchmark_preserves_frozen_tables_and_discovery(output, language, prefix, suffix):
frozen = json.loads((SITE_ROOT / 'tests/fixtures/historical_asr_tables.json').read_text())
record = next(item for item in frozen['pages'] if item['language'] == language)
route = '/' + prefix + 'docs/historical-asr-benchmark.html'
path = output / route.lstrip('/')
assert path.is_file(), route
page = BeautifulSoup(path.read_text(), 'html.parser')
article = page.select_one('.docs-article')
tables = article.select('table')
assert len(tables) == 3
assert len(article.select('h2')) == 4
for table, expected in zip(tables, record['tables']):
actual = [[' '.join(cell.get_text().split()) for cell in row.select('th,td')]
for row in table.select('tr')]
assert actual == [[cell['text'] for cell in row] for row in expected['rows']]
assert page.select_one('[data-source-link]')['href'].endswith(
f'/docs/benchmark/historical_asr{suffix}.md')
links = {link['href'] for link in article.select('a[href]')}
for slug in ('agent-integration', 'vllm', 'model-selection', 'benchmark-method', 'realtime-benchmark'):
assert '/' + prefix + 'docs/' + slug + '.html' in links
assert ('/docs/' if prefix else '/en/docs/') + 'historical-asr-benchmark.html' in links
assert 'https://github.com/QwenAudio/Fun-ASR/issues/106' in links
assert 'https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512' in links
original_path = urlsplit(record['requested_url']).path.removeprefix('/FunASR/')
assert f"https://github.com/modelscope/FunASR/blob/{frozen['source_commit']}/{original_path}" in links
index = BeautifulSoup((output / prefix / 'docs/index.html').read_text(), 'html.parser')
assert index.select_one(f'a[href="{route}"]')
search = json.loads((output / f'search-{language}.json').read_text())
assert any(row['url'] == route for row in search)
@pytest.mark.parametrize('suffix', ['', '_zh'])
def test_historical_benchmark_does_not_advertise_missing_scripts_as_runnable(suffix):
root = SITE_ROOT.parents[1]
source_path = root / f'docs/benchmark/historical_asr{suffix}.md'
assert source_path.is_file()
source = source_path.read_text()
blocks = re.findall(r'^```(\w+)\n(.*?)^```', source, re.M | re.S)
assert len(blocks) == 2 and all(language == 'text' for language, _ in blocks)
commands = [line for _, block in blocks for line in block.splitlines() if line.startswith('python ')]
assert commands == ['python benchmark/run_full_benchmark.py',
'python benchmark/run_remaining.py',
'python benchmark/fix_sensevoice_cer.py']
for command in commands:
assert not (root / command.split()[1]).exists()
for marker in ('11,539', '11,541', '169.6x', '211.8x', 'RTFx', '2026-09-07'):
assert marker in source
for marker in (('来源不完整', '不是测量日期', '不能直接执行', '不计算 CER/WER') if suffix
else ('incomplete provenance', 'not the measurement date', 'cannot be run directly', 'does not compute CER/WER')):
assert marker in source
if not suffix:
assert 'Reproduce the 184-file long-audio benchmark' not in (root / 'docs/use_case_showcase.md').read_text()
web-pages/product-site/tests/test_legacy_doc_anchors.py
+10
View File
@@ -21,6 +21,7 @@ OLD_IDS = {
'deployment-matrix.html': 'matrix choices runtime-diagnostics checklist help',
# Independently inventoried from public EN/ZH pages on 2026-09-07.
'agent.html': 'server sdk workflows mcp voice subtitle',
'benchmark.html': 'summary table method choose',
}
PAGES = [prefix + name for name in OLD_IDS for prefix in ('', 'zh/')]
@@ -151,6 +152,15 @@ def test_agent_legacy_fragments_keep_their_topics(page, targets):
assert pages[page] == dict(zip('server sdk workflows mcp voice subtitle'.split(), targets))
@pytest.mark.parametrize('page,targets', [
('benchmark.html', ['historical-summary', 'historical-results', 'provenance-and-limitations', 'choosing-a-current-path']),
('zh/benchmark.html', ['历史概览', '历史结果', '来源与限制', '当前选型']),
])
def test_benchmark_legacy_fragments_keep_their_topics(page, targets):
pages = json.loads((SITE / 'data/legacy_doc_anchors.json').read_text())['pages']
assert pages[page] == dict(zip('summary table method choose'.split(), targets))
@pytest.mark.parametrize('heading', ('<h2 id="different">Topic</h2>', '<p id="topic">Not a heading</p>'))
def test_missing_or_nonheading_target_fails_explicitly(heading):
from export_docs import insert_legacy_anchors