diff --git a/docs/README.md b/docs/README.md index 292ea9834..44c2c7236 100644 --- a/docs/README.md +++ b/docs/README.md @@ -51,6 +51,7 @@ interfaces. Choose the matching protocol before selecting a client. ## Evaluate and Operate +- [Historical ASR benchmark](benchmark/historical_asr.md) / [中文](benchmark/historical_asr_zh.md): preserved results and provenance limits - [Performance methodology](benchmark/rtf_reproducibility.md) and [realtime WebSocket benchmarks](benchmark/realtime_ws_benchmark.md) - [Troubleshooting](troubleshooting.md) / [中文](troubleshooting_zh.md) - [Use cases](use_case_showcase.md), [community integrations](community_projects.md), and [repository responsibilities](repository_roles.md) diff --git a/docs/benchmark/historical_asr.md b/docs/benchmark/historical_asr.md new file mode 100644 index 000000000..492c3a8e5 --- /dev/null +++ b/docs/benchmark/historical_asr.md @@ -0,0 +1,113 @@ +# Historical ASR Benchmark + +[中文](historical_asr_zh.md) + +This is a historical record with incomplete provenance, retained for readers of +earlier FunASR comparisons. It is not a new measurement, a universal leaderboard, +or a guarantee for a current checkpoint, machine or deployment. For a new +evaluation, start with the [performance methodology](rtf_reproducibility.md). + +## Historical Summary + +The table below preserves the original report's wording and numbers. Its +"best" labels refer only to that report, not to all available models or hardware. + +| Metric | Result | +| --- | --- | +| Dataset | 184 long-form Chinese audio files, 11,539 s total, 192.3 min. | +| GPU | NVIDIA H100 80GB HBM3. | +| Best GPU speed | SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run. | +| Best CPU speed | SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime. | +| Baseline | OpenAI Whisper-large-v3: 13.4x realtime on GPU. | + +The **169.6x full run and 211.8x initial run are separate reported results**. +The original page does not disclose the measurement date. The source was +checked on **2026-09-07**, which is a snapshot audit date, not the measurement date. + +## Historical Results + +All values and notes in this table are archived claims from the original report. +The notes are not current API capability guarantees. In particular, a model's +raw tags do not imply that an HTTP endpoint returns those tags, and an old +timestamp limitation must not replace the current [model selection guide](../model_selection.md). + +| Model | Device | RTF | Speed | CER | Notes | +| --- | --- | --- | --- | --- | --- | +| SenseVoice-Small | GPU | 0.005896 | 169.6x | 7.81% | ASR + language / emotion / event tags; CER after tag stripping. | +| Paraformer-Large | GPU | 0.008359 | 119.6x | 10.18% | Fast non-autoregressive Chinese ASR with VAD/punctuation pipeline. | +| Fun-ASR-Nano | GPU | 0.058803 | 17.0x | 8.06% | LLM-based ASR for Chinese, English, Japanese, seven Chinese dialect groups, and 26 regional accents; supports hotwords. Reliable checkpoint-native timestamps are not available ([#106](https://github.com/QwenAudio/Fun-ASR/issues/106)). | +| GLM-ASR-Nano | GPU | 0.026974 | 37.1x | 31.07% | LLM-based multilingual ASR. | +| Whisper-large-v3-turbo (OpenAI) | GPU | 0.021708 | 46.1x | 21.71% | OpenAI Whisper implementation. | +| Whisper-large-v3 (OpenAI) | GPU | 0.074694 | 13.4x | 20.02% | Baseline for large Whisper quality. | +| SenseVoice-Small | CPU | 0.057988 | 17.2x | 7.81% | CPU run from the remaining benchmark script. | +| Paraformer-Large | CPU | 0.064056 | 15.6x | 10.18% | CPU viable for batch jobs. | +| Fun-ASR-Nano | CPU | 0.274318 | 3.6x | 8.06% | LLM-based model is heavier but still above realtime. | + +The repeated CPU/GPU CER values do not establish independent per-device scoring. +The raw predictions, references and scoring program are not available in the +audited record. The tag-stripping statement is preserved as a historical claim, +not as newly verified scorer output. Numerical precision and rounded speed/RTF +pairs are retained without recalculation. + +## Provenance and Limitations + +The [original English HTML](https://github.com/modelscope/FunASR/blob/67d63b80a246dc33749e43904c294e0409cd9183/benchmark.html) +is pinned to its historical GitHub Pages commit. Its bytes matched the archived +public page during the source audit. That establishes the source of this table, +not the correctness or reproducibility of the underlying measurements. + +The old report describes RTF as total inference time divided by total audio +duration, and speed as its reciprocal. The latter is also called RTFx: + +```text +RTF = total inference time / total audio duration +RTFx = total audio duration / total inference time = 1 / RTF +``` + +The following commands are **historical text and cannot be run directly from +the audited checkout**. All three referenced files were absent at FunASR source +revision `386f6f9106684ba5a114e796147db4396a09eab5`; no replacement scripts or +reproduction data are supplied by this document. + +```text +python benchmark/run_full_benchmark.py +python benchmark/run_remaining.py +python benchmark/fix_sensevoice_cer.py +``` + +The audited report does not provide a CPU model/thread count, dataset membership +and reference manifest, exact checkpoint revisions, software/driver versions, +per-file predictions or timing logs, or a complete timing scope covering warmup, +I/O and preprocessing. Without these materials, the old table is not directly +reproducible and its CPU-versus-GPU headline is not a production-wide guarantee. + +This record's **11,539 seconds** must remain separate from the **11,541 seconds** +reported in the [vLLM methodology](rtf_reproducibility.md). Both mention 184 files, +but that alone does not prove identical dataset membership. Do not merge their +rows or silently normalize the two-second discrepancy. + +## Choosing a Current Path + +The following is the **original recommendation table, retained as historical +context only**. It is not a newly validated recommendation or performance ranking. + +| Need | Recommended model | +| --- | --- | +| Fastest production transcription | SenseVoice-Small or Paraformer-Large. | +| CPU batch transcription | SenseVoice-Small first; Paraformer-Large for Chinese production pipelines. | +| Chinese/English/Japanese LLM-style recognition with dialect and accent coverage | Fun-ASR-Nano; use the separate [Fun-ASR-MLT-Nano](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512) checkpoint for 31 languages, and use [vLLM](../vllm_guide.md) for higher LLM decoding throughput. | +| OpenAI-compatible local endpoint | [funasr-server](../agent_integration.md) with model alias `sensevoice`, `paraformer`, or `fun-asr-nano`. | + +For current decisions, use the [model and capability guide](../model_selection.md), +[Agent integration contracts](../agent_integration.md), and +[vLLM deployment guide](../vllm_guide.md). The separate MLT checkpoint's language +coverage must not be attributed to Fun-ASR-Nano. Evaluate your own audio, runtime +and end-to-end latency before selecting a deployment. + +Use the [performance methodology](rtf_reproducibility.md) for new measurements +and the [WebSocket benchmark guide](realtime_ws_benchmark.md) for concurrent +realtime services. The current +[migration timing helper](../../examples/migration/benchmark_funasr.py) measures +FunASR on your own audio; it does not compute CER/WER, run Whisper or reproduce +the missing historical scripts. Keep your timing scope, failed files and quality +evaluation separate and explicit. diff --git a/docs/benchmark/historical_asr_zh.md b/docs/benchmark/historical_asr_zh.md new file mode 100644 index 000000000..5bbfe7b8f --- /dev/null +++ b/docs/benchmark/historical_asr_zh.md @@ -0,0 +1,98 @@ +# 历史 ASR 评测 + +[English](historical_asr.md) + +这是一份**来源不完整的历史记录**，用于帮助读者理解早期 FunASR 对比结果。 +它不是新测量、通用排行榜，也不是对当前 checkpoint、机器或部署的保证。 +开展新的评测请先阅读[性能评测方法](rtf_reproducibility.md)。 + +## 历史概览 + +下表保留原报告的措辞与数字。“最佳”等标签仅指该份报告，不能推广到所有模型或硬件。 + +| 指标 | 结果 | +| --- | --- | +| 数据集 | 184 条中文长音频，总时长 11,539 秒，约 192.3 分钟。 | +| GPU | NVIDIA H100 80GB HBM3. | +| 最佳 GPU 速度 | SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run. | +| 最佳 CPU 速度 | SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime. | +| 基线 | OpenAI Whisper-large-v3：GPU 上 13.4 倍实时。 | + +**完整运行的 169.6x 与初次运行的 211.8x 是两个独立报告的结果**。 +原页面没有披露测量日期。**2026-09-07 是本次来源快照核对日期，不是测量日期**。 + +## 历史结果 + +下表全部数值和说明都是原报告的历史表述，**不是当前 API 能力保证**。 +模型原始输出包含标签，不代表 HTTP 接口一定返回这些标签；旧时间戳限制也不能代替 +当前的[模型选型说明](../model_selection_zh.md)。 + +| 模型 | 设备 | RTF | 速度 | CER | 说明 | +| --- | --- | --- | --- | --- | --- | +| SenseVoice-Small | GPU | 0.005896 | 169.6x | 7.81% | ASR + 语种 / 情感 / 事件标签；CER 已去除标签后计算。 | +| Paraformer-Large | GPU | 0.008359 | 119.6x | 10.18% | 高速非自回归中文 ASR，适合 VAD/标点生产流水线。 | +| Fun-ASR-Nano | GPU | 0.058803 | 17.0x | 8.06% | LLM-based 中/英/日 ASR，另覆盖 7 类中文方言和 26 种地域口音，支持热词；不提供可靠的 checkpoint 原生时间戳（[#106](https://github.com/QwenAudio/Fun-ASR/issues/106)）。 | +| GLM-ASR-Nano | GPU | 0.026974 | 37.1x | 31.07% | LLM-based 多语种 ASR。 | +| Whisper-large-v3-turbo (OpenAI) | GPU | 0.021708 | 46.1x | 21.71% | OpenAI Whisper 实现。 | +| Whisper-large-v3 (OpenAI) | GPU | 0.074694 | 13.4x | 20.02% | 基线 for large Whisper quality. | +| SenseVoice-Small | CPU | 0.057988 | 17.2x | 7.81% | CPU 结果来自 remaining benchmark 脚本。 | +| Paraformer-Large | CPU | 0.064056 | 15.6x | 10.18% | CPU 上可用于批量任务。 | +| Fun-ASR-Nano | CPU | 0.274318 | 3.6x | 8.06% | LLM-based 模型更重，但仍高于实时。 | + +CPU/GPU 行重复出现相同 CER，不能证明曾分别独立评分。审计材料中没有原始预测、 +参考文本或评分程序。“去除标签后计算”保留为历史说法，不是本次验证过的评分输出。 +原始精度以及经过舍入的速度/RTF 数值对均原样保留，不重新计算。 + +## 来源与限制 + +[原中文 HTML](https://github.com/modelscope/FunASR/blob/67d63b80a246dc33749e43904c294e0409cd9183/zh/benchmark.html) +固定到历史 GitHub Pages 提交；本次核对时，该文件与旧公网快照逐字节一致。 +这能确认表格出处，但不能证明原始测量正确或可复现。 + +原报告将 RTF 定义为总推理时间除以总音频时长，速度为其倒数，后者也称 RTFx： + +```text +RTF = total inference time / total audio duration +RTFx = total audio duration / total inference time = 1 / RTF +``` + +以下命令仅是**历史文本，不能直接执行**：在本次审计的 FunASR 源码版本 +`386f6f9106684ba5a114e796147db4396a09eab5` 中，三个文件都不存在。 +本页没有提供替代脚本或复现数据。 + +```text +python benchmark/run_full_benchmark.py +python benchmark/run_remaining.py +python benchmark/fix_sensevoice_cer.py +``` + +审计的原报告未披露 CPU 型号/线程数、数据成员与参考文本清单、精确 checkpoint +revision、软件/驱动版本、逐文件预测和计时日志，以及是否包含预热、I/O、预处理等 +完整计时范围。缺少这些材料，旧表无法直接复现，“CPU 对比 GPU”的标题也不能视作 +面向所有生产环境的保证。 + +这份记录的 **11,539 秒**必须与 [vLLM 评测方法](rtf_reproducibility.md) 中的 +**11,541 秒**分开引用。两者都提到 184 个文件，并不能证明数据成员完全一致； +不要合并两份表格的结果，也不要自行消除两秒差异。 + +## 当前选型 + +下面是**原报告的建议表，仅保留为历史上下文**，不是新验证的推荐或性能排名。 + +| 需求 | 推荐模型 | +| --- | --- | +| 最快生产转写 | SenseVoice-Small 或 Paraformer-Large。 | +| CPU 批量转写 | 优先 SenseVoice-Small；中文生产流水线可选 Paraformer-Large。 | +| 中/英/日及中文方言、口音的 LLM-style 识别 | Fun-ASR-Nano；需要 31 语种时使用独立的 [Fun-ASR-MLT-Nano](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512) checkpoint，并使用 [vLLM](../vllm_guide_zh.md) 提升 LLM 解码吞吐。 | +| OpenAI 兼容本地服务 | 使用 [funasr-server](../agent_integration_zh.md)，模型别名为 `sensevoice`、`paraformer` 或 `fun-asr-nano`。 | + +当前决策请参考[模型与能力说明](../model_selection_zh.md)、 +[Agent 接口契约](../agent_integration_zh.md)和 [vLLM 部署说明](../vllm_guide_zh.md)。 +独立 MLT checkpoint 的语种覆盖不能归到 Fun-ASR-Nano；选型前请使用自己的音频、 +运行时与端到端延迟要求进行评估。 + +新的测量按[性能评测方法](rtf_reproducibility.md)记录；并发实时服务参考 +[WebSocket 压测说明](realtime_ws_benchmark.md)。当前的 +[迁移计时工具](../../examples/migration/benchmark_funasr.py)只测量 FunASR 在自有音频 +上的表现，**不计算 CER/WER**、不运行 Whisper，也不复现缺失的历史脚本。 +请分别记录计时范围、失败文件和质量评估。 diff --git a/docs/index.rst b/docs/index.rst index 91e660c49..8671321b7 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -60,6 +60,8 @@ entry to these guides. Source Markdown remains in this repository. :maxdepth: 1 :caption: Evaluate and Operate + benchmark/historical_asr + benchmark/historical_asr_zh benchmark/rtf_reproducibility benchmark/realtime_ws_benchmark troubleshooting diff --git a/docs/use_case_showcase.md b/docs/use_case_showcase.md index d4747a729..09a5cb64e 100644 --- a/docs/use_case_showcase.md +++ b/docs/use_case_showcase.md @@ -8,7 +8,7 @@ FunASR is useful far beyond a single offline transcription command. This page co |---|---|---| | Try FunASR in a browser | [Colab quickstart](../examples/colab/) | Run a public sample and upload your own audio before setting up a local environment. | | Transcribe one file locally | [README quick start](../README.md#quick-start) and [model selection guide](./model_selection.md) | Verify install, model choice, and model download in minutes. | -| Compare accuracy and speed | [Benchmark report](https://modelscope.github.io/FunASR/benchmark.html) | Reproduce the 184-file long-audio benchmark before choosing a model. | +| Compare accuracy and speed | [Historical benchmark](./benchmark/historical_asr.md) and [current methodology](./benchmark/rtf_reproducibility.md) | Read the historical results and provenance limits, then benchmark representative local audio before choosing a model. | | Migrate from Whisper/cloud ASR | [Migration guide](./migration_from_whisper.md) | Map existing pipelines to FunASR, benchmark representative audio, and plan a safe rollout. | | Build a private speech API | [OpenAI-compatible API example](../examples/openai_api/), [Gradio browser demo](../examples/openai_api/GRADIO.md), [client recipes](../examples/openai_api/CLIENTS.md), [JavaScript/TypeScript recipes](../examples/openai_api/JAVASCRIPT.md), and [workflow recipes](../examples/openai_api/WORKFLOWS.md) | Reuse LangChain, Dify, n8n, AutoGen, and other OpenAI-style clients without sending audio to a cloud ASR provider. | | Reuse an existing integration | [Community integrations](./community_projects.md) | Start from verified upstream paths for voice agents, local assistants, desktop subtitles, model serving, and Rust VAD. | diff --git a/web-pages/product-site/assets/css/experience.css b/web-pages/product-site/assets/css/experience.css index d76fbbd46..8e39eb294 100644 --- a/web-pages/product-site/assets/css/experience.css +++ b/web-pages/product-site/assets/css/experience.css @@ -158,6 +158,8 @@ td, th { padding: 14px; } .docs-article pre code { background: none; color: inherit; white-space: pre; padding: 0; } .docs-article :not(pre) > code { padding: 2px 5px; border-radius: 4px; background: #f0f4f1; font-size: .88em; } .docs-article table { display: block; overflow-x: auto; max-width: 100%; width: 100%; overflow-wrap: normal; } +.docs-article #historical-results ~ table td:nth-child(6), +.docs-article #\5386\53f2\7ed3\679c ~ table td:nth-child(6) { min-width: 260px; } .docs-article blockquote { border-left: 3px solid #7ea98e; background: #f3f8f4; margin: 20px 0; padding: 8px 20px; color: #52655a; } .docs-toc { position: sticky; top: 104px; max-height: calc(100vh - 130px); overflow-y: auto; border-left: 1px solid var(--line); padding-left: 18px; font-size: 12px; } .docs-toc strong { color: var(--ink); } diff --git a/web-pages/product-site/data/documentation.json b/web-pages/product-site/data/documentation.json index 4e5c503e6..37859ef65 100644 --- a/web-pages/product-site/data/documentation.json +++ b/web-pages/product-site/data/documentation.json @@ -33,6 +33,7 @@ {"slug": "troubleshooting", "group": "operate", "zh": "安装与推理故障排查", "en": "Troubleshooting", "source_zh": "docs/troubleshooting_zh.md", "source_en": "docs/troubleshooting.md"}, {"slug": "security", "group": "operate", "zh": "服务安全边界", "en": "Service security boundaries", "source_zh": "examples/openai_api/SECURITY_zh.md", "source_en": "examples/openai_api/SECURITY.md"}, {"slug": "benchmark-method", "group": "operate", "zh": "可复现性能评测", "en": "Reproducible performance tests", "source_zh": "docs/benchmark/rtf_reproducibility.md", "source_en": "docs/benchmark/rtf_reproducibility.md"}, + {"slug": "historical-asr-benchmark", "group": "operate", "zh": "历史 ASR 评测", "en": "Historical ASR benchmark", "source_zh": "docs/benchmark/historical_asr_zh.md", "source_en": "docs/benchmark/historical_asr.md"}, {"slug": "realtime-benchmark", "group": "operate", "zh": "实时 WebSocket 压测", "en": "Realtime WebSocket benchmarking", "source_zh": "docs/benchmark/realtime_ws_benchmark.md", "source_en": "docs/benchmark/realtime_ws_benchmark.md"}, {"slug": "agent-integration", "group": "ecosystem", "zh": "Agent 集成", "en": "Agent integration", "source_zh": "docs/agent_integration_zh.md", "source_en": "docs/agent_integration.md"}, {"slug": "use-cases", "group": "ecosystem", "zh": "应用场景与示例", "en": "Use cases & examples", "source_zh": "docs/use_case_showcase_zh.md", "source_en": "docs/use_case_showcase.md"}, diff --git a/web-pages/product-site/data/legacy_doc_anchors.json b/web-pages/product-site/data/legacy_doc_anchors.json index a0bc12711..862b73ebf 100644 --- a/web-pages/product-site/data/legacy_doc_anchors.json +++ b/web-pages/product-site/data/legacy_doc_anchors.json @@ -1,6 +1,10 @@ { "source_ref": "origin/gh-pages", "source_revision": "871ac766876c2f511550478d88c86aa3daab3172", + "additional_source_revisions": { + "benchmark.html": "67d63b80a246dc33749e43904c294e0409cd9183", + "zh/benchmark.html": "67d63b80a246dc33749e43904c294e0409cd9183" + }, "notes": "Published fragments map to the nearest current topic heading. Consolidated quickstart topics use workflow sections; removed standalone runtime diagnostics use the deployment readiness checklist. Source articles and API reference pages are not rewritten by this compatibility layer.", "pages": { "tutorial.html": { @@ -148,6 +152,18 @@ "voice": "desktop-voice-input", "subtitle": "subtitle-generation" }, + "benchmark.html": { + "summary": "historical-summary", + "table": "historical-results", + "method": "provenance-and-limitations", + "choose": "choosing-a-current-path" + }, + "zh/benchmark.html": { + "summary": "历史概览", + "table": "历史结果", + "method": "来源与限制", + "choose": "当前选型" + }, "zh/agent.html": { "server": "http-服务", "sdk": "sdk-与-curl", diff --git a/web-pages/product-site/export_docs.py b/web-pages/product-site/export_docs.py index 83a78ad14..4d4b80d71 100644 --- a/web-pages/product-site/export_docs.py +++ b/web-pages/product-site/export_docs.py @@ -18,6 +18,7 @@ ALIASES = { 'vllm': 'vllm.html', 'deployment-matrix': 'deployment-matrix.html', 'agent-integration': 'agent.html', + 'historical-asr-benchmark': 'benchmark.html', } diff --git a/web-pages/product-site/tests/browser/documentation.spec.ts b/web-pages/product-site/tests/browser/documentation.spec.ts index 593d1e72f..3650f3773 100644 --- a/web-pages/product-site/tests/browser/documentation.spec.ts +++ b/web-pages/product-site/tests/browser/documentation.spec.ts @@ -93,6 +93,46 @@ test('public speech sample is playable and waveform has actual pixels', async ({ expect(await page.locator('.hero-image').evaluate((image: HTMLImageElement) => image.naturalWidth)).toBeGreaterThan(1000); }); +for (const prefix of ['', '/en']) { + for (const width of [390, 1440]) { + test(`Historical ASR benchmark is discoverable and readable ${prefix || '/zh'} ${width}`, async ({ page }) => { + await page.setViewportSize({ width, height: 900 }); + const errors: string[] = []; + page.on('pageerror', error => errors.push(error.message)); + await page.goto(`${prefix}/docs/`); + await page.locator('[data-doc-search] input').fill(prefix ? 'Historical ASR' : '历史 ASR'); + const route = `${prefix}/docs/historical-asr-benchmark.html`; + await page.locator(`.search-results a[href="${route}"]`).click(); + await expect(page).toHaveURL(new RegExp(`${route}$`)); + await expect(page.locator('.docs-article h2')).toHaveCount(4); + await expect(page.locator('.docs-article table')).toHaveCount(3); + const results = page.locator('.docs-article table').nth(1); + await expect(results.locator('tr')).toHaveCount(10); + const notes = await results.locator('td:nth-child(6)').evaluateAll(nodes => nodes.map(node => ({ + width: node.getBoundingClientRect().width, + rowHeight: node.parentElement!.getBoundingClientRect().height, + }))); + expect(notes.every(note => note.width >= 260 && note.rowHeight < 240)).toBeTruthy(); + const cell = results.locator('td').filter({ hasText: /^0\.005896$/ }); + await cell.scrollIntoViewIfNeeded(); + const numberHeight = await cell.evaluate(node => { + const range = document.createRange(); + range.selectNodeContents(node); + return range.getBoundingClientRect().height; + }); + expect(numberHeight).toBeLessThan(32); + for (const table of await page.locator('.docs-article table').all()) { + await table.scrollIntoViewIfNeeded(); + expect(await page.evaluate(() => document.documentElement.scrollWidth - innerWidth)).toBeLessThanOrEqual(1); + } + const peer = `${prefix ? '' : '/en'}/docs/historical-asr-benchmark.html`; + await page.locator(`.docs-article a[href="${peer}"]`).click(); + await expect(page).toHaveURL(new RegExp(`${peer}$`)); + expect(errors).toEqual([]); + }); + } +} + for (const prefix of ['', '/en']) { for (const width of [390, 1440]) { test(`Agent integration is discoverable and readable ${prefix || '/zh'} ${width}`, async ({ page, context }) => { diff --git a/web-pages/product-site/tests/fixtures/historical_asr_tables.json b/web-pages/product-site/tests/fixtures/historical_asr_tables.json new file mode 100644 index 000000000..57e4791a6 --- /dev/null +++ b/web-pages/product-site/tests/fixtures/historical_asr_tables.json @@ -0,0 +1,807 @@ +{ + "schema_version": 1, + "source_commit": "67d63b80a246dc33749e43904c294e0409cd9183", + "status": "Frozen historical HTML cells, not new measurements", + "pages": [ + { + "language": "en", + "requested_url": "https://modelscope.github.io/FunASR/benchmark.html", + "fetched_at_utc": "2026-09-07T11:03:49.943245+00:00", + "sha256": "e11c0d3842555f8f376d8ee658ee1dc4c45e92c08f2d420b3b60913fde6d8186", + "tables": [ + { + "section_id": "summary", + "rows": [ + [ + { + "text": "Metric", + "links": [] + }, + { + "text": "Result", + "links": [] + } + ], + [ + { + "text": "Dataset", + "links": [] + }, + { + "text": "184 long-form Chinese audio files, 11,539 s total, 192.3 min.", + "links": [] + } + ], + [ + { + "text": "GPU", + "links": [] + }, + { + "text": "NVIDIA H100 80GB HBM3.", + "links": [] + } + ], + [ + { + "text": "Best GPU speed", + "links": [] + }, + { + "text": "SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run.", + "links": [] + } + ], + [ + { + "text": "Best CPU speed", + "links": [] + }, + { + "text": "SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime.", + "links": [] + } + ], + [ + { + "text": "Baseline", + "links": [] + }, + { + "text": "OpenAI Whisper-large-v3: 13.4x realtime on GPU.", + "links": [] + } + ] + ] + }, + { + "section_id": "table", + "rows": [ + [ + { + "text": "Model", + "links": [] + }, + { + "text": "Device", + "links": [] + }, + { + "text": "RTF", + "links": [] + }, + { + "text": "Speed", + "links": [] + }, + { + "text": "CER", + "links": [] + }, + { + "text": "Notes", + "links": [] + } + ], + [ + { + "text": "SenseVoice-Small", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.005896", + "links": [] + }, + { + "text": "169.6x", + "links": [] + }, + { + "text": "7.81%", + "links": [] + }, + { + "text": "ASR + language / emotion / event tags; CER after tag stripping.", + "links": [] + } + ], + [ + { + "text": "Paraformer-Large", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.008359", + "links": [] + }, + { + "text": "119.6x", + "links": [] + }, + { + "text": "10.18%", + "links": [] + }, + { + "text": "Fast non-autoregressive Chinese ASR with VAD/punctuation pipeline.", + "links": [] + } + ], + [ + { + "text": "Fun-ASR-Nano", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.058803", + "links": [] + }, + { + "text": "17.0x", + "links": [] + }, + { + "text": "8.06%", + "links": [] + }, + { + "text": "LLM-based ASR for Chinese, English, Japanese, seven Chinese dialect groups, and 26 regional accents; supports hotwords. Reliable checkpoint-native timestamps are not available (#106).", + "links": [ + "https://github.com/QwenAudio/Fun-ASR/issues/106" + ] + } + ], + [ + { + "text": "GLM-ASR-Nano", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.026974", + "links": [] + }, + { + "text": "37.1x", + "links": [] + }, + { + "text": "31.07%", + "links": [] + }, + { + "text": "LLM-based multilingual ASR.", + "links": [] + } + ], + [ + { + "text": "Whisper-large-v3-turbo (OpenAI)", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.021708", + "links": [] + }, + { + "text": "46.1x", + "links": [] + }, + { + "text": "21.71%", + "links": [] + }, + { + "text": "OpenAI Whisper implementation.", + "links": [] + } + ], + [ + { + "text": "Whisper-large-v3 (OpenAI)", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.074694", + "links": [] + }, + { + "text": "13.4x", + "links": [] + }, + { + "text": "20.02%", + "links": [] + }, + { + "text": "Baseline for large Whisper quality.", + "links": [] + } + ], + [ + { + "text": "SenseVoice-Small", + "links": [] + }, + { + "text": "CPU", + "links": [] + }, + { + "text": "0.057988", + "links": [] + }, + { + "text": "17.2x", + "links": [] + }, + { + "text": "7.81%", + "links": [] + }, + { + "text": "CPU run from the remaining benchmark script.", + "links": [] + } + ], + [ + { + "text": "Paraformer-Large", + "links": [] + }, + { + "text": "CPU", + "links": [] + }, + { + "text": "0.064056", + "links": [] + }, + { + "text": "15.6x", + "links": [] + }, + { + "text": "10.18%", + "links": [] + }, + { + "text": "CPU viable for batch jobs.", + "links": [] + } + ], + [ + { + "text": "Fun-ASR-Nano", + "links": [] + }, + { + "text": "CPU", + "links": [] + }, + { + "text": "0.274318", + "links": [] + }, + { + "text": "3.6x", + "links": [] + }, + { + "text": "8.06%", + "links": [] + }, + { + "text": "LLM-based model is heavier but still above realtime.", + "links": [] + } + ] + ] + }, + { + "section_id": "choose", + "rows": [ + [ + { + "text": "Need", + "links": [] + }, + { + "text": "Recommended model", + "links": [] + } + ], + [ + { + "text": "Fastest production transcription", + "links": [] + }, + { + "text": "SenseVoice-Small or Paraformer-Large.", + "links": [] + } + ], + [ + { + "text": "CPU batch transcription", + "links": [] + }, + { + "text": "SenseVoice-Small first; Paraformer-Large for Chinese production pipelines.", + "links": [] + } + ], + [ + { + "text": "Chinese/English/Japanese LLM-style recognition with dialect and accent coverage", + "links": [] + }, + { + "text": "Fun-ASR-Nano; use the separate Fun-ASR-MLT-Nano checkpoint for 31 languages, and use vLLM for higher LLM decoding throughput.", + "links": [ + "https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512", + "vllm.html" + ] + } + ], + [ + { + "text": "OpenAI-compatible local endpoint", + "links": [] + }, + { + "text": "funasr-server with model alias sensevoice, paraformer, or fun-asr-nano.", + "links": [ + "agent.html" + ] + } + ] + ] + } + ] + }, + { + "language": "zh", + "requested_url": "https://modelscope.github.io/FunASR/zh/benchmark.html", + "fetched_at_utc": "2026-09-07T11:03:50.243350+00:00", + "sha256": "af8cf07e9cf06c9ce5865b54f34581a2911ede699f750776634d8b6d22183462", + "tables": [ + { + "section_id": "summary", + "rows": [ + [ + { + "text": "指标", + "links": [] + }, + { + "text": "结果", + "links": [] + } + ], + [ + { + "text": "数据集", + "links": [] + }, + { + "text": "184 条中文长音频，总时长 11,539 秒，约 192.3 分钟。", + "links": [] + } + ], + [ + { + "text": "GPU", + "links": [] + }, + { + "text": "NVIDIA H100 80GB HBM3.", + "links": [] + } + ], + [ + { + "text": "最佳 GPU 速度", + "links": [] + }, + { + "text": "SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run.", + "links": [] + } + ], + [ + { + "text": "最佳 CPU 速度", + "links": [] + }, + { + "text": "SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime.", + "links": [] + } + ], + [ + { + "text": "基线", + "links": [] + }, + { + "text": "OpenAI Whisper-large-v3：GPU 上 13.4 倍实时。", + "links": [] + } + ] + ] + }, + { + "section_id": "table", + "rows": [ + [ + { + "text": "模型", + "links": [] + }, + { + "text": "设备", + "links": [] + }, + { + "text": "RTF", + "links": [] + }, + { + "text": "速度", + "links": [] + }, + { + "text": "CER", + "links": [] + }, + { + "text": "说明", + "links": [] + } + ], + [ + { + "text": "SenseVoice-Small", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.005896", + "links": [] + }, + { + "text": "169.6x", + "links": [] + }, + { + "text": "7.81%", + "links": [] + }, + { + "text": "ASR + 语种 / 情感 / 事件标签；CER 已去除标签后计算。", + "links": [] + } + ], + [ + { + "text": "Paraformer-Large", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.008359", + "links": [] + }, + { + "text": "119.6x", + "links": [] + }, + { + "text": "10.18%", + "links": [] + }, + { + "text": "高速非自回归中文 ASR，适合 VAD/标点生产流水线。", + "links": [] + } + ], + [ + { + "text": "Fun-ASR-Nano", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.058803", + "links": [] + }, + { + "text": "17.0x", + "links": [] + }, + { + "text": "8.06%", + "links": [] + }, + { + "text": "LLM-based 中/英/日 ASR，另覆盖 7 类中文方言和 26 种地域口音，支持热词；不提供可靠的 checkpoint 原生时间戳（#106）。", + "links": [ + "https://github.com/QwenAudio/Fun-ASR/issues/106" + ] + } + ], + [ + { + "text": "GLM-ASR-Nano", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.026974", + "links": [] + }, + { + "text": "37.1x", + "links": [] + }, + { + "text": "31.07%", + "links": [] + }, + { + "text": "LLM-based 多语种 ASR。", + "links": [] + } + ], + [ + { + "text": "Whisper-large-v3-turbo (OpenAI)", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.021708", + "links": [] + }, + { + "text": "46.1x", + "links": [] + }, + { + "text": "21.71%", + "links": [] + }, + { + "text": "OpenAI Whisper 实现。", + "links": [] + } + ], + [ + { + "text": "Whisper-large-v3 (OpenAI)", + "links": [] + }, + { + "text": "GPU", + "links": [] + }, + { + "text": "0.074694", + "links": [] + }, + { + "text": "13.4x", + "links": [] + }, + { + "text": "20.02%", + "links": [] + }, + { + "text": "基线 for large Whisper quality.", + "links": [] + } + ], + [ + { + "text": "SenseVoice-Small", + "links": [] + }, + { + "text": "CPU", + "links": [] + }, + { + "text": "0.057988", + "links": [] + }, + { + "text": "17.2x", + "links": [] + }, + { + "text": "7.81%", + "links": [] + }, + { + "text": "CPU 结果来自 remaining benchmark 脚本。", + "links": [] + } + ], + [ + { + "text": "Paraformer-Large", + "links": [] + }, + { + "text": "CPU", + "links": [] + }, + { + "text": "0.064056", + "links": [] + }, + { + "text": "15.6x", + "links": [] + }, + { + "text": "10.18%", + "links": [] + }, + { + "text": "CPU 上可用于批量任务。", + "links": [] + } + ], + [ + { + "text": "Fun-ASR-Nano", + "links": [] + }, + { + "text": "CPU", + "links": [] + }, + { + "text": "0.274318", + "links": [] + }, + { + "text": "3.6x", + "links": [] + }, + { + "text": "8.06%", + "links": [] + }, + { + "text": "LLM-based 模型更重，但仍高于实时。", + "links": [] + } + ] + ] + }, + { + "section_id": "choose", + "rows": [ + [ + { + "text": "需求", + "links": [] + }, + { + "text": "推荐模型", + "links": [] + } + ], + [ + { + "text": "最快生产转写", + "links": [] + }, + { + "text": "SenseVoice-Small 或 Paraformer-Large。", + "links": [] + } + ], + [ + { + "text": "CPU 批量转写", + "links": [] + }, + { + "text": "优先 SenseVoice-Small；中文生产流水线可选 Paraformer-Large。", + "links": [] + } + ], + [ + { + "text": "中/英/日及中文方言、口音的 LLM-style 识别", + "links": [] + }, + { + "text": "Fun-ASR-Nano；需要 31 语种时使用独立的 Fun-ASR-MLT-Nano checkpoint，并使用 vLLM 提升 LLM 解码吞吐。", + "links": [ + "https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512", + "vllm.html" + ] + } + ], + [ + { + "text": "OpenAI 兼容本地服务", + "links": [] + }, + { + "text": "使用 funasr-server，模型别名为 sensevoice、paraformer 或 fun-asr-nano。", + "links": [ + "agent.html" + ] + } + ] + ] + } + ] + } + ] +} diff --git a/web-pages/product-site/tests/test_documentation.py b/web-pages/product-site/tests/test_documentation.py index 6f6433e0b..3eaa73455 100644 --- a/web-pages/product-site/tests/test_documentation.py +++ b/web-pages/product-site/tests/test_documentation.py @@ -209,3 +209,58 @@ def test_agent_recipes_are_bilingual_and_use_existing_script_options(): 'examples/voice_input/funasr_input.py', 'examples/subtitle/generate_subtitle.py'} assert recipes[0] == recipes[1] + + +@pytest.mark.parametrize('language,prefix,suffix', [('en', 'en/', ''), ('zh', '', '_zh')]) +def test_historical_benchmark_preserves_frozen_tables_and_discovery(output, language, prefix, suffix): + frozen = json.loads((SITE_ROOT / 'tests/fixtures/historical_asr_tables.json').read_text()) + record = next(item for item in frozen['pages'] if item['language'] == language) + route = '/' + prefix + 'docs/historical-asr-benchmark.html' + path = output / route.lstrip('/') + assert path.is_file(), route + page = BeautifulSoup(path.read_text(), 'html.parser') + article = page.select_one('.docs-article') + tables = article.select('table') + assert len(tables) == 3 + assert len(article.select('h2')) == 4 + for table, expected in zip(tables, record['tables']): + actual = [[' '.join(cell.get_text().split()) for cell in row.select('th,td')] + for row in table.select('tr')] + assert actual == [[cell['text'] for cell in row] for row in expected['rows']] + assert page.select_one('[data-source-link]')['href'].endswith( + f'/docs/benchmark/historical_asr{suffix}.md') + links = {link['href'] for link in article.select('a[href]')} + for slug in ('agent-integration', 'vllm', 'model-selection', 'benchmark-method', 'realtime-benchmark'): + assert '/' + prefix + 'docs/' + slug + '.html' in links + assert ('/docs/' if prefix else '/en/docs/') + 'historical-asr-benchmark.html' in links + assert 'https://github.com/QwenAudio/Fun-ASR/issues/106' in links + assert 'https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512' in links + original_path = urlsplit(record['requested_url']).path.removeprefix('/FunASR/') + assert f"https://github.com/modelscope/FunASR/blob/{frozen['source_commit']}/{original_path}" in links + index = BeautifulSoup((output / prefix / 'docs/index.html').read_text(), 'html.parser') + assert index.select_one(f'a[href="{route}"]') + search = json.loads((output / f'search-{language}.json').read_text()) + assert any(row['url'] == route for row in search) + + +@pytest.mark.parametrize('suffix', ['', '_zh']) +def test_historical_benchmark_does_not_advertise_missing_scripts_as_runnable(suffix): + root = SITE_ROOT.parents[1] + source_path = root / f'docs/benchmark/historical_asr{suffix}.md' + assert source_path.is_file() + source = source_path.read_text() + blocks = re.findall(r'^```(\w+)\n(.*?)^```', source, re.M | re.S) + assert len(blocks) == 2 and all(language == 'text' for language, _ in blocks) + commands = [line for _, block in blocks for line in block.splitlines() if line.startswith('python ')] + assert commands == ['python benchmark/run_full_benchmark.py', + 'python benchmark/run_remaining.py', + 'python benchmark/fix_sensevoice_cer.py'] + for command in commands: + assert not (root / command.split()[1]).exists() + for marker in ('11,539', '11,541', '169.6x', '211.8x', 'RTFx', '2026-09-07'): + assert marker in source + for marker in (('来源不完整', '不是测量日期', '不能直接执行', '不计算 CER/WER') if suffix + else ('incomplete provenance', 'not the measurement date', 'cannot be run directly', 'does not compute CER/WER')): + assert marker in source + if not suffix: + assert 'Reproduce the 184-file long-audio benchmark' not in (root / 'docs/use_case_showcase.md').read_text() diff --git a/web-pages/product-site/tests/test_legacy_doc_anchors.py b/web-pages/product-site/tests/test_legacy_doc_anchors.py index 56ce76686..f1c953b1e 100644 --- a/web-pages/product-site/tests/test_legacy_doc_anchors.py +++ b/web-pages/product-site/tests/test_legacy_doc_anchors.py @@ -21,6 +21,7 @@ OLD_IDS = { 'deployment-matrix.html': 'matrix choices runtime-diagnostics checklist help', # Independently inventoried from public EN/ZH pages on 2026-09-07. 'agent.html': 'server sdk workflows mcp voice subtitle', + 'benchmark.html': 'summary table method choose', } PAGES = [prefix + name for name in OLD_IDS for prefix in ('', 'zh/')] @@ -151,6 +152,15 @@ def test_agent_legacy_fragments_keep_their_topics(page, targets): assert pages[page] == dict(zip('server sdk workflows mcp voice subtitle'.split(), targets)) +@pytest.mark.parametrize('page,targets', [ + ('benchmark.html', ['historical-summary', 'historical-results', 'provenance-and-limitations', 'choosing-a-current-path']), + ('zh/benchmark.html', ['历史概览', '历史结果', '来源与限制', '当前选型']), +]) +def test_benchmark_legacy_fragments_keep_their_topics(page, targets): + pages = json.loads((SITE / 'data/legacy_doc_anchors.json').read_text())['pages'] + assert pages[page] == dict(zip('summary table method choose'.split(), targets)) + + @pytest.mark.parametrize('heading', ('

Topic

', '

Not a heading

')) def test_missing_or_nonheading_target_fails_explicitly(heading): from export_docs import insert_legacy_anchors