Files
FunASR/native-transformers.html
T
2026-09-09 17:33:15 +00:00

345 lines
20 KiB
HTML

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<title>Transformers quickstart - FunASR</title>
<meta content="Transformers quickstart" name="description"/>
<meta content="#ffffff" name="theme-color"/>
<link href="https://www.funasr.com/en/docs/native-transformers.html" rel="canonical"/>
<link href="https://www.funasr.com/docs/native-transformers.html" hreflang="zh-CN" rel="alternate"/>
<link href="https://www.funasr.com/en/docs/native-transformers.html" hreflang="en" rel="alternate"/>
<meta content="website" property="og:type"/>
<meta content="Transformers quickstart - FunASR" property="og:title"/>
<meta content="Transformers quickstart" property="og:description"/>
<meta content="https://www.funasr.com/en/docs/native-transformers.html" property="og:url"/>
<link href="assets/css/site.7be954c8e106.css" rel="stylesheet"/>
<link href="assets/css/experience.34d6e632686e.css" rel="stylesheet"/>
<script type="application/ld+json">{"@context": "https://schema.org", "@type": "SoftwareApplication", "applicationCategory": "DeveloperApplication", "codeRepository": "https://github.com/modelscope/FunASR", "dateModified": "2026-09-01", "license": "https://github.com/modelscope/FunASR/blob/main/LICENSE", "name": "FunASR", "operatingSystem": "Linux, macOS, Windows", "url": "https://www.funasr.com/en/docs/native-transformers.html"}</script></head>
<body data-copy-icon="assets/lucide/copy.ea80e566c7a1.svg" data-language="en" data-route="/en/docs/native-transformers.html">
<a class="skip-link" href="#main">Skip to main content</a>
<header class="site-header">
<a aria-label="FunASR home" class="brand" href="https://www.funasr.com/en/">
<span aria-hidden="true" class="brand-mark">F</span>
<span>FunASR</span>
</a>
<button aria-controls="primary-navigation" aria-expanded="false" class="icon-button menu-button" data-menu-toggle="" title="Open navigation" type="button">
<img alt="" src="assets/lucide/menu.90a31b3ab1cf.svg"/>
</button>
<nav aria-label="Primary navigation" class="primary-navigation" data-primary-nav="" id="primary-navigation">
<a href="https://www.funasr.com/en/">Product</a>
<a href="https://www.funasr.com/en/deploy/">Deploy</a>
<a href="https://www.funasr.com/en/models.html">Models</a>
<a href="https://www.funasr.com/en/benchmarks.html">Benchmarks</a>
<a href="https://www.funasr.com/en/ecosystem.html">Ecosystem</a>
<a href="https://www.funasr.com/en/blog/">Blog</a>
<a href="https://www.funasr.com/en/docs/">Docs</a>
<a href="https://www.funasr.com/en/donors.html">Donors</a>
</nav>
<div class="header-actions">
<a class="icon-link" href="https://www.funasr.com/docs/native-transformers.html" title="简体中文">
<img alt="" src="assets/lucide/languages.ae9048feb2c7.svg"/>
<span>中文</span>
</a>
<a aria-label="GitHub" class="icon-button" href="https://www.funasr.com/go/github" title="GitHub">
<img alt="" src="assets/lucide/github.793554449a30.svg"/>
</a>
</div>
</header>
<main id="main">
<div class="docs-layout">
<aside class="docs-sidebar">
<a class="docs-home" href="https://www.funasr.com/en/docs/">Documentation</a>
<details class="docs-navigation" open="">
<summary class="docs-navigation-toggle">Browse topics</summary>
<details open="">
<summary>Get started</summary>
<nav aria-label="Get started">
<a aria-current="page" href="https://www.funasr.com/en/docs/native-transformers.html">Transformers quickstart</a>
<a href="https://www.funasr.com/en/docs/installation.html">Installation &amp; environment checks</a>
<a href="https://www.funasr.com/en/docs/quickstart.html">Your first transcription</a>
<a href="https://www.funasr.com/en/docs/python-api.html">Python SDK: AutoModel</a>
<a href="https://www.funasr.com/en/docs/model-selection.html">Choose your first model</a>
<a href="https://www.funasr.com/en/docs/command-line.html">Command-line transcription</a>
<a href="https://www.funasr.com/en/docs/migration.html">Migrate from Whisper &amp; cloud APIs</a>
</nav>
</details>
<details open="">
<summary>Models &amp; capabilities</summary>
<nav aria-label="Models &amp; capabilities">
<a href="https://www.funasr.com/en/docs/moss-transcribe-diarize.html">MOSS transcription &amp; diarization</a>
<a href="https://www.funasr.com/en/docs/model-zoo.html">Model Zoo</a>
<a href="https://www.funasr.com/en/docs/streaming-vad.html">Streaming voice activity detection</a>
<a href="https://www.funasr.com/en/docs/keyword-spotting.html">Keyword spotting</a>
<a href="https://www.funasr.com/en/docs/speaker-emotion.html">Speakers and emotion tags</a>
</nav>
</details>
<details open="">
<summary>Train &amp; extend</summary>
<nav aria-label="Train &amp; extend">
<a href="https://www.funasr.com/en/docs/training.html">Fine-tuning &amp; evaluation</a>
<a href="https://www.funasr.com/en/docs/model-registration.html">Custom model registration</a>
</nav>
</details>
<details open="">
<summary>Deploy &amp; integrate</summary>
<nav aria-label="Deploy &amp; integrate">
<a href="https://www.funasr.com/en/docs/docker.html">Containers &amp; image selection</a>
<a href="https://www.funasr.com/en/docs/runtime-guide.html">Serving entry points</a>
<a href="https://www.funasr.com/en/docs/http-server.html">Python HTTP service</a>
<a href="https://www.funasr.com/en/docs/kubernetes.html">Kubernetes deployment</a>
<a href="https://www.funasr.com/en/docs/deployment-matrix.html">Runtime &amp; deployment matrix</a>
<a href="https://www.funasr.com/en/docs/vllm.html">vLLM split-engine inference</a>
<a href="https://www.funasr.com/en/docs/official-native-vllm.html">Official native vLLM functional validation (2026-09-07)</a>
<a href="https://www.funasr.com/en/docs/native-vllm.html">Historical community native vLLM record (2026-08-13)</a>
</nav>
</details>
<details open="">
<summary>Evaluate &amp; operate</summary>
<nav aria-label="Evaluate &amp; operate">
<a href="https://www.funasr.com/en/docs/troubleshooting.html">Troubleshooting</a>
<a href="https://www.funasr.com/en/docs/security.html">Service security boundaries</a>
<a href="https://www.funasr.com/en/docs/benchmark-method.html">Reproducible performance tests</a>
<a href="https://www.funasr.com/en/docs/historical-asr-benchmark.html">Historical ASR benchmark</a>
<a href="https://www.funasr.com/en/docs/realtime-benchmark.html">Realtime WebSocket benchmarking</a>
</nav>
</details>
<details open="">
<summary>Applications &amp; ecosystem</summary>
<nav aria-label="Applications &amp; ecosystem">
<a href="https://www.funasr.com/en/docs/agent-integration.html">Agent integration</a>
<a href="https://www.funasr.com/en/docs/use-cases.html">Use cases &amp; examples</a>
<a href="https://www.funasr.com/en/docs/community.html">Community integrations</a>
<a href="https://www.funasr.com/en/docs/repositories.html">Repositories &amp; contribution paths</a>
<a href="https://www.funasr.com/en/docs/javascript.html">JavaScript clients</a>
<a href="https://www.funasr.com/en/docs/gradio.html">Gradio browser client</a>
<a href="https://www.funasr.com/en/docs/workflows.html">Workflow integrations</a>
</nav>
</details>
<details open="">
<summary>Protocols &amp; reference</summary>
<nav aria-label="Protocols &amp; reference">
<a href="https://www.funasr.com/en/docs/service-api.html">HTTP API contract</a>
<a href="https://www.funasr.com/en/docs/websocket-protocol.html">C++ WebSocket protocol</a>
<a href="https://www.funasr.com/en/docs/onnx-output.html">ONNX binary output contract</a>
</nav>
</details>
<a href="https://www.funasr.com/en/deploy/">Deployment guides →</a>
</details>
</aside>
<div class="docs-body">
<nav aria-label="Breadcrumbs" class="breadcrumbs"><a href="https://www.funasr.com/en/">FunASR</a><span>/</span><a href="https://www.funasr.com/en/docs/">Docs</a><span>/</span><span>Transformers quickstart</span></nav>
<header class="docs-title"><h1>Transformers quickstart</h1><a class="source-link" data-source-link="" href="https://github.com/modelscope/FunASR/blob/main/docs/transformers_native.md">View / edit source ↗</a></header>
<article class="docs-article"><div class="source-title-anchor" id="fun-asr-nano-with-transformers"></div>
<p><a href="https://www.funasr.com/docs/native-transformers.html">简体中文</a> | English</p>
<p>Transcribe Chinese, English and Japanese with <code>AutoProcessor</code> and
<code>AutoModelForSpeechSeq2Seq</code>. <strong>Transformers 5.17.0 is a released package with
native Fun-ASR-Nano support.</strong> No source checkout, FunASR toolkit or
checkpoint-provided Python code is needed.</p>
<p><a href="https://huggingface.co/spaces/FunAudioLLM/Fun-ASR-Nano">Try the Space</a> ·
<a href="https://colab.research.google.com/github/QwenAudio/Fun-ASR/blob/main/examples/colab/fun_asr_nano_transformers.ipynb">Open the notebook</a> ·
<a href="https://github.com/QwenAudio/Fun-ASR/tree/main/examples/transformers">Runnable examples</a></p>
<h2 id="get-your-first-transcript">Get your first transcript</h2>
<p>Start in an isolated environment. This reproducible installation is for Linux
x86-64, Python 3.12 and <strong>CPU</strong>; do not upgrade an existing vLLM or FunASR service
environment in place. Torch and torchaudio must match because the native
feature extractor uses <code>torchaudio.compliance.kaldi.fbank</code>.</p>
<pre id="code-0"><code class="language-bash">python3.12 -m venv .venv-funasr-native
. .venv-funasr-native/bin/activate
python -m pip install --index-url https://download.pytorch.org/whl/cpu \
'torch==2.10.0+cpu' 'torchaudio==2.10.0+cpu'
python -m pip install 'transformers==5.17.0' 'numpy==1.26.4' \
'librosa==0.11.0' 'soundfile==0.13.1' \
'huggingface-hub==1.30.0' 'tokenizers==0.23.2'
python -m pip check
</code></pre>
<p>Run this Python example. It downloads about 1.66 GB of model weights on first
use and transcribes a short, pinned official English sample. CPU float32 is
explicit; no GPU is needed for this example.</p>
<!-- native-example: transcribe -->
<pre id="code-1"><code class="language-python">import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
torch.set_num_threads(4)
model_id = "FunAudioLLM/Fun-ASR-Nano-2512-hf"
revision = "d93b302ee7fd505e1b3576120fc142fc6f7820e1"
audio = "https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512/resolve/272c57b82523ada6fd87095e955f8e29100979ab/example/en.mp3"
processor = AutoProcessor.from_pretrained(
model_id, revision=revision, trust_remote_code=False, token=False
)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
model_id, revision=revision, trust_remote_code=False, token=False,
dtype=torch.float32,
).to("cpu").eval()
inputs = processor.apply_transcription_request(
audio=audio, language="en",
processor_kwargs={
"return_tensors": "pt",
"audio_kwargs": {"sampling_rate": 16000},
"text_kwargs": {"padding": True},
},
)
with torch.inference_mode():
generated = model.generate(**inputs, max_new_tokens=128, do_sample=False)
new_tokens = generated[:, inputs.input_ids.shape[1]:]
print(processor.batch_decode(new_tokens, skip_special_tokens=True)[0])
</code></pre>
<p>The model returns the prompt and generated tokens together. Remove the full
input tensor width before decoding. For this sample, the observed raw output
was: “The tribal chieftain called for the boy, and presented him with fifty
pieces of gold.” This is a functional example, not a quality guarantee.</p>
<h2 id="your-files-keywords-and-batches">Your files, keywords and batches</h2>
<p>The <a href="https://github.com/QwenAudio/Fun-ASR/tree/main/examples/transformers">command-line example</a>
adds explicit input validation, sample-rate conversion and completion checks.
Clone the <strong>QwenAudio/Fun-ASR code repository</strong> (not an HF weights repository),
install the <a href="https://github.com/QwenAudio/Fun-ASR/tree/main/examples/transformers">independent native requirements</a>,
and run these commands from its root directory:</p>
<pre id="code-2"><code class="language-bash">python examples/transformers/transcribe.py recording.wav --language zh --keywords 开放时间
python examples/transformers/transcribe.py chinese.wav english.wav --language zh en
</code></pre>
<p>It averages multichannel audio to mono and resamples to 16 kHz with <code>soxr_hq</code>;
it never overwrites the original file. It accepts 1-4 files with at most
60 seconds of total audio. Longer recordings, empty input and non-finite samples
are rejected, not silently trimmed.</p>
<p>For your own application, pass a waveform, local path or supported URL to
<code>processor.apply_transcription_request</code>. A raw waveform must have its true
sample rate; do not label 48 kHz samples as 16 kHz. Use <code>language="zh"</code>,
<code>"en"</code> or <code>"ja"</code>. For a batch, use an audio list, matching language list and
<code>text_kwargs={"padding": True}</code>. Preserve input order when decoding.</p>
<p><code>keywords</code> supplies vocabulary hints and <code>prompt</code> supplies context. They are
not enforced vocabulary and are not the toolkit's <code>hotword</code> parameter. In our
Chinese sample, adding <code>开放时间</code> did not correct the raw output <code>开饭时间</code>.
Use representative recordings and human references to assess quality.</p>
<p>The short example bounds generation at 128 new tokens. Missing EOS can indicate
truncation; increasing the limit does not prove coverage of a long recording.
The CLI reports <code>reached_eos</code> and exits unsuccessfully for missing EOS or empty
text. EOS itself is not proof that all words were recognized.</p>
<h2 id="from-an-example-to-a-service">From an example to a service</h2>
<table>
<thead>
<tr>
<th>Need</th>
<th>Checkpoint and interface</th>
</tr>
</thead>
<tbody>
<tr>
<td>Native Transformers in Python</td>
<td><code>FunAudioLLM/Fun-ASR-Nano-2512-hf</code>, <code>AutoProcessor</code> + <code>AutoModelForSpeechSeq2Seq</code></td>
</tr>
<tr>
<td>FunASR pipelines and HTTP adapters</td>
<td><code>FunAudioLLM/Fun-ASR-Nano-2512</code>, <code>funasr.AutoModel</code>; <a href="https://www.funasr.com/en/docs/deployment-matrix.html">deployment matrix</a></td>
</tr>
<tr>
<td>Native vLLM</td>
<td><code>FunAudioLLM/Fun-ASR-Nano-2512-vllm</code>; <a href="https://www.funasr.com/en/docs/official-native-vllm.html">native vLLM guide</a></td>
</tr>
<tr>
<td>C++ / edge</td>
<td>Converted GGUF files; <a href="https://github.com/modelscope/FunASR/blob/main/runtime/llama.cpp/README.md">llama.cpp</a></td>
</tr>
</tbody>
</table>
<p>These are different artifacts, not interchangeable model IDs. Native
Transformers support does not add an HTTP server, queue, streaming protocol,
word timestamps or speaker identities. This native export omits the CTC
timestamp branch. The base Nano checkpoint covers zh/en/ja and Chinese
dialects/accents; the <strong>31-language MLT</strong> checkpoint is separate.</p>
<p>For transcription with anonymous speakers and segment timestamps, see the
third-party OpenMOSS <a href="https://www.funasr.com/en/docs/moss-transcribe-diarize.html">MOSS guide</a>. For other tasks,
use the <a href="https://www.funasr.com/en/docs/model-zoo.html">Model Zoo</a>. Validate GPU dtype, attention
backends, memory, concurrency and quality on the actual target hardware.</p>
<h2 id="troubleshooting-without-loading-weights">Troubleshooting without loading weights</h2>
<p>If loading fails, check the active interpreter, <code>transformers.__version__</code>,
the <code>-hf</code> model ID and model revision first. Version 5.16.1 lacks this model;
upgrade in the new environment, not an unrelated running service. The native
classes live in <code>transformers.models.fun_asr_nano</code>.</p>
<p>This optional synthetic-silence check downloads the processor, not model weights:</p>
<!-- native-example: processor -->
<pre id="code-3"><code class="language-python">import numpy as np
from transformers import AutoProcessor
model_id = "FunAudioLLM/Fun-ASR-Nano-2512-hf"
revision = "d93b302ee7fd505e1b3576120fc142fc6f7820e1"
processor = AutoProcessor.from_pretrained(
model_id, revision=revision, trust_remote_code=False, token=False
)
inputs = processor.apply_transcription_request(
audio=np.zeros(16000, dtype=np.float32), language="en",
processor_kwargs={
"return_tensors": "pt",
"audio_kwargs": {"sampling_rate": 16000},
},
)
print({name: tuple(value.shape) for name, value in inputs.items()})
</code></pre>
<p>Reject empty recordings and batches before calling the processor. Synthetic
preprocessing success is <strong>not an accuracy or capacity result</strong>.</p>
<h2 id="reproducibility-and-sources">Reproducibility and sources</h2>
<p>Verified on <strong>2026-09-09</strong> using the actual released 5.17.0 wheel, Python 3.12.3,
matching torch/torchaudio 2.10.0+cpu, float32 and four Torch threads on an Intel
Xeon Platinum 8480+. Official English URL, Chinese waveform, Chinese keywords
and a padded Chinese/English batch all returned non-empty text and EOS before
128 tokens. These two short public samples do not establish CER/WER, GPU
compatibility or serving capacity.</p>
<p>The public native revision is
<code>d93b302ee7fd505e1b3576120fc142fc6f7820e1</code>; it resolves to
<code>FunAsrNanoForConditionalGeneration</code> with <code>trust_remote_code=False</code>.
<code>model.safetensors</code> is 1,659,773,320 bytes, SHA256
<code>1bbb6dcc5d8b75084a399d48c4d4b0f3aa1d3f09f2616ae40ed2c4fba03d89c9</code>.
Audio samples come from the original repository's
<a href="https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512/tree/272c57b82523ada6fd87095e955f8e29100979ab/example">fixed example directory</a>,
not an invented <code>-hf/example</code> path.</p>
<p><a href="https://github.com/huggingface/transformers/pull/46180">PR #46180</a> merged as
<code>fc501343edfccdc840eb8594a6cafa8185c2de53</code> before the stable package was
published. Earlier source-only instructions described that interval; new
installations should use the released package above.
See the <a href="https://huggingface.co/docs/transformers/v5.17.0/en/model_doc/fun_asr_nano">official 5.17.0 model documentation</a>
and <a href="https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512-hf">native model card</a>.</p>
<p>Keep versions, input hashes and raw results with application evaluations.
Never attach private recordings or credentials to a public issue.</p></article>
<nav aria-label="Adjacent documents" class="docs-next">
<a href="https://www.funasr.com/en/docs/installation.html">Installation &amp; environment checks →</a> </nav>
</div>
<aside class="docs-toc"><strong>On this page</strong><div class="toc">
<ul>
<li><a href="#fun-asr-nano-with-transformers">Fun-ASR-Nano with Transformers</a><ul>
<li><a href="#get-your-first-transcript">Get your first transcript</a></li>
<li><a href="#your-files-keywords-and-batches">Your files, keywords and batches</a></li>
<li><a href="#from-an-example-to-a-service">From an example to a service</a></li>
<li><a href="#troubleshooting-without-loading-weights">Troubleshooting without loading weights</a></li>
<li><a href="#reproducibility-and-sources">Reproducibility and sources</a></li>
</ul>
</li>
</ul>
</div>
</aside></div>
</main>
<footer class="site-footer">
<div>
<strong>FunASR</strong>
<p>Open speech infrastructure for production deployment.</p>
</div>
<nav aria-label="Official repositories" class="footer-repositories" data-footer-repositories="">
<strong>Official repositories</strong>
<div class="footer-links">
<a href="https://www.funasr.com/go/fun-asr">Fun-ASR</a>
<a href="https://www.funasr.com/go/sensevoice">SenseVoice</a>
<a href="https://www.funasr.com/go/funclip">FunClip</a>
</div>
</nav>
<div class="footer-links">
<a href="https://www.funasr.com/en/deploy/">Deploy</a>
<a href="https://www.funasr.com/en/docs/">Docs</a>
<a href="https://www.funasr.com/go/releases">Releases</a>
<a href="https://www.funasr.com/en/donors.html">Donors</a>
</div>
</footer>
<script defer="" src="assets/js/site.d446f5fbcfe2.js"></script>
<script defer="" src="assets/js/experience.3f688a12942a.js"></script>
</body>
</html>