docs: restore vLLM guide — add English version and Chinese (vllm_guide_zh.md)

The original docs/vllm_guide.md was accidentally emptied. Restored the full content as docs/vllm_guide_zh.md (Chinese) and created docs/vllm_guide.md (English translation) as the primary guide.
2026-10-12 02:19:18 +00:00 · 2026-05-26 02:25:58 +08:00
parent 7d3c2161db
commit bb92a779fd
2 changed files with 1204 additions and 0 deletions
@@ -0,0 +1,602 @@
+# FunASR vLLM Inference Engine Guide
+
+FunASR integrates the vLLM high-throughput inference engine to accelerate autoregressive decoding of LLM-based ASR models. Three modes are supported: offline batch inference, streaming SDK inference, and production-grade WebSocket real-time service.
+
+---
+
+## Table of Contents
+
+1. [Overview](#1-overview)
+2. [Installation & Environment](#2-installation--environment)
+3. [Offline Batch Inference](#3-offline-batch-inference)
+4. [Streaming SDK Inference](#4-streaming-sdk-inference)
+5. [Real-time WebSocket Service](#5-real-time-websocket-service)
+6. [Performance Comparison](#6-performance-comparison)
+7. [API Reference](#7-api-reference)
+8. [FAQ](#8-faq)
+
+---
+
+## 1. Overview
+
+### Supported Models
+
+| Model | vLLM Support | Notes |
+|-------|-------------|-------|
+| **FunASRNano** | ✓ | Qwen3-0.6B LLM, supports offline and streaming |
+| **LLMASR** | ✓ | Whisper + Qwen/Vicuna/LLaMA |
+| **GLMASR** | ✓ | GLM-ASR-Nano |
+| **QwenAudioWarp** | ✓ | Qwen-Audio |
+| Paraformer | ✗ | Non-autoregressive model (CIF predictor), no LLM decoding |
+| SenseVoice | ✗ | Whisper-like encoder-decoder, not LLM |
+| Conformer/Transformer | ✗ | CTC/attention decoding, not LLM |
+
+### Architecture
+
+```
+┌────────────────────────────────────────────────────────┐
+│                   FunASR + vLLM                         │
+├────────────────────────────────────────────────────────┤
+│                                                        │
+│  Audio ──→ Frontend ──→ Audio Encoder ──→ Adaptor      │
+│            (fbank)      (SenseVoice)     (Transformer) │
+│                                │                       │
+│                                ▼                       │
+│                       Audio Embeddings                  │
+│                                │                       │
+│  Text Prompt ──→ Tokenize ──→ Embed ──→ [Concat]      │
+│  (hotwords/language/itn)                    │          │
+│                                             ▼          │
+│                                    ┌──────────────┐    │
+│                                    │  vLLM Engine │    │
+│                                    │  (Qwen3 etc) │    │
+│                                    │  TP parallel │    │
+│                                    └──────┬───────┘    │
+│                                           ▼            │
+│                                    Generated Text      │
+│                                                        │
+│  (Optional) CTC Forced Alignment ──→ Character-level   │
+│                                      timestamps        │
+└────────────────────────────────────────────────────────┘
+```
+
+### Three Usage Modes
+
+| Mode | Entry Point | Use Case |
+|------|-------------|----------|
+| [Offline Batch Inference](#3-offline-batch-inference) | `AutoModelVLLM` / `FunASRNanoVLLM` | Large-scale transcription, batch processing |
+| [Streaming SDK Inference](#4-streaming-sdk-inference) | `FunASRNanoStreamingVLLM` | Real-time subtitles (SDK integration) |
+| [WebSocket Real-time Service](#5-real-time-websocket-service) | `serve_realtime_ws.py` | Production deployment, multi-client access |
+
+---
+
+## 2. Installation & Environment
+
+### Dependencies
+
+```bash
+pip install funasr>=1.3.0
+pip install vllm>=0.12.0
+pip install safetensors tiktoken websockets regex
+
+# FunASR development mode install (requires auto_model_vllm)
+cd /path/to/FunASR && pip install -e .
+```
+
+### Hardware Requirements
+
+| Configuration | Minimum | Recommended |
+|--------------|---------|-------------|
+| GPU Memory | 8GB | 16GB+ |
+| CUDA | 11.8 | 12.0+ |
+| GPU Count | 1 | 2+ (tensor parallel) |
+
+---
+
+## 3. Offline Batch Inference
+
+Best for large-scale audio transcription and offline batch processing. vLLM's batching capability provides the greatest advantage in this scenario.
+
+### Universal Interface (Recommended)
+
+```python
+from funasr.auto.auto_model_vllm import AutoModelVLLM
+
+model = AutoModelVLLM(
+    model="FunAudioLLM/Fun-ASR-Nano-2512",
+    hub="ms",                    # or "hf"
+    tensor_parallel_size=2,      # multi-GPU parallel
+    gpu_memory_utilization=0.8,
+)
+
+results = model.generate(
+    ["audio1.wav", "audio2.wav"],
+    language="中文",
+    hotwords=["张三", "北京"],
+)
+for r in results:
+    print(f"[{r['key']}] {r['text']}")
+```
+
+### Direct Interface
+
+```python
+from funasr.models.fun_asr_nano.inference_vllm import FunASRNanoVLLM
+
+engine = FunASRNanoVLLM.from_pretrained(
+    model="FunAudioLLM/Fun-ASR-Nano-2512",
+    tensor_parallel_size=4,
+)
+
+results = engine.generate(
+    inputs="wav.scp",  # supports scp/jsonl/file list
+    hotwords=["开放时间"],
+    language="中文",
+    max_new_tokens=512,
+)
+```
+
+### Command Line
+
+```bash
+cd examples/industrial_data_pretraining/fun_asr_nano
+
+# Single file
+python demo_vllm.py --input audio.wav --language 中文
+
+# Batch + multi-GPU
+python demo_vllm.py --input wav.scp --tensor-parallel-size 4 --batch-size 32
+
+# With hotwords + save results
+python demo_vllm.py --input audio.wav --hotwords 张三 北京 --output results.jsonl
+```
+
+---
+
+## 4. Streaming SDK Inference
+
+Processes audio in 720ms chunks incrementally, outputting progressively stable recognition results. Suitable for SDK-integrated real-time subtitle scenarios.
+
+### Design Principle
+
+```
+Audio stream (720ms chunks)
+    │ Cumulative re-encoding (each chunk contains all audio from start to current)
+    ▼
+┌──────────────────────────┐
+│ Stage 1: First 10 chunks │  ← No prev_text, batch generation
+│ Find stable output       │
+└──────────┬───────────────┘
+           ▼
+┌──────────────────────────┐
+│ Stage 2: Subsequent      │  ← Use stable output as prev_text
+└──────────┬───────────────┘
+           ▼
+Each chunk: [fixed region (confirmed)] + [8-char unfixed (may change)]
+```
+
+### Usage
+
+```python
+from funasr.models.fun_asr_nano.inference_vllm_streaming import FunASRNanoStreamingVLLM
+
+engine = FunASRNanoStreamingVLLM.from_pretrained(
+    model="FunAudioLLM/Fun-ASR-Nano-2512",
+    chunk_ms=720,
+    rollback_chars=8,
+)
+
+for result in engine.streaming_generate("audio.wav", language="中文"):
+    if result["is_final"]:
+        print(f"Final: {result['text']}")
+    else:
+        print(f"[{result['audio_duration_ms']:.0f}ms] Confirmed: {result['fixed_text']}")
+```
+
+### Output Characteristics
+
+| Accumulated Audio | Output Quality |
+|-------------------|---------------|
+| < 1.5s | Empty or noise |
+| 1.5-3.0s | Partially correct |
+| > 3.0s | Accurate output |
+
+> Note: `repetition_penalty=1.3` is hardcoded internally to prevent short-chunk repetition degradation.
+
+---
+
+## 5. Real-time WebSocket Service
+
+Production-grade real-time speech recognition service integrating VAD segmentation + vLLM inference + speaker diarization + hotwords.
+
+### Inference Logic
+
+**Core design: VAD endpoint-based segment decoding (not fixed chunks)**
+
+```
+Audio stream ──→ StreamingVAD (60ms) ──→ Endpoint detected ──→ vLLM decode segment
+                                    │                              │
+                                    │                              ▼
+                                    │                     locked_sentences (locked)
+                                    │                              │
+                                    │                              ▼
+                                    │                     SPK assign (speaker assignment)
+                                    │
+                                    └──→ Not ended ──→ vLLM partial decode (every 0.48s)
+                                                              │
+                                                              ▼
+                                                       partial text (preview, will be overwritten)
+```
+
+**Two inference paths:**
+
+| Path | Trigger Condition | Output |
+|------|-------------------|--------|
+| Confirmed segment decode | VAD detects silence endpoint | Locked to sentences, never changes |
+| Partial preview | Every 0.48s + new audio ≥ 960ms | Temporary text, overwritten anytime |
+
+**Dynamic VAD thresholds:**
+
+| Accumulated Duration | Silence Threshold | Effect |
+|---------------------|-------------------|--------|
+| ≤ 5s | 2.0s | Preserve short sentences |
+| 5-10s | 1.5s | Normal segmentation |
+| 10-15s | 1.0s | Start tightening |
+| 15-30s | 0.8s | Faster splitting |
+| 30-45s | 0.4s | Prevent overly long segments |
+| > 45s | 0.1s | Force split |
+
+**STOP final processing:**
+1. Feed remaining audio to VAD (is_final=True)
+2. Force-end currently speaking segment
+3. Global SPK re-clustering (correct speaker IDs)
+4. Return `is_final: true`
+
+### Deployment
+
+```bash
+cd examples/industrial_data_pretraining/fun_asr_nano
+
+# Single GPU
+CUDA_VISIBLE_DEVICES=0 python serve_realtime_ws.py --port 10095 --language 中文
+
+# Multi-GPU
+CUDA_VISIBLE_DEVICES=0,1 python serve_realtime_ws.py \
+    --port 10095 --tensor-parallel-size 2 --language 中文
+
+# Full parameters
+python serve_realtime_ws.py \
+    --port 10095 \
+    --model FunAudioLLM/Fun-ASR-Nano-2512 \
+    --hub ms \
+    --device cuda:0 \
+    --decode-interval 0.48 \
+    --hotword-file hotword_list \
+    --language 中文 \
+    --dtype bf16 \
+    --tensor-parallel-size 1 \
+    --gpu-memory-utilization 0.8 \
+    --max-model-len 2048
+```
+
+### Clients
+
+| Client | Usage |
+|--------|-------|
+| Browser | Open `client_mic.html` (microphone/file/hotwords/speakers) |
+| Python CLI | `python client_python.py --server ws://localhost:10095 --mic` |
+| Test script | `python client_test.py --server ws://localhost:10095 --file audio.wav` |
+
+Remote access requires SSH port forwarding: `ssh -L 10095:localhost:10095 <server>`
+
+### WebSocket Protocol
+
+**Client → Server:**
+
+| Message | Format | Description |
+|---------|--------|-------------|
+| Start session | `"START"` | Initialize session |
+| Set hotwords | `"HOTWORDS:word1,word2"` | Optional |
+| Set language | `"LANGUAGE:中文"` | Optional |
+| Audio data | `bytes` | PCM16, 16kHz, mono |
+| End session | `"STOP"` | Trigger final decoding |
+
+**Server → Client:**
+
+```json
+// Events
+{"event": "started"}
+{"event": "hotwords_set", "hotwords": ["word1", "word2"]}
+{"event": "language_set", "language": "中文"}
+{"event": "stopped"}
+
+// Real-time results
+{
+    "sentences": [{"text": "confirmed text", "start": 1700, "end": 5500, "spk": 0}],
+    "partial": "currently speaking...",
+    "partial_start_ms": 5800,
+    "duration_ms": 7200,
+    "is_final": false
+}
+
+// Final result
+{
+    "sentences": [{"text": "...", "start": ..., "end": ..., "spk": ...}, ...],
+    "partial": "",
+    "partial_start_ms": 0,
+    "duration_ms": 10000,
+    "is_final": true
+}
+```
+
+**Interaction sequence:**
+
+```
+Client                          Server
+  │── "START" ─────────────────→│
+  │←─ {"event":"started"} ──────│
+  │── "HOTWORDS:张三,北京" ────→│
+  │←─ {"event":"hotwords_set"} ─│
+  │── "LANGUAGE:中文" ─────────→│
+  │←─ {"event":"language_set"} ─│
+  │── [audio bytes] ───────────→│
+  │←─ {sentences,partial} ──────│  (real-time push)
+  │── [audio bytes] ───────────→│
+  │←─ {sentences,partial} ──────│
+  │── "STOP" ──────────────────→│
+  │←─ {sentences,is_final:true} │  (final result)
+  │←─ {"event":"stopped"} ──────│
+```
+
+### Hotword File
+
+Default filename `hotword_list` (specified via `--hotword-file`), one word per line:
+
+```
+张三
+李四
+北京大学
+```
+
+Can also be set dynamically via WebSocket: `HOTWORDS:word1,word2,word3`
+
+---
+
+## 6. Performance Comparison
+
+### Offline Inference
+
+| Configuration | 5.6s Audio Latency | Relative Speedup |
+|--------------|-------------------|-----------------|
+| PyTorch (baseline) | 0.89s | 1x |
+| vLLM 1-GPU | 0.30s | **3x** |
+| vLLM 2-GPU TP | ~0.20s | **4.5x** |
+
+### Batch Throughput
+
+| Batch Size | 1-GPU | 2-GPU | 4-GPU |
+|-----------|-------|-------|-------|
+| 1 | ~1.5x | ~2x | ~2.5x |
+| 16 | ~4x | ~7x | ~12x |
+| 32 | ~5x | ~9x | ~15x |
+
+### WebSocket Real-time Service
+
+| Metric | Value |
+|--------|-------|
+| RTF | < 0.08 |
+| First-word latency | ~480ms |
+| Total time for 30s audio | ~2.3s |
+| Concurrency | Multiple WebSocket connections |
+
+### VAD + vLLM Pipeline
+
+| Scenario | PyTorch Sequential | vLLM Batch | Speedup |
+|----------|-------------------|------------|---------|
+| 10 segments × 5s | ~9s | ~1.5s | **6x** |
+| 20 segments × 5s | ~18s | ~2.5s | **7x** |
+
+---
+
+## 7. API Reference
+
+### AutoModelVLLM
+
+```python
+from funasr.auto.auto_model_vllm import AutoModelVLLM
+```
+
+Universal entry point that automatically detects model type and selects the corresponding vLLM implementation.
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `model` | - | Model name (hub) or local path |
+| `hub` | `"ms"` | `"ms"` / `"hf"` |
+| `device` | `"cuda:0"` | Audio encoder device |
+| `dtype` | `"bf16"` | Precision |
+| `tensor_parallel_size` | `1` | vLLM GPU parallel count |
+| `gpu_memory_utilization` | `0.8` | KV Cache memory ratio |
+| `max_model_len` | `4096` | Maximum sequence length |
+
+### AutoModelVLLM.generate()
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `inputs` | - | Audio path/list/numpy/tensor |
+| `language` | `None` | Language hint |
+| `hotwords` | `None` | Hotword list |
+| `itn` | `True` | Inverse text normalization |
+| `max_new_tokens` | `512` | Maximum generated tokens |
+| `temperature` | `0.0` | Sampling temperature |
+| `repetition_penalty` | `1.0` | Repetition penalty |
+
+**Returns**: `[{"key": str, "text": str, "timestamps": [...]}]`
+
+### FunASRNanoStreamingVLLM.from_pretrained()
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `model` | `"FunAudioLLM/Fun-ASR-Nano-2512"` | Model name or path |
+| `hub` | `"ms"` | Model source |
+| `device` | `"cuda:0"` | Device |
+| `dtype` | `"bf16"` | Precision |
+| `tensor_parallel_size` | `1` | GPU parallel count |
+| `gpu_memory_utilization` | `0.8` | Memory ratio |
+| `max_model_len` | `2048` | Sequence length |
+| `chunk_ms` | `720` | Chunk duration |
+| `rollback_chars` | `8` | Rollback character count |
+
+### streaming_generate()
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `audio_input` | - | Audio path/data |
+| `chunk_ms` | `720` | Chunk size |
+| `rollback_chars` | `8` | Rollback characters |
+| `hotwords` | `None` | Hotwords |
+| `language` | `None` | Language |
+| `max_new_tokens` | `200` | Max tokens per chunk |
+| `temperature` | `0.0` | Sampling temperature |
+
+**Yields**: `{"text", "fixed_text", "is_final", "chunk_idx", "audio_duration_ms"}`
+
+> `repetition_penalty=1.3` is hardcoded internally.
+
+### serve_realtime_ws.py Parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `--port` | `10095` | WebSocket port |
+| `--model` | `FunAudioLLM/Fun-ASR-Nano-2512` | Model |
+| `--hub` | `ms` | Source |
+| `--device` | `cuda:0` | Device |
+| `--decode-interval` | `0.48` | Partial decode interval (seconds) |
+| `--hotword-file` | `hotword_list` | Hotword file |
+| `--language` | `None` | Language |
+| `--dtype` | `bf16` | Precision |
+| `--tensor-parallel-size` | `1` | GPU parallel |
+| `--gpu-memory-utilization` | `0.8` | Memory ratio |
+| `--max-model-len` | `2048` | Sequence length |
+
+---
+
+## 8. FAQ
+
+### Q: Why is the first startup slow?
+vLLM needs to initialize KV Cache and perform CUDA Graph warmup, taking approximately 60-90 seconds. Subsequent inferences respond instantly.
+
+### Q: How to handle CUDA OOM?
+- Reduce `gpu_memory_utilization` (e.g., 0.6)
+- Increase `tensor_parallel_size` to distribute across multiple GPUs
+- Reduce `max_model_len`
+
+### Q: Can Paraformer use vLLM?
+No. Paraformer is a non-autoregressive model (CIF predictor) where all tokens are generated in parallel without KV-cache. vLLM only accelerates autoregressive LLM decoding.
+
+### Q: What's the difference between WebSocket service and streaming_generate?
+
+| | WebSocket Service | streaming_generate |
+|---|---|---|
+| Segmentation | VAD natural endpoints | Fixed 720ms chunks |
+| Inference | Decode entire VAD segment | Cumulative re-encode all audio |
+| Accuracy | Higher | Lower for first 3s |
+| Use case | Production deployment | SDK integration |
+
+### Q: Why is output empty for the first few seconds of streaming?
+Normal. The model needs ~3 seconds of accumulated audio to produce meaningful output. This is a model characteristic, not a vLLM limitation.
+
+### Q: What audio formats are supported?
+wav, mp3, flac and other mainstream formats. Sample rate is automatically converted to 16kHz.
+
+### Q: Browser cannot access microphone?
+Chrome requires HTTPS or localhost. For remote servers, use SSH port forwarding: `ssh -L 10095:localhost:10095 <server>`
+
+### Q: Will multiple concurrent connections interfere with each other?
+No. Each WebSocket connection has an independent session (VAD/ASR state isolation). vLLM handles scheduling internally.
+
+---
+
+## Appendix: DynamicStreamingVAD
+
+`funasr.models.fsmn_vad_streaming.dynamic_vad.DynamicStreamingVAD` is a generic dynamic-threshold streaming VAD wrapper that adjusts silence splitting thresholds dynamically based on the accumulated duration of the current speech segment, built on top of fsmn-vad.
+
+### Design Motivation
+
+fsmn-vad uses a fixed silence threshold (800ms) by default. In practice:
+- Short utterances (e.g., "okay") need longer silence before splitting, otherwise sentences get fragmented
+- Long segments (e.g., 30s+ meeting speeches) need faster splitting, otherwise ASR input becomes too long and quality degrades
+
+### Usage
+
+```python
+from funasr import AutoModel
+from funasr.models.fsmn_vad_streaming.dynamic_vad import DynamicStreamingVAD
+
+vad_model = AutoModel(model="fsmn-vad", device="cuda:0")
+
+# Use default threshold configuration
+vad = DynamicStreamingVAD(vad_model)
+
+# Or customize thresholds
+vad = DynamicStreamingVAD(
+    vad_model,
+    silence_schedule=[
+        (3000, 1500),       # accumulated <=3s: wait 1.5s silence
+        (10000, 800),       # accumulated 3-10s: wait 0.8s
+        (float('inf'), 300), # accumulated >10s: wait 0.3s
+    ],
+    speech_noise_thres=0.5,
+)
+```
+
+#### Streaming Call
+
+```python
+import torch
+
+for audio_chunk in audio_stream:  # real-time audio stream
+    segments = vad.feed(torch.from_numpy(audio_chunk).float())
+    for seg in segments:
+        print(f"Speech: {seg[0]}-{seg[1]}ms")
+
+# At the end
+final_segments = vad.finalize()
+```
+
+#### Non-streaming Call
+
+```python
+segments = vad.process(full_audio_tensor)
+for seg in segments:
+    print(f"Speech: {seg[0]}-{seg[1]}ms")
+```
+
+### Default Threshold Configuration
+
+| Accumulated Duration | Silence Threshold | Description |
+|---------------------|-------------------|-------------|
+| ≤ 5s | 2.0s | Preserve short sentences |
+| 5-10s | 1.5s | Normal segmentation |
+| 10-15s | 1.0s | Start tightening |
+| 15-30s | 0.8s | Faster splitting |
+| 30-45s | 0.4s | Prevent overly long segments |
+| > 45s | 0.1s | Force split |
+
+### Parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `vad_model` | - | fsmn-vad instance loaded via FunASR AutoModel |
+| `chunk_size_ms` | 60 | VAD internal processing chunk size |
+| `speech_noise_thres` | 0.5 | Speech/noise discrimination threshold |
+| `speech_to_sil_thres_ms` | 150 | Speech-to-silence base time |
+| `silence_schedule` | See table above | Dynamic threshold config `[(upper_ms, silence_ms), ...]` |
+| `sample_rate` | 16000 | Sample rate |
+
+### Properties
+
+| Property | Description |
+|----------|-------------|
+| `vad.is_speaking` | Whether currently in speech state |
+| `vad.current_duration_ms` | Current segment accumulated duration |
+| `vad.current_threshold_ms` | Current silence threshold in use |
@@ -0,0 +1,602 @@
+# FunASR vLLM 推理引擎指南
+
+FunASR 集成 vLLM 高吞吐推理引擎，用于加速 LLM-based ASR 模型的自回归解码。支持离线批量推理、SDK 流式推理、生产级 WebSocket 实时服务三种模式。
+
+---
+
+## 目录
+
+1. [概述](#1-概述)
+2. [安装与环境](#2-安装与环境)
+3. [离线批量推理](#3-离线批量推理)
+4. [流式 SDK 推理](#4-流式-sdk-推理)
+5. [实时 WebSocket 服务](#5-实时-websocket-服务)
+6. [性能对比](#6-性能对比)
+7. [API 参考](#7-api-参考)
+8. [FAQ](#8-faq)
+
+---
+
+## 1. 概述
+
+### 适用模型
+
+| 模型 | vLLM 支持 | 说明 |
+|------|-----------|------|
+| **FunASRNano** | ✓ | Qwen3-0.6B LLM，支持离线和流式 |
+| **LLMASR** | ✓ | Whisper + Qwen/Vicuna/LLaMA |
+| **GLMASR** | ✓ | GLM-ASR-Nano |
+| **QwenAudioWarp** | ✓ | Qwen-Audio |
+| Paraformer | ✗ | 非自回归模型（CIF predictor），无 LLM 解码 |
+| SenseVoice | ✗ | Whisper-like encoder-decoder，非 LLM |
+| Conformer/Transformer | ✗ | CTC/attention 解码，非 LLM |
+
+### 架构
+
+```
+┌────────────────────────────────────────────────────────┐
+│                   FunASR + vLLM                         │
+├────────────────────────────────────────────────────────┤
+│                                                        │
+│  Audio ──→ Frontend ──→ Audio Encoder ──→ Adaptor      │
+│            (fbank)      (SenseVoice)     (Transformer) │
+│                                │                       │
+│                                ▼                       │
+│                       Audio Embeddings                  │
+│                                │                       │
+│  Text Prompt ──→ Tokenize ──→ Embed ──→ [Concat]      │
+│  (hotwords/language/itn)                    │          │
+│                                             ▼          │
+│                                    ┌──────────────┐    │
+│                                    │  vLLM Engine │    │
+│                                    │  (Qwen3 etc) │    │
+│                                    │  TP 并行加速  │    │
+│                                    └──────┬───────┘    │
+│                                           ▼            │
+│                                    Generated Text      │
+│                                                        │
+│  (可选) CTC Forced Alignment ──→ 字级别时间戳          │
+└────────────────────────────────────────────────────────┘
+```
+
+### 三种使用模式
+
+| 模式 | 入口 | 适用场景 |
+|------|------|---------|
+| [离线批量推理](#3-离线批量推理) | `AutoModelVLLM` / `FunASRNanoVLLM` | 大规模转写、批量处理 |
+| [流式 SDK 推理](#4-流式-sdk-推理) | `FunASRNanoStreamingVLLM` | 实时字幕展示（SDK 集成） |
+| [WebSocket 实时服务](#5-实时-websocket-服务) | `serve_realtime_ws.py` | 生产部署、多端接入 |
+
+---
+
+## 2. 安装与环境
+
+### 依赖
+
+```bash
+pip install funasr>=1.3.0
+pip install vllm>=0.12.0
+pip install safetensors tiktoken websockets regex
+
+# FunASR 开发模式安装（需要 auto_model_vllm）
+cd /path/to/FunASR && pip install -e .
+```
+
+### 硬件要求
+
+| 配置 | 最低要求 | 推荐 |
+|------|---------|------|
+| GPU 显存 | 8GB | 16GB+ |
+| CUDA | 11.8 | 12.0+ |
+| GPU 数量 | 1 | 2+（tensor parallel） |
+
+---
+
+## 3. 离线批量推理
+
+适用于大规模音频转写、离线批量处理。vLLM 的批处理能力在此场景优势最大。
+
+### 通用接口（推荐）
+
+```python
+from funasr.auto.auto_model_vllm import AutoModelVLLM
+
+model = AutoModelVLLM(
+    model="FunAudioLLM/Fun-ASR-Nano-2512",
+    hub="ms",                    # 或 "hf"
+    tensor_parallel_size=2,      # 多卡并行
+    gpu_memory_utilization=0.8,
+)
+
+results = model.generate(
+    ["audio1.wav", "audio2.wav"],
+    language="中文",
+    hotwords=["张三", "北京"],
+)
+for r in results:
+    print(f"[{r['key']}] {r['text']}")
+```
+
+### 直接接口
+
+```python
+from funasr.models.fun_asr_nano.inference_vllm import FunASRNanoVLLM
+
+engine = FunASRNanoVLLM.from_pretrained(
+    model="FunAudioLLM/Fun-ASR-Nano-2512",
+    tensor_parallel_size=4,
+)
+
+results = engine.generate(
+    inputs="wav.scp",  # 支持 scp/jsonl/文件列表
+    hotwords=["开放时间"],
+    language="中文",
+    max_new_tokens=512,
+)
+```
+
+### 命令行
+
+```bash
+cd examples/industrial_data_pretraining/fun_asr_nano
+
+# 单文件
+python demo_vllm.py --input audio.wav --language 中文
+
+# 批量 + 多卡
+python demo_vllm.py --input wav.scp --tensor-parallel-size 4 --batch-size 32
+
+# 带热词 + 保存结果
+python demo_vllm.py --input audio.wav --hotwords 张三 北京 --output results.jsonl
+```
+
+---
+
+## 4. 流式 SDK 推理
+
+将音频按 720ms chunk 逐步处理，输出逐步稳定的识别结果。适用于 SDK 集成实时字幕场景。
+
+### 设计原理
+
+```
+音频流（720ms chunks）
+    │ 累积重编码（每个 chunk 包含从头到当前的全部音频）
+    ▼
+┌──────────────────────┐
+│ Stage 1: 前 10 chunk  │  ← 无 prev_text，批量生成
+│ 找到稳定输出          │
+└──────────┬───────────┘
+           ▼
+┌──────────────────────┐
+│ Stage 2: 后续 chunk   │  ← 用稳定输出作 prev_text
+└──────────┬───────────┘
+           ▼
+每个 chunk: [fixed 区域（确认）] + [8字 unfixed（可能变）]
+```
+
+### 用法
+
+```python
+from funasr.models.fun_asr_nano.inference_vllm_streaming import FunASRNanoStreamingVLLM
+
+engine = FunASRNanoStreamingVLLM.from_pretrained(
+    model="FunAudioLLM/Fun-ASR-Nano-2512",
+    chunk_ms=720,
+    rollback_chars=8,
+)
+
+for result in engine.streaming_generate("audio.wav", language="中文"):
+    if result["is_final"]:
+        print(f"最终: {result['text']}")
+    else:
+        print(f"[{result['audio_duration_ms']:.0f}ms] 确认: {result['fixed_text']}")
+```
+
+### 输出特性
+
+| 累积音频 | 输出质量 |
+|---------|---------|
+| < 1.5s | 空或噪声 |
+| 1.5-3.0s | 部分正确 |
+| > 3.0s | 准确输出 |
+
+> Note: `repetition_penalty=1.3` 内部硬编码，防止短 chunk 重复退化。
+
+---
+
+## 5. 实时 WebSocket 服务
+
+生产级实时语音识别服务，集成 VAD 分句 + vLLM 推理 + 说话人分离 + 热词。
+
+### 推理逻辑
+
+**核心设计：基于 VAD 端点逐段解码（非固定 chunk）**
+
+```
+音频流 ──→ StreamingVAD (60ms) ──→ 检测到端点 ──→ vLLM 解码整段
+                                │                      │
+                                │                      ▼
+                                │             locked_sentences（锁定）
+                                │                      │
+                                │                      ▼
+                                │             SPK assign（说话人分配）
+                                │
+                                └──→ 未结束 ──→ vLLM partial decode（每0.48s）
+                                                      │
+                                                      ▼
+                                               partial text（预览，会覆盖）
+```
+
+**两条推理路径：**
+
+| 路径 | 触发条件 | 输出 |
+|------|---------|------|
+| 确认段解码 | VAD 检测到静音端点 | 锁定到 sentences，永不改变 |
+| Partial 预览 | 每 0.48s + 新音频 ≥ 960ms | 临时文字，随时覆盖 |
+
+**动态 VAD 阈值：**
+
+| 累积时长 | 静音阈值 | 效果 |
+|---------|---------|------|
+| ≤ 5s | 2.0s | 短句不切碎 |
+| 5-10s | 1.5s | 正常分句 |
+| 10-15s | 1.0s | 开始收紧 |
+| 15-30s | 0.8s | 较快切分 |
+| 30-45s | 0.4s | 防止过长 |
+| > 45s | 0.1s | 强制切分 |
+
+**STOP 最终处理：**
+1. 剩余音频喂给 VAD（is_final=True）
+2. 强制结束当前在说话的段
+3. 全局 SPK 重聚类（修正说话人 ID）
+4. 返回 `is_final: true`
+
+### 部署
+
+```bash
+cd examples/industrial_data_pretraining/fun_asr_nano
+
+# 单卡
+CUDA_VISIBLE_DEVICES=0 python serve_realtime_ws.py --port 10095 --language 中文
+
+# 多卡
+CUDA_VISIBLE_DEVICES=0,1 python serve_realtime_ws.py \
+    --port 10095 --tensor-parallel-size 2 --language 中文
+
+# 完整参数
+python serve_realtime_ws.py \
+    --port 10095 \
+    --model FunAudioLLM/Fun-ASR-Nano-2512 \
+    --hub ms \
+    --device cuda:0 \
+    --decode-interval 0.48 \
+    --hotword-file 热词列表 \
+    --language 中文 \
+    --dtype bf16 \
+    --tensor-parallel-size 1 \
+    --gpu-memory-utilization 0.8 \
+    --max-model-len 2048
+```
+
+### 客户端
+
+| 客户端 | 用法 |
+|--------|------|
+| 浏览器 | 打开 `client_mic.html`（麦克风/文件/热词/说话人） |
+| Python CLI | `python client_python.py --server ws://localhost:10095 --mic` |
+| 测试脚本 | `python client_test.py --server ws://localhost:10095 --file audio.wav` |
+
+远程访问需 SSH 端口转发：`ssh -L 10095:localhost:10095 <server>`
+
+### WebSocket 协议
+
+**客户端 → 服务端：**
+
+| 消息 | 格式 | 说明 |
+|------|------|------|
+| 开始会话 | `"START"` | 初始化 session |
+| 设置热词 | `"HOTWORDS:词1,词2"` | 可选 |
+| 设置语种 | `"LANGUAGE:中文"` | 可选 |
+| 音频数据 | `bytes` | PCM16, 16kHz, mono |
+| 结束会话 | `"STOP"` | 触发最终解码 |
+
+**服务端 → 客户端：**
+
+```json
+// 事件
+{"event": "started"}
+{"event": "hotwords_set", "hotwords": ["词1", "词2"]}
+{"event": "language_set", "language": "中文"}
+{"event": "stopped"}
+
+// 实时结果
+{
+    "sentences": [{"text": "已确认", "start": 1700, "end": 5500, "spk": 0}],
+    "partial": "正在说...",
+    "partial_start_ms": 5800,
+    "duration_ms": 7200,
+    "is_final": false
+}
+
+// 最终结果
+{
+    "sentences": [{"text": "...", "start": ..., "end": ..., "spk": ...}, ...],
+    "partial": "",
+    "partial_start_ms": 0,
+    "duration_ms": 10000,
+    "is_final": true
+}
+```
+
+**交互时序：**
+
+```
+Client                          Server
+  │── "START" ─────────────────→│
+  │←─ {"event":"started"} ──────│
+  │── "HOTWORDS:张三,北京" ────→│
+  │←─ {"event":"hotwords_set"} ─│
+  │── "LANGUAGE:中文" ─────────→│
+  │←─ {"event":"language_set"} ─│
+  │── [audio bytes] ───────────→│
+  │←─ {sentences,partial} ──────│  (实时推送)
+  │── [audio bytes] ───────────→│
+  │←─ {sentences,partial} ──────│
+  │── "STOP" ──────────────────→│
+  │←─ {sentences,is_final:true} │  (最终结果)
+  │←─ {"event":"stopped"} ──────│
+```
+
+### 热词文件
+
+默认文件名 `热词列表`（`--hotword-file` 指定），一行一个词：
+
+```
+张三
+李四
+北京大学
+```
+
+也可通过 WebSocket 动态设置：`HOTWORDS:词1,词2,词3`
+
+---
+
+## 6. 性能对比
+
+### 离线推理
+
+| 配置 | 5.6s 音频延迟 | 相对加速 |
+|------|-------------|---------|
+| PyTorch (baseline) | 0.89s | 1x |
+| vLLM 1-GPU | 0.30s | **3x** |
+| vLLM 2-GPU TP | ~0.20s | **4.5x** |
+
+### 批量吞吐
+
+| Batch Size | 1-GPU | 2-GPU | 4-GPU |
+|-----------|-------|-------|-------|
+| 1 | ~1.5x | ~2x | ~2.5x |
+| 16 | ~4x | ~7x | ~12x |
+| 32 | ~5x | ~9x | ~15x |
+
+### WebSocket 实时服务
+
+| 指标 | 数值 |
+|------|------|
+| RTF | < 0.08 |
+| 首字延迟 | ~480ms |
+| 30s 音频总耗时 | ~2.3s |
+| 并发 | 多 WebSocket 连接 |
+
+### VAD + vLLM Pipeline
+
+| 场景 | PyTorch 串行 | vLLM 批量 | 加速 |
+|------|-------------|-----------|------|
+| 10 段 x 5s | ~9s | ~1.5s | **6x** |
+| 20 段 x 5s | ~18s | ~2.5s | **7x** |
+
+---
+
+## 7. API 参考
+
+### AutoModelVLLM
+
+```python
+from funasr.auto.auto_model_vllm import AutoModelVLLM
+```
+
+通用入口，自动检测模型类型并选择对应的 vLLM 实现。
+
+| 参数 | 默认值 | 说明 |
+|------|--------|------|
+| `model` | - | 模型名（hub）或本地路径 |
+| `hub` | `"ms"` | `"ms"` / `"hf"` |
+| `device` | `"cuda:0"` | 音频编码器设备 |
+| `dtype` | `"bf16"` | 精度 |
+| `tensor_parallel_size` | `1` | vLLM GPU 并行数 |
+| `gpu_memory_utilization` | `0.8` | KV Cache 显存比例 |
+| `max_model_len` | `4096` | 最大序列长度 |
+
+### AutoModelVLLM.generate()
+
+| 参数 | 默认值 | 说明 |
+|------|--------|------|
+| `inputs` | - | 音频路径/列表/numpy/tensor |
+| `language` | `None` | 语种提示 |
+| `hotwords` | `None` | 热词列表 |
+| `itn` | `True` | 逆文本正则化 |
+| `max_new_tokens` | `512` | 最大生成 token |
+| `temperature` | `0.0` | 采样温度 |
+| `repetition_penalty` | `1.0` | 重复惩罚 |
+
+**返回**: `[{"key": str, "text": str, "timestamps": [...]}]`
+
+### FunASRNanoStreamingVLLM.from_pretrained()
+
+| 参数 | 默认值 | 说明 |
+|------|--------|------|
+| `model` | `"FunAudioLLM/Fun-ASR-Nano-2512"` | 模型名或路径 |
+| `hub` | `"ms"` | 模型来源 |
+| `device` | `"cuda:0"` | 设备 |
+| `dtype` | `"bf16"` | 精度 |
+| `tensor_parallel_size` | `1` | GPU 并行数 |
+| `gpu_memory_utilization` | `0.8` | 显存比例 |
+| `max_model_len` | `2048` | 序列长度 |
+| `chunk_ms` | `720` | chunk 时长 |
+| `rollback_chars` | `8` | 回退字符数 |
+
+### streaming_generate()
+
+| 参数 | 默认值 | 说明 |
+|------|--------|------|
+| `audio_input` | - | 音频路径/数据 |
+| `chunk_ms` | `720` | chunk 大小 |
+| `rollback_chars` | `8` | 回退字符 |
+| `hotwords` | `None` | 热词 |
+| `language` | `None` | 语种 |
+| `max_new_tokens` | `200` | 每 chunk 最大 token |
+| `temperature` | `0.0` | 采样温度 |
+
+**Yields**: `{"text", "fixed_text", "is_final", "chunk_idx", "audio_duration_ms"}`
+
+> `repetition_penalty=1.3` 内部硬编码。
+
+### serve_realtime_ws.py 参数
+
+| 参数 | 默认值 | 说明 |
+|------|--------|------|
+| `--port` | `10095` | WebSocket 端口 |
+| `--model` | `FunAudioLLM/Fun-ASR-Nano-2512` | 模型 |
+| `--hub` | `ms` | 来源 |
+| `--device` | `cuda:0` | 设备 |
+| `--decode-interval` | `0.48` | partial 解码间隔（秒） |
+| `--hotword-file` | `热词列表` | 热词文件 |
+| `--language` | `None` | 语种 |
+| `--dtype` | `bf16` | 精度 |
+| `--tensor-parallel-size` | `1` | GPU 并行 |
+| `--gpu-memory-utilization` | `0.8` | 显存比例 |
+| `--max-model-len` | `2048` | 序列长度 |
+
+---
+
+## 8. FAQ
+
+### Q: 首次启动为什么慢？
+vLLM 需要初始化 KV Cache 和 CUDA Graph warmup，约 60-90 秒。后续推理即时响应。
+
+### Q: CUDA OOM 怎么办？
+- 减小 `gpu_memory_utilization`（如 0.6）
+- 增加 `tensor_parallel_size` 分摊到多卡
+- 减小 `max_model_len`
+
+### Q: Paraformer 能用 vLLM 吗？
+不能。Paraformer 是非自回归模型（CIF predictor），所有 token 并行生成，不使用 KV-cache。vLLM 只加速自回归 LLM 解码。
+
+### Q: WebSocket 服务和 streaming_generate 有什么区别？
+
+| | WebSocket 服务 | streaming_generate |
+|---|---|---|
+| 分句 | VAD 自然端点 | 固定 720ms chunk |
+| 推理 | 每个 VAD 段整体解码 | 累积重编码全部音频 |
+| 准确率 | 更高 | 前 3s 较低 |
+| 场景 | 生产部署 | SDK 集成 |
+
+### Q: 流式推理前几秒输出为空？
+正常。模型需要 ~3 秒累积音频才能产生有意义输出，这是模型特性而非 vLLM 限制。
+
+### Q: 支持哪些音频格式？
+wav、mp3、flac 等主流格式，采样率自动转为 16kHz。
+
+### Q: 浏览器无法使用麦克风？
+Chrome 要求 HTTPS 或 localhost。远程服务器用 SSH 端口转发：`ssh -L 10095:localhost:10095 <server>`
+
+### Q: 多个并发连接会互相影响吗？
+不会。每个 WebSocket 连接有独立的 session（VAD/ASR 状态隔离）。vLLM 内部会自动调度。
+
+---
+
+## 附录：DynamicStreamingVAD
+
+`funasr.models.fsmn_vad_streaming.dynamic_vad.DynamicStreamingVAD` 是通用的动态阈值流式 VAD 封装，
+在 fsmn-vad 基础上根据当前语音段的累积时长动态调整静音切分阈值。
+
+### 设计动机
+
+fsmn-vad 默认使用固定静音阈值（800ms）。实际场景中：
+- 短句（如"好的"）需要等更长的静音才切，否则会把一句话切碎
+- 长段（如会议发言 30s+）需要更快切分，否则 ASR 输入过长导致质量下降
+
+### 用法
+
+```python
+from funasr import AutoModel
+from funasr.models.fsmn_vad_streaming.dynamic_vad import DynamicStreamingVAD
+
+vad_model = AutoModel(model="fsmn-vad", device="cuda:0")
+
+# 使用默认阈值配置
+vad = DynamicStreamingVAD(vad_model)
+
+# 或自定义阈值
+vad = DynamicStreamingVAD(
+    vad_model,
+    silence_schedule=[
+        (3000, 1500),       # 累积 <=3s: 等 1.5s 静音
+        (10000, 800),       # 累积 3-10s: 等 0.8s
+        (float('inf'), 300), # 累积 >10s: 等 0.3s
+    ],
+    speech_noise_thres=0.5,
+)
+```
+
+#### 流式调用
+
+```python
+import torch
+
+for audio_chunk in audio_stream:  # 实时音频流
+    segments = vad.feed(torch.from_numpy(audio_chunk).float())
+    for seg in segments:
+        print(f"Speech: {seg[0]}-{seg[1]}ms")
+
+# 结束时
+final_segments = vad.finalize()
+```
+
+#### 非流式调用
+
+```python
+segments = vad.process(full_audio_tensor)
+for seg in segments:
+    print(f"Speech: {seg[0]}-{seg[1]}ms")
+```
+
+### 默认阈值配置
+
+| 累积时长 | 静音阈值 | 说明 |
+|---------|---------|------|
+| ≤ 5s | 2.0s | 短句不切碎 |
+| 5-10s | 1.5s | 正常分句 |
+| 10-15s | 1.0s | 开始收紧 |
+| 15-30s | 0.8s | 较快切分 |
+| 30-45s | 0.4s | 防止过长 |
+| > 45s | 0.1s | 强制切分 |
+
+### 参数
+
+| 参数 | 默认值 | 说明 |
+|------|--------|------|
+| `vad_model` | - | FunASR AutoModel 加载的 fsmn-vad 实例 |
+| `chunk_size_ms` | 60 | VAD 内部处理 chunk 大小 |
+| `speech_noise_thres` | 0.5 | 语音/噪声判别阈值 |
+| `speech_to_sil_thres_ms` | 150 | 语音转静音基础时间 |
+| `silence_schedule` | 见上表 | 动态阈值配置 `[(上限ms, 静音ms), ...]` |
+| `sample_rate` | 16000 | 采样率 |
+
+### 属性
+
+| 属性 | 说明 |
+|------|------|
+| `vad.is_speaking` | 当前是否在语音状态中 |
+| `vad.current_duration_ms` | 当前段已累积时长 |
+| `vad.current_threshold_ms` | 当前使用的静音阈值 |