diff --git a/docs/moss_transcribe_diarize.md b/docs/moss_transcribe_diarize.md index 6467de591..02ed5d460 100644 --- a/docs/moss_transcribe_diarize.md +++ b/docs/moss_transcribe_diarize.md @@ -21,7 +21,9 @@ This guide was verified against: - source: `OpenMOSS/MOSS-Transcribe-Diarize@cb765f2b0fe6f7a298aa2002e2281ae693d1f3c3` - model: `OpenMOSS-Team/MOSS-Transcribe-Diarize@e8681d68e7042738ffca8ac8212bc8fcb1131ab8` - license: Apache-2.0 in the pinned upstream source and model metadata -- vLLM nightly index: `68b4a1d582818e67adc903bf1b8fc5a5447da2fa` +- vLLM release: `v0.27.1@6e448d0ea9bf3d88d898b65449ca6dc2aec170ac` +- vLLM CUDA 12.9 x86_64 wheel SHA-256: + `bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b` Pin all three revisions. The model uses `trust_remote_code`; do not execute a floating model revision in a production service. @@ -76,36 +78,48 @@ model = AutoModel( backend="vllm", vllm_base_url="http://127.0.0.1:8898/v1", vllm_model="moss-transcribe-diarize", + vllm_response_format="diarized_json", disable_update=True, ) result = model.generate("audio.wav")[0] ``` The vLLM adapter sends the documented OpenAI-compatible multipart request and -normalizes the returned raw text. Authentication can be supplied with -`vllm_api_key`; keep it in a secret store rather than source code. +normalizes the official speaker-attributed `segments` directly into +`sentence_info`. Authentication can be supplied with `vllm_api_key`; keep it +in a secret store rather than source code. + +`vllm_response_format="diarized_json"` is the structured production path on +the pinned vLLM revision. The compatibility default remains `json` so existing +clients keep the exact tagged generation in `raw_text`; in structured mode, +`raw_text` is the cleaned `text` returned by vLLM and the authoritative speaker +metadata is in `sentence_info`. ## Choose a serving path | Path | Environment | Response contract | Use when | |---|---|---|---| -| vLLM | CUDA 12 (`cu129`) or CUDA 13 (`cu130`) | `response_format=json` returns raw `[start][Sxx]text[end]` text | You already operate vLLM or need its scheduler | +| vLLM | CUDA 12 (`cu129`) or CUDA 13 (`cu130`) | `response_format=diarized_json` returns OpenAI-compatible speaker segments; `response_format=json` keeps raw `[start][Sxx]text[end]` text | You already operate vLLM or need its scheduler | | SGLang Omni | CUDA 13 in the current upstream guide | `response_format=verbose_json` returns parsed segments | You need structured speaker segments directly from the API | | Transformers | PyTorch process | Python objects and raw tagged text | Evaluation, debugging, or custom preprocessing | The two HTTP backends share `/v1/audio/transcriptions`, but their documented -response formats are not interchangeable. Do not promise vLLM `segments` -without testing the exact vLLM revision. +response formats are not interchangeable. The pinned vLLM revision supports +`diarized_json`; test the exact server revision before promising `segments`. ## vLLM -Create an isolated environment and install the upstream-pinned nightly build: +Create an isolated environment and install the verified vLLM release: ```bash uv venv --python 3.12 .venv-moss -uv pip install --python .venv-moss/bin/python -U 'vllm[audio]' \ - --torch-backend=auto \ - --extra-index-url https://wheels.vllm.ai/68b4a1d582818e67adc903bf1b8fc5a5447da2fa/cu129 +curl -fL \ + https://github.com/vllm-project/vllm/releases/download/v0.27.1/vllm-0.27.1%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl \ + -o vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl +echo "bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl" \ + | sha256sum -c - +uv pip install --python .venv-moss/bin/python --torch-backend=auto \ + "vllm[audio] @ file://$PWD/vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl" ``` The `audio` extra is required. A plain `vllm` install can start the server but @@ -124,13 +138,13 @@ CUDA_VISIBLE_DEVICES=0 .venv-moss/bin/vllm serve \ --port 8898 ``` -Submit audio and validate the speaker-tagged text: +Submit audio and validate the official speaker-attributed response: ```bash curl -fsS http://127.0.0.1:8898/v1/audio/transcriptions \ -F file=@audio.wav \ -F model=moss-transcribe-diarize \ - -F response_format=json \ + -F response_format=diarized_json \ -F temperature=0 \ | tee moss-transcription.json @@ -140,19 +154,36 @@ import json with open("moss-transcription.json", encoding="utf-8") as stream: payload = json.load(stream) text = payload.get("text", "") +segments = payload.get("segments", []) assert text.strip(), payload -assert "[S01]" in text, text -print(text) +assert segments, payload +assert all( + isinstance(item.get("speaker"), str) + and item.get("text") + and item.get("start") <= item.get("end") + for item in segments +), payload +print(text, sorted({item["speaker"] for item in segments})) PY ``` +For an audit of the model's compact tagged generation, repeat the request with +`response_format=json` and verify `[S01]` appears in the returned `text`. + For long recordings, test `max_completion_tokens` against the longest expected meeting. A larger limit can increase memory use and tail latency. ### Reproduced vLLM contract -The FunASR ecosystem validation used vLLM -`0.23.1rc1.dev949+g68b4a1d58`, Torch `2.11.0+cu129`, and one H100 80GB: +The structured-response validation used vLLM 0.27.1, Torch +`2.13.0+cu129`, and one H100 80GB. The 15.1685-second two-speaker probe +(`43dccc068506439cb633b382b6b98185baa837363d08cc5f7152ca89b0fdc3c8`) +returned two `diarized_json` segments labelled `S01` and `S02`; the same +request through the FunASR adapter returned two monotonic `sentence_info` +entries with the same speakers. + +Earlier raw-response validation on vLLM +`0.23.1rc1.dev949+g68b4a1d58`, Torch `2.11.0+cu129`, and one H100 80GB found: - the bundled 6.000-second sample (`ea03e1f473ad1618a03da3327a545369cb8f6f06cb0f4115535e5a866167d47e`) @@ -165,6 +196,12 @@ This proves the pinned API and speaker-return contract on those inputs. It is not a diarization accuracy result, overlap test, throughput benchmark, or production capacity claim. +The older nightly commit +`68b4a1d582818e67adc903bf1b8fc5a5447da2fa` predates vLLM +[`#48543`](https://github.com/vllm-project/vllm/pull/48543). It accepts +`response_format=json` but returns HTTP 400 for `diarized_json`; upgrade to the +verified release instead of silently retrying an expensive transcription. + The FunASR adapter was also tested on both `backend="hf"` and `backend="vllm"` with model revision `e8681d68e7042738ffca8ac8212bc8fcb1131ab8`. The 15.1685-second two-speaker diff --git a/docs/moss_transcribe_diarize_zh.md b/docs/moss_transcribe_diarize_zh.md index c0409a759..324d8d68e 100644 --- a/docs/moss_transcribe_diarize_zh.md +++ b/docs/moss_transcribe_diarize_zh.md @@ -19,7 +19,9 @@ MOSS-Transcribe-Diarize 会联合生成转写、时间戳和 `[S01]` 等说话 - 源码：`OpenMOSS/MOSS-Transcribe-Diarize@cb765f2b0fe6f7a298aa2002e2281ae693d1f3c3` - 模型：`OpenMOSS-Team/MOSS-Transcribe-Diarize@e8681d68e7042738ffca8ac8212bc8fcb1131ab8` - 许可证：固定上游源码和模型元数据中的 Apache-2.0 -- vLLM nightly 索引：`68b4a1d582818e67adc903bf1b8fc5a5447da2fa` +- vLLM 发布版：`v0.27.1@6e448d0ea9bf3d88d898b65449ca6dc2aec170ac` +- vLLM CUDA 12.9 x86_64 wheel SHA-256： + `bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b` 三者都要固定。模型依赖 `trust_remote_code`，生产服务不要执行浮动的模型 revision。 @@ -68,34 +70,46 @@ model = AutoModel( backend="vllm", vllm_base_url="http://127.0.0.1:8898/v1", vllm_model="moss-transcribe-diarize", + vllm_response_format="diarized_json", disable_update=True, ) result = model.generate("audio.wav")[0] ``` -vLLM 适配器发送官方文档中的 OpenAI-compatible multipart 请求，并统一解析返回文本。 -认证可通过 `vllm_api_key` 提供；请放在 secret store，不要写入源码。 +vLLM 适配器发送官方文档中的 OpenAI-compatible multipart 请求，并把官方返回的 +speaker `segments` 直接映射到 `sentence_info`。认证可通过 `vllm_api_key` 提供； +请放在 secret store，不要写入源码。 + +固定 vLLM revision 的生产结构化路径是 +`vllm_response_format="diarized_json"`。为保持已有客户端兼容，默认值仍是 `json`， +此时 `raw_text` 保留原始标签生成；结构化模式下，`raw_text` 是 vLLM 返回的清理后 +`text`，权威说话人信息位于 `sentence_info`。 ## 选择服务后端 | 路径 | 环境 | 响应契约 | 适用场景 | |---|---|---|---| -| vLLM | CUDA 12（`cu129`）或 CUDA 13（`cu130`） | `response_format=json` 返回原始 `[start][Sxx]text[end]` 文本 | 已运行 vLLM，或需要其调度能力 | +| vLLM | CUDA 12（`cu129`）或 CUDA 13（`cu130`） | `response_format=diarized_json` 返回 OpenAI-compatible 说话人 segments；`response_format=json` 保留原始 `[start][Sxx]text[end]` 文本 | 已运行 vLLM，或需要其调度能力 | | SGLang Omni | 当前上游指南为 CUDA 13 | `response_format=verbose_json` 返回解析后的 segments | API 需要直接返回结构化说话人分段 | | Transformers | PyTorch 进程 | Python 对象和原始标签文本 | 评测、排障或自定义预处理 | 两个 HTTP 后端都使用 `/v1/audio/transcriptions`，但官方文档中的响应格式不能混为 -一谈。没有对 exact vLLM revision 做验证时，不要承诺 vLLM 返回 `segments`。 +一谈。固定的 vLLM revision 支持 `diarized_json`；承诺 `segments` 前仍要验证实际 +部署的 exact server revision。 ## vLLM -创建隔离环境并安装上游固定的 nightly build： +创建隔离环境并安装已验证的 vLLM 发布版： ```bash uv venv --python 3.12 .venv-moss -uv pip install --python .venv-moss/bin/python -U 'vllm[audio]' \ - --torch-backend=auto \ - --extra-index-url https://wheels.vllm.ai/68b4a1d582818e67adc903bf1b8fc5a5447da2fa/cu129 +curl -fL \ + https://github.com/vllm-project/vllm/releases/download/v0.27.1/vllm-0.27.1%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl \ + -o vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl +echo "bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl" \ + | sha256sum -c - +uv pip install --python .venv-moss/bin/python --torch-backend=auto \ + "vllm[audio] @ file://$PWD/vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl" ``` 必须安装 `audio` extra。只安装 plain `vllm` 时服务可以启动，但因为没有音频解码器， @@ -113,13 +127,13 @@ CUDA_VISIBLE_DEVICES=0 .venv-moss/bin/vllm serve \ --port 8898 ``` -提交音频并验证说话人标签文本： +提交音频并验证官方结构化说话人响应： ```bash curl -fsS http://127.0.0.1:8898/v1/audio/transcriptions \ -F file=@audio.wav \ -F model=moss-transcribe-diarize \ - -F response_format=json \ + -F response_format=diarized_json \ -F temperature=0 \ | tee moss-transcription.json @@ -129,18 +143,34 @@ import json with open("moss-transcription.json", encoding="utf-8") as stream: payload = json.load(stream) text = payload.get("text", "") +segments = payload.get("segments", []) assert text.strip(), payload -assert "[S01]" in text, text -print(text) +assert segments, payload +assert all( + isinstance(item.get("speaker"), str) + and item.get("text") + and item.get("start") <= item.get("end") + for item in segments +), payload +print(text, sorted({item["speaker"] for item in segments})) PY ``` +如需审计模型原始紧凑标签生成，请把请求改为 `response_format=json`，并验证返回 +`text` 中包含 `[S01]`。 + 长音频需要按业务最长会议验证 `max_completion_tokens`。增大上限也会影响显存和尾延迟。 ### 已复现的 vLLM 契约 -FunASR 生态验证使用 vLLM `0.23.1rc1.dev949+g68b4a1d58`、Torch -`2.11.0+cu129` 和一张 H100 80GB： +结构化响应验证使用 vLLM 0.27.1、Torch `2.13.0+cu129` 和一张 H100 80GB。 +15.1685 秒双说话人样例 +（`43dccc068506439cb633b382b6b98185baa837363d08cc5f7152ca89b0fdc3c8`） +返回两个 `diarized_json` segments，标签为 `S01`、`S02`；同一请求通过 FunASR +适配器后，返回两个时间单调且说话人一致的 `sentence_info`。 + +较早的 raw-response 验证使用 vLLM `0.23.1rc1.dev949+g68b4a1d58`、Torch +`2.11.0+cu129` 和一张 H100 80GB，结果如下： - 仓库内 6.000 秒样例 （`ea03e1f473ad1618a03da3327a545369cb8f6f06cb0f4115535e5a866167d47e`） @@ -152,6 +182,11 @@ FunASR 生态验证使用 vLLM `0.23.1rc1.dev949+g68b4a1d58`、Torch 这证明固定版本对这些输入满足 API 和说话人返回契约，不是 diarization 精度、重叠语音、 吞吐或生产容量结论。 +旧 nightly commit `68b4a1d582818e67adc903bf1b8fc5a5447da2fa` 早于 vLLM +[`#48543`](https://github.com/vllm-project/vllm/pull/48543)：它支持 +`response_format=json`，但 `diarized_json` 会返回 HTTP 400。应升级到已验证发布版， +不要静默重试一次成本较高的转写请求。 + FunASR 适配器还分别使用 `backend="hf"` 和 `backend="vllm"`，按模型 revision `e8681d68e7042738ffca8ac8212bc8fcb1131ab8` 做了真实测试。15.1685 秒的双说话人 样例（`43dccc068506439cb633b382b6b98185baa837363d08cc5f7152ca89b0fdc3c8`）通过 diff --git a/funasr/models/moss_transcribe_diarize/model.py b/funasr/models/moss_transcribe_diarize/model.py index fcc302be5..0f00dfb1a 100644 --- a/funasr/models/moss_transcribe_diarize/model.py +++ b/funasr/models/moss_transcribe_diarize/model.py @@ -7,6 +7,7 @@ https://github.com/OpenMOSS/MOSS-Transcribe-Diarize import copy import io +import math import mimetypes import os import re @@ -127,6 +128,66 @@ def _result_from_transcript(key, transcript): } +def _result_from_diarized_response(key, response): + """Normalize vLLM's OpenAI-compatible diarized JSON response.""" + text = response.get("text") if isinstance(response, dict) else None + segments = response.get("segments") if isinstance(response, dict) else None + if not isinstance(text, str) or not isinstance(segments, list): + raise RuntimeError( + "vLLM diarized_json response requires text and segments fields" + ) + + sentence_info = [] + timestamps = [] + for index, segment in enumerate(segments): + if not isinstance(segment, dict): + raise RuntimeError( + "vLLM diarized_json segment %d must be an object" % index + ) + start = segment.get("start") + end = segment.get("end") + segment_text = segment.get("text") + speaker = segment.get("speaker") + numeric = ( + isinstance(start, (int, float)) + and not isinstance(start, bool) + and isinstance(end, (int, float)) + and not isinstance(end, bool) + ) + if ( + not numeric + or not math.isfinite(start) + or not math.isfinite(end) + or start < 0 + or end < start + or not isinstance(segment_text, str) + or not isinstance(speaker, str) + or not speaker + ): + raise RuntimeError( + "vLLM diarized_json segment %d has an invalid contract" % index + ) + timestamp = [int(round(start * 1000)), int(round(end * 1000))] + timestamps.append(timestamp) + sentence_info.append( + { + "start": timestamp[0], + "end": timestamp[1], + "text": segment_text, + "sentence": segment_text, + "spk": speaker, + "timestamp": [timestamp], + } + ) + return { + "key": key, + "text": text, + "raw_text": text, + "timestamp": timestamps, + "sentence_info": sentence_info, + } + + @tables.register("model_classes", "MOSS-Transcribe-Diarize") @tables.register("model_classes", DEFAULT_MODEL) class MossTranscribeDiarize(nn.Module): @@ -157,6 +218,9 @@ class MossTranscribeDiarize(nn.Module): self.vllm_model = kwargs.get("vllm_model", DEFAULT_MODEL) self.vllm_api_key = kwargs.get("vllm_api_key", "EMPTY") self.vllm_timeout = float(kwargs.get("vllm_timeout", 600.0)) + self.vllm_response_format = kwargs.get("vllm_response_format", "json").lower() + if self.vllm_response_format not in {"json", "diarized_json"}: + raise ValueError("vllm_response_format must be 'json' or 'diarized_json'") self.http_session = kwargs.get("http_session") self.hf_model = None @@ -240,10 +304,18 @@ class MossTranscribeDiarize(nn.Module): results = [] for index, audio in enumerate(inputs): if self.backend == "vllm": - transcript = self._transcribe_vllm(audio, **kwargs) + response = self._transcribe_vllm(audio, **kwargs) + if self.vllm_response_format == "diarized_json": + results.append( + _result_from_diarized_response(keys[index], response) + ) + else: + results.append( + _result_from_transcript(keys[index], response["text"]) + ) else: transcript = self._transcribe_hf(audio, **kwargs) - results.append(_result_from_transcript(keys[index], transcript)) + results.append(_result_from_transcript(keys[index], transcript)) return results, { "batch_data_time": max(audio_duration, 1e-9), "forward": time.perf_counter() - started, @@ -324,7 +396,7 @@ class MossTranscribeDiarize(nn.Module): filename, payload, content_type = self._multipart_audio(audio) data = { "model": self.vllm_model, - "response_format": "json", + "response_format": self.vllm_response_format, "temperature": str(kwargs.get("temperature", 0)), } prompt = kwargs.get("prompt") @@ -345,11 +417,15 @@ class MossTranscribeDiarize(nn.Module): ) response.raise_for_status() result = response.json() - if not isinstance(result, dict) or not isinstance(result.get("text"), str): + if not isinstance(result, dict): + raise RuntimeError("vLLM transcription response must be a JSON object") + if self.vllm_response_format == "json" and not isinstance( + result.get("text"), str + ): raise RuntimeError( "vLLM transcription response did not contain a text field" ) - return result["text"] + return result @staticmethod def _multipart_audio(audio): diff --git a/tests/test_moss_transcribe_diarize_docs.py b/tests/test_moss_transcribe_diarize_docs.py index 8b2a956de..219d663c2 100644 --- a/tests/test_moss_transcribe_diarize_docs.py +++ b/tests/test_moss_transcribe_diarize_docs.py @@ -25,15 +25,19 @@ def test_moss_guides_pin_upstream_and_separate_serving_contracts(guide: Path) -> "cb765f2b0fe6f7a298aa2002e2281ae693d1f3c3", "OpenMOSS-Team/MOSS-Transcribe-Diarize", "e8681d68e7042738ffca8ac8212bc8fcb1131ab8", - "68b4a1d582818e67adc903bf1b8fc5a5447da2fa", + "6e448d0ea9bf3d88d898b65449ca6dc2aec170ac", + "bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b", "vllm[audio]", - "0.23.1rc1.dev949+g68b4a1d58", + "vLLM 0.27.1", + "68b4a1d582818e67adc903bf1b8fc5a5447da2fa", "dbb32bcfed2e8226bedf64248a9f4a44685b293a4696d18fb4cfa701b04db912", "43dccc068506439cb633b382b6b98185baa837363d08cc5f7152ca89b0fdc3c8", "S01 -> S02 -> S01", "/v1/audio/transcriptions", "response_format=json", + "response_format=diarized_json", "response_format=verbose_json", + 'vllm_response_format="diarized_json"', "[S01]", "Apache-2.0", "sentence_info", diff --git a/tests/test_moss_transcribe_diarize_model.py b/tests/test_moss_transcribe_diarize_model.py index 938a6fa93..55176dae3 100644 --- a/tests/test_moss_transcribe_diarize_model.py +++ b/tests/test_moss_transcribe_diarize_model.py @@ -138,6 +138,89 @@ class MossVllmBackendTest(unittest.TestCase): self.assertEqual(content_type, "audio/wav") response.raise_for_status.assert_called_once_with() + def test_normalizes_official_vllm_diarized_json_response(self): + session = MagicMock() + response = MagicMock() + response.json.return_value = { + "task": "transcribe", + "duration": 3.5, + "text": "hello again", + "segments": [ + { + "type": "transcript.text.segment", + "id": "seg_0", + "start": 0.25, + "end": 1.5, + "text": "hello", + "speaker": "S01", + }, + { + "type": "transcript.text.segment", + "id": "seg_1", + "start": 2.0, + "end": 3.5, + "text": "again", + "speaker": "S02", + }, + ], + } + session.post.return_value = response + + model = MossTranscribeDiarize( + backend="vllm", + vllm_base_url="http://vllm.test:8000/v1", + vllm_response_format="diarized_json", + http_session=session, + ) + results, _ = model.inference( + [np.zeros(1600, dtype=np.float32)], key=["meeting"] + ) + + self.assertEqual( + session.post.call_args.kwargs["data"]["response_format"], + "diarized_json", + ) + self.assertEqual(results[0]["text"], "hello again") + self.assertEqual(results[0]["raw_text"], "hello again") + self.assertEqual(results[0]["timestamp"], [[250, 1500], [2000, 3500]]) + self.assertEqual( + [(item["spk"], item["text"]) for item in results[0]["sentence_info"]], + [("S01", "hello"), ("S02", "again")], + ) + + def test_rejects_unsupported_vllm_response_format(self): + with self.assertRaisesRegex(ValueError, "json.*diarized_json"): + MossTranscribeDiarize( + backend="vllm", + vllm_base_url="http://vllm.test:8000/v1", + vllm_response_format="verbose_json", + ) + + def test_rejects_malformed_vllm_diarized_segment(self): + session = MagicMock() + response = MagicMock() + response.json.return_value = { + "text": "broken", + "segments": [ + { + "start": 2.0, + "end": 1.0, + "text": "broken", + "speaker": "S01", + } + ], + } + session.post.return_value = response + model = MossTranscribeDiarize( + backend="vllm", + vllm_base_url="http://vllm.test:8000/v1", + vllm_response_format="diarized_json", + http_session=session, + ) + + with self.assertRaisesRegex(RuntimeError, "segment 0.*invalid contract"): + model.inference([np.zeros(1600, dtype=np.float32)], key=["meeting"]) + if __name__ == "__main__": unittest.main() diff --git a/web-pages/product-site/data/deployments.json b/web-pages/product-site/data/deployments.json index 80bdd6afc..a83d79487 100644 --- a/web-pages/product-site/data/deployments.json +++ b/web-pages/product-site/data/deployments.json @@ -1,6 +1,6 @@ { "schema_version": 1, - "verified": "2026-08-29", + "verified": "2026-08-30", "deployments": [ { "id": "moss-transcribe-diarize", @@ -13,10 +13,10 @@ "models": ["OpenMOSS-Team/MOSS-Transcribe-Diarize (third-party Apache-2.0 model)"], "operating_systems": ["Linux"], "interfaces": ["OpenAI-compatible HTTP", "vLLM", "SGLang Omni", "Transformers"], - "tested": {"funasr": "AutoModel adapter@a3a2de0a; third-party model@e8681d68", "runtime": "Transformers 5.16.1 + vLLM 0.23.1rc1.dev949+g68b4a1d58 / Torch 2.11.0+cu129 / H100 80GB", "verified": "2026-08-29"}, + "tested": {"funasr": "AutoModel diarized_json adapter; third-party model@e8681d68", "runtime": "vLLM 0.27.1 / Torch 2.13.0+cu129 / H100 80GB", "verified": "2026-08-30"}, "commands": { "install": [ - "uv venv --python 3.12 .venv-moss && uv pip install --python .venv-moss/bin/python -U 'vllm[audio]' --torch-backend=auto --extra-index-url https://wheels.vllm.ai/68b4a1d582818e67adc903bf1b8fc5a5447da2fa/cu129", + "uv venv --python 3.12 .venv-moss && curl -fL https://github.com/vllm-project/vllm/releases/download/v0.27.1/vllm-0.27.1%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl -o vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl && echo 'bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl' | sha256sum -c - && uv pip install --python .venv-moss/bin/python --torch-backend=auto \"vllm[audio] @ file://$PWD/vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl\"", "HF_HUB_ENABLE_HF_TRANSFER=1 hf download OpenMOSS-Team/MOSS-Transcribe-Diarize --revision e8681d68e7042738ffca8ac8212bc8fcb1131ab8 --local-dir .models/moss-transcribe-diarize" ], "launch": [ @@ -27,9 +27,9 @@ "curl -fsS http://127.0.0.1:8898/v1/models" ], "smoke": [ - "curl -fsS http://127.0.0.1:8898/v1/audio/transcriptions -F file=@runtime/llama.cpp/tests/sample.wav -F model=moss-transcribe-diarize -F response_format=json -F temperature=0 | tee /tmp/moss-transcription.json", - "python - <<'PY'\nimport json\n\nwith open('/tmp/moss-transcription.json', encoding='utf-8') as stream:\n payload = json.load(stream)\ntext = payload.get('text', '')\nassert text.strip(), payload\nassert '[S01]' in text, text\nprint(text)\nPY", - "python - <<'PY'\nfrom funasr import AutoModel\n\nmodel = AutoModel(model='OpenMOSS-Team/MOSS-Transcribe-Diarize', backend='vllm', vllm_base_url='http://127.0.0.1:8898/v1', vllm_model='moss-transcribe-diarize', disable_update=True)\nresult = model.generate('runtime/llama.cpp/tests/sample.wav')[0]\nassert result['raw_text'] and result['sentence_info'], result\nprint(result['text'])\nprint(result['sentence_info'])\nPY" + "curl -fsS http://127.0.0.1:8898/v1/audio/transcriptions -F file=@runtime/llama.cpp/tests/sample.wav -F model=moss-transcribe-diarize -F response_format=diarized_json -F temperature=0 | tee /tmp/moss-transcription.json", + "python - <<'PY'\nimport json\n\nwith open('/tmp/moss-transcription.json', encoding='utf-8') as stream:\n payload = json.load(stream)\ntext = payload.get('text', '')\nsegments = payload.get('segments', [])\nassert text.strip() and segments, payload\nassert all(isinstance(item.get('speaker'), str) and item.get('text') and item.get('start') <= item.get('end') for item in segments), payload\nprint(text, sorted({item['speaker'] for item in segments}))\nPY", + "python - <<'PY'\nfrom funasr import AutoModel\n\nmodel = AutoModel(model='OpenMOSS-Team/MOSS-Transcribe-Diarize', backend='vllm', vllm_base_url='http://127.0.0.1:8898/v1', vllm_model='moss-transcribe-diarize', vllm_response_format='diarized_json', disable_update=True)\nresult = model.generate('runtime/llama.cpp/tests/sample.wav')[0]\nassert result['raw_text'] and result['sentence_info'], result\nprint(result['text'])\nprint(result['sentence_info'])\nPY" ] }, "evidence": [ @@ -37,22 +37,24 @@ {"label": "upstream source pinned at cb765f2", "url": "https://github.com/OpenMOSS/MOSS-Transcribe-Diarize/commit/cb765f2b0fe6f7a298aa2002e2281ae693d1f3c3"}, {"label": "official model snapshot pinned at e8681d68", "url": "https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize/tree/e8681d68e7042738ffca8ac8212bc8fcb1131ab8"}, {"label": "official vLLM and SGLang Omni serving guide", "url": "https://github.com/OpenMOSS/MOSS-Transcribe-Diarize/blob/cb765f2b0fe6f7a298aa2002e2281ae693d1f3c3/README.md"}, + {"label": "vLLM official diarized_json response contract", "url": "https://github.com/vllm-project/vllm/pull/48543"}, + {"label": "verified vLLM 0.27.1 release", "url": "https://github.com/vllm-project/vllm/releases/tag/v0.27.1"}, {"label": "FunASR ecosystem integration guide", "url": "https://github.com/modelscope/FunASR/blob/main/docs/moss_transcribe_diarize.md"}, {"label": "FunASR AutoModel adapter merge and exact-head CI", "url": "https://github.com/modelscope/FunASR/pull/3558"} ], "benchmarks": [ { "model": "OpenMOSS-Team/MOSS-Transcribe-Diarize@e8681d68", - "runtime": "vLLM 0.23.1rc1.dev949+g68b4a1d58 / Torch 2.11.0+cu129", + "runtime": "vLLM 0.27.1 / Torch 2.13.0+cu129", "hardware": "NVIDIA H100 80GB HBM3", - "workload": "OpenAI-compatible transcription API smoke test plus synthetic A-to-B-to-A speaker-return probe", - "audio": "Bundled 6.000 s Mandarin sample; synthetic 19.147 s file built as sample A + 0.8 s silence + sample B + 0.8 s silence + sample A", - "settings": "Pinned model and vLLM revisions; vllm[audio]; response_format=json; temperature=0; one request at a time", - "timing_scope": "Client wall time after /health became ready; local HTTP and decoding included; model download and engine startup excluded", - "result": "Both requests returned HTTP 200. The 6.000 s sample produced S01 with timestamps 0.96-5.94. The 19.147 s probe produced S01 -> S02 -> S01 with timestamps covering 0.96-19.08. Observed client times were 1.034 s and 0.171 s respectively.", - "qualification": "A functional API contract and speaker-return probe on synthetic concatenation, not a diarization accuracy study, overlap test, throughput benchmark, or production capacity promise.", + "workload": "OpenAI-compatible diarized_json API smoke plus FunASR AutoModel structured-response mapping", + "audio": "15.1685 s two-speaker probe; SHA256 43dccc068506439cb633b382b6b98185baa837363d08cc5f7152ca89b0fdc3c8", + "settings": "Pinned model revision; vLLM 0.27.1; vllm[audio]; response_format=diarized_json; temperature=0; max_completion_tokens=768", + "timing_scope": "Functional response validation after /health became ready; no latency or throughput claim", + "result": "Direct vLLM HTTP and the FunASR adapter each returned two non-empty monotonic segments with speakers S01 and S02.", + "qualification": "A structured response contract smoke, not a diarization accuracy study, overlap test, throughput benchmark, or production capacity promise.", "source": "https://github.com/modelscope/FunASR/blob/main/docs/moss_transcribe_diarize.md", - "verified": "2026-08-29" + "verified": "2026-08-30" } ], "translations": { @@ -62,11 +64,11 @@ "fit": ["会议、访谈、播客和通话的多人长音频", "希望通过同一个模型输出文本、时间戳与说话人编号", "已有 NVIDIA GPU，并需要 OpenAI 兼容的音频转写接口"], "not_fit": ["实时流式字幕或低延迟端点检测", "CPU、桌面边缘设备或 Windows 原生部署", "要求 FunASR 自有模型权重的项目"], "selection_reason": "模型端到端联合生成转写与说话人标签，用户不必再拼接外部 VAD、ASR 和 diarization 管线；FunASR AutoModel 为 Transformers 与 vLLM 提供统一的结构化结果。", - "primary_limitation": "这是 OpenMOSS 的第三方模型，不是 FunASR 模型；“无需外部 VAD”不表示模型内部没有分块或分段。当前 vLLM 路径固定 nightly commit，升级前必须重跑真实多人长音频。", + "primary_limitation": "这是 OpenMOSS 的第三方模型，不是 FunASR 模型；“无需外部 VAD”不表示模型内部没有分块或分段。当前 vLLM 路径固定 0.27.1 发布版，升级前必须重跑真实多人长音频。", "status_label": "社区验证", - "operations": ["固定模型 revision、FunASR adapter merge、vLLM nightly commit、CUDA/Torch 与 trust_remote_code 审计结果", "用真实会议验证说话人一致性、重叠语音、长静音、时间戳和最大生成长度", "FunASR AutoModel 保留 raw_text，并统一返回 text、timestamp 和带 spk 的 sentence_info；直接调用 vLLM 时 json 仍是原始 [Sxx] 标记文本"], + "operations": ["固定模型 revision、FunASR adapter merge、vLLM 0.27.1、CUDA/Torch 与 trust_remote_code 审计结果", "用真实会议验证说话人一致性、重叠语音、长静音、时间戳和最大生成长度", "FunASR AutoModel 可用 vLLM diarized_json 直接统一 text、timestamp 和带 spk 的 sentence_info；json 兼容路径仍在 raw_text 保留原始 [Sxx] 标记文本"], "security": ["把服务绑定内网，由网关完成认证、TLS、限流及音频大小/时长限制", "固定并审计 trust_remote_code 对应的模型 revision，不执行浮动 main 代码", "隔离模型缓存、上传临时目录和生成日志，按数据保留策略清理原始音频"], - "troubleshooting": ["启动前确认 CUDA 12 使用 cu129 nightly 源，CUDA 13 使用上游文档列出的 cu130 源", "长音频被截断时提高 max_completion_tokens，并同时监控显存、延迟和输出完整性", "直接调用 vLLM 时 json 没有 segments 字段是当前接口边界；使用 FunASR AutoModel 归一化结果，或改用并单独验证 SGLang Omni verbose_json"] + "troubleshooting": ["CUDA 12 使用已校验 SHA256 的 vLLM 0.27.1 cu129 wheel；其他 CUDA 栈需按上游发布资产重新验证", "长音频被截断时提高 max_completion_tokens，并同时监控显存、延迟和输出完整性", "需要结构化说话人 segments 时使用 vLLM diarized_json；旧 68b4a1d5 nightly 会返回 HTTP 400，json 只返回原始 [Sxx] 标签文本"] }, "en": { "name": "MOSS unified transcription and diarization", @@ -74,11 +76,11 @@ "fit": ["Multi-speaker meetings, interviews, podcasts, and calls", "One model output for text, timestamps, and speaker identifiers", "NVIDIA GPU deployments that need an OpenAI-compatible audio transcription endpoint"], "not_fit": ["Realtime streaming captions or low-latency endpoint detection", "CPU, desktop-edge, or native Windows deployments", "Projects that require FunASR-owned model weights"], "selection_reason": "The model jointly emits transcription and speaker labels so users do not need to assemble an external VAD, ASR, and diarization pipeline; FunASR AutoModel provides one structured result across Transformers and vLLM.", - "primary_limitation": "This is an OpenMOSS third-party model, not a FunASR model; no external VAD requirement does not imply an absence of internal segmentation. The vLLM path pins a nightly commit and must be revalidated on real long multi-speaker audio before upgrades.", + "primary_limitation": "This is an OpenMOSS third-party model, not a FunASR model; no external VAD requirement does not imply an absence of internal segmentation. The vLLM path pins release 0.27.1 and must be revalidated on real long multi-speaker audio before upgrades.", "status_label": "Community verified", - "operations": ["Pin the model revision, FunASR adapter merge, vLLM nightly commit, CUDA/Torch stack, and trust_remote_code audit", "Validate speaker consistency, overlap, long silence, timestamps, and generation limits on real meetings", "FunASR AutoModel preserves raw_text and normalizes text, timestamp, and sentence_info with spk; direct vLLM json still returns raw [Sxx]-tagged text"], + "operations": ["Pin the model revision, FunASR adapter merge, vLLM 0.27.1, CUDA/Torch stack, and trust_remote_code audit", "Validate speaker consistency, overlap, long silence, timestamps, and generation limits on real meetings", "FunASR AutoModel can map vLLM diarized_json directly into text, timestamp, and sentence_info with spk; the json compatibility path still preserves raw [Sxx]-tagged text in raw_text"], "security": ["Bind workers to a private network and put authentication, TLS, rate limits, and audio size/duration limits at the gateway", "Pin and audit the trust_remote_code model revision instead of executing a floating main revision", "Isolate model caches, upload directories, and generation logs; remove source audio according to retention policy"], - "troubleshooting": ["Use the cu129 nightly source for CUDA 12 and the upstream cu130 source for CUDA 13", "If long audio is truncated, raise max_completion_tokens while monitoring memory, latency, and output completeness", "A direct vLLM json response without segments is the current API boundary; normalize it with FunASR AutoModel or use and separately validate SGLang Omni verbose_json"] + "troubleshooting": ["For CUDA 12, use the SHA256-verified vLLM 0.27.1 cu129 wheel; revalidate another CUDA stack against its upstream release asset", "If long audio is truncated, raise max_completion_tokens while monitoring memory, latency, and output completeness", "Use vLLM diarized_json for structured speaker segments; the old 68b4a1d5 nightly returns HTTP 400, while json only returns raw [Sxx]-tagged text"] } } }, diff --git a/web-pages/product-site/tests/browser/product-site.spec.ts b/web-pages/product-site/tests/browser/product-site.spec.ts index ce9b933c3..28bf45818 100644 --- a/web-pages/product-site/tests/browser/product-site.spec.ts +++ b/web-pages/product-site/tests/browser/product-site.spec.ts @@ -186,7 +186,9 @@ for (const viewport of [ await expect(page.getByText('FunASR AutoModel', { exact: false }).first()).toBeVisible(); await expect(page.getByText('OpenMOSS', { exact: false }).first()).toBeVisible(); await expect(page.getByText('sentence_info', { exact: false }).first()).toBeVisible(); + await expect(page.getByText('diarized_json', { exact: false }).first()).toBeVisible(); await expect(page.locator('a[href="https://github.com/modelscope/FunASR/pull/3558"]')).toBeVisible(); + await expect(page.locator('a[href="https://github.com/vllm-project/vllm/pull/48543"]')).toBeVisible(); await expect(page.locator('a[href="/en/deploy/moss-transcribe-diarize.html"]')).toBeVisible(); const layout = await page.evaluate(() => ({ diff --git a/web-pages/product-site/tests/test_registry.py b/web-pages/product-site/tests/test_registry.py index b5a2297ea..9e01efdad 100644 --- a/web-pages/product-site/tests/test_registry.py +++ b/web-pages/product-site/tests/test_registry.py @@ -107,14 +107,15 @@ def test_moss_transcribe_diarize_contract_tracks_third_party_upstream(valid_regi 'OpenMOSS-Team/MOSS-Transcribe-Diarize (third-party Apache-2.0 model)' ] assert entry['tested'] == { - 'funasr': 'AutoModel adapter@a3a2de0a; third-party model@e8681d68', - 'runtime': 'Transformers 5.16.1 + vLLM 0.23.1rc1.dev949+g68b4a1d58 / Torch 2.11.0+cu129 / H100 80GB', - 'verified': '2026-08-29', + 'funasr': 'AutoModel diarized_json adapter; third-party model@e8681d68', + 'runtime': 'vLLM 0.27.1 / Torch 2.13.0+cu129 / H100 80GB', + 'verified': '2026-08-30', } install = '\n'.join(entry['commands']['install']) assert 'vllm[audio]' in install - assert 'wheels.vllm.ai/68b4a1d582818e67adc903bf1b8fc5a5447da2fa/cu129' in install + assert 'vllm-0.27.1%2Bcu129' in install + assert 'bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b' in install assert 'OpenMOSS-Team/MOSS-Transcribe-Diarize' in install assert 'e8681d68e7042738ffca8ac8212bc8fcb1131ab8' in install @@ -123,10 +124,11 @@ def test_moss_transcribe_diarize_contract_tracks_third_party_upstream(valid_regi assert '--trust-remote-code' in launch smoke = '\n'.join(entry['commands']['smoke']) assert '/v1/audio/transcriptions' in smoke - assert 'response_format=json' in smoke - assert '[S01]' in smoke + assert 'response_format=diarized_json' in smoke + assert "payload.get('segments'" in smoke assert "from funasr import AutoModel" in smoke assert "backend='vllm'" in smoke + assert "vllm_response_format='diarized_json'" in smoke assert "result['raw_text']" in smoke assert "result['sentence_info']" in smoke @@ -141,6 +143,7 @@ def test_moss_transcribe_diarize_contract_tracks_third_party_upstream(valid_regi 'e8681d68e7042738ffca8ac8212bc8fcb1131ab8' ) in evidence_urls assert 'https://github.com/modelscope/FunASR/pull/3558' in evidence_urls + assert 'https://github.com/vllm-project/vllm/pull/48543' in evidence_urls english = entry['translations']['en'] assert 'OpenMOSS' in english['summary'] @@ -151,14 +154,16 @@ def test_moss_transcribe_diarize_contract_tracks_third_party_upstream(valid_regi assert 'FunASR AutoModel API' not in ' '.join(english['not_fit']) assert 'sentence_info' in english['operations'][-1] assert 'raw_text' in english['operations'][-1] + assert 'diarized_json' in english['operations'][-1] + assert 'diarized_json' in english['troubleshooting'][-1] assert 'internal segmentation' in english['primary_limitation'] assert 'FunASR model' in english['primary_limitation'] assert any( benchmark['hardware'] == 'NVIDIA H100 80GB HBM3' - and '6.000 s' in benchmark['audio'] - and '19.147 s' in benchmark['audio'] - and 'S01 -> S02 -> S01' in benchmark['result'] - and 'functional API contract' in benchmark['qualification'] + and '15.1685 s' in benchmark['audio'] + and '43dccc068506439cb633b382b6b98185baa837363d08cc5f7152ca89b0fdc3c8' in benchmark['audio'] + and 'S01 and S02' in benchmark['result'] + and 'contract smoke' in benchmark['qualification'] for benchmark in entry['benchmarks'] ) @@ -167,7 +172,7 @@ def test_audio_cpp_contract_tracks_mainline_nano_and_sensevoice(valid_registry): entry = next(item for item in valid_registry['deployments'] if item['id'] == 'audio-cpp') llama_cpp = next(item for item in valid_registry['deployments'] if item['id'] == 'llama-cpp') - assert valid_registry['verified'] == '2026-08-29' + assert valid_registry['verified'] == '2026-08-30' assert entry['maturity'] == 'community-verified' assert entry['selector_rank'] > llama_cpp['selector_rank'] assert entry['tested'] == { @@ -333,7 +338,7 @@ def test_sensevoice_native_server_contract_tracks_merged_runtime(valid_registry) llama_cpp = next(item for item in valid_registry['deployments'] if item['id'] == 'llama-cpp') audio_cpp = next(item for item in valid_registry['deployments'] if item['id'] == 'audio-cpp') - assert valid_registry['verified'] == '2026-08-29' + assert valid_registry['verified'] == '2026-08-30' assert entry['maturity'] == 'production-verified' assert llama_cpp['selector_rank'] < entry['selector_rank'] < audio_cpp['selector_rank'] assert entry['tested'] == {