QwenLM/qwen-code
1
0 Fork 0
mirror of https://github.com/QwenLM/qwen-code.git synced 2026-11-07 06:19:43 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

feat(voice): voice dictation with native capture, streaming, and biasing (#5502)

Browse Source 
* feat(voice): voice dictation with native capture, streaming, and biasing

Add voice dictation for the prompt input:
- /voice [hold|tap|off|status] command + general.voice.{enabled,mode,language,protocol} settings; push-to-talk via Space, /model --voice to pick the model
- Native microphone capture (@qwen-code/audio-capture, miniaudio N-API) with arecord/SoX fallback, silence auto-stop, cold-start warm-up, and macOS permission query
- Batch transcription via DashScope Qwen-ASR (OpenAI-compatible chat/completions + input_audio) with language + keyterm biasing and an echo guard
- Live streaming over the DashScope realtime WebSocket (fun-asr-realtime / paraformer-realtime-v2) with interim text and an input-level waveform, behind voice.protocol=dashscope-realtime
- Rich VoiceIndicator UI (state, level meter, live partial transcript)
- Cross-platform prebuilds via prebuildify + node-gyp-build and a CI matrix

Co-authored-by: Qwen-Coder <[email protected]>

* feat(cli): route voice ASR by model

* feat(cli): polish voice realtime parity

* ci: fix voice workflow checks

* fix(cli): address voice review blockers

* fix(cli): harden voice transcription failures

* fix(cli): address voice PR review blockers

* fix(cli): harden voice review follow-ups

* fix(voice): handle realtime review blockers

* fix(cli): add voice command i18n keys

* fix(cli): address voice review blockers

* fix(cli): address voice review follow-ups

* fix(voice): harden review edge cases

* fix(voice): address release and realtime review blockers

* fix(voice): address realtime suggestion followups

* fix(voice): handle realtime review followups

* fix(voice): address realtime review blockers

* fix(voice): address review feedback

* fix(voice): address recorder review feedback

* docs(voice): document ssrf guard boundary

* fix(voice): address review feedback

* fix(voice): preserve warm recorder session safety

* fix(voice): address stream review suggestions

* fix(voice): address multi-round review findings

Realtime/streaming:
- Salvage an already-committed transcript when the WebSocket closes right
  after finish() instead of rejecting the whole dictation
  (qwenAsrRealtimeSession, voiceStreamSession) + regression tests.

useVoiceInput state machine:
- Single-shot finalize guard so a tap-stop racing the silence auto-stop can't
  double-stop the recorder and surface a spurious failure.
- Reset mountedRef on (re)mount so StrictMode (DEBUG) can't freeze the voice UI.
- Widen the hold-mode first-press release window above common key-repeat delays.

Model selection:
- Reject ids with no ASR transport at /model --voice and in the model dialog via
  a new isSelectableVoiceModel; move resolveVoiceTransport into voiceModel so the
  record-time config resolver stays transport-agnostic (+ tests).

macOS mic permission:
- Surface the not-determined state in voice warmup so the first dictation isn't
  silently lost behind the TCC dialog.

Native packaging:
- Make the audio-capture native install non-fatal (falls back to SoX/arecord) so
  a voice-only build failure can't break npm ci.
- Download audio-capture prebuilds before building standalone release archives.

Co-authored-by: Qwen-Coder <[email protected]>

* fix(voice): address review blockers

* fix(voice): update batch recording audio level

* fix(voice): tap-mode transcript loss, stream leaks, and dead keyterm cleanup

- Tap-mode dictation submitted a stale empty buffer.text, wiping the just-
  inserted transcript and sending nothing: thread the resulting prompt text
  through onSubmit(text) instead of reading buffer.text back synchronously
  after the async insert (useVoiceInput, InputPrompt).
- Streaming finalize leaked the WebSocket session when recorder.drain() threw:
  abort the session before propagating the error.
- voiceStreamSession: reject the connect promise when 'task-finished' arrives
  before 'task-started' instead of hanging forever in 'transcribing'.
- Remove dead keyterm enrichment (project/branch/recent-file paths were
  unreachable after the privacy fix) and the now-inert CJK echo guard, plus the
  tests that asserted that removed behavior; fix the misleading "OpenAI prompt
  field" comment.
- Add the missing 'Voice Model' and macOS mic-permission i18n keys to
  en/zh/zh-TW (they were falling back to English; check-i18n doesn't flag keys
  absent from en).

Co-authored-by: Qwen-Coder <[email protected]>

* fix(voice): reject partial stream transcripts

* fix(cli): let dialogs consume voice keys first

* fix(voice): reject incomplete qwen realtime transcripts

* fix(voice): handle stream review blockers

* fix(voice): salvage qwen realtime transcript on close

* fix(voice): sanitize streamed transcript text

* test(audio): run audio capture tests in CI

* fix(voice): address review blockers

* fix(voice): report stream close while recording

* fix(voice): reduce keyterm echo false positives

* fix(voice): handle realtime close diagnostics

* fix(voice): address realtime review blockers

* fix(lint): allow legacy voice filenames

* chore(cli): rename voice files to kebab-case

---------

Co-authored-by: Qwen-Coder <[email protected]>
This commit is contained in:
qqqysandQwen-Coder authored and GitHub committed 2026-06-22 14:33:36 +08:00
1 parent 4990fb093a
commit b9c5e3566b
75 files changed
+102404 -75
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/audio-capture-prebuilds.yml
+88
View File
@@ -0,0 +1,88 @@
name: 'Audio Capture Prebuilds'
# Cross-compiles the @qwen-code/audio-capture native addon (miniaudio + N-API)
# into prebuilds/<platform>-<arch>/*.node for every supported target, then
# bundles them into a single `audio-capture-prebuilds` artifact.
#
# The publish job (in release.yml) should download that artifact into
# packages/audio-capture/prebuilds/ before `npm publish`, e.g.:
#
# - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
# with:
# name: audio-capture-prebuilds
# path: packages/audio-capture/prebuilds
#
# N-API is ABI-stable, so one prebuild per platform/arch works across Node
# versions and `node-gyp-build` selects the right one at require time.
on:
workflow_call:
workflow_dispatch:
permissions:
contents: 'read'
defaults:
run:
working-directory: 'packages/audio-capture'
jobs:
build:
name: 'prebuild ${{ matrix.os }} (${{ matrix.arch }})'
runs-on: '${{ matrix.runner }}'
strategy:
fail-fast: false
matrix:
include:
- os: 'macos-14'
runner: 'macos-14'
arch: 'arm64'
- os: 'macos-13'
runner: 'macos-13'
arch: 'x64'
- os: 'ubuntu-latest'
runner: 'ubuntu-latest'
arch: 'x64'
- os: 'ubuntu-24.04-arm'
runner: 'ubuntu-24.04-arm'
arch: 'arm64'
- os: 'windows-latest'
runner: 'windows-2022'
arch: 'x64'
steps:
- uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2
- uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0
with:
node-version: '22'
# Install only this package's deps (skip the node-gyp-build install hook;
# prebuildify does its own compile below).
- name: 'Install build deps'
run: 'npm install --no-workspaces --ignore-scripts --no-audit --no-fund'
- name: 'Build prebuild'
run: 'npm run prebuildify'
- name: 'Upload prebuild'
uses: 'actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a' # v7.0.1
with:
name: 'prebuilds-${{ matrix.os }}-${{ matrix.arch }}'
path: 'packages/audio-capture/prebuilds/'
if-no-files-found: 'error'
collect:
name: 'collect prebuilds'
needs: 'build'
runs-on: 'ubuntu-latest'
steps:
- name: 'Merge per-platform prebuilds'
uses: 'actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c' # v8.0.1
with:
pattern: 'prebuilds-*'
merge-multiple: true
path: 'packages/audio-capture/prebuilds'
- name: 'List collected prebuilds'
run: 'find prebuilds -type f'
- name: 'Upload combined prebuilds'
uses: 'actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a' # v7.0.1
with:
name: 'audio-capture-prebuilds'
path: 'packages/audio-capture/prebuilds/'
if-no-files-found: 'error'
.github/workflows/ci.yml
+4 -1
View File
@@ -158,7 +158,7 @@ jobs:
# Stay running on release-sync PRs so the required Test contexts still
# report; the per-step skip_ci guards below make them no-op (pass) there.
if: '${{ !cancelled() }}'
runs-on: '${{ matrix.os }}'
runs-on: '${{ matrix.runner }}'
permissions:
contents: 'read'
checks: 'write'
@@ -168,12 +168,15 @@ jobs:
matrix:
include:
- os: 'macos-latest'
runner: 'macos-latest'
node-version: '22.x'
upload-coverage: 'false'
- os: 'ubuntu-latest'
runner: 'ubuntu-latest'
node-version: '22.x'
upload-coverage: 'true'
- os: 'windows-latest'
runner: 'windows-2022'
node-version: '22.x'
upload-coverage: 'false'
steps:
.github/workflows/release.yml
+29
View File
@@ -284,6 +284,13 @@ jobs:
run: |-
QWEN_SANDBOX=docker npx vitest run --root ./integration-tests interactive
audio_capture_prebuilds:
name: 'Audio Capture Prebuilds'
needs: 'prepare'
if: |-
${{ github.repository == 'QwenLM/qwen-code' }}
uses: './.github/workflows/audio-capture-prebuilds.yml'
publish:
name: 'Publish Release'
runs-on: 'ubuntu-latest'
@@ -292,10 +299,15 @@ jobs:
- 'quality'
- 'integration_none'
- 'integration_docker'
- 'audio_capture_prebuilds'
if: |-
${{
always() &&
needs.prepare.result == 'success' &&
(
github.repository != 'QwenLM/qwen-code' ||
needs.audio_capture_prebuilds.result == 'success'
) &&
(
github.event.inputs.force_skip_tests == 'true' ||
(
@@ -384,11 +396,28 @@ jobs:
npm run bundle
npm run prepare:package
- name: 'Download audio capture prebuilds'
if: |-
${{ github.repository == 'QwenLM/qwen-code' }}
uses: 'actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c' # v8.0.1
with:
name: 'audio-capture-prebuilds'
path: 'packages/audio-capture/prebuilds'
- name: 'Build Standalone Archives'
env:
RELEASE_VERSION: '${{ needs.prepare.outputs.release_version }}'
run: 'npm run package:standalone:release -- --version "${RELEASE_VERSION}" --out-dir dist/standalone'
- name: 'Publish @qwen-code/audio-capture'
if: |-
${{ github.repository == 'QwenLM/qwen-code' }}
working-directory: 'packages/audio-capture'
run: |-
npm publish --access public --tag=${{ needs.prepare.outputs.npm_tag }} ${{ needs.prepare.outputs.is_dry_run == 'true' && '--dry-run' || '' }}
env:
NODE_AUTH_TOKEN: '${{ secrets.NPM_TOKEN }}'
- name: 'Publish @qwen-code/qwen-code'
working-directory: 'dist'
run: |-
.gitignore
+2
View File
@@ -52,6 +52,8 @@ Thumbs.db
# Ignore built ts files
dist
packages/audio-capture/build/
packages/audio-capture/prebuilds/
# Docker folder to help skip auth refreshes
.docker
esbuild.config.js
+1
View File
@@ -63,6 +63,7 @@ const external = [
'@lydell/node-pty-linux-x64',
'@lydell/node-pty-win32-arm64',
'@lydell/node-pty-win32-x64',
'@qwen-code/audio-capture',
'@teddyzhu/clipboard',
'@teddyzhu/clipboard-darwin-arm64',
'@teddyzhu/clipboard-darwin-x64',
package-lock.json
Generated
+519 -13
View File
@@ -2702,6 +2702,36 @@
"node": ">= 8"
}
},
"node_modules/@npmcli/agent": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/@npmcli/agent/-/agent-3.0.0.tgz",
"integrity": "sha512-S79NdEgDQd/NGCay6TCoVzXSj74skRZIKJcpJjC5lOq34SZzyI6MqtiiWoiVWoVrTcGjNeC4ipbh1VIHlpfF5Q==",
"dev": true,
"license": "ISC",
"dependencies": {
"agent-base": "^7.1.0",
"http-proxy-agent": "^7.0.0",
"https-proxy-agent": "^7.0.1",
"lru-cache": "^10.0.1",
"socks-proxy-agent": "^8.0.3"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/@npmcli/fs": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@npmcli/fs/-/fs-4.0.0.tgz",
"integrity": "sha512-/xGlezI6xfGO9NwuJlnwz/K14qD1kCSAGtacBHnGzeAIuJGazcp45KP5NuyARXoKb7cwulAGWVsbeSxdG/cb0Q==",
"dev": true,
"license": "ISC",
"dependencies": {
"semver": "^7.3.5"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/@open-draft/deferred-promise": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/@open-draft/deferred-promise/-/deferred-promise-2.2.0.tgz",
@@ -3389,6 +3419,10 @@
"resolved": "packages/acp-bridge",
"link": true
},
"node_modules/@qwen-code/audio-capture": {
"resolved": "packages/audio-capture",
"link": true
},
"node_modules/@qwen-code/channel-base": {
"resolved": "packages/channels/base",
"link": true
@@ -6552,6 +6586,16 @@
"dev": true,
"license": "BSD-2-Clause"
},
"node_modules/abbrev": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/abbrev/-/abbrev-3.0.1.tgz",
"integrity": "sha512-AO2ac6pjRB3SJmGJo+v5/aK6Omggp6fsLrs6wN9bd35ulu4cCwaAU9+7ZhXjeqHVkaHThLuzH0nZr0YpCDhygg==",
"dev": true,
"license": "ISC",
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/abort-controller": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/abort-controller/-/abort-controller-3.0.0.tgz",
@@ -7454,7 +7498,6 @@
"integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"buffer": "^5.5.0",
"inherits": "^2.0.4",
@@ -7627,7 +7670,6 @@
}
],
"license": "MIT",
"optional": true,
"dependencies": {
"base64-js": "^1.3.1",
"ieee754": "^1.1.13"
@@ -7683,6 +7725,30 @@
"node": ">=8"
}
},
"node_modules/cacache": {
"version": "19.0.1",
"resolved": "https://registry.npmjs.org/cacache/-/cacache-19.0.1.tgz",
"integrity": "sha512-hdsUxulXCi5STId78vRVYEtDAjq99ICAUktLTeTYsLoTE6Z8dS0c8pWNCxwdrk9YfJeobDZc2Y186hD/5ZQgFQ==",
"dev": true,
"license": "ISC",
"dependencies": {
"@npmcli/fs": "^4.0.0",
"fs-minipass": "^3.0.0",
"glob": "^10.2.2",
"lru-cache": "^10.0.1",
"minipass": "^7.0.3",
"minipass-collect": "^2.0.1",
"minipass-flush": "^1.0.5",
"minipass-pipeline": "^1.2.4",
"p-map": "^7.0.2",
"ssri": "^12.0.0",
"tar": "^7.4.3",
"unique-filename": "^4.0.0"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/call-bind": {
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/call-bind/-/call-bind-1.0.8.tgz",
@@ -9876,6 +9942,16 @@
"node": ">= 0.8"
}
},
"node_modules/encoding": {
"version": "0.1.13",
"resolved": "https://registry.npmjs.org/encoding/-/encoding-0.1.13.tgz",
"integrity": "sha512-ETBauow1T35Y/WZMkio9jiM0Z5xjHHmJ4XmjZOq1l/dXz3lr2sRn87nJy20RupqSh1F2m3HHPSp8ShIPQJrJ3A==",
"license": "MIT",
"optional": true,
"dependencies": {
"iconv-lite": "^0.6.2"
}
},
"node_modules/encoding-sniffer": {
"version": "0.2.1",
"resolved": "https://registry.npmjs.org/encoding-sniffer/-/encoding-sniffer-0.2.1.tgz",
@@ -9911,6 +9987,16 @@
"url": "https://github.com/fb55/entities?sponsor=1"
}
},
"node_modules/env-paths": {
"version": "2.2.1",
"resolved": "https://registry.npmjs.org/env-paths/-/env-paths-2.2.1.tgz",
"integrity": "sha512-+h1lkLKhZMTYjog1VEpJNG7NZJWcuc2DDk/qsqSTRRCOXiLjeQ1d1/udrUGhqMxUgAlwKNZ0cf2uqan5GLuS2A==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6"
}
},
"node_modules/environment": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/environment/-/environment-1.1.0.tgz",
@@ -9923,6 +10009,13 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/err-code": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/err-code/-/err-code-2.0.3.tgz",
"integrity": "sha512-2bmlRpNKBxT/CRmPOlyISQpNj+qSeYvcym/uT0Jx2bMOlKLtSy1ZmLuVxSEKKyor/N5yhvp/ZiG1oE3DEYMSFA==",
"dev": true,
"license": "MIT"
},
"node_modules/error-ex": {
"version": "1.3.4",
"resolved": "https://registry.npmjs.org/error-ex/-/error-ex-1.3.4.tgz",
@@ -10975,6 +11068,13 @@
"node": ">=12.0.0"
}
},
"node_modules/exponential-backoff": {
"version": "3.1.3",
"resolved": "https://registry.npmjs.org/exponential-backoff/-/exponential-backoff-3.1.3.tgz",
"integrity": "sha512-ZgEeZXj30q+I0EN+CbSSpIyPaJ5HVQD18Z1m+u1FXbAeT94mr1zw50q4q6jiiC447Nl/YTcIYSAftiGqetwXCA==",
"dev": true,
"license": "Apache-2.0"
},
"node_modules/express": {
"version": "5.2.1",
"resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz",
@@ -11503,8 +11603,7 @@
"resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",
"integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==",
"dev": true,
"license": "MIT",
"optional": true
"license": "MIT"
},
"node_modules/fs-extra": {
"version": "11.3.1",
@@ -11531,6 +11630,19 @@
"node": ">= 10.0.0"
}
},
"node_modules/fs-minipass": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/fs-minipass/-/fs-minipass-3.0.3.tgz",
"integrity": "sha512-XUBA9XClHbnJWSfBzjkm6RvPsyg3sryZt06BEQoXcF7EK/xpGaQYJgQKDJSUH5SGZ76Y7pFx1QBnXz09rU5Fbw==",
"dev": true,
"license": "ISC",
"dependencies": {
"minipass": "^7.0.3"
},
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
}
},
"node_modules/fs.realpath": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz",
@@ -12471,6 +12583,13 @@
"entities": "^4.4.0"
}
},
"node_modules/http-cache-semantics": {
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/http-cache-semantics/-/http-cache-semantics-4.2.0.tgz",
"integrity": "sha512-dTxcvPXqPvXBQpq5dUr6mEMJX4oIEFv6bwom3FDwKRDsuIjjJGANqhBuoAn9c1RQJIdAKav33ED65E2ys+87QQ==",
"dev": true,
"license": "BSD-2-Clause"
},
"node_modules/http-errors": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz",
@@ -14815,6 +14934,29 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/make-fetch-happen": {
"version": "14.0.3",
"resolved": "https://registry.npmjs.org/make-fetch-happen/-/make-fetch-happen-14.0.3.tgz",
"integrity": "sha512-QMjGbFTP0blj97EeidG5hk/QhKQ3T4ICckQGLgz38QF7Vgbk6e6FTARN8KhKxyBbWn8R0HU+bnw8aSoFPD4qtQ==",
"dev": true,
"license": "ISC",
"dependencies": {
"@npmcli/agent": "^3.0.0",
"cacache": "^19.0.1",
"http-cache-semantics": "^4.1.1",
"minipass": "^7.0.2",
"minipass-fetch": "^4.0.0",
"minipass-flush": "^1.0.5",
"minipass-pipeline": "^1.2.4",
"negotiator": "^1.0.0",
"proc-log": "^5.0.0",
"promise-retry": "^2.0.1",
"ssri": "^12.0.0"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/markdown-it": {
"version": "14.1.0",
"resolved": "https://registry.npmjs.org/markdown-it/-/markdown-it-14.1.0.tgz",
@@ -16014,6 +16156,136 @@
"node": ">=16 || 14 >=14.17"
}
},
"node_modules/minipass-collect": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/minipass-collect/-/minipass-collect-2.0.1.tgz",
"integrity": "sha512-D7V8PO9oaz7PWGLbCACuI1qEOsq7UKfLotx/C0Aet43fCUB/wfQ7DYeq2oR/svFJGYDHPr38SHATeaj/ZoKHKw==",
"dev": true,
"license": "ISC",
"dependencies": {
"minipass": "^7.0.3"
},
"engines": {
"node": ">=16 || 14 >=14.17"
}
},
"node_modules/minipass-fetch": {
"version": "4.0.1",
"resolved": "https://registry.npmjs.org/minipass-fetch/-/minipass-fetch-4.0.1.tgz",
"integrity": "sha512-j7U11C5HXigVuutxebFadoYBbd7VSdZWggSe64NVdvWNBqGAiXPL2QVCehjmw7lY1oF9gOllYbORh+hiNgfPgQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"minipass": "^7.0.3",
"minipass-sized": "^1.0.3",
"minizlib": "^3.0.1"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
},
"optionalDependencies": {
"encoding": "^0.1.13"
}
},
"node_modules/minipass-flush": {
"version": "1.0.7",
"resolved": "https://registry.npmjs.org/minipass-flush/-/minipass-flush-1.0.7.tgz",
"integrity": "sha512-TbqTz9cUwWyHS2Dy89P3ocAGUGxKjjLuR9z8w4WUTGAVgEj17/4nhgo2Du56i0Fm3Pm30g4iA8Lcqctc76jCzA==",
"dev": true,
"license": "BlueOak-1.0.0",
"dependencies": {
"minipass": "^3.0.0"
},
"engines": {
"node": ">= 8"
}
},
"node_modules/minipass-flush/node_modules/minipass": {
"version": "3.3.6",
"resolved": "https://registry.npmjs.org/minipass/-/minipass-3.3.6.tgz",
"integrity": "sha512-DxiNidxSEK+tHG6zOIklvNOwm3hvCrbUrdtzY74U6HKTJxvIDfOUL5W5P2Ghd3DTkhhKPYGqeNUIh5qcM4YBfw==",
"dev": true,
"license": "ISC",
"dependencies": {
"yallist": "^4.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/minipass-flush/node_modules/yallist": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz",
"integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==",
"dev": true,
"license": "ISC"
},
"node_modules/minipass-pipeline": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/minipass-pipeline/-/minipass-pipeline-1.2.4.tgz",
"integrity": "sha512-xuIq7cIOt09RPRJ19gdi4b+RiNvDFYe5JH+ggNvBqGqpQXcru3PcRmOZuHBKWK1Txf9+cQ+HMVN4d6z46LZP7A==",
"dev": true,
"license": "ISC",
"dependencies": {
"minipass": "^3.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/minipass-pipeline/node_modules/minipass": {
"version": "3.3.6",
"resolved": "https://registry.npmjs.org/minipass/-/minipass-3.3.6.tgz",
"integrity": "sha512-DxiNidxSEK+tHG6zOIklvNOwm3hvCrbUrdtzY74U6HKTJxvIDfOUL5W5P2Ghd3DTkhhKPYGqeNUIh5qcM4YBfw==",
"dev": true,
"license": "ISC",
"dependencies": {
"yallist": "^4.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/minipass-pipeline/node_modules/yallist": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz",
"integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==",
"dev": true,
"license": "ISC"
},
"node_modules/minipass-sized": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/minipass-sized/-/minipass-sized-1.0.3.tgz",
"integrity": "sha512-MbkQQ2CTiBMlA2Dm/5cY+9SWFEN8pzzOXi6rlM5Xxq0Yqbda5ZQy9sU75a673FE9ZK0Zsbr6Y5iP6u9nktfg2g==",
"dev": true,
"license": "ISC",
"dependencies": {
"minipass": "^3.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/minipass-sized/node_modules/minipass": {
"version": "3.3.6",
"resolved": "https://registry.npmjs.org/minipass/-/minipass-3.3.6.tgz",
"integrity": "sha512-DxiNidxSEK+tHG6zOIklvNOwm3hvCrbUrdtzY74U6HKTJxvIDfOUL5W5P2Ghd3DTkhhKPYGqeNUIh5qcM4YBfw==",
"dev": true,
"license": "ISC",
"dependencies": {
"yallist": "^4.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/minipass-sized/node_modules/yallist": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz",
"integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==",
"dev": true,
"license": "ISC"
},
"node_modules/minizlib": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/minizlib/-/minizlib-3.1.0.tgz",
@@ -16031,8 +16303,7 @@
"resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
"integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==",
"dev": true,
"license": "MIT",
"optional": true
"license": "MIT"
},
"node_modules/mlly": {
"version": "1.8.0",
@@ -16244,7 +16515,6 @@
"integrity": "sha512-KdHvFWZjEKDf0cakgFjebl371GPsISX2oZHcuyKqM7DtogIsHrqKeLTo8wBHxaXRAQlY2PsPlZmfo+9ZCxEREQ==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"semver": "^7.3.5"
},
@@ -16322,6 +16592,68 @@
"webidl-conversions": "^3.0.0"
}
},
"node_modules/node-gyp": {
"version": "11.5.0",
"resolved": "https://registry.npmjs.org/node-gyp/-/node-gyp-11.5.0.tgz",
"integrity": "sha512-ra7Kvlhxn5V9Slyus0ygMa2h+UqExPqUIkfk7Pc8QTLT956JLSy51uWFwHtIYy0vI8cB4BDhc/S03+880My/LQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"env-paths": "^2.2.0",
"exponential-backoff": "^3.1.1",
"graceful-fs": "^4.2.6",
"make-fetch-happen": "^14.0.3",
"nopt": "^8.0.0",
"proc-log": "^5.0.0",
"semver": "^7.3.5",
"tar": "^7.4.3",
"tinyglobby": "^0.2.12",
"which": "^5.0.0"
},
"bin": {
"node-gyp": "bin/node-gyp.js"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/node-gyp-build": {
"version": "4.8.4",
"resolved": "https://registry.npmjs.org/node-gyp-build/-/node-gyp-build-4.8.4.tgz",
"integrity": "sha512-LA4ZjwlnUblHVgq0oBF3Jl/6h/Nvs5fzBLwdEF4nuxnFdsfajde4WfxtJr3CaiH+F6ewcIB/q4jQ4UzPyid+CQ==",
"license": "MIT",
"bin": {
"node-gyp-build": "bin.js",
"node-gyp-build-optional": "optional.js",
"node-gyp-build-test": "build-test.js"
}
},
"node_modules/node-gyp/node_modules/isexe": {
"version": "3.1.5",
"resolved": "https://registry.npmjs.org/isexe/-/isexe-3.1.5.tgz",
"integrity": "sha512-6B3tLtFqtQS4ekarvLVMZ+X+VlvQekbe4taUkf/rhVO3d/h0M2rfARm/pXLcPEsjjMsFgrFgSrhQIxcSVrBz8w==",
"dev": true,
"license": "BlueOak-1.0.0",
"engines": {
"node": ">=18"
}
},
"node_modules/node-gyp/node_modules/which": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/which/-/which-5.0.0.tgz",
"integrity": "sha512-JEdGzHwwkrbWoGOlIHqQ5gtprKGOenpDHpxE9zVR1bWbOtYRyPPHMe9FaP6x61CmNaTThSkb0DAJte5jD+DmzQ==",
"dev": true,
"license": "ISC",
"dependencies": {
"isexe": "^3.1.1"
},
"bin": {
"node-which": "bin/which.js"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/node-releases": {
"version": "2.0.27",
"resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.27.tgz",
@@ -16343,6 +16675,22 @@
"node": ">=20"
}
},
"node_modules/nopt": {
"version": "8.1.0",
"resolved": "https://registry.npmjs.org/nopt/-/nopt-8.1.0.tgz",
"integrity": "sha512-ieGu42u/Qsa4TFktmaKEwM6MQH0pOWnaB3htzh0JRtx84+Mebc0cbZYN5bC+6WTZ4+77xrL9Pn5m7CV6VIkV7A==",
"dev": true,
"license": "ISC",
"dependencies": {
"abbrev": "^3.0.0"
},
"bin": {
"nopt": "bin/nopt.js"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/normalize-package-data": {
"version": "7.0.1",
"resolved": "https://registry.npmjs.org/normalize-package-data/-/normalize-package-data-7.0.1.tgz",
@@ -17537,7 +17885,6 @@
"os": [
"darwin"
],
"peer": true,
"engines": {
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
}
@@ -17770,6 +18117,37 @@
"node": ">=10"
}
},
"node_modules/prebuildify": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/prebuildify/-/prebuildify-6.0.1.tgz",
"integrity": "sha512-8Y2oOOateom/s8dNBsGIcnm6AxPmLH4/nanQzL5lQMU+sC0CMhzARZHizwr36pUPLdvBnOkCNQzxg4djuFSgIw==",
"dev": true,
"license": "MIT",
"dependencies": {
"minimist": "^1.2.5",
"mkdirp-classic": "^0.5.3",
"node-abi": "^3.3.0",
"npm-run-path": "^3.1.0",
"pump": "^3.0.0",
"tar-fs": "^2.1.0"
},
"bin": {
"prebuildify": "bin.js"
}
},
"node_modules/prebuildify/node_modules/npm-run-path": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/npm-run-path/-/npm-run-path-3.1.0.tgz",
"integrity": "sha512-Dbl4A/VfiVGLgQv29URL9xshU8XDY1GeLy+fsaZ1AA8JDSfjvr5P5+pzRbWqRSBxk6/DW7MIh8lTM/PaGnP2kg==",
"dev": true,
"license": "MIT",
"dependencies": {
"path-key": "^3.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/prelude-ls": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/prelude-ls/-/prelude-ls-1.2.1.tgz",
@@ -17831,6 +18209,16 @@
"dev": true,
"license": "MIT"
},
"node_modules/proc-log": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/proc-log/-/proc-log-5.0.0.tgz",
"integrity": "sha512-Azwzvl90HaF0aCz1JrDdXQykFakSSNPaPoiZ9fm5qJIMHioDZEi7OAdRwSm6rSoPtY3Qutnm3L7ogmg3dc+wbQ==",
"dev": true,
"license": "ISC",
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/process": {
"version": "0.11.10",
"resolved": "https://registry.npmjs.org/process/-/process-0.11.10.tgz",
@@ -17848,6 +18236,30 @@
"dev": true,
"license": "MIT"
},
"node_modules/promise-retry": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/promise-retry/-/promise-retry-2.0.1.tgz",
"integrity": "sha512-y+WKFlBR8BGXnsNlIHFGPZmyDf3DFMoLhaflAnyZgV6rG6xu+JwesTo2Q9R6XwYmtmwAFCkAk3e35jEdoeh/3g==",
"dev": true,
"license": "MIT",
"dependencies": {
"err-code": "^2.0.2",
"retry": "^0.12.0"
},
"engines": {
"node": ">=10"
}
},
"node_modules/promise-retry/node_modules/retry": {
"version": "0.12.0",
"resolved": "https://registry.npmjs.org/retry/-/retry-0.12.0.tgz",
"integrity": "sha512-9LkiTwjUh6rT555DtE9rTX+BKByPfrMzEAtnlEtdEwr3Nkffwiihqe2bWADg+OQRjt9gl6ICdmB/ZFDCGAtSow==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 4"
}
},
"node_modules/prompts": {
"version": "2.4.2",
"resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz",
@@ -18459,7 +18871,6 @@
"integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"inherits": "^2.0.3",
"string_decoder": "^1.1.1",
@@ -19622,6 +20033,47 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/smart-buffer": {
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/smart-buffer/-/smart-buffer-4.2.0.tgz",
"integrity": "sha512-94hK0Hh8rPqQl2xXc3HsaBoOXKV20MToPkcXvwbISWLEs+64sBq5kFgn2kJDHb1Pry9yrP0dxrCI9RRci7RXKg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 6.0.0",
"npm": ">= 3.0.0"
}
},
"node_modules/socks": {
"version": "2.8.9",
"resolved": "https://registry.npmjs.org/socks/-/socks-2.8.9.tgz",
"integrity": "sha512-LJhUYUvItdQ0LkJTmPeaEObWXAqFyfmP85x0tch/ez9cahmhlBBLbIqDFnvBnUJGagb0JbIQrkBs1wJ+yRYpEw==",
"dev": true,
"license": "MIT",
"dependencies": {
"ip-address": "^10.1.1",
"smart-buffer": "^4.2.0"
},
"engines": {
"node": ">= 10.0.0",
"npm": ">= 3.0.0"
}
},
"node_modules/socks-proxy-agent": {
"version": "8.0.5",
"resolved": "https://registry.npmjs.org/socks-proxy-agent/-/socks-proxy-agent-8.0.5.tgz",
"integrity": "sha512-HehCEsotFqbPW9sJ8WVYB6UbmIMv7kUUORIF2Nncq4VQvBfNBLibW9YZR5dlYCSUhwcD628pRllm7n+E+YTzJw==",
"dev": true,
"license": "MIT",
"dependencies": {
"agent-base": "^7.1.2",
"debug": "^4.3.4",
"socks": "^2.8.3"
},
"engines": {
"node": ">= 14"
}
},
"node_modules/source-map": {
"version": "0.6.1",
"resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz",
@@ -19691,6 +20143,19 @@
"dev": true,
"license": "BSD-3-Clause"
},
"node_modules/ssri": {
"version": "12.0.0",
"resolved": "https://registry.npmjs.org/ssri/-/ssri-12.0.0.tgz",
"integrity": "sha512-S7iGNosepx9RadX82oimUkvr0Ct7IjJbEbs4mJcTxst8um95J3sDYU1RBEOvdu6oL1Wek2ODI5i4MAw+dZ6cAQ==",
"dev": true,
"license": "ISC",
"dependencies": {
"minipass": "^7.0.3"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/stack-utils": {
"version": "2.0.6",
"resolved": "https://registry.npmjs.org/stack-utils/-/stack-utils-2.0.6.tgz",
@@ -20497,7 +20962,6 @@
"integrity": "sha512-mDAjwmZdh7LTT6pNleZ05Yt65HC3E+NiQzl672vQG38jIrehtJk/J3mNwIg+vShQPcLF/LV7CMnDW6vjj6sfYQ==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"chownr": "^1.1.1",
"mkdirp-classic": "^0.5.2",
@@ -20510,8 +20974,7 @@
"resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",
"integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",
"dev": true,
"license": "ISC",
"optional": true
"license": "ISC"
},
"node_modules/tar-stream": {
"version": "2.2.0",
@@ -20519,7 +20982,6 @@
"integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"bl": "^4.0.3",
"end-of-stream": "^1.4.1",
@@ -21307,6 +21769,32 @@
"url": "https://opencollective.com/unified"
}
},
"node_modules/unique-filename": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/unique-filename/-/unique-filename-4.0.0.tgz",
"integrity": "sha512-XSnEewXmQ+veP7xX2dS5Q4yZAvO40cBN2MWkJ7D/6sW4Dg6wYBNwM1Vrnz1FhH5AdeLIlUXRI9e28z1YZi71NQ==",
"dev": true,
"license": "ISC",
"dependencies": {
"unique-slug": "^5.0.0"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/unique-slug": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/unique-slug/-/unique-slug-5.0.0.tgz",
"integrity": "sha512-9OdaqO5kwqR+1kVgHAhsp5vPNU0hnxRa26rBFNfNgM7M6pNtgzeBn3s/xbyCQL3dcjzOatcef6UUHpB/6MaETg==",
"dev": true,
"license": "ISC",
"dependencies": {
"imurmurhash": "^0.1.4"
},
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/unist-util-find-after": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/unist-util-find-after/-/unist-util-find-after-5.0.0.tgz",
@@ -22642,6 +23130,23 @@
"node": ">=22"
}
},
"packages/audio-capture": {
"name": "@qwen-code/audio-capture",
"version": "0.18.3",
"hasInstallScript": true,
"dependencies": {
"node-gyp-build": "^4.8.4"
},
"devDependencies": {
"node-gyp": "^11.2.0",
"prebuildify": "^6.0.1",
"typescript": "^5.3.3",
"vitest": "^3.1.1"
},
"engines": {
"node": ">=22"
}
},
"packages/channels/base": {
"name": "@qwen-code/channel-base",
"version": "0.18.5",
@@ -22732,6 +23237,7 @@
"@iarna/toml": "^2.2.5",
"@modelcontextprotocol/sdk": "^1.25.2",
"@qwen-code/acp-bridge": "file:../acp-bridge",
"@qwen-code/audio-capture": "file:../audio-capture",
"@qwen-code/channel-base": "file:../channels/base",
"@qwen-code/channel-dingtalk": "file:../channels/dingtalk",
"@qwen-code/channel-feishu": "file:../channels/feishu",
packages/audio-capture/binding.gyp
+28
View File
@@ -0,0 +1,28 @@
{
"targets": [
{
"target_name": "audio_capture",
"sources": ["native/audio_capture.cc"],
"include_dirs": ["native"],
"conditions": [
["OS=='mac'", {
"sources": ["native/mac_permission.mm"],
"libraries": [
"-framework CoreAudio",
"-framework AudioToolbox",
"-framework AudioUnit",
"-framework CoreFoundation",
"-framework AVFoundation",
"-framework Foundation"
]
}],
["OS=='linux'", {
"libraries": ["-ldl", "-lpthread", "-lm"]
}],
["OS=='win'", {
"libraries": ["winmm.lib", "ole32.lib", "uuid.lib", "ksuser.lib"]
}]
]
}
]
}
packages/audio-capture/install.js
+29
View File
@@ -0,0 +1,29 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
// Build (or locate a prebuild for) the native microphone addon. A failed or
// impossible build is intentionally NON-FATAL: voice input falls back to the
// SoX/arecord recorder, so installing the CLI must never break for this optional
// capability — not on a platform without a prebuild or a C/C++ toolchain, and
// not for the many users who never use voice. This keeps a voice-only compile
// failure from turning every `npm ci` (including unrelated CI jobs) red.
import { spawnSync } from 'node:child_process';
import process from 'node:process';
const result = spawnSync('node-gyp-build', [], {
stdio: 'inherit',
shell: true,
});
if (result.status !== 0) {
process.stderr.write(
'[audio-capture] native microphone backend unavailable; ' +
'voice input will fall back to SoX/arecord.\n',
);
}
// Always exit 0 so a missing native backend never fails the install.
process.exit(0);
packages/audio-capture/native/audio_capture.cc
+421
View File
@@ -0,0 +1,421 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
#include <node_api.h>
#include <algorithm>
#include <atomic>
#include <cmath>
#include <cstdint>
#include <cstring>
#include <mutex>
#include <stdexcept>
#include <string>
#include <vector>
#define MINIAUDIO_IMPLEMENTATION
#include "miniaudio.h"
#if defined(__APPLE__)
// Implemented in mac_permission.mm (AVFoundation). 0=notDetermined, 1=restricted,
// 2=denied, 3=authorized.
extern "C" int qwen_mic_authorization_status(void);
#endif
namespace {
// Silence auto-stop: stop after this much trailing silence below the amplitude
// threshold (fraction of int16 full-scale). Mirrors the SoX `silence` effect
// (3% / 2.0s). Only armed after speech is first detected, so leading silence
// before the user speaks does not trigger a stop.
constexpr double kSilenceThreshold = 0.03 * 32768.0;
constexpr double kSilenceDurationSecs = 2.0;
// Reserve room for the 44-byte WAV header so ToWav output stays under 10 MiB.
constexpr size_t kMaxPcmBytes = 10 * 1024 * 1024 - 44;
constexpr size_t kMaxPcmSamples = kMaxPcmBytes / sizeof(int16_t);
struct RecorderState {
ma_device device;
std::atomic<bool> deviceInitialized{false};
std::atomic<bool> recording{false};
uint32_t sampleRate = 16000;
uint32_t channels = 1;
std::vector<int16_t> pcm;
size_t pcmSize = 0;
std::mutex mutex;
std::atomic<bool> silenceDetectionEnabled{false};
std::atomic<bool> speechStarted{false};
std::atomic<uint64_t> silentFrames{0};
// Recent input level (0..1), guarded by mutex — drives the waveform.
double level = 0.0;
// Read from the JS thread.
std::atomic<bool> silenceDetected{false};
};
RecorderState gRecorder;
void StopRecorderDevice() {
if (!gRecorder.recording.load() || !gRecorder.deviceInitialized.load()) {
return;
}
ma_device_stop(&gRecorder.device);
ma_device_uninit(&gRecorder.device);
gRecorder.recording.store(false);
gRecorder.deviceInitialized.store(false);
}
void CleanupRecorder(void*) {
StopRecorderDevice();
}
bool GetBoolProperty(
napi_env env,
napi_value object,
const char* key,
bool fallback) {
bool hasProperty = false;
napi_has_named_property(env, object, key, &hasProperty);
if (!hasProperty) return fallback;
napi_value value;
napi_get_named_property(env, object, key, &value);
napi_valuetype type;
napi_typeof(env, value, &type);
if (type != napi_boolean) return fallback;
bool result = fallback;
napi_get_value_bool(env, value, &result);
return result;
}
void Throw(napi_env env, const std::string& message) {
napi_throw_error(env, nullptr, message.c_str());
}
uint32_t GetUint32Property(
napi_env env,
napi_value object,
const char* key,
uint32_t fallback) {
bool hasProperty = false;
napi_has_named_property(env, object, key, &hasProperty);
if (!hasProperty) return fallback;
napi_value value;
napi_get_named_property(env, object, key, &value);
napi_valuetype type;
napi_typeof(env, value, &type);
if (type != napi_number) return fallback;
uint32_t result = fallback;
napi_get_value_uint32(env, value, &result);
return result;
}
void WriteU16(std::vector<uint8_t>& out, uint16_t value) {
out.push_back(static_cast<uint8_t>(value & 0xff));
out.push_back(static_cast<uint8_t>((value >> 8) & 0xff));
}
void WriteU32(std::vector<uint8_t>& out, uint32_t value) {
out.push_back(static_cast<uint8_t>(value & 0xff));
out.push_back(static_cast<uint8_t>((value >> 8) & 0xff));
out.push_back(static_cast<uint8_t>((value >> 16) & 0xff));
out.push_back(static_cast<uint8_t>((value >> 24) & 0xff));
}
void WriteAscii(std::vector<uint8_t>& out, const char* value) {
out.insert(out.end(), value, value + std::strlen(value));
}
std::vector<uint8_t> ToWav(
const std::vector<int16_t>& pcm,
uint32_t sampleRate,
uint32_t channels) {
const uint16_t bitsPerSample = 16;
const uint16_t blockAlign = static_cast<uint16_t>(channels * 2);
const uint32_t byteRate = sampleRate * blockAlign;
const uint32_t dataBytes = static_cast<uint32_t>(pcm.size() * sizeof(int16_t));
std::vector<uint8_t> wav;
wav.reserve(44 + dataBytes);
WriteAscii(wav, "RIFF");
WriteU32(wav, 36 + dataBytes);
WriteAscii(wav, "WAVE");
WriteAscii(wav, "fmt ");
WriteU32(wav, 16);
WriteU16(wav, 1);
WriteU16(wav, static_cast<uint16_t>(channels));
WriteU32(wav, sampleRate);
WriteU32(wav, byteRate);
WriteU16(wav, blockAlign);
WriteU16(wav, bitsPerSample);
WriteAscii(wav, "data");
WriteU32(wav, dataBytes);
const uint8_t* bytes = reinterpret_cast<const uint8_t*>(pcm.data());
wav.insert(wav.end(), bytes, bytes + dataBytes);
return wav;
}
void DataCallback(
ma_device* device,
void* output,
const void* input,
ma_uint32 frameCount) {
(void)output;
if (!input) return;
auto* state = static_cast<RecorderState*>(device->pUserData);
const auto* samples = static_cast<const int16_t*>(input);
const size_t sampleCount = static_cast<size_t>(frameCount) * state->channels;
double sumAbs = 0.0;
for (size_t i = 0; i < sampleCount; ++i) {
sumAbs += std::abs(static_cast<double>(samples[i]));
}
const double meanAbs = sampleCount > 0 ? sumAbs / sampleCount : 0.0;
if (
state->silenceDetectionEnabled.load() &&
!state->silenceDetected.load()) {
if (meanAbs >= kSilenceThreshold) {
state->speechStarted.store(true);
state->silentFrames.store(0);
} else if (state->speechStarted.load()) {
const uint64_t silentFrames =
state->silentFrames.load() + frameCount;
state->silentFrames.store(silentFrames);
const uint64_t needed =
static_cast<uint64_t>(state->sampleRate * kSilenceDurationSecs);
if (silentFrames >= needed) {
state->silenceDetected.store(true);
}
}
}
std::lock_guard<std::mutex> lock(state->mutex);
state->level = meanAbs / 32768.0;
const size_t remaining = state->pcmSize < kMaxPcmSamples
? kMaxPcmSamples - state->pcmSize
: 0;
const size_t toCopy = std::min(sampleCount, remaining);
if (toCopy > 0) {
std::copy(samples, samples + toCopy, state->pcm.begin() + state->pcmSize);
state->pcmSize += toCopy;
}
if (toCopy < sampleCount) {
state->silenceDetected.store(true);
}
}
napi_value StartRecording(napi_env env, napi_callback_info info) {
size_t argc = 1;
napi_value args[1];
napi_get_cb_info(env, info, &argc, args, nullptr, nullptr);
if (gRecorder.recording.load()) {
Throw(env, "Native audio capture is already recording.");
return nullptr;
}
uint32_t sampleRate = 16000;
uint32_t channels = 1;
bool silenceDetection = false;
if (argc > 0) {
napi_valuetype type;
napi_typeof(env, args[0], &type);
if (type == napi_object) {
sampleRate = GetUint32Property(env, args[0], "sampleRate", sampleRate);
channels = GetUint32Property(env, args[0], "channels", channels);
silenceDetection =
GetBoolProperty(env, args[0], "silenceDetection", silenceDetection);
}
}
if (sampleRate == 0 || sampleRate > 192000 || channels == 0 ||
channels > 2) {
Throw(env, "Native audio capture requires 1 or 2 channels and a sample rate.");
return nullptr;
}
ma_device_config config = ma_device_config_init(ma_device_type_capture);
config.capture.format = ma_format_s16;
config.capture.channels = channels;
config.sampleRate = sampleRate;
config.dataCallback = DataCallback;
config.pUserData = &gRecorder;
{
std::lock_guard<std::mutex> lock(gRecorder.mutex);
gRecorder.sampleRate = sampleRate;
gRecorder.channels = channels;
gRecorder.silenceDetectionEnabled.store(silenceDetection);
gRecorder.speechStarted.store(false);
gRecorder.silentFrames.store(0);
gRecorder.level = 0.0;
gRecorder.silenceDetected.store(false);
if (gRecorder.pcm.size() < kMaxPcmSamples) {
gRecorder.pcm.resize(kMaxPcmSamples);
}
gRecorder.pcmSize = 0;
}
ma_result result = ma_device_init(nullptr, &config, &gRecorder.device);
if (result != MA_SUCCESS) {
Throw(env, "Native audio capture failed to initialize the input device.");
return nullptr;
}
gRecorder.deviceInitialized.store(true);
result = ma_device_start(&gRecorder.device);
if (result != MA_SUCCESS) {
ma_device_uninit(&gRecorder.device);
gRecorder.deviceInitialized.store(false);
Throw(env, "Native audio capture failed to start the input device.");
return nullptr;
}
gRecorder.recording.store(true);
napi_value undefined;
napi_get_undefined(env, &undefined);
return undefined;
}
napi_value StopRecording(napi_env env, napi_callback_info info) {
(void)info;
if (!gRecorder.recording.load() || !gRecorder.deviceInitialized.load()) {
Throw(env, "Native audio capture is not recording.");
return nullptr;
}
StopRecorderDevice();
std::vector<int16_t> pcm;
uint32_t sampleRate = 16000;
uint32_t channels = 1;
{
std::lock_guard<std::mutex> lock(gRecorder.mutex);
pcm.assign(gRecorder.pcm.begin(), gRecorder.pcm.begin() + gRecorder.pcmSize);
gRecorder.pcmSize = 0;
sampleRate = gRecorder.sampleRate;
channels = gRecorder.channels;
}
if (pcm.empty()) {
Throw(env, "Native audio capture produced empty audio.");
return nullptr;
}
std::vector<uint8_t> wav = ToWav(pcm, sampleRate, channels);
napi_value buffer;
napi_create_buffer_copy(env, wav.size(), wav.data(), nullptr, &buffer);
return buffer;
}
napi_value IsRecording(napi_env env, napi_callback_info info) {
(void)info;
napi_value value;
napi_get_boolean(env, gRecorder.recording.load(), &value);
return value;
}
napi_value SilenceDetected(napi_env env, napi_callback_info info) {
(void)info;
napi_value value;
napi_get_boolean(env, gRecorder.silenceDetected.load(), &value);
return value;
}
// Return (and clear) PCM captured since the last call — for streaming uploads.
napi_value DrainAudio(napi_env env, napi_callback_info info) {
(void)info;
std::vector<int16_t> pcm;
{
std::lock_guard<std::mutex> lock(gRecorder.mutex);
pcm.assign(gRecorder.pcm.begin(), gRecorder.pcm.begin() + gRecorder.pcmSize);
gRecorder.pcmSize = 0;
}
napi_value buffer;
const size_t bytes = pcm.size() * sizeof(int16_t);
napi_create_buffer_copy(
env,
bytes,
bytes == 0 ? nullptr : reinterpret_cast<const uint8_t*>(pcm.data()),
nullptr,
&buffer);
return buffer;
}
// Recent input level (0..1) for the waveform display.
napi_value AudioLevel(napi_env env, napi_callback_info info) {
(void)info;
double level;
{
std::lock_guard<std::mutex> lock(gRecorder.mutex);
level = gRecorder.level;
}
napi_value value;
napi_create_double(env, level, &value);
return value;
}
napi_value MicrophoneAuthorizationStatus(
napi_env env,
napi_callback_info info) {
(void)info;
const char* status = "unknown";
#if defined(__APPLE__)
switch (qwen_mic_authorization_status()) {
case 3:
status = "granted";
break;
case 2: // denied
case 1: // restricted (e.g. MDM) — unusable, treat as denied
status = "denied";
break;
case 0:
status = "prompt"; // not yet determined; capture will trigger the dialog
break;
default:
status = "unknown";
break;
}
#endif
napi_value value;
napi_create_string_utf8(env, status, NAPI_AUTO_LENGTH, &value);
return value;
}
napi_value Init(napi_env env, napi_value exports) {
napi_add_env_cleanup_hook(env, CleanupRecorder, nullptr);
napi_property_descriptor descriptors[] = {
{"startRecording", nullptr, StartRecording, nullptr, nullptr, nullptr,
napi_default, nullptr},
{"stopRecording", nullptr, StopRecording, nullptr, nullptr, nullptr,
napi_default, nullptr},
{"isRecording", nullptr, IsRecording, nullptr, nullptr, nullptr,
napi_default, nullptr},
{"silenceDetected", nullptr, SilenceDetected, nullptr, nullptr, nullptr,
napi_default, nullptr},
{"drainAudio", nullptr, DrainAudio, nullptr, nullptr, nullptr,
napi_default, nullptr},
{"audioLevel", nullptr, AudioLevel, nullptr, nullptr, nullptr,
napi_default, nullptr},
{"microphoneAuthorizationStatus", nullptr, MicrophoneAuthorizationStatus,
nullptr, nullptr, nullptr, napi_default, nullptr},
};
napi_define_properties(
env,
exports,
sizeof(descriptors) / sizeof(descriptors[0]),
descriptors);
return exports;
}
} // namespace
NAPI_MODULE(NODE_GYP_MODULE_NAME, Init)
packages/audio-capture/native/mac_permission.mm
+31
View File
@@ -0,0 +1,31 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
// macOS microphone authorization query via AVFoundation. Compiled only on mac
// (see binding.gyp). Returns the AVAuthorizationStatus as a small int so the
// C++ addon can map it without linking ObjC itself:
// 0 = notDetermined, 1 = restricted, 2 = denied, 3 = authorized
#import <AVFoundation/AVFoundation.h>
extern "C" int qwen_mic_authorization_status(void) {
if (@available(macOS 10.14, *)) {
AVAuthorizationStatus status =
[AVCaptureDevice authorizationStatusForMediaType:AVMediaTypeAudio];
switch (status) {
case AVAuthorizationStatusAuthorized:
return 3;
case AVAuthorizationStatusDenied:
return 2;
case AVAuthorizationStatusRestricted:
return 1;
case AVAuthorizationStatusNotDetermined:
default:
return 0;
}
}
// Pre-10.14 had no mic gate — treat as granted.
return 3;
}
packages/audio-capture/native/miniaudio.h
+93468
View File
File diff suppressed because it is too large. Load diff
packages/audio-capture/package.json
+45
View File
@@ -0,0 +1,45 @@
{
"name": "@qwen-code/audio-capture",
"version": "0.18.3",
"description": "Native microphone capture backend for Qwen Code voice input",
"type": "module",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
}
},
"files": [
"dist",
"!dist/**/*.test.*",
"prebuilds",
"native/audio_capture.cc",
"native/mac_permission.mm",
"native/miniaudio.h",
"binding.gyp",
"install.js"
],
"scripts": {
"install": "node install.js",
"prebuildify": "prebuildify --napi --strip",
"build": "node-gyp rebuild && tsc --build",
"build:ts": "tsc --build",
"test": "vitest run",
"test:ci": "vitest run",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"node-gyp-build": "^4.8.4"
},
"devDependencies": {
"node-gyp": "^11.2.0",
"prebuildify": "^6.0.1",
"typescript": "^5.3.3",
"vitest": "^3.1.1"
},
"engines": {
"node": ">=22"
}
}
packages/audio-capture/src/index.ts
+94
View File
@@ -0,0 +1,94 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { createRequire } from 'node:module';
import { dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { getPlatformBackendName } from './platform.js';
export type MicrophoneAuthorizationStatus =
| 'granted'
| 'denied'
| 'prompt'
| 'unknown';
export interface AudioCaptureOptions {
sampleRate: number;
channels: number;
/** Flag sustained silence so the caller can auto-stop (tap mode). */
silenceDetection?: boolean;
}
export interface NativeAudioCaptureBackend {
startRecording: (options: AudioCaptureOptions) => void;
stopRecording: () => Uint8Array;
isRecording: () => boolean;
/** True once sustained silence was detected. Absent on older addons. */
silenceDetected?: () => boolean;
/** Return & clear PCM captured since the last call (for streaming uploads). */
drainAudio?: () => Uint8Array;
/** Recent input level 0..1 (for waveform display). */
audioLevel?: () => number;
microphoneAuthorizationStatus: () => MicrophoneAuthorizationStatus;
}
interface NativeBinding {
startRecording: (options?: Partial<AudioCaptureOptions>) => void;
stopRecording: () => Uint8Array;
isRecording: () => boolean;
silenceDetected?: () => boolean;
drainAudio?: () => Uint8Array;
audioLevel?: () => number;
microphoneAuthorizationStatus?: () => MicrophoneAuthorizationStatus;
}
const nativeRequire = createRequire(import.meta.url);
// dist/index.js → package root, which holds prebuilds/ and build/.
const packageRoot = dirname(dirname(fileURLToPath(import.meta.url)));
function loadBinding(): NativeBinding {
try {
// Throws on unsupported platforms before touching the native layer.
getPlatformBackendName();
// node-gyp-build picks the matching prebuild from prebuilds/<platform>-<arch>,
// falling back to a local build/Release compile — no compiler needed when a
// prebuilt binary ships for the host.
const loadPrebuild = nativeRequire('node-gyp-build') as (
dir: string,
) => NativeBinding;
return loadPrebuild(packageRoot);
} catch (error) {
throw new Error(
'Native audio capture addon could not be loaded. Reinstall ' +
'@qwen-code/audio-capture, or run "npm run build" in packages/audio-capture. ' +
`(${error instanceof Error ? error.message : String(error)})`,
);
}
}
export function createNativeAudioCaptureBackend(
binding: NativeBinding = loadBinding(),
): NativeAudioCaptureBackend {
const silenceDetected = binding.silenceDetected;
const drainAudio = binding.drainAudio;
const audioLevel = binding.audioLevel;
return {
startRecording: (options) => {
binding.startRecording(options);
},
stopRecording: () => binding.stopRecording(),
isRecording: () => binding.isRecording(),
...(drainAudio ? { drainAudio: () => drainAudio.call(binding) } : {}),
...(audioLevel ? { audioLevel: () => audioLevel.call(binding) } : {}),
...(silenceDetected
? { silenceDetected: () => silenceDetected.call(binding) }
: {}),
microphoneAuthorizationStatus: () =>
binding.microphoneAuthorizationStatus?.() ?? 'unknown',
};
}
export { getPlatformBackendName } from './platform.js';
packages/audio-capture/src/platform.test.ts
+22
View File
@@ -0,0 +1,22 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, expect, it } from 'vitest';
import { getPlatformBackendName } from './platform.js';
describe('getPlatformBackendName', () => {
it('maps macOS, Linux, and Windows to native audio backends', () => {
expect(getPlatformBackendName('darwin')).toBe('coreaudio');
expect(getPlatformBackendName('linux')).toBe('alsa-pulse');
expect(getPlatformBackendName('win32')).toBe('wasapi');
});
it('rejects unsupported platforms with a useful error', () => {
expect(() => getPlatformBackendName('freebsd')).toThrow(
'Native audio capture is not available for freebsd.',
);
});
});
packages/audio-capture/src/platform.ts
+22
View File
@@ -0,0 +1,22 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
export type NativeAudioBackendName = 'coreaudio' | 'alsa-pulse' | 'wasapi';
export function getPlatformBackendName(
platform: NodeJS.Platform = process.platform,
): NativeAudioBackendName {
switch (platform) {
case 'darwin':
return 'coreaudio';
case 'linux':
return 'alsa-pulse';
case 'win32':
return 'wasapi';
default:
throw new Error(`Native audio capture is not available for ${platform}.`);
}
}
packages/audio-capture/tsconfig.json
+10
View File
@@ -0,0 +1,10 @@
{
"extends": "../../tsconfig.json",
"compilerOptions": {
"outDir": "dist",
"rootDir": "src",
"types": ["node", "vitest/globals"]
},
"include": ["src/**/*.ts"],
"exclude": ["node_modules", "dist"]
}
packages/audio-capture/vitest.config.ts
+14
View File
@@ -0,0 +1,14 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
environment: 'node',
include: ['src/**/*.test.ts'],
},
});
packages/cli/package.json
+1
View File
@@ -45,6 +45,7 @@
"@iarna/toml": "^2.2.5",
"@modelcontextprotocol/sdk": "^1.25.2",
"@qwen-code/acp-bridge": "file:../acp-bridge",
"@qwen-code/audio-capture": "file:../audio-capture",
"@qwen-code/channel-base": "file:../channels/base",
"@qwen-code/channel-dingtalk": "file:../channels/dingtalk",
"@qwen-code/channel-qqbot": "file:../channels/qqbot",
packages/cli/src/config/keyBindings.test.ts
+6
View File
@@ -55,5 +55,11 @@ describe('keyBindings config', () => {
const config: KeyBindingConfig = defaultKeyBindings;
expect(config[Command.HOME]).toBeDefined();
});
it('should bind voice push-to-talk to bare Space by default', () => {
expect(defaultKeyBindings[Command.VOICE_PUSH_TO_TALK]).toEqual([
{ key: 'space', ctrl: false, meta: false },
]);
});
});
});
packages/cli/src/config/keyBindings.ts
+2
View File
@@ -43,6 +43,7 @@ export enum Command {
// Text input
SUBMIT = 'submit',
NEWLINE = 'newline',
VOICE_PUSH_TO_TALK = 'voicePushToTalk',
// External tools
OPEN_EXTERNAL_EDITOR = 'openExternalEditor',
@@ -196,6 +197,7 @@ export const defaultKeyBindings: KeyBindingConfig = {
{ key: 'return', shift: true },
{ key: 'j', ctrl: true },
],
[Command.VOICE_PUSH_TO_TALK]: [{ key: 'space', ctrl: false, meta: false }],
// External tools
[Command.OPEN_EXTERNAL_EDITOR]: [
packages/cli/src/config/settingsSchema.test.ts
+29
View File
@@ -29,6 +29,7 @@ describe('SettingsSchema', () => {
'security',
'advanced',
'plansDirectory',
'voiceModel',
];
expectedSettings.forEach((setting) => {
@@ -146,6 +147,34 @@ describe('SettingsSchema', () => {
expect(getSettingsSchema().plansDirectory.showInDialog).toBe(false);
});
it('should have voice model setting in schema', () => {
const voiceModel = getSettingsSchema().voiceModel;
expect(voiceModel).toBeDefined();
expect(voiceModel.type).toBe('string');
expect(voiceModel.category).toBe('Model');
expect(voiceModel.default).toBe('');
expect(voiceModel.requiresRestart).toBe(false);
expect(voiceModel.showInDialog).toBe(false);
});
it('should have voice dictation settings under general', () => {
const voice =
getSettingsSchema().general.properties.voice.properties ?? {};
expect(voice.enabled.type).toBe('boolean');
expect(voice.enabled.default).toBe(false);
expect(voice.mode.type).toBe('enum');
expect(voice.mode.default).toBe('hold');
expect(
voice.mode.options?.map((o: { value: string }) => o.value),
).toEqual(['hold', 'tap']);
expect(voice.language.type).toBe('string');
expect(voice.language.default).toBe('');
});
it('should have unique categories', () => {
const categories = new Set();
packages/cli/src/config/settingsSchema.ts
+55
View File
@@ -365,6 +365,50 @@ const SETTINGS_SCHEMA = {
description: 'Enable Vim keybindings',
showInDialog: true,
},
voice: {
type: 'object',
label: 'Voice Dictation',
category: 'General',
requiresRestart: false,
default: {},
description: 'Voice dictation settings.',
showInDialog: false,
properties: {
enabled: {
type: 'boolean',
label: 'Voice Dictation',
category: 'General',
requiresRestart: false,
default: false,
description: 'Enable voice dictation in the prompt input.',
showInDialog: false,
},
mode: {
type: 'enum',
label: 'Voice Dictation Mode',
category: 'General',
requiresRestart: false,
default: 'hold',
description:
'How push-to-talk behaves: "hold" to talk while held, or "tap" to start and tap (or pause) to stop and submit.',
showInDialog: false,
options: [
{ value: 'hold', label: 'Hold to talk' },
{ value: 'tap', label: 'Tap to toggle' },
],
},
language: {
type: 'string',
label: 'Voice Dictation Language',
category: 'General',
requiresRestart: false,
default: '',
description:
'Preferred spoken language for voice transcription (e.g. "english", "chinese"). Leave empty to auto-detect.',
showInDialog: false,
},
},
},
enableAutoUpdate: {
type: 'boolean',
label: 'Enable Auto Update',
@@ -1134,6 +1178,17 @@ const SETTINGS_SCHEMA = {
showInDialog: true,
},
voiceModel: {
type: 'string',
label: 'Voice Model',
category: 'Model',
requiresRestart: false,
default: '',
description:
'Model used for voice transcription. Set with /model --voice. Leave empty to keep voice dictation disabled until a voice model is selected.',
showInDialog: false,
},
model: {
type: 'object',
label: 'Model',
packages/cli/src/i18n/locales/en.js
+52
View File
@@ -1411,8 +1411,56 @@ export default {
// ============================================================================
'Switch the model for this session (--fast for suggestion model, [model-id] to switch immediately).':
'Switch the model for this session (--fast for suggestion model, [model-id] to switch immediately).',
'Switch the model for this session (--fast for suggestion model, --voice for voice transcription model, [model-id] to switch immediately).':
'Switch the model for this session (--fast for suggestion model, --voice for voice transcription model, [model-id] to switch immediately).',
'Set a lighter model for prompt suggestions and speculative execution':
'Set a lighter model for prompt suggestions and speculative execution',
'Toggle voice dictation input': 'Toggle voice dictation input',
'Set the model for voice transcription':
'Set the model for voice transcription',
'Select Fast Model': 'Select Fast Model',
'Select Voice Model': 'Select Voice Model',
'Voice Model': 'Voice Model',
'Selected voice model is unavailable.':
'Selected voice model is unavailable.',
"Voice model '{{model}}' is configured more than once. Remove duplicate model ids before selecting it for voice transcription.":
"Voice model '{{model}}' is configured more than once. Remove duplicate model ids before selecting it for voice transcription.",
'Voice dictation: {{status}} (mode: {{mode}}, {{modelText}}).':
'Voice dictation: {{status}} (mode: {{mode}}, {{modelText}}).',
'model: {{voiceModel}}': 'model: {{voiceModel}}',
'no voice model selected': 'no voice model selected',
'Voice dictation disabled.': 'Voice dictation disabled.',
'Usage: /voice [hold|tap|off|status]': 'Usage: /voice [hold|tap|off|status]',
'No voice model selected. Run /model --voice to choose one before enabling voice dictation.':
'No voice model selected. Run /model --voice to choose one before enabling voice dictation.',
'Voice dictation enabled (tap mode). Tap Space at an empty prompt to start, tap again or pause to stop and submit, using {{voiceModel}}.':
'Voice dictation enabled (tap mode). Tap Space at an empty prompt to start, tap again or pause to stop and submit, using {{voiceModel}}.',
'Voice dictation enabled (hold mode). Hold Space at an empty prompt to dictate with {{voiceModel}}.':
'Voice dictation enabled (hold mode). Hold Space at an empty prompt to dictate with {{voiceModel}}.',
'No models are configured.': 'No models are configured.',
'Configured models: {{models}}.': 'Configured models: {{models}}.',
'Configure a unique model id in settings.modelProviders or run /model --voice to select an available model.':
'Configure a unique model id in settings.modelProviders or run /model --voice to select an available model.',
"Voice model '{{modelName}}' is not configured.":
"Voice model '{{modelName}}' is not configured.",
"Voice model '{{modelName}}' cannot be used for transcription.":
"Voice model '{{modelName}}' cannot be used for transcription.",
"Voice model '{{modelName}}' cannot be used for transcription. Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.":
"Voice model '{{modelName}}' cannot be used for transcription. Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.",
'Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.':
'Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.',
'Microphone access is denied. Enable it for your terminal in System Settings → Privacy & Security → Microphone, then restart voice dictation.':
'Microphone access is denied. Enable it for your terminal in System Settings → Privacy & Security → Microphone, then restart voice dictation.',
'Voice dictation is not supported on {{platform}}.':
'Voice dictation is not supported on {{platform}}.',
'Voice dictation needs microphone access, which is unavailable in this WSL session. Use WSLg/PulseAudio, or run Qwen Code on a host with a microphone.':
'Voice dictation needs microphone access, which is unavailable in this WSL session. Use WSLg/PulseAudio, or run Qwen Code on a host with a microphone.',
'Voice dictation needs microphone access. macOS will ask the first time you record — approve it, then start again. Your first recording may be empty while the dialog is open.':
'Voice dictation needs microphone access. macOS will ask the first time you record — approve it, then start again. Your first recording may be empty while the dialog is open.',
'Voice: recording': 'Voice: recording',
'Voice: transcribing': 'Voice: transcribing',
'listening…': 'listening…',
'transcribing…': 'transcribing…',
'Content generator configuration not available.':
'Content generator configuration not available.',
'Authentication type not available.': 'Authentication type not available.',
@@ -1614,6 +1662,10 @@ export default {
audio: 'audio',
video: 'video',
'not set': 'not set',
'Current voice model: {{voiceModel}}\nUse "/model --voice <model-id>" to set voice model.':
'Current voice model: {{voiceModel}}\nUse "/model --voice <model-id>" to set voice model.',
"Voice model '{{modelName}}' is ambiguous. Configure a unique model id before using /model --voice.":
"Voice model '{{modelName}}' is ambiguous. Configure a unique model id before using /model --voice.",
none: 'none',
unknown: 'unknown',
// ============================================================================
packages/cli/src/i18n/locales/zh-TW.js
+50
View File
@@ -1210,8 +1210,54 @@ export default {
'生成摘要失敗 - 未從 LLM 響應中接收到文本內容',
'Switch the model for this session (--fast for suggestion model, [model-id] to switch immediately).':
'切換此會話的模型（--fast 可設置建議模型）',
'Switch the model for this session (--fast for suggestion model, --voice for voice transcription model, [model-id] to switch immediately).':
'切換此會話的模型（--fast 可設置建議模型，--voice 可設置語音轉寫模型，[model-id] 可立即切換）',
'Set a lighter model for prompt suggestions and speculative execution':
'設置用於輸入建議和推測執行的輕量模型',
'Toggle voice dictation input': '切換語音聽寫輸入',
'Set the model for voice transcription': '設定語音轉寫模型',
'Select Fast Model': '選擇快速模型',
'Select Voice Model': '選擇語音模型',
'Voice Model': '語音模型',
'Selected voice model is unavailable.': '所選語音模型不可用。',
"Voice model '{{model}}' is configured more than once. Remove duplicate model ids before selecting it for voice transcription.":
"語音模型 '{{model}}' 被配置了多次。請先移除重複的模型 ID，再將其選為語音轉寫模型。",
'Voice dictation: {{status}} (mode: {{mode}}, {{modelText}}).':
'語音聽寫：{{status}}（模式：{{mode}}，{{modelText}}）。',
'model: {{voiceModel}}': '模型：{{voiceModel}}',
'no voice model selected': '未選擇語音模型',
'Voice dictation disabled.': '語音聽寫已停用。',
'Usage: /voice [hold|tap|off|status]': '用法：/voice [hold|tap|off|status]',
'No voice model selected. Run /model --voice to choose one before enabling voice dictation.':
'未選擇語音模型。請先執行 /model --voice 選擇模型，再啟用語音聽寫。',
'Voice dictation enabled (tap mode). Tap Space at an empty prompt to start, tap again or pause to stop and submit, using {{voiceModel}}.':
'語音聽寫已啟用（點按模式）。在空輸入框中點按 Space 開始，再點按一次或停頓後停止並提交，使用 {{voiceModel}}。',
'Voice dictation enabled (hold mode). Hold Space at an empty prompt to dictate with {{voiceModel}}.':
'語音聽寫已啟用（按住模式）。在空輸入框中按住 Space，使用 {{voiceModel}} 聽寫。',
'No models are configured.': '未設定模型。',
'Configured models: {{models}}.': '已設定模型：{{models}}。',
'Configure a unique model id in settings.modelProviders or run /model --voice to select an available model.':
'請在 settings.modelProviders 中設定唯一的模型 ID，或執行 /model --voice 選擇可用模型。',
"Voice model '{{modelName}}' is not configured.":
"語音模型 '{{modelName}}' 未設定。",
"Voice model '{{modelName}}' cannot be used for transcription.":
"語音模型 '{{modelName}}' 不能用於轉寫。",
"Voice model '{{modelName}}' cannot be used for transcription. Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.":
"語音模型 '{{modelName}}' 不能用於轉寫。請在 settings.modelProviders 中設定帶 baseUrl 的 OpenAI 相容模型。",
'Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.':
'請在 settings.modelProviders 中設定帶 baseUrl 的 OpenAI 相容模型。',
'Microphone access is denied. Enable it for your terminal in System Settings → Privacy & Security → Microphone, then restart voice dictation.':
'麥克風存取被拒絕。請在系統設定 → 隱私權與安全性 → 麥克風中允許目前終端機存取，然後重新啟動語音聽寫。',
'Voice dictation is not supported on {{platform}}.':
'語音聽寫不支援 {{platform}}。',
'Voice dictation needs microphone access, which is unavailable in this WSL session. Use WSLg/PulseAudio, or run Qwen Code on a host with a microphone.':
'語音聽寫需要麥克風存取，但目前 WSL 會話不可用。請使用 WSLg/PulseAudio，或在具備麥克風的主機上執行 Qwen Code。',
'Voice dictation needs microphone access. macOS will ask the first time you record — approve it, then start again. Your first recording may be empty while the dialog is open.':
'語音聽寫需要麥克風存取。macOS 會在你首次錄音時彈出授權請求——請同意後重新開始。彈窗開啟期間的首次錄音可能為空。',
'Voice: recording': '語音：錄音中',
'Voice: transcribing': '語音：轉寫中',
'listening…': '聆聽中…',
'transcribing…': '轉寫中…',
'Content generator configuration not available.': '內容生成器配置不可用',
'Authentication type not available.': '認證類型不可用',
'No models available for the current authentication type ({{authType}}).':
@@ -1371,6 +1417,10 @@ export default {
audio: '音頻',
video: '視頻',
'not set': '未設置',
'Current voice model: {{voiceModel}}\nUse "/model --voice <model-id>" to set voice model.':
'當前語音模型：{{voiceModel}}\n使用 "/model --voice <model-id>" 設置語音模型。',
"Voice model '{{modelName}}' is ambiguous. Configure a unique model id before using /model --voice.":
"語音模型 '{{modelName}}' 不唯一。請先配置唯一的模型 ID，再使用 /model --voice。",
none: '無',
unknown: '未知',
'Manage folder trust settings': '管理檔案夾信任設置',
packages/cli/src/i18n/locales/zh.js
+50
View File
@@ -1324,8 +1324,54 @@ export default {
// ============================================================================
'Switch the model for this session (--fast for suggestion model, [model-id] to switch immediately).':
'切换此会话的模型（--fast 可设置建议模型）',
'Switch the model for this session (--fast for suggestion model, --voice for voice transcription model, [model-id] to switch immediately).':
'切换此会话的模型（--fast 可设置建议模型，--voice 可设置语音转写模型，[model-id] 可立即切换）',
'Set a lighter model for prompt suggestions and speculative execution':
'设置用于输入建议和推测执行的轻量模型',
'Toggle voice dictation input': '切换语音听写输入',
'Set the model for voice transcription': '设置语音转写模型',
'Select Fast Model': '选择快速模型',
'Select Voice Model': '选择语音模型',
'Voice Model': '语音模型',
'Selected voice model is unavailable.': '所选语音模型不可用。',
"Voice model '{{model}}' is configured more than once. Remove duplicate model ids before selecting it for voice transcription.":
"语音模型 '{{model}}' 被配置了多次。请先移除重复的模型 ID，再将其选为语音转写模型。",
'Voice dictation: {{status}} (mode: {{mode}}, {{modelText}}).':
'语音听写：{{status}}（模式：{{mode}}，{{modelText}}）。',
'model: {{voiceModel}}': '模型：{{voiceModel}}',
'no voice model selected': '未选择语音模型',
'Voice dictation disabled.': '语音听写已禁用。',
'Usage: /voice [hold|tap|off|status]': '用法：/voice [hold|tap|off|status]',
'No voice model selected. Run /model --voice to choose one before enabling voice dictation.':
'未选择语音模型。请先运行 /model --voice 选择模型，再启用语音听写。',
'Voice dictation enabled (tap mode). Tap Space at an empty prompt to start, tap again or pause to stop and submit, using {{voiceModel}}.':
'语音听写已启用（点击模式）。在空输入框中点击 Space 开始，再点击一次或停顿后停止并提交，使用 {{voiceModel}}。',
'Voice dictation enabled (hold mode). Hold Space at an empty prompt to dictate with {{voiceModel}}.':
'语音听写已启用（按住模式）。在空输入框中按住 Space，使用 {{voiceModel}} 听写。',
'No models are configured.': '未配置模型。',
'Configured models: {{models}}.': '已配置模型：{{models}}。',
'Configure a unique model id in settings.modelProviders or run /model --voice to select an available model.':
'请在 settings.modelProviders 中配置唯一的模型 ID，或运行 /model --voice 选择可用模型。',
"Voice model '{{modelName}}' is not configured.":
"语音模型 '{{modelName}}' 未配置。",
"Voice model '{{modelName}}' cannot be used for transcription.":
"语音模型 '{{modelName}}' 不能用于转写。",
"Voice model '{{modelName}}' cannot be used for transcription. Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.":
"语音模型 '{{modelName}}' 不能用于转写。请在 settings.modelProviders 中配置带 baseUrl 的 OpenAI 兼容模型。",
'Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.':
'请在 settings.modelProviders 中配置带 baseUrl 的 OpenAI 兼容模型。',
'Microphone access is denied. Enable it for your terminal in System Settings → Privacy & Security → Microphone, then restart voice dictation.':
'麦克风访问被拒绝。请在系统设置 → 隐私与安全性 → 麦克风中允许当前终端访问，然后重新启动语音听写。',
'Voice dictation is not supported on {{platform}}.':
'语音听写不支持 {{platform}}。',
'Voice dictation needs microphone access, which is unavailable in this WSL session. Use WSLg/PulseAudio, or run Qwen Code on a host with a microphone.':
'语音听写需要麦克风访问，但当前 WSL 会话不可用。请使用 WSLg/PulseAudio，或在带麦克风的宿主机上运行 Qwen Code。',
'Voice dictation needs microphone access. macOS will ask the first time you record — approve it, then start again. Your first recording may be empty while the dialog is open.':
'语音听写需要麦克风访问。macOS 会在你首次录音时弹出授权请求——请同意后重新开始。弹窗打开期间的首次录音可能为空。',
'Voice: recording': '语音：录音中',
'Voice: transcribing': '语音：转写中',
'listening…': '聆听中…',
'transcribing…': '转写中…',
'Content generator configuration not available.': '内容生成器配置不可用',
'Authentication type not available.': '认证类型不可用',
'No models available for the current authentication type ({{authType}}).':
@@ -1513,6 +1559,10 @@ export default {
audio: '音频',
video: '视频',
'not set': '未设置',
'Current voice model: {{voiceModel}}\nUse "/model --voice <model-id>" to set voice model.':
'当前语音模型：{{voiceModel}}\n使用 "/model --voice <model-id>" 设置语音模型。',
"Voice model '{{modelName}}' is ambiguous. Configure a unique model id before using /model --voice.":
"语音模型 '{{modelName}}' 不唯一。请先配置唯一的模型 ID，再使用 /model --voice。",
none: '无',
unknown: '未知',
// ============================================================================
packages/cli/src/services/BuiltinCommandLoader.ts
+2
View File
@@ -64,6 +64,7 @@ import { terminalSetupCommand } from '../ui/commands/terminalSetupCommand.js';
import { themeCommand } from '../ui/commands/themeCommand.js';
import { toolsCommand } from '../ui/commands/toolsCommand.js';
import { vimCommand } from '../ui/commands/vimCommand.js';
import { voiceCommand } from '../ui/commands/voice-command.js';
import { setupGithubCommand } from '../ui/commands/setupGithubCommand.js';
import { insightCommand } from '../ui/commands/insightCommand.js';
import { statuslineCommand } from '../ui/commands/statuslineCommand.js';
@@ -161,6 +162,7 @@ export class BuiltinCommandLoader implements ICommandLoader {
toolsCommand,
settingsCommand,
vimCommand,
voiceCommand,
setupGithubCommand,
terminalSetupCommand,
insightCommand,
packages/cli/src/ui/AppContainer.tsx
+3
View File
@@ -1064,6 +1064,7 @@ export const AppContainer = (props: AppContainerProps) => {
const {
isModelDialogOpen,
isFastModelMode,
isVoiceModelMode,
openModelDialog,
closeModelDialog,
} = useModelCommand();
@@ -3394,6 +3395,7 @@ export const AppContainer = (props: AppContainerProps) => {
isMemoryDialogOpen,
isModelDialogOpen,
isFastModelMode,
isVoiceModelMode,
isTrustDialogOpen,
activeArenaDialog,
isPermissionsDialogOpen,
@@ -3528,6 +3530,7 @@ export const AppContainer = (props: AppContainerProps) => {
isMemoryDialogOpen,
isModelDialogOpen,
isFastModelMode,
isVoiceModelMode,
isTrustDialogOpen,
activeArenaDialog,
isPermissionsDialogOpen,
packages/cli/src/ui/commands/modelCommand.test.ts
+309 -1
View File
@@ -45,7 +45,7 @@ describe('modelCommand', () => {
it('should have the correct name and description', () => {
expect(modelCommand.name).toBe('model');
expect(modelCommand.description).toBe(
'Switch the model for this session (--fast for suggestion model, [model-id] to switch immediately).',
'Switch the model for this session (--fast for suggestion model, --voice for voice transcription model, [model-id] to switch immediately).',
);
});
@@ -592,6 +592,314 @@ describe('modelCommand', () => {
});
});
it('should open the voice model dialog for /model --voice in interactive mode', async () => {
const mockConfig = createMockConfig({
model: 'qwen-plus',
authType: AuthType.USE_OPENAI,
});
mockContext.services.config = mockConfig as Config;
const result = await modelCommand.action!(mockContext, '--voice');
expect(result).toEqual({
type: 'dialog',
dialog: 'voice-model',
});
});
it('should return current voice model outside interactive mode', async () => {
mockContext = createMockCommandContext({
executionMode: 'non_interactive',
invocation: { args: '--voice' },
services: {
config: createMockConfig({
model: 'qwen-max',
authType: AuthType.USE_OPENAI,
}),
settings: {
merged: { voiceModel: 'qwen3-asr-flash' } as Record<string, unknown>,
},
},
});
const result = await modelCommand.action!(mockContext, '--voice');
expect(result).toEqual({
type: 'message',
messageType: 'info',
content:
'Current voice model: qwen3-asr-flash\nUse "/model --voice <model-id>" to set voice model.',
});
});
it('should set voice model without switching the main model', async () => {
const setValue = vi.fn();
const switchModel = vi.fn();
mockContext = createMockCommandContext({
invocation: {
raw: '/model --voice qwen3-asr-flash',
name: 'model',
args: '--voice qwen3-asr-flash',
},
services: {
config: {
getContentGeneratorConfig: vi.fn().mockReturnValue({
model: 'qwen-plus',
authType: AuthType.USE_OPENAI,
}),
getAllConfiguredModels: vi.fn().mockReturnValue([
{
id: 'qwen3-asr-flash',
label: 'qwen3-asr-flash',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
},
]),
switchModel,
},
settings: createMockSettings(setValue),
},
});
const result = await modelCommand.action!(
mockContext,
'--voice qwen3-asr-flash',
);
expect(setValue).toHaveBeenCalledWith(
expect.any(String),
'voiceModel',
'qwen3-asr-flash',
);
expect(switchModel).not.toHaveBeenCalled();
expect(setValue).not.toHaveBeenCalledWith(
expect.any(String),
'model.name',
expect.any(String),
);
expect(setValue).not.toHaveBeenCalledWith(
expect.any(String),
'fastModel',
expect.any(String),
);
expect(result).toEqual({
type: 'message',
messageType: 'info',
content: 'Voice Model: qwen3-asr-flash',
});
});
it('should reject unavailable voice models', async () => {
const setValue = vi.fn();
mockContext = createMockCommandContext({
invocation: {
raw: '/model --voice missing-model',
name: 'model',
args: '--voice missing-model',
},
services: {
config: {
getContentGeneratorConfig: vi.fn().mockReturnValue({
model: 'qwen-plus',
authType: AuthType.USE_OPENAI,
}),
getAllConfiguredModels: vi.fn().mockReturnValue([
{
id: 'qwen3-asr-flash',
label: 'qwen3-asr-flash',
authType: AuthType.USE_OPENAI,
},
]),
},
settings: createMockSettings(setValue),
},
});
const result = await modelCommand.action!(
mockContext,
'--voice missing-model',
);
expect(setValue).not.toHaveBeenCalled();
expect(result).toEqual({
type: 'message',
messageType: 'error',
content:
"Voice model 'missing-model' is not configured.\n" +
'Configured models: qwen3-asr-flash.\n' +
'Configure a unique model id in settings.modelProviders or run /model --voice to select an available model.',
});
});
it('should reject voice models that cannot use the transcription endpoint', async () => {
const setValue = vi.fn();
mockContext = createMockCommandContext({
invocation: {
raw: '/model --voice qwen3-coder',
name: 'model',
args: '--voice qwen3-coder',
},
services: {
config: {
getContentGeneratorConfig: vi.fn().mockReturnValue({
model: 'qwen-plus',
authType: AuthType.USE_OPENAI,
}),
getAllConfiguredModels: vi.fn().mockReturnValue([
{
id: 'qwen3-coder',
label: 'qwen3-coder',
authType: AuthType.USE_OPENAI,
},
]),
},
settings: createMockSettings(setValue),
},
});
const result = await modelCommand.action!(
mockContext,
'--voice qwen3-coder',
);
expect(result).toEqual({
type: 'message',
messageType: 'error',
content:
"Voice model 'qwen3-coder' cannot be used for transcription. Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.",
});
expect(setValue).not.toHaveBeenCalled();
});
it('should reject non OpenAI-compatible voice models', async () => {
const setValue = vi.fn();
mockContext = createMockCommandContext({
invocation: {
raw: '/model --voice claude-sonnet',
name: 'model',
args: '--voice claude-sonnet',
},
services: {
config: {
getContentGeneratorConfig: vi.fn().mockReturnValue({
model: 'qwen-plus',
authType: AuthType.USE_OPENAI,
}),
getAllConfiguredModels: vi.fn().mockReturnValue([
{
id: 'claude-sonnet',
label: 'claude-sonnet',
authType: AuthType.USE_ANTHROPIC,
baseUrl: 'https://anthropic.example/v1',
},
]),
},
settings: createMockSettings(setValue),
},
});
const result = await modelCommand.action!(
mockContext,
'--voice claude-sonnet',
);
expect(result).toEqual({
type: 'message',
messageType: 'error',
content:
"Voice model 'claude-sonnet' cannot be used for transcription. Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.",
});
expect(setValue).not.toHaveBeenCalled();
});
it('should reject duplicate voice model ids as ambiguous', async () => {
const setValue = vi.fn();
mockContext = createMockCommandContext({
invocation: {
raw: '/model --voice qwen3-asr-flash',
name: 'model',
args: '--voice qwen3-asr-flash',
},
services: {
config: {
getContentGeneratorConfig: vi.fn().mockReturnValue({
model: 'qwen-plus',
authType: AuthType.USE_OPENAI,
}),
getAllConfiguredModels: vi.fn().mockReturnValue([
{
id: 'qwen3-asr-flash',
label: 'first',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://one.example/v1',
},
{
id: 'qwen3-asr-flash',
label: 'second',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://two.example/v1',
},
]),
},
settings: createMockSettings(setValue),
},
});
const result = await modelCommand.action!(
mockContext,
'--voice qwen3-asr-flash',
);
expect(setValue).not.toHaveBeenCalled();
expect(result).toEqual({
type: 'message',
messageType: 'error',
content:
"Voice model 'qwen3-asr-flash' is ambiguous. Configure a unique model id before using /model --voice.",
});
});
it('should treat colon-containing voice model values as literal ids', async () => {
const setValue = vi.fn();
mockContext = createMockCommandContext({
invocation: {
raw: '/model --voice openai:qwen3-asr-flash',
name: 'model',
args: '--voice openai:qwen3-asr-flash',
},
services: {
config: {
getContentGeneratorConfig: vi.fn().mockReturnValue({
model: 'qwen-plus',
authType: AuthType.USE_OPENAI,
}),
getAllConfiguredModels: vi.fn().mockReturnValue([
{
id: 'qwen3-asr-flash',
label: 'qwen3-asr-flash',
authType: AuthType.USE_OPENAI,
},
]),
},
settings: createMockSettings(setValue),
},
});
const result = await modelCommand.action!(
mockContext,
'--voice openai:qwen3-asr-flash',
);
expect(setValue).not.toHaveBeenCalled();
expect(result).toMatchObject({
type: 'message',
messageType: 'error',
content: expect.stringContaining(
"Voice model 'openai:qwen3-asr-flash' is not configured.",
),
});
});
it('should not treat model IDs prefixed with --fast as the --fast flag', async () => {
const setValue = vi.fn();
const switchModel = vi.fn().mockResolvedValue(undefined);
packages/cli/src/ui/commands/modelCommand.ts
+116 -10
View File
@@ -21,6 +21,10 @@ import {
} from '@qwen-code/qwen-code-core';
import type { LoadedSettings } from '../../config/settings.js';
import { parseAcpModelOption } from '../../utils/acpModelUtils.js';
import {
formatUnsupportedVoiceModelMessage,
isSelectableVoiceModel,
} from '../voice/voice-model.js';
const MAIN_MODEL_CONFIGURATION_HINT =
'Configure models in settings.modelProviders and ensure the required environment variables are set. In interactive mode, run /auth to configure or switch providers, or run /model without arguments to choose from configured models.';
@@ -112,6 +116,30 @@ function formatUnavailableFastModelMessage(
);
}
function formatUnavailableVoiceModelMessage(
modelName: string,
availableModels: AvailableModel[],
): string {
const availableModelIds = Array.from(
new Set(availableModels.map((model) => model.id)),
);
const availableModelsLine =
availableModelIds.length === 0
? t('No models are configured.')
: t('Configured models: {{models}}.', {
models: availableModelIds.join(', '),
});
return (
t("Voice model '{{modelName}}' is not configured.", { modelName }) +
'\n' +
`${availableModelsLine}\n` +
t(
'Configure a unique model id in settings.modelProviders or run /model --voice to select an available model.',
)
);
}
// Get an array of the available model IDs as strings
function getAvailableModelIds(context: CommandContext) {
const { services } = context;
@@ -129,27 +157,35 @@ export const modelCommand: SlashCommand = {
completionPriority: 100,
get description() {
return t(
'Switch the model for this session (--fast for suggestion model, [model-id] to switch immediately).',
'Switch the model for this session (--fast for suggestion model, --voice for voice transcription model, [model-id] to switch immediately).',
);
},
argumentHint: '[--fast] [<model-id>]',
argumentHint: '[--fast|--voice] [<model-id>]',
kind: CommandKind.BUILT_IN,
supportedModes: ['interactive', 'non_interactive', 'acp'] as const,
completion: async (context, partialArg) => {
if (partialArg && '--fast'.startsWith(partialArg)) {
return [
if (partialArg) {
const flagCompletions = [
{
value: '--fast',
description: t(
'Set a lighter model for prompt suggestions and speculative execution',
),
},
];
} else if (partialArg.trim()) {
// Include model IDs matching the partial argument
return getAvailableModelIds(context).filter((id) =>
id.startsWith(partialArg.trim()),
);
{
value: '--voice',
description: t('Set the model for voice transcription'),
},
].filter((item) => item.value.startsWith(partialArg));
if (flagCompletions.length > 0) {
return flagCompletions;
}
if (partialArg.trim()) {
return getAvailableModelIds(context).filter((id) =>
id.startsWith(partialArg.trim()),
);
}
return null;
} else {
return null;
}
@@ -171,6 +207,76 @@ export const modelCommand: SlashCommand = {
// Handle --fast flag: /model --fast <modelName>
const args = context.invocation?.args?.trim() || actionArgs.trim();
const isVoiceModelCommand =
args === '--voice' || args.startsWith('--voice ');
if (isVoiceModelCommand) {
const modelName = args.replace('--voice', '').trim();
if (!modelName) {
if (context.executionMode !== 'interactive') {
const voiceModel =
context.services.settings?.merged?.voiceModel?.trim() ||
t('not set');
return {
type: 'message',
messageType: 'info',
content: t(
'Current voice model: {{voiceModel}}\nUse "/model --voice <model-id>" to set voice model.',
{ voiceModel },
),
};
}
return {
type: 'dialog',
dialog: 'voice-model',
};
}
if (!settings) {
return {
type: 'message',
messageType: 'error',
content: t('Settings service not available.'),
};
}
const availableModels = config.getAllConfiguredModels();
const matches = availableModels.filter((model) => model.id === modelName);
if (matches.length === 0) {
return {
type: 'message',
messageType: 'error',
content: formatUnavailableVoiceModelMessage(
modelName,
availableModels,
),
};
}
if (matches.length > 1) {
return {
type: 'message',
messageType: 'error',
content: t(
"Voice model '{{modelName}}' is ambiguous. Configure a unique model id before using /model --voice.",
{ modelName },
),
};
}
if (!isSelectableVoiceModel(matches[0]!)) {
return {
type: 'message',
messageType: 'error',
content: formatUnsupportedVoiceModelMessage(modelName),
};
}
persistSetting(settings, 'voiceModel', modelName);
return {
type: 'message',
messageType: 'info',
content: t('Voice Model') + ': ' + modelName,
};
}
const isFastModelCommand = args === '--fast' || args.startsWith('--fast ');
if (isFastModelCommand) {
const modelName = args.replace('--fast', '').trim();
packages/cli/src/ui/commands/types.ts
+1
View File
@@ -179,6 +179,7 @@ export interface OpenDialogActionReturn {
| 'memory'
| 'model'
| 'fast-model'
| 'voice-model'
| 'subagent_create'
| 'subagent_list'
| 'skills_manage'
packages/cli/src/ui/commands/voice-command.test.ts
+268
View File
@@ -0,0 +1,268 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, it, expect, vi } from 'vitest';
import { voiceCommand } from './voice-command.js';
import { createMockCommandContext } from '../../test-utils/mockCommandContext.js';
import { SettingScope, type LoadedSettings } from '../../config/settings.js';
import type { CommandContext } from './types.js';
function createSettings(
merged: Record<string, unknown>,
setValue = vi.fn(),
options: Partial<LoadedSettings> = {},
): LoadedSettings {
return {
merged,
isTrusted: false,
workspace: { settings: {} },
setValue,
...options,
} as unknown as LoadedSettings;
}
describe('voice-command', () => {
it('has the expected metadata', () => {
expect(voiceCommand.name).toBe('voice');
expect(voiceCommand.argumentHint).toBe('[hold|tap|off|status]');
});
it('prompts for a voice model before enabling', async () => {
const setValue = vi.fn();
const context = createMockCommandContext({
services: {
settings: createSettings({}, setValue),
},
});
const result = await voiceCommand.action!(context, '');
expect(result).toEqual({
type: 'message',
messageType: 'warning',
content:
'No voice model selected. Run /model --voice to choose one before enabling voice dictation.',
});
expect(setValue).not.toHaveBeenCalled();
});
it('enables hold-mode voice dictation when a voice model is selected', async () => {
const setValue = vi.fn();
const context = createMockCommandContext({
services: {
settings: createSettings({ voiceModel: 'qwen3-asr-flash' }, setValue),
},
});
const result = await voiceCommand.action!(context, '');
expect(setValue).toHaveBeenCalledWith(
SettingScope.User,
'general.voice.mode',
'hold',
);
expect(setValue).toHaveBeenCalledWith(
SettingScope.User,
'general.voice.enabled',
true,
);
expect(result).toEqual({
type: 'message',
messageType: 'info',
content:
'Voice dictation enabled (hold mode). Hold Space at an empty prompt to dictate with qwen3-asr-flash.',
});
});
it('toggles voice dictation off when bare /voice is used while enabled', async () => {
const setValue = vi.fn();
const context = createMockCommandContext({
services: {
settings: createSettings(
{ general: { voice: { enabled: true, mode: 'tap' } } },
setValue,
),
},
});
const result = await voiceCommand.action!(context, '');
expect(setValue).toHaveBeenCalledTimes(1);
expect(setValue).toHaveBeenCalledWith(
SettingScope.User,
'general.voice.enabled',
false,
);
expect(result).toEqual({
type: 'message',
messageType: 'info',
content: 'Voice dictation disabled.',
});
});
it('enables tap mode when /voice tap is used', async () => {
const setValue = vi.fn();
const context = createMockCommandContext({
services: {
settings: createSettings({ voiceModel: 'qwen3-asr-flash' }, setValue),
},
});
const result = await voiceCommand.action!(context, 'tap');
expect(setValue).toHaveBeenCalledWith(
SettingScope.User,
'general.voice.mode',
'tap',
);
expect(setValue).toHaveBeenCalledWith(
SettingScope.User,
'general.voice.enabled',
true,
);
expect(result).toEqual({
type: 'message',
messageType: 'info',
content:
'Voice dictation enabled (tap mode). Tap Space at an empty prompt to start, tap again or pause to stop and submit, using qwen3-asr-flash.',
});
});
it('preserves tap mode when bare /voice re-enables voice dictation', async () => {
const setValue = vi.fn();
const context = createMockCommandContext({
services: {
settings: createSettings(
{
voiceModel: 'qwen3-asr-flash',
general: { voice: { enabled: false, mode: 'tap' } },
},
setValue,
),
},
});
const result = await voiceCommand.action!(context, '');
expect(setValue).toHaveBeenCalledWith(
SettingScope.User,
'general.voice.mode',
'tap',
);
expect(result).toEqual({
type: 'message',
messageType: 'info',
content:
'Voice dictation enabled (tap mode). Tap Space at an empty prompt to start, tap again or pause to stop and submit, using qwen3-asr-flash.',
});
});
it('disables voice dictation even when no voice model is selected', async () => {
const setValue = vi.fn();
const context = createMockCommandContext({
services: {
settings: createSettings({}, setValue),
},
});
const result = await voiceCommand.action!(context, 'off');
expect(setValue).toHaveBeenCalledWith(
SettingScope.User,
'general.voice.enabled',
false,
);
expect(result).toEqual({
type: 'message',
messageType: 'info',
content: 'Voice dictation disabled.',
});
});
it('writes to workspace scope when workspace owns voice enabled', async () => {
const setValue = vi.fn();
const context = createMockCommandContext({
services: {
settings: createSettings(
{
voiceModel: 'qwen3-asr-flash',
general: { voice: { enabled: false } },
},
setValue,
{
isTrusted: true,
workspace: {
settings: { general: { voice: { enabled: false } } },
},
} as Partial<LoadedSettings>,
),
},
});
await voiceCommand.action!(context, 'hold');
expect(setValue).toHaveBeenCalledWith(
SettingScope.Workspace,
'general.voice.enabled',
true,
);
expect(setValue).toHaveBeenCalledWith(
SettingScope.Workspace,
'general.voice.mode',
'hold',
);
});
it('reports the current voice status', async () => {
const context = createMockCommandContext({
services: {
settings: createSettings({
voiceModel: 'qwen3-asr-flash',
general: { voice: { enabled: true } },
}),
},
}) as CommandContext;
const result = await voiceCommand.action!(context, 'status');
expect(result).toEqual({
type: 'message',
messageType: 'info',
content: 'Voice dictation: enabled (mode: hold, model: qwen3-asr-flash).',
});
});
it('reports the configured mode in status', async () => {
const context = createMockCommandContext({
services: {
settings: createSettings({
voiceModel: 'qwen3-asr-flash',
general: { voice: { enabled: true, mode: 'tap' } },
}),
},
}) as CommandContext;
const result = await voiceCommand.action!(context, 'status');
expect(result).toEqual({
type: 'message',
messageType: 'info',
content: 'Voice dictation: enabled (mode: tap, model: qwen3-asr-flash).',
});
});
it('returns usage for unknown arguments', async () => {
const context = createMockCommandContext();
const result = await voiceCommand.action!(context, 'maybe');
expect(result).toEqual({
type: 'message',
messageType: 'warning',
content: 'Usage: /voice [hold|tap|off|status]',
});
});
});
packages/cli/src/ui/commands/voice-command.ts
+157
View File
@@ -0,0 +1,157 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { SettingScope } from '../../config/settings.js';
import { t } from '../../i18n/index.js';
import { CommandKind, type SlashCommand } from './types.js';
import { getVoiceUnavailableReason } from '../voice/voice-availability.js';
type VoiceMode = 'hold' | 'tap';
function getVoiceModel(settings: {
merged?: { voiceModel?: unknown };
}): string | undefined {
const value = settings.merged?.voiceModel;
if (typeof value !== 'string') {
return undefined;
}
const trimmed = value.trim();
return trimmed.length > 0 ? trimmed : undefined;
}
function isVoiceEnabled(settings: {
merged?: { general?: { voice?: { enabled?: unknown } } };
}): boolean {
return settings.merged?.general?.voice?.enabled === true;
}
function getVoiceMode(settings: {
merged?: { general?: { voice?: { mode?: unknown } } };
}): VoiceMode {
return settings.merged?.general?.voice?.mode === 'tap' ? 'tap' : 'hold';
}
function getVoiceScope(settings: {
isTrusted?: boolean;
workspace?: { settings?: { general?: { voice?: { enabled?: unknown } } } };
}): SettingScope {
return settings.isTrusted === true &&
typeof settings.workspace?.settings?.general?.voice?.enabled === 'boolean'
? SettingScope.Workspace
: SettingScope.User;
}
export const voiceCommand: SlashCommand = {
name: 'voice',
get description() {
return t('Toggle voice dictation input');
},
argumentHint: '[hold|tap|off|status]',
kind: CommandKind.BUILT_IN,
supportedModes: ['interactive'] as const,
action: (context, args) => {
const settings = context.services.settings;
const command = args.trim().toLowerCase();
if (command === 'status') {
const voiceModel = getVoiceModel(settings);
const status = isVoiceEnabled(settings) ? 'enabled' : 'disabled';
const modelText = voiceModel
? t('model: {{voiceModel}}', { voiceModel })
: t('no voice model selected');
return {
type: 'message',
messageType: 'info',
content: t(
'Voice dictation: {{status}} (mode: {{mode}}, {{modelText}}).',
{
status,
mode: getVoiceMode(settings),
modelText,
},
),
};
}
if (command === 'off') {
settings.setValue(
getVoiceScope(settings),
'general.voice.enabled',
false,
);
return {
type: 'message',
messageType: 'info',
content: t('Voice dictation disabled.'),
};
}
if (command !== '' && command !== 'hold' && command !== 'tap') {
return {
type: 'message',
messageType: 'warning',
content: t('Usage: /voice [hold|tap|off|status]'),
};
}
if (command === '' && isVoiceEnabled(settings)) {
settings.setValue(
getVoiceScope(settings),
'general.voice.enabled',
false,
);
return {
type: 'message',
messageType: 'info',
content: t('Voice dictation disabled.'),
};
}
const voiceModel = getVoiceModel(settings);
if (!voiceModel) {
return {
type: 'message',
messageType: 'warning',
content: t(
'No voice model selected. Run /model --voice to choose one before enabling voice dictation.',
),
};
}
const unavailableReason = getVoiceUnavailableReason();
if (unavailableReason) {
return {
type: 'message',
messageType: 'warning',
content: unavailableReason,
};
}
const scope = getVoiceScope(settings);
const mode: VoiceMode =
command === 'tap'
? 'tap'
: command === 'hold'
? 'hold'
: getVoiceMode(settings);
settings.setValue(scope, 'general.voice.mode', mode);
settings.setValue(scope, 'general.voice.enabled', true);
return {
type: 'message',
messageType: 'info',
content:
mode === 'tap'
? t(
'Voice dictation enabled (tap mode). Tap Space at an empty prompt to start, tap again or pause to stop and submit, using {{voiceModel}}.',
{ voiceModel },
)
: t(
'Voice dictation enabled (hold mode). Hold Space at an empty prompt to dictate with {{voiceModel}}.',
{ voiceModel },
),
};
},
};
packages/cli/src/ui/components/DialogManager.tsx
+1
View File
@@ -268,6 +268,7 @@ export const DialogManager = ({
<ModelDialog
onClose={uiActions.closeModelDialog}
isFastModelMode={uiState.isFastModelMode}
isVoiceModelMode={uiState.isVoiceModelMode}
/>
);
}
packages/cli/src/ui/components/InputPrompt.test.tsx
+139
View File
@@ -26,6 +26,7 @@ import type { UseInputHistoryReturn } from '../hooks/useInputHistory.js';
import { useInputHistory } from '../hooks/useInputHistory.js';
import type { UseReverseSearchCompletionReturn } from '../hooks/useReverseSearchCompletion.js';
import { useReverseSearchCompletion } from '../hooks/useReverseSearchCompletion.js';
import { useVoiceInput } from '../hooks/use-voice-input.js';
import * as clipboardUtils from '../utils/clipboardUtils.js';
import { createMockCommandContext } from '../../test-utils/mockCommandContext.js';
import stripAnsi from 'strip-ansi';
@@ -54,6 +55,7 @@ vi.mock('../hooks/useShellHistory.js');
vi.mock('../hooks/useCommandCompletion.js');
vi.mock('../hooks/useInputHistory.js');
vi.mock('../hooks/useReverseSearchCompletion.js');
vi.mock('../hooks/use-voice-input.js');
vi.mock('../utils/clipboardUtils.js');
vi.mock('../contexts/UIStateContext.js', () => ({
useUIState: vi.fn(() => ({ isFeedbackDialogOpen: false, messageQueue: [] })),
@@ -184,6 +186,7 @@ describe('InputPrompt', () => {
const mockedUseReverseSearchCompletion = vi.mocked(
useReverseSearchCompletion,
);
const mockedUseVoiceInput = vi.mocked(useVoiceInput);
beforeEach(() => {
vi.resetAllMocks();
@@ -325,6 +328,12 @@ describe('InputPrompt', () => {
mockedUseReverseSearchCompletion.mockReturnValue(
mockReverseSearchCompletion,
);
mockedUseVoiceInput.mockReturnValue({
status: 'idle',
interimText: '',
audioLevel: 0,
handleKeypress: vi.fn(() => false),
});
props = {
buffer: mockBuffer,
@@ -351,6 +360,136 @@ describe('InputPrompt', () => {
};
});
it('routes Space through voice input when the prompt is empty', async () => {
const handleVoiceKeypress = vi.fn(() => true);
mockedUseVoiceInput.mockReturnValue({
status: 'idle',
interimText: '',
audioLevel: 0,
handleKeypress: handleVoiceKeypress,
});
const { stdin, unmount } = renderWithProviders(<InputPrompt {...props} />);
stdin.write(' ');
await waitFor(() => {
expect(handleVoiceKeypress).toHaveBeenCalled();
});
expect(props.buffer.handleInput).not.toHaveBeenCalled();
unmount();
});
it('keeps normal Space typing when the prompt already has text', async () => {
const handleVoiceKeypress = vi.fn(() => true);
mockedUseVoiceInput.mockReturnValue({
status: 'idle',
interimText: '',
audioLevel: 0,
handleKeypress: handleVoiceKeypress,
});
props.buffer.setText('hello');
const { stdin, unmount } = renderWithProviders(<InputPrompt {...props} />);
stdin.write(' ');
await waitFor(() => {
expect(props.buffer.handleInput).toHaveBeenCalled();
});
expect(handleVoiceKeypress).not.toHaveBeenCalled();
unmount();
});
it('lets Space stop voice recording even when shell mode is active', async () => {
const handleVoiceKeypress = vi.fn(() => true);
mockedUseVoiceInput.mockReturnValue({
status: 'recording',
interimText: '',
audioLevel: 0,
handleKeypress: handleVoiceKeypress,
});
props.shellModeActive = true;
const { stdin, unmount } = renderWithProviders(<InputPrompt {...props} />);
stdin.write(' ');
await waitFor(() => {
expect(handleVoiceKeypress).toHaveBeenCalled();
});
expect(props.buffer.handleInput).not.toHaveBeenCalled();
unmount();
});
it('does not route voice keys while the background dialog is open', async () => {
const handleVoiceKeypress = vi.fn(() => true);
mockedUseVoiceInput.mockReturnValue({
status: 'recording',
interimText: '',
audioLevel: 0,
handleKeypress: handleVoiceKeypress,
});
mockedUseBackgroundTaskViewState.mockReturnValue({
entries: [],
selectedIndex: 0,
dialogMode: 'list',
dialogOpen: true,
pillFocused: false,
livePanelFocused: false,
livePanelSelectedIndex: 0,
});
const { stdin, unmount } = renderWithProviders(<InputPrompt {...props} />);
stdin.write(' ');
await waitFor(() => {
expect(handleVoiceKeypress).not.toHaveBeenCalled();
});
expect(props.buffer.handleInput).not.toHaveBeenCalled();
unmount();
});
it('lets the feedback dialog consume option keys before active voice input', async () => {
const handleVoiceKeypress = vi.fn(() => true);
mockedUseVoiceInput.mockReturnValue({
status: 'recording',
interimText: '',
audioLevel: 0,
handleKeypress: handleVoiceKeypress,
});
mockedUseUIState.mockReturnValue({
isFeedbackDialogOpen: true,
messageQueue: [],
pendingGeminiHistoryItems: [],
} as unknown as ReturnType<typeof useUIState>);
const { stdin, unmount } = renderWithProviders(<InputPrompt {...props} />);
stdin.write('1');
await waitFor(() => {
expect(handleVoiceKeypress).not.toHaveBeenCalled();
});
expect(props.buffer.handleInput).not.toHaveBeenCalled();
unmount();
});
it('lets non-voice keys fall through while voice recording is active', async () => {
const handleVoiceKeypress = vi.fn(() => false);
mockedUseVoiceInput.mockReturnValue({
status: 'recording',
interimText: '',
audioLevel: 0,
handleKeypress: handleVoiceKeypress,
});
const { stdin, unmount } = renderWithProviders(<InputPrompt {...props} />);
stdin.write('a');
await waitFor(() => {
expect(handleVoiceKeypress).toHaveBeenCalled();
});
expect(props.buffer.handleInput).toHaveBeenCalled();
unmount();
});
// Two microtask yields are intentional: Ink 7 + React 19 split a render
// pass across two ticks (one to flush state updates into the reconciler,
// a second for the resulting effects to settle). A single Promise.resolve
packages/cli/src/ui/components/InputPrompt.tsx
+180 -21
View File
@@ -47,6 +47,7 @@ import { SCREEN_READER_USER_PREFIX } from '../textConstants.js';
import { useShellFocusState } from '../contexts/ShellFocusContext.js';
import { useUIState } from '../contexts/UIStateContext.js';
import { useUIActions } from '../contexts/UIActionsContext.js';
import { useSettings } from '../contexts/SettingsContext.js';
import { useKeypressContext } from '../contexts/KeypressContext.js';
import {
useAgentViewState,
@@ -61,6 +62,22 @@ import { FEEDBACK_DIALOG_KEYS } from '../FeedbackDialog.js';
import { BaseTextInput } from './BaseTextInput.js';
import type { RenderLineOptions } from './BaseTextInput.js';
import { getApprovalModePromptStyle } from './approvalModeVisuals.js';
import {
useVoiceInput,
type VoiceTranscriber,
} from '../hooks/use-voice-input.js';
import { createVoiceRecorder } from '../voice/voice-recorder.js';
import {
assertVoiceBaseUrlNetworkAllowed,
isKeytermEcho,
isStreamingVoiceModel,
resolveVoiceStreamConfig,
transcribeVoiceAudio,
} from '../voice/voice-transcriber.js';
import { openQwenAsrRealtimeStream } from '../voice/qwen-asr-realtime-session.js';
import { openVoiceStream } from '../voice/voice-stream-session.js';
import { openVoiceStreamWithRetry } from '../voice/voice-stream-retry.js';
import { VoiceIndicator } from './VoiceIndicator.js';
/**
* Represents an attachment (e.g., pasted image) displayed above the input prompt
@@ -151,6 +168,7 @@ export const InputPrompt: React.FC<InputPromptProps> = ({
const isShellFocused = useShellFocusState();
const uiState = useUIState();
const uiActions = useUIActions();
const settings = useSettings();
const { pasteWorkaround } = useKeypressContext();
const { agents, agentTabBarFocused } = useAgentViewState();
const { setAgentTabBarFocused } = useAgentViewActions();
@@ -290,6 +308,116 @@ export const InputPrompt: React.FC<InputPromptProps> = ({
isFocused: isShellFocused,
});
const rawVoiceModel = settings.merged.voiceModel;
const voiceModel =
typeof rawVoiceModel === 'string' && rawVoiceModel.trim().length > 0
? rawVoiceModel.trim()
: undefined;
const voiceEnabled =
settings.merged.general?.voice?.enabled === true && Boolean(voiceModel);
const voiceMode =
settings.merged.general?.voice?.mode === 'tap' ? 'tap' : 'hold';
// handleSubmitAndClear is defined below; bridge with a ref so tap-mode voice
// can submit the prompt once the transcript is inserted.
const voiceSubmitRef = useRef<(text: string) => void>(() => {});
const transcribeVoice = useCallback<VoiceTranscriber>(
(audio, { voiceModel }) =>
transcribeVoiceAudio(audio, { config, settings, voiceModel }),
[config, settings],
);
const voiceMicWarnedStatusRef = useRef<string | null>(null);
const voiceRecorderRef = useRef<ReturnType<
typeof createVoiceRecorder
> | null>(null);
const getVoiceRecorder = useCallback(() => {
voiceRecorderRef.current ??= createVoiceRecorder();
return voiceRecorderRef.current;
}, []);
const warmupVoice = useCallback(() => {
const recorder = getVoiceRecorder();
void Promise.resolve(recorder.warmup?.()).catch(() => {});
void Promise.resolve(recorder.microphoneStatus?.())
.then((status) => {
if (voiceMicWarnedStatusRef.current === status) {
return;
}
if (status === 'denied') {
voiceMicWarnedStatusRef.current = status;
uiState.historyManager?.addItem(
{
type: 'error',
text: t(
'Microphone access is denied. Enable it for your terminal in System Settings → Privacy & Security → Microphone, then restart voice dictation.',
),
},
Date.now(),
);
} else if (status === 'prompt') {
// notDetermined: macOS raises the permission dialog on first capture,
// so that first recording can come back empty. Tell the user once
// instead of letting it look like a silent no-op.
voiceMicWarnedStatusRef.current = status;
uiState.historyManager?.addItem(
{
type: 'info',
text: t(
'Voice dictation needs microphone access. macOS will ask the first time you record — approve it, then start again. Your first recording may be empty while the dialog is open.',
),
},
Date.now(),
);
}
})
.catch(() => {});
}, [getVoiceRecorder, uiState.historyManager]);
const voiceStreaming = voiceModel ? isStreamingVoiceModel(voiceModel) : false;
const openVoiceStreamSession = useCallback(
(callbacks: {
onInterim: (text: string) => void;
onError?: (error: Error) => void;
}) => {
if (!voiceModel) {
return Promise.reject(new Error('No voice model selected.'));
}
const streamConfig = resolveVoiceStreamConfig({
config,
settings,
voiceModel,
});
return assertVoiceBaseUrlNetworkAllowed(streamConfig)
.then(() =>
openVoiceStreamWithRetry(() =>
streamConfig.transport === 'qwen-asr-realtime'
? openQwenAsrRealtimeStream(streamConfig, callbacks)
: openVoiceStream(streamConfig, callbacks),
),
)
.then((session) => ({
...session,
finish: async () => {
const transcript = await session.finish();
return isKeytermEcho(transcript, streamConfig.keytermsContext)
? ''
: transcript;
},
}));
},
[config, settings, voiceModel],
);
const voiceInput = useVoiceInput({
enabled: voiceEnabled,
mode: voiceMode,
voiceModel,
buffer,
addItem: uiState.historyManager?.addItem,
createRecorder: getVoiceRecorder,
transcribe: transcribeVoice,
onSubmit: (text) => voiceSubmitRef.current(text),
warmup: warmupVoice,
streaming: voiceStreaming,
openStream: voiceStreaming ? openVoiceStreamSession : undefined,
});
const resetCompletionState = completion.resetCompletionState;
const dismissCompletion = completion.dismissCompletion;
const resetReverseSearchCompletionState =
@@ -427,6 +555,9 @@ export const InputPrompt: React.FC<InputPromptProps> = ({
],
);
// Tap-mode voice dictation submits the prompt after the transcript lands.
voiceSubmitRef.current = (text) => handleSubmitAndClear(text);
const customSetTextAndResetCompletionSignal = useCallback(
(newText: string) => {
buffer.setText(newText);
@@ -553,6 +684,31 @@ export const InputPrompt: React.FC<InputPromptProps> = ({
const handleInput = useCallback(
(key: Key): boolean => {
// When the Background tasks dialog is open, swallow every key so
// nothing reaches the composer buffer — the dialog's own keypress
// handler owns selection, open/close, and stop actions. Keep this ahead
// of active voice handling so modal UI remains the key owner.
if (bgDialogOpen) {
return true;
}
// Handle feedback dialog keyboard interactions before global voice
// handling so modal UI gets first chance to consume the key.
if (uiState.isFeedbackDialogOpen) {
// If it's one of the feedback option keys (1-4), let FeedbackDialog handle it
if ((FEEDBACK_DIALOG_KEYS as readonly string[]).includes(key.name)) {
return true;
} else {
// For any other key, close feedback dialog temporarily and continue with normal processing
uiActions.temporaryCloseFeedbackDialog();
// Continue processing the key for normal input handling
}
}
if (voiceInput.status !== 'idle') {
return voiceInput.handleKeypress(key);
}
// When the Arena tab bar or background pill has focus, block
// non-printable keys so arrow keys and shortcuts don't interfere.
// Printable characters fall through to BaseTextInput's default
@@ -646,16 +802,6 @@ export const InputPrompt: React.FC<InputPromptProps> = ({
return true; // consume non-printable keys
}
// When the Background tasks dialog is open, swallow every key so
// nothing reaches the composer buffer — the dialog's own keypress
// handler owns selection, open/close, and stop actions. Unlike
// the tab bar we do NOT let printable chars type through, because
// the dialog doesn't auto-close on printable input and users
// would leak text into the hidden composer.
if (bgDialogOpen) {
return true;
}
// TODO(jacobr): this special case is likely not needed anymore.
// We should probably stop supporting paste if the InputPrompt is not
// focused.
@@ -716,16 +862,16 @@ export const InputPrompt: React.FC<InputPromptProps> = ({
return true;
}
// Handle feedback dialog keyboard interactions when dialog is open
if (uiState.isFeedbackDialogOpen) {
// If it's one of the feedback option keys (1-4), let FeedbackDialog handle it
if ((FEEDBACK_DIALOG_KEYS as readonly string[]).includes(key.name)) {
return true;
} else {
// For any other key, close feedback dialog temporarily and continue with normal processing
uiActions.temporaryCloseFeedbackDialog();
// Continue processing the key for normal input handling
}
if (
!shellModeActive &&
!reverseSearchActive &&
!commandSearchActive &&
!showCompletionSuggestions &&
!isAttachmentMode &&
buffer.text.length === 0 &&
voiceInput.handleKeypress(key)
) {
return true;
}
// Helper: pop all queued messages into the input buffer,
@@ -1469,6 +1615,7 @@ export const InputPrompt: React.FC<InputPromptProps> = ({
exportCompletion,
isHistoryRestoredText,
showCompletionSuggestions,
voiceInput,
],
);
@@ -1677,6 +1824,13 @@ export const InputPrompt: React.FC<InputPromptProps> = ({
? (statusColor ?? theme.border.focused)
: theme.border.default;
const voiceStatusLabel =
voiceInput.status === 'recording'
? t('Voice: recording')
: voiceInput.status === 'transcribing'
? t('Voice: transcribing')
: undefined;
const prefixNode = (
<Text
color={statusColor ?? theme.text.accent}
@@ -1730,6 +1884,11 @@ export const InputPrompt: React.FC<InputPromptProps> = ({
))}
</Box>
)}
<VoiceIndicator
status={voiceInput.status}
interimText={voiceInput.interimText}
audioLevel={voiceInput.audioLevel}
/>
<BaseTextInput
buffer={buffer}
onSubmit={handleSubmitAndClear}
@@ -1739,7 +1898,7 @@ export const InputPrompt: React.FC<InputPromptProps> = ({
prefix={prefixNode}
prefixWidth={prefixWidth}
borderColor={borderColor}
topRightLabel={uiState.sessionName || undefined}
topRightLabel={voiceStatusLabel ?? uiState.sessionName ?? undefined}
isActive={!isEmbeddedShellFocused}
renderLine={renderLineWithHighlighting}
/>
packages/cli/src/ui/components/ModelDialog.test.tsx
+78
View File
@@ -565,6 +565,84 @@ describe('<ModelDialog />', () => {
expect(props.onClose).toHaveBeenCalledTimes(1);
});
it('stores the plain model id in voice model mode without switching models', async () => {
const switchModel = vi.fn();
const setFastModel = vi.fn();
const { props, mockSettings } = renderComponent(
{ isVoiceModelMode: true },
{
getAuthType: vi.fn(() => AuthType.USE_OPENAI),
getModel: vi.fn(() => 'qwen3.7-max'),
switchModel,
getAllConfiguredModels: vi.fn(() => [
{
id: 'qwen3-asr-flash',
label: 'qwen3-asr-flash',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
},
{
id: 'qwen3.7-max',
label: 'qwen3.7-max',
authType: AuthType.USE_OPENAI,
},
]),
getContentGeneratorConfig: vi.fn(() => ({
authType: AuthType.USE_OPENAI,
model: 'qwen3.7-max',
})),
setFastModel,
} as unknown as Partial<Config>,
);
const selectProps = mockedSelect.mock.calls[0][0];
await selectProps.onSelect(selectProps.items[0].value);
expect(mockSettings.setValue).toHaveBeenCalledWith(
SettingScope.User,
'voiceModel',
'qwen3-asr-flash',
);
expect(switchModel).not.toHaveBeenCalled();
expect(setFastModel).not.toHaveBeenCalled();
expect(mockSettings.setValue).not.toHaveBeenCalledWith(
SettingScope.User,
'model.name',
expect.any(String),
);
expect(props.onClose).toHaveBeenCalledTimes(1);
});
it('does not store a voice model without a transcription baseUrl', async () => {
const switchModel = vi.fn();
const { props, mockSettings } = renderComponent(
{ isVoiceModelMode: true },
{
getAuthType: vi.fn(() => AuthType.USE_OPENAI),
getModel: vi.fn(() => 'qwen3.7-max'),
switchModel,
getAllConfiguredModels: vi.fn(() => [
{
id: 'qwen3-coder',
label: 'qwen3-coder',
authType: AuthType.USE_OPENAI,
},
]),
getContentGeneratorConfig: vi.fn(() => ({
authType: AuthType.USE_OPENAI,
model: 'qwen3.7-max',
})),
} as unknown as Partial<Config>,
);
const childOnSelect = mockedSelect.mock.calls[0][0].onSelect;
await childOnSelect(`${AuthType.USE_OPENAI}::qwen3-coder`);
expect(mockSettings.setValue).not.toHaveBeenCalled();
expect(switchModel).not.toHaveBeenCalled();
expect(props.onClose).not.toHaveBeenCalled();
});
it('highlights the cross-auth row for a bare fast-model setting', () => {
// `/model --fast deepseek-v4-flash` validates across all providers and
// persists the bare model id. When the dialog re-opens, it must locate
packages/cli/src/ui/components/ModelDialog.tsx
+81 -12
View File
@@ -26,6 +26,10 @@ import { UIStateContext, type UIState } from '../contexts/UIStateContext.js';
import { useSettings } from '../contexts/SettingsContext.js';
import { getPersistScopeForModelSelection } from '../../config/modelProvidersScope.js';
import { t } from '../../i18n/index.js';
import {
formatUnsupportedVoiceModelMessage,
isSelectableVoiceModel,
} from '../voice/voice-model.js';
function formatModalities(modalities?: InputModalities): string {
if (!modalities) return t('text-only');
@@ -80,6 +84,7 @@ function parseModelSelectionKey(key: string): {
interface ModelDialogProps {
onClose: () => void;
isFastModelMode?: boolean;
isVoiceModelMode?: boolean;
}
function maskApiKey(apiKey: string | undefined): string {
@@ -202,6 +207,7 @@ function DetailRow({
export function ModelDialog({
onClose,
isFastModelMode,
isVoiceModelMode,
}: ModelDialogProps): React.JSX.Element {
const config = useContext(ConfigContext);
const uiState = useContext(UIStateContext);
@@ -343,6 +349,7 @@ export function ModelDialog({
// In fast model mode, default to the currently configured fast model
const fastModelSetting = settings?.merged?.fastModel as string | undefined;
const voiceModelSetting = settings?.merged?.voiceModel as string | undefined;
const parsedFastModelSetting = useMemo(() => {
if (!isFastModelMode) return undefined;
try {
@@ -357,9 +364,10 @@ export function ModelDialog({
: config?.getModel() || MAINLINE_CODER_MODEL;
// Check if current model is a runtime model
// Runtime snapshot ID is already in $runtime|${authType}|${modelId} format
const activeRuntimeSnapshot = isFastModelMode
? undefined // fast model is never a runtime model
: config?.getActiveRuntimeModelSnapshot?.();
const activeRuntimeSnapshot =
isFastModelMode || isVoiceModelMode
? undefined // fast and voice models are never runtime model selections
: config?.getActiveRuntimeModelSnapshot?.();
const currentBaseUrl = config
?.getModelsConfig()
.getGenerationConfig()?.baseUrl;
@@ -381,21 +389,36 @@ export function ModelDialog({
({ model }) => model.id === parsedFastModelSetting.modelId,
)
: undefined;
const preferredVoiceModelEntry =
isVoiceModelMode && voiceModelSetting
? availableModelEntries.find(
({ model }) => model.id === voiceModelSetting,
)
: undefined;
const preferredKey = activeRuntimeSnapshot
? activeRuntimeSnapshot.id
: preferredFastModelEntry
: preferredVoiceModelEntry
? buildModelSelectionKey(
preferredFastModelEntry.authType,
preferredFastModelEntry.model.id,
preferredFastModelEntry.model.baseUrl,
preferredVoiceModelEntry.authType,
preferredVoiceModelEntry.model.id,
preferredVoiceModelEntry.model.baseUrl,
)
: authType
? buildModelSelectionKey(authType, preferredModelId, currentBaseUrl)
: '';
: preferredFastModelEntry
? buildModelSelectionKey(
preferredFastModelEntry.authType,
preferredFastModelEntry.model.id,
preferredFastModelEntry.model.baseUrl,
)
: authType
? buildModelSelectionKey(authType, preferredModelId, currentBaseUrl)
: '';
useKeypress(
(key) => {
if (key.name === 'escape' || (key.name === 'left' && isFastModelMode)) {
if (
key.name === 'escape' ||
(key.name === 'left' && (isFastModelMode || isVoiceModelMode))
) {
onClose();
}
},
@@ -438,6 +461,45 @@ export function ModelDialog({
return value === selected;
},
);
if (isVoiceModelMode) {
if (!selectedEntry) {
setErrorMessage(t('Selected voice model is unavailable.'));
return;
}
const voiceModel = selectedEntry.model.id;
if (!isSelectableVoiceModel(selectedEntry.model)) {
setErrorMessage(formatUnsupportedVoiceModelMessage(voiceModel));
return;
}
const matchingEntries = availableModelEntries.filter(
({ model }) => model.id === voiceModel,
);
if (matchingEntries.length > 1) {
setErrorMessage(
t(
"Voice model '{{model}}' is configured more than once. Remove duplicate model ids before selecting it for voice transcription.",
{ model: voiceModel },
),
);
return;
}
const scope = getPersistScopeForModelSelection(settings);
settings.setValue(scope, 'voiceModel', voiceModel);
uiState?.historyManager.addItem(
{
type: 'success',
text: `${t('Voice Model')}: ${voiceModel}`,
},
Date.now(),
);
onClose();
return;
}
hydrateApiKeyEnvFromSettings(settings, selectedEntry?.model.envKey);
// Fast model mode: save authType:modelId so duplicate model ids across
@@ -583,6 +645,7 @@ export function ModelDialog({
uiState,
setErrorMessage,
isFastModelMode,
isVoiceModelMode,
availableModelEntries,
],
);
@@ -597,7 +660,13 @@ export function ModelDialog({
padding={1}
width="100%"
>
<Text bold>{t('Select Model')}</Text>
<Text bold>
{isVoiceModelMode
? t('Select Voice Model')
: isFastModelMode
? t('Select Fast Model')
: t('Select Model')}
</Text>
{!hasModels ? (
<Box marginTop={1} flexDirection="column">
packages/cli/src/ui/components/VoiceIndicator.test.tsx
+32
View File
@@ -0,0 +1,32 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, expect, it } from 'vitest';
import { render } from 'ink-testing-library';
import { VoiceIndicator } from './VoiceIndicator.js';
describe('<VoiceIndicator />', () => {
it('sanitizes streaming interim transcript text', () => {
const { lastFrame } = render(
<VoiceIndicator
status="recording"
interimText={'partial\x1b[8m hidden\x1b[0m'}
/>,
);
const frame = lastFrame() ?? '';
expect(frame).toContain('partial\\u001b[8m hidden\\u001b[0m');
expect(frame).not.toContain('\x1b[8m');
});
it('renders the meter when audioLevel is NaN', () => {
const { lastFrame } = render(
<VoiceIndicator status="recording" audioLevel={Number.NaN} />,
);
expect(lastFrame()).toContain('░'.repeat(16));
});
});
packages/cli/src/ui/components/VoiceIndicator.tsx
+61
View File
@@ -0,0 +1,61 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { Box, Text } from 'ink';
import type { VoiceInputStatus } from '../hooks/use-voice-input.js';
import { t } from '../../i18n/index.js';
import { escapeAnsiCtrlCodes } from '../utils/textUtils.js';
interface VoiceIndicatorProps {
status: VoiceInputStatus;
/** Live partial transcript (streaming only). */
interimText?: string;
/** Recent input level 0..1. */
audioLevel?: number;
}
const METER_WIDTH = 16;
// Speech mean-abs level is small (~0.03–0.1 of full scale); amplify for display.
const LEVEL_GAIN = 12;
function meter(level: number): string {
if (!Number.isFinite(level)) level = 0;
const norm = Math.max(0, Math.min(1, level * LEVEL_GAIN));
const filled = Math.round(norm * METER_WIDTH);
return '█'.repeat(filled) + '░'.repeat(METER_WIDTH - filled);
}
/** Live voice dictation indicator: state, input-level meter, and partial text. */
export function VoiceIndicator({
status,
interimText,
audioLevel = 0,
}: VoiceIndicatorProps): React.JSX.Element | null {
if (status === 'idle') {
return null;
}
return (
<Box flexDirection="column" paddingLeft={1}>
<Box>
{status === 'recording' ? (
<>
<Text color="redBright">{'● '}</Text>
<Text color="cyan">{meter(audioLevel)}</Text>
<Text color="gray">{' ' + t('listening…')}</Text>
</>
) : (
<Text color="yellow">{'✦ ' + t('transcribing…')}</Text>
)}
</Box>
{interimText ? (
<Text dimColor wrap="truncate-end">
{escapeAnsiCtrlCodes(interimText)}
</Text>
) : null}
</Box>
);
}
packages/cli/src/ui/contexts/UIActionsContext.tsx
+4 -1
View File
@@ -44,7 +44,10 @@ export interface UIActions {
notifyStatusLineSettingsChanged: (config: StatusLinePresetConfig) => void;
closeMemoryDialog: () => void;
closeModelDialog: () => void;
openModelDialog: (options?: { fastModelMode?: boolean }) => void;
openModelDialog: (options?: {
fastModelMode?: boolean;
voiceModelMode?: boolean;
}) => void;
openArenaDialog: (type: Exclude<ArenaDialogType, null>) => void;
closeArenaDialog: () => void;
handleArenaModelsSelected?: (models: string[]) => void;
packages/cli/src/ui/contexts/UIStateContext.tsx
+1
View File
@@ -59,6 +59,7 @@ export interface UIState {
isMemoryDialogOpen: boolean;
isModelDialogOpen: boolean;
isFastModelMode: boolean;
isVoiceModelMode: boolean;
isTrustDialogOpen: boolean;
activeArenaDialog: ArenaDialogType;
isPermissionsDialogOpen: boolean;
packages/cli/src/ui/hooks/slashCommandProcessor.test.ts
+19
View File
@@ -565,6 +565,25 @@ describe('useSlashCommandProcessor', () => {
expect(mockOpenModelDialog).toHaveBeenCalled();
});
it('should handle "dialog: voice-model" action', async () => {
const command = createTestCommand({
name: 'voicemodelcmd',
action: vi
.fn()
.mockResolvedValue({ type: 'dialog', dialog: 'voice-model' }),
});
const result = setupProcessorHook([command]);
await waitFor(() => expect(result.current.slashCommands).toHaveLength(1));
await act(async () => {
await result.current.handleSlashCommand('/voicemodelcmd');
});
expect(mockOpenModelDialog).toHaveBeenCalledWith({
voiceModelMode: true,
});
});
it('awaits direct resume session switching before returning handled', async () => {
const actions = createMockActions();
let resolveResume: (() => void) | undefined;
packages/cli/src/ui/hooks/slashCommandProcessor.ts
+7 -1
View File
@@ -108,7 +108,10 @@ export interface SlashCommandProcessorActions {
openMemoryDialog: () => void;
openSettingsDialog: () => void;
openStatusLineDialog: () => void;
openModelDialog: (options?: { fastModelMode?: boolean }) => void;
openModelDialog: (options?: {
fastModelMode?: boolean;
voiceModelMode?: boolean;
}) => void;
openTrustDialog: () => void;
openPermissionsDialog: () => void;
openApprovalModeDialog: () => void;
@@ -790,6 +793,9 @@ export const useSlashCommandProcessor = (
case 'fast-model':
actions.openModelDialog({ fastModelMode: true });
return { type: 'handled' };
case 'voice-model':
actions.openModelDialog({ voiceModelMode: true });
return { type: 'handled' };
case 'trust':
actions.openTrustDialog();
return { type: 'handled' };
packages/cli/src/ui/hooks/use-voice-input.test.ts
+836
View File
@@ -0,0 +1,836 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { renderHook, act, waitFor } from '@testing-library/react';
import { describe, it, expect, vi } from 'vitest';
import { useVoiceInput } from './use-voice-input.js';
import type { Key } from './useKeypress.js';
function deferred<T>() {
let resolve!: (value: T | PromiseLike<T>) => void;
let reject!: (reason?: unknown) => void;
const promise = new Promise<T>((res, rej) => {
resolve = res;
reject = rej;
});
return { promise, resolve, reject };
}
const voiceKey: Key = {
name: 'space',
sequence: ' ',
ctrl: false,
meta: false,
shift: false,
paste: false,
};
const escapeKey: Key = {
name: 'escape',
sequence: '\x1b',
ctrl: false,
meta: false,
shift: false,
paste: false,
};
function createBuffer(text = '') {
const testBuffer = {
text,
insert: vi.fn((value: string) => {
testBuffer.text += value;
}),
};
return testBuffer;
}
let buffer = createBuffer();
describe('use-voice-input', () => {
it('warms up the backend when voice is enabled', () => {
buffer = createBuffer();
const warmup = vi.fn();
renderHook(() =>
useVoiceInput({
enabled: true,
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: vi.fn(),
transcribe: vi.fn(),
warmup,
}),
);
expect(warmup).toHaveBeenCalled();
});
it('does not warm up the backend when voice is disabled', () => {
buffer = createBuffer();
const warmup = vi.fn();
renderHook(() =>
useVoiceInput({
enabled: false,
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: vi.fn(),
transcribe: vi.fn(),
warmup,
}),
);
expect(warmup).not.toHaveBeenCalled();
});
it('does not intercept Space when voice input is disabled', () => {
buffer = createBuffer();
const { result } = renderHook(() =>
useVoiceInput({
enabled: false,
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: vi.fn(),
transcribe: vi.fn(),
}),
);
expect(result.current.handleKeypress(voiceKey)).toBe(false);
});
it('tap mode: records on first Space, transcribes and submits on second', async () => {
buffer = createBuffer('explain');
const stop = vi.fn().mockResolvedValue({
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
});
const recorder = { start: vi.fn().mockResolvedValue(undefined), stop };
const createRecorder = vi.fn(() => recorder);
const transcribe = vi.fn().mockResolvedValue('the diff');
const onSubmit = vi.fn();
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder,
transcribe,
onSubmit,
}),
);
await act(async () => {
expect(result.current.handleKeypress(voiceKey)).toBe(true);
});
expect(createRecorder).toHaveBeenCalledTimes(1);
// Tap mode arms silence auto-stop.
expect(recorder.start).toHaveBeenCalledWith({
silenceDetection: true,
onAutoStop: expect.any(Function),
});
expect(result.current.status).toBe('recording');
await act(async () => {
expect(result.current.handleKeypress(voiceKey)).toBe(true);
});
await waitFor(() => {
expect(transcribe).toHaveBeenCalledWith(
{
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
},
{ voiceModel: 'qwen3-asr-flash' },
);
expect(buffer.insert).toHaveBeenCalledWith(' the diff');
expect(onSubmit).toHaveBeenCalledTimes(1);
expect(result.current.status).toBe('idle');
});
});
it('sanitizes ASR transcripts before inserting and submitting', async () => {
buffer = createBuffer();
const recorder = {
start: vi.fn().mockResolvedValue(undefined),
stop: vi.fn().mockResolvedValue({
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
}),
};
const transcribe = vi
.fn()
.mockResolvedValue('send this\x1b[8m hidden\x1b[0m');
const onSubmit = vi.fn();
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: () => recorder,
transcribe,
onSubmit,
}),
);
await act(async () => {
result.current.handleKeypress(voiceKey);
result.current.handleKeypress(voiceKey);
});
await waitFor(() => {
expect(buffer.insert).toHaveBeenCalledWith(
'send this\\u001b[8m hidden\\u001b[0m',
);
expect(onSubmit).toHaveBeenCalledWith(
'send this\\u001b[8m hidden\\u001b[0m',
);
});
});
it('hold mode: starts the recorder without silence detection', async () => {
buffer = createBuffer();
const recorder = {
start: vi.fn().mockResolvedValue(undefined),
stop: vi.fn().mockResolvedValue({
data: new Uint8Array([1]),
mimeType: 'audio/wav',
}),
};
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'hold',
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: () => recorder,
transcribe: vi.fn().mockResolvedValue('hold text'),
onSubmit: vi.fn(),
}),
);
await act(async () => {
result.current.handleKeypress(voiceKey);
});
// Hold mode never arms silence auto-stop; release is driven by key repeats.
expect(recorder.start).toHaveBeenCalledWith({
silenceDetection: false,
onAutoStop: expect.any(Function),
});
expect(result.current.status).toBe('recording');
});
it('updates audio level while recording with a batch model', async () => {
vi.useFakeTimers();
try {
buffer = createBuffer();
const recorder = {
start: vi.fn().mockResolvedValue(undefined),
stop: vi.fn().mockResolvedValue({
data: new Uint8Array([1]),
mimeType: 'audio/wav',
}),
audioLevel: vi.fn().mockReturnValue(0.42),
};
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'hold',
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: () => recorder,
transcribe: vi.fn().mockResolvedValue('hold text'),
}),
);
await act(async () => {
result.current.handleKeypress(voiceKey);
await Promise.resolve();
});
await act(async () => {
await vi.advanceTimersByTimeAsync(100);
});
expect(recorder.audioLevel).toHaveBeenCalled();
expect(result.current.audioLevel).toBe(0.42);
} finally {
vi.useRealTimers();
}
});
it('hold mode: finalizes after key repeat stops', async () => {
vi.useFakeTimers();
try {
buffer = createBuffer();
const recorder = {
start: vi.fn().mockResolvedValue(undefined),
stop: vi.fn().mockResolvedValue({
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
}),
};
const transcribe = vi.fn().mockResolvedValue('hold text');
const onSubmit = vi.fn();
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'hold',
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: () => recorder,
transcribe,
onSubmit,
}),
);
await act(async () => {
expect(result.current.handleKeypress(voiceKey)).toBe(true);
});
expect(result.current.status).toBe('recording');
await act(async () => {
await vi.advanceTimersByTimeAsync(400);
expect(result.current.handleKeypress(voiceKey)).toBe(true);
});
await act(async () => {
await vi.advanceTimersByTimeAsync(249);
});
expect(recorder.stop).not.toHaveBeenCalled();
await act(async () => {
await vi.advanceTimersByTimeAsync(1);
});
await act(async () => {
await Promise.resolve();
await Promise.resolve();
});
expect(recorder.stop).toHaveBeenCalledTimes(1);
expect(transcribe).toHaveBeenCalledWith(
{
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
},
{ voiceModel: 'qwen3-asr-flash' },
);
expect(buffer.insert).toHaveBeenCalledWith('hold text');
expect(onSubmit).not.toHaveBeenCalled();
expect(result.current.status).toBe('idle');
} finally {
vi.useRealTimers();
}
});
it('surfaces recorder errors without inserting text', async () => {
buffer = createBuffer();
const addItem = vi.fn();
const recorder = {
start: vi.fn().mockResolvedValue(undefined),
stop: vi.fn().mockRejectedValue(new Error('microphone denied')),
};
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash',
buffer,
addItem,
createRecorder: () => recorder,
transcribe: vi.fn(),
}),
);
await act(async () => {
result.current.handleKeypress(voiceKey);
result.current.handleKeypress(voiceKey);
});
await waitFor(() => {
expect(addItem).toHaveBeenCalledWith(
{
type: 'error',
text: 'Voice transcription failed: microphone denied',
},
expect.any(Number),
);
expect(buffer.insert).not.toHaveBeenCalled();
expect(result.current.status).toBe('idle');
});
});
it('does not restart recording during the start error cooldown', async () => {
buffer = createBuffer();
const addItem = vi.fn();
const createRecorder = vi.fn(() => ({
start: vi.fn().mockRejectedValue(new Error('missing recorder')),
stop: vi.fn(),
}));
const now = vi.spyOn(Date, 'now').mockReturnValue(1000);
try {
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
voiceModel: 'qwen3-asr-flash',
buffer,
addItem,
createRecorder,
transcribe: vi.fn(),
}),
);
await act(async () => {
expect(result.current.handleKeypress(voiceKey)).toBe(true);
});
await waitFor(() => {
expect(addItem).toHaveBeenCalledTimes(1);
});
expect(createRecorder).toHaveBeenCalledTimes(1);
now.mockReturnValue(1500);
act(() => {
expect(result.current.handleKeypress(voiceKey)).toBe(true);
});
expect(createRecorder).toHaveBeenCalledTimes(1);
now.mockReturnValue(3100);
act(() => {
expect(result.current.handleKeypress(voiceKey)).toBe(true);
});
expect(createRecorder).toHaveBeenCalledTimes(2);
} finally {
now.mockRestore();
}
});
it('waits for recorder startup before stopping on a quick second Space', async () => {
buffer = createBuffer();
const start = deferred<void>();
const stop = vi.fn().mockResolvedValue({
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
});
const recorder = { start: vi.fn(() => start.promise), stop };
const transcribe = vi.fn().mockResolvedValue('hello');
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: () => recorder,
transcribe,
}),
);
act(() => {
result.current.handleKeypress(voiceKey);
result.current.handleKeypress(voiceKey);
});
expect(stop).not.toHaveBeenCalled();
await act(async () => {
start.resolve();
await start.promise;
});
await waitFor(() => {
expect(stop).toHaveBeenCalledTimes(1);
expect(transcribe).toHaveBeenCalled();
expect(result.current.status).toBe('idle');
});
});
it('waits for a pending stream session before finalizing quick streaming input', async () => {
buffer = createBuffer();
const start = deferred<void>();
const streamOpen = deferred<{
pushAudio: (pcm: Uint8Array) => void;
finish: () => Promise<string>;
abort: () => void;
}>();
const stop = vi
.fn()
.mockRejectedValue(
new Error('Native audio capture produced empty audio.'),
);
const drain = vi
.fn()
.mockReturnValueOnce(new Uint8Array([1, 2, 3]))
.mockReturnValue(new Uint8Array(0));
const recorder = { start: vi.fn(() => start.promise), stop, drain };
const streamSession = {
pushAudio: vi.fn(),
finish: vi.fn().mockResolvedValue('streamed text'),
abort: vi.fn(),
};
const transcribe = vi.fn();
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash-realtime',
buffer,
createRecorder: () => recorder,
transcribe,
streaming: true,
openStream: vi.fn(() => streamOpen.promise),
}),
);
act(() => {
result.current.handleKeypress(voiceKey);
result.current.handleKeypress(voiceKey);
});
await act(async () => {
start.resolve();
await start.promise;
});
expect(transcribe).not.toHaveBeenCalled();
expect(streamSession.finish).not.toHaveBeenCalled();
await act(async () => {
streamOpen.resolve(streamSession);
await streamOpen.promise;
});
await waitFor(() => {
expect(streamSession.pushAudio).toHaveBeenCalledWith(
new Uint8Array([1, 2, 3]),
);
expect(streamSession.finish).toHaveBeenCalledTimes(1);
expect(transcribe).not.toHaveBeenCalled();
expect(buffer.insert).toHaveBeenCalledWith('streamed text');
expect(result.current.status).toBe('idle');
});
});
it('stops the recorder when streaming requires an unsupported backend', async () => {
buffer = createBuffer();
const addItem = vi.fn();
const stop = vi.fn().mockResolvedValue({
data: new Uint8Array([1]),
mimeType: 'audio/wav',
});
const recorder = {
start: vi.fn().mockResolvedValue(undefined),
stop,
supportsStreaming: vi.fn(() => false),
};
const openStream = vi.fn();
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash-realtime',
buffer,
addItem,
createRecorder: () => recorder,
transcribe: vi.fn(),
streaming: true,
openStream,
}),
);
await act(async () => {
result.current.handleKeypress(voiceKey);
});
await waitFor(() => {
expect(stop).toHaveBeenCalledTimes(1);
expect(openStream).not.toHaveBeenCalled();
expect(addItem).toHaveBeenCalledWith(
{
type: 'error',
text: 'Voice transcription failed: Streaming voice transcription requires native audio capture. Install/rebuild @qwen-code/audio-capture or switch voiceModel to qwen3-asr-flash for batch transcription.',
},
expect.any(Number),
);
expect(result.current.status).toBe('idle');
});
});
it('reports streaming pump errors without throwing from the timer', async () => {
vi.useFakeTimers();
try {
buffer = createBuffer();
const addItem = vi.fn();
const stop = vi.fn().mockResolvedValue({
data: new Uint8Array([1]),
mimeType: 'audio/wav',
});
const recorder = {
start: vi.fn().mockResolvedValue(undefined),
stop,
supportsStreaming: vi.fn(() => true),
drain: vi.fn(() => {
throw new Error('native drain failed');
}),
};
const streamSession = {
pushAudio: vi.fn(),
finish: vi.fn().mockResolvedValue('streamed text'),
abort: vi.fn(),
};
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash-realtime',
buffer,
addItem,
createRecorder: () => recorder,
transcribe: vi.fn(),
streaming: true,
openStream: vi.fn().mockResolvedValue(streamSession),
}),
);
await act(async () => {
result.current.handleKeypress(voiceKey);
});
await act(async () => {
await Promise.resolve();
await vi.advanceTimersByTimeAsync(100);
});
expect(streamSession.abort).toHaveBeenCalledTimes(1);
expect(stop).toHaveBeenCalledTimes(1);
expect(addItem).toHaveBeenCalledWith(
{
type: 'error',
text: 'Voice transcription failed: native drain failed',
},
expect.any(Number),
);
expect(result.current.status).toBe('idle');
} finally {
vi.useRealTimers();
}
});
it('stops an active recorder when unmounted without transcribing', async () => {
buffer = createBuffer();
const stop = vi.fn().mockResolvedValue({
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
});
const recorder = { start: vi.fn().mockResolvedValue(undefined), stop };
const transcribe = vi.fn();
const { result, unmount } = renderHook(() =>
useVoiceInput({
enabled: true,
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: () => recorder,
transcribe,
}),
);
act(() => {
result.current.handleKeypress(voiceKey);
});
unmount();
await waitFor(() => {
expect(stop).toHaveBeenCalledTimes(1);
});
expect(transcribe).not.toHaveBeenCalled();
});
it('treats an empty-audio capture as a silent no-op (no error item)', async () => {
buffer = createBuffer();
const addItem = vi.fn();
const recorder = {
start: vi.fn().mockResolvedValue(undefined),
stop: vi
.fn()
.mockRejectedValue(
new Error('Native audio capture produced empty audio.'),
),
};
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash',
buffer,
addItem,
createRecorder: () => recorder,
transcribe: vi.fn(),
}),
);
await act(async () => {
result.current.handleKeypress(voiceKey);
result.current.handleKeypress(voiceKey);
});
await waitFor(() => {
expect(result.current.status).toBe('idle');
});
expect(addItem).not.toHaveBeenCalled();
expect(buffer.insert).not.toHaveBeenCalled();
});
it('cancels transcription and ignores a late transcript', async () => {
buffer = createBuffer();
const transcript = deferred<string>();
const recorder = {
start: vi.fn().mockResolvedValue(undefined),
stop: vi.fn().mockResolvedValue({
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
}),
};
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: () => recorder,
transcribe: vi.fn(() => transcript.promise),
}),
);
await act(async () => {
result.current.handleKeypress(voiceKey);
result.current.handleKeypress(voiceKey);
});
await waitFor(() => {
expect(result.current.status).toBe('transcribing');
});
act(() => {
expect(result.current.handleKeypress(escapeKey)).toBe(true);
});
expect(result.current.status).toBe('idle');
await act(async () => {
transcript.resolve('late text');
await transcript.promise;
});
expect(buffer.insert).not.toHaveBeenCalled();
});
it('ignores stale transcription when a singleton recorder starts a new session', async () => {
buffer = createBuffer();
const transcript = deferred<string>();
const recorder = {
start: vi.fn().mockResolvedValue(undefined),
stop: vi.fn().mockResolvedValue({
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
}),
};
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder: () => recorder,
transcribe: vi.fn(() => transcript.promise),
}),
);
await act(async () => {
result.current.handleKeypress(voiceKey);
result.current.handleKeypress(voiceKey);
});
await waitFor(() => {
expect(result.current.status).toBe('transcribing');
});
act(() => {
result.current.handleKeypress(escapeKey);
});
await act(async () => {
result.current.handleKeypress(voiceKey);
await Promise.resolve();
});
expect(result.current.status).toBe('recording');
await act(async () => {
transcript.resolve('stale text');
await transcript.promise;
});
expect(buffer.insert).not.toHaveBeenCalled();
expect(result.current.status).toBe('recording');
});
it('does not wait forever for a previous stop before starting again', async () => {
vi.useFakeTimers();
try {
buffer = createBuffer();
const firstRecorder = {
start: vi.fn().mockResolvedValue(undefined),
stop: vi.fn(() => new Promise<never>(() => {})),
};
const secondRecorder = {
start: vi.fn().mockResolvedValue(undefined),
stop: vi.fn(),
};
const createRecorder = vi
.fn()
.mockReturnValueOnce(firstRecorder)
.mockReturnValueOnce(secondRecorder);
const { result } = renderHook(() =>
useVoiceInput({
enabled: true,
mode: 'tap',
voiceModel: 'qwen3-asr-flash',
buffer,
createRecorder,
transcribe: vi.fn(),
}),
);
await act(async () => {
result.current.handleKeypress(voiceKey);
});
act(() => {
result.current.handleKeypress(escapeKey);
});
act(() => {
result.current.handleKeypress(voiceKey);
});
expect(secondRecorder.start).not.toHaveBeenCalled();
await act(async () => {
await vi.advanceTimersByTimeAsync(5000);
});
expect(secondRecorder.start).toHaveBeenCalledTimes(1);
} finally {
vi.useRealTimers();
}
});
});
packages/cli/src/ui/hooks/use-voice-input.ts
+628
View File
@@ -0,0 +1,628 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { useCallback, useEffect, useRef, useState } from 'react';
import { createDebugLogger } from '@qwen-code/qwen-code-core';
import type { TextBuffer } from '../components/shared/text-buffer.js';
import { Command, keyMatchers } from '../keyMatchers.js';
import type { HistoryItemWithoutId } from '../types.js';
import { escapeAnsiCtrlCodes } from '../utils/textUtils.js';
import type { VoiceStreamSession } from '../voice/voice-stream-session.js';
import type { Key } from './useKeypress.js';
export interface RecordedVoiceAudio {
data: Uint8Array;
mimeType: string;
}
export interface VoiceRecorderStartOptions {
/** Enable amplitude-based auto-stop after sustained silence (tap mode). */
silenceDetection?: boolean;
/** Invoked if the recorder stops itself (silence detected) before stop(). */
onAutoStop?: () => void;
}
export type MicrophonePermission = 'granted' | 'denied' | 'prompt' | 'unknown';
export interface VoiceRecorder {
start: (options?: VoiceRecorderStartOptions) => Promise<void> | void;
stop: () => Promise<RecordedVoiceAudio>;
/** Optional: pre-load the backend so the first start() isn't cold. */
warmup?: () => void | Promise<void>;
/** Optional: query OS microphone permission (macOS TCC). */
microphoneStatus?: () => Promise<MicrophonePermission>;
/** Optional (streaming): return & clear PCM captured since the last call. */
drain?: () => Uint8Array;
/** Optional: whether this recorder can provide streaming PCM chunks. */
supportsStreaming?: () => boolean;
/** Optional: recent input level 0..1 for the waveform. */
audioLevel?: () => number;
}
export type VoiceTranscriber = (
audio: RecordedVoiceAudio,
context: { voiceModel: string },
) => Promise<string>;
export type VoiceInputStatus = 'idle' | 'recording' | 'transcribing';
/** hold = hold-to-talk (release stops, dictation only). tap = tap to start, tap/silence to stop+submit. */
export type VoiceInputMode = 'hold' | 'tap';
const VOICE_ERROR_RETRY_DELAY_MS = 2000;
const PREVIOUS_STOP_WAIT_TIMEOUT_MS = 5000;
// Terminals emit no key-up event. In hold mode we infer release from the gap
// between auto-repeat keypresses: arm a longer timer on first press (covers the
// OS initial-repeat delay), then a short timer on each repeat. When repeats stop
// (key released) the timer fires and we finalize. Requires terminal key repeat.
// Kept above typical OS initial key-repeat delays so a held key isn't cut off
// before its first auto-repeat arrives (which would truncate the utterance).
const HOLD_FIRST_PRESS_RELEASE_MS = 800;
const HOLD_REPEAT_RELEASE_MS = 250;
const debugLogger = createDebugLogger('VOICE_INPUT');
interface UseVoiceInputArgs {
enabled: boolean;
mode?: VoiceInputMode;
voiceModel?: string;
buffer: Pick<TextBuffer, 'text' | 'insert'>;
addItem?: (item: HistoryItemWithoutId, timestamp: number) => void;
createRecorder: () => VoiceRecorder;
transcribe: VoiceTranscriber;
/**
* Called after a tap-mode transcript is inserted, to submit the prompt.
* Receives the resulting prompt text — buffer.insert dispatches async, so the
* submit handler must not read it back from buffer.text synchronously.
*/
onSubmit?: (text: string) => void;
/** Pre-load the recorder backend when voice turns on (avoids cold-start race). */
warmup?: () => void | Promise<void>;
/** Enable live streaming transcription (requires openStream + a drain-capable recorder). */
streaming?: boolean;
/** Open a streaming session; the hook pumps drained PCM into it while recording. */
openStream?: (callbacks: {
onInterim: (text: string) => void;
onError?: (error: Error) => void;
}) => Promise<VoiceStreamSession>;
}
interface UseVoiceInputReturn {
status: VoiceInputStatus;
/** Live partial transcript during streaming (empty otherwise). */
interimText: string;
/** Recent input level 0..1 during recording (for a waveform). */
audioLevel: number;
handleKeypress: (key: Key) => boolean;
}
function formatError(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
function waitForPreviousStop(stopPromise: Promise<void> | null): Promise<void> {
if (!stopPromise) {
return Promise.resolve();
}
return new Promise<void>((resolve) => {
const timer = setTimeout(resolve, PREVIOUS_STOP_WAIT_TIMEOUT_MS);
void stopPromise
.catch(() => undefined)
.then(() => {
clearTimeout(timer);
resolve();
});
});
}
function insertTranscript(
buffer: Pick<TextBuffer, 'text' | 'insert'>,
transcript: string,
): string | null {
const text = escapeAnsiCtrlCodes(transcript).trim();
if (!text) {
return null;
}
const needsSpace = buffer.text.length > 0 && !/\s$/.test(buffer.text);
const segment = needsSpace ? ` ${text}` : text;
// buffer.text is the pre-insert (rendered) value and buffer.insert dispatches
// async, so compute the resulting prompt text now for callers that submit
// immediately (tap mode) instead of reading the stale buffer.text back.
const resulting = buffer.text + segment;
buffer.insert(segment);
return resulting;
}
function isCancelKey(key: Key): boolean {
return key.name === 'escape' || (key.ctrl && key.name === 'c');
}
function logRecorderStopError(error: unknown): void {
debugLogger.warn('[voice] recorder stop failed:', error);
}
export function useVoiceInput({
enabled,
mode = 'hold',
voiceModel,
buffer,
addItem,
createRecorder,
transcribe,
onSubmit,
warmup,
streaming,
openStream,
}: UseVoiceInputArgs): UseVoiceInputReturn {
const [status, setStatus] = useState<VoiceInputStatus>('idle');
const [interimText, setInterimText] = useState('');
const [audioLevel, setAudioLevelState] = useState(0);
const statusRef = useRef<VoiceInputStatus>('idle');
const recorderRef = useRef<VoiceRecorder | null>(null);
const startPromiseRef = useRef<Promise<void> | null>(null);
const stopPromiseRef = useRef<Promise<void> | null>(null);
const retryAfterErrorAtRef = useRef(0);
const mountedRef = useRef(true);
const cancelRecordingRef = useRef<() => void>(() => {});
const releaseTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
const finalizeRef = useRef<(submit: boolean) => void>(() => {});
const streamSessionRef = useRef<VoiceStreamSession | null>(null);
const streamSessionPromiseRef = useRef<Promise<VoiceStreamSession> | null>(
null,
);
const pumpTimerRef = useRef<ReturnType<typeof setInterval> | null>(null);
const sessionIdRef = useRef(0);
const isStreaming = streaming === true && typeof openStream === 'function';
const clearPump = useCallback(() => {
if (pumpTimerRef.current) {
clearInterval(pumpTimerRef.current);
pumpTimerRef.current = null;
}
}, []);
const resetStreamUi = useCallback(() => {
if (mountedRef.current) {
setInterimText('');
setAudioLevelState(0);
}
}, []);
const setVoiceStatus = useCallback((next: VoiceInputStatus) => {
statusRef.current = next;
if (mountedRef.current) {
setStatus(next);
}
}, []);
const clearReleaseTimer = useCallback(() => {
if (releaseTimerRef.current) {
clearTimeout(releaseTimerRef.current);
releaseTimerRef.current = null;
}
}, []);
const reportError = useCallback(
(error: unknown) => {
retryAfterErrorAtRef.current = Date.now() + VOICE_ERROR_RETRY_DELAY_MS;
addItem?.(
{
type: 'error',
text: `Voice transcription failed: ${formatError(error)}`,
},
Date.now(),
);
},
[addItem],
);
const stopRecorderQuietly = useCallback(async (recorder: VoiceRecorder) => {
try {
await recorder.stop();
} catch (error) {
logRecorderStopError(error);
}
}, []);
const isCurrentSession = useCallback(
(recorder: VoiceRecorder, sessionId: number) =>
recorderRef.current === recorder && sessionIdRef.current === sessionId,
[],
);
const handleStreamError = useCallback(
(recorder: VoiceRecorder, sessionId: number, error: Error) => {
if (!isCurrentSession(recorder, sessionId)) {
return;
}
clearReleaseTimer();
clearPump();
streamSessionRef.current?.abort();
streamSessionRef.current = null;
streamSessionPromiseRef.current = null;
recorderRef.current = null;
startPromiseRef.current = null;
void stopRecorderQuietly(recorder);
setVoiceStatus('idle');
resetStreamUi();
reportError(error);
},
[
clearPump,
clearReleaseTimer,
isCurrentSession,
reportError,
resetStreamUi,
setVoiceStatus,
stopRecorderQuietly,
],
);
const updateAudioLevel = useCallback(
(recorder: VoiceRecorder, sessionId: number) => {
if (!isCurrentSession(recorder, sessionId)) return;
const level = recorder.audioLevel?.();
if (typeof level === 'number' && mountedRef.current) {
setAudioLevelState(level);
}
},
[isCurrentSession],
);
const startRecording = useCallback(
(silenceDetection: boolean) => {
const recorder = createRecorder();
const sessionId = ++sessionIdRef.current;
recorderRef.current = recorder;
setVoiceStatus('recording');
if (mountedRef.current) {
setInterimText('');
setAudioLevelState(0);
}
const startPromise = waitForPreviousStop(stopPromiseRef.current)
.then(() =>
Promise.resolve(
recorder.start({
silenceDetection,
onAutoStop: () => finalizeRef.current(true),
}),
),
)
.then(async () => {
if (!isStreaming) {
if (statusRef.current !== 'recording') {
return;
}
pumpTimerRef.current = setInterval(() => {
updateAudioLevel(recorder, sessionId);
}, 100);
return;
}
// Streaming: open the WS session and pump drained PCM into it while
// recording, surfacing partial transcripts live.
if (!isCurrentSession(recorder, sessionId)) {
return;
}
if (recorder.supportsStreaming?.() === false) {
throw new Error(
'Streaming voice transcription requires native audio capture. Install/rebuild @qwen-code/audio-capture or switch voiceModel to qwen3-asr-flash for batch transcription.',
);
}
const streamPromise = openStream!({
onInterim: (text) => {
if (mountedRef.current && isCurrentSession(recorder, sessionId)) {
setInterimText(text);
}
},
onError: (error) => handleStreamError(recorder, sessionId, error),
});
streamSessionPromiseRef.current = streamPromise;
const session = await streamPromise;
if (!isCurrentSession(recorder, sessionId)) {
session.abort();
void stopRecorderQuietly(recorder);
return;
}
streamSessionRef.current = session;
if (statusRef.current !== 'recording') {
return;
}
pumpTimerRef.current = setInterval(() => {
try {
if (!isCurrentSession(recorder, sessionId)) return;
const active = recorderRef.current;
if (!active) return;
const pcm = active.drain?.();
if (pcm && pcm.length > 0) session.pushAudio(pcm);
updateAudioLevel(active, sessionId);
} catch (error) {
clearPump();
clearReleaseTimer();
const pendingStreamPromise = streamSessionRef.current
? null
: streamSessionPromiseRef.current;
streamSessionRef.current?.abort();
streamSessionRef.current = null;
streamSessionPromiseRef.current = null;
void pendingStreamPromise
?.then((pendingSession) => pendingSession.abort())
.catch(() => {});
const active = recorderRef.current;
recorderRef.current = null;
startPromiseRef.current = null;
if (active) {
void stopRecorderQuietly(active);
}
setVoiceStatus('idle');
resetStreamUi();
reportError(error);
}
}, 100);
});
startPromiseRef.current = startPromise;
void startPromise.catch((error: unknown) => {
if (
isCurrentSession(recorder, sessionId) &&
statusRef.current === 'recording'
) {
recorderRef.current = null;
startPromiseRef.current = null;
clearReleaseTimer();
clearPump();
void stopRecorderQuietly(recorder);
streamSessionRef.current?.abort();
streamSessionRef.current = null;
void streamSessionPromiseRef.current
?.then((session) => session.abort())
.catch(() => {});
streamSessionPromiseRef.current = null;
setVoiceStatus('idle');
resetStreamUi();
if (!(error instanceof Error && /empty audio/i.test(error.message))) {
reportError(error);
}
}
});
},
[
clearPump,
clearReleaseTimer,
createRecorder,
handleStreamError,
isCurrentSession,
isStreaming,
openStream,
reportError,
resetStreamUi,
setVoiceStatus,
stopRecorderQuietly,
updateAudioLevel,
],
);
// Stop the active recorder, transcribe, and insert. In tap mode (submit) the
// prompt is auto-submitted; in hold mode the transcript is inserted only.
const finalize = useCallback(
(submit: boolean) => {
const recorder = recorderRef.current;
// Single-shot guard: a tap-stop and the native silence auto-stop can both
// fire. setVoiceStatus below flips statusRef synchronously, so the second
// concurrent call sees a non-'recording' status here and bails — avoiding
// a double recorder.stop() and the spurious "transcription failed" error.
if (!recorder || !voiceModel || statusRef.current !== 'recording') {
return;
}
const sessionId = sessionIdRef.current;
clearReleaseTimer();
clearPump();
const startPromise = startPromiseRef.current ?? Promise.resolve();
const wasStreaming = isStreaming;
setVoiceStatus('transcribing');
void startPromise
.then(async () => {
const session =
streamSessionRef.current ??
(wasStreaming ? await streamSessionPromiseRef.current : null);
if (session) {
// Push any remaining audio, tear down the device, then flush the
// stream and await the final transcript.
try {
const pcm = recorder.drain?.();
if (pcm && pcm.length > 0) session.pushAudio(pcm);
await stopRecorderQuietly(recorder);
} catch (error) {
// drain()/pushAudio can throw (native crash); abort so the
// WebSocket isn't left open until the server idle-timeout.
session.abort();
throw error;
}
return session.finish();
}
const audio = await recorder.stop();
return transcribe(audio, { voiceModel });
})
.then((transcript) => {
if (!mountedRef.current || !isCurrentSession(recorder, sessionId)) {
return;
}
const inserted = insertTranscript(buffer, transcript);
if (submit && inserted !== null) {
// Pass the resulting prompt text explicitly — buffer.text hasn't
// re-rendered yet after the async insert.
onSubmit?.(inserted);
}
})
.catch((error: unknown) => {
if (!isCurrentSession(recorder, sessionId)) {
return;
}
if (wasStreaming) {
void stopRecorderQuietly(recorder);
}
// A too-short/empty capture (quick tap, or cold-start race) isn't a
// real failure — silently return to idle instead of a scary error.
if (error instanceof Error && /empty audio/i.test(error.message)) {
return;
}
reportError(error);
})
.finally(() => {
if (!isCurrentSession(recorder, sessionId)) return;
recorderRef.current = null;
streamSessionRef.current = null;
streamSessionPromiseRef.current = null;
startPromiseRef.current = null;
setVoiceStatus('idle');
resetStreamUi();
});
},
[
buffer,
clearPump,
clearReleaseTimer,
isCurrentSession,
stopRecorderQuietly,
isStreaming,
onSubmit,
reportError,
resetStreamUi,
setVoiceStatus,
transcribe,
voiceModel,
],
);
finalizeRef.current = finalize;
const cancelRecording = useCallback(() => {
sessionIdRef.current += 1;
clearReleaseTimer();
clearPump();
const session = streamSessionRef.current;
streamSessionRef.current = null;
session?.abort();
void streamSessionPromiseRef.current
?.then((pendingSession) => pendingSession.abort())
.catch(() => {});
streamSessionPromiseRef.current = null;
resetStreamUi();
const recorder = recorderRef.current;
setVoiceStatus('idle');
if (!recorder) {
return;
}
const startPromise = startPromiseRef.current ?? Promise.resolve();
recorderRef.current = null;
startPromiseRef.current = null;
const stopPromise = startPromise
.then(async () => {
await recorder.stop();
})
.catch(logRecorderStopError)
.finally(() => {
if (recorderRef.current === null) {
setVoiceStatus('idle');
}
});
stopPromiseRef.current = stopPromise;
void stopPromise.finally(() => {
if (stopPromiseRef.current === stopPromise) {
stopPromiseRef.current = null;
}
});
}, [clearPump, clearReleaseTimer, resetStreamUi, setVoiceStatus]);
cancelRecordingRef.current = cancelRecording;
const armReleaseTimer = useCallback(
(ms: number) => {
clearReleaseTimer();
releaseTimerRef.current = setTimeout(() => {
releaseTimerRef.current = null;
finalizeRef.current(false);
}, ms);
},
[clearReleaseTimer],
);
// Preload the recorder backend when voice turns on, so the first keypress
// isn't delayed by a cold native-module load (which would otherwise race the
// hold-release timer and capture nothing).
useEffect(() => {
if (enabled && voiceModel) {
void Promise.resolve(warmup?.()).catch(() => {});
}
}, [enabled, voiceModel, warmup]);
useEffect(() => {
if (enabled && voiceModel) {
return;
}
cancelRecording();
}, [cancelRecording, enabled, voiceModel]);
useEffect(() => {
// Reset on (re)mount so StrictMode's mount→unmount→remount (active when
// DEBUG is set) doesn't leave mountedRef stuck false, which would no-op
// every gated setState and freeze the voice UI.
mountedRef.current = true;
return () => {
mountedRef.current = false;
cancelRecordingRef.current();
};
}, []);
const handleKeypress = useCallback(
(key: Key): boolean => {
if (
!enabled ||
!voiceModel ||
!keyMatchers[Command.VOICE_PUSH_TO_TALK](key)
) {
if (statusRef.current !== 'idle' && isCancelKey(key)) {
cancelRecording();
return true;
}
return false;
}
if (statusRef.current === 'idle') {
if (Date.now() < retryAfterErrorAtRef.current) {
return true;
}
if (mode === 'hold') {
startRecording(false);
armReleaseTimer(HOLD_FIRST_PRESS_RELEASE_MS);
} else {
startRecording(true);
}
return true;
}
if (statusRef.current === 'recording') {
if (mode === 'hold') {
// Auto-repeat keypress while held: keep alive until repeats stop.
armReleaseTimer(HOLD_REPEAT_RELEASE_MS);
} else {
finalize(true);
}
return true;
}
return true;
},
[
armReleaseTimer,
cancelRecording,
enabled,
finalize,
mode,
startRecording,
voiceModel,
],
);
return { status, interimText, audioLevel, handleKeypress };
}
packages/cli/src/ui/hooks/useModelCommand.test.ts
+15 -1
View File
@@ -24,19 +24,33 @@ describe('useModelCommand', () => {
expect(result.current.isModelDialogOpen).toBe(true);
});
it('should open the model dialog in voice model mode', () => {
const { result } = renderHook(() => useModelCommand());
act(() => {
result.current.openModelDialog({ voiceModelMode: true });
});
expect(result.current.isModelDialogOpen).toBe(true);
expect(result.current.isVoiceModelMode).toBe(true);
expect(result.current.isFastModelMode).toBe(false);
});
it('should close the model dialog when closeModelDialog is called', () => {
const { result } = renderHook(() => useModelCommand());
// Open it first
act(() => {
result.current.openModelDialog();
result.current.openModelDialog({ voiceModelMode: true });
});
expect(result.current.isModelDialogOpen).toBe(true);
expect(result.current.isVoiceModelMode).toBe(true);
// Then close it
act(() => {
result.current.closeModelDialog();
});
expect(result.current.isModelDialogOpen).toBe(false);
expect(result.current.isVoiceModelMode).toBe(false);
});
});
packages/cli/src/ui/hooks/useModelCommand.ts
+14 -3
View File
@@ -9,17 +9,26 @@ import { useState, useCallback } from 'react';
interface UseModelCommandReturn {
isModelDialogOpen: boolean;
isFastModelMode: boolean;
openModelDialog: (options?: { fastModelMode?: boolean }) => void;
isVoiceModelMode: boolean;
openModelDialog: (options?: {
fastModelMode?: boolean;
voiceModelMode?: boolean;
}) => void;
closeModelDialog: () => void;
}
export const useModelCommand = (): UseModelCommandReturn => {
const [isModelDialogOpen, setIsModelDialogOpen] = useState(false);
const [isFastModelMode, setIsFastModelMode] = useState(false);
const [isVoiceModelMode, setIsVoiceModelMode] = useState(false);
const openModelDialog = useCallback(
(options?: { fastModelMode?: boolean }) => {
setIsFastModelMode(options?.fastModelMode ?? false);
(options?: { fastModelMode?: boolean; voiceModelMode?: boolean }) => {
const voiceModelMode = options?.voiceModelMode ?? false;
setIsFastModelMode(
voiceModelMode ? false : (options?.fastModelMode ?? false),
);
setIsVoiceModelMode(voiceModelMode);
setIsModelDialogOpen(true);
},
[],
@@ -28,11 +37,13 @@ export const useModelCommand = (): UseModelCommandReturn => {
const closeModelDialog = useCallback(() => {
setIsModelDialogOpen(false);
setIsFastModelMode(false);
setIsVoiceModelMode(false);
}, []);
return {
isModelDialogOpen,
isFastModelMode,
isVoiceModelMode,
openModelDialog,
closeModelDialog,
};
packages/cli/src/ui/keyMatchers.test.ts
+2
View File
@@ -50,6 +50,8 @@ describe('keyMatchers', () => {
key.name === 'return' && !key.ctrl && !key.meta && !key.paste,
[Command.NEWLINE]: (key: Key) =>
key.name === 'return' && (key.ctrl || key.meta || key.paste),
[Command.VOICE_PUSH_TO_TALK]: (key: Key) =>
key.name === 'space' && !key.ctrl && !key.meta,
[Command.OPEN_EXTERNAL_EDITOR]: (key: Key) =>
key.ctrl && (key.name === 'x' || key.sequence === '\x18'),
[Command.PASTE_CLIPBOARD_IMAGE]: (key: Key) =>
packages/cli/src/ui/voice/arecord-recorder.ts
+163
View File
@@ -0,0 +1,163 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { spawn, type ChildProcess } from 'node:child_process';
import { mkdtemp, readFile, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import type {
RecordedVoiceAudio,
VoiceRecorder,
VoiceRecorderStartOptions,
} from '../hooks/use-voice-input.js';
// arecord has no built-in silence trigger, so silenceDetection is ignored —
// recording runs until stop(). On WSL/headless Linux the binary exists but
// open() fails because there is no ALSA card; we probe by spawning and racing
// a short timer: still alive after the grace window => the device opened.
const DEVICE_OPEN_GRACE_MS = 200;
function toArecordError(error: Error): Error {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') {
return new Error(
'arecord is not installed or not on PATH. Install alsa-utils and try again.',
);
}
return error;
}
class ArecordRecorder implements VoiceRecorder {
private child: ChildProcess | null = null;
private tmpDir: string | null = null;
private filePath: string | null = null;
private closeResult: {
code: number | null;
signal: NodeJS.Signals | null;
} | null = null;
private closePromise: Promise<{
code: number | null;
signal: NodeJS.Signals | null;
}> | null = null;
private async cleanup(): Promise<void> {
const tmpDir = this.tmpDir;
if (tmpDir) {
await rm(tmpDir, { recursive: true, force: true });
}
this.child = null;
this.tmpDir = null;
this.filePath = null;
this.closeResult = null;
this.closePromise = null;
}
async start(_options: VoiceRecorderStartOptions = {}): Promise<void> {
this.tmpDir = await mkdtemp(path.join(tmpdir(), 'qwen-voice-'));
this.filePath = path.join(this.tmpDir, 'recording.wav');
const child = spawn('arecord', [
'-q',
'-f',
'S16_LE',
'-r',
'16000',
'-c',
'1',
'-t',
'wav',
this.filePath,
]);
this.child = child;
let stderr = '';
child.stderr?.on('data', (chunk: Buffer) => {
stderr += chunk.toString();
});
this.closePromise = new Promise((resolve) => {
child.once('close', (code, signal) => {
this.closeResult = { code, signal };
resolve(this.closeResult);
});
});
try {
await new Promise<void>((resolve, reject) => {
let settled = false;
const grace = setTimeout(() => {
// Process is still alive after the grace window: it opened the device.
settled = true;
resolve();
}, DEVICE_OPEN_GRACE_MS);
child.once('error', (error: Error) => {
if (settled) return;
settled = true;
clearTimeout(grace);
reject(toArecordError(error));
});
child.once('close', (code) => {
if (settled) return;
settled = true;
clearTimeout(grace);
const detail = stderr.trim();
reject(
new Error(
`arecord could not open an audio device (exit code ${code ?? 'unknown'})${
detail ? `: ${detail}` : ''
}.`,
),
);
});
});
} catch (error) {
await this.cleanup();
throw error;
}
}
async stop(): Promise<RecordedVoiceAudio> {
if (!this.child || !this.tmpDir || !this.filePath) {
throw new Error('Voice recorder was not started.');
}
const child = this.child;
const filePath = this.filePath;
const closePromise = this.closePromise;
try {
if (!this.closeResult) {
child.kill('SIGINT');
}
const closeResult =
this.closeResult ?? (closePromise ? await closePromise : null);
if (
closeResult &&
closeResult.code !== 0 &&
closeResult.signal !== 'SIGINT'
) {
throw new Error(
`Voice recorder failed with exit code ${closeResult.code ?? 'unknown'}.`,
);
}
const data = await readFile(filePath);
if (data.byteLength === 0) {
throw new Error('Voice recorder produced empty audio.');
}
return {
data,
mimeType: 'audio/wav',
};
} finally {
await this.cleanup();
}
}
}
export function createArecordRecorder(): VoiceRecorder {
return new ArecordRecorder();
}
packages/cli/src/ui/voice/native-audio-recorder.test.ts
+160
View File
@@ -0,0 +1,160 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, expect, it, vi } from 'vitest';
import { createNativeAudioRecorder } from './native-audio-recorder.js';
describe('createNativeAudioRecorder', () => {
it('records mono 16k audio through the native capture backend', async () => {
const backend = {
startRecording: vi.fn(),
stopRecording: vi.fn(() => new Uint8Array([1, 2, 3])),
isRecording: vi.fn(() => false),
microphoneAuthorizationStatus: vi.fn(() => 'unknown' as const),
};
const recorder = createNativeAudioRecorder({
loadBackend: () => backend,
});
await recorder.start();
const audio = await recorder.stop();
expect(backend.startRecording).toHaveBeenCalledWith({
sampleRate: 16000,
channels: 1,
silenceDetection: false,
});
expect(backend.stopRecording).toHaveBeenCalledTimes(1);
expect(audio).toEqual({
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
});
});
it('polls the silence flag and auto-stops when silence is detected', async () => {
vi.useFakeTimers();
try {
let silent = false;
const backend = {
startRecording: vi.fn(),
stopRecording: vi.fn(() => new Uint8Array([1])),
isRecording: vi.fn(() => true),
silenceDetected: vi.fn(() => silent),
microphoneAuthorizationStatus: vi.fn(() => 'unknown' as const),
};
const onAutoStop = vi.fn();
const recorder = createNativeAudioRecorder({
loadBackend: () => backend,
});
await recorder.start({ silenceDetection: true, onAutoStop });
expect(backend.startRecording).toHaveBeenCalledWith({
sampleRate: 16000,
channels: 1,
silenceDetection: true,
});
await vi.advanceTimersByTimeAsync(400);
expect(onAutoStop).not.toHaveBeenCalled();
silent = true;
await vi.advanceTimersByTimeAsync(400);
expect(onAutoStop).toHaveBeenCalledTimes(1);
} finally {
vi.useRealTimers();
}
});
it('auto-stops when native silence polling fails', async () => {
vi.useFakeTimers();
try {
const backend = {
startRecording: vi.fn(),
stopRecording: vi.fn(() => new Uint8Array([1])),
isRecording: vi.fn(() => true),
silenceDetected: vi.fn(() => {
throw new Error('silence poll failed');
}),
microphoneAuthorizationStatus: vi.fn(() => 'unknown' as const),
};
const onAutoStop = vi.fn();
const recorder = createNativeAudioRecorder({
loadBackend: () => backend,
});
await recorder.start({ silenceDetection: true, onAutoStop });
await vi.advanceTimersByTimeAsync(200);
expect(onAutoStop).toHaveBeenCalledTimes(1);
} finally {
vi.useRealTimers();
}
});
it('rejects a second start while already recording', async () => {
const backend = {
startRecording: vi.fn(),
stopRecording: vi.fn(() => new Uint8Array([1])),
isRecording: vi.fn(() => true),
microphoneAuthorizationStatus: vi.fn(() => 'unknown' as const),
};
const recorder = createNativeAudioRecorder({
loadBackend: () => backend,
});
await recorder.start();
await expect(recorder.start()).rejects.toThrow(/already recording/);
expect(backend.startRecording).toHaveBeenCalledTimes(1);
});
it('rejects concurrent starts while the backend is loading', async () => {
const backend = {
startRecording: vi.fn(),
stopRecording: vi.fn(() => new Uint8Array([1])),
isRecording: vi.fn(() => true),
microphoneAuthorizationStatus: vi.fn(() => 'unknown' as const),
};
type Backend = typeof backend;
let resolveBackend: (backend: Backend) => void = () => {};
const loadBackend = vi.fn(
() =>
new Promise<Backend>((resolve) => {
resolveBackend = resolve;
}),
);
const recorder = createNativeAudioRecorder({ loadBackend });
const firstStart = recorder.start();
await expect(recorder.start()).rejects.toThrow(/already recording/);
resolveBackend(backend);
await firstStart;
expect(loadBackend).toHaveBeenCalledTimes(1);
expect(backend.startRecording).toHaveBeenCalledTimes(1);
});
it('allows retry after backend loading fails', async () => {
const backend = {
startRecording: vi.fn(),
stopRecording: vi.fn(() => new Uint8Array([1])),
isRecording: vi.fn(() => true),
microphoneAuthorizationStatus: vi.fn(() => 'unknown' as const),
};
const loadBackend = vi
.fn()
.mockRejectedValueOnce(new Error('load failed'))
.mockResolvedValueOnce(backend);
const recorder = createNativeAudioRecorder({ loadBackend });
await expect(recorder.start()).rejects.toThrow('load failed');
await recorder.start();
expect(loadBackend).toHaveBeenCalledTimes(2);
expect(backend.startRecording).toHaveBeenCalledTimes(1);
});
});
packages/cli/src/ui/voice/native-audio-recorder.ts
+134
View File
@@ -0,0 +1,134 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import type { NativeAudioCaptureBackend } from '@qwen-code/audio-capture';
import { createDebugLogger } from '@qwen-code/qwen-code-core';
import type {
RecordedVoiceAudio,
VoiceRecorder,
VoiceRecorderStartOptions,
} from '../hooks/use-voice-input.js';
// Native silence detection sets a flag we poll for; older addons lack it.
const SILENCE_POLL_INTERVAL_MS = 200;
const debugLogger = createDebugLogger('VOICE_NATIVE_RECORDER');
interface NativeAudioRecorderOptions {
loadBackend?: () =>
| NativeAudioCaptureBackend
| Promise<NativeAudioCaptureBackend>;
}
class NativeAudioRecorder implements VoiceRecorder {
private backend: NativeAudioCaptureBackend | null = null;
private silencePoll: ReturnType<typeof setInterval> | null = null;
private starting = false;
constructor(
private readonly loadBackend: () =>
| NativeAudioCaptureBackend
| Promise<NativeAudioCaptureBackend>,
) {}
private clearSilencePoll(): void {
if (this.silencePoll) {
clearInterval(this.silencePoll);
this.silencePoll = null;
}
}
// Pay the dynamic-import + dlopen cost up front so the first start() is warm.
async warmup(): Promise<void> {
await this.loadBackend();
}
async microphoneStatus(): Promise<
'granted' | 'denied' | 'prompt' | 'unknown'
> {
const backend = await this.loadBackend();
return backend.microphoneAuthorizationStatus();
}
drain(): Uint8Array {
return this.backend?.drainAudio?.() ?? new Uint8Array(0);
}
supportsStreaming(): boolean {
return typeof this.backend?.drainAudio === 'function';
}
audioLevel(): number {
return this.backend?.audioLevel?.() ?? 0;
}
async start(options: VoiceRecorderStartOptions = {}): Promise<void> {
if (this.backend || this.starting) {
throw new Error('Native voice recorder is already recording.');
}
this.starting = true;
let backend: NativeAudioCaptureBackend;
try {
backend = await this.loadBackend();
const silenceDetection = options.silenceDetection === true;
backend.startRecording({
sampleRate: 16000,
channels: 1,
silenceDetection,
});
this.backend = backend;
this.starting = false;
const { onAutoStop } = options;
if (silenceDetection && onAutoStop && backend.silenceDetected) {
this.silencePoll = setInterval(() => {
try {
if (this.backend?.silenceDetected?.()) {
this.clearSilencePoll();
onAutoStop();
}
} catch (error) {
this.clearSilencePoll();
debugLogger.warn(
'[voice] silence detection failed, auto-stopping:',
error,
);
onAutoStop();
}
}, SILENCE_POLL_INTERVAL_MS);
}
} catch (error) {
this.starting = false;
throw error;
}
}
async stop(): Promise<RecordedVoiceAudio> {
this.clearSilencePoll();
if (!this.backend) {
throw new Error('Native voice recorder was not started.');
}
const data = this.backend.stopRecording();
this.backend = null;
return {
data,
mimeType: 'audio/wav',
};
}
}
async function loadDefaultBackend(): Promise<NativeAudioCaptureBackend> {
const { createNativeAudioCaptureBackend } = await import(
'@qwen-code/audio-capture'
);
return createNativeAudioCaptureBackend();
}
export function createNativeAudioRecorder(
options: NativeAudioRecorderOptions = {},
): VoiceRecorder {
return new NativeAudioRecorder(options.loadBackend ?? loadDefaultBackend);
}
packages/cli/src/ui/voice/qwen-asr-realtime-session.test.ts
+459
View File
@@ -0,0 +1,459 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { afterEach, describe, expect, it, vi } from 'vitest';
import {
deriveQwenRealtimeUrl,
openQwenAsrRealtimeStream,
} from './qwen-asr-realtime-session.js';
class FakeSocket {
readonly OPEN = 1;
readyState = this.OPEN;
bufferedAmount = 0;
readonly sent: Array<string | Uint8Array> = [];
private readonly handlers = new Map<
string,
Array<(...args: unknown[]) => void>
>();
send(data: string | Uint8Array): void {
this.sent.push(data);
}
close(): void {
this.readyState = 3;
}
on(event: string, cb: (...args: unknown[]) => void): void {
const handlers = this.handlers.get(event) ?? [];
handlers.push(cb);
this.handlers.set(event, handlers);
}
emit(event: string, ...args: unknown[]): void {
for (const handler of this.handlers.get(event) ?? []) {
handler(...args);
}
}
}
function parseSent(socket: FakeSocket, index: number): Record<string, unknown> {
return JSON.parse(String(socket.sent[index]));
}
describe('qwen-asr-realtime-session', () => {
afterEach(() => {
vi.useRealTimers();
});
it('derives the Qwen realtime endpoint from the provider host', () => {
expect(
deriveQwenRealtimeUrl(
'https://dashscope.aliyuncs.com/compatible-mode/v1',
'qwen3-asr-flash-realtime',
),
).toBe(
'wss://dashscope.aliyuncs.com/api-ws/v1/realtime?model=qwen3-asr-flash-realtime',
);
expect(
deriveQwenRealtimeUrl(
'http://localhost:8080/dashscope/v1',
'qwen3-asr-flash-realtime',
),
).toBe(
'ws://localhost:8080/dashscope/api-ws/v1/realtime?model=qwen3-asr-flash-realtime',
);
});
it('streams PCM chunks as base64 events and resolves the completed transcript', async () => {
const socket = new FakeSocket();
const createWebSocket = vi.fn(() => socket);
const interim = vi.fn();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
apiKey: 'sk-test',
model: 'qwen3-asr-flash-realtime',
language: 'zh',
keytermsContext: 'grep regex OAuth',
},
{ onInterim: interim },
{ createWebSocket },
);
expect(createWebSocket).toHaveBeenCalledWith(
'wss://dashscope.example/api-ws/v1/realtime?model=qwen3-asr-flash-realtime',
{ headers: { Authorization: 'Bearer sk-test' } },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.created', event_id: 'created' }),
false,
);
expect(parseSent(socket, 0)).toMatchObject({
type: 'session.update',
session: {
input_audio_format: 'pcm',
sample_rate: 16000,
input_audio_transcription: {
language: 'zh',
corpus_text: 'grep regex OAuth',
},
turn_detection: null,
},
});
socket.emit(
'message',
JSON.stringify({ type: 'session.updated', event_id: 'updated' }),
false,
);
const session = await sessionPromise;
session.pushAudio(new Uint8Array([1, 2, 3]));
expect(parseSent(socket, 1)).toMatchObject({
type: 'input_audio_buffer.append',
audio: 'AQID',
});
socket.emit(
'message',
JSON.stringify({
type: 'conversation.item.input_audio_transcription.text',
text: 'hello',
stash: ' world',
}),
false,
);
expect(interim).toHaveBeenCalledWith('hello world');
const transcriptPromise = session.finish();
expect(parseSent(socket, 2)).toMatchObject({
type: 'input_audio_buffer.commit',
});
expect(parseSent(socket, 3)).toMatchObject({
type: 'session.finish',
});
socket.emit(
'message',
JSON.stringify({
type: 'conversation.item.input_audio_transcription.completed',
transcript: 'hello world',
}),
false,
);
socket.emit(
'message',
JSON.stringify({ type: 'session.finished', event_id: 'finished' }),
false,
);
await expect(transcriptPromise).resolves.toBe('hello world');
});
it('drops realtime audio chunks when the socket buffer is backed up', async () => {
const socket = new FakeSocket();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'qwen3-asr-flash-realtime',
},
{},
{ createWebSocket: () => socket },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.updated', event_id: 'updated' }),
false,
);
const session = await sessionPromise;
socket.bufferedAmount = 1024 * 1024 + 1;
session.pushAudio(new Uint8Array([1, 2, 3]));
expect(socket.sent).toEqual([]);
});
it('resumes sending after realtime socket backpressure clears', async () => {
const socket = new FakeSocket();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'qwen3-asr-flash-realtime',
},
{},
{ createWebSocket: () => socket },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.updated', event_id: 'updated' }),
false,
);
const session = await sessionPromise;
socket.bufferedAmount = 1024 * 1024 + 1;
session.pushAudio(new Uint8Array([1, 2, 3]));
socket.bufferedAmount = 0;
session.pushAudio(new Uint8Array([4, 5, 6]));
expect(socket.sent).toHaveLength(1);
expect(parseSent(socket, 0)).toMatchObject({
type: 'input_audio_buffer.append',
audio: 'BAUG',
});
});
it('rejects when the server finishes before the session is ready', async () => {
const socket = new FakeSocket();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'qwen3-asr-flash-realtime',
},
{},
{ createWebSocket: () => socket },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.finished', event_id: 'finished' }),
false,
);
await expect(sessionPromise).rejects.toThrow(
'Qwen ASR realtime session finished before it was ready.',
);
expect(socket.readyState).toBe(3);
});
it('rejects finish when the server never sends session.finished', async () => {
vi.useFakeTimers();
const socket = new FakeSocket();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'qwen3-asr-flash-realtime',
},
{},
{ createWebSocket: () => socket },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.updated', event_id: 'updated' }),
false,
);
const session = await sessionPromise;
const transcriptPromise = session.finish();
void transcriptPromise.catch(() => undefined);
await vi.advanceTimersByTimeAsync(60_000);
await expect(transcriptPromise).rejects.toThrow(
'Qwen ASR realtime finish timed out.',
);
expect(socket.readyState).toBe(3);
});
it('resolves finish when the server already finished the session', async () => {
const socket = new FakeSocket();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'qwen3-asr-flash-realtime',
},
{},
{ createWebSocket: () => socket },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.updated', event_id: 'updated' }),
false,
);
const session = await sessionPromise;
socket.emit(
'message',
JSON.stringify({
type: 'conversation.item.input_audio_transcription.completed',
transcript: 'server finished first',
}),
false,
);
socket.emit(
'message',
JSON.stringify({ type: 'session.finished', event_id: 'finished' }),
false,
);
await expect(session.finish()).resolves.toBe('server finished first');
});
it('preserves transcription failures that arrive before finish', async () => {
const socket = new FakeSocket();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'qwen3-asr-flash-realtime',
},
{},
{ createWebSocket: () => socket },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.updated', event_id: 'updated' }),
false,
);
const session = await sessionPromise;
socket.emit(
'message',
JSON.stringify({
type: 'conversation.item.input_audio_transcription.failed',
error: { message: 'server failed' },
}),
false,
);
await expect(session.finish()).rejects.toThrow('server failed');
});
it('notifies transcription failures that arrive while recording', async () => {
const socket = new FakeSocket();
const onError = vi.fn();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'qwen3-asr-flash-realtime',
},
{ onError },
{ createWebSocket: () => socket },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.updated', event_id: 'updated' }),
false,
);
const session = await sessionPromise;
socket.emit(
'message',
JSON.stringify({
type: 'conversation.item.input_audio_transcription.failed',
error: { message: 'server failed while recording' },
}),
false,
);
expect(onError).toHaveBeenCalledWith(
new Error('server failed while recording'),
);
expect(socket.readyState).toBe(3);
await expect(session.finish()).rejects.toThrow(
'server failed while recording',
);
});
it('notifies and rejects finish when the realtime socket closed before finish', async () => {
const socket = new FakeSocket();
const onError = vi.fn();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'qwen3-asr-flash-realtime',
},
{ onError },
{ createWebSocket: () => socket },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.updated', event_id: 'updated' }),
false,
);
const session = await sessionPromise;
socket.emit('close');
expect(onError).toHaveBeenCalledWith(
new Error(
'Qwen ASR realtime connection closed unexpectedly. Transcript may be incomplete.',
),
);
await expect(session.finish()).rejects.toThrow(
'Qwen ASR realtime connection closed unexpectedly.',
);
});
it('rejects committed transcript when the socket closes before session.finished', async () => {
const socket = new FakeSocket();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'qwen3-asr-flash-realtime',
},
{},
{ createWebSocket: () => socket },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.updated', event_id: 'updated' }),
false,
);
const session = await sessionPromise;
socket.emit(
'message',
JSON.stringify({
type: 'conversation.item.input_audio_transcription.completed',
transcript: 'hello world',
}),
false,
);
const transcriptPromise = session.finish();
socket.emit('close');
await expect(transcriptPromise).rejects.toThrow(
'Qwen ASR realtime connection closed unexpectedly.',
);
});
it('sanitizes and caps realtime server error messages', async () => {
const socket = new FakeSocket();
const sessionPromise = openQwenAsrRealtimeStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'qwen3-asr-flash-realtime',
},
{},
{ createWebSocket: () => socket },
);
socket.emit(
'message',
JSON.stringify({ type: 'session.updated', event_id: 'updated' }),
false,
);
const session = await sessionPromise;
const transcriptPromise = session.finish();
const longMessage = `bad\x1b[8mhidden\x1b[0m ${'x'.repeat(300)}`;
socket.emit(
'message',
JSON.stringify({
type: 'error',
error: { message: longMessage },
}),
false,
);
await expect(transcriptPromise).rejects.toThrow(
'bad\\u001b[8mhidden\\u001b[0m',
);
await expect(transcriptPromise).rejects.not.toThrow('x'.repeat(220));
});
});
packages/cli/src/ui/voice/qwen-asr-realtime-session.ts
+313
View File
@@ -0,0 +1,313 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { randomUUID } from 'node:crypto';
import { createDebugLogger } from '@qwen-code/qwen-code-core';
import WebSocket from 'ws';
import type {
SocketLike,
VoiceStreamCallbacks,
VoiceStreamConfig,
VoiceStreamSession,
} from './voice-stream-session.js';
import { deriveWebSocketBase } from './voice-stream-session.js';
import { escapeAnsiCtrlCodes } from '../utils/textUtils.js';
export interface QwenRealtimeDeps {
createWebSocket?: (
url: string,
options: { headers: Record<string, string> },
) => SocketLike;
}
const CONNECT_TIMEOUT_MS = 8000;
const FINISH_TIMEOUT_MS = 60_000;
const MAX_BUFFERED_AUDIO_BYTES = 1024 * 1024;
const MAX_SERVER_ERROR_MESSAGE_LENGTH = 200;
const debugLogger = createDebugLogger('VOICE_QWEN_REALTIME');
export function deriveQwenRealtimeUrl(baseUrl: string, model: string): string {
return `${deriveWebSocketBase(baseUrl)}/api-ws/v1/realtime?model=${encodeURIComponent(model)}`;
}
function appendTranscript(existing: string, next: string): string {
const text = next.trim();
if (!text) return existing;
return existing ? `${existing} ${text}` : text;
}
function toError(error: unknown): Error {
return error instanceof Error ? error : new Error(String(error));
}
function formatServerErrorMessage(raw: unknown): string {
const text = typeof raw === 'string' ? raw : 'Qwen ASR realtime failed.';
return escapeAnsiCtrlCodes(text).slice(0, MAX_SERVER_ERROR_MESSAGE_LENGTH);
}
export function openQwenAsrRealtimeStream(
config: VoiceStreamConfig,
callbacks: VoiceStreamCallbacks = {},
deps: QwenRealtimeDeps = {},
): Promise<VoiceStreamSession> {
const createWebSocket =
deps.createWebSocket ??
((url, options) =>
new WebSocket(url, {
headers: options.headers,
}) as unknown as SocketLike);
return new Promise<VoiceStreamSession>((resolve, reject) => {
const ws = createWebSocket(
deriveQwenRealtimeUrl(config.baseUrl, config.model),
{
headers: config.apiKey
? { Authorization: `Bearer ${config.apiKey}` }
: {},
},
);
let opened = false;
let openSettled = false;
let committed = '';
let finishPromise: Promise<string> | null = null;
let finishResolve: ((text: string) => void) | null = null;
let finishReject: ((error: unknown) => void) | null = null;
let finishTimer: ReturnType<typeof setTimeout> | null = null;
let connectTimer: ReturnType<typeof setTimeout> | null = null;
let finishedTranscript: string | null = null;
let terminalError: Error | null = null;
let failed = false;
let backpressureWarned = false;
const sendJson = (body: Record<string, unknown>) => {
ws.send(JSON.stringify({ event_id: randomUUID(), ...body }));
};
const close = () => {
try {
ws.close();
} catch {
/* ignore */
}
};
const clearFinishTimer = () => {
if (finishTimer) {
clearTimeout(finishTimer);
finishTimer = null;
}
};
const clearConnectTimer = () => {
if (connectTimer) {
clearTimeout(connectTimer);
connectTimer = null;
}
};
const fail = (error: unknown) => {
if (failed) return;
failed = true;
const normalized = toError(error);
clearConnectTimer();
clearFinishTimer();
close();
if (finishReject) {
finishReject(normalized);
finishResolve = null;
finishReject = null;
return;
}
if (!openSettled) {
openSettled = true;
reject(normalized);
return;
}
terminalError = normalized;
callbacks.onError?.(normalized);
};
connectTimer = setTimeout(() => {
if (!opened) fail(new Error('Qwen ASR realtime connection timed out.'));
}, CONNECT_TIMEOUT_MS);
const sendSessionUpdate = () => {
sendJson({
type: 'session.update',
session: {
input_audio_format: 'pcm',
sample_rate: 16000,
input_audio_transcription: {
...(config.language ? { language: config.language } : {}),
...(config.keytermsContext
? { corpus_text: config.keytermsContext }
: {}),
},
turn_detection: null,
},
});
};
ws.on('message', (...args: unknown[]) => {
const data = args[0];
const isBinary = args[1] === true;
if (isBinary) return;
let msg: {
type?: string;
text?: unknown;
stash?: unknown;
transcript?: unknown;
error?: { code?: string; message?: string; param?: string };
};
try {
msg = JSON.parse(String(data));
} catch (error) {
debugLogger.warn(
'[voice] failed to parse Qwen ASR realtime message:',
error,
);
return;
}
switch (msg.type) {
case 'session.created':
sendSessionUpdate();
break;
case 'session.updated':
opened = true;
openSettled = true;
clearConnectTimer();
resolve({
pushAudio: (pcm) => {
if (ws.readyState !== ws.OPEN || pcm.length === 0) return;
if ((ws.bufferedAmount ?? 0) > MAX_BUFFERED_AUDIO_BYTES) {
if (!backpressureWarned) {
backpressureWarned = true;
debugLogger.warn(
'[voice] dropping Qwen ASR realtime audio due to socket backpressure.',
);
}
return;
}
backpressureWarned = false;
sendJson({
type: 'input_audio_buffer.append',
audio: Buffer.from(pcm).toString('base64'),
});
},
finish: () => {
if (finishPromise) return finishPromise;
finishPromise = new Promise<string>((res, rej) => {
if (finishedTranscript !== null) {
res(finishedTranscript);
return;
}
if (terminalError) {
rej(terminalError);
return;
}
finishResolve = res;
finishReject = rej;
finishTimer = setTimeout(() => {
fail(new Error('Qwen ASR realtime finish timed out.'));
}, FINISH_TIMEOUT_MS);
try {
sendJson({ type: 'input_audio_buffer.commit' });
sendJson({ type: 'session.finish' });
} catch (error) {
fail(error);
}
});
return finishPromise;
},
abort: close,
});
break;
case 'conversation.item.input_audio_transcription.text': {
const text = typeof msg.text === 'string' ? msg.text : '';
const stash = typeof msg.stash === 'string' ? msg.stash : '';
const preview = `${text}${stash}`.trim();
callbacks.onInterim?.([committed, preview].filter(Boolean).join(' '));
break;
}
case 'conversation.item.input_audio_transcription.completed':
if (typeof msg.transcript === 'string') {
committed = appendTranscript(committed, msg.transcript);
callbacks.onInterim?.(committed);
}
break;
case 'conversation.item.input_audio_transcription.failed':
fail(
new Error(
formatServerErrorMessage(
msg.error?.message ??
msg.error?.code ??
'Qwen ASR realtime transcription failed.',
),
),
);
break;
case 'session.finished':
if (!openSettled) {
fail(
new Error(
'Qwen ASR realtime session finished before it was ready.',
),
);
break;
}
failed = true;
clearFinishTimer();
finishedTranscript = committed.trim();
finishResolve?.(finishedTranscript);
finishResolve = null;
finishReject = null;
close();
break;
case 'error':
fail(
new Error(
formatServerErrorMessage(
msg.error?.message ??
msg.error?.code ??
'Qwen ASR realtime request failed.',
),
),
);
break;
default:
break;
}
});
ws.on('error', fail);
ws.on('close', () => {
clearConnectTimer();
clearFinishTimer();
if (failed) return;
if (!openSettled) {
openSettled = true;
reject(new Error('Qwen ASR realtime connection closed.'));
return;
}
if (finishReject) {
finishReject(
new Error(
'Qwen ASR realtime connection closed unexpectedly. Transcript may be incomplete.',
),
);
finishResolve = null;
finishReject = null;
} else {
const error = new Error(
'Qwen ASR realtime connection closed unexpectedly. Transcript may be incomplete.',
);
terminalError ??= error;
callbacks.onError?.(terminalError);
}
});
});
}
packages/cli/src/ui/voice/sox-recorder.test.ts
+169
View File
@@ -0,0 +1,169 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { EventEmitter } from 'node:events';
import path from 'node:path';
import { describe, it, expect, vi, beforeEach } from 'vitest';
const mocks = vi.hoisted(() => ({
spawn: vi.fn(),
mkdtemp: vi.fn(),
readFile: vi.fn(),
rm: vi.fn(),
}));
vi.mock('node:child_process', () => ({
default: { spawn: mocks.spawn },
spawn: mocks.spawn,
}));
vi.mock('node:fs/promises', () => ({
default: {
mkdtemp: mocks.mkdtemp,
readFile: mocks.readFile,
rm: mocks.rm,
},
mkdtemp: mocks.mkdtemp,
readFile: mocks.readFile,
rm: mocks.rm,
}));
import { createSoxRecorder } from './sox-recorder.js';
class FakeChildProcess extends EventEmitter {
stderr = new EventEmitter();
kill = vi.fn();
}
async function startRecorder(recorder: ReturnType<typeof createSoxRecorder>) {
const startPromise = Promise.resolve(recorder.start());
await Promise.resolve();
const child = mocks.spawn.mock.results.at(-1)?.value as FakeChildProcess;
child.emit('spawn');
await startPromise;
return child;
}
describe('createSoxRecorder', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('records mono 16k wav audio with sox and returns the file bytes', async () => {
const child = new FakeChildProcess();
mocks.spawn.mockReturnValue(child);
mocks.mkdtemp.mockResolvedValue('/tmp/qwen-voice-abc');
mocks.readFile.mockResolvedValue(Buffer.from([1, 2, 3]));
const recorder = createSoxRecorder();
await startRecorder(recorder);
expect(mocks.spawn).toHaveBeenCalledWith('sox', [
'-d',
'-r',
'16000',
'-c',
'1',
'-b',
'16',
path.join('/tmp/qwen-voice-abc', 'recording.wav'),
]);
const stopPromise = recorder.stop();
expect(child.kill).toHaveBeenCalledWith('SIGINT');
child.emit('close', 0);
await expect(stopPromise).resolves.toEqual({
data: Buffer.from([1, 2, 3]),
mimeType: 'audio/wav',
});
expect(mocks.rm).toHaveBeenCalledWith('/tmp/qwen-voice-abc', {
recursive: true,
force: true,
});
});
it('rejects promptly when sox closed before stop is requested', async () => {
const child = new FakeChildProcess();
mocks.spawn.mockReturnValue(child);
mocks.mkdtemp.mockResolvedValue('/tmp/qwen-voice-abc');
const recorder = createSoxRecorder();
await startRecorder(recorder);
child.emit('close', 1);
const stopResult = Promise.race([
recorder.stop().then(
() => 'resolved',
(error: unknown) =>
error instanceof Error ? `rejected:${error.message}` : 'rejected',
),
new Promise<string>((resolve) => setTimeout(() => resolve('pending'), 0)),
]);
await expect(stopResult).resolves.toMatch(/^rejected:/);
expect(mocks.rm).toHaveBeenCalledWith('/tmp/qwen-voice-abc', {
recursive: true,
force: true,
});
});
it('includes sox stderr when recording fails', async () => {
const child = new FakeChildProcess();
mocks.spawn.mockReturnValue(child);
mocks.mkdtemp.mockResolvedValue('/tmp/qwen-voice-abc');
const recorder = createSoxRecorder();
await startRecorder(recorder);
child.stderr.emit('data', Buffer.from('permission denied\n'));
child.emit('close', 2);
await expect(recorder.stop()).rejects.toThrow(
'Voice recorder failed with exit code 2: permission denied.',
);
});
it('caps captured sox stderr used in failure messages', async () => {
const child = new FakeChildProcess();
mocks.spawn.mockReturnValue(child);
mocks.mkdtemp.mockResolvedValue('/tmp/qwen-voice-abc');
const recorder = createSoxRecorder();
await startRecorder(recorder);
child.stderr.emit('data', Buffer.from('x'.repeat(5000)));
child.emit('close', 2);
await expect(recorder.stop()).rejects.toThrow(
`Voice recorder failed with exit code 2: ${'x'.repeat(4096)}.`,
);
});
it('explains how to fix a missing sox executable', async () => {
const child = new FakeChildProcess();
mocks.spawn.mockReturnValue(child);
mocks.mkdtemp.mockResolvedValue('/tmp/qwen-voice-abc');
const recorder = createSoxRecorder();
const startPromise = Promise.resolve(recorder.start());
const startResult = startPromise.then(
() => 'resolved',
(error: unknown) =>
error instanceof Error ? `rejected:${error.message}` : 'rejected',
);
await Promise.resolve();
const error = Object.assign(new Error('spawn sox ENOENT'), {
code: 'ENOENT',
});
child.emit('error', error);
await expect(startResult).resolves.toBe(
'rejected:SoX is not installed or not on PATH. Install SoX and try again.',
);
expect(mocks.rm).toHaveBeenCalledWith('/tmp/qwen-voice-abc', {
recursive: true,
force: true,
});
});
});
packages/cli/src/ui/voice/sox-recorder.ts
+168
View File
@@ -0,0 +1,168 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { spawn, type ChildProcess } from 'node:child_process';
import { mkdtemp, readFile, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import type {
RecordedVoiceAudio,
VoiceRecorder,
VoiceRecorderStartOptions,
} from '../hooks/use-voice-input.js';
// SoX `silence` effect: stop after 2.0s below 3% amplitude (matches CC).
const SILENCE_EFFECT_ARGS = ['silence', '1', '0.1', '3%', '1', '2.0', '3%'];
const MAX_STDERR_LENGTH = 4096;
function toSoxError(error: Error): Error {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') {
return new Error(
'SoX is not installed or not on PATH. Install SoX and try again.',
);
}
return error;
}
function formatSoxExitError(code: number | null, stderr: string): Error {
const detail = stderr.trim();
return new Error(
`Voice recorder failed with exit code ${code ?? 'unknown'}${
detail ? `: ${detail}` : ''
}.`,
);
}
class SoxRecorder implements VoiceRecorder {
private child: ChildProcess | null = null;
private tmpDir: string | null = null;
private filePath: string | null = null;
private spawnError: Error | null = null;
private stopRequested = false;
private onAutoStop: (() => void) | null = null;
private closeResult: {
code: number | null;
signal: NodeJS.Signals | null;
} | null = null;
private stderr = '';
private closePromise: Promise<{
code: number | null;
signal: NodeJS.Signals | null;
}> | null = null;
private async cleanup(): Promise<void> {
const tmpDir = this.tmpDir;
if (tmpDir) {
await rm(tmpDir, { recursive: true, force: true });
}
this.child = null;
this.tmpDir = null;
this.filePath = null;
this.spawnError = null;
this.stopRequested = false;
this.onAutoStop = null;
this.closeResult = null;
this.closePromise = null;
this.stderr = '';
}
async start(options: VoiceRecorderStartOptions = {}): Promise<void> {
this.stopRequested = false;
this.onAutoStop = options.onAutoStop ?? null;
this.tmpDir = await mkdtemp(path.join(tmpdir(), 'qwen-voice-'));
this.filePath = path.join(this.tmpDir, 'recording.wav');
this.child = spawn('sox', [
'-d',
'-r',
'16000',
'-c',
'1',
'-b',
'16',
this.filePath,
...(options.silenceDetection ? SILENCE_EFFECT_ARGS : []),
]);
const child = this.child;
child.stderr?.on('data', (chunk: Buffer) => {
if (this.stderr.length < MAX_STDERR_LENGTH) {
this.stderr = (this.stderr + chunk.toString()).slice(
0,
MAX_STDERR_LENGTH,
);
}
});
this.closePromise = new Promise((resolve) => {
child.once('close', (code, signal) => {
this.closeResult = { code, signal };
// SoX exited on its own with a clean status while we were still
// recording => the silence effect fired. Notify the hook to finalize.
if (!this.stopRequested && code === 0 && this.onAutoStop) {
this.onAutoStop();
}
resolve(this.closeResult);
});
});
try {
await new Promise<void>((resolve, reject) => {
child.once('spawn', resolve);
child.once('error', (error: Error) => {
this.spawnError = toSoxError(error);
reject(this.spawnError);
});
});
} catch (error) {
await this.cleanup();
throw error;
}
}
async stop(): Promise<RecordedVoiceAudio> {
if (!this.child || !this.tmpDir || !this.filePath) {
throw new Error('Voice recorder was not started.');
}
this.stopRequested = true;
const child = this.child;
const filePath = this.filePath;
const closePromise = this.closePromise;
try {
if (this.spawnError) {
throw this.spawnError;
}
if (!this.closeResult) {
child.kill('SIGINT');
}
const closeResult =
this.closeResult ?? (closePromise ? await closePromise : null);
if (
closeResult &&
closeResult.code !== 0 &&
closeResult.signal !== 'SIGINT'
) {
throw formatSoxExitError(closeResult.code, this.stderr);
}
const data = await readFile(filePath);
if (data.byteLength === 0) {
throw new Error('Voice recorder produced empty audio.');
}
return {
data,
mimeType: 'audio/wav',
};
} finally {
await this.cleanup();
}
}
}
export function createSoxRecorder(): VoiceRecorder {
return new SoxRecorder();
}
packages/cli/src/ui/voice/voice-availability.test.ts
+62
View File
@@ -0,0 +1,62 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, it, expect } from 'vitest';
import { getVoiceUnavailableReason } from './voice-availability.js';
describe('getVoiceUnavailableReason', () => {
it('is available on supported desktop platforms', () => {
expect(
getVoiceUnavailableReason({ platform: 'darwin', env: {} }),
).toBeUndefined();
expect(
getVoiceUnavailableReason({ platform: 'linux', env: {} }),
).toBeUndefined();
expect(
getVoiceUnavailableReason({ platform: 'win32', env: {} }),
).toBeUndefined();
});
it('flags unsupported platforms', () => {
expect(
getVoiceUnavailableReason({
platform: 'aix' as NodeJS.Platform,
env: {},
}),
).toMatch(/not supported/);
});
it('flags WSL without PulseAudio', () => {
expect(
getVoiceUnavailableReason({
platform: 'linux',
env: { WSL_DISTRO_NAME: 'Ubuntu' },
}),
).toMatch(/WSL/);
});
it('allows WSL when PulseAudio is configured', () => {
expect(
getVoiceUnavailableReason({
platform: 'linux',
env: {
WSL_DISTRO_NAME: 'Ubuntu',
PULSE_SERVER: '/mnt/wslg/PulseServer',
},
}),
).toBeUndefined();
});
it('allows WSLg when the PulseAudio socket exists', () => {
expect(
getVoiceUnavailableReason({
platform: 'linux',
env: { WSL_DISTRO_NAME: 'Ubuntu' },
fileExists: (path) => path === '/mnt/wslg/PulseServer',
}),
).toBeUndefined();
});
});
packages/cli/src/ui/voice/voice-availability.ts
+62
View File
@@ -0,0 +1,62 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { existsSync } from 'node:fs';
import process from 'node:process';
import { t } from '../../i18n/index.js';
// Pre-flight environment check so users get a clear message at /voice-enable
// time, instead of a cryptic recorder failure when no microphone can work
// (e.g. WSL without WSLg/PulseAudio, or an unsupported platform).
export interface VoiceEnvironment {
platform: NodeJS.Platform;
env: NodeJS.ProcessEnv;
fileExists?: (path: string) => boolean;
}
const SUPPORTED_PLATFORMS: readonly NodeJS.Platform[] = [
'darwin',
'linux',
'win32',
];
function isWsl(env: NodeJS.ProcessEnv): boolean {
return Boolean(env['WSL_DISTRO_NAME'] || env['WSL_INTEROP']);
}
/**
* Returns a human-readable reason if voice dictation cannot work in this
* environment, or undefined when it should be available.
*/
export function getVoiceUnavailableReason(
environment: VoiceEnvironment = {
platform: process.platform,
env: process.env,
},
): string | undefined {
const { platform, env } = environment;
const fileExists = environment.fileExists ?? existsSync;
if (!SUPPORTED_PLATFORMS.includes(platform)) {
return t('Voice dictation is not supported on {{platform}}.', {
platform,
});
}
if (
platform === 'linux' &&
isWsl(env) &&
!env['PULSE_SERVER'] &&
!fileExists('/mnt/wslg/PulseServer')
) {
return t(
'Voice dictation needs microphone access, which is unavailable in this WSL session. Use WSLg/PulseAudio, or run Qwen Code on a host with a microphone.',
);
}
return undefined;
}
packages/cli/src/ui/voice/voice-keyterms.test.ts
+22
View File
@@ -0,0 +1,22 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, it, expect } from 'vitest';
import { buildVoiceKeyterms } from './voice-keyterms.js';
describe('buildVoiceKeyterms', () => {
it('returns the static global vocabulary', () => {
const terms = buildVoiceKeyterms();
expect(terms).toContain('TypeScript');
expect(terms).toContain('worktree');
});
it('does not include project- or branch-derived terms (no metadata sent)', () => {
const terms = buildVoiceKeyterms();
expect(terms).not.toContain('qwen-code');
expect(terms).not.toContain('mvp');
});
});
packages/cli/src/ui/voice/voice-keyterms.ts
+50
View File
@@ -0,0 +1,50 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
// Static vocabulary-biasing hints sent to the transcription provider to improve
// accuracy on domain-specific terms a generic STT model tends to mangle. Sent as
// a leading system message (batch) or `corpus_text` (realtime) — not the OpenAI
// `prompt` field. Project/branch/recent-file enrichment is intentionally omitted
// so no project metadata is sent to the ASR provider. Mirrors Claude Code's
// voice keyterms feature.
const GLOBAL_KEYTERMS = [
'Qwen',
'MCP',
'grep',
'regex',
'localhost',
'codebase',
'TypeScript',
'JavaScript',
'JSON',
'YAML',
'OAuth',
'webhook',
'gRPC',
'dotfiles',
'subagent',
'worktree',
'stdout',
'stderr',
'async',
'await',
'API',
'CLI',
'npm',
'pnpm',
'commit',
'rebase',
'refactor',
'endpoint',
'middleware',
'schema',
'tokenizer',
];
/** The static keyterm vocabulary sent to the ASR provider for biasing. */
export function buildVoiceKeyterms(): string[] {
return [...GLOBAL_KEYTERMS];
}
packages/cli/src/ui/voice/voice-model.test.ts
+53
View File
@@ -0,0 +1,53 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, expect, it } from 'vitest';
import { AuthType, type AvailableModel } from '@qwen-code/qwen-code-core';
import {
isSelectableVoiceModel,
isTranscribableVoiceModel,
} from './voice-model.js';
function model(overrides: Partial<AvailableModel>): AvailableModel {
return {
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
...overrides,
} as AvailableModel;
}
describe('voice model guards', () => {
it('isTranscribableVoiceModel accepts any OpenAI model with a baseUrl (transport-agnostic)', () => {
expect(isTranscribableVoiceModel(model({}))).toBe(true);
// Custom ids resolve at the config layer; transport is enforced separately.
expect(isTranscribableVoiceModel(model({ id: 'custom:asr' }))).toBe(true);
});
it('isTranscribableVoiceModel rejects runtime models and empty baseUrls', () => {
expect(isTranscribableVoiceModel(model({ isRuntimeModel: true }))).toBe(
false,
);
expect(isTranscribableVoiceModel(model({ baseUrl: '' }))).toBe(false);
});
it('isSelectableVoiceModel accepts ids with a real ASR transport', () => {
expect(isSelectableVoiceModel(model({}))).toBe(true);
expect(
isSelectableVoiceModel(model({ id: 'qwen3-asr-flash-realtime' })),
).toBe(true);
});
it('isSelectableVoiceModel rejects ids with no ASR transport', () => {
// The core D fix: a chat/non-ASR id can no longer be persisted as voice.
expect(isSelectableVoiceModel(model({ id: 'gpt-4o' }))).toBe(false);
expect(isSelectableVoiceModel(model({ id: 'custom:asr' }))).toBe(false);
expect(
isSelectableVoiceModel(model({ id: 'qwen3-asr-flash-filetrans' })),
).toBe(false);
});
});
packages/cli/src/ui/voice/voice-model.ts
+65
View File
@@ -0,0 +1,65 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { AuthType, type AvailableModel } from '@qwen-code/qwen-code-core';
import { t } from '../../i18n/index.js';
export type VoiceTransport =
| 'qwen-asr-chat'
| 'qwen-asr-realtime'
| 'dashscope-task-realtime'
| 'unsupported';
/** Map a model id to the ASR transport it uses, or 'unsupported'. */
export function resolveVoiceTransport(model: string): VoiceTransport {
const id = model.toLowerCase();
if (/^qwen3-asr-flash-realtime(?:-|$)/.test(id)) {
return 'qwen-asr-realtime';
}
if (/^qwen3-asr-flash(?:-\d{4}-\d{2}-\d{2})?$/.test(id)) {
return 'qwen-asr-chat';
}
if (/^(fun-asr|paraformer).*realtime(?:-|$)/.test(id)) {
return 'dashscope-task-realtime';
}
return 'unsupported';
}
/**
* A model that can be used as a voice transcription provider: an OpenAI-compatible,
* non-runtime model with a baseUrl. Transport-agnostic on purpose — the record-time
* config resolver also uses this and then enforces the exact baseUrl/transport rules.
*/
export function isTranscribableVoiceModel(model: AvailableModel): boolean {
return (
model.authType === AuthType.USE_OPENAI &&
model.isRuntimeModel !== true &&
typeof model.baseUrl === 'string' &&
model.baseUrl.trim().length > 0
);
}
/**
* Selection guard for `/model --voice` and the model dialog: transcribable AND a
* model id we actually have an ASR transport for. This stops a non-ASR id (e.g. a
* chat model picked by mistake) from being persisted as the voice model, where it
* would report "enabled" in /voice status yet throw on every dictation.
*/
export function isSelectableVoiceModel(model: AvailableModel): boolean {
return (
isTranscribableVoiceModel(model) &&
resolveVoiceTransport(model.id) !== 'unsupported'
);
}
export function formatUnsupportedVoiceModelMessage(modelName: string): string {
return t(
"Voice model '{{modelName}}' cannot be used for transcription. Configure an OpenAI-compatible model with baseUrl in settings.modelProviders.",
{
modelName,
},
);
}
packages/cli/src/ui/voice/voice-recorder.test.ts
+178
View File
@@ -0,0 +1,178 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, expect, it, vi } from 'vitest';
import type { VoiceRecorder } from '../hooks/use-voice-input.js';
import { createVoiceRecorder } from './voice-recorder.js';
const debugLogger = vi.hoisted(() => ({
warn: vi.fn(),
}));
vi.mock('@qwen-code/qwen-code-core', () => ({
createDebugLogger: () => debugLogger,
}));
function recorder(overrides: Partial<VoiceRecorder> = {}): VoiceRecorder {
return {
start: vi.fn().mockResolvedValue(undefined),
stop: vi.fn().mockResolvedValue({
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
}),
...overrides,
};
}
describe('createVoiceRecorder', () => {
it('uses native audio capture before shell fallbacks', async () => {
const nativeRecorder = recorder();
const soxRecorder = recorder();
const voiceRecorder = createVoiceRecorder({
createNativeRecorder: vi.fn(() => nativeRecorder),
createSoxRecorder: vi.fn(() => soxRecorder),
platform: 'darwin',
});
await voiceRecorder.start();
const audio = await voiceRecorder.stop();
expect(nativeRecorder.start).toHaveBeenCalledTimes(1);
expect(nativeRecorder.stop).toHaveBeenCalledTimes(1);
expect(soxRecorder.start).not.toHaveBeenCalled();
expect(audio).toEqual({
data: new Uint8Array([1, 2, 3]),
mimeType: 'audio/wav',
});
});
it('falls back to sox when native audio capture is unavailable', async () => {
const nativeRecorder = recorder({
start: vi.fn().mockRejectedValue(new Error('native unavailable')),
});
const soxRecorder = recorder();
const voiceRecorder = createVoiceRecorder({
createNativeRecorder: vi.fn(() => nativeRecorder),
createSoxRecorder: vi.fn(() => soxRecorder),
platform: 'darwin',
});
await voiceRecorder.start();
await voiceRecorder.stop();
expect(nativeRecorder.start).toHaveBeenCalledTimes(1);
expect(nativeRecorder.stop).not.toHaveBeenCalled();
expect(soxRecorder.start).toHaveBeenCalledTimes(1);
expect(soxRecorder.stop).toHaveBeenCalledTimes(1);
});
it('prefers sox before arecord on Linux so tap mode can auto-stop on silence', async () => {
const nativeRecorder = recorder({
start: vi.fn().mockRejectedValue(new Error('native unavailable')),
});
const arecordRecorder = recorder();
const soxRecorder = recorder();
const voiceRecorder = createVoiceRecorder({
createNativeRecorder: vi.fn(() => nativeRecorder),
createArecordRecorder: vi.fn(() => arecordRecorder),
createSoxRecorder: vi.fn(() => soxRecorder),
platform: 'linux',
});
await voiceRecorder.start({ silenceDetection: true });
await voiceRecorder.stop();
expect(soxRecorder.start).toHaveBeenCalledWith({
silenceDetection: true,
});
expect(soxRecorder.stop).toHaveBeenCalledTimes(1);
expect(arecordRecorder.start).not.toHaveBeenCalled();
});
it('reports streaming support from the active native recorder', async () => {
const nativeRecorder = recorder({
supportsStreaming: vi.fn(() => true),
});
const voiceRecorder = createVoiceRecorder({
createNativeRecorder: vi.fn(() => nativeRecorder),
createSoxRecorder: vi.fn(() => recorder()),
platform: 'darwin',
});
await voiceRecorder.start();
expect(voiceRecorder.supportsStreaming?.()).toBe(true);
});
it('reports no streaming support after falling back to shell recording', async () => {
const nativeRecorder = recorder({
start: vi.fn().mockRejectedValue(new Error('native unavailable')),
});
const soxRecorder = recorder({
drain: vi.fn(() => new Uint8Array(0)),
});
const voiceRecorder = createVoiceRecorder({
createNativeRecorder: vi.fn(() => nativeRecorder),
createSoxRecorder: vi.fn(() => soxRecorder),
platform: 'darwin',
});
await voiceRecorder.start();
expect(voiceRecorder.supportsStreaming?.()).toBe(false);
});
it('reuses backend instances across warmup, status, and start', async () => {
const nativeRecorder = recorder({
warmup: vi.fn().mockResolvedValue(undefined),
microphoneStatus: vi.fn().mockResolvedValue('granted'),
});
const createNativeRecorder = vi.fn(() => nativeRecorder);
const voiceRecorder = createVoiceRecorder({
createNativeRecorder,
createSoxRecorder: vi.fn(() => recorder()),
platform: 'darwin',
});
await voiceRecorder.warmup?.();
await expect(voiceRecorder.microphoneStatus?.()).resolves.toBe('granted');
await voiceRecorder.start();
expect(createNativeRecorder).toHaveBeenCalledTimes(1);
expect(nativeRecorder.warmup).toHaveBeenCalledTimes(1);
expect(nativeRecorder.microphoneStatus).toHaveBeenCalledTimes(1);
expect(nativeRecorder.start).toHaveBeenCalledTimes(1);
});
it('logs warmup failures and still lets start fall back', async () => {
const nativeRecorder = recorder({
warmup: vi.fn().mockRejectedValue(new Error('native load failed')),
start: vi.fn().mockRejectedValue(new Error('native unavailable')),
});
const soxRecorder = recorder();
const voiceRecorder = createVoiceRecorder({
createNativeRecorder: vi.fn(() => nativeRecorder),
createSoxRecorder: vi.fn(() => soxRecorder),
platform: 'darwin',
});
await voiceRecorder.warmup?.();
await voiceRecorder.start();
expect(debugLogger.warn).toHaveBeenCalledWith(
'[voice] recorder warmup failed:',
new Error('native load failed'),
);
expect(soxRecorder.start).toHaveBeenCalledTimes(1);
});
});
packages/cli/src/ui/voice/voice-recorder.ts
+127
View File
@@ -0,0 +1,127 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import process from 'node:process';
import { createDebugLogger } from '@qwen-code/qwen-code-core';
import type {
RecordedVoiceAudio,
VoiceRecorder,
VoiceRecorderStartOptions,
} from '../hooks/use-voice-input.js';
import { createArecordRecorder } from './arecord-recorder.js';
import { createNativeAudioRecorder } from './native-audio-recorder.js';
import { createSoxRecorder } from './sox-recorder.js';
const debugLogger = createDebugLogger('VOICE_RECORDER');
interface VoiceRecorderOptions {
createNativeRecorder?: () => VoiceRecorder;
createArecordRecorder?: () => VoiceRecorder;
createSoxRecorder?: () => VoiceRecorder;
platform?: NodeJS.Platform;
}
class FallbackVoiceRecorder implements VoiceRecorder {
private activeRecorder: VoiceRecorder | null = null;
private readonly recorders = new Map<() => VoiceRecorder, VoiceRecorder>();
constructor(private readonly factories: Array<() => VoiceRecorder>) {}
private recorderFor(factory: () => VoiceRecorder): VoiceRecorder {
let recorder = this.recorders.get(factory);
if (!recorder) {
recorder = factory();
this.recorders.set(factory, recorder);
}
return recorder;
}
async start(options?: VoiceRecorderStartOptions): Promise<void> {
const errors: string[] = [];
for (const factory of this.factories) {
const recorder = this.recorderFor(factory);
try {
await recorder.start(options);
this.activeRecorder = recorder;
return;
} catch (error) {
errors.push(error instanceof Error ? error.message : String(error));
}
}
throw new Error(
`Voice recording is unavailable. ${errors.filter(Boolean).join(' ')}`,
);
}
async stop(): Promise<RecordedVoiceAudio> {
const recorder = this.activeRecorder;
if (!recorder) {
throw new Error('Voice recorder was not started.');
}
this.activeRecorder = null;
return recorder.stop();
}
// Best-effort preload of any backend that supports it (the native one).
async warmup(): Promise<void> {
for (const factory of this.factories) {
try {
await this.recorderFor(factory).warmup?.();
} catch (error) {
debugLogger.warn('[voice] recorder warmup failed:', error);
// Ignore — start() will still try/fall back at record time.
}
}
}
drain(): Uint8Array {
return this.activeRecorder?.drain?.() ?? new Uint8Array(0);
}
supportsStreaming(): boolean {
return this.activeRecorder?.supportsStreaming?.() ?? false;
}
audioLevel(): number {
return this.activeRecorder?.audioLevel?.() ?? 0;
}
// Permission from the first backend that can report it (the native one).
async microphoneStatus(): Promise<
'granted' | 'denied' | 'prompt' | 'unknown'
> {
for (const factory of this.factories) {
const recorder = this.recorderFor(factory);
if (recorder.microphoneStatus) {
try {
return await recorder.microphoneStatus();
} catch {
// Try the next backend.
}
}
}
return 'unknown';
}
}
export function createVoiceRecorder(
options: VoiceRecorderOptions = {},
): VoiceRecorder {
const platform = options.platform ?? process.platform;
const factories: Array<() => VoiceRecorder> = [
options.createNativeRecorder ?? createNativeAudioRecorder,
];
// arecord (ALSA) only exists on Linux; elsewhere it would just fail fast.
if (platform === 'linux') {
factories.push(options.createSoxRecorder ?? createSoxRecorder);
factories.push(options.createArecordRecorder ?? createArecordRecorder);
} else {
factories.push(options.createSoxRecorder ?? createSoxRecorder);
}
return new FallbackVoiceRecorder(factories);
}
packages/cli/src/ui/voice/voice-stream-retry.test.ts
+98
View File
@@ -0,0 +1,98 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { describe, expect, it, vi } from 'vitest';
import { openVoiceStreamWithRetry } from './voice-stream-retry.js';
import type { VoiceStreamSession } from './voice-stream-session.js';
function session(): VoiceStreamSession {
return {
pushAudio: vi.fn(),
finish: vi.fn().mockResolvedValue('ok'),
abort: vi.fn(),
};
}
describe('openVoiceStreamWithRetry', () => {
it('retries once when opening the realtime stream fails before use', async () => {
vi.useFakeTimers();
try {
const opened = session();
const open = vi
.fn<() => Promise<VoiceStreamSession>>()
.mockRejectedValueOnce(new Error('early connect failed'))
.mockResolvedValueOnce(opened);
const result = openVoiceStreamWithRetry(open);
const assertion = result.then((value) => {
expect(value).toBe(opened);
});
await vi.advanceTimersByTimeAsync(200);
await assertion;
expect(open).toHaveBeenCalledTimes(2);
} finally {
vi.useRealTimers();
}
});
it('throws the second open error after the retry is exhausted', async () => {
vi.useFakeTimers();
try {
const second = new Error('still failing');
const open = vi
.fn<() => Promise<VoiceStreamSession>>()
.mockRejectedValueOnce(new Error('first failure'))
.mockRejectedValueOnce(second);
const result = openVoiceStreamWithRetry(open);
const assertion = result.then(
() => {
throw new Error('Expected retry to reject.');
},
(error) => {
expect(error).toBe(second);
},
);
await vi.advanceTimersByTimeAsync(200);
await assertion;
expect(open).toHaveBeenCalledTimes(2);
} finally {
vi.useRealTimers();
}
});
it('does not retry non-retryable request errors', async () => {
const error = new Error('401 Unauthorized');
const open = vi
.fn<() => Promise<VoiceStreamSession>>()
.mockRejectedValueOnce(error);
await expect(openVoiceStreamWithRetry(open)).rejects.toBe(error);
expect(open).toHaveBeenCalledTimes(1);
});
it('does not retry permanent not found errors', async () => {
const error = new Error('404 Not Found');
const open = vi
.fn<() => Promise<VoiceStreamSession>>()
.mockRejectedValueOnce(error);
await expect(openVoiceStreamWithRetry(open)).rejects.toBe(error);
expect(open).toHaveBeenCalledTimes(1);
});
it('does not retry rate limit errors', async () => {
const error = new Error('429 rate limit exceeded');
const open = vi
.fn<() => Promise<VoiceStreamSession>>()
.mockRejectedValueOnce(error);
await expect(openVoiceStreamWithRetry(open)).rejects.toBe(error);
expect(open).toHaveBeenCalledTimes(1);
});
});
packages/cli/src/ui/voice/voice-stream-retry.ts
+42
View File
@@ -0,0 +1,42 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import type { VoiceStreamSession } from './voice-stream-session.js';
import { createDebugLogger } from '@qwen-code/qwen-code-core';
const RETRY_DELAY_MS = 200;
const debugLogger = createDebugLogger('VOICE_STREAM');
function delay(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function isRetryable(error: unknown): boolean {
const message = error instanceof Error ? error.message : String(error);
if (
/\b(400|401|403|404|410|422|429)\b|unauthori[sz]ed|forbidden|model_not_supported|rate.?limit/i.test(
message,
)
) {
return false;
}
return true;
}
export async function openVoiceStreamWithRetry(
open: () => Promise<VoiceStreamSession>,
): Promise<VoiceStreamSession> {
try {
return await open();
} catch (error) {
if (!isRetryable(error)) {
throw error;
}
debugLogger.debug('[voice] stream open failed, retrying:', error);
await delay(RETRY_DELAY_MS);
return open();
}
}
packages/cli/src/ui/voice/voice-stream-session.test.ts
+248
View File
@@ -0,0 +1,248 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { afterEach, describe, expect, it, vi } from 'vitest';
import { deriveStreamUrl, openVoiceStream } from './voice-stream-session.js';
class FakeSocket {
readonly OPEN = 1;
readyState = this.OPEN;
bufferedAmount = 0;
readonly sent: Array<string | Uint8Array> = [];
private readonly handlers = new Map<
string,
Array<(...args: unknown[]) => void>
>();
send(data: string | Uint8Array): void {
this.sent.push(data);
}
close(): void {
this.readyState = 3;
}
on(event: string, cb: (...args: unknown[]) => void): void {
const handlers = this.handlers.get(event) ?? [];
handlers.push(cb);
this.handlers.set(event, handlers);
}
emit(event: string, ...args: unknown[]): void {
for (const handler of this.handlers.get(event) ?? []) {
handler(...args);
}
}
}
function startSession(
socket: FakeSocket,
callbacks: Parameters<typeof openVoiceStream>[1] = {},
) {
const sessionPromise = openVoiceStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'paraformer-realtime-v2',
},
callbacks,
{ createWebSocket: () => socket },
);
socket.emit('open');
socket.emit(
'message',
JSON.stringify({ header: { event: 'task-started' } }),
false,
);
return sessionPromise;
}
describe('voice-stream-session', () => {
afterEach(() => {
vi.useRealTimers();
});
it('derives websocket URLs from https/http base URLs with path prefixes', () => {
expect(
deriveStreamUrl('https://dashscope.aliyuncs.com/compatible-mode/v1'),
).toBe('wss://dashscope.aliyuncs.com/api-ws/v1/inference');
expect(deriveStreamUrl('http://localhost:8080/dashscope/v1')).toBe(
'ws://localhost:8080/dashscope/api-ws/v1/inference',
);
});
it('rejects finish when the task stream closes unexpectedly', async () => {
const socket = new FakeSocket();
const session = await startSession(socket);
const transcriptPromise = session.finish();
socket.emit('close');
await expect(transcriptPromise).rejects.toThrow(
'Voice stream connection closed unexpectedly.',
);
});
it('notifies immediately when the task stream closes while recording', async () => {
const socket = new FakeSocket();
const onError = vi.fn();
await startSession(socket, { onError });
socket.emit('close');
expect(onError).toHaveBeenCalledWith(
expect.objectContaining({
message:
'Voice stream connection closed unexpectedly. Transcript may be incomplete.',
}),
);
});
it('rejects committed transcript when the stream closes before task-finished', async () => {
const socket = new FakeSocket();
const session = await startSession(socket);
socket.emit(
'message',
JSON.stringify({
header: { event: 'result-generated' },
payload: {
output: { sentence: { text: 'hello world', sentence_end: true } },
},
}),
false,
);
const transcriptPromise = session.finish();
socket.emit('close');
await expect(transcriptPromise).rejects.toThrow(
'Voice stream connection closed unexpectedly.',
);
});
it('rejects when task-finished arrives before task-started', async () => {
const socket = new FakeSocket();
const sessionPromise = openVoiceStream(
{
baseUrl: 'https://dashscope.example/v1',
model: 'paraformer-realtime-v2',
},
{},
{ createWebSocket: () => socket },
);
socket.emit('open');
socket.emit(
'message',
JSON.stringify({ header: { event: 'task-finished' } }),
false,
);
await expect(sessionPromise).rejects.toThrow(
'Voice stream finished before it started.',
);
});
it('resolves finish when task-finished arrives before finish is called', async () => {
const socket = new FakeSocket();
const session = await startSession(socket);
socket.emit(
'message',
JSON.stringify({
header: { event: 'result-generated' },
payload: {
output: { sentence: { text: 'hello world', sentence_end: true } },
},
}),
false,
);
socket.emit(
'message',
JSON.stringify({ header: { event: 'task-finished' } }),
false,
);
await expect(session.finish()).resolves.toBe('hello world');
});
it('drops audio chunks when the socket buffer is backed up', async () => {
const socket = new FakeSocket();
const session = await startSession(socket);
socket.sent.length = 0;
socket.bufferedAmount = 1024 * 1024 + 1;
session.pushAudio(new Uint8Array([1, 2, 3]));
session.pushAudio(new Uint8Array([4, 5, 6]));
expect(socket.sent).toEqual([]);
socket.bufferedAmount = 0;
session.pushAudio(new Uint8Array([7]));
expect(socket.sent).toEqual([new Uint8Array([7])]);
});
it('rejects finish when the task never finishes', async () => {
vi.useFakeTimers();
const socket = new FakeSocket();
const session = await startSession(socket);
const transcriptPromise = session.finish();
void transcriptPromise.catch(() => undefined);
await vi.advanceTimersByTimeAsync(60_000);
await expect(transcriptPromise).rejects.toThrow(
'Voice stream finish timed out.',
);
expect(socket.readyState).toBe(3);
});
it('rejects finish when the server sends task-failed', async () => {
const socket = new FakeSocket();
const session = await startSession(socket);
const transcriptPromise = session.finish();
socket.emit(
'message',
JSON.stringify({
header: {
event: 'task-failed',
error_code: '429',
error_message: 'rate limited',
},
}),
false,
);
await expect(transcriptPromise).rejects.toThrow('rate limited');
await expect(transcriptPromise).rejects.toThrow(
'wss://dashscope.example/api-ws/v1/inference',
);
});
it('sanitizes and caps task-failed server messages', async () => {
const socket = new FakeSocket();
const session = await startSession(socket);
const transcriptPromise = session.finish();
const longMessage = `bad\x1b[8mhidden\x1b[0m ${'x'.repeat(300)}`;
socket.emit(
'message',
JSON.stringify({
header: {
event: 'task-failed',
error_code: '500',
error_message: longMessage,
},
}),
false,
);
await expect(transcriptPromise).rejects.toThrow(
'bad\\u001b[8mhidden\\u001b[0m',
);
await expect(transcriptPromise).rejects.not.toThrow('x'.repeat(220));
});
});
packages/cli/src/ui/voice/voice-stream-session.ts
+349
View File
@@ -0,0 +1,349 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { randomUUID } from 'node:crypto';
import { createDebugLogger } from '@qwen-code/qwen-code-core';
import WebSocket from 'ws';
import { escapeAnsiCtrlCodes } from '../utils/textUtils.js';
// Streaming ASR over the DashScope realtime "task" WebSocket protocol
// (paraformer-realtime / fun-asr-realtime). Audio is pushed as raw binary PCM
// (s16le, 16 kHz, mono) and transcripts arrive as `result-generated` events;
// `payload.output.sentence.sentence_end` marks a finalized sentence.
export interface VoiceStreamConfig {
/** HTTPS base URL of the configured provider; its host derives the wss URL. */
baseUrl: string;
apiKey?: string;
/** A realtime model id, e.g. paraformer-realtime-v2 / fun-asr-realtime. */
model: string;
/** Optional BCP-47-ish language code (paraformer language_hints). */
language?: string;
/** Optional contextual bias text for providers that support corpus prompts. */
keytermsContext?: string;
}
export interface VoiceStreamCallbacks {
/** The full running transcript (committed sentences + current partial). */
onInterim?: (text: string) => void;
/** Terminal stream errors that arrive while recording, before finish(). */
onError?: (error: Error) => void;
}
export interface VoiceStreamSession {
pushAudio: (pcm: Uint8Array) => void;
/** Flush, wait for the final result, and return the full transcript. */
finish: () => Promise<string>;
abort: () => void;
}
export interface SocketLike {
readyState: number;
OPEN: number;
bufferedAmount?: number;
send: (data: string | Uint8Array) => void;
close: () => void;
on: (event: string, cb: (...args: unknown[]) => void) => void;
}
export interface VoiceStreamDeps {
createWebSocket?: (
url: string,
options: { headers: Record<string, string> },
) => SocketLike;
}
const CONNECT_TIMEOUT_MS = 8000;
const FINISH_TIMEOUT_MS = 60_000;
const MAX_BUFFERED_AUDIO_BYTES = 1024 * 1024;
const MAX_SERVER_ERROR_MESSAGE_LENGTH = 200;
const debugLogger = createDebugLogger('VOICE_STREAM_SESSION');
export function deriveWebSocketBase(baseUrl: string): string {
const url = new URL(baseUrl);
const wsScheme = url.protocol === 'https:' ? 'wss:' : 'ws:';
let prefix = url.pathname.replace(/\/+$/, '');
if (prefix.endsWith('/compatible-mode/v1')) {
prefix = prefix.slice(0, -'/compatible-mode/v1'.length);
} else if (prefix.endsWith('/v1')) {
prefix = prefix.slice(0, -'/v1'.length);
}
return `${wsScheme}//${url.host}${prefix}`;
}
export function deriveStreamUrl(baseUrl: string): string {
return `${deriveWebSocketBase(baseUrl)}/api-ws/v1/inference`;
}
function formatServerErrorMessage(raw: unknown): string {
const text = typeof raw === 'string' ? raw : 'unknown';
return escapeAnsiCtrlCodes(text).slice(0, MAX_SERVER_ERROR_MESSAGE_LENGTH);
}
export function openVoiceStream(
config: VoiceStreamConfig,
callbacks: VoiceStreamCallbacks = {},
deps: VoiceStreamDeps = {},
): Promise<VoiceStreamSession> {
const createWebSocket =
deps.createWebSocket ??
((url, options) =>
new WebSocket(url, {
headers: options.headers,
}) as unknown as SocketLike);
return new Promise<VoiceStreamSession>((resolve, reject) => {
const streamUrl = deriveStreamUrl(config.baseUrl);
const ws = createWebSocket(streamUrl, {
headers: config.apiKey
? { Authorization: `Bearer ${config.apiKey}` }
: {},
});
const taskId = randomUUID();
let started = false;
let settled = false;
let committed = '';
let finishPromise: Promise<string> | null = null;
let finishResolve: ((text: string) => void) | null = null;
let finishReject: ((error: unknown) => void) | null = null;
let finishTimer: ReturnType<typeof setTimeout> | null = null;
let connectTimer: ReturnType<typeof setTimeout> | null = null;
let terminalError: Error | null = null;
let finishedTranscript: string | null = null;
let backpressureWarned = false;
const clearFinishTimer = () => {
if (finishTimer) {
clearTimeout(finishTimer);
finishTimer = null;
}
};
const clearConnectTimer = () => {
if (connectTimer) {
clearTimeout(connectTimer);
connectTimer = null;
}
};
const fail = (error: unknown) => {
if (settled) return;
settled = true;
const normalized =
error instanceof Error ? error : new Error(String(error));
clearConnectTimer();
clearFinishTimer();
try {
ws.close();
} catch {
/* ignore */
}
if (finishReject) {
finishReject(normalized);
finishResolve = null;
finishReject = null;
} else {
terminalError = normalized;
if (!started) {
reject(normalized);
} else {
callbacks.onError?.(normalized);
}
}
};
connectTimer = setTimeout(() => {
if (!started) fail(new Error('Voice stream connection timed out.'));
}, CONNECT_TIMEOUT_MS);
ws.on('open', () => {
ws.send(
JSON.stringify({
header: { action: 'run-task', task_id: taskId, streaming: 'duplex' },
payload: {
task_group: 'audio',
task: 'asr',
function: 'recognition',
model: config.model,
parameters: {
format: 'pcm',
sample_rate: 16000,
...(config.language ? { language_hints: [config.language] } : {}),
},
input: {},
},
}),
);
});
ws.on('message', (...args: unknown[]) => {
const data = args[0];
const isBinary = args[1] === true;
if (isBinary) return;
let msg: {
header?: {
event?: string;
error_code?: string;
error_message?: string;
};
payload?: {
output?: {
sentence?: {
text?: unknown;
sentence_end?: boolean;
heartbeat?: boolean;
};
};
};
};
try {
msg = JSON.parse(String(data));
} catch (error) {
debugLogger.warn('[voice] failed to parse stream message:', error);
return;
}
const event = msg.header?.event;
if (event === 'task-started') {
started = true;
clearConnectTimer();
resolve({
pushAudio: (pcm) => {
if (
ws.readyState === ws.OPEN &&
pcm.length > 0 &&
(ws.bufferedAmount ?? 0) <= MAX_BUFFERED_AUDIO_BYTES
) {
backpressureWarned = false;
ws.send(pcm);
} else if (pcm.length > 0 && !backpressureWarned) {
backpressureWarned = true;
debugLogger.warn(
'[voice] dropping DashScope audio due to socket backpressure',
);
}
},
finish: () => {
if (finishPromise) return finishPromise;
finishPromise = new Promise<string>((res, rej) => {
if (finishedTranscript !== null) {
res(finishedTranscript);
return;
}
if (terminalError) {
rej(terminalError);
return;
}
finishResolve = res;
finishReject = rej;
finishTimer = setTimeout(() => {
fail(new Error('Voice stream finish timed out.'));
}, FINISH_TIMEOUT_MS);
try {
ws.send(
JSON.stringify({
header: {
action: 'finish-task',
task_id: taskId,
streaming: 'duplex',
},
payload: { input: {} },
}),
);
} catch (error) {
fail(error);
}
});
return finishPromise;
},
abort: () => {
try {
ws.close();
} catch {
/* ignore */
}
},
});
} else if (event === 'result-generated') {
const sentence = msg.payload?.output?.sentence;
if (
sentence &&
!sentence.heartbeat &&
typeof sentence.text === 'string'
) {
if (sentence.sentence_end) {
committed = committed
? `${committed} ${sentence.text}`
: sentence.text;
callbacks.onInterim?.(committed);
} else {
const running = committed
? `${committed} ${sentence.text}`
: sentence.text;
callbacks.onInterim?.(running);
}
}
} else if (event === 'task-finished') {
if (!started) {
// Out-of-order finish before task-started: the connect promise only
// resolves on task-started, so reject it instead of hanging forever.
fail(new Error('Voice stream finished before it started.'));
return;
}
finishedTranscript = committed.trim();
settled = true;
clearConnectTimer();
clearFinishTimer();
try {
ws.close();
} catch {
/* ignore */
}
finishResolve?.(finishedTranscript);
finishResolve = null;
finishReject = null;
} else if (event === 'task-failed') {
clearConnectTimer();
fail(
new Error(
`Voice stream failed at ${streamUrl} task ${taskId} (${msg.header?.error_code ?? 'error'}): ${formatServerErrorMessage(
msg.header?.error_message,
)}`,
),
);
}
});
ws.on('error', (...args: unknown[]) => {
clearConnectTimer();
const error = args[0];
fail(error instanceof Error ? error : new Error(String(error)));
});
ws.on('close', () => {
clearConnectTimer();
clearFinishTimer();
if (settled) return;
if (started && finishReject) {
settled = true;
finishReject(
new Error(
'Voice stream connection closed unexpectedly. Transcript may be incomplete.',
),
);
finishResolve = null;
finishReject = null;
} else if (!started) {
fail(new Error('Voice stream closed before it started.'));
} else {
const err = new Error(
'Voice stream connection closed unexpectedly. Transcript may be incomplete.',
);
terminalError ??= err;
callbacks.onError?.(err);
}
});
});
}
packages/cli/src/ui/voice/voice-transcriber.test.ts
+848
View File
@@ -0,0 +1,848 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import { afterEach, beforeEach, describe, it, expect, vi } from 'vitest';
import { AuthType, type Config } from '@qwen-code/qwen-code-core';
import type { LoadedSettings } from '../../config/settings.js';
import {
isStreamingVoiceModel,
isKeytermEcho,
resolveVoiceStreamConfig,
resolveVoiceTranscriptionConfig,
resolveVoiceTransport,
transcribeVoiceAudio,
} from './voice-transcriber.js';
function createConfig(models: ReturnType<Config['getAllConfiguredModels']>) {
return {
getAllConfiguredModels: vi.fn().mockReturnValue(models),
} as unknown as Config;
}
function createSettings(
env: Record<string, string> = {},
apiKey?: string,
): LoadedSettings {
return {
merged: { env, security: { auth: { apiKey } } },
} as unknown as LoadedSettings;
}
async function lookupPublicHost(): Promise<{ address: string }> {
return { address: '8.8.8.8' };
}
describe('voice-transcriber', () => {
beforeEach(() => {
vi.stubEnv('OPENAI_API_KEY', '');
});
afterEach(() => {
vi.unstubAllEnvs();
});
it('resolves a plain voice model id from configured models', () => {
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]);
expect(
resolveVoiceTranscriptionConfig({
config,
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash',
}),
).toEqual({
model: 'qwen3-asr-flash',
baseUrl: 'https://dashscope.example/v1',
apiKey: 'sk-test',
});
});
it('routes known voice models by model id instead of user protocol', () => {
expect(resolveVoiceTransport('qwen3-asr-flash')).toBe('qwen-asr-chat');
expect(resolveVoiceTransport('qwen3-asr-flash-2026-02-10')).toBe(
'qwen-asr-chat',
);
expect(resolveVoiceTransport('qwen3-asr-flash-realtime')).toBe(
'qwen-asr-realtime',
);
expect(resolveVoiceTransport('qwen3-asr-flash-realtime-2026-02-10')).toBe(
'qwen-asr-realtime',
);
expect(resolveVoiceTransport('fun-asr-realtime')).toBe(
'dashscope-task-realtime',
);
expect(resolveVoiceTransport('fun-asr-flash-8k-realtime')).toBe(
'dashscope-task-realtime',
);
expect(resolveVoiceTransport('paraformer-realtime-v2')).toBe(
'dashscope-task-realtime',
);
expect(resolveVoiceTransport('qwen3-asr-flash-filetrans')).toBe(
'unsupported',
);
});
it('does not rewrite qwen3-asr-flash to a realtime model', () => {
expect(isStreamingVoiceModel('qwen3-asr-flash')).toBe(false);
expect(() =>
resolveVoiceStreamConfig({
config: createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash',
}),
).toThrow(/does not support streaming/);
});
it('keeps realtime model ids on their matching streaming transport', () => {
const qwenStreamConfig = resolveVoiceStreamConfig({
config: createConfig([
{
id: 'qwen3-asr-flash-realtime',
label: 'Qwen ASR Realtime',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash-realtime',
});
expect(qwenStreamConfig).toEqual({
transport: 'qwen-asr-realtime',
model: 'qwen3-asr-flash-realtime',
baseUrl: 'https://dashscope.example/v1',
apiKey: 'sk-test',
keytermsContext: expect.stringContaining('Qwen'),
});
const funStreamConfig = resolveVoiceStreamConfig({
config: createConfig([
{
id: 'fun-asr-realtime',
label: 'Fun ASR Realtime',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'fun-asr-realtime',
});
expect(funStreamConfig.transport).toBe('dashscope-task-realtime');
expect(funStreamConfig.keytermsContext).toBeUndefined();
});
it('does not include project path metadata in voice keyterms', () => {
const config = {
...createConfig([
{
id: 'qwen3-asr-flash-realtime',
label: 'Qwen ASR Realtime',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
getProjectRoot: vi.fn(() => '/tmp/secret-project-codename'),
} as unknown as Config;
const streamConfig = resolveVoiceStreamConfig({
config,
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash-realtime',
});
expect(streamConfig.keytermsContext).toContain('Qwen');
expect(streamConfig.keytermsContext).not.toContain('secret');
expect(streamConfig.keytermsContext).not.toContain('codename');
});
it('treats colon-containing voice model values as literal model ids', () => {
const config = createConfig([
{
id: 'custom:asr',
label: 'Custom ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://asr.example/v1',
},
]);
expect(
resolveVoiceTranscriptionConfig({
config,
settings: createSettings(),
voiceModel: 'custom:asr',
}).model,
).toBe('custom:asr');
});
it('rejects duplicate voice model ids', () => {
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'A',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://a.example/v1',
},
{
id: 'qwen3-asr-flash',
label: 'B',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://b.example/v1',
},
]);
expect(() =>
resolveVoiceTranscriptionConfig({
config,
settings: createSettings(),
voiceModel: 'qwen3-asr-flash',
}),
).toThrow("Voice model 'qwen3-asr-flash' is ambiguous.");
});
it('rejects non OpenAI-compatible voice models', () => {
const config = createConfig([
{
id: 'claude-sonnet',
label: 'Claude Sonnet',
authType: AuthType.USE_ANTHROPIC,
baseUrl: 'https://anthropic.example/v1',
},
]);
expect(() =>
resolveVoiceTranscriptionConfig({
config,
settings: createSettings(),
voiceModel: 'claude-sonnet',
}),
).toThrow("Voice model 'claude-sonnet' cannot be used for transcription.");
});
it('falls back to the OpenAI auth apiKey when the model has no envKey', () => {
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.aliyuncs.com/compatible-mode/v1',
},
]);
expect(
resolveVoiceTranscriptionConfig({
config,
settings: createSettings({}, 'sk-from-settings'),
voiceModel: 'qwen3-asr-flash',
}),
).toEqual({
model: 'qwen3-asr-flash',
baseUrl: 'https://dashscope.aliyuncs.com/compatible-mode/v1',
apiKey: 'sk-from-settings',
});
});
it('falls back to the primary auth apiKey for DashScope intl and US hosts', () => {
for (const baseUrl of [
'https://dashscope-intl.aliyuncs.com/compatible-mode/v1',
'https://dashscope-us.aliyuncs.com/compatible-mode/v1',
]) {
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl,
},
]);
expect(
resolveVoiceTranscriptionConfig({
config,
settings: createSettings({}, 'sk-from-settings'),
voiceModel: 'qwen3-asr-flash',
}).apiKey,
).toBe('sk-from-settings');
}
});
it('does not forward the primary auth apiKey to third-party voice hosts', () => {
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'Custom ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://asr.example/v1',
},
]);
expect(
resolveVoiceTranscriptionConfig({
config,
settings: createSettings({}, 'sk-primary'),
voiceModel: 'qwen3-asr-flash',
}),
).toEqual({
model: 'qwen3-asr-flash',
baseUrl: 'https://asr.example/v1',
});
});
it('does not forward OPENAI_API_KEY to third-party voice hosts without envKey', () => {
vi.stubEnv('OPENAI_API_KEY', 'sk-openai');
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'Custom ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://asr.example/v1',
},
]);
expect(
resolveVoiceTranscriptionConfig({
config,
settings: createSettings(),
voiceModel: 'qwen3-asr-flash',
}),
).toEqual({
model: 'qwen3-asr-flash',
baseUrl: 'https://asr.example/v1',
});
});
it('rejects invalid voice base URLs', () => {
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'dashscope.example/v1',
},
]);
expect(() =>
resolveVoiceTranscriptionConfig({
config,
settings: createSettings(),
voiceModel: 'qwen3-asr-flash',
}),
).toThrow("Voice model 'qwen3-asr-flash' has an invalid baseUrl.");
});
it('rejects non-https voice URLs', () => {
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'http://dashscope.aliyuncs.com/compatible-mode/v1',
},
]);
expect(() =>
resolveVoiceTranscriptionConfig({
config,
settings: createSettings({}, 'sk-primary'),
voiceModel: 'qwen3-asr-flash',
}),
).toThrow(/must use an https baseUrl/);
});
it('rejects private-network voice URLs', () => {
for (const baseUrl of [
'https://10.0.0.5/v1',
'https://172.16.0.5/v1',
'https://192.168.1.5/v1',
'https://169.254.169.254/v1',
'https://0.0.0.0/v1',
'https://[fe80::1]/v1',
'https://[::ffff:169.254.169.254]/v1',
]) {
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'Private ASR',
authType: AuthType.USE_OPENAI,
baseUrl,
},
]);
expect(() =>
resolveVoiceTranscriptionConfig({
config,
settings: createSettings(),
voiceModel: 'qwen3-asr-flash',
}),
).toThrow(/private-network baseUrl/);
}
});
it('rejects voice model hosts that resolve to private-network IPs', async () => {
await expect(
transcribeVoiceAudio(
{ data: new Uint8Array([1, 2, 3]), mimeType: 'audio/wav' },
{
config: createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://asr.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash',
lookupHost: vi.fn().mockResolvedValue({ address: '10.0.0.8' }),
fetchFn: vi.fn(),
},
),
).rejects.toThrow(/private-network address/);
});
it('rejects voice model hosts when DNS safety lookup fails', async () => {
await expect(
transcribeVoiceAudio(
{ data: new Uint8Array([1, 2, 3]), mimeType: 'audio/wav' },
{
config: createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://asr.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash',
lookupHost: vi.fn().mockRejectedValue(new Error('NXDOMAIN')),
fetchFn: vi.fn(),
},
),
).rejects.toThrow(/DNS lookup failed for asr\.example/);
});
it('allows localhost voice URLs for development', () => {
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'Local ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'http://localhost:8080/v1',
},
]);
expect(
resolveVoiceTranscriptionConfig({
config,
settings: createSettings(),
voiceModel: 'qwen3-asr-flash',
}).baseUrl,
).toBe('http://localhost:8080/v1');
});
it('strips userinfo from voice base URLs', () => {
const config = createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl:
'https://user:[email protected]/compatible-mode/v1',
},
]);
expect(
resolveVoiceTranscriptionConfig({
config,
settings: createSettings({}, 'sk-primary'),
voiceModel: 'qwen3-asr-flash',
}).baseUrl,
).toBe('https://dashscope.aliyuncs.com/compatible-mode/v1');
});
it('keeps terse speech that happens to use a few keyterms', () => {
expect(
isKeytermEcho(
'commit schema endpoint async',
'commit schema endpoint async await api cli npm grep regex json refactor',
),
).toBe(false);
});
it('keeps longer speech that only mentions a small slice of keyterms', () => {
expect(
isKeytermEcho(
'commit schema endpoint async await api cli npm',
[
'commit',
'schema',
'endpoint',
'async',
'await',
'api',
'cli',
'npm',
'grep',
'regex',
'json',
'refactor',
'middleware',
'tokenizer',
'typescript',
'javascript',
'yaml',
'oauth',
'grpc',
'worktree',
'subagent',
'stdout',
'stderr',
'localhost',
'codebase',
'dotfiles',
'webhook',
'qwen',
'mcp',
].join(' '),
),
).toBe(false);
});
it('posts audio to chat/completions as input_audio content', async () => {
const fetchFn = vi.fn().mockResolvedValue({
ok: true,
json: vi.fn().mockResolvedValue({
choices: [{ message: { content: 'hello world' } }],
}),
});
const text = await transcribeVoiceAudio(
{ data: new Uint8Array([1, 2, 3]), mimeType: 'audio/wav' },
{
config: createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1/',
envKey: 'DASHSCOPE_API_KEY',
},
]),
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash',
lookupHost: lookupPublicHost,
fetchFn,
},
);
expect(text).toBe('hello world');
const [url, init] = fetchFn.mock.calls[0];
expect(url).toBe('https://dashscope.example/v1/chat/completions');
expect(init.method).toBe('POST');
expect((init.headers as Record<string, string>)['Authorization']).toBe(
'Bearer sk-test',
);
const body = JSON.parse(init.body as string);
expect(body.model).toBe('qwen3-asr-flash');
const userMsg = body.messages.find(
(m: { role: string }) => m.role === 'user',
);
expect(userMsg.content[0].type).toBe('input_audio');
expect(userMsg.content[0].input_audio.data).toMatch(
/^data:audio\/wav;base64,/,
);
expect(userMsg.content[0].input_audio.format).toBe('wav');
});
it('derives input_audio format from the recorder mime type', async () => {
const fetchFn = vi.fn().mockResolvedValue({
ok: true,
json: vi
.fn()
.mockResolvedValue({ choices: [{ message: { content: 'hello' } }] }),
});
await transcribeVoiceAudio(
{ data: new Uint8Array([1, 2, 3]), mimeType: 'audio/webm;codecs=opus' },
{
config: createConfig([
{
id: 'qwen3-asr-flash',
label: 'Custom ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://asr.example/v1',
},
]),
settings: createSettings(),
voiceModel: 'qwen3-asr-flash',
lookupHost: lookupPublicHost,
fetchFn,
},
);
const [, init] = fetchFn.mock.calls[0];
const body = JSON.parse(init.body as string);
const userMsg = body.messages.find(
(m: { role: string }) => m.role === 'user',
);
expect(userMsg.content[0].input_audio.data).toMatch(
/^data:audio\/webm;codecs=opus;base64,/,
);
expect(userMsg.content[0].input_audio.format).toBe('webm');
});
it('sends asr_options.language and a keyterms context message', async () => {
const fetchFn = vi.fn().mockResolvedValue({
ok: true,
json: vi
.fn()
.mockResolvedValue({ choices: [{ message: { content: 'hi' } }] }),
});
const config = {
getAllConfiguredModels: vi.fn().mockReturnValue([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
// Non-existent dir => getGitBranch fails fast (no real git subprocess).
getProjectRoot: vi.fn().mockReturnValue('/no/such/voice/project'),
} as unknown as Config;
const settings = {
merged: {
env: { DASHSCOPE_API_KEY: 'sk-test' },
security: { auth: {} },
general: { voice: { language: 'english' } },
},
} as unknown as LoadedSettings;
await transcribeVoiceAudio(
{ data: new Uint8Array([1, 2, 3]), mimeType: 'audio/wav' },
{
config,
settings,
voiceModel: 'qwen3-asr-flash',
lookupHost: lookupPublicHost,
fetchFn,
},
);
const body = JSON.parse(fetchFn.mock.calls[0][1].body as string);
expect(body.asr_options.language).toBe('en'); // english -> en
expect(body.asr_options.enable_itn).toBe(true);
const sys = body.messages.find(
(m: { role: string }) => m.role === 'system',
);
expect(sys.content[0].type).toBe('text');
expect((sys.content[0].text as string).length).toBeGreaterThan(0);
});
it('rejects audio over the size limit without calling the API', async () => {
const fetchFn = vi.fn();
await expect(
transcribeVoiceAudio(
{
data: new Uint8Array(10 * 1024 * 1024 + 1),
mimeType: 'audio/wav',
},
{
config: createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash',
lookupHost: lookupPublicHost,
fetchFn,
},
),
).rejects.toThrow(/too long/);
expect(fetchFn).not.toHaveBeenCalled();
});
it('redacts and truncates failed batch transcription responses', async () => {
const fetchFn = vi.fn().mockResolvedValue({
ok: false,
status: 500,
statusText: 'Internal Server Error',
text: vi
.fn()
.mockResolvedValue(
`Bearer sk-secret Invalid API key: sk-test ${'x'.repeat(500)}`,
),
});
let error: unknown;
try {
await transcribeVoiceAudio(
{ data: new Uint8Array([1, 2, 3]), mimeType: 'audio/wav' },
{
config: createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash',
lookupHost: lookupPublicHost,
fetchFn,
},
);
} catch (caught) {
error = caught;
}
expect(error).toBeInstanceOf(Error);
const message = (error as Error).message;
expect(message).toContain('Bearer [REDACTED]');
expect(message).not.toContain('sk-secret');
expect(message).not.toContain('sk-test');
expect(message).toMatch(/\.\.\.$/);
});
it('sends an inference timeout signal and reports timeout clearly', async () => {
let signal: AbortSignal | undefined;
const fetchFn = vi.fn((_url: RequestInfo | URL, init?: RequestInit) => {
signal = init?.signal ?? undefined;
return Promise.reject(new DOMException('TimeoutError', 'TimeoutError'));
});
await expect(
transcribeVoiceAudio(
{ data: new Uint8Array([1, 2, 3]), mimeType: 'audio/wav' },
{
config: createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash',
lookupHost: lookupPublicHost,
fetchFn,
},
),
).rejects.toThrow(
'Voice transcription timed out after 60s. Check ASR service health and retry.',
);
expect(signal).toBeInstanceOf(AbortSignal);
});
it('ignores legacy protocol settings and routes batch models by model id', async () => {
const fetchFn = vi.fn();
const settings = {
merged: {
env: { DASHSCOPE_API_KEY: 'sk-test' },
security: { auth: {} },
general: { voice: { protocol: 'dashscope-realtime' } },
},
} as unknown as LoadedSettings;
fetchFn.mockResolvedValue({
ok: true,
json: vi
.fn()
.mockResolvedValue({ choices: [{ message: { content: 'hi' } }] }),
});
await expect(
transcribeVoiceAudio(
{ data: new Uint8Array([1, 2, 3]), mimeType: 'audio/wav' },
{
config: createConfig([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
settings,
voiceModel: 'qwen3-asr-flash',
lookupHost: lookupPublicHost,
fetchFn,
},
),
).resolves.toBe('hi');
expect(fetchFn).toHaveBeenCalledTimes(1);
});
it('drops an echoed keyterm list instead of inserting it', async () => {
// What the model returns on non-speech audio: our bias terms verbatim.
const echoed =
'grep regex TypeScript JSON OAuth subagent worktree endpoint middleware schema';
const fetchFn = vi.fn().mockResolvedValue({
ok: true,
json: vi
.fn()
.mockResolvedValue({ choices: [{ message: { content: echoed } }] }),
});
const config = {
getAllConfiguredModels: vi.fn().mockReturnValue([
{
id: 'qwen3-asr-flash',
label: 'Qwen ASR',
authType: AuthType.USE_OPENAI,
baseUrl: 'https://dashscope.example/v1',
envKey: 'DASHSCOPE_API_KEY',
},
]),
getProjectRoot: vi.fn().mockReturnValue('/no/such/voice/project'),
} as unknown as Config;
const text = await transcribeVoiceAudio(
{ data: new Uint8Array([1, 2, 3]), mimeType: 'audio/wav' },
{
config,
settings: createSettings({ DASHSCOPE_API_KEY: 'sk-test' }),
voiceModel: 'qwen3-asr-flash',
lookupHost: lookupPublicHost,
fetchFn,
},
);
expect(text).toBe('');
});
});
packages/cli/src/ui/voice/voice-transcriber.ts
+551
View File
@@ -0,0 +1,551 @@
/**
* @license
* Copyright 2025 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
import process from 'node:process';
import { lookup as dnsLookup } from 'node:dns/promises';
import { isIP } from 'node:net';
import { createDebugLogger } from '@qwen-code/qwen-code-core';
import type { AvailableModel, Config } from '@qwen-code/qwen-code-core';
import type { LoadedSettings } from '../../config/settings.js';
import type { RecordedVoiceAudio } from '../hooks/use-voice-input.js';
import { buildVoiceKeyterms } from './voice-keyterms.js';
import type { VoiceStreamConfig } from './voice-stream-session.js';
import {
formatUnsupportedVoiceModelMessage,
isTranscribableVoiceModel,
resolveVoiceTransport,
} from './voice-model.js';
const DEFAULT_OPENAI_API_KEY = 'OPENAI_API_KEY';
const INFERENCE_TIMEOUT_MS = 60_000;
const MIN_KEYTERM_ECHO_TOKENS = 8;
const MIN_KEYTERM_SET_ECHO_RATIO = 0.3;
const debugLogger = createDebugLogger('VOICE_TRANSCRIBER');
export { resolveVoiceTransport };
export type { VoiceTransport } from './voice-model.js';
export type VoiceStreamingTransport =
| 'qwen-asr-realtime'
| 'dashscope-task-realtime';
type VoiceHostLookup = (
hostname: string,
) => Promise<{ address: string } | Array<{ address: string }>>;
function readVoiceLanguage(settings: LoadedSettings): string | undefined {
const language = (
settings.merged.general as { voice?: { language?: unknown } } | undefined
)?.voice?.language;
if (typeof language !== 'string') {
return undefined;
}
const trimmed = language.trim();
return trimmed.length > 0 ? trimmed : undefined;
}
export interface VoiceTranscriptionConfig {
model: string;
baseUrl: string;
apiKey?: string;
}
export interface ResolvedVoiceStreamConfig extends VoiceStreamConfig {
transport: VoiceStreamingTransport;
}
interface ResolveVoiceTranscriptionConfigArgs {
config: Config;
settings: LoadedSettings;
voiceModel: string;
}
interface TranscribeVoiceAudioArgs extends ResolveVoiceTranscriptionConfigArgs {
fetchFn?: typeof fetch;
lookupHost?: VoiceHostLookup;
}
function trimTrailingSlashes(value: string): string {
return value.replace(/\/+$/, '');
}
function readSettingsEnv(
settings: LoadedSettings,
envKey: string,
): string | undefined {
const env = settings.merged.env as Record<string, unknown> | undefined;
const value = env?.[envKey];
return typeof value === 'string' && value.trim().length > 0
? value.trim()
: undefined;
}
function isQwenBaseUrl(baseUrl: string): boolean {
try {
const hostname = new URL(baseUrl).hostname.toLowerCase();
return (
hostname === 'dashscope.aliyuncs.com' ||
hostname === 'dashscope-intl.aliyuncs.com' ||
hostname === 'dashscope-us.aliyuncs.com' ||
hostname.endsWith('.dashscope.aliyuncs.com') ||
hostname.endsWith('.dashscope-intl.aliyuncs.com') ||
hostname.endsWith('.dashscope-us.aliyuncs.com')
);
} catch {
return false;
}
}
function normalizeBaseUrl(baseUrl: string, modelName: string): string {
let url: URL;
try {
url = new URL(baseUrl);
} catch {
throw new Error(`Voice model '${modelName}' has an invalid baseUrl.`);
}
url.username = '';
url.password = '';
return trimTrailingSlashes(url.toString());
}
function normalizeHostname(hostname: string): string {
return hostname.toLowerCase().replace(/^\[|\]$/g, '');
}
function isLoopbackHost(hostname: string): boolean {
const host = normalizeHostname(hostname);
return host === 'localhost' || host === '127.0.0.1' || host === '::1';
}
// Blocks IP-literal private networks only. Hostname DNS resolution and
// rebinding protection require an async lookup or socket-level remoteAddress check.
function isPrivateNetworkIp(hostname: string): boolean {
const host = normalizeHostname(hostname);
if (isLoopbackHost(host)) {
return false;
}
const ipv4Mapped = host.match(/^::ffff:(\d+\.\d+\.\d+\.\d+)$/);
if (ipv4Mapped) {
return isPrivateNetworkIp(ipv4Mapped[1]!);
}
if (host.startsWith('::ffff:')) {
return true;
}
if (isIP(host) === 4) {
const [first = 0, second = 0] = host.split('.').map(Number);
return (
first === 0 ||
first === 10 ||
first === 127 ||
(first === 169 && second === 254) ||
(first === 172 && second >= 16 && second <= 31) ||
(first === 192 && second === 168) ||
(first === 100 && second >= 64 && second <= 127)
);
}
if (isIP(host) === 6) {
return (
host === '::' ||
host.startsWith('fe80:') ||
host.startsWith('fc') ||
host.startsWith('fd')
);
}
return false;
}
async function defaultLookupHost(
hostname: string,
): Promise<Array<{ address: string }>> {
return dnsLookup(hostname, { all: true });
}
export async function assertVoiceBaseUrlNetworkAllowed(
voiceConfig: VoiceTranscriptionConfig,
lookupHost?: VoiceHostLookup,
): Promise<void> {
const hostname = new URL(voiceConfig.baseUrl).hostname;
if (isLoopbackHost(hostname) || isIP(normalizeHostname(hostname)) !== 0) {
return;
}
let result: { address: string } | Array<{ address: string }>;
try {
result = await (lookupHost ?? defaultLookupHost)(hostname);
} catch {
throw new Error(
`Voice model '${voiceConfig.model}': DNS lookup failed for ${hostname}. Cannot verify network safety.`,
);
}
const records = Array.isArray(result) ? result : [result];
if (records.some((record) => isPrivateNetworkIp(record.address))) {
throw new Error(
`Voice model '${voiceConfig.model}' resolved to a private-network address.`,
);
}
}
function readApiKey(
settings: LoadedSettings,
model: AvailableModel,
baseUrl: string,
): string | undefined {
if (!model.envKey && !isQwenBaseUrl(baseUrl)) {
return undefined;
}
const envKey = model.envKey ?? DEFAULT_OPENAI_API_KEY;
const envValue = process.env[envKey];
if (envValue && envValue.trim().length > 0) {
return envValue.trim();
}
const settingsEnvValue = readSettingsEnv(settings, envKey);
if (settingsEnvValue) {
return settingsEnvValue;
}
if (!model.envKey && isQwenBaseUrl(baseUrl)) {
const authApiKey = settings.merged.security?.auth?.apiKey;
return typeof authApiKey === 'string' && authApiKey.trim().length > 0
? authApiKey.trim()
: undefined;
}
return undefined;
}
export function resolveVoiceTranscriptionConfig({
config,
settings,
voiceModel,
}: ResolveVoiceTranscriptionConfigArgs): VoiceTranscriptionConfig {
const matches = config
.getAllConfiguredModels()
.filter((model) => model.id === voiceModel);
if (matches.length === 0) {
throw new Error(
`Voice model '${voiceModel}' is not configured. Run /model --voice to choose a configured model.`,
);
}
if (matches.length > 1) {
throw new Error(`Voice model '${voiceModel}' is ambiguous.`);
}
const model = matches[0];
if (!isTranscribableVoiceModel(model)) {
throw new Error(formatUnsupportedVoiceModelMessage(voiceModel));
}
const baseUrl = model.baseUrl?.trim();
if (!baseUrl) {
throw new Error(`Voice model '${voiceModel}' does not define a baseUrl.`);
}
const normalizedBaseUrl = normalizeBaseUrl(baseUrl, voiceModel);
const parsedBaseUrl = new URL(normalizedBaseUrl);
const isLocalhost = isLoopbackHost(parsedBaseUrl.hostname);
if (parsedBaseUrl.protocol !== 'https:' && !isLocalhost) {
throw new Error(
`Voice model '${voiceModel}' must use an https baseUrl. Voice audio must not be transmitted in cleartext.`,
);
}
if (isPrivateNetworkIp(parsedBaseUrl.hostname)) {
throw new Error(
`Voice model '${voiceModel}' must not use a private-network baseUrl.`,
);
}
const apiKey = readApiKey(settings, model, normalizedBaseUrl);
if (model.envKey && !apiKey) {
throw new Error(`Voice model '${voiceModel}' requires ${model.envKey}.`);
}
return {
model: voiceModel,
baseUrl: normalizedBaseUrl,
...(apiKey ? { apiKey } : {}),
};
}
export function isStreamingVoiceModel(model: string): boolean {
const transport = resolveVoiceTransport(model);
return (
transport === 'qwen-asr-realtime' || transport === 'dashscope-task-realtime'
);
}
/** Build a streaming (WebSocket) config from the configured voice provider. */
export function resolveVoiceStreamConfig(
args: ResolveVoiceTranscriptionConfigArgs,
): ResolvedVoiceStreamConfig {
const base = resolveVoiceTranscriptionConfig(args);
const transport = resolveVoiceTransport(base.model);
if (
transport !== 'qwen-asr-realtime' &&
transport !== 'dashscope-task-realtime'
) {
throw new Error(
`Voice model '${base.model}' does not support streaming transcription.`,
);
}
const language = resolveLanguageCode(readVoiceLanguage(args.settings));
const keytermsContext =
transport === 'qwen-asr-realtime' ? buildKeytermsContext() : undefined;
return {
transport,
baseUrl: base.baseUrl,
model: base.model,
...(base.apiKey ? { apiKey: base.apiKey } : {}),
...(language ? { language } : {}),
...(keytermsContext ? { keytermsContext } : {}),
};
}
// Common spoken-language names → the codes Qwen-ASR's asr_options.language wants.
const LANGUAGE_CODES: Record<string, string> = {
english: 'en',
chinese: 'zh',
mandarin: 'zh',
cantonese: 'yue',
japanese: 'ja',
korean: 'ko',
french: 'fr',
german: 'de',
spanish: 'es',
italian: 'it',
portuguese: 'pt',
russian: 'ru',
arabic: 'ar',
};
function resolveLanguageCode(language: string | undefined): string | undefined {
if (!language) {
return undefined;
}
const lower = language.toLowerCase();
if (LANGUAGE_CODES[lower]) {
return LANGUAGE_CODES[lower];
}
// Already a short code (en / zh / yue). Unknown free text → let it auto-detect.
return /^[a-z]{2,3}$/.test(lower) ? lower : undefined;
}
function buildKeytermsContext(): string | undefined {
try {
const keyterms = buildVoiceKeyterms();
return keyterms.length > 0 ? keyterms.join(' ') : undefined;
} catch {
return undefined;
}
}
interface ChatCompletionResponse {
choices?: Array<{ message?: { content?: unknown } }>;
}
function tokenize(value: string): string[] {
return value
.toLowerCase()
.replace(/[^\p{L}\p{N}\s]/gu, ' ')
.split(/\s+/)
.filter(Boolean);
}
/**
* On non-speech audio (silence/noise) Qwen-ASR can hallucinate the keyterm
* context back as the transcript. Detect that — a multi-word result whose tokens
* are almost entirely keyterms — so the bias list never lands in the prompt.
* Short results are left alone so genuine terse utterances ("grep regex") pass.
*/
export function isKeytermEcho(
transcript: string,
keytermsContext?: string,
): boolean {
if (!keytermsContext) {
return false;
}
const tokens = tokenize(transcript);
if (tokens.length < 4) {
return false;
}
const keyset = new Set(tokenize(keytermsContext));
const overlap = tokens.filter((t) => keyset.has(t)).length;
const transcriptRatio = overlap / tokens.length;
const keytermRatio = overlap / keyset.size;
const isEcho =
overlap >= MIN_KEYTERM_ECHO_TOKENS &&
transcriptRatio >= 0.9 &&
keytermRatio >= MIN_KEYTERM_SET_ECHO_RATIO;
if (isEcho) {
debugLogger.debug(
`[voice] dropped likely keyterm echo: transcriptRatio=${transcriptRatio.toFixed(2)} keytermRatio=${keytermRatio.toFixed(2)} text="${transcript}"`,
);
}
return isEcho;
}
// Qwen-ASR caps each audio file at 10 MB / 5 minutes. Our 16 kHz mono 16-bit WAV
// is ~32 KB/s, so guard before encoding to give a clear error on overlong holds.
const MAX_AUDIO_BYTES = 10 * 1024 * 1024;
function escapeRegExp(value: string): string {
return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}
function sanitizeResponseDetails(raw: string, apiKey?: string): string {
let redacted = raw.replace(/Bearer\s+\S+/gi, 'Bearer [REDACTED]');
if (apiKey) {
redacted = redacted.replace(
new RegExp(escapeRegExp(apiKey), 'g'),
'[REDACTED]',
);
}
return redacted.length > 200 ? `${redacted.slice(0, 200)}...` : redacted;
}
function inputAudioFormat(mimeType: string): string {
return mimeType.split(';', 1)[0]?.replace(/^audio\//, '') || 'wav';
}
/**
* Transcribe via the DashScope/Qwen-ASR OpenAI-compatible protocol: the audio
* is sent as an `input_audio` chat message and the transcript comes back as the
* assistant message content. (DashScope does NOT serve the Whisper-style
* `/audio/transcriptions` endpoint — it 404s.) Keyterm biasing goes in a leading
* system message with structured content; language/itn go in `asr_options`.
*/
async function transcribeViaQwenAsr(
audio: RecordedVoiceAudio,
voiceConfig: VoiceTranscriptionConfig,
options: { language?: string; keytermsContext?: string },
fetchFn: typeof fetch,
): Promise<string> {
if (audio.data.byteLength > MAX_AUDIO_BYTES) {
throw new Error(
'Recording is too long for transcription (max ~5 minutes / 10 MB). Try a shorter dictation.',
);
}
const dataUrl = `data:${audio.mimeType};base64,${Buffer.from(audio.data).toString('base64')}`;
const messages: unknown[] = [];
if (options.keytermsContext) {
messages.push({
role: 'system',
content: [{ type: 'text', text: options.keytermsContext }],
});
}
messages.push({
role: 'user',
content: [
{
type: 'input_audio',
input_audio: {
data: dataUrl,
format: inputAudioFormat(audio.mimeType),
},
},
],
});
const asrOptions: Record<string, unknown> = { enable_itn: true };
if (options.language) {
asrOptions['language'] = options.language;
}
const headers: Record<string, string> = {
'Content-Type': 'application/json',
};
if (voiceConfig.apiKey) {
headers['Authorization'] = `Bearer ${voiceConfig.apiKey}`;
}
let response: Response;
try {
response = await fetchFn(
`${trimTrailingSlashes(voiceConfig.baseUrl)}/chat/completions`,
{
method: 'POST',
headers,
body: JSON.stringify({
model: voiceConfig.model,
messages,
asr_options: asrOptions,
}),
signal: AbortSignal.timeout(INFERENCE_TIMEOUT_MS),
},
);
} catch (error) {
if (error instanceof DOMException && error.name === 'TimeoutError') {
throw new Error(
`Voice transcription timed out after ${INFERENCE_TIMEOUT_MS / 1000}s. Check ASR service health and retry.`,
);
}
throw error;
}
if (!response.ok) {
let details = '';
try {
details = sanitizeResponseDetails(
await response.text(),
voiceConfig.apiKey,
);
} catch {
details = '';
}
if (/model_not_supported|unsupported model/i.test(details)) {
throw new Error(
'This voice model cannot be used for batch transcription. Use qwen3-asr-flash for batch or choose a realtime voice model such as qwen3-asr-flash-realtime / fun-asr-realtime / paraformer-realtime-v2.',
);
}
const suffix = details ? `: ${details}` : '';
throw new Error(
`Voice transcription request failed (${response.status} ${response.statusText})${suffix}`,
);
}
const json = (await response.json()) as ChatCompletionResponse;
const content = json.choices?.[0]?.message?.content;
if (typeof content !== 'string') {
throw new Error('Voice transcription response did not include text.');
}
const text = content.trim();
// Drop the result if the model just echoed our keyterm bias back (happens on
// non-speech audio) so the term list never gets inserted into the prompt.
if (isKeytermEcho(text, options.keytermsContext)) {
return '';
}
return text;
}
export async function transcribeVoiceAudio(
audio: RecordedVoiceAudio,
args: TranscribeVoiceAudioArgs,
): Promise<string> {
const voiceConfig = resolveVoiceTranscriptionConfig(args);
await assertVoiceBaseUrlNetworkAllowed(voiceConfig, args.lookupHost);
const fetchFn = args.fetchFn ?? fetch;
const language = resolveLanguageCode(readVoiceLanguage(args.settings));
const keytermsContext = buildKeytermsContext();
const transport = resolveVoiceTransport(voiceConfig.model);
switch (transport) {
case 'qwen-asr-chat':
return transcribeViaQwenAsr(
audio,
voiceConfig,
{ language, keytermsContext },
fetchFn,
);
case 'qwen-asr-realtime':
case 'dashscope-task-realtime':
throw new Error(
`Voice model '${voiceConfig.model}' requires streaming transcription.`,
);
case 'unsupported':
default:
throw new Error(
`Voice model '${voiceConfig.model}' is not a supported transcription model.`,
);
}
}
packages/cli/tsconfig.json
+4 -1
View File
@@ -10,7 +10,9 @@
"@qwen-code/qwen-code-core": ["../core/src/index.ts"],
"@qwen-code/qwen-code-core/*": ["../core/src/*"],
"@qwen-code/acp-bridge": ["../acp-bridge/src/index.ts"],
"@qwen-code/acp-bridge/*": ["../acp-bridge/src/*"]
"@qwen-code/acp-bridge/*": ["../acp-bridge/src/*"],
"@qwen-code/audio-capture": ["../audio-capture/src/index.ts"],
"@qwen-code/audio-capture/*": ["../audio-capture/src/*"]
}
},
"include": [
@@ -96,6 +98,7 @@
"references": [
{ "path": "../core" },
{ "path": "../acp-bridge" },
{ "path": "../audio-capture" },
{ "path": "../channels/base" },
{ "path": "../channels/telegram" },
{ "path": "../channels/weixin" },
packages/cli/vitest.config.ts
+4
View File
@@ -61,6 +61,10 @@ export default defineConfig({
__dirname,
'../acp-bridge/src/workspacePaths.ts',
),
'@qwen-code/audio-capture': path.resolve(
__dirname,
'../audio-capture/src/index.ts',
),
},
},
test: {
packages/vscode-ide-companion/schemas/settings.schema.json
+29
View File
@@ -44,6 +44,30 @@
"type": "boolean",
"default": false
},
"voice": {
"description": "Voice dictation settings.",
"type": "object",
"properties": {
"enabled": {
"description": "Enable voice dictation in the prompt input.",
"type": "boolean",
"default": false
},
"mode": {
"description": "How push-to-talk behaves: \"hold\" to talk while held, or \"tap\" to start and tap (or pause) to stop and submit. Options: hold, tap",
"enum": [
"hold",
"tap"
],
"default": "hold"
},
"language": {
"description": "Preferred spoken language for voice transcription (e.g. \"english\", \"chinese\"). Leave empty to auto-detect.",
"type": "string",
"default": ""
}
}
},
"enableAutoUpdate": {
"description": "Enable automatic update checks and installations on startup.",
"type": "boolean",
@@ -482,6 +506,11 @@
"type": "string",
"default": ""
},
"voiceModel": {
"description": "Model used for voice transcription. Set with /model --voice. Leave empty to keep voice dictation disabled until a voice model is selected.",
"type": "string",
"default": ""
},
"model": {
"description": "Settings related to the generative model.",
"type": "object",
scripts/build.js
+13 -10
View File
@@ -42,12 +42,13 @@ const cliOnly = process.argv.includes('--cli-only');
// 2. web-templates (embeddable web templates - used by cli)
// 3. channel-base (base channel infrastructure - used by channel adapters and cli)
// 4. channel adapters (depend on channel-base)
// 5. acp-bridge (depends on core - used by cli)
// 6. cli (depends on core, acp-bridge, web-templates, channel packages)
// 7. webui (shared UI components - used by vscode companion)
// 8. sdk (build-time devDep on acp-bridge for shared constants)
// 9. web-shell (depends on webui and sdk)
// 10. vscode-ide-companion (depends on webui)
// 5. audio-capture (native microphone backend used by cli)
// 6. acp-bridge (depends on core - used by cli)
// 7. cli (depends on core, acp-bridge, web-templates, channel packages)
// 8. webui (shared UI components - used by vscode companion)
// 9. sdk (build-time devDep on acp-bridge for shared constants)
// 10. web-shell (depends on webui and sdk)
// 11. vscode-ide-companion (depends on webui)
const buildOrder = [
'packages/core',
'packages/web-templates',
@@ -58,6 +59,7 @@ const buildOrder = [
'packages/channels/feishu',
'packages/channels/qqbot',
'packages/channels/plugin-example',
'packages/audio-capture',
'packages/acp-bridge',
'packages/cli',
...(cliOnly
@@ -71,10 +73,11 @@ const buildOrder = [
];
for (const workspace of buildOrder) {
execSync(`npm run build --workspace=${workspace}`, {
stdio: 'inherit',
cwd: root,
});
const command =
workspace === 'packages/audio-capture'
? `npm run build:ts --workspace=${workspace}`
: `npm run build --workspace=${workspace}`;
execSync(command, { stdio: 'inherit', cwd: root });
// After cli is built, generate the JSON Schema for settings
// so the vscode-ide-companion extension can provide IntelliSense
scripts/prepare-package.js
+1
View File
@@ -193,6 +193,7 @@ if (result.signal) {
config: rootPackageJson.config,
dependencies: {},
optionalDependencies: {
'@qwen-code/audio-capture': rootPackageJson.version,
'@lydell/node-pty': '1.2.0-beta.10',
'@lydell/node-pty-darwin-arm64': '1.2.0-beta.10',
'@lydell/node-pty-darwin-x64': '1.2.0-beta.10',
scripts/tests/package-assets.test.js
+6
View File
@@ -59,8 +59,14 @@ describe('package asset scripts', () => {
const distPackageJson = JSON.parse(
readFileSync(path.join(rootDir, 'dist', 'package.json'), 'utf8'),
);
const rootPackageJson = JSON.parse(
readFileSync(path.join(rootDir, 'package.json'), 'utf8'),
);
expect(distPackageJson.files).toContain('examples');
expect(distPackageJson.optionalDependencies).toMatchObject({
'@qwen-code/audio-capture': rootPackageJson.version,
});
expect(
existsSync(
path.join(rootDir, 'dist', 'examples', 'mcp-server', 'package.json'),